In this chapter we present several notions used throughout this thesis. In particular we formalize the risk minimization framework in which fall our algorithmic contributions. From a theoretical standpoint we present two frameworks interested in deriving generalization guarantees for risk minimization. As we will see in the next chapter these two frameworks have been successfully extended to the metric learning problem. We use them to theoretically analyse our contributions demonstrating the ability of our algorithms to learn metrics able to generalize well. From a more practical point of view we present several loss functions and regularization terms which can be used in the risk minimization framework and we introduce a formal definition of the notion of metric considered in this thesis. Finally we present the nearest neighbour classifier and the domain adaptation setting which will be used to empirically demonstrate the interest of most of our contributions.

4 Sometimes we also consider that we have access to a second non labelled training set from the source domain.

Introduction

I Background

Introduction

Machine Learning is a field of Artificial Intelligence aiming at acquiring new knowledge from data. This new knowledge generally takes the form of a model, learned from a limited number of observed examples, and able to generalize well to future queries. In other words the goal is to learn how to automatically solve a problem from a finite set of observations. For example, the objective in spam detection is to use the annotated mail box of a user to learn how to separate solicited emails from unsolicited ones; in tracking the underlying problem is to follow an object in a video; in face recognition the goal is to identify a person in a set of images. . . . The large diversity of problems raised in machine learning has attracted a lot of attention in the past and still deserves a lot of active research.

In this thesis we are mainly interested in Supervised Learning problems. The idea behind this paradigm is that the examples are accompanied by a label. This label can be either a value or a class and it corresponds to the solution of the problem for the given example. As illustrative examples, let us consider the problems of house pricing and poisonous mushrooms recognition. For the former the goal is to predict the price of a house, each example then corresponds to a set of characteristics of a particular building while the label is its price. For the latter we want to recognize from images poisonous mushrooms from edible ones, each example is the picture of a mushroom while the label is its class, i.e. poisonous or not. From these examples one can see the importance, in supervised learning, of generalization to new data. Indeed, the labels for the training examples are already known, if the model cannot find the correct labels for new examples then its interest becomes limited. Note that the previous examples correspond to two widely studied problems in supervised learning, namely regression and classification. The difference between the two is that the goal of the former is to predict a continuous value while the objective of the latter is to guess the correct class.

Supervised learning is not the sole paradigm existing in machine learning. In fact it can be opposed to Unsupervised Learning where examples are unlabelled. For example a widely studied problem in this paradigm is called clustering. The underlying goal is to obtain a meaningful partition of the space where the examples share common properties. The performance of unsupervised learning algorithms is often difficult to assess as, contrary to the supervised learning case, there is no labels to provide an obvious feedback on the model.

Drawing from these two paradigms, the idea behind Semi-Supervised Learning is to consider two sets of examples where the first one is labelled while the second one is not. In this case the goal is often to use the labelled examples to help solve an unsupervised learning task or to consider the unlabelled examples to aid in a supervised learning problem.

So far we have considered that the goal of a machine learning approach is to solve a single task. Taking a different point of view the idea behind Transfer Learning is to transfer some knowledge learned on a so-called source to a so-called target. Following this idea, in Domain Adaptation the goal is to transfer the model learned on a source task to solve a different but related target problem. For example, in the spam detection problem, the two tasks could be to detect unsolicited emails from the mailboxes of two different users. The two users share the same problem but their email distributions might differ, e.g. because they did not subscribe to the same mailing lists. In this case the goal is to adapt the model learned from one of the users to the other.

In this manuscript we will see that despite being mainly interested in supervised learning problems, several of our contributions also share some ties with the other paradigms presented here.

When presenting the supervised learning paradigm we stressed the fact that a model, learned with a limited number of training examples, should generalize well to new examples. One way to verify this property is to evaluate the learned model on a set of new test examples independent from training examples and for which the solution to the problem is known. However the number of test examples that can be obtained is often limited. It might make this approach insufficient to ensure that the model generalizes well. Other approaches are then necessary. To this extent note that a common assumption in machine learning is that the task that we want to solve is completely defined by an unknown distribution from which the training examples are drawn. Then, one possible solution is to use a cross-validation procedure where the idea is to partition the learning sample into k parts. The model is learned on k -1 parts and tested on the last one. This procedure is repeated k times, i.e. until each part was used as a test set, and the accuracy is averaged over the different test samples. Anyway, this procedure still requires a significant amount of examples to be valid. Another possibility following the assumption evoked previously consists in theoretically studying the learning algorithm in order to derive so-called generalization bounds. The idea behind these bounds is to show that the true error of the model, i.e. its error on the unknown distribution, is bounded by its empirical error, i.e. its error on the training sample, plus a term which decreases when the size of the training set increases. Obtaining such bounds guarantees that models learned by the concerned algorithm generalize reasonably well.

Many different approaches have been proposed to solve supervised learning problems. Among these several rely on a notion of distance or similarity between the examples to learn a model. A very representative example is the nearest neighbour classifier which is based on the idea that two similar examples should share the same label. Another example is the support vector machine algorithm. It proposes to classify examples depending on their similarity to landmarks points called the support vectors. In these two examples the notion of similarity is of critical importance. However different tasks often call for different measures of similarity. As an example recall the two examples used previously in this introduction, it does not make sense to compare houses and mushrooms in the same way. Manually choosing an appropriate measure of similarity can be tedious and difficult. However it might be possible to automatically infer it from the data. This is the underlying idea behind Metric Learning which is the field of interest of this thesis.

We identify several limits of the current approaches in metric learning. First some methods propose to make use of side informations to help during the learning process. However there is no theoretical understanding of the impact of such information on the learned metric. Second the intrinsic properties of the learned metrics are often the same. Indeed metrics are usually learned with the objective of bringing closer similar examples while pushing far away dissimilar ones. In some cases it might be interesting to consider different kinds of constraints. One example is to obtain a metric whose behaviour is not limited to the examples but is more global in the sense that it is, for example, able to move masses of examples together. A third limit to current approaches is that there is often no theoretical justifications on the proposed approaches, i.e. there is no guarantees on the generalization ability of the learned metrics.

Contributions: Learning Metrics with Controlled Behaviour

In this thesis we propose several approaches to learn metrics whose behaviour is controlled. First we propose to use side informations in the form of a reference metric to either strictly or loosely guide the learned metric. Hence in our first contribution we propose to address the problem of regressing the values of a reference metric only accessible through a limited training set. In our second contribution we theoretically study how using a reference metric coming from a related but different problem can help during the learning process. In particular we derive several measures of goodness of the reference metric for the problem at hand. Second we propose two approaches able to consider new kinds of constraints for metric learning. Hence in our third contribution we consider that the training examples should not be moved with respect to each other but rather with respect to some virtual points which lay in the output space of the learned metric. By this way it is possible to carefully control the movement of each example. In our fourth contribution we build upon our third contribution and on recent advances in Optimal Transport to propose a new approach to learn a metric able to move masses of examples across the space. As a last remark, note that throughout this thesis we put particular emphasis on providing theoretically sound approaches.

Outline

In the first part of this thesis we propose some preliminary informations. In the first chapter we introduce some concepts that are used throughout the manuscript while in the second chapter we propose a review of the state of the art in metric learning.

Chapter 1 The first chapter of this thesis is dedicated to the presentation of several notions and tools used throughout it. The first part of the chapter presents the risk minimization framework which is the basis of all our algorithmic contributions. The second part is dedicated to the theoretical analysis of algorithms. More precisely we present two frameworks used to derive generalization bounds. They correspond to the uniform stability and the Rademacher complexity frameworks. The third part is interested in the notion of losses and regularization terms. These elements are core in the formulation of a regularized risk minimization problem.

Through several examples we show that there exist a wide range of possibilities with different properties. This second part is also interested in the formal definition of the notion of metric as a general term to design a similarity, a dissimilarity or a distance. As for the losses and regularization terms, several examples are presented. The last part of this first chapter introduces other useful notions such as the nearest neighbour classifier, which we often use with our leaned metrics, and the domain adaptation setting in which two of our contributions are evaluated.

Chapter 2

The second chapter of this thesis corresponds to a review of metric learning.

Here we present the main approaches which have made the success of the field. We propose to divide this review in four parts answering four basic questions about metric leaning problems. In the first part we consider the problem of the kind of metrics which can be learned. Then in the second part we answer the question of how, technically, these metrics can be learned. In the third part of this chapter we review some approaches deriving theoretical guarantees for metric learning. In the last part we present several works interested in making use of metric learning for different kind of applications ranging from classification to clustering or domain adaptation.

In the second part of this thesis we present our first two contributions which are interested in using reference metrics to help during the metric learning process.

Chapter 3 In the third chapter of this thesis we present our first contribution. It corresponds to a metric learning method able to approximate an existing metric. The first part of this chapter is dedicated to the presentation of the main optimization problem considered. It corresponds to a regression of the values of a metric. Furthermore we show that when the reference metric is too complex it is possible to use local metric learning to obtain a better approximation. In the second part we present a theoretical analysis of the approach both in the global and the local settings. It shows that the metrics learned by our algorithm generalize well. In the third and fourth parts we consider the problem of learning perceptual color differences to show the interest of our approach in a real life application.

Chapter 4

The fourth chapter of this thesis is dedicated to our second contribution. As in the third chapter it corresponds to a metric learning approach able to use some knowledge given by an existing metric. The difference is that, this time, we do not want to approximate this metric but we want to use some information that it carries to help during the learning process. This contribution is thus strongly related to the field of transfer learning/domain adaptation. The chapter is divided in seven parts. In the first part we present the framework of Metric Hypothesis Transfer Learning which corresponds to a minimization problem equipped with a biased regularization term. In the second, third and fourth parts we propose a theoretical analysis of metric hypothesis transfer learning using three different theoretical frameworks. It allows us to derive different notions of goodness of the reference metric. In the fifth part of the chapter we summarize the different bounds and in the sixth part we present several loss functions and regularization terms which fall into our framework. In the last part we show that this framework can be used in practice to obtain competitive results on several widely used transfer learning problems.

In the last part of this thesis we introduce our last two contributions where we propose new ways to control the behaviour of the learned metric.

Chapter 5 In the fifth chapter we present our third contribution. Here instead of using standard similarity and dissimilarity constraints we propose to consider that the metric should bring the examples closer to virtual points defined a priori. It allows us to learn a metric with a regression and to reduce the number of constraints considered. In the first part of the chapter we present our algorithm. In the second part we address the problem of selecting the virtual points and defining the constraints. In the third part we propose a theoretical analysis of the algorithm showing that learning a metric with our approach is founded but also that it is possible to obtain some links with a standard metric learning method. In the last part we validate our approach with several experiments.

Chapter 6

The sixth chapter introduces the last contribution of this thesis. It corresponds to a new method able to learn a metric that moves masses of examples by approximating the transformation corresponding to the solution of an Optimal Transport problem. In the first part of this chapter we formally introduce the problem of optimal transport. In the second part we present our formulation while in the third we propose an efficient way to optimize it. In the fourth part we discuss some theoretical aspects showing that if standard assumptions made in the optimal transport community are correct, then our approach is founded. In the last part we empirically validate our approach on a domain adaptation and an image editing problem.

Notations

In this thesis R and R + respectively represent the sets of real and non negative real numbers. A vector is denoted by a bold lower case letter. For example x ∈ R d is a d-dimensional real valued column vector. For i ∈ {1, . . . , d}, x(i) is its i-th feature. In a similar fashion, matrices are denoted by a bold upper case letter. For examples M ∈ R d×d is a real valued matrix with d rows and d columns. We also denote by S the set of symmetric real valued Introduction matrices and by S + the set of symmetric positive semi-definite matrices. Let i ∈ {1, . . . , d} and j ∈ {1, . . . , d }, M(i, j) corresponds to the value at row i and column j while M(i, ) and M(, j) are respectively the row and column vectors of indices i and j. x T and M T stand for the transpose of vector x and matrix M. •, • represents the dot product between two vectors while •, • F corresponds to the Frobenius product1 between two matrices.

We are interested in supervised learning. Hence throughout this thesis we consider that we are working in a domain T corresponding to the space Z = X × Y equipped with a probability distribution D T . In this case X ⊆ R d is the example space while Y is the label space, e.g. Y = {-1, 1} in a binary classification problem. We consider that we have access to a set of n examples T = {z i = (x i , y i )} n i=1 with x i ∈ X and y i ∈ Y. The fact that the examples of the set T are drawn i.i.d. from the distribution D T is denoted as T ∼ D T . In matrix form we write T = (X, y) where X contains one example per row and y is a column vector of the labels.

We denote by [•] + the hinge loss function, i.e. [x] + = max(0, x), E X∼D T [X] corresponds to the expectation of the random variable X drawn from the distribution D T , Pr (E) denotes the probability of an event E and D T (x) corresponds to the probability of drawing x from distribution D T . We denote the composition of two functions as g •f , i.e. (g •f )(x) = g(f (x)).

All the notations are summarized in Table 1.

In the various mathematical proofs of this thesis we propose to explain the derivations step by step by adding a justification surrounded by brackets and flushed on the right between the concerned lines. Matrix X(i, ),X(, j) Row i and column j of matrix X X(i, j)

Entry in row i and column j of matrix X x(i) Entry i in vector x y Label X , Y, H, M Input Space, Output Space, Hypothesis Space, Space of Metrics In this chapter we are interested in supervised learning problems. We consider that we have access to a domain T which corresponds to the space Z = X ×Y equipped with the probability distribution D T . In this case X ⊆ R d is the example space and Y is the label space. The goal is to find the correct relation between the examples in X and the labels in Y. In other words we are looking for an hypothesis h : X → Y coming from an hypothesis space H and able to solve the problem of associating each example in X with the correct value in Y. A key assumption in machine learning is that the distribution D T is unknown and that we only have access to it through a finite size sample T = {z i = (x i , y i )} n i=1 . This sample of size n is assumed to be representative of the true distribution and is called the training set. One of the key objective is then to use T to learn an hypothesis h which generalizes well to new examples drawn from the distribution D T . To present this notion of generalization we start by introducing two notions of risk of an hypothesis. Before that we address the problem of assessing the performance of an hypothesis with respect to an example.

One of the most intuitive way to assess the performance of a model for a given problem is to measure its error. However the notion of error can vary from one problem to another. To illustrate this let us consider an example z = (x, y) and the two problems of classification and regression. In classification the label space Y is discrete and of limited size, e.g. in binary classification Y = {-1, 1}. The goal is to correctly choose the class of an object. Hence an error is the prediction of the wrong class, i.e. h(x) = y. In regression the label space Y is continuous, i.e. Y = R. The goal is to return the correct value given an example. Hence an error is the prediction of a value far from the ground truth, i.e. h(x) y or h(x) y. In this thesis we consider that the error, or the risk, is defined with respect to a loss function able to quantify it. More formally we consider that each problem is associated with a loss function l : H × Z → R + which, given an hypothesis h ∈ H and an example z = (x, y) ∼ D T is able to return a positive real value in R + . This value is a numerical representation of the error committed by the hypothesis on the example. It should be large if the error is significant and small otherwise. Considering the same examples as before we can define intuitive loss functions. On the one hand in classification we can consider the following loss, called the 0/1 loss and presented in Figure 1.2:

l (h, z) = 0 if h(x) = y, 1 otherwise. (1.1)
On the other hand in regression we can consider the following loss function, called the absolute loss and presented in Figure 1.3:

l (h, z) = |h(x) -y| . (1.2)
Note that we come back to this notion of loss function in Section 1.4 where we give a formal definition and several examples. We can now define the notions of empirical and true risk in the two following definitions.

Definition 1.1 (Empirical risk). Given a loss function l : H × Z → R + and a set of examples T , the empirical risk of an hypothesis h is defined as:

LT (h) = 1 n z∈T l (h, z) .
It corresponds to the average error of the hypothesis on the training set.

Definition 1.2 (True risk). Given a loss function l : H × Z → R + and a distribution D T , the true risk of an hypothesis h is defined as:

L T (h) = E z∼D T l (h, z) .
It corresponds to the expected error of the hypothesis on the whole distribution. The goal of a supervised learning algorithm is to learn an hypothesis with the smallest possible true risk. However this quantity is only theoretical and cannot be computed. Indeed in practice we do not have access to the distribution D T but only to the so called training set T which is assumed to be a good empirical approximation of D T . Hence we say that an algorithm generalizes well when the difference between the true risk and the empirical risk is small. In Section 1.3 we will see that even if this difference cannot be computed it can sometimes be upper bounded by a small value in a generalization bound.

We have seen that we have access to the empirical risk but not to the true risk. However we argued that the ideal hypothesis should have the smallest possible true risk. Hence in Section 1.2 we discuss the problem of empirically learning an hypothesis and we see that directly minimizing the empirical risk might not always be a good idea. In Section 1.3 we consider a theoretical standpoint and we address the problem of linking the empirical risk of an hypothesis to its true risk. In Sections 1.4 and 1.5 we give some formal definitions and examples of several notions that will be used throughout this thesis.

Learning by Risk Minimization

In this first section we address the problem of learning an hypothesis h when we only have access to a training set T and not to the whole distribution D T . We first propose to tackle the problem by simply minimizing the empirical risk. We will see that this Empirical Risk Minimization approach presents some drawbacks. We then turn our attention to two other frameworks, namely Structural Risk Minimization and Regularized Risk Minimization, which have been specifically designed to alleviate these drawbacks.

Empirical Risk Minimization (ERM)

The idea of ERM is to select the best hypothesis h ∈ H minimizing the empirical risk over the training set T . The objective is to solve the following optimization problem: arg min h∈H LT (h). (1.3) This approach allows us to learn an hypothesis with a small empirical error. However we have no information about its true risk. What can happen is that the hypothesis is very good on the training set but do not generalize well to unseen examples, i.e. it has a big true risk despite its small empirical risk. This is not a desirable property as we recall that our goal is to learn an hypothesis with a small true risk. The problem described above is called over-fitting. It often arises when the considered hypotheses are too complex for the problem at hand. Indeed they are more prone to noise fitting than simpler ones. To overcome this we can follow Occam's razor which says that among a set of hypotheses able to explain a phenomenon, choosing the simplest one is better. Hence the idea is to limit the complexity of the hypothesis class H. However we also have to be careful to not limit the hypothesis class too much as it may lead to a situation where the empirical risk blows up. This second issue is called under-fitting. To sum up, the best case Figure 1.1: Illustration of the bias-variance trade-off problem. On the left of the plot there is a high risk of over-fitting while one the right there is a high risk of under-fitting. scenario is to find the correct trade-off between a small empirical risk and a simple hypothesis. This trade-off is called the bias-variance trade-off and is illustrated in Figure 1. 1. In ERM the candidates are chosen uniformly from the space H. It implies that all the hypotheses are considered to have the same complexity. Hence, to avoid over-fitting, H must be chosen very carefully. However this space is often defined beforehand and does not depend on the data. It implies that one has to resort to a costly trial and error procedure to select H in a correct way. It makes ERM hard to use in practice and opened the door for new frameworks that we present below.

Structural Risk Minimization (SRM)

In ERM the bias-variance trade-off is hard to satisfy since hypotheses are chosen uniformly from H. In SRM instead of considering a single hypothesis space H, we consider an infinite number of hypothesis spaces of increasing complexity such that H 1 ⊆ H 2 , • • • . The idea is then to solve the following optimization problem: arg min h∈H i ,i∈N LT (h) + pen(H i ) (1.4) where pen(H i ) is a term penalizing the complexity of the hypothesis space H i . In this case we consider hypotheses by increasing complexity. We then select the one with the best trade-off between empirical risk and complexity. In other words, the goal is to minimize the empirical risk while avoiding over-fitting by selecting the simplest hypothesis.

Regularized Risk Minimization (RRM)

RRM takes the idea of ordering the hypothesis in terms of their complexity one step further and considers the complexity of each hypothesis individually. The idea is to consider a single, large hypothesis space H and to solve the following optimization problem:

arg min h∈H LT (h) + λ h (1.5)
where h is a measure of the complexity of the hypothesis h. Furthermore λ is an hyperparameter which controls the trade-off between low error and low complexity. Choosing the value of λ can be difficult in practice. However several heuristics have been proposed to cope with this problem. We can for example cite the leave-one-out or the cross-validation approaches.

Most of the algorithms presented in this thesis build upon the last framework presented in this section. As such in Section 1.4 we give a formal definition of a loss function and a regularization term. Furthermore we provide several examples which have been successfully used in the literature. Note that using a slight abuse of language we will often say that an optimization problem is an algorithm in itself since, in this thesis, we do not put much emphasis on the problem of developing efficient solvers. Implicitly we refer to an algorithm able to solve it. In the next section we propose to take a more theoretical point of view on the problem of learning an hypothesis with small true risk.

Deriving Generalization Guarantees

In the previous section we have presented empirical solutions to learn an hypothesis with small empirical risk and small true risk. In this section we consider a more theoretical point of view. We will show that, under some conditions on the algorithm used to learn the hypothesis, it is possible to derive what is called a generalization bound. The idea of these bounds is to show that the true risk of an hypothesis is upper bounded by its empirical risk plus a small quantity which usually depends on the complexity of the hypothesis and the number of examples in the training set. Furthermore these bounds build upon the PAC-Learning framework [START_REF] Leslie | A theory of the learnable[END_REF] and as such are probabilistic bounds which hold true almost everywhere. For δ > 0 they have the following form: .6) It means that deriving a probabilistic generalization bound boils down to showing that the probability that the true risk is upper bounded by the empirical risk plus a small quantity ε(n, H, δ) is greater than 1 -δ. This is illustrated in Figure 1.1. The key point is then to obtain the value of ε(H, n, δ). It can be seen as a measure of the generalization ability of the learned hypothesis. As stated before this quantity should be small and depends on three elements:

Pr L T (h) ≤ LT (h) + ε(n, H, δ) ≥ 1 -δ. ( 1
• • The hypothesis class H: to be more precise ε(H, n, δ) depends on the complexity of the hypothesis. As hinted by Occam's razor, the value of ε(H, n, δ) should increase if the hypothesis is more complex. Note that depending on the framework we either consider the complexity of the learned hypothesis h or the overall complexity of the hypothesis class H.

• The probability δ: these bounds are generally based on concentration inequalities such as McDiarmid's inequality (McDiarmid, 1989) or Bennett's inequality [START_REF] Bennett | Probability inequalities for the sum of independent random variables[END_REF]. They are probabilistic bounds which hold true with probability 1 -δ. The value of ε(H, n, δ) increases when δ decreases. Indeed, if we want the bound to hold everywhere we have to take more particular cases into account which loosen the result.

Several frameworks have been proposed to derive generalization bounds. The main differences between these different frameworks is the concentration inequality considered and how they handle the complexity of the hypothesis class. In this thesis we consider two frameworks to derive generalization bounds, namely the uniform stability framework [START_REF] Bousquet | Stability and generalization[END_REF] and the Rademacher complexity framework [START_REF] Peter | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF]. These two frameworks are presented below. Note that there is several other possible approaches that we will not detail here as they are less relevant to this thesis. We can for example cite the uniform convergence framework, the VC-dimension framework [START_REF] Vapnik | On the uniform convergence of relative frequencies of events to their probabilities[END_REF]; [START_REF] Vapnik | Estimation of Dependences Based on Empirical Data[END_REF] or the algorithmic robustness framework (Xu andMannor, 2010, 2012). Also see [START_REF] Boucheron | Concentration inequalities[END_REF] for a survey on concentration inequalities and [START_REF] Langford | Tutorial on practical prediction theory for classification[END_REF] for a general tutorial on prediction theory.

Uniform Stability

We first present the Uniform Stability framework introduced by [START_REF] Bousquet | Stability and generalization[END_REF]. This framework is applicable to any algorithm which is uniformly stable, i.e. which respects the following definition: (Bousquet and Elisseeff, 2002b, Definition 6)). Let T ∼ D T be a training set of size n and z ∼ D T be any example. Let T i be the training set obtained by replacing example i in T by z. Let A be an algorithm which returns hypothesis h T when learning with the training set T and h T i when learning with the training set T i . An algorithm A has uniform stability β with respect to its loss function l if the following holds: ∀i ∈ {1, . . . , n} , sup

Definition 1.3 (Uniform Stability
z ∼D T l h T , z -l h T i , z ≤ β n . (1.7)
The idea is to say that an algorithm is uniformly stable if under small changes in the training set, the difference in the errors of the learned hypotheses is bounded. Furthermore the term on the r.h.s. should decrease as the number of examples increases. Note that this property should hold for all size n training sets and only the hypotheses effectively learned by the algorithm are considered, i.e. if some hypotheses in H are never learned by the algorithm under any training set then these hypotheses do not impact the result. As a consequence this framework has the nice property to focus on hypotheses that will really be learned by the considered algorithm. The value of β usually depends on the loss function and the regularization term. In Chapters 3, 4 and 5 we use this framework and show that the proposed algorithms are uniformly stable. Note that in their paper [START_REF] Bousquet | Stability and generalization[END_REF] consider several definitions for the notion of stability. However they show that the notion of uniform stability is the strongest one and that it implies all the others.

Using the McDiarmid's inequality (Theorem A.1) it is possible to show that a β-uniformly stable algorithm generalizes well: Theorem 1.1 (Generalization bound (Bousquet and Elisseeff, 2002b, Theorem 12)). Let A be an algorithm with uniform stability β with respect to a bounded loss function 0 ≤ l ≤ B, for all z ∈ Z and all sets T . Then given a randomly drawn sample T and given that h T is the solution given by A, for any n ≥ 1, and any δ ∈ (0, 1), the following bound holds with probability at least 1 -δ:

L T (h T ) ≤ LT (h T ) + β n + (2β + B) ln( 1 δ ) 2n
.

(1.8)

The bounds derived using this framework converge in O 1

√

n . Note that this rate is standard for generalization bounds.

This framework has been shown to be applicable to a wide range of algorithms such as Support Vector Machines or Regularized Least Square Regression [START_REF] Bousquet | Stability and generalization[END_REF]. Due to the proof techniques used to derive the bound, the main limitation of this framework lays in the kind of regularization terms that it can handle. For example Xu et al. (2012) have shown that algorithms based on sparsity inducing regularization terms are not stable.

Rademacher Complexity

We now switch our attention to the Rademacher Complexity framework introduced by [START_REF] Peter | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF]. This framework is based on the notion of Rademacher complexity defined as follows: Definition 1.4 (Rademacher Complexity (Shalev-Shwartz and Ben-David, 2014b, Equation (26.4))). Let T ∼ D T and let F be a function space such that f : X → R. Let σ be a vector of n Rademacher Variables, i.e. variables which can take a value of either 1 or -1 with probability 1 2 . The empirical Rademacher complexity is defined as follows:

Rn (F) = E σ sup f ∈F 1 n n i=1 σ i f (x i ) x 1 , • • • , x n .
(1.9)

The Rademacher complexity is then defined as:

R(F) = E T ∼D T Rn (F) (1.10)
where the expectation is taken over size n sets.

The idea of empirical Rademacher complexity is to measure the capacity of the functions in F at fitting random noise. This noise is generated by the Rademacher variables and all the possibilities are considered through the expectation. Note that the supremum is considered over all the possible functions in F. It implies that instead of considering only the hypotheses learned by the algorithm as for the uniform stability framework, the Rademacher complexity framework considers the complexity of the whole hypothesis class, i.e. some hypotheses which are never learned by the considered algorithm might impact the bound. Furthermore Rademacher complexity is defined in expectation with respect to all the training sets of size n and is not specific to the training set T considered during the learning process.

Using the McDiarmid's inequality (Theorem A.1) it is possible to derive a generalization bound based on the Rademacher complexity.

Theorem 1.2 (Generalization bound (Shalev-Shwartz and Ben-David, 2014b, Theorem 26.5)).

Let A be an algorithm with Rademacher Complexity R(F) with respect to a bounded loss function 0 ≤ l ≤ B. Note that F = {f = l • h} with h ∈ H. Then, for any n ≥ 1, any δ ∈ (0, 1) and any h ∈ H, the following bound holds with probability at least 1 -δ over the random draw of the sample T : .11) On the one hand this bound holds for any hypothesis h ∈ H and for a wide range of regularization terms including several sparsity inducing ones. On the other hand the uniform stability based bound only holds for the hypothesis learned by the algorithm and for a limited number of regularization terms. The price paid by the Rademacher complexity framework to obtain such a behaviour is the convergence of the complexity related term. Indeed in the uniform stability framework this term was decreasing in O 1 n while in the Rademacher complexity framework it can often be shown that R(F) ≤ O 1 √ n . Note that overall both bounds converge in O 1 √ n due to the probabilistic term. We make use of the Rademacher complexity framework in Chapter 4.

L T (h) ≤ LT (h) + 2R(F) + B 2 ln( 2 δ ) n . ( 1
In this section we presented two frameworks that can be used to derive generalization bounds. In the next section we formally define several notions used throughout this thesis. These include loss functions, regularization terms and metrics. These notions are accompanied by several illustrating examples.

Loss Functions, Regularization Terms and Metrics

The performance of regularized risk minimization primarily depends on the chosen loss function and regularization term. In this section we present a formal definition of these notions along with some examples. We also introduce the notion of metric as a way to compare learning examples. This is often a key component of machine learning methods.

Loss Functions and Regularization Terms

Loss functions and regularization terms are fundamental building blocks of regularized risk minimization algorithms. We start by presenting the formal definitions of what we consider as a loss function and a regularization term. Note that both of these definitions exhibit the property of Hypothesis Ordering. This is key to select the best hypothesis, i.e. an hypothesis with low error (with respect to the loss function) and as simple as possible (with respect to the regularization term). Hence given a training set, changing the loss or the regularization can lead one to learn different hypotheses. After the formal definitions we present several examples used by state of the art approaches.

Loss Function: Definition Definition 1.5 (Loss function). Let T be a domain corresponding to the space Z equipped with the probability distribution D T . Let H be an hypothesis space of candidates able to give a solution to the problem associated with the domain T . A loss function is any function l : H × Z → R + such that:

1. ∀h ∈ H, ∀z ∈ Z, l (h, z) ≥ 0 (Non-negativity), 2. ∀h 1 , h 2 ∈ H, ∀z ∈ Z, l (h 1 , z) ≤ l (h 2 , z) implies that h 1 gives a better prediction than h 2
on example z (Hypothesis ordering).

A loss function can take its values in [0, u] rather than R + . It is then said to be upper bounded or bounded.

Loss Function: Examples

Depending on the problem at hand different loss functions should be used. Here we propose to consider two different problems which have already been introduced before, namely classification and regression. We first propose several examples of loss functions mainly used in classification. These are depicted in Figure 1.2. 0/1 loss This loss is probably the most intuitive one as the idea is simply to count the number of errors of the hypothesis. This loss returns 1 if the hypothesis makes an incorrect prediction and 0 otherwise:

∀h ∈ H, ∀z ∈ Z, l (h, z) = 0 if h(x) = y, 1 otherwise.
(1.12)

The main drawback of this loss is that it is not convex and not differentiable everywhere. As such an optimization problem based on it is hard to solve and thus this loss is not used in practice. One solution is to use a surrogate loss. The idea is to upper-bound the 0/1 loss with a convex function which is easier to include in an optimization problem. To present several examples of surrogate loss functions we start by defining ε ∈ R as the degree of agreement between the prediction h(x) and the ground truth y. The value of ε mainly depends on the confidence of the prediction, see Figure 1.2.

Hinge loss It is defined as follows:

∀h ∈ H, ∀z ∈ Z, l (h, z) = [1 -ε] + = max {0, 1 -ε} (1.13)
This loss has, for example, been successfully used in [START_REF] Cortes | Support-vector networks[END_REF]. Logistic loss It is defined as follows:

∀h ∈ H, ∀z ∈ Z, l (h, z) = log(1 + exp(-ε)) log(2) (1.14)
This loss has, for example, been successfully used in [START_REF] Friedman | Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)[END_REF].

Exponential loss It is defined as follows:

∀h ∈ H, ∀z ∈ Z, l (h, z) = exp(-ε) (1.15)
This loss has, for example, been successfully used in [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF].

We also propose some loss functions which can be used in regression. These are depicted in Figure 1.3. Note that in this case the degree of agreement ε between the prediction and the ground truth is defined as the residual, i.e. ε = h(x) -y.

Square loss It is defined as follows:

∀h ∈ H, ∀z ∈ Z, l (h, z) = ε 2 (1.16)
This loss has, for example, been successfully used in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF].

Chapter 1. Preliminaries

Absolute loss It is defined as follows:

∀h ∈ H, ∀z ∈ Z, l (h, z) = |ε| (1.17)

A survey on the use of this loss can be found in Dielman (2005). Note that this loss is not differentiable everywhere and as such can be harder to use in an optimization problem.

Huber loss It is parametrized by δ and defined as follows:

∀h ∈ H, ∀z ∈ Z, l (h, z) = 1 2 ε 2 if |ε| ≤ δ, δ |ε| -1 2 δ otherwise.
( 1.18) This loss has been proposed by [START_REF] Peter | Robust estimation of a location parameter[END_REF] and has been designed to be more robust to outliers than the square loss while still being differentiable everywhere.

We now turn our attention to regularization terms.

Regularization Term: Definition Definition 1.6 (Regularization term). Let H be an hypothesis space. A regularization term is any function

• : H → R + such that: 1. ∀h ∈ H, h ≥ 0 (Non-negativity), 2. ∀h 1 , h 2 ∈ H, h 1 ≤ h 2 implies
that h 1 is less complex than h 2 (Hypothesis ordering).

A regularization term can take its values in [0, u] rather than R + . It is then said to be upper bounded or bounded. As our notation suggests, most of the time we choose the regularization term as a norm over the hypothesis space. Definition 1.7 (Norm). Let X ⊆ R d be a d-dimensional vector space. A norm is any function

• : X → R + such that: 1. ∀x ∈ X , x ≥ 0 (Non-negativity), 2. ∀x ∈ X , x = 0 ⇔ x = 0 where 0 is the zero vector (Separate points), 3. ∀x ∈ X , ∀a ∈ R, ax ≤ |a| x (Absolute homogeneity), 4. ∀x, x ∈ X , x + x ≤ x + x (Triangle inequality).
For the sake of simplicity we presented the definition of a norm with respect to a vector space. However it can be easily extended to the notion of metric or hypothesis as long as the different properties are respected. Furthermore a norm can take its values in [0, u] rather than R + . It is then said to be upper bounded or bounded.

Regularization Term: Examples

As stated before regularization terms are often defined as norms. This is for example true when we consider that the hypothesis space is a vector space (as in linear classification or regression). Hence we now propose several examples of norms.

p norms The p norms are parametrised by a value p: .19) For particular values of p we retrieve some well known norms depicted in Figure 1.4.

∀x ∈ R d , x p = d i=1 |x i | p 1 p . ( 1
• 1 norm: If p = 1 it corresponds to the 1 norm. The 1 norm has been widely used for its sparsity inducing properties [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. However, it is not differentiable everywhere and thus is harder to use in practice.

• 2 norm: If p = 2 it corresponds to the 2 norm. The 2 norm is strongly convex and differentiable everywhere. As such it has been used in many practical applications [START_REF] Cortes | Support-vector networks[END_REF]. It tends to penalize large values.

• Max norm or ∞ norm: If p = ∞ it corresponds to max norm or ∞ norm.
Note that for p < 1 this norm is not convex and thus is hard to use in an optimization problem.

p,q norm The p,q norm is a generalization to matrices of the vectors p norm. The idea is to apply a p norm on each row of the matrix and then to apply the q norm on the vector composed of the values obtained for each row:

M p,q = ( M(1, ) p , • • • , M(d, ) p ) q (1.20)
When p = q = 2 we retrieve the Frobenius norm which is a natural extension to the matrix case of the 2 norm. Note that it is also possible to use this norm for vectors by separating the different features into several groups. For example, the 2,1 norm has been used to induce sparsity constraints on groups of features [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF].

Schatten p norms The Schatten p norms are norms obtained by applying the p norms to σ the vector of singular values of the matrix: .21) If p = 2 it corresponds to the Frobenius norm. If p = ∞ its the spectral norm and if p = 1 it corresponds to the nuclear or trace norm. The latter norm has been used for its capacity in producing low rank matrices [START_REF] Wang | On the schatten norm for matrix based subspace learning and classification[END_REF].

M p = σ p . ( 1
We have presented a formal definition of loss functions and regularization terms along with several examples. We now turn our interest to the notion of metric.

Metrics

As mentioned before metrics are often a key component of machine learning algorithms as a way to compare examples. Before switching to the formal definition of what we consider as a metric we cite several well known algorithms which heavily rely on this notion:

• k-Nearest Neighbours [START_REF] Cover | Nearest neighbor pattern classification[END_REF] • Support Vector Machines [START_REF] Cortes | Support-vector networks[END_REF]: The idea behind this classification algorithm is to assume that there exist an high dimensional space in which the problem is linearly separable. This space is induced by a kernel which is a kind of metric.

• k-Means [START_REF] Stuart | Least squares quantization in PCM[END_REF]: The goal of this clustering algorithm is to partition the space into k regions whose members share a similar meaning. To achieve this, the idea is to randomly select k centres and to associate each example to its closest centre. The centres are then updated and the algorithm proceeds iteratively until convergence. The notion of closeness is controlled by a metric.

In this thesis we consider as a metric any similarity or dissimilarity which respect Definition 1.8. It includes but is not limited to the notion of Distance, Definition 1.9, and the notion of Kernel, Definition 1.10.

Metrics: Definitions

We start by presenting the general notion of similarity and dissimilarity that we consider in this thesis1 . Definition 1.8 ((Dis)Similarity). Let X ⊆ R d be a d-dimensional vector space. A (dis)similarity is any pairwise function k : X × X → R. We say that a (dis)similarity is symmetric if

∀x, x ∈ X , k(x, x ) = k(x , x).
A similarity should return a large positive value when two examples are similar and a large negative value otherwise. Conversely a dissimilarity should return a large negative value when two examples are similar and a large negative value otherwise. A (dis)similarity is said to be lower bounded, respectively upper bounded, if instead of taking its values in R it takes its values in an interval [l, +∞[, respectively ]-∞, u], such that -∞ < l, u < +∞. When a (dis)similarity is lower and upper bounded, with l ≤ u, we simply say that it is bounded.

A particular kind of lower bounded dissimilarity is a distance.

Definition 1.9 (Distance). Let X ⊆ R d be a d-dimensional vector space. A distance is a lower bounded dissimilarity function d : X × X → R + such that: 1. ∀x, x ∈ X , d(x, x ) ≥ 0 (Non-negativity), 2. ∀x, x ∈ X , d(x, x ) = 0 ⇔ x = x (Identity of indiscernible), 3. ∀x, x ∈ X , d(x, x ) = d(x , x) (Symmetry), 4. ∀x, x , x ∈ X , d(x, x ) ≤ d(x, x ) + d(x , x ) (Triangle inequality).
As in the case of (dis)similarities, a distance can take its values in [0, u] and is then said to be upper bounded or bounded. Note the similarities between the definition of a distance and the definition of a norm. These two notions are closely related:

• Given a norm, the function x, x → xx is a distance.

• Given a distance, if it further respects the two following properties:

1. ∀x, x , x ∈ X , d(x + x , x + x ) = d(x, x ), 2. ∀x, x ∈ X , ∀a ∈ R, d(tx, tx ) = |t| d(x, x ) then the function x → d(x, 0) is a norm.
The notion of kernel is a particular kind of similarity. Definition 1.10 (Kernel). Let X ⊆ R d be a d-dimensional vector space and K be an Hilbert space. A symmetric similarity function k(•) is a kernel if there exists a function φ : X → K such that: ∀x, x ∈ X , k x, x = φ(x) , φ x .

(1.22)

Equivalently, k(•) is a kernel if it is positive semi-definite: ∀x 1 , • • • , x n ∈ X , ∀c 1 , • • • , c n ∈ R, n i=1 n j=1 c i c j k(x i , x j ) ≥ 0. (1.23)
Note that K can be very high dimensional or even infinite. In this case φ(•) is intractable. However it is still possible to compute the value of the kernel through the expression of k(•). This is called the kernel trick2 . This trick has for example been successfully used in [START_REF] Cortes | Support-vector networks[END_REF]; Schölkopf et al. (1997).

Metrics: Examples

We give several examples of well-known metrics.

Minkowski distances

The Minkowski distances is a family of distances induced by the p norms and as such parametrised by a value p:

d x, x = d i=1 x i -x i p 1 p = x -x p .
(1.24)

For particular values of p we retrieve some well known distances.

• Manhattan distance: If p = 1 it corresponds to the Manhattan distance induced by the 1 norm.

• Euclidean distance: If p = 2 it corresponds to the Euclidean distance induced by the 2 norm.

• Chebyshev distance: If p = ∞ it corresponds to the Chebyshev distance induced by the ∞ norm or max norm.

Mahalanobis distances

The Mahalanobis distances is a family of distances parametrised by a matrix M such that: .25) To obtain a proper distance, the matrix M has to be positive definite. If the matrix is positive semi-definite (PSD)3 then it is a pseudo distance, i.e. the constraint on the identity of indiscernibles is relaxed and it is only required that ∀x ∈ X , d(x, x) = 0. Note that if M = I the identity matrix, it corresponds to the Euclidean distance. In its original definition [START_REF] Chandra | On the generalized distance in statistics[END_REF] the Mahalanobis distance was using the inverse variance-covariance matrix of the examples, i.e. M = Σ -1 . The intuition behind the Mahalanobis distance is to reweight the features of the examples. As such using a Cholesky decomposition such that M = L T L, one can see that the Mahalanobis distance corresponds to the Euclidean distance in a space linearly dependent on X .

d x, x = (x -x ) T M(x -x ). ( 1

Bilinear similarities

The bilinear similarities is a family of similarities parametrised by a matrix M and which is strongly related to the dot product:

k x, x = x, Mx = M T x, x (1.26)
If M = I it corresponds to the dot product in the original space. Similarly if M = 1 x 2 x 2 I it corresponds to the cosine similarity. While in general there is no constraints on M, we can choose it to be positive semi-definite and, using a Cholesky decomposition, one can see that it corresponds to the dot product in a new space linearly dependent on X .

In Chapter 2 we propose a review of several metric learning methods whose goal is to learn the parameters of M, either for the Mahalanobis distance or the Bilinear similarity.

Kernels Kernels are defined with respect to a function k(•) and sometimes it is possible to explicitly compute the feature map φ(•):

• Linear kernel: It corresponds to the dot product in the original space ∀x, x ∈ X , k x, x = x T x = x, x .

(1.27)

• Polynomial kernel: It is parametrized by its order p and a bias c .28) It is possible to compute the feature map explicitly. For example, for two dimensional vectors and p = 2 each example is implicitly mapped to a 6 dimensional vector: .29) Note that small values of p are often preferred for this kernel as it becomes numerically unstable when p tends to infinity.

∀x, x ∈ X , k x, x = x T x + c p . ( 1
φ(x) = x 2 1 x 2 2 √ 2x 1 x 2 √ 2cx 1 √ 2cx 2 c T . ( 1
• Gaussian kernel: It is parametrized by its width σ and the feature map is infinite dimensional and thus intractable. .30) When using the Gaussian kernel if one has access to a training set T of n examples a standard heuristic is to set σ to the mean of all the euclidean distances between the examples [START_REF] Kar | Similarity-based learning via data driven embeddings[END_REF].

∀x, x ∈ X , k x, x = exp - x -x 2 2 2σ 2 . ( 1

Other Notions

In this section we propose to consider other notions that will be used throughout the thesis but do not fall in any of the previous sections. Hence we present the nearest neighbour classifier and the domain adaptation setting. We use the former in our experiments as the classifier making use of our metrics while we evaluate two of our contributions in the latter.

Nearest Neighbours Classifier

As mentioned in the introduction metric learning algorithms are often used as a preprocessing step to improve the performance of another algorithm. In this thesis we propose to consider the nearest neighbour classifier [START_REF] Cover | Nearest neighbor pattern classification[END_REF] as this subsequent approach. It is probably one of the most intuitive method in classification. The idea stems from the saying birds of a feather flock together, i.e. if two examples are close to each other they probably share the same label. To formally present this approach we consider that we want to classify z ∼ D T using the training set T and a measure of closeness between the examples under the form of a distance d.

1 nearest neighbour (1-NN) The idea behind the 1-NN classifier is to predict the class ŷ for z as ŷ = y i where z i is the closest example of z in T , i.e. the example z i satisfying: .31) k nearest neighbours (k-NN) The idea behind the k-NN classifier is that instead of only considering the closest example of z as for the 1-NN, one can select the k closest examples and set ŷ as the majority class among these k nearest neighbours.

z i = arg min z ∈T d z, z . ( 1
The nearest neighbours algorithm is illustrated in Figure 1.5. Note that here we considered that the closeness between the examples is determined by a distance. However this algorithm can be used with any metric and, in particular, learned ones.

Domain Adaptation Setting

Throughout this chapter we considered a supervised learning setting. If most of our contributions in this thesis fall into this first setting, we will also show that two of them are well suited to solve domain adaptation problems (See Chapters 4 and 6). In this subsection we propose to quickly introduce this setting. Domain adaptation is a special case of transfer learning [START_REF] Pan | A survey on transfer learning[END_REF] where the goal is to adapt a model learned on a source domain to a target domain. Formally we consider that we have access to two domains, the source domain S defined as the space Z s = X s ×Y s equipped with the probability distribution D S and the target domain T defined as the space Z t = X t × Y t equipped with the probability distribution D T . These two domains are considered to be different but related. Hence the tasks associated with the two domains are the same but there is a sort of shift between the two distributions. For the adaptation to be possible we further assume that this shift is not prohibitively large. As an illustrative example we consider the Office-Caltech dataset [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF] which is used as a benchmark in the domain adaptation community. The task consists in classifying the images of 10 kind of objects. The shift comes from the fact that the pictures come from 4 different domains:

• Amazon: the objects are presented on a white background,

• DSLR: the pictures are taken in an office environment with a high-end camera,

• Webcam: the pictures are taken in an office environment with a low resolution webcam,

• Caltech: the pictures of the objects come from the Caltech256 dataset [START_REF] Griffin | Caltech-256 object category dataset[END_REF].

It defines 12 different tasks where the domain are paired and alternatively used as the source and the target.

In domain adaptation we consider that we have access to two training sets. The first one, S, is labelled and comes from the source domain 4 while the second one, T , comes from the target domain. We then consider two different settings:

• unsupervised domain adaptation: there is no supervision in T ,

• semi-supervised domain adaptation: only a small amount of examples are labelled in T .

When solving a domain adaptation problem the goal is to estimate and overcome the shift between the distributions [START_REF] Shai Ben-David | A theory of learning from different domains[END_REF]. To this extent several different strategies have been proposed in the literature. Among these we can cite for example the reweighting approaches where the idea is to put more emphasis on the source examples which are mixed with the target examples in the input space (See e.g. [START_REF] Mansour | Domain adaptation: Learning bounds and algorithms[END_REF]). We can also cite iterative approaches where the idea is to learn a classifier on the source domain, label the target domain and replace some examples in the source domain by the newly labelled target examples. This process is repeated several times until a convergence criterion is met (See e.g. [START_REF] Bruzzone | Domain adaptation problems: A DASVM classification technique and a circular validation strategy[END_REF]). As a last example another strategy consists in learning a common representation space for the source and the target where the shift between the two domains does not exist (See e.g. [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF]; [START_REF] Hoffman | Efficient learning of domain-invariant image representations[END_REF]). This last strategy is often the motivating idea behind the metric learning methods interested in addressing the domain adaptation problem (See Section 2.5).

Conclusion

In this chapter we presented several fundamental notions used throughout this thesis. We started by introducing the risk minimization framework which will be used, in its regularized form, in all of our contributions. Then we addressed the problem of deriving generalization bounds. We presented two frameworks respectively based on the uniform stability principle and the notion of Rademacher complexity. Next we proposed a formal definition and some well known examples of the notion of loss function and regularization term. We also clarified the notion of metric as we consider it in this thesis. Lastly we introduced the nearest neighbour algorithm and the domain adaptation setting which will be used to assess the performance of several of our contributions.

In the next chapter we propose a non exhaustive review of the field of metric learning by answering four fundamental questions on the problem.

Introduction

As mentioned before the idea behind metric learning is to automatically learn, from the data, a metric adapted to the task at hand. This chapter is a non exhaustive review of this field as we put the focus on the most relevant methods for this thesis. We propose to explore the different existing approaches by answering four basic questions on metric learning.

• What kind of metrics is it possible to learn? We will see that most of the methods are interested in learning either a Mahalanobis distance or a bilinear similarity. The most common approaches to include some non linearity in the process are to learn multiple metrics across the space or to learn a linear metric in a kernel induced space.

• How are the metrics effectively learned? We will see that optimization problems in batch learning setting are widely used in metric learning and that approaches mainly varies in function of the kind of constraints used, the loss function considered and the regularization term. Nevertheless several approaches also proposed to learn a metric in an online fashion.

• Are there any theoretical guarantees on the learned metrics? We presented two frameworks used to derive generalization bounds in Chapter 1. We will see that these can be extended to the metric learning setting. Furthermore it is sometimes possible to evaluate the impact of a metric on the subsequent algorithm. • In which tasks are the metrics used? We will see that many approaches are interested in solving classification or semi-supervised clustering problems. However some works also considered different tasks such as image retrieval, face recognition or domain adaptation.

In Chapter 1 we were mainly focused on supervised learning problems. In this chapter, and unless stated otherwise, we consider the setting consisting in learning a metric for a classification problem. Formally we consider a domain T which corresponds to the space Z = X × Y equipped with the probability distribution D T . We further consider that X ⊆ R d , i.e. we are working with real valued vectors, and that we only have access to a training set

T = {z i = (x i , y i )} n i=1 of n training examples.
In Figure 2.1 we propose a diagram summarizing the outline of this review. In Section 2.2 we present several works whose goal is to learn a specific metric. Next in Section 2.3 we review different learning procedures applicable to metric learning. In Section 2.4 we consider the problem of deriving theoretical guarantees. Finally in Section 2.5 we focus on the applications making use of the learned metrics before concluding in Section 2.6

Metrics

Several metrics have been considered in the field. We present a short description of the most popular ones here.

Mahalanobis Distance

In their pioneering work [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF] propose to learn the parameter matrix M of a Mahalanobis distance. Popularized by Large Margin Nearest Neighbour (LMNN) [START_REF] Kilian Q Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF] and Information Theoretic Metric Learning (ITML) [START_REF] Davis | Informationtheoretic metric learning[END_REF], it is probably the most studied metric in the community. We presented it in Section 1.4 but we recall it here for the sake of readability:

d M x, x = (x -x ) T M(x -x ) with x, x ∈ X , M ∈ S d×d + . (2.1)
Note that to avoid the difficulties linked to the use of a square root, i.e. it is concave and only defined on R + , a lot of approaches have focused on learning the quadratic version of d M , denoted d 2 M . To obtain a proper distance the matrix M has to be positive semi-definite. This constraint can be hard to satisfy in practice as it often requires some costly projections on the positive semi-definite cone 1 [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF]. However this constraints also provides a nice interpretation of the metric. Indeed using a Cholesky decomposition one can write M = L T L with L ∈ R d ×d . It implies that the Mahalanobis distance is the standard euclidean distance in a new space linearly dependent on X :

d L x, x = (Lx -Lx ) T (Lx -Lx ) with x, x ∈ X , L ∈ R d ×d .
(

Following this idea and to avoid the positive semi-definite constraint on M some approaches propose to directly learn the matrix L [START_REF] Goldberger | Neighbourhood components analysis[END_REF].

Another appealing property of the matrix M stemming from its positive semi-definiteness is that it can be written as a combination of rank 1 matrices:

M = k i=1 u i u T i (2.3)
with the u i ∈ R d are linearly independent vectors 2 and k is the rank of M. Using this property several approaches propose to learn either a weighted combination of given rank 1 matrices [START_REF] Shi | Sparse compositional metric learning[END_REF] or the matrices themselves [START_REF] Shen | Positive semidefinite metric learning with boosting[END_REF][START_REF] Shen | Positive semidefinite metric learning using boosting-like algorithms[END_REF].

Depending on the form of the matrix M the Mahalanobis distance can have some appealing properties. For example if this matrix is diagonal the distance can be seen as a reweigthing of the input features [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF]. Similarly when M is low rank (k < d) then in the decomposition presented above the matrix L is rectangular with d < d. It implies that the examples are projected in a lower dimensional space, i.e. it is equivalent to performing some dimensionality reduction on the data. Following this idea some approaches have then been interested in learning low rank matrices using some sparsity inducing norms such as the 1 A projection onto the positive semi-definite cone requires an eigenvalue decomposition whose computational cost is roughly in O d 3 making it intractable when d becomes large.

2 These vectors can for example be the eigenvectors of the matrix times the square root of the corresponding eigenvalue.

trace norm [START_REF] Ying | Sparse metric learning via smooth optimization[END_REF], the capped trace norm [START_REF] Huo | Robust and effective metric learning using capped trace norm: Metric learning via capped trace norm[END_REF] or a Fantope based norm [START_REF] Marc T Law | Fantope regularization in metric learning[END_REF].

Bilinear Similarity

Apart from the Mahalanobis distance, the second most popular metric is probably the bilinear similarity which is also parametrized by a matrix M. For example [START_REF] Mustafa Qamar | Similarity learning for nearest neighbor classification[END_REF] consider the following similarity:

k M x, x = x T Mx N (x, x ) with x, x ∈ X , M ∈ R d×d (2.4)
where M can be either diagonal, symmetric or simply a square matrix and N (x, x ) is a normalization parameter. Following this idea [START_REF] Mustafa | Online and batch learning of generalized cosine similarities[END_REF] propose to use a generalized cosine similarity:

k M x, x = x T Mx √ x T Mx √ x T Mx with x, x ∈ X , M ∈ S d×d + . (2.5)
However the positive semi-definite constraint can be too restrictive in practice. These two similarities can be seen as particular forms of the bilinear similarity presented in Section 1.4 and recalled here for the sake of readability: .6) This more general form has for example been used in [START_REF] Chechik | Large scale online learning of image similarity through ranking[END_REF][START_REF] Chechik | Large scale online learning of image similarity through ranking[END_REF]; [START_REF] Kulis | What you saw is not what you get: Domain adaptation using asymmetric kernel transforms[END_REF]; [START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF].

k M x, x = x T Mx with x, x ∈ X , M ∈ R d×d + . ( 2

Multiple Metrics

The two metrics presented above are linearly dependent on the input space. However it is sometimes not sufficient to capture the idiosyncrasies of the data. Hence learning a non linear metric becomes necessary. One possible approach is then to learn multiple linear metrics across the space. One basic strategy is local metric learning which consists in dividing the input space in K clusters C 1 , . . . , C K and to learn one metric in each cluster. For example one can obtain K Mahalanobis distances d M 1 , . . . , d M K . The distance can then be computed between two examples x, x ∈ X as:

d M x, x = K i=1 w x,x (i)d M i x, x (2.7)
where w x,x (i) is the weight of the distance d M i in the combination when considering the two examples x and x . Hence the methods interested in learning multiple metrics mainly vary on the way they cluster the input space, the metric learned and the way they choose the weights of the metrics with respect to the examples.

• In Multi-Metric Large Margin Nearest Neighbour (MM-LMNN) [START_REF] Kilian | Fast solvers and efficient implementations for distance metric learning[END_REF] propose to either set one partition for each class or to use the k-means algorithm.

In each partition they propose to use LMNN (Weinberger et al., 2005) to learn a Mahalanobis distance. Finally the distance d M (x, x ) between two examples only depends on the cluster in which x falls. It implies that the global distance is not symmetric if x and x do not fall in the same cluster.

• [START_REF] Semerci | Mixtures of large margin nearest neighbor classifiers[END_REF] propose to learn a Mixture of LMNN (MoLMNN) by alternatively learning the partition of the space and the transformation matrix of a Mahalanobis distance. Furthermore they propose to use a soft partitioning of the space where the transformation of one example depends on several local transformations.

• In Large Margin Local Metric Learning (LMLML) [START_REF] Bohné | Large margin local metric learning[END_REF] propose to learn one Mahalanobis distance for each of the K components of a Gaussian mixture model.

For two examples x and x the weight of each metric depends on the degree of membership of the two examples to each component.

• In Parametric Local Metric Learning (PLML) [START_REF] Wang | Parametric local metric learning for nearest neighbor classification[END_REF] propose to select anchor points defined as the means of clusters constructed by the k-means algorithm. Then they express each example in the training set as a weighted combination of the anchor points and they use these weights to learn one basis metric for each anchor point. Note that the global metric is not symmetric.

• Instead of partitioning the input space Chang andYeung (2004, 2007) propose to learn one linear transformation for each example but to compute their effective transformations as a learned weighted combination of the transformations of their neighbours.

Other Non Linear Metrics

To include some non linearity in the model some approaches propose to consider intrinsically non linear metrics. For example [START_REF] Kedem | Non-linear metric learning[END_REF] proposed to build upon LMNN [START_REF] Kilian Q Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF] Another possible approach to include some non linearity, used for example in [START_REF] Davis | Informationtheoretic metric learning[END_REF], is to consider learning a linear metrics in a space non linearly dependent on the input space using for example a kernel.

In this thesis we consider the problem of learning Mahalanobis distances in Chapters 4, 5 and 6. In Chapter 4 we also consider learning bilinear similarities while in Chapter 3 we propose to learn multiple Mahalanobis distances.

Learning Approaches

A classification problem becomes easier to solve when, in the data, the intra class variance is low and the inter class variance is high. In other words it is desirable to have all the examples of the same class close to each other and all the examples of different classes far from each other. Building upon this idea most of the existing works in metric learning try to learn the best metric such that the aforementioned constraints are respected.

Pair Based Constraints

Given a labelled training set T , a first approach consists in considering the examples by pairs and defining similarity and dissimilarity constraints as follows:

• A set of pairs of similar examples:

P sim = {(z, z ) s.t. z, z ∈ T, y = y },
• A set of pairs of dissimilar examples:

P dis = {(z, z ) s.t. z, z ∈ T, y = y }.
Alternatively we can consider a single set of pairs of examples:

• P pair = (z, z , δ yy ) s.t. z, z ∈ T, δ yy = 1 if y = y , δ yy = -1 if y = y .
A good metric should be able to bring closer to each other all the similar examples while pushing far away all the dissimilar ones. For example, using empirical risk minimization (Section 1. In practice we often fix a value γ sim when y = y and a value γ dis when y = y . Many approaches in metric learning are based on a similar idea, i.e. they try to obtain a metric that approximates these constraints. However the optimization problem presented here is non convex and non differentiable and is thus hard to optimize in practice. Most of the approaches then consider surrogate losses (Section 1.4) which are easier to handle. They also make use of various regularization terms to enforce different properties on the metrics.

• In their pioneering work [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF]

M∈S d×d + (x,x )∈P sim d 2 M x, x (2.8)
s.t.

(x,x )∈P dis d M x, x ≥ 1.
(2.9)

Here the margin between similar examples is implicitly set to 0 while the margin between dissimilar examples is set to 1.

• [START_REF] Goldberger | Neighbourhood components analysis[END_REF] proposed Neighbourhood Components Analysis (NCA). It is a method based on a non convex optimization problem where the idea is to directly learn the transformation matrix L of a Mahalanobis distance. To this extent they first propose to define for each example in the training set T the probability that an example x j is in the neighbourhood of an example x i as: .10) From this, assuming that p ii = 0, they compute the probability that the example x i is correctly classified as: .11) They then try to find the metric which maximizes the probability of correctly classifying the examples: arg max .12) To solve this optimization problem, the authors propose to use a gradient based approach and precise that some care should be taken to avoid local maxima. In this work the authors consider that all the examples of the same class should be similar while all the examples of different classes should be dissimilar. There is no explicit notion of margin.

p ij = exp -d 2 L (x i , x j ) z k ∈T z k =z i exp -d 2 L (x i , x k ) . ( 2
p i = z j ∈T y i =y j p ij . ( 2
L∈R d ×d z i ∈T p i . ( 2
• [START_REF] Globerson | Metric learning by collapsing classes[END_REF] proposed Maximally Collapsing Metric Learning (MCML) where the idea is to learn a Mahalanobis distance able to collapse all the similar examples in a single point and to push the dissimilar examples infinitely far away. To this extent the authors propose a convex optimization problem based on the Kullback-Leibler divergence. As in NCA [START_REF] Goldberger | Neighbourhood components analysis[END_REF] they first propose to consider for each example in the training set T the probability that an example x j is in the neighbourhood of an example x i as: .13) They also define the ideal probability that they want to achieve between two examples as: .14) Following this the authors propose to learn a metric minimizing the Kullback-Leibler divergence between the empirical and the ideal probability distributions: arg min .15) To solve this convex optimization problem the authors propose a gradient based approach with projections onto the constraints.

p ij = exp -d 2 M (x i , x j ) z k ∈T z k =z i exp -d 2 M (x i , x k ) . ( 2
p * ij ∝ 1 y i = y j 0 y i = y j . ( 2
M∈S d×d + KL p ij p * ij . ( 2
• Information-Theoretic Metric Learning (ITML) [START_REF] Davis | Informationtheoretic metric learning[END_REF] is among the most famous Mahalanobis distance learning approaches. It is based on the log det divergence and the idea is to learn a metric which is close to a known prior metric M S using the following optimization problem: arg min

M∈S d×d + Tr MM -1 S -log det MM -1 S -n (2.16) s.t. Tr M(x -x )(x -x ) T ≤ γ sim (x, x ) ∈ P sim (2.17) Tr M(x -x )(x -x ) T ≥ γ dis (x, x ) ∈ P dis . (2.18)
The log det divergence is a particular Bregman divergence with the nice property that if the divergence is finite and the prior matrix is positive semi-definite then the learned matrix is also guaranteed to be positive semi-definite. It implies that this optimization problem does not require projections on the semi-definite cone.

• [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF] propose to learn a Mahalanobis distance using the following optimization problem:

arg min

M∈S d×d + ,Tr(M)≤η(d) 2 n(n -1) z i ,z j ∈T i<j δ y i y j 1 -d 2 M (x i , x j ) + + λ 1 2 M 2 F (2.19)
where η(d) is shown to be sublinear in d, i.e. η(d) ∼ O (d p ) with p < 1.

• Log-determinant regularized Distance Metric Learning (L-DML) [START_REF] Zha | Robust distance metric learning with auxiliary knowledge[END_REF]) is a Mahalanobis distance learning method which is able to make use of some auxiliary knowledge in the form of given metrics. The idea is to use a variant of ITML [START_REF] Davis | Informationtheoretic metric learning[END_REF] to accommodate several prior metrics rather than a single one. Hence they consider that they have access to a set of prior matrices

M 1 , • • • , M k ∈ S d×d +
and use the following optimization problem: arg min

MS d×d + ,µ≥0 k i=1 µ i Tr M -1 i M -log det(M) + λ sim (z,z )∈P sim d M x, x -λ dis (z,z )∈P dis d M x, x + λ µ µ 2 2 s.t. k i=1 µ i = 1
where the vector µ controls the impact of each prior matrix and the λ parameters control the trade-off between the different terms.

Triplet Based Constraints

Sometimes pair based constraints are not sufficient to capture the relationships between the constraints. Another common trend in metric learning is to consider triplet based constraints:

• P tri = {(z, z , z ) s.t. z, z , z ∈ T, y = y , y = y }.
Using empirical risk minimization (Section 1.2), learning a Mahalanobis distance could be done by solving the following optimization problem: arg min

M∈S d×d + (z,z ,vz )∈P tri 1 d M (x,x )≥d M (x,x )+γ + λ M (2.20)
where γ is the desired margin between the two distances and 1 is the indicator function.

In practice this margin is often set to 1 (Weinberger et al., 2005;[START_REF] Ying | Sparse metric learning via smooth optimization[END_REF][START_REF] Shi | Sparse compositional metric learning[END_REF]. The underlying idea is that similar examples should be closer to each other than dissimilar ones (See Figure 2.2(b)). Once again many approaches in metric learning are based on a similar idea and make use of a variation of the previous optimization problem.

M∈S d×d + λ (x,x )∈P sim d 2 M x, x + (1 -λ) (x,x ,x )∈P tri 1 + d 2 M x, x -d 2 M x, x + (2.21)
where λ is a parameter controlling the balance between the term which brings closer the target examples and the term which moves the impostors with respect to the target examples. Satisfying the positive semi-definiteness of M is costly and general solvers do not handle this optimization problem efficiently. Hence the authors propose a gradient based specific solver which makes use of the possible decomposition M = L T L. Several metric learning approaches are based on the same formulation albeit in different contexts such as local metric learning [START_REF] Kilian | Fast solvers and efficient implementations for distance metric learning[END_REF][START_REF] Semerci | Mixtures of large margin nearest neighbor classifiers[END_REF] or learning intrinsically non linear metrics [START_REF] Kedem | Non-linear metric learning[END_REF].

• [START_REF] Ying | Sparse metric learning via smooth optimization[END_REF] propose to learn a low rank Mahalanobis distance. The idea is to use the trace norm as a regularization term in the following optimization problem: arg min

M∈S d×d + (z,z ,z )∈P tri 1 + d 2 M x, x -d 2 M x, x + + λTr (M) 2 . (2.22)
As seen in Section 1.4 the trace norm corresponds to the sum of the eigenvalues of the matrix. It implies that to obtain a small trace norm the eigenvalues should be minimized and thus go to 0, i.e. the matrix becomes low rank.

• Sparse Compositional Metric Learning (SCML) [START_REF] Shi | Sparse compositional metric learning[END_REF]) is a Mahalanobis distance learning method based on the idea that any positive semi-definite matrix can be decomposed as a set of rank 1 positive semi-definite matrices. To learn a metric the authors consider that they have access to a set of rank 1 matrices, the bases

B = b i b T i s.t. b i ∈ R d m i=1
. This set can, for example, be obtained thanks to a Fisher discriminant analysis. The goal is then to learn the vector w which combines the bases. In the general case the authors propose to solve the following optimization problem:

arg min w≥0 1 n (z,z ,z )∈P tri 1 + d 2 M x, x -d 2 M x, x + + λ w 1 (2.23)
where M = m i=1 w(i)b i b T i . The regularization term tends to promote sparse combination vectors in order to minimize the number of bases needed to compute the matrix.

Note that instead of learning the d 2 parameters of the matrix M, this method only requires to learn the sparse vector w of size m. Hence it greatly reduces the computational cost since m will, most of the time, be smaller than d 2 .

Quadruplet Based Constraints

Introduced by [START_REF] Marc T Law | Quadruplet-wise image similarity learning[END_REF] in Quadruplet-wise Metric Learning (Qwise) the underlying idea is that in some particular cases pair or triplet based constraints are not sufficient. As a motivating example they propose the problem of smiling faces. They consider 4 examples ordered as follows z ≺ z ∼ z ≺ z , i.e. z is not smiling at all, z is smiling a lot and z and z both smile a little. In this case pair or triplet based constraints cannot completely capture the relations between the examples since they are not fully determined, e.g. it is unknown if z is closer to z or to z . To solve this problem [START_REF] Marc T Law | Quadruplet-wise image similarity learning[END_REF] propose to use quadruplet based constraints of the form:

P quad = (z, z , z , z ) s.t.
z and z are more similar to each other than z and z .

These constraints can also be extended to take into account a margin γ: P quad = (z, z , z , z , γ) s.t. z and z are more similar than z and z by a margin γ .

Quadruplet based constraints can accommodate the motivating example by considering that z and z should be closer to each other than z and z should. These constraints have been used in a several approaches.

• Law et al. (2014) introduce a Fantope regularization for Mahalanobis distance learning.

One of the limits of the trace norm regularization is the fact that it tends to reduce all the eigenvalues of the matrix. However reducing the values of the highest eigenvalues does not reduce the rank and might decrease the performance of the metric. Hence [START_REF] Marc T Law | Fantope regularization in metric learning[END_REF] propose to consider a regularization of the form Tr (WM) where W is in the convex hull of the set of rank k projection matrices, called a Fantope. This matrix can be built by first computing the eigenvalue decomposition M = V T ΣV with Σ a diagonal matrix containing the eigenvalues and V containing the eigenvectors and then by setting W = V T Σ V where Σ is obtained from Σ by replacing the k smallest eigenvalues by 1 and the others by 0. In other words, the idea is to consider that only the k smallest eigenvalues should be minimized. Using this idea the authors propose to solve the following optimization problem: arg min .24) Note that one of the difficulties when optimizing this kind of problem is that W depends on the current value of M and as such it should be updated during the optimization process. Hence the authors propose to consider a sub gradient descent approach and to update W at each iteration.

M∈S d×d + (z,z ,z ,z ,γ)∈P quad γ + d 2 M x, x -d 2 M x , x + + λTr (WM) . ( 2
• [START_REF] Huo | Robust and effective metric learning using capped trace norm: Metric learning via capped trace norm[END_REF] propose a Capped Trace norm regularization term for Mahalanobis metric learning. This regularization term can be written as 1 2 i min(σ(i), C) where σ is the vector of singular values of M and C is a constant threshold. The idea is to limit the impact of the highest singular values in the optimization problem and thus to promote the minimization of the smallest singular values. The authors propose to use the following optimization problem: arg min

M∈S d×d + (z,z ,z ,z ,γ)∈P quad γ + d 2 M x, x -d 2 M x , x + + λ 2 i min(σ i , C). (2.25)
In its original form the proposed optimization problem is not convex but the authors tackle this issue by solving an equivalent convex optimization formulation. The Capped Trace norm regularization is close in spirit to the Fantope regularization [START_REF] Marc T Law | Fantope regularization in metric learning[END_REF]. However the authors show that it is less sensitive to hyper parameters.

Online Learning

In

d M t+1 = f d Mt , (x t+1 , x t+1 ) (2.26)
where f is a function able to combine the current metric with the new pair to learn a new metric.

• Pseudo-Metric Online Learning Algorithm (POLA) [START_REF] Shalev-Shwartz | Online and batch learning of pseudometrics[END_REF]

M t+ 1 2 , b t+ 1 2 = arg min M∈R d×d ,b∈R M t -M 2 F + (b t -b) 2 s.t. δ y t+1 y t+1 d 2 Mt x t+1 , x t+1 -b t + 1 + = 0, M t+1 , b t+1 = arg min M∈S d×d + ,b≥1 M t+ 1 2 -M 2 F + (b t+ 1 2 -b) 2 .
(2.27)

The first step consists in searching for the (matrix, threshold)-pair achieving a loss of 0 on the new pair of examples while staying as close as possible to the current solution.

The second step consists in projecting the new solution onto the set of admissible solutions, i.e. positive semi-definite matrix and threshold greater than 1. Note that a kernel version of this approach is also proposed allowing one to learn non-linear metrics in an online fashion.

• LogDet Exact Gradient Optimization (LEGO) [START_REF] Jain | Online metric learning and fast similarity search[END_REF]) is a Mahalanobis distance learning approach. For a new pair of examples (x t+1 , x t+1 , δ y t+1 y t+1 ) they propose to use a formulation based on ITML [START_REF] Davis | Informationtheoretic metric learning[END_REF] where they set the prior matrix M S to the current matrix M t at each iteration. Hence they obtain the matrix M t+1 by solving the following optimization problem:

M t+1 = arg min M∈S d×d + Tr MM -1 t -log det MM -1 t -d + λ δ y t+1 y t+1 d M x t+1 , x t+1 -γ y t+1 y t+1 + .
• [START_REF] Chechik | Large scale online learning of image similarity through ranking[END_REF][START_REF] Chechik | Large scale online learning of image similarity through ranking[END_REF] proposed Online Algorithm for Scalable Image Similarity (OASIS) a bilinear similarity learning approach specifically designed to handle large datasets of images. In this work the authors work with triplet based constraints and given a new triplet (x t+1 , x t+1 , x t+1 ) they propose to update the metric in the following way:

M t+1 = arg min M∈R d×d 1 2 M -M t 2 F + λ 1 -k Mt x t+1 , x t+1 + k Mt x t+1 , x t+1 + . (2.28)
The idea is to update the matrix for each new triplet while staying close to the matrix obtained during the previous iteration. This trade-off is controlled by a parameter λ.

The initial matrix is selected as the identity matrix M 0 = I. The matrix M can be either unconstrained, symmetric or PSD depending on the problem. This method has been shown to be computationally efficient thanks to the specifically developed solver.

Other Approaches

Other approaches than the one presented above have been considered to learn metrics. For example, [START_REF] Shen | Positive semidefinite metric learning with boosting[END_REF][START_REF] Shen | Positive semidefinite metric learning using boosting-like algorithms[END_REF] propose to use the theory of boosting to learn a Mahalanobis distance. The main idea is to notice that any positive semi definite matrix can be decomposed as a combination of rank 1 matrices which can be used and combined as weak learners. In a subsequent work [START_REF] Bi | Adaboost on low-rank psd matrices for metric learning[END_REF] propose a substantial seep-up of the approach. Another approach was proposed by [START_REF] Mustafa Qamar | Similarity learning for nearest neighbor classification[END_REF] who learn a similarity by using a variant of the voted perceptron algorithm [START_REF] Freund | Large margin classification using the perceptron algorithm[END_REF].

In this thesis we are mainly interested in batch optimization problems based on regularized risk minimization. In Chapters 3 and 4 we consider pair based constraints. In Chapters 5 and 6 we propose two approaches which are not based on standard metric learning constraints as they are able to consider each example individually.

Theoretical Guarantees

Metric learning is most of the time used as a preprocessing step before other algorithms. As such when considering theoretical guarantees for metric learning two questions may arise. On the one hand studying the generalization ability of the metric is crucial to ensure that distances computed between new examples will be correct. On the other hand considering the impact of the metric on the subsequent algorithm is important as it is a way to theoretically show the interest of learning it.

Generalization Bounds for Metric Learning

Generalization bounds for metric learning are harder to derive than in standard approaches. Indeed one of the common assumptions when proving this kind of guarantees is that the examples are drawn i.i.d. from a probability distribution (See Section 1.3). However in metric learning most of the time the loss functions are defined with respect to pairs or triplets of examples as presented above. One of the issue is then that even if the examples are drawn i.i.d. from D T there is no guarantee that this is also the case for the pairs or the triplets. More precisely if the examples are drawn identically and independently from D T , one can only assume that the pairs are independent but not that they are identically distributed. Nevertheless the two particular frameworks presented in Section 1.3 are based on the McDiarmid's inequality (McDiarmid, 1989) that only needs to assume that the examples are independent. Using adapted definitions of uniform stability [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF] and Rademacher complexity [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] these two frameworks have been successfully extended to metric learning with pair based constraints. Note that the robustness framework has also been considered for metric learning (Bellet and Habrard, 2015) but we do not present it here.

Uniform stability for metric learning To extend the uniform stability framework to metric learning, [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF] propose to adapt the definition of uniform stability as follows.

Definition 2.1 (Uniform Stability for Metric Learning). Let T ∼ D T be a size n training set and z ∼ D T be an example. Let T i be the training set obtained by replacing example i in T by z. Let A be an algorithm which returns a metric h T when learning with the training set T and h T i when learning with the training set T i . An algorithm A has uniform stability β with respect to its loss function l (•) if the following holds: .29) Given this definition, they show that using the same proof technique that in [START_REF] Bousquet | Stability and generalization[END_REF] it is possible to obtain a generalization bound for metric learning similar, up to some constants, to the one presented in Theorem 1.1. As an example of a practical use of the framework, [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF] show that their algorithm, presented in Section 2.3, is uniformly stable and thus that the metric learned with their method generalizes well to new pairs of examples. Definition 2.2 (Rademacher Complexity for Metric Learning). Let T ∼ D T be a set of size n such that the pairs (z i , z n 2 +i ) are i.i.d.. Let σ be a vector of n Rademacher Variables, i.e. variables which can take a value of either 1 or -1 with probability 1 2 . Let • be a norm and • * its dual norm4 . The Rademacher average is respectively defined for Mahalanobis distance learning and bilinear similarity learning:

∀i ∈ {1, . . . , n} , sup z ,z ∼D T l h T , z , z -l h T i , z , z ≤ β n . ( 2

Rademacher complexity for metric learning

Rn ( • ) = 1 n 2 E σ n 2 i=1 σ i (x i -x n 2 +i )(x i -x n 2 +i ) T * (2.30) Rn ( • ) = 1 n 2 E σ n 2 i=1 σ i x i x T n 2 +i * . ( 2

.31)

The Rademacher complexity for Metric Learning is then defined as:

R( • ) = E T ∼D T Rn ( • ) (2.32)
where the expectation is taken over size n training sets.

Note that even if the i.i.d. property of the pairs is relaxed when using the McDiarmid concentration inequality, it is still needed when computing the Rademacher complexity. However, using properties of U-statistics (See e.g. [START_REF] Clémençon | Ranking and empirical minimization of u-statistics[END_REF]), [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] show that this is not an issue in practice since it is possible, in expectation over all the possible training sets of size n, to reduce a pair based approach to the case of i.i.d. random variable blocks as required in this definition 5 . Using this new definition of Rademacher complexity for metric learning [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] derive a generalization bound which is close in spirit to the one presented in Theorem 1.2.

Impact on a Subsequent Algorithm

As we have seen in Chapter 1, metrics are used in a wide range of applications and choosing a good metric through metric learning or other means can be seen as a preprocessing step for classic problems such as clustering or classification. Hence the question of the impact of the metric on the algorithm which makes use of it may arise. Such problems have been for example addressed by [START_REF] Balcan | Improved guarantees for learning via similarity functions[END_REF] who propose to define the goodness of a metric as its capacity to determine the similarity between the examples and a set of so called reasonable examples. They then show that this can be directly related to the performance of a linear classifier making use of the metric. Building upon this framework, several works propose to learn a good metric [START_REF] Bellet | Learning good edit similarities with generalization guarantees[END_REF][START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF] or even to jointly learn the metric and its associated classifier [START_REF] Nicolae | Joint semisupervised similarity learning for linear classification[END_REF]. Using a different approach [START_REF] Guo | Guaranteed classification via regularized similarity learning[END_REF] propose to learn a metric specifically designed to improve the performance of a linear SVM [START_REF] Vapnik | Statistical Learning Theory[END_REF]. Building upon the Rademacher Complexity framework they show that the true risk of the classifier is bounded by the empirical risk of the metric. Once again, the problem of considering the impact of the learned metric on an algorithm making use of it is beyond the scope of this thesis.

In this thesis we theoretically justify the approaches presented in Chapters 3, 4 and 5 by deriving generalization bounds based either on the uniform stability or Rademacher complexity frameworks.

Applications

If many approaches consider learning a metric for a clustering or classification task, some methods are specifically designed to help solve other kind of tasks such as image retrieval or domain adaptation. We provide a quick non exhaustive overview here.

Semi-supervised clustering

The idea behind semi-supervised clustering [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF][START_REF] Chang | Locally linear metric adaptation for semi-supervised clustering[END_REF] is that instead of having the labels of the examples as in classification, we have only access to similarity and dissimilarity constraints.

Classification A lot of approaches have been interested in learning a metric for classification [START_REF] Semerci | Mixtures of large margin nearest neighbor classifiers[END_REF][START_REF] Davis | Informationtheoretic metric learning[END_REF][START_REF] Nicolae | Joint semisupervised similarity learning for linear classification[END_REF]. Similarly, [START_REF] Kilian Q Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF]; [START_REF] Wang | Parametric local metric learning for nearest neighbor classification[END_REF]; [START_REF] Goldberger | Neighbourhood components analysis[END_REF][START_REF] Mustafa Qamar | Similarity learning for nearest neighbor classification[END_REF] propose to put a special emphasis on learning a metric specifically designed for a nearest neighbour classifier (See Section 1.5). It results in methods where only a subset of the constraints are considered as in LMNN (Weinberger et al., 2005) presented in Section 2.3. Other approaches consider the problem of learning low rank matrices to simplify the subsequent classification algorithm [START_REF] Ying | Sparse metric learning via smooth optimization[END_REF][START_REF] Shi | Sparse compositional metric learning[END_REF] or learning a metric inducing a sparse classifier [START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF].

Image retrieval Metric learning has also been used to improve image retrieval algorithms where the goal is, given a query image, to retrieve the most similar images in a given set [START_REF] Chechik | Large scale online learning of image similarity through ranking[END_REF][START_REF] Marc T Law | Fantope regularization in metric learning[END_REF][START_REF] Chang | Locally smooth metric learning with application to image retrieval[END_REF][START_REF] Huo | Robust and effective metric learning using capped trace norm: Metric learning via capped trace norm[END_REF]. Similarly the idea behind face recognition is to be able to recognize a query person in a set of images [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF][START_REF] Bohné | Large margin local metric learning[END_REF]Cao et al., 2013b;[START_REF] Zha | Robust distance metric learning with auxiliary knowledge[END_REF].

Domain adaptation

In domain adaptation we assume that we have access to two domains, the source S and the target T , and that we want to adapt from the source to the target (See Section 1.5). One way to perform domain adaptation is to bring the two domains closer to each other, i.e. to align the source and target examples such that any classifier leaned on the source can also be applied on the target. Several approaches in metric learning thus propose to learn a metric to bring examples from the source closer to the examples from the target.

• [START_REF] Saenko | Adapting visual category models to new domains[END_REF] propose to use ITML [START_REF] Davis | Informationtheoretic metric learning[END_REF] presented in Section 2.3 to learn a Mahalanobis distance which brings closer the two domains. They consider a semi-supervised domain adaptation problem, i.e. some of the target examples are labelled. To generate the constraints they propose to randomly select examples from the source and the target and simply consider them as similar if they share the same label and as dissimilar otherwise. Learning a Mahalanobis distance might sometimes not be sufficient to overcome the shift between the two domains. Hence they also consider the kernelized version of ITML in order to obtain a non linear metric.

• In Asymmetric Regularized Cross-domain transformation (ARC-t) [START_REF] Kulis | What you saw is not what you get: Domain adaptation using asymmetric kernel transforms[END_REF] propose to learn a bilinear similarity between the source and the target domain for a semi-supervised domain adaptation task. The interest is that instead of modifying the source and the target domain at the same time as in [START_REF] Saenko | Adapting visual category models to new domains[END_REF], they simply move one domain closer to the other. They propose to use the following optimization problem:

arg min

M∈R d×d (z s ,z t )∈P sim γ sim -k M x s , x t 2 + + (z s ,z t )∈P dis k M x s , x t -γ dis 2 + + λ 2 M 2 F
where the two examples from the source and the target are similar if they share the same label and are dissimilar otherwise. Also note that in this work the bilinear similarity is considered to be oriented in the sense that the source example always multiply the matrix M on the left while the target example always multiply M on the right. To consider more complex transformations between the source and the target they propose to kernelize their approach. et al., 2006) where the idea is that the respective means of the source and target samples should be close even after the projection. They propose to learn the transformation matrix L which minimizes the following optimization problem:

arg min

L∈R d ×d (z s ,z s )∈P sim d 2 L x s , x s + 1 n z s ∈S Lz s - 1 m z t ∈T Lz t 2 2 s.t. (z s ,z s )∈P dis d L x s , x s ≥ 1.
They also propose a kernelization of their approach based on a Kernel Principal Component Analysis (KPCA) (Schölkopf et al., 1997) in order to learn a non linear metric.

Other tasks As a last remark note that metric learning has also been used to improve the performance of a kernel [START_REF] Kilian | Metric learning for kernel regression[END_REF] where the idea is to consider distance based kernels and to optimize the corresponding metric or in a multi-task setting [START_REF] Parameswaran | Large margin multi-task metric learning[END_REF] where the idea is to learn one metric for each task under the constraint that all these metrics share a common basis.

In this thesis we demonstrate the interest of our algorithms in a wide range of applications. In Chapter 3 we propose to learn a metric for perceptual color differences and we show the interest of this metric in a segmentation task. In Chapter 4 we consider the problem of learning a metric with auxiliary knowledge and we apply our framework to a semi-supervised domain adaptation task. In Chapter 5 we propose a new framework for machine learning and we demonstrate its good performances for classification problems. Lastly in Chapter 6 the interest of the algorithm is demonstrated on two tasks, namely unsupervised domain adaptation and seamless copy in images.

Conclusion

In this chapter we proposed a non exhaustive review of metric learning. We chose to consider 4 different questions and to study several approaches proposing different answers to these problems. First we noticed that several kinds of metrics can be learned with metric learning approaches. The most popular one is the Mahalanobis distance while the bilinear similarity has also been widely studied. To include some non linearity several methods propose to learn multiple metrics while others consider either learning a linear metric in a kernel induced space or directly learning an intrinsically non linear metric. Second we presented many approaches interested in learning a metric in a batch setting by using an optimization problem making use of pair, triplet or quadruplet based constraints. We also considered several methods addressing the problem of learning a metric in an online fashion. Third we introduced two methods to derive generalization guarantees in metric learning. These approaches are respectively based on the uniform stability and the Rademacher complexity frameworks. We also recalled several methods interested in the theoretical impact of a metric on the subsequent algorithm. Fourth we considered different tasks that can be solved with the help of metric learning, namely classification, clustering, image retrieval or domain adaptation.

This chapter concludes the first part of this thesis that was dedicated to the presentation of several notions which will be used throughout our contributions. In the next part we address the problem of controlling the behaviour of a metric such that it either follow or stay close to a reference metric. In our first contribution we propose to learn a metric able to approximate a reference distance from a limited number of examples.

Part II

Metric Learning with a Reference Metric

Abstract

In this chapter we are interested in the problem of estimating an unknown reference metric from a set of pairs of examples. A solution to this problem could be to use metric learning to automatically approximate the values of the reference metric. However most of the algorithms proposed in metric learning are more interested in correctly estimating the relative closeness of the examples rather than the actual distance. In this chapter we propose a new local metric learning algorithm to learn a Mahalanobis distance which correctly approximates a reference metric. Using the uniform stability framework we derive generalization guarantees on the learned model showing that our method is theoretically founded. Furthermore we evaluate our approach in a computer vision problem, namely the computation of perceptual color differences. Having perceptual differences between scene colors is key in many computer vision applications such as image segmentation or visual salient region detection. Nevertheless, most of the times, we only have access to the rendered image colors, without any means to go back to the true scene colors. There are two main existing approaches to tackle this problem. On the one hand, one can compute a complex perceptual distance between rendered image colors. However it makes the distance dependent on the acquisition conditions and thus far from the scene color differences. On the other hand one can estimate the scene colors from the rendered image colors and then evaluate perceptual distances. However it implies the knowledge of the acquisition conditions which is an unreasonable assumption for most of the applications. Our approach allows us to learn a metric which is invariant to the acquisition conditions and computed only from rendered image colors. Our experimental evaluation shows its great ability (i) to generalize to new colors and devices and (ii) to deal with segmentation tasks.

Introduction

In recent years, metric learning has mainly been interested in learning metrics able to estimate the relative similarities between examples. This can be attributed to the fact that using similarity and dissimilarity constraints is usually the way to go when learning a metric (See Section 2.3). It implies that automatically learned distances are able to return a small value when comparing similar examples and a large value when comparing dissimilar examples. In this context the exact value is often out of interest. For example when using a nearest neighbour algorithm what really matters is the ordering of the examples rather than the exact distance. However there are some cases where learning a distance able to return a specific value could be of interest. This is for example the case when one wants to approximate an existing distance to simplify its computation or when one has access to pairs of examples alongside their distances but no way to compute the distance between new examples. To sum up here we are more interested in regressing the values of a metric than in learning the best metric for a subsequent algorithm as it is often the case in metric learning (See Section 2.5).

In this chapter we present a new algorithm to deal with the problem of learning a metric able to approximate a reference distance. We propose to learn a Mahalanobis distance which corresponds to a linear transformation of the input space (See Section 2.2). However there is no guarantee that the metric we want to approximate can effectively be embedded in an euclidean space. In other words a linear metric might not be sufficient. Following previous works in local metric learning (Section 2.2) we propose to learn several metrics across the input space. More precisely we consider a hard partitioning of the space (Weinberger and Saul, 2008) and learn one metric for each cluster. To deal with the problem of examples which do not fall in the same cluster we also learn a so called global metric. Moreover we show that our approach is theoretically founded. Indeed we build upon the framework of uniform stability to show that the global and each local metric generalize well. Furthermore, combining these generalization bounds we derive a global bound which holds for the whole model.

We evaluate our method on the computer vision problem of learning perceptual color differences, i.e. differences between colors which are proportional to the color difference perceived by human observers. A metric with such a property is highly desirable for most of computer vision applications and especially for visual saliency detection [START_REF] Achanta | Saliency detection using maximum symmetric surround[END_REF] or image segmentation [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF]. The main drawbacks of existing methods for computing perceptual color differences is that they are either dependent on the acquisition conditions or make unrealistic assumptions to be usable in a practical context. Using our approach we propose to approximate a perceptual color difference. Furthermore, we create a new dataset specifically dedicated to the task. It allows us to go a step further by learning a metric which is mostly invariant to acquisition conditions. This last point is empirically demonstrated by showing the ability of the learned metric to generalize to new colors and to new cameras, i.e. to new acquisition conditions. Furthermore we illustrate the good behaviour of our distance in a standard segmentation task. This chapter is organised as follows. First in Section 3.2 we present our local metric learning algorithm. Then in Section 3.3 we derive generalisation bounds which theoretically show the good behaviour of our approach. Section 3.4 is dedicated to the problem of learning color differences in computer vision and to our new dataset created to learn perceptual distances. Finally in Section 3.5 we empirically evaluate our approach before concluding in Section 3.6.

Regressing the Values of a Reference Metric by Local Metric Learning

In this section we present our metric learning framework whose objective is to approximate a reference distance ∆ : X × X → R d . It aims at optimizing K local metrics plus one global metric. Let T be the domain equipped with the distribution D T over the space X × X × R where X ∈ R d is a vector space and R ∈ R + is the set of values the reference metric can take.

We consider that we have access to a training set of pairs and their distance:

T = (x i , x i , ∆(x i , x i )) n i=1 . (3.1)
For the sake of simplicity, when the examples are clear from the context we replace ∆(x i , x i ) by ∆.

To learn a local metric we first divide the space of examples, i.e. X , in K local parts using a clustering algorithm. From this, we deduce K +1 regions defining a partition C 0 , C 1 , . . . , C K over the possible pairs of examples, i.e. over X × X . A pair (x, x ) belongs to a region C j , 1 ≤ j ≤ K if both x and x belong to the same cluster j, otherwise it is assigned to region C 0 . In other words, each region C j corresponds to pairs related to cluster j, while C 0 contains the remaining pairs whose points do not belong to the same cluster. It gives us a finite-size training sample of n j pairs for each region:

T j = (x i , x i , ∆) n j i=1 . (3.2)
To approximate ∆ we independently learn a Mahalanobis distance in every C j , j = 0, 1, . . . , K. We define a loss function l on any matrix M and any pair of examples as:

l M, (x, x , ∆) = (x -x ) T M(x -x ) -∆ 2 . (3.3)
Here we consider the non differentiable absolute loss rather than a more classic loss based on the 2 norm. It makes the optimization problem harder to solve. However we will see in Section 3.4 that in our application we are mainly interested in having a good metric for small

T j = {(x i , x i , ∆)} n j i=1 , j = 0, 1 . . . , K. for j = 0 → K do
Learn M T j by solving the convex optimization Problem (3.5) using T j end end Algorithm 1: Local metric learning values of the reference distance. In this case the absolute loss is more adapted as it penalizes more small approximation errors which are more likely to happen when dealing with small distances. We denote the empirical error over the set T j by:

LT j (M) = 1 n j (x,x ,∆)∈T j l M, (x, x , ∆) . (3.4)
Finally we suggest to learn the matrix M T j minimizing LT j via the following regularized problem: .5) where λ j > 0 is a regularization parameter. It is worth noting that our optimization problem takes the form of a simple regularized least absolute deviation formulation. The interest of using the least absolute deviation, rather than a regularized least square, comes from the fact that it enables accurate estimates of small ∆ values.

arg min M 0 LT j (M) + λ j M 2 F . ( 3
The pseudo-code of our metric learning algorithm is presented in Algorithm 1. Note that to solve the convex Problem (3.5), we use a classic interior points approach. Moreover, parameter λ j can be tuned by cross-validation.

Discussion about Local versus Global Metric

Note that in our approach, the metrics learned in the K regions C 1 , . . . , C K are local metrics while the one learned for region C 0 is rather a global metric considering pairs that do not fall in the same region. Beyond the fact that such a setting will allow us to derive generalization guarantees on our algorithm, it constitutes a straightforward solution to deal with examples at test time that would not be concerned by the same local metric in the input space. In this case, we make use of the matrix M T 0 associated to partition C 0 . Another possible solution may consist in resorting to a Gaussian embedding of the local metrics. However, because this solution would imply learning additional parameters, we suggest here to make use of this simple and efficient (parameters-wise) strategy. In the segmentation experiments, we noticed that M T 0 is used in only ∼20% of the cases. Finally, note that if K = 1, this boils down to learning only one global metric over the whole training sample.

In the next section, we prove generalization guarantees for our approach.

Theoretical Analysis

We now provide a generalization bound justifying that the metrics learned with our approach will generalize well. It is derived by considering (i) a multinomial distribution over the regions, and (ii) per region generalization guarantees that are obtained with the uniform stability framework presented in Section 1.3.

First of all we assume that the training sample T = ∪ K j=0 T j is drawn from an unknown distribution D T over a domain T such that for any (x, x , ∆) ∼ D T , ∆ ≤ ∆ max , with ∆ max the maximum distance value used in our context. We assume that given any two examples x and x we have xx 2 ≤ 1, i.e. the examples are normalized1 . The K+1 regions C 0 , . . . , C K define a partition of the support of D T where Pr(C j ) is the probability that a pair of examples falls in region C j . In C j , let D T j be the marginal distribution and D j = max (x,x ,∆)∼D T j xx 2 be the maximum distance between two examples.

Let M T = {M T 0 , M T 1 , . . . , M T K } be the K + 1 matrices learned by Algorithm 1. We define the true error associated to M T by:

L T (M T ) = K j=0 L T j (M T j ) Pr(C j ) (3.6)
where

L T j (M T j ) = E (x,x ,∆)∼D T j l M T j , (x, x , ∆) (3.7)
is the local true risk for region C j . The empirical error over T of size n is defined as:

LT (M T ) = 1 n K j=0 n j LT j (M T j ) (3.8)
where LT j (M T j ), Equation (3.4), is the empirical risk over T j , i.e. for region C j .

Generalization Bound per Region C j

For any learned local matrix M T j , we provide a bound on its associated local true risk L T j (M T j ) in function of the empirical risk LT j (M T j ) over T j . To this end we use the uniform stability framework presented in Section 1.3. Note that this theoretical analysis is based on the work of [START_REF] Bousquet | Stability and generalization[END_REF]. Hence to show a generalization bound in each region C j we use the McDiarmid's inequality (Theorem A.1) on the estimation error, i.e. the difference between the true risk and the empirical risk. Before that we need to show that our algorithm to learn a metric in each region is uniformly stable which requires the loss to be bounded and k-lipschitz (Definition A.1). First of all, our loss function, Equation (3.3), is bounded and k-lipschitz as shown in the two following lemmas. Lemma 3.1 (Bounded loss function). For any 0 ≤ j ≤ K, let M T j be the metric learned for region C j with the training set T j , we have that for any example (x, x , ∆) ∼ D T j :

0 ≤ l M T j , (x, x , ∆) ≤ B j , (3.9
)

with B j = max ∆max √ λ j , ∆ 2 max .
Proof. The proof of this lemma can be found in Appendix B.1.

Lemma 3.2 (k-lipschitz continuity). Let M T j and M T j be two matrices for a region C j and (x, x , ∆) be an example. Our loss

l M T j , (x, x , ∆) is k-lipschitz continuous with k = D 2 j .
Proof. The proof of this lemma can be found in Appendix B.2.

We can show that our approach is uniformly stable in the sense of Definition 1.3 for each region C j . Lemma 3.3 (Uniform stability per region C j ). Given two training samples T j and T i j of n j examples where T i j is obtained by replacing example i from T j by another example drawn independently from D T j . Let M T j and M T i j be the respective optimal solutions of Problem (3.5) when learning with T j and T i j . In region C j our problem is β j uniformly stable with

β j = 2D 4 j λ j .
Proof. The proof of this lemma can be found in Appendix B.3.

Using Lemma 3.3 about the stability of our algorithm and McDiarmid's inequality (Theorem A.1) we can derive our generalization bound. Let R T j = L T j (M T j ) -LT j (M T j ) be the estimation error for Problem (3.5) when learning with training set T j . To apply McDiarmid's inequality we need to bound

E T j ∼D T j R T j and R T j -R T i j
. This is done in the two following lemmas.

Lemma 3.4 (Bound on E T j ∼D T j R T j ). For any β j uniformly stable learning method of estimation error R T j = L T j (M T j ) -LT j (M T j ) for a training set T j , we have:

E T j ∼D T j R T j ≤ β j n j . (3.10)
Proof. The proof of this lemma can be found in Appendix B.4.

Lemma 3.5 (Bound on R T j -R T i j
). For any β j uniformly stable learning method of estimation error R T j = L T j (M T j ) -LT j (M T j ) for a training set T j and any B j bounded loss function we have:

R T j -R T i j ≤ 2β j + B j n j . (3.11)
Proof. The proof of this lemma can be found in Appendix B.5.

We can now show that Problem (3.5) generalize well for each region C j .

Lemma 3.6 (Generalization bound per region C j ). For any matrix M T j learned with Problem (3.5) with the training set T j in region C j , we have with probability 1 -δ:

L T j (M T j ) -LT j (M T j ) ≤ 2D 4 j λ j n j + 4D 4 j λ j + B j ln( 2 δ ) 2n j . (3.12) Proof. Using the McDiarmid inequality (Theorem A.1) on R T j = L T j (M T j ) -LT j (M T j )
, the estimation error, coupled with Lemma 3.5 for the c i values we have:

Pr R T j -E T j ∼D T j R T j ≥ ≤ 2 exp   - 2 2 n j i=1 2β j +B j n j 2    ≤ 2 exp - 2 2 1 n j (2β j + B j ) 2 .
Then, by setting:

δ = 2 exp - 2 2 1 n j (2β j + B j ) 2
we obtain:

= (2β j + B j ) ln 2 δ 2n j and: Pr R T j -E T j ∼D T j R T j < > 1 -δ.
Then, with probability 1 -δ:

R T j < E T j ∼D T j R T j + ⇔ L T j (M T j ) -LT j (M T j ) < E T j ∼D T j R T j + (2β j + B j ) ln 2 δ 2n j (Lemma 3.4.) ⇔ L T j (M T j ) -LT j (M T j ) < β j n j + (2β j + B j ) ln 2 δ 2n j . (3.13)
Noting that Lemmas 3.5 and 3.4 also hold for R T j = LT j (M T j ) -L T j (M T j ) and using similar arguments than above we obtain with probability 1 -δ that:

LT j (M T j ) -L T j (M T j ) < β j n j + (2β j + B j ) ln 2 δ 2n j . (3.14)
From Equations (3.13) and (3.14) we deduce that with probability 1 -δ we have:

L T j (M T j ) -LT j (M T j ) < β j n j + (2β j + B j ) ln 2 δ 2n j .
Replacing β j by its value gives the lemma.

This lemma shows that good generalization is achieved in each region with a convergence rate in O 1 √ n . When the region is compact, the quantity D j is rather small making the bound tighter. However we will see in the next section that for the generalization of Algorithm 1 there is a trade-off between D j and the number of regions K.

Generalization Bound for Algorithm 1

The generalization bound of our algorithm is based on the fact that the different marginals D T j can be interpreted as the parameters of a multinomial distribution. Then, we have that (n 0 , n 1 , . . . , n K ) is an i.i.d. multinomial random variable with parameters n = K j=0 n j and (Pr(C 0 ), Pr(C 1 ), . . . , Pr(C K )). Our result makes use of the Bretagnolle-Huber-Carol concentration inequality for multinomial distributions [START_REF] Aad | Weak convergence[END_REF] which is recalled in Proposition A.1 for the sake of completeness (this result has also been used in other contexts (Xu and Mannor, 2012)).

Theorem 3.1 (Generalization bound for Algorithm 1). Let C 0 , C 1 , . . . , C K be the regions considered, then for any set of metrics M T = {M T 0 , . . . , M T K } learned by Algorithm 1 from a data sample T of n triplets, we have with probability at least 1 -δ that

L T (M T ) ≤ LT (M T ) + B 2(K + 1) ln 2 + 2 ln 2 δ n + 2(KD 4 + 1) λn + 4(KD 4 + 1) λ + (K + 1)B ln( 4(K+1) δ ) 2n (3.15)
where B = max 0≤j≤K B j is a global bound on the loss function, D = max 1≤j≤K D j is the maximum euclidean distance in a region except C 0 and λ = min 0≤j≤K λ j is the minimum regularization parameter among the K + 1 learning problems used in Algorithm 1.

Proof. Let n j be the number of points of T that fall into the partition C j . (n 0 , n 1 , . . . , n K ) is an i.i.d. multinomial random variable with parameters n and (Pr(C 0 ), Pr(C 1 ), . . . , Pr(C K )).

L T (M T ) -LT (M T ) = K j=0 L T j (M T j ) Pr(C j ) -LT (M T ) = K j=0 L T j (M T j ) Pr(C j ) - K j=0 L T j (M T j ) n j n + K j=0 L T j (M T j ) n j n -LT (M T ) (Triangle inequality.) ≤ K j=0 L T j (M T j ) Pr(C j ) - K j=0 L T j (M T j ) n j n + K j=0 L T j (M T j ) n j n -LT (M T ) (Triangle inequality.) ≤ K j=0 L T j (M T j ) Pr(C j ) - n j n + K j=0 L T j (M T j ) n j n - K j=0 n j n LT j (M T j ) (Lemma 3.1 coupled with the definition of B.) ≤ K j=0 B Pr(C j ) - n j n + K j=0 n j n L T j (M T j ) -LT j (M T j ) (Proposition A.1 with probability 1 -δ 2 .) ≤ B 2(K + 1) ln 2 + 2 ln 2 δ n + K j=0 n j n L T j (M T j ) -LT j (M T j ) (Lemma 3.6 with probability 1 -δ 2(K+1) in each region.) ≤ B 2(K + 1) ln 2 + 2 ln 2 δ n + K j=0 n j n   2D 4 j λ j n j + 4D 4 j λ j + B j ln( 4(K+1) δ ) 2n j   (Definition of B, D, λ, D 0 = 1 and noting that √ n j ≤ √ n.) ≤ B 2(K + 1) ln 2 + 2 ln 2 δ n + 2(KD 4 + 1) λn + 4(KD 4 + 1) λ + (K + 1)B ln( 4(K+1) δ ) 2n
.

Finally, the union bound (Theorem A.2) gives the theorem with probability 1 -δ.

This result justifies that good generalization is achieved globally with a standard convergence rate in O 1 √ n . We can remark that if the local regions C 1 , . . . , C K are rather small (i.e. D is significantly smaller than 1), then the last part of the bound will not suffer too much on the number of regions. On the other hand, there is also a trade-off between the number/size of regions considered and the number of instances falling in each region. It is important to have enough examples to learn good models.

Learning Perceptual Color Differences

In computer vision, the evaluation of color differences is required for many applications. For example, in image segmentation, the basic idea is to merge two neighbour pixels in the same region if the difference between their colors is "small" and to split them into different regions otherwise [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF]. Likewise, for visual salient region detection, the color difference between one pixel and its neighbourhood is also the main used information [START_REF] Achanta | Saliency detection using maximum symmetric surround[END_REF], as well as for edge and corner detection (Van de [START_REF] Van De Weijer | Boosting color saliency in image feature detection[END_REF][START_REF] Van De Weijer | Edge and corner detection by photometric quasi-invariants[END_REF]. Similarly, in order to evaluate the quality of color images, [START_REF] Xue | Perceptual fidelity aware mean squared error[END_REF] have shown that the pixel-wise mean square difference between the original and distorted image provides very good results. As a last example, the orientation of gradient which is the most widely used feature for image description (SIFT [START_REF] David G Lowe | Distinctive image features from scale-invariant keypoints[END_REF], HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]) is evaluated as the ratio between vertical and horizontal differences.

Depending on the application requirement, the used color difference may have different properties. For material edge detection, it has to be robust to local photometric variations such as highlights or shadows (Van de [START_REF] Van De Weijer | Edge and corner detection by photometric quasi-invariants[END_REF]. For gradient-based color descriptors, it has to be robust to acquisition condition variations [START_REF] Gertjan | Performance evaluation of local colour invariants[END_REF][START_REF] Van De Sande | Evaluating color descriptors for object and scene recognition[END_REF] or discriminative [START_REF] Van De Weijer | Boosting color saliency in image feature detection[END_REF]. For most applications and especially for visual saliency detection [START_REF] Achanta | Saliency detection using maximum symmetric surround[END_REF], image segmentation [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF] or image quality assessment [START_REF] Xue | Perceptual fidelity aware mean squared error[END_REF], the color difference has to be above all perceptual, i.e. proportional to the color difference perceived by human observers. As such a large amount of work has been done by color scientists around perceptual color differences [START_REF] Wyszecki | Color Science: Concepts and Methods, Quantitative Data and Formulas[END_REF][START_REF] Huang | Evaluation of threshold color differences using printed samples[END_REF][START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF], where the required inputs of the proposed distances are either reflectance spectra or the device-independent color components CIE XYZ [START_REF] Wyszecki | Color Science: Concepts and Methods, Quantitative Data and Formulas[END_REF]. These features are obtained with particular devices such as spectrophotometer or photoelectric colorimeter [START_REF] Wyszecki | Color Science: Concepts and Methods, Quantitative Data and Formulas[END_REF]. It is known that neither the euclidean distance between reflectance spectra nor the euclidean distance between XYZ vectors are perceptual, i.e. these distances can be higher for two colors that look similar than for two colors that look different. Consequently, some color spaces such as CIELAB or CIELUV have been designed to be more perceptually uniform. In those spaces, specific color difference equations have been proposed to further improve perceptual uniformity over the simple euclidean distance [START_REF] Huang | Evaluation of threshold color differences using printed samples[END_REF]. The ∆E 00 [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF]) distance is one nice example of such a distance. It corresponds to the difference perceived by a human looking at the two considered colors under standard viewing conditions recommended by the CIE (illuminant D65, illuminance of 1000 lx, etc.). However, it is worth noting that in most of the computer vision applications, the available information does not take the form of a reflectance spectra or some device-independent components, as assumed above. Indeed, the classical acquisition devices are cameras that use iterative complex transforms from the irradiance (amount of light) collected by each CCD sensor cell to the pixel intensity of the output image (Kim et al., 2012b). These device-dependent transforms are color filtering, white-balancing, gamma correction, demosaicing, compression, etc. (Xiong et al., 2012b) which are designed to provide pleasant images and not to accurately measure colors. Consequently, the available RGB components in color images do not allow us to get back to the original spectra or XYZ components. To overcome this limitation, two main strategies have been suggested in the literature: either by applying a default transformation from RGB components to L * a * b * (CIELAB space) or L * u * v * (CIELUV space) assuming a given configuration, or by learning a coordinate transform to actual L * a * b * components under particular conditions.

Using default transformations A classic strategy consists in using a default transformation from the available RGB components to XYZ and then to L * a * b * or L * u * v * [START_REF] Achanta | Saliency detection using maximum symmetric surround[END_REF][START_REF] Arbelaez | Contour detection and hierarchical image segmentation[END_REF][START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF][START_REF] Khan | Discriminative color descriptors[END_REF][START_REF] Mojsilovic | A computational model for color naming and describing color composition of images[END_REF]. This default transformation assumes an average gamma correction of 2.2 [START_REF] Stokes | A standard default color space for the internet: sRGB[END_REF], color primaries close to ITU-R BT.709 (Union, 2000) and D65 illuminant (Daylight). Finally, from the estimated L * a * b * or L * u * v * (denoted L * a * b * and L * u * v * respectively) of two pixels, one can make use of the euclidean distance. In the case of L * a * b * , one can use L * a * b * to estimate more complex and accurate distances such as ∆E 00 via its estimate ∆E 00 [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF], that will be used in our experimental study as a baseline. This default approach provides a perceptual distance between the colors in the rendered image (called image-wise color distance) and not between the colors as they appear to a human observer looking at the real scene (called scene-wise color distance). For some applications such as image quality assessment, it is required to use the image-wise color distances since only the rendered image colors need to be compared, whatever the scene colors. But for a lot of other applications such as image segmentation or saliency detection, we claim that a scene-wise perceptual color distance should be used. Indeed, in these cases, the aim is to be able to evaluate distances as they would have been perceived by a human observing the scene and not after the camera transformations. Note that some solutions exist (Kim et al., 2012a) to get back to scene colors from RGB camera outputs, thus avoiding using a default transformation, but they require calibrated acquisition conditions (known illumination, known sensor sensitivities, RAW data available, . . . ).

Learning coordinate transforms to L * a * b * For applications requiring the distances between the colors in the scene, the acquisition conditions are calibrated first and then the images are acquired under these particular conditions [START_REF] Re Larraín | Use of digital images to estimate cie color coordinates of beef[END_REF][START_REF] Leon | Color measurement in l? a? b? units from rgb digital images[END_REF]. Therefore, the camera position and the light color, intensity and positions are fixed and a set of images of different color patches are acquired. Meanwhile, under the same exact condi-tions, a colorimeter measures the actual L * a * b * components (in the scene) for each of these patches. [START_REF] Leon | Color measurement in l? a? b? units from rgb digital images[END_REF] learn then the best transform from camera RGB to actual L * a * b * components with a neural network. [START_REF] Re Larraín | Use of digital images to estimate cie color coordinates of beef[END_REF] first apply the default transform presented before from camera RGB to L * a * b * and then learn a polynomial regression (until quadratic term) from the L * a * b * to the true L * a * b * . However, it is worth mentioning that in both cases the learned transforms are accurate only under these acquisition conditions. Thus, these approaches can not be applied on most of the computer vision applications where such an information is unavailable.

Using the metric learning method presented in Section 3.2 we propose to estimate scenewise color distances from non calibrated rendered image colors. Furthermore, we go a step further towards an invariant color distance. This invariance property means that, considering one image representing two color patches, the distance is predicting how much difference would have perceived a human observer looking at the two real patches under standard fixed viewing conditions, such as the ones recommended by the CIE (Commission Internationale de l'Eclairage) in the context of color difference assessment [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF]. In other words, whatever the acquisition device or the illuminant, an invariant scene-wise distance should return stable values. To the best of our knowledge, no previous work has both underlined and answered the problem of the approximations that are made during the estimation of perceptual color differences in the very frequent case of non calibrated acquisitions. It implies that no suitable dataset exists for the problem at hand. Hence we propose a new dataset specifically designed to learn a perceptual distance which is invariant across acquisition conditions.

Creating the Dataset

Given two color patches, we want to design a perceptual distance not disturbed by the acquisition conditions. So we propose to use pairs of patches for which we can measure the true perceptual distance under standard viewing conditions and to image them under different other conditions.

The choice of the patches is key in this work since all the distances will be learned from these pairs. Consequently, the colors of the patches have to be well distributed in the RGB cube in order to be able to well approximate the color distance between two new pairs that have not been seen in the training set. Moreover, as we would like to learn a local perceptual distance, we need pairs of patches whose colors are close from each other. According to [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF], ∆E 00 seems to be a good candidate for that because it is designed to compare similar colors. Finally, since hue, chroma and luminance differences impact the perceptual color difference [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF], the patches have to be chosen so that all these three variations are represented among the pairs. Given these three requirements, we propose to use two different well-known sets of patches, namely the Farnsworth-Munsell 100 hue test and the Munsell atlas (see Figure 3.1). The Farnsworth-Munsell 100 hue test is one of the most famous color vision tests which consists in ordering 84 patches in the correct order and any misplacement can point to some sort of color vision deficiency. Since these 84 patches are well distributed on the hue wheel, their colors will cover a large area of the RGB cube when imaging them under an important range of acquisition conditions. Furthermore, consecutive patches are known to have very small color differences and then, learning perceptual distances from such pairs is a good purpose. This set is constituting the main part of our dataset. However, the colors of these patches first, are not highly saturated and second, they mostly exhibit hue variations and relatively small luminance and chroma differences. In order to cope with these weaknesses, we add to this dataset the 238 patches constituting the Munsell Student Color Set [START_REF] Albert | A pigment color system and notation[END_REF]. These patches are characterized by more saturated colors and the pairs of similar patches mostly exhibit luminance and chroma variations (since only the 5 principal and 5 intermediate hues are provided in this student set).

To build the dataset, we first use a spectroradiometer (Minolta CS 1000) in order to measure the spectra of each color patch of the Farnsworth set, the spectra of the Munsell atlas patches being available online2 . Five measurements have been done in our light cabinet and the final spectra are the average of each measurement. From these spectra, we evaluate the L * a * b * coordinates of each patch under D65 illuminant. Then, we evaluate the distance ∆E 00 between all the pairs of color patches [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF] Note that the available ∆E 00 have been evaluated in the standard viewing conditions recommended by the CIE for color difference assessment and we would like to obtain these reference distances whatever the acquisition conditions. Consequently, we propose to use 4 different cameras, namely Kodak DCS Pro 14n, Konica Minolta Dimage Z3, Nikon Coolpix S6150 and Sony DCR-SR32 and a large variety of lights, viewpoints and backgrounds (since background also perturbs the colors of the patches). For each camera, we acquire 50 images of each Farnsworth pair and 15 of each Munsell pair (overall, 41, 800 imaged pairs). Finally, after all these measurements and acquisitions, we have for each image of a pair, two image rendered RGB vectors and one reference distance ∆E 00 .

In the next section, using this dataset, we evaluate the approach presented in Section 3.2.

Experiments

Evaluating the interest of a metric can be done in two ways:

• assessing the quality of the metric itself,

• measuring its impact once plugged in an application.

In the following, we evaluate the generalization ability of the learned metric on our dataset and we measure its contribution in a color segmentation application but first we give a brief overview of how we learn a metric on our dataset.

Learning the Metric

From our dataset of 41, 800 pairs and their reference distance ∆E 00 we can draw training sets T of varying size depending on our needs. Given T we learn a set of K + 1 local distances M T = {M T 0 , M T 1 , . . . , M T K }. Note that the local metrics are relatively simple since they correspond to 3 × 3 matrices. Furthermore, in all our experiments we consider a large amount of the training pairs. It makes our algorithm rather insensible to the choice of λ. Therefore, we chose to fix λ = 1.

The learned metric for a given training set T is denoted by ∆ T . For two examples x, x it is computed as follows:

∆ T (x, x ) = (x -x ) T M T j (x -x ) if x and x fall in the same cluster C j , 1 ≤ j ≤ K, (x -x ) T M T 0 (x -x ) otherwise.
(3.16)

Evaluation on our Dataset

To empirically evaluate the generalization ability of the metric, we conduct two experiments.

On the one hand we assess the behaviour of our approach when it is applied to new unseen colors. On the other hand we consider the problem of patches coming from a new unseen camera, i.e. of new acquisition conditions. All the results presented are averaged over 5 runs.

To estimate the performance of our metric we use two criteria that we want to make as small as possible. These two criteria are computed over a test set T = {(x i , x i , ∆E 00 )} n i=1 independent from the training set T . The first criterion is the mean absolute difference between the learned metric ∆ T and the reference metric ∆E 00 : As a second criterion, we use the STRESS3 measure [START_REF] Melgosa | Performance of recent advanced colordifference formulas using the standardized residual sum of squares index[END_REF] which is widely used by the computer vision community as a way to compare color differences. It is defined as follows:

mean = 1 n (x,x ,∆E 00 )∈T |∆ T -∆E 00 | . ( 3
STRESS = 100 (x,x ,∆E 00 )∈T (∆E 00 -F ∆ T ) 2 (x,x ,∆E 00 )∈T F 2 ∆ 2 T with F = (x,x ,∆E 00 )∈T ∆E 2 00 (x,x ,∆E 00 )∈T ∆E 00 ∆ T . (3.18)
Roughly speaking the STRESS evaluates quadratic differences between the learned metric ∆ T and the reference ∆. We compare our approach to the state of the art where ∆ T is replaced by ∆E 00 [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF] in both criteria, i.e. transforming from rendered image RGB to L * a * b * and computing the ∆E 00 distance.

Generalization to Unseen Colors

In this experiment, we perform a 6-fold cross validation procedure over the set of patches.

Thus we obtain, on average, 27927 training pairs and 13873 testing pairs. The results are shown on Figure 3.2(a) according to an increasing number of clusters (from 1 to 70). We can see that using our learned metric ∆ T instead of the state of the art estimate ∆E 00 [START_REF] Sharma | The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[END_REF] enables significant improvements according to both criteria (the baselines are 1.70 for the mean and 48.05 for the STRESS). Note that from 50 clusters onward, the quality of the learned metric declines slightly while remaining much better than ∆E 00 . Figure 3.2(a) shows that K = 20 seems to be a good compromise between a high algorithmic complexity (the higher K, the larger the number of learned metrics) and good performances of the models. When K = 20, using a Student's t test over the mean absolute differences and a Fisher test over the STRESS, our method is significantly better than the state of the art with a p-value < 1 -10 . Figure 3.2(a) also emphasizes the interest of learning several local metrics. Indeed, optimizing 20 local metrics rather than only one is significantly better with a p-value smaller than 0.001 for both criteria.

Generalization to Unseen Cameras

In this experiment, our model is learned according to a 4-fold cross validation procedure such that each fold corresponds to the pairs coming from a given camera. Thus we learn the metric on a set of 31350 pairs and test it on a set of 10450 pairs. This task is more complicated than generalizing to unseen colors. Indeed when generalizing to unseen colours even if the metric has never seen a given colour before it has been learned on similar examples. Contrarily the acquisition conditions highly depend on the kind of camera used and can vastly differ from one camera to another [START_REF] Ilie | Ensuring color consistency across multiple cameras[END_REF]. Given that we use a limited number of cameras there is no guarantee that similar acquisition conditions have been seen before. The results are presented in Figure 3.2(b). We can note that our approach always outperforms the state of the art for the mean criterion (of baseline 1.70). Regarding the STRESS, we are on average better when using between 5 to 60 clusters. Beyond 65 clusters, the performances decrease significantly. This behaviour likely describes an overfitting phenomenon due to the fact that a lot of local metrics have been learned that are more and more specialized for 3 out of 4 cameras, and unable to generalize well to the fourth one. For this series of experiments, K = 20 is still a good value to deal with the trade-off between complexity and efficiency. Using a Student's t test over the mean absolute differences and a Fisher test over the STRESS, our method is significantly better with p-values respectively < 1 -10 and < 0.006. The interest of learning several local metrics rather than only one is still confirmed. Applying statistical comparison tests between K = 20 and K = 1 leads to small p-values < 0.001.

Thus for both series of experiments, K = 20 appears to be a good number of clusters and allows significant improvements. Therefore, we suggest to take this value in the next section to tackle a segmentation problem. Before that, let us finish this section by geometrically showing Figure 3.3: Interest of learning local metrics. We took 27 points uniformly distributed on the RGB cube. Around each point we plotted an ellipsoid where the surface corresponds to the RGB colors lying at a learned distance of 1. In this case we used the metric learned by our algorithm using K = 20. the interest of learning local metrics. Figure 3.3 shows ellipsoids uniformly distributed in the RGB space whose surface corresponds to the RGB colors lying at the corresponding learned local distance of 1 from the center of the ellipsoid. It is worth noting that the variability of the shapes and orientations of the ellipsoids is high, meaning that each local metric could capture local specificities of the color space. The experimental results presented in the next section will prove this claim.

Application to Image Segmentation

In this experiment, we evaluate the performance of our approach in a color based image segmentation application. We propose to use the approach proposed by [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF] that suggests a nice extension of the classic mean-shift algorithm [START_REF] Fukunaga | The estimation of the gradient of a density function, with applications in pattern recognition[END_REF]) by accounting for color information. Furthermore, the authors show that the more perceptual the used distance, the better the results. Especially, by using the default transform from the available camera RGB to the L * u * v * space, they significantly improve the segmentation results over the simple RGB coordinates. Our aim is not to propose a new segmentation algorithm but to use the exact algorithm proposed by [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF] working in the RGB space and to replace in their code (publicly available) the distance between two colors with our learned color distance ∆ T . This way, we can compare the perceptual property of our distance with this of the recommended default approach (euclidean distance in the L * u * v * space).

Therefore, we take exactly the same protocol as [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF]. We use the same 200 images taken from the well-known Berkeley dataset and the associated ground-truth that is constituted by 1087 segmented images provided by humans. In order to assess the quality of the segmentation, as recommended by [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF], we use the average Boundary Displacement Error (BDE) and the Probabilistic Rand Index (PRI). Note that the better the quality of the segmentation, the lower the BDE and the higher the PRI. The segmentation algorithm proposed in [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF] has one main parameter which is the color distance threshold under which two neighbour pixels (or sets of pixels) have to be merged in the same segment. As in [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF], we plot the evolution of the quality criteria versus the average segment size (see Figures 3.4(a) and 3.4(b)). For comparison, we have run the code from [START_REF] Bitsakos | An experimental study of color-based segmentation algorithms based on the mean-shift concept[END_REF] for the parameters providing the best results in their paper, namely "CMS Luv/N.", corresponding to their color mean-shift (CMS) applied in the L * u * v * color space. The results of CMS applied in the RGB color space with the classical euclidean distance are plotted as "CMS RGB/N." and those of CMS applied with our color distance in the RGB color space are plotted as "CMS Local Metric/N.".

For both criteria, we can see that our learned color distance significantly improves the quality of the results over the two other approaches, i.e. it provides a segmentation that is closer to the one computed by humans. This is truer when the segment size is increasing (right part of the plots). It is important to understand that increasing the average segment size (moving to the right on the plots) is like merging neighbour segments in the images. So by analysing the curves, we can see that for the classic approaches ("CMS Luv/N." and "CMS RGB/N."), it seems that the segments that are merged together when moving to the right on the plot are not the ones that would be merged by humans. That is why both criteria are worst (BDE increases and PRI decreases) on the right for these methods. On the other hand, it seems that our distance is more accurate when merging neighbour segments since for high average segment sizes, our results are much better. This point can be observed in Figure 3.5, where the segment size is high, i.e. when the number of clusters is low [START_REF] Kilian | Metric learning for kernel regression[END_REF], the segmentation provided by RGB or L * u * v * are far from the ground truth, unlike our approach which provides nice results. To get the same perceptual result, both methods require about 500 clusters. We further illustrate the performance of our method in Figure 3.6. As explained before, the number of segments in the resulting images is not a parameter of the algorithm, as a consequence it is not easy to obtain images with the same number of segments for the three algorithms (RGB, L * u * v * and Metric learning). Thus, given an image, by modifying the color distance threshold, we tried to obtain the same segment numbers as the corresponding ground truth for the three algorithms. However, the color mean-shift algorithm provides some very small segments, specially for the RGB and L * u * v * color spaces. Consequently, for each test, in Figure 3.6, we have mentioned between brackets, first, the number of segments, and second, the number of segments whose size is more than 150 pixels. For a fair comparison, we use this last number as reference for each image, i.e. this number is almost constant and close to the ground truth for each row. It is worth mentioning that the ground truth segmentation has always very few segments. Thus, starting from a large number of small segments, the algorithm is grouping them by considering their color differences. Consequently, the used color distance is crucial when we want to obtain small number of segments as provided by the ground truth. We can see in Figure 3.6 that when working in the RGB or L * u * v * color spaces, some segments that are perceptually different are merged while some other similar segments are not. Most of the time, the color mean-shift is working well when using our distance.

Conclusion

In this chapter, we presented a new local metric learning approach able to approximate a reference distance. Based on a hard clustering of the space, the main idea is to minimize the absolute difference between the learned and the reference distances. Building upon the uniform stability framework we proved that this method is theoretically founded. It is guaranteed to generalize well if a sufficient number of examples is used. We have applied our framework to the problem of perceptual color differences where the idea is to have a metric which is invariant to acquisition conditions but also which is proportional to the human perception of color differences. To this end we proposed a new dataset specifically tailored for this problem. We have shown that with a sufficient number of clusters our approach allows us to learn a metric which is substantially better than the state of the art distances. Similarly we have demonstrated the interest of the learned metric in a segmentation application.

Even though Figure 3.3 shows ellipsoids that tend to be locally regular leading to a certain spatial continuity, our model does not explicitly deal with this issue. Hence one of the main drawback of the proposed approach is that the learned metric might not be smooth across the space. To deal with this problem we could consider a soft clustering of the space [START_REF] Semerci | Mixtures of large margin nearest neighbor classifiers[END_REF] where each example is associated to several local metrics depending on its degree of membership to each cluster. It would reduce the risk of having discontinuities in the values of the metric across the space, notably near the borders of the clusters. Another approach, explored for example in [START_REF] Zantedeschi | Metric learning as convex combinations of local models with generalization guarantees[END_REF], would consist in learning a smooth combination of the local metrics as an independent post processing step.

In our framework we only considered the Frobenius norm as a regularization term. One interesting perspective would be to consider other regularization terms such as the mixed norm or the nuclear norm. Indeed learning low rank matrices could reduce the computational cost of the metrics, especially in case of a very high dimensional input spaces. One of the drawbacks is that, as we have seen in Section 1.3, algorithms which make use of sparse regularization terms are not stable. It implies that our theoretical analysis would not hold and that other frameworks, such as the Rademacher complexity one, would have to be considered.

In this chapter we have studied the problem of learning an approximation of a reference metric. We assumed to have only access to this reference through its values over a limited number of examples. In the next chapter we consider to have fully access to the reference metric, i.e. we have the parameter matrix of a Mahalanobis distance or a Bilinear Similarity, and we want to use this reference to help learning another metric. More precisely we consider a transfer learning problem where the reference metric is either given or learned on a source domain, i.e. a source metric, and we want to learn a metric on a different but related domain, i.e. a target metric.

Metric Hypothesis Transfer Learning

This chapter is based on the following publication Michaël Perrot and Amaury Habrard. A theoretical analysis of metric hypothesis transfer learning. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1708-1717, 2015d

Abstract

We consider the problem of transferring some a priori knowledge in the context of supervised metric learning approaches. More precisely we consider biased optimization problems which make use of a source metric coming from a different but related problem to help learning in the presence of few data. While this setting has been shown to be empirically successful, no theoretical evidence exists to justify it. In this chapter we propose to close this gap by providing a theoretical analysis of this framework based on three different approaches. First we propose an on-average-replace-two-stability model allowing us to prove on average fast generalization rates when an auxiliary source metric is used to bias the regularization term. Second we consider a notion of algorithmic stability adapted to the regularized metric learning setting and we prove probabilistic generalization bounds which show the interest of considering biased weighted regularized formulations. We also provide a solution to estimate the associated weight that we evaluate in two experimental tasks (i) standard metric learning and (ii) transfer learning with few labelled target data. Third we derive a generalization bound related to the Rademacher complexity of the metric class taking into account the source metric considered by the algorithm. This vanishing bound underlines the interest of using a good source metric by showing that, when the source metric perfectly solves the problem, learning is no longer a necessity. To justify the interest of the framework we also provide several examples of loss functions and regularization terms which fall under one or more of our theoretical analyses.

Introduction

Recently, there is a growing interest for methods able to take into account some background knowledge when learning a metric [START_REF] Parameswaran | Large margin multi-task metric learning[END_REF]Cao et al., 2013a;[START_REF] Bohné | Large margin local metric learning[END_REF]. This is in particular the case for supervised regularized metric learning approaches where the regularization term is biased with respect to an auxiliary metric given under the form of a matrix. The main objective here is to make use of this a priori knowledge in a setting where only few labelled data are available to help learning. For example, in the context of learning a PSD matrix M plugged into a Mahalanobis-like distance, let I be the identity matrix used as an auxiliary knowledge, M -I is a biased regularization often considered. This regularization can be interpreted as follows: learn M while trying to stay close to the Euclidean distance, or from another standpoint try to learn a matrix M which performs better than I. Other standard matrices can be used such as Σ -1 the inverse of the variance-covariance matrix [START_REF] Chandra | On the generalized distance in statistics[END_REF]. If we take the 0 matrix, we retrieve the classic unbiased regularization setting.

Another useful setting comes when I is replaced by any auxiliary matrix M S learned from another task. It then corresponds to a transfer learning approach (See Section 1.5) where the biased regularization can be interpreted as transferring the knowledge brought by M S to help learning M. This setting is appropriate when the distributions over training and testing domains are different but related. Domain adaptation strategies [START_REF] Shai Ben-David | A theory of learning from different domains[END_REF] propose to make use of the relationship between the training examples, called the source domain, and the testing instances, called the target domain to infer a model. However, it is sometimes not possible to have access to all the training examples, for example when some new domains are acquired incrementally. In this context, transferring the information directly from the model learned from the source domain without any other access to the source domain is of crucial importance. We call this setting Metric Hypothesis Transfer Learning in reference to the Hypothesis Transfer Learning model introduced in [START_REF] Kuzborskij | Stability and hypothesis transfer learning[END_REF] in the context of classic supervised learning.

If metric hypothesis transfer learning has been shown to work well empirically, it has, to the best of our knowledge, never been investigated from a theoretical standpoint. In this chapter, we propose to bridge this gap by providing a theoretical analysis showing that supervised regularized metric learning approaches using a biased regularization are well-founded. This analysis is based on three different theoretical frameworks which allows us to underline the different properties of biased regularization based algorithms and to derive three measures of goodness of the source metric. The latter quantities are important in the sense that they give a founded way of estimating the interest of a source metric for a particular problem.

• On average stability: The first theoretical framework that we consider is derived from a notion of stability called on-average-replace-one-stability [START_REF] Ben | Domain adaptation as learning with auxiliary information[END_REF].

As in other stability frameworks the idea is to verify that small changes in the training does not significantly change the output of the algorithm. The main difference is that this property is considered on average over all the size n training sets. This approach allows us to derive a bound showing that on average the metric learned with a biased regularization will be better than the source metric with a convergence rate in O 1 n . It also implies a first theoretical notion of goodness of the source metric.

• Uniform stability for metric learning: The second theoretical framework that we consider has been proposed by [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF] This chapter is organized as follows. First we present the metric hypothesis transfer learning setting considered here in Section 4.2. Then we present our theoretical analysis based on three frameworks in Sections 4.3,4.4 and 4.5. We summarize the different bounds in Section 4.6. In Section 4.7 we propose several examples of loss functions and regularization terms that can be used in our framework. Next, in Section 4.8, we empirically demonstrate the interest of using a good source metric as defined in Section 4.4. We conclude in Section 4.9.

Metric Hypothesis Transfer Learning with Biased Regularization

In this section we present the framework of metric hypothesis transfer learning considered in this chapter. First of all, let T be a domain equipped with a probability distribution D T defined over X × Y, where X ⊆ R d and Y is the label set. Our goal is to learn a metric (as considered in Section 1.4) parametrized by a d × d matrix M. Let M be a metric class. Using a slight abuse of notations we denote the fact that a metric is in M by M ∈ M. Here, M can simply be the set of real matrices of dimension d × d or can be more restrictive and only consider symmetric positive semi-definite matrices. To learn we consider that we have access to T = {z i = (x i , y i )} n i=1 a set of n examples drawn i.i.d. from D T . We also assume that we have access to some additional information under the form of an auxiliary d × d matrix M S .

We call this additional information source metric or source hypothesis to denote the fact that, in a transfer learning setting, this metric can come from a different but related source domain S.

We consider all the algorithms learning M by solving the following optimization problem:

arg min M∈M LT (M) + λ M -M S 2 (4.1)
where M -M S 2 is a biased regularization term which ensures that there is a transfer of informations between M and M S . In Section 4.7 we consider several regularization terms allowing the transfer of different properties of the metric. λ is a trade-off parameter between risk and regularization. The empirical risk of a metric M over a training set T is:

LT (M) = 1 n(n -1) z∈T z ∈T z =z l M, z, z (4.2)
where l (M, z, z ) is a pairwise loss quantifying the error of the metric M when presented with the examples z and z . In Section 4.7 we present several loss functions which can be used in metric hypothesis transfer learning. The true risk of M over the distribution D T is:

L T (M) = E z,z ∼D T l M, z, z . (4.3)
In this chapter we consider a theoretical analysis of this framework of metric hypothesis transfer learning using three different theoretical approaches. As mentioned before several loss functions and regularization terms can be considered. However depending on the theoretical framework considered some assumptions have to be made on these and may differ from one approach to another. Similarly here we only considered a general framework able to handle any metric parametrized by a matrix M but it might sometimes be necessary to further restrain the range of metrics considered. For the sake of readability we postpone the definition of these different constraints to the beginning of each section.

On Average Stability Analysis

To derive our first bound for metric hypothesis transfer learning we propose a new notion of stability which is an adaptation to metric learning of the notion of on-average-replace-onestability (Shalev-Shwartz and Ben-David, 2014a) that we recall in Definition A.5 for the sake of completeness.

Assumptions In this section we make the following assumptions. We only consider metrics as Mahalanobis distances parametrized by a matrix M positive semi definite. The loss function has to be positive, convex in M (Definition A.6) and k-lipschitz (Definition A.1). As a last constraint we consider that the regularization term is the Frobenius norm (Section 1.4).

We now turn our attention to the derivation of the bound. First of all we introduce our new notion of on-average-replace-two-stability. Indeed Definition A.5 allows one to perform an on average analysis over the expected loss, however its formulation is not tailored to metric learning approaches that work with pairs of examples. Thus we propose an adaptation allowing us to derive sharp bounds for metric learning. Definition 4.1 (On-average-replace-two-stability). Let n be the number of examples considered during the learning step. Let : N → R be monotonically decreasing with respect to n and let U (1, n) be the uniform distribution over 1, . . . , n. A metric learning algorithm is on-average-replace-two-stable with rate (n) if for any distribution D T :

E T ∼D T i,j∼U (1,n) z,z ∼D T l M T i j , z i , z j -l (M T , z i , z j ) ≤ (n) (4.4)
where M T , respectively M T i j , is the optimal solution when learning with the training set T , respectively T i j . T i j is obtained by replacing z i , the i th example of T , by z to get a training set T i and then by replacing z j , the j th example of T i , by z .

This property ensures that, given two examples, learning with or without them will not imply a big change in the hypothesis prediction. Note that the property is required to be true on average over all the possible training sets of size n. Furthermore note that when this definition holds, it implies E T ∼D T L T (M T ) -LT (M T ) ≤ (n). Using this definition we derive a bound on the expected true risk of our algorithm. Before that we show, in the next theorem, that our algorithm is on-average-replace-two-stable. Proof. The proof of this lemma can be found in Appendix C.1.

We can now bound the expected true risk of our algorithm.

Theorem 4.1 (On average bound). For any positive, convex, k-lipschitz loss and for M T optimal solution of Problem (4.1) when learning with the training set T , we have:

E T ∼D T [L T (M T )] ≤ L T (M S ) + 8k 2 λn (4.5)
where the expected value is taken over size-n training sets.

Proof. Let T be any training set of size n, we have:

E T ∼D T [L T (M T )] = E T ∼D T [L T (M T )] + E T LT (M T ) -E T LT (M T ) = E T ∼D T LT (M T ) + E T L T (M T ) -LT (M T ) (E T ∼D T L T (M T ) -LT (M T ) ≤ 8k 2 λn (Lemma 4.1).) ≤ E T ∼D T LT (M T ) + 8k 2 λn ( LT (M T ) ≤ LT (M T ) + λ M T -M S 2 F ≤ LT (M S ) + λ M S -M S 2 F .
)

≤ E T ∼D T LT (M S ) + 8k 2 λn .
Noting that E T ∼D T LT (M S ) = L T (M S ) gives the theorem.

This bound shows that with a sufficient but rather reasonable number of examples, i.e. with a convergence rate in O(1/n), we will on average obtain a metric which is at least as good as the source hypothesis. It underlines the interest of having a good source metric. However this notion of goodness depends on the true risk of the source metric on the target domain:

G 1 (M S ) . = L T (M S ).
This quantity can not be computed as it depends of the unknown distribution D T and thus it cannot be used to explicitly choose a good metric. Furthermore the stability condition considered here is in expectation over all the possible training sets. It implies that it will probably not be possible to obtain an empirical measure of goodness in this particular setting.

In the next section we propose to address this problem and we consider the different but related framework of uniform stability to derive a generalization bound where the goodness of the source metric is empirical and thus can be estimated.

Uniform Stability Analysis

The second framework that we propose to use to analyse metric hypothesis transfer learning is the uniform stability framework adapted to metric learning presented by [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF] (Section 2.4). In this section we will show that this framework allows us to derive a probabilistic generalization bound where, depending on the loss function, the goodness of the source metric can be empirically estimated.

Assumptions We make the following assumptions. First we only consider metrics as Mahalanobis distances parametrized by a metric M positive semi definite. Second the loss function has to be convex in M (Definition A.6), positive, k-lipschitz (Definition A.1) and (σ, m)-admissible (Definition A.2). Third the regularization term is the Frobenius norm (Section 1.4).

We can now present our generalization bound. We divide this section as follows. First we present the bound for general losses. Then we consider a particular example where we show that the goodness of the source metric can be empirically estimated and we deduce an approach to weight the importance of the source hypothesis in order to obtain a tighter generalization bound.

Generalization Bound for General Loss Functions

We now derive our generalization bound for general loss functions based on the work of [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF]. To this extent we use the McDiarmid's inequality (Theorem A.1) on the estimation error, i.e. the difference between the true risk and the empirical risk. Before that we show that our algorithm is uniformly stable with respect to Definition 2.1 in the next lemma.

E T ∼D T [R T ] ≤ 2β n . (4.6)
Proof. The proof of this lemma can be found in Appendix C.3.

This lemma shows that the expected value of the estimation error over all the possible training sets of size n is bounded. In the next lemma we show that the difference in estimation error between two training sets which only vary by one example is also bounded.

|R T -R T i | ≤ 2β + 4σ + 2m n . (4.7)
Proof. The proof of this lemma can be found in Appendix C.4.

Using the fact that our algorithm is uniformly stable, we can now derive generalization guarantees as stated in the following theorem.

Pr R T -E T ∼D T [R T ] ≥ ≤ 2 exp   - 2 2 n i=1 2β+4σ+2m n 2    ≤ 2 exp - 2 2 1 n (2β + 4σ + 2m) 2 .
Then, by setting:

δ = 2 exp - 2 2 1 n (2β + 4σ + 2m) 2
we obtain:

= (2β + 4σ + 2m) ln 2 δ 2n
and thus:

Pr R T -E T ∼D T [R T ] < > 1 -δ.
Then, with probability 1 -δ: .11) This bound shows that with a convergence rate in O 1

R T < E T ∼D T [R T ] + (4.9) ⇔ L T (M T ) -LT (M T ) < E T ∼D T [R T ] + (4.10) (Lemma 4.3.) ⇒ L T (M T ) -LT (M T ) < 2β n + (2β + 4σ + 2m) ln 2 δ 2n . ( 4

√

n the true risk of our algorithm is bounded above by its empirical risk. In the next section, we consider a particular example of loss function where we show that the goodness of the source metric can be empirically estimated. This extension allows us to introduce a natural weighting of the source metric in order to improve the proposed bound.

Refinement with Weighted Source Hypothesis

In this section we propose to study the problem of weighting the source metric to improve the generalization bound. However in its current form the generalization bound does not explicitly include any information about the source metric. In the following we notice that, for a particular loss, the goodness of the source metric appears in the (σ, m)-admissibility of this loss.

First of all we consider the loss presented in Example 4.1) instantiated with the hinge function. For any metric M and any two labelled examples z, z ∼ D T we have:

l M, z, z = δ yy (x -x ) T M(x -x ) -γ yy + (4.12)
where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between the examples. In Example 4.1, given that the hinge loss is 1-lipschitz, we show that this loss is positive, convex, k-lipschitz with:

k = sup z,z ∼D T x -x 2 2
and (σ, m)-admissible with:

σ = sup z,z ∼D T γ yy , m = 2 sup z,z ∼D T x -x 2 2   LT (M S ) λ + M S F   .
Using this we can now apply Theorem 4.2, to obtain a generalization bound which includes a measure of the goodness of the source metric.

Theorem 4.3 (Generalization bound with Loss (4.12) ). For any matrix M T learned with Problem (4.1) with the training set T and Loss (4.12), we have with probability 1 -δ:

L T (M T ) ≤ LT (M T ) + 8D 2 λn +   8D 2 λ + 4 sup z,z ∼D T γ yy + 4D   LT (M S ) λ + M S F     ln 2 δ 2n .
where

D = sup z,z ∼D T x -x 2 2 .
Proof. This theorem comes from the application of Theorem 4.2 with specific values of k, σ and m.

This theorem is a refinement of Theorem 4.2 in the case of a specific loss. Hence the convergence rate is still in O 1 √ n and one of the most important difference is the presence of the term:

G 2 (M S ) . =   LT (M S ) λ + M S F   .
This term can be seen as a measure of the goodness of the source metric. It mainly depends on the quality of the source hypothesis M S . The product G 2 (M S ) ln( 2δ ) 2n which appears in the bound implies that as the number of examples available for learning increases, the quality of the source metric is of decreasing importance. A similar result has already been stated in domain adaptation or transfer learning in [START_REF] Shai Ben-David | A theory of learning from different domains[END_REF]; [START_REF] Kuzborskij | Stability and hypothesis transfer learning[END_REF] where they show that as the number of target examples increases, the necessity of having source examples decreases.

Given a source hypothesis M S , it is possible to optimize it with respect to the bound derived in Theorem 4.3. Indeed, note that G 2 (M S ) corresponds to a trade-off between the complexity of the source metric and its performance on the training set. The lower the value of this term, the tighter the bound.

Following this, we propose a way to minimize the right hand side of the generalization bound and more specifically G 2 (M S ) by adding a weighting parameter 0 ≤ ω T ≤ 1 on the source metric M S . This parameter is a simple way to control the trade-off between complexity and performance of the source metric thanks to a reweighting. It can be assessed by means of the following optimization problem:

ω T = arg min 0≤ω≤1 C(ωM S ) (4.13)
Note that the bound derived in Theorem 4.3 holds whatever the value of M S . Thus replacing it with ω T M S does not impact the theoretical study proposed in this section.

Interpretation of the value of ω T We can distinguish three main cases. First if the source hypothesis performs poorly on the training set at hand we expect ω T to be as small as possible to reduce the importance of M S . In a sense, we tend to go back to the classical case were M S = 0. Second if the source hypothesis is complex and performs well, we expect ω T to be rather small to reduce the complexity of the hypothesis while keeping a good performance on the training set. Third if the source hypothesis is simple and performs well, we expect ω T to be closer to one since M S is already a good choice. Note that we choose ω T ≤ 1 to limit the potential increase in complexity of the learned matrix.

Learning ω T Problem (4.13) is highly non differentiable1 and non convex. However, it remains simple in the sense that we have only one parameter to estimate and we used a classical sub-gradient descent to solve it. Even if it is not convex, our empirical study (Section 4.8 shows no need to perform many restarts to output a good solution: we always found almost the same solution. As a consequence, we applied only one optimization procedure in our experiments. Note that ω T is influenced by both the values of the margin and the regularization parameter and thus should be tuned accordingly each time. However, computing ω T by solving Problem (4.13) is not too costly. The process can even be sped up by computing the value of the source metric between the examples beforehand.

In this section, using the uniform stability framework for metric learning we have shown that our approach generalizes well with a convergence rate in O 1 √ n . Furthermore, given a specific loss, we have shown that the use of a weighting parameter to control the importance of the source metric is theoretically founded. Indeed it boils down to optimizing a notion of goodness of the source metric for the problem at hand. However the right hand side of the bound derived in Theorem 4.3 does not go to 0 when the source metric is a perfect fit for the problem at hand, i.e. it is not a vanishing bound. It implies that even with the perfect source hypothesis we might learn a metric which performs slightly worse than the source metric in terms of generalization to new examples. In the next section we consider the Rademacher complexity framework to derive a vanishing generalization bound when the source metric is a perfect fit.

Rademacher Complexity Analysis

The third theoretical framework that we consider in this chapter is the Rademacher complexity framework adapted to metric learning by [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF]. More precisely we further adapt this approach to take into account the source metric. It allows us to obtain a vanishing generalization bound which implies that when the source metric is a perfect fit, learning is no longer necessary.

Assumptions In this section instead of only considering the Mahalanobis distance we consider all the metrics parametrized by a matrix M ∈ M, and denoted by the function k M , such that given two vectors x and x the metric can be written as:

k M x, x = g(x, x ), M (4.14)
where g is a function over a pair of examples and •, • is the Frobenius product2 . If this definition seems restrictive, it is in fact fulfilled by the main metrics used in Metric Learning (See Section 2.1) such as the squared Mahalanobis distance:

(x -x ) T M(x -x ) = (x -x )(x -x ) T , M , (4.15)
or the bilinear similarity:

x T Mx = xx T , M . (4.16)
The loss function has to be positive, convex (Definition A.6) and k-lipschitz with respect to the metric (Definition 4.2) which is a slight adaptation of the k-lipschitzness presented in Definition A.1. The regularization term has to be convex and must have a well defined dual norm (Definition A.4).

We can now derive our generalization bound based on the Rademacher complexity framework. First of all we present our notion of k-lipschitz continuity with respect to the metric.

Then we present the notion of Rademacher complexity considered here along with a refinement of the metric class M considered. These definitions allow us to state our generalization bound linked to the Rademacher complexity of the metric class. Next we bound this Rademacher complexity showing that the bound depends on the source metric. We then discuss the implications of the bound and we show that if the source metric M S is a good fit then the rate of convergence is improved.

l M, z, z -l M , z, z ≤ k k M x, x -k M x, x (4.17)
This k-lipschitz continuity property ensures that given two metrics the difference in losses is bounded by the difference between the metrics. If the loss is differentiable with respect to the metric it can also be seen as a bound on the magnitude of the first derivative Srebro et al. (2010a). Note that the k-lipschitz property is usually considered with respect to the parameters of the metric rather than the metric as it is the case here. However our definition lipschitzness implies the standard definition when metrics of the form presented in Equation (4.14) are considered.

The Rademacher complexity used here is an adaptation of the definition given in [START_REF] Lei | Generalization analysis of multi-modal metric learning[END_REF] to the case of metric hypothesis transfer learning. The idea is to take into account the source metric.

Rn (M) . = 1 n 2 E σ sup M∈M n 2 i=1 σ i k M-M S x i , x n 2 +i (4.18)
and the Rademacher Complexity of M as:

R n (M) = E T ∼D T Rn (M). (4.19)
Instead of considering the complexity of a metric class with respect to its ability to fit random noise [START_REF] Lei | Generalization analysis of multi-modal metric learning[END_REF], this definition measures the complexity of the metric class with respect to its ability to differ from the source metric.

We now define formally the refinement of the metric class considered in our analysis and taking into account the source metric. Definition 4.4 (Metric class and source metric). We define a metric class dependent on M S as follows:

M S = M ∈ M : M -M S ≤ G 3 (M S ) λ (4.20)
where G 3 (M S ) = sup z,z ∼D T l (M S , z, z ) and M is the metric class used in Problem (4.1).

We now prove that any metric M T learned by Problem (4.1) belongs to M S . Proof. By the convexity of the loss and the optimality of M T we have:

LT (M T ) + λ M T -M S 2 ≤ LT (M S ) (Positive loss.) ⇒ λ M T -M S 2 ≤ LT (M S ) ⇒ M T -M S ≤ LT (M S ) λ ( LT (M S ) ≤ sup z,z ∼D T l (M S , z, z ).) ⇒ M T -M S ≤ G 3 (M S ) λ with G 3 (M S ) = sup z,z ∼D T l (M S , z, z ).
Noting that M T ∈ M gives the lemma.

To prove our generalization bound based on the Rademacher complexity of the source metric, we use the McDiarmid's inequality (Theorem A.1) and follow a similar strategy as in [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF]. Let R T = sup M∈M S L T (M) -LT (M) be the estimation error for Problem (4.1) when learning with training set T . Note that this estimation error is different from the one used in the previous sections. Indeed here we consider the worst error over the whole metric class rather than the error of the learned hypothesis. To apply McDiarmid's inequality we need to bound E T ∼D T [R T ] and |R T -R T i |. This is done in the two following lemmas. we have:

E T ∼D T [R T ] ≤ 2kR n (M S ).
Proof. The proof of this lemma can be found in Appendix C.5.

This lemma shows that the expected value of the estimation error over all the possible training sets of size n is bounded. In the next lemma we show that the difference in estimation error between two training sets which only vary by one example is also bounded.

|R T -R T i | ≤ 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S )
λ n where • * is the dual norm of the regularization term (Definition A.4).

Proof. The proof of this lemma can be found in Appendix C.6.

We can now present our generalization bound.

L T (M T ) ≤ LT (M T ) + 2kR n (M S ) + 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ ln 2 δ 2n .
Proof. Coupling Lemma 4.7 with McDiarmid's inequality (Theorem A.1) applied on the estimation error R T = sup M∈M S L T (M) -LT (M) we have with probability 1 -δ:

R T ≤ E T ∼D T [R T ] + 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ ln 2 δ 2n (Lemma 4.5.) L T (M T ) -LT (M T ) ≤ E T ∼D T [R T ] + 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ ln 2 δ 2n (Lemma 4.6.) L T (M T ) -LT (M T ) ≤ 2kR n (M S ) + 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ ln 2 δ 2n .
This generalization bound shows that the generalization ability of a metric learned with Problem (4.1) depends on the Rademacher complexity of the source metric class and on G 3 (M S ), its worst possible error over the distribution. In the next subsection we show that the Rademacher complexity of the source metric class also depends on G 3 (M S ). This value is thus a good candidate to measure the goodness of the source metric.

Rademacher Complexity and Source Metric

One of the critical quantities in the bound presented in Theorem 4.4 is the Rademacher complexity of the source metric class M S which depends on the source metric M S . In the next lemma we show that the Rademacher complexity of the source metric class depends on G 3 (M S ). Lemma 4.8 (Bounding the Rademacher complexity of M S ). Let M S be a metric class which depends on a source metric M S as in Definition 4.4 then we have that:

R n (M S ) ≤ G 3 (M S ) λ R n ( • * ) with: R n ( • * ) = E T ∼D T 1 n 2 E σ n 2 i=1 σ i g(x i , x n 2 +i ) * (4.21)
where • * is the dual norm of the regularization term (Definition A.4).

Proof. We consider the empirical Rademacher complexity of the metric class M S :

Rn (M S ) = 1 n 2 E σ sup M∈M S n 2 i=1 σ i k M-M S x i , x n 2 +i
(Equation (4.14).)

≤ 1 n 2 E σ sup M∈M S n 2 i=1 σ i g(x i , x n 2 +i ), M -M S (Trace linearity.) ≤ 1 n 2 E σ sup M∈M S n 2 i=1 σ i g(x i , x n 2 +i ), M -M S
(Cauchy-Schwarz's inequality (Theorem A.3).)

≤ 1 n 2 E σ sup M∈M S n 2 i=1 σ i g(x i , x n 2 +i ) * M -M S (M ∈ M S (Lemma 4.4).) ≤ G 3 (M S ) λ 1 n 2 E σ n 2 i=1 σ i g(x i , x n 2 +i ) *
Taking the expectation over all size n training sets on both sides of the last inequality gives:

R n (M S ) ≤ G 3 (M S ) λ E T ∼D T 1 n 2 E σ n 2 i=1 σ i g(x i , x n 2 +i ) * . Setting R n ( • * ) = E T ∼D T 1 n 2 E σ n 2 i=1 σ i g(x i , x n 2 +i )
Rademacher complexity and source metric M S . We have that M S ∈ M S and the metric class is centred 3 around M S with the radius being dependent on the worst case performance of M S , i.e. G 3 (M S ). It means that changing M S will impact the metric class and, consequently, its Rademacher complexity. On the one hand if the source metric is good, i.e. if G 3 (M S ) is low, then the considered metrics cannot go too far away from the source and thus the Rademacher complexity will be small. On the other hand if the source metric is bad, i.e. if G 3 (M S ) is high, then the considered metrics can go far away from the source and thus the Rademacher complexity will be higher.

We have shown how the source metric impacts the Rademacher complexity of the metric class and that G 3 (M S ) is a measure of goodness of the source metric. We now study the impact of the source metric on the rate of convergence of the bound presented in Theorem 4.4.

Goodness of M S

In this subsection we propose to study the bound presented in Theorem 4.4 with respect to the source metric M S . We show that when this source metric is good then it has a positive impact on the rate of convergence of the bound. In some case it might result in a faster rate of convergence in O 1 n . We also study the impact of the source metric on the number of examples needed to obtain a true risk at most equal to the empirical risk plus .

First of all note that the key quantity related to the source metric in the bound is:

G 3 (M S ) . = sup z,z ∼D T l(M S , z, z ).
We will use this quantity to define the goodness of a source metric. If it is small, respectively large, then the metric is good, respectively bad, for the problem at hand. This quantity represents the worst case loss of the source metric over all the examples.

Source metric and convergence. We consider the case where we want L T (M T ) ≤ LT (M T ) + and we seek n the number of examples needed to obtain such an convergence.

Let m ≥ 1 such that G 3 (M S ) ≤ 1 m and define α ∈ R such that R n ( • * ) ≤ α √ n then the bound in Theorem 4.4 gives: = 1 √ nm   2kα √ λ + 2 ln 2 δ √ 2 1 + sup z,z ∼D T 2k g(x, x ) * √ λ   (4.22)
which implies:

nm = 1 2   2kα √ λ + 2 ln 2 δ √ 2 1 + sup z,z ∼D T 2k g(x, x ) * √ λ   2 . ( 4

.23)

The quantity in the out-most brackets is a constant with respect to n, m and . It shows that a smaller can be obtained by increasing the number of examples or by using a better source metric. We can then consider several cases:

• m ≥ n: In this case the source metric is a good fit since G 3 (M S ) ≤ 1 n and the bound exhibits a fast rate of convergence with ≤ O 1 n . However this result is not fully informative in the sense that the constraint imposed on the source metric can be stronger than the bound, i.e. it might be better to directly use the source metric than to learn a new metric.

• m → ∞: In this case the source metric is a perfect fit, i.e. G 3 (M S ) = 0 and by convexity of the loss, the metric learned by Problem (4.1) is the source metric. It implies that the empirical risk is equal to 0 and since → 0 the right hand side of the bound also tends to 0 which implies that the bound is vanishing and translates the fact that no learning is necessary.

• m < n: In this case we have G 3 (M S ) > 1 n the bound is not worse than classic generalization bounds for metric learning as it exhibits a convergence rate in O 1 √ n .

Note that here we chose to bound the goodness of the source metric by a quantity which depends on the number of examples. It might be surprising in the sense that this measure is a constant with respect to M S . However it reflects the fact that when one has access to a sufficient number of examples, it is harder to obtain a meaningful source metric.

Comparison with similar bounds. Albeit not in the context of metric learning, the work presented in [START_REF] Kuzborskij | Learning by transferring from auxiliary hypotheses[END_REF] presents a generalization bound which is close to ours4 . However, the condition obtained to derive a fast rate is different. In this work, they propose to bound by O 1 n the true risk of the source hypothesis rather than the worst case loss. It might seem less restrictive as it considers the whole distribution and, thus, makes it easier to have a good source hypothesis. However it is important to note that, besides the fast rate, our condition allows us to guarantee that the empirical risk of the learned metric will be small. Indeed our condition implies that the error of the source metric will be low on the training set, then by convexity of the loss, the learned metric will have a better performance than the source metric. Such an analysis is not possible if we link the goodness of the source hypothesis to its true risk as it might happen that we have access to a training set where the empirical risk of the source hypothesis is greater than its true risk. Note also that their framework is more restrictive than ours since we allow to deal with possibly non smooth lispschitz functions and relaxes the strong convexity. While their result uses standard supervised learning losses, one question is to know if we can derive similar results for metric learning with pairwise losses. We cannot provide a clear answer but this issue is [START_REF] Bousquet | Concentration inequalities and empirical processes theory applied to the analysis of learning algorithms[END_REF] which requires to have some informations about the variance of the studied random variable. Its application to pairwise losses seems difficult and would require a specific study. In our framework, we used the McDiarmid's concentration inequality which does not require any information about the variance of the studied random variable but rather about the impact of a small change in the training set that is easier to consider in a metric learning context.

(MS ) ≤ O 1 √ n Convergence rate O 1 n O 1 √ n O 1 √ n Goodness Theoretical Empirical Theoretical G i (M S ) L T (M S ) LT (M S ) λ + M S F sup z,z ∼D T l(M S ,
In this section we have presented a vanishing generalization bound showing that a good source metric is beneficial and can significantly increase the rate of convergence of the bound. Furthermore when the metric is a perfect fit the bound shows that learning is not necessary. In the next section we propose a comparison of the bounds and a discussion on their implications.

Summary of the Bounds

In this section we propose a summary of the bounds derived in the previous sections. We recall their main characteristics in Table 4.1.

Nature of the bound In the average stability case the bound obtained is an exact one since its derivation does not rely on any concentration inequality. In the uniform stability and Rademacher complexity cases the bounds are probabilistic since they come from the application of the McDiarmid inequality (Theorem A.1).

Impact of the complexity of the metric class In the Rademacher complexity case the impact of the complexity of the metric class is in

O 1 √ n while this impact is improved to O 1 √ n
for the two stability based bounds. It can be explained by the fact that the Rademacher complexity bound measures the complexity of the metric class with respect to all the possible metrics in the class, even the ones which might never be learned by the algorithm. Hence it is a worst case result. On the other hand the stability based approaches only consider the metrics which can be learned given a training set. Hence it is closer to an exact result.

Convergence rate In the average stability case the goal is to bound the expected true risk of the algorithm over all the possible training sets while in the uniform stability and Rademacher complexity case the goal is to bound the true risk of a metric learned on any particular training set, in other words these bounds should also hold in the worst case scenario. It explains why the convergence rate is better in the average stability case.

Goodness of the source metric On the one hand in the average stability and the Rademacher complexity cases the measure of goodness of the source metric is theoretical and cannot be computed in practice making it unfit to derive an algorithm to choose a good source metric. On the other hand in the uniform stability case, the quantity involving the source metric is empirical. Following this we derived an algorithm to weight the importance of the source metric with respect to its goodness.

Applicability of the bound

The three bounds require different assumptions on the metric, the loss function and the regularization. The Rademacher complexity bound is the least constrained of the three approaches as it is applicable to several kind of regularization terms and can be used with different kind of metrics. The two stability bounds are more constrained as they only hold for a Mahalanobis distance learned with a Frobenius norm regularization term.

In the next section we propose several examples of loss functions and regularization terms which can be used in one or several of our frameworks.

Examples

In this section we present several examples of popular loss functions and regularization terms in metric learning. We show that each example falls in one or more of the theoretical frameworks presented here. It demonstrates the wide range of applicability of the metric hypothesis transfer learning framework proposed here.

First of all note that in Sections 4.3 and 4.4 we only consider learning a Mahalanobis distance and this distance respects the assumption made on the metric of Section 4.5 ). Hence in this section we only consider metrics which satisfy this property. Furthermore we always consider positive and convex losses. It implies a bound on the regularization term presented in the next lemma. Lemma 4.9 (Bounded regularization). Let M T be the optimal solution returned by Problem (4.1) with training set T and a positive and convex loss. We have:

M T -M S ≤ LT (M S ) λ .
Proof. The proof of this lemma can be found in Appendix C.7.

Examples of Loss Functions

Overall we successively considered the following properties for the loss function: First we propose to consider L-lipschitz functions for dissimilarity and similarity learning. For each example we prove that all the previous properties hold. where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k = L,

• k-lipschitz continuous with k = L sup z,z ∼D T g(x, x ) * ,

• (σ, m)-admissible with

   σ = sup z,z ∼D T γ yy m = 2L sup z,z ∼D T g(x, x ) * LT (M S ) λ + M S .
Proof. The proof of this example can be found in Appendix C.8.

This loss has, for example, been successfully used in [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF]. Similarly if f (a) = [a] + then it corresponds to the loss used in Section 4.

l M, z, z = f 1 -δ yy k M (x, x ) γ yy (4.25)
where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k

= L inf z,z ∼D T |γ yy | , • k-lipschitz continuous with k = L inf z,z ∼D T |γ yy | sup z,z ∼D T g(x, x ) * , • (σ, m)-admissible with    σ = 0 m = 2 L inf z,z ∼D T |γ yy | sup z,z ∼D T g(x, x ) * LT (M S ) λ + M S .
Proof. The proof of this example can be found in Appendix C.9.

This loss has, for example, been successfully used before with f (a) = [a] + in [START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF]; [START_REF] Nicolae | Joint semisupervised similarity learning for linear classification[END_REF] albeit in a slightly different context.

We now turn our interest to H-smooth (Definition A.3), B-bounded functions as defined in Srebro et al. (2010b). Note that the results are close in spirit to the one obtained for L-lipschitz functions. where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k = √ 12HB,

• k-lipschitz continuous with k = √ 12HB sup z,z ∼D T g(x, x ) * , • (σ, m)-admissible with σ = 0 m = B .
Proof. The proof of this example can be found in Appendix C.10. where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k =

√ 12HB inf z,z ∼D T |γ yy | , • k-lipschitz continuous with k = √ 12HB inf z,z ∼D T |γ yy | sup z,z ∼D T g(x, x ) * , • (σ, m)-admissible with σ = 0 m = B .
Proof. The proof of this example can be found in Appendix C.11.

Smooth losses have, for example, been successfully used with f (a) = (a) 2 in Srebro et al. (2010a); [START_REF] Kuzborskij | Learning by transferring from auxiliary hypotheses[END_REF] albeit in the different context of classification.

We have presented several loss functions which can be used in our framework. In the next subsection we turn our attention toward several regularization terms.

Examples of Regularizations

Due to several technical issues we have to use the Frobenius norm when considering the two stability frameworks. However in the Rademacher complexity framework the only constraint is that the dual norm of the regularization should be well defined, i.e. its Rademacher average should be bounded above by a term which decreases in O 1 √ n . [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] have shown that this condition is fulfilled by several norms that we recall in Table 4.2 7 . These norms have been successfully used before as non biased regularization terms [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF][START_REF] Ying | Sparse metric learning via smooth optimization[END_REF][START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF][START_REF] Shen | Positive semidefinite metric learning using boosting-like algorithms[END_REF][START_REF] Nicolae | Joint semisupervised similarity learning for linear classification[END_REF] but also as biased regularization terms [START_REF] Parameswaran | Large margin multi-task metric learning[END_REF]Cao et al., 2013a;[START_REF] Bohné | Large margin local metric learning[END_REF].

We now propose to discuss the impact of the regularization term on the transfer taking place between the source metric M S and the learned metric M T . First of all note that this differ with the kind of metric considered as the interpretation of the values of the matrix may change. Here we consider that we are learning a Mahalanobis distance or a Bilinear similarity where each entry of the matrix can be seen as a measure of the importance of the relation between two features. Hence it gives us the following possible interpretations for the different norms.

• Frobenius norm: The Frobenius norm will encourage small element-wise changes in the values of the matrix M T with respect to the matrix M S . It implies that the relations between the features will keep the same order of magnitude and only slightly change.

• 1 -norm: The 1 -norm is an element-wise sparsity inducing norm. It implies that from the source metric to the learned metric only a limited number of entries in the matrix will change. It means that some of the relations between features will be kept the same while the other relations will be able to change more than with a Frobenius norm.

• 2,1 -norm: The 2,1 -norm is a column wise sparsity inducing norm. It implies that this norm will encourage the learned metric M T to keep intact whole rows (and columns if the matrix is symmetric) of M S . In other words, for some features, the relationships between this feature and the others as encoded by the source metric will be kept in the learned metric.

• Trace norm: The interest of using a trace norm is mainly to obtain low rank matrices.

In the case of biased regularization it translates into obtaining a low rank difference between M T and M S . However it does not always imply that the rank difference between the learned metric and the source metric will be small. It seems that, in a biased regularization case, the trace norm is less interesting than the other norms.

• F • F R n ( • * ) ≤ 2 sup z,z ∼D T g(x,x ) 2 F √ n • 1 • ∞ • 2,1 • 2,∞ • Tr • Spec
In this section we have presented several examples of loss functions and regularization terms which can be used in our framework. In the next section we propose to empirically study it in several experiments. On the one hand we propose to evaluate the interest of optimizing the goodness of the metric as defined in Section 4.4. On the other hand we demonstrate that using our approach in a semi-supervised domain adaptation task leads to state of the art results.

Experiments

In Section 4.4, instantiating Problem (4.1) with the hinge loss (Example 4.1) and the Frobenius norm, we have derived an empirical measure of the goodness of a metric. It leads to the development of an optimization problem to learn the weight of the source metric (Equation (4.13)). In this section we consider two empirical studies depending on the choice of the source metric. First, using some well-known distances as a source metric, we show that our framework performs well on classic supervised metric learning tasks of the UCI database and we empirically demonstrate the interest of learning the ω parameter. Second, we apply our framework with weighted source metric in a semi-supervised Domain Adaptation task. We show that, using only source information through a learned metric, our method is able to compete with state of the art algorithms which have access to the examples of the source dataset.

Setup In all our experiments we use limited training dataset, making it difficult to apply any kind of cross-validation to set the parameters. Thus we propose to fix them as follows. First the positive and negative margin are respectively set to the 5 th and 95 th percentile of the training set possible distances computed with the source metric as proposed in [START_REF] Davis | Informationtheoretic metric learning[END_REF]. Next we set λ such that the two terms of Equation (4.13) are equals, i.e. we balance the complexity and performance importance with respect to the source metric. The ω parameter is then learned using Problem (4.13). In all the experiments we plug our metric in a 1-nearest neighbour classifier to classify the examples of the test set. Furthermore, the significance of the results is assessed with a paired samples t-test considering that an approach is significantly better when the p-value is lower than 0.05.

Classic Supervised Metric Learning

First we start by conducting experiments on several UCI datasets [START_REF] Lichman | UCI machine learning repository[END_REF], namely breast, pima, scale and wine. We propose to consider three source metrics:

• Zero: No source hypothesis,

• Identity: Euclidean distance,

• Mahalanobis: Inverse of the variance-covariance matrix computed on the training set [START_REF] Chandra | On the generalized distance in statistics[END_REF].

For the last two source metric we propose two experiments, one where we set ω = 1 and one where we learn ω using Problem (4.13). The goal of this experiment is to show the interest of automatically setting ω. We consider a 1-nearest neighbour (1-NN) classifier using the Euclidean Distance as the baseline and also report the results of two well known metric learning algorithms, namely ITML [START_REF] Davis | Informationtheoretic metric learning[END_REF] and LMNN [START_REF] Kilian Q Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF].

The results averaged over 10 runs are reported in Table 4.3. For each run we randomly draw a training set containing 20% of the data available for each class and we test the metric on the remaining 80% of data. These experiments highlight the interest of learning the ω parameter. When we consider the performance of our approach with and without learning ω, we mainly notice the following facts. First, learning ω always leads to an improvement on all the datasets and the final result is better than the 1-NN classifier. Second, learning ω when considering the identity matrix as the source metric seems to be of limited interest as the differences in accuracy are only significant for the wine dataset. This can be justified by the fact that, in this case, it only consists of a rescaling of the diagonal of the matrix and it does not change much the behaviour of the distance. Finally, learning ω when considering the variance-covariance matrix as the source metric leads to a significant improvement of the performance of the metric except on the breast dataset. This is particularly true for the wine dataset with a gain of nearly 14% in accuracy. It can be explained by the fact that, for this dataset, we are learning with less than 40 examples. Thus the original Mahalanobis distance does not carry as much information as in the other datasets and is thus of a lower quality. Learning ω allows us to compensate this drawback and to obtain results which are even better than ITML or LMNN.

Metric Learning for Semi-Supervised Domain Adaptation

In this section we consider a semi-supervised domain adaptation task with the Office-Caltech dataset. This dataset consists of four domains: Amazon (A), Caltech (C), DSLR (D) and Webcam (W) for which 10 classes are considered. This leads to consider 12 different adaptation problems when we alternatively take each domain as the source or the target dataset. The results are averaged over 20 runs. In each run the training set is composed of 8 labelled source examples (20 if the source is Amazon) and 3 labelled target examples for each class. The testing set corresponds to the remaining target examples. In these experiments we use the same splits as the ones considered in [START_REF] Hoffman | Efficient learning of domain-invariant image representations[END_REF] since they are freely available from the authors website and we follow their experimental setup. The data is normalized thanks to the zscore and the dimensionality of the examples is reduced to 20 thanks to a simple PCA. The results are presented in Table 4.48 where we compare the performance of our method to 6 baselines: • GFK: another DA approach [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF].

The last two methods need the source sample while in our case we only use a source metric learned from the source instances. For our biased regularization framework we consider 3 possible metrics learned on the source examples, namely Mahalanobis, ITML and LMNN. These source metrics are weighted by ω T which is learned using Problem (4.13). These results show that metric hypothesis transfer learning can perform well in a semisupervised domain adaptation setting. Indeed, we perform better than directly plugging the metrics learned by LMNN and ITML in a 1-NN classifier. Moreover, we obtain accuracies which are competitive with state of the art approaches like MMDT or GFK while using less information. If we compare our approach using LMNN as the source metric with MMDT, we note that MMDT is significantly better than our approach on 4 out of 12 tasks while we are significantly better on 3 and 5 end as a draw. Hence we can conclude that our method presents a similar level of performance than MMDT. Similarly, if we compare our approach using LMNN as the source metric with GFK, we obtain that GFK is significantly better than Table 4.4: Metric Learning for Semi-Supervised Domain Adaptation. For the sake of readability we design the considered domains by their initials. S → T stands for adaptation from the source domain to the target domain. Each time we consider the mean and standard deviation over 20 runs. For each task, the best result is highlighted with a bold font.

Baselines

Our approach Task 1-NNS our approach on 3 tasks, we are significantly better on 7 and 2 lead to a draw. Hence, we can conclude that our approach performs better than GFK.

If we compare the performances of both ITML and LMNN as metrics used directly in a nearest neighbour classifier one can intuitively expect ITML to be a better source hypothesis than LMNN since its results are better. However, in practice, using the metric learned by LMNN as the source hypothesis yields better results. This suggests that using a learned source model that tends to over-fit reasonably the learning source sample can be of potential interest in a transfer learning context. Indeed LMNN does not use a regularization term in its formulation is thus prone to over-fitting. Since the parameter ω penalizes the source metric with respect to its complexity it may limit the impact of the source metric to what is needed for the transfer. Nevertheless, this point deserves further investigation.

Conclusion

In this chapter we formalised and theoretically analysed the metric hypothesis transfer learning framework. This framework takes into account a source hypothesis information to help learning by means of a biased regularization. This regularization can be interpreted into two ways: (i) when the source metric is an a priori known metric such as the identity matrix, the objective is to infer a new metric that performs better than the source metric, (ii) when the source metric has been learned from another domain, the formulation allows one to transfer the knowledge from the source metric to the new domain. This last interpretation refers to a transfer learning setting where the learner does not have access to source examples and can only make use of the source model in the presence of few labelled data.

In our theoretical analysis we considered three different frameworks. First the on average stability framework allowed us to derive an exact bound showing that with a convergence rate in O 1 n the learned metric will, on average over all the size n training sets, be as good as the source metric. Second the uniform stability framework leads to a probabilistic generalization bound where, given a specific loss, an empirical measure of the goodness of the source metric can be obtained. From this we proposed an algorithm to optimally weight the source metric in order to optimize the bound in a theoretically sound way. Third we used the Rademacher complexity framework to address both problems of considering different regularization terms and obtaining a vanishing bound, i.e. a bound which implies that learning is no longer necessary when the source metric is the perfect fit.

To further demonstrate the interest of metric hypothesis transfer learning we proposed several examples of loss functions and regularization terms which can be used with our theoretical analysis. We also discussed the impact of the different regularization terms on the transfer of informations between the learned metric and the source metric. Lastly we proposed an empirical evaluation of our source metric weighting method. On the one hand we considered a classic metric learning task where we showed that weighting the source metric to minimize the bound is indeed beneficial. On the other hand, in a semi-supervised domain adaptation task we demonstrated the good behaviour of the metric hypothesis transfer learning framework. Indeed we obtained results comparable to state of the art approaches which fully make use of the source examples while we only have access to the source metric.

As stated in [START_REF] Kuzborskij | Learning by transferring from auxiliary hypotheses[END_REF] in another context, our results stress the importance of choosing a good source hypothesis. Perspectives of this work include further empirical investigations on the interest of using metric hypothesis transfer learning. In particular empirically studying the impact of the regularization on the transferred informations between the source and learned metric could be of interest. Here we considered the case where we have a single source metric. One interesting perspective could be to consider more complex strategies to learn ω and, for example, to study the multi-source case where several source metrics are available at once. Note that one of the main limitations of this work is the fact that we did not manage to obtain a vanishing bound highlighting an empirical goodness criterion for the source metric. Indeed in Section 4.4 we obtained an empirical criterion but the bound is not a vanishing one while in Section 4.5 we obtained a vanishing bound but the associated goodness criterion is theoretical in the sense that it cannot be computed. Hence a perspective could be to develop a vanishing bound with empirical goodness. It would imply obtaining a theoretically sound way to choose the source metric. Furthermore it would probably give further insights on the trade-off between goodness of the source metric and number of examples.

In the first part of this thesis we have been interested in learning in the presence of an auxiliary metric serving as a reference during the learning process. On the one hand in Chapter 3 we studied the case where this metric is only available through its values and we want to learn a good approximation. On the other hand in Chapter 4 we considered that this metric is accessible through its parameters matrix and that it can help us learning a better metric for a given task. In other words in this first part we constrained the learned metric to have a behaviour similar to the reference metric. Hence we implicitly, through the reference metric, controlled the behaviour of the learned metric. In the next part of this thesis we propose to address the problem of explicitly controlling the behaviour of a metric. To this end in Chapter 5 we propose a new metric learning framework based on virtual points. In other words instead of learning a metric based on similarity constraints between examples we propose to explicitly control the target position of the examples by bringing them closer to these virtual points.

Part III

Metric Learning with Controlled Behaviour

Chapter 5

Regressive Virtual Metric Learning

This chapter is based on the following publication Michaël Perrot and Amaury Habrard. Regressive virtual metric learning. In Advances in Neural Information Processing Systems (NIPS-15), pages 1810-1818, 2015c

Abstract

In this chapter we are interested in supervised metric learning of Mahalanobis like distances. Existing approaches mainly focus on learning a new distance using similarity and dissimilarity constraints between examples. Here instead of bringing closer examples of the same class and pushing far away examples of different classes we propose to move the examples with respect to virtual points. Hence, each example is brought closer to an a priori defined virtual point reducing the number of constraints to satisfy and explicitly controlling the expected behaviour of the metric for each example. We show that our problem admits a closed form solution which can be kernelized. We provide a theoretical analysis proving the generalization ability of the metric learned with our approach and establishing some links with other classic metric learning methods. Furthermore we propose an efficient solution to the difficult problem of selecting virtual points based in part on recent works in optimal transport. Lastly, we evaluate our approach on several state of the art datasets. set of constraints is quadratic in the number of examples which can be prohibitive when the number of examples increases. One heuristic is then to select only a subset of the constraints but selecting such a subset is not trivial.

Introduction

In this chapter, we propose to consider a new kind of constraints where each example is associated with an a priori defined virtual point. It allows us to consider the metric learning problem as a simple regression where we try to minimize the differences between learning examples and virtual points. Figure 5.1 illustrates the differences between our approach and a classic metric learning approach. It can be noticed that our algorithm only uses a linear number of constraints. However defining these constraints by hand can be tedious and difficult. To overcome this problem, we present two approaches to automatically define them. The first one is based on some recent advances in the field of optimal transport while the second one uses a class-based representation space.

Moreover, thanks to its regression-based formulation, our approach can be easily kernelized allowing us to deal efficiently with non linear transformations which is a nice advantage in comparison to some metric learning methods. We also provide a theoretical analysis showing the generalization ability of the metrics learned with our approach and establishing some relationships with a classic metric learning formulation. This chapter is organised as follows. In Section 5.2 we present our framework to learn a metric when the virtual points are known. Then in Section 5.3 we address the problem of selecting these virtual points. In Section 5.4 we theoretically analyse our approach by deriving a generalization bound and by showing some links with a classic metric learning approach. In Section 5.5 we empirically demonstrate the interest of our approach. We conclude in Section 5.6.

Learning a Metric Using Virtual Points

The main idea behind our algorithm is to bring closer the learning examples to a set of virtual points. Here we assume that we have access to a set of n learning pairs (x,v) where x is a learning example and v is a virtual point associated to x. We present both the linear and kernelized formulations of our approach called Regressive Virtual Metric Learning (RVML). It boils down to solve a regression in closed form, the main originality being the introduction of virtual points.

Given a probability distribution D T defined over X × Y where X ⊆ R d and Y is a finite label set, let T = {(x i , y i )} n i=1 be a set of examples drawn i.i.d. from D T . Let f V : X ×Y → V, where V ⊆ R d is the space of virtual points, be the function which associates each example to a virtual point. We consider the learning set

V = {(x i , v i )} n i=1 where v i = f V (x i , y i ).
We denote by D V the probability distribution defined on X × V obtained from the distribution

D T after applying f V , i.e. Pr D V (x, v) = Pr D T (x, y|v = f V (x, y)). Thus it is equivalent to obtain the set of examples V = {(x i , v i )} n
i=1 from T after applying f V and to draw V i.i.d. from D V . For the sake of simplicity denote by X = (x 1 , . . . , x n ) T and V = (v 1 , . . . , v n ) T the matrices containing respectively one example and its associated virtual point on each line.

We consider that the function f V is known. We come back to its definition in Section 5.3. Our goal is to learn a Mahalanobis distance through its linear transformation interpretation (Section 1.4). More precisely we want to learn the linear transformation matrix L such that M = LL T . We consider the following optimisation problem: arg min

L∈R d×d LV (L) + λ L 2 F (5.1)
where LV (L) is the empirical risk defined as follows:

LV (L) = 1 n (x,v)∈V l (L, (x, v)) with l (L, (x, v)) = x T L -v T 2 2 .
Note that we can also write the empirical risk in matrix form:

LV (L) = 1 n XL -V 2 F . (5.2)
We also define the true risk of a matrix L as:

L V (L) = E (x,v)∼D V l (L, (x, v)) . (5.
3)

The idea is to learn a new space of representation where each example is close to its associated virtual point. Note that L is a d × d matrix and if d < d we also perform dimensionality reduction.

Theorem 5.1 (Optimal solution of Problem (5.1)). The optimal solution of Problem (5.1) can be found in closed form. Furthermore, we can derive two equivalent solutions: .5) (the virtual points here). Hence, it is possible to compute the projection of an example x of dimension d in a new space of dimension d :

L V = X T X + λnI -1 X T V (5.4) ⇔ L V = X T XX T + λnI -1 V. ( 5
φ(x) L V k = φ(x) T Φ T X (K X + λnI) -1 V = k T X,x (K X + λnI) -1 V.
where k X,x = (k(x, x 1 ) , . . . , k(x, x n )) T is the similarity vector to the examples with respect to k. Recall that we are interested in learning a distance between examples and not in the prediction of the virtual points which only serve as a way to bring closer similar examples and push far away dissimilar examples.

In this section we presented our linear and kernelized metric learning approaches when the virtual points are given. If these points could be chosen by hand, we believe that an automatic solution would be preferable. We propose to address this problem in the next section.

Choosing the Virtual Points

Previously, we assumed to have access to the function f V : X × Y → V. In this subsection, we present two methods for generating automatically the set of virtual points and the mapping f V .

Using Optimal Transport on the Learning Set

In this first approach we propose to generate the virtual points associated to the examples of a training set T as follows. First we use a variation of the landmark selection method proposed in [START_REF] Kar | Similarity-based learning via data driven embeddings[END_REF] to choose in T a set T of n landmarks. Then we use a recent variation of the optimal transport problem proposed by Courty et al. (2014b) to associate each example z i ∈ T to the landmarks z 1 , . . . , z n ∈ T with weights W(i, 1), . . . , W(i, n ) such that: j W(i, j) = 1.

(5.11)

The virtual points are then defined as a weighted combination of the landmarks. Let X be the matrix form of T , for an example z i ∈ T we define f V as:

f V (x i , y i ) = W(i, )X .
(5.12)

In the following we present our landmark selection method based on the work of [START_REF] Kar | Similarity-based learning via data driven embeddings[END_REF] and the variation of the optimal transport problem (Courty et al., 2014b) that we consider.

Landmark Selection

To select the set T we propose an adaptation of the selection method of [START_REF] Kar | Similarity-based learning via data driven embeddings[END_REF] allowing us to take into account some diversity among the landmarks. Our approach is a fully automatic procedure and is summarized in Algorithm 2 and works as follows. First we assume without loss of generality that the examples are centred in 0 and we select as the first landmark the example z ∈ T furthest from 0. Each new landmark is selected as the example z ∈ T with the largest minimum distance with the landmarks in T . To avoid explicitly choosing the number of landmarks we propose to stop the selection process under two conditions:

• the number of landmarks is greater than the number of classes,

• the maximum distance between an example and a landmark is lower than the mean of pairwise distances between all the examples of T .

input : T = {(x i , y i )} n i=1 a set of examples; Y the label set. output: T a subset of T begin µ = mean of distances between all the examples of T x max = arg max x∈T x -0 2 T = {x max } T = T \ T ε = max x∈T min x ∈T x -x 2 while |T | < |Y| or ε > µ do x max = arg max x∈T x ∈T x -x 2 T = T ∪ {x max } T = T \ T ε = max x∈T min x ∈T
xx 2 end end Algorithm 2: Selecting T from a set of examples T .

Optimal Transport

Assume that you have an input distribution and an output distribution, the goal of optimal transport [START_REF] Villani | Optimal transport: old and new[END_REF] is to align the two distributions at a minimal cost. We come back to this general problem in Section 6.2 and instead we consider a particular case of discrete optimal transport where the idea is to transport the examples of an input set T toward the examples of an output set T at a minimal cost. Here we start by presenting the solution to this problem proposed by Courty et al. (2014b) before explaining how we use it to associate the training examples to the landmarks.

In the particular case of discrete optimal transport we assume that each example in T has a mass of 1 n where n is the number of examples in T . Similarly each example in T has a mass of 1 n . We also consider that we have a cost matrix C ∈ R n×n + where each entry C(i, j) represents the cost of moving example x i ∈ T toward example x j ∈ T . For example we can set C(i, j) = x i -x j 2

. The goal is then to learn a weight matrix Γ ∈ R n×n + which minimizes the transport cost Γ, C F between the two sets. Note that this matrix should take into account the mass associated with each example, i.e. it has to respect the constraints:

∀x i ∈ T, Γ(i, )1 n = 1 n , ∀x j ∈ T , Γ(, j)1 n = 1 n (5.13)
where 1 n and 1 n are vectors of 1 of size n and n respectively. To learn the matrix Γ Courty et al. (2014b) propose to use the following regularized optimization problem:

arg min

Γ∈R n×n + Γ, C F - 1 λ h(Γ) + η j c Γ(y i = c, j) p q s.t. ∀x i ∈ T, Γ(i, )1 n = 1 n , ∀x j ∈ T , Γ(, j)1 n = 1 n .
The two regularization terms have different objectives.

• -1 λ h(Γ) where h(Γ) =i,j Γ(i, j) log(Γ(i, j)) is the entropy of gamma: this regularization term has been proposed by Cuturi (2013b). It allows one to solve the transportation problem more efficiently by using the Sinkhorn-Knopp algorithm [START_REF] Philip | The sinkhorn-knopp algorithm: convergence and applications[END_REF]. Furthermore by setting the value of the parameter λ it is possible to control the sparsity of the matrix Γ. On the one hand if the matrix is sparse it implies that each example from T will be associated to a small number of examples in T . On the other hand if the matrix is full it implies that each example from T will be associated to each example in T .

• η j c Γ(y i = c, j) p q where Γ(y i = c, j) corresponds to the lines of the j th column of Γ where the class of the input is c: this term has been proposed by Courty et al. (2014b). Its goal is to prevent input examples of different classes to move toward the same output examples by promoting group sparsity in the matrix Γ. This is done thanks to the functions • p q corresponding to a q -norm to the power of p used here with q = 1 and p = 1 2 (See the p,q -norm in Section 1.4). Once the matrix Γ has been learned it is possible to compute the image xi of an input example x i ∈ T as follows: xi = n Γ(i, )X (5.14) where X is the matrix form of T with one example per line. In this case multiplying Γ by n ensures that j Γ(i, j) = 1 and thus the image xi can be seen as a linear combination of the output examples. Note that the transport might imply non linear transformations of the input space. Indeed there is no guarantee that there exists a matrix T such that ∀x i ∈ T, xi = Tx i .

Here we propose to use this optimal transport approach to learn the matrix Γ between the learning examples T and the landmarks T . We then obtain the weight matrix used to compute the virtual points as W = n Γ. Note that in this case our metric learning approach can be seen as a an approximation of the result given by the optimal transport1 .

Using a Class-based Representation Space

For this second approach, we propose to define virtual points as the unit vectors of a space of dimension |Y|, i.e. the number of classes in the problem. Let e j ∈ R |Y| be such a unit vector (1 ≤ j ≤ |Y|), i.e. a vector where all the attributes are 0 except for one attribute j which is set to 1, to which we associate a class label from Y. In this case the e j vectors can also be seen as the vertices of a standard (|Y| -1)-simplex. For any learning example (x i , y i ), we define f V (x i , y i ) = e #y i where #y i = j if e j is mapped with the class y i . Thus, we have exactly |Y| virtual points, each one corresponding to a unit vector and a class label.

We call this approach the class-based representation space method. If the number of classes is smaller than the number of dimensions used to represent the learning examples, then our method will also perform dimensionality reduction. Furthermore, our approach will try to project all the examples of one class on the same axis while examples of other classes will tend to be projected on different axes. The underlying intuition behind the new space defined by L V is to make each attribute discriminant for one class. The interest of this approach is illustrated in Figure 5.2.

In this section we proposed two methods to define the virtual points but other approaches could be considered. For example [START_REF] Kusner | Stochastic neighbor compression[END_REF] proposed a solution to compress a dataset by considering only a small number of examples in order to speed up nearest neighbours classification. Using this compressed dataset could be a way to define virtual points which summarize well the behaviour of the examples in each class. In the next section we show that our approach is theoretically founded.

Theoretical Analysis

In this section, we propose to theoretically show the interest of our approach by proving that the learned metric generalizes well, Section 5.4.1, and that it is possible to link it to a more classic metric learning formulation, Section 5.4.2.

Generalization Bound

In this section we show that a metric learned with Problem (5.1) generalizes well. To this extent we use the uniform stability framework presented in Section 1.3. In the following we assume that x 2 ≤ B x and v 2 ≤ B v . Before proving that our approach is uniformly stable we start by presenting two lemmas showing that our loss is bounded and k-lipschitz continuous (Definition A.1).

Lemma 5.1 (Bounded loss function). Let L V be the metric learned with Problem (5.1) with training set V , we have that for any example (x, v) ∼ D V :

l (L V , (x, v)) ≤ B with B = B 2 v 1 + Bx √ λ 2 .
Proof. The proof of this lemma can be found in Appendix D.2.

Lemma 5.2 (k-lipschitz continuity). Our loss is k-lipschitz with k = 2B v B x 1 + Bx √ λ .
Proof. The proof of this lemma can be found in Appendix D.3.

We can now show that our algorithm is uniformly stable (Definition 1.3).

Lemma 5.3 (Uniform stability). Our algorithm has a uniform stability in

β = 8B 2 v B 2 x λn 1 + Bx √ λ 2 .
Proof. The proof of this lemma can be found in Appendix D.4.

We can now prove our generalization bound.

Theorem 5.2 (Generalization bound). Let v 2 ≤ B v for any v ∈ V and x 2 ≤ B x for any x ∈ X . Let L V be the optimal solution of Problem (5.1). With probability 1 -δ we have:

L V (L V ) ≤ LV (L V ) + 8B 2 v B 2 x λn 1 + B x √ λ 2 + 16B 2 x λ + 1 B 2 v 1 + B x √ λ 2 ln 1 δ 2n .
Proof. This theorem is a direct application of Theorem 1.1 [START_REF] Bousquet | Stability and generalization[END_REF] using the bound on the loss presented in Lemma 5.1 and the uniform stability of our algorithm proven in Lemma 5.3.

We obtain a rate of convergence in O 1 √ n which is standard with this kind of bounds.

Kernelized case Recall that in the linear case we assumed that x 2 ≤ B x . In the kernelized case, we only have to bound φ(x) 2 where φ is the projection function associated to the used kernel. A common assumption [START_REF] Audiffren | Stability of multi-task kernel regression algorithms[END_REF] when studying kernels is that ∃κ such that 0 < κ < ∞ and k(x, x) ≤ κ 2 . Hence, we have φ(x) 2 2 ≤ κ 2 . Thus setting B x = κ allows us to use the same proof than in the linear case leading us to the same generalization bound (the only difference being the value of B x ).

Link with a Classic Metric Learning Formulation

In this section we show that it is possible to bound the true risk of a classic metric learning approach with the empirical risk of our formulation. Most of the classic metric learning approaches make use of a notion of margin between similar and dissimilar examples. Hence, similar examples have to be close to each other, i.e. at a distance smaller than a margin γ 1 , and dissimilar examples have to be far from each other, i.e. at a distance greater than a margin γ -1 . Let (x, y) and (x , y ) be two examples from X × Y, using this notion of margin, we consider the following loss [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF]: 5.15) where δ yy = 1 if y = y and -1 otherwise, γ yy = γ 1 if y = y and γ -1 otherwise and d is the standard euclidean distance. The latter is the desired margin between examples. As introduced before, we consider that γ yy takes a big value when the examples are dissimilar, i.e. when δ yy = -1, and a small value when the examples are similar, i.e. when δ yy = 1.

l (L, (x, y), (x , y )) = δ yy (d 2 L T x, L T x -γ yy ) + (
In the following we set, up to some constants, the margin between similar examples as the maximum distance between two virtual points associated to the same class and the margin between dissimilar examples as the minimum distance between two virtual points associated to different classes. Then we show that it is possible to bound, up to a constant factor, the true risk associated with the previous loss by the empirical risk of our approach.

Theorem 5.3 (Link with a classic metric learning approach). Let D T be a distribution over X × Y. Let V ⊂ R d be a finite set of virtual points and f V is defined as f

V (x, y) = v, v ∈ V. Let v 2 ≤ B v for any v ∈ V and x 2 ≤ B x for any x ∈ X . Let γ 1 = 2 max x,x ,y=y v -v 2 2 and γ -1 = 1 2 min x,x ,y =y v -v 2 2
. Let L V be the optimal solution of Problem (5.1), we have with probability 1 -δ:

E (x,y),(x ,y )∼D T δ yy (d 2 L T V x, L T V x -γ yy ) + ≤ 8   LV (L V ) + 8B 2 v B 2 x λn 1 + B x √ λ 2 + 16B 2 x λ + 1 B 2 v 1 + B x √ λ 2 ln 1 δ 2n )   .
Proof. First of all, let us consider two examples x and x and their associated virtual points v and v . Using the fact that distances respect the triangle inequality, one can obtain:

1 2 d 2 v, v + γ -1 + (γ -1 ≤ 1 2 d 2 (v, v ).) ≤ 4d 2 L T V x, v + 4d 2 v , L T V x .
(5.17)

Noting that we obtain the same inequality for similar and dissimilar examples and taking the expectation on both sides gives:

E (x,y),(x ,y )∼D T δ yy (d 2 L T V x, L T V x -γ δ yy ) + (5.18) ≤ E (x,y),(x ,y )∼D T 4d 2 L T V x, v + 4d 2 v , L T V x = E (x,y),(x ,y )∼D T 4d 2 L T V x, v + E (x,y),(x ,y )∼D T 4d 2 v , L T V x = 8 E (x,y)∼D T d 2 L T V x, v = 8L V (L V ).
Applying Theorem 5.2 to the last inequality gives the theorem.

In Theorem 5.3, we can notice that the margins are related to the distances between virtual points and correspond to the ideal margins, i.e. the margins that we would like to achieve after the learning step. In practice, we can also define γ1 and γ-1 the observed margins obtained after the learning step. All the similar examples are in a sphere centred in their corresponding virtual point and of diameter γ1 = 2 max (x,v)∈V x T L V -v T 2 . Similarly, the distance between spheres of dissimilar examples is γ-1 = min v,v ,v =v vv 2 -γ1 . As a consequence, even if we do not use cannot-link constraints, our algorithm is able to push reasonably far away dissimilar examples by minimizing the diameter of the sphere around similar examples.

In this section we have shown that the metrics learned by our algorithm generalize well and that our method can be theoretically linked to a classic metric learning approach. In the next section we empirically show the interest of our approach for several classification problems.

Experiments

In this section we propose an empirical evaluation of our method. On the one hand in Subsection 5.5.1 we compare it to several methods on a task of metric learning for classification and we provide some graphics showing 2D projections of the space learned by RVML-Lin-Class and RVML-RBF-Class on one dataset illustrating the capability of these approaches to learn discriminative attributes. On the other hand in Subsection 5.5.2 we further study the interest of explicitly choosing the virtual points using the methods presented in Section 5.3.

In all these experiments we consider 13 different datasets coming from either the UCI [START_REF] Lichman | UCI machine learning repository[END_REF] repository or used in recent works in metric learning [START_REF] Kedem | Non-linear metric learning[END_REF]; [START_REF] Shi | Sparse compositional metric learning[END_REF]; [START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF]. For isolet, splice and svmguide1 we have access to a standard training/test partition, for the other datasets we use a 70% training/30% test partition, we perform the experiments on 10 different splits and we average the result. We normalize the examples with respect to the training set by subtracting for each attribute its mean and dividing by 3 times its standard deviation. We set our regularization parameter λ with a 5-fold cross validation on the training set. After the metric learning step, we use a 1-nearest neighbour classifier to assess the performance of the metric and we report the accuracy obtained. Note that we also report the mean accuracy over the 13 tasks. Even if we are conscious that the different datasets consider different classification problems, it gives a rough idea of the global performance of the compared approaches.

Metric Learning for Classification

Here we consider the problem of learning a metric for a classification task. We consider two sets of experiments. In the first set we consider our linear formulation used with the two virtual points selection methods presented in this chapter, namely RVML-Lin-OT based on optimal transport (Section 5.3.1) and RVML-Lin-Class using the class-based representation space method (Section 5.3.2). We compare our two approaches to three baselines:

• 1-NN: A 1-nearest neighbour classifier without metric learning,

• LMNN [START_REF] Kilian Q Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF],

• SCML [START_REF] Shi | Sparse compositional metric learning[END_REF].

In a second set we consider the kernelized versions of RVML, namely RVML-RBF-OT and RVML-RBF-Class, based respectively on optimal transport and class-based representation space methods with a RBF kernel with the parameter σ fixed as the mean of all pairwise training set euclidean distances [START_REF] Kar | Similarity-based learning via data driven embeddings[END_REF]. We compare them to non linear methods using a KPCA with a RBF kernel3 as a pre-process. The number of dimensions is fixed as the one of the original space for high dimensional datasets (more than 100 attributes), to 3 times the original dimension when the dimension is smaller (between 5 and 100 attributes) and to 4 times the original dimension for the lowest dimensional datasets (less than 5 attributes). We also consider some local metric learning methods. Hence we compare our approach with 4 non-linear baselines:

• 1-NN-KPCA: A 1-nearest neighbour classifier in the KPCA space without metric learning,

• LMNN-KPCA: LMNN in the KPCA-space,

• GB-LMNN: A non linear version of LMNN [START_REF] Kedem | Non-linear metric learning[END_REF],

• SCMLLOCAL: The local version of SCML [START_REF] Shi | Sparse compositional metric learning[END_REF].

For all the baselines (linear and non linear), we use the implementations available online letting them handle hyper-parameters tuning.

The results for linear methods are presented in Table 5.1 while Table 5.2 gives the results obtained with the non linear approaches. In each table, the best result on each line is highlighted with a bold font while the second to best result is underlined. A star indicates either that the best baseline is significantly better than our best result or that our best result is significantly better than the best baseline according to classic significance tests (the p-value being fixed at 0.05).

We can make the following remarks. In the linear setting, our approaches are very competitive with state of the art approaches and RVML-Lin-OT tends to be the best on average even though SCML also performs very well on some datasets (the difference is not significant). RVML-Lin-Class performs slightly less on average. Considering now the non linear methods, our approaches improve their performance and are significantly better than the others on average, RVML-RBF-Class has the best average behaviour in this setting. These experiments show that our regressive formulation is very competitive and is even able to improve state of the art performances in a non linear setting.

Considering the virtual points selection, we can observe that the OT formulation performs better than the class-based representation space one in the linear case, while it is the opposite Table 5.1: Comparison of our approach with several baselines in the linear setting. The best result is highlighted with a bold font while the second to best result is underlined. A star indicates that the best result for the baseline is significantly better than our best result or that our best result is significantly better than the best baseline result. in the non linear case. We think that this can be explained by the fact that the OT approach generates more virtual points in a potentially non linear way which brings more expressiveness for the linear case. On the other hand, in the non linear one, the relative small number of virtual points used by the class-based method seems to induce a better regularization.

Baselines

To illustrate the capability of RVML-Lin-Class and RVML-RBF-Class to learn discriminative attributes we propose to select two dimensions out of the 26 of the space learned by these approaches on the isolet dataset. We selected 3 pairs of axis and the images obtained are presented in Figure 5.2. On the same line, we plot two images corresponding to the same axis pair: on the left column for RVML-Lin-Class and on the right column for RVML-RBF-Class. Note that for each axis, there is only one class for which the value of the attribute tends to be 1, for all the other classes this feature tends to be 0. Furthermore, we can note that the kernelized version of our metric outputs a more discriminative space: the examples are brought closer to their corresponding virtual point than in the linear version.

Interest of Explicitly Choosing Virtual Points

In the previous subsection we have seen that our approach is very competitive. Here we demonstrate the interest of explicitly choosing the virtual points.

Class based virtual points

In [START_REF] Globerson | Metric learning by collapsing classes[END_REF] the authors propose to collapse similar examples on a single point, an implicit virtual point, while pushing far away dissimilar examples. This behaviour can, in fact, be achieved by any margin based metric learning approach by setting the margin between similar examples to 0 and the margin between dissimilar examples to a high value. Thus to illustrate the interest of using explicit virtual points, we propose to compare our approach to Information Theoretic Metric Learning, ITML [START_REF] Davis | Informationtheoretic metric learning[END_REF], when considering the aforementioned margins (ITML-Collapse). For the sake of completeness we also consider ITML with tuned margins (ITML). The results are presented in Table 5.3 and show that, on average, ITML-Collapse and ITML are less accurate than RVML-Lin-Class hinting that considering explicit virtual points is better than considering implicit ones but also that learning a metric where each axis is discriminative is indeed beneficial for classification.

Optimal transport based virtual points

To further assess the interest of using our OT based formulation to select virtual points and associate them to examples, we propose to compare it with a random based approach (Random). In this latter setting, we randomly select a subset of examples for each class to act as virtual points and we randomly associate each example of this class to these virtual points. The results in the linear case are presented in Table 5.4 while the results in the non linear case are presented in Table 5.5. Overall, randomly selecting the virtual points is less accurate than using the OT based formulation. This is especially true in the linear case where the metric is less expressive than in the

Conclusion

In this chapter we presented a new metric learning approach based on a regression and aiming at bringing closer the learning examples to some a priori defined virtual points. The number of constraints has the advantage of growing linearly with the size of the learning set in opposition to the quadratic grow of standard must-link cannot-link approaches. Moreover, our method can be solved in closed form and can be easily kernelized allowing us to deal with non linear problems. Additionally, we proposed two methods to define the virtual points: one making use of recent advances in the field of optimal transport and one based on unit vectors of a class-based representation space allowing one to perform directly some dimensionality reduction. Theoretically, we have shown that the metrics learned with our approach generalize well and that we are able to link our empirical risk to the true risk of a classic metric learning formulation. Finally, we empirically show that explicitly choosing the virtual points is important and that our approach is competitive with the state of the art in the linear case and outperforms some classic approaches in the non linear one. We think that this work opens the door to design new metric learning formulations, in particular the definition of the virtual points can bring a way to control some particular properties of the metric (rank, locality, discriminative power, . . . ). As a consequence, this aspect opens new issues which are in part related to landmark selection problems but also to the ability to embed expressive semantic constraints to satisfy by means of the virtual points. Other perspectives include the development of a specific solver, of online versions, the use of low rank-inducing norms or the conception of new local metric learning methods. Another direction would be to study similarity learning extensions to perform linear classification with generalization guarantees on the classifier such as in [START_REF] Bellet | Similarity learning for provably accurate sparse linear classification[END_REF]; [START_REF] Balcan | Improved guarantees for learning via similarity functions[END_REF].

In this chapter we have addressed the problem of explicitly controlling the behaviour of a metric by introducing the notion of virtual points. It allows us to design metrics with a behaviour well tailored to the task at hand, for example classification. However choosing these virtual points can be difficult and the methods proposed in Section 5.3 might not always be satisfactory. In the next chapter we propose to build upon RVML and the optimal transport based virtual points to design a new algorithm able to learn a transformation which brings closer two distributions in a principled way.

Chapter 6

Mapping Estimation for Discrete Optimal Transport

This chapter is based on the following publication Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaury Habrard. Mapping estimation of discrete optimal transport. In Advances in Neural Information Processing Systems (NIPS-16), 2016

Abstract

In this chapter we propose to address the problem of learning a transformation from a Mahalanobis distance which follows some particular geometric transformations. Such a metric could be very beneficial for domain adaptation problems where the goal is to align the source and the target domains. Here we propose to consider geometric transformations which come from the result of an optimal transport problem. Indeed it is a reasonable procedure to align distributions and it has been shown to perform well in domain adaptation. Most of the computational approaches of optimal transport use the Kantorovich relaxation of the problem to learn a probabilistic coupling Γ between the training examples but do not address the problem of learning the transport map f S→T linked to the original Monge problem. Consequently, the fact that the coupling can only be used to transform training examples and not for out of samples ones lowers the potential usage of such methods. In this chapter we propose to combine the most interesting features of each method and we propose a new framework to estimate the transport map, also called the mapping, of a coupling. This estimation takes the form of a matrix L which corresponds to a new metric in the source domain. In this case we show that our approach is similar to RVML, presented in Chapter 5, where we define the transformation of the examples induced by the coupling as the virtual points for each example. However instead of considering that this coupling is defined a priori, we jointly learn it along the metric. It results in a jointly convex formulation which can be efficiently optimized and has the beneficial effect of smoothing the result of optimal transport. Empirically, we show the interest and the relevance of our method in two tasks, namely unsupervised domain adaptation and image editing.

Introduction

Many metric learning approaches have focused on learning a linear transformation in the form of a Mahalanobis distance or a bilinear similarity (See Section 2.2). One can notice that these methods do not try to control this transformation with respect to some particular geometric transformations but rather try to bring closer similar examples and push far away dissimilar ones. However considering other kind of transformations might be relevant for some problems. This is for example the case in domain adaptation (See Section 1.5) where one has to estimate and overcome the shift between a source and a target distribution. In this context, a few works in metric learning have proposed to learn a metric in order to move closer source and target instances [START_REF] Saenko | Adapting visual category models to new domains[END_REF][START_REF] Kulis | What you saw is not what you get: Domain adaptation using asymmetric kernel transforms[END_REF]. However these methods often require some sort of supervision to associate the examples with each other and, as mentioned before, remain limited by the kind of constraints considered.

Among approaches able to align distributions, an interesting solution is to consider optimal transport based methods which have recently shown their interest in domain adaptation. The idea is to learn a transformation of the source examples such that the source and the target are aligned. This transformation takes the form of a coupling of minimal cost between source and target where the cost function is for example the euclidean distance between the examples. One of the main drawbacks of using this coupling in optimal transport is that it can only be used to map source examples which have been seen during the training process and it is not applicable to out-of-sample examples. Hence, despite showing good performances in practice (Courty et al., 2014b) this approach cannot be used when new examples have to be mapped from source to target domains.

In this chapter we propose to consider the best of both worlds by learning a transformation whose behaviour is controlled by the transport map implied by the coupling Γ of a discrete optimal transport problem. Our formulation is based on a jointly convex optimization problem which admits two appealing interpretations. One the one hand it can be seen as learning a linear mapping regularized by an optimal transport map1 . On the other hand we can also see the approach as the computation of the optimal transport map regularized with respect to the definition of a mapping. Furthermore under some mild conditions on the set of transformations considered we will show some ties between this approach and RVML developed in Chapter 5. This formulation can be efficiently solved thanks to an alternating block-coordinate descent and actually benefits the two models. On the one hand we obtain smoother optimal transport maps which are compliant with a linear mapping usable as an out-of-sample transformation. This learned transformation is able to take into account some geometrical information captured by optimal transport. Another important aspect of our contribution is that it is in fact not limited to linear mappings as it can be kernelized. In this case it conveniently expresses non linear and out-of-sample transformations, thus enhancing the faithfulness to the true optimal transport map. See Figure 6 method. We also provide a brief discussion on the theoretical challenges behind our approach and we provide some empirical evidence of its interest in domain adaptation and in image editing.

The remaining of this chapter is organized as follows. Section 6.2 is dedicated to a presentation of the problem of optimal transport. In Section 6.3 we present our approach to jointly learn the coupling and the corresponding general transformation. Section 6.4 presents the optimisation scheme used to solve our formulation. Here we consider several possible transformations showing that our approach is close to RVML. In Section 6.5 we discuss some theoretical aspects of this work. In Section 6.6 we show the good behaviour of our approach in several experiments before concluding in Section 6.7.

Optimal Transport

In this section we present the problem of optimal transport. We start by recalling several recent approaches which successfully make use of it before formalising the problem.

In recent years optimal transport [START_REF] Villani | Optimal transport: old and new. Grund. der mathematischen Wissenschaften[END_REF] has received a lot of attention in the machine learning community (e.g. [START_REF] Canas | Learning probability measures with respect to optimal transport metrics[END_REF]Cuturi, 2013a;[START_REF] Solomon | Wasserstein propagation for semisupervised learning[END_REF][START_REF] Frogner | Learning with a wasserstein loss[END_REF]). This gain of interest comes from several nice properties of optimal transport when used as a divergence to compare discrete distributions. On the one hand it provides a sound and theoretically grounded way of comparing multivariate probability distributions without the need of estimating parametric versions. On the other hand by considering the geometry of the underlying space through a cost metric, it can encode useful informations about the nature of the problem. Optimal transport is usually expressed as an optimal cost functional but it also enjoys a dual variational formulation (Villani, 2009, Chapter 5).

Optimal transport has been proven to be useful in several settings. As a first example it corresponds to the Wasserstein distance in the space of probability distributions. Using this distance it is possible to compute means and barycentres [START_REF] Cuturi | Fast computation of wasserstein barycenters[END_REF][START_REF] Benamou | Iterative bregman projections for regularized transportation problems[END_REF] or to perform a PCA in the space of probability measures [START_REF] Seguy | Principal geodesic analysis for probability measures under the optimal transport metric[END_REF]. This distance has also been used in subspace identification problems for analysing the differences between distributions [START_REF] Mueller | Principal differences analysis: Interpretable characterization of differences between distributions[END_REF], in graph based semi-supervised learning to propagate histogram labels across nodes [START_REF] Solomon | Wasserstein propagation for semisupervised learning[END_REF] or as a way to define a loss function for multi-label learning [START_REF] Frogner | Learning with a wasserstein loss[END_REF]. As a second example optimal transport enjoys a variety of bounds for the convergence rate of empirical to population measures which can be used to derive new probabilistic bounds for the performance of unsupervised learning algorithms such as k-means [START_REF] Canas | Learning probability measures with respect to optimal transport metrics[END_REF]). As a last example optimal transport is a mean of interpolation between distributions (McCann, 1997) that has been used in Bayesian inference [START_REF] Reich | A nonparametric ensemble transform method for bayesian inference[END_REF], color transfer [START_REF] Ferradans | Regularized discrete optimal transport[END_REF] or domain adaptation (Courty et al., 2014a).

On the computational side, one of the major gain for optimal transport is the recent development of regularized versions that lead to efficient algorithms Cuturi (2013a); [START_REF] Benamou | Iterative bregman projections for regularized transportation problems[END_REF]; [START_REF] Cuturi | A smoothed dual approach for variational wasserstein problems[END_REF]. Most of optimal transport formulations are based on the computation of a (probabilistic) coupling matrix that can be seen as a bipartite graph between the bins of the distributions. This coupling, also denoted transportation matrix suffers from some drawbacks: it is always restricted to the data samples used to compute this map. In other words when a new dataset (or sample) is available, one has to recompute an optimal transport problem to deal with the new instances which can be prohibitive from some applications in particular when the task is similar or related. From a machine learning standpoint, this also means that we do not know how to have a good approximation of an optimal transport map computed from a small sample that can be generalized to unseen data. This is particularly critical when one considers large scale applications, or even medium-scales such as image editing problems. In this chapter, we bridge this gap by learning an explicit transformation that can be interpreted as a good approximation of the transport. As far as we know, this is the first approach that addresses directly this problem of out-of-sample mapping.

Formalisation

In this subsection we propose a more formal presentation of the problem of optimal transport. We present Monge's and Kantorovich's formulations which answer the problem of finding a map of minimal cost. We also present the notion of Barycentric mapping which corresponds to the transformation implied by the coupling of Kantorovich's formulation. First of all let S and T be the source and target domains respectively defined as the distribution D S on the space X s and the distribution D T over the space X t . In this chapter we upper-script with s any element associated with the source domain and with t any element associated with the target domain.

Monge problem Let X s ∈ R d s and X t ∈ R d t be two separable metric spaces such that any probability measure on X s (or X t ) is a Radon measure. By considering a cost function c : X s × X t → [0, ∞[, Monge's formulation of the optimal transport problem is to find a transform map f S→T : X s → X t (also known as a push-forward operator) between two probability measures D S on X s and D T on X t realizing the infimum of the following function inf

X s c(x s , f S→T (x s )) dD S (x s ), f S→T #D S = D T . (6.1)
When reaching this infimum, the corresponding map f S→T is an optimal transport map. It associates one point from X s to a single point in X t . Therefore, the existence of this map is not always guaranteed, as when for example D S is a Dirac and D T is not. As such, the existence of solutions for this problem can in general not be established when D S and D T are supported on a different number of Diracs. Yet, in a machine learning context, data samples usually form discrete distributions, but can be seen as observations of a regular, continuous (with respect to the Lebesgue measure) underlying distribution, thus fulfilling existence conditions (see (Villani, 2009, Chapter 9)). As such, assuming for the existence of f S→T calls for a relaxation of the previous problem.

Kantorovich relaxation

The Kantorovitch formulation of the optimal transportation Kantorovich ( 1942) is a convex relaxation of the Monge problem. Let us define Π as the set of all probabilistic couplings ∈ P(X s × X t ) the space of all joint distributions with marginals D S and D T . The Kantorovitch problem seeks for a general coupling f X s ×X t ∈ Π between X s and X t solving the following problem:

arg min f X s ×X t ∈Π X s ×X t c x s , x t df X s ×X t x s , x t (6.2)
The optimal coupling always exists (Villani, 2009, Theorem 4.1). This leads to a simple writing of the optimal transport problem in the discrete case, i.e. whenever D S and D T are only accessible through discrete samples S = {x s i } n s i=1 and T = {x t i } n t i=1 (designed by X s and X t in matrix form with one example on each line). The corresponding empirical distributions can be written as D S = n s i=1 p s i δ x s i and D T = nt i=1 p t i δ x t i where δ x is the Dirac function at location x ∈ X . p s i and p t i are probability masses associated to the i-th sample and belong to the probability simplex, i.e. n s i=1 p s i = n t i=1 p t i = 12 . Let Π be the set of probabilistic couplings between the two empirical distributions defined as:

Π = Γ ∈ (R + ) n s ×n t | Γ1 n t = D S , Γ T 1 n s = D T (6.3)
where 1 n is a n-dimensional vector of ones. Problem (6.2) becomes:

arg min Γ∈ Π Γ, C F (6.4)
where •, • F is the Frobenius dot product3 and C ≥ 0 is the n s × n t cost matrix related to the function c.

Barycentric mapping Once the probabilistic coupling Γ has been computed one needs to perform the transformation of examples from X s to X t . This transformation can be conveniently expressed with respect to the set of examples X t as the following barycentric mapping Reich (2013); Courty et al. (2014a);[START_REF] Ferradans | Regularized discrete optimal transport[END_REF]:

x s i = arg min x s ∈X s n t j=1 Γ(i, j)c x s , x t j . (6.5)
where x s i is the image of example x s i with coupling Γ. When the cost function is the squared euclidean distance4 , this barycentre corresponds to a weighted average and the sample is mapped into the convex hull of the target samples. For all source samples, this barycentric mapping can therefore be expressed as:

X s = f Γ (X s ) . = diag(Γ1 n t ) -1 ΓX t . ( 6.6)
In the rest of the chapter we will focus on an uniform sampling5 hence X s = n s ΓX t . The main drawback of the mapping ((6.6)) is that it does not allow the projection of out-of-sample examples which do not have been seen during the learning process of Γ. It means that to transport a new example x s ∼ D S one has to compute the coupling matrix Γ again using this new example. Also, while some authors consider specific regularization of Γ Cuturi (2013a); Courty et al. (2014a) to control the nature of the coupling, inducing specific properties of the transformation f S→T (i.e. regularity, divergence free, etc.) is hard to achieve.

In the next section we present a relaxation of the optimal transport problem, which consists in jointly learning Γ and f S→T . We derive the corresponding optimization problem, and show its usefulness in specific scenarios.

General Framework

In this chapter we propose to solve the problem of optimal transport by jointly learning the matrix Γ and the transformation function f S→T . First of all we denote H the space of transformations. Let X s and X t be matrices where each line is an example drawn from D S and D T . We propose the following optimisation problem: arg min

f S→T ∈H Γ∈ Π f (Γ, f S→T ) = 1 n s d t f S→T (X s ) -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ f S→T d s d t R(f S→T ) (6.7)
where f S→T (X s ) is a short-hand for the application of f S→T on each example in X s , R is a regularization term on f S→T and λ Γ , λ f S→T are hyper-parameters controlling the trade-off between the different terms in the optimization problem. The constants in front of each term normalize their values to be of the same order of magnitude. The first term in Problem (6.7) depends on both f S→T and Γ and controls the closeness between the transformation induced by f S→T and the barycentric interpolation obtained from Γ. The second term only depends on Γ and corresponds to the standard optimal transport loss. The third term regularizes f S→T to ensure a better generalization of the learned transformation.

In the next section we propose an efficient solution to optimize Problem (6.7) and we discuss several possible choices for the set of transformations H, in particular we show a link with RVML (Chapter 5).

Optimisation

A standard approach to solve Problem (6.7) is to use a block-coordinate descent [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF] where the idea is to alternatively optimize for f S→T and Γ. In the next theorem we show that under some mild assumptions on the regularization term R and the function space H this problem is jointly convex. In this case we are guaranteed to converge to the optimal solution if the formulation is strictly convex with respect to f S→T and Γ respectively. While this is not the case for Γ in our formulation, our algorithm works well in practice and a small regularization term can be added if theoretical convergence is required6 . Theorem 6.1. Let H be a convex space and R be a convex regularization. Problem (6.7) is jointly convex in f S→T and Γ.

Proof. First of all recall that a sum of jointly convex functions is jointly convex. Hence it is sufficient to show that the three terms of Problem (6.7) are jointly convex. We note:

f 1 (Γ, f S→T ) = 1 n s d t f S→T (X s ) -n s ΓX t 2 F , f 2 (Γ) = λ Γ max(C) Γ, C F , f 3 (f S→T ) = λ f S→T d s d t R(f S→T ) .
Note that by construction f 2 and f 3 are jointly convex in Γ and f S→T . We will show that f 1 is also jointly convex. Let g(Γ, f S→T ) = f S→T (X s ) -n s ΓX t F , we want to show that:

g tΓ + (t -1)Γ , tf S→T + (1 -t)f S→T ≤ tg(Γ, f S→T ) + (1 -t)g Γ , f S→T .
We have:

(tf S→T + (1 -t)f S→T )(X s ) -n s (tΓ + (t -1)Γ )X t F (Triangle inequality and definition of H.) ≤ tf S→T (X s ) -tn s ΓX t F + (1 -t)f S→T (X s ) -(1 -t)n s Γ X t F (t ∈ [0, 1].) ≤ t f S→T (X s ) -n s ΓX t F + (1 -t) f S→T (X s ) -n s Γ X t F
Furthermore noting that g is convex and positive we have:

g tΓ + (t -1)Γ , tf S→T + (1 -t)f S→T 2 (∀x ∈ R + , x → x 2 is non decreasing.) ≤ tg(Γ, f S→T ) + (1 -t)g Γ , f S→T 2 (∀x ∈ R, x → x 2 is convex.) ≤ t [g(Γ, f S→T )] 2 + (1 -t) g Γ , f S→T 2 .
Noting that f 1 (Γ, f S→T ) = 1 n s d t g(Γ, f S→T ) 2 concludes the proof.

As discussed above we propose to solve Problem (6.7) using a block-coordinates approach. As such we derive an efficient way to solve the problem for Γ when f S→T is fixed and for f S→T when Γ is fixed.

Solving for Γ with f S→T fixed In this case Problem (6.7) becomes:

arg min Γ∈ Π f (Γ, f S→T ) = 1 n s d t f S→T (X s ) -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ f S→T d s d t R(f S→T ) (6.8)
where f S→T is the current transformation. To solve such an optimization problem a common approach is to use the Frank-Wolfe algorithm [START_REF] Ferradans | Regularized discrete optimal transport[END_REF]; [START_REF] Frank | An algorithm for quadratic programming[END_REF].

It is a procedure for solving any convex constrained optimization problems with a convex and continuously differentiable objective function over a compact convex subset of any vector space. This algorithm can find an approximation of the optimal solution in O(1/ ) iterations [START_REF] Jaggi | Revisiting frank-wolfe: Projection-free sparse convex optimization[END_REF]. The approach is detailed in Algorithm 3.

Γ k+ 1 2 ∈ Π Γ k+ 1 2 , ∇f (Γ k , f S→T ) F with ∇f (Γ, f S→T ) = λ Γ max(C) C - 2 d t f S→T (X s ) X t T + 2 d t n s ΓX t X t T .
Find the optimal step α k satisfying the Armijo rule that minimizes

f (1 -α)Γ k + αΓ k+ 1 2 , f S→T . Update Γ k+1 = (1 -α k )Γ k + α k Γ k+ 1
2 and k = k + 1. until convergence end Algorithm 3: Updating Γ with the Frank-Wolfe algorithm.

Solving for f S→T with Γ fixed In this case Problem (6.7) becomes: arg min

f S→T ∈H f (Γ, f S→T ) = 1 n s d t f S→T (X s ) -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ f S→T d s d t R(f S→T ) (6.9)
where Γ is the current mapping between the examples. The solution to this optimization problem depends on H and R. This is discussed in detail in the next subsection.

Choosing H

In the previous subsection we presented our method when considering a general set of functions H. We now turn our attention toward several possibilities for the choice of H. On the one hand we propose to define H as a set of linear transformations from X s to X t . On the other hand using the kernel trick, we propose to consider non-linear transformations. Furthermore in both cases we consider the biased and non biased settings. In this case our approach boils down to learn a transformation matrix L. It can then be seen as using RVML (Chapter 5) where the virtual points are defined thanks to the barycentric mapping associated to the current coupling.

Linear transformations A first way to define H is to consider linear transformations induced by a d s × d t real matrix L:

H = f S→T : ∃L ∈ R d s ×d t s.t. ∀x s ∈ X s , f S→T (x s ) = x sT L . (6.10)
Furthermore, we define R(f S→T ) = L -I 2 F where I is the identity matrix. We choose to bias L toward I in order to ensure that the examples are not moved too far away from their initial position. In this case we can rewrite optimization problem (6.7) as: arg min (6.11) According to Algorithm 4 a part of our procedure requires to solve optimization problem (6.11) when Γ is fixed. One solution is to use the following closed form for L: (6.12) where (•) -1 is the matrix inverse (Moore-Penrose pseudo-inverse when the matrix is singular).

L∈R d s ×d t ,Γ∈ Π 1 n s d t X s L -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ L d s d t L -I 2 F .
L = 1 n s d t X sT X s + λ L d s d t I -1 1 n s d s X sT n s ΓX t + λ L d s d t I
In the previous definition of H, we considered non biased linear transformations. However it is sometimes desirable to add a bias to the transformation.

Biased linear transformations In the biased linear case H becomes: (6.13) In this case, Problem (6.7) becomes: (6.14) As in the non biased case, it is possible to find a closed form solution for L b T when Γ is fixed:

H = f S→T : ∃L ∈ R d s ×d t , ∃b ∈ R d t s.t. ∀x s ∈ X s , f S→T (x s ) = x sT L + b T = x sT 1 L b T .
arg min     L b T     ∈R d s +1×d t ,Γ∈ Π 1 n s d t X s 1 L b T -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ L d s d t L -I 2 F .
L b T = 1 n s d t X s 1 T X s 1 + λ L d s d t I 0 0 T 0 -1 1 n s d t X s 1 T n s ΓX t + λ L d s d t I 0 T (6.15)
Non-linear transformations In some cases a linear transformation is not sufficient to approximate the optimal transport. Hence, we propose to consider non-linear transformations. To do this, let φ be a non-linear function associated to a kernel function k : X s × X s → R such that k(x s , x s ) = φ(x s ), φ(x s ) H . We can then define H for a given set of examples X s as: (6.16) where k X s x sT is a short-hand for the vector k(x s , x s 1 ) k(x, x s 2 ) • • • k(x s , x s n s ) where x s 1 , • • • , x s n s are the rows of X s . In this case optimization problem (6.7) becomes:

H = f S→T : ∃L ∈ R n s ×d t s.t. ∀x s ∈ X s , f S→T (x s ) = k X s x sT L
arg min

L∈R n s ×d t ,Γ∈ Π 1 n s d t k X s (X s ) L -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ L n s d t k X s (•) L 2 F . (6.17)
where k X s (•) is a short-hand for the vector k(•,

x s 1 ) • • • k(•, x s n s ) = φ(x s 1 ) • • • φ(x s n s ) .
As in the linear case there is a closed form solution for L when Γ is fixed: (6.18) As in the linear case it might be interesting to use a bias.

L = 1 n s d t k X s (X s ) + λ L n s d t I -1 1 n s d t n s ΓX t .
Biased non-linear transformations In the biased non-linear case H becomes:

H = f S→T : ∃L ∈ R n s ×d t , ∃b ∈ R d t s.t. ∀x s ∈ X s , f S→T (x s ) = k X s x sT 1 L b T (6.19)
Optimization problem (6.7) can be rewritten as: (6.20) As in the non biased case, it is possible to find a closed form solution for L b T when Γ is fixed:

arg min      L b T      ∈R n s +1×d t ,Γ∈ Π 1 n s d t k X s (X s ) 1 L b T -n s ΓX t 2 F + λ Γ max(C) Γ, C F + λ L n s d t k X s (•) L 2 F .
L b T = 1 n s d t K X s X s 1 T K X s X s 1 + λ L d s d t K X s X s 0 0 T 0 -1 1 n s d t K X s X s 1 T n s ΓX t (6.21) where K X s X s = k X s (X s ).
A summary of our approach can be found in Algorithm 4.

Discussion on Theoretical Aspects

In this section we propose to discuss some theoretical considerations about our framework and more precisely on the quality of the learned transformation denoted by f L to show its dependence on the matrix L. To assess this quality we consider the Frobenius norm between f L and the true transport map, denoted f X s →X t , that we would obtain if we could solve the input : X s , X t source and target examples and λ Γ , λ L hyper parameters. output: L, Γ. begin Initialize k = 0, Γ 0 ∈ Π and L 0 = I repeat Learn Γ k+1 with fixed L k using a Frank-Wolfe approach (Algorithm 3). Learn L k+1 using Equations (6.12), (6.15), (6.18) Monge problem. Let f Γ be the empirical barycentric mapping using the probabilistic coupling Γ learned between X s and X t . Similarly let f X s ×X t be the theoretical barycentric mapping associated with the probabilistic coupling learned on D S , D T the whole distributions and which corresponds to the solution of Kantorovich's problem. Using a slight abuse of notations we denote by f Γ (x s ) and f X s ×X t (x s ) the projection of x s ∈ X s by these barycentric mappings. We have the following simple theorem on the quality of the learned transformation. Theorem 6.2 (Bound on the quality of the learned transformation). With high probability we have:

E x s ∼D S f L (x s ) -f X s →X t (x s ) 2 F ≤ 4 x s ∈X s f L (x s ) -f Γ (x s ) 2 F + O 1 √ n s + 4 x s ∈X s f Γ (x s ) -f X s ×X t (x s ) 2 F + 2 E x s ∼D S f X s ×X t (x s ) -f X s →X t (x s ) 2 F (6.22)
Proof.

E x s ∼D S f L (x s ) -f X s →X t (x s ) 2 F (Triangle inequality.) ≤ E x s ∼D S f L (x s ) -f X s ×X t (x s ) F + f X s ×X t (x s ) -f X s →X t (x s ) F 2 ((a + b) 2 ≤ 2a 2 + 2b 2 .) ≤ 2 E x s ∼D S f L (x s ) -f X s ×X t (x s ) 2 F + 2 E x s ∼D S f X s ×X t (x s ) -f X s →X t (x s ) 2 F
Furthermore considering that H is as proposed in Section 6.4 and using Theorem 5.2 in Chapter 5 we have with high probability that:

E x s ∼D S f L (x s ) -f X s →X t (x s ) 2 F ≤ 2 x s ∈X s f L (x s ) -f X s ×X t (x s ) 2 F + O 1 √ n s + 2 E x s ∼D S f X s ×X t (x s ) -f X s →X t (x s ) 2 F . (Triangle inequality.) ≤ 2 x s ∈X s f L (x s ) -f Γ (x s ) F + f Γ (x s ) -f X s ×X t (x s ) F 2 + O 1 √ n s + 2 E x s ∼D S f X s ×X t (x s ) -f X s →X t (x s ) 2 F . ((a + b) 2 ≤ 2a 2 + 2b 2 .) ≤ 4 x s ∈X s f L (x s ) -f Γ (x s ) 2 F + O 1 √ n s + 4 x s ∈X s f Γ (x s ) -f X s ×X t (x s ) 2 F + 2 E x s ∼D S f X s ×X t (x s ) -f X s →X t (x s ) 2 F (6.23)
From Inequality (6.22) we deduce that there are three key quantities related to the quality of the learned transformation f L :

• x s ∈X s f L (x s ) -f Γ (x s ) 2 F + O 1
√ ns : This first quantity is the difference between the transformation and the empirical barycentric mapping with respect to the Frobenius norm. It is the one that we minimize in Problem (6.7) and should be as small as possible to obtain a better approximation f L . Furthermore, by definition, the coupling used to compute the empirical barycentric mapping of this term also appears in Problem (6.7).

•

x s ∈X s f Γ (x s ) -f X s ×X t (x s ) 2 F : This second quantity is the difference between the learned barycentric mapping and the theoretical one which could be obtained by learning on the whole distribution. We expect this quantity to decrease uniformly with respect to the number of examples as it corresponds to a measure of how well a mapping learned on a limited set reflects the true mapping.

• E x s ∼D S f X s ×X t (x s ) -f X s →X t (x s ) 2
F : This third quantity is the difference between the theoretical barycentric mapping and the true transformation. We expect this quantity to be small as it characterizes that a barycentric mapping using a coupling learned on the whole distributions is a good approximation of the true transport map.

Note that we only expect the second and third term to be small but we do not prove it. Indeed these quantities are difficult to bound because of a lack of theoretical results related to these in the literature. Nevertheless we think that this discussion opens the door for new theoretical perspectives to use OT in a Machine Learning setting but these are beyond the scope of this thesis.

Experiments

In this section we propose to experimentally validate our approach on two tasks. The first one is an unsupervised domain adaptation one while the second one deals with the problem of seamless copy in images.

Application in Unsupervised Domain Adaptation

In this first experiment we show the interest of our approach in an unsupervised domain adaptation task. [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF]. Note that we also used this dataset in Chapter 4 but with the original SIFT features [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF].

Datasets

Methods We consider 6 different baselines. The first one is a simple 1-Nearest-Neighbour (1-NN) using the original source examples only. The second and third ones are two widely used domain adaptation approaches, namely Geodesic Flow Kernel (GFK) [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF] and Subspace Alignment (SA) [START_REF] Fernando | Unsupervised visual domain adaptation using subspace alignment[END_REF]. The fourth to sixth baselines are OT based approaches: The classic OT method (OT), OT with an entropy based regularization (OTE) Cuturi (2013a) and OT with a 1,2 regularization (L1L2) Courty et al. (2014a). We present the results of our approach with the linear (OTLin) and kernel (OTKer) versions of the transformation. We also consider their biased counterparts (*B). For all the baselines the idea is to apply the learned transformation on the source and then to use a 1-NN classifier on the labelled source examples to classify the target examples.

Experimental setup We consider the following experimental setup for all the methods and datasets. All the results presented in this section are averaged over 10 trials. For each trial we consider three sets of examples, a labelled source training set denoted X s , y s , an unlabelled target training set denoted X t train and a labelled target testing set X t test , y t test . The model is learned on X s , y s and X t train and evaluated on X t test , y t test with a 1-NN learned on X s , y s . All the hyper-parameters are tuned according to a grid search from the source and target training instances from a reverse validation procedure close to [START_REF] Bruzzone | Domain adaptation problems: A DASVM classification technique and a circular validation strategy[END_REF]; [START_REF] Zhong | Cross validation framework to choose amongst models and datasets for transfer learning[END_REF] and presented in Algorithm 5. We use this approach to affect a score to all the possible instantiations of hyper parameters and we then select the best among these. As a score we compute an average accuracy of a two fold method. The idea is to split the source training set in two halves. From one half we learn a model f that is used to label the target training set. These new labels are then used to label the second half of the source training set to obtain a first accuracy. The role of the two halves are then reversed to obtain a second accuracy. The model f is learned with an algorithm A λ using some hyper parameters λ and is able to bring closer the source and the target examples. For example, with our linear mapping learned from our regularized OT formulation, we have f X t = X t and f (X s ) = f L (X s ) = X s L. For the hyper parameters of the compared methods we use the following ranges: For GFK and SA we choose the dimension of the subspace as d ∈ {3, 6, . . . , 30}, for L1L2 and OTE we set the parameter for entropy regularization in {10 -6 , 10 -5 , . . . , 10 5 }, for L1L2 we choose the class related parameter η ∈ {10 -5 , 10 -4 , • • • , 10 2 }, for all our methods we choose λ L , λ Γ ∈ {10 -3 , 10 -2 , . . . , 10 0 }.

In this algorithm f is any model able to bring closer the source and the target. For example, with our linear mapping learned from our regularized OT formulation, we have

f X t = X t and f (X s ) = f L (X s ) = X s L.
input : (X s , y s ) source examples and their labels, X t target examples, A λ a learning procedure using hyper-parameters λ. output: Average accuracy of A λ . begin Split (X s , y s ) in two halves (X s1 , y s1 ) and (X s2 , y s2 ).

Learn f 1 = A λ (X s1 , y s1 , X t ) and set y t 1 the pseudo-labels of f 1 (X t ) obtained from a 1NN learned on (f 1 X s1 , y s1 ).

Set s 1 the accuracy of a 1NN learned on (f 1 (X t ) , y t 1 ) and evaluated on (f 1 X s2 , y s2 ) .

Learn f 2 = A λ (X s2 , y s2 , X t ) and set y t 2 the pseudo-labels of f 2 (X t ) obtained from a 1NN learned on (f 2 X s2 , y s2 ).

Set s 2 the accuracy of a 1NN learned on (f 2 (X t ) , y t 2 ) and evaluated on (f 2 X s1 , y s1 ) .

return s 1 +s 2 2 . end Algorithm 5: Circular validation.

The results on the Moons dataset are presented in Table 6.1 and those for Office-Caltech are given in Table 6.2. A first important remark is that for both datasets the results obtained by using the barycentric mapping with f Γ and the results obtained by directly using the transformation f L are almost the same. It shows that our method allows us to learn a function f L that is a good approximation of f Γ and that f Γ is well adapted to the class of transformations H. In terms of accuracy, our approach tends to give the best results in most of the cases which shows that we are effectively able to move closer the distributions in a relevant way. For the Moons datasets, the last four approaches (including ours) based on OT obtain similar results until 40 degrees while other DA methods fail to obtain good results at 20 degrees. Beyond 50 degrees, our approach tends to obtain significantly better results (more than 10 points of accuracy) and is more stable when the difficulty of the problem increases. For Office-Caltech, our results are significantly better than other approaches which clearly illustrates the potential of our method for difficult DA tasks. As a conclusion, forcing the OT to learn a smoother map f L allows the approach to get a better robustness.

f Γ f L f Γ f L f Γ f L f Γ D → W

Seamless Copy in Images with Gradient Adaptation

We propose here a direct application of our mapping estimation in the context of image editing. While several papers using optimal transport are focusing on color adaptation [START_REF] Ferradans | Regularized discrete optimal transport[END_REF]; [START_REF] Solomon | Convolutional wasserstein distances[END_REF], we explore here a new variant in the domain of image editing: the seamless editing or cloning in images. In this context, one may desire to import a region from a given source image to a target image. As a direct copy of the region leads to inaccurate results in the final image nearby the boundaries of the copied selection, a very popular method, proposed by Pérez and co-workers [START_REF] Pérez | Poisson image editing[END_REF], allows to seamlessly blend the target image and the selection. This technique, coined as Poisson Image Editing, operates in the gradient domain of the image. Hence, the gradients of the selection operate as a guidance field for an image reconstruction based on membrane interpolation with appropriate boundary conditions extracted from the target image.

Let f be an unknown scalar function (usually a component of the color space of the image) defined on a given region of the image Ω. Let f t be the target image defined everywhere apart from the interior of Ω. The Poisson editing method operates by solving for f as the following variational optimization problem with Dirichlet boundary conditions:

min f Ω |∇f -v| 2 with f | ∂Ω = f t | ∂Ω . (6.24)
Here, v is the guidance field, which is usually given as the gradient from the source image f s over the domain Ω, i.e. v = ∇f s | Ω . One can show that the unique solution to this problem is the solution of the following Poisson equation [START_REF] Pérez | Poisson image editing[END_REF]: (6.25) Using appropriate first order discretization of the Laplacian operator, solving for this problem amounts to solve a big sparse linear system, which can be performed efficiently with multigrid solvers. Though appealing, this technique is prone to errors due to local contrast change or false colors resulting from the integration. While some solutions combining both gradient and color domains exist [START_REF] Deng | ACCV, chapter Color-Aware Regularization for Gradient Domain Image Manipulation[END_REF], this editing technique usually requires the source and target images to have similar colors and contrast. Here, we propose to enhance the generality of this technique by forcing the gradient distribution from the source image to follow the gradient distribution in the target image. As a result, the seamless cloning not only blends smoothly the copied region in the target domain, but also constraints the color dynamics to that of the target image. Hence, a part of the style of the target image is preserved. We start by learning a transfer function f L : R 6 → R 6 with our method, where 6 denotes the vertical and horizontal components of gradient per color. Following our method which aligns the distribution of gradients in the source image to the target one, we then solve for the following system: .26) When dealing with images, the number of source and target gradients are largely exceeding tens of thousands and it is mandatory to consider methods that scale appropriately. As such, our technique can readily learn the transfer function f L over a limited set of gradients and generalizes appropriately to unseen gradients. Several illustrations of this method are proposed in a context of face swapping in Figure 6.2. As one can observe, the original method of Poisson image editing [START_REF] Pérez | Poisson image editing[END_REF] (3rd column) tends to preserve the color dynamic of the original image and fails in copying the style of the target image. Our method was tested with a linear and kernel version of f L , that was learned with only 500 gradients sampled randomly from both sources (λ L = 10 -2 , λ L = 10 3 for respectively the linear and kernel versions, and λ Γ = 10 -7 for both cases). As a general qualitative comment, one can observe that the kernel version of f L is better at preserving the dynamics of the gradient, while the linear version tends to flatten the colors. In this low-dimensional space, this illustrates the need of a non-linear transform. We also illustrate one case of failure of our approach in Figure 6.3 where it is not possible to produce the same vast swaths of colors as in the target image since our method does not modify the spatial arrangement of the gradient. Regarding the computational time, the gradient adaptation is of the same order of magnitude as the Poisson equation solving, and each example are computed in less than 30s on a standard personal laptop.

∆f = div v over Ω, with f | ∂Ω = f t | ∂Ω .
∆f = div f L (v) over Ω, with f | ∂Ω = f t | ∂Ω . ( 6

Conclusion

In this chapter we proposed a solution to learn a transformation from a Mahalanobis distance whose behaviour is controlled by the geometric transformation induced by a transport map. We considered a jointly convex approach to learn both the coupling Γ and the transformation f L . From an optimal transport point of view this transformation can be seen as an approximation of the transport map given by Γ and allows us to project out-of-samples examples not seen during the learning process. Furthermore, jointly learning the coupling and the transformation allows us to regularize the transport by enforcing a certain smoothness on the transport map. We presented some theoretical considerations on the generalization ability of the learned transformation f L . Hence we discussed that under the assumption that the barycentric mapping generalizes well and is a good estimate of the true transformation, then f L learned with our method should be a good approximation of the true transformation. We have shown that our approach is efficient in practice on two different tasks, namely domain adaptation and image editing. On the one hand, in the domain adaptation task, we obtained better results than standard optimal transport based approaches. Furthermore the results obtained by the coupling Γ and the transformation f L are almost identical validating the approach. On the other hand, in a Computer Vision task, we have shown that the transformation f L can be efficiently used on out-of-samples examples leading to visually smoother and better results than the standard approaches.

The framework presented in this chapter opens the door to several perspectives. First, from a theoretical standpoint the bound proposed raises some questions on the generalization ability of the barycentric mapping and on the estimation of the quality of the true barycentric mapping with respect to the target transformation. On a more practical side, note that in recent years regularized optimal transport has encountered a growing interest and several methods have been proposed to control the behaviour of the transport. As long as these regularization terms are convex, one could imagine use them in our framework. Another perspective could be to use our framework in a mini-batch setting where instead of learning from the whole dataset we can estimate a single function f S→T from several couplings Γ optimized on different splits of the examples. We also believe that our framework could allow the use of the notion of optimal transport in deep architectures as, contrary to the coupling Γ, the function f L can be used on out-of-samples examples. As a last perspective we think that our framework could be used to learn some metrics in some other contexts such as unsupervised learning.

be computed and used to select the best source metric among a set of candidates. As an illustration it allowed us to propose an algorithm to weight the importance of the source metric. We demonstrated the wide range of applicability of the metric hypothesis transfer learning framework by proving that several loss functions and regularization terms fall into our theoretical analysis. Furthermore we empirically evaluated it on metric learning and semi-supervised domain adaptation tasks.

Most of metric learning approaches use similarity and dissimilarity constraints to learn a metric but do not explicitly control the behaviour of the underlying transformation. In our third contribution we addressed this problem by proposing a new approach where the desired destination of the examples is explicitly chosen through so-called virtual points. It allowed us to carefully control the learned metric and thus to design more problem specific models. For example for classification we proposed class based virtual points where the metric is learned such that each axis is discriminative for a particular class. We showed that our approach can easily be kernelized making it able to learn very expressive metrics. We also proposed a theoretical study demonstrating that our approach can be tied to a classic metric learning method. Lastly we empirically demonstrated its good performance on several well known datasets.

In our fourth contribution we addressed a problem similar to the third one. However instead of explicitly controlling the behaviour of each example individually we proposed to force a metric to follow a particular geometric transformation. Hence we considered transformations implied by the coupling learned by a discrete optimal transport problem which is particularly relevant for domain adaptation tasks. We proposed a solution to jointly learn this coupling and an associated metric through its underlying transformation. We derived an efficient optimization scheme and we showed that this approach could be further interpreted as a modification of our third contribution where the transformation and the virtual points are jointly learned. We empirically demonstrated the good behaviour of our approach on unsupervised domain adaptation and seamless copy tasks.

Perspectives

We have already presented specific perspectives for each of our contributions. In this part, we rather try to discuss more general future works that can represent some new research directions from the work presented in this thesis.

From an algorithmic standpoint our contributions are mainly based on batch optimization problems. A first perspective would be to extend the concepts presented here to the online learning setting. Following this idea it could be interesting to develop some mechanisms able to detect a potential drift in the distribution of the examples and to automatically change the behaviour of the metric accordingly. Such an approach could for example be used when learning a metric to solve a problem of tracking of objects in videos where the variations in the scene might call for different behaviours. Another perspective would be to consider active learning to improve the control over the metric. For example when learning a transformation it could be interesting to obtain some feedback from the user to verify that the examples are moving in the correct direction. A motivating example could be the problem of domain adaptation, where active learning has already proven to be useful [START_REF] Berlind | Active nearest neighbors in changing environments[END_REF], where obtaining some carefully selected feedbacks could ensure that the metric is correctly estimating the shift between the distributions.

On a more theoretical standpoint we can notice that in this thesis we were mainly interested in the generalization ability of the learned metric and not in its impact on the subsequent application. Following the latter idea [START_REF] Balcan | Improved guarantees for learning via similarity functions[END_REF] have demonstrated that the error of a linear classifier is tied to a measure of goodness of the similarity used to learn it. This goodness is related to the capacity of a metric at bringing closer similar examples and pushing far away dissimilar ones. However when learning a metric with controlled behaviour this measure might not be adapted. For example when learning a metric with respect to a reference metric (Chapters 3 and 4) one would probably be more interested in considering a measure telling if the learned metric is better than the reference one. Similarly when learning a transformation for a domain adaptation task (Chapter 4 and 6) one would probably put its focus on the ability of the metric at aligning the source and target distributions. It implies that a measure of the goodness of the metric is task dependent. An interesting perspective would be to consider some theoretical frameworks able to take into account a measure of goodness related to the task at hand and to prove that a good metric is indeed beneficial.

Another theoretical perspective would be to derive generalization bounds with a fast rate of convergence in the presence of additional informations. In Chapter 4 we proposed a first solution to this problem using the Rademacher complexity framework along with the additional information that is the goodness of a source metric. However this solution is not satisfying in the sense that the constraint on the source metric was somehow stronger than the result obtained on the learned metric. Nevertheless this is still encouraging in the sense that it shows that under strong assumptions it is possible to obtain a fast rate of convergence. Thus, if one manages to obtain weaker assumptions (See e.g. Srebro et al. (2010c)) it might be possible to obtain more meaningful results.

≤ max ∆ max λ j , ∆ 2 max (B.3) Setting B j = max ∆max √ λ j , ∆ 2 max
gives the lemma.

B.2 Proof of Lemma 3.2

Lemma (k-lipschitz continuity). Let M T j and M T j be two matrices for a region C j and (x, x , ∆) be an example. Our loss l M T j , (x, x , ∆) is k-lipschitz continuous with k = D 2 j .

Proof.

l M T j , (x, x , ∆) -l M T j , (x, x , ∆) = (x -x ) T M T j (x -x ) -∆ 2 -(x -x ) T M T j (x -x ) -∆ 2 (Triangle inequality.) ≤ (x -x ) T M T j (x -x ) -(x -x ) T M T j (x -x ) = (x -x ) T (M T j -M T j )(x -x ) (Cauchy-Shwartz inequality (Theorem A.3).) ≤ x -x 2 2 M T j -M T j F (D j = max (x,x ,∆)∼D T j x -x 2 .) ≤ D 2 j M T j -M T j F
Setting k = D 2 j gives the lemma.

B.3 Proof of Lemma 3.3

To prove Lemma 3.3 we need the following technical lemma.

Lemma B.1. Let F T j (M) = LT j (M) + λ j M 2 F and F T i j (M) = LT i j (M) + λ j M 2
F be the functions minimized in Problem (3.5) where T j and T i j are two training samples of n j examples. T i j is obtained by replacing example i from T j by another example drawn independently from D T j . Let M T j and M T i j be their respective minimizers, and λ j be the regularization parameter used in our algorithm. Let ∆ M T j = M T j -M T i j , then, we have, for any t ∈ [0, 1],

M T j 2 F -M T j -t∆ M T j 2 F + M T i j 2 F -M T i j + t∆ M T j 2 F ≤ 2kt λ j n j ∆ M T j F . (B.4)
Proof. LT i j is a convex function, thus, for any t ∈ [0, 1], we can write:

LT i j (M T j -t∆ M T j ) -LT i j (M T j ) ≤ t LT i j (M T i j ) -LT i j (M T j ) , (B.5) LT i j (M T i j + t∆ M T j ) -LT i j (M T i j ) ≤ t LT i j (M T j ) -LT i j (M T i j ) . (B.6)
By summing Inequalities (B.5) and (B.6) we obtain

LT i j (M T j -t∆ M T j ) -LT i j (M T j ) + LT i j (M T i j + t∆ M T j ) -LT i j (M T i j ) ≤ 0. (B.7)
Since M T j and M T i j are minimizers of F T j and F T i j , we can write:

F T j M T j -F T j M T j -t∆ M T j ≤ 0, (B.8) F T i j M T i j -F T i j M T i j + t∆ M T j ≤ 0. (B.9)
By summing Inequalities (B.8) and (B.9), we obtain:

LT j (M T j ) -LT j (M T j -t∆ M T j ) + λ j M T j 2 F -λ j M T j -t∆ M T j 2 F + LT i j (M T i j ) -LT i j (M T i j + t∆ M T j ) + λ j M T i j 2 F -λ j M T i j + t∆ M T j 2 F ≤ 0. (B.10)
We can now sum Inequalities (B.7) and (B.10) to obtain:

LT j (M T j ) -LT i j (M T j ) -LT j (M T j -t∆ M T j ) + LT i j (M T j -t∆ M T j ) + λ j M T j 2 F -λ j M T j -t∆ M T j 2 F + λ j M T i j 2 F -λ j M T i j + t∆ M T j 2 F ≤ 0. (B.11)
From Inequality (B.11), we can write:

λ j M T j 2 F -λ j M T j -t∆ M T j 2 F + λ j M T i j 2 F -λ j M T i j + t∆ M T j 2 F ≤ C (B.12) with C = LT i j (M T j ) -LT j (M T j ) + LT j (M T j -t∆ M T j ) -LT i j (M T j -t∆ M T j ).
We are now looking for a bound on C:

C ≤ LT j (M T j -t∆ M T j ) -LT j (M T j ) + LT i j (M T j ) -LT i j (M T j -t∆ M T j ) = 1 n j (x,x ,∆)∈T j l M T j -t∆ M T j , (x, x , ∆) -l M T j , (x, x , ∆) + (x i ,x i ,∆)∈T i j l M T j , (x i , x i , ∆) -l M T j -t∆ M T j , (x i , x i , ∆) ≤ 2k 2 λ j n j . (B.16) In particular, since k = D 2 j , sup (x,x ,∆)∼D T j l M T j , (x, x , ∆) -l M T i j , (x, x , ∆) ≤ 2D 4 j λ j n j . (B.17)
The last inequality matches the definition of uniform stability, Definition 1.3. Thus setting

β j = 2D 4 j λ j gives the lemma.
B.4 Proof of Lemma 3.4 Lemma (Bound on E T j ∼D T j R T j ). For any β j uniformly stable learning method of estimation error R T j = L T j (M T j ) -LT j (M T j ) for a training set T j , we have:

E T j ∼D T j R T j ≤ β j n j . (B.18)
Proof. First of all note that:

E T j ∼D T j LT j (M T j ) = 1 n j (x i ,x i ,∆)∈T j E T j ∼D T j l M T j , (x i , x i , ∆) = 1 n j (x i ,x i ,∆)∈T j E T j ,(x,x ,∆)∼D T j l M T i j , (x, x , ∆) = E T j ,(x,x ,∆)∼D T j l M T i j , (x, x , ∆)
The second to last equality comes from the fact that the pairs are drawn independently from D T j and thus changing one example with another does not change the expectation. From this equality we deduce:

E T j ∼D T j R T j = E T j ∼D T j L T j (M T j ) -LT j (M T j ) = E T j ,(x,x ,∆)∼D T j l M T j , (x, x , ∆) -l M T i j , (x, x , ∆) ≤ E T j ,(x,x ,∆)∼D T j l M T j , (x, x , ∆) -l M T i j , (x, x , ∆) (Uniform stability (Lemma 3.3).) ≤ β j n
Using the k-lipschitz continuity of the loss, we have:

sup z,z ∼D T l M T , z, z -l M T i , z, z ≤ k ∆ M F ≤ 4k 2 λn .
Setting β = 4k 2 λ concludes the proof.

C.3 Proof of Lemma 4.3

Lemma (Bound on E T ∼D T [R T ]). For any β uniformly stable learning method of estimation error R T = L T (M T ) -LT (M T ) for a training set T , we have:

E T ∼D T [R T ] ≤ 2β n .
Proof. First of all note that:

E T ∼D T LT (M T ) = 1 n(n -1) z i ∈T z j ∈T z i =z j E T ∼D T [l (M T , z i , z j )] = 1 n(n -1) z i ∈T z j ∈T z i =z j E T,z,z ∼D T l M T i j , z, z = E T,z,z ∼D T l M T i j , z, z
The second to last equality comes from the fact that the examples are drawn independently from D T and thus changing one example with another twice does not change the expectation.

From this equality we deduce: have:

E T ∼D T [R T ] = E T ∼D T L T (M T ) -LT (M T ) = E T,z,z ∼D T l M T , z, z -l M T i j , z, z = E T,z,z ∼D T l M T , z, z -l M T i , z, z + l M T i , z, z -l M T i j , z, z (Triangle inequality.) ≤ E T,z,z ∼D T l M T , z, z -l M T i , z, z + E T,z,z ∼D T l M T i , z, z -l M T i j , z, z (
E T ∼D T [R T ] ≤ 2kR n (M S ).
Proof. Using standard properties on Rademacher variables [START_REF] Peter | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF]; Shalev-Shwartz and Ben-David (2014a) and U-statistics [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] we have that:

E T ∼D T sup M∈M S L T (M) -LT (M) = E T ∼D T sup M∈M S   L T (M) - 1 n(n -1) n i=1 n j=1,j =i l (M, z i , z j )   (U-Statistics, Lemma 7 in Cao et al. (2016).) ≤ E T ∼D T sup M∈M S   LT (M) - 1 n 2 n 2 i=1 l M, z i , z n 2 +i    ≤ E T ∼D T sup M∈M S    E T ∼D T LT (M) - 1 n 2 n 2 i=1 l M, z i , z n 2 +i    ≤ E T,T ∼D T sup M∈M S    LT (M) - 1 n 2 n 2 i=1 l M, z i , z n 2 +i    (U-Statistics, Lemma 7 in Cao et al. (2016).) ≤ E T,T ∼D T sup M∈M S    1 n 2 n 2 i=1 l M, z i , z n 2 +i -l M, z i , z n 2 +i   
(Equation (26.9) in Shalev-Shwartz and Ben-David (2014a).)

≤ 1 n 2 E T,T ∼D T σ sup M∈M S n 2 i=1 σ i l M, z i , z n 2 +i -l M, z i , z n 2 +i
(End of the proof of Lemma 26.2 in Shalev-Shwartz and Ben-David (2014a).)

≤ 2 n 2 E T ∼D T ,σ sup M∈M S    n 2 i=1 σ i l M, z i , z n 2 +i   
(k-lipschitzness and Lemma 26.9 in Shalev-Shwartz and Ben-David (2014a).)

≤ 2 n 2 E T ∼D T ,σ sup M∈M S    n 2 i=1 σ i kk M x i , x n 2 +i    (C.8) Note that ∀b ∈ R n , E σ n i=1 σ i b i = 0. It implies that ∀a ∈ A ⊆ R n we have: E σ sup a∈A n i=1 σ i a i = E σ n i=1 σ i b i + E σ sup a∈A n i=1 σ i a i (b does not depend on A.) = E σ sup a∈A n i=1 σ i (a i + b i ). (C.9) Applying (C.9) to (C.8) with bi = -k M S xi, x n 2 +i
gives:

E T ∼D T sup M∈M S L T (M) -LT (M) ≤ 2 n 2 E T ∼D T ,σ sup M∈M S    n 2 i=1 σ i k k M-M S x i , x n 2 +i    (Definition 4.3.) ≤ 2kR n (M S ).
|R T -R T i | ≤ 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ n
where • * is the dual norm of the regularization term (Definition A.4).

Proof. First of all note that from Definition 4.2 and Equation (4.14) we have that for any two examples z, z ∼ D T :

l M, z, z -l M S , z, z ≤ k g(x, x ), M -M S (Cauchy-Schwartz's inequality (Theorem A.3).) ⇒ l M, z, z ≤ l M S , z, z + k g(x, x ) * M -M S (M ∈ M S .) ⇒ l M, z, z ≤ l M S , z, z + k g(x, x ) * G 3 (M S ) λ (Taking the supremum over z, z ∼ D T .) ⇒ l M, z, z ≤ G 3 (M S ) + sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ (C.10)
where • * represents the dual norm of the regularization term (Definition A.4) and G 3 (M S ) = sup z,z ∼D T l (M S , z, z ).

|R T -R T i | = sup M∈M S L T (M) -LT (M) -sup M∈M S L T (M) -LT i (M) ≤ sup M∈M S LT (M) -LT i (M) = 1 n(n -1) sup M∈M S z∈T z ∈T z =z l M, z, z - z i ∈T i z i ∈T i z i =z i l M, z i , z i
(T and T i only differ by one example, ∀j = i, z j = z i j and z

j = z i j .) = 1 n(n -1) sup M∈M S z ∈T z i =z l M, z i , z - z ∈T i z i i =z l M, z i i , z + z∈T z =z i l M, z, z i - z∈T i z =z i i l M, z, z i i (Triangle inequality.) ≤ 1 n(n -1) sup M∈M S      z ∈T,T i z i =z ,z i i =z l M, z i i , z -l M, z i , z + z∈T,T i z =z i ,z =z i i l M, z, z i i -l M, z, z i      (Positive loss and Inequality (C.10)) ≤ 2G 3 (M S ) + 2 sup z,z ∼D T k g(x, x ) * G 3 (M S ) λ n . (f is L-lipschitz.) ≤ L δ yy [k M x, x -γ yy ] -δ yy [k M x, x -γ yy ] (δ yy ∈ {-1, 1}.) ≤ L k M x, x -k M x, x (
C.12) (Equation (4.14) and Cauchy-Schwartz's inequality (Theorem A.3).)

≤ L sup z,z ∼D T g(x, x ) * M -M (C.13)
Inequalities (C.12) and (C.13) respectively prove the lipschitzness with respect to the metric and the matrix. Lastly we show that the loss is (σ, m)-admissible. Given four examples z, z , z , z ∼ D T and M T the learned metric when learning with T we have: where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

l M T , z, z -l M T , z , z ≤ f δ yy [k M T x, x -γ yy ] -f δ y y [k M T x , x -γ y y ] (f is L-lipschitz.) ≤ L δ yy [k M T x, x -γ yy ] -δ y y [k M T x , x -γ y y ] (Triangle inequality.) ≤ L δ yy k M T x, x -δ y y k M T x , x + δ yy γ yy -δ y y γ y y (δ yy , δ y y ∈ {-1, 1}.) ≤ 2L sup z,z ∼D T k M T x,
• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k =

L inf z,z ∼D T |γ yy | , • k-lipschitz continuous with k = L inf z,z ∼D T |γ yy | sup z,z ∼D T g(x, x ) * , • (σ, m)-admissible with    σ = 0 m = 2 L inf z,z ∼D T |γ yy | sup z,z ∼D T g(x, x ) * LT (M S ) λ + M S .
Proof. First the loss is positive and convex by construction.

Then we prove that the loss function is k-lipschitz. Given two metrics k M and k M we have: )

l M, z, z -l M , z, z ≤ f 1 -δ yy k M (x, x ) γ yy -f 1 -δ yy k M (x, x ) γ yy (f is L-lipschitz.) ≤ L 1 -δ yy k M (x, x ) γ yy -1 -δ yy k M (x, x ) γ yy (δ yy ∈ {-1, 1}.) ≤ L inf z,z ∼D T γ yy k M x, x -k M x, x (
≤ L inf z,z ∼D T γ yy sup z,z ∼D T g(x, x ) * M -M (C.16)
Inequalities (C.15) and (C.16) respectively prove the lipschitzness with respect to the metric and the matrix. Lastly we show that the loss is (σ, m)-admissible. Given four examples z, z , z , z ∼ D T and M T the learned metric when learning with T we have:

l M T , z, z -l M T , z , z ≤ f 1 -δ yy k M T (x, x ) γ yy -f 1 -δ y y k M T (x , x ) γ y y (f is L-lipschitz.) ≤ L δ yy k M T (x, x ) γ yy -δ y y k M T (x , x ) γ y y (δ yy , δ y y ∈ {-1, 1}.) ≤ 2 L inf z,z ∼D T γ yy sup z,z ∼D T k M T x, x
(Equation (4.14) and Cauchy-Schwartz's inequality (Theorem A.3).)

≤ 2 L inf z,z ∼D T γ yy sup z,z ∼D T g(x, x ) * M T (Triangle inequality.) ≤ 2 L inf z,z ∼D T γ yy sup z,z ∼D T g(x, x ) * ( M T -M S + M S )
(Bounded regularization (Lemma 4.9).)

≤ 2 L inf z,z ∼D T γ yy sup z,z ∼D T g(x, x ) *   LT (M S ) λ + M S   .
Setting σ = 0 and m = 2 where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k = √ 12HB,

• k-lipschitz continuous with k = √ 12HB sup z,z ∼D T g(x, x ) * , • (σ, m)-admissible with σ = 0 m = B .
Proof. First the loss is positive and convex by construction. Then we prove that the loss function is k-lipschitz. Given two metrics k M and k M we have: C.12 Proofs of Table 4

l M, z, z -l M , z, z ≤ f δ yy [k M x, x -γ yy ] -f δ yy [k M x, x -γ yy ] (Lemma A.1 with f which is H-smooth and B bounded.) ≤ √ 12HB 1 -δ yy k M (x, x ) γ yy -1 -δ yy k M (x, x ) γ yy (δ yy ∈ {-1, 1}.) ≤ √ 12HB inf z,z ∼D T γ yy k M x, x -k M x, x (
R n ( • * ) ≤ 2 sup z,z ∼D T g(x, x ) 2 F √ n . (C.23)
Proof. The results presented here have already been proven in [START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] in a slightly less general setting where g(x, x ) = (x -x )(x -x ) T . We recall the proof below for the sake of completeness.

For the Frobenius norm, from the definition of Rademacher Averages we have:

R n ( • F ) = E T ∼D T 1 n 2 E σ n 2 i=1 σ i g(x i , x n 2 +i ) F (Jensen's inequality (Theorem A.4).) ≤ E T ∼D T 1 n 2 E σ n 2 i=1 σ i g(x i , x n 2 +i ) 2 F (Definition of • F .) ≤ E T ∼D T 1 n 2 E σ j,k    n 2 i=1 σ i g(x i , x n 2 +i )    2 j,k
(Standard properties on Rademacher Variables.)

≤ E T ∼D T 1 n 2 E σ n 2 i=1 j,k σ i g(x i , x n 2 +i ) 2 j,k (Definition of • F .) ≤ E T ∼D T 1 n 2 n 2 i=1 g(x i , x n 2 +i ) 2 F ≤ E T ∼D T 1 n 2 n 2 sup z,z ∼D T g(x, x ) 2 F ≤ 2 sup z,z ∼D T g(x, x ) F √ n .
Noting that the ∞ norm, the 2,∞ norm and the spectral norm are always smaller than the Frobenius norm and that the Rademacher Average is increasing when the value of the norm is increasing gives the example.

= x T L -v T 2 -x T L -v T 2 x T L -v T 2 + x T L -v T 2 (Triangle inequality.) ≤ x T L -v T -x T L + v T 2 x T L -v T 2 + x T L -v T 2
(Bounded loss (Lemma 5.1).)

≤ L -L F 2B v B x 1 + B x √ λ . Setting k = 2B v B x 1 + Bx √ λ
gives the lemma.

D.4 Proof of Lemma 5.3

To prove Lemma 5.3 we need the following technical lemma.

Lemma D.2. Let F V (L) = LV (L) + λ L 2 F and F i V (L) = LV (L) + λ L 2
F be the functions minimized in Problem (5.1) where V and V i are two training samples of n examples. V i is obtained by replacing example i from V by another example drawn independently from D V . Let L V and L V i be their respective minimizers, and λ be the regularization parameter used in our algorithm. Let ∆

L = L V -L V i , then, we have, for any t ∈ [0, 1], L V 2 F -L V -t∆ L 2 F + L V i 2 F -L V i + t∆ L 2 F ≤ 4tB v B x λn 1 + B x √ λ ∆ L F . (D.1)
Proof. This proof is similar to the proof of Lemma 20 in Bousquet and Elisseeff (2002a) which we recall here for the sake of completeness. First, note that LV is a convex function, thus, for any t ∈ [0, 1], we have:

LV i (L V -t∆ L ) -LV i (L V ) ≤ t( LV i (L V i ) -LV i (L V )) (D.2) LV i (L V i + t∆ L ) -LV i (L V i ) ≤ t( LV i (L V ) -LV i (L V i )) (D.3)
Summing Inequalities (D.2) and (D.3) gives:

LV i (L V -t∆ L ) -LV i (L V ) + LV i (L V i + t∆ L ) -LV i (L V i ) ≤ 0 (D.4) L V and L V i respectively minimize F V and F V i (L), we have: F V (L V ) -F V (L V -t∆ L ) ≤ 0 (D.5) F V i (L V i ) -F V i (L V i + t∆ L ) ≤ 0 (D.6)
Summing Inequalities (D.4), (D.5) and (D.6) gives:

LV i (L V -t∆ L ) -LV i (L V ) + LV (L V ) -LV (L V -t∆ L ) + λ L V 2 F -λ L V -t∆ L 2 F + λ L V i 2 F -λ L V i + t∆ L 2 F ≤ 0. (D.7)
From Equation (D.7), we can write: Dans cette thèse nous sommes principalement intéressés par les problèmes d'Apprentissage Supervisé. L'idée derrière ce paradigme est que les exemples sont accompagnés d'une étiquette. Celle-ci peut être une valeur ou une classe et correspond à la solution du problème pour l'exemple associé. Pour illustrer cela, nous considérons le problème de prédiction du prix des habitations et celui de la reconnaissance de champignons empoisonnés. Dans le premier cas, le but est de prédire le prix d'une maison, chaque exemple correspond alors à un ensemble de caractéristiques du bâtiment tandis que l'étiquette correspond à son prix. Dans le second cas, le but est de reconnaitre, à partir d'images, les champignons mangeables de ceux qui sont empoisonnés. Chaque exemple est alors la photo d'un champignon tandis que l'étiquette correspond à sa classe, i.e. empoissonné ou pas. De ces exemples, nous remarquons l'importance, en apprentissage supervisé, de la généralisation aux nouvelles données. En effet, les étiquettes des exemples d'apprentissage étant données, l'intérêt d'un modèle qui n'est pas capable de prédire la bonne étiquette pour un nouvel exemple est limité. Notons que les deux exemples précédents correspondent à des problèmes largement étudiés en apprentissage supervisé : la régression et la classification. La différence entre les deux est que le but du premier est de prédire une valeur continue tandis que l'objectif du second est de déterminer une classe discrète.

λ L V 2 F -λ L V -t∆ L 2 F + λ L V i 2 F -λ L V i + t∆ L 2 F ≤ C (D.8)
L'apprentissage supervisé n'est pas le seul paradigme existant en apprentissage automatique. Il peut, en fait, être opposé à celui de l'apprentissage non supervisé où les exemples ne sont pas associés à des étiquettes. Par exemple un problème largement étudié est celui du partitionnement de données où l'objectif est d'obtenir une séparation sensée de l'espace, c'est à dire de regrouper les exemples qui partagent des propriétés communes. La performance des algorithmes d'apprentissage non supervisé est difficile à évaluer en pratique. En effet, contrairement à l'apprentissage supervisé, il n'y a pas d'étiquettes donnant un retour évident sur le modèle appris.

S'inspirant de ces deux paradigmes, l'idée derrière l'Apprentissage Semi-Supervisé est de considérer deux ensembles d'apprentissage, le premier est étiqueté tandis que le second ne l'est pas. Dans ce cas, le but est souvent d'utiliser les exemples étiquetés pour aider à résoudre une tâche d'apprentissage non supervisé ou d'utiliser les exemples non étiquetés pour aider à résoudre une tâche d'apprentissage supervisé.

Jusque là nous avons considéré que le but des différentes approches d'apprentissage automatique est de résoudre une tâche unique. Prenant un point de vue différent, l'idée derrière l'Apprentissage par Transfert est de transférer la connaissance apprise sur un problème source à un problème cible. De façon similaire, l'idée derrière l'Adaptation de Domaine est de transférer le modèle appris sur une tâche source pour résoudre un problème cible qui est différent mais relié. Par exemple, dans le problème de la détection de spams, les deux tâches peuvent être de détecter les mails non désirés dans les boîtes de deux utilisateurs différents. Ces deux utilisateurs rencontrent le même problème mais la distribution de leurs mails diffère, e.g. ils ne sont pas abonnés aux mêmes listes de diffusion. Dans ce cas le but est d'adapter le modèle appris pour un des utilisateurs à l'autre.

Dans ce manuscrit nous verrons que même si nous nous intéressons principalement à des problèmes d'apprentissage supervisé, plusieurs de nos contributions sont aussi liées aux différents paradigmes présentés ici.

Lors de la présentation du paradigme de l'apprentissage supervisé nous avons insisté sur le fait qu'un modèle, appris en utilisant un nombre limité d'exemples d'apprentissage, devrait pouvoir généraliser à de nouveaux exemples. Une façon de vérifier cette propriété est d'évaluer le modèle appris sur un nouvel ensemble d'exemples de test indépendants des exemples d'apprentissage et pour lesquels la solution au problème est connue. Cependant, le nombre d'exemples qui peuvent être obtenus est souvent limité. Cela implique que cette approche ne suffit bien souvent pas à assurer que le modèle généralise bien. D'autres méthodes sont alors nécessaires. Pour cela, notons d'abord qu'une supposition commune en apprentissage automatique est que la tâche que nous cherchons à résoudre est complètement définie par une distribution inconnue à partir de laquelle les exemples d'apprentissage sont tirés. Par suite, une solution possible est d'utiliser une procédure de validation croisée où l'idée est de séparer l'ensemble d'apprentissage en k parties. Le modèle est alors appris sur k -1 parties et testé sur la dernière. Cette procédure est alors répétée k fois, i.e. jusqu'à ce que chaque partie est été utilisée comme partie de test, et la précision est obtenue comme moyenne des différentes répétitions. Cette procédure requiert aussi un nombre significatif d'exemples pour être pertinente. Une autre possibilité qui suit de la supposition évoquée précédemment consiste à procéder à une analyse théorique de l'algorithme d'apprentissage et de dériver des bornes appelées bornes en généralisation. L'idée de ces bornes est de montrer que l'erreur réelle du modèle appris, i.e. son erreur sur la distribution inconnue, est bornée par son erreur empirique, i.e. son erreur sur l'ensemble d'apprentissage, à laquelle s'ajoute un terme qui décroit avec l'augmentation de la taille de l'ensemble d'apprentissage. L'obtention de telles bornes est une garantie que les modèles appris par l'algorithme concerné généralisent raisonnablement bien.

De nombreuses approches ont été proposées pour résoudre les problèmes posés par l'apprentissage supervisé. Parmi celles-ci plusieurs dépendent fortement d'une notion de distance ou de similarité entre les exemples pour apprendre un modèle. Un exemple très représentatif est le classificateur des plus proches voisins qui est basé sur l'idée que deux exemples similaires devraient partager la même étiquette. Un autre exemple est l'algorithme des machines à vecteurs de support qui propose de classer les exemples en fonction de leur similarité à des points spécifiques nommés vecteurs de support. Dans ces deux exemples la notion de similarité utilisée est d'une importance critique. Cependant des tâches différentes requièrent souvent des mesures de similarité différentes. Par exemple, considérant les exemples évoqués précédemment dans cette introduction, il semble mal venu de comparer les habitations et les champignons de la même façon. Manuellement choisir une mesure de similarité appropriée peut être fastidieux et difficile. Cependant il devrait être possible de l'inférer de façon automatique à partir des données. C'est l'idée derrière l'Apprentissage de Métriques qui correspond au problème auquel nous allons nous intéresser dans cette thèse.

Nous identifions plusieurs limites des approches actuelles en apprentissage de métriques. Tout d'abord plusieurs méthodes proposent d'utiliser des informations supplémentaires pour aider durant le processus d'apprentissage. Cependant il n'y a pas de compréhension théorique de l'impact de ces informations sur la métrique apprise. Ensuite les propriétés intrinsèques des métriques apprises sont souvent les mêmes. En effet celles-ci sont généralement apprises avec l'idée de rapprocher les exemples similaires et d'éloigner les exemples dissimilaires. Dans certains cas il pourrait être intéressant de considérer d'autres types de contraintes. Un exemple est l'obtention d'une métrique dont le comportement n'est pas limité aux exemples mais est plus global dans le sens où elle est, par exemple, capable de bouger des blocs d'exemples en tant que tels. Enfin, une troisième limite des approches actuelles est qu'elles ne sont, bien souvent, pas justifiées théoriquement, i.e. aucune garantie n'est proposée concernant la Dans cette thèse nous proposons plusieurs approches pour apprendre des métriques dont le comportement est contrôlé. Dans une première partie nous proposons d'utiliser une information supplémentaire qui prend la forme d'une métrique de référence ou métrique source pour guider de façon stricte ou plus relâchée la métrique apprise. Ainsi, dans notre première contribution, nous nous intéressons au problème de la régression des valeurs d'une métrique de référence uniquement accessible à travers un ensemble d'apprentissage de taille limité. Dans notre seconde contribution nous étudions de façon théorique comment utiliser une métrique de référence venant d'un problème lié mais différent peut aider lors du processus d'apprentissage. En particulier nous dérivons plusieurs mesures de l'apport de la métrique source pour le problème considéré. Dans une seconde partie nous proposons deux approches capables de considérer de nouvelles formes de contraintes pour l'apprentissage de métriques. Ainsi, dans notre troisième contribution, nous considérons que les exemples d'apprentissage ne devraient pas bouger les uns par rapport aux autres mais plutôt par rapport à des points virtuels qui se trouvent déjà dans l'espace induit par la métrique apprise. Cette méthode nous permet de contrôler de manière précise le mouvement de chaque exemple. Dans notre quatrième contribution nous étendons notre troisième contribution et considérons de récentes avancées dans le domaine du Transport Optimal pour proposer une nouvelle approche pour apprendre une métrique capable de bouger des blocs d'exemples dans l'espace. Enfin, remarquons que dans cette thèse nous proposons, autant que possible, des approches qui sont justifiées théoriquement.

Plan général

Dans la première partie de cette thèse nous présentons plusieurs éléments préliminaires. Dans le premier chapitre nous introduisons des concepts qui seront utilisés tout au long de ce manuscrit tandis que dans le second chapitre nous proposons une revue de l'état de l'art en apprentissage de métriques.

Chapitre 1 Le premier chapitre de cette thèse est dédié à la présentation de plusieurs notions et outils utilisés dans celle-ci. La première partie de ce chapitre présente le cadre de travail de la minimisation du risque sur lequel sont basées toutes nos contributions algorithmiques. La seconde partie est dédiée à l'analyse théorique des algorithmes. Plus précisément nous présentons deux cadres théoriques utilisés pour dériver des bornes en généralisation et basés respectivement sur la stabilité uniforme et la complexité de Rademacher. La troisième partie s'intéresse à la notion de fonction de perte et de terme de régularisation qui sont des éléments clés de l'apprentissage par minimisation du risque. Au travers de plusieurs exemples nous montrons qu'il existe de nombreux choix avec des propriétés différentes. Cette troisième partie s'intéresse aussi à la présentation d'une définition formelle de la notion de métrique comme terme général pour désigner une similarité, une dissimilarité ou une distance. De la même façon que pour les fonctions de perte et les termes de régularisation plusieurs exemples sont présentés. La dernière partie de ce premier chapitre introduit plusieurs autres notions utiles telles que le classificateur des plus proches voisins, que nous utiliserons souvent avec les métriques que nous apprenons, et le problème de l'adaptation de domaine que nous utilisons pour évaluer deux de nos contributions.

Chapitre 2 Le second chapitre de cette thèse correspond à une revue de l'état de l'art en apprentissage de métriques. Dans celle-ci nous présentons les approches principales qui ont fait le succès de ce domaine. Nous proposons de diviser cette revue en quatre parties qui correspondent aux réponses à quatre questions basiques sur les problèmes de l'apprentissage de métriques. Dans la première partie nous considérons les différentes sortes de métriques pouvant être apprises. Ensuite, dans la seconde partie, nous répondons à la question de savoir comment ces métriques peuvent être apprises en pratique. Dans la troisième partie de ce chapitre nous présentons plusieurs approches qui s'intéressent aux questions théoriques liées à l'apprentissage de métriques. Enfin, dans la dernière partie, nous présentons plusieurs travaux qui s'intéressent à l'utilisation de l'apprentissage de métriques dans des applications qui vont de la classification au partitionnement en passant par l'adaptation de domaine.

Dans la seconde partie de cette thèse nous présentons nos deux premières contributions. Elles s'intéressent à l'utilisation d'une métrique de référence comme aide lors du processus d'apprentissage.

Chapitre 3 Dans le troisième chapitre de cette thèse nous présentons notre première contribution. Elle correspond à une méthode d'apprentissage capable d'approximer une métrique existante. La première partie de ce chapitre est dédiée à la présentation du problème d'optimisation qui correspond à une régression des valeurs d'une métrique. De plus nous montrons que quand la métrique de référence est trop complexe, il est possible d'utiliser une approche locale pour obtenir une meilleure approximation. Dans la deuxième partie nous analysons théoriquement notre approche dans le cas global mais aussi dans le cas local. Cela montre que les métriques apprises par notre approche généralisent bien. Dans les troisième et quatrième parties de ce chapitre nous considérons le problème de l'apprentissage de distances couleur perceptuelles pour montrer l'intérêt de notre approche dans une application réelle.

Chapitre 4 Le quatrième chapitre de cette thèse est dédié à notre seconde contribution. Comme dans le troisième chapitre il s'agit d'une approche d'apprentissage de métriques capable de prendre en compte la connaissance donnée par une métrique de référence. La principale différence est que, dans ce chapitre, le but n'est pas d'approximer cette métrique mais plutôt de l'utiliser pour aider au cours du processus d'apprentissage. Cette contribution est ainsi fortement liée aux domaines de l'apprentissage par transfert et de l'adaptation de domaine. Ce chapitre est divisé en sept parties. Dans la première nous présentons le cadre de travail de l'apprentissage de métriques par transfert d'hypothèses qui correspond à un problème de minimisation avec terme de régularisation biaisé. Dans les deuxième, troisième et quatrième parties nous proposons une analyse théorique du cadre de travail proposé en utilisant trois approches théoriques différentes. Cela nous permet de dériver plusieurs mesures de l'apport de la métrique de référence. Dans la cinquième partie de ce chapitre nous résumons les différentes bornes et dans la sixième partie nous présentons plusieurs fonctions de perte et termes de régularisation pouvant être utilisés dans notre cadre de travail. Dans la dernière partie nous montrons que ce cadre peut être utilisé en pratique pour obtenir des résultats compétitifs sur plusieurs tâches d'apprentissage par transfert.

Dans la dernière partie de cette thèse nous introduisons nos deux dernières contributions où nous proposons de nouvelles façons de contrôler le comportement des métriques apprises.

Chapitre 5 Dans ce cinquième chapitre nous présentons notre troisième contribution. Dans celle-ci, plutôt que d'utiliser les contraintes classiques de similarités et dissimilarités nous proposons de considérer que la métrique devrait rapprocher les exemples d'apprentissage de points virtuels définis à priori. Cela nous permet d'apprendre une métrique à l'aide d'une régression et de réduire le nombre de contraintes considérées. Dans la première partie de ce chapitre nous présentons notre algorithme. Dans la seconde nous adressons le problème de sélectionner les points virtuels et de définir les contraintes. Dans la troisième partie nous proposons une analyse théorique de l'algorithme proposé et nous montrons, d'une part, qu'apprendre une métrique avec notre approche est fondé et, d'autre part, qu'il est possible de dériver des liens avec une approche plus classique d'apprentissage de métriques. Dans la dernière partie nous validons empiriquement l'intérêt de notre approche.

Chapitre 6 Le sixième chapitre de cette thèse introduit la dernière contribution de celleci. Il s'agit d'une nouvelle méthode capable d'apprendre une métrique pouvant bouger des blocs d'exemples en approximant la transformation correspondant à la solution d'un problème de transport optimal. Dans la première partie de ce chapitre nous introduisons de manière formelle le problème du transport optimal. Dans la deuxième partie nous présentons notre formulation tandis que dans la troisième nous proposons une approche efficace pour l'optimiser. Dans la quatrième partie de ce chapitre nous proposons une discussion théorique qui montre que si les suppositions classiques faites dans le domaine du transport optimal sont correctes alors notre approche est fondée. Dans la dernière partie nous proposons une validation empirique de notre méthode sur des problèmes d'adaptation de domaine et d'édition d'images.

E.2 Résumé du Chapitre 1

Dans ce chapitre nous présentons plusieurs notions essentielles à la bonne compréhension de cette thèse. En particulier nous formalisons le cadre de travail de l'apprentissage par minimisation empirique du risque sur lequel sont basées nos différentes contributions algorithmiques. De la même façon nous présentons deux cadres théoriques qui permettent de dériver des bornes en généralisation pour la minimisation du risque. Comme nous le verrons dans le second chapitre ces deux cadres théoriques ont été étendus avec succès au problème de l'apprentissage de métriques. Dans ce manuscrit nous les utiliserons pour démontrer que nos algorithmes sont capables d'apprendre des métriques qui généralisent bien. D'un point de vue plus pratique nous présentons plusieurs fonctions de perte et termes de régularisation pouvant être utilisés dans le cadre de la minimisation du risque. Nous proposons aussi une définition formelle de la notion de métrique considérée dans cette thèse. Pour finir nous présentons l'algorithme de classification des plus proches voisins et le problème de l'adaptation de domaine qui seront utilisés pour empiriquement démontrer l'intérêt de la plupart de nos contributions.

E.3 Résumé du Chapitre 2

Dans ce chapitre nous proposons une revue, non exhaustive, de l'état de l'art en apprentissage de métriques. Ainsi nous nous intéressons tout particulièrement aux méthodes proches de nos contributions. Cela correspond à des approches qui apprennent le même genre de métriques, considèrent des façons similaires d'effectuer l'étape d'apprentissage, dérivent le même genre de bornes en généralisation ou apprennent une métrique pour résoudre les mêmes tâches.

E.4 Résumé du Chapitre 3

Dans ce chapitre nous nous intéressons au problème de l'estimation d'une métrique de référence inconnue à partir d'un ensemble de paires d'exemples. Une solution à ce problème est d'utiliser l'apprentissage de métriques pour approximer de façon automatique les valeurs de cette métrique de référence. Cependant, la plupart des algorithmes d'apprentissage de métriques s'intéressent à l'estimation de la proximité relative des exemples d'apprentissage plutôt qu'à la distance effective qui les sépare. Dans ce chapitre nous proposons un nouvel algorithme d'apprentissage de métriques locales nous permettant, à l'aide d'une distance de Mahalanobis, d'approximer de façon précise une métrique de référence. En utilisant le cadre théorique de la stabilité uniforme nous dérivons des bornes en généralisation sur le modèle appris qui montrent que notre méthode est fondée théoriquement. De plus nous évaluons notre approche sur un problème de vision par ordinateur consistant à calculer des différences de couleurs qui soient perceptuellement uniformes. Avoir des distances qui reflètent la perception humaine des couleurs de la scène est essentiel dans les applications de vision par ordinateur comme la segmentation d'images ou la détection d'objets saillants. Cependant, dans la plupart des cas, il est uniquement possible d'avoir accès aux couleurs de l'image sans aucun moyen de revenir aux couleurs de la scène. Il existe deux approches principales permettant de résoudre ce problème. D'un côté, il est possible de calculer directement une distance perceptuelle entre les couleurs de l'image considérée. Cependant cette distance est coûteuse à calculer et dépend des conditions d'acquisition ce qui implique qu'elle est bien souvent loin des différences entre couleurs de la scène. D'un autre côté, il est possible d'estimer les couleurs de la scène à partir de celles de l'image puis de calculer une distance perceptuellement uniforme. Cependant, cela implique une connaissance sur les conditions d'acquisition qui n'est pas raisonnable pour la plupart des applications. Notre approche nous permet d'apprendre une métrique qui est à la fois invariante aux conditions d'acquisition et calculable à partir des couleurs des images. Nous évaluons l'intérêt de cette dernière en montrant sa capacité à (i) généraliser à de nouvelles couleurs et de nouveaux appareils photographiques et (ii) aider dans un problème de segmentation.

E.5 Résumé du Chapitre 4

Nous considérons le problème du transfert de connaissances à priori dans le contexte de l'apprentissage supervisé de métriques. De façon plus précise nous considérons des problèmes à régularisation biaisée qui utilisent un métrique de référence, une métrique source, venant d'un problème différent mais relié et pouvant potentiellement aider lors de l'apprentissage d'une métrique avec peu de données. Si ce cadre a déjà été appliqué avec succès de manière empirique, il n'existe pas de cadre théorique justifiant une telle approche. Dans ce chapitre, nous proposons de résoudre ce problème en proposant une analyse théorique basée sur trois approches différentes. Tout d'abord nous présentons une nouvelle définition de la stabilité, on-average-replace-two-stability, qui nous permet de montrer des bornes en généralisation en moyenne avec un taux de convergence rapide lorsqu'une métrique source auxiliaire est utilisée pour biaiser le terme de régularisation. Ensuite nous considérons une notion de stabilité algorithmique adaptée au cadre de l'apprentissage de métriques régularisé et nous prouvons une borne en généralisation probabiliste montrant l'intérêt d'utiliser un terme de régularisation biaisé avec pondération de la métrique source. Nous proposons une solution algorithmique à ce problème de pondération que nous évaluons (i) dans un problème d'apprentissage de métriques classique et (ii) dans un problème d'apprentissage par transfert avec peu de données cibles. Enfin nous dérivons une borne en généralisation basée sur la complexité de Rademacher de la classe de métriques considérée en prenant notamment en compte la métrique de référence. Cette borne souligne l'intérêt d'utiliser une bonne métrique source en montrant que, lorsque celle-ci est une solution idéale au problème, l'apprentissage n'est plus nécessaire. Pour justifier l'intérêt de ce cadre de travail nous proposons plusieurs exemples de fonctions de perte et de termes de régularisation qui peuvent être utilisés dans le cadre d'une ou plusieurs des approches théoriques considérées.

E.6 Résumé du Chapitre 5

Dans ce chapitre nous nous intéressons à l'apprentissage supervisé de distances de type Mahalanobis. Les approches existantes cherchent principalement à apprendre un nouvel espace de représentation en fonction de contraintes prenant en compte des informations de similarité et de dissimilarité entre les exemples. Ici, au lieu de rapprocher où d'éloigner les exemples selon ce type de contraintes, nous proposons d'introduire le concept de points virtuels nous servant de supports pour le déplacement des exemples d'apprentissage. Ainsi, les exemples d'apprentissage sont rapprochés d'un point virtuel qui leur a été affecté à priori permettant alors de réduire le nombre de contraintes à satisfaire et de contrôler de façon explicite le comportement de la métrique pour chaque exemple. Nous montrons que l'approche proposée peut être résolue en forme close et qu'il est alors possible de travailler dans l'espace induit par un noyau. Nous proposons deux analyses théoriques, la première prouvant la capacité de généralisation des métriques apprises avec notre méthode et la seconde établissant des liens avec une approche d'apprentissage de métriques classique. De plus nous proposons deux solutions efficaces au difficile problème de la sélection des points virtuels, l'une d'elle étant basée sur de récentes avancées dans le domaine du transport optimal. Pour finir, nous évaluons notre approche sur plusieurs jeux de données classiques en apprentissage de métriques.

E.7 Résumé du Chapitre 6

Dans ce chapitre nous proposons d'adresser le problème de l'apprentissage d'une transformation, induisant une distance de Mahalanobis, qui approxime une transformation géométrique particulière. Une telle métrique pourrait être très bénéfique dans le contexte de l'adaptation de domaine où le but est d'aligner les domaines sources et cibles. Ici nous proposons de considérer des transformations géométriques induites par la résolution d'un problème de transport optimal. En effet, il s'agit d'une procédure raisonnable pour aligner des distributions et sa capacité à résoudre des problèmes d'adaptation de domaine a déjà été démontrée. La plupart des approches en transport optimal utilisent la formulation donnée par Kantorovich et apprennent un couplage probabiliste Γ entre les différents exemples d'apprentissage. Cependant elles n'abordent pas le problème de l'apprentissage de la transformation f S→T liée au problème de Monge. En conséquence le couplage appris ne peut-être utilisé que sur les exemples d'apprentissage et pas sur de nouveaux exemples ce qui réduit l'intérêt potentiel de telles approches. Dans ce chapitre nous proposons de combiner l'apprentissage de métriques et le transport optimal dans un nouveau cadre de travail nous permettant d'apprendre conjointement le couplage et une approximation de la transformation correspondante. Cette approximation prend la forme d'une matrice L correspondant à une nouvelle métrique dans le domaine source. Dans ce cas nous montrons que notre approche est liée à RVML, présenté dans le Chapitre 5, où les points virtuels associés à chaque exemple sont définis comme le résultat du couplage induit par le transport. Cependant, plutôt que de considérer que le couplage est défini a priori, nous proposons de l'apprendre en même temps que la métrique.

Ainsi, nous obtenons une formulation jointe et convexe pouvant être optimisée de façon efficace et ayant le bénéfice de lisser le résultat du transport optimal. En pratique nous montrons l'intérêt de notre méthode pour deux tâches, l'une en adaptation de domaine et l'autre en édition d'images.

E.8 Conclusion

Dans cette thèse nous avons adressé le problème de l'apprentissage de métriques à comportement contrôlé. Nous avons considéré deux types de contrôle sur la métrique apprise. D'une part, nous avons considéré le problème de l'apprentissage par rapport à une métrique de référence donnée soit sous la forme d'une distance pour un nombre limité de paires d'exemples soit directement sous la forme d'un modèle. D'autre part, nous avons considéré le problème de l'apprentissage de la transformation induite par une distance de Mahalanobis soit pour contrôler de façon précise le mouvement de chaque exemple soit pour approximer une transformation géométrique. Nos différentes contributions sont à la fois algorithmiques et théoriques.

Résumé des Contributions

La plupart des algorithmes d'apprentissage de métriques s'intéressent à l'obtention de métriques capables de rapprocher les exemples similaires tout en éloignant les exemples dissimilaires. Cependant, il peut parfois être intéressant de prédire une valeur précise entre deux exemples. C'est par exemple le cas lorsque l'on a accès à un nombre limité de paires d'exemples pour lesquelles la valeur d'une métrique de référence est connue. Dans notre première contribution nous avons adressé le problème de l'approximation de cette métrique de référence. Nous avons proposé une approche d'apprentissage de métriques locales que nous avons analysé théoriquement pour montrer que si le modèle a été appris avec un nombre suffisant d'exemples, il généralise bien. De plus nous avons évalué notre approche sur le problème de vision par ordinateur qu'est l'estimation de distances couleur perceptuelles. Pour cela nous avons créé un nouveau jeu de données spécialement dédié à cette tâche. Nos résultats empiriques ont montré le bon comportement de notre approche ainsi que sa capacité à approximer la métrique de référence. Le nouveau jeu de données ainsi que la distance perceptuellement uniforme apprise sont distribués gratuitement (Perrot et al., 2014a).

Plusieurs approches d'apprentissage de métriques montrent de façon empirique l'intérêt d'utiliser une information supplémentaire, sous forme d'une métrique source, mais ne prouvent pas ces bénéfices de façon théorique. Dans notre deuxième contribution nous avons proposé de résoudre ce problème. Ainsi, nous avons formalisé le cadre de travail de l'apprentissage de métriques par transfert d'hypothèse où l'idée est de prendre en compte une métrique source dans un terme de régularisation biaisé. Nous avons proposé une analyse théorique de ce cadre nous permettant de dériver trois mesures différentes de l'apport d'une métrique source. Ces mesures représentent différents moyens d'évaluer l'intérêt de la métrique de référence pour le problème considéré. Deux de ces mesures sont théoriques et donc difficiles à utiliser en pratique. La troisième, cependant, est empirique ce qui signifie qu'elle peut être utilisée pour sélectionner la meilleur métrique source dans un ensemble. Pour illustrer cela nous avons proposé un algorithme de pondération de l'importance de la métrique source. Nous avons de plus démontré l'intérêt de notre cadre de travail en montrant que de nombreuses fonctions de perte et termes de régularisations pouvaient être utilisés. Enfin nous l'avons empiriquement évalué sur un problème d'apprentissage de métriques classique mais aussi sur une tâche d'adaptation de domaine semi-supervisé.

La plupart des approches d'apprentissage de métriques utilisent des contraintes de similarité et dissimilarité pour apprendre une métrique mais ne contrôlent pas de façon explicite le comportement de la transformation induite. Dans notre troisième contribution nous avons adressé ce problème en proposant une nouvelle approche où la destination des exemples, après projection par la transformation, est choisie de manière explicite à l'aide de points virtuels. Cela nous a permis de contrôler de manière précise la métrique apprise et donc d'apprendre des modèles plus adaptés à la tâche considérée. Par exemple, pour un problème de classification, nous avons proposé des points virtuels basés sur les différentes classes de telle façon que chaque axe de l'espace de projection de la métrique apprise soit discriminant pour une classe particulière. Nous avons montré que notre approche peut facilement apprendre à partir de l'espace induit par un noyau et donc apprendre des métriques très expressives. Nous avons aussi proposé une étude théorique montrant des liens entre notre méthode et une approche classique d'apprentissage de métriques. Enfin, nous avons démontré empiriquement ses bonnes performances sur plusieurs jeux de données classiques.

Dans notre quatrième contribution nous avons abordé un problème similaire à celui de notre troisième contribution. Cependant, au lieu de contrôler de façon explicite le comportement de chaque exemple individuellement, nous avons proposé de forcer la métrique à suivre une transformation géométrique particulière. Ainsi, nous avons considéré des transformations induites par le couplage appris par un problème de transport optimal discret, ce qui est d'un intérêt tout particulier pour des tâches d'adaptation de domaine. Nous avons proposé une solution pour apprendre de façon jointe ce couplage et la transformation induite par une métrique. Nous avons dérivé une méthode d'optimisation efficace et nous avons montré que cette approche pouvait être reliée à notre troisième contribution où les points virtuels et la transformation sont appris de façon jointe. Nous avons empiriquement démontré le bon comportement de notre approche pour un problème d'adaptation de domaine non supervisé ainsi que pour une tâche d'édition d'images.

Perspectives

Nous avons déjà présenté des perspectives spécifiques pour chacune de nos contributions. Dans cette partie, nous proposons plutôt de considérer des travaux futurs généraux qui peuvent représenter de nouvelles directions de recherche découlant des éléments présentés dans cette thèse. D'un point de vue algorithmique nos contributions sont principalement basées sur des problèmes d'optimisation directs. Une première perspective serait d'étendre les concepts présentés dans ce manuscrit au contexte de l'apprentissage en ligne. Ainsi, il pourrait être intéressant de développer des mécanismes capables de détecter de potentiels changements dans la distribution des exemples et d'alors changer automatiquement le comportement de la métrique considérée. Une telle approche pourrait, par exemple, être utilisée dans un contexte de suivi d'objets dans des vidéos où les variations dans la scène peuvent potentiellement appeler à des comportements différents de la métrique. Une autre perspective serait de considérer l'apprentissage actif pour améliorer le contrôle de la métrique. Par exemple, lorsque l'on apprend une transformation, il pourrait être intéressant d'obtenir un retour de l'utilisateur pour vérifier que les exemples sont correctement déplacés. Un domaine d'application pourrait être celui de l'adaptation de domaine, où l'apprentissage actif à déjà fait ses preuves [START_REF] Berlind | Active nearest neighbors in changing environments[END_REF], où obtenir des retours sur des exemples bien choisis pourrait assurer que la métrique estime de façon correcte les différences entre les distributions. D'un point de vue plus théorique notons que dans cette thèse nous nous sommes principalement intéressés à la capacité de généralisation des métriques apprises et pas à leur impact sur l'application dans laquelle elles sont utilisées. Partant de cette dernière idée, [START_REF] Balcan | Improved guarantees for learning via similarity functions[END_REF] ont montré que l'erreur d'un classificateur linéaire était liée à une mesure de l'apport de la similarité utilisée pour l'apprendre. Cette mesure de l'apport d'une métrique est reliée à sa capacité à rapprocher les exemples similaires et à éloigner les exemples dissimilaires. Cependant, lors de l'apprentissage d'une métrique à comportement contrôlé cette mesure ne sera pas forcément adaptée. Par exemple, lors de l'apprentissage d'une métrique à l'aide d'une métrique de référence ( Chapitres 3 et 4 ) il serait probablement plus intéressant de considérer une mesure prenant en compte cette information supplémentaire. De la même façon, lors de l'apprentissage d'une transformation pour une tâche d'adaptation de domaine ( Chapitres 4 et 6 ) il serait probablement plus intéressant de se focaliser sur la capacité de la métrique à aligner la source et la cible. Cela implique que la mesure de l'apport de la métrique dépend de la tâche considérée. Une perspective intéressante pourrait être de considérer des cadres théoriques capables de prendre en compte une notion d'apport reliée à la tâche considérée et de montrer qu'une bonne métrique est en effet bénéfique.

Une autre perspective théorique est la dérivation de bornes à convergence rapides en présence d'informations supplémentaires. Dans le Chapitre 4 nous avons proposé une première solution à ce problème en utilisant la complexité de Rademacher et l'information supplémentaire qu'est la mesure de l'apport de la métrique source dérivée. Cependant, cette solution n'est pas satisfaisante dans le sens où la contrainte imposée sur la métrique source était plus forte que le résultat obtenu sur la métrique apprise. Dans tous les cas, cela reste un résultat encourageant puisqu'il montre qu'en utilisant des suppositions fortes il est possible d'obtenir un taux de convergence rapide. Ainsi, s'il est possible d'obtenir des suppositions plus faibles ( Voir e.g. Srebro et al. (2010c) ) il pourrait être possible de dériver des résultats plus significatifs.

Abstract Many Machine Learning algorithms make use of a notion of distance or similarity between examples to solve various problems such as classification, clustering or domain adaptation. Depending on the tasks considered these metrics should have different properties but manually choosing an adapted comparison function can be tedious and difficult. A natural trend is then to automatically tailor such metrics to the task at hand. This is known as Metric Learning and the goal is mainly to find the best parameters of a metric under some specific constraints. Standard approaches in this field usually focus on learning Mahalanobis distances or Bilinear similarities and one of the main limitations is that the control over the behaviour of the learned metrics is often limited. Furthermore if some theoretical works exist to justify the generalization ability of the learned models, most of the approaches do not come with such guarantees. In this thesis we propose new algorithms to learn metrics with a controlled behaviour and we put a particular emphasis on the theoretical properties of these algorithms. We propose four distinct contributions which can be separated in two parts, namely (i) controlling the metric with respect to a reference metric and (ii) controlling the underlying transformation corresponding to the learned metric. Our first contribution is a local metric learning method where the goal is to regress a distance proportional to the human perception of colors. Our approach is backed up by theoretical guarantees on the generalization ability of the learned metrics. In our second contribution we are interested in theoretically studying the interest of using a reference metric in a biased regularization term to help during the learning process. We propose to use three different theoretical frameworks allowing us to derive three different measures of goodness for the reference metric. These measures give us some insights on the impact of the reference metric on the learned one. In our third contribution we propose a metric learning algorithm where the underlying transformation is controlled. The idea is that instead of using similarity and dissimilarity constraints we associate each learning example to a so-called virtual point belonging to the output space associated with the learned metric. We theoretically show that metrics learned in this way generalize well but also that our approach is linked to a classic metric learning method based on pairs constraints. In our fourth contribution we also try to control the underlying transformation of a learned metric. However instead of considering a point-wise control we consider a global one by forcing the transformation to follow the geometrical transformation associated to an optimal transport problem. From a theoretical standpoint we propose a discussion on the link between the transformation associated with the learned metric and the transformation associated with the optimal transport problem. On a more practical side we show the interest of our approach for domain adaptation but also for a task of seamless copy in images.

Résumé De nombreux algorithmes en Apprentissage Automatique utilisent une notion de distance ou de similarité entre les exemples pour résoudre divers problèmes tels que la classification, le partitionnement ou l'adaptation de domaine. En fonction des tâches considérées ces métriques devraient avoir des propriétés différentes mais les choisir manuellement peut-être fastidieux et difficile. Une solution naturelle est alors d'adapter automatiquement ces métriques à la tâche considérée. Il s'agit alors d'un problème connu sous le nom d'Apprentissage de Métriques et où le but est principalement de trouver les meilleurs paramètres d'une métrique respectant des contraintes spécifiques. Les approches classiques dans ce domaine se focalisent habituellement sur l'apprentissage de distances de Mahalanobis ou de similarités Bilinéaires et l'une des principales limitations est le fait que le contrôle du comportement de ces métriques est souvent limité. De plus, si des travaux théoriques existent pour justifier de la capacité de généralisation des modèles appris, la plupart des approches ne présentent pas de telles garanties. Dans cette thèse nous proposons de nouveaux algorithmes pour apprendre des métriques à comportement contrôlé et nous mettons l'accent sur les propriétés théoriques de ceux-ci. Nous proposons quatre contributions distinctes qui peuvent être séparées en deux parties: (i) contrôler la métrique apprise en utilisant une métrique de référence et (ii) contrôler la transformation induite par la métrique apprise. Notre première contribution est une approche locale d'apprentissage de métriques où le but est de régresser une distance proportionnelle à la perception humaine des couleurs. Notre approche est justifiée théoriquement par des garanties en généralisation sur les métriques apprises. Dans notre deuxième contribution nous nous sommes intéressés à l'analyse théorique de l'intérêt d'utiliser une métrique de référence dans un terme de régularisation biaisé pour aider lors du processus d'apprentissage. Nous proposons d'utiliser trois cadres théoriques différents qui nous permettent de dériver trois mesures différentes de l'apport de la métrique de référence. Ces mesures nous donnent un aperçu de l'impact de la métrique de référence sur celle apprise. Dans notre troisième contribution nous proposons un algorithme d'apprentissage de métriques où la transformation induite est contrôlée. L'idée est que, plutôt que d'utiliser des contraintes de similarité et de dissimilarité, chaque exemple est associé à un point virtuel qui appartient déjà à l'espace induit par la métrique apprise. D'un point de vue théorique nous montrons que les métriques apprises de cette façon généralisent bien mais aussi que notre approche est liée à une méthode plus classique d'apprentissage de métriques basée sur des contraintes de paires. Dans notre quatrième contribution nous essayons aussi de contrôler la transformation induite par une métrique apprise. Cependant, plutôt que considérer un contrôle individuel pour chaque exemple, nous proposons une approche plus globale en forçant la transformation à suivre une transformation géométrique associée à un problème de transport optimal. D'un point de vue théorique nous proposons une discussion sur le lien entre la transformation associée à la métrique apprise et la transformation associée au problème de transport optimal. D'un point de vue plus pratique nous montrons l'intérêt de notre approche pour l'adaptation de domaine mais aussi pour l'édition d'images.

  The number of examples n: as the number of examples increases the value of ε(H, n, δ) should decrease. Furthermore a desirable property is to have lim n→+∞ ε(H, n, δ) = 0. Indeed when we have access to all the examples L T (h) = LT (h).

Figure 1 . 2 :

 12 Figure 1.2: Graphical representation of commonly used loss functions for the classification problem.

Figure 1 . 3 :

 13 Figure 1.3: Graphical representation of commonly used loss functions for the regression problem.

Figure 1 . 4 :

 14 Figure 1.4: Two dimensional representation of the p norm for different values of p. Each ball represents all the points with a norm of 1.

  : The idea behind this classification algorithm is to consider that examples which are close to each other share the same label. Hence to predict the label of a new example the algorithm considers its k nearest examples in the training set and chooses the majority label. Here the notion of metric is critical as one has to compare any new examples to the training examples.

Figure 1 . 5 :

 15 Figure 1.5: The goal is to classify the black point. The 1-NN algorithm,Figure 1.5(a), selects the closest example and classify the point as blue. The 3-NN classifier,Figure 1.5(b), chooses the majority class among the 3 closest examples and classifies the point as red.

Figure 2 . 1 :

 21 Figure 2.1: Metric learning in four questions.

Figure 2 . 2 :

 22 Figure 2.2: Illustration of the notion of margin for pair based constraints 2.2(a) and triplet based constraints 2.2(b).

  To extend the Rademacher complexity framework, Cao et al. (2016) propose a new definition of the Rademacher complexity specifically tailored for metric learning. It corresponds to the expected value over size n training sets of the Rademacher averages of the dual norm (Definition A.4) of the regularization term.

•

  [START_REF] Geng | DAML: domain adaptation metric learning[END_REF] proposed Domain Adaptation Metric Learning (DAML) a Mahalanobis distance learning approach for unsupervised domain adaptation. They propose to learn a metric able to well separate similar and dissimilar examples in the source domain while keeping the source and the target domain close to each other. Hence they define the set of similar examples P sim as all the pairs of source examples sharing the same label. Similarly they define the set of dissimilar examples P dis as all the pairs of source examples with different labels. To keep the source and the target domain close to each other they propose to use the Maximum Mean Discrepancy (MMD) (Borgwardt

  This chapter is based on the following publication Michaël Perrot, Amaury Habrard, Damien Muselet, and Marc Sebban. Modeling perceptual color differences by local metric learning. In European Conference on Computer Vision (ECCV-15), pages96-111. Springer International Publishing, 2014b

  input : A training set T of n pairs of examples; The number of clusters K ≥ 2 output: 1 global and K local Mahalanobis distances begin Run K-means to deduce, from T , K + 1 training subsets

Figure 3 . 1 :

 31 Figure 3.1: Some images from our dataset showing (first row) the 84 Farnsworth-Munsell patches and (second row) the 238 Munsell patches under different conditions.

  Generalization to new cameras.

Figure 3 .

 3 Figure 3.2: 3.2(a) Generalization of the learned metrics to new colors; 3.2(b) Generalization of the learned metrics to new cameras. For 3.2(a) and 3.2(b), we plotted the Mean and STRESS values as a function of the number of clusters. The horizontal dashed line represents the STRESS baseline of ∆E 00 . For the sake of readability we have not plotted the mean baseline of ∆E 00 at 1.70.

Figure 3 .

 3 Figure 3.4: 3.4(a) Boundary Displacement Error (lower is better) versus the average segment size. 3.4(b) Probabilistic Rand Index (higher is better) versus the average segment size.

Figure 3 . 5 :

 35 Figure 3.5: Segmentation illustration. When the number of clusters is low (around 50), the segmentation provided by RGB or L * u * v * are far from the ground truth, unlike our approach which provides nice results. To get the same perceptual result, both methods require about 500 clusters.

Figure 3 . 6 :

 36 Figure 3.6: Illustration of segmentation provided by the color mean-shift algorithm applied in the RGB components (third column), on L * u * v * components (fourth column) and by using our learned distance directly in the RGB components (fifth column). First column represents the original image and the second one the ground truth.

Lemma 4 . 1 (

 41 On-average-replace-two-stability). Given n the number of training examples, drawn i.i.d. from D T , considered and a k-lipschitz loss function, any algorithm solving Problem (4.1) is on-average-replace-two-stable with (n) = 8k 2 λn .

Lemma 4 . 2 (

 42 Uniform stability). Given a positive, convex, k-lipschitz loss and a training sample T of n examples drawn i.i.d. from D T , an algorithm solving Problem (4.1) has a uniform stability in β = 4k 2 λ . Proof. The proof of this lemma can be found in Appendix C.2. Using Lemma 4.2 about the stability of our algorithm and McDiarmid's inequality (Theorem A.1) we can derive our generalization bound. Let R T = L T (M T ) -LT (M T ) be the estimation error for Problem (4.1) when learning with training set T . To apply McDiarmid's inequality we need to bound E T ∼D T [R T ] and |R T -R T i |. This is done in the two following lemmas. Lemma 4.3 (Bound on E T ∼D T [R T ]). For any β uniformly stable learning method of estimation error R T = L T (M T ) -LT (M T ) and any training set T , we have:

Lemma 4 . 4 (

 44 Bound on |R T -R T i |).For any β uniformly stable learning method of estimation error R T = L T (M T ) -LT (M T ), for any training set T and any (σ, m)-admissible loss function we have:

Theorem 4 . 2 (

 42 Generalization bound). For any matrix M T learned with Problem (4.1) with the training set T and any positive, convex, k-lipschitz and (σ, m)-admissible loss function, we have with probability 1 -δ: L T (M T ) ≤ LT (M T ) Using McDiarmid's inequality (Theorem A.1) on the estimation error R T = L T (M T )-LT (M T ) coupled with Lemma 4.4 for the estimation of the constants c i we have:

Definition 4 . 2

 42 (k-lipschitz continuity with respect to the metric). A loss function l (M, z, z ) is k-lipschitz continuous with respect to the metric if for any two matrices M, M ∈ M and any two examples z, z there exists k ≥ 0 such that:

Definition 4 . 3 (

 43 Rademacher Complexity). Let M be a metric class and σ i : i = 1, • • • , n 2 be a sequence of independent Rademacher variables, that is, Pr(σ i = +1) = Pr(σ i = -1) = 1 2 . Let {x i : i = 1, • • • , n} be an i.i.d. sequence of examples. Then the empirical Rademacher complexity of M is defined as:

Lemma 4 . 5 (

 45 Metric class and optimal solution). Let M T be the optimal solution returned by Problem (4.1) with training set T . We have M T ∈ M S where M S is defined as inDefinition 4.4.

Lemma 4 . 6 (

 46 Bound on E T ∼D T [R T ]). For any positive, convex and k-lipschitz (Definition 4.2) loss function and any algorithm with estimation error R T = sup M∈M S L T (M) -LT (M)

Lemma 4 . 7 (

 47 Bound on |R T -R T i |). For any positive, convex and k-lipschitz continuous (Definition 4.2) loss function, any metric satisfying Equation (4.14) and any algorithm of estimation error R T = sup M∈M S L T (M) -LT (M) we have:

Theorem 4 . 4 (

 44 Generalization bound). With probability 1 -δ, for any matrix M T learned with Problem (4.1), for any positive, convex, and k-lipschitz continuous (Definition 4.2) loss function and any metric satisfying Equation (4.14) we have:

Example 4 . 1 (

 41 Positive, convex, L-lipschitz functions for dissimilarity learning). Let f (a) be a positive, convex, L-lipschitz function. Given a dissimilarity (Definition 1.8) k M parametrized by M ∈ M and any two examples z, z ∼ D T we define a loss as: l M, z, z = f δ yy k M x, x -γ yy (4.24)

4 5 .

 5 As a last example if f (a) = |a| we retrieve a loss function close to the one used in Chapter 3 6 . Example 4.2 (Positive, convex, L-lipschitz functions for similarity learning). Let f (a) be a positive, convex, L-lipschitz function. Given a similarity (Definition 1.8) k M parametrized by M ∈ M and any two examples z, z ∼ D T we define a loss as:

Example 4 . 3 (

 43 Positive, convex, H-smooth, B-bounded functions for dissimilarity learning). Let f (a) be a positive, convex, H-smooth, B-bounded function. Given a dissimilarity (Definition 1.8) k M parametrized by M ∈ M and any two examples z, z ∼ D T we define a loss as: l M, z, z = f δ yy k M x, x -γ yy(4.26)

Example 4 . 4 (

 44 Positive, convex, H-smooth, B-bounded functions for similarity learning). Let f (a) be a positive, convex, H-smooth, B-bounded function. Given a similarity (Definition 1.8) k M parametrized by M ∈ M and any two examples z, z ∼ D T we define a loss as: l M, z, z = f 1 -δ yy k M (x, x ) γ yy (4.27)

• 1 -

 1 NN S : a 1-NN using the source examples, • 1-NN T : a 1-NN using the target examples, • LMNN T : a 1-NN on the target examples using the metric learned by LMNN on the source examples, • ITML T : a 1-NN on the target examples using the metric learned by ITML on the source examples, • MMDT: a domain adaptation method Hoffman et al. (2013),

  Most of the existing approaches in metric learning use constraints of type must-link and cannot-link between learning examples (See Section 2.3). For example, in a supervised classification task, the goal is to bring closer examples of the same class and to push far away examples of different classes. The idea is that the learned metric should affect a high value to dissimilar examples and a low value to similar examples. Then, this new distance can be used in a classification algorithm like a nearest neighbour classifier. Note that in this case the

  (a) classic must-link cannot-link approach.(b) Our virtual point-based regression formulation.

Figure 5 . 1 :

 51 Figure 5.1: Arrows denote the constraints used by each approach for one particular example in a binary classification task. The classic metric learning approach in Figure 5.1(a) uses O(n 2 ) constraints bringing closer examples of the same class and pushing far away examples of different classes. On the contrary, our approach presented in Figure 5.1(b) moves the examples to the neighborhood of their corresponding virtual point, in black, using only O(n) constraints.

Figure 5 . 2 :

 52 Figure 5.2: In the learned space from the isolet dataset, we randomly select 2 attributes three times and plot the 2D projection on each pair. The first line corresponds to features 1 and 20, the second line to features 7 and 14 and the third line to features 2 and 23. The left column corresponds to the space learned by RVML-Lin-Class (linear) and the right column to the one learned by RVML-RBF-Class (non linear).

Figure 6 . 1 :

 61 Figure 6.1: Illustration of our approach on the clown dataset when learning a linear transformation (top) and a non linear transformation obtained by kernelization (bottom). In both cases we considered the same original data, depicted in the first column, where the blue crosses correspond to the source examples and the red crosses to the target examples. The second and third column respectively show the couplings and the transformations jointly learned by our approach. The fourth column demonstrates the generalization ability of the transformations on new examples. Note that the couplings cannot be used on these examples that have not been seen during the learning process.

  input : The current values of Γ and f S→T . output: The new value of Γ. begin Initialize k = 0 and Γ 0 = Γ repeat Solve Γ k+ 1 2 = arg min

  We consider two domain adaptation datasets namely the Moons dataset[START_REF] Bruzzone | Domain adaptation problems: A DASVM classification technique and a circular validation strategy[END_REF] and the Office-Caltech dataset[START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF]. The Moons dataset is a binary classification task which consists of 2 domains. The source domain corresponds to two intertwined moons, each one representing one class. The target domain is built by rotating the source domain. The rotation angle ranges from 10 to 90 degrees leading to 9 different adaptation tasks of increasing difficulty. In this dataset the examples are of dimension 2 and we consider 300 source examples and 300 target examples for training and 1000 target examples for testing. The Office-Caltech dataset is a 10 class image classification task which consists of 4 domains. These domains are amazom (A), dslr (D), webcam (W) and Caltech10 (C) and corresponds to images coming from different sources. Following this, there are 12 adaptation tasks where each domain is in turn considered as the source or the target (denoted source → target in the results). During the training process we consider all the examples from the source domain and half of the examples from the target domain, the other half being used as the test set. To represent the images we use deep learning features of size 4096 named decaf6

Figure 6 . 2 :

 62 Figure 6.2: Illustrations of seamless copies with gradient adaptation. Each row is composed of the source image, the corresponding selection zone Ω described as a binary mask, and the target image. We compare here the linear (4th column) and kernel (5th column) versions of the map f L with the original method of Pérez et al. (2003) (3rd column).

Figure 6 . 3 :

 63 Figure 6.3: Illustration of failure of style adaptation.

Lemma

  (Bound on |R T -R T i |). For any β uniformly stable learning method of estimation error R T = L T (M T ) -LT (M T ) for a training set T and any (σ, m)-admissible loss function C.5 Proof of Lemma 4.6 Lemma (Bound on E T ∼D T [R T ]). For any positive, convex and k-lipschitz (Definition 4.2) loss function and any algorithm with estimation error R T = sup M∈M S L T (M) -LT (M) we

C. 6

 6 Proof of Lemma 4.7 Lemma (Bound on |R T -R T i |). For any positive, convex and k-lipschitz continuous (Definition 4.2) loss function, any metric satisfying Equation (4.14) and any algorithm of estimation error R T = sup M∈M S L T (M) -LT (M) we have:

Example

  x + δ yy -δ y y sup z,z ∼D T γ yy (Equation (4.14) and Cauchy-Schwartz's inequality (Theorem A.3).) ≤ 2L sup z,z ∼D T g(x, x ) * M T + δ yy -δ y y sup z,z ∼D T γ yy (Triangle inequality.) ≤ 2L sup z,z ∼D T g(x, x ) * ( M T -M S + M S ) + δ yy -δ y y sup z,z ∼D T yy -δ y y sup z,z ∼D T γ yy Setting σ = sup z,z ∼D T γ yy and m = 2L sup z,z ∼D T g(x, x ) (Positive, convex, L-lipschitz functions for similarity learning). Let f (a) be a positive, convex, L-lipschitz function. Given a similarity (Definition 1.8) k M parametrized by M ∈ M and any two examples z, z ∼ D T we define a loss as: l M, z, z = f 1 -δ yy k M (x, x ) γ yy (C.14)

  C.15) (Equation (4.14) and Cauchy-Schwartz's inequality (Theorem A.3).

LExample

  inf z,z ∼D T |γ yy | sup z,z ∼D T g(x, x ) * LT (M S ) λ (Positive, convex, H-smooth, B-bounded functions for dissimilarity learning). Let f (a) be a positive, convex, H-smooth, B-bounded function. Given a dissimilarity (Definition 1.8) k M parametrized by M ∈ M and any two examples z, z ∼ D T we define a loss as: l M, z, z = f δ yy k M x, x -γ yy (C.17)

  z ∼D T γ yy sup z,z ∼D T g(x, x ) * M -M (C.22) Inequalities (C.21) and (C.22) respectively prove the lipschitzness with respect to the metric and the matrix.Lastly we show that the loss is (σ, m)-admissible. Given four examples z, z , z , z ∼ D T , M T the learned metric when learning with T and the fact that the loss function is positive and B-bounded we have:l M T , z, z -l M T , z , z ≤ BSetting σ = 0 and m = B gives the example.

  Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Regressing the Values of a Reference Metric by Local Metric Learning . . . . 3.3 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Table of Contents 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Metric Hypothesis Transfer Learning 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Metric Hypothesis Transfer Learning with Biased Regularization . . . . . . . 4.3 On Average Stability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Uniform Stability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Rademacher Complexity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Summary of the Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	II Metric Learning with a Reference Metric 3 Metric Approximation Learning in Perceptual Colour Learning List of Publications 3.1 Conclusion and Perspectives A Theorems, Lemmas and Definitions
	B Proofs of Chapter 3
	C Proofs of Chapter 4
	D Proofs of Chapter 5

1 Preliminaries 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Learning by Risk Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Deriving Generalization Guarantees . . . . . . . . . . . . . . . . . . . . . . . 1.4 Loss Functions, Regularization Terms and Metrics . . . . . . . . . . . . . . . 1.5 Other Notions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Metric Learning 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Learning Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Theoretical Guarantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 3.4 Learning Perceptual Color Differences . . . . . . . . . . . . . . . . . . . . . . 3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III Metric Learning with Controlled Behaviour 5 Regressive Virtual Metric Learning 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Learning a Metric Using Virtual Points . . . . . . . . . . . . . . . . . . . . . 5.3 Choosing the Virtual Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Mapping Estimation for Discrete Optimal Transport 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Optimal Transport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 General Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Optimisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Discussion on Theoretical Aspects . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Illustration of the bias-variance trade-off problem . . . . . . . . . . . . . . . . 1.2 Illustration of several loss functions (classification) . . . . . . . . . . . . . . . 1.3 Illustration of several loss functions (regression) . . . . . . . . . . . . . . . . . 1.4 Illustration of the p norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Illustration of the nearest neighbour algorithm . . . . . . . . . . . . . . . . .

	Table of Contents
	E French Translations
	Bibliography
	List of Figures
	1.1

2.1 Metric learning in four questions. . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Illustration of the notion of margin . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Illustration of proposed dataset . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Generalization ability of the metrics . . . . . . . . . . . . . . . . . . . . . . . 3.3 Interest of learning local metrics . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Performance of the metric in a segmentation task . . . . . . . . . . . . . . . . 3.5 Illustration of the segmentation results . . . . . . . . . . . . . . . . . . . . . . 3.6 Illustration of the segmentation results . . . . . . . . . . . . . . . . . . . . . . 5.1 Illustration of the intuition behind the virtual points . . . . . . . . . . . . . . 5.2 Illustration of RVML on the Isolet dataset . . . . . . . . . . . . . . . . . . . . 6.1 Illustration of the transformations . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Successful seamless copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Unsuccessful seamless copy . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 1 :

 1 Notations.

	Notation	Description
	R, R +	Sets of real and non negative real numbers
	B, C	Constants
	x	Scalar
	x	Vector
	X

  ,δ yy )∈P pair 1 δ yy d M (x,x )≥δ yy γ yy + λ M where the margin γ yy is a measure of closeness between the examples and 1 is the indicator function whose value is 1 if the condition is true and 0 otherwise 3 . The idea is that similar examples should be at a distance lower than γ yy while dissimilar examples should be at a distance greater than γ yy (See Figure2.2(a)). Note that γ yy depends on the examples and can thus have a different value for each pair.

	2), learning a Mahalanobis distance could be done by solving one of the following
	two optimization problems:
	arg min + M∈S d×d	(z,z )∈P sim	1 d M (x,x )≥γ yy +	(z,z )∈P dis	1 d M (x,x )≤γ yy + λ M
	arg min
	M∈S d×d +	(z,z

  the previous subsections we considered a batch setting where all the examples are available at the same time. However in some cases the examples arrive in a stream like fashion. Contrary to classification or regression where the examples can be considered independently from each other, in metric learning most of the approaches work with pairs or triplet of examples.Hence a natural assumption is that these pairs or triplets are given one after the other. The goal is then to learn a metric which is able to change when more and more pairs or triplets are available. Formally assume that we have a sequence of pairs (x 1 , x 1 ), . . . , (x t , x t ), (x t+1 , x t+1 ) and that from the first t examples we were able to learn a Mahalanobis distance d Mt . The goal is to learn a distance d M t+1 such that:

  is interested in learning a Mahalanobis distance and a parameter b corresponding to the threshold between similar and dissimilar examples. It is assumed that pairs of examples arrive one after the other. Given a new example (x t+1 , x t+1 , δ y t+1 y t+1 ) they propose to update the matrix M t and the threshold b t successively solving the following two optimization problems:

  . Since we need patch pairs whose colors are similar, following the CIE recommendations (CIE Standard DS 014-6/E:2012), we select among the C 2 84 + C 2 238 available pairs only the 223 that are characterized by a Euclidean distance in the CIELAB space (denoted ∆E ab ) less than 5.

  . It corresponds to the uniform stability framework presented in Section 1.3 but adapted to the case of metric learning as shown in Section 2.4. It allows us to derive a probabilistic generalization bound. The main interest of this bound is that, in some cases, it involves an empirical quantity related to the source metric. It implies a natural notion of goodness of the source metric which can be optimized. We empirically confirm the interest of this measure in two experiments. • Rademacher complexity for metric learning: The third theoretical framework that we consider is a slight adaptation of the one proposed by Cao et al. (2016) (See Section 2.4). The latter is in itself an adaptation to metric learning of the Rademacher complexity framework presented in Section 1.3. It allows us to derive a vanishing bound with respect to the source metric. It means that if the source metric is a perfect fit for the problem at hand the bound shows that learning is no longer necessary. It also gives a theoretical measure of goodness of the source metric.

Table 4 . 1 :

 41 Summary of the different bounds.

	Bound	On Average Stability (Section 4.3)	Uniform Stability (Section 4.4)	Rademacher Complexity (Section 4.5)
	Nature of the bound	Exact	Probabilistic (1 -δ) Probabilistic (1 -δ)
	Impact of the complexity of the metric class	8k 2 λn	8k 2 λn	2kRn

Table 4 .

 4 2: Examples of regularization terms.

	Norm •	Dual Norm • *	Rademacher Average R n ( • * ) (See Appendix C.12 for a proof)

Table 4 . 3 :

 43 Results of the experiments conducted on the UCI datasets. Each value corresponds to the mean and standard deviation over 10 runs. For each dataset we highlight the best result using a bold font. Approaches with the suffix -ω 1 do not learn ω but fix it to 1.

			Baselines				Our approach
	Dataset	1-NN	ITML	LMNN	IDENTITY	IDENTITY-ω 1 MAHALANOBIS MAHALANOBIS-ω 1
	Breast 95.31 ± 1.11 95.40 ± 1.37 95.60 ± 0.92 96.06 ± 0.77	95.75 ± 0.87	95.71 ± 0.84	94.76 ± 1.38
	Pima	67.92 ± 1.95 68.13 ± 1.86 67.90 ± 2.05 67.87 ± 1.57	67.54 ± 1.99	68.37 ± 2.00	66.31 ± 2.37
	Scale	78.73 ± 1.69 87.31 ± 2.35 86.20 ± 2.83 80.98 ± 1.51	80.82 ± 1.27	81.35 ± 1.17	80.88 ± 1.43
	Wine	93.40 ± 2.70 93.82 ± 2.63 93.47 ± 1.80 95.42 ± 1.71	95.07 ± 1.68	94.31 ± 2.01	80.56 ± 5.75

Table 5 .

 5 2: Comparison of our approach with several baselines in the non linear case. The best result is highlighted with a bold font while the second to best result is underlined. A star indicates that the best result for the baseline is significantly better than our best result or that our best result is significantly better than the best baseline result.

	Our approach

Table 5 .

 5 3: Comparison between a method with explicit virtual points (RVML-Lin-Class) and a method with implicit virtual points (ITML-Collapse). The best result is highlighted with a bold font.

	Base	RVML-Lin-Class ITML-Collapse	ITML
	Amazon	73.09 ± 2.49	57.97 ± 3.36	65.91 ± 2.64
	Breast	95.34 ± 0.95	94.56 ± 1.41	95.49 ± 1.15
	Caltech	55.41 ± 2.55	37.34 ± 2.01	47.31 ± 2.75
	DSLR	75.29 ± 5.08	77.25 ± 4.15	77.25 ± 4.91
	Ionosphere	82.74 ± 2.81	85.75 ± 6.23	88.11 ± 1.68
	Isolet	94.61	74.53	92.88
	Letters	95.51 ± 0.26	95.67 ± 0.30	95.00 ± 0.64
	Pima	69.57 ± 2.85	71.08 ± 2.13	70.26 ± 1.38
	Scale	87.94 ± 1.99	87.51 ± 4.39	87.67 ± 2.71
	Splice	78.44	66.80	71.49
	Svmguide1	85.25	94.62	95.00
	Wine	98.18 ± 1.48	85.91 ± 3.74	96.91 ± 1.93
	Webcam	88.60 ± 2.69	97.64 ± 2.43	86.56 ± 2.88
	mean	83.07	78.97	82.30

Table 5 .

 5 4: Comparison of our OT based formulation to a random selection approach when learning a linear metric. The best result is highlighted with a bold font.

		OT based approach		Random
	Base	RVML-Lin-OT	1 VP per class 2 VP per class 5 VP per class
	Amazon	71.62 ± 1.34	74.23 ± 2.15	72.92 ± 2.31	70.31 ± 2.82
	Breast	95.24 ± 1.21	95.34 ± 0.95	95.29 ± 1.32	94.90 ± 1.92
	Caltech	52.51 ± 2.41	55.09 ± 2.38	53.63 ± 2.12	49.59 ± 1.69
	DSLR	74.71 ± 5.27	70.59 ± 6.06	63.53 ± 5.08	52.16 ± 8.68
	Ionosphere	87.36 ± 3.12	82.74 ± 2.81	88.40 ± 4.05	90.28 ± 3.33
	Isolet	91.40	92.75	94.16	92.43
	Letters	90.25 ± 0.60	89.90 ± 1.02	90.54 ± 1.24	91.13 ± 0.74
	Pima	70.48 ± 3.19	69.57 ± 2.85	69.35 ± 2.44	69.26 ± 2.60
	Scale	90.05 ± 2.13	88.10 ± 2.57	89.47 ± 2.99	89.21 ± 2.68
	Splice	84.64	78.44	78.94	80.87
	Svmguide1	94.83	85.25	86.90	94.70
	Wine	98.55 ± 1.67	98.55 ± 1.43	97.64 ± 2.43	98.00 ± 1.34
	Webcam	88.60 ± 3.63	88.92 ± 3.21	86.24 ± 2.95	81.18 ± 3.56
	mean	83.86	82.27	82.08	81.08
	kernelized case and thus requires more meaningful virtual points. Hence, selecting virtual
	points and correctly associating them to the examples is key to obtain a good performance.

Table 5 .

 5 5: Comparison of our OT based formulation to a random selection approach when learning a non linear metric. The best result is highlighted with a bold font.

		OT based approach		Random
	Base	RVML-RBF-OT	1 VP per class 2 VP per class 5 VP per class
	Amazon	73.51 ± 0.83	75.74 ± 2.35	72.68 ± 2.02	70.07 ± 2.86
	Breast	95.73 ± 0.97	95.73 ± 1.07	95.83 ± 0.80	95.58 ± 1.38
	Caltech	54.39 ± 1.89	58.33 ± 2.05	53.98 ± 3.18	50.35 ± 1.89
	DSLR	70.39 ± 4.48	65.29 ± 7.51	58.24 ± 7.79	48.82 ± 8.03
	Ionosphere	90.66 ± 3.10	90.57 ± 3.05	89.25 ± 3.73	90.38 ± 3.26
	Isolet	95.96	96.99	96.54	95.25
	Letters	91.26 ± 0.50	91.77 ± 0.43	91.87 ± 0.52	92.04 ± 0.62
	Pima	69.35 ± 2.95	70.82 ± 4.60	71.26 ± 2.84	70.00 ± 2.56
	Scale	95.19 ± 1.46	93.39 ± 2.19	91.96 ± 1.69	91.32 ± 1.95
	Splice	88.51	88.37	88.46	87.22
	Svmguide1	95.67	95.03	95.55	95.88
	Wine	98.91 ± 1.53	97.82 ± 1.88	97.27 ± 1.96	97.82 ± 1.67
	Webcam	88.71 ± 4.28	87.31 ± 2.99	83.01 ± 3.28	76.67 ± 4.78
	mean	85.25	85.17	83.53	81.65

  Joint Learning of L and Γ.

	or (6.21) with fixed Γ k+1 .
	Set k = k + 1.
	until convergence
	end
	Algorithm 4:

Table 6 . 1 :

 61 Accuracy on the Moons dataset. The best result for each angle is highlighted with a bold font.

	Angle 1-NN GFK	SA	OT	L1L2 OTE	OTLin f L f Γ	OTLinB f L f Γ	OTKer f L f Γ	OTKerB f L f Γ
	10	99.99 99.86 99.99 97.88 99.56 99.95 100.	100.	100.	100.	100.	100.	100. 100.
	20	93.08 95.75 93.08 94.96 98.74 100.	100. 99.96 100. 99.97 100.	100.	100. 100.
	30	83.98 92.55 83.98 90.62 98.36 100. 99.82	99.9	99.78 99.86 99.99 100. 99.99 100.
	40	77.07 90.85 74.41 83.73 95.8 99.98 98.32 98.65	98.1	98.46 99.65 99.73 99.63 99.74
	50	61.73 90.22 73.13 77.75 87.69 87.29	97.8	97.56 97.48	97.5	99.12 99.23 99.11 99.14
	60	41.21 79.37 72.35 71.0	88.3	86.35 96.42 97.22 95.84 97.04 96.59	96.8	96.62 96.81
	70	23.08 61.05 72.27 64.48 89.03 77.46 88.04 94.66 88.21 94.32 80.77 81.54 82.45 83.06
	80	20.72 36.16 72.31 57.34 73.6	58.79 76.91 81.01 76.58 80.74 73.96 74.13 73.94 74.24
	90	19.4 43.08 34.16 50.97 58.1	51.31 67.88 67.96 67.13 68.06 56.32 55.77 57.57 55.42

Table 6 .

 6 2: Accuracy on the Office-Caltech dataset. The best result for each task is highlighted with a bold font.

	Task	1-NN GFK	SA	OT	L1L2 OTE	OTLin f L	OTLinB	OTKer	OTKerB

  C 51.81 69.73 78.99 68.09 67.85 68.03 77.15 77.15 77.43 77.43 69.1 69.17 69.27 69.31 W → D 99.25 99.75 99.63 74.13 94.38 94.38 99.38 99.38 99.75 99.75 97.25 97.25 96.88 96.88

		89.47 93.31 95.56 76.95 95.7	95.7	97.28 97.28 97.28 97.28 98.41 98.48 98.48 98.48
	D → A 62.52 77.23	88.5	70.83 74.9 74.85 85.73 85.73 85.75 85.75 89.92	89.9	89.54 89.54
	D → W → A 62.5	72.38 79.25	67.6 71.33 71.35 81.46 81.46 81.38 81.38	78.5	78.35 78.52 78.81
	W → C	59.5	63.74 55.02	63.1 67.78 67.78 75.87 75.87 75.41 75.41 72.71	72.7	65.12 63.26
	A → D 65.25 75.88 83.75 64.63 70.13 70.5	80.63 80.63 80.38	80.5	65.63	65.5	71.88	71.5
	A → W 56.75 68.01 74.57 66.82 67.15 67.28 74.64 74.64 74.37 74.37 66.36 64.77	70.	68.87
	A → C 70.09 75.71	79.2	70.43 74.06 74.31 81.81 81.81	81.6	81.63 84.38 84.43 84.49 84.47
	C → D 75.88	79.5	85.	66.	69.75 70.25 87.13 87.13 87.25 87.25 70.13	70.	78.63 78.63
	C → W 65.17 70.66 74.44 59.21 63.77 63.77 78.28 78.28 78.48 78.48	80.	80.4	73.51 73.38
	C → A 85.79 87.13 89.33 75.25 76.63 76.67 89.94 89.94 89.71 89.71 82.38 82.15 83.56 83.48
	Mean	70.33 77.75 81.94 68.59 74.45 74.57 84.11 84.11 84.07 84.08 79.56 79.43 79.99 79.72

  6.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

	Conclusion et Perspectives	157
	Liste des Publications	161
	A Théorèmes, Lemmes et Définitions	163
	B Preuves du Chapitre 3	167
	C Preuves du Chapitre 4	173
	D Preuves du Chapitre 5	189
	E Traductions Françaises	193
	E.1 Introduction

L'Apprentissage Automatique est un domaine de l'Intelligence Artificielle dont le but est d'acquérir de nouvelles connaissances à partir de données. Ces nouvelles connaissances prennent généralement la forme d'un modèle, appris à partir d'un nombre limité d'exemples observés et capable de bien généraliser à de futures requêtes. En d'autres termes, le but est d'apprendre comment résoudre un problème de façon automatique à partir d'un nombre fini d'observations. Par exemple, l'objectif de la détection de spams est d'utiliser la boite mail annotée d'un utilisateur pour apprendre comment séparer les emails désirés des autres; en suivi d'objets le problème est de suivre un élément donné dans une vidéo; en reconnaissance de visages le but est d'identifier une personne dans un ensemble d'images. . . . La diversité des problèmes posés en apprentissage automatique a attiré beaucoup d'attention dans le passé et mérite que l'on continue à s'y intéresser activement.

Table des Matières

 des capacité à généraliser des métriques apprises.

	Contributions: Apprendre des Métriques avec un Comportement Contrôlé

A, B F = Tr A T B

Here we only consider metrics for feature vectors. However there also exist some metrics for structured data but this is beyond the scope of this thesis. We refer the interested reader toBellet et al. (2015).

Note that even if the feature map is tractable it is most of the times more interesting to compute the value of the kernel through the expression of k(•).

To denote the fact that a matrix is positive semi-definite we interchangeably use the notation M 0 or M ∈ S d×d + where d is the dimension of the matrix.

It is another way to write the 0/1 loss presented in Section 1.4.

See A.4 for a formal definition.

An example of this is given in Section 4.5 where we use the Rademacher complexity to derive a generalization bound for metric hypothesis transfer learning.

Note that in the case of color differences studied in Section 3.4, we work in the RGB cube and any patch belongs to [0;

255]3 . It is then easy to normalize each coordinate by 255 √ 3 to meet the assumption.

https://www.uef.fi/spectral/spectral-database

STandardized REsidual Sum of Squares.

To avoid this problem, we can use the classic relaxation with slack variables.

A, B = Tr(A T B)

To the best of our knowledge it is the only work with a fast rate in an Hypothesis Transfer Learning setting.

The hinge loss is positive, convex and 1-lipschitz.

The absolute value is positive, convex and 1-lipschitz.

Note that[START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF] also proved tighter results (with respect to the constants) than the one presented here but this is beyond the scope of this analysis.

Note that we also report the mean accuracy over the 12 tasks. Even if we are conscious that the problems are different, it gives a rough idea of the global performance of the compared approaches.

In Chapter 6 we elaborate upon this idea by jointly learning the metric and the optimal transport.

Legendre identity is (a + b) 2 -(a -b) 2 = 4ab from which we deduce a 2 + b 2 ≥ 2ab.

With the σ parameter fixed as previously to the mean of all pairwise training set euclidean distances.

This optimal transport map is implied by the coupling Γ but cannot be computed.

If we consider that the examples are drawn i.i.d. we have pi = 1 n for every example.

A, B F = Tr(A T B)

c(x, x ) = xx 2 2

In other words the examples are drawn i.i.d. from DS and DT .

For example this regularization term could be the Frobenius norm.

Proof. The proof of this theorem can be found in Appendix D.1.

From Equation (5.4) we deduce the matrix M V : (5.6) Note that M V is PSD by construction:

So far, we have focused on the linear setting. We now present a kernelized version, showing that it is possible to learn a metric in a very high dimensional space without an explicit projection.

Let φ(x) be a projection function and k(x, x ) = φ(x) T φ(x ) be its associated kernel (Section 1.4). For the sake of readability, let K X = Φ X Φ T X where Φ X = (φ(x 1 ) , . . . , φ(x n )) T . We also define V k the kernelized version of V . Given the solution matrix L V presented in Equation (5.5), we have:

Then, M V k the kernelized version of the matrix M V is defined such that:

The squared Mahalanobis distance can be written as:

(5.7)

Thus we can obtain: 5.8) the kernelized version by considering that: 5.10) where k X,x = (k(x, x 1 ) , . . . , k(x, x n )) T is the similarity vector to the examples with respect to k. Note that it is also possible to obtain a kernelized version of L V :

This result is close to a previous one already derived in [START_REF] Cortes | A general regression technique for learning transductions[END_REF] in a structured output setting. The main difference is the fact that we do not use a kernel on the output

Conclusion and Perspectives

In this thesis we addressed the problem of learning a metric with a controlled behaviour. We considered two kinds of control on the learned metric. On the one hand we addressed the problem of learning with respect to a reference metric available either under the form of a given distance between a limited number of pairs or directly through its model. On the other hand we considered the problem of learning the underlying transformation of a Mahalanobis distance either able to precisely move the examples toward a set destination or controlled by a geometric transformation. Our contributions have taken the form of algorithmic and theoretical solutions.

Summary of the Contributions

Most of metric learning algorithm are interested in learning metrics able to bring closer similar examples and to push far away dissimilar ones. However, in some cases, one might be interested in predicting an exact value between two examples. This is for example the case when one has access to a limited number pairs for which the value of a reference metric is known. In our first contribution we addressed the problem of learning an approximation of this reference metric. We proposed a local metric learning algorithm and we theoretically analysed it showing that with a sufficient number of examples the learned model generalizes well. Furthermore we evaluated our approach on the computer vision problem of perceptual color differences. To this end we created a new dataset specifically designed for the problem at hand. Our empirical results showed the good behaviour of our approach and its ability to correctly approximate a reference metric. The dataset and the perceptually uniform distance that we learned are freely distributed (Perrot et al., 2014a).

Several approaches in metric learning empirically demonstrated the interest of using side information in the form of a source metric without theoretically proving that it was indeed beneficial. In the second contribution we proposed to bridge this gap. Hence we formalised the metric hypothesis transfer learning framework where the idea is to take into account a source metric in a biased regularization term. We proposed a theoretical analysis of this framework and, considering several theoretical approaches, we derived three different measures of goodness for the source metric. These measures are ways to evaluate the interest of a source metric for the problem at hand. Two of these measures are theoretical and thus are hard to use in practice. The third one, however, is empirical which means that it can Appendix A

Theorems, Lemmas and Definitions

We present here several Theorems, Lemmas and Definitions used throughout the thesis but not presented in the main text for the sake of readability.

A.1 Properties of Loss Functions

Definition A.1 (k-lipschitz continuity). A loss function l (h, z) is k-lipschitz with respect to its first argument if, for any hypotheses h, g ∈ H and any example z, there exists k ≥ 0 such that:

The k-lipschitz property ensures that the loss deviation does not exceed the deviation between two hypotheses h and g with respect to a positive constant k.

Definition A.2 ((σ, m)-admissibility). A loss function for metric learning l (M, z, z ) is (σ, m)-admissible, with respect to M, if it is convex with respect to its first argument and if for any two pairs of examples z, z and z , z , we have:

where δ yy = 1 if y = y and -1 otherwise, i.e. δ yy -δ y y ∈ {0, 2}.

This property bounds the difference between the losses of two pairs of examples by a value only related to the labels plus a constant independent from the matrix M. Definition A.3 (H-smooth loss (Srebro et al., 2010c)). A function f : R → R is H-smooth if it is twice differentiable and its first derivative is H-lipschitz continuous (Definition A.1).

Lemma A.1 (Srebro et al. (2010b, Lemma B.1)). For any H-smooth non-negative function f : R → R and t, r ∈ R we have that:

(A.3)

A.2 Properties of Norms

Definition A.4 (Dual norm). Let • be a norm over a normed space M. The dual norm • * is defined as:

A.3 Properties of Algorithms

Definition A.5 (On-average-replace-one-stability (Shalev-Shwartz and Ben-David, 2014a)).

Let n be the number of examples considered during the learning step. Let : N → R be monotonically decreasing with respect to n and let U (1, n) be the uniform distribution over 1, . . . , n. An algorithm is on-average-replace-two-stable with rate (n) if for any distribution D T :

where h T , respectively h T i , is the optimal solution when learning with the training set T , respectively T i . T i is obtained by replacing z i , the i th example of T , by z.

A.4 Concentration Inequalities

Theorem A. 1 (McDiarmid's inequality (McDiarmid, 1989)). Let X 1 , ..., X n be n independent random variables taking values in X and let

then for any > 0, 1 (Van Der Vaart and Wellner (1996)). Let (n 0 , n 1 , . . . , n K ) an i.i.d. multinomial random variable with parameters n = K j=0 n j and (P (C 0 ), P (C 1 ), . . . , P (C K )). By the Breteganolle-Huber-Carol inequality we have:

hence with probability at least 1 -δ,

(A.9)

A.5 Other Theorems

Theorem A.2 (Union bound). Given a countable set of events E 1 , E 2 , E 3 , . . .:

Theorem A.3 (Cauchy-Schwarz inequality). Let X be a vector space equipped with an inner product •, • defining a norm • . Let x, x ∈ X , then: .6 (Convexity). A function f is convex if for all w, u, and α ∈ [0, 1] we have:

Definition A.7 (c-strong convexity). A function f is c-strongly convex if for all w, u, and α ∈ [0, 1] we have:

Theorem A.4 (Jensen's inequality). For any convex function f of a random variable X we have:

Proofs of Chapter 3

B.1 Proof of Lemma 3.1

Lemma (Bounded loss function). For any 0 ≤ j ≤ K, let M T j be the metric learned for region C j with the training set T j , we have that for any example (x, x , ∆) ∼ D T j :

Proof. First of all note that the absolute value is always positive which gives the first inequality. Furthermore M T j is an optimal solution of Problem (3.5). Hence we have:

We can now prove the second inequality of the lemma:

(Difference between two positive values.)

Combining this bound on C with Equation (B.12) and dividing both sides by λ j gives the lemma.

We are now ready to prove Lemma 3.3.

Lemma (Uniform stability per region C j ). Given two training samples T j and T i j of n j examples where T i j is obtained by replacing example i from T j by another example drawn independently from D T j . Let M T j and M T i j be the respective optimal solutions of Problem (3.5) when learning with T j and T i j . In region C j our problem is β j uniformly stable with

Proof. By setting t = 1 2 in Lemma B.1, one can obtain for the left hand side:

and thus

Since our loss is k-lipschitz (Lemma 3.2) we have:

B.5 Proof of Lemma 3.5

Lemma (Bound on R T j -R T i j

). For any β j uniformly stable learning method of estimation error R T j = L T j (M T j ) -LT j (M T j ) for a training set T j and any B j bounded loss function we have:

(Definition of L T j and triangle inequality.)

(Definition of LT j and triangle inequality.)

(T j and T i j only differ by one pair.)

Proofs of Chapter 4

C.1 Proof of Lemma 4.1

Before proving the lemma we show that the biased Frobenius norm is strongly convex (Definition A.7).

Lemma C.1 (Strong convexity of the biased Frobenius norm). The biased Frobenius norm is 2-strongly convex.

Proof.

(2-strong convexity of the non biased Frobenius norm.)

We can now prove Lemma 4.1.

Lemma (On-average-replace-two-stability). Given n the number of training examples, drawn i.i.d. from D T , considered and a k-lipschitz loss function, any algorithm solving Problem (4.1) is on-average-replace-two-stable with (n) = 8k 2 λn . Proof. Let M T , respectively M T i j , be the optimal solution when learning with the training set T , respectively T i j . Let z k , z i k , z i j k respectively be the k th examples of training sets T, T i , T i j . We have:

(Adding and removing the same quantity.)

(Adding and removing the same quantity.)

-

(Triangle inequality and k-lipschitz continuity.)

(Convex loss and optimality of M T i j when learning with T i j .)

Furthermore, from the 2-strong convexity of the biased Frobenius norm used as a regularization term (Lemma C.1) we deduce that Problem (4.1) is 2λ-strongly convex (Definition A.7). Given M T the optimal solution of Problem (4.1) when learning with T , we have:

Combining the two inequalities we obtain:

The last inequality is obtained thanks to the k-lipschitzness of the loss. Taking the expectation on both sides gives the lemma.

C. Proof. Let ∆ M = M T -M T i where M T is the optimal solution when learning with set T and M T i is the optimal solution when learning with set T i . The empirical risk is convex by sum of convex functions, thus

Summing up the two inequalities gives:

Problem (4.1) is convex by sum of convex functions, thus:

Summing Inequalities (C.2) and (C.3) gives:

From Inequality (C.4) we have:

where

We are now looking for a bound on C:

(T and T i only differ by one example.)

Furthermore, setting t = 1 2 in the left hand side of Inequality (C.5), we have:

Following this we have:

we have:

(T and T i only differ by one example, ∀j = i, z j = z i j and z j = z i j .)

Noting that by definition sup z,z ,z ,z ∼D T |yy -y y | ≤ 2 gives the lemma.

C.7 Proof of Lemma 4.9

Lemma (Bounded regularization). Let M T be the optimal solution returned by Problem (4.1)

with training set T and a positive and convex loss. We have:

Proof. By the convexity of the loss and the optimality of M T we have:

C where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k = L,

• k-lipschitz continuous with k = L sup z,z ∼D T g(x, x ) * ,

• (σ, m)-admissible with

Proof. First the loss is positive and convex by construction.

Then we prove that the loss function is k-lipschitz. Given two metrics k M and k M we have: where δ yy = 1 if y = y and -1 otherwise and γ yy is the desired margin between examples. This loss is:

• Positive,

• Convex,

• k-lipschitz continuous with respect to the metric with k =

Proof. First the loss is positive and convex by construction. Then we prove that the loss function is k-lipschitz. Given two metrics k M and k M we have: Theorem (Optimal solution of Problem (5.1)). The optimal solution of Problem (5.1) can be found in closed form. Furthermore, we can derive two equivalent solutions:

Proof. Problem (5.1) is a classic regularized regression problem admitting a closed form solution [START_REF] Cortes | A general regression framework for learning string-to-string mappings[END_REF]. We recall the derivation here for the sake of completeness. Let F V (L) = LV (L) + λ L 2 F be the function optimised in Problem (5.1). First we consider its derivative with respect to L:

Then we set this derivative to zero to obtain:

Finally Equation (5.5) comes from using Taylor expansions as proposed in [START_REF] Cortes | A general regression framework for learning string-to-string mappings[END_REF].

D.2 Proof of Lemma 5.1

Before proving Lemma 5.1 we need the following technical lemma showing that the Frobenius norm of the optimal solution of Problem (5.1) is bounded.

Lemma D.1 (Bounded Frobenius norm). Let L V be an optimal solution of Problem (5.1), we have:

Proof. Since L is an optimal solution of Problem (5.1) and by convexity of the loss we have:

Lemma (Bounded loss function). Let L V be the metric learned with Problem (5.1) with training set V , we have that for any example (x, v) ∼ D V :

Proof.

l (L, (x, v)) = x T Lv T 2 2 (Triangle inequality and standard norm properties.)

( v 2 ≤ B v , x 2 ≤ B x and Lemma D.1.)

gives the lemma.

D.3 Proof of Lemma 5.2

Lemma (k-lipschitz continuity). Our loss is k-lipschitz with k = 2B v B x 1 + Bx √ λ .

Proof.

Using Lemma 5.2 we can bound C:

(V and V i only differ by one example.)

(Loss k-lipschitz (Lemma 5.2).)

We can now prove the lemma.

Lemma (Uniform stability). Our algorithm has a uniform stability in

Proof. By setting t = 1 2 in Lemma D.2, one can obtain for the left hand side:

and thus:

From Lemma 5.2 we have:

gives the lemma.

Appendix E

French Translations