Je tiens ici à remercier tous ceux qui ont contribué au succès de ce projet doctoral.

Mes premiers remerciements vont à mon maître de thèse, Stephan, qui m'a guidé dans mes premiers pas dans le monde de la recherche. Les compétences que j'ai acquises pendant ces quatre années à Télécom Paris, ainsi que la qualité du travail de recherche effectué, sont en grande partie le fruit de son enseignement, son soutien et sa direction. Je te remercie particulièrement pour ta disponibilité au quotidien, ainsi que pour ta présence à des moments inoubliables de ma thèse.

Je remercie aussi tout particulièrement Aurélien, qui malgré la distance qui nous sépare a suivi de très près ce projet doctoral, et a été une force motrice essentielle de ce projet. Je ne peux qu'admirer ta capacité et ta détermination à toujours réévaluer avec justesse le travail accompli, et j'espère emporter cette aptitude avec moi pour mes projets futurs. Je n'ai aucun doute qu'elle sera promesse de succès, et j'espère l'appliquer dès que possible sous ton oeil averti.

Je remercie Stéphane (avec un e), qui a assuré la continuité du suivi industriel du projet. Par son dynamisme, il a porté la dimension industrielle de ma thèse, et m'a enseigné des compétences professionnelles qui ne s'apprennent pas dans les livres, qui me seront sûrement très utiles à l'avenir. Je dois aussi saluer ta curiosité et ta constance, qui ont été le moteur de nombreuses réunions de suivi.

Je tiens à remercier ma famille et l'ensemble de mes proches, pour leur soutien durant ma thèse, et plus généralement durant l'ensemble de mon aventure francilienne.

Chapter 1 Summary

Context

The thesis originates from a collaboration between the french grande école Télécom Paris and the multinational company IDEMIA. Precisely, the project relies on a CIFRE contract (Industrial Agreements for Training through Research), a type of contract introduced in 1981 by the french government to strengthen the ties between research institutions and private companies. The research work is thus supervised by both parties, which is made possible in our case by frequent interactions.

Finalement, je tiens à remercier formellement mes supérieurs immédiats en entreprise, dans l'ordre chronologique: Julien Bohné, Anouar Mellakh et Vincent ici présent. Merci Julien pour ton enthousiasme, et pour m'avoir mis le pied à l'étrier dans ma première expérience en apprentissage profond dans l'industrie. Merci Anouar pour ta présence rassurante et tes conseils. Vincent, je te remercie pour ta bienveillance et ta curiosité, qui je le crois contribuent à créer un environnement propice au bon déroulement d'une thèse CIFRE. Je vous souhaite à tous les trois une très bonne continuation dans vos projets professionnels futurs.

Je tiens aussi à remercier Anne Sabourin, pour m'avoir enseigné une certaine rigueur scientifique, que ce soit en réunion de suivi de thèse, ou en donnant de ton temps aux tableaux blancs de Rue Barrault. Je ne peux pas nier ta contribution à ma thèse, et à ma formation. I want to thank Marc Sebban and Robert Williamson, for having reviewed the thesis. Their advice contributed to the quality of the final version of the manuscript, and to the permanent improvement of my writing skills. I also thank Florence d'Alché Buc, for teaching me machine learning in the classroom five years ago, as well as for accepting to be the president of my thesis jury today. Last, but not least, I want to thank Oldaric-Ambrym Maillard and Isabel Valera for accepting to join my jury.

Merci aux anciens doctorants (Anna, Albert, Igor, Guillaume P., Nicolas, . . . ) du labo, qui m'auront beaucoup aidé directement en début de thèse, et qui ont ensuite souvent incarné un exemple à suivre.

Merci aux doctorants de "ma promotion" (Hamid, Pierre L., . . . ) pour ces débats animés sur des sujets variés tels que le sport, les sujets de société, la mode, en passant par les mathématiques. Je remercie particulièrement Mastane, sans qui l'école d'été à Buenos Aires n'aurait pas aussi été ce road-trip légendaire. Je retiens ton invitation dans cette ville hispanophone un peu plus animée que Neuquen.

Merci aux nouveaux doctorants (Pierre C., Émile (not on mute), Kimia, Amaury, Guillaume, Anas, Myrto, . . . ) du labo, sans qui ma fin de thèse n'aurait pas été la même. Malgré la distance qui nous sépare du point zéro, je suis convaincu qu'entre ces murs nous sommes au bon endroit.

Merci à mes collègues en entreprise, que ce soit les nouveaux (Baptiste, Arnaud, Damien, Dora, . . . ) et les moins nouveaux (Emine, Jonathan, Éric, . . . ). Je serais toujours partant pour un petit afterwork à la Défense. Je n'oublie bien sûr pas Richard, mon "camarade de CIFRE", auquel je souhaite bon courage pour la dernière ligne droite.

Merci à mes amis passés par la thèse (Yannick, Guillaume C., . . . ), ainsi qu'aux doctorants des quatre coins du monde rencontrés en école d'été. Leur authenticité m'a permis de mettre ma propre expérience en perspective dans les moments difficiles. Leur enthousiasme a été communicatif.

Introduction

Biometrics is the discipline of distinguishing individuals based on physical or behavioral attributes such as fingerprints, face, irises and voice. In the modern world, biometrics has many essential applications, such as border crossing, electronic commerce and welfare disbursement. While its mainstream usage is recent, the discipline is not new. Indeed, in the late 19th century, the French law enforcement officer Alphonse Bertillon proposed a personal identification system based on the measurement of specific bony parts of the body [START_REF] Jain | Introduction to Biometrics[END_REF].

Today, the most widespread biometric measurement is fingerprint recognition, followed by face and iris recognition. All of them rely on the acquisition of images of specific parts the body. Hence, while biometrics is seen by many authors as a distinct field of science, its history and development is tightly related with that of computer vision, the interdisciplinary scientific field that seeks to enable computers to gain high-level understanding of digital images.

In the early 2010s, the performance of computer vision systems started to dramatically improve [START_REF] Goodfellow | Deep Learning[END_REF], due to the development of general-purpose computing on graphical processing units (GPGPU). It enabled the widespread adoption of neural networks -statistical models composed of layers that summarize information -as their training strongly benefits from very fast matrix multiplication. Training neural networks consists in finding the parameters of the network that minimize a loss function with gradient descent algorithms. Those modify iteratively the network parameters, by adding a small quantity that is negatively proportional to the gradient at each step. The growing interest for neural networks has fueled a huge corpus of literature. Most papers propose a better architecture for the model, suggest an improvement of the optimization method or introduce a better loss function for a particular use case.

Recent literature has proposed many loss functions for biometrics, built on the intuition that a more stringent separation of identities in an embedding space leads to improvements in performance [START_REF] Wang | Deep face recognition: A survey[END_REF]. The flagship problem of biometrics is 1:1 verification, which seeks to verify the identity of an individual by comparing a live measurement with reference data using a similarity measure. For example, the entry of an individual to a restricted area can require the conformity of the measurement with a personal identification card. Performance in 1:1 verification is evaluated using the ROC curve, a functional criterion that summarizes the quality of a similarity measure. For a set of tests, the ROC curve gives all false acceptance and false rejection rates attainable by thresholding the similarity. It is the golden standard for evaluating score functions. In the thesis, we argue for using the bipartite ranking literature to design loss functions for 1:1 verification, which is seen as scoring on pairs of observations. While both scoring and pairwise learning are addressed in the literature, their simultaneous examination is new and raises distinct challenges.

The recent dramatic improvements in accuracy of many machine learning applications foreshadow the emergence of new markets, issued from the maturation of formerly very experimental technologies. One such market is facial recognition, which has recorded, and is expected to maintain, exponential growth. Its development brought media coverage about the potential misuses and systemic biases of the technology, on top of the usual privacy concerns. In that context, practitioners and governmental agencies have recorded differences in accuracy between ethnicities in face recognition [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. One common explanation is that available face databases for training face recognition systems fail to represent the general population. The performance discrepancy raises the broader issue of fairness, a common concern with automated decisions that has received increasing attention in the recent machine learning literature [START_REF] Barocas | Fairness and Machine Learning[END_REF]. Some observers commented that predictive algorithms run the risk of being merely "opinions embedded in mathematics". In their view, practitioners should not focus on predictive performance alone, but also enforce the conformity of their system with a set of moral values. Biometrics is also concerned with fairness. Indeed, even when the representativeness of the training database in terms of gender is accounted for, women still account for more false positives than men, possibly due to societal norms regarding appearance. This may lead to systemic discrimination, notably when considering systems that flag people of interest. In the thesis, we first propose to correct biases using importance weights, which adjusts the distribution of the training data. The correction covers several cases where train and test data do not match, and assumes the availability of auxiliary information about their relationship. The level of generality that we propose is new, and covers many important applications in biometrics. Then, we propose to modify loss functions to incorporate explicitly fairness considerations when learning a score for bipartite ranking. While fairness in classification has received a lot of attention, fair bipartite ranking has not. Considering our scoring perspective on 1:1 verification, this work is an intermediary step to explicitly incorporate fairness constraints in this biometric problem.

In general, many issues from the machine learning literature arise simultaneously in the design of biometric systems. The objective of this thesis is to identify and address several of those from the perspective of statistical machine learning, in the hope of providing security guarantees under probabilistic assumptions, and proposing sensible solutions to biometric systems manufacturers. Generalized inverse of a càdlàg function N (resp. R) Natural (resp. real) numbers N ˚" Nt0u, R `" tx P R | x ě 0u, R ˚" R `zt0u

Table 1.1: Summary of notations.

In that regard, the usual statistical machine learning literature deals with simple problems, such as classification or regression [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF]. However, the problems tackled in biometrics involve both pairwise learning and a functional criterion, and thus require a specific analysis.

Chapter outline. The current chapter summarizes the contributions of the main parts II and III of the thesis, and skips the remaining part (Part I). Part I is in the thesis for clarity reasons and only contains technical preliminaries to the theoretical results of the other parts. The current chapter is organized as follows: firstly, Section 1.3 extends on the ideas presented in the short introduction above and provides a detailed outline for the thesis. Secondly, Section 1.4 focuses on Part II and discusses similarity ranking. Thirdly, Section 1.5 summarizes the contributions of Part III on the broad topic of reliable machine learning. Finally, Section 1.6 details the perspectives of the thesis. Section 1.3 is a developed introduction that focuses on important aspects of biometrics that are addressed in the machine learning literature. More precisely, it first presents the relationship between biometrics and metric learning, as well as the impact of deep learning on both fields.

It then delves into the nature of bias in facial recognition, and details potential dangers of bias through the lens of algorithmic fairness.

Section 1.4 is a short summary of Part II. It focuses on the idea of considering similarity learning as scoring on a product space. We name this view similarity ranking. In that regard, it begins with theoretical guarantees for that problem. Then, it proposes strategies with guaranteed statistical accuracy to reduce the computational complexity of similarity ranking, and ends with several practical, gradient-based, approaches.

Section 1.5 is a short summary of Part III, which deals with reliability in machine learning. As such, it first proposes a strategy to predict an ordered list of probable classes from multiclass classification data, instead of focusing on classification precision alone. Then, it gives strategies to deal with the lack of representativeness of databases. It ends with a proposal to enforce fairness for the scoring problem.

Section 1.6 presents the perspectives of the thesis. Specifically, it stresses the importance of practical illustrations of this work for biometric practitioners, and discusses several possible extensions of our analyses.

The notations adopted throughout the thesis are summarized in Table 12.1.

Recent Challenges in Biometrics

The recent advances in deep learning have brought rapid changes to the state-of-the-art in biometrics. In that regard, the new subfield of deep learning metric has proposed many derivable losses, but none of them relate to the ranking-based evaluation of biometrics systems. At the same time, the advances have stirred public debate about biometric technologies, and in particular on face recognition. While most issues concern the usage of facial recognition algorithms, one is its recently-measured racial bias. The machine learning research community can propose technical solutions for that problem.

Introduction to Biometrics

Societal value. Biometrics responds to the need to establish the identity of a person with high confidence. It has become crucial in the modern world, since one interact with an always increasing number of people. However, its roots can be traced as early as the late 19th century, with the first recorded uses of fingerprints for identification (Jain et al., 2011, Section 1.8). Today, biometrics are widely used in forensics and other governmental applications, as well as by various industries, such as the banking sector. One example of large-scale biometrics application is the Aadhaar project, managed by the Unique IDentification Authority of India (UIDAI), which has assigned national identification numbers and recorded the biometrics (irises, faces and fingerprints) of over one billion people (Jain et al., 2011, Section 1.6).

Formal objectives. The objective of biometric systems is to compare two measurements px, x 1 q in an input space X , e.g. two fingerprints or two faces, and to decide whether both of them originate from the same individual. It is generally done by means of a pairwise similarity function s : X ˆX Ñ R `, that quantifies the likelihood that x and x 1 originate from the same person. The decision is taken by thresholding the similarity value, i.e. the pair px, x 1 q is a match if spx, x 1 q ą t, where t is a threshold in R `. Two flagship problems can be identified in biometrics: verification and identification (Jain et al., 2011, Section 1.3). The verification problem is also referred to as 1:1 matching or 1:1 authentication. It is illustrated by the use case of border crossing, where an official compares a document x 1 with a live measurement x. As such, it consists in making a decision on a pair px, x 1 q. On the other hand, identification is illustrated by the use case of automatic surveillance, where a live measurement x is compared to a database. Precisely, it consists in finding the existence of a match to x in a database of N P N observations D N " tx i u N i"1 Ă X . If a match exists, it needs to return the relevant elements in D N . Identification is also referred to as 1:N matching or 1:N authentication. The number of enlisted people N can be large, for example in the millions.

Operational steps. To compare an observation x with a large database D N , one needs the capacity to match quickly low-memory representations of elements in X . It requires the derivation of efficient intermediary representations of the input data. Biometric systems can usually be split in three distinct processes: 1) the acquisition of the input data, called enrollment, 2) the feature extraction, sometimes referred to as the encoding of the data, 3) and the matching of the encodings. See [START_REF] Jain | Introduction to Biometrics[END_REF] (Section 1.2) for more details. In the context of fingerprint recognition, the enrollment phase covers the acquisition of the raw input, post-processing steps, as well as quality verifications on the final image. Feature extraction consists in applying usual computer vision techniques, followed by domain-specific techniques to extract specific points in the fingerprint image. For example, Gabor filters (Szeliski, 2011, Section 3.4.1) are used to derive ridge orientation maps (Jain et al., 2011, Chapter 2) from raw fingerprint images. Characteristic points called minutiae are then extracted from that intermediate representation. Finally, matching relies on the evaluation of a distance between the point clouds of two images. We refer to Section 2 of [START_REF] Jain | Introduction to Biometrics[END_REF] for more details.

Feature extraction. The module that has received the most attention in biometrics research is the feature extraction module. For example, research in automatic fingerprint recognition has dedicated many man-hours to finding as much discriminative information as possible in the images. In the context of facial recognition, the feature extraction part was first based on EigenFaces [START_REF] Turk | Face recognition using eigenfaces[END_REF], an application of the principal component analysis (PCA) to natural images of faces. Then, it combined common computer vision descriptors -such as Local Binary Patterns (LBP) and Scale-Invariant Feature Transform (SIFT) image descriptorswith dimensionality reduction techniques. Finally, it now relies on end-to-end -models that perform a task using raw input -deep convolutional neural networks, a type of neural network well suited for images [START_REF] Wang | Deep face recognition: A survey[END_REF].

Metric learning algorithms serve to train the feature extraction module. Recently, the advent of deep learning algorithms has completely shifted the focus of biometric researchers from a combination of domain-specific feature extraction and linear metric learning, to end-to-end deep metric learning. Researchers in biometrics thus have to follow closely recent developments in machine learning -and especially deep learning -to stay competitive.

Deep Metric Learning for Biometrics

Metric learning or similarity learning is the machine learning problem that that seeks to learns how similar two objects are. We refer to Bellet et al. (2015a) for a survey. In biometrics, the supervision for those algorithms originates from a database of n images tx i u n i"1 Ă X , with each image x i having an associated identity y i P t1, . . . , Ku with K ď n and pn, Kq P N 2 .

Linear metric learning. The first metric learning algorithms and a large part of the literature focus on linear metric learning. It refers to distances or similarity functions s : X ˆX Ñ R that are linear functions of their inputs. Those mainly rely on the use of a Mahalanobis distance -a distance function R d ˆRd Ñ R `for d P N parameterized by a positive semidefinite matrix M P R dˆd -with a few other linear metric learning methods using other combinations of M and the input x, x 1 . The Mahalanobis distance d M between the points x and x 1 satisfies:

d M px, x 1 q " b px ´x1 q J M px ´x1 q.
The Cholesky decomposition (Petersen and Pedersen, 2008, Section 5.5.1) implies that one can write M " LL J , where L is a lower triangular matrix. This result justifies seeing Mahalanobis distances as computing a simple Euclidean distance over a transformation of the inputs, since d M px, x 1 q " Lx ´Lx 1 2 , where ¨ 2 is the standard Euclidean distance. Notable approaches for learning Mahalanobis distances include the Mahalanobis Metric for Clustering (MMC) algorithm in 2002 [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF], the Neighborhood Component Analysis (NCA) algorithm in 2004 [START_REF] Goldberger | Neighbourhood components analysis[END_REF], and the Large Margin Nearest Neighbor (LMNN) algorithm in 2009 [START_REF] Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF]. Several authors have considered the extension of linear metric learning algorithms. For instance, they proposed the kernelization of linear metric learning methods, as well as the use of local linear metrics (Bellet et al., 2015a, Section 5). These extensions turned out to be useful for facial recognition practitioners. For exemple, [START_REF] Bohné | Large margin local metric learning[END_REF] considered applying the algorithm MM-LMNN [START_REF] Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF], which learns a local linear metric, to facial recognition.

Deep metric learning (DML). Due to the development of general-purpose computing on graphical processing units (GPGPU), the training and use of very deep neural networks became practical. It has had the effect of dramatically improving the performance of computer vision, notably for the task of large-scale classification on the challenge ILSVRC (ImageNet Large Scale Visual Recognition Challenge). The most salient breakthrough for that challenge happened in 2012 and is presented in [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. Results with deep networks led to important advances in face recognition as early as 2014 [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF]. Deep facial recognition encodes raw data x to a vector epxq, where e : X Ñ R d is a non-linear function and corresponds to the output of a neural network. Then, a simple distance is computed between epxq and epx 1 q to decide whether x and x 1 match. The encoding e is optimized by gradient descent to minimize a loss function. On the contrary to other popular biometries such as fingerprints and irises, finding manually the important distinctive features of a face is difficult. Therefore, an end-to-end gradient-based approach is very well-suited to face recognition. Still, authors have already proposed deep metric learning approaches for other biometries [START_REF] Minaee | Fingernet: Pushing the limits of fingerprint recognition using convolutional neural network[END_REF].

Loss functions for DML. Deep metric learning has replaced the sequence of hand-crafted features and dimensionality reduction by end-to-end models, as illustrated in the very influential paper of [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. Therefore, a lot of research focuses on better network architectures. Simultaneously, the advent of gradient-based learning has paved the way for extensive research in the design of loss functions. Early facial recognition systems used the usual softmax cross-entropy loss, an usual classification loss in deep learning that seeks to separates identities (Goodfellow et al., 2016, Section 3.13). Since then, many other loss functions have been proposed, such as ArcFace [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF]. We refer to Figure 5 of [START_REF] Wang | Deep face recognition: A survey[END_REF] for an overview of losses for facial recognition. Their objective is either to: increase the margin between the identities to increase inter-group variance, group all observations of each identity together to decrease intra-group variance, or combine both approaches as does the triplet loss [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. Practicioners have reported that summing different losses, while adjusting the proportion of each loss was necessary to optimize for performance [START_REF] Parkhi | Deep face recognition[END_REF].

Ranking-based evaluation. The performance of facial recognition systems is measured on the ROC curve, as shown in the evaluations of commercial face recognition systems by [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. Those were conducted by the National Institute of Standards and Technology (NIST), an agency of the United States Department of Commerce. The ROC curve is a standard for evaluating scores functions for bipartite ranking, a problem that seeks to assign higher scores to elements associated with a label `1 than to elements with label ´1. We refer to [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF] for a survey of bipartite ranking. In that context, similarity functions can be seen as score functions on the product space. That observation suggests that exploring the vast corpus of research on bipartite ranking is a justified approach to find better loss functions for facial recognition, which is what we undertake in both Part II and in Chapter 8 of Part III. Facial recognition has observed rapid improvements in accuracy and does not require the individual to be cooperative. Therefore, the technology has recently gathered attention in the media and in public opinion. On top of the usual privacy concerns, commentators have expressed growing concerns over the possible unreliability or unfairness of facial recognition.

Reliability of Biometrics

The recent advances in facial recognition have confirmed the maturation of the technology, which foreshadows its deployment and has provoked widespread debate about it. [START_REF] Gates | Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance[END_REF] warned about the tendency of the public to extrapolate about the ubiquity of such systems, which creates an illusion of surveillance that changes behavior. However, the technical hurdles enunciated by [START_REF] Gates | Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance[END_REF] seem much weaker today. Precisely, papers such as [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] show the ability of facial recognition systems to handle very loosely controlled acquisition conditions. Also, observers have forecasted a compound annual growth rate (CAGR)i.e. an average geometric progression -of 14.5 % per year between 2020 and 2027 for the global facial recognition market. In that context, the issues regarding the deployment of facial recognition and other machine learning technologies belong to the domain of the legislator, but the decisions of the model are of the responsibility of the machine learning practicioner.

Bias in facial recognition. In the specific case of facial recognition, the NIST has precisely Figure 1.1: This graph shows that fixing a threshold for a fixed false positive rate (called False Matching Rate (FMR) in facial recognition) of α " 10 ´5 on the general population can give higher false positive rate when used on another type of population. Specifically, it gives α " 10 ´2.8 for the population that originates from Polynesia. quantified important negative differentials in recognition performance for people of color, with Caucasians as the reference class. Figure 1.1 illustrates that and is extracted from [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. The observation was echoed by many news outlets and referred to as "racial bias" in 2019. The main justification was that the face databases used for the training of such systems are generally composed of European and northern American personalities, which are mostly Caucasians and do not represent the general population. Authors interpreted the observation as an "other-race effect" for automatic face recognition, an idea introduced in [START_REF] Furl | Face recognition algorithms and the other-race effect: Computational mechanisms for a developmental contact hypothesis[END_REF] and [START_REF] Petersen | An other-race effect for face recognition algorithms[END_REF], that states that humans generally struggle to recognize ethnicities different from their own. Some authors have presented strategies to correct this specific issue explicitly [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF]. More generally, a broad literature on bias in machine learning can be invoked to tackle that issue [START_REF] Bolukbasi | Man is to computer programmer as woman is to homemaker? debiasing word embeddings[END_REF][START_REF] Zhao | Men also like shopping: Reducing gender bias amplification using corpus-level constraints[END_REF][START_REF] Hendricks | Women also snowboard: Overcoming bias in captioning models[END_REF][START_REF] Liu | Transfer learning by sample selection bias correction and its application in communication specific emitter identification[END_REF][START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF]. Chapter 9 of Part III contributes to that effort, by providing a general reweighting scheme that addresses usual representativeness problems in biometrics.

Limitations of dataset representativeness. While having a database that represents the target population is important, it cannot be expected to correct for inherent biases in the training data. Precisely, even if a social group identified by protected attributes such as race or religion is significantly poorer on average than another social group, it can be deemed as immoral to refuse a loan on the basis that an applicant belongs to the former group. In that context, observers have bluntly qualified predictive algorithms of "opinions embedded in math" (O 'Neil, 2016).

Algorithmic fairness. A large body of research [START_REF] Agarwal | A reductions approach to fair classification[END_REF][START_REF] Woodworth | Learning non-discriminatory predictors[END_REF]Zafar et al., 2017a[START_REF] Zafar | Fairness constraints: A flexible approach for fair classification[END_REF][START_REF] Menon | The cost of fairness in binary classification[END_REF][START_REF] Bechavod | Learning fair classifiers: A regularization-inspired approach[END_REF] has emerged on the topic of fairness in machine learning -also called algorithmic fairness -which is a new field of study. It seeks to add explicit constraints to the training phase of machine learning models, so that blunt optimizations for accuracy do not lead to a reproduction of systemic societal biases. Early influential works date all the way back to 2012 [START_REF] Dwork | Fairness through awareness[END_REF]. More recently, authors have worked on a textbook dedicated to the subject [START_REF] Barocas | Fairness and Machine Learning[END_REF].

Fairness in facial recognition. In facial recognition, including fairness constraints when learning models can correct for social groups being harder to identify than others. It is a necessity in many practical cases. For example, if a system designed to flag persons of interest has a higher rate of false acceptance for a specific ethnicity, it may interpreted as automatic racial profiling.

Fairness literature. The literature in fairness for classification is broad, but there is little work in specific settings, such as ranking or similarity learning. Notable exceptions include [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF] for ranking, which only modifies a score function with a post-processing step to satisfy a criterion of fairness. This suggests an opportunity for new approaches to fairness that are specifically tailored for those important problems, which we address in Chapter 10 of Part III.

As a whole, the thesis participates in the dialogue between the machine learning community and the biometrics community. It proposes a stylized and theory-oriented view of challenges of biometrics, which leverages recent literature to study the specifici.e. pairwise and functional -criteria used in biometrics. We hope that our perspective on biometric problems will have valuable impacts on practice, and underline that the derivation of statistical guarantees for biometric systems constitutes an important tool to ensure their security.

Thesis outline

The thesis is divided in three parts. Part I of the thesis contains technical preliminaries, which provide all of the intermediary results necessary to derive the theoretical contributions of the thesis. It is featured in the thesis for clarity reasons. Part II and Part III focus on our contributions. Part II delves into the idea of viewing similarity learning as a scoring problem on a product space. Part III engages with the broad idea of reliable machine learning.

Part I is divided in three chapters. The first chapter (Chapter 2) is a quick introduction to statistical learning theory. It presents the necessary results to derive generalization guarantees in the easy setting of binary classification. Precisely, it details the derivation of finite-sample bounds on the excess error of the empirical minimizer. Most of our theoretical contributions can be interpreted as extensions of those results, but our contributions concern more complicated problems. The second chapter (Chapter 3) deals with all of the necessary results that relate to the idea of ranking in machine learning. Our contributions build on two topics related to rankings: bipartite ranking and ranking aggregation. Indeed, our work extends existing bipartite ranking guarantees to the similarity ranking setting presented in Part II, ranking aggregation is used for bagging ranking trees, and the guarantees of the first chapter of Part III are built on a parametric model for rankings, which can be considered as a tool for ranking aggregation. Finally, the third and last chapter (Chapter 4) features a short introduction to important results in U -statistics, an essential component of all pairwise learning problems. As such, Chapter 4 is a prerequisite to our similarity ranking guarantees, and is involved in the estimation of risk functionals in standard bipartite ranking.

Part II explores the idea of considering the biometrics verification problem as ranking pairs of instances. It is divided in three chapters. The first chapter (Chapter 5) presents formally the idea of seeing similarity learning through the lens of ROC optimization. It proposes novel guarantees for the problem of pointwise ROC optimization, which seeks to optimize for true positive rates under an upper bound on the false positive rate. This analysis paves the way for an extension of the guarantees of the TreeRank algorithm to similarity learning, that we deliver. Using numerical simulations, we provide the first empirical illustration of fast learning rates, tailored here to the specific case of pointwise ROC optimization for similarity ranking. Due to the prohibitive number of pairs involved in the computations, the propositions of the first chapter are not practical for any large-scale application. To correct this, statisticians proposed sampling approximations for U -statistics, which is an useful approach in similarity ranking. The second chapter (Chapter 6) extends that proposition to settings where the dataset is distributed. Finally, the third chapter (Chapter 7) is rather prospective and proposes simple numerical toy experiments on similarity ranking, that address the optimization aspect of the problem. The extension of those experiments will be the subject of future work.

Part III is the last part of the thesis. It revolves around the idea of reliable machine learning and 1.4. Similarity Ranking is also divided in three chapters. The first chapter (Chapter 8) derives learning guarantees for predicting an ordering over possible classes with only multiclass classification data, which relies on the use of a One-Versus-One (OVO) strategy. That problem often arises in noisy problems, i.e. those for which the top-1 ranked class has a good chance to be a false positive. One is then interested in the most likely classes, as is often the case in forensics. The second chapter (Chapter 9) proposes techniques to correct bias between the train and test sample, with auxiliary information on the difference between the two. It relies on an application of the well-known principle of importance sampling in statistics. Finally, the third chapter (Chapter 10) proposes an unification of a class of fairness constraints, as well as a new and more restrictive fairness constraint, that is more suited to practical situations. It also features theoretical guarantees and gradient-based approaches for learning under both types of constraints.

The last chapter of the thesis (Chapter 11) contains a summary of the contributions of Part II and Part III, as well as a detailed account of the most promising directions for future work. It ends with a general conclusion on the thesis.

The next two sections of this chapter -Section 1.4 and Section 1.5 -summarize the contributions of the thesis. Each section focuses respectively on the first and second part of the thesis, and is divided in subsections which summarize each chapter of the part. Finally, Section 1.6 sums up the perspectives of the thesis.

Similarity Ranking

Similarity learning plays a key role in many machine learning problems such as clustering, classification or dimensionality reduction. It is especially important when one considers openworld problems,e.g. situations when a model encounters classes after deployment that were not available during training [START_REF] Chen | Lifelong Machine Learning[END_REF] -which is the case for any biometric application. In this section, we consider metric learning from the perspective of scoring pairs of instances, which is coherent with the evaluation of many systems based on metric learning techniques.

Similarity Ranking Theory

Bipartite ranking/scoring considers a set of elements associated to a binary label, and seeks to rank those with label `1 higher than those with label ´1. To derive an order on an input space X , bipartite ranking is generally tackled by learning a score function s : X Ñ R [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF]. On the other hand, the field of metric/similarity learning (Bellet et al., 2015a) is the task of learning a similarity -or equivalently, a distance -s : X ˆX Ñ R on the product space X ˆX . While metric learning algorithms were originally evaluated w.r.t. their relevance for a clustering task [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF], today practitioners use performance indicators derived from the ROC curve, the golden standard for evaluating scoring functions in bipartite ranking. Therefore, our work introduces as similarity ranking the idea of learning similarities for a ranking objective.

A functional criterion: the ROC curve. In the multi-class classification setting, introduce a random pair pX, Y q P X ˆt1, . . . , Ku, with K P N the number of classes, as well as an independent copy pX 1 , Y 1 q of pX, Y q. Then, one can define a random variable Z " 2 ¨ItY " Y 1 u ´1 that is equal to 1 if both pairs belong to the same class and ´1 otherwise. The ROC curve of a similarity function is then equal to the PP-plot t P R Þ Ñ p s H s ptq, s G s ptqq, where, for all t P R: s H s ptq :" PtspX, X 1 q ą t | Z " ´1u and s G s ptq :" PtspX, X 1 q ą t | Z " `1u.

s H s ptq and s G s ptq are respectively the false positive and true positive rate associated to the similarity s. Under continuity assumptions, the ROC curve writes as the graph of the function α P p0, 1q Þ Ñ ROC s pαq " s G s ptq ˝s H ´1 s pαq. Previous approaches for similarity learning optimize an empirical evaluation of the Area Under the ROC Curve (AUC) of the similarity function s [START_REF] Mcfee | Metric learning to rank[END_REF][START_REF] Huo | Cross-modal metric learning for auc optimization[END_REF].

Pointwise ROC optimization (pROC). The AUC is a global summary of the ROC curve which penalizes ranking errors regardless of their position in the list (Clémençon et al., 2008, Proposition B.2). Other criteria focus on the top of the list [START_REF] Clémençon | Ranking the best instances[END_REF][START_REF] Huo | Cross-modal metric learning for auc optimization[END_REF], and their study is the subject of the Ranking the Best literature (Menon and Williamson, 2016, Section 9). In our work, we consider optimizing the true positive rate attained by a similarity under an upper bound α P p0, 1q on its false positive rate. This setting is relevant in biometric applications, as security guarantees are specified with a limit on the false positive rate of the system. We refer to this problem as pointwise ROC optimization (pROC). Considering the risks: R ´psq :" ErspX, X 1 q | Z " ´1s and R `psq :" ErspX, X 1 q | Z " `1s, with S a proposed family of similarities, the pROC problem writes: max sPS R `psq subject to R ´psq ď α.

(1.1)

We denote a solution of Eq. (1.1) as s ˚. Clémençon and Vayatis (2010) studied the equivalent of Eq. (1.1) in bipartite ranking. This problem is analogous to Neyman-Pearson classification [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF], and bears close resemblance to the minimum-volume set problem [START_REF] Scott | Learning minimum volume sets[END_REF]. When S is the class of all measurable functions, a solution of Eq. (1.1) writes as a super-level set of the posterior probability η : x, x 1 Þ Ñ PtpX, X 1 q " px, x 1 q | Y " Y 1 u, which is a consequence of the Neyman-Pearson fundamental lemma [START_REF] Lehmann | Testing Statistical Hypotheses[END_REF], Theorem 3.2.1).

Pairwise estimators. The analysis of Clémençon and Vayatis (2010) relies on the fact that natural estimators of R ´psq and R `psq are standard empirical means in the case of bipartite ranking, However, it is not true in similarity ranking. Consider a sample D n " tpX i , Y i qu n i"1 composed of n i.i.d. copies of the pair pX, Y q, then the natural estimators of R ´psq and R `psq based on D n write:

R ń psq :" 1 n ´ÿ iăj ItY i ‰ Y j u ¨spX i , X j q, (1.2)
R ǹ psq :" 1 n `ÿ iăj ItY i " Y j u ¨spX i , X j q, (1.3)

where n `:" ř iăj ItY i " Y j u and n ´:" npn ´1q{2 ´n`. The quantities in Eq. (1.2) and Eq. (1.3) are not sums of independent random variables, hence the analysis of Clémençon and Vayatis (2010) breaks down. However, they are ratios of the well-known U -statistics [START_REF] Lee | U -statistics: Theory and practice[END_REF][START_REF] De La Pena | Decoupling: from Dependence to Independence[END_REF].

Generalization guarantees for pROC. The empirical counterpart of pROC (Eq. (1.1)) writes:

max sPS R ǹ psq subject to R ń psq ď α `φ, (1.4)
where φ ě 0 is a term, which tolerates the variations of R ń psq around its expectation R ´psq. We denote by s n a solution of Eq. (1.4). The generalization of standard concentration inequalities to U -statistics enables us to extend the uniform guarantees of Clémençon and Vayatis (2010).

Precisely, we simultaneously ensure with high probability: that R `ps ˚q´R `ps n q is upper-bounded by a quantity of order n ´1{2 and that R ´ps n q ď α `φn with φ n " Opn ´1{2 q. In summary, we show that both excesss risk are bounded with the standard learning speed in n ´1{2 without assumptions on the distribution of the data.

Fast generalization speeds for pROC. In the case of binary classification, [START_REF] Mammen | Asympotical minimax recovery of the sets with smooth boundaries[END_REF] have shown that, under a noise assumption parameterized by a P p0, 1q on the data distribution, fast convergence speeds in Opn ´1{p2´aq q hold. The fast speed is a consequence of an upper-bound on the variance of the excess risk, that is derived from the noise assumption. The analysis of Clémençon and Vayatis (2010) built on those ideas to propose an upper bound on R `ps ˚q ´R`p s n q in Opn ´p2`aq{4 q with similar guarantees in Opn ´1{2 q for R ´, for pROC in bipartite ranking. Compared to the binary classification setting, pROC has lower learning speeds, which comes from the bilateral nature of pROC. Our work extend the fast speeds of Clémençon and Vayatis (2010) from bipartite ranking to the case of similarity ranking. Incidentally, the result is true under a much weaker assumption. Precisely, it relies on the second Hoeffding's decomposition for U -statistics [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF], which implies that the deviation of the excess risk consists mainly in that of its Hájek projection. Since the Hájek projection is a variancereducing transformation of a U -statistic (van der Vaart, 2000, Section 11), weaker assumptions imply the upper-bound on the variance required for fast learning speeds. Clémençon et al. (2008) featured that usage of the properties of U -statistics to derive fast convergence speeds, but never applied it to problems that feature a random constraint.

Empirical illustration of fast speeds. Our work also contains the first experimental illustration of fast speeds of convergence, on our specific similarity ranking problem. The illustration relies on the generation of data that satisfies the noise assumption for different noise parameters a P p0, 1q, followed by a comparison of their empirical learning rates. We chose the data distribution and the proposed family S so that the optimal similarity s ˚is known and the empirical minimizer s n can be found exactly. For that matter, we set S as the decision stumps on a fixed transformation of the data.

Limitations of pROC. While the pROC problem echoes practical considerations in biometrics, where systems are deployed to work at a fixed rate of false positives α, its empirical resolution is hard in practice. Few exceptions rely on fixed partitioning of the input space [START_REF] Scott | Learning minimum volume sets[END_REF]. In many situations, the false positive rate α for a system is unknown in advance, thus optimizing for the wrong α may not yield satisfying outcomes at deployment.

TreeRank for bipartite ranking. The TreeRank algorithm for bipartite ranking was introduced in Clémençon and Vayatis (2009). TreeRank learns a piecewise constant score function s Dn , built to provide an adaptive piecewise linear estimation of the optimal ROC curve.

As implied by optimal solutions of Eq. (1.1), the optimal ROC curve ROC ˚is that of the posterior probability η. TreeRank recursively splits the input space X and optimizes greedily the AUC at each split, thus forming a binary tree of depth D n of nested partitions of the input space X . Under specific assumptions, Clémençon and Vayatis (2009) have proven uniform bounds in supremum norm between the optimal ROC curve and that of s Dn when D n " a logpnq, i.e. that with large probability: sup αPr0,1s ROC s Dn pαq ´ROC ˚pαq ď expp´λ a logpnqq, (

where λ is a constant specified by the user.

TreeRank for similarity ranking. Our work proposes an extension of the TreeRank algorithm for learning similarities, by considering recursive splits of the product space X ˆX . To ensure that the similarity s is symmetric, we consider only symmetric splits with respect to the two arguments in the input space X Ă R d , by splitting on the following simple reparametrization of X ˆX :

f :px, x 1 q Þ Ñ ˆ|x ´x1 | x `x1 ˙.
Using the same extensions of classic concentrations inequalities to U -statistics as before, we extended the proof of Eq. (1.5) to similarity ranking. Our analysis provides a theoreticallysupported approach to learning similarities that approach the optimal ROC curve in supremum norm.

While we have proven theoretical guarantees for approaches to similarity ranking, the estimators involved in the computation of the risk functionals require performing sums of very large numbers of terms. The induced computational cost makes the practical application of such approaches prohibitive. For example, calculating of R ń psq require summing n ´terms, which is quadratic in n when K is constant. In typical biometric applications, the number of samples per class is fixed. Hence, the proportion of negative pairs n ´over all pairs n 2 is even higher than the case K constant. The next section exploits recent analyses in the approximation of U -statistics to alleviate that problem.

Distributed U -Statistics

Most biometric applications learn on large-scale datasets. For facial recognition, the largest dataset released to the public contains 8.2 million images [START_REF] Guo | Ms-celeb-1m: A dataset and benchmark for large-scale face recognition[END_REF] and private datasets are much larger. The scale of facial recognition datasets justifies the computational concerns described in Section 1.4.1, as the number of negative pairs is higher than 50 trillions (10 12 ) for [START_REF] Guo | Ms-celeb-1m: A dataset and benchmark for large-scale face recognition[END_REF]. Besides this restriction on the number of operations, these datasets often cannot be contained in the random-access memory (RAM) of a single machine. In our work, we proposed an approach to tackle the estimation of U -statistics in a distributed environment, which deals with those two limitations simultaneously.

Incomplete U -statistics. The idea of alleviating the computational complexity of U -statistics is not new, as [START_REF] Blom | Some properties of incomplete U-statistics[END_REF] proposed summing over a small finite number of B pairs selected by sampling with replacement to in the set of all pairs to form incomplete U -statistics in 1976. [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] derived an upper bound on the deviation between an incomplete Ustatistic U B and the complete U -statistic U n that holds with high probability. In the case of one sample U -statistics of degree two -averages of all possible pairs formed with a sample -, the bound implies that using the estimator U B with only B " n pairs suffices to recover a usual learning rate of order n ´1{2 , instead of summing all npn ´1q{2 pairs as for U n . This result implies that one can extend the proofs for similarity ranking presented above so that they work with incomplete U -statistics, which makes the setting practical in large-scale learning applications.

Distributed environments. In cases where the data does not fit on a single machine, the recent technological progresses regarding distributed databases and parallel computation made the deployment of distributed machine learning accessible, largely due to the development of frameworks for cluster computing, such as Apache Spark [START_REF] Zaharia | Spark: Cluster computing with working sets[END_REF] or Petuum [START_REF] Xing | Petuum: A New Platform for Distributed Machine Learning on Big Data[END_REF]. These frameworks abstracted away the network and communication aspects of distributed algorithms. As such, they eased the deployment of distributed algorithms, but have restricted the types of operations that can be efficiently achieved, generally in order to guarantee algorithmic properties. At the same time, [START_REF] Jordan | On statistics, computation and scalability[END_REF] urged statisticians to guide practitioners of large-scale machine learning, by studying the implications of distribution on estimation, particularly to put in perspective the gains in computational time with the potential losses in statistical accuracy. Our work addresses this issue, by proposing several estimators for U -statistics in a distributed setting and comparing their variances. In that context, we propose time versus variance tradeoffs.

Probabilistic framework. Introduce two independent i.i.d. samples D n " tX 1 , . . . , X n u Ă X and Q m " tZ 1 , . . . , Z m u Ă Z of respectively n P N and m P N elements, such that D n and Q m can have different distributions. The complete two-sample U -statistic of kernel h : X ˆZ Ñ R associated with those samples writes as:

U n phq :" 1 nm n ÿ k"1 m ÿ l"1
hpX k , Z l q, (1.6)

with n " pn, mq. On the other hand, the incomplete counterpart of U n phq based on B pairs writes as:

U B phq :" 1 B ÿ pk,lqPD B hpX k , Z l q, (1.7)
where D B is a set of B elements selected at random in the set of all pairs tpk, lq | pk, lq P t1, . . . , nu ˆt1, . . . , muu. For large-scale data, the full datasets D n and Q m can not fit on a single machine, which makes the direct computation of Eq. (1.6) and Eq. (1.7) impossible. In that context, the standard approach is to distribute the data on N P N workers. For a standard mean, the simple computation of the average of the local means for each worker yields the same estimator as in the centralized setting. The computation is not that simple for U -statistics, since each worker can only form pairs with the local sample without network communication.

Distributed estimators for U -statistics. We first introduced two simple estimators in the distributed setting that do not require network communication: the average U n,N of N complete U -statistics on each local sample, and the average U n,N,B of N incomplete U -statistics formed with B randomly selected pairs on each local sample. Using the second Hoeffding decomposition, we derive an analytical formulation for the variances of those estimators, following the steps of [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF] for U n . Their expression shows that the estimators U n,N and U n,N,B have a limited accuracy, i.e. a minimum variance, that can be significantly higher than that of U n for a specific h and specific distributions of X 1 and Z 1 .

Similarity Ranking

Repartition for distributed estimators. This difference in variance between distributed estimators and U n comes from the fact that many pairs involved in the computation of the latter are not involved in that of the former. To counterbalance this effect, we propose to average estimators computed between repartitioning procedures, that reassign the observations to each of the clusters at random, so that every pair involved in U n has a chance to be seen. We propose two estimators with repartitioning: U n,N,T (resp. U n,N,B,T ) that averages T estimators U n,N (resp. U n,N,B ) computed on T different partitioning of the data. As T grows, the variance of those estimators approach the variance of U n by above.

Relative variance of our estimates. We provide analytical expressions for the variance of the estimators U n , U B , U n,N , U n,N,B , U n,N,T and U n,N,B,T , for several ways of distributing the data on the workers. In that regard, we first consider sampling without replacement (SWOR), which is relevant when splitting all data on several machines for space constraints. Then, we consider with replacement (SWR), which is relevant when selecting batches of data to compute several estimates in parallel, e.g. in minibatch gradient descent. We assume for both settings that each worker contains n{N elements of the sample D n and m{N elements of Q m , a setting we call prop. Relaxing that assumption implies that there is a nonzero probability to have no elements of either D n or Q m in a worker. In that case, one has to provide a default value for the estimator of that worker. Hence, the variance has no simple interpretable analytic form, so we provide empirical evidence that the observed variances are of the same order. Additionally, we characterize the parameters h, n, m and the distributions of X 1 , Z 1 for which repartitioning is important.

Learning with distributed U -statistics. [START_REF] Papa | SGD algorithms based on incomplete u-statistics: Largescale minimization of empirical risk[END_REF] studied stochastic gradient descent for incomplete U -statistics. While we do not extend their analysis to our distributed estimators, our work considers the minimization of U -statistics with stochastic gradient descent in a distributed environment, by estimating the gradient with U n,N,B and repartitioning the data every n r timesteps. We provide empirical evidence that lowering n r gives a better solution to the optimization process on average. Also, the variance of the loss of the final solution is much smaller, which shows the increased robustness obtained by repartitioning the data.

While Section 1.4.1 and this section account respectively for the generalization and scalability aspect of similarity ranking, the next section focuses on the optimization aspect. As such, the next section is more prospective and gives optimization strategies for similarity ranking problems on toy examples.

Practical Similarity Ranking

The development of the deep metric learning literature is motivated by a need for criteria that align with the requirements of biometric identification, and can be optimized by gradient descent.

Most of those criteria focus on better separating the identities with some heuristic, such as that all instances associated to a same identity should be mapped to the same point in a representation space, see the center loss [START_REF] Wen | A discriminative feature learning approach for deep face recognition[END_REF]. While those criteria are sensible, they are poorly connected with the evaluation of biometric systems, which is based on the ROC curve. The ROC curve is a functional made to assess the capacity of a scoring function to distinguish positive from negative instances, or in biometrics the capacity of a similarity function to distinguish matching instances from non-matching instances. The connection between bipartite ranking and biometrics motivates us to propose practical approaches for learning similarity functions, that optimize a measure derived from the ROC curve.

Previous analyses of similarity ranking. Viewing similarity learning as pairwise bipartite ranking is extensively discussed in other parts of the thesis under the name similarity ranking. Indeed, we provided generalization results for the pointwise ROC optimization (pROC) problem, sometimes referred to as Neyman-Pearson classification [START_REF] Scott | Learning minimum volume sets[END_REF]. The pROC problem consists in finding a score s and a threshold t for it, that give the highest possible true positive rate under an upper-bound on the false positive rate. Additionally, we provided an extension of the analysis and methodology of the TreeRank algorithm, which was originally shown to learn score functions s that approximate the optimal ROC in supremum norm (Clémençon and Vayatis, 2009), to learn similarity functions. Finally, we motivated theoretically such methods, as we prove that one can replace the computationally intensive estimators involved by incomplete U -statistics, which dealt with the computational limitations associated to similarity ranking.

Limitations of previous analyses. Our theoretical results justify considering the optimization of the pROC problem or the use of the TreeRank algorithm. However, they do not imply obvious approaches to optimize for the pROC problem, nor guarantee the empirical performance of our extension of the TreeRank algorithm. This chapter of the thesis provides illustrations of possible approaches, that are demonstrated to work on simple synthetic datasets. Deriving an extensive empirical evaluation of those approaches is an interesting direction for future work.

Solving pROC for linear similarities. While many papers discuss the pROC problem [START_REF] Scott | Learning minimum volume sets[END_REF]Clémençon and Vayatis, 2010;[START_REF] Rigollet | Neyman-pearson classification, convexity and stochastic constraints[END_REF], there lacks practical approaches to it, with the exception of a few propositions based on recursive partitioning, see e.g. [START_REF] Scott | Learning minimum volume sets[END_REF]. Introduce a sample of n data points as D n :" tpx i , y i qu n i"1 Ă X ˆt1, . . . , Ku, where y i is the identity of an observation x i P X and X Ă R d . In the case of similarity ranking, pROC at level α P p0, 1q over a class of proposed functions S, with s : X ˆX Ñ R for any s P S, writes:

max sPS 1 n `ÿ iăj
Ity i " y j u ¨spx i , x j q subject to 1 n ´ÿ iăj Ity i ‰ y j u ¨spx i , x j q ď α.

(1.8)

In the general case, Eq. (1.8) can be very hard to solve. For example, if S is composed of indicators of sets in X ˆX , then Eq. (1.8) might not be differentiable, nor continuous. However, if the family S has a simple form, solving Eq. (1.8) may be much easier. For example, we propose an explicit analytical solution when S is the set of all bounded bilinear similarities px, x 1 q Þ Ñ x J Ax with A P R dˆd and A F ď 1 where ¨ F is the Frobenius norm. While we show that this approach gives sensible solutions to pointwise ROC optimization for very specific distributions, the general case requires more flexible families of functions.

Solving pROC with gradient descent. To address pROC with more complex proposed families, we propose a gradient descent based approach to minimize a counterpart of the excess error for minimum volume set estimation of [START_REF] Scott | Learning minimum volume sets[END_REF], that is adapted for bipartite ranking. While we demonstrate its efficiency with a simple linear classifier and a toy example, the approach is flexible enough to accomodate for more complex models, as well as an extension to similarity ranking. However, we will have to demonstrate its performance empirically on more eloquent examples. Introduce as H and G the distributions of respectively X | Y " ´1 and X | Y " `1, as well as the optimal rejection region for pointwise ROC optimization as R α, then our relaxation of the excess error writes:

max pw,bqPR d ˆR ´GpR αq ´p Gpw, bq ¯``´p Hpw, bq ´α¯`s .t. w 2 2 `b2 ď 1. (1.9)
where p G and p H are relaxed empirical versions of respectively G and H, and pxq `:" maxpx, 0q for any x P R. By minimizing the objective of Eq. (12.9) and projecting the weights on the unit ball, our method recovers optimal rejection regions by gradient descent, for our toy example. Though the quantity GpR αq is unknown, we conjecture that the results are not extremely sensitive to this value, and that reasonable approximations can be proposed for it in most applications. The pROC problem is a principled approach to learning similarities with a ranking objective, but does not address ranking as a global problem. Precisely, it focuses on recovering a single level set of the score function, as demonstrated by the fundamental lemma of Neyman-Pearson, whereas bipartite ranking consists in recovering an order relation between any two points of the input space.

TreeRank for similarity learning. The TreeRank algorithm of Clémençon and Vayatis (2009) deals with bipartite ranking as a global problem. Precisely, it tackles bipartite ranking with a recursive splitting procedure, that solves binary classification problems with asymmetrically weighted errors between positives and negatives instances. As presented before, our work extended the algorithm to the case of similarity ranking, as well as the supremum norm guarantees for the distance of the ROC of the learned score to the optimal ROC. To build intuition on our TreeRank variant for similarity ranking, we illustrate visually the shape of our proposed symmetric regions for splitting the space.

Practical considerations for TreeRank. Two drawbacks of TreeRank are : 1) its dependence on the initial splits of the space, which jeopardize performance if the proposed splitting family is too limited, 2) and its discontinuous nature, which is inconsistent with natural hypotheses of many practical settings. A first response to these issues is to extend the idea of random forests proposed in [START_REF] Breiman | Random forests[END_REF] to TreeRank, which is proposed in Clémençon et al. (2013). We show on a toy dataset with continuous likelihood ratio dG{dH that averaged ranking trees can correct for both the initial mispecification of the proposed family and the discrete nature of ranking trees. Precisely, the averaged score function that we obtain gives a ROC curve that is almost undistinguishable from the optimal ROC, despite both its finite -but numerous -set of values, and the inadequacy of each tree with respect to the true likelihood ratio.

These proposals sketch a way for new algorithms specifically designed for solving the similarity ranking problem, which is a principled approach to dealing with the biometric identification problem. However, they are merely illustrations on very simple problems. Finding new approaches for practical similarity ranking, ideally for large-scale experiments that correspond to operational scenarios in biometrics, is an exciting track for future work.

Reliable Machine Learning

Besides the similarity learning aspect, biometrics and particularly facial recognition incarnate many important issues in machine learning, as shown in the reports of the NIST (National Institute of Standards and Technology) on industrial facial recognition benchmarks [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. Precisely, facial recognition is confronted with issues regarding the robustness of predictions, bias in the training data, as well as algorithmic fairness. The subsections of this section tackle all of those issues sequentially. While there is extensive literature on those general topics, our work deals with challenging and underexamined settings that directly apply to biometric problems. Precisely: 1) for prediction robustness, we consider learning an ordered list of identities, as do particular biometric identification problems, 2) for training data bias, we reweight training instances using high-level information, which can be nationality in the context of border control, 3) for fairness, we focus on score functions as a gateway to similarity functions, as done in Part II.

Ranking the Most Likely Labels

In biometrics systems for law enforcement, an human expert often analyzes several of the most likely suspects proposed by a system. More generally, hard classification problems focus on the most likely labels, such as the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) challenge, where [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and subsequent papers consider the accuracy at top-5 alongside the usual (top-1) prediction accuracy. In our work, we propose an approach to tackle predicting an ordered list of labels from classification data. We refer to that problem by the name of label ranking (LR). Precisely, we propose to use the well-known One-versus-One approach to classification, and derive guarantees for that approach.

Probabilistic framework for label ranking (LR). As usual in the multiclass classification setting, consider a random pair pX, Y q P X ˆY with Y " t1, . . . , Ku, as well as the risk Lpgq " PtgpXq ‰ Y u associated to a classifier g : X Ñ Y. The optimal classifier for Lpgq in the class of all measurable functions is the well-known Bayes classifier g ˚, defined as follows:

g ˚pxq :" arg max kPt1,...,Ku η k pxq, where η k : x Þ Ñ PtY " k | X " xu
is the posterior probability of class k for any k P t1, . . . , Ku. We refer to the task of finding an ordered list of the most likely labels as label ranking (LR). It amounts to associating to any x P X a permutation σ x P S K , such that:

η σ ˚´1 x p1q ą η σ ˚´1 x p2q ą ¨¨¨ą η σ ˚´1 x pKq .
(1.10)

We denote by σ X the corresponding random permutation, i.e. σ X satisfies, for any σ P S K , Ptσ X " σu " PtX P A σ u with A σ " tx P X | σ x " σu.

On ranking median regression (RMR). Another well-known problem in statistical literature that concerns predicting a ranking over a set of labels is ranking median regression (RMR) [START_REF] Tsoumakas | Mining multi-label data[END_REF][START_REF] Vembu | Label ranking algorithms: A survey[END_REF]Clémençon et al., 2018). RMR considers a random pair pX, Σq P X ˆSK , hence Σ is a random permutation. RMR learns from data a ranking rule s : X Ñ S K that minimizes the risk: (1.11) where d : S K ˆSK Ñ R `is a symmetric loss function. The distance d quantifies the distance between rankings. The most well-known distance is the Kendall τ distance d τ , which is equal to the number of pairwise disagreements between the two permutations.

Rpsq :" ErdpΣ, spXqqs,
Optimal solution of RMR. Previous work on RMR (Clémençon et al., 2018) has shown that: the optimal minimizer of Eq. (1.11) for measurable ranking rules has a simple analytical formulation for the distance d τ , under an assumption called the Strict Stochastic Transitivity (SST). SST assumes that the pairwise probabilities p k,l pxq :" PtΣpkq ă Σplq | X " xu and p l,k pxq :" 1 ´pk,l pxq for any 1 ď k ă l ď K satisfy: for all x P X and pi, j, kq P t1, . . . , Ku 2 with i ‰ j, we have p i,j pxq ‰ 1{2 and:

p i,j pxq ą 1{2 and p j,k pxq ą 1{2 ñ p i,k pxq ą 1{2.
Under the SST assumption, the optimal ranking rule for d τ writes:

s X pkq " 1 `ÿ l‰k Itp k,l pXq ă 1{2u.
(1.12)

LR as RMR with partial information. While the characterization of the optimal element is a welcome extension of usual learning theory [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF] to the RMR problem, the whole ranking Σ is not available when dealing with classification data. However, our work shows that if we consider the random permutation Σ to be generated by a conditional BLTP model [START_REF] Korba | Learning from ranking data : theory and methods[END_REF] with preference vector ηpXq " pη 1 pXq, . . . , η K pXqq, then it is possible to build Σ so that it satisfies Y " Σ ´1p1q almost surely. Based on this observation, we propose to consider LR as a RMR problem with the partial information Σ ´1p1q about the full random permutation Σ.

Optimal solutions of LR with One-versus-One (OVO). One can calculate the expressions of the pairwise probabilities p k,l pxq's under the conditional BTLP model with preference vector ηpxq. Precisely, we have p k,l pxq " η k pxq{pη k pxq `ηl pxqq for any x P X and k ă l. Remark that p k,l pxq corresponds to the probability of predicting k against l for the One-versus-One (OVO) problem of classifying k against l. The One-versus-One (OVO) approach is a well-studied approach [START_REF] Hastie | Classification by pairwise coupling[END_REF][START_REF] Moreira | Improved pairwise coupling classification with correcting classifiers[END_REF][START_REF] Allwein | Reducing multiclass to binary: A unifying approach for margin classifiers[END_REF][START_REF] Fürnkranz | Round robin classification[END_REF] to tackle multiclass classification using binary classification algorithms. OVO consists in learning KpK ´1q{2 decision functions, i.e. a classifier for each class k against l with k ă l, and taking the majority vote of the KpK ´1q{2 classifiers. The Bayes classifier for the pk, lq OVO problem is g k,l : x Þ Ñ 2 ¨Itp k,l pxq ě 1{2u ´1. Hence, Eq. (1.12) reduces to:

s X pkq " 1 `ÿ l‰k Itg k,l pXq " ´1u. (1.13)
Notice that s X corresponds to σ X in Eq. (1.10), as soon as all of the η k pXq are distinct. We showed that a combination of the optimal solutions of all of the KpK ´1q{2 OVO problems imply an optimal ranking rule s X . Hence, we can probably derive a good solution of the LR problem from good solutions of all of the OVO problems.

Guarantees for LR with OVO. We propose a solution to LR, that uses a combination of the solutions of all empirical OVO classification problems. Then, we derive theoretical guarantees for that solution. Introduce a sample D n " tpX i , Y i qu n i"1 of n i.i.d. copies of the random pair pX, Y q, as well as the notation Y k,l,i " ItY i " lu ´ItY i " ku for any k ă l and any i P t1, . . . , nu. The empirical risk p L k,l of g : X Ñ t´1, `1u for OVO classification of k versus l writes:

p L k,l pgq :" 1 n k `nl ÿ i:YiPtk,lu ItgpX i q ‰ Y k,l,i u,
where n k " ř n i"1 ItY i " ku for any k P t1, . . . , Ku. We denote the minimizer of p L k,l over the fixed proposed class G of binary classifiers as p g k,l . Following Eq. (1.13), an empirical solution of LR writes: p s X pkq :" 1 `ÿ k‰l Itp g k,l pXq " ´1u.

A simple union bound implies:

Ptp s X ‰ s X u ď ÿ kăl Ptp g k,l pXq ‰ g k,l pXqu.
(1.14)

Eq. (1.14) shows that the sum of the probabilities of not predicting the optimal class for each OVO problem bounds the probability of not predicting the optimal list of labels in LR. At the same time, a consequence of usual hypotheses for the derivation of fast generalization speedsfirst presented in [START_REF] Mammen | Asympotical minimax recovery of the sets with smooth boundaries[END_REF] and reviewed in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] -is an upper-bound of the r.h.s. quantity in Eq. (1.14) by the excess errors of the k versus l classification problems. Under a standard noise assumption, we provide usual fast convergence bounds in Opn ´1{p2´aq q for the excess error of each k versus l classification problem, where a P p0, 1q is a noise parameter. From Eq. (1.14), these results imply a convergence bound in Opn ´a{p2´aq q for the quantity Ptp s X ‰ s X u, which is slower than classification learning speeds, due to the inherent complexity of label ranking.

Implications of the analysis. The counterpart of the RMR error Eq. (1.11) for LR would be the following risk:

Rpsq :" ErdpspXq, σ X qs, (1.15)
for a ranking rule s : X Ñ S K . Note that, for any bounded distance d:

dpσ, σ 1 q ď Itσ ‰ σ 1 u ˆmax σ0,σ1PS K dpσ 0 , σ 1 q, (1.16)
for any σ, σ 1 P S K . Eq. (1.16) implies an extension of our guarantees for Ptp s X ‰ s X u to the risk Eq. (1.15) of the empirical minimizer. Incidentally, our analysis of LR provides the first generalization bounds for the OVO approach to multiclass classification, by considering the specific case k " 1 for the top-k classification guarantees that we provide.

In conclusion, we proposed the new yet natural label ranking (LR) problem, which consists in learning to predict an ordered list of most likely labels from multiclass classification data. While [START_REF] Korba | A structured prediction approach for label ranking[END_REF] and [START_REF] Brinker | A reduction of label ranking to multiclass classification[END_REF] give practical approaches to ranking median regression (RMR) with partial information, our theoretical guarantees are new. Our analysis fits nicely into the usual empirical risk minimization framework and exploits recent results on RMR. A byproduct of our analysis is the first generalization bounds for the OVO approach to multiclass classification.

Selection Bias Correction

In statistical learning problems, the distribution P 1 of the training data Z 1 1 , . . . , Z 1 n may differ from that of the testing data P . This setup constitutes a particular case of transfer learning [START_REF] Pan | A survey on transfer learning[END_REF][START_REF] Ben-David | A theory of learning from different domains[END_REF][START_REF] Storkey | When training and test sets are different: characterizing learning transfer[END_REF][START_REF] Redko | Advances in Domain Adaptation Theory[END_REF]. Notably in facial recognition problems, the training population is frequently not representative of the testing population, as underlined in [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF]. Auxiliary information in the form of high level characteristics is often available, such as the nationality associated to a portrait in facial recognition. Our work addresses learning with biased data from the lens of Empirical Risk Minimization (ERM). In that regard, we propose an approach based on importance sampling that deals with: 1) classification problems where class probabilities differ between the training and testing step, 2) situations where the data originates from stratified populations that are represented differently between training and testing, 3) PU learning, the problem of learning with a sample of positive and unlabeled data (du Plessis et al., 2014), 4) and learning with censored data [START_REF] Fleming | Counting processes and survival analysis[END_REF]. Our analysis is supported by strong empirical evidence for classification on the ImageNet database [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], for which we formed strata information from higher level concepts than the predicted classes.

Weighted Empirical Risk Minimization (WERM). The goal of learning algorithms is generally to find a parameter θ P Θ, that minimizes the expected risk Rpθq " E P r pθ, Zqs on the testing data, with : Θ ˆZ Ñ R `is a measurable loss function. To approximate Rpθq, we propose a weighted estimator r R w,n on the training data:

r R w,n pθq :" 1 n n ÿ i"1 w i ¨ pθ, Z 1 i q,
where w " pw 1 , . . . , w n q is a weight vector. When P 1 " P and w " p1, . . . , 1q, then r R w,n pθq is the usual empirical risk, as well as an unbiased estimator of the expected test risk Rpθq. When P 1 ‰ P and P is absolutely continuous with respect to P 1 , the importance sampling method (Wasserman, 2010, Section 25.3) introduces optimal weights w ˚:" pw 1 , . . . , w nq that satisfy w i :" ΦpZ i q for any i P t1, . . . , nu, where Φpzq :" pdP {dP 1 qpzq for any z P Z. Since the function Φ denotes the likelihood ratio between P 1 and P , r R w ˚,n is an unbiased estimator of R.

Generalization guarantees for WERM. We then propose usual generalization guarantees in Opn ´1{2 q that depend on the supremum norm Φ 8 of the likelihood ratio over the input space Z. Our guarantees show that generalization is better when the two distributions P 1 and P are similar. In the general case, the likelihood ratio Φ is unknown, which limits the applicability of the technique. Additionally, Φ is a function over the input space Z, which makes its estimation impractical. Our work presents situations for which the likelihood function Φ has a simple form, under the assumption that auxiliary information about the relationship between the distributions P 1 and P is available.

WERM for strata frequencies. In the context multiclass classification, i.e. when Z " pX, Y q P X ˆY with Y " t1, . . . , Ku and K P N is the number of classes, Φ has a simple form when the proportion p k " P Z"P tY " ku of each class k in the test dataset is known.

In that context, the optimal weights w ˚satisfy w i " p Yi {p 1 Yi for any i P t1, . . . , nu, where p 1 k " P P 1 tY " ku is the train set proportion of class k for any k P t1, . . . , Ku. Note that the p 1 k 's can be estimated from the training data. This reweighting strategy does not depends on a classification objective, but applies to any case with known: proportions of each strata for some stratification in the test distribution, and stratum associated to each sample in the training set.

WERM for Positive Unlabeled (PU) learning. Positive Unlabeled (PU) learning has received increasing attention in the statistical learning literature recently (du Plessis and Sugiyama, 2014;du Plessis et al., 2014[START_REF] Du Plessis | Convex formulation for learning from positive and unlabeled data[END_REF][START_REF] Kiryo | Positive-unlabeled learning with nonnegative risk estimator[END_REF][START_REF] Bekker | Beyond the selected completely at random assumption for learning from positive and unlabeled data[END_REF]. PU learning considers a binary classification problem, i.e. Z " pX, Y q P X ˆt´1, `1u, and learns a classification decision with a sample of positive instances and one of unlabeled instances. The unlabeled sample is a mixture of both negative and positive data with fixed proportions. Our work shows that, by reweighting the instances of those samples, we obtain an unbiased estimator of the risk R. We provide statistical guarantees for the minimizer of our estimated risk.

Experiments on the ImageNet dataset. Finally, we provide convincing numerical evidence of the effectiveness of weighted ERM on the ImageNet dataset, a database used to benchmark large-scale visual classification algorithms [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. ImageNet classes are based on the WordNet lexical database for english [START_REF] Fellbaum | WordNet: an electronic lexical database[END_REF]. As such, they can be regrouped in several higher-level concepts, which constituted the strata of our weighted classification experiment. For example, the class "flamingo" would belong to the stratum "bird". Using high-level information to reweight the training data greatly increases performance on the testing set, measured in terms of top-1 and top-5 classification accuracy.

Ensuring the representativeness of a database may help to obtain fair predictions, but there is growing interest in mechanisms that correct explicitly for inherent biases of training data.

Learning Fair Scoring Functions

Learning a classifier under fairness constraints has received a great deal of attention [START_REF] Dwork | Fairness through awareness[END_REF]Zafar et al., 2017a;[START_REF] Donini | Empirical risk minimization under fairness constraints[END_REF][START_REF] Barocas | Fairness and Machine Learning[END_REF][START_REF] Williamson | Fairness risk measures[END_REF]1.5. Reliable Machine Learning McNamara et al., 2019). However, proposed approaches for fairness in ranking either only correct the scores with a post-processing step [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF][START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF][START_REF] Zehlike | Fa*ir: A fair top-k ranking algorithm[END_REF][START_REF] Celis | Ranking with fairness constraints[END_REF], or tackle original notions of fairness, such as fairness in exposure for the sequential presentation of rankings (Singh andJoachims, 2018, 2019). Many fairness for ranking papers have proposed different constraints based on the Area Under the ROC Curve (AUC), a standard measure of performance in ranking. Our work first proposes: an unified framework for AUC-based fairness constraints, generalization guarantees for the minimization of a loss that incorporates any of those constraints, as well as a practical optimization procedure for that loss based on gradient descent. Then, we show the limitation of AUC-based fairness constraints, and propose stronger ROC-based constraints. Finally, we prove generalization guarantees and an extension of our gradient-based optimization procedure to learning with the new ROC-based constraints.

Framework for fair bipartite ranking. The standard fairness framework for binary classification considers a triplet of random variables pX, Y, Zq P X ˆt´1, 1u ˆt0, 1u, where X is the input random variable Y is the binary output random variable, and Z encodes the membership to a protected group. In bipartite ranking, one learns a score function s : X Ñ R and evaluates it with respect to where it projects the negatives Y " ´1 relatively to the positives Y " `1 on the real line. In the context of fairness, the influence of Z on the distribution of the scores is important. Hence, we introduce the following conditional distributions of a given score s for any z P t0, 1u:

H s ptq :" P tspXq ď t | Y " ´1u and H pzq s ptq :" P tspXq ď t | Y " ´1, Z " zu , G s ptq :" P tspXq ď t | Y " `1u
and G pzq s ptq :" P tspXq ď t | Y " `1, Z " zu . While the ROC curve serves to evaluate the ranking performance of a score function, it is also a general tool to assess the differences between two distributions functions h and g over R. In this context, the ROC curve is known as the probability-probability (PP) plot of h and g. The Area Under the ROC Curve (AUC) is a scalar summary of the ROC curve, that is omnipresent in the ranking literature. It generally serves to evaluate the performance of bipartite ranking algorithms (Clémençon et al., 2008). Formally, the ROC and AUC between the two c.d.f.'s h and g, writes:

ROC h,g : α P r0, 1s Þ Ñ 1 ´g ˝h´1 p1 ´αq and AUC h,g :"

ż 1 0 ROC h,g pαq dα.
Differences in the distributions of the positives (or negatives) between the protected groups leads to discrepancies in the error rates of those groups, as observed for facial recognition in [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF].

Unified criterion for AUC-based fairness. To correct discrepancies of error rates between protected groups, many authors -principally from the recommendation systems communityhave proposed fairness constraints based on the AUC [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF][START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF][START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF]. With Dpsq :"

pH p0q s , H p1q s , G p0q
s , G p1q s q J , those constraints write:

AUC α J Dpsq,β J Dpsq " AUC α 1J Dpsq,β 1J Dpsq , (1.17)
for different values of pα, β, α 1 , β 1 q P pPq 4 , where P " tv | v P R 4 `, 1 J v " 1u denotes the 4-simplex. Our work shows that: if the fairness constraint Eq. (1.17) is satisfied when the distribution X|Y " y, Z " z does not depend on z P t0, 1u for both y " ´1 and y " `1, then Eq. (1.17) writes as the linear combination Γ J Cpsq " 0 of five elementary fairness constraints Cpsq " pC 1 psq, . . . , C 5 psqq with Γ P R 5 . Our general definition for AUC-based fairness constraints englobes all proposed measures of fairness based on the AUC and can serve to to derive new ones. More importantly, it paves the way for considering flexible fair scoring approaches based on AUC constraints.

Learning under AUC-based constraints. We integrate the AUC-based fairness constraint as a penalization in the objective function, and maximize the objective: max sPS AUC Hs,Gs ´λ Γ J Cpsq ,

(1.18) on a family of scoring functions S. The parameter λ dictates the trade-off between ranking accuracy and fairness. We provide theoretical guarantees on the generalization error of the criterion in Eq. (1.18), which are proven with concentration inequalities on the deviations of U -statistics. We propose a gradient-based algorithm that simply optimizes a relaxed version of Eq. (1.18). Introduce the notation r ¨to denote the relaxation of an empirical estimation AUC with the logistic function σ : x Þ Ñ 1{p1 `e´x q, our relaxation of the loss with a specific AUC constraint writes as:

r L λ psq :" Ć AUC Hs,Gs ´λ ¨c ´Ć AUC H p0q s ,G p0q s ´Ć AUC H p1q s ,G p1q s ¯, (1.19)
where c P r´1, 1s is a parameter that changes during the learning process. The parameter c is modified every fixed n adapt gradient steps by a fixed value, depending on whether the difference in AUC's in the constraint of Eq. (1.19) is evaluated as positive or negative on a validation set.

Limitations of AUC-based constraints. The equality between two AUC's constrains the distributions involved. Specifically, from the mean value theorem, it imposes an equality point for the ROC curves involved. However, that point is unknown a priori. Many applications -and in particular biometrics -focus on the performance of the system for small false positive rates, i.e. specific regions of the ROC curve. Enforcing the equality of ROC curves in those regions implies that the classifiers obtained by thresholding the score are fair in a classification sense. To enforce ROC's to be equal in a specific region, we can focus on a few selected points, which is motivated by discrete approximation results. For these reasons, we introduce ROC-based constraints, which enforce the equality of two ROC curves at specific points. , their deviation to the diagonal writes:

∆ F,α psq :" ROC F p0q s ,F p1q s pαq ´α,
for F P tH, Gu. Instead of AUC-based fairness constraints, we enforce ∆ F,α psq " 0 with specific values α F " rα p1q F , . . . , α pm F q F s of α for any F P tH, Gu. For that matter, introduce a loss L Λ that incorporates those as constraints with strength λ F "

" λ p1q F , . . . , λ pm F q F ı
for F P tH, Gu, as:

L Λ psq :" AUC Hs,Gs ´mH ÿ k"1 λ pkq H ∆ H,α pkq H psq ´mG ÿ k"1 λ pkq G ∆ G,α pkq G psq ,
where Λ :" pα, λ H , λ G q. We extend our generalization guarantees to ROC-based fairness constraints with an uniform control of the ROC curves. To prove the result, we consider an of empirical processes indexed by the family of points α P r0, 1s. We also propose a strategy for optimization that is analogous to the one used for Eq. (1.19). Our strategy features threshold parameters, that are modified in the same way as c.

Experimental results. We provide strong empirical evidence of our approach. Fig. 1.2 gives an overview of the fairness versus accuracy trade-offs achieved with our methods on specific data. As more restrictive fairness constraints are introduced in our experiments, we observe simultaneously that: 1) the area under ROC Hs,Gsi.e. the AUC as a ranking accuracy measure -decreases, and 2) the conditional distributions of the score change coherently with our fairness constraints. Notice that, for high scores, the difference in the conditional distribution of the score between sensitive groups is much higher with AUC-based constraints than with ROC-based constraints.

In conclusion, we have proposed new approaches to tackle algorithmic fairness in ranking. First, we regrouped AUC-based constraints under a single general definition. Then, we proposed theoretical guarantees, and a practical method for learning with any of those constraints by gradient descent. Afterwards, we pointed at the limitations of AUC-based constraints, and proposed a ROC-based approach that corresponds better to operational settings. Finally, we extended our generalization guarantees and practical optimization method to the new and more flexible ROC-based constraint.

Perspectives

In conclusion, the thesis addresses important problems in biometrics from the point of view of statistical learning theory. Our work proposes original ideas for these problems, and supports pαq " α for any α P r0, 1{4s and any F P tH, Gu.

those ideas with theoretical results. These results can be interpreted as security guarantees, that hold under probabilistic assumptions. Our work is a much-needed answer to the rapidly increasing volume of experimental machine learning literature that biometrics researchers have to follow. Biometric identification, and facial recognition in particular, incarnate many machine learning topics simultaneously, such as pairwise learning, sample bias or ranking. For this reason, we considered stylized versions of those problems, as their simultaneous examination would obscure our discourse, and runs the risk of being disregarded as anecdotal by the machine learning community. The richness of the topics tackled by the thesis is a result of this imperative. Broadening this spectrum could be envisaged, for example by considering the extension of ranking the best criteria (Menon and Williamson, 2016, Section 9) to the similarity ranking problem presented above.

From a biometrics perspective, the most important perspective in this thesis is to realize the potential impact of the methods presented, by providing strong empirical evidence of their relevance in practical settings. Indeed, while the rapid adoption of machine learning techniques by private companies has boosted the growth of the field, it also directed most of the attention to papers that propose unequivocal solutions to specific industrial problems. A notorious example for face recognition is [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. In that context, the promotion of this work will require finding and presenting pedagogically large-scale experiments that address precisely practical use-cases, which is a promising direction for future work.

Finally, we could extend the different topics considered in the thesis. In the context of fairness for ranking, one of the limitations of our analysis comes from the absence of an analytical expression for the optimal score function under a fairness condition. However, it is provided in the case of fair regression by [START_REF] Chzhen | Fair Regression via Plug-in Estimator and Recalibration With Statistical Guarantees[END_REF] for example. In bipartite ranking, overcoming this hurdle would pave the way for an extension of the partition-based algorithms of Clémençon and Vayatis (2009) to learning under fairness constraints. Another possibility concerns the extension of the techniques presented here to the case of similarity ranking. Indeed, that extension gives a framework that matches operational considerations in biometrics very closely, and would be justified by the current interest in methods to explicitly correct biases for facial recognition. The experimental component of that work would be supported by the availability of well-suited face databases [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF]. Another possibility is to address the limitations of our work on weighted empirical risk minimization, by considering cases where the nature of the difference between the train and test set is not covered by our work. For example, in Sugiyama et al. (2007), the authors propose to estimate the likelihood ratio using a small sample of the test set as auxiliary information. Besides the above examples, more extensions of each topic in the thesis could be considered.

In conclusion, the richness of the issues that arise in biometrics is fertile grounds for new theory and new practices in machine learning. This richness spurred the creation of this thesis and can inspire future research.

Introduction

Statistical learning theory is described in [START_REF] Bousquet | Introduction to statistical learning theory[END_REF] (Chapter 1) as a mathematical framework for studying the problem of inference, that is of gaining knowledge about some modeled phenomenon from data. It can be considered as a mathematical tool that justifies the soundness of performing machine learning, i.e. of using data collected in some specific context, e.g. in the past, to make decisions in another context, e.g. in the future. For that to be sensible, one has to make assumptions as to how the data collected -called training data -relates to the testing data, and those assumptions are referred to as inductive bias. Those assumptions express naturally in a probabilistic language. A common probabilistic assumption is that all observations are independent and originate from the same distribution (i.i.d.). Since risk measures in machine learning are usually mathematical expectations of a loss function, the i.i.d. assumption implies that natural estimators of that true risk write as averages of the loss on the training data.

Statistical estimation has extensively studied the convergence of standard averages. While classic statistical literature focuses on their asymptotic properties as the sample size grows to infinity [START_REF] Van Der | Asymptotic Statistics[END_REF], the limitation of that analysis for finite samples has lead authors to quantify the random fluctuations of finite averages of i.i.d. variables with concentration inequalities [START_REF] Boucheron | Concentration Inequalities -A Nonasymptotic Theory of Independence[END_REF]. As a result, recent theoretical results in statistical machine learning originate from bounds on the deviation between a finite-sample estimate and a target value, that hold with some probability [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF].

In machine learning, authors have shown that the difference between the true risk of a minimizer of the empirical risk and that of a reference solution can also be bounded using concentration inequalities. In this thesis, we refer to those bounds as learning bounds/rates, or as generalization bounds/guarantees/rates, This chapter details the derivation of such bounds for binary classification, the flagship problem in machine learning.

Firstly, Section 2.2 details the context and mathematical framework necessary to introduce learning bounds. Secondly, Section 2.3 shows how to derive an uniform learning bound, i.e. bounds that do not depend on the distribution of the data. Those bounds hold for the most complicated distributions P possible, but tend to be quite loose for smooth distributions. Thirdly, Section 2.4 responds to this weakness by deriving tighter bounds under mild assumptions on the data. Finally, Section 2.5 discusses the extension of such results to the more complicated statistics involved in the thesis and indexes the uses of the tools and results of this chapter in the thesis.

A Probabilistic Setting for Binary Classification

Statistical learning considers data that belongs to a input space X , also called feature space, that is often a d-dimensional Euclidean space, i.e. X " R d . That information serves to gain knowledge about some value that belongs to an output space Y. To study the relationship between the collected features and the output value, those are modeled as a pair of dependent random variables r.v. pX, Y q P X ˆY that follow some unknown joint probability distribution P defined on a probability space pΩ, A, Pq. We refer to the two first chapters of [START_REF] Shorack | Probability for Statisticians[END_REF] for a thorough definition of these mathematical objects. The random variable Y is referred to as the output r.v., while X is referred to as the input r.v.. We introduce the distribution of X as F , called the marginal distribution of X (Papoulis, 1965, Chapter 6, page 171). Mathematically, statistical learning focuses seeks to recover some information about distribution of the output r.v. Y for any possible value of the input r.v. X, i.e. about the conditional random variable Y |X " x for any x P A, with A Ă X such that (s.t.) PtX P Au " 1. The probability distribution P can be decomposed as F ˆPY |X , with P Y |X the conditional distribution of Y given X. The conditional distribution P Y |X is referred to as the posterior probability of Y , and its existence is justified in [START_REF] Shorack | Probability for Statisticians[END_REF] (Chapter 8, Theorem 5.2).

Learning with binary output Y . Many learning problems assume that the output random variable Y is binary, i.e. Y " t´1, `1u, such as binary classification or bipartite ranking, that last problem being the subject of Chapter 3. Under that assumption, the posterior probability P Y |X is summarized by the mapping η with:

η : X Ñ r0, 1s, x Þ Ñ PtY " `1 | X " xu,
which is referred to as the regression function or as the Bayes score in the bipartite ranking literature. The distribution P is completely characterized by the pair pF, ηq. Another way to specify P is to define the proportion of positive instances p " PtY " `1u and the distribution of the conditional random variables X|Y " y, denoted as G for y " `1 and as H for y " ´1. Then, the two characterizations are related by the identities G " ηF {p, H " p1 ´ηqF {p1 ´pq and p " ş ηpxqdF pxq. They imply F " pG `p1 ´pqH and η{p1 ´ηq " pG{pp1 ´pqHq.

Example 2.1. To illustrate this section, we provide an example with X " r0, 1s. We introduce a posterior probability η, that depends on the parameter a P p0, 1q: @x P r0, 1s,

ηpxq " 1 2 `1 2 sgnp2x ´1q|1 ´2x| 1´a a .
See Fig. 2.1(a) for a representation of the characterizations of P for a " 1{2.

Binary classification. We now focus on the binary classification problem, and refer to Section 2.5 for a discussion on the extension of those results to other settings. The goal of binary classification As F, H, G are all absolute continuous measures, we represented the Radon-Nikodym derivative w.r.t. the Lebesgue measure of those distributions [START_REF] Shorack | Probability for Statisticians[END_REF], Section 2, Chapter 4). On the other hand, the two right-hand side figures represent the approximations of G and H that we obtain using histograms of n independent realizations of pX, Y q with n " 200 and n " 1000.

is to find a function g : X Ñ Y " t´1, `1u in a class of measurable candidate functions G that minimizes the misclassification error Rpgq, with:

Rpgq :" PtgpXq ‰ Y u.
The following proposition characterizes the best classifier in the set of all measurable functions.

Definition 2.2 (Bayes classifier).

The Bayes classifier is defined as the function g ˚: X Ñ Y s.t.: @x P X , g ˚pxq :" 2 ¨Itηpxq ą 1{2u ´1.

For any measurable g : X Ñ Y, we have Rpg ˚q ď Rpgq.

Furthermore, we have the following decomposition of the excess risk:

Rpgq ´Rpg ˚q " Er|2ηpXq ´1| ¨ItgpXq ‰ g ˚pX qus.

(2.1)

Proof. Note that:

Rpgq " ErErItgpXq ‰ Y u|Xss " ErηpXq `p1 ´2ηpXqqItgpXq " `1us.

Since: for any x P X and g : X Ñ Y, p1 ´2ηpxqq pItgpxq " `1u ´Itg ˚pxq " `1uq " |2ηpxq ´1| ¨Itgpxq ‰ g ˚pxqu,

we have that Rpgq ě Rpg ˚q for any measurable g.

The Bayes classifier makes classification mistakes unless the classes are separable. Indeed, note that Rpg ˚q " ErminpηpXq, 1´ηpXqqs, which means that g ˚has a positive risk unless ηpXq P t0, 1u a.s.. A sensible goal when selecting a candidate function in G is thus to find g P G with Rpgq as close as possible to Rpg ˚q, which is measured by the notion of excess risk, defined as:

Epgq :" Rpgq ´Rpg ˚q.

Remark 2.3. Then, the gap in performance between g : and g ˚is quantified by the approximation error, defined as:

Rpg : q ´Rpg ˚q ě 0.

Controlling the approximation error generally requires an assumption on the smoothness of the target function. To introduce this type of assumption, the regression function η can for example be assumed to belongs to a Sobolev space (Jost, 2011, Appendix A.1). While results exist in the case of regression [START_REF] Smale | Estimating the approximation error in learning theory[END_REF], bounding the approximation error in classification remains an open problem.

Generalization error. Since P is unknown, we cannot directly measure the risk R. " P , we define the empirical risk R n on the data D n as:

R n pgq :" 1 n n ÿ i"1 ItgpX i q ‰ Y i u. (2.2)
We denote by g n the minimizer of R n over G. The precision of the approximation of R by R nand as a result, the choice of g n -depends on the difference between the empirical distribution and the true distribution, illustrated in Fig. 2.1. A quantity of interest is the risk Rpg n q of the minimizer of the empirical risk, as it quantifies the performance of the classifier learned from data. Observe, that, since R n pg n q ď R n pg : q by definition of g n , we have:

Epg n q ď Rpg : q ´Rpg ˚q `Rpg n q ´Rn pg n q `Rn pg : q ´Rpg : q, ď Rpg : q ´Rpg ˚q `2 sup gPG |R n pgq ´Rpgq|.

(2.3)

Eq. ( 2.

3) shows that one can bound the excess of risk of the empirical minimizer by the approximation error plus the uniform deviation E n pGq :" sup gPG |R n pgq ´Rpgq| of R n pgq ´Rpgq over G, which quantifies the generalization error. The bound can be seen as a formalization of the bias-variance tradeoff presented in [START_REF] Domingos | A few useful things to know about machine learning[END_REF], where the bias term corresponds to the approximation error, while the variance corresponds to the generalization error. When we increase the size of the proposed family G, the approximation error decreases, while the generalization error increases.

PAC-bounds. PAC (probably approximately correct)-style bounds are bounds that are satisfied for any outcome ω P A δ , for some A δ Ă Ω with PtA δ u ě 1 ´δ. The learning bounds presented throughout the thesis consist in PAC bounds for similar types of uniform deviations over diverse families of functions. All of the bounds of the thesis concern the generalization error.

Uniform Learning Bounds

Most learning bounds are based on the Chernoff bound, which bounds the tail PtZ ą tu of a r.v. Z using its moment generating function λ Þ Ñ Ere λZ s, as shown below.

Proposition 2.4 (The Chernoff bound). [START_REF] Boucheron | Concentration Inequalities -A Nonasymptotic Theory of Independence[END_REF], section 2.1) For any real-valued r.v. Z and any t P R,

P tZ ą tu ď inf λą0 E " e λpZ´tq ı " inf λą0 ´E " e λpZ´ErZsq ı ¨eλpErZs´tq ¯. (2.4)
Proof. The proof is a simple combination of an increasing transformation with Markov's inequality. Let λ ą 0, then t Þ Ñ e λt is increasing and the simple bound Itx ą 1u ď x holds for any x P R, hence:

P tZ ą tu " P e λZ ą e λt ( " E " Ite λpZ´tq ą 1u ı ď E " e λpZ´tq ı , (2.5)
and minimizing the right-hand side of Eq. (2.5) gives the result.

By substituting Z with E n pGq in Eq. (2.4), we see that controlling the uniform deviation can relies on a combination of two separate results. Specifically, we rely on a result on ErZs, a term that measures the expressivity of the proposed class G, and on a term Z ´ErZs, a term that measures the robustness of the process of selecting g from data. For sensible families G, both of these quantities decrease to zero when n increases, but with different convergence speeds. Typically, the parameter λ is selected to balance the speeds of convergence of the two terms.

Basic Concentration Inequalities

Three distinct but related types of presentation exist for any result of Section 2.3.1. First, two equivalent formulations exist for presenting bounds on the tail PtZ ą tu of a r.v. Z. Precisely, consider a bound on the tail of a r.v. Z that writes PtZ ą tu ď f ptq, and define t δ as the solution in t of δ " f ptq with some δ ą 0. The bound PtZ ą tu ď f ptq is then equivalent to a simple inequality of the form Z ď t δ , that holds with probability (w.p.) larger than (ě) 1 ´δ. For example, Corollary 2.6 below is provided as such. Another type is as a bound on the moment generating function t Þ Ñ Ere tZ s of a r.v. Z, which relates directly to the tail of Z with Chernoff's bound (Proposition 2.4). The moment generating bound implies tail bounds, and all theorems can be stated in any of the three forms.

The most famous concentration inequality is Hoeffding's inequality [START_REF] Györfi | Principles of Nonparametric Learning[END_REF], Theorem 1.2), a PAC bound of the order Opn ´1{2 q on the deviation of the mean of bounded i.i. The independence between the Z i 's, see [START_REF] Shorack | Probability for Statisticians[END_REF] (Chapter 8, Theorem 1.1), followed by Lemma 2.5, implies that:

E " e λp s Zn´Er s Znsq ı " n ź i"1 E " e pλ{nqpZ1´ErZ1sq
ı " e λ 2 pb´aq 2 {p8nq .

Minimizing the r.h.s. of Eq. (2.6) in λ gives:

Pt s Z n ´Er s Z n s ą tu ď e ´2t 2 n{pb´aq 2 . (2.7)
Inverting that bound, i.e. solving in t such that δ is equal to the r.h.s. of Eq. (2.7) implies the result.

We can bound the tail of s Z n ´Er s Z n s using the union bound -also referred to as the subadditivity of measures or Boole's inequality [START_REF] Shorack | Probability for Statisticians[END_REF], Chapter 1, Proposition 1.2) -between Corollary 2.6 and its application to the family ´Z1 , . . . , ´Zn . It writes:

Pt s Z n ´Er s Z n s ą tu ď Pt s Z n ´Er s Z n s ą tu `PtEr s Z n s ´s Z n ą tu ď 2e ´2t 2 n{pb´aq 2 .
The Hoeffding inequality on its own implies guarantees for finite families of functions G, as presented in [START_REF] Bousquet | Introduction to statistical learning theory[END_REF] (section 3.4) and shown in Corollary 2.7 below:

Corollary 2.7. (Bousquet et al., 2003, section 3.4) Assume that G is of cardinality N . Then, we have that: for any δ ą 0 and n P N ˚, w.p. ě 1 ´δ,

Rpg n q ´Rpg : q ď d 2 n log ˆ2N δ ˙.
Proof. The result presented in Lemma 2.5 implies bounds on the expectation of the maximum of a family of random variables, as shown in Lemma 2.8 below. It is useful when considering minimizers of the empirical risk, to upper-bound the value of ErE n pGqs presented in Section 2.1.

Lemma 2.8. (Györfi, 2002, Lemma 1.3) Let σ ą 0, and assume that Z 1 , . . . , Z m are m P N ˚real-valued r.v.'s such that for any t ą 0 and 1 ď j ď m, Ere tZj s ď e t 2 σ 2 {2 , then:

E " max jďm Z j  ď σ a logp2mq.
We refer to [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] (Lemma 1.3) for the proof, as well as the remark that the tail of max jďm |Z j | is bounded by applying Lemma 2.8 to the family tZ 1 , ´Z1 , . . . , Z m , ´Zm u of 2m elements. Now that we have presented inequalities to deal with the term ErE n pGqs, we introduce one that controls the deviation of E n pGq from its mean, and refer to [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] (Lemma 1.4) for the proof.

Theorem 2.9 (McDiarmid's inequality). (Györfi, 2002, Lemma 1.4) Let Z 1 , . . . , Z n be a collection of n P N ˚i.i.d. random variables. Assume that some function g : ImpZ 1 q n Ñ R satisfies the bounded difference assumption:

sup z1,...,znPImpZ1q z 1 i PImpZ1q gpz 1 , . . . z n q ´gpz 1 , . . . , z 1 i , . . . , z n q ď c i , 1 ď i ď n,
then, for all t ą 0:

PtgpZ 1 , . . . , Z n q ´ErgpZ 1 , . . . , Z n qs ą tu ď exp

" ´2t 2 ř n i"1 c 2 i * .

Complexity of Classes of Functions

This section presents several ways to control the complexity of a class of proposed functions G, referred to as capacity measures in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF]. Those include Rademacher averages, the concept of shatter coefficient and VC dimension, as well as covering numbers. First, we introduce the notion of Rademacher average and relate it to the notion of VC dimension. For an introduction to covering numbers for statistical learning theory, we refer to [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (Section 5.1).

Definition 2.10 (Rademacher average).

Let G be a set of binary classifiers, i.e. @g P G, g : X Ñ t´1, `1u. The Rademacher average of G is defined as:

R n pGq :" E « sup gPG 1 n n ÿ i"1 σ i ItgpX i q ‰ Y i u | pX i , Y i q n i"1 ff ,
where σ " pσ i q n i"1 is a set of n i.i.d. Rademacher variables, i.e. Ptσ i " `1u " Ptσ i " ´1u " 1{2 for any i P t1, . . . , nu.

The Rademacher average measures the capacity of a family of functions to fit random noise, and bounds the expectation of E n pGq, as shown in Proposition 2.11 below.

Proposition 2.11. For any class of binary classifiers G and any n P N ˚, we have:

E " sup gPG |R n pgq ´Rpgq|  ď 2E rR n pGqs .
Proof. The proof is based on a trick called symmetrization, as presented for example in [START_REF] Bousquet | Introduction to statistical learning theory[END_REF] (Section 4.4). It requires the introduction of an unobserved sample

D 1 n " pX 1 i s, Y 1 i sq n i"1 i.i.d.
" P , referred to as ghost sample or virtual sample. We denote by R 1 n the estimate of the same form as Eq. (2.2), but estimated on D 1 n . Using the simple property sup Er¨s ď Ersupp¨qs, followed by the triangle inequality, gives:

E rE n pGqs " E " sup gPG R n pgq ´ErR 1 n pgqs G  ď E " sup gPG R n pgq ´R1 n pgq  , ď E « E « sup gPG 1 n n ÿ i"1 σ i `ItgpX i q ‰ Y i u ´ItgpX 1 i q ‰ Y 1 i u ˘ | pX i , Y i q n i"1 ffff , ď 2E rR n pGqs .
Remark 2.12. If X is a continuous random variable and G is the family of all measurable functions, we have: R n pGq ě 1{2 a.s. It follows from the observation that: we have almost surely.

R n pGq " 1 n ¨E « max ˜n ÿ i"1 σ i , n ´n ÿ i"1 σ i ¸ff ě 1 2 ,
The result implies that the bound in Proposition 2.11 is uninformative.

On the other hand, an estimation of the Rademacher average of G stump (see Remark 2.3) gives, by averaging over 200 random draws of the σ i 's, values of 0.13 for n " 100 and 0.04 for n " 1000.

It shows that, for simple families of functions, the Rademacher average decreases quickly as n grows.

The notion of Rademacher average is a standard analysis tool and is involved in many important results. Those include the derivation of sharp bounds for the supremum of empirical processes (Györfi, 2002, Section 1.4.6) or the derivation of data-dependent bounds (Boucheron et al., 2005, Theorem 3.2). However, it does not relate directly to simple properties of the proposed family.

Introduce the shatter coefficient S G pnq of a family of functions G as the maximal number of ways to split any collection of n elements in X ˆY with G. Formally:

S G pnq " sup tpxi,yiqu n i"1 PpX ˆYq n |tpItgpx 1 q ‰ y 1 u, . . . , Itgpx n q ‰ y n uq | g P Gu|.
(2.8)

Note that the shatter coefficient does not depend on the sample D n , and is bounded by 2 n . A set of n points is said to be shattered by a family G if its shatter coefficient is equal to 2 n , with the convention that the empty set is always shattered.

Lemma 2.8 relates the Rademacher average to the shatter coefficient, which in turn enables us to replace the supremum over G with a maximum over a set of S G pnq vectors, as detailed in Corollary 2.13 below.

Corollary 2.13 (Massart's Lemma). [START_REF] Györfi | Principles of Nonparametric Learning[END_REF], see the proof of Theorem 1.9) Let G be a class of measurable family of binary classifiers, i.e. for any g P G, g : X Ñ t´1, `1u.

For any n P N ˚, we have:

R n pGq ď c 2 logp2S G p2nqq n .
The combination of Proposition 2.11 and Corollary 2.13 is referred to as the Vapnik-Chervonenkis inequality (Györfi, 2002, Theorem 1.9, page 13). When the class G is finite of size N , then S G pnq ď N , which implies a bound on E " sup gPG |R n ´R| ‰ that decreases in Opn ´1{2 q. The shatter coefficient gives a simple bound on the Rademacher average by removing the expectation. However, it depends on n and does not have an intuitive form for simple families of functions. On the other hand, the VC-dimension is intuitive and summarizes the complexity of a class of functions by a single coefficient.

Definition 2.14 (VC-dimension). (Bousquet et al., 2003, Definition 2, page 189) The VC-dimension V of a proposed class G is defined as the maximum number of points that a class can shatter, i.e. as the largest n P N ˚such that:

S G pnq " 2 n .
Any class of functions with finite VC-dimension is called a VC-class of functions.

Remark 2.15. The VC-dimension of the set of decision stumps introduced in Remark 2.3 is 2, as one can always separate two distinct points, but not the set tp0.2, `1q, p0.3, ´1q, p0.4, `1qu Ă X ˆY. We refer to [START_REF] Mohri | Foundations of Machine Learning[END_REF] (Section 3.3,pages 41 to 45) for many examples of proposed families and a discussion on their VC-dimension.

The following lemma relates the VC-dimension to the shatter coefficient.

Lemma 2.16 (Vapnik-Chervonenkis-Sauer-Shelah lemma). (Bousquet et al., 2003, Lemma 1, page 190) Assume that G is a VC-class of functions with VC-dimension V . Then: for all n P N,

S G pnq ď V ÿ k"0 ˆn k ˙ď pn `1q V .
The last inequality of Lemma 2.16 is simply a consequence of the binomial theorem:

pn `1q V " V ÿ k"1 n k k! V ! pV ´kq! ě V ÿ k"1 n k k! ě V ÿ k"1 n! pn ´kq! 1 k! " n ÿ k"1 ˆn k ˙,
as detailed in Györfi (2002) (Theorem 1.13, page 19). Even though [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] also contains a proof of the first inequality, we refer here to the clearer proof of Shalev-Shwartz and Ben-David (2014) (page 74, lemma 6.10).

All of the results of Section 2.3.2 come into play to control the quantity E rE n pGqs introduced in Proposition 2.4. They are necessary to derive a first, basic learning bound, presented in the section below.

Uniform Generalization Bounds

By joining the results of Section 2.3.2 to control E rE n pGqs (Proposition 2.4) with Theorem 2.9 for the deviation of E n pGq to its mean, we can derive a simple uniform learning bound.

First, combining Proposition 2.11, Corollary 2.13 and Lemma 2.16 gives Proposition 2.17 below.

Proposition 2.17. Assume that G is a VC-class of functions with VC-dimension V , then, for any n P N ˚, we have:

E rE n pGqs ď c 8 logp2q `8V logp1 `2nq n .
Proposition 2.17 gives a bound on E rE n pGqs. We now deal with the other term in Proposition 2.4.

The following corollary is a simple application of Theorem 2.9 to the random variable .

Corollary 2.18. Let G be a class of binarybe classifiers. With Z " E n pGq " sup gPG |R n pgq ´Rpgq|, we have that, for any n P N

˚:

E " e λpZ´ErZsq ı ď e λ 2 n{8 .

Proof. Introduce R 1 n as the estimator where for a fixed i P t1, . . . , nu, pX i , Y i q was substituted by some value pX 1 i , Y 1 i q P X ˆY in the estimator of the risk R n , as well as E 1 n pGq :" sup gPG |R 1 n pgq ´Rpgq|. Simple triangle inequalities for the absolute value, combined with usual properties of the supremum imply:

E n pGq ´E1 n pGq ď sup gPG R n pgq ´R1 n pgq , ď 1 n sup gPG ItgpX i q ‰ Y i u ´ItgpX 1 i q ‰ Y 1 i u ď 1 n ,
which implies the result with Theorem 2.9.

Joining the results of Proposition 2.4 with those of Proposition 2.17 and Corollary 2.18 gives Proposition 2.19 below. Note the usual order in Opn ´1{2 q for statistical learning theory, as the term in logpnq is negligible in front of the other terms. A more advanced technique for controlling the complexity of G, called chaining, gives sharper bounds in Opn ´1{2 q without the logarithmic term logpnq. Here, we report that result without the proof, but refer to [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] (Theorem 1.16 and 1.17) for it.

Proposition 2.19. (Boucheron et al., 2005, Theorem 3.4) Let G be a class of binary classifiers. Assume that G has finite VC dimension V , then: for any δ ą 0 and n P N ˚, w.p. ě 1 ´δ:

Rpg n q ´Rpg : q ď c 2 log p1{δq n `2c 8 logp2q `8V logp1 `2nq n .
A more refined version of this inequality can be proven using a chaining argument, as explained in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (Theorem 3.4 therein) and proven in [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] (Section 1.4.6). It states that: for any δ ą 0 and n P N ˚, w.p ě 1 ´δ,

Rpg n q ´Rpg : q ď c 2 log p1{δq n `2C c V n ,
where C ą 0 is an universal constant.

Proof. It follows from Eq. (2.3), that:

PtRpg n q ´Rpg : q ą tu ď P t2 ¨En pGq ą tu .

Combining Proposition 2.17 and Corollary 2.18, then minimizing the bound in Proposition 2.4 in λ gives, with C V,n " 8 logp2q `8V logp1 `2nq:

P tE n pGq ą tu ď exp " ´2 ´?nt ´aC V,n ¯2* .
(2.9)

Inverting the bound gives the result.

While the bound holds for any possible probability distributions P , it is loose for smooth distributions. Indeed, in many cases, the uniform deviation of the empirical risk sup gPG |R n pgq ´Rpgq| over G is a very poor bound for Rpg n q ´Rpg : q, as mentioned in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (Section 5.2). The looseness of the bound of this section is illustrated in Remark 2.20.

Remark 2.20. For the distributions defined in Example 2.1, we illustrate in Fig. 2.2 the bound of Proposition 2.19 for the family of decision stumps in R as proposed family G.

Firstly, observe that the bound is very loose. Our distribution is not very complicated and our proposed family G stump is very favorable for the problem at hand, which may explain that fact.

Secondly, while the order of the slope is the same for any distribution P , the error rate seems to decrease faster when a is close to 1 than when a is close to 0. Precisely, the learning rate for easily separated distributions seems faster than Opn ´1{2 q. It is proven in Section 2.4.

Faster Learning Bounds

This section introduces the derivation of learning rates faster than Opn ´1{2 q under a noise assumption on the distribution of the data. Firstly, we introduce concentration inequalities that involve the variance of random variables. Secondly, we introduce noise conditions, and explain how they induce an upper-bound on the variance of the excess loss. Finally, we show that combining these two first results gives a fixed-point inequality, which implies fast learning rates.

The methodology presented here is limited to finite classes of functions G and to problems for which the Bayes classifier g ˚belongs to G. In this section, we also discuss the generalization of those results to general classes of functions of controlled complexity, as well as to those that do not contain the Bayes classifier.

We introduce the family F as the family of the excess losses of all elements in G. Formally, the 2.4. Faster Learning Bounds family of functions F satisfies that: for any f P F there exist some g P G, such that,

f pX, Y q :" ItgpXq ‰ Y u ´Itg ˚pX q ‰ Y u.
The bound of Proposition 2.19 relies on the uniform (for any g P G) boundedness of the random variable f pX, Y q. However, in many situations, as we choose a candidate g that approaches g

˚, the variance of f pX, Y q may be very small in front of the range of f pX, Y q. Hence, the next section presents concentrations inequalities that use the variance of random variables. They serve to derive tighter bounds based on hypotheses on the distribution of the data.

Sharper Concentration Inequalities

The derivation of fast learning bounds relies on concentration inequalities that involve the variance of random variables. The simplest one is Bernstein's inequality [START_REF] Boucheron | Concentration Inequalities -A Nonasymptotic Theory of Independence[END_REF] Popoviciu's inequality on variances gives an upper bound on the variance of a bounded random variable [START_REF] Egozcue | The variance upper bound for a mixed random variable[END_REF]. Precisely, it states that the variance σ 2 of a r.v. Z such that m ď Z ď M is smaller than pM ´mq 2 {4. Plugging this result into Theorem 2.21 gives a similar result as Corollary 2.6.

Bernstein's inequality does not give a guarantee on the supremum of a family of functions, but on an empirical mean, as does Hoeffding's inequality. A direct consequence of Hoeffding's inequality were guarantees on a finite family of functions, as shown in Corollary 2.7. Similarly, only the derivation of guarantees on a finite family of functions are a direct application of the combination of Bernstein's inequality and noise conditions on P .

Extending those results to more general classes of functions requires Talagrand's inequality, presented in Theorem 2.22 below. We refer to [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (Section 5.3, Theorem 5.4) for a simple presentation of the result, and to [START_REF] Boucheron | Concentration Inequalities -A Nonasymptotic Theory of Independence[END_REF] (Theorem 7.9, page 225) for the proof. We denote by P n the empirical distribution associated to D n . For any f P F, we have P f :" Erf pX, Y qs and P n f :" p1{nq ř n i"1 f pX i , Y i q Theorem 2.22 (Talagrand's inequality). Let b ą 0 and set F a family of functions X Ñ R. Assume that, for any f P F, P f ´f ď b. Then for all n P N ˚: w.p ě 1 ´δ,

sup f PF pP f ´Pn f q ď 2E « sup f PF pP f ´Pn f q ff `c 2psup f PF Varpf qq logp1{δq n `4b logp1{δq 3n .
The formulations of Bernstein's inequality and Talagrand's inequality are very similar. Their main difference is that the first deals with simple means, while the second handles the supremum of empirical processes. Note that the first term in the bound of Talagrand's inequality can be controlled in Section 2.3.2.

These inequalities will prove useful when combined with a clever upper bound of the variance of the excess loss, i.e. the variance of f pX, Y q. Conveniently, that quantity is upper-bounded by a function of the excess risk Rpgq ´Rpg ˚q, using the noise conditions introduced in the following section.

Noise Conditions

The simplest noise assumption to derive faster bounds is called Massart's noise condition (Boucheron et al., 2005, page 340). It assumes that there is always a clear separation between the two classes for the classification decision. Formally, it assumes that ηpXq is always far from 1{2.

Assumption 2.23 (Massart's noise assumption).

There exists h ą 0, s.t.

|2ηpXq ´1| ě h a.s.

A less restrictive assumption is called the Mammen-Tsybakov noise condition, see (Boucheron et al., 2005, page 340). It gives a bound on the proportion of X , as measured by the distribution of X, for which the classification decision is hard to take. It is parameterized by a parameter a P r0, 1s. If a distribution satisfies it for a close to 1, the two classes are easily separated. If it only satisfies the bound when a is close to 0, the two classes are hard to separate.

Assumption 2.24 (Mammen-Tsybakov's noise assumption).

There exists B ą 0 and a P r0, 1s, such that:

P t|2ηpXq ´1| ď tu ď Bt a 1´a .
Remark 2.25. The proof relies on Eq. (2.10) and an equivalent formulation of the Mammen-Tsybakov noise assumption. The equivalent formulation states that: there exists c ą 0 and a P r0, 1s, such that:

PtgpXq ‰ g ˚pX qu ď c

Distribution-dependent Generalization Bounds

The reasoning presented here can be found in [START_REF] Bousquet | Introduction to statistical learning theory[END_REF] Let pc 1 , c 2 , s, qq P R ˚4 `be real positive numbers. Then the equation x s ´c1 x q ´c2 " 0 has a unique positive zero x ˚. In addition, x ˚ď maxtp2c 1 q 1{ps´qq , p2c 2 q 1{s u.

Using Lemma 2.27, we can prove a fast learning bound for finite families of functions.

Proposition 2.28. Assume that G is a finite family of N functions and that the Mammen-Tsybakov noise assumption Eq. (2.11) is satisfied with constant parameter c and noise parameter a. For any δ ą 0, we have that, for any n P N ˚: w.p. ě 1 ´δ,

Rpg n q ´Rpg ˚q ď ˆ8c logpN {δq n ˙1{p2´aq `8c logpN {δq 3n .

Proof. A straightforward application of Lemma 2.27 to Eq. (2.13), followed by the bound maxpx, yq ď x `y for any x, y ą 0 implies:

Rpg n q ´Rpg ˚q ď max ˜ˆ8c logpN {δq n ˙1{p2´aq , 8c logpN {δq 3n ¸, ď ˆ8c logpN {δq n ˙1{p2´aq `8c logpN {δq 3n .
Proposition 2.28 gives a learning bound of the order Opn ´1{p2´aq q, under a noise condition parameterized by a P r0, 1s. When a is close to 0, we recover the order Opn ´1{2 q derived in Section 2.3.3. On the other hand, when a is close to zero, we approach the fast rate Opn ´1q. Observe in Fig. 2.3 that the learning rate of the fast bound seems much more aligned with the empirical learning rate than the slow learning rate of Section 2.3.

In real problems, the distribution of the data is unknown. Hence, the satisfaction of a noise assumption can not be verified. Still, fast learning speeds justify the looseness of the bounds derived in Section 2.3, constitute strong evidence of the soundness of learning from empirical data, and give possible tighter generalization bounds.

Connections to Present Work

The results in this thesis are modeled after those of this chapter. However, the majority of our work -specifically Chapter 5 and Chapter 7 of Part II and Chapter 10 of Part III -tackles the problems of scoring or similarity learning. Scoring concerns learning a score function s : X Ñ R, while similarity learning concerns learning a function s : X ˆX Ñ R. Hence, both bipartite ranking and similarity learning focus on real-valued functions. To derive generalization results for real-valued functions, tools that control the complexity of a real-valued family of functions S are required. In this section, we present notions that extend the results of the chapter to real-valued functions, and detail the implications of the chapter for our results in the thesis.

VC-major classes of functions. To quantify the complexity of classes of real-valued functions, the notion of VC-major class is used throughout the thesis. It builds on top of the notion of superlevel-set at some level t P R, defined as tx P X | spxq ą tu for some s : X Ñ R. Chapter 3

Selected Ranking Problems

Summary: This chapter is a short introduction to selected problems that deal with ranking data in machine learning. Its purpose is to provide the preliminaries to our theoretical results: on similarity ranking (Chapter 5) and on fair ranking (Chapter 10). Additionally, it puts in perspective the results involved in our analysis of label ranking (Chapter 8), and provides the intuitions for our propositions for practical similarity ranking (Chapter 7).

Ranking data in a broad sense is for example encountered in: bipartite ranking/scoring, ranking aggregation and learning-to-rank. Bipartite ranking considers a set of elements associated to a binary label, and seeks to associate a score to each instance, such that those with label `1 have higher score than those with label ´1. A typical application is credit scoring. Ranking aggregation is the problem of summarizing several individual rankings by a representative ranking. Finally, learning-to-rank associates an ordering over candidate elements to a query, and is the usual theoretical framework for studying search engines algorithms. In this chapter, the majority of the results focus on bipartite ranking. Those results are finite-sample bounds, and an extension of the analysis of Chapter 2. Precisely, we present at length the ROC criterion, pointwise ROC optimization and the theoretical guarantees for the TreeRank algorithm. Then, we briefly present the ranking aggregation problem, and precise its relationship to probabilistic models for ranking. Finally, we detail the implications throughout the thesis of the results presented here. We refer to [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF] for a more detailed overview of bipartite ranking.

Introduction

Rankings are a very natural way to provide information about a set of elements, which explains the ubiquity of machine learning problems related to finding ordered sets of those elements. The thesis relates to three different settings connected to that idea, with some papers making interesting connections between them. Those settings are: bipartite ranking or scoring, ranking aggregation and learning-to-rank.

The bipartite ranking/scoring problem considers a set of elements associated to a binary label, and seeks to rank those with label `1 higher than those with label ´1. This is generally performed by learning a mapping from the input space to the real line, called a score function. For example, this setting applies to any situation with a fixed budget to assign to the observations that are the most likely to be positive, e.g. in credit scoring. On the other hand, the binary classification problem described in Chapter 2 recovers a set of relevant instances, but does not compare positive instances among one another. We refer to [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF] for an overview of bipartite ranking.

Ranking aggregation seeks a consensus between several observed orderings of items. It relies on finding an ordering that summarizes best all observed orderings. A flagship problem of ranking aggregation is social choice theory, a theoretical framework for combining individual opinions. We refer to Korba (2018) (Part 1) for an introduction to ranking aggregation.

Finally, learning-to-rank refers to learning models that return an ordered list of instances in a database, that are relevant for a specific query. It can be performed by learning a notion of similarity between queries and candidate instances, and returning a list of items ordered by decreasing similarity with the query. The design of search engines is an example of direct application of this setting. We refer to [START_REF] Liu | Learning to Rank for Information Retrieval[END_REF] for an introduction to learning-to-rank.

In this section, we will focus first on bipartite ranking (Section 3.2) and then briefly introduce ranking aggregation (Section 3.3). Our introduction to bipartite ranking introduces the ROC curve, presents the problem of pointwise ROC optimization, also known as Neyman-Pearson classification, and finally recalls the theoretical guarantees for the TreeRank algorithm for bipartite ranking. The ranking aggregation section swiftly presents the problem and focuses on its relation to probabilistic models for ranking. Section 3.4 details the implications of the results of this chapter on the original work of the thesis.

Bipartite Ranking

Introduction

Bipartite ranking relies on the same probabilistic framework as binary classification, presented in Chapter 2. Specifically, one considers a random pair pX, Y q " P in X ˆY where X is the input space, and Y " t´1, `1u is the output space. Generally the input space X is a d-dimensional Euclidean space R d . As stated in Section 2.1, the distribution of the pair pX, Y q is completely summarized by the pair pF, ηq, where F is the marginal distribution of X and η is the posterior probability ηpxq :" PtY " 1 | X " xu, as well as by the triplet pp, H, Gq, where p is the proportion of positive instances p " PtY " `1u, H is the distribution of X|Y " ´1 and G that of X|Y " `1. An usual approach to bipartite ranking is to project items on the real line with a score function s : X Ñ R, and to return the ordering of those items by decreasing score.

To quantify the capacity of the score s to split positive from negative instances, we introduce the distribution of the random variables spXq|Y " ´1 and spXq|Y " `1 as, respectively:

H s ptq :" HpspXq ď tq " PtspXq ď t | Y " ´1u, G s ptq :" GpspXq ď tq " PtspXq ď t | Y " `1u.
With s F " 1 ´F the survival function associated to a distribution F , the quantities Ď H s ptq and Ď G s ptq are known respectively as the false positive rate (FPR) and the true positive rate (TPR) at threshold t. Considering that an instance x is classified as positive when spxq ą t, they correspond respectively to the proportion of negative instances wrongly classified as positive and the proportion of positive instances rightly classified as positive.

The goal of bipartite ranking is to order items by a notion of quality, quantified by the probability that an item x P X is positive, which is simply the posterior probability ηpxq. While one could consider solving the bipartite ranking problem by constructing an approximation of η from data, bipartite ranking is only interested in recovering the order induced by the posterior probability, not in the values of the posterior probability. A widespread criterion to measure the accuracy of the order induced by a score function is the Receiver Operating Characteristic curve (ROC) ( Definition 4 in Clémençon and Vayatis (2009)) a functional criterion defined as the parametric curve:

R Ñ r0, 1s 2 , t Þ Ñ `s H s ptq, s G s ptq ˘. (3.1)
The above definition implies that the ROC of an increasing transform T ˝s of the score s is the same as that of the score s. Since T ˝s induces the same order as s, the ROC only evaluates the induced order on supppF q of the score s, and not its value.

We formalize the order over a random sample D n " tX i u n i"1 i.i.d.

" P with a permutation σ P S n , as the order induced by s on D n , with the highest score on top and ties broken arbitrarily. Specifically, the permutation σ maps each i P t1, . . . , nu to the position of spX i q in the list of decreasing spX j q's, which writes:

spX σ ´1p1q q ě ¨¨¨ě spX σ ´1pnq q,
The permutation σ ´1 is the inverse of σ, hence σ ´1piq is the element in rank i for the ranking σ. Rather than a direct evaluation of the score s, the empirical version of the ROC is an evaluation of the permutation σ.

Consider an independent copy pX 1 , Y 1 q of the random pair pX, Y q. If the score perfectly separates the positives and the negatives, i.e. spXq ą spX 1 q|Y " `1, Y 1 " ´1 a.s., then the ROC curve is included in the plot of a step function, defined as tpt 1 , t 2 q | t 1 " 0 or t 2 " 1u. If the score does not separates the positives from the negatives at all, i.e. spXq|Y " ´1 and spXq|Y " `1 have the same distribution, then the ROC curve is included in the plot of the identity function, defined as tpt, tq | t P r0, 1su.

As defined in Eq. (3.1), the ROC curve is not continuous in general. One approach to solve that limitation is to consider the ROC curve as the evaluation of the score s, where the equality spxq " spx 1 q between the score of two elements induces a random order between those two elements. That extension connects two consecutive disconnected dots of the ROC curve with a straight line.

Remark 3.1. With the pF, ηq defined in Example 2.1 from Chapter 2. Introduce the noise coefficient parameter a P r0, 1s. Then, with s : x Þ Ñ x, we have s H s ptq " 2p1 ´tq ´s G s ptq, and:

s G s ptq " 1 ´t `a 2 " 1 ´|1 ´2t| 1 a ı .
A graphical representation of the ROC curves is featured in Fig. 3.1.

Gs(t)

ROCH s, Gs (f) a " 3{4 Figure 3.1: Illustration of the ROC curve of the score function s : x P r0, 1s Þ Ñ x for three distributions with three different levels of separability of the distributions spXq|Y " ´1 and spXq|Y " `1. Here, we denote by H (resp. G) the distribution spXq|Y " ´1 (resp. spXq|Y " `1). Here, the distributions F, H, G are all absolutely continuous, and are thus represented by their Radon-Nykodim derivatives w.r.t. the Lebesgue measure.

Another parametrization of the ROC curve is as the plot of the function:

ROC Hs,Gs : r0, 1s Ñ r0, 1s, α Þ Ñ s G s ˝s H ´1 s pαq,
where the generalized inverse of a decreasing function f writes f

´1 : t Þ Ñ suptλ | f pλq ą tu.
Introducing the definition of the generalized inverse of an increasing function f as f ´1ptq " inf tλ | f pλq ě tu, we have:

s H ´1 s pαq " sup tλ | 1 ´α ą H s pλqu " inf tλ | H s pλq ě 1 ´αu " H ´1 s p1 ´αq.
The quantity s H ´1 s pαq is a p1 ´αq-quantile of the random variable spXq|Y " ´1, since a property of the quantile function, specifically Lemma 21.1 of [START_REF] Van Der | Asymptotic Statistics[END_REF], implies:

PtspXq ď s H ´1 s pαq | Y " ´1u " PtspXq ď H ´1 s p1 ´αq | Y " ´1u " H s ˝H´1 s p1 ´αq ě 1 ´α.
Hence, we introduce the notation QpZ, αq to denote the quantile of order 1 ´α of any random variable Z conditioned on the event Y " ´1, and we have QpspXq, αq " s H ´1 s pαq. A score function s 1 is considered better than another score function s 2 whenever its ROC is higher, i.e. when the ROC of s 1 uniformly dominates that of s 2 . Introducing the notation ROCps, αq :" ROC Hs,Gs pαq for any score function s and any α P r0, 1s, it is verified when ROCps 1 , αq ě ROCps 2 , αq for any α P r0, 1s. Proposition 3.3 in Section 3.2.2 below proves the existence of optimal score functions for uniform dominance of ROC's, and shows that η is an optimal score function. Notice that the optimal element for the ranking problem involves the posterior probability η, as in the binary classification setting described in Section 2.1. Introducing the notations H ˚:" H η and G ˚:" G η , we write the ROC curve of η as ROC ˚:" ROC Hη,Gη , and similarly the p1 ´αq-quantile of ηpXq|Y " ´1 as Q ˚pαq :" QpηpXq, αq " s H ´1 η pαq. The invariance property under strictly increasing transforms of the ROC implies that any score s P S ˚, with S ˚" tT ˝η | T : r0, 1s Ñ R is increasing u, satisfies ROC Hs,Gs " ROC ˚.

The order defined by the uniform dominance between two ROC curves is not total, since there exists pairs of score functions that can not be compared. Indeed, the ROC a score function s 1 can strictly dominate that of another score function s 2 for all values α of a subset A of r0, 1s and the ROC of s 2 strictly dominate that of s 1 on another subset of A. Additionally, the estimation of the ROC curve from empirical data gives a random function. Results on the estimation of the ROC curve often involve the theory of empirical processes, as do for example the strong convergence and strong approximations theorems for the ROC curve of [START_REF] Hsieh | Nonparametric and semiparametric estimation of the receiver operating characteristic curve[END_REF]. The work of Bertail et al. (2008) draws on that analysis to derive a bootstrap procedure to derive tight confidence bands for the ROC curve.

Due to these two reasons, i.e. the absence of total order on the ROC curve and its functional nature, practitioners often consider summaries of the ROC curve instead of the full functional criterion. The most popular one is the Area under the ROC Curve (AUC), defined below. Definition 3.2 (Area under the ROC curve (AUC)). (Clémençon et al., 2008, Proposition B.2). The Area under the ROC Curve (AUC) of a score function s writes: AUC Hs,Gs :"

ż 1 0 ROC Hs,Gs pαq dα.
The AUC is equal to the proportion of correctly ranked pairs of independent elements. Formally, with pX, Y q and pX 1 , Y 1 q i.i.d. copies of the distribution P , we have:

AUC Hs,Gs " PtspXq ě spX 1 q | Y " `1, Y 1 " ´1u.
We write AUCpsq " AUC Hs,Gs for any score s, and AUC ˚:" AUC Hη,Gη .

The AUC encourages viewing the ranking problem as a binary classification problem over X ˆX on the pairs pX, X 1 q with different label, i.e. that satisfies Y ‰ Y 1 . In that framework, the binary objective is pY ´Y 1 q{2q. The ubiquity of the AUC implies that it is often considered as the standard performance measure for the bipartite ranking problem. Most of the papers about bipartite ranking focus on maximizing that quantity. A theoretical analysis of the AUC optimization procedure is provided in Clémençon et al. (2008). Many algorithm deal with the bipartite ranking problem by optimizing the AUC, such as RankSVM [START_REF] Joachims | Optimizing search engines using clickthrough data[END_REF], RankBoost [START_REF] Freund | An efficient boosting algorithm for combining preferences[END_REF] and RankNet [START_REF] Burges | Learning to rank using gradient descent[END_REF].

The estimation of the AUC involves a special type of statistics, different in nature from the standard averages studied in Chapter 2. Specifically, studying the empirical AUC uses estimators called U -statistics and presented in Chapter 4, which in their simplest form are averages on all possible pairs of elements from a random sample. Essential results for those statistics are provided in Chapter 4. Formally, introduce a sequence of n i.i.d. pairs with the same distribution as pX, Y q, i.e. D n " tpX i , Y i qu

n i"1 i.i.d.
" P , a natural estimator of the AUC writes:

z AUC Hs,Gs :" 1 n `n´ÿ Yi"`1 ÿ Yj "´1 ItspX i q ą spX j qu, (3.2)
with n `:" ř n i"1 ItY i " `1u and n ´:" n ´n`. While the AUC is a simple summary of the ROC curve, a drawback of the AUC in practical problems is that it considers all inversions to be equivalent. For example, an inversion that puts only one negative instance above the next positive instance that is higher on the list, will contribute as much to decreasing the AUC if the positive instance is at the top or if in the middle of the list, see Eq. (3.2). However, for many practical problems, having an accurate ranking is way more important at the beginning of the list than at the bottom. Therefore, a corpus of literature addresses the problem of ranking the best instances, which introduces many criteria that consider errors at the top as more critical than errors at the bottom.

Those criteria include popular accuracy measures from the information retrieval community, such as the Discounted Gain Criterion (DCG) or the Mean Reciprocal Rank (MRR) [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF], section 9.2), as well as propositions from the statistical machine learning community, such as the the p-norm push [START_REF] Rudin | Ranking with a p-norm push[END_REF]. In [START_REF] Clémençon | Ranking the best instances[END_REF], ranking the best is seen through the lens of a combination of two problems: finding the best instances and ordering that subset of instances well.

In this section, we first discuss the task of finding the optimal score function for maximizing the true positive rate under an upper-bound α P r0, 1s on the false positive rate. We refer to this task as pointwise ROC optimization pROC at level α. Then, we present theoretical guarantees for pROC, that were proven in Theorem 10 and Theorem 12 of Clémençon and Vayatis (2010). Those guarantees are modeled after those presented in Chapter 2. Finally, we recall a few results on the algorithm TreeRank, a flexible approach for solving the ranking problem by partitioning the input space. We refer to Clémençon and Vayatis (2009) for an extensive analysis of TreeRank.

Pointwise ROC Optimization (pROC)

Given a proposed family of score functions S, we define pointwise ROC optimization at level α P r0, 1s as the task of finding the score s P S and threshold t P R, such that the test spxq ą t has false positive rate constrained by α, and the highest true positive rate s G s ptq as possible. (3.4)

A solution of Eq. (3.4) is R α :" tx P X |s α pxq ą t α u.

Eq. (3.3) is also very similar to minimum volume set (MV-set) estimation, an approach in anomaly detection. MV-set estimation consists in finding a set of minimum volume, as measured by the Lebesgue measure on the Euclidean space X , in which a random variable X P X falls with probability greater than 1 ´α. The Lebesgue measure λ is a generalization of the notion of volume to general Euclidean spaces, defined formally in [START_REF] Shorack | Probability for Statisticians[END_REF] (Chapter 5, Example 1.1). Formally, with F the distribution of the r.v. X and for the proposed family of sets A Ă PpX q, the minimum volume set problem writes:

min APA λpAq such that F pAq ě 1 ´α. (3.5)
The main difference between Eq. (3.3) and Eq. (3.5) is that the objective λpAq of Eq. (3.5) does not have to be replaced by an empirical quantity. Indeed, the Lesbegue measure λpAq is known, unlike the constraint F pAq or the quantities GpRq and HpRq in Eq. (3.4), which all have to be estimated from data. We refer to [START_REF] Polonik | Measuring Mass Concentrations and Estimating Density COntour Clusters -An Excess Mass Approach[END_REF][START_REF] Polonik | Minimum volume sets and generalized quantile processes[END_REF] and [START_REF] Scott | Learning minimum volume sets[END_REF] for more details on the minimum volume set problem. The similarities between Eq. (3.4) and Eq. (3.5) have led authors to consider the mass volume curve, the counterpart of the ROC curve in the context of anomaly detection. We refer to [START_REF] Clémençon | Scoring anomalies: a m-estimation formulation[END_REF] and [START_REF] Clémençon | Mass volume curves and anomaly ranking[END_REF] for more details on the mass volume curve.

The problem presented in Eq. (3.3) can be considered in the standard hypothesis testing framework, presented in Wasserman (2010) (Section 11). In that light, consider the hypothesis H 0 : Y " ´1 against the alternative H 1 : Y " `1. The true positive rate s G s ptq then corresponds to the power of the test spXq ď t, while the false positive rate s H s ptq corresponds to its type I error. Solving Eq. (3.3) for the proposed class of all measurable functions corresponds to finding the most powerful test for rejecting H 0 at the level of significance α, i.e. the test with the highest power and a type I error below or equal to α. An application of the fundamental lemma of Neyman-Pearson [START_REF] Lehmann | Testing Statistical Hypotheses[END_REF], Theorem 3.2.1) implies the expression of an optimal solution of Eq. (3.3) in the space of all measurable functions. We present that solution in the proposition below, along an expression of a notion of pointwise excess risk at level α for the ROC curve.

Proposition 3.3 (Optimal score function). (Clémençon and Vayatis, 2009, Proposition 6) (Clémençon and Vayatis, 2010, Proposition 5). For any measurable scoring function s, we have: Itspxq ą tu dt.

@α P p0,
(3.8) Clémençon and Vayatis (2010) proposed to find a good solution of the ranking problem by combining the solutions of several pointwise ROC optimization problems, as those are super-level sets of the optimal score function η. Eq. (3.8) suggests a simple combination strategy.

The optimal solution for binary classification, i.e. the Bayes classifier g ˚: x Þ Ñ 2¨Itηpxq ą 1{2u´1, consists in thresholding the Bayes score η by a threshold t " 1{2 to separate positive and negative instances. Similarly, the proof of Proposition 3.3 implies that the optimal solution of the pointwise ROC optimization problem at level α Eq. (3.3) also consists in thresholding the Bayes score function, as the optimal rejection region of the corresponding test is R α. In that case, the value of the threshold is unknown, as t α " Q ˚pαq depends on the distribution of the random variable ηpXq | Y " ´1.

Introduce the weighted classification cost for binary classification, as:

R c pgq :" c ¨PtgpXq " ´1, Y " `1u `p1 ´cq ¨PtgpXq " `1, Y " ´1u.

(3.9)

Eq. (3.9) can be seen as a weighted sum of the false negative rate G s ptq and false positive rate s H s ptq for the classifier defined by g : x Þ Ñ spxq ą t. The proposition below shows that g c : x Þ Ñ 2 ¨I tηpxq ą 1 ´cu ´1 is a minimizer of R c pgq in the class of all measurable functions. It is proven in a similar way as the optimality of the Bayes classifier for binary classification, presented in Chapter 2. Its proof is thus omitted.

Proposition 3.4 (Optimal classifier for weighted classification). For any c P r0, 1s, we have that R c pgq ě R c pg c q for any measurable classification function g : X Ñ t´1, `1u.

The optimal solution of weighted classification also writes as a thresholding operation on the Bayes score, with a threshold t c :" 1 ´c. The optimal solution of pointwise ROC optimization Eq. ( 3.3) at level α is then the same as the optimal solution of Eq. (3.9) for an unknown cost asymmetry parameter c " 1 ´Q˚p αq. Hence, a common strategy for optimal ranking is to combine many solutions of the problem of classification with asymmetric costs, as done for example in [START_REF] Bach | Considering cost asymmetry in learning classifiers[END_REF].

Generalization Guarantees for pROC

In this section, we derive guarantees for empirical solutions of the pointwise ROC optimization problem. We first derive an uniform learning bound, followed by a fast learning bound under a noise assumption on the data distribution. The uniform learning bound is found in [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF] (Proposition 1) for Neyman-Pearson classification and is a direct consequence of Theorem 10 of Clémençon and Vayatis (2010), while the fast learning bound is Theorem 12 in Clémençon and Vayatis (2010) HpRq " HpRq `p p HpRq ´HpRqq implies that very often, very good solutions of Eq. (3.4) will not satisfy the constraint of Eq. (3.10) if φpn, δq " 0, depending on the generated sample D n . [START_REF] Rigollet | Neyman-pearson classification, convexity and stochastic constraints[END_REF] take a conservative approach and investigate Eq. (3.10) with a negative tolerance term φpn, δq, so that it enforces an empirical solution p R α that satisfies the constraint Hp p R α q ď α.

Under a complexity assumption on the proposed family S, Clémençon and Vayatis (2010) prove learning bounds for the pointwise ROC optimization problem, as presented in Proposition 3.5 below. In Proposition 3.5, the complexity assumption is that S is a VC-major class of functions, which means that the super-level sets R of the score functions in S form a VC-class of sets, see Definition 2.14 and Definition 2.30 in Chapter 2.

Proposition 3.5. (Consequence of Theorem 10 in Clémençon and Vayatis (2010)).

See Proposition 1 of [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF] for the same result in Neyman-Pearson classification.) Let α P p0, 1q. Assume that:

• R is a VC-class of functions with VC-dimension V ,

• " minpp, 1 ´pq ą 0,

• H η is continuous at Q ˚pαq and R α P R.

For all pδ, nq P p0, 1q ˆN˚, set:

φpn, δq " 1 ˜c 2 logp2{δq n `c 8 logp2q `8V logp1 `2nq n ¸.
Then, for all pδ, nq P p0, 1q ˆN˚, we simultaneously have w.p. ě 1 ´δ:

Hp p R α q ď α `2φpn, δ{2q and Gp p R α q ě GpR αq ´2φpn, δ{2q.

(3.11)

The proof of Proposition 3.5 is slightly different than that of Clémençon and Vayatis (2010) (Theorem 10), as the quantities p H and p G satisfy:

for any R P R, p HpRq "

ř n i"1 ItY i " ´1, X i P Ru ř n i"1 ItY i " ´1u
and p GpRq "

ř n i"1 ItY i " `1, X i P Ru ř n i"1 ItY i " ´1u
, thus are ratios of averages, while Clémençon and Vayatis (2010) (Theorem 10) assumed n ´and n `to be deterministic. However, the core argument remains the same. It relies on the fact that PAC-style bounds for the supremum of empirical processes p GpRq ´GpRq and p HpRq ´HpRq over R imply the bilateral control of errors presented in Eq. (3.11). Proposition 3.5 recovers the standard learning bound in Opn ´1{2 q presented in Section 2.3 for binary classification, for both the constraint and the objective, Using the same tools as in Section 2.4, Clémençon and Vayatis (2010) (Section 5.2) prove faster speeds of convergence for the pointwise ROC optimization problem under a noise assumption on the data given below. The noise assumption is very similar to the Mammen-Tsybakov assumption for binary classification, but concerns the distribution of ηpxq around Q ˚pαq instead of 1{2, see Assumption 2.24.

Assumption 3.6. (Clémençon and Vayatis, 2010, noise assumption) Let α P p0, 1q. There exists B ą 0 and a P r0, 1s, s.t.:

P t|ηpXq ´Q˚p αq| ď tu ď B ¨t a 1´a .
Clémençon and Vayatis (2010) (Remark 14) explains that Assumption 3.6 is satisfied with a " 1{2 and B " 2 ¨sup t f ˚ptq as soon as ηpXq has a bounded density f ˚. Indeed, with F ˚the c.d.f. of ηpXq, the finite increments theorem gives:

P t|ηpXq ´Q˚p αq| ď tu " F ˚pQ ˚pαq `tq ´F ˚pQ ˚pαq ´tq ď Bt.

As in Section 2.4.2, the noise assumption (Assumption 3.6) implies a control on the variance of a notion of excess loss, presented in the lemma below. The implication is proven in Clémençon and Vayatis (2010), in a similar manner as Proposition 2.26 in Chapter 2, using equivalent formulations of the Mammen-Tsybakov assumption that can be found in [START_REF] Bousquet | Introduction to statistical learning theory[END_REF] (Definition 7).

Lemma 3.7. (Clémençon and Vayatis, 2010, Lemma 11) Suppose that H η is continuous at Q ˚pαq and that Assumption 3.6 is fulfilled. Set for all R P R, s 2 α pRq :" Var rItY " `1u pItX P R αu ´ItX P Ruqs .

Then we have: @R P R, s 2 α pRq ď c rpp1 ´Q˚p αqqpGpR αq ´GpRqq `Q˚p αqp1 ´pqpHpRq ´αqs a .

Similarly to Section 2.4.3, the control on the variance can be used to derive fast bounds on a finite family of functions with Bernstein's inequality, as proven in Clémençon and Vayatis (2010).

Proposition 3.8. (Clémençon and Vayatis, 2010, Theorem 12) Assume that the assumptions of Proposition 3.5 are satisfied and that Assumption 3.6 is fulfilled for some α P p0, 1q with noise parameter a P r0, 1s. Then, for all δ ą 0, there exists C :" CpR, δ, α, p, Q ˚pαqq and n 0 :" n 0 pR, δ, α, p, Q ˚pαqq, such that @n ě n 0 , we simultaneously have w.p ě 1 ´δ:

Hp p R α q ď α `2φpn, δ{2q and ROC ˚pαq ´Gp p R α q ď Cn ´2`a 4 .

Proposition 3.8 above proves bounds of the order Opn ´p2`aq{4 q for the true positive rate under bounds in Opn ´1{2 q for the false positive rate. It is slower than the usual fast learning rate proven in Section 2.4, due to the bilateral control of the two types of errors.

TreeRank

TreeRank is a recursive algorithm that gives a solution of the scoring problem based on the adaptive approximation of the ROC curve of the optimal score function. Like other decision tree-based algorithms, it recursively partitions the input space into cells. However, the output of TreeRank is a piecewise constant scoring function, and the inverse image of its values corresponds to leaves of the associated tree. In its simplest form, the algorithm generates a full binary tree of depth D, which gives a partition of the space X in 2 D cells C D,k , indexed by k P t0, . . . , 2 D ´1u. TreeRank is a greedy algorithm, since the splitting procedure is designed to optimize the AUC at each step.

The algorithm is described in Clémençon and Vayatis (2009) and [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF]. We refer the reader to [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF] (Section 4) for approaches to prune the full ranking tree, and to Clémençon et al. (2013) for a description of an ensemble method for aggregating TreeRank-based score functions. The aggregate is named Ranking Forest, as a reference of the notion of Random Forest introduced in [START_REF] Breiman | Random forests[END_REF]. The aggregation of the random trees is based on the idea of aggregating the order induced by each of the trees, see Section 3.3 below for a short introduction to ranking aggregation, but in practice often consists of simply averaging the scores. In this section, we present theoretical results for the TreeRank procedure that can be found in Clémençon and Vayatis (2009), as the thesis extends those results to a variant of TreeRank adapted to the problem of similarity learning.

In the case of a full binary tree T of depth D, the score function writes as:

s T pxq :" 2 D ´1 ÿ k"0 p2 D ´kq ¨Itx P C D,k u,
and the score s T induces a total order on the cells tC D,k u 2 D ´1 k"0 . At the beginning of the formation of the ranking tree, a cell C 0,0 contains all of the input space, hence all instances of X have the same score. To form a tree of depth D " 1, C 0,0 is split in two cells C 1,0 andC 1,1 . In general, we introduce the notation corresponding to the cell C d,k of each node, such that:

@d, k P t1, . . . , Du ˆt0, . . . , 2 d ´1u with k " 2m, m P N, C d,k Y C d,k`1 " C d´1,m .
The parametrization of the nodes by the tuple pd, kq is best understood by considering the problem of visiting all the nodes of a tree, see Sedgewick and Wayne (2011) (Chapter 29). Going through the nodes pd, kq by increasing d and then by increasing k implements a depth first search (DFS). On the other hand, going through the nodes pd, kq by increasing k, then by increasing d implements a breadth first search (BFS). For example, for a tree of depth 2, it gives:

(DFS) : p0, 0q ă p1, 0q ă p2, 0q ă p2, 1q ă p1, 1q ă . . .

(BFS) : p0, 0q ă p1, 0q ă p1, 1q ă p2, 0q ă p2, 1q ă . . .
where pd, kq ă pd 1 , k 1 q means that the node pd, kq is visited first.

As shown by the definition of the ROC curve, see Eq. (3.1), the ROC curve of a piecewise constant score function is a set of points on the ROC plane. A common response in the analysis of piecewise constant scores, is to consider its ROC to be the broken line that connects those points, which corresponds to considering an element to be above another with probability 1{2 when their scores are equal, as precised below Eq. (3.1).

We introduce the alternative notations for the proportion of positives (resp. proportion of negatives) contained by a set C as βpCq (resp. αpCq), i.e. for any C Ă X , βpCq :" GpCq and αpCq :" HpCq. Then, Clémençon and Vayatis (2009) (Remark 13) provides an expression for the AUC of the piecewise constant score function s τ as:

AUCps τ q " 1 2 2 D ´1 ÿ k"1 pαpC D,k q `αpC D,k´1 qq βpR D,k´1 q, (3.12)
where R d,j " Y j k"0 C d,k for all d ě 0, j P t0, . . . , 2 d ´1u. The ROC of the output s D of TreeRank is thus a piecewise linear function, whose ROC (ROCps D , ¨q) is designed to approach that of a piecewise linear approximation of the optimal ROC curve ROC ˚. Clémençon and Vayatis (2009) introduce a score function s D , whose ROC (ROCps D , ¨q) is the piecewise linear approximation of ROC ˚. Hence, deriving theoretical guarantees for TreeRank consists in two steps. First, under regularity assumptions on the optimal ROC curve, we quantify how the piecewise linear interpolation α Þ Ñ ROCps D , αq approximates the optimal ROC curve ROC ˚. Second, we show that the ROC of s D can recover a good approximation of the interpolation α Þ Ñ ROCps D , αq learned from data, under an assumption on the complexity of the proposed family C used to split the cells. The algorithm for splitting the cells is referred to as LeafRank and its nature dictates the complexity of C. The distance between the ROC curves of s D , s D and s ˚P S ˚is measured with the following quantities, for any two score functions s 1 , s 2 :

d 1 ps 1 , s 2 q " ż 1 0 |ROCps 1 , αq ´ROCps 2 , αq| dα, d 8 ps 1 , s 2 q " sup αPp0,1q
|ROCps 1 , αq ´ROCps 2 , αq|.

Proposition 3.3 implies that for any score function s, we have d 1 ps ˚, sq " AUCps ˚q ´AUCpsq.

In general, the approximation error of the piecewise linear interpolation of a function f that is twice differentiable on ra, bs is controlled using its second derivative, as shown in [START_REF] Davis | Interpolation and approximation[END_REF] (Theorem 3.1.1) and reminded in the proposition below. This property is a consequence of the generalized Rolle's theorem (Davis, 1975, Theorem 1.6.3). Proposition 3.9. Assume that f : R Ñ R is twice differentiable on ra, bs. Formally, for any x such that a ď x ď b, we have:

f pxq ´ˆf paq `f pbq ´f paq b ´a px ´aq ˙ ď pb ´aq 2 8 max aăuăb f 2 puq .
By the definition of the optimal ROC curve, points of ROC ˚correspond to error rates of super-level sets of the posterior probability η. Therefore, the candidate splitting class for the interpolation procedure writes as a super level-set on the value ηpxq:

C ˚" ttx P X : ηpxq ą tu : t P r0, 1su, as expected from Proposition 3.3. The interpolation procedure seeks to find a set C t P C ˚that satisfies C t " tx P X : ηpxq ą tu, with t P r0, 1s, such that adding the point pαpC t q, βpC t qq between the two points pα d,k , β d,k q and pα d,k`1 , β d,k`1 q of ROC ˚maximizes the AUC increment Λ d,k pC t q{2, with:

Λ d,k pC t q :" pα d,k`1 ´αd ,k q ¨βpC t q ´pβ d,k`1 ´βd ,k q ¨αpC t q, " pα d,k`1 ´αd ,k q ¨s G ˚ptq ´pβ d,k`1 ´βd ,k q ¨s H ˚ptq. (3.13)
The quantity Λ d,k pC t q is the AUC increment of the split, since the magnitude of the cross-product between the vectors pαpC t q, βpC t q, 0q and pα 0,1 ´α0 ,0 , β 0,1 ´β0 ,0 , 0q is equal to the area of the parallelogram that the vectors span. Maximizing Λ d,k pC t q in t yields the following equality, if both s H ˚and s G ˚are differentiable and the derivative of s H ˚is nonzero:

s G ˚1 ptq s H ˚1 ptq " β d,k`1 ´βd ,k α d,k`1 ´αd ,k , (3.14)
Eq. (3.14) is an equality between the derivative of ROC ˚and the slope between the points pα d,k , β d,k q and pα d,k`1 , β d,k`1 q of ROC ˚.

Both Eq. (3.14) and the approximation result presented in Proposition 3.9 justify the derivation of an expression for the first and second derivative of the ROC curve, provided in Proposition 3.11. Proposition 3.11 requires the following technical assumptions.

Assumption 3.10. (Clémençon and Vayatis, 2009, Assumptions A1 and A2) The three following assumptions hold true:

1. The distributions G and H are equivalent, i.e. each of them is absolutely continuous w.r.t. the other (Shorack, 2000, Definition 1.3, Chapter 4).

2. The likelihood ratio dG{dHpXq is bounded. This property is equivalent from Bayes' theorem to ηpXq ă 1 a.s..

3.

The distribution of ηpXq is absolutely continuous w.r.t. the Lebesgue measure.

Proposition 3.11 (Derivatives of ROC ˚). (Clémençon and Vayatis, 2009, Proposition 6) Assume that Assumption 3.10 is satisfied, and that there exists c ą 0 such that H ˚1 puq ě c for any u P supppH ˚1 q, where supppH ˚1 q is the support of H ˚1 . Then, the optimal ROC curve ROC is twice differentiable on r0, 1s with bounded derivatives, and we have: @α P r0, 1s,

d dα ROC ˚pαq " 1 ´p p ¨Q˚p αq 1 ´Q˚p αq , d 2 dα 2 ROC ˚pαq " 1 ´p p ¨Q˚1 pαq p1 ´Q˚p αqq 2 ,
where Q ˚1 pαq " ´1{H ˚1 pQ ˚pαqq. Note that Q ˚pαq ď Q ˚p0q " ηpXq 8 ă 1, which implies the boundedness of the derivatives.

The procedure to define s D is summarized in Fig. 3.3. The expression of ∆ d`1,2k in Fig. 3.3 is justified by combining Eq. (3.13) with the following expression of the likelihood ratio of ηpXq | Y " ´1 and ηpXq | Y " `1:

dG dH ˚puq " 1 ´p p ¨u 1 ´u ,
found in the proof of Proposition 3.11, and maximizing Λ d,k pC t q in t. The definition of s d gives, combined with Proposition 3.9 and Lemma 18 of Clémençon and Vayatis (2009), a control for both of the distances d 1 and d 8 between the ROC of the piecewise constant s d and the ROC of an optimal score function, as shown in Proposition 3.12 below. Proposition 3.12. (Clémençon and Vayatis, 2009, Proposition 10) Assume that the same assumptions as Proposition 3.11 hold. Then, the sequence of piecewise constant scoring functions ps d q dě1 , satisfies: for any d ě 1,

AUC ˚´AUCps d q " d 1 ps ˚, s d q ď C ¨2´2d , d 8 ps ˚, s d q ď C ¨2´2d .
Now, we present guarantees for the distance between the ROC's of the output of the TreeRank algorithm (ROCps D , ¨q) and the piecewise linear approximation of ROC ˚(ROCps D , 9 q). As TreeRank's scoring rule is learned from empirical data, we introduce the empirical proportion of negatives (resp. positives) as p α (resp. p β) on the set C as:

p αpCq :" 1 n ´n ÿ i"1 ItX i P C, Y i " ´1u and p βpCq :" 1 n `n ÿ i"1 ItX i P C, Y i " `1u.
TreeRank relies on a proposed family of sets C, that has finite VC-dimension. The TreeRank procedure is described in Fig. 3.4, and follows the same ideas as Fig. 3.3 but without knowledge of the data distributions and optimal ROC curve ROC ˚. See Fig. 3.2 for a visual representation of the construction of a ranking tree. The following lemma defines a recurrence hypothesis that implies Theorem 3.15. Combined with Proposition 3.12, Theorem 3.15 implies guarantees for the TreeRank algorithm.

Example 3.13. Consider that the random variable X is uniformly distributed on X " r0, 1s 2 and that ηpxq :" Itx 1 `x2 ą 1u. Then, the data is separable and the optimal ROC (ROC ˚) is the unit step. In Initialization (d " 0 and d " 1). For the extremal points, we set

@d P N, α d,0 " β d,0 " αpHq " 0 and α d,2 d " β d,2 d " αpC0,0q " 1. From d to d `1, for d ě 1.
We are given the collection of points tpα d,k , β d,k qu k"0,...,2 d ´1. On each interval pα d,k , α d,k`1 q, we find a point on ROC ˚that maximizes the incremental AUC quantity Λ d,k , given by:

α d`1,2k`1 " s H ˚p∆ d`1,2k`1 q, and β d`1,2k`1 " s G ˚p∆ d`1,2k`1 q,
where:

∆ d`1,2k`1 " ppβ d,k`1 ´βd ,k q p1 ´pq ´αd ,k`1 ´αd ,k ¯`p ´βd ,k`1 ´βd ,k ¯.
Moreover, the previous cut-off point is renamed:

α d`1,2k " α d,k and β d`1,2k " β d,k and also ∆ d`1,2k " ∆ d,k .
Output.

For each level d, the resulting partition is given by the class of sets:

C d,k " tx P X : ∆ d,k`1 ă ηpxq ď ∆ d,k u,
for all k " 0, . . . , 2 d ´1 with the convention that ∆ d,0 " 0 and ∆ d,2 d " 1 for all d ě 0.

The TreeRank Algorithm

Input. Maximal depth D ě 1 of the scoring tree, class A of measurable subsets of X , training dataset Dn " tpX1, Y1q, . . . , pXn, Ynqu.

1. (Initialization.) Set C0,0 " X , α d,0 " β d,0 " 0 and α d,2 d " β d,2 d " 1 for all d ě 0.

2. (Iterations.) For d " 0, . . . , D ´1 and k " 0, . . . , 2 d ´1:

(a) (Optimization step.) Set the entropic measure:

p Λ d,k`1 pCq " pα d,k`1 ´αd,k qp αpCq ´pβ d,k`1 ´βd,k q p βpCq.
Find the best subset C d`1,2k of the cell C d,k in the AUC sense:

C d`1,2k " arg max CPA, CĂC d,k p Λ d,k`1 pCq . (3.15) Then, set C d`1,2k`1 " C d,k zC d`1,2k . (b) (Update.) Set α d`1,2k`1 " α d,k `p αpC d`1,2k q, α d`1,2k`2 " α d,k`1 , β d`1,2k`1 " β d,k `p βpC d`1,2k q, β d`1,2k`2 " β d,k`1 .
3. (Output.) After D iterations, get the piecewise constant score function:

sDpxq " as LeafRank for differents values of the depth D. As TreeRank is a greedy algorithm, it can not recover from an initial bad split in that case. Our illustrations shows the necessity for the LeafRank algorithm is adapted to type of data considered.

2 D ´1 ÿ k"0 p2 D ´kq Itpxq P C D,k u. ( 3
Lemma 3.14. (Clémençon and Vayatis, 2009, Lemma 19) Assume that the assumptions of Proposition 3.12 hold true and that the class C of subset candidates contains all level sets tR αu αPr0,1s and is of VC-dimension V and is intersection stable, i.e., @C, C 1 P C 2 : C X C 1 P C. Then, there exists positive constants κ 1 , κ 2 , c 1 and c 2 such that: for all δ ą 0, we have w.p. ě 1 ´δ, that @d P N ˚, @n P N ˚,

|AUCps d q ´AUCpp s d q| ď κ d´1 1 Bpd, n, δq,
and @k P t0, . . . , 2 d´1 ´1u:

αpC d,2k q ´αpC d,2k q ` βpC d,2k q ´βpC d,2k q ď κ d 2 Bpd `1, n, δq
, where @pd, n, δq P N ˆNˆs0, 1r:

Bpd, n, δq " ˆc2 1 V n ˙1 2d `ˆc 2 2 logp1{δq n ˙1 2d .
Lemma 3.14 implies the following theorem.

Theorem 3.15. (Clémençon and Vayatis, 2009, Theorem 15) Assume that the assumptions of Lemma 3.14 hold. For all δ ą 0, there exists a constant c 0 and universal constants c 1 , c 2 such that w.p. ě 1 ´δ, we have for all D ě 1, n P N,

d 1 pp s D , s D q ď c D 0 ˜ˆc 2 1 V n ˙1 2D `ˆc 2 2 logp1{δq n ˙1 2D ¸, d 8 pp s D , s D q ď c D 0 ˜ˆc 2 1 V n ˙1 2pD`1q `ˆc 2 2 logp1{δq n ˙1 2pD`1q
¸.

Ranking aggregation

Combining Theorem 3.15 and Proposition 3.12 gives guarantees on the ability of TreeRank to recover the optimal ROC curve.

Corollary 3.16. (Clémençon and Vayatis, 2009, Corollary 16) Assume that the assumptions of Lemma 3.14 hold. Choose D " D n , so that D n " a logpnq as n Ñ 8. Then, for all δ ą 0, there exists a constant κ such that, w.p. ě 1 ´δ: for all n P N, d i pp s Dn , s ˚q ď expp´κ a logpnqq, i P t1, 8u.

Ranking aggregation

Introduction

As pointed out in Korba ( 2018) (Part 1), ranking data is less considered in the machine learning literature than other types of data because the space of rankings is not a vector space. Hence, many approaches in machine learning cannot directly apply to ranking data. Introduce the space of all permutations of t1, . . . , Ku as S K . The basic notion of mean or median of a sample of n elements D n " tσ i u n i"1 P S K in the space of permutations does not have a clear definition, but it does on vector spaces. The mean or the median of a sample in a vector space gives a sensible summary of that sample, and ranking aggregation can be defined as the problem of finding a sensible summary in S K for the sample D n .

Approaches to ranking aggregation can be separated in two groups: those that introduce a distance on the space of rankings and seek a ranking σ ˚that minimizes an expected distance to elements of the sample D n , and those that model explicitly the generation of the σ i 's and derive a notion of true ranking from the estimated parameters of the model.

The natural average X of a sample in a inner product space is the minimizer of the average distance between the points of the sample and X, which justifies the distance-based approach. Precisely, introduce a random sample Q n " tX i u n i"1 in the space X with a scalar product x¨, ¨y and an associated norm ¨ . Then, the average s X of the random sample Q n is the solution of the following problem:

min xPX Epxq with Epxq :" n ÿ i"1 X i ´x 2 .
Indeed, simple calculus implies that Epxq ´Ep s Xq " n s X ´x 2 ě 0 with s X " p1{nq ř n i"1 X i . The most typical example of a distance-based approach to ranking aggregation is the Kemeny ranking aggregation [START_REF] Kemeny | Mathematics without numbers[END_REF]. Formally, it writes on the sample D n , as:

min σPS K n ÿ i"1 d τ pσ, σ i q, (3.17)
where d τ is the Kendall τ distance between two permutations, defined for any two σ, σ 1 P S K as the number of pairwise disagreements over t1, . . . , Ku. Formally:

d τ pσ, σ 1 q :" ÿ kăl Itpσpkq ´σplqqpσ 1 pkq ´σ1 plqq ă 0u, " ÿ kăl Itσ ˝σ1 ´1pkq ą σ ˝σ1 ´1plqu.
Since d τ pσ, σ 1 q depends only on the permutation σ ˝σ1 ´1, authors often consider a function d τ : S K Ñ R defined as the distance between σ and the identity permutation, with:

d τ pσq :" ÿ kăl Itσpkq ą σplqu.
A minimizer σ ˚of Eq. (3.17) is called a Kemeny consensus. The Kemeny consensus for an infinite sample, i.e. the solution of min σPS K Erd τ pσ, Σqs, is referred to as Kemeny median in Korba (2018) (Chapter 1). While the Kemeny ranking aggregation procedure satisfies good properties, computing the Kemeny consensus is a NP-hard problem (Cormen et al., 2009, Section 34). We refer to Korba (2018) (Chapter 2 and 3) for an overview of methods that give a good approximation of the Kemeny consensus σ ˚. Now, we focus on the notion of aggregation derived from probabilistic models for ranking data, as those are more related to the thesis.

Probabilistic Rankings Models

Probabilistic models for ranking data model the distribution of a random permutation Σ P S K by real-valued parameters. In this context, we consider the observations to aggregate D n " tσ i u n i"1 Ă S K as i.i.d. realizations of the random variable Σ. The most simple distribution considers the r.v. Σ to be a slightly modified permutation of a reference permutation σ ˚, with the strength of the modifications proportional to a dispersion parameter λ. It is the Mallows model presented below.

Definition 3.17 (The Mallows model). [START_REF] Korba | Learning from ranking data : theory and methods[END_REF], Section 2.2.1) The Mallows model is parameterized by a reference ranking σ ˚and a dispersion parameter λ P R `. Let σ P S K , then:

PtΣ " σu " 1 Z exp p´λ ¨dτ pσ, σ ˚qq ,
where Z " ř σPS K exp p´λ ¨dτ pσ, σ ˚qq.

Now, we provide a simple technique to sample from the Mallows model, referred to in Mesaoudi-Paul et al. ( 2018) (Section 2.1). Notice that:

d τ pσ, σ ˚q " ÿ 1ďkălďK Itσ ˝σ˚´1 pkq ą σ ˝σ˚´1 plqu,
which implies that the Kendall τ distance d τ pσ, σ ˚q can be seen as the number of wrongly rankedelements when resorting the reference ranking σ ˚with σ. Precisely, for l P t2, Ku, introduce the position of the l-th element in the reordering of tσ ˚´1 p1q, . . . , σ ˚´1 pl ´1qu by σ as the following value:

V l pσ | σ ˚q :" ÿ 1ďkăl
Itσ ˝σ˚´1 pkq ą σ ˝σ˚´1 plqu.

The function j Þ Ñ V l pσ | σ ˚q is a well-known notion in combinatorics defined as the Lehmer code of the permutation σ ˝σ˚´1 . Then, the distribution of Σ writes as:

PtΣ " σu " 1 Z K ź l"2 exp p´λV l pσ | σ ˚qq . (3.18)
Eq. (3.18) implies that generating from the Mallows model can be done by constructing a random reordering of the reference ranking σ ˚. Precisely, consider an empty list and add to this list the element σ ˚´1 p1q. With probability expp´λq, insert the second element σ ˚´1 p2q before that first element, i.e. in position V 2 pσ | σ ˚q, with positions starting at 0. Then, insert sequentially all of the remaining elements σ ˚´1 plq at position V l pσ | σ ˚q P t0, . . . , l ´1u for all l P t3, . . . , Ku, where the Lehmer code is generated with, for any r P t0, . . . , l ´1u:

PtV l pσ | σ ˚q " r | V l´1 . . . , V 1 u 9 e ´λr ,
and 9 denotes proportionality. This approach for generating from a Mallows model is called the Repeated Insertion Model (RIM) and was introduced by [START_REF] Doignon | The repeated insertion model for rankings: Missing link between two subset choice models[END_REF]. We refer to that last publication for details.

Another popular model is the Thurstone model, which simply generates K independent random variables in R with different distributions, and outputs the natural order Σ on R of those K variables. Hence, sampling from the Thurstone model is straightforward.

Definition 3.18 (The Thurstone model). [START_REF] Korba | Learning from ranking data : theory and methods[END_REF], Section 2.2.1) Given the independent random variables X 1 , X 2 , . . . , X K with different real-valued distributions X k " P k for k P t1, . . . , Ku, the probability to observe a ranking σ in the Thurstone model is defined as:

PtΣ " σu " PtX σ ´1p1q ă X σ ´1 p2q ă ¨¨¨ă X σ ´1pKq u.
Finally, the Plackett-Luce model is a generalization to full rankings of the pairwise comparison model of Bradley-Terry. That last model assumes that Σ follows the property below, for a preference vector pw 1 , . . . , w K q:

PtΣpkq ą Σplqu " w k w k `wl .
Definition 3.19 is thus very often referred to as the Bradley-Terry-Luce-Plackett (BTLP) model.

It was introduced in [START_REF] Bradley | Rank analysis of incomplete block designs: I. the method of paired comparisons[END_REF], [START_REF] Luce | Individual Choice Behavior[END_REF] and [START_REF] Plackett | The analysis of permutations[END_REF] and is presented below.

Definition 3.19 (The Plackett-Luce model). [START_REF] Korba | Learning from ranking data : theory and methods[END_REF], Section 2.2.1) Given a preference vector w " pw 1 , . . . , w K q, the Plackett-Luce model states that:

PtΣ " σu " K´1 ź k"1 w σ ´1pkq ř K l"k w σ ´1plq . (3.19)
As expected from the expression of Eq. (3.19), a natural way to sample from the Plackett-Luce model is to sample sequentially the Σ ´1pkq for each k P t1, . . . , Ku, i.e. the elements at each rank k for the permutation Σ. First, note that PtΣ ´1p1q " lu " w l {p ř K j"1 w j q, which means that Σ ´1p1q follows a multinomial distribution of size 1 with support S 1 :" t1, . . . , Ku and parameters tw l {p ř K j"1 w j qu K l"1 . Let k P t2, . . . , Ku and denote by S k the remaining elements, i.e. S k :" S 1 ztΣ ´1p1q, . . . , Σ ´1pk ´1qu, then, for any l P S k :

PtΣ ´1pkq " l | Σ ´1p1q, . . . , Σ ´1pk ´1qu " w l ř jPS k w j .
The limitations of the parametric models presented have led authors to consider non-parametric models for rankings, which we do not detail here. We refer to Korba (2018) (Section 2.2.2) for more details. Under the assumption below on a probabilistic model on rankings, the Kemeny consensus writes as a function of the parameters of the model.

Assumption 3.20 (Strict Stochastic Transitivity (SST)). [START_REF] Korba | Learning from ranking data : theory and methods[END_REF], Definition 1.2) For any pk, lq P t1, . . . , Ku 2 , introduce the pairwise probability as p k,l " PtΣpkq ă Σplqu.

The Strong Stochastic Transitivity (SST) assumption states that: for any k, l, m P t1, . . . , Ku,

p k,l ą 1{2 and p l,m ą 1{2 ñ p k,m ą 1{2.
Under Assumption 3.20, the Kemeny median is equal to:

σ ˚pkq " 1 `ÿ l‰k Itp k,l ă 1{2u for any k P t1, . . . , Ku,
see [START_REF] Korba | Learning from ranking data : theory and methods[END_REF] (Proposition 1.3). Assume that the SST assumption holds true. For a model, we can explicit the pairwise probabilities p k,l 's involved in the expression of the Kemeny median as functions of its parameters. Therefore, we can derive a sensible solution of the ranking aggregation problem directly from the estimation of the parameters.

Connections to Present Work

The theory of bipartite ranking presented in this chapter is extended in Part II to our similarity ranking problem (Chapter 5). That theory also helps to grasp the origin of our practical propositions for similarity ranking (Chapter 7). In Part III, Chapter 10 considers bipartite ranking under fairness constraints. Finally, Chapter 8 relies on probabilistic models for interpreting classification data as partial information about rankings on labels.

In Part II, Chapter 5 frames similarity learning as scoring pairs of features in the product space X ˆX from the most similar to the least similar, and names this problem similarity ranking. This point of view is natural, as many applications evaluate the performance of similarity functions with metrics based on induced rankings, such as the ROC curve. In this context, Chapter 5 derives guarantees for the pointwise ROC optimization problem and the TreeRank algorithm in the case of similarity ranking. Precisely, it extends the results of Proposition 3.5, Proposition 3.8 and Lemma 3.14, which involves results on U -statistics (Chapter 4).

In Part III, Chapter 10 discusses learning fair scoring functions, with constraints derived from ROC-based criteria. As such, it leverages the discussion on bipartite ranking of this chapter, combined with a decomposition of the ROC [START_REF] Hsieh | Nonparametric and semiparametric estimation of the receiver operating characteristic curve[END_REF], Theorem 2.1) not presented in this chapter. Finally, Chapter 8 considers the problem of providing a list of the possible labels for an instance, ordered by the likelihood of the label. For that matter, it models the relationship between random labels and an ordering of the labels. Hence, Chapter 8 leverages probabilistic models for rankings (Section 3.3.2), and combines their properties with fast generalization bounds in binary classification (Chapter 2)

Chapter 4

U-statistics

Summary: This chapter is a short introduction to U -statistics, which in their simplest form are averages of the evaluation of a real-valued function on all npn ´1q{2 pairs that can be formed with a sample of size n. It provides the necessary tools for for our theoretical results on similarity ranking (Chapter 5) and distributed U -statistics (Chapter 6). To a lesser extent, it is involved in the analysis of pairwise functionals in our work on fair ranking (Chapter 10).

The complexity of dealing with U -statistics arises from the dependence of the terms involved in the sum. To extend the results for empirical processes to U -statistics, many statistical tools were introduced. In this chapter, we first detail the derivation of finite-sample guarantees on the supremum of the deviations of U -statistics, following a similar analysis as finite-sample bounds for binary classification (Chapter 2). That analysis also implies analytical expressions for the variance of U -statistics. The main drawback of standard U -statistics is their computational complexity as the sample size n grows.

Authors have thus proposed to consider incomplete U -statistics, which average a random sample of B pairs selected by sampling with replacement. We detail in this chapter the guarantees of [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] for incomplete U -statistics, as well as their expression of the variance. Finally, we detail the involvement throughout the thesis of the theory of U -statistics presented here. We refer to [START_REF] Lee | U -statistics: Theory and practice[END_REF] and de la Pena and Giné (1999) for a more detailed presentation of U -statistics.

Introduction

Most of the statistical machine learning literature focuses on averages of i.i.d. random variables, since usual risks can be estimated with that type of estimators, for example in binary classification (Chapter 2) or in bipartite ranking (Chapter 3). For other problems, estimators of the risk can not -or should not -be introduced as averages of i.i.d. variables. For example, in the case of metric learning (Bellet et al. (2015a)), empirical evaluations of the risk rely on evaluations of the distance between a pair of independent copies of a random variable. If a realization of the same random variable is involved in two pairs, the terms averaged in an estimator of the risk are not independent, an the whole analysis of Chapter 2 and Chapter 3 do not apply. Computing the average of a carefully selected set of pairs may restore the independence between the elements of the sum. However, the limitations over the possible number of selected independent pairs leads to an inaccurate estimator.

Averages over all pairs of independent random variables are well-known in the literature and can be considered as the simplest form of a broad family of statistics referred to as U -statistics. U -statistics are minimum variance unbiased estimators (MVUE) of their expected value [START_REF] Lee | U -statistics: Theory and practice[END_REF], and were first introduced by [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF]. [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF] proposed decompositions for U -statistics that imply generalizations of the usual finite-sample learning bounds of Chapter 2 to U -statistics, as well as analytical expressions of their variance.

The main drawback of U -statistics is their computational cost as the number of samples n grows, since the number of terms that they average is quadratic in n. A natural approach to dealing with the computational cost of a full U -statistic are incomplete U -statistics, which average a fixed number of pairs, selected at random in the set of all possible pairs. They were introduced in [START_REF] Blom | Some properties of incomplete U-statistics[END_REF] and we refer to [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] for a detailed account of their statistical accuracy. In this chapter, we focus on one-sample U -statistics of degree 2i.e. U -statistics formed with only one i.i.d. sample and pairs of elements of that sample -and give a more general presentation in Section 6.2.1 of Chapter 6. We refer to [START_REF] Lee | U -statistics: Theory and practice[END_REF] and de la Pena and Giné (1999) for a more details on U -statistics.

Section 4.2 introduces formally U -statistics, and gives a few statistical properties that motivate their study. Section 4.3 presents the theoretical tools for the extension of the learning bounds presented in Chapter 2 to the case of U -statistics. Additionally, those same tools imply expressions of their variance. In Section 4.4, we provide similar results for incomplete U -statistics. Section 4.5 details the implications of the results of this chapter on the original work of the thesis.

Preliminaries

Introduce a random variable X " P in an Euclidean space X as well as an independent copy X 1 of X. Consider the estimation of a value that depends of P , denoted by θpP q P R, such that:

θpP q " ErhpX, X 1 qs, (4.1)
where h is a measurable function X ˆX Ñ R, called a kernel. The value θpP q is called a population characteristic of P in [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF], and the function θ is called a functional of P .

Examples of quantities that write as Eq. ( 4.1) include:

• the variance of a distribution: X " R and hpX, X 1 q " pX ´X1 q 2 {2,

• the Gini mean difference [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF], page 297): X " R and hpX, X 1 q " |X ´X1 |,

• Kendall's τ : X " r0, 1s ˆr0, 1s and hppX 1 , X 2 q, pX 1 1 , X 1 2 qq " ItpX 1 ´X1 1 qpX 2 ´X1 2 q ą 0u, as well as the estimators of the AUC involved in Chapter 3 and all of the statistics considered in the similarity ranking setting presented in Part II.

A simple example of U -statistic is the natural estimator of the population characteristic θpP q.

Formally, consider a sample D n :" tX i u n i"1 composed of n i.i.d. copies of X. The U -statistic U n phq constitutes an unbiased estimator of U phq :" ErU n phqs " θpP q, with:

U n phq :" 2 npn ´1q ÿ iăj hpX i , X j q. (4.2)
In the rest of the chapter, we assume without loss of generality that the kernel h is symmetric, since we can define the symmetric kernel h 1 as:

h 1 px, x 1 q :" 1 2 phpx, x 1 q `hpx 1 , xqq,
such that U n ph 1 q is a U -statistic with symmetric kernel, and also an unbiased estimator of θpP q.

The estimator U n phq is said to be symmetric, which means that computing it on a permutation of the sample D n yields the same value. This property is true irrelevant of the distribution P .

The following property shows the unicity of a symmetric unbiased estimator in a large class of distributions P over X is unique.

Theorem 4.1. [START_REF] Lee | U -statistics: Theory and practice[END_REF], Theorem 2, Section 1.1, Chapter 1). Consider the class of distributions P of all distributions with finite support in X . Then, U n phq is the unique symmetric unbiased estimator of θpP q for any P P P.

The main complexity when studying U n phq is the dependence of the terms involved in the summation in Eq. (4.2). For example, the summation involves the two dependent terms hpX 1 , X 2 q 4.3. Properties of Simple U -Statistics and hpX 1 , X 3 q. Hence, all of the results for standard empirical means introduced in Chapter 2 do not extend to U -statistics. For that reason, other estimators of θpP q could be considered, such as the average V n phq of tn{2u independent terms:

V n phq :" 1 tn{2u tn{2u ÿ i"1 h `Xi , X i`tn{2u ˘. (4.3)
Using V n phq instead of U n phq requires less computational power as we average Opnq terms instead of Opn 2 q for U n phq. Additionally, V n phq is covered by the analysis introduced in Chapter 2. However, the estimator V n phq is less precise than U n phq since it averages way less terms. In that regard, the importance of U -statistics is formally justified by Theorem 4.2 below.

Theorem 4.2. [START_REF] Lee | U -statistics: Theory and practice[END_REF], Theorem 3, Section 1.1, Chapter 1)

The statistic U n phq is a minimum variance unbiased estimator (MVUE) of the parameter θpP q.

Properties of Simple U -Statistics

As presented in Eq. ( 4.3), estimators of θpF q that are simple empirical averages exist, but have higher variance than U -statistics. The first Hoeffding decomposition (Proposition 4.3 below) shows that U -statistics can be written as the mean of n! dependent simple empirical averages. It is proven simply by rearranging of the terms of the sum. Lemma 4.4 exploits that decomposition, to provide a first approach for extending the results of Chapter 2 on standard averages to U -statistics.

Proposition 4.3 (First Hoeffding decomposition). (Hoeffding, 1963, Equation 5.5) or [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF], Equation 15)

The U -statistic U n phq writes as the mean of n! dependent standard averages. Those averages correspond to evaluations of V n phq (Eq. (4.3)) on a permuted sample D n . Precisely, we have:

U n phq " 1 n! ÿ σPSn 1 tn{2u tn{2u ÿ i"1 h `Xσpiq , X σpi`tn{2uq ˘.
Lemma 4.4. (Clémençon et al., 2008, Lemma A.1) Let th t u tPT be a family of symmetric measurable functions indexed by t P T , with T some set. Assume that ψ is a convex non-decreasing function. Then, we have:

E « ψ ˜sup tPT 2 npn ´1q ÿ iăj h t pX i , X j q ¸ff ď E » -ψ ¨sup tPT 1 tn{2u tn{2u ÿ i"1 h t pX i , X tn{2u`i q 'fi fl ,
under the assumption that the suprema are measurable and the expected values exist.

The main implication of Lemma 4.4 is that the results of Section 2.3.1 hold for U -statistics, a consequence of the Chernoff bound (Proposition 2.4-Chapter 2) that is presented below. The drawback of that analysis is that it gives the same bounds for both U n phq and the inaccurate estimator V n phq (Eq. ( 4.3)). Corollary 4.5 below is an extension of Proposition 2.19 (in Chapter 2) to U -statistics. Its proof is a simple combination of Lemma 4.4, Proposition 2.19 and the inequalities pn ´1q{2 ď tn{2u ď n{2.

Corollary 4.5. (Clémençon et al., 2016, Proposition 2) Let H be a collection of symmetric kernels, that are indicators of a VC-class of sets of VCdimension V . Then, for all δ P p0, 1q and n P N such that n ě 2, we have: w.p. ě 1 ´δ,

sup hPH |U n phq ´U phq| ď c logp1{δq n ´1 `c 16 logp2q `16V logp1 `nq n ´1 .
A more refined version of this inequality can be proven using a chaining argument, see Proposition 2.19 (Chapter 2). It states that, for all δ P p0, 1q and n P N such that n ě 2, we have: w.p. ě 1 ´δ, w.p ě 1 ´δ,

sup hPH |U n phq ´U phq| ď c logp1{δq n ´1 `2C c 2V n ,
where C ą 0 is an universal constant.

As in Chapter 2, tighter bounds are obtained by considering the variance of U -statistics. To compute that variance, we use a linearization technique presented below, called the second Hoeffding decomposition (Proposition 4.6). It writes the estimator U n phq as the sum of two decorrelated terms, and implies directly a simple expression for the variance of U n phq. Precisely, it decomposes U n phq as its Hájek projection -its projection on standard averages w.r.t. the L 2 norm -plus a remainder, that satisfies a "degenerate" property presented in Definition 4.8 below. The second Hoeffding decomposition is proven simply by rearranging of the terms of the sum.

Proposition 4.6 (Second Hoeffding decomposition). (Hoeffding, 1948, Equation 5.18) or [START_REF] Lee | U -statistics: Theory and practice[END_REF], Theorem 3, Section 1.3, Chapter 1)

The U -statistic U n phq can be written as the sum between an empirical process and a degenerate U -statistic. Formally:

U n phq ´U phq " 2T n phq `Wn phq,
where:

T n phq :" 1 n n ÿ i"1 h 0 pX i q and W n phq :" 2 npn ´1q ÿ iăj h 1 pX i , X j q,
with: h 0 pxq " Erhpx, X 1 qs ´U phq and h 1 px, x 1 q :" hpx, x 1 q ´U phq ´h0 pxq ´h0 px 1 q.

Corollary 4.7. Since: @i, j, k, l P t1, . . . , nu, with i ‰ j, k ‰ l and pi, jq ‰ pk, lq,

Covph 1 pX i , X j q, h 1 pX k , X l qq " 0, Covph 0 pX i q, h 0 pX k qq " 0,
and Covph 0 pX i q, h 1 pX k , X l qq " 0,

we have that:

VarpU n phqq " 4σ 2 0 n `2σ 2 1 npn ´1q .
The U -statistic W n phq in Proposition 4.6 is a special type of U -statistic called a degenerate U -statistic. Its variance is low compared to T n phq, as implied by Corollary 4.7.

Definition 4.8 (Degenerate U -statistic). [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF], Equation 5.11) or (van der Vaart, 2000, Section 12.3) Assume that for any x P X , we have Erhpx, Xqs " 0. Or equivalently, that h 0 " 0. Then, U n phq is said to be a degenerate U -statistic -or stationary of order 1 -and it satisfies:

VarpU n phqq " Opn ´2q. Clémençon et al. (2008) presented fast learning bounds for empirical risk minimization with U -statistics. For that matter, they introduced a noise assumption (as in Section 2.4) to derive bounds for the term T n phq. They deal with the term W n phq with specific properties of degenerate U -statistics.

For T n phq, Clémençon et al. (2008) give a noise condition on the variance of T n phq directly. However, in Section 2.4.2 of Chapter 2, a condition on the distribution implies a condition on some variance. By Jensen's inequality, we have Varph 0 pXqq ď VarphpX, X 1 qq. It shows that any term in the Hájek projection T n phq has lower variance than any term of the quantity V n phq in Eq. (4.3) (van der Vaart, 2000, Chapter 11). As a result of the variance reduction property of the Hájek projection, fast learning bounds hold under weaker assumptions on the data distribution. This property explains the noise condition of Clémençon et al. (2008). A straightforward application of the analysis used to derive fast learning speeds in Chapter 2 achieves to bound T n phq.

For W n phq, we first present fast learning bounds for a finite family of functions, which follow from an union bound of results on single functions (Proposition 4.9). Then, we provide a result for more general classes of functions (Proposition 4.10).

Proposition 4.9. Consequence of (de la Pena and Giné, 1999, Theorem 4.1.13) Assume that h satisfies h 8 ď M h . Then, for any pn, δq P N ˚ˆp0, 1q: w.p ě 1 ´δ,

|W n phq| ď 4cM h logp4{δq n .
where c ą 0 is an universal constant.

Proof. We refer to de la Pena and Giné (1999) (Theorem 4.1.13) for the full proof of Proposition 4.9, but detail the application of the result. With the kernel h 1 , it gives:

Pt|W n phq| ą tu ď 4 exp ˆ´nt 4cM h ˙,
since |h 1 | ď 4M h . Proposition 4.9 follows from inverting the bound.

Proposition 4.10. (Arcones and Giné, 1994, Theorem 3.2) Assume that the family of kernels H is of VC-dimension V , and that any h P H satisfies h 8 ď M H . Then, for any pn, δq P N ˆp0, 1q such that n ě 2: w.p ě 1 ´δ,

sup hPH |W n phq| ď 8M H logpc{δq c 1 pn ´1q ,
where c, c 1 are constants that depend of V .

Proof. We refer to (Arcones and Giné, 1994, Theorem 3.2) for the full proof of Proposition 4.10, but detail the application of the result. [START_REF] Arcones | U-processes indexed by Vapnik-Červonenkis classes of functions with applications to asymptotics and bootstrap of U-statistics with estimated parameters[END_REF] refers to degenerate U -statistics as P -canonical U -statistics, see Equation 2.11 therein. On the other hand, the VC-dimension implies an upper bound on the covering number of a family of functions, as shown in [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] (Theorem 1.17). Applying the result to the family of h 1 's derived from the elements h P H, gives, since h 1 8 ď 4M H :

P " n ´1 8M H ¨sup hPH |W n phq| ě t * ď ce ´c1 t ,
where c and c 1 are positive constants that depend on V . Proposition 4.10 follows from inverting the bound.

To derive fast learning bounds for U -statistics, it suffices to: 1) repeat the analysis for standard averages (Chapter 2) on T n phq, while taking into account the specificities regarding the noise condition, 2) combine those with concentration bounds for degenerate U -statistics

Properties of Incomplete U -Statistics

The main drawback of the complete U -statistic U n phq is its computational cost for large n, since the number of terms to average is npn ´1q{2 (Eq. (4.2)). Thus, a popular idea is to consider the incomplete U -statistic U B phq, which averages the value of h over B pairs pX i , X j q selected at random in the set of all pairs (Section 4.3 of [START_REF] Lee | U -statistics: Theory and practice[END_REF] or [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF]). Formally, we have:

U B phq :" 1 B ÿ pi,jqPD B hpX i , X j q,
where D B is a set of cardinality B built by sampling with replacement in the set of all possible pairs Λ :" tpi, jq : i ă ju of size #Λ " npn ´1q{2. Another approach to reduce the computational cost of U -statistics is to consider estimating U phq with the full U -statistic U m phq on a smaller sample of size m. Replacing U n phq by U m phq in Corollary 4.5 gives a bound of the order Opm ´1{2 q.

The following result gives an expression of the variance of the incomplete U -statistic U B phq, which tends to that of U n phq when B tends to infinity, as expected.

Theorem 4.11. [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF], Equation 21) For any n ą 1 and B P N ˚, the variance of the incomplete U -statistic U B phq can be decomposed as:

VarpU B phqq " ˆ1 ´1 B ˙VarpU n phqq `1 B VarphpX 1 , X 2 qq.
Besides the expression of the variance, Clémençon et al. ( 2016) (Theorem 6) derived an upperbound on the deviation of the incomplete U -statistic U B phq from the complete one U n phq.

Theorem 4.12. (Clémençon et al., 2016, Theorem 6) Let H be a collection of kernels, that satisfies the assumptions of Corollary 4.5, then, for any n ą 1 and B P N ˚, w.p. ě 1 ´δ,

sup hPH |U B phq ´Un phq| ď c 2 V logp1 `npn ´1q{2q `logp2{δq B .
The combination of Corollary 4.5 and Theorem 4.12 implies a simple learning bound for incomplete U -statistics. A striking fact, underlined in Clémençon et al. ( 2016) (Section 3.1), is that it suffices to select B " Opnq pairs to recover the standard learning bound in Opn ´1{2 q presented in Chapter 2. For the same number of evaluated pairs B " mpm ´1q{2 " Opnq, computing the complete U -statistic U m phq on a sample of m pairs, gives a significantly slower bound in Opn ´1{4 q.

While the observation of [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] gives significant guarantees for large n, in practical cases (finite n) usual learning bounds are quite loose and do not give information about the relative accuracy of an incomplete U -statistics with respect to a complete U -statistic.

Connections to Present Work

Theoretical results on U -statistics are involved in Chapter 5 and Chapter 6 of Part II, as well as in Chapter 10 of Part III. Precisely, Chapter 5 extends the proofs for bipartite ranking seen in Chapter 3 to the case of similarity ranking, which involves pairwise learning. Chapter 6 introduces estimators of U -statistics in a distributed environment. Finally, Chapter 10 discusses fairness for bipartite ranking, and relies on the empirical AUC as a performance measure.

In Part II, Chapter 5 leverages standard learning bounds for U -statistics (Corollary 4.5) to derive uniform generalization bounds for similarity ranking, defined as the problem of learning a similarity on the product space X ˆX with a pairwise bipartite ranking objective. The second Hoeffding decomposition (Proposition 4.6), combined with bounds on degenerate U -statistics (Proposition 4.9 or Proposition 4.10), implies faster but data-dependent guarantees. Additionally, our convergence bound for incomplete U -statistics (Theorem 4.12) implies theoretical results for scalable sampling-based methods for similarity ranking. Those results are not a direct application of this chapter, since the pointwise ROC optimization problem presented in Chapter 5 is a constrained optimization. In Chapter 6, the implications on the variance of U -statistics (Theorem 4.11) of the second Hoeffding decomposition (Proposition 4.6) are essential to compute the variance of our distributed estimators for U -statistics.

In Part III, Chapter 10 involves standard learning bounds for U -statistics (Corollary 4.5) to derive generalization guarantees for maximizing the Area Under the ROC Curve (AUC), a pairwise functional criterion for ranking.

Chapter 5

Similarity Ranking Theory

Summary: The performance of biometric systems -and that of many machine learning techniques -depends on the choice of an appropriate similarity or distance measure on the input space. Similarity learning (or metric learning) aims at building such a measure from training data so that observations with the same (resp. different) label are as close (resp. far) as possible. In this chapter, similarity learning is investigated from the perspective of pairwise bipartite ranking, where the goal is to rank the elements of a database by decreasing order of the probability that they share the same label with some query data point, based on the similarity scores. We study this novel perspective on similarity learning, that we call similarity ranking, through a rigorous probabilistic framework. Our results are an extension of the analysis of bipartite ranking provided in Chapter 3, but the pairwise nature of similarity ranking involves results on U -statistics (Chapter 4). We provide an extensive study of the generalization of pointwise ROC optimization for similarity ranking, completed by an empirical illustration of the fast learning rates that we prove. Then, we proceed to extend the theoretical guarantees of the TreeRank algorithm to similarity ranking with the theory of U -statistics.

Introduction

Similarity (or distance) functions play a key role in many machine learning algorithms for problems ranging from classification (e.g., k-nearest neighbors) and clustering (e.g., k-means) to dimensionality reduction (van der Maaten and Hinton, 2008) and ranking [START_REF] Chechik | Large scale online learning of image similarity through ranking[END_REF]. They are also essential to biometric identification algorithms. The success of such methods is heavily dependent on the relevance of the similarity function to the task and dataset of interest. This has motivated the research in similarity and distance metric learning (Bellet et al., 2015a), a line of work which consists in automatically learning a similarity function from data. This training data often comes in the form of pairwise similarity judgments derived from labels, such as positive (resp. negative) pairs composed of two instances with same (resp. different) label.

Most existing algorithms can then be framed as unconstrained optimization problems where the objective is to minimize some average loss function over the set of similarity judgments, see for instance [START_REF] Goldberger | Neighbourhood components analysis[END_REF][START_REF] Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF]Bellet et al., 2015a). Some generalization bounds for this class of methods have been derived, accounting for the specific dependence structure found in the training similarity judgments [START_REF] Jin | Regularized distance metric learning: Theory and algorithm[END_REF]Bellet and Habrard, 2015;[START_REF] Cao | Generalization bounds for metric and similarity learning[END_REF][START_REF] Mason | Learning low-dimensional metrics[END_REF][START_REF] Verma | Sample complexity of learning mahalanobis distance metrics[END_REF]. We refer to [START_REF] Kulis | Metric Learning: A Survey[END_REF] and Bellet et al. (2015a) for detailed surveys on similarity and metric learning.

In this chapter, we study similarity learning from the perspective of pairwise bipartite ranking, where the goal is to rank the elements of a database by decreasing order of the probability that they share the same label with some query data point. This problem is motivated by many concrete applications: for instance, biometric identification aims to check the claimed identity of an individual by matching their biometric information (e.g. a photo taken at an airport) with a large reference database of authorized people (e.g. of passport photos) [START_REF] Jain | Introduction to Biometrics[END_REF]. Given a similarity function and a threshold, the database elements are ranked in decreasing order of similarity score with the query, and the matching elements are those whose score is above the threshold. In this context, performance criteria are related to the ROC curve associated with the similarity function, i.e. the relation between the false positive rate and the true positive rate.

Previous approaches have empirically tried to optimize the Area under the ROC curve (AUC) of the similarity function McFee and Lanckriet (2010); [START_REF] Huo | Cross-modal metric learning for auc optimization[END_REF], without establishing any generalization guarantees. The AUC is a global summary of the ROC curve which penalizes pairwise ranking errors regardless of the positions in the list. More local versions of the AUC (e.g., focusing on the top of the list) are difficult to optimize in practice and lead to complex nonconvex formulations [START_REF] Clémençon | Ranking the best instances[END_REF][START_REF] Huo | Cross-modal metric learning for auc optimization[END_REF].

In this chapter, we focus on a specific performance criterion, namely pointwise ROC optimization, which aims at maximizing the true positive rate under a fixed false positive rate. This objective, formulated as a constrained optimization problem, naturally expresses the operational constraints present in many practical scenarios. For instance, in biometric applications such as the one outlined above, the verification system is typically set to keep the proportion of people falsely considered a match below a predefined acceptable threshold (see e.g., [START_REF] Jain | Biometric identification[END_REF][START_REF] Jain | An introduction to biometric recognition[END_REF].

Specifically, we derive statistical guarantees for the approach of solving the constrained optimization problem corresponding to the empirical version of our theoretical objective, based on a dataset of n labeled data points. As the empirical quantities involved are not i.i.d. averages but rather in the form of U -statistics [START_REF] Lee | U -statistics: Theory and practice[END_REF], our results rely on concentration bounds developed for U -processes Clémençon et al. (2008). We first derive a learning rate of order Op1{ ? nq which holds without any assumption on the data distribution. We then show that one can obtain faster rates under a low-noise assumption on the data distribution, which has the form of a margin criterion involving the conditional quantile.

The chapter is organized as follows. Section 5.2 introduces the proposed probabilistic framework for similarity learning, draws connections to existing approaches and introduces the pointwise ROC optimization problem. In Section 5.3, we derive universal and fast learning rates for the minimizer of the empirical version of our problem, with Section 5.3.3 illustrating the different generalization speeds on a simple toy example. Finally, Section 5.4 presents the extension of the well-known TreeRank algorithm for learning score functions to the case of learning similarity functions as well as an extension of its theoretical guarantees based on the theory of U -statistics.

Similarity Ranking

In this section, we formulate the supervised similarity learning problem from the perspective of pairwise bipartite ranking, and highlight connections with some popular metric and similarity learning algorithms of the literature.

Similarity Learning as Pairwise Ranking

We consider the (multi-class) classification setting. The random variable Y denotes the output label with values in the discrete set t1, . . . , Ku with K ě 1, and X is the input random variable, taking its values in a feature space X Ă R d with d ě 1 and modeling some information hopefully useful to predict Y . We denote by F the marginal distribution of X and by ηpxq " pη 1 pxq, . . . , η K pxqq the posterior probability, where η k pxq " PtY " k | X " xu for x P X and k P t1, . . . , Ku. The distribution of the random pair pX, Y q is entirely characterized by P " pF, ηq. The probability of occurrence of an observation with label k P t1, . . . , Ku is assumed to be strictly positive and denoted by p k " PtY " ku, and the conditional distribution of X given Y " k is denoted by F k . Equipped with these notations, we have F " ř K k"1 p k F k . Optimal similarity measures. The objective of similarity learning can be informally formulated as follows: the goal is to learn, from a training sample D n " tpX 1 , Y 1 q, . . . , pX n , Y n qu composed of n ě 1 independent copies of pX, Y q, a (measurable) similarity measure s : X ˆX Ñ R such that given two independent pairs pX, Y q and pX 1 , Y 1 q drawn from P , the larger the similarity spX, X 1 q between two observations, the more likely they are to share the same label.

The set of all similarity measures is denoted by S. The class S ˚of optimal similarity rules naturally corresponds to the set of strictly increasing transforms T of the pairwise posterior probability ηpx, x1 q " PtY " Y 1 | pX, X 1 q " px, x 1 qu, where pX 1 , Y 1 q denotes an independent copy of pX, Y q: tT ˝η | T : Impηq Ñ R `borel, strictly increasingu, and where Impηq denotes the support of ηpX, X 1 q's distribution. With the notations previously introduced, we have ηpx, x 1 q " ř K k"1 η k pxqη k px 1 q for all px, x 1 q P X 2 . A similarity rule s ˚P S defines the optimal preorder 1 ĺ ˚on the product space X ˆX : for any px 1 , x 2 , x 3 , x 4 q P X 4 , x 1 and x 2 are more similar to each other than x 3 and x 4 if and only if (i.f.f.) ηpx 1 , x 2 q ě ηpx 3 , x 4 q, and one writes px 3 , x 4 q ĺ ˚px 1 , x 2 q in this case. For any x P X , s ˚also defines a preorder ĺ x on the input space X , permitting to rank optimally all possible observations by increasing degree of similarity to x: for all px 1 , x 2 q P X 2 , x 1 is more similar to x than x 2 (one writes x 2 ĺ x x 1 ) i.f.f. px, x 2 q ĺ ˚px, x 1 q, meaning that ηpx, x 2 q ď ηpx, x 1 q. We point out that, despite its simplicity, this framework covers a wide variety of applications, such as the biometric identification problem mentioned earlier in the introduction.

Similarity learning as pairwise bipartite ranking. In view of the objective formulated above, similarity learning can be seen as a bipartite ranking problem on the product space X ˆX where, given two independent realizations pX, Y q and pX 1 , Y 1 q of P , the input r.v. is the pair pX, X 1 q and the binary label is Z " 2ItY " Y 1 u ´1. See Chapter 3 for a statistical learning view of bipartite ranking.

ROC analysis is the gold standard to evaluate the performance of a similarity measure s in this context, i.e. to measure how close the preorder induced by s is to ĺ

˚. The ROC curve of s is the PP-plot t P R `Þ Ñ p s H s ptq, s G s ptqq,
where, for all t ě 0,

H s ptq " PtspX, X 1 q ď t | Z " ´1u and G s ptq " PtspX, X 1 q ď t | Z " `1u,
where possible jumps are connected by line segments and s F denotes the survival function of any distribution F over R, i.e. s

F " 1 ´F . Hence, it can be viewed as the graph of a continuous function α P p0, 1q Þ Ñ ROC s pαq, where ROC s pαq " s G s ˝H´1 s pαq at any point α P p0, 1q such that s H s ˝s H ´1 s pαq " α. The curve ROC s reflects the ability of s to discriminate between pairs with same labels and pairs with different labels: the stochastically smaller than H s the distribution G s is, the higher the associated ROC curve. Note that it corresponds to the type I error vs power plot of the statistical test ItspX, X 1 q ą tu when the null hypothesis stipulates that X and X 1 have different marginal distribution (i.e. Y ‰ Y 1 ). A similarity measure s 1 is said to be more accurate than another similarity s 2 when ROC s2 pαq ď ROC s1 pαq for any α P p0, 1q.

Pointwise ROC optimization. In many applications, one is interested in finding a similarity function which optimizes the ROC curve at a particular point α P p0, 1q. The superlevel sets of similarity functions in S ˚define the solutions of pointwise ROC optimization problems in this context. In the above framework, it indeed follows from Neyman Pearson's lemma that the test statistic of type I error less than α with maximum power is the indicator function of the set R α " tpx, x 1 q P X 2 : ηpx, x 1 q ě Q αu, where Q α is the conditional quantile of the r.v. ηpX, X 1 q given Z " ´1 at level 1 ´α. Restricting our attention to similarity functions bounded by 1, this corresponds to the unique solution of the following problem:

max s:X 2 Ñr0,1s, borel R `psq subject to R ´psq ď α, (5.1)
where R `psq " ErspX, X 1 q | Z " `1s is referred to as positive risk and R ´psq " ErspX, X 1 q | Z " ´1s as the negative risk.

Connection to Metric Learning

We point out that the similarity learning framework described above can be equivalently described in terms of learning a dissimilarity measure (or pseudo distance metric) D : X ˆX Ñ R `. In this case, the pointwise ROC optimization problem (5.1) translates into:

min D:X 2 Ñr0,1s E " DpX, X 1 q | Z " `1‰ subject to E " DpX, X 1 q | Z " ´1‰ ě 1 ´α. (5.2)
A large variety of practical similarity and distance metric learning algorithms have been proposed in the literature, all revolving around the same idea that a good similarity function should output large scores for pairs of points in the same class, and small scores for pairs with different label. They differ from one another by the class of metric/similarity functions considered, and by the kind of objective function they optimize (see Bellet et al., 2015a, for a comprehensive review).

In any case, ROC curves are commonly used to evaluate metric learning algorithms when the number of classes is large (see for instance [START_REF] Guillaumin | Is that you? metric learning approaches for face identification[END_REF]; [START_REF] Köstinger | Large scale metric learning from equivalence constraints[END_REF]; [START_REF] Shen | Positive Semidefinite Metric Learning Using Boosting-like Algorithms[END_REF], which makes our framework very relevant in practice. Several popular algorithms optimize an empirical version of Problems (5.1)-( 5.2), often in their unconstrained version as in [START_REF] Liu | Constrained metric learning via distance gap maximization[END_REF] and [START_REF] Xie | Large scale distributed distance metric learning[END_REF]. We argue here in favor of the constrained version as the parameter α has a direct correspondence with the point ROC S pαq of the ROC curve, unlike the unconstrained case presented below.

A remarkable fact is that the superlevel set R α of the pairwise posterior probability ηpx, x 1 q is the measurable subset R of X 2 that minimizes the cost-sensitive classification risk:

pp1 ´Qα qP pX, X 1 q R R | Z " `1( `p1 ´pqQ αP pX, X 1 q P R | Z " ´1( , (5.3)
where p " PtZ " `1u "

ř K k"1 p 2 k .
Hence, the solution of Eq. ( 5.1) corresponds to the solution of the minimization of Eq. ( 5.3). Notice however that the asymmetry factor, namely the quantile Q α, is unknown in practice, just like the r.v. ηpX, X 1 q. For this reason, one typically considers the problem of maximizing: R `psq ´λR ´psq,

(5.4) for different values of the constant λ ą 0. The performance in terms of ROC curve can only be analyzed a posteriori, and the value λ thus needs to be tuned empirically by model selection techniques. We focus here on the constrained version as the parameter α has a direct correspondence with the point ROC s pαq of the ROC curve, unlike the unconstrained version.

Interestingly, our framework sheds light on MMC, the seminal metric learning algorithm of [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF] originally designed for clustering with side information. MMC solves the empirical version of (5.2) with α fixed to 0. This is because MMC optimizes over a class of distance functions with unbounded values, hence modifying α does not change the solution (up to a scaling factor). We note that by choosing a bounded family of distance functions, one can use the same formulation to optimize the pointwise ROC curve.

Statistical Guarantees for Generalization

Pointwise ROC optimization problems have been investigated from a statistical learning perspective by [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF] and Clémençon and Vayatis (2010) in the context of binary classification, which we presented in Chapter 3. The major difference with the present framework lies in the pairwise nature of the quantities appearing in Problem (5.1) and, consequently, in the complexity of its empirical version.

In this section, we present the extension of those results to the problem of learning similarities. First, we deal with the derivation of uniform learning rates, i.e. learning rates that do not depend on the data distribution, which relies on the extension of usual concentration inequalities for U -statistics. Second, we present data-dependent fast rates, which requires less restrictive assumptions on the data distribution than in bipartite ranking, which stems from the variancereducing property of the Hájek projection of a U -statistic.

Uniform Rates for Pointwise ROC Optimization

Natural statistical estimates for the positive risk R `psq and the negative risk R ´psq (5.1) computed on the training sample D n " tpX 1 , Y 1 q, . . . , pX n , Y n qu are given by:

p R ǹ psq " 1 n `ÿ 1ďiăjďn spX i , X j q ¨ItY i " Y j u, (5.5) p R ń psq " 1 n ´ÿ 1ďiăjďn spX i , X j q ¨ItY i ‰ Y j u, (5.6)
where n `" ř 1ďiăjďn ItY i " Y j u " npn ´1q{2 ´n´. It is important to note that these quantities are not i.i.d. averages, since several pairs involve each i.i.d. sample. This breaks the analysis presented in Chapter 3. We can however observe that:

U ǹ psq :" 2n ǹpn ´1q ¨p R ǹ psq and U ń psq :" 2n ńpn ´1q ¨p R ń psq (5.7)
are U -statistics of degree two -averages of a function of the pairs of an i.i.d. sample (see Chapter 4) -with respective symmetric kernels:

h `ppx, yq, px 1 , y 1 qq " spx, x 1 q ¨Ity " y 1 u and h ´ppx, yq, px 1 , y 1 qq " spx, x 1 q ¨Ity ‰ y 1 u.

We will therefore be able to use existing representation tricks to derive concentration bounds for U -processes (collections of U -statistics indexed by classes of kernel functions), under appropriate complexity conditions, see e.g. [START_REF] Dudley | Uniform Central Limit Theorems[END_REF].

We thus investigate the generalization ability of solutions obtained by solving the empirical version of Problem (5.1), where we also restrict the domain to a subset S of similarity functions bounded by 1, and we assume S has controlled complexity (e.g. finite VC dimension). Finally, we replace the target level α by α `φ, where φ ě 0 is some tolerance parameter that should be of the same order as the maximal deviation sup sPS | p R ń psq ´R´p sq|. This leads to the following empirical problem:

max sPS p R ǹ psq subject to p R ń psq ď α `φ.
(5.8)

We are now ready to state our universal learning rate, describing the generalization capacity of solutions of the constrained optimization program (5.8) under specific conditions for the class S 0 of similarity functions and a suitable choice of the tolerance parameter Φ. This result can be established by combining Corollary 4.5 in Chapter 4, an extension of usual concentrations inequalities for empirical processes to U -statistics, with the derivations of Clémençon and Vayatis (2010, Theorem 10 therein).

Theorem 5.1. Suppose that the proposed family S is a VC-major class of functions of finite VC-dimension V , and that spx, x 1 q ď 1 for all s P S and any px, x 1 q P X 2 . Assume also that there exists a constant κ P p0, 1q such that κ ď ř k"1 p 2 k ď 1 ´κ. For all δ P p0, 1q and n ą 1, set

φpδ, nq " 2Cκ ´1c V n `2κ ´1p1 `κ´1 q c logp3{δq n ´1 ,
where C is a positive universal constant, and consider a solution p s n of the constrained minimization problem (5.8) with φ " φpn, δ{2q. Then, for any δ P p0, 1q, we have simultaneously, @n ě 1 `4κ ´2 logp3{δq, w.p. ě 1 ´δ: R `pp s n q ě ROC s ˚pαq ´φpn, δ{2q ´!ROC s ˚pαq ´sup sPS: R ´psqďα R `psq ) , (5.9) and R ´pp s n q ď α `φpn, δ{2q.

(5.10) Combining this proposition with Corollary 4.5 in Chapter 4, gives that: @n ą 1, w.p. ě 1 ´δ,

sup sPS p R ǹ psq ´R`p sq ď 2C p c V n `3 p c logp3{δq n ´1 .
Similarly, we obtain, with q " 1 ´p:

sup sPS p R ń psq ´R´p sq ď q ´1 2n ńpn ´1q ´q `q´1 sup sPS U ń psq ´ErU ń psqs ,
which gives with the exact same reasoning that: @n ą 1, w.p. ě 1 ´δ,

sup sPS p R ń psq ´R´p sq ď 2C q c V n `3 q c logp3{δq n ´1 .
which gives the right order of convergence. The proof is then finished by following the proof of Theorem 10 in Clémençon and Vayatis (2010).

The last term on the right hand side of (5.9) should be interpreted as the bias of the statistical learning problem (5.8), which depends on the richness of class S. This term vanishes when Itpx, x 1 q P R αu belongs to S. Choosing a class yielding a similarity rule of highest true positive rate with large probability can be tackled by means of classical model selection techniques, based on resampling methods or complexity penalization.

Except for the minor condition stipulating that the probability of occurrence of "positive pairs" ř K k"1 p 2 k stays bounded away from 0 and 1, the generalization bound stated in Theorem 5.1 holds whatever the probability distribution of pX, Y q. Beyond such universal results, we investigate situations where rates faster than Op1{

? nq can be achieved by solutions of (5.8), which are presented in the section below.

Fast Rates for Pointwise ROC Optimization

Such fast rates results exist for binary classification under the so-called Mammen-Tsybakov noise condition, as presented in Chapter 2. By means of a variant of the Bernstein inequality for U -statistics, we can establish fast rate bounds under the following condition on the data distribution.

Assumption 5.2. Let α P p0, 1q. There exist a P r0, 1s and a constant c ą 0 such that, almost surely,

E X 1 " ηpX, X 1 q ´Qα ´a‰ ď c.
This noise condition is similar to that introduced by [START_REF] Mammen | Asympotical minimax recovery of the sets with smooth boundaries[END_REF] for the binary classification framework, except that the threshold 1{2 is replaced here by the conditional quantile Q α. It characterizes "nice" distributions for the problem of ROC optimization at point α: it essentially ensures that the pairwise posterior probability is bounded away from Q α with high probability.

The condition expressed in Assumption 5.2 is weaker than that for standard bipartite ranking, i.e. Assumption 3.6, which is possible thanks to the variance reducing property of Hájek projections. For example, the noise condition is automatically fulfilled for any a P p0, 1q when, for almost every point x with respect to the measure induced by X, ηpx, X 1 q has an absolutely continuous distribution and bounded density. Intuitively, this assumption means that the problem of ranking elements modeled by X according to their similarity with an element x P A is somewhat easy (almost-surely). The implication is proven precily in the result below, with a proof that follows the same arguments as Clémençon et al. (2008) (Corollary 8).

Proposition 5.3. Assume that there exist A Ă X , PpX P Aq " 1, such that for all x P A, the random variable ηpx, Xq has an absolutely continuous distribution on r0, 1s and its density is bounded by B. Then: for any ą 0,

E X 1 " ηpX, X 1 q ´Qα ´1` ı ď 2B almost surely,
which implies that the fast rate of convergence of Theorem 5.5 applies for any a P p0, 1q.

Proof. Let x P A and h x be the density of ηpx, Xq, with h x ď B. Hence, for any a P p0, 1q,

E X 1 " |ηpx, Xq ´Qα | ´aı " ż 1 0 |z ´Qα | ´ah x pzqdz, ď B ˜ż Q α 0 pQ α ´zq ´adz `ż 1 Q α pz ´Qα q ´adz ¸, ď B ˆQ˚1´a α 1 ´a `p1 ´Qα q 1´a 1 ´a ˙, ď 2B 1 ´a .
The fast bounds result is based on the second Hoeffding decomposition (see Proposition 4.6 in Chapter 4) of the U -statistic U ǹ (see Eq. (5.7)), with an independent analysis of the sum of i.i.d. terms and the degenerate U -statistic (see Definition 4.8 in Chapter 4) remainder. Denote by s ˚px, x 1 q " Itpx, x 1 q P R αu the optimal similarity function. We assume for simplicity that it belongs to S, but the result can be extended using the local analysis arguments presented in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (Section 5.3.5). For any s P S, the statistic:

∆ n psq " `U ǹ psq ´E " U ǹ psq ‰˘´`U ǹ ps ˚q ´E " U ǹ ps ˚q‰˘,
(5.11) is a U -statistic based on D n with kernel Q s given by:

Q s ppx, yq, px 1 , y 1 qq " I y " y 1 ( `spx, x 1 q ´s˚p x, x 1 q ˘´E " I Y " Y 1 ( `spX, X 1 q ´s˚p X, X 1 q ˘‰ .
The second Hoeffding decomposition of U -statistics, presented in Chapter 4, leads to the following decomposition of the U -statistic ∆ n psq:

∆ n psq " 2T n psq `Wn psq, (5.12) where:

T n psq " 1 n n ÿ i"1
q s pX i , Y i q and W n psq " 2 npn ´1q ÿ 1ďiăjďn p q s ppX i , Y i q, pX j , Y j qq , with: q s px, yq " E rQ s ppX, Y q, px, yqqs , p q s `px, yq, px 1 , y 1 q ˘" Q s `px, yq, px 1 , y 1 q ˘´q s px, yq ´qs px 1 , y 1 q.

While in the binary classification or bipartite ranking settings, the noise condition implied a bound on the variance of an empirical process, our weak similarity ranking noise condition bounds the variance of the Hájek projection of a U -statistic, as seen in the lemma below.

Lemma 5.4. Suppose that the assumptions of Theorem 5.1 are satisfied and that Assumption 5.2 holds true with parameter a P r0, 1s. Then, for any s P S, we have:

Varpq s pX, Y qq ď c " p1 ´Qα qppR `ps ˚q ´R`p sqq `p1 ´pqQ αpR ´psq ´R´p s ˚qq ‰ a .

(5.13)

Proof. For any s P S and u P r0, 1s, we introduce R s,u :" tpx, x 1 q | spx, x 1 q ě uu. Then, s writes as an integral of indicators of its level sets (van der Vaart and Wellner, 1996, Lemma 2.6.13):

spX, X 1 q " ż 1 0 ItpX, X 1 q P R s,u u du.

(5.14)

Additionally, we have s ˚pX, X 1 q " ItpX, X 1 q P R αu. The definition of q s implies that:

Var pq s pX, Y qq " Var " `EX 1 ,Y 1 " ItY " Y 1 upspX, X 1 q ´s˚p X, X 1 qq ‰˘2 ı .
Therefore, the fact that VarpZq ď ErZ 2 s gives:

Var pq s pX, Y qq ď E X,Y " `EX 1 ,Y 1 " ItY " Y 1 upspX, X 1 q ´s˚p X, X 1 qq ‰˘2 ı .
(5.15) Injecting Eq. (5.14) into Eq. (5.15) gives, from Fubini's theorem:

Var pq s pX, Y qq ď E X,Y « ˆż 1 0 E X 1 ,Y 1 " ItY " Y 1 u `ItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αu ˘‰ du ˙2ff .
Jensen's inequality applied to the integral over u, followed by Fubini's theorem gives:

Var pq s pX, Y qq ď ż 1 0 E X,Y " `EX 1 ,Y 1 " ItY " Y 1 u `ItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αu ˘‰˘2 ı du.
We introduce γ (we hide dependencies for readability reasons) as:

γ : " ItY " Y 1 u `ItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αu "

ItpX, X 1 q P R s,u ∆R αu `1 ´2ItpX, X 1 q P R αu ˘ItY " Y 1 u, then |γ| ď ItpX, X 1 q P R s,u ∆R αu and the bound implies that:

Var pq s pX, Y qq ď ż 1 0 E X " `EX 1 " ItpX, X 1 q P R s,u ∆R αu ‰˘2 ı du.
(5.16)

Cauchy-Schwartz's inequality gives:

E X 1 " ItpX, X 1 q P R s,u ∆R αu ‰ ď ´EX 1 r ηpX, X 1 q ´Qα ´as ¯1 2 ˆ`E X 1 " ItpX, X 1 q P R s,u ∆R αu ¨ ηpX, X 1 q ´Qα a ‰˘1 2 ,
and Assumption 5.2 implies:

E X 1 " ItpX, X 1 q P R s,u ∆R αu ‰ ď ? c ˆ`E X 1 " ItpX, X 1 q P R s,u ∆R αu ηpX, X 1 q ´Qα a ‰˘1 2 .
Plugging that last result into Eq. (5.16) gives:

Var pq s pX, Y qq ď c ¨ż 1 0 E X " E X 1 " ItpX, X 1 q P R s,u ∆R αu ¨ ηpX, X 1 q ´Qα a ‰‰ du.
Since a P r0, 1s, the function x Þ Ñ x a is concave. Therefore, Jensen's inequality applied for integration with respect to X 1 , X and u sequentially, gives:

Var pq s pX, Y qq ď c ˆż 1 0 E " ItpX, X 1 q P R s,u ∆R αu ¨ ηpX, X 1 q ´Qα ‰ du ˙a .
(5.17)

The right-hand-side of Eq. (5.17) is very similar to the expression of the excess risk in binary classification presented in Eq. (2.1). We now relate it to the right-hand side of Eq. (5.13):

R `psq ´R`p s ˚q " p1{pq ¨E " ItY " Y 1 upspX, X 1 q ´s˚p X, X 1 qq ‰ , " p1{pq ¨ż 1 0 E " ηpX, X 1 qpItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αuq ‰ du,
by Fubini's theorem. Simple calculus gives:

R `psq ´R`p s ˚q " p1{pq ¨ż 1 0 E "`1 ´2 ¨ItpX, X 1 q P R αu ˘¨ηpX, X 1 q ¨ItpX, X 1 q P R s,u ∆R αu ‰ du,
The definition of Q α implies that:

`1 ´2 ¨ItpX, X 1 q P R αu ˘`ηpX, X 1 q ´Qα ˘" ´ ηpX, X 1 q ´Qα , and we have:

R `psq ´R`p s ˚q " ´p1{pq ¨ż 1 0 E " ηpX, X 1 q ´Qα ¨ItpX, X 1 q P R s,u ∆R αu ‰ du (5.18) `pQ α{pq ¨ż 1 0 E " ItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αu ‰ du, since:
ItpX, X 1 q P R s,u ∆R αu ¨p1 ´2 ¨ItpX, X 1 q P R αuq " ItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αu.

Eq. (5.14) combined with Fubini's theorem gives:

ż 1 0 E " ItpX, X 1 q P R s,u u ´ItpX, X 1 q P R αu ‰ du " ErspX, X 1 qs ´Ers ˚pX, X 1 qs, (5.19)
" ppR `psq ´R`p s ˚qq `p1 ´pqpR ´psq ´R´p s ˚qq.

Combining Eq. (5.19), Eq. (5.18) and Eq. (5.17) and rearranging the terms completes the proof.

Using this property, we can derive fast learning rates.

Theorem 5.5. Suppose that the assumptions of Theorem 5.1 and Lemma 5.4 are satisfied, that the class S is finite of size M , and that the optimal similarity rule s αpx, x 1 q " Itpx, x 1 q P R αu belongs to S. Fix δ ą 0. Then, there exists a constant C 1 , depending on δ, κ, Q α, a, c and M such that, for any n ą 1, w.p. ě 1 ´δ, ROC s ˚pαq ´R`p p s n q ď C 1 n ´p2`aq{4 and R ´pp s n q ď α `2φpn, δ{2q.

Proof. The component W n psq is a degenerate U -statistic (see Definition 4.8 in Chapter 4), meaning that, for all px, yq, E rp q s ppx, yq, pX, Y qqs " 0 almost-surely.

The supremum of the family tW n psqu sPS is controlled using the results in Chapter 4, precisely Proposition 4.9 for finite families and Proposition 4.10 for more general classes of families. Precisely, from de la Pena and Giné (1999) (Theorem 4.1.13 therein), we have since W n psq is a degenerate U -statistic, that: w.p. ě 1 ´δ,

sup sPS |W n psq| ď 2C logp4M {δq n , (5.20)
where C is an universal constant. This result shows that the second term in the decomposition (5.12) is uniformly negligible with respect to the first term T n psq.

We consider Lemma 5.4 as an analogue of Lemma 11 of Clémençon and Vayatis (2010) where C 1 " 2C `4{3.

The proof of Theorem 5.1 may be adapted to the finite class setting. Formally, introducing the tolerance term:

φpn, δq 1 " 2κ ´1p1 `κ´1 q c logp2pM `1q{δq n ´1 ,
where M is the cardinal of the proposition class S, we have simultaneously w.p ě 1 ´δ, that: R `pp s n q ě sup sPS: R ´psqďα R `psq ´φpn, δ{2q 1 and R ´pp s n q ď α `φpn, δ{2q 1 , (5.23) sup sPS R ǹ psq ´R`p sq ď φpn, δ{2q 1 .

(5.24)

Eq. ( 5.24) implies that s ˚satisfies the constraint of the ERM problem Eq. (5.8), hence R ǹ pp s n q Ŕǹ ps ˚q ě 0. It follows that :

∆ n pp s n q "
2n ǹpn ´1q `Rǹ pp s n q ´Rǹ ps ˚q˘`p `R`p s ˚q ´R`p p s n q ˘, ě p `R`p s ˚q ´R`p p s n q ˘.

(5.25)

Let p s n be a solution of Eq. (5.8) with φpn, δ 1 {2q 1 , where δ 1 " 2pM `1qδ{p9M `4q. Introducing K δ,M " p9M `4q{δ, we combine Lemma 5.4, Eq. (5.22), Eq. (5.23) and Eq. (5.25), to obtain that: w.p. ě 1 ´δ,

p `R`p s ˚q ´R`p p s n q ď c 2c log K δ,M n ´rp1 ´Qα qppR `ps ˚q ´Rpp s n qqs a{2 `rQ αp1 ´pqφpn, δ 1 {2q 1 s a{2 C1 log K δ,M n .
(5.26)

The highest order term on the right-hand side is in Opn ´1{2 φpn, δ 1 {2q 1a{2 q which is Opn ´p2`aq{4 q.

Eq. (5.26) is a fixed point equation in R `ps ˚q ´R`p p s n q. Finding an upper bound on the solution of this fixed-point equation is done by invoking Lemma 7 of [START_REF] Cucker | Best choices for regularization parameters in learning theory: On the bias-variance problem[END_REF], which can be found as Lemma 2.27 in Chapter 2. Applying Lemma 2.27 to Eq. (5.26) concludes the proof.

Remark 5.6. We state here the result for the case where S is of finite cardinality M . Proving this result for more general classes of functions S can be tackled by the localization argument expressed in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (pages 341-346 therein).

The proof is based on the same argument as that of Clémençon and Vayatis (2010) (Theorem 12), except that it involves a sharp control of the fluctuations of the U -statistic estimates of the true positive rate excess ROC s ˚pαq ´R`p sq over the class S. The reduced variance property of U -statistics plays a crucial role in the analysis, which essentially relies on the Hoeffding decomposition [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF].

Illustration of the Fast Rates

In this section, we provide numerical evidence of the fast rates of Theorem 5.5, which shows that when Assumption 5.2 is verified, faster rates of generalization can be achieved. Showing the existence of fast rates experimentally requires us to design a problem for which η satisfies Assumption 5.2, which is not trivial due to the pairwise nature of the involved quantities. Such practical evidence of fast rates is rarely found in the literature.

We consider a simple scenario where we focus on pointwise ROC optimization at level α P p0, 1q, and we have X " r0, 1s, F " 1 (X follows an uniform distribution over r0, 1s), K " 2, p 1 " p 2 " 1{2 and for any x P r0, 1{2s, F 1 px `1{2q " 2 ´F1 pxqi.e. F 1 symmetric in the point p1{2, 1qand:

F 1 pxq " # 2C if x P r0, ms, 1 ´|1 ´2x| p1´aq{a if x P pm, 1{2s,
where C P p0, 1{2q and m P p0, 1{2q satisfy:

C " 1 2 ´?1 ´2α 4m `ap1 ´2mq a ´1 4m .
The complex expression of F 1 arises naturally when trying to design a suitable distribution. The variable m was fixed, and C is a solution of:

ż 1´ηpx,x 1 qą 1 2 p1 ´ηpx, x 1 qq dx dx 1 " α 2 , (5.27)
since we fixed Q α " 1{2. Eq. (5.27) is a simple quadratic equation since the posterior probability η expresses in F 1 as: ηpx, x 1 q " 1 2 `1 2 pF 1 pxq ´1q `F1 px 1 q ´1˘.

(5.28)

For C to satisfy 0 ă C ă 1{2, the variables pm, α, aq need to be restricted, as shown by Fig. 5.1. Precisely, we see that experimental parameters pm, α, aq are valid if their corresponding point is below the orange curve and above the dark blue curve. We see that excessively low values of m restrict severely the possible values of pα, aq. The points should be under the red curve if possible, since then F 1 is increasing and assures that P p|ηpX, X 1 q ´Qα | ď tq is smooth on a larger neighborhood of 0. Empirical distributions of |ηpX, X 1 q ´Qα | are displayed in Fig. 5.2 for α " 0.26, m " 0.35. Figure 5.4 shows example distributions of the data.

Finally, we have:

P ` ηpX, X 1 q ´Qα ď t ˘" P ` pF 1 pxq ´1qpF 1 px 1 q ´1q ď 2t ˘, " ż I ! |2x ´1| ¨|2x ´1| ď 2p4tq a 1´a ) dx dx 1 , " 2p4tq a 1´a " 1 ´logp2q ´a 1 ´a logp4tq  .
(5.29)

When t is small enough, Eq. (5.29) gives P p|ηpX, X 1 q ´Qα | ď tq of the order ´ta{p1´aq logptq.

Ignoring the log term, the result shows that our distribution satisfies the strong Mammen-Tsybakov assumption (Assumption 3.6 of Chapter 3) and not our weak assumption (Assumption 5.2). Due to the logarithm term in the noise condition, we expect the generalization speeds to be slightly worse than Opn ´p2`aq{4 q.

Now that the distribution of the data is set, we need to set the class of functions on which we optimize Eq. (5.8). For all t P r0, 1s, we define the proposed family S as px, x 1 q Þ Ñ Itpx, x 1 q P S t u | 0 ď t ď 1 ( where S t " r0, ts 2 Y r1 ´t, 1s 2 for any t P r0, 1s. The sets S t are illustrated by Fig. 5.3. The risks R `psq, R ´psq of an element s of S that is an indicator of a set S t can be expressed in closed form with the expression of η. Indeed: R `psq " 2 ż St ηpx, x 1 q dx dx 1 " λpS t q `żSt pF 1 pxq ´1q `F1 px 1 q ´1˘d x dx 1 , using Eq. (5.28), where λ is the usual area measure (i.e. Lebesgue measure over r0, 1s 2 ). The right-hand side integral is easily developed since it is a sum of integrals over squares included in r0, 1s ˆr0, 1s. We now describe the processus of choosing an optimal empirical function p s n for a set of observations pX i , Y i q. For all pairs X i , X j , we derive the quantity S i,j " min pmax p1 ´Xi , 1 ´Xj q , max pX i , X j qq. Let σ t1, . . . , npn ´1q{2u Þ Ñ t1, . . . , nu 2 be the function that orders the quantities S i,j increasingly, i.e. S σp1q ď ¨¨¨ď S σpnpn´1q{2q . Choosing an optimal empirical function p s n in S requires to solve the pointwise ROC optimization problem for pS i,j , Z i,j q iăj and proposed functions:

P n (X ∈ bin | Y = i), i ∈ {1, 2} a = 0.8 µ 1 (x)
" x Þ Ñ I " x ď S σpiq `Sσpi`1q 2 * ˇˇ0 ď i ď npn ´1q 2 * ,
where S σp0q " 0 and S σppnpn´1q{2q`1q " 1 by convention. It can be solved in O `n2 log n ˘time and we always neglect the tolerance parameter Φ, i.e. we set Φ " 0.

For all a P t1{10, . . . , 9{10u, we generate 512 data points and compute the generalization error ROC S ˚pαq ´R`p p s n q for the n first data points, where n P t64, 128, 256, 512u, and repeat the operation 1000 times. We introduce Q a,n as the 90-quantile of the 1000 realizations of ROC S ˚pαq ´R`p p s n q for a given pn, aq. The coefficients pC a , D a q of the regression Q a,n " D a `Ca ˆlogpnq are estimated. Fig. 5.6 shows the learning speeds C a given the noise parameters a's. The estimation of the C a 's is illustrated by Fig. 5.5 and Fig. 5.6 summarizes the experiments for the case α " 0.26, m " 0.35 and a P r0.1, 0.9s. There is a clear downward trend when a increases, illustrating the fast rates in practice.

Solving Pointwise ROC Optimization

Though the formulation of pointwise ROC optimization is natural, this constrained optimization problem is very difficult to solve in practice, as discussed at length in [START_REF] Vogel | A probabilistic theory of supervised similarity learning for pointwise ROC curve optimization[END_REF]. As far as we know, the only papers that tackle the problem directly are [START_REF] Scott | Learning minimum volume sets[END_REF] (Section 7.2), which is based on a fixed partition of the input space, or [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF] (Section VI-B), which is based on recursive partitioning of the input space. In Chapter 7, we propose a gradient-descent approach for learning to optimize for a specific point of the ROC curve.

These difficulties suggest the extension to the similarity ranking framework of the TreeRank approach for ROC optimization (Clémençon and Vayatis (2009) and [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF]), recalled below. Indeed, in the standard (non pairwise) statistical learning setup for bipartite ranking, whose probabilistic framework is the same as that of binary classification and stipulates that training data are i.i.d. labeled observations, this recursive technique builds (piecewise constant) scoring functions s, whose accuracy can be guaranteed in terms of sup norm, i.e. for which D 8 ps, s ˚q can be controlled where:

D p ps, s ˚q " ||ROC s ´ROC ˚|| p ,
with s ˚P S ˚and p P r1, `8s. It is the essential purpose of the next section to prove that this remains true when the training observations are of the form tppX i , X j q, Z i,j q : 1 ď i ă j ď nu, where Z i,j " 2ItY i " Y j u ´1 for 1 ď i ă j ď n, and are thus far from being independent. Regarding the implementation of TreeRank for pairs, attention should be paid to the fact that the splitting rules for recursive partitioning of the space X ˆX must ensure that the decision functions produced by the algorithm fulfill the symmetric property.

The TreeRank Algorithm for Learning Similarities

Because they offer a visual model summary in the form of an easily interpretable binary tree graph, decision trees, see e.g. [START_REF] Breiman | Classification and Regression Trees[END_REF] or [START_REF] Quinlan | Induction of Decision Trees[END_REF], remain very popular among practitioners. In general, predictions are computed through a hierarchical combination of elementary rules comparing the value taken by a (quantitative) component of the input information (the split variable) to a certain threshold (the split value). In contrast to (supervised) learning problems such as classification/regression, which are of local nature, predictive rules for a global problem such as similarity learning cannot be described by a simple (tree-structured) partition of X ˆX : the (symmetric) cells corresponding to the terminal leaves of the binary decision tree must be sorted in order to define a similarity function.

TreeRank on a Product Space

We define a similarity tree as a binary tree whose leaves all correspond to symmetric subsets C of the product space X ˆX (i.e. @px, x 1 q P X 2 ; px, x 1 q P C ô px 1 , xq P C) and is equipped with a 'leftto-right' orientation, that defines a tree-structured collection of similarity functions. Incidentally, the symmetry property makes it a specific ranking tree, using the terminology introduced in Clémençon and Vayatis (2009). The root node of a tree T J of depth J ě 0 corresponds to the whole space X ˆX : C 0,0 " X 2 , while each internal node pj, kq with j ă J and k P t0, . . . , 2 j ´1u represents a subset C j,k Ă X 2 , whose left and right siblings respectively correspond to (symmetric) disjoint subsets C j`1,2k and C j`1,2k`1 such that C j,k " C j`1,2k Y C j`1,2k`1 . Equipped with the left-to-right orientation, any subtree T Ă T J defines a preorder on X 2 : the degree of similarity being the same for all pairs px, x 1 q lying in the same terminal cell of T . Figure 5.7 represents a fully grown tree of depth 3 with its associated scores.

Figure 5.7: A piecewise constant similarity function described by an oriented binary subtree T . For any pair px, x 1 q P X 2 , the similarity score s T px, x 1 q can be computed very fast in a top-down manner using the heap structure: starting from the initial value 2 J at the root node, at each internal node C j,k , the score remains unchanged if px, x 1 q moves down to the left sibling and one subtracts 2 J´pj`1q from it if px, x 1 q moves down to the right.

The similarity function related to the oriented tree T can be written as:

@px, x 1 q P X 2 , s T px, x 1 q " ÿ C j,k : terminal leaf of T 2 J ˆ1
´k 2 j ˙¨Itpx, x 1 q P C j,k u.

(5.30)

Observe that its symmetry results from that of the C j,k 's. The ROC curve of the similarity function s T px, x 1 q is the piecewise linear curve connecting the knots: p0, 0q and

˜k ÿ l"0 HpC j,l q, k ÿ l"0 GpC j,l q ¸for all terminal leaf C j,k of T ,
denoting by H and G the conditional distribution of pX, X 1 q given Z " ´1 and Z " `1, respectively. Setting p " PtZ " `1u "

ř k p 2 k , we have G " p1{pq ř k p 2 k ¨Fk b F k and H " p1{p1 ´pqq ř k‰l p k p l ¨Fk b F l .
A statistical version can be computed by replacing the HpC j,l q or GpC j,l q by their empirical counterpart.

The TreeRank algorithm, a bipartite ranking technique optimizing the ROC curve in a recursive fashion, has been introduced in Clémençon and Vayatis (2009) and its properties have been investigated in [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF] at length. Its output consists of a tree-structured scoring rule (5.30) whose ROC curve can be viewed as a piecewise linear approximation of ROC ˚obtained by a Finite Element Method with implicit scheme and is proved to be nearly optimal in the D 1 sense under mild assumptions. The growing stage is performed as follows. At the root, one starts with a constant similarity function s 1 px, x 1 q " Itpx, x 1 q P C 0,0 u " 1 and after m " 2 j `k iterations, 0 ď k ă 2 j , the current similarity function is:

s m px, x 1 q " 2k´1 ÿ l"0 pm ´lq ¨Itpx, x 1 q P C j`1,l u `2j ´1 ÿ l"k
pm ´k ´lq ¨Itpx, x 1 q P C j,l u and the cell C j,k is split so as to form a refined version of the similarity function,

s m`1 px, x 1 q " 2k ÿ l"0 pm ´lq ¨Itpx, x 1 q P C j`1,l u `2j ´1 ÿ l"k`1
pm ´k ´lq ¨Itpx, x 1 q P C j,l u, namely, with maximum (empirical) AUC. Therefore, it happens that this problem boils down to solve a cost-sensitive binary classification problem on the set C j,k , see subsection 3.3 in [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF]. Indeed, one may write the AUC increment as AUCps m`1 q ´AUCps m q " 1 2 HpC j,k qGpC j,k q ˆp1 ´ΛpC j`1,2k | C j,k qq, (5.31)

where ΛpC j`1,2k | C j,k q :" GpC j,k zC j`1,2k q{GpC j,k q `HpC j`1,2k q{HpC j,k q.

Setting p " GpC j,k q{pHpC j,k q `GpC j,k qq, the crucial point of the TreeRank approach is that the quantity 2pp1 ´pqΛpC j`1,2k | C j,k q can be interpreted as the cost-sensitive error of a classifier on C j,k predicting positive label for any pair lying in C j`1,2k and negative label fo all pairs in C j,k zC j`1,2k with cost p (respectively, 1 ´p) assigned to the error consisting in predicting label `1 given Z " ´1 (resp., label ´1 given Z " `1), balancing thus the two types of error. Hence, at each iteration of the similarity tree growing stage, the TreeRank algorithm calls a cost-sensitive binary classification algorithm, termed LeafRank, in order to solve a statistical version of the problem above (replacing the theoretical probabilities involved by their empirical counterparts) and split C j,k into C j`1,2k and C j`1,2k`1 . As described at length in [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF], one may use cost-sensitive versions of celebrated binary classification algorithms such as CART or SVM for instance as LeafRank procedure, the performance depending on their ability to capture the geometry of the level sets R α of the posterior probability ηpx, x 1 q. As highlighted above, in order to apply the TreeRank approach to similarity learning, a crucial feature the LeafRank procedure implemented must have is the capacity to split a region in subsets that are both stable under the reflection px, x 1 q P X 2 Þ Ñ px 1 , xq. This point is discussed in the next section. Rate bounds for the TreeRank method in the sup norm sense are also established therein in the statistical framework of similarity learning, when the set of training examples tppX i , X j q, Z i,j qu iăj is composed of non independent observations with binary labels, formed from the original multi-class classification dataset D n . From a statistical perspective, a learning algorithm can be derived from the recursive approximation procedure recalled in the previous section, simply by replacing the quantities HpCq and GpCq, by their respective (σ " ´1 and σ " `1) empirical counterparts based on the dataset D n :

p F σ,n pCq " 1 n σ ÿ iăj
ItpX i , X j q P C, Z i,j " σ1u, (5.32)

with n σ " p2{pnpn ´1qqq ř iăj ItZ i,j " σ1u, C Ă X ˆX any borelian. Observe incidentally that the quantities (5.32) are by no means i.i.d. averages, but take the form of ratios of U -statistics of degree two (i.e. averages over pairs of observations, cf [START_REF] Lee | U -statistics: Theory and practice[END_REF]), see section 3 in [START_REF] Vogel | A probabilistic theory of supervised similarity learning for pointwise ROC curve optimization[END_REF]. For this reason, a specific rate bound analysis (ignoring bias issues) guaranteeing the accuracy of the TreeRank approach in the similarity learning framework is carried out in the following subsections.

The symmetry property of the function (5.30) output by the learning algorithm is directly inherited from that of the candidate subsets C P A of the product space X ˆX among which the C d,k 's are selected. We new explain at length how to perform the optimization step Eq. (5.31) for similarity ranking ((3.15) for its bipartite ranking version) in practice in the similarity learning context. As recalled above, maximizing (5.31) boils down to finding the best classifier on C d,k Ă X 2 of the form:

g C|C d,k px, x 1 q " Itpx, x 1 q P Cu ´Itpx, x 1 q P CzC d,k u with C Ă C d,k , C P A,
in the empirical AUC sense, that is to say minimizing a statistical version of the cost-sensitive classification error based on tppX i , X j q, Z i,j q : 1 ď i ă j ď n, pX i , X j q P C d,k u:

ΛpC | C d,k q " Ptg C|C d,k pX, X 1 q " 1 | Z " ´1u PtpX, X 1 q P C d,k | Z " ´1u `Ptg C|C d,k pX, X 1 q " ´1 | Z " 1u PtpX, X 1 q P C d,k | Z " 1u ,
which is straightforwardly estimated from data in the same manner as in Chapter 3, see Section 3.2.4.

Learning a Symmetric Function

In [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF], it is highlighted that, in the standard ranking bipartite setup, any (cost-sensitive) classification algorithm (e.g. Neural Networks, CART, Random Forest, SVM, nearest neighbours) can be possibly used for splitting, whereas, in the present framework, classifiers are defined on product spaces and the symmetry issue must be addressed. For simplicity, assume that X is a subset of the space R q , q ě 1, whose canonical basis is denoted by pe 1 , . . . , e q q. Denote by P V px, x 1 q the orthogonal projection of any point px, x 1 q in R q ˆRq equipped with its usual Euclidean structure onto the subspace V " Spanppe 1 , e 1 q, . . . , pe q , e q qq. Let W be V 's orthogonal complement in R q ˆRq . For any px, x 1 q P X 2 , write x " px 1 , . . . , x q q and x 1 " px 1 1 , . . . , x 1 q q. Introduce f px, x 1 q :" pf 1 px, x 1 q, . . . , f 2q px, x 1 qq P R 2q , which satisfies, for any j P t1, . . . , qu: f j px, x 1 q :" px j `x1 j q{ ? 2 and f j`q px, x 1 q " sgn ´xlpx,x 1 q ´x1 lpx,x 1 q ¯px j ´x1 j q{ ? 2, (5.33)

where lpx, xq :" arg max lPt1,...,qu |x l ´x1 l | and sgnp¨q : R Ñ t´1, 0, `1u is the sign function. Observe that, by construction, f px, x 1 q " f px 1 , xq for all px, x 1 q P X 2 . The first q components of f px, x 1 q are the coordinates of the projection P V px, x 1 q of px, x 1 q onto the subspace V in an orthonormal basis of V (say tp1{ ? 2qpe 1 , e 1 q, . . . , p1{ ? 2qpe q , e q qu for instance) its last components are formed by a simple data-dependent transformation of the coordinates of the projection P W px, x 1 q " ppx 1 ´x1 1 q{ ? 2, . . . , px q ´x1 q q{ ? 2q of px, x 1 q onto W expressed in a given orthonormal basis (say tp1{ ? 2qpe 1 , ´e1 q, . . . , p1{ ? 2qpe q , ´eq qu for instance). The following lemma guarantees useful properties of our symmetric transformation of the input space for the derivation of symmetric classifiers.

Lemma 5.7. Introduce f defined in Eq. (5.33). Let s : X 2 Ñ R. Then, s is symmetric if and only if there exists k : R q ˆRq `Ñ R such that: @px, x 1 q P X 2 , spx, x 1 q " pk ˝f qpx, x 1 q.

Proof. We first prove the implication. Assume that s is symmetric. Let px, x 1 q P X 2 . Observe that:

x " ? 2 2 " P V px, x 1 q `PW px, x 1 q ‰ and x 1 " ? 2 2

" P V px, x 1 q ´PW px, x 1 q ‰ .
We denote by f p1q px, x 1 q (resp. f p2q px, x 1 q) the first (resp. last) q components of f px, x 1 q. Denote by γpx, x 1 q :" sgnpx lpx,xq ´x1 lpx,x 1 q q, observe that P W px, x 1 q " γpx, x 1 q¨f p2q px, x 1 q. If γpx, x 1 q " `1, we have:

spx, x 1 q " s ˆ?2 2 " f p1q px, x 1 q `f p2q px, x 1 q ı , ? 2 2 " f p1q px, x 1 q ´f p2q px, x 1 q ı ˙, (5.34)
If γpx, x 1 q " ´1, the symmetry of s implies that:

spx, x 1 q " spx 1 , xq " s ˆ?2 2 " f p1q px, x 1 q `f p2q px, x 1 q ı , ? 2 2 " f p1q px, x 1 q ´f p2q px, x 1 q ı ˙.
Finally, if γpx, x 1 q " 0, then x " x 1 and Eq. (5.34) is also true. We have proven from the symmetry of s implies that we can write spx, x 1 q as Eq. (5.34), thus that for any x, x 1 P X 2 , we have spx, x 1 q " pk ˝f qpx, x 1 q.

We now prove the converse. Assume that there exist k : R q ˆRq `Ñ R such that @px, x 1 q P X 2 , spx, x 1 q " pk˝f qpx, x 1 q. However for any px, x 1 q P X 2 , f px 1 , xq " f px, x 1 q, thus f is symmetric, and s is symmetric.

In order to get splits that are symmetric w.r.t. the reflection px, x 1 q Þ Ñ px 1 , xq, we propose to build directly classifiers of the form pk ˝f qpx, x 1 q. In practice, this splitting procedure referred to as Symmetric LeafRank and summarized below simply consists in using as input space R q ˆRq `rather than R 2q and considering as training labeled observations the dataset tpf pX i , X j q, Z i,j q : 1 ď i ă j ď n, pX i , X j q P C d,k u when running a cost-sensitive classification algorithm. Figure 5.8 represents a split produced by the LeafRank procedure. Just like in the original version of the TreeRank method, the growing stage can be followed by a pruning procedure, where children of a same parent node are recursively merged in order to produce a similarity subtree that maximizes an estimate of the AUC criterion, based on cross-validation usually, one may refer to section 4 in [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF] for further details. In addition, as in the standard bipartite ranking context, the Ranking Forest approach (Clémençon et al., 2013), an ensemble learning technique based on TreeRank that combines aggregation and randomization, can be implemented to dramatically improve stability and accuracy of similarity tree models both at the same time, while preserving their advantages (e.g. scalability, interpretability).

Symmetric LeafRank

• Input. Pairs tppXi, Xjq, Zi,jq : 1 ď i ă j ď n, pXi, Xjq P C d,k u lying in the (symmetric) region to be split. Classification algorithm A.

• Cost. Compute the number of positive pairs lying in the region C d,k

p "

ř 1ďiăjďn ItpXi, Xjq P C d,k , Zi,j " `1u ř 1ďiăjďn ItpXi, Xjq P C d,k u • Cost-sensitive classification.
Based on the labeled observations tpf pXi, Xjq, Zi,jq : 1 ď i ă j ď n, pXi, Xjq P C d,k u , run algorithm A with cost p for the false positive error and cost 1 ´p for the false negative error to produce a (symmetric) classifier gpx, x 1 q on C d,k .

• Output Define the subregions:

C d`1,2k " tpx, x 1 q P C d,k : gpx, x 1 q " `1u and C d`1,2k`1 " C d,k zC d`1,2k .

Generalization Ability -Rate Bound Analysis

We now prove that the theoretical guarantees formulated in the ROC space equipped with the sup norm that have been established for the TreeRank algorithm in the standard bipartite ranking setup in Clémençon and Vayatis (2009) remain valid in the similarity learning framework. The rate bound result stated below is the analogue of Corollary 1 in Clémençon and Vayatis (2009). The following technical assumptions are involved:

• the feature space X is bounded;

• α Þ Ñ ROC ˚pαq is twice differentiable with a bounded first order derivative;

• the class A is intersection stable, i.e. @pC, C 1 q P A 2 , C X C 1 P A;

• the class A has finite VC dimension V ă `8;

• we have tpx, x 1 q P X 2 : ηpx, x 1 q ě qu P A for any q P r0, 1s.

Theorem 5.8. Assume that the conditions above are fulfilled. Choose D " D n so that D n " ? log n, as n Ñ 8, and let s Dn denote the output of the Similarity TreeRank algorithm. Then, for all δ ą 0, there exists a constant λ s.t., w.p. ě 1 ´δ, we have for all n ě 2: D 8 ps Dn , s ˚q ď expp´λ ? log nq.

Proof. The proof is based on usual control of U -statistics, as provided in Corollary 4.5 of Chapter 4 and introduced in Clémençon et al. ( 2016) (Proposition 2). This crucial result permits to control the deviation of the progressive outputs of the Similarity TreeRank algorithm and those of the nonlinear approximation scheme (based on the true quantities) investigated in Clémençon and Vayatis (2009). The proof can be thus derived by following line by line the argument of Corollary 1 in Clémençon and Vayatis (2009).

This universal logarithmic rate bound may appear slow at first glance but attention should be paid to the fact that it directly results from the hierarchical structure of the partition induced by the tree construction and the global nature of the similarity learning problem. As pointed out in Clémençon and Vayatis (2009) (see Remark 14 therein), the same rate bound holds true for the deviation in sup norm between the empirical ROC curve z ROCps Dn , .q output by the TreeRank algorithm and the optimal curve ROC ˚.

Conclusion

We have introduced a rigorous probability framework to study similarity learning from the novel perspective of pairwise bipartite ranking and pointwise ROC optimization. We derived statistical guarantees for generalization that do not depend on the distribution, as well as stronger data-dependent guarantees that do. As far as we know, we have provided the first empirical illustration of fast generalization bounds using simulated data, which confirms our theoretical results. Finally, we presented an extension of the well-known TreeRank algorithm for bipartite ranking and extended its theoretical analysis to similarity ranking, using results on U -statistics.

Our study opens promising directions of future work. We are especially interested in extending our results to allow the rejection of queries from unseen classes (e.g., unknown identities) at test time (see for instance [START_REF] Bendale | Towards open world recognition[END_REF]. This could be achieved by incorporating a loss function to encourage the score of all positive pairs to be above some fixed threshold, below which we would reject the query.

The similarity ranking setting presented in this chapter is a direct mathematical translation of the 1:1 biometric identification/verification problem. While our generalization bounds give security guarantees for that problem, they do not imply new practical approaches to 1:1 verification. The next chapters focus on that aspect. Precisely, Chapter 6 focuses on alleviating the scalability issues encountered when dealing with large-scale problems, using approximation methods of U -statistics combined with distributed learning. Additionally, Chapter 7 sketches practical gradient-descent approaches to similarity ranking.

Chapter 6

Distributed U -Statistics Summary: Pairwise learning problems involves functionals on pairs of observations, as does the specific case of similarity ranking of Chapter 5, which is a direct formalization of the 1:1 biometric verification problem. These problems often involve summing a quadratic number of observations, which implies a prohibitive number of operations for any modern large scale application. Hence, they requires sensible approximations that do not sacrifice statistical accuracy. Most functionals in these problems are U -statistics, a well-studied type of statistic, for which a popular sampling-based approximation is incomplete U -statistics, presented in Section 4.4 of Chapter 4. The extension of those sampling approximations to similarity ranking (Chapter 5) is a direct consequence of Section 4.4, but it assumes that all of the data is accessible for the computation of the U -statistic. However, large-scale learning often requires data-parallelism or learns with small random subsamples of the data. For that reason, this chapter introduces techniques to estimate U -statistics in a distributed environment, and studies their compromise between statistical accuracy and computing time. We prove the benefits brought by our techniques in terms of variance reduction, and extend our results to design distributed gradient descent algorithms for tuplewise empirical risk minimization. Our analysis builds on top of the properties for U -statistics presented in Chapter 4. Our results are supported by numerical experiments in pairwise statistical estimation and learning on synthetic and real-world datasets.

Introduction

Statistical machine learning has seen dramatic development over the last decades, due partially to the availability of massive datasets, observed in particular in biometrics. For example, the largest public facial recognition dataset now contains 8.2 million images [START_REF] Guo | Ms-celeb-1m: A dataset and benchmark for large-scale face recognition[END_REF], and private datasets are much larger. Another contributing factor is the increasing need to perform predictive/inference/optimization tasks in a wide variety of domains. Those two elements have given a considerable boost to the field and led to successful applications. In parallel to those advances, there has been an ongoing technological progress in the architecture of data repositories and distributed systems, allowing to process ever larger (and possibly complex, high-dimensional) data sets gathered on distributed storage platforms. This trend is illustrated by the development of many easy-to-use cluster computing frameworks for large-scale distributed data processing. These frameworks implement the data-parallel setting, in which data points are partitioned across different machines which operate on their partition in parallel. Some striking examples are Apache Spark [START_REF] Zaharia | Spark: Cluster computing with working sets[END_REF] and Petuum [START_REF] Xing | Petuum: A New Platform for Distributed Machine Learning on Big Data[END_REF], the latter being fully targeted to machine learning. The goal of such frameworks is to abstract away the network and communication aspects in order to ease the deployment of distributed algorithms on large computing clusters and on the cloud, at the cost of some restrictions in the types of operations and parallelism that can be efficiently achieved. However, these limitations as well as those arising from network latencies or the nature of certain memory-intensive operations are often ignored or incorporated in a stylized manner in the mathematical description and analysis of statistical learning algorithms (see e.g. [START_REF] Balcan | Distributed learning, communication complexity and privacy[END_REF][START_REF] Iii | Protocols for learning classifiers on distributed data[END_REF]; Bellet et al. (2015b); [START_REF] Arjevani | Communication complexity of distributed convex learning and optimization[END_REF]). The implementation of statistical methods proved to be theoretically sound may thus be hardly feasible in a practical distributed system, and seemingly minor adjustments to scale-up these procedures can turn out to be disastrous in terms of statistical performance, see e.g. the discussion in [START_REF] Jordan | On statistics, computation and scalability[END_REF]. This greatly restricts their practical interest in some applications and urges the statistics and machine learning communities to get involved with distributed computation more deeply [START_REF] Bekkerman | Scaling up Machine Learning: Parallel and Distributed Approaches[END_REF].

In this chapter, we propose to study these issues in the context of tuplewise estimation and learning problems, where the statistical quantities of interest are not basic sample means but come in the form of averages over all pairs (or more generally, d-tuples) of data points. Such data functionals are known as U -statistics [START_REF] Lee | U -statistics: Theory and practice[END_REF][START_REF] De La Pena | Decoupling: from Dependence to Independence[END_REF], and many empirical quantities describing global properties of a probability distribution fall in this category (e.g. the sample variance, the Gini mean difference, Kendall's tau coefficient). U -statistics are also natural empirical risk measures in several learning problems such as ranking (Clémençon et al., 2008), metric learning [START_REF] Vogel | A probabilistic theory of supervised similarity learning for pointwise ROC curve optimization[END_REF], cluster analysis [START_REF] Clémençon | A statistical view of clustering performance through the theory of U-processes[END_REF] and risk assessment [START_REF] Bertail | Incomplete generalized U -statistics for food risk assessment[END_REF]. In the similarity ranking problem of the preceding chapter (Chapter 5), all estimators are combinations of U -statistics. The behavior of these statistics is well-understood and a sound theory for empirical risk minimization based on U -statistics is now documented in the machine learning literature (Clémençon et al., 2008), but the computation of a U -statistic poses a serious scalability challenge as it involves a summation over an exploding number of pairs (or d-tuples) as the dataset grows in size. In the centralized (single machine) setting, this can be addressed by appropriate subsampling methods, which have been shown to achieve a nearly optimal balance between computational cost and statistical accuracy [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF]. Unfortunately, naive implementations in the case of a massive distributed dataset either greatly damage the accuracy or are inefficient due to a lot of network communication (or disk I/O). This is due to the fact that, unlike basic sample means, a U -statistic is not separable across the data partitions.

Our main contribution is to design and analyze distributed methods for statistical estimation and learning with U -statistics that guarantee a good trade-off between accuracy and scalability. Our approach incorporates an occasional data repartitioning step between parallel computing stages in order to circumvent the limitations induced by data partitioning over the cluster nodes. The number of repartitioning steps allows to trade-off between statistical accuracy and computational efficiency. To shed light on this phenomenon, we first study the setting of statistical estimation, precisely quantifying the variance of estimates corresponding to several strategies. Thanks to the use of Hoeffding's decomposition [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF], our analysis reveals the role played by each component of the variance in the effect of repartitioning. We then discuss the extension of these results to statistical learning and design efficient and scalable stochastic gradient descent algorithms for distributed empirical risk minimization. Finally, we carry out some numerical experiments on pairwise estimation and learning tasks on synthetic and real-world datasets to support our results from an empirical perspective.

The chapter is structured as follows. Section 6.2 reviews background on U -statistics and their use in statistical estimation and learning, and discuss the common practices in distributed data processing. Section 6.3 deals with statistical tuplewise estimation: we introduce our general approach for the distributed setting and derive (non-)asymptotic results describing its accuracy. Section 6.4 extends our approach to statistical tuplewise learning. We provide experiments supporting our results in Section 6.5, and we conclude in Section 6.6.

Background

In this section, we first review the definition and properties of U -statistics, and discuss some popular applications in statistical estimation and learning. We then discuss the recent randomized methods designed to scale up tuplewise statistical inference to large datasets stored on a single machine. Finally, we describe the main features of cluster computing frameworks. 6.2. Background

U -Statistics: Definition and Applications

U -statistics are the natural generalization of i.i.d. sample means to tuples of points. We state the definition of U -statistics in their generalized form, where points can come from K ě 1 independent samples. Note that we recover classic sample mean statistics in the case where K " d 1 " 1. Definition 6.1. (Generalized U -statistic) Let K ě 1 and pd 1 , . . . , d K q P N ˚K . For each k P t1, . . . , Ku, let X t1, ..., n k u " pX pkq 1 , . . . , X pkq n k q be an independent sample of size n k ě d k composed of i.i.d. random variables with values in some measurable space X k with distribution F k pdxq. Let h : X d1 1 ˆ¨¨¨ˆX d K K Ñ R be a measurable function, square integrable with respect to the probability distribution F " F bd1 1 b ¨¨¨b F bd K K . Assume w.l.o.g. that hpx p1q , . . . , x pKq q is symmetric within each block of arguments x pkq (valued in X d k k ). The generalized (or K-sample) U -statistic of degrees pd 1 , . . . , d K q with kernel H is defined as

U n phq " 1 ś K k"1 `nk d k ˘ÿ I1 . . . ÿ I K hpX p1q I1 , X p2q I2 , . . . , X pKq I K q, (6.1)
where ř I k denotes the sum over all `nk

d k ˘subsets X pkq I k " pX pkq i1 , . . . , X pkq i d k q related to a set I k of d k indexes 1 ď i 1 ă . . . ă i d k ď n k and n " pn 1 , . . . , n K q.
The U -statistic U n phq is known to have minimum variance among all unbiased estimators of the parameter mphq " E " hpX

p1q 1 , . . . , X p1q d1 , . . . , X pKq 1 , . . . , X pKq d K q ‰ .
The price to pay for this low variance is a complex dependence structure exhibited by the terms involved in the average (6.1), as each data point appears in multiple tuples. The (non) asymptotic behavior of U -statistics and U -processes (i.e., collections of U -statistics indexed by classes of kernels) can be investigated by means of linearization techniques [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF] combined with decoupling methods (de la Pena and Giné, 1999), reducing somehow their analysis to that of basic i.i.d. averages or empirical processes. One may refer to [START_REF] Lee | U -statistics: Theory and practice[END_REF] for an account of the asymptotic theory of U -statistics, and to van der Vaart (2000) (Chapter 12 therein) and de la Pena and Giné (1999) for nonasymptotic results.

U -statistics are commonly used as point estimators for inferring certain global properties of a probability distribution as well as in statistical hypothesis testing. Popular examples include the (debiased) sample variance, obtained by setting K " 1, d 1 " 2 and hpx 1 , x 2 q " px 1 ´x2 q 2 , the Gini mean difference, where K " 1, d 1 " 2 and hpx 1 , x 2 q " |x 1 ´x2 |, and Kendall's tau rank correlation, where K " 2, d 1 " d 2 " 1 and hppx 1 , y 1 q, px 2 , y 2 qq " Itpx 1 ´x2 q ¨py 1 ´y2 q ą 0u.

U -statistics also correspond to empirical risk measures in statistical learning problems such as clustering [START_REF] Clémençon | A statistical view of clustering performance through the theory of U-processes[END_REF], metric learning [START_REF] Vogel | A probabilistic theory of supervised similarity learning for pointwise ROC curve optimization[END_REF] and multipartite ranking [START_REF] Clémençon | Building confidence regions for the ROC surface[END_REF]. The generalization ability of minimizers of such criteria over a class H of kernels can be derived from probabilistic upper bounds for the maximal deviation of collections of centered U -statistics under appropriate complexity conditions on H (e.g., finite VC dimension) (Clémençon et al., 2008;[START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF].

Below, we describe the example of multipartite ranking used in our numerical experiments (Section 6.5). We refer to [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] for details on more learning problems involving U -statistics.

Example 6.2 (Multipartite Ranking). Consider items described by a random vector of features X P X with associated ordinal labels Y P t1, . . . , Ku, where K ě 2. The goal of multipartite ranking is to learn to rank items in the same preorder as that defined by the labels, based on a training set of labeled examples. Rankings are generally defined through a scoring function s : X Ñ R transporting the natural order on the real line onto X . Given K independent samples, the empirical ranking performance of spxq is evaluated by means of the empirical VUS (Volume Under the ROC Surface) criterion [START_REF] Clémençon | Building confidence regions for the ROC surface[END_REF]:

z V U Spsq " 1 ś K k"1 n k n1 ÿ i1"1 . . . n K ÿ i K "1 ItspX p1q i1 q ă . . . ă spX pKq i K qu, (6.2)
which is a K-sample U -statistic of degree p1, . . . , 1q with kernel h s px 1 , . . . , x K q " Itspx 1 q ă . . . ă spx K qu.

In Chapter 5, we analyzed the learning rates achieved by minimizing functionals akin to Ustatistics. In the large-scale setting, solving this problem is computationally costly due to the very large number of training pairs. In particular, given a sample tpX i , Y i qu n i"1 Ă X ˆt1, . . . , Ku, averages over the positive pairs, i.e. of the same class, and the negatives pairs are sums over respectively ř K k"1 n k pn k ´1q{2 and ř kăl n k n l pairs, with n k " ř n i"1 ItY " ku for any k P t1, . . . , Ku. In the setting where we have a large number of (rather balanced) classes, as in a biometric identification example where a class corresponds to an identity, the number of pairs is quadratic in the number of observations, which makes the approach unpractical. One can extend the theoretical guarantees of Chapter 5 to learning with sampling-based approximations with much lower computational cost, as detailed in [START_REF] Vogel | A probabilistic theory of supervised similarity learning for pointwise ROC curve optimization[END_REF] (Section 4), which is achieved using incomplete U -statistics.

Large-Scale Tuplewise Inference with Incomplete U -statistics

The cost related to the computation of the U -statistic (6.1) rapidly explodes as the sizes of the samples increase. Precisely, the number of terms involved in the summation is `n1

d1 ˘ˆ¨¨¨ˆ`n K d K
˘, which is of order Opn d1`...`d K q when the n k 's are all asymptotically equivalent. Whereas computing U -statistics based on subsamples of smaller size would severely increase the variance of the estimation, the notion of incomplete generalized U -statistic [START_REF] Blom | Some properties of incomplete U-statistics[END_REF] enables to significantly mitigate this computational problem while maintaining a good level of accuracy. Note incidentally that the subsets of indices can be selected by means of other sampling schemes [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF], but sampling with replacement is often preferred due to its simplicity. In practice, the parameter B should be picked much smaller than the total number of tuples to reduce the computational cost. Like (6.1), the quantity (6.3) is an unbiased estimator of mphq but its variance is naturally larger:

Varp r U B phqq " ´1 ´1 B ¯VarpU n phqq `1 B VarphpX p1q 1 , . . . , X pKq d K qq. (6.4)
The recent work in [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] has shown that the maximal deviations between (6.1) and ( 6.3) over a class of kernels H of controlled complexity decrease at a rate of order Op1{ ? Bq as B increases. An important consequence of this result is that sampling B " Opnq terms is sufficient to preserve the learning rate of order a log n{n of the minimizer of the complete risk (6.1), whose computation requires to average Opn d1`...`d K q terms. In contrast, the distribution of a complete U -statistic built from subsamples of reduced sizes n 1 k drawn uniformly at random is quite different from that of an incomplete U -statistic based on B "

ś K k"1 `n1
k d k ˘terms sampled with replacement in Λ, although they involve the summation of the same number of terms. Empirical minimizers of such a complete U -statistic based on subsamples achieve a much slower learning rate of Op a logpnq{n 1{pd1`...`d K q q. We refer to [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] for details and additional results.

We have seen that approximating complete U -statistics by incomplete ones is a theoretically and practically sound approach to tackle large-scale tuplewise estimation and learning problems. However, as we shall see later, the implementation is far from straightforward when data is stored and processed in standard distributed computing frameworks, whose key features are recalled below.

Practices in Distributed Data Processing

Data-parallelism, i.e. partitioning the data across different machines which operate in parallel, is a natural approach to store and efficiently process massive datasets. This strategy is especially appealing when the key stages of the computation to be executed can be run in parallel on each partition of the data. As a matter of fact, many estimation and learning problems can be reduced to (a sequence of) local computations on each machine followed by a simple aggregation step. This is the case of gradient descent-based algorithms applied to standard empirical risk minimization problems, as the objective function is nicely separable across individual data points. Optimization algorithms operating in the data-parallel setting have indeed been largely investigated in the machine learning community, see [START_REF] Bekkerman | Scaling up Machine Learning: Parallel and Distributed Approaches[END_REF]; [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF]; [START_REF] Arjevani | Communication complexity of distributed convex learning and optimization[END_REF]; [START_REF] Smith | CoCoA: A General Framework for Communication-Efficient Distributed Optimization[END_REF] and references therein for some recent work.

Because of the prevalence of data-parallel applications in large-scale machine learning, data analytics and other fields, the past few years have seen a sustained development of distributed data processing frameworks designed to facilitate the implementation and the deployment on computing clusters. Besides the seminal MapReduce framework [START_REF] Dean | Mapreduce: simplified data processing on large clusters[END_REF], which is not suitable for iterative computations on the same data, one can mention Apache Spark [START_REF] Zaharia | Spark: Cluster computing with working sets[END_REF], Apache Flink [START_REF] Carbone | Apache flink™: Stream and batch processing in a single engine[END_REF] and the machine learning-oriented Petuum [START_REF] Xing | Petuum: A New Platform for Distributed Machine Learning on Big Data[END_REF]. In these frameworks, the data is typically first read from a distributed file system (such as HDFS, Hadoop Distributed File System) and partitioned across the memory of each machine in the form of an appropriate distributed data structure. The user can then easily specify a sequence of distributed computations to be performed on this data structure (map, filter, reduce, etc.) through a simple API which hides low-level distributed primitives (such as message passing between machines). Importantly, these frameworks natively implement fault-tolerance (allowing efficient recovery from node failures) in a way that is also completely transparent to the user.

While such distributed data processing frameworks come with a lot of benefits for the user, they also restrict the type of computations that can be performed efficiently on the data. In the rest of this chapter, we investigate these limitations in the context of tuplewise estimation and learning problems, and propose solutions to achieve a good trade-off between accuracy and scalability.

Distributed Tuplewise Statistical Estimation

In this section, we focus on the problem of tuplewise statistical estimation in the distributed setting (an extension to statistical learning is presented in Section 6.4). We consider a set of N ě 1 workers in a complete network graph (i.e., any pair of workers can exchange messages). For convenience, we assume the presence of a master node, which can be one of the workers and whose role is to aggregate estimates computed by all workers.

For ease of presentation, we restrict our attention to the case of two sample U -statistics of degree p1, 1q (K " 2 and d 1 " d 2 " 1), see Remark 6.7 in Section 6.3.3 for extensions to the general case. We denote by D n " tX 1 , . . . , X n u the first sample and by Q m " tZ 1 , . . . , Z m u the second sample (of sizes n and m respectively). These samples are distributed across the N workers. For i P t1, . . . , N u, we denote by R i the subset of data points held by worker i and, unless otherwise noted, we assume for simplicity that all subsets are of equal size |R i | " pn `mq{N P N. The notations R X i and R Z i respectively denote the subset of data points held by worker i from D n and Q m , with R X i Y R Z i " R i . We denote their (possibly random) cardinality by n i " |R X i | and m i " |R Z i |. Given a kernel h, the goal is to compute a good estimate of the parameter U phq " ErhpX 1 , Z 1 qs while meeting some computational and communication constraints.

Naive Strategies

Before presenting our approach, we start by introducing two simple (but ineffective) strategies to compute an estimate of U phq. The first one is to compute the complete two-sample U -statistic associated with the full samples D n and Q m :

U n phq " 1 nm n ÿ k"1 m ÿ l"1
hpX k , Z l q, (6.5)

with n " pn, mq. While U n phq has the lowest variance among all unbiased estimates that can be computed from pD n , Q m q, computing it is a highly undesirable solution in the distributed setting where each worker only has access to a subset of the dataset. Indeed, ensuring that each possible pair is seen by at least one worker would require massive data communication over the network. Note that a similar limitation holds for incomplete versions of (6.5) as defined in Definition 6.3.

A feasible strategy to go around this problem is for each worker to compute the complete Ustatistic associated with its local subsample R i , and to send it to the master node who averages all contributions. This leads to the estimate

U n,N phq " 1 N N ÿ i"1 U Ri phq where U Ri phq " 1 n i m i ÿ kPR X i ÿ lPR Z i hpX k , Z l q.
(6.6)

Note that if minpn i , m i q " 0, we simply set U Ri phq " 0.

Alternatively, as the R i 's may be large, each worker can compute an incomplete U -statistic r U B,Ri phq with B terms instead of U Ri , leading to the estimate

r U n,N,B phq " 1 N N ÿ i"1 r U B,Ri phq where r U B,Ri phq " 1 B ÿ pk,lqPR i,B
hpX k , Z l q, (6.7)

with R i,B a set of B pairs built by sampling uniformly with replacement from the local subsample R X i ˆRZ i . As shown in Section 6.3.3, strategies (6.6) and (6.7) have the undesirable property that their accuracy decreases as the number of workers N increases. This motivates our proposed approach, introduced in the following section.

Proposed Approach

The naive strategies presented above are either accurate but very expensive (requiring a lot of communication across the network), or scalable but potentially inaccurate. The approach we promote here is of disarming simplicity and aims at finding a sweet spot between these two extremes. The idea is based on repartitioning the dataset a few times across workers (we keep the repartitioning scheme abstract for now and postpone the discussion of concrete choices to subsequent sections). By alternating between parallel computation and repartitioning steps, one considers several estimates based on the same data points. This allows to observe a greater diversity of pairs and thereby refine the quality of our final estimate, at the cost of some additional communication.

Formally, let T be the number of repartitioning steps. We denote by R t i the subsample of worker i after the t-th repartitioning step, and by U R t i phq the complete U -statistic associated with R t i . At each step t P t1, . . . , T u, each worker i computes U R t i phq and sends it to the master node. After T steps, the master node has access to the following estimate:

p U n,N,T phq " 1 T T ÿ t"1 U t n,N phq, (6.8)
where U t n,N phq " p1{N q

ř N i"1 U R t i phq.
Similarly as before, workers may alternatively compute incomplete U -statistics r U B,R t i phq with B terms. The estimate is then:

r U n,N,B,T phq " 1 T T ÿ t"1 r U t n,N,B phq, (6.9)
where r U t n,N,B phq " p1{N q

ř N i"1 r U B,R t i phq.
These statistics, and those introduced in Section 6.3.1 which do not rely on repartition, are summarized in Figure 6.1. Of course, the repartitioning operation is rather costly in terms of runtime so T should be kept to a reasonably small value. We illustrate this trade-off by the analysis presented in the next section. Figure 6.1: Graphical summary of the statistics that we compare: with/without repartition and with/without subsampling. Note that tpσ t , π t qu T t"1 denotes a set of T independent couples of random permutations in S n ˆSm .

Analysis

In this section, we analyze the statistical properties of the various estimators introduced above. We focus here on repartitioning by proportional sampling without replacement (prop-SWOR). Prop-SWOR creates partitions that contain the same proportion of elements of each sample: specifically, it ensures that at any step t and for any worker i, |R t i | " pn`mq{N with |R t,X i | " n{N and |R t,Z i | " m{N . We discuss the practical implementation of this repartitioning scheme as well as some alternative choices in Section 6.3.5 and Section 6.3.4.

All estimators are unbiased when repartitioning is done with prop-SWOR. We will thus compare their variance. Our main technical tool is a linearization technique for U -statistics known as Hoeffding's Decomposition (see [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF]; Clémençon et al. (2008); Clémençon et al. ( 2016)). Definition 6.4. (Hoeffding's decomposition) Let h 1 pxq " Erhpx, Z 1 qs, h 2 pzq " ErhpX 1 , zqs and h 0 px, zq " hpx, zq ´h1 pxq ´h2 pzq `U phq. U n phq ´U phq can be written as a sum of three orthogonal terms:

U n phq ´U phq " T n phq `Tm phq `Wn phq, where T n phq " p1{nq ř n k"1 h 1 pX k q ´U phq and T m phq " p1{mq ř n l"1 h 2 pZ l q ´U phq are sums of independent r.v.'s while W n phq " pnmq

´1 ř n k"1 ř m l"1 h 0 pX k , Z l q is a degenerate U -statistic (i.e. ErhpX 1 , Z 1 q|X 1 s " U phq and ErhpX 1 , Z 1 q|Z 1 s " U phq).
This decomposition is very convenient as the two terms T n phq and T m phq are decorrelated and the analysis of W n phq (a degenerate U -statistic) is well documented [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF]Clémençon et al., 2008;[START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF]. It will allow us to decompose the variance of the estimators of interest into single-sample components σ 2 1 " Varph 1 pXqq and σ 2 2 " Varph 2 pZqq on the one hand, and a pairwise component σ 2 0 " Varph 0 pX 1 , Z 1 qq on the other hand. Denoting

σ 2 " VarphpX 1 , Z 1 qq, we have σ 2 " σ 2 0 `σ2 1 `σ2
2 . It is well-known that the variance of the complete U -statistic U n phq can be written as

VarpU n phqq " σ 2 1 n `σ2 2 m `σ2 0 nm .
Our first result gives the variance of the estimators which do not rely on a repartitioning of the data with respect to the variance of U n phq.

Theorem 6.5. If the data is distributed over workers using prop-SWOR, we have:

VarpU n,N phqq " VarpU n phqq `pN ´1q σ 2 0 nm , Varp r U n,N,B phqq " ˆ1 ´1 B ˙VarpU n,N phqq `σ2 N B .
Proof. First, consider VarpU n,N q. Hoeffding's decomposition implies that:

U n,N phq ´U phq " T n phq `Tm phq `1 N N ÿ k"1 1 n 0 m 0 ÿ iPR X k ÿ jPR Z k h 0 pX i , Z j q,
as well as the following properties, for any @ppk, lq, pi, jqq P pt1, . . . , nu ˆt1, . . . , muq 2 , Covph 1 pX k q, h 2 pZ l qq " 0, Covph 1 pX j q, h 0 pX k , Z l qq " 0 and Covph 2 pZ j q, h 0 pX k , Z l qq " 0, Covph 0 pX i , Z j q, h 0 pX k , Z l qq " 0 if pi, jq ‰ pk, lq, (6.10) which imply the result. The variance of the complete U-statistic U n is just the special case N " 1 of the variance U n,N . Explicitely,

VarpU n,N phqq " σ 2 1 n `σ2 2 m `N σ 2 0 nm . Now for r U n,N,B phq, since r U n,N,B conditioned upon the data has expectation U n,N phq, i.e. E " r U n,N,B phq|D n , Q m , pR k q N k"1 ı " U n,N phq, the law of total variance implies, Varp r U n,N,B phqq " VarpU n,N phqq `ErVarp r U n,N,B phq|D n , Q m , pR k q N k"1 qs, " VarpU n,N phqq `1 N ErVarp r U R1,B phq|D n , Q m , pR k q N
k"1 qs, (Since the draws of B pairs on different workers are independent),

" VarpU n,N phqq `1 N " ´1 B VarpU R1 q `1 B VarphpX, Zqq  ,
(See [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF])

" ˆ1 ´1 B ˙VarpU n,N phqq `1 N B VarphpX, Zqq, which concludes our proof. Explicitly, Varp r U n,N,B phqq " ˆ1 ´1 B ˙ˆσ 2 1 n `σ2 2 m `N σ 2 0 nm ˙`1 N B
Var phpX, Zqq .

Theorem 6.5 precisely quantifies the excess variance due to the distributed setting if one does not use repartitioning. Two important observations are in order. First, the variance increase is proportional to the number of workers N , which clearly defeats the purpose of distributed processing. Second, this increase only depends on the pairwise component σ 2 0 of the variance. In other words, the average of U -statistics computed independently over the local partitions contains all the information useful to estimate the single-sample contributions, but fails to accurately estimate the pairwise contributions. The resulting estimates thus lead to significantly larger variance when the choice of kernel and the data distributions imply that σ 2 0 is large compared to σ 1 2 and/or σ 2 1 . The extreme case happens when U n phq is a degenerate U -statistic, i.e. σ 2 1 " σ 2 2 " 0 and σ 2 0 ą 0, which is verified for example when hpx, zq " x ¨z and X, Z are both centered random variables.

We now characterize the variance of the estimators that leverage data repartitioning steps. Theorem 6.6. If the data is distributed and repartitioned between workers using prop-SWOR, we have:

Varp p U n,N,T phqq " VarpU n phqq `pN ´1q σ 2 0 nmT , Varp r U n,N,B,T phqq " Varp p U n,N,T phqq ´1 T B VarpU n,N phqq `σ2 N T B .
Proof. We first detail the derivation of Varp p U n,N,T phqq. Define the Bernouilli r.v. t i pkq as equal to one if X k is in partition i at time t, and similarly γ t i plq is equal to one if Z l is in partition i at time t. Note that for t ‰ t 1 , t i pkq and t1 i1 pk 1 q are independent, as well as γ t i plq and γ t1 i1 pl 1 q. Additionally, t i pkq and γ t1 i1 plq are independent for any t, t 0 P t1, . . . , T u 2 . Hoeffding's decomposition implies:

U t n,N phq ´Un phq " 1 N N ÿ i"1 1 nm n ÿ k"1 m ÿ l"1 pN 2 t i pkqγ t i plq ´1qh 0 pX k , Z l q.
The law of total variance, the fact that conditioned upon the data p U n,N,T phq is an average of T independent experiments and the properties of Eq. (6.10) imply:

Var ´p U n,N,T phq ¯" Var pU n phqq `E " Var ´p U n,N,T phq|D n , Q m ¯ı , " Var pU n phqq `1 T E " Var `U t n,N phq|D n , Q m ˘‰ , " Var pU n phqq `N 2 σ 2 0 nmT N ÿ i1,i2"1 Cov ` t i1 p1qγ t i1 p1q, t i2 p1qγ t i2 p1q ˘. (6.11)
On the other hand, observe that:

Cov ` t i1 p1qγ t i1 p1q, t i2 p1qγ t i2 p1q ˘" # ´N ´4 if i 1 ‰ i 2 , N ´2 ´N ´4 if i 1 " i 2 . (6.12)
The result is obtained by plugging Eq. (6.12) in Eq. (6.11). Explicitly,

Var ´p U n,N,T phq ¯" Var pU n phqq `N ´1 nmT σ 2 0 .
Using that Er r U n,N,B,T phq|D n , Q m , , γs " p U n,N,T phq, we now compute Varp r U n,N,B,T phqq by decomposing it as the variance of its conditional expectation plus the expectation of its conditional variance. It writes:

Varp r U n,N,B,T phqq " Var ´p U n,N,T phq ¯`E " Var ´r U n,N,B,T |D n , Q m , , γ ¯ı " Var ´p U n,N,T phq ¯`1 N T E " Var ´r U B,R t i |D n , Q m , , γ ¯ı
(Since the draws of B pairs on different workers are independent)

" Var ´p U n,N,T phq ¯`1 N T " ´1 B Var ´UR t i ¯`1 B VarphpX, Zqq  (See Clémençon et al. (2016).) " VarphpX, Zqq N T B `ˆ1 ´1 T B ˙ˆσ 2 1 n `σ2 2 m ˙`σ 2 0 nm " 1 `N ´1 T ´N T B  ,
which gives the desired result after reorganizing the terms.

See Fig. 6.2 for illustrations of these formulas with two example cases. The figure represent the variance of each estimator as a function of the number of evaluated pairs for proportional SWOR.

Theorem 6.5 and Theorem 6.6 show that the influence of the kernel h and the distribution of X and Z on the expression of the variances is summarized by the variance derived from pairwise information σ 2 0 , that issued from instances of the majority class σ 2 1 and that from instances of the minority class σ 2 2 . The value of repartitioning arises from the fact that the term that account for pairwise variance for p U n,N,T phq is almost T times lower that than of U n phq. Repartitioning is thus interesting when the pairwise variance term is significant in front of the other terms. Practical settings are presented in Section 6.5. Figure 6.2: Theoretical variance as a function of the number of evaluated pairs for different estimators under prop-SWOR, with n " 100, 000, m " 200 and N " 100.

Since n " m, it suffices that σ 2 2 be low for the influence of pairwise variance to be significant, which is illustrated in Fig. 6.2. It shows that for a same number of pairs evaluated, one can only expect a very limited precision for the estimators that do not use redistribution. The left-hand side figure in Fig. 6.2 shows the importance of redistribution in some cases. Indeed, it shows that for a same number of pairs, one can only expect a very limited precision for the estimators VarpU n,N phqq and Varp r U n,N,B phqq, compared to those presented in Theorem 6.6 that use redistribution. When σ 2 2 is not small, the variance of U n mostly originates from the rarity of the minority class which implies that redistributing does not bring estimates that are significantly more accurate.

On the other hand, the right-hand side figure in Fig. 6.2 shows that this only holds for certain types of statistics, and that for different parameters σ 2 0 , σ 2 1 , σ 2 2 , redistribution of the instances on the workers does not lead to more accurate estimators.

Finally, the results of Theorem 6.6 imply that, in the case of proportional SWOR and for a fixed budget of evaluated pairs, using all pairs on each worker is always a dominant strategy over using incomplete estimators, i.e. p U n,N,T is always preferable to r U n,N,B,T . This is shown by developments of the difference Varp r U n,N,B,T phqq ´Varp p U n,N,T0 phqq, while imposing N BT " nmT 0 {N . Indeed, ∆ :"Var ´r U n,N,B,T phq ¯´Var ´p U n,N,T0

¯,

"σ 2 0 " N ´1 nm ˆ1 T ´1 T 0 ˙´N nmT B `1 N T B  `σ2 1 T B " 1 N ´1 n  `σ2 2 T B " 1 N ´1 m  , "σ 2 0 " N ´1 nmT ˆ1 ´1 B ˙`1 T B ˆ1 N 2 ´1 nm ˙ `σ2 1 T B " 1 N ´1 n  `σ2 2 T B " 1 N ´1 m  ,
which shows that ∆ ą 0. Note that computing complete U -statistics also require fewer repartitioning steps to evaluate the same number of pairs (i.e., T 0 ď T ).

Optimization processes, such as stochastic gradient descent are often interested in small random batches of data over many timesteps, and the dominance of p U n,N,T over r U n,N,B,T supports the hypothesis that redistributing the instances infrequently between timesteps is enough to correct for the losses in precision incurred from redistributing the data. Remark 6.7 (Extension to high-order U -statistics). The extension of our analysis to general U -statistics is straightforward (see [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] for a review of the relevant technical tools). We stress the fact that the benefits of repartitioning are even stronger for higher-order U -statistics (K ą 2 and/or larger degrees) because higher-order components of the variance are also affected.

Extension to Sampling With Replacement

While the use of prop-SWR is not very natural in a standard distributed setting, it is relevant in cases where workers have access to joint database that they can efficiently subsample. We have the following results for the variance of estimates based on prop-SWR. Theorem 6.8. If the data is distributed between workers with prop-SWR, and denoting n 0 " pn{N, m{N q, we have:

VarpU n,1 phqq " σ 2 1 n ˆ2 ´1 n ˙`σ 2 2 m ˆ2 ´1 m ˙`σ 2 0 nm " 4 ´2 ˆ1 n `1 m ˙`1 nm  , VarpU n,N phqq " VarpU n,1 phqq `σ2 0 nm pN ´1q ˆ1 ´1 n ˙ˆ1 ´1 m ˙, Varp r U n,N,B phqq " Var pU n,N phqq `1 N B " σ 2 ´Var pU n0,1 phqq ‰ .
Proof. First we derive the variance of U n,N phq. Since ErU n,N phq|D n , Q m s " U n phq, the law of total variance implies:

VarpU n,N phqq " VarpU n phqq `E rVar pU n,N phq|D n , Q m qs , " VarpU n phqq `1 N E rVar pU R1 phq|D n , Q m qs .
Introduce pkq (resp. γplq) as the random variable that is equal to the number of times that k has been sampled in cluster 1 for the D n elements (resp. that l has been sampled in cluster 1 for the Q m elements). The random variable pkq (resp. γplq) follows a binomial distribution with parameters pn{N, 1{nq (resp. pm{N, 1{mq). Note that the and γ are independent and that ř n k"1 pkq " n{N and ř m l"1 γplq " m{N . It follows that:

U R1 phq ´Un phq " U phq `1 n n ÿ k"1 pN pkq ´1q ph 1 pX k q ´U phqq `1 m m ÿ l"1 pN γplq ´1q ph 2 pZ l q ´U phqq `1 nm n ÿ k"1 m ÿ l"1 `N 2 pkqγplq ´1˘h 0 pX k , Z l q,
which implies, using the results of Eq. (6.10),

E rVar pU R1 phq|D n , Q m qs " N 2 σ 2 1 n Varp p1qq `N 2 σ 2 2 m Varpγp1qq `N 4 σ 2 0 nm Varp p1qγp1qq. (6.13)
The mean and variance of a binomial distribution is known. Since p1q and γp1q are independent,

Varp p1qq " 1 N ˆ1 ´1 n ˙, Varpγp1qq " 1 N ˆ1 ´1 m ˙, Varp p1qγp1qq " 1 N 2 "ˆ1 ´1 n ˙ˆ1 ´1 m ˙`1 N ˆ2 ´1 n ´1 m ˙ . (6.14)
Plugging Eq. (6.14) into Eq. (6.13) gives the result. Explicitly,

Var pU n,N phqq " σ 2 1 n ˆ2 ´1 n ˙`σ 2 2 m ˆ2 ´1 m σ2 0 nm "ˆ3 ´1 n ´1 m ˙`N ˆ1 ´1 n ˙ˆ1 ´1 m ˙ .
Now we derive the variance of r U n,N,B phq. Note that Er r

U n,N,B |D n , Q m , , γs " U n,N phq, hence: Varp r U n,N,B q " VarpU n,N phqq `E " Var ´r U n,N,B |D n , Q m , , γ ¯ı , " VarpU n,N phqq `1 N E " Var ´r U B,R1 |D n , Q m , , γ ¯ı . (6.15)
Conditioned upon D n , Q m , , γ, the statistic r U B,R1 is an average of B independent experiments. Introducing δ k,l as equal to 1 if the pair pk, lq is selected in worker 1 as the 1th pair of r U B,R1 , and ∆ k,l its expected value, i.e. ∆ k,l :" Erδ k,l s " N 2 pkqγplq{nm, it implies:

Var ´r U B,R1 | D n , Q m , , γ ¯" 1 B Var ˜n ÿ k"1 m ÿ l"1 δ k,l hpX k , Z l q | D n , Q m , , γ ¸. (6.16)
From the definition of δ k,l we have δ k,l δ k1,l1 " 0 as soon as k ‰ k 1 or l ‰ l 1 , writing the right-hand-side of Eq. (6.16) as the second order moment minus the squared means gives:

Var ´r U B,R1 | D n , Q m , , γ ¯" 1 B m ÿ k"1 n ÿ l"1 ∆ k,j h 2 pX k , Z l q ´1 B ˜n ÿ k"1 m ÿ l"1
∆ k,l hpX k , Z l q ¸2 . (6.17)

Taking the expectation of Eq. ( 6.17) gives:

E " Var ´r U B,R1 |D n , Q m , , γ ¯ı " 1 B " Erh 2 pX, Zqs ´ErU 2 R1 s ‰ , " 1 B rVarphpX, Zqq ´VarpU R1 qs . (6.18)
Plugging Eq. ( 6.18) into Eq. ( 6.15) gives:

Varp r U n,N,B q " VarphpX, Zqq BN `VarpU n,N phqq ´VarpU Ri q BN ,
and we can conclude from preceding results, since U Ri is simply U n0,1 with n 0 " pn{N, m{N q.

Theorem 6.9. If the data is distributed and repartitioned between workers with prop-SWR, we have:

Varp p U n,N,T phqq " VarpU n phqq `1 T rVarpU n,N phqq ´VarpU n phqqs , Varp r U n,N,B,T phqq " Var ´p U n,N,T phq ¯`1 N BT " σ 2 ´VarpU n0,1 phqq ‰ . Proof. Since E " p U n,N,T phq|D n , Q m ı "
U n phq the law of total covariances followed by the fact that, conditioned upon D n , Q, the statistic p U n,N,T phq is an average of T independent random variables, implies:

Var ´p U n,N,T phq ¯" VarpU n phqq `1 T E rVar pU n,N phq|D n , Q m qs .
The calculations above give the result. Explicitly, Var ´p U n,N,T phq ¯" VarpU n phqq `1 T rVarpU n,N phqq ´VarpU n phqqs .

We now derive the variance of r U n,N,B,T . Since

E " r U n,N,B,T phq|D n , Q m , , γ ı " p U n,N,T phq,
the law of total covariance followed by the calculations above imply the result:

Var ´r U n,N,B,T phq ¯" Var ´p U n,N,T phq ¯`E " Var ´r U n,N,B,T phq|D n , Q m , , γ ¯ı , " Var ´p U n,N,T phq ¯`1 T E " Var ´r U n,N,B phq|D n , Q m , , γ ¯ı , " Var ´p U n,N,T phq ¯`1 N BT rVarphpX, Zqq ´VarpU Ri qs .
10 4 10 5 # pairs 10 -4 Variance σ 2 0 = 1.00, σ 2 1 = 0.25, σ 2 2 = 0.00

10 4 10 5 # pairs 10 -2 2 × 10 -2 3 × 10 -2 σ 2 0 = 1.00, σ 2 1 = 0.00, σ 2 2 = 0.25 U n U n,N U n,1 U n,N,B U n,N,T U n,N,B,T B = 250 B = 1, 000 B = 4, 000 T = 2 T = 8 U n,N,T varying T U n,N,B varying B U n,N,1,T varying T Figure 6
.3: Theoretical variances as a function of the number of evaluated pairs for different estimators under prop-SWR, with n " 100, 000, m " 200 and N " 100.

Fig. 6.3 gives a visual illustration of these results. First note that they are similar to those obtained for prop-SWOR in Fig. 6.2. Yet, the right-hand side figure shows that r U n,N,B,T can have a significantly lower variance than p U n,N,T , for the same number of evaluated pairs. This comes from the fact that r U n,N,B,T works on bootstrap re-samples of the data than p U n,N,T and hence better corrects for the loss of information due to sampling with replacement (at the cost of more communication or disk reads). To stress this, we also represented U n,1 , i.e. the point that gives the variance of a complete estimator based on one bootstrap re-sample of the data.

Extension to Simple SWOR

The analysis above assumes that repartitioning is done using prop-SWOR, which has the advantage of exactly preserving the proportion of points from the two samples D n and Q m even in the event of significant imbalance in their size. However, a naive implementation of prop-SWOR requires some coordination between workers at each repartitioning step. To avoid exchanging many messages, we propose that the workers agree at the beginning of the protocol on a numbering of the workers, a numbering of the points in each sample, and a random seed to use in a pseudorandom number generator. This allows the workers to implement prop-SWOR without any further coordination: at each repartitioning step, they independently draw the same two random permutations over t1, . . . , nu and t1, . . . , mu using the common random seed and use these permutations to assign each point to a single worker.

Of course, other repartitioning schemes can be used instead of prop-SWOR. A natural choice is sampling without replacement (SWOR), which does not require any coordination between workers. However, the partition sizes generated by SWOR are random. This is a concern in the case of imbalanced samples, where the probability that a worker i does not get any point from the minority sample (and thus no pair to compute a local estimate) is non-negligible. For these reasons, it is difficult to obtain exact and concise theoretical variances for the SWOR case, but the results with SWOR should not deviate too much from the theoretical predictions given in Theorem 6.5 and Theorem 6.6 obtained with prop-SWOR, as illustrated numerically by the following example. Consider the kernel hpx, zq " x ¨z and random variables in R that follow a normal law X " N pµ X , σ X q and Z " N pµ Z , σ Z q. In that setting, note that σ

2 1 " µ 2 Z σ 2 X , σ 2 2 " µ 2 X σ 2 Z and σ 2 0 " σ 2 X σ 2
Z , which means that by tweaking the parameters µ X , µ Z , σ X , σ Z , one can obtain any possible value of σ 1 , σ 2 , σ 0 . 10 4 10 5 # pairs 10 -4 Variance σ 2 0 = 1.00, σ 2 1 = 0.25, σ 2 2 = 0.00 The results, shown in Fig. 6.4 are very similar to those obtained for prop-SWOR in Fig. 6.2. The fact that SWOR has slightly lower variance is expected, since when no pairs are available the default value is always 0. This makes the estimator give a stable prediction, but also makes it biased.

10 4 10 5 # pairs 10 -2 9 × 10 -3 σ 2 0 = 1.00, σ 2 1 = 0.00, σ 2 2 = 0.25 U n U n,N U n,N,B U n,N,T U n,N,B,T B = 250 B = 1, 000 B = 4, 000 T = 2 T = 8
Finally, we note that deterministic repartitioning schemes may be used in practice for simplicity.

For instance, the repartition method in Apache Spark relies on a deterministic shuffle which preserves the size of the partitions.

Extensions to Stochastic Gradient Descent for ERM

The results of Section 6.3 can be extended to statistical learning in the empirical risk minimization framework. In such problems, given a class of kernels H, one seeks the minimizer of (6.6) or (6.8) depending on whether repartition is used.1 Under appropriate complexity assumptions on H (e.g., of finite VC dimension), excess risk bounds for such minimizers can be obtained by combining our variance analysis of Section 6.3 with the control of maximal deviations based on Bernstein-type concentration inequalities as done in Clémençon et al. (2008) and [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF].

Gradient-based Empirical Minimization of U -statistics

In the setting of interest, the class of kernels to optimize over is indexed by a real-valued parameter θ P R q representing the model. Adapting the notations of Section 6.3, the kernel h : X 1 ˆX2 ˆRq Ñ R then measures the performance of a model θ P R q on a given pair, and is assumed to be convex and smooth in θ. Empirical Risk Minimization (ERM) aims at finding θ P R q minimizing

U n pθq " 1 nm n ÿ k"1 m ÿ l"1
hpX k , Z l ; θq. (6.19)

Numerical Results

The minimizer can be found by means of Gradient Descent (GD) techniques.2 Starting at iteration s " 1 from an initial model θ 1 P R q and given a learning rate γ ą 0, GD consists in iterating over the following update:

θ s`1 " θ s ´γ∇ θ U n pθ s q. (6.20)

Note that the gradient ∇ θ U n pθq is itself a U -statistic with kernel given by ∇ θ H, and its computation is very expensive in the large-scale setting. In this regime, Stochastic Gradient Descent (SGD) is a natural alternative to GD which is known to provide a better trade-off between the amount of computation and the performance of the resulting model [START_REF] Bottou | The tradeoffs of large scale learning[END_REF]. Following the discussion of Section 6.2.2, a natural idea to implement SGD is to replace the gradient ∇ θ U n pθq in (6.20) by an unbiased estimate given by an incomplete U -statistic. The work of [START_REF] Papa | SGD algorithms based on incomplete u-statistics: Largescale minimization of empirical risk[END_REF] shows that SGD converges much faster than if the gradient is estimated using a complete U -statistic based on subsamples with the same number of terms.

However, as in the case of estimation, the use of standard complete or incomplete U -statistics turns out to be impractical in the distributed setting. Building upon the arguments of Section 6.3, we propose a more suitable strategy.

Repartitioning for Stochastic Gradient Descent

The approach we propose is to alternate between SGD steps using within-partition pairs and repartitioning the data across workers. We introduce a parameter n r P Z `corresponding to the number of iterations of SGD between each redistribution of the data. For notational convenience, we let rpsq :" rs{n r s so that for any worker i, R rpsq i denotes its data partition at iteration s ě 1 of SGD.

Given a local batch size B, at each iteration s of SGD, we propose to adapt the strategy (6.9) by having each worker i compute a local gradient estimate using a set R s i,B of B randomly sampled pairs in its current local partition R rpsq i :

∇ θ r U B,R rpsq i pθ s q " 1 B ÿ pk,lqPR s i,B ∇ θ hpX k , Z l ; θ s q.
This local estimate is then sent to the master node who averages all contributions, leading to the following global gradient estimate:

∇ θ r U n,N,B pθ s q " 1 N N ÿ i"1 ∇ θ r U B,R rpsq i pθ s q. (6.21)
The master node then takes a gradient descent step as in (6.20) and broadcasts the updated model θ s`1 to the workers.

Following our analysis in Section 6.3, repartitioning the data allows to reduce the variance of the gradient estimates, which is known to greatly impact the convergence rate of SGD (see e.g. [START_REF] Bubeck | Convex optimization: Algorithms and complexity[END_REF], Theorem 6.3 therein). When n r " `8, data is never repartitioned and the algorithm minimizes an average of local U -statistics, leading to suboptimal performance. On the other hand, n r " 1 corresponds to repartitioning at each iteration of SGD, which minimizes the variance but is very costly and makes SGD pointless. We expect the sweet spot to lie between these two extremes: the dominance of p U n,N,T over r U n,N,B,T established in Section 6.3.3, combined with the common use of small batch size B in SGD, suggests that occasional redistributions are sufficient to correct for the loss of information incurred by the partitioning of data. We illustrate these trade-offs experimentally in the next section.

Numerical Results

In this section, we illustrate the importance of repartitioning for estimating and optimizing the Area Under the ROC Curve (AUC) through a series of numerical experiments. The corresponding

10 -3 10 -2 10 -1 1 2 3 Rel. var. U n U n,N U n,N,T Figure 6
.5: Relative variance estimated over 5000 runs, n " 5000, m " 50, N " 10 and T " 4.

Results are divided by the true variance of U n expressed from (6.22) and Theorem 6.5.

Figure 6.6: Learning dynamics for different repartition frequencies computed over 100 runs.

U -statistic is the two-sample version of the multipartite ranking VUS introduced in Example 6.2 (Section 6.2.1). The first experiment focuses on the estimation setting considered in Section 6.3.

The second experiment shows that redistributing the data across workers, as proposed in Section 6.4, allows for more efficient mini-batch SGD. All experiments use prop-SWOR and are conducted in a simulated environment.

Estimation experiment. We seek to illustrate the importance of redistribution for estimating two-sample U -statistics with the concrete example of the AUC. The AUC is obtained by choosing the kernel hpx, zq " Itz ă xu, and is widely used as a performance measure in bipartite ranking and binary classification with class imbalance. Recall that our results of Section 6.3.3 highlighted the key role of the pairwise component of the variance σ 2 0 being large compared to the singlesample components. In the case of the AUC, this happens when the data distributions are such that the expected outcome using single-sample information is far from the truth, e.g. in the presence of hard pairs. We illustrate this on simple discrete distributions for which we can compute σ 2 0 , σ 2 1 and σ 2 2 in closed form. Consider positive points X P t0, 2u, negative points Z P t´1, `1u and P pX " 2q " q, P pZ " `1q " p. It follows that:

σ 2
1 " p 2 qp1 ´qq, σ 2 2 " p1 ´qq 2 pp1 ´pq, and σ 2 " pp1 ´p `pqqp1 ´qq. (6.22)

Assume that the scoring function has a small probability to assign a low score to a positive instance or a large score to a negative instance. In our formal setting, this translates into letting p " 1 ´q " for a small ą 0, which implies that σ 2 0 {pσ 2 1 `σ2 2 q " p1 ´ q{p2 q Ñ 8 as Ñ 0. We thus expect that as the true AUC U phq " 1 ´ 2 gets closer to 1, repartitioning the dataset becomes more critical to achieve good relative precision. This is confirmed numerically, as shown in Fig. 6.5. Note that in practice, settings where the AUC is very close to 1 are very common as they correspond to well-functioning systems, such as face recognition systems.

Learning experiment. We now turn to AUC optimization, which is the task of learning a scoring function s : X Ñ R that optimizes the VUS criterion (6.2) with K " 2 in order to discriminate between a negative and a positive class. We learn a linear scoring function s w,b pxq " w J x `b, and optimize a continuous and convex surrogate of (6.2) based on the hinge loss. The resulting loss function to minimize is a two-sample U-statistic with kernel g w,b px, zq " maxp0, 1 `sw,b pxq ´sw,b pzqq indexed by the parameters pw, bq of the scoring function, to which we add a small L2 regularization term of 0.05 w 2 2 with ¨ 2 2 the Euclidean norm. We use the shuttle dataset, a classic dataset for anomaly detection. 3 It contains roughly 49,000 points in dimension 9, among which only 7% (approx. 3,500) are anomalies. A high accuracy is expected for this dataset. To monitor the generalization performance, we keep 20% of the data as our test set, corresponding to 700 points of the minority class and approx. 9,000 points of the majority class. The test performance is measured with complete statistics over the 6.3 million pairs. The training set consists of the remaining data points, which we distribute over N " 100 workers. This leads to approx. 10, 200 pairs per worker. The gradient estimates are calculated following (6.21) with batch size B " 100. We use an initial step size of 0.01 with a momentum of 0.9. As there are more than 100 million possible pairs in the training dataset, we monitor the training loss and accuracy on a fixed subset of 4.5 ˆ10 5 randomly sampled pairs. Fig. 6.6 shows the evolution of the continuous loss and the true AUC on the training and test sets along the iteration for different values of n r , from n r " 1 (repartition at each iteration) to n r " `8 (no repartition). The lines are the median at each iteration over 100 runs, and the shaded area correspond to confidence intervals for the AUC and loss value of the testing dataset. We can clearly see the benefits of repartition: without it, the median performance is significantly lower and the variance across runs is very large. The results also show that occasional repartitions (e.g., every 25 iterations) are sufficient to mitigate these issues significantly.

Conclusion

We tackled the distributed estimation of U -statistics. We showed that the pairwise nature of U -statistics implies that naive distributed estimators may fail for some settings. We proposed new unbiased estimators, based on the idea of repartitioning the data on the available machines. Using analytical expressions, we showed that they are efficient in terms of variance, and compared their computational cost to non-distributed and naive distributed estimators of U -statistics. Finally, we proposed using the repartition procedure for minimizing U -statistics with gradient descent in a distributed environment, and confirmed the relevance of our proposition with numerical experiments.

We envision several further research questions on the topic of distributed tuplewise learning. We would like to provide a rigorous convergence rate analysis of the general distributed SGD algorithm introduced in Section 6.4. This is a challenging task because each series of iterations executed between two repartition steps can be seen as optimizing a slightly different objective function. It would also be interesting to investigate settings where the workers hold sensitive data that they do not want to share in the clear due to privacy concerns.

Similarity ranking is the pairwise bipartite ranking view of similarity ranking, introduced and studied in Chapter 5. Precisely, Chapter 5 gave guarantees for the pointwise ROC optimization or the TreeRank procedure for similarity ranking. The techniques presented in this chapter alleviate the extreme computational costs of estimating naively the quantities involved in similarity 3 http://odds.cs.stonybrook.edu/shuttle-dataset/ ranking. However, the propositions for scalability of this chapter do not address the challenges specific to the optimization aspect of similarity ranking. The next chapter focuses on that aspect by means of new procedures, toy examples and illustrations, and leaves aside the scaling considerations as they were thoroughly addressed in this chapter.

Chapter 7

Practical Similarity Ranking Summary: Chapter 5 introduced similarity ranking as the formulation of similarity learning as a pairwise bipartite ranking problem. In similarity ranking, the goal is that the larger the probability that two observations share the same label, the larger the similarity measure between them. Chapter 5 provided analyses on the generalization of selected similarity ranking problems, and Chapter 6 addressed the scalability issues associated to similarity learning. To tackle practically the problems presented in Chapter 5, optimization challenges remain. In this regard, this chapter first focuses on the pointwise ROC optimization (pROC) problem. Precisely, we give an analytical solution to empirical pROC for similarity ranking and for very specific class of functions. Then, we propose a more general gradient-based approach to pROC for bipartite ranking, that we illustrate on a toy example. The extension of that approach to similarity ranking is straightforward. The remainder of the chapter focuses on the TreeRank algorithm -see Chapter 5 for a formal presentation -and first provides illustrations of it in the pairwise setting. Then, we show that ranking forests, an ensemble method for TreeRank, can correct for a misspecficiation of its proposed splitting region. Finally, we discuss the limitations and extensions of our propositions, as well as the idea of deriving practical approaches for similarity learning from our theory of similarity ranking, a promising direction for future work.

Introduction

With the advent of deep neural networks for vision tasks, a new sub-branch of metric learning has emerged, referred to as deep metric learning. Today, all modern facial recognition algorithms are trained with deep metric learning algorithms, and some deep learning-based propositions exist for other biometries [START_REF] Minaee | Fingernet: Pushing the limits of fingerprint recognition using convolutional neural network[END_REF]). Deep metric learning can be roughly summarized as the proposition of losses to learn a transformation of the data where natural distances have a semantic meaning, unlike in the input space X . Formally, deep metric learning consists in learning an embedding e : X Ñ R d . where R d is a low-dimensional space. Then, the distance between the points x, x 1 P X can be written as dpepxq, epx 1 qq, where d : R d ˆRd Ñ R is some simple distance function. In practice, the distance d is usually either the Euclidean distance or the cosine similarity.

The common idea behind all of these losses is to separate classes/identities, but they differ by the way they tackle the problem. In this section, we review several losses used in facial recognition, and refer to [START_REF] Wang | Deep face recognition: A survey[END_REF] for more details.

The original deep metric learning approach to facial recognition used the classical softmax cross-entropy (SCE). SCE is applied to a transformation of the embedding by a linear classifier l : epxq Þ Ñ W J epxq where W P R d,K and K is the number of classes in the training dataset. Introduce a sample of n data points as D n " tpx i , y i qu n i"1 Ă X ˆt1, . . . , Ku, as well as the notations e i :" epx i q P R d and l piq :" lpepx i qq " pl piq 1 , . . . , l piq K q for any i P t1, . . . , nu. Then, the softmax cross-entropy on the observation x i writes:

L SCE :" ´log ¨exp ´lpiq yi řK k"1 exp ´lpiq k ¯' . (7.1)
Eq. ( 7.1) is simply the average of the cross-entropies between the softmax of the vectors l piq and the one-hot-encoding pIty i " 1u, . . . , Ity i " Kuq of y i for any i.

Separating the identities in the embedding space using the softmax cross-entropy makes intuitive sense, but is not based on the idea of computing the distance of a pair of observations, i.e. the end purpose of the trained system. Other losses, notably the contrastive loss and triplet loss, are based on the idea of computing distances between embeddings. With the margin parameters p ´, `q, the contrastive loss on the pair of observations px i , y i q, px j , y j q writes:

L cons :" z i,j ¨max `0, e i ´ej 2
´ `˘`p1 ´zi,j q ¨max `0, ´´ e i ´ej 2 ˘, (7.2) and z i,j " 2 ¨Ity i " y j u ´1 for any pi, jq P t1, . . . , nu 2 . The triplet loss writes is expressed on a triplet ppx i , y i q, px j , y j q, px k , y k qq of observations that satisfies y i " y j and y i ‰ y k . It writes:

L triplet :" max ´0, e i ´ej 2 2 ´ e i ´ek 2 2
`α¯, (7.3)

In Eq. ( 7.3), the point px i , y i q is named the anchor point and α is a margin parameter. Another loss, called the center loss, seeks to align the embeddings of the observations of each class on a center c k , for any k P t1, . . . , Ku. It is expressed as follows on a single point x i :

L center :" 1 2 x i ´cyi 2 2 . (7.4)
Many other losses have been proposed in the literature, such as for example the vMF loss [START_REF] Hasnat | von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification[END_REF], the ArcFace loss [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF] or the AMS loss (Wang et al., 2018). We again refer to [START_REF] Wang | Deep face recognition: A survey[END_REF] for an exhaustive presentation of those losses.

A recent experimental study, see [START_REF] Musgrave | A metric learning reality check[END_REF], of a non-negligible number of the losses presented in the literature has recently shown that they perform comparably when one corrects for all of the differences in models and implementations. This study suggests that most of the improvements advertised by those papers are not due to the loss functions themselves, but to the evolution of other practices in deep learning. Additionally, practitioners often consider summing a selection of those losses in different proportions when training facial recognition systems, see e.g. [START_REF] Parkhi | Deep face recognition[END_REF]. Finding the right combination requires performing extensive cross-validation for models that take several weeks to train, which is extremely costly in terms of time and computational resources. We refer to [START_REF] Strubell | Energy and policy considerations for deep learning in NLP[END_REF] for a discussion on the energy costs of deep learning algorithms.

Although these losses rely on the sound intuitions that minimizing the intra-class variations (Eq. (7.2), Eq. ( 7.3), Eq. (7.4)), maximizing the inter-class variations (Eq. ( 7.2) Eq. ( 7.3)) or splitting the identities (Eq. (7.1)) will improve performance, none of them relates with the evaluation of biometric systems. The evaluation of those systems essentially considers similarity learning as a ranking problem, as explained in Chapter 5. Precisely, those systems are evaluated using the ROC curve, which summarizes all attainable compromises between false positive rate and true positive rate for all possible thresholds on the similarity. In this chapter, we provide simple proof-of-concept toy experiments that draw on Chapter 5 to propose more relevant approaches to deep metric learning for biometrics. Precisely, we propose approaches designed to solve the similarity ranking problem, i.e. the problem of scoring on a product space (Chapter 5).

The chapter is organized as follows. Firstly, we discuss approaches to pointwise ROC optimization. Precisely, we show that for simple similarities and specific data, one can exactly optimize for specific points of the ROC curve. In a more general case, we then propose a gradient-based algorithm that explicitly optimizes for a specific point of the ROC curve. Secondly, we discuss approaches based on the TreeRank algorithm, a recursive splitting algorithm that optimizes for ranking in the ROC sense, as proven in Chapter 5. In that context, we first illustrate the output of the TreeRank algorithm for similarity ranking to provide intuition about its behavior. Then, we show that ranking forests -a bagging procedure for TreeRank -can correct for the flaws of TreeRank. Precisely, our experiment shows that averaging ranking trees can yield a score that is indiscernible from the optimal score, even when the splitting algorithm is ill-suited for the problem. However, this setting is not covered by the analysis of Chapter 5, as we assumed that the proposed family contains the optimal splitting rules. Additionally, our experiment shows that an average of ranking trees can approximate a continuous output score, despite the fine-grained but discrete output of TreeRank.

Practical Algorithms for Pointwise ROC Optimization

Many approaches to solve the ranking problem consist in finding a score function s : X Ñ R that maximizes the Area under the ROC curve (AUC), such as RankSVM in [START_REF] Joachims | Optimizing search engines using clickthrough data[END_REF], RankBoost in [START_REF] Freund | An efficient boosting algorithm for combining preferences[END_REF] or RankNet in [START_REF] Burges | Learning to rank using gradient descent[END_REF]. We refer to Liu (2011) (Part II) for a review of most practical approaches in ranking. However, the minimization of the AUC considers that ordering correctly the instances in the middle of the ranked list is as important as ordering those at the top, which conflicts with many applications that focus on the first best results. Some authors have proposed functionals or algorithms that focus on the top of the list, such as the p-norm push [START_REF] Rudin | Ranking with a p-norm push[END_REF] or the approach to optimize accuracy at the top presented in [START_REF] Boyd | Accuracy at the top[END_REF]. Additionally, Neyman-Pearson classification [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF] concerns the optimization of the true positive rate under a constrained false positive rate.

We also refer to this problem as pointwise ROC optimization (pROC). A thorough theoretical analysis of that problem was conducted in Chapter 5. Consider a sample of n data points as D n " tpx i , y i qu n i"1 Ă X ˆt1, . . . , Ku. For any similarity function s : X ˆX Ñ R `, introduce the estimators: p R ń psq " 1 n ´ÿ iăj spx i , x j q ¨Ity i ‰ y j u, and p R ǹ psq " 1 n `ÿ iăj spx i , x j q ¨Ity i " y j u, with n `:" ř iăj Ity i " y j u and n ´:" npn ´1q{2 ´n`. The pROC problem then writes:

(pROC): max

sPS p R ǹ psq subject to p R ń psq ď α, (7.5)
where S is a proposed family of similarity functions. While this problem encompasses very natural settings, such as the optimization of a biometric system for a specific level of security, there is a lack of practical approaches for it in the literature. Notable exceptions include Scott and Nowak (2006) (Section 7.2) and [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF] (Section VI-B), which are both based on partitioning the input space X . In that context, the objective of this section is to propose new approaches to pointwise ROC optimization. It first presents a simple experiment that learns linear similarities on simulated data, then proposes a generic gradient-descent based approach.

Exact Resolution for Bilinear Similarities

We illustrate on synthetic data that solving (7.5) for different values of α can optimize for different regions of the ROC curve. Let X Ă R d and let S be the set of bilinear similarities with norm-constrained matrices:

S " " s A : px, x 1 q Þ Ñ 1 2 `1 `xJ Ax 1 ˘ˇ A 2 F ď 1 * , where A 2 F " ř d i,j"1 a 2 ij is the Frobenius norm of A.
Note that when the data is scaled, i.e. }x} 2 " 1 for all x P X , we have s A px, x 1 q P r0, 1s for all x, x 1 P X and all s A P S. Our simple experiment features K " 3 classes, and observations belong to the sphere in R 3 . Following the notations of Chapter 5, we denote by θ x,c k the angle between the element x and the centroid c k of class k and set for all k P t1, 2, 3u: as well as c 1 :" pcospπ{3q, sinpπ{3q, 0q, c 2 :" e 2 and c 3 :" e 3 , with pe 1 , e 2 , e 3 q the vectors of the standard basis of R 3 . See Figure 7.1(a) for a graphical representation of the data. With the proposed family S, Eq. (7.5) writes:

F k pxq 9 I ! θ x,c k ă π 4 ) and p k " 1 3 ,
max A 1 n `ÿ iăj
I ty i " y j u ¨sA px i , x j q , such that 1 n ´ÿ iăj I ty i ‰ y j u ¨sA px i , x j q ď α and ||A|| 2 F ď 1.

Introduce the matrices N, P P R dˆd , defined as:

P " 1 2n `ÿ 1ďiăjďn I ty i " y j u ¨`x i x J j `xj x J i ˘, N " 1 2n ´ÿ 1ďiăjďn I ty i ‰ y j u ¨`x i x J j `xj x J i ˘.
With β " 2α ´1, one can further write Eq. (7.5) as: The solutions of the problem above can be expressed in closed form using Lagrangian duality. In particular, when the constraints are saturated, the solution s Aα is an increasing transformation of s Aα with A α " P ´λα N , where λ α is a positive Lagrange multiplier that decreases in α. By decreasing α, we trade-off the information contained in the positive pairs (α large, λ α close to zero), for that in the negative pairs (α small, λ α large). Changing α results in optimizing for different areas of the ROC curve, see Figure 7.1(b).

A Gradient-descent Approach for General Similarities

In this section, we propose an approach to learn a score function that solves pointwise ROC optimization by gradient descent in the case of bipartite ranking. Then, we discuss its extension to similarity ranking. Bipartite ranking considers a sample D n " tpx i , y i qu n i"1 Ă X ˆt´1, `1u and seeks to learn a score function s : X Ñ R `that ranks positive elements tx i | y i " `1, 1 ď i ď nu above negative elements tx i | y i " ´1, 1 ď i ď nu.

We consider the resolution of Eq. (7.5) with a linear frontier on a simple 2-dimensional example. In our example, an obvious solution always exists, but is very different when α varies in p0, 1q. To achieve this result, we built our sample tpx i , y i qu n i"1 by sampling n i.i.d. copies of a random pair pX, Y q P X ˆt´1, `1u. The r.v. X has the distribution F on r0, 1s 2 and the posterior probability ηpxq :" PtY " `1 | X " xu. They satisfy, for any x P X such that x " px 1 , x 2 q J :

F pxq " p4{πq ¨Itx 2 1 `x2 2 ď 1, 0 ď x 1 , 0 ď x 2 u, ηpxq " p2{πq ¨arctanpx 2 {x 1 q,
We call this distribution the quarter-circle distribution. We provide a representative sample in Fig. 7.2.

Denote by H and G respectively the distributions of the conditional random variables X|Y " ´1 and X|Y " `1. Consider the region R θ " tx P r0, 1s 2 | arctanpx 2 {x 2 q ď θu. Then, simple calculations give:

GpR θ q " P pX P R θ | Y " `1q " 4 π ˆθ ´θ2 π ˙, HpR θ q " P pX P R θ | Y " ´1q " 4θ 2 π 2 .
Hence, the optimal rejection region R α for pointwise ROC optimization at level α simply satisfies R α :" tx | θ x ď ? απ{2u, and we have:

GpR αq " PtX P R α | Y " `1u " 2 ? α ´α, HpR αq " PtX P R α | Y " ´1u " α.
Given a region defined by a linear classifier, the oriented distance to the separator plane of the region defines a linear score function. The oriented distance to the region is the distance to the boundary of the region, with a negative sign if the point belongs to the region and a positive sign otherwise. We provide a visualization of an example sample with the optimal rejection regions in Fig. 7.2. Fig. 7.3 shows the ROC curves of the score function associated to the optimal regions. With the parameterized logistic function σ λ : t Þ Ñ 1{p1 `expp´λtq, we can define relaxed versions of the true positive rate and false positive rate for the linear score function s w,b x Þ Ñ w J x as, respectively:

p Gpw, bq :" 1 n `ÿ Yi"`1 σ λ `wJ x i `b˘a nd p Hpw, bq :" 1 n ´ÿ Yi"´1 σ λ `wJ x i `b˘.
Drawing inspiration from the excess error for minimum volume set estimation of [START_REF] Scott | Learning minimum volume sets[END_REF] For the toy dataset, we use a standard gradient descent approach to minimize the objective of Eq. (7.6), combined with regular projections on the unit ball to satisfy the constraint. A representative experimental result is summarized in Fig. 7.4.

Another approach to optimize for different locations on the ROC curve proposed using classification with asymmetric costs [START_REF] Bach | Considering cost asymmetry in learning classifiers[END_REF], i.e. weighting differently errors on positive and negative instances. An extension of those results could lay the foundations for differentiable losses for similarity ranking, adapted to operational constraints that can be framed as pointwise ROC optimization. Clearly, the sampling strategies described in Chapter 6 can directly contribute to the efficient learning of such similarities. Another proposition of Chapter 5 is an approach to approximate the optimal scoring function by recursive partitioning of the input space, named the TreeRank algorithm, which is the subject of the next section.

Empirical Evaluation of TreeRank

The TreeRank algorithm, first introduced in Clémençon and Vayatis (2009), tackles bipartite ranking by recursively splitting the input space. At each step, TreeRank solves classification with asymmetric costs on a specific region of the input space, and with different weights. TreeRank was first introduced with a general splitting method, referred to as LeafRank, yet only implemented with coordinate splits or combinations of those. In this section, we first discuss the extension of the TreeRank algorithm to learning similarities functions. Precisely, we illustrate the output of TreeRank for similarity ranking, i.e. with symmetric LeafRank, on the product space X ˆX " R ˆR. Then, we show the capacities of averages of scores learned with TreeRank, so-called "ranking forests", to approximate smooth score functions despite the discrete nature of each individual score. Additionally, our experiment demonstrates that averaging random ranking trees can correct for a misspecification of the LeafRank algorithm. Our analysis does not cover misspecified splitting families of sets, as it always assumes that the optimal split belongs to the family. Finally, we underline that the flexibility of TreeRank implies that LeafRank could use complex proposed regions, possibly learned using usual gradient descent algorithms. In that view, TreeRank constitute a methodology for solving bipartite ranking as a global problem, and considering variants of the original algorithm is a promising route for future work.

Symmetric Proposal Regions

Following the recommendations of Chapter 5, we introduce a symmetric transformation of the input space, i.e. any function f : X ˆX Ñ Impf q such that f px, x 1 q " f px 1 , xq. Then, a natural way to extend the TreeRank algorithm to similarity functions, is to choose for LeafRank the collection of regions C, such that:

C :" x, x 1 P X ˆX | f px, x 1 q P D ( DPD ,
where D Ă PpImpf qq. The i-th element of the vector f px, x 1 q is written f i px, x 1 q. A first proposition, is to consider the transformation: Introduce the collection C diag defined as follows:

f px, x 1 q " p x ´x1 , x `x1 q J .
C diag :" ! x, x 1 P X ˆX M σf i px, x 1 q ě σA )
where i P t1, . . . , Du, σ P t´1, `1u and A P R. The collection C diag corresponds to the set of all coordinate splits in the transformed input space. Another possible approach is to set:

f px, x 1 q " px _ x 1 , x ^x1 q J ,
where x _ x 1 and x ^x1 respectively stand for the element-wise maximum and minimum of x and x 1 . We introduce the collection C sq of regions in X ˆX as follows:

C sq :" ! x, x 1 P X ˆX M `σf i px, x 1 q ě σA ˘b `σf i`D px, x 1 q ď σA ˘)
where i P t1, . . . , Du, σ P t´1, `1u, A P R and b is the standard XOR.

We illustrate with Fig. 7.5 the results of the outcome of the TreeRank algorithm with either one of these two proposed families, in a simple case where X " r0, 1s, F " 1 (F is the uniform distribution), K " 2 and:

PtY " 2|X " xu " 0.6 ¨Itx ě 0.5u `0.2.

As shown in Fig. 7.5, the output of those decision functions is discrete, an undesirable property in many applications. Furthermore, the nature of the algorithm implies that the output score is very dependent on the first splits. This property may impair the performance of TreeRank if the proposed family of splits is underspecified. An extension of TreeRank is the notion of ranking forests, which addresses to some extent both of these issues.

Approaching Continuous Scores with Ranking Forests

Ranking forests is the extension by Clémençon et al. (2013) for the TreeRank algorithm of the random forests proposed in [START_REF] Breiman | Random forests[END_REF]. The main idea of ranking forests is to aggregate the scores of several runs of a slightly randomized TreeRank algorithm on a subset of the data, by means of a ranking aggregation procedure as argued in Clémençon et al. (2013) or simply by averaging each score function. While the output of a ranking forest is not continuous as well in theory, averaging enough trees may produce a ranking rule that is hardly distinguishable numerically from a continuous score function.

To illustrate the properties of ranking forests, we show that they can recover an optimal score for the quarter-circle distribution presented above. We considered fitting each individual tree on a randomly selected subsample of 500 elements of a training set of n " 4, 000 points. Final performance is evaluated on an independently generated set of n " 4, 000 points as well. We learned in total 30 full binary trees, each of depth 4. Each LeafRank algorithm was composed of a full binary tree of recursive coordinate splits and depth 3. The aggregation of the scores was their simple average. Results are summarized in Fig. 7.7 and show that the ROC of our model is indistinguishable from the known optimal ROC on the test set. Additionally, Fig. 7.6 shows that our output score is hardly distinguishable from a continuous function on the square r0, 1s 2 . The take-home message from this example is: even though the proposed family of the LeafRank procedure is misspecified for our distribution, which breaks down the mathematical analysis of TreeRank, we can retrieve the optimal solution by averaging biased solutions. While this example is conclusive on our simple example, we can doubt its straightforward application to learning in high-dimensional spaces and/or large samples, due to its computational complexity and/or potential inaccuracy.

In that regard, more complicated splitting methods could be chosen, such as a linear decision function. In that example, LeafRank could consists in fitting an asymmetrically weighted SVM. Another approach would be learning a split of the input space using a gradient-based approach. To obtain a continous output, future work could consider smoothing the output score of TreeRank.

Conclusion

This chapter first proposed simple illustrations of optimization procedures for the pointwise ROC optimization problem presented in Chapter 5, on naive but explanatory toy examples. While the generality of those approaches is yet to prove, our examples suggest possible solutions to a problem that is rarely addressed practically in the literature, apart from few notable examples that are based on recursive partitioning. The chapter then illustrated the extension of the TreeRank algorithm to learning similarity functions instead of score functions, presented formally in Chapter 5. That approach is theoretically justified, and specifically tackles solving similarity rankingi.e. similarity learning viewed as ranking on a product space, presented in Chapter 5 -but we address its limitations in practical scenarios. Those limitations can be alleviated with for example ranking forests.

In general, this chapter is more prospective than the others and proposes to address the optimization challenges induced by Chapter 5. As such, it initiates a dialogue between the broad statistical ranking literature of the second part of the 2000s and the rapidly expanding deep metric learning literature. It is motivated by the systematic evaluation of biometric systems with tools designed for ranking, as well as the recent rapid increase of propositions for deep metric learning that are backed solely by empirical results. In that regard, it is a promising direction for future work.

The recent improvements in similarity learning due to deep neural networks have fostered the adoption of biometric technologies, and in particular that of facial recognition. In that regard, while most of the early concerns focused on the privacy risks induced by the technology, recent research and news outlets insisted on the risks induced by its lack of reliability. The subject of the next part (Part III) of the thesis is to propose machine learning techniques to address those risks.

Precisely, we first propose a method to solve the label ranking problem with classification data (Chapter 8), which stands as a principled approach to the specific identification problem of recovering most likely suspects. Secondly, Chapter 9 addresses the issue of representativeness of a statistical population, a recurring topic in biometrics, using a reweighting scheme. Finally, while reweighting corrects some form of bias due to representation, other type of bias are integrated in the ground truth data, and can only be corrected with stronger explicit constraints on the learning procedure. In that regard, Chapter 10 proposes theoretical analyses and techniques, that address balancing predictive performance and flexible user-defined fairness criteria.

Part III

Reliable Machine Learning

129 Chapter 8

Ranking the Most Likely Labels Summary: In multiclass classification, the goal is to learn a classification rule g : R q Ñ Y, that accurately maps the random variable X P R q to the label Y P Y " t1, . . . , Ku. In a wide variety of situations, the task targeted may be more ambitious, and consists in sorting all the possible label values y that may be assigned to X by decreasing order of the posterior probability η y pXq " PtY " y | Xu. This chapter is devoted to the analysis of this statistical learning problem, referred to as label ranking here. While Part II of the thesis focuses on similarity ranking, a direct mathematization of the 1:1 biometric identification problem, label ranking corresponds to a different operational setting in biometrics. Indeed, label ranking is the problem of returning a list of most likely suspects. Formally, label ranking can be viewed as a specific variant of ranking median regression (RMR). Specifically, rather than observing a random permutation Σ assigned to the input vector X and drawn from a Bradley-Terry-Luce-Plackett model with conditional preference vector pη 1 pXq, . . . , η K pXqq, the sole information available for training a label ranking rule is the label Y ranked on top, namely Σ ´1p1q. Inspired by recent results in RMR, we prove that under noise conditions, the One-Versus-One (OVO) approach to classification yields, as a by-product, an optimal ranking of the labels with large probability. Our theoretical analysis builds on top of finite-bounds for binary classification (Chapter 2) and probabilistic models for ranking (Chapter 4).

Introduction

In the standard formulation of the multiclass classification problem, pX, Y q is a random pair defined on a probability space pΩ, F, Pq with unknown joint probability distribution P , where Y is a label valued in Y " t1, . . . , Ku with K ě 3 and the r.v. X takes its values in a possibly high-dimensional Euclidean space, say R q with q ě 1 and models some input information that is expected to be useful to predict the output variable Y . The objective pursued is to build from training data D n " tpX 1 , Y 1 q, . . . , pX n , Y n qu, supposed to be independent copies of the generic pair pX, Y q, a (measurable) classifier g : R q Ñ Y that nearly minimizes the misclassification error:

Lpgq :" PtY ‰ gpXqu.

(8.1)

Let ηpxq " pη 1 pxq, . . . , η K pxqq be the vector of posterior probabilities: η k pxq " PtY " k | X " xu, for x P R q and k P t1, . . . , Ku. For simplicity, we assume here that the distribution of the r.v. ηpXq is continuous, so that the η k pXq's are pairwise distinct with probability one, i.e. for all pk, lq P t1, . . . , Ku with k ă l, then Ptη k pXq " η l pXqu " 0. Denoting by S K the symmetric group of order K (i.e. the group of permutations of t1, . . . , Ku), another natural statistical learning goal in this setup, halfway between multiclass classification and estimation of the posterior probability function ηpxq and referred to as label ranking throughout the chapter, is to learn, from the training data D n , a ranking rule s, i.e. a measurable mapping s : R q Ñ S K , such that the permutation spXq sorts, with 'high probability', all possible label values k in Y by decreasing order of the posterior probability η k pXq, that is to say in the same order as the permutation σ X defined by: @x P R q ,

η σ ˚´1 x p1q pxq ą η σ ˚´1 x p2q pxq ą . . . ą η σ ˚´1 x pKq pxq, (8.3)
and σ X is a random permutation that satisfies Ptσ X " σu " PtX P tx P R q | σ x " σuu for any σ P S K . This label ranking corresponds to a specific operational setting in biometrics, but not to the similarity ranking setting studied in Part II, since label ranking can be seen as returning a list of most likely suspects. Equipped with this notation, observe that g ˚pxq " σ ˚´1

x p1q for all x P R q . Given a loss function d : S K ˆSK Ñ R `(i.e. a symmetric measurable mapping s.t. dpσ, σq " 0 for all σ P S K ), one may formulate label ranking as the problem of finding a ranking rule s which minimizes the ranking risk :

Rpsq " E rd pspXq, σ X qs .

(8.4)

Except when K " 2 and in the case when the loss function d considered only measures the capacity of the ranking rule to recover the label that is ranked first, that is to say when dpσ, σ 1 q " Itσ ´1p1q ‰ σ 1´1 p1qu (in this case, Rpsq " Ptg ˚pX q ‰ spXq ´1p1qu), the nature of the label ranking problem significantly differs from that of multiclass classification.

There is no natural empirical counterpart of the risk (8.4) based on the observations D n , which makes the ERM strategy inapplicable in a straightforward fashion. It is the goal of the present chapter to show that the label ranking problem can be solved, under appropriate noise conditions, by means of the One-Versus-One (OVO) approach to multiclass classification. The learning strategy proposed is directly inspired from recent advances in consensus ranking and ranking median regression (RMR), see [START_REF] Korba | A learning theory of ranking aggregation[END_REF] and Clémençon et al. (2018). an output r.v. Σ that takes its values in the group S K (in recommending systems, Σ may represent the preferences over a set of items indexed by k P t1, . . . , Ku of a given user, whose profile is described by the features X). The goal is to find a ranking rule s that minimizes ErdpspXq, Σqs, that is to say, for any x P R q , a consensus/median ranking spxq P S K related to the conditional distribution of Σ given X " x w.r.t. the metric dp., .q.

In this chapter, by means of a coupling technique, we show that the label ranking problem stated above can be viewed as a variant of RMR where the output ranking is very partially observed in the training stage, through the label ranked first solely. While previous authors [START_REF] Korba | A structured prediction approach for label ranking[END_REF] or [START_REF] Brinker | A reduction of label ranking to multiclass classification[END_REF]) tackled RMR with partial information, they do not feature the strong theoretical guarantees that we provide. Based on this analogy, the main result of the chapter shows that the OVO method permits to recover the optimal label ranking with high probability, provided that noise conditions are fulfilled for all binary classification subproblems. Incidentally, the analysis carried out provides statistical guarantees in the form of (possibly fast) learning rate bounds for the OVO approach to multiclass classification under the hypotheses stipulated. Our contribution focuses on label ranking, but builds on top of both the OVO approach and recent literature in RMR. Finally, various numerical experiments corroborate empirically the theoretical results established in this chapter.

The chapter is organized as follows. In section 8.2, the OVO methodology for multiclass classification is recalled at length, together with recent results in RMR. The main results of the chapter are stated in section 8.3: principally, a coupling result connecting label ranking to RMR and statistical guarantees for the OVO approach to label ranking in the form of nonasymptotic probability bounds. Numerical experiments are displayed in section 8.4, while some concluding remarks are collected in section 8.5.

Preliminaries

To begin with, we recall the OVO approach for defining a multiclass classifier from binary classifiers. Basic hypotheses and results related to Ranking Median Regression (RMR) are next briefly described.

From Binary to Multiclass Classification

A classifier g is entirely characterized by the collection of subsets of the feature space R q : pS g p1q, . . . , S g pKqq, where S g pkq " tx P R q : gpxq " ku for k P t1, . . . , Ku. Observe that the S k 's are pairwise disjoint and their union is equal to R q . Hence, they form a partition of R q , except that it may happen that a certain subset S k pgq is empty, i.e. a certain label k is never predicted by g.

The OVO approach. Partitioning the feature space R q in more than two subsets may lead to practical difficulties and certain learning algorithms such as Support Vector Machines (SVM's) are originally tailored to the binary situation (i.e. to the case K " 2). In this case, a natural way to extend such algorithms is the 'One-Versus-One' approach to multi-class classification [START_REF] Hastie | Classification by pairwise coupling[END_REF][START_REF] Moreira | Improved pairwise coupling classification with correcting classifiers[END_REF][START_REF] Allwein | Reducing multiclass to binary: A unifying approach for margin classifiers[END_REF][START_REF] Fürnkranz | Round robin classification[END_REF][START_REF] Wu | Probability estimates for multi-class classification by pairwise coupling[END_REF]. It consists in running the binary classification algorithm KpK ´1q{2 times once for each binary subproblem. For any 1 ď k ă l ď K, the binary subproblem of k against l is based on the fraction of the training data with labels in tk, lu only:

D k,l " tpX i , Y i q | Y i P tk, lu, i " 1, . . . , nu ,
and the binary classification algorithm outputs a classification rule g k,l : R q Ñ t´1, `1u with risk:

L k,l pg k,l q " PtY k,l ‰ g k,l pXq | Y P tk, luu,
as small as possible, where Y k,l " ItY " lu ´ItY " ku. The OVO approach combines, for any possible input value x P R q , the binary predictions g k,l pxq to produce a multi-class classifier s g : R q Ñ t1, . . . , Ku with minimum risk Rps gq. A possible fashion of combining the results of the KpK ´1q{2 'duels' is to take as predicted label which has won the largest number of duels (and stipulate a rule for breaking possible ties). The rationale behind this OVO approach lies in the fact that: g ˚pxq " arg max kPt1, ..., Ku N k pxq, (8.5)

where, for all pk, xq P t1, . . . , Ku ˆRq , N k pxq denotes the number of duels won by label k with optimal/Bayes classifiers for all binary subproblems, namely:

N k pxq " ÿ lăk Itg l,k pxq " `1u `ÿ kăl Itg k,l pxq " ´1u,
where g l,m pxq " 2Itη m pxq{pη m pxq `ηl pxqq ą 1{2u ´1 is the minimizer of the risk L l,m for l ă m. The proof is straightforward. Indeed, it suffices to observe that, for all i P t1, . . . , Ku, N σx piq " K ´i.

Remark 8.1. (One-Versus-All) An alternative to the OVO approach in order to reduce multiclass classification to binary subproblems and apply the SVM methodology consists in comparing each class to all of the others in K two-class duels. A test point is classified as follows: the signed distances from each of the K separating hyperplanes are computed, the winner being simply the class corresponding to the largest signed distance. However, other rules have been proposed in [START_REF] Vapnik | Statistical Learning Theory[END_REF] and in [START_REF] Weston | Support vector machines for multi-class pattern recognition[END_REF].

Label Ranking. As underlined in the Introduction section, rather than learning to predict the most likely label given X, it may also be desirable to rank all possible labels according to their conditional likelihood. The goal is then to recover the permutation σ X defined through (8.3). Practically, this boils down to building a predictive rule spxq from the training data pX 1 , Y 1 q, . . . , pX n , Y n q that maps R q to S K and minimizes the ranking risk (8.4), where dp., .q is an appropriate loss function defined on S K ˆSK . For instance, one may consider Itσ ‰ σ 1 u or the Hamming distance ř K k"1 Itσpkq ‰ σ 1 pkqu to measure the dissimilarity between two permutations σ and σ 1 in S K . Classic metrics on S K (see [START_REF] Deza | Metrics on permutations[END_REF]) also provide natural choices for the loss function, including:

• the Kendall τ distance: @pσ, σ 1 q P S 2 K , d τ pσ, σ 1 q " ÿ iăj Itpσpiq ´σpjqq ¨pσ 1 piq ´σ1 pjqq ă 0u,

• the Spearman footrule: @pσ, σ 1 q P S 2 K ,

d 1 pσ, σ 1 q " K ÿ i"1 ˇˇσpiq ´σ1 piq ˇˇ,
• the Spearman ρ distance: @pσ, σ 1 q P S 2 K ,

d 2 pσ, σ 1 q " K ÿ i"1 pσpiq ´σ1 piqq 2 .
As shall be explained below, the label ranking problem can be viewed as a variant of the standard ranking median regression problem.

Ranking Median Regression

This problem of minimizing (8.4) shares some similarity with that referred to as ranking median regression in Clémençon et al. (2018), also called label ranking sometimes, see e.g. [START_REF] Tsoumakas | Mining multi-label data[END_REF] and [START_REF] Vembu | Label ranking algorithms: A survey[END_REF]. In this supervised learning problem, the output associated with the input variable X is a random vector Σ taking its values in S K (expressing the preferences on a set of items indexed by k P t1, . . . , Ku of a user with a profile characterized by X drawn at random in a certain statistical population) and the goal pursued is to learn from independent copies pX 1 , Σ 1 q, . . . , pX n , Σ n q of the pair pX, Σq a (measurable) ranking rule s : R q Ñ S K that nearly minimizes:

Rpsq :" ErdpΣ, spXqqs.

(8.6)

The name ranking median regression arises from the fact that any rule mapping X to a median of Σ's conditional distribution given X w.r.t. the metric/loss d (refer to [START_REF] Korba | A learning theory of ranking aggregation[END_REF] for a statistical learning formulation of the consensus/median ranking problem) is a minimizer of (8.6), see Proposition 5 in Clémençon et al. (2018). In certain situations, the minimizer of (8.6) is unique and a closed analytic form can be given for the latter, based on the pairwise probabilities: p i,j pxq " PtΣpiq ă Σpjq | X " xu :" 1 ´pj,i pxq for 1 ď i ă j ď K and x P R q .

Assumption 8.2. For all x P R q , we have: @pi, k, lq P t1, . . . , Ku 3 , p i,j pxq ‰ 1{2 and p i,j pxq ą 1{2 and p j,k pxq ą 1{2 ñ p i,k pxq ą 1{2. (8.7) Indeed, when choosing the Kendall τ distance d τ as loss function, it has been shown that, under Assumption 8.2, referred to as strict stochastic transitivity, the minimizer of (8.6) is almost-surely unique and given by: @k P t1, . . . , Ku, with probability one:

s X pkq " 1 `ÿ l‰k Itp k,l pXq ă 1{2u. (8.8)
Remark 8.3. (Conditional BTLP model) A Bradley-Terry-Luce-Plackett model for Σ's conditional distribution given X, P Σ|X , assumes the existence of a hidden preference vector wpXq " pw 1 pXq, . . . , w K pXqq, where w k pXq ą 0 is interpreted as a preference score for item k of a user with profile X, see e.g. [START_REF] Bradley | Rank analysis of incomplete block designs: I. the method of paired comparisons[END_REF], [START_REF] Luce | Individual Choice Behavior[END_REF] or [START_REF] Plackett | The analysis of permutations[END_REF]. The conditional distribution of Σ ´1 given X can be defined sequentially as follows: Σ ´1p1q is distributed according to a multinomial distribution of size 1 with support S 1 " t1, . . . , Ku and parameters w k pXq{ ř l w l pXq and, for k ą 1, Σ ´1pkq is distributed according to a multinomial distribution of size 1 with support S k " S 1 ztΣ ´1p1q, . . . , Σ ´1pk ´1qu with parameters w l pXq{ ř mPS k w m pXq, l P S k . The conditional pairwise probabilities are given by p k,l pXq " w k pXq{pw k pXq `wl pXqq and one may easily check that Assumption 8.2 is fulfilled as soon as the w k pXq's are pairwise distinct with probability one. In this case, s ˚pX q is the permutation that sorts the w k pXq's in decreasing order.

In Clémençon et al. (2018), certain situations where empirical risk minimizers over classes of ranking rules fulfilling appropriate complexity assumptions can be proved to achieve fast learning rates (i.e. faster than Op1{ ? nq) have been investigated. More precisely, denoting by ess inf Z the essential infimum of any real valued r.v.Z, the following 'noise condition' related to conditional pairwise probabilities was considered.

Assumption 8.4. The pairwise probabilities p i,j satisfy: 2018)), a bound for the probability that the empirical risk minimizer differs from the optimal ranking rule at a random point X can be immediately derived.

H "

Label Ranking

We now describe at length the connection between label ranking and RMR and state the main results of the chapter.

Label Ranking as RMR

The major difference of the multi-class classification context with label ranking lies in the fact that only the partial information σ ˚´1 X p1q is observable in the presence of noise -under the form of the random label Y assigned to X (σ ˚´1 X p1q is the mode of the conditional distribution of Y given X) -in order to mimic the optimal rule σ X .

Lemma 8.5. Let pX, Y q be a random pair on the probability space pΩ, F. Pq. One may extend the sample space so as to build a random variable Σ that takes its values in S K and whose conditional distribution given X is a BTLP model with preference vector ηpXq " pη 1 pXq, . . . , η K pXqq such that Y " Σ ´1p1q with probability one. (8.10) l Ñ X k is transitive (i.e. l Ñ k and k Ñ X m ñ l Ñ X m), in other words that the scoring function:

p spXqpkq :" 1 `ÿ k‰l I tp g k,l pXq " ´1u , (8.14)

" 1 `ÿ k‰l I tk Ñ X lu , for k P t1, . . . , Ku, defines a permutation, which, in addition, coincides with σ X , cf Eq. (8.12). The equivalence between the transitivity of Ñ X , the acyclicity of G X and the membership of p spXq in S K is straightforward, but we refer to Theorem 5's proof of Eq. ( 9) in [START_REF] Korba | A learning theory of ranking aggregation[END_REF] for more details.

Remark 8.7. The quantity (8.14) can be related to the Copeland score, see [START_REF] Copeland | A reasonable social welfare function[END_REF]: the score p spXqpkq of label k being equal to 1 plus the number of duels it has lost, while its Copeland score C X pkq is its number of victories minus its number of defeats, so that p spXqp.q " pK `1 ´CX p.qq {2.

OVO Approach to Label Ranking

Inputs. Class G of classifier candidates. Training classification dataset Dn " tpX1, Y1q, . . . , pXn, Ynqu. Query point x P R q .

1. (Binary classifiers.) For k ă l, based on D k,l " tpXi, Yiq : Yi P tk, lu, i " 1, . . . , nu, compute the ERM solution to the binary classification problem:

p g k,l " arg min gPG p L k,l pgq.
2. (Scoring.) Compute the predictions p g k,l pxq and the score for the query point x: p spxqpkq " 1 `ÿ l‰k Itp g k,l pxq " ´1u.

Output. Break arbitrarily possible ties in order to get a prediction p σx in SK at x from p spxq. While the rest of the paper focuses on events where G X is transitive, the end of this subsection present practical approaches in the opposite case. If G X is not transitive, or equivalently when p spXq R S K , one may build a ranking p σ X from the scoring function (8.14) by breaking ties in an arbitrary fashion, as proposed below for simplicity. Alternatives could be considered of course. The issue of building a ranking/permutation of the labels in t1, . . . , Ku from (8.14) can be connected with the feedback set problem for directed graphs, see e.g. [START_REF] Battista | Graph Drawing: Algorithms for the Visualization of Graphs[END_REF]: for a directed graph, a minimal feedback arcset is a set of edges of smallest cardinality such that a directed acyclic graph is obtained when reversing the edges in it. We refer to e.g. [START_REF] Festa | Feedback Set Problems[END_REF] for algorithms.

Statistical Guarantees for Label Ranking

It is the purpose of the subsequent analysis to show that, provided that the conditions listed below are fulfilled, the ranking rule σ X can be fully recovered with high probability through the OVO approach previously described. We denote by F the marginal distribution of the input variable X, by F k the conditional distribution of X given Y " k and set p k " PtY " ku for k P t1, . . . , Ku.

Assumption 8.8. There exists α P r0, 1s and B ą 0 such that: for all k ă l and t ě 0, P t|2 ¨ηk,l pXq ´1| ă tu ď Bt α 1´α .

where β " pB 1´α q{p p1 ´αq 1´α α α q. The proof only differs slightly from that of (Bousquet et al., 2003, page 202 therein), as it features implications of 8.10. Set β 0 " βpp k `pl q α , Equation (8.16) implies, for any f P F k,l , that, with T pf q " ? β 0 ¨pP f q α{2 : Varpf q ď P gpXq ‰ g k,l pXq ( ď βpL k,l pgq ´Lk ,l q α " βpp k `pl q α ¨pP f q α " T 2 pf q.

(8.17)

The function T pf q controls the variance of the elements in F k,l , and is used to reweights its instances before applying Talagrand's inequality.

The complexity of the proposed family of functions is controlled using the notion of Rademacher average, presented in Definition 2.10 of Chapter 2 in a the restricted setting of classifiers. Let F be a class of functions, its Rademacher average R n pFq is defined as:

R n pFq :" E σ sup f PF 1 n n ÿ i"1 σ i f pX i , Y i q .
Introduce F k,l as the star-hull of F k,l , i.e. F k,l " tαf : α P r0, 1s, f P F k,l u, we define two functions that characterize the properties of the problem of interest, and are required to apply Theorem 5.8 of [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF]:

wprq " sup f PF k,l :P f ďr T pf q and ψprq " ER n tf P F k,l : T pf q ď ru.

(8.18)

Finally Theorem 5.8 of [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] implies that, for all δ ą 0, with r 0 pδq the solution of:

r " 4ψpwprqq `2wprq d 2 log 2 δ n `16 log 2 δ 3n , (8.19)
we have that, for any k ă l, pk, lq P t1, . . . , Ku 2 and any n P N ˚, w.p. ě 1 ´δ,

L k,l pp g k,l q ´Lk ,l ď 2 ˆinf gPG L k,l pgq ´Lk ,l
˙`r 0 pδq p k `pl .

Now, we can conclude by combining this result with properties of w and ψ, that originate from the noise assumption and the control on the complexity of G, respectively. Assumption 8.9 states that the proposed class G is of VC-dimension V . Permanence properties of VC-classes of functions, see van der Vaart and Wellner (1996) (section 2.6.5), imply that F k,l is also VC. It follows from P. [START_REF] Bartlett | Localized rademacher complexities[END_REF] that:

ψprq ď Cr c V n log n,
where C ą 0 is an universal constant.

Plugging this result into Equation (8.19) gives:

r 0 pδq ď 2wpr 0 pδqq ? n « 2C a V log n, `c2 log 2 δ ff `16 log 2 δ 3n .
Combining it with the definition of w in (8.18) and the control on the variance laid forth in (8.17) yields:

r 0 pδq ď rr 0 pδqs α{2 2 ? β 0 ? n « 2C a V log n `c2 log 2 δ ff `16 log 2 δ 3n . (8.20)
Equation (8.20) is a variational inequality, and an upper bound on the solution can be derived directly from [START_REF] Zou | Learning from uniformly ergodic markov chains[END_REF] (Lemma 2). It writes:

r 0 pδq ď max # ˆ16β 0 n ˙1 2´α " 2C a V log n `a2 logp2{δq ı 2 2´α , 32 logp2{δq 3n + .
Using the convexity of x Þ Ñ x 2{p2´αq , the right-hand side of the above inequality can be upperbounded, which leads to:

r 0 pδq ď 2 ¨max # ˆ16β 0 n ˙1 2´α " `4C 2 V log n ˘1 2´α `p2 logp2{δqq 1 2´α ı , 32 logp2{δq 3n + . (8.21)
Assumption 8.10 implies that pp k `pl q ´1 ď 1{ . Introducing r ˚pδq " pp k `pl q ´1r 0 pδq, Equation (8.21) combined with the definition of β 0 implies:

r ˚pδq ď 2 ¨max # ˆ16β 2´2α n ˙1 2´α " `4C 2 V log n ˘1 2´α `p2 logp2{δqq 1 2´α ı , 32 logp2{δq 3 n + . (8.22)
Introduce n 0 pδ, α, , B, V q as the lowest n such that the first term in the maximum in Equation (8.22) dominates the second term, it satisfies:

n 1´α 2´α 0 " `4C 2 V logpn 0 q ˘1 2´α `p2 logp2{δqq 1 2´α ı ě 32 logp2{δq 3 r16β α s 1 2´α , (8.23)
and so does any n ě n 0 .

To conclude, we have proven that for any δ P p0, 1q, for any n ě n 0 pδ, α, , B, V q, we have that w.p ě 1 ´δ,

L k,l pp g k,l q ´Lk ,l ď 2 ˆinf gPG L k,l pgq ´Lk ,l ˙`r ˚pδq, with r ˚pδq " 2 ˆ16β n 2´2α ˙1 2´α " `4C 2 V log n ˘1 2´α `p2 logp2{δqq 1 2´α ı . (8.24)
The following result builds on top of Lemma 8.11 to provide nonasymptotic bounds for the ranking risk R P pp σ X q of the OVO ranking rule in the case where the loss function is Itσ ‰ σ 1 u, i.e. for the probability of error. Extension to any other loss function dp., .q is straightforward, insofar as we obviously have dpσ X , p σ X q ď max pσ,σ 1 qPS 2 K dpσ, σ 1 q ˆItp σ X ‰ σ X u with probability one.

Theorem 8.12. Suppose that Assumptions 8.8-8.10 are fulfilled. Then, for all δ P p0, 1q, we have @n ě n 0 pδ, α, , B, V q, with probability (w.p.) at least 1 ´δ:

P tp σ X ‰ σ X | D n u ď β ε # ˆK 2 ˙rα n ˜δ `K 2 ˘¸`ÿ kăl 2 ˆinf gPG L k,l pgq ´Lk ,l ˙α+ ,
where X denotes a r.v. drawn from F , independent from the training data D n , L k,l " L k,l pg k,l q, β " βpα, Bq and with h :" hpB, a, q,

r n pδq " 2 p1{pnhqq 1 2´α ˆ"`6 4C 2 V log n ˘1 2´α `p32 logp2{δqq 1 2´α ı .
Proof. Fix δ P 0, 1 and let 1 ď k ă l ď K. Assumption 8.8 implies that the Mammen-Tsybakov noise condition is fulfilled for the binary classification problem related to the pair pX, Y q given that Y P tk, lu. When Assumptions 8.9-8.10 are also satisfied, a possibly fast rate bound for the risk excess of the empirical risk minimizer p g k,l can be established, as stated in Lemma 8.11.

Observe that the probabilities appearing in this proof are conditional probabilities given the training sample D n and, as a consequence, must be considered as random variables. However, to simplify notations, we omit to write the conditioning w.r.

č kăl p g k,l pXq " g k,l pXq ( Ă tσ X " p σ X u ,
with probability one, combining the bound above with the union bound gives that, for all δ P p0, 1q, w.p. ě 1 ´δ:

P tσ X ‰ p σ X u ď ÿ kăl P p g k,l pXq ‰ g k,l pXq ( ď β ε # ˆK 2 ˙rα n ˜δ `K 2 ˘¸`ÿ kăl 2 ˆinf gPG L k,l pgq ´Lk ,l
˙α+ .

Hence, for the RMR problem related to the partially observed BTLP model detailed in subsection 8.3.1, the rate bound achieved by the OVO ranking rule in Theorem 8.12 is of order n ´α{p2´αq , ignoring the bias term and the logarithmic factors. In the case α " 1, it is exactly the same rate as that attained by minimizers of the ranking risk in the standard RMR setup, as stated in Proposition 7 in [START_REF] Korba | A learning theory of ranking aggregation[END_REF]. Whereas situations where the OVO multi-class classification may possibly lead to 'inconsistencies' (i.e. where the binary relationship Ñ X is not transitive) have been exhibited many times in the literature, no probability bound for the excess of classification risk of the general OVO classifier, built from ERM applied to all binary subproblems, is documented to the best our knowledge. Hence, attention should be paid to the fact that, as a by-product of the argument of Theorem 8.12's proof, generalization bounds for the OVO classifier:

s gpXq " p σ ´1 X p1q,
can be established, as stated in Corollary 8.14 below. More generally, the statistical performance of the label ranking rule p σ x produced by the method described in subsection 8.3.2 can be assessed for other risks. For instance, rather than just comparing the true label Y assigned to X to the label p σ ´1 X p1q ranked first, as in OVO classification approach, one could consider k pY, p σ X q, with k py, σq " Ity R tσ ´1p1q, . . . , σ ´1pkquu for all py, σq P t1, . . . , Ku ˆSK , equal to 1 when Y does not appear in the top k list and to 0 otherwise, where k is fixed in t1, . . . , Ku. For any ranking rule s : R q Ñ S K , the corresponding risk is then:

W k psq " Er k pY, spXqqs. (8.29)
Set W k " min s W k psq, where the minimum is taken over the set of all possible ranking rules s.

The argument leading to Theorem 10 can be adapted to prove a rate bound for the risk excess of the OVO ranking rule σ . :

x P R q Þ Ñ σ x .
Proposition 8.13. Let k P t1, . . . , Ku be fixed. Then:

W k " W k pσ . q.
Suppose in addition that Assumptions 8.8-8.10 are fulfilled. Then, for all δ P p0, 1q, we have @n ě 1 w.p. ě 1 ´δ:

W k pp σ . q ´W k ď β ε ˆK k ˙kpK ´kq ˆ˜r α n ˜δ `K 2 ˘¸`2 ¨max m‰l ˆinf gPG L l,m pgq ´Ll ,m ˙α¸.
Proof. Let us first show that W k " W k pσ . q.

For any ranking rule s and all x P R q , we define:

Top k pspxqq " tspXq ´1p1q, . . . , spXq ´1pkqu, and also set Top k pxq " Top k pσ x q. Indeed, for any ranking rule s, we can write:

W k psq " E rE r k pY, spXqq | Xss ,
and we almost-surely have: Hence, the collection of optimal ranking rules regarding the risk (8.29) coincides with the set of ranking rules such that (8.31) holds true with probability one. Observe that, with probability one,

E r k pY, spXqq | Xs " K ÿ l"1 η l pXqItl R Top k pspXqqu. ( 8
I tY R Top k pspXqqu ´I Y R Top k pXq ( ď I Top k pXq ‰ Top k pspXqq ( ,
for any ranking rule spxq, so that:

W k psq ´W k ď P Top k pspXqq ‰ Top k pXq ( .
In addition, notice that:

W k pp σ X q ´W k ď P Top k pXq ‰ Top k pp σ X q ( " ÿ LĂY: #L"k P Top k pXq " L, Top k pXq ‰ Top k pp σ X q ( , ď ÿ LĂY: #L"k ÿ lPL, mRL P p g l,m pXq ‰ g l,m pXq ( , ď β ε ˆK k ˙kpK ´kq ˆ˜r α n ˜δ `K 2 ˘¸`2 ¨max m‰l ˆinf gPG L l,m pgq ´Ll ,m
˙α¸, using (8.27).

As stated above, since we have W 1 psq " Rpg s q where g s : x Þ Ñ pspxqq ´1p1q for any label ranking rule spxq, in the case k " 1 the result above provides a generalization bound for the excess of misclassification risk of the OVO classifier s gpxq " p σ ´1

x p1q, that we provide in Corollary 8.14 below.

Corollary 8.14. Suppose that Assumptions 8.8-8.10 are fulfilled. Then, for all δ P p0, 1q, we have @n ě n 0 pδ, α, , B, V q, w.p. ě 1 ´δ:

Lps gq ´L˚ď β ε KpK ´1q ˆ˜r α n ˜δ `K 2 ˘¸`2 ¨max k‰l ˆinf gPG L k,l pgq ´Lk ,l
˙α¸.

Experimental Results

Proof. It suffices to observe that we almost-surely have:

Lps gq ´L˚" E X r|ηpXq ´1{2| ˆIts gpXq ‰ g ˚pX qus ď P ts gpXq ‰ g ˚pX qu " P p σ ´1 X p1q ‰ σ ˚´1 X p1q ( ď P tp σ X ‰ σ X u and to apply next the bound stated in Theorem 8.12.

Experimental Results

This section first illustrates the results of Theorem 8.12 using simulated datasets, of which distributions satisfy Assumption 8.8, for certain values of the noise parameter α, highlighting the impact/relevance of this condition. In the experiments based on real data next displayed, the OVO approach to top-k classification, cf Eq. (8.29), is shown to surpass rankings that rely on the scores output by multiclass classification algorithms.

Synthetic Data Experiments

In this subsection, we illustrate the fast bounds proposed in this chapter by selecting and generating data from very specific distributions. Introduce the function h α , which for any α P r0, 1s:

h α pxq " 1 2 `1 2 pxq|2x ´1| 1´α α ,
where pxq " 2I t2x ą 1u ´1. It has good properties with regard to the usual Mammen-Tsybakov noise condition introduced in [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF]. We define a warped version h α,x0 of this function h α such that:

h α,x0 pxq " $ & % h α,x0 " h α ´x 2x0 ¯if x ă x 0 , h α,x0 " h α ´1 2 `x´x0 2p1´x0q ¯if x ě x 0 .
We use this function to define the η k 's by recursion, and assume that X follows a uniform distribution on the interval r0, 1s. Introduce a depth parameter D and assume that the variable Y belong to K " 2 D`1 classes. Let b pdq 2 pkq describe the decomposition in base 2 of the value k, i.e. k " ř D d"0 2 b pdq 2 pkq . We introduce an evenly spaced grid over r0, 1s as the set of values

x pd,kq " D ÿ d"1 2 ´bpdq 2 pkq ,
for k P t0, 2 D`1 u. Finally, we set:

η k pxq " D ź d"0 h α,x pd,kq pxq.
By varying the parameter α, one can set the classification problems to be more complicated or more easy. If α is close to 1, the problems are very simple. If α is close to 0, the problems are more arduous.

To implement the procedure described in Figure 8.1, we learn decision stumps in r0, 1s, i.e. we optimize over the family of functions G " tg s, | s P r0, 1s, P t´1, `1uu, where, for any x P r0, 1s:

g s, pxq " 2I tpx ´sq ě 0u ´1.

n P ď iPt1,2,3,4u t10 i , 3 ˆ10 i u Y t10 5 u.
One sees that learning is fast when α is close to 1, as expected. Figure 8.5 shows that the average Kendall's τ distance decreases quickly when α is close to 1, as does the the proportion of cycles in predictions, see Figure 8.3. On the other hand, due to the difficulty of predicting a complete permutation, the influence of the noise parameter on the evolution of the probability of error when n grows is more subtle, see Figure 8.4.

Real Data Experiments

The MNIST dataset is composed of 28 ˆ28 grayscale images of digits and labels being the value of the digits. In this experiment, we learn to predict the value of the digit between K " 10 classes corresponding to digits between 0 and 9. The dataset contains 60, 000 images for training and 10, 000 images for testing, all equally distributed within the classes. This dataset has been praised for its accessibility, but was recently criticized for being too easy, which led to the introduction of the dataset Fashion-MNIST, see [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF]. It has the same structure as MNIST, with regard to train and test splits, number of classes and and image size. It consists in classifying types of clothing apparel, e.g. dress, coat and sandals, and is harder to classify than MNIST.

Our experiments aim to show that the OVO approach for top-k classification, cf Eq. ( 8.29), can surpass rankings relying on the scores output by multiclass classification algorithms. For that matter, we evaluated the performances of both approaches using a logistic regression to solve binary classification in the OVO case and multiclass classification in the other. For that matter, we relied on the implementations provided by the python package scikit-learn, specifically the LogisticRegressionCV class. The dimensionality of the data was reduced using standard PCA with enough components to retain 95% of the variance for both datasets, which makes for 153 components for MNIST and 187 components for Fashion-MNIST.

Results are summarized in Table 8.1. They show that the OVO approach performs better than the logistic regression for the top-1 accuracy, i.e. classification accuracy, as well as for the top-5 accuracy. While the OVO approach requires us to train KpK ´1q{2 " 45 models, those are trained with less data and output values. Both approaches end up requiring a similar amount of time to be trained.

Conclusion

In this chapter, a statistical problem halfway between multiclass classification and posterior probability estimation, referred to as label ranking here, was considered. The goal was to design a method to rank, for any test observation X, all the labels y that can be possibly assigned to it by 0.0 0.5 1.0 x decreasing order of magnitude of the (unknown) posterior probability PtY " y | Xu. Formulated as a specific ranking median regression problem with incomplete observations, this problem was shown to have a solution that takes the form of a Copeland score, involving pairwise comparisons only. Based on this crucial observation, it was proved that the OVO procedure for multiclass classification permits to build, from training classification/labeled data, the optimal ranking with high probability, under appropriate hypotheses. This was also empirically supported by numerical experiments. Remarkably, the analysis carried out here incidentally provided a rate bound for the OVO classifier.

The analysis presented here advocates for a different method than usual class probabilities to predict an ordered list of the most likely labels for some observation. As such, it fits into our larger effort directed at finding more suitable loss functions for specific problems in biometrics, which increases the reliability of biometric systems. A flaw of said systems is their impaired generalization when deployed on different statistical populations than that used for training. In that regard, some authors believe that automatic facial recognition might suffer from an other-race effect, which refers to the better capacity of humans to distinguish faces of their own ethnicity, rather than those of others [START_REF] Furl | Face recognition algorithms and the other-race effect: Computational mechanisms for a developmental contact hypothesis[END_REF]. That effect is usually explained by a more important exposure to faces of our own ethnicity. It suggests that increasing the representation of some populations during training process might help adapting the biometric system to specific Chapter 9

Selection Bias Correction

Summary: We consider statistical learning problems, when the distribution P 1 of the training observations Z 1 1 , ..., Z 1 n differs from the test distribution P but is still defined on the same measurable space as P and dominates it. This problem often arises in biometrics, as companies often have large databases, but must tailor their systems to specific populations. Our mathematization of 1:1 identification presented in Part II does not cover that topic. In the unrealistic case where the likelihood ratio Φpzq " dP {dP 1 pzq is known, one may straightforwardly extend Empirical Risk Minimization (ERM) to this transfer learning setup using Importance Sampling (IS), i.e. by minimizing a weighted version of the risk functional computed on the 'biased' training data Z 1 i with weights ΦpZ 1 i q. The importance function Φpzq is generally unknown in practice, but frequently takese.g. when learning: with class imbalance, in a stratified population or with only positive and unlabeled data -a simple form and is directly estimated from the Z 1 i 's and some auxiliary information on P . Using the same tools as for finite-sample generalization bounds for binary classification (Chapter 2), we then prove that the usual generalization capacity of ERM is preserved when plugging the resulting estimates of the ΦpZ 1 i q's into the weighted empirical risk. We provide experiments, that show on ImageNet -an image dataset based on the hierarchical lexical database WordNet -that correcting bias on high-level categories leads to significant performance improvements for the classification task.

Introduction

Prediction problems are of major importance in statistical learning. The main paradigm of predictive learning is Empirical Risk Minimization (ERM in abbreviated form), see e.g. [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF]. In the standard setup, Z is a random variable (r.v. in short) that takes its values in a feature space Z with distribution P , Θ is a parameter space and : Θ ˆZ Ñ R `is a (measurable) loss function. The risk is then defined by: @θ P Θ, R P pθq " E P r pθ, Zqs , (9.1) and more generally for any measure Q on Z: R Q pθq " ş Z pθ, zqdQpzq. In most practical situations, the distribution P involved in the definition of the risk is unknown and learning is based on the sole observation of an independent and identically distributed (i.i.d.) sample Z 1 , . . . , Z n drawn from P and the risk (9.1) must be replaced by an empirical counterpart (or a possibly smoothed/penalized version of it), typically:

p R P pθq " 1 n n ÿ i"1 pθ, Z i q " R p Pn pθq, (9.2)
where p P n " p1{nq ř n i"1 δ Zi is the empirical measure of P and δ z denotes the Dirac measure at any point z. With the design of successful algorithms such as neural networks, support vector machines or boosting methods to perform ERM, the practice of predictive learning has recently received a significant attention and is now supported by a sound theory based on results in empirical process theory. The performance of minimizers of (9.2) can be indeed studied by means of concentration inequalities, quantifying the fluctuations of the maximal deviations sup θPΘ | p R P pθq ´RP pθq| under various complexity assumptions for the functional class F " t pθ, ¨q : θ P Θu (e.g. VC dimension, metric entropies, Rademacher averages), see [START_REF] Boucheron | Concentration Inequalities -A Nonasymptotic Theory of Independence[END_REF] for instance. Although, in the Big Data era, the availability of massive digitized information to train predictive rules is an undeniable opportunity for the widespread deployment of machine-learning solutions, the poor control of the data acquisition process one is confronted with in many applications puts practitioners at risk of jeopardizing the generalization ability of the rules produced by the algorithms implemented. Bias selection issues in machine-learning are now the subject of much attention in the literature, see [START_REF] Bolukbasi | Man is to computer programmer as woman is to homemaker? debiasing word embeddings[END_REF], [START_REF] Zhao | Men also like shopping: Reducing gender bias amplification using corpus-level constraints[END_REF], [START_REF] Hendricks | Women also snowboard: Overcoming bias in captioning models[END_REF], [START_REF] Liu | Transfer learning by sample selection bias correction and its application in communication specific emitter identification[END_REF] or [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF]. In the context of face analysis, a research area including a broad range of applications such as face detection, face recognition or face attribute detection, machine learning algorithms trained with baised training data, e.g. in terms of gender or ethnicity, raise concerns about fairness in machine learning. Unfair algorithms may induce systemic undesired disadvantages for specific social groups, see [START_REF] Das | Mitigating bias in gender, age and ethnicity classification: A multi-task convolution neural network approach[END_REF] for further details. Several examples of bias in deep learning based face recognition systems are discussed in [START_REF] Nagpal | Deep learning for face recognition: Pride or prejudiced?[END_REF].

Throughout the present chapter, we consider the case where the i.i.d. sample Z 1 1 , . . . , Z 1 n available for training is not drawn from P but from another distribution P 1 , with respect to which P is absolutely continuous, and the goal pursued is to set theoretical grounds for the application of ideas behind Importance Sampling (IS in short) methodology to extend the ERM approach to this learning setup. This setup often arises in biometrics. Indeed, companies often have large databases, but must tailor their systems to specific populations. We highlight that the problem under study is a very particular case of Transfer Learning (see e.g. [START_REF] Pan | A survey on transfer learning[END_REF], [START_REF] Ben-David | A theory of learning from different domains[END_REF], [START_REF] Storkey | When training and test sets are different: characterizing learning transfer[END_REF] and [START_REF] Redko | Advances in Domain Adaptation Theory[END_REF]), a research area currently receiving much attention in the literature and encompassing general situations where the information/knowledge one would like to transfer may take a form in the target space very different from that in the source space (referred to as domain adaptation).

Weighted ERM (WERM). In this chapter, we investigate conditions guaranteeing that values for the parameter θ that nearly minimize (9.1) can be obtained through minimization of a weighted version of the empirical risk based on the Z 1 i 's, namely:

r R w,n pθq " R r Pw,n pθq, (9.3)
where r P w,n " p1{nq ř n i"1 w i δ Z 1 i and w " pw 1 , . . . , w n q P R n `is a certain weight vector. Of course, ideal weights w ˚are given by the likelihood function Φpzq " pdP {dP 1 qpzq: w i " ΦpZ 1 i q for i P t1, . . . , nu. In this case, the quantity (9.3) is obviously an unbiased estimate of the true risk (9.1): (9.4) and generalization bounds for the R P -risk excess of minimizers of r R w ˚,n can be directly established by studying the concentration properties of the empirical process related to the Z 1 i 's and the class of functions tΦp¨q pθ, ¨q : θ P Θu (see section 9.2 below). However, the importance function Φ is unknown in general, just like distribution P . It is the major purpose of this chapter to show that, in far from uncommon situations, the (ideal) weights w i can be estimated from the Z 1 i s combined with auxiliary information on the target population P . As shall be seen below, such favorable cases include in particular classification problems where class probabilities in the test stage differ from those in the training step, risk minimization in stratified populations (see [START_REF] Bekker | Beyond the selected completely at random assumption for learning from positive and unlabeled data[END_REF]), with strata statistically represented in a different manner in the test and training populations or even positive-unlabeled learning (PU-learning, see e.g. du Plessis et al. ( 2014)). In each of these cases, we show that the stochastic process obtained by plugging the weight estimates in the weighted empirical risk functional (9.3) is much more complex than a simple empirical process (i.e. a collection of i.i.d. averages) but can be however studied by means of linearization techniques, in the spirit of the ERM extensions established in Clémençon et al. (2008) or Clémençon and Vayatis (2008). Learning rate bounds for minimizers of the corresponding risk estimate are proved and, beyond these theoretical guarantees, the performance of the weighted ERM approach is supported by convincing numerical results.

E P 1 " R r P w ˚,n pθq ı " R P pθq,
The chapter is structured as follows. In section 9.2, the ideal case where the importance function Φ is known is preliminarily considered and a first basic example where the optimal weights can be easily inferred and plugged into the risk without deteriorating the learning rate is discussed. The main results of the chapter are stated in section 9.3, which shows that the methodology promoted can be applied to two important problems in practice, risk minimization in stratified populations and PU-learning, with generalization guarantees. Illustrative numerical experiments are displayed in section 9.5, while some concluding remarks are collected in section 9.6.

Importance Sampling -Risk Minimization with Biased Data

Here and throughout, the indicator function of any event E is denoted by ItEu, the sup norm of any bounded function h : Z Ñ R by ||h|| 8 . We place ourselves in the framework of statistical learning based on biased training data previously introduced. To begin with, we consider the unrealistic situation where the importance function Φ is known, insofar as we shall subsequently develop techniques aiming at mimicking the minimization of the ideally weighted empirical risk:

r R w ˚,n pθq " 1 n n ÿ i"1 w i pθ, Z 1 i q, (9.5)
namely the (unbiased) Importance Sampling estimator of (9.1) based on the instrumental data Z 1 1 , . . . , Z 1 n . The following result describes the performance of minimizers r θ n of (9.5). Since the goal of this chapter is to promote the main ideas of the approach rather than to state results with the highest level of generality, we assume throughout the chapter for simplicity that and Φ are both bounded functions. For σ 1 , . . . , σ n independent Rademacher random variables (i.e. symmetric t´1, 1u-valued r.v.'s), independent from the Z 1 i 's, we define the Rademacher average associated to any class of function F 0 as:

R 1 n pF 0 q :" E σ « sup f PF0 p1{nq ˇˇˇˇn ÿ i"1 σ i ¨f pZ 1 i q ˇˇˇˇff .
This quantity can be bounded by metric entropy methods under appropriate complexity assumptions on the class F 0 , it is for instance of order 1{ ? n when F 0 is a VC-major class with finite VC dimension, see e.g. [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF]. In particular, we are interested in the family F :" tz P Z Þ Ñ pθ, zq : θ P Θu.

Lemma 9.1. We have @n ě 1, with probability (w.p) at least 1 ´δ:

R P p r θ nq ´min θPΘ R P pθq ď 4||Φ|| 8 E " R 1 n pFq ‰ `2||Φ|| 8 sup pθ,zqPΘˆZ pθ, zq c 2 logp1{δq n .
Proof. Let δ P p0, 1q. Applying the classic maximal deviation bound stated in Theorem 3.2 of [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] to the bounded class K " tz P Z Þ Ñ Φpzqlpθ, zq : θ P Θu, we obtain that, w.p ě 1 ´δ:

sup θPΘ ˇˇr R w ˚,n pθq ´E " r R w ˚,n pθq ıˇˇˇď 2E " R 1 n pKq ‰ `||Φ|| 8 sup pθ,zqPΘˆZ | pθ, zq| c 2 logp1{δq n .
In addition, by virtue of the contraction principle, we have R 1 n pKq ď ||Φ|| 8 R 1 n pFq almost-surely. The desired result can be thus deduced from the bound above combined with the classic bound:

R P p θn q ´min θPΘ R P pθq ď 2 sup θPΘ ˇˇr R w ˚,n pθq ´E " r R w ˚,n pθq ıˇˇˇ.
Of course, when P 1 " P , we have Φ " 1 and the bound stated above simply describes the performance of standard empirical risk minimizers. The proof is based on the standard bound:

R P p r θ nq ´min θPΘ R P pθq ď 2 sup θPΘ ˇˇr R w ˚,n pθq ´E " r R w ˚,n pθq ıˇˇˇ,
combined with basic concentration results for empirical processes. Of course, the importance function Φ is generally unknown and must be estimated in practice. As illustrated by the elementary example below (related to binary classification, in the situation where the probability of occurence of a positive instance significantly differs in the training and test stages), in certain statistical learning problems with biased training distribution, Φ takes a simplistic form and can be easily estimated from the Z 1 i 's combined with auxiliary information on P . Binary classification with varying class probabilities. The flagship problem in supervised learning corresponds to the simplest situation, where Z " pX, Y q, Y being a binary variable valued in t´1, `1u say, and the r.v. X takes its values in a measurable space X and models some information hopefully useful to predict Y . The parameter space Θ is a set G of measurable mappings (i.e. classifiers) g : X Ñ t´1, `1u and the loss function is given by pg, px, yqq " Itgpxq ‰ yu for all g in G and any px, yq P X ˆt´1, `1u. The distribution P of the random pair pX, Y q can be either described by X's marginal distribution F and the posterior probability ηpxq " PtY " `1 | X " xu or else by the triplet pp, F `, F ´q where p " PtY " `1u and F σ pdxq is X's conditional distribution given Y " σ1 with σ P t´, `u. It is very common that the fraction of positive instances in the training dataset is significantly lower than the rate p expected in the test stage, supposed to be known here. We thus consider the case where the distribution P 1 of the training data pX 1 1 , Y 1 1 q, . . . , pX 1 n , Y 1 n q is described by the triplet pp 1 , F `, F ´q with p 1 ă p. The likelihood function takes the simple following form: Φpx, yq " Ity " `1u p p 1 `Ity " ´1u

1 ´p 1 ´p1 :" φpyq,
which reveals that it depends on the label y solely, and the ideally weighted empirical risk process is:

r R w ˚,n pgq " p p 1 1 n ÿ i:Y 1 i "1 ItgpX 1 i q " ´1u `1 ´p 1 ´p1 1 n ÿ i:Y 1 i "´1
ItgpX 1 i q " `1u.

(9.6)

In general the theoretical rate p 1 is unknown and one replaces (9.6) with:

r R p w ˚,n pgq " p n 1 `ÿ i:Y 1 i "1 ItgpX 1 i q " ´1u `1 ´p n 1 ´ÿ i:Y 1 i "´1
ItgpX 1 i q " `1u, (9.7)

where n 1 `" ř n i"1 ItY 1 i " `1u " n ´n1

´, p w i " p φpY 1 i q and p φpyq " Ity " `1unp{n 1 ``Ity " ´1unp1 ´pq{n 1

´. The stochastic process above is not a standard empirical process but a collection of sums of two ratios of basic averages. However, the following result provides a uniform control of the deviations between the ideally weighted empirical risk and that obtained by plugging the empirical weights into the latter.

Lemma 9.2. Let ε P p0, 1{2q. Suppose that p 1 P pε, 1 ´εq. For any δ P p0, 1q and n P N ˚, we have w.p ě 1 ´δ:

sup gPG ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇď 2 ε 2 c logp2{δq 2n ,
as soon as n ě 2 logp2{δq{ε 2 .

Proof. Apply twice the Taylor expansion:

1 x " 1 a ´x ´a a 2 `px ´aq 2 xa 2 ,
so as to get:

1 n 1 `{n " 1 p 1 ´n1 `{n ´p1 p 12 `pn 1 `{n ´p1 q 2 p 12 n 1 `{n , 1 n 1 ´{n " 1 1 ´p1
´n1 ´{n ´1 `p1 p1 ´p1 q 2 `pn 1 ´{n ´1 `p1 q 2 p1 ´p1 q 2 n 1 ´{n .

This yields the decomposition:

r R p w ˚,n pgq ´r R w ˚,n pgq " ´p p 12 ˆn1 ǹ ´p1 ˙1 n n ÿ i"1 ItgpX 1 i q " ´1, Y 1 i " `1u ´1 ´p p1 ´p1 q 2 ˆn1 ń ´1 `p1 ˙1 n n ÿ i"1 ItgpX 1 i q " `1, Y 1 i " ´1u `ppn 1 `{n ´p1 q 2 p 12 n 1 `{n 1 n n ÿ i"1 ItgpX 1 i q " ´1, Y 1 i " `1u `p1 ´pqpn 1 ´{n ´1 `p1 q 2 p1 ´p1 q 2 n 1 ´{n 1 n n ÿ i"1 ItgpX 1 i q " `1, Y 1 i " ´1u.
We deduce that:

ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇď |n 1 `{n ´p1 | ε 2 ˆ1 `|n 1 `{n ´p1 | ˆp n 1 `{n `1 ´p 1 ´n1 `{n

˙˙.

By virtue of Hoeffding inequality, we obtain that, for any δ P p0, 1q, we have w.p ě 1 ´δ:

ˇˇn 1 `{n ´p1 ˇˇď c logp2{δq 2n ,
so that, in particular, mintn 1 `{n, 1 ´n1 `{nu ě ε ´alogp2{δq{p2nq. This yields the desired result.

Consequently, minimizing (9.7) nearly boils down to minimizing (9.6). Combining Lemmas 9.2 and 9.1, we immediately get the generalization bound stated in the result below.

Corollary 9.3. Suppose that the hypotheses of Lemma 9.2 are fulfilled. Let n P N ˚and r g n be any minimizer of r R p w ˚,n over the class G. We have w.p. ě 1 ´δ:

R P pr g n q ´inf gPG R P pgq ď 2 maxpp, 1 ´pq ε ˜2ErR 1 n pGqs `c 2 logp2{δq n ¸`4 ε 2 c logp4{δq 2n ,
as soon as n ě 2 logp4{δq{ε 2 ; where R 1 n pGq " p1{nqE σ rsup gPG |

ř n i"1 σ i ItgpX 1 i q ‰ Y 1 i u|s.
Proof. Observe first that ||Φ|| 8 ď maxpp, 1 ´pq{ε and

R P pr g n q ´inf gPG R P pgq ď 2 sup gPG ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇ`2 sup gPG ˇˇr R w ˚,n pgq ´RP pgq ˇˇ.
The result then directly follows from the application of Lemmas 9.1-9.2 combined with the union bound.

Hence, some side information (i.e. knowledge of parameter p) has permitted to weight the training data in order to build an empirical risk functional that approximates the target risk and to show that minimization of this risk estimate yields prediction rules with optimal (in the minimax sense) learning rates. The purpose of the subsequent analysis is to show that this remains true for more general problems. Observe in addition that the bound in Corollary 9.3 deteriorates as ε decays to zero: the method used here is not intended to solve the few shot learning problem, where almost no training data with positive labels is available (i.e. p 1 « 0). As shall be seen in subsection 9.3.2, alternative estimators of the importance function must be considered in this situation.

Remark 9.4. Although the quantity (9.7) can be viewed as a cost-sensitive version of the empirical classification risk based on the pX 1 i , Y 1 i q's (see e.g. [START_REF] Bach | Considering cost asymmetry in learning classifiers[END_REF]), we point out that the goal pursued here is not to achieve an appropriate trade-off between type I and type II errors in the P 1 classification problem as in biometric applications for instance (i.e. optimization of the pF `, F ´q-ROC curve at a specific point) but to transfer knowledge gained in analyzing the biased data drawn from P 1 to the classification problem related to distribution P . Related work. We point out that the natural idea of using weights in ERM problems that mimic those induced by the importance function has already been used in Sugiyama et al. (2007) for covariate shift adaptation problems (i.e. supervised situations, where the conditional distribution of the output given the input information is the same in the training and test domains), when, in contrast to the framework considered here, a test sample is additionally available (a method for estimating directly the importance function based on Kullback-Leibler divergence minimization is proposed, avoiding estimation of the test density). Importance sampling estimators have been also considered in [START_REF] Garcke | Importance weighted inductive transfer learning for regression[END_REF] in the setup of inductive transfer learning (the tasks between source and target are different, regardless of the similarities between source and target domains), where the authors have proposed two methods to approximate the importance function, among which one is again based on minimizing the Kullback-Leibler divergence between the two distributions. In [START_REF] Cortes | Sample selection bias correction theory[END_REF], the sample selection bias is assumed to be independent from the label, which is not true under our stratum-shift assumption or for the PU learning problem (see section 8.3). Lemma 9.1 assumes that the exact importance function is known, as does [START_REF] Cortes | Learning bounds for importance weighting[END_REF]. The next section introduces new results for more realistic settings where it has to be learned from the data.

Weighted Empirical Risk Minimization -Generalization Guarantees

Through two important and generic examples, relevant for many applications, we show that the approach sketched above can be applied to general situations, where appropriate auxiliary information on the target distribution is available, with generalization guarantees. We work in this section under the condition that the loss function is bounded assuming sup pθ,zqPΘˆZ pθ, zq ď L.

Statistical Learning from Biased Data in a Stratified Population

A natural extension of the simplistic problem considered in section 9.2 is multiclass classification in a stratified population. The random labels Y and Y 1 are supposed to take their values in t1, . . . , Ju say, with J ě 1, and each labeled observation pX, Y q belongs to a certain random stratum S in t1, . . . , Ku with K ě 1. Again, the distribution P of a random element Z " pX, Y, Sq may be described by the parameters tpp j,k , F j,k q : 1 ď j ď J, 1 ď k ď Ku where F j,k is the conditional distribution of X given pY, Sq " pj, kq and p j,k " P pX,Y,Sq"P tY " j, S " ku.

Then, we have:

dP px, y, sq " J ÿ j"1 K ÿ k"1
Ity " j, s " kup j,k dF j,k pxq, and considering a distribution P 1 with F j,k " F 1 j,k but possibly different class-stratum probabilities p 1 j,k , the likelihood function becomes:

dP dP 1 px, y, sq " J ÿ j"1 K ÿ k"1 p j,k p 1 j,k
Ity " j, s " ku :" φpy, sq.

A more general framework can actually encompass this specific setup by defining 'meta-strata' in t1, . . . , Ju ˆt1, . . . , Ku. Strata may often correspond to categorical input features in practice.

The formalism introduced below is more general and includes the example considered in the preceding section, where strata are defined by labels.

Learning from biased stratified data. Consider a general mixture model, where distributions P and P 1 are stratified over K ě 1 strata. Namely, Z " pX, Sq and Z 1 " pX 1 , S 1 q with auxiliary random variables S and S 1 (the strata) valued in t1, . . . , Ku. We place ourselves in a stratumshift context, assuming that the conditional distribution of X given S " k is the same as that of X 1 given S 1 " k, denoted by F k pdxq, for any k P t1, . . . , Ku. However, stratum probabilities p k " PpS " kq and p 1 k " PpS 1 " kq may possibly be different. In this setup, the likelihood function depends only on the strata and can be expressed in a very simple form, as follows:

dP dP 1 px, sq " K ÿ k"1 Its " ku p k p 1 k :" φpsq.
In this case, the ideally weighted empirical risk writes:

r R w ˚,n pθq " 1 n n ÿ i"1 pθ, Z 1 i q K ÿ k"1 ItS 1 i " ku p k p 1 k .
If the strata probabilities p k 's for the test distribution are known, an empirical counterpart of the ideal empirical risk above is obtained by simply plugging estimates of the p 1 k 's computed from the training data:

r R p w ˚,n pθq " n ÿ i"1 pθ, Z 1 i q K ÿ k"1 ItS 1 i " ku p k n 1 k , (9.8)
with n 1 k "

ř n i"1 ItS 1 i " ku, p w i " p φpS 1 i q and p φpsq " ř K k"1 Its " kunp k {n 1 k .
A bound for the excess of risk associated to Eq. (9.8) is given in Theorem 9.6, that can be viewed as a generalization of Corollary 9.3. It relies on Lemma 9.1 and Lemma 9.5 below.

Lemma 9.5. Let ε P p0, 1{2q. Suppose that p 1 k P pε, 1 ´εq for k P t1, . . . , Ku. For any δ P p0, 1q and n P N ˚, we have w.p. ě 1 ´δ:

sup θPΘ ˇˇr R p w ˚,n pθq ´r R w ˚,n pθq ˇˇď 2L ε 2 c logp2K{δq 2n ,
as soon as n ě 2 logp2K{δq{ε 2 , where L " sup pθ,zqPΘˆZ pθ, zq.

Proof. Apply the Taylor expansion

1 x " 1 a ´x ´a a 2 `px ´aq 2 xa 2 ,
so as to get for all k P t1, . . . , Ku

1 n 1 k {n " 1 p 1 k ´n1 k {n ´p1 k p 12 k `pn 1 k {n ´p1 k q 2 p 12 k n 1 k {n . This yields the decomposition r R p w ˚,n pθq ´r R w ˚,n pθq " 1 n n ÿ i"1 pθ, Z 1 i q K ÿ k"1 ItS 1 i " ku ˆ´p k p 12 k ˆn1 k n ´p1 k ˙`p k pn 1 k {n ´p1 k q 2 p 12 k n 1 k {n ˙.
We deduce that

ˇˇr R p w ˚,n pθq ´r R w ˚,n pθq ˇˇď L ε 2 K ÿ k"1 |n 1 k {n ´p1 k |p k ˆ1 `|n 1 k {n ´p1 k | n 1 k {n ˙.
By virtue of Hoeffding inequality, we obtain that, for any k P t1, . . . , Ku and δ P p0, 1q, we have w.p ě 1 ´δ:

ˇˇn 1 k {n ´p1 k ˇˇď c logp2{δq 2n ,
so that, by a union bound, max k tn 1 k {nu ě ε ´alogp2K{δq{p2nq. This yields the desired result.

Theorem 9.6. Let ε P p0, 1{2q and assume that p 1 k P pε, 1 ´εq for k " 1, . . . , K. Let r θ n be any minimizer of r R p w ˚,n as defined in (9.8) over class Θ. For any n P N ˚and δ P p0, 1q, we have w.p ě 1 ´δ:

R P p r θ nq ´inf θPΘ R P pθq ď 2 max k p k ε ˜2ErR 1 n pFqs `Lc 2 logp2{δq n ¸`4L ε 2 c logp4K{δq 2n ,
as soon as n ě 2 logp4K{δq{ 2 ; where R 1 n pFq " p1{nqE σ rsup θPΘ | ř n i"1 σ i pθ, Z 1 i q|s, and the loss is bounded by L " sup pθ,zqPΘˆZ pθ, zq.

Proof. Observe first that ||Φ|| 8 ď max k p k {ε and

R P p r θ nq ´inf θPΘ R P pθq ď 2 sup θPΘ ˇˇr R p w ˚,n pθq ´r R w ˚,n pθq ˇˇ`2 sup θPΘ ˇˇr R w ˚,n pθq ´RP pθq ˇˇ.
The result then directly follows from the application of Lemmas 9.1 and 9.5 combined with the union bound.

Just like in Corollary 9.3, the bound in Theorem 9.6 explodes when ε vanishes, which corresponds to the situation where a stratum k P t1, . . . , Ku is very poorly represented in the training data, i.e. when p 1 k ăă p k . Again, as highlighted by the experiments carried out, reweighting the losses in a frequentist (ERM) approach guarantees good generalization properties in a specific setup only, i.e. when the training information, though biased, is sufficiently informative.

Positive-Unlabeled Learning

Relaxing the stratum-shift assumption made in the previous subsection, the importance function becomes more complex and writes:

Φpx, sq " dP dP 1 px, sq " K ÿ k"1 Its " ku p k p 1 k dF k dF 1 k pxq,
where F k and F 1 k are respectively the conditional distributions of X given S " k and of X 1 given S 1 " k. The Positive-Unlabeled (PU) learning problem, which has recently been the subject of much attention (see e.g. du Plessis et al. (2014[START_REF] Du Plessis | Convex formulation for learning from positive and unlabeled data[END_REF], [START_REF] Kiryo | Positive-unlabeled learning with nonnegative risk estimator[END_REF]), provides a typical example of this situation. Re-using the notations introduced in section 9.2, in the PU problem, the testing and training distributions P and P 1 are respectively described by the triplets pp, F `, F ´q and pq, F `, F q, where F " pF ``p1 ´pqF ´is the marginal distribution of X. Hence, the objective pursued is to solve a binary classification task, based on the sole observation of a training sample pooling data with positive labels and unlabeled data, q denoting the theoretical fraction of positive data among the dataset. As noticed in du Plessis et al. (2014) (see also du [START_REF] Du Plessis | Convex formulation for learning from positive and unlabeled data[END_REF], [START_REF] Kiryo | Positive-unlabeled learning with nonnegative risk estimator[END_REF]), the likelihood/importance function can be expressed in a simple manner. Precisely, @px, yq P X ˆt´1, `1u, Φpx, yq " p q Ity " `1u `1 1 ´q Ity " ´1u ´p 1 ´q dF dF pxqIty " ´1u. (9.9)

Based on an i.i.d. sample pX 1 1 , Y 1 1 q, . . . , pX 1 n , Y 1 n q drawn from P 1 combined with the knowledge of p (which can also be estimated from PU data, see e.g. du Plessis and Sugiyama (2014)) and using that F ´" p1{p1 ´pqqpF ´pF `q, one may obtain estimators of q, F `and F by computing n 1 `{n " p1{nq

ř n i"1 ItY 1 i " `1u, p F `" p1{n 1 `q ř n i"1 ItY 1 i " `1uδ X 1 i and p F " p1{n 1 ´q ř n i"1 ItY 1 i " ´1uδ X 1
i . However, plugging these quantities into (9.9) do not permit to get a statistical version of the importance function, insofar as the probability measures p F `and p F are mutually singular with probability one, as soon as F `is continuous. Of course, as proposed in du Plessis et al. ( 2014), one may use statistical methods (e.g. kernel smoothing) to build distribution estimators, that ensures absolute continuity but are subject to the curse of dimensionality. However, WERM can still be applied in this case, by observing that: @g P G, R P pgq " ´p `EP 1 " 2p q ItgpX 1 q " ´1, Y 1 " `1u `1 1 ´q ItgpX 1 q " `1, Y 1 " ´1u  , (9.10) which leads to the weighted empirical risk:

2p n 1 `ÿ i:Y 1 i "`1 ItgpX 1 i q " ´1u `1 n 1 ´ÿ i:Y 1 i "´1
ItgpX 1 i q " `1u.

(9.11) Minimization of (9.11) yields rules r g n whose generalization ability regarding the binary problem related to pp, F `, F ´q can be guaranteed, as shown by the following result Theorem 9.8 implied by combining Lemmas 9.1 and 9.7, the form of the weighted empirical risk in this case being quite similar to (9.7).

Lemma 9.7. Let ε P p0, 1{2q. Suppose that q P pε, 1 ´εq. For any δ P p0, 1q and n ě 2 logp2{δq{ε 2 , we have w.p ě 1 ´δ:

sup gPG ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇď 2p2p `1q ε 2 c logp2{δq 2n .
Proof. Apply twice the Taylor expansion

1 x " 1 a ´x ´a a 2 `px ´aq 2 xa 2 , so as to get 1 n 1 `{n " 1 q ´n1 `{n ´q q 2 `pn 1 `{n ´qq 2 q 2 n 1 `{n , 1 n 1 ´{n " 1 1 ´q ´n1 ´{n ´1 `q p1 ´qq 2 `pn 1 ´{n ´1 `qq 2 p1 ´qq 2 n 1 ´{n .
This yields the decomposition

r R p w ˚,n pgq ´r R w ˚,n pgq " ´2p q 2 ˆn1 ǹ ´q˙1 n n ÿ i"1 ItgpX 1 i q " ´1, Y 1 i " `1u ´1 p1 ´qq 2 ˆn1 ń ´1 `q˙1 n n ÿ i"1 ItgpX 1 i q " `1, Y 1 i " ´1u `2ppn 1 `{n ´qq 2 q 2 n 1 `{n 1 n n ÿ i"1 ItgpX 1 i q " ´1, Y 1 i " `1u `pn 1 ´{n ´1 `qq 2 p1 ´qq 2 n 1 ´{n 1 n n ÿ i"1 ItgpX 1 i q " `1, Y 1 i " ´1u. We deduce that ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇď |n 1 `{n ´q| ε 2 ˆ2p `1 `|n 1 `{n ´q| ˆ2p n 1 `{n `1 1 ´n1 `{n ˙˙.
By virtue of Hoeffding inequality, we obtain that, for any δ P p0, 1q, we have w.p. ě 1 ´δ:

ˇˇn 1 `{n ´qˇˇď c logp2{δq 2n ,
so that, in particular, mintn 1 `{n, 1 ´n1 `{nu ě ε ´alogp2{δq{p2nq. This yields the desired result.

Theorem 9.8. Let ε P p0, 1{2q. Suppose that q P pε, 1 ´εq. Let r g n be any minimizer of the weighted empirical risk (9.11) over class G. For any δ P p0, 1q and n ě 2 logp4{δq{ε 2 , we have w.p ě 1 ´δ:

R P pr g n q ´inf gPG R P pgq ď 2 maxp2p, 1q ε ˜2ErR 1 n pGqs `c 2 logp2{δq n ¸`4p2p `1q ε 2 c logp4{δq 2n , where R 1 n pGq " p1{nqE σ rsup gPG | ř n i"1 σ i ItgpX 1 i q ‰ Y 1 i u|s.
Proof. Observe first that ||Φ|| 8 ď maxp2p, 1q{ε and

R P pr g n q ´inf gPG R P pgq ď 2 sup gPG ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇ`2 sup gPG ˇˇr R w ˚,n pgq ´RP pgq ˇˇ,
with weighted empirical risk r R w ˚,n pgq defined in (9.11). The result then directly follows from the application of Lemmas 9.1 and 9.7 combined with the union bound.

In the next section, we show that better generalization bounds can be obtained when one has an estimator p η of the posterior probability η, which paves the way for iterative procedures for PU learning.

Alternative Approach for Positive-Unlabeled Learning

In the usual PU learning approach, observe that: Φpx, yq " p q Ity " `1u `1 ´ηpxq 1 ´q Ity " ´1u, (9.12) in the case when an estimate p ηpxq of ηpxq is available, one can perform WERM using the empirical weight function:

p Φpx, yq " np n 1 `Ity " `1u `1 ´p ηpxq 1 ´n1 `{n
Ity " ´1u. (9.13)

A bound that describes how this approach generalizes, depending on the accuracy of estimate p η, can be easily established, which is summarized in Theorem 9.10, again a direct consequence of Lemmas 9.1 and 9.9. Lemma 9.9. Let weights p w ˚be defined as in Theorem 9.10. Let ε P p0, 1{2q. Suppose that q P pε, 1 ´εq. For any δ P p0, 1q and n ě 2 logp2{δq{ε 2 , we have w.p ě 1 ´δ:

sup gPG ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇď 2 ε 2 c logp2{δq 2n `2 sup xPX |p ηpxq ´ηpxq|.
Proof. Apply twice the Taylor expansion:

1 x " 1 a ´x ´a a 2 `px ´aq 2 xa 2 ,
so as to get:

1 n 1 `{n " 1 q ´n1 `{n ´q q 2 `pn 1 `{n ´qq 2 q 2 n 1 `{n , 1 n 1 ´{n " 1 1 ´q ´n1 ´{n ´1 `q p1 ´qq 2 `pn 1 ´{n ´1 `qq 2 p1 ´qq 2 n 1 ´{n .
This yields the decomposition:

r R p w ˚,n pgq ´r R w ˚,n pgq " ´p q 2 ˆn1 ǹ ´q˙1 n n ÿ i"1 ItgpX 1 i q " ´1, Y 1 i " `1u ´1 p1 ´qq 2 ˆn1 ń ´1 `q˙1 n n ÿ i"1 p1 ´p ηpX 1 i qqItgpX 1 i q " `1, Y 1 i " ´1u `ppn 1 `{n ´qq 2 q 2 n 1 `{n 1 n n ÿ i"1 ItgpX 1 i q " ´1, Y 1 i " `1u `pn 1 ´{n ´1 `qq 2 p1 ´qq 2 n 1 ´{n 1 n n ÿ i"1 p1 ´p ηpX 1 i qqItgpX 1 i q " `1, Y 1 i " ´1u `1 p1 ´qqn n ÿ i"1 pηpX 1 i q ´p ηpX 1 i qqItgpX 1 i q " `1, Y 1 i " ´1u.
We deduce that:

ˇˇr R p w ˚,n pgq ´r R w ˚,n pgq ˇˇď |n 1 `{n ´q| ε 2 ˆ1 `|n 1 `{n ´q| ˆ1 n 1 `{n `1 1 ´n1 `{n ˙ṅ 1 ´{n 1 ´q sup xPX |p ηpxq ´ηpxq|.
By virtue of Hoeffding inequality, we obtain that, for any δ P p0, 1q, we have w.p. ě 1 ´δ:

ˇˇn 1 `{n ´qˇˇď c logp2{δq 2n ,
so that, in particular, mintn 1 `{n, 1 ´n1 `{nu ě ε ´alogp2{δq{p2nq. Moreover, still under this event, n 1 ´{n ď 2p1 ´qq if n ě logp2{δq{p2ε 2 q. This yields the desired result.

Theorem 9.10. Let p w i " p ΦpX 1 i , Y 1 i q for all i P t1, . . . , nu with p Φ defined in (9.13). Suppose that the hypotheses of Lemma 9.9 are fulfilled. Let r g n be any minimizer of r R p w ˚,n over class G. For any δ P p0, 1q and n ě 2 logp4{δq{ε 2 , we have w.p ě 1 ´δ:

R P pr g n q ´inf gPG R P pgq ď 2 maxpp, 1 ´pq ε ˜ErR 1 n pGqs `c 2 logp2{δq n 4 ε 2 c logp4{δq 2n `4 sup xPX |p ηpxq ´ηpxq|.
Proof. Observe first that ||Φ|| 8 ď max k p k {ε and

R P p r θ nq ´inf θPΘ R P pθq ď 2 sup θPΘ ˇˇr R p w ˚,n pθq ´r R w ˚,n pθq ˇˇ`2 sup θPΘ ˇˇr R w ˚,n pθq ´RP pθq ˇˇ.
The result then directly follows from the application of Lemmas 9.1-9.9 combined with the union bound.

Learning from Censored Data

Another important example of sample bias is the censorship setting where the learner has only access to (right) censored targets minpY 1 , C 1 q instead of Y 1 . Intuitively, this situation occurs when Y 1 is a duration/date, e.g. the date of death of a patient modeled by covariates X 1 , and the study happens at a (random) date C 1 . Hence if C 1 ď Y 1 , then we know that the patient is still alive at time C 1 but the target time Y 1 remains unknown. This problem has been extensively studied (see e.g. [START_REF] Fleming | Counting processes and survival analysis[END_REF], [START_REF] Andersen | Statistical models based on counting processes[END_REF] and the references therein for the asymptotic theory and [START_REF] Ausset | Empirical risk minimization under random censorship: Theory and practice[END_REF] for finite-sample guarantees): we show here that it is an instance of WERM. Formally, we respectively denote by P and P 1 the testing and training distributions of the r.v.'s pX, minpY, Cq, ItY ď Cuq and pX 1 , minpY 1 , C 1 q, ItY 1 ď C 1 uq both valued in R d ˆR`ˆt 0, 1u (with Y, Y 1 , C, C 1 all nonnegative r.v.'s) and such that the pairs pX, Y q and pX 1 , Y 1 q share the same distribution Q. Moreover, C ą Y with probability 1 (i.e. the testing data are never censored) and Y 1 and C 1 are assumed to be conditionally independent given X 1 . Hence, for all px, y, δq P R d ˆR`ˆt 0, 1u:

dP px, y, δq " δdQpx, yq, and

δdP 1 px, y, δq " δPpC 1 ě yqdPpX 1 " x, Y 1 " y|C 1 ě yq " δS C 1 py|xqdQpx, yq,
where S C 1 py|xq " PpC 1 ě y|X 1 " xq denotes the conditional survival function of C 1 given X 1 . Then, the importance function is: @px, y, δq P R d ˆR`ˆt 0, 1u, Φpx, y, δq " dP dP 1 px, y, δq " δ S C 1 py|xq .

In survival analysis, the ratio δ{S C 1 py|xq is called IPCW (inverse of the probability of censoring weight) and S C 1 py|xq can be estimated by using the Kaplan-Meier approach, see [START_REF] Kaplan | Nonparametric estimation from incomplete observations[END_REF].

Extension to Iterative WERM

As highlighted in Remark 2, the importance function can be expressed as a function of the ideal decision function in certain situations: Eq. ( 9.12) involves the regression function ηpxq, that defines the optimal (Bayes) classifier g ˚pxq " 2Itηpxq ě 1{2u ´1. This simple observation paves the way for a possible incremental application of the WERM approach: in the case where the solution of the WERM problem considered outputs an estimate of the optimal decision function, it can be next re-used for defining and solving a novel WERM problem. Whereas binary classification based on PU data only aims at recovering a single level set of the posterior probability ηpxq, it is not the case of a more ambitious statistical learning problem, referred to as bipartite ranking, for which such an incremental version of WERM can be described.

Bipartite ranking based on PU data. In bipartite ranking, the statistical challenge consists of ranking all the instances x P X through a scoring function s : X Ñ R in the same order as the likelihood ratio ΨpXq " pdF `{dF ´qpX q, or, equivalently, as the regression function ηpxq " PtY " `1 | X " xu, x P X : the higher the score spXq, the more likely one should observe Y " `1. Let S " ts : X Ñ R measurableu denotes the set of all scoring functions on the input space X . A classical way of measuring "how much stochastically larger" a c.d.f. g on R is than another one, h say, consists in drawing the "probability-probability plot":

t P R Þ Ñ p1 ´hptq, 1 ´gptqq,
with the convention that possible jumps are connected by line segments (in order to guarantee the continuity of the curve). Equipped with this convention, this boils down to plot the graph of the mapping ROC h,g : α P p0, 1q Þ Ñ ROC h,g " 1 ´g ˝h´1 p1 ´αq, where Γ ´1puq " inftt P R : Γptq ě uu denotes the pseudo-inverse of any c.d.f. Γptq on R. The closer to the left upper corner of the unit square r0, 1s 2 , the larger the c.d.f. g is compared to h in a stochastic sense. This approach is known as ROC analysis. The gold standard for evaluating the ranking performance of a scoring function s is thus the ROC curve:

ROC s :" ROC Hs,Gs ,

where G s and H s denote the conditional c.d.f. of spXq given Y " `1 and given Y " ´1 respectively, i.e. the images of the class probability distributions F `and F ´by the mapping spxq. Indeed, it follows from a standard Neyman-Pearson argument that the ROC curve ROC of strictly increasing transforms of ηpxq is optimal with respect to this criterion in the sense that: @α P p0, 1q, ROC s pαq ď ROC ˚pαq, for any scoring function s. We set S ˚" tT ˝η : T : p0, 1q :Ñ Ru. A summary quantity of this functional criterion that is widely used in practice is the Area Under the ROC Curve (AUC in short), given by: AUCpsq "

ż 1 α"0 ROC s pαq dα,
for s P S. Beyond its scalar nature, an attractive property of this criterion lies in the fact that it can be interpreted in a probabilistic manner, insofar as we have the relation: @s P S, AUCpsq " P spXq ă spX 1 q | pY, Y 1 q " p´1, `1q ( `1 2 P spXq " spX 1 q | pY, Y 1 q " p´1, `1q

( .
Denoting by pX i , Y i q, i P t1, 2u, independent copies of the pair pX, Y q and placing ourselves in the situation where spXq's probability distribution is continuous, as observed in Clémençon et al. (2008), we have AUCpsq " 1 ´LP psq{p2pp1 ´pqq, where L P psq :" P tpspX 1 q ´spX 2 qqpY 1 ´Y2 qq ă 0u , is the ranking risk, the theoretical rate of discording pairs namely, that can be viewed as a pairwise classification risk. Hence, bipartite ranking can be formulated as the problem of learning a scoring function s that minimizes the ranking risk

L P psq " E P 1 bP 1 " dP dP 1 pX 1 1 , Y 1 1 q dP dP 1 pX 1 2 , Y 1 2 q ˆI pspX 1 1 q ´spX 1 2 qqpY 1 1 ´Y 1 2 qq ă 0 (  .
Now, using Eq. ( 9.12) and the fact that η " pΨ{p1 ´p `pΨq, we have:

dP dP 1 px, yq " Φpx, yq " p q
Ity " `1u `1 ´ηpxq 1 ´q Ity " ´1u, " p q Ity " `1u `1 ´p p1 ´qqp1 ´p `pΨpxqq Ity " ´1u.

Therefore, it has been shown in Clémençon and Vayatis (2009) From this formula, it is the easy to see how an incremental use of the WERM could be implemented.

• Start from an initial guess s for the optimal scoring functions (e.g. solve the empirical ranking risk minimization problem ignoring the bias issue)

• Estimate Φ s from the pX 1 i , Y 1 i q's and the knowledge of p, observing that one is not confronted with the curse of dimensionality in this case

• Solve the Weighted Empirical Ranking Risk Minimization problem using the weight function p Φ s px 1 , y 1 q p Φ s px 2 , y 2 q, which produces a new scoring function s and iterate.

Investigating the performance of such an incremental procedure will be the subject of future research.

Numerical Experiments

This section illustrates the impact of reweighting by the likelihood ratio on classification performances, as a special case of the general strategy presented in Section 9.2. A first simple illustration on known probability distributions highlights the impact of the shapes of the distributions on the importance of reweighting. This example illustrates in the infinite-sample case that separable or almost separable data do not require reweighting, in contrast to noisy data. Since the distribution shapes are unknown for real data, we infer that reweighting will have variable effectiveness, depending on the dataset. We detail other experiments that illustrate the effectiveness of the reweighting approach on real data. Precisely, we first use reweighting on a shift in the class probabilities for the well-known dataset MNIST between training and testing. Secondly, uses the structure of ImageNet to illustrate reweighting with a stratified population and strata distribution bias or strata bias.

Importance of Reweighting for Simple Distributions

Introduce a random pair pX, Y q in r0, 1s ˆt´1, `1u where X | Y " `1 has for p.d.f. f `pxq and X | Y " ´1 has for p.d.f. f ´pxq, with, for some α ą 0 and β ą 0:

f ´pxq :" p1 `βqp1 ´xq β and f `pxq :" p1 `αqx α .

As in Section 9.2, the train and test datasets have different class probabilities p 1 and p for Y " `1.

The loss is defined as pθ, zq " Itpx ´θqy ě 0u where θ ą 0 is a learned parameter. The true risk can be explicitly calculated. For θ ą 0, we have: R P pθq " pθ 1`α `p1 ´pqp1 ´θq 1`β , and the optimal threshold θ p can be found by derivating the risk R P pθq. The derivative is zero when θ satisfies:

pp1 `αqθ α " p1 ´pqp1 `βqp1 ´θq β . (9.14) Solving Eq. (9.14) is straightforward for well-chosen values of α, β, which are detailed in Table 9.1. The excess error Epp 1 , pq " R P pθ p1 q ´RP pθ p q for the diagonal entries of The results of Fig. 9.1 show that the optimum for the train distribution is significantly different from the optimum for the test distribution when the problem involves Bayes noise.

Generalities on Real Data Experiments

Strategy to induce bias in balanced datasets. In the two real data experiments below, the same strategy is used to induce class distribution bias or strata bias. Since both experiments involve a small test dataset, it is kept intact, while we discard elements of the train dataset to induce bias between the train and test datasets. The bias is parameterized by a single parameter γ, such that when γ is close to one, there is little strata or class bias, while when γ approaches 0, bias is extreme.

The bias we induce is inspired by a power law, which is often used to model unequal distributions. Each value of a modality, i.e. a possible value of the stratum or class of an instance, is given by one of the values of the power law at random. Formally, the target train distribution tp 1 k u K k"1 over a modality S P t1, . . . , Ku, is defined for all 1 ď k ď K as:

p 1 k " γ ´tK{2u σpkq p k ř K l"1 γ ´tK{2u σpkq p k ,
where σ is a random permutation in t1, . . . , Ku.

To generate a train dataset with modality distribution tp 1 k u K k"1 , we sample instances from the original train data set D n " tpX 1 i , Y 1 i , S 1 i qu n i"1 , where Y 1 i is the class, S 1 i is the modality. For the MNIST experiment, S 1 i " Y 1 i , while for Section 8.4, the value S 1 i is the stratum of the instance i. The output of the train dataset is noted as D n , see Algorithm 1 for the detailed algorithm of the train dataset generation.

Algorithm 1 Biased training dataset generation

Input: D n " tpX 1 i , Y 1 i , S 1 i qu n i"1 , tp 1 k u K k"1 Output: D n D Ð H # Initialize the result index set. for k " 1, . . . , K do I k Ð ti | 1 ď i ď n, S 1 i " ku # Count the candidates for each modality. end for m samp Ð minp#I 1 , . . . , #I K q while m samp ą 0 do m 1 , . . . , m K Ð Mpm samp , p 1 , . . . , p K q # M is the multinomial law. end while for k " 1, . . . , K do D k Ð RandSetpm k , I k q # RandSetpn, Xq is a random subset of n elements of X. I k Ð I k zD k end for m samp Ð min p#I 1 , . . . , #I K q D Ð D Y ˆK Ť k"1 D k Ḋn Ð tpX 1 i , Y 1 i , S 1 i q | i P Du Return D n
Models Both MNIST and ImageNet experiments compare two models: a linear model and a multilayer perceptron (MLP) with one hidden layer. Given a classification problem of input x of dimension d with K classes, precisely with d " 784, K " 10 for MNIST and d " 2048, K " 1000 for ImageNet data, a linear model simply learns the weights matrix W P R dˆK and the bias vector b P R K and outputs logits l " W J x `b. On the other hand, the MLP has a hidden layer of dimension h " tpd `Kq{2u and learns the weights matrices W 1 P R d,h , W 2 P th, Ku and bias vectors b 1 P R h , b 2 P R K and outputs logits l " W J 2 hpW J 1 x `b1 q `b2 where h is the ReLU function, i.e. h : x Þ Ñ maxpx, 0q. The number of parameters for each dataset and each model is summarized in Table 9

Cost function

The cost function is the Softmax Cross-Entropy (SCE), which is the most used classification loss in deep learning. Specifically, given logits l " pl 1 , . . . , l K q P R K , the softmax function is γ : R k Ñ r0, 1s K with γ " pγ 1 , . . . , γ K q and for all k P t1, . . . , Ku,

γ k : l Þ Ñ exppl k q ř K j"0 exppl j q
.

Given an instance with logits l and ground truth class value y, the expression of the softmax cross-entropy cpl, yq is cpl, yq "

K ÿ k"1
Ity " ku log pγ k plqq .

The loss that is reweighted depending on the cases as described in Section 8.3 is this quantity cpl, yq. The loss on the test set is never reweighted, since the test set is the target distribution. The weights and bias of the model that yield the logits are tuned using backpropagation on this loss averaged on random batches of B elements of the training data summed with the regularization term λ ¨P where λ is a hyperparameter that controls the strength of the regularization.

Preprocessing, optimization, parameters The images of ILSVRC were encoded using the implementation of ResNet50 provided by the library keras1 , see [START_REF] Chollet | [END_REF], by taking the flattened output of the last convolutional layer.

Optimization is performed using a momentum batch gradient descent algorithm, which updates the parameters θ t at timestep t with an update vector v t by performing the following operations:

v t " γv t´1 `η∇Cpθ t´1 q, θ t " θ t´1 ´vt ,
where η is the step size and γ is the momentum, as explained in [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF].

The parameters of the learning processes are summarized in Table 9.3. The weight decay parameters λ were cross-validated by trying values on a logarithmic scale, e.g. for ImageNet t10 ´4, 10 ´3, 10 ´2, 10 ´1, 1u and then trying more fine-grained values between the two best results, e.g. for ImageNet 10 ´3 was best and 10 ´2 was second best so we tried t0.002, 0.003, 0.004, 0.005u. The standard deviation initialization of the weights was chosen by trial-and-error to avoid overflows. The step size was fixed after trying different values to have fast convergence while keeping good convergence properties.

Classes Bias Experiment for MNIST

The impact of the bias correction in the multi-class supervised learning setting described in Section 2 is illustrated on a widely used dataset for studying classification tasks: the MNIST dataset.

The MNIST dataset is composed of images X P R d of digits and labels being the value of the digits. In our experiment, we learn to predict the value of the digit so we have K " 10 classes corresponding to digits between 0 and 9. The dataset contains 60, 000 images for training and 10, 000 images for testing, all equally distributed within the classes. There is therefore no class bias between train and test samples in the original dataset.

Bias between classes is induced using the power law strategy described above. We deal with the classification task associated to pX, Y q with a linear model or a MLP with one hidden layer that optimizes the softmax cross-entropy (SCE) using momentum gradient descent. We compare the uniform weighting of each instance in the train set (corresponding to the case where there is no reweighting described in Eq. ( 9.2)) with the reweighting of each instance using the proportion of each label Y for the train and test datasets as described in Eq. (9.7).

The optimization dynamics are summarized in Fig. 9.3. We report the median over 100 runs of these values for the test set and a fixed random sample of the train set. For the test set, we represent 95% confidence-intervals in a lighter tone. The x-axis corresponds to the number of iterations of the learning process. For the uniform weights, we see that the misclassification rate is pretty low for the train set, but poor for the test set. By reweighting the instances, we see that we favor low error over the test set, which gives a miss probability reduced by half.

Strata Reweighting Experiments for ImageNet

We focus here on the learning from biased stratified data setting introduced in Section 9.3.1 by leveraging the ImageNet Large Scale Visual Recognition Challenge (ILSVRC); a well-known benchmark for the image classification task, see [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] for more details.

The challenge consists in learning a classifier from 1.3 million training images spread out over 1,000 classes. Performance is evaluated using the validation dataset of 50,000 images of ILSVRC as our test dataset. ImageNet is an image database organized according to the WordNet hierarchy, which groups nouns in sets of related words called synsets. In that context, images are examples of very precise nouns, e.g. flamingo, which are contained in a larger synset, e.g. bird. Table 9.4: Original categories used to construct the strata for the experiment of Section 8.4.

The impact of reweighting in presence of strata bias is illustrated on the ILSVRC classification problem with broad significance synsets for strata. We detail the data preprocessing necessary to assign strata to the ILSVRC data. These were constructed using a list of 27 high-level categories found on the ImageNet website2 and copied in Table 9.4. Each ILSVRC image has a ground truth low level synset, either from the name of the training instance, or in the validation textfile for the validation dataset, that is provided by the ImageNet website. The ImageNet API3 provides the hierarchy of synsets in the form of is-a relationships, e.g. a flamingo is a bird. Using this information, for each synset in the validation and training database, we gathered all of its ancestors in the hierarchy that were in the table Table 9.4. Most of the synsets had only one ancestor, which then accounts for one stratum. Some of the synsets had no ancestors, or even several ancestors in the table, which accounts in for extra strata, either a no-category stratum or a strata composed of the union of several ancestors. The final distribution of the dataset over the created strata is summarized by Figure 9.4. Observe the presence of a no strata stratum and of unions of two high-level synsets strata, e.g. n00015388 n01905661. A definition provided by the API of each of the synsets involved in the strata is given in Table 9.5.

To do this, we encode the data using deep neural networks. Specifically our encoding is the flattened output of the last convolutional layer of the network ResNet50 introduced in [START_REF] He | Deep residual learning for image recognition[END_REF]. It was trained for classification on the training dataset of ILSVRC. The encodings X 1 , . . . , X n belong to a 2,048-dimensional space.

A total of 33 strata are derived from a list of high-level categories provided by ImageNet 4 . By default, strata probabilities p k and p 1 k for 1 ď k ď K are equivalent between training and testing datasets, meaning that reweighting by Φ would have little to no effect. Since our testing data n 0 0 0 1 5 3 8 8 _ n 0 1 8 6 1 7 7 8 n o _ s t r a t a n 0 3 1 8 3 0 8 0 n 0 3 1 2 2 7 4 8 n 0 4 5 2 4 3 1 3 n 0 0 0 1 5 3 8 8 _ n 0 1 9 0 5 6 6 1 n 0 0 0 1 5 3 8 8 _ n 0 1 5 0 3 0 6 1 n 0 4 3 4 1 6 8 6 n 0 0 0 1 5 3 8 8 _ n 0 1 6 6 1 0 9 1 n 0 3 1 8 3 0 8 0 _ n 0 3 8 0 0 9 3 3 n 0 3 4 0 5 7 2 5 n 1 3 1 3 4 9 4 7 n 0 0 0 1 5 3 8 8 _ n 0 2 5 1 2 0 5 3 n 0 7 7 0 7 4 5 1 n 0 4 4 5 1 8 1 8 n 0 2 7 2 9 8 3 7 n 0 9 2 8 7 9 6 8 n 0 4 2 8 5 1 4 6 n 0 7 5 6 6 3 4 0 n 0 0 0 1 5 3 8 8 _ n 0 1 6 2 7 4 2 4 n 1 2 9 9 2 8 6 8 n 0 4 5 1 6 6 7 2 n 0 3 3 0 9 8 0 8 n 0 0 0 1 5 3 8 8 n 0 2 7 2 9 8 3 7 _ n 0 3 1 8 3 0 8 0 n 0 0 0 1 7 2 2 2 _ n 1 1 6 6 9 9 2 1 n 0 3 1 8 3 0 8 0 _ n 0 4 4 5 1 8 1 8 n 0 3 1 8 3 0 8 0 _ n 0 4 5 2 4 3 1 3 n 0 4 3 4 1 6 8 6 _ n 0 4 5 2 4 3 1 3 n 0 7 5 6 6 3 4 0 _ n 0 7 7 0 7 4 5 1 n 0 4 2 8 5 1 4 6 _ n 0 4 5 2 4 3 1 3 n 0 3 1 2 2 7 4 8 _ n 0 3 1 8 3 0 8 is the validation data of ILSVRC, we have around 25 times more training than testing data. Introducing a strata bias parameter 0 ď γ ď 1, we set the strata train probabilities such that p 1 k " γ 1´tK{2u{k p k before renormalization and remove train instances so that the train set has the right distribution over strata. When γ is close to one, there is little to no strata bias. In contrast, when γ approaches 0, strata bias is extreme.

The models used are a linear model and a multilayer perceptron (MLP) with one hidden layer. We report significantly better performance when reweighting using the strata information, compared to the case where the strata information is ignored, see Table 9.6. For comparison, we added two reference experiments: one which reweights the train instances by the class probabilities, which we do not know in a stratified population experiment, and one with more data and no strata bias because it uses all of the ILSVRC train data. The dynamics of the learning process can be found in Fig. 9.6 for the linear model and in Fig. 9.7 for the MLP model. The dominance of the linear model over the MLP can be justified by the much higher number of parameters to estimate for the MLP.

Conclusion

In this chapter, we have considered specific transfer learning problems, where the distribution of the test data P differs from that of the training data, P 1 , and is absolutely continuous with respect to the latter. This setup encompasses many situations in practice, where the data acquisition process is not perfectly controlled. In this situation, a simple change of measure shows that the target risk may be viewed as the expectation of a weighted version of the basic empirical risk, with ideal weights given by the importance function Φ " dP {dP 1 , unknown in practice. Throughout this chapter, we have shown that, in statistical learning problems corresponding to a wide variety of practical applications, these ideal weights can be replaced by statistical versions based solely on the training data combined with very simple information about the target distribution. The generalization capacity of rules learned from biased training data by minimization of the weighted empirical risk has been established, with learning bounds. These theoretical results are also illustrated with several numerical experiments.

While some biases can be corrected by addressing the representativeness of data, some have other intrisic reasons. For example, in many settings, differentials in performance are observed even though representativeness is not an issue, such as with respect to age in facial recognition.

In those type of situations, one may wish to learn a model that corrects for those differentials with explicit constraints, while simultaneously performing the usual optimization for predictive performance. It is the subject of fairness in machine learning, which we address in the next chapter in the context of fairness in bipartite ranking.

Chapter 10

Learning Fair Scoring Functions Summary: In Part II, we framed similarity learning as a scoring problem on the product space, which fits with the usual evaluation of the 1:1 biometric identification problem. In many instances, the simple reweighting procedure of Chapter 9 does not suffices to correct for discrepancies in error rates between sensitive groups. Facial recognition practitioners have confirmed the importance of those discrepancies (Chapter 1). This concern resonates with the idea of fairness, which has received a lot of attention in classification, but not so much in the important problem of bipartite scoring/ranking. However, bipartite ranking/scoring is a preliminary step to our formalization of 1:1 verification (Part II). In this chapter, we propose a flexible approach to group fairness for bipartite ranking, the standard learning task of scoring binary labeled data. We argue that the functional nature of the ROC curve, the gold standard measure of ranking performance in this context, leads to several ways of formulating fairness constraints. We introduce general classes of fairness conditions based on AUC and ROC curves, and establish generalization bounds for scoring functions learned under such constraints. Our analysis is in the form of finite-sample bounds (Chapter 2) and is supported by usual results on bipartite ranking (Chapter 3) and U -statistics (Chapter 4). Beyond the theoretical formulation and results, we design practical learning algorithms and illustrate our approach with numerical experiments on real and synthetic data.

Introduction

With the availability of data at ever finer granularity through the Internet-of-Things and the development of technological bricks to efficiently store and process this data, the infatuation with machine learning and artificial intelligence is spreading to nearly all fields (science, transportation, energy, medicine, security, banking, insurance, commerce, etc.). Expectations are high. AI is supposed to allow for the development of personalized medicine that will adapt a treatment to the patient's genetic traits. Autonomous vehicles will be safer and be in service for longer. There is no denying the opportunities, and we can rightfully hope for an increasing number of successful deployments in the near future. However, AI will keep its promises only if certain issues are addressed. In particular, machine learning systems that make significant decisions for humans, regarding for instance credit lending in the banking sector [START_REF] Chen | Fair lending needs explainable models for responsible recommendation[END_REF], diagnosis in medicine [START_REF] Deo | Machine learning in medicine[END_REF] or recidivism prediction in criminal justice [START_REF] Rudin | The age of secrecy and unfairness in recidivism prediction[END_REF], should guarantee that they do not penalize certain groups of individuals. In the specific case of facial recognition, practitioners and governmental agencies have recorded differences in accuracy between ethnicities [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF], which is expected to penalize people of color.

Hence, stimulated by the societal demand, notions of fairness in machine learning and guarantees that they can be fulfilled by decision-making models trained under appropriate constraints have recently been the subject of a good deal of attention in the literature, see e.g. [START_REF] Dwork | Fairness through awareness[END_REF] or [START_REF] Kleinberg | Inherent trade-offs in the fair determination of risk scores[END_REF] among others. Fairness constraints are generally modeled by means of a (qualitative) sensitive variable, indicating membership to a certain group (e.g., ethnicity, gender). The vast majority of the work dedicated to algorithmic fairness in machine learning focuses on binary classification. In this context, fairness constraints force the classifiers to have the same true positive rate (or false positive rate) across the sensitive groups. For instance, [START_REF] Hardt | Equality of opportunity in supervised learning[END_REF] and [START_REF] Pleiss | On fairness and calibration[END_REF] propose to modify a pre-trained classifier in order to fulfill such constraints without deteriorating classification performance. Other work incorporates fairness constraints in the learning stage (see e.g., [START_REF] Agarwal | A reductions approach to fair classification[END_REF]; [START_REF] Woodworth | Learning non-discriminatory predictors[END_REF]; Zafar et al. (2017a[START_REF] Zafar | Fairness constraints: A flexible approach for fair classification[END_REF]; [START_REF] Menon | The cost of fairness in binary classification[END_REF]; [START_REF] Bechavod | Learning fair classifiers: A regularization-inspired approach[END_REF]; [START_REF] Williamson | Fairness risk measures[END_REF]). In addition to algorithms, statistical guarantees (in the form of generalization bounds) are crucial for fair machine learning, as they ensure that the desired fairness will be met at deployment. Such learning guarantees have been established in [START_REF] Donini | Empirical risk minimization under fairness constraints[END_REF] for the case of fair classification.

The present chapter is also devoted to algorithmic fairness, but for a different problem: namely, learning scoring functions from binary labeled data. This statistical learning problem, known as bipartite ranking, is of considerable importance in applications. It covers in particular tasks such as credit scoring in banking, pathology scoring in medicine or recidivism scoring in criminal justice, for which fairness requirements are a major concern [START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF]. While it can be formulated in the same probabilistic framework as binary classification, bipartite ranking is not a local learning problem: the goal is not to guess whether a binary label Y is positive or negative from an input observation X but to rank any collection of observations X 1 , . . . , X n by means of a scoring function s : X Ñ R so that observations with positive labels are ranked higher with large probability. Due to the global nature of the task, evaluating the performance is itself a challenge. The gold standard measure, the ROC curve, is functional: it is the PP-plot of the false positive rate vs the true positive rate (the higher the curve, the more accurate the ranking induced by s). Sup-norm optimization of the ROC curve has been investigated in [START_REF] Clémençon | A statistical view of clustering performance through the theory of U-processes[END_REF]Vayatis (2009, 2010), while most of the literature focuses on the maximization of scalar summaries of the ROC curve such as the AUC criterion [START_REF] Agarwal | Generalization bounds for the area under the ROC curve[END_REF]Clémençon et al., 2008;[START_REF] Zhao | Online AUC maximization[END_REF] or alternative measures [START_REF] Rudin | Ranking with a p-norm push[END_REF][START_REF] Clémençon | Ranking the best instances[END_REF][START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF].

We propose a thorough study of fairness in bipartite ranking, where the goal is to guarantee that sensitive variables have little impact on the rankings induced by a scoring function. Similar to ranking performance, there are various possible options to measure the fairness of a scoring function. We start by introducing a general family of AUC-based fairness constraints, which encompasses recently proposed notions [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF][START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF][START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF] in a unified framework and enables the design of generic methods and generalization bounds. Then, we argue that the AUC is not always appropriate to characterize fairness as two ROC curves with very different shapes may have the same AUC. This motivates our design of richer definitions of fairness for scoring functions related to the ROC curves themselves. Crucially, this novel functional view of fairness has strong implications for fair classification: classifiers obtained by thresholding such fair scoring functions approximately satisfy definitions of classification fairness for a wide range of thresholds. We establish the first generalization bounds for learning fair scoring functions under both AUC and ROC-based fairness constraints, following in the footsteps of [START_REF] Donini | Empirical risk minimization under fairness constraints[END_REF] for fair classification. Due to the complex nature of the ranking measures, our proof techniques largely differ from [START_REF] Donini | Empirical risk minimization under fairness constraints[END_REF] and require non standard technical tools (e.g. to control deviations of ratios of U -statistics). Beyond our theoretical analysis, we propose efficient training algorithms based on gradient descent and illustrate the practical relevance of our approach on synthetic and real datasets.

The chapter is organized as follows. Section 10.2 reviews bipartite ranking and fairness for ranking. In Section 10.3, we study AUC-based fairness constraints. We then analyze richer ROC-based constraints in Section 10.4. Section 10.5 presents numerical experiments and we conclude in Section 10.6.

Background and Related Work

In this section, we introduce the main concepts involved in the subsequent analysis.

Background and Related Work

Probabilistic framework. We place ourselves in the same binary classification setting as Chapter 2 and Chapter 2, thus introduce a random pair pX, Y q characterized by the triplet pp, H, Gq or the pair pF, ηq (see Section 2.2 in Chapter 2). In the context of fairness, we consider a third random variable Z which denotes the sensitive attribute taking values in t0, 1u. The pair pX, Y q is said to belong to salient group 0 (resp. 1) when Z " 0 (resp. Z " 1). The distribution of the triplet pX, Y, Zq can be expressed as a mixture of the distributions of X, Y |Z " z. Following the conventions described in Chapter 2, we introduce the quantities p z , G pzq , H pzq as well as F pzq , η pzq . For instance, p 0 " PtY " `1|Z " 0u and the distribution of X|Y " `1, Z " 0 is written G p0q , i.e. for A Ă X , G p0q pAq " PtX P A|Y " `1, Z " 0u. We denote the probability of belonging to group z by q z :" PtZ " zu, with q 0 " 1 ´q1 .

Bipartite Ranking

The goal of bipartite ranking is to learn an order relationship on X for which positive instances are ranked higher than negative ones with high probability. This order is defined by transporting the natural order on the real line to the feature space through a scoring function s : X Ñ R. Given any distribution F over X and a scoring function s, we write as F s the cumulative distribution function of spXq when X follows F . Additionally, we introduce H s and G s , defined in Section 3.2 of Chapter 3. ROC analysis. In bipartite ranking, one focuses on the ability of the scoring function s to separate positive from negative data. This is reflected by ROC Hs,Gs (see Eq. (3.1) of Chapter 3), which gives the false negative rate vs. false positive rate of binary classifiers g s,t : x Þ Ñ 2 ¨Itspxq ą tu ´1 obtained by thresholding s at all possible thresholds t P R. The global summary AUC Hs,Gs (see Definition 3.2 of Chapter 3) serves as a standard performance measure (Clémençon et al., 2008).

ROC curves can more generally be used to visualize the dissimilarity between two real-valued distributions in many applications, e.g. anomaly detection, medical diagnosis, information retrieval. We present a more general definition in Definition 10.1 below.

Definition 10.1. (ROC curve) Let g and h be two cumulative distribution functions on R. The ROC curve related to the distributions gpdtq and hpdtq is the graph of the mapping ROC h,g : α P r0, 1s Þ Ñ 1 ´g ˝h´1 p1 ´αq. When gpdtq and hpdtq are continuous, it can alternatively be defined as the parametric curve t P R Þ Ñ p1 ´hptq, 1 ´gptqq.

The L 1 distance of ROC h,g to the diagonal conveniently quantifies the deviation from the homogeneous case, leading to a generalization of the classic area under the ROC curve (AUC) criterion, AUC h,g :" ş ROC h,g pαqdα " PtS ą S 1 u `1 2 PtS " S 1 u, where S and S 1 denote independent random variables, drawn from hpdtq and gpdtq respectively.

Empirical estimates. In practice, the scoring function s is learned based on a training set tpX i , Y i qu n i"1 of n i.i.d. copies of the random pair pX, Y q. Let n `and n ´be the number of positive and negative data points respectively, which are sums of i.i.d. random (indicator) variables. We introduce by p G s and p H s the empirical counterparts of G s and H s , defined formally in Section 3.2.3 of Chapter 3. For any two distributions F, F 1 over R, we denote the empirical counterparts of AUC F,F 1 and ROC F,F 1 by:

z AUC F,F 1 :" AUC p F , x F 1 and z ROC F,F 1 p¨q :" ROC p F , x F 1 p¨q,
respectively. In particular, considering z AUC Hs,Gs defined in Eq. (3.2) of Chapter 3, we have z AUC Hs,Gs " AUC y Hs, x

Gs . Empirical risk minimization for bipartite ranking typically consists in maximizing z AUC Hs,Gs over a class of scoring functions (see e.g. Clémençon et al. (2008); [START_REF] Zhao | Online AUC maximization[END_REF]).

Fairness in Binary Classification

In binary classification, the goal is to learn a mapping function g : X Þ Ñ t´1, `1u that predicts the output label Y from the input random variable X as accurately as possible as measured by a loss function Lpgq. One common example is the probability of error Lpgq " PtgpXq ‰ Y u, which is minimized by the Bayes classifier g ˚, where g ˚pxq " 2Itηpxq ą 1{2u ´1. Any classifier g can be defined by its unique acceptance set A g :" tx P X | gpxq " `1u Ă X .

Existing notions of fairness for binary classification (see [START_REF] Zafar | Fairness constraints: A flexible approach for fair classification[END_REF], for a detailed treatment) aim to ensure that g makes similar predictions (or errors) for the two groups. We mention here the common fairness definitions that depend on the ground truth label Y . Parity in mistreatment requires that the proportion of errors is the same for the two groups:

M p0q pgq " M p1q pgq, (10.1)
where M pzq pgq :" PtgpXq ‰ Y | Z " zu. While this requirement is natural, it considers that all errors are equal: in particular, one can have a high false positive rate (FPR) H p1q pA g q for one group and a high false negative rate (FNR) G p0q pA g q for the other. This can be considered unfair when acceptance is an advantage, e.g. for job applications. A solution to this issue is to consider parity in false positive rates, which writes:

H p0q pA g q " H p1q pA g q, (10.2)
as well as parity in false negative rates, which writes: G p0q pA g q " G p1q pA g q.

(10.3)

We refer to [START_REF] Zafar | Fairness constraints: A flexible approach for fair classification[END_REF] for a detailed treatment of fairness definitions for classification.

Remark 10.2 (Connection to bipartite ranking). A score function s : X Ñ R induces an infinite collection of binary classifiers tg s,t pxq :" signpspxq ´tqu tPR . While one could fix a threshold t P R and try to enforce fairness on g s,t , we are interested in notions of fairness for the score function itself, independently of a particular choice of threshold.

Fairness in Ranking

Fairness for rankings has only recently become a research topic of interest, and most of the work originates from the informational retrieval and recommender systems communities. Given a set of items with known relevance scores, they aim to extract a (partial) ranking that balances utility and notions of fairness at the group or individual level, or through a notion of exposure over several queries [START_REF] Zehlike | Fa*ir: A fair top-k ranking algorithm[END_REF][START_REF] Celis | Ranking with fairness constraints[END_REF][START_REF] Biega | Equity of attention: Amortizing individual fairness in rankings[END_REF][START_REF] Singh | Fairness of exposure in rankings[END_REF]. [START_REF] Singh | Policy learning for fairness in ranking[END_REF] and [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF] extend the above work to the learning to rank framework, where the task is to learn relevance scores and ranking policies from a certain number of observed queries that consist of query-item features and item relevance scores (which are typically not binary). This framework is fundamentally different from the bipartite ranking setting considered here.

AUC constraints. In a setting closer to ours, [START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF] introduces measures to quantify the fairness of a known scoring function on binary labeled data (they do not address learning). Similar definitions of fairness are also considered by [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF], and by [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] in a classification context. Below, we present these fairness measures in the unified form of equalities between two AUCs. In general, the AUC can be seen as a measure of homogeneity between distributions (Clémençon et al., 2009). Eq. ( 10.4) is referred to as intra-group pairwise or subgroup AUC fairness and Eq. (10.5) as pairwise accuracy [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF] or Background Positive Subgroup Negative (BNSP) AUC fairness [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. Eq. ( 10.4) requires the ranking performance to be equal within groups, which is relevant for instance in applications where groups are ranked separately (e.g., candidates for two types of jobs). Eq. (10.5) enforces that positive instances from either group have the same probability of being ranked higher than a negative example, and can be seen as the ranking counterpart of parity in false negative rates in binary classification [START_REF] Hardt | Equality of opportunity in supervised learning[END_REF], see Eq. ( 10.3). [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] and [START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF] The work of [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] refers to Eq. (10.6) as Backgroup Positive Subgroup Negative (BPSN) AUC, and can be seen as the ranking counterpart of parity in false positive rates in classification [START_REF] Hardt | Equality of opportunity in supervised learning[END_REF], see Eq. ( 10.2). The Average Equality Gap (AEG) [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] can be written as AUCpG s , G pzq s q ´1{2 for z P t0, 1u. Eq. (10.7) thus corresponds to an AEG of zero, i.e. the scores of the positives of any group are not stochastically larger than those of the other. [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF] and [START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF] also define the inter-group pairwise fairness or xAUC parity:

AUC H p0q s ,G p1q s " AUC H p1q s ,G p0q s , (10.8)
which imposes that the positives of a group can be distinguished from the negatives of the other group as effectively for both groups. Below, we generalize these AUC-based definitions and derive generalization bounds and algorithms for learning scoring functions under such fairness constraints.

Fair Scoring via AUC Constraints

In this section, we give a thorough treatment of the problem of statistical learning of scoring functions under AUC-based fairness constraints. First, we introduce a general family of AUCbased fairness definitions which encompasses those presented in Section 10.2.3. We then derive the first generalization bounds for the bipartite ranking problem under such AUC-based fairness constraints. Finally, we propose a practical algorithm to learn such fair scoring functions.

A Family of AUC-based Fairness Definitions

Many sensible fairness definitions can be expressed in terms of the AUC between two distributions. We now introduce a framework to formulate AUC-based fairness constraints as a linear combination of 5 elementary fairness constraints, and prove its generality. Given a scoring function s, we introduce the vector Cpsq " pC 1 psq, . . . , C 5 psqq J , where the C l psq's, l P t1, . . . , 5u, are elementary fairness measurements. More precisely, the value of |C 1 psq| (resp. |C 2 psq|) quantifies the resemblance of the distribution of the negatives (resp. positives) between the two sensitive attributes:

C 1 psq " AUC H p0q s ,H p1q s ´1{2, C 2 psq " 1{2 ´AUC G p0q s ,G p1q s
, while C 3 psq, C 4 psq and C 5 psq measure the difference in ability of a score to discriminate between positives and negatives for any two pairs of sensitive attributes:

C 3 psq " AUC H p0q s ,G p0q s ´AUC H p0q s ,G p1q s , C 4 psq " AUC H p0q s ,G p1q s ´AUC H p1q s ,G p0q s , C 5 psq " AUC H p1q s ,G p0q s ´AUC H p1q s ,G p1q s
.

The family of fairness constraints we consider is then the set of linear combinations of the C l psq " 0:

C Γ psq : Γ J Cpsq " ř 5 l"1 Γ l C l psq " 0, with Γ " pΓ 1 , . . . , Γ 5 q J P R 5 . (10.9)

Theorem 10.3 (Informal). The family pC Γ psqq ΓPR 5 compactly captures all relevant AUC-based fairness constraints, including (but not limited to) those proposed in previous work. Now, we detail the mathematical derivation of the formal statement of this result and give examples of new fairness constraints that can be expressed with Eq. (10.9). We will show that the family pC Γ psqq ΓPR 5 covers a wide array of possible fairness constraints in the form of equalities of the AUC's between mixtures of the distributions Dpsq, with:

Dpsq :" pH p0q s , H p1q s , G p0q s , G p1q s q J .
Denote by pe 1 , e 2 , e 3 , e 4 q the canonical basis of R 4 , as well as the constant vector 1 " ř 4 k"1 e k . Introducing the probability vectors α, β, α 1 , β 1 P P where P " tv | v P R 4 `, 1 J v " 1u, we define the following constraint: AUC α J Dpsq,β J Dpsq " AUC α 1J Dpsq,β 1J Dpsq .

(10.10) Theorem 10.4 below formalizes Theorem 10.3, and rules out AUC constraints that are not satisfied when H p0q " H p1q and G p0q " G p1q . Such undesirable fairness constraints are those which actually give an advantage to one of the groups, such as AUC G p0q s G p1q s " 2AUC Hs,Gs ´1 which is a special case of Eq. ( 10.10) that requires the scores of the positives of group 1 to be higher than those of group 0.

Theorem 10.4. The following propositions are equivalent:

1. Eq. (10.10) is satisfied for any measurable scoring function s when H p0q " H p1q , G p0q " G p1q and F pηpXq " pq ă 1, 2. Eq. (10.10) is equivalent to C Γ psq for some Γ P R 5 , 3. pe 1 `e2 q J rpα ´α1 q ´pβ ´β1 qs " 0.

Proof. Denote Dpsq " pD 1 psq, D 2 psq, D 3 psq, D 4 psqq J :" pH p0q s , H p1q s , G p0q
s , G p1q s q J . For any pi, jq P t1, . . . , 4u 2 , we introduce the notation:

AUC Di,Dj : s Þ Ñ AUC Dipsq,Dj psq .
Introduce a function M such that M psq P R 4ˆ4 for any s : X Ñ R, and for any pi, jq P t1, . . . , 4u, the pi, jq coordinate of M writes:

M i,j " AUC Di,Dj ´1 2 .
First note that, for any s, M psq is antisymmetric i.e. M j,i psq " ´Mi,j psq for any pi, jq P t1, . . . , 4u 2 . Then, with pα, βq P P 2 , we have that:

AUC α J D,β J D " α J M β `1 2 " xM, αβ J y `1 2 ,
where xM, M 1 y " trpM J M 1 q is the standard dot product between matrices. Eq. (10.10) writes as:

xM, αβ J ´α1 β 1 J y " 0. (10.11)

Case of α " α 1 and β ´β1 " δpe i ´ej q.

Consider the specific case where α " α 1 and β ´β1 " δpe i ´ej q with i ‰ j and δ ‰ 0, then xM, αpβ ´β1 q J y " δK pαq i,j , where:

K pαq i,j :" xM, αpe i ´ej q J y " 4 ÿ k"1 α k " AUC D k ,Di ´AUC D k ,Dj ‰ , " pα i `αj q " 1 2 ´AUC Di,Dj  `ÿ kRti,ju α k " AUC D k ,Di ´AUC D k ,Dj ‰ .
The preceding definition implies that K pαq i,j " ´Kpαq j,i . Using ř K k"1 α k " 0, we can express every K pαq i,j as a linear combinations of the C l 's plus a remainder, precisely:

K pαq 1,2 " ´pα 1 `α2 q C 1 ´α3 pC 3 `C4 q ´α4 pC 4 `C5 q, K pαq 1,3 " ˆ1 2 ´AUC D1,D3 ˙`α 2 p´C 1 `C3 `C4 q `α4 p´C 2 `C3 q, K pαq 1,4 " ˆ1 2 ´AUC D1,D4 ˙`α 2 p´C 1 `C4 `C5 q `α3 pC 2 ´C3 ´C4 q, K pαq 2,3 " ˆ1 2 ´AUC D2,D3 ˙`α 1 pC 1 ´C3 ´C4 q `α4 p´C 2 `C5 q, K pαq 2,4 " ˆ1 2 ´AUC D2,D4 ˙`α 1 pC 1 ´C4 ´C5 q `α3 pC 2 ´C5 q, K pαq 3,4 " pα 3 `α4 q C 2 `α1 C 3 `α2 C 5 .
Hence, it suffices that ti, ju " t1, 2u or ti, ju " t3, 4u for Eq. ( 10.11) to be equivalent to C Γ for some Γ P R 5 .

Case of α " α 1 .

Any of the β ´β1 writes as a positive linear combination of e i ´ej with i ‰ j, since:

β ´β1 " 1 4 ÿ i‰j `βi `β1 j ˘pe i ´ej q , which means that, since K pαq i,j " ´Kpαq j,i : xM, αpβ ´β1 q J y " 1 4 ÿ i‰j `βi `β1 j ˘Kpαq i,j " 1 4 ÿ iăj `rβ i ´βj s ´rβ 1 i ´β1 j s ˘Kpαq i,j . (10.12)
Note that any linear combination of the K

γ 1 ¨Kpαq 1,3 `γ2 ¨Kpαq 1,4 `γ3 ¨Kpαq 2,3 `γ4 ¨Kpαq 2,4
, where γ P R 4 with 1 J γ " 0 writes as a weighted sum of the C l for l P t1, . . . , 5u.

Hence, it suffices that β 1 `β2 " β 1 1 `β1 2 for Eq. ( 10.12) to be equivalent to some C Γ for some Γ P R 5 .

General case.

Note that, using the antisymmetry of M and Eq. (10.12):

xM, αβ J ´α1 β 1 J y " xM, αpβ ´β1 q J y `xM, pα ´α1 qβ 1J y, " xM, αpβ ´β1 q J y ´xM, β 1 pα ´α1 q J y, " 1 4

ÿ iăj " `rβ i ´βj s ´rβ 1 i ´β1 j s ˘Kpαq i,j ´`rα i ´αj s ´rα 1 i ´α1 j s ˘Kpβ 1 q i,j ı .
Hence, it suffices that pe 1 `e2 q J rpα ´α1 q ´pβ ´β1 qs " 0 for Eq. ( 10.11) to be equivalent to some C Γ for some Γ P R 5 .

Conclusion.

We denote the three propositions of Theorem 10.4 as P 1 , P 2 and P 3 .

Assume that H p0q " H p1q , G p0q " G p1q and F pηpXq " 1{2q ă 1, then C l " 0 for any l P t1, . . . , 5u, which gives:

xM psq, αβ J ´α1 β 1 J y " 1 4 ˆ1 2 ´AUC Hs,Gs ˙¨ÿ iPt1,2u ÿ jPt3,4u "`r β i ´βj s ´rβ 1 i ´β1 j s ˘´`r α i ´αj s ´rα 1 i ´α1 j s ˘‰' , " ˆ1 2
´AUC Hs,Gs ˙pe 1 `e2 q J rpα ´α1 q ´pβ ´β1 qs, Table 10.1: Value of Γ " pΓ l q 5 l"1 for all of the AUC-based fairness constraints in the chapter for the general formulation of Eq. (10.9).

Eq. Γ 1 Γ 2 Γ 3 Γ 4 Γ 5
(10.4) 0 0

1 3 1 3 1 3
(10.5) 0 0 q 0 p1´p 0 q 1´p 0 q 1 p1´p 1 q 1´p

(10.6) 0 0

q 0 p 0 2p 1 2 q 1 p 1 2p (10.7) 0 1 0 0 0 (10.8) 0 0 0 1 0 (10.13) 0 p 0 1 ´p0 0 0
It is known that:

AUC Hη,Gη " 1 2 `1 4pp1 ´pq ij ηpxq ´ηpx 1 q dF pxqdF px 1 q, which means that AUC Hη,Gη " 1{2 implies that ηpXq " p almost surely (a.s.), and the converse is true.

Assume P 1 is true, then AUC Hη,Gη ą 1{2, hence pe 1 `e2 q J rpα ´α1 q ´pβ ´β1 qs " 0 because Eq. ( 10.11) is verified for η, and we have shown P 1 ùñ P 3 .

Assume P 3 is true, then xM, αβ J ´α1 β 1 J y writes as a linear combination of the C l 's, l P t1, . . . , 5u, and we have shown that P 3 ùñ P 2 .

Assume P 2 is true, then observe that if H p0q " H p1q and G p0q " G p1q , then any C Γ is satisfied for any Γ P R 5 , and we have shown that P 2 ùñ P 1 , which concludes the proof.

Recovering existing AUC-based fairness constraints. All AUC-based fairness constraints proposed in previous work (see Section 10.2.3) write as instances of our general definition for a specific choice of Γ, see Table 10.1. Note that Γ might depend on the quantities q 0 , p 0 , q 1 , p 1 .

Expressing new AUC-based fairness constraints. Relevant fairness constraints that have not been considered in previous work can be expressed using our general formulation. Denoting F p0q s " p1 ´p0 qH p0q s `p0 G p0q s , consider for instance the following constraint:

AUC F p0q s ,G p0q s " AUC F p0q s ,G p1q s .
(10.13)

It equalizes the expected position of the positives of each group with respect to a reference group (here group 0). Another fairness constraint of interest is based on the rate of misranked pairs when one element is in a specific group:

Eps, zq :"p1{2q ¨PtspXq " spXq | Y ‰ Y 1 , Z " zu `PtpspXq ´spXqqpY ´Y 1 q ą 0 | Y ‰ Y 1 , Z " zu.
The equality Eps, 0q " Eps, 1q can be seen as the analogue of parity in mistreatment for the task of ordering pairs, see Eq. (10.1). It is easy to see that this constraint can be written in the form of Eq. ( 10.10) and that point 1 of Theorem 10.4 holds, hence it is equivalent to C Γ psq for some Γ P R 5 .

As we show below, our unifying framework enables the design of general problem formulations, statistical guarantees and algorithms which can then be instantiated to the specific notion of AUC-based fairness that the practitioner is interested in.

Learning Problem and Statistical Guarantees

We now formulate the problem of fair ranking based on the fairness definitions introduced above.

Introducing fairness as a hard constraint is tempting, but may be costly in terms of ranking performance. In general, there is indeed a trade-off between the ranking performance and the level of fairness.

For a family of scoring functions S and some instantiation Γ of our general fairness definition in Eq. ( 10.9), we thus define the learning problem as follows:

max sPS AUC Hs,Gs ´λ Γ J Cpsq , (10.14)

where λ ě 0 is a hyperparameter balancing ranking performance and fairness.

For the sake of simplicity and concreteness, we focus on the special case of the fairness definition in Eq. ( 10.4) -one can easily extend our analysis to any other instance of our general definition in Eq. (10.9). Thus, we denote by s λ the scoring function that maximizes the objective L λ psq of Eq. (10.14), where:

L λ psq :" AUC Hs,Gs ´λˇˇA UC H p0q s ,G p0q s ´AUC H p1q s ,G p1q s ˇˇ. Given a training set tpX i , Y i , Z i qu n i"1 of n i.i.d.
p H pzq s ptq " p1{n pzq ´q ř n i"1 I tZ i " z, Y i " ´1, spX i q ď tu , p G pzq s ptq " p1{n pzq `q ř n i"1 I tZ i " z, Y i " `1, spX i q ď tu .
Recalling the notation z AUC F,F 1 :" AUC p F , p F 1 from Section 10.2.1, the empirical problem writes:

p L λ psq :" z AUC Hs,Gs ´λˇˇz AUC H p0q s ,G p0q s ´z AUC H p1q s ,G p1q s ˇˇ.
We denote its maximizer by p s λ . We can now state our statistical learning guarantees for fair ranking.

Theorem 10.5. Assume the class of functions S is VC-major with finite VC-dimension V ă `8 and that there exists ą 0 s.t. min zPt0,1u,yPt´1,1u PtY " y, Z " zu ě . Then, for any δ ą 0, for all n ą 1, we have with probability (w.p.) at least 1 ´δ:

2 ¨rL λ ps λq ´Lλ pp s λ qs ď C c V n ¨p4λ `1{2q `c logp13{δq n ´1 ¨´4λ `p4λ `2q ¯`Opn ´1q.
Proof. Usual arguments imply that: L λ ps λq ´Lλ pp s λ q ď 2 ¨sup sPS p L λ psq ´Lλ psq . Introduce the quantities:

p ∆ " sup sPS z AUC Hs,Gs ´AUC Hs,Gs , p ∆ 0 " sup sPS z AUC H p0q s ,G p0q s ´AUC H p0q s ,G p0q s , and p ∆ 1 " sup sPS z AUC H p1q s ,G p1q s ´AUC H p1q s ,G p1q s
.

The triangular inequality implies that:

sup sPS p L λ psq ´Lλ psq ď p ∆ `λ p ∆ 0 `λ p ∆ 1 .
Case of p ∆: Note that:

z AUC Hs,Gs " pnpn ´1q{2n `n´q ¨p U K psq, where p U K psq " 2 npn ´1q ÿ 1ďiăjďn KppspX i q, Y i , Z i q, pspX j q, Y j , Z j qq,
and Kppt, y, zq, pt 1 , y 1 , z 1 qq " Itpy ´y1 qpt ´t1 q ą 0u `p1{2q ¨Ity ‰ y 1 , t " t 1 u. The quantity p U K psq is a known type of statistic and is called a U -statistic, see e.g. [START_REF] Lee | U -statistics: Theory and practice[END_REF] for an overview. We write U K psq :" Er p U K psqs " 2pp1 ´pqAUC Hs,Gs .

Introducing p m :" n `n´{ n 2 ´pp1 ´pq, we have that, since sup sPS | p U K psq| ď 2n `n´{ pnpn ´1qq:

p ∆ ď npn ´1q 2n `n´´1 2pp1 ´pq ¨sup sPS p U K psq `1 2pp1 ´pq ¨sup sPS p U K psq ´UK psq , ď 1 pp1 ´pq p m `n`nń 2 pn ´1q `1 2pp1 ´pq ¨sup sPS p U K psq ´UK psq .
The properties of the shatter coefficient described in [START_REF] Györfi | Principles of Nonparametric Learning[END_REF] (Theorem 1.12 therein) and the fact that S is VC-major, imply that the class of sets:

tppx, yq, px 1 , y 1 qq | pspxq ´spx 1 qqpy ´y1 q ą 0u sPS , is VC with dimension V . The right-hand side term above is covered by Corollary 3 of Clémençon et al. (2008), presented in the preliminaries as Corollary 4.5 in Chapter 4, and we deal now with the left-hand side term.

Hoeffding's inequality implies that, for all n ě 1, w.p. ě 1 ´δ,

n ǹ ´p ď d logp 2 δ q 2n . (10.15)
Since n ´" n ´n`, we have that:

p m " p1 ´2pq ´nǹ ´p¯´´n ǹ ´p¯2 .
It follows that:

p m `n`nń 2 pn ´1q ď | p m| `1 4pn ´1q ď p1 ´2pq c logp2{δq 2n `An pδq,
where:

A n pδq " logp2{δq 2n `1 4pn ´1q " O `n´1 ˘.
Finally, a union bound between Corollary 3 of Clémençon et al. (2008) and Eq. ( 10.15) gives that, using the upper-bound 1{p2nq ď 1{pn ´1q: for any n ą 1, w.p. ě 1 ´δ,

pp1 ´pq ¨p ∆ ď C c V n `2p1 ´pq c logp3{δq n ´1 `An p2δ{3q. (10.16)
Case of p ∆ 0 : Note that:

z AUC H p0q s ,G p0q s " ´npn ´1q{2n p0q `np0q ´¯¨p U K p0q psq,
where K p0q ppt, y, zq, pt 1 , y 1 , z 1 qq " Itz " 0, z 1 " 0u ¨Kppt, y, zq, pt 1 , y 1 , z 1 qq. We denote:

U K p0q psq :" Er p U K p0q psq " 2q 2 0 p 0 p1 ´p0 q ¨AUC H p0q s ,G p0q s
.

Following the proof of the bound for p ∆, introducing p m 0 :" n p0q `np0q ´{n

2 ´q2 0 p 0 p1 ´p0 q, p ∆ 0 ď 1 q 2 0 p 0 p1 ´p0 q p m 0 `np0q `np0q ń2 pn ´1q `1 2q 2 0 p 0 p1 ´p0 q ¨sup sPS p U K p0q psq ´UK p0q psq .
The right-hand side term above is once again covered by Lemma 1. We deal now with the left-hand side term, note that:

p m 0 " n p0q `np0q n 2 ´q2 0 p 0 ´¨« n p0q ǹ ff 2 ´q2 0 p 2 0 ', " q 0 p 0 ˆnp0q n ´q0 ˙`q 0 p1 ´2p 0 q ˜np0q ǹ ´q0 p 0 ņp0q ǹ ´q0 p 0 ¸ˆn p0q n ´q0 ˙´˜n p0q ǹ ´q0 p 0 ¸2 .
A union bound of two Hoeffding inequalities gives that for any n ą 1, w.p. ě 1 ´δ, we have simultaneously:

n p0q n ´q0 ď c logp4{δq 2n and n p0q ǹ ´q0 p 0 ď c logp4{δq 2n .
(10.17)

It follows that:

p m 0 `np0q `np0q ń2 pn ´1q ď | p m 0 | ` `np0q ˘2
4n 2 pn ´1q ď q 0 p1 ´p0 q c logp4{δq 2n `Bn pδq, where

B n pδq " 1 4pn ´1q `logp4{δq n .
Finally, a union bound between (Clémençon et al., 2008, Corollary 3) and Eq. ( 10.17) gives, using the upper-bound 1{p2nq ď 1{pn ´1q: for any n ą 1, w.p. ě 1 ´δ,

q 2 0 p 0 p1 ´p0 q ¨p ∆ 0 ď C c V n `p1 `q0 p1 ´p0 qq c logp5{δq n `Bn p4δ{5q. (10.18) Case of p ∆ 1 :
One can prove a similar result as Eq. ( 10.18) for p ∆ 1 : for any n ą 1, w.p. ě 1 ´δ,

q 2 1 p 1 p1 ´p1 q ¨p ∆ 1 ď C c V n `p1 `q1 p1 ´p1 qq c logp5{δq n `Bn p4δ{5q. (10.19) Conclusion:
Under the assumption min zPt0,1u min yPt´1,1u PtY " y, Z " zu ě , note that minpp, 1 ´pq ě 2 . A union bound between Eq. (10.16), Eq. ( 10.18), and Eq. (10.19). gives that: for any δ ą 0 and n ą 1, w.p. ě 1 ´δ,

2 ¨pL λ ps λq ´Lλ pp s λ qq ď C c V n ¨ˆ4λ `1 2 ˙`c logp13{δq n ´1 ¨´4λ `p4λ `2q ¯`Opn ´1q,
which concludes the proof.

Theorem 10.5 establishes a learning rate of Op1{ ? nq for our problem of ranking under AUC-based fairness constraints, which holds for any distribution of pX, Y, Zq as long as the probability of observing each combination of label and group is bounded away from zero. As the natural estimate of the AUC involves sums of dependent random variables, the proof of Theorem 10.5 does not follow from usual concentration inequalities on standard averages. Indeed, it requires controlling the uniform deviation of ratios of U -processes indexed by a class of functions of controlled complexity.

Training Algorithm

Maximizing directly p L λ by gradient ascent (GA) is not feasible, since the criterion is not continuous, hence not differentiable. Hence, we decided to approximate any indicator function x Þ Ñ Itx ą 0u by a logistic function σ : x Þ Ñ 1{p1 `e´x q.

We learn with stochastic gradient descent using batches B N of N elements sampled with replacement in the training set D n " tpX i , Y i , Z i qu n i"1 , with B N " tpx i , y i , z i qu N i"1 . We assume the existence of a small validation dataset V m , with V m " tpx pvq i , y pvq i , z pvq i qu m i"1 . In practice, one splits a total number of instances n `m between the train and validation dataset.

The approximation of z AUC Hs,Gs on the batch writes:

Ć AUC Hs,Gs " 1 N `N´ÿ iăj σ rpspx i q ´spx j qqpy i ´yj qs ,
where N `:" ř N i"1 Ity i " `1u and N ´:" N ´N`i s the number of positive instances in the batch. Similarly, we denote by N pzq `:" N pzq ´N pzq ´the number of positive instances of group z in the batch, with:

N pzq :" N ÿ i"1 Itz i " zu and N pzq `:" N ÿ i"1 Itz i " z, y i " `1u.
Due to the high number of term involved involved in the summation, the computation of Ć AUC Hs,Gs can be very expensive, and we rely on approximations called incomplete U-statistics, which simply average a random sample of B nonzero terms of the summation, see [START_REF] Lee | U -statistics: Theory and practice[END_REF]. We refer to [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] and [START_REF] Papa | SGD algorithms based on incomplete u-statistics: Largescale minimization of empirical risk[END_REF] for details on their statistical efficiency and use in the context of SGD algorithms, respectively. Formally, we define the incomplete approximation with B P N pairs of Ć AUC Hs,Gs as:

Ć AUC pBq Hs,Gs :" 1 B ÿ pi,jqPD B σ rpspx i q ´spx j qqpy i ´yj qs ,
where D B is a random set of B pairs in the set of all possible pairs tpi, jq | 1 ď i ă j ď N u.

Here, we give more details on our algorithm for the case of the AUC-based constraint Eq. ( 10.4).

The generalization to other AUC-based fairness constraints is straightforward. For any z P t0, 1u the relaxation of z AUC H pzq ,G pzq on the batch writes:

Ć AUC H pzq s ,G pzq s " 1 N pzq `N pzq ´ÿ iăj zi"zj "z
σ rpspx i q ´spx j qqpy i ´yj qs .

Similarly as Ć AUC Hs,Gs , we introduce the sampling-based approximations Ć AUC

pBq H pzq s ,G pzq s
for any z P t0, 1u.

To minimize the absolute value in Eq. (10.14), we introduce a parameter c P r´1, `1s, which is modified slightly every n adapt iterations so that it has the same sign as the evaluation of Γ J Cpsq on V m . This allows us to write a cost in the form of a weighted sum of AUC's, with weights that vary during the optimization process. Precisely, it is defined as:

r L λ,c psq :" ´1 ´Ć AUC Hs,Gs ¯`λ ¨c ´Ć AUC H p1q s ,G p1q s ´Ć AUC H p0q s ,G p0q s ¯`λ reg 2 ¨ W 2 2 ,
where λ reg is a regularization parameter and W 2 2 is the sum of the squared L 2 norms of all of the weights of the model. The sampling-based approximation of r L λ,c writes:

r L pBq λ,c psq :" ˆ1 ´Ć AUC pBq Hs,Gs ˙`λ ¨c ˆĆ AUC pBq H p1q s ,G p1q s ´Ć AUC pBq H p0q s ,G p0q s ˙`λ reg 2 ¨ W 2 2 .
The algorithm is detailed in Algorithm 2, where sng is the sign function, i.e. sgnpxq " 2Itx ą 0u´1 for any x P R.

Richer ROC-based Fairness Constraints

The equality between two AUC's considered as fairness constraints in Section 10.3 only quantifies a stochastic order between distributions, not the equality between these distributions. In particular, two very different distributions can be indistinguishable in terms of AUC. As a matter of fact, for continuous ROCs, the equality between their two AUCs only implies that the two ROCs intersect at one unknown point. As a consequence, AUC-based fairness can only guarantee that there exists some threshold t P R that induces a non-trivial classifier g s,t :" signpspxq ´tq satisfying a notion of fairness for classification, as detailed below in Section 10.4.1. Unfortunately, the value of t and the corresponding point α of the ROC curve are not known in advance and are difficult to control.

Algorithm 2 Practical algorithm for learning with the AUC-based constraint Eq. (10.4).

Input: training set D n , validation set V m c Ð 0 for i " 1 to n iter do B N Ð N observations sampled with replacement from D n s Ð updated scoring function using a gradient-based algorithm ( e.g. ADAM), using the derivative of r L pBq λ,c psq on B N if pn iter mod n adapt q " 0 then

∆AUC Ð z AUC pBvq H p1q s ,G p1q s ´z AUC pBvq H p0q s ,G p0q s computed on V m c Ð c `sgnp∆AUCq ¨∆c c Ð minp1, maxp´1, cqq end if end for Output: scoring function s
To see why this can be a problem for fairness, consider for instance pairwise accuracy fairness in Eq. (10.5), which specifies that the probability of scoring a positive instance of a given group higher than a negative example should be the same across groups. Despite this, it is possible for positives from one group to appear less often in the top 1% scores than positives from the other group (via "compensation" in other regions of the score distribution). In a use-case where only top 1% individuals get some advantage (or obtain better advantages), this would be unfair for one group. Learning with AUC-based constraints can thus lead to scoring functions that are inadequate for the use-case of interest. These limitations serve as a motivation to introduce new ROC-based fairness constraints in Section 10.4.2.

Limitations of AUC-based Constraints

In this section, we clarify the relationship between known propositions for fairness in classification on the one hand, and our AUC-based and ROC-fairness for bipartite ranking on the other hand. In a nutshell, we show that: (i) if a scoring function s satisfies an AUC-based fairness constraint, there exists a certain threshold t such that the classifier g s,t obtained by thresholding s at t satisfies fair classification constraints, and (ii) ROC-based fairness constraints allow to directly control the value of t for which g s,t is fair, and more generally to achieve classification fairness for a whole range of thresholds, which is useful to address task-specific operational constraints.

Pointwise ROC equality and fairness in binary classification. As mentioned in the main text, a scoring function s : X Ñ R induces an infinite family of binary classifiers g s,t : x Þ Ñ 2 ¨Itspxq ą tu ´1 indexed by thresholds t P R. The following proposition shows that one of those classifiers satisfies a fairness constraint as soon as appropriate group-wise ROC curves are equal for some value α P r0, 1s.

Proposition 10.6. Under appropriate conditions on the score function s (i.e., s P S where S satisfies Assumption 10.9), we have that:

• If p 0 " p 1 and s satisfies: (10.20) for some α P r0, 1s, then there exists pt 0 , t 1 q P p0, T q 2 , s.t. M p0q pg s,t0 q " M p1q pg s,t1 q, which resembles parity in mistreatment (see Eq. 10.1).

ROC H p0q s ,G p0q s pαq " ROC H p1q s ,G p1q s pαq,
• If s satisfies:

ROC Hs,G p0q s pαq " ROC Hs,G p1q s pαq, (10.21)

for some α P r0, 1s, then g s,t satisfies fairness in FNR (see 10.3) for some threshold t P p0, T q.

• If s satisfies: (10.22) for some α P r0, 1s, then g s,t satisfies parity in FPR (see 10.2) for some threshold t P p0, T q.

ROC H p0q s ,Gs pαq " ROC H p1q s ,Gs pαq,
Proof. We go over each case.

Case of Eq. (10.20). Eq. ( 10.20) also writes:

G p0q s ˝´H p0q s ¯´1 pαq " G p1q s ˝´H p1q s ¯´1 pαq,
Introduce t z " pH pzq s q ´1pαq then G pzq s pt z q " H pzq s pt z q " α for any z P t0, 1u, since H pzq s is increasing. Also, M pzq pg s,tz q " Ptg s,tz pXq ‰ Y | Z " zu, " p z G pzq s pt z q `p1 ´pz qp1 ´Hpzq s pt z qq, " p2α ´1qp z `p1 ´αq, which implies the result.

Case of Eq. (10.21). Eq. ( 10.21) also writes:

G p0q s ˝H´1 s pαq " G p1q s ˝H´1
s pαq, which translates to:

G p0q `spXq ď H ´1 s pαq ˘" G p1q `spXq ď H ´1 s pαq ˘,
hence g s,t satisfies fairness in FNR (Eq. ( 10.3)) for the threshold t " H ´1 s pαq. Case of Eq. (10.22). Eq. ( 10.22) also writes: G s ˝pH p0q s q ´1pαq " G s ˝pH p1q s q ´1pαq, which implies, since G s , H p0q s and H p1q s are increasing:

H p0q s ˝pH p0q s q ´1pαq " H p1q s ˝pH p0q s q ´1pαq, and:

H p0q ´spXq ą pH p0q s q ´1pαq ¯" H p1q ´spXq ą pH p0q s q ´1pαq ¯, hence g s,t satisfies fairness in FPR (Eq. (10.2)) for the threshold t " pH p0q s q ´1pαq.

Relation with AUC-based fairness. For continuous ROCs, the equality between their two AUCs implies that the two ROCs intersect at some unknown point, as shown by Proposition 10.7 (a simple consequence of the mean value theorem), proven below. Theorem 3.3 in [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] corresponds to the special case of Proposition 10.7 when h " g, h 1 ‰ g 1 .

Proposition 10.7. Let h, g, h 1 , g 1 be c.d.f.s on R such that ROC h,g and ROC h 1 ,g 1 are continuous. If AUC h,g " AUC h 1 ,g 1 , then there exists α P p0, 1q, such that ROC h,g pαq " ROC h 1 ,g 1 pαq.

Proof. Consider ψ : r0, 1s Þ Ñ r´1, 1s: ψpαq " ROC h,g pαq ´ROC h 1 ,g 1 pαq, it is continuous, hence integrable, and with:

Ψptq "

ż t 0 ψpαqdt,
Note that Ψp1q " AUC h,g ´AUC h 1 ,g 1 " 0 " Ψp0q. The mean value theorem implies that there exists α P p0, 1q such that: ROC h,g pαq " ROC h 1 ,g 1 pαq.

Proposition 10.7, combined with Proposition 10.6, implies that when a scoring function s satisfies some AUC-based fairness constraint, there exists a threshold t P R inducing a non-trivial classifier g s,t :" signpspxq ´tq that satisfies some notion of fairness for classification at some unknown threshold t. For example, it is straightforward from Proposition 10.6 and Proposition 10.7 that:

• Eq. ( 10.4) implies parity in mistreatment for some thresholds,

• Eq. (10.5), Eq. (10.7) and Eq. ( 10.13) all imply parity in FNR for some threshold,

• Eq. (10.6) implies parity in FPR for some threshold.

The principal drawback of AUC-based fairness constraints is that it guarantees the existence of a single (unknown) t for which the fair binary classification properties are verified by g s,t , and that the corresponding ROC point α cannot be easily controlled. Furthermore, one can impose the equalities Eq. ( 10.20), Eq. ( 10.21) and Eq. ( 10.22) for several values of α such that thresholding the score behaves well for many critical situations. Specifically, under Assumption 10.9, we prove in Proposition 10.8 below that pointwise constraints over a discretization of the interval of interest approximates its satisfaction on the whole interval. This behavior, confirmed by our empirical results (see Sections 10.5 and 10.5.3), is relevant for many real-world problems that requires fairness in binary classification to be satisfied for a whole range of thresholds t in a specific region. For instance, in biometric verification, one is interested in low false positive rates (i.e., large thresholds t). We refer to [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF] for an evaluation of the fairness of facial recognition systems in the context of 1:1 verification. is differentiable, and its derivative is bounded by B{b. Indeed, for any K 1 , K 2 P K, since K 1 is continuous and increasing, the inverse function theorem implies that pK 1 q ´1 is differentiable. It follows that K 2 ˝K´1

1 is differentiable and that its derivative satisfies:

`K2 ˝K´1 1 ˘1 " K 1 2 ˝K´1 1 K 1 1 ˝K´1 1 ď B b .
Let k P t0, . . . , mu, and α P rα The aforementioned condition of equality between the distribution of the positives (resp. negatives) of the two groups are equivalent to satisfying ∆ G,α psq " 0 ( resp. ∆ H,α psq " 0) for any α P r0, 1s. When both of those conditions are satisfied, all of the AUC-based fairness constraints covered by Eq. ( 10.9) are verified, as it is easy to see that C l psq " 0 for all l P t1, . . . , 5u. Furthermore, guarantees on the fairness of classifiers g s,t induced by s hold for all possible thresholds t. While this strong property is desirable, it is challenging to enforce in practice due to its functional nature, and in many cases it may only be achievable by completely jeopardizing the ranking performance.

pkq F , α pk`1q F s. Since α Þ Ñ ∆ F,
We thus propose to implement the satisfaction of a finite number of fairness constraints on ∆ H,α psq and ∆ G,α psq for specific values of α. Let m H , m G P N be the number of constraints for the negatives and the positives respectively, as well as α H " rα p1q H , . . . , α pm H q H s P r0, 1s m H and α G " rα p1q G , . . . , α pm G q G s P r0, 1s m G the points at which they apply (sorted in strictly increasing order). With the notation Λ :" pα, λ H , λ G q, we can introduce the criterion L Λ psq, defined as:

L Λ psq " AUC Hs,Gs ´řm H k"1 λ pkq H ˇˇ∆ H,α pkq H psq ˇˇ´ř m G k"1 λ pkq G ˇˇ∆ G,α pkq G psq ˇˇ, λ H " rλ p1q H , . . . , λ pm H q H s P R m H `, λ G " rλ p1q G , . . . , λ pm G q G s P R m G
`being trade-off hyperparameters. This criterion is flexible enough to address the limitations of AUC-based constraints outlined above. In particular, a practitioner can choose the points in α H and α G so as to guarantee the fairness of classifiers obtained by thresholding the scoring function at the desired trade-offs between false negative/false positive rates. Furthermore, in applications where the threshold used at deployment can vary in a whole interval, as in biometric verification [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF], we show in Proposition 10.8 under some regularity assumption on the ROC curve (see Assumption 10.9 in Section 10.4.3), if a small number of fairness constraints m F are satisfied at discrete points of the interval for F P tH, Gu, then one obtains guarantees in sup norm on α Þ Ñ ∆ F,α (and therefore fair classifiers) in the entire interval.

Statistical Guarantees and Training Algorithm

We now prove statistical guarantees for the maximization of p L Λ psq, the empirical counterpart of L Λ : pαq ´α for any α P r0, 1s. We denote by s Λ the maximizer of L Λ over S, and p s Λ the maximizer of p L Λ over S.

z AUC Hs,Gs ´řm H k"1 λ pkq H ˇˇp ∆ H,α pkq H psq ˇˇ´ř m G k"1 λ pkq G ˇˇp ∆ G,
Our analysis relies on the following regularity assumption on the ROC curve.

Assumption 10.9. The class S of scoring functions take values in p0, T q for some T ą 0, and the family of c.d.f.s K " tG pzq s , H pzq s : s P S, z P t0, 1uu satisfies: (a) any K P K is continuously differentiable, and (b) there exists b, B ą 0 s.t. @pK, tq P K ˆp0, T q, b ď |K 1 ptq| ď B. The latter condition is satisfied when scoring functions do not have flat or steep parts, see [START_REF] Clémençon | Ranking the best instances[END_REF] (Remark 7) for a discussion.

Theorem 10.10. Under Assumption 10.9 and those of Theorem 10.5, for any δ ą 0, n ą 1, w.p. ě 1 ´δ:

2 ¨rL Λ ps Λq ´LΛ pp s Λ qs ď C p1{2 `2 C Λ,K q b V n `2 p1 `3C Λ,K q b logp19{δq n´1 `Opn ´1q, where C Λ,K " p1 `B{bqp s λ H `s λ G q, with s λ H " ř m H k"1 λ pkq H and s λ G " ř m G k"1 λ pkq G .
Proof. Usual arguments imply that: L Λ ps Λq ´LΛ pp s Λ q ď 2 ¨sup sPS p L Λ psq ´LΛ psq . As for Theorem 10.5, the triangle inequality implies that: ¨U pzq n,s ptq for any t P p0, T q. First note that, using the same type of arguments as for Theorem 10.5: sup s,αPSˆr0,1s

p L Λ psq
|T 2 ps, αq| ď sup s,tPSˆp0,T q p G p1q s ptq ´Gp1q s ptq ,

ď 1 q 1 p 1 n p1q ǹ ´q1 p 1 `1 q 1 p 1 ¨sup s,tPSˆp0,T q p U p1q n,s ptq ´U p1q n,s ptq .
The same arguments as for Eq. (10.25) apply, which means that: for any δ ą 0, n ą 0, w.p. ě 1 ´δ, (10.27) where C is an universal constant.

sup s,tPSˆp0,T q p U p1q n,s ptq ´U p1q n,s ptq ď C c V n `c 2 logp2{δq n ,
A union bound of Eq. ( 10.27) and a standard Hoeffding inequality for n p1q `finally imply that: for any δ ą 0, n ą 1, w.p. ě 1 ´δ,

sup sPS |T 2 ps, αq| ď C q 1 p 1 c V n `3 q 1 p 1 c logp4{δq 2n .
(10.28)

Conclusion.

Combining Eq. ( 10.26) and Eq. ( 10.28), one obtains that: for any δ ą 0, n ą 1, w.p. ě 1 ´δ, sup s,αPSˆr0,1s p ∆ G,α psq ´∆G,α psq ď C ˆ1 q 1 p 1 `B bq 0 p 0 ˙c V n `ˆ3 q 1 p 1 `3B bq 0 p 0 ˙c logp8{δq 2n .

(10.29) and a result with similar form can be shown for sup s,αPSˆr0,1s p ∆ H,α psq ´∆H,α psq by following the same steps.

Under the assumption min zPt0,1u min yPt´1,1u PtY " y, Z " zu ě , a union bound between Eq. (10.29), its equivalent for p ∆ H,α and Eq. ( 10.16) gives, with the upper-bound 1{p2nq ď 1{pn´1q: for any δ ą 0, n ą 1, w.p. ě 1 ´δ,

2 ¨pL Λ ps Λq ´LΛ pp s Λ qq ď 2 ˆ1 `3p s λ H `s λ G q " 1 `B b ˙c logp19{δq n ´1 `C ˆ1 2 `2 p s λ H `s λ G q " 1 `B b ˙c V n `Opn ´1q,
which concludes the proof.

Theorem 10.10 generalizes the learning rate of Op1{ ? nq of Theorem 10.5 to ranking under ROC-based constraints. Like Theorem 10.5, its proof relies on results on U -processes, but further requires a study of the deviations of the empirical ROC curve seen as ratios of empirical processes indexed by S ˆr0, 1s. In that regard, our analysis builds upon the decomposition proposed in [START_REF] Hsieh | Nonparametric and semiparametric estimation of the receiver operating characteristic curve[END_REF], which enables the derivation of uniform bounds over S ˆr0, 1s from results on standard empirical processes [START_REF] Van Der Vaart | Weak convergence and empirical processes[END_REF].

Training Algorithm

The approach presented here follows the same broad principles as that of Section 10. psq| " 0. This constraint can be seen as one that imposes equality between the true positive rates and false positive rates for the problem of discriminating between the negatives (resp. positives) of sensitive group 1 against those of sensitive group 0 when F " H (resp. F " G). An approximation of this problem's false positive rate (resp. true positive rate) at threshold t is r F

Experiments

We have experimented with our algorithms on real and synthetic data, with various AUC and ROC-based fairness constraints. This section covers extensively all results and details about our experimental setup. In general, our empirical results show that our approaches consistently balance ranking performance and the chosen notion of fairness.

Experimental Details

Scoring function and optimization. We used a simple neural network of various depth D (D " 0 corresponds to linear scoring function, while D " 2 corresponds to a network of 2 hidden layers) where each layer has the same width d (the dimension of the input space), except for the output layer which outputs a real score. We used ReLU's as activation functions. To center and scale the output score we used batch normalization (BN) (see Goodfellow et al., 2016, Section 8.7.1 therein) with fixed values γ " 1, β " 0 for the output value of the network. Algorithm 4 gives a formal description of the network architecture. The intuition for normalizing the output score is that the ranking losses only depend on the relative value of the score between instances, and the more classification-oriented losses of ROC-based constraints only depend on a threshold on the score. Empirically, we observed the necessity of renormalization for the algorithm with ROC-based constraints, as the loss pkq F is zero when p F p0q s pt pkq F q " p F p1q s pt pkq F q P t0, 1u, which leads to scores that drift away from zero during the learning process, as it seeks to satisfy the constraint imposed by pkq F . All of the network weights were initialized using a simple centered normal random variable with standard deviation 0.01.

Algorithm 4 Network architecture.

Input: observation x " h 2 0 P R d , for k " 1 to D do Linear layer: h k " W J k h 2 k´1 `bk with W k P R d,d , b k P R d,1 learned by GD, ReLu layer: h 2 k " maxp0, h 1 k q
where max is an element-wise maximum, end for

Linear layer: h D`1 " w J D`1 h 2 D `bD`1 with w D`1 P R d,1 , b D`1 P R learned by GD, BN layer: h 1 D`1 " ph D`1 ´µD`1 q{σ D`1
, with µ D`1 P R, σ D`1 P R running averages, Output: score spxq of x, with spxq " h 1 D`1 P R.

For both AUC-based and ROC-based constraints, optimization was done with the ADAM algorithm. It has an adaptative step size, so we did not modify its default parameters. Refer to [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF] for more details on gradient descent optimization algorithms.

Other details. For all experiments, we set aside 40% of the data for validation, i.e. m " t0.40pm `nqu with t¨u the floor function, the batch size to N " 100 and the parameters of the loss changed every n adapt " 50 iterations. For any sampling-based approximation computed on a batch B N , we set B " 100, and B v " 10 5 for those on a validation set V m . The value ∆c was always fixed to 0.01 and ∆t to 0.001. We used linear scoring functions, i.e. D " 0, for the synthetic data experiments, and networks with D " 2 for real data.

The experiments were implemented in Python, and relied extensively on the libraries numpy, TensorFlow [START_REF] Abadi | Tensorflow: Large-scale machine learning on heterogeneous distributed systems[END_REF], scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] and matplotlib for plots.

Synthetic Data Experiments

The following examples introduce data distributions that we use to illustrate the relevance of our approach.

Example 10.11. Let X " r0, 1s 2 . For any x " px 1 x 2 q J P X , let F p0q pxq " F p1q pxq " 1, as well as η p0q pxq " x 1 and η p1q pxq " x 2 . We have F " 1 (F is the uniform distribution) and ηpxq " q 0 x 1 `q1 x 2 . Consider linear scoring functions of the form s c pxq " cx 1 `p1 ´cqx 2 parameterized by c P r0, 1s. for any z P t0, 1u as a function of c, illustrating the trade-off between fairness and ranking performance.

Example 10.12. Set X " r0, 1s 2 . For any x P X with x " px 1 x 2 q J , set: F p0q pxq " p16{πq ¨Itx 2 `y2 ď 1{2u, F p1q pxq " p16{3πq ¨It1{2 ď x 2 `y2 ď 1u, and η p0q pxq " η p1q pxq " p2{πq ¨arctanpx 2 {x 1 q.

For all of the synthetic data experiments, our objective is to show that the learning procedure recovers the optimal scoring function when the dataset is large enough. Each of the 100 runs that we perform uses independently generated train, validation and test datasets. The variation that we report on 100 runs hence includes that of the data generation process, which is small since we use large samples. Precisely, for each run, we chose a total of n `m " 10, 000 points for the train and validation sets and a test dataset of size n test " 20, 000. Both algorithms ran for n iter " 10, 000 iterations, and with the same regularization strength λ reg " 0.01.

Example 10.11. First, we illustrate learning with the AUC constraint in Eq. ( 10.4) on the simple problem in Example 10.11. Our experiment shows that we can effectively find trade-offs between ranking accuracy and satisfying Eq. (10.4) using the procedure described in Algorithm 2.

The final solutions of Algorithm 2 with two different values of λ, parameterized by c, are shown in Fig. 10.2. A representation of the value of the corresponding scoring functions on r0, 1s ˆr0, 1s is provided in Fig. 10.3. The median ROC curves for two values of λ over 100 independent runs are shown in Fig. 10.4, with pointwise 95% confidence intervals.

Example 10.12. Example 10.12 allows to compare ROC-based and ROC-based approaches. The former uses Eq. ( 10.4) as constraint and the latter penalizes ∆ H,3{4 psq ‰ 0. The goal of our experiment with Example 10.12 is to show that Algorithm 3 can effectively learn a scoring function s for which the α corresponding to a classifier g s,tα that is fair in FPR is specified in advance, and that the solution can be significantly different from those obtained with AUC-based constraints, i.e. Algorithm 2.

In practice, we compare the solutions of optimizing the AUC without constraint, i.e. Algorithm 2 with λ " 0 with those of Algorithm 2 with λ " 1 and Algorithm 3 where we impose ∆ H,3{4 psq " 0 with strength λ H " 1. To illustrate the results, we introduce the following family of scoring functions s c pxq " ´c ¨x1 `p1 ´cq ¨x2 , parameterized by c P r0, 1s.

In practice, we observe that the different constraints lead to scoring functions with specific tradeoffs between fairness and performance, as summarized in Table 10.2. Results with AUC-based fairness are the same for λ " 0 and λ " 1 because the optimal scoring function for ranking satisfies Eq. (10.4). Fig. 10.5 shows that the AUC-based constraint has no effect on the solution, unlike the ROC-based constraint which is successfully enforced by Algorithm 3. Fig. 10.6 gives two possible scoring functions with Algorithm 3. The median ROC curves for two values of λ H over 100 independent runs are shown in Fig. 10.7, with pointwise 95% confidence intervals.

Real Data Experiments

Datasets. We evaluate our algorithms on four datasets that have been commonly used in the fair machine learning literature. Those are the following: 10.4), a fair solution exists for c " 1{2, but the ranking performance for c ă 1{2 is significantly higher. Parameters. For Algorithm 2, we select different AUC-based fairness constraints depending on the dataset. In the case of Compas (recidivism prediction), being labeled positive is a disadvantage so the approach with AUC-based fairness uses the constraint in Eq. (10.6) to balance FPRs (by forcing the probabilities that a negative from a given group is mistakenly ranked higher than a positive to be the same across groups). Conversely for German (credit scoring), a positive label is an advantage, so we choose Eq. (10.5) to balance FNRs. For Bank and Adult, the problem has no clear connotation so we select Eq. (10.8) to force the same ranking accuracy when comparing the positives of a group with the negatives of another.

Inspired by the consideration that many operational settings focus on learning a good score for small FPR rates, the ROC-based approach is configured to simultaneously align the distribution of FPR and TPR for low FPRs between both groups by penalizing solutions with high |∆ H,1{8 psq|, |∆ H,1{4 psq|, |∆ G,1{8 psq| and |∆ G,1{4 psq|.

Precisely, for every run of Algorithm 3, we set:

m G " m H " 2, α p1q G " α p1q H " 1 8 , α p1q G " α p2q H " 1 4 , λ p1q G " λ p2q G " λ and λ p1q H " λ p2q H " λ.
For all algorithms, we chose the parameter λ from the candidate set P t0, 0.25, 0.5, 1, 5, 10u, where λ " 0 corresponds to the case without constraint. Denoting by r s the output of Algorithm 2 or Algorithm 3, we selected the parameter λ reg of the L2 regularization that maximizes the criterion L λ pr sq (resp. L Λ pr sq) on the validation dataset over the following candidate regularization strength set:

λ reg P t1 ˆ10 ´3, 5 ˆ10 ´3, 1 ˆ10 ´2, 5 ˆ10 ´2, 1 ˆ10 ´1, 5 ˆ10 ´1, 1u.
The selected parameters are summarized in Table 10.4. Results are summarized in Table 10.5, where AUC denotes the ranking accuracy AUC Hs,Gs , and ∆AUC denotes the absolute difference of the terms in the AUC-based fairness constraint of interest. We also report on the values of ∆ F,1{8 and ∆ F,1{4 for any F P tH, Gu and refer the reader to the ROC curves in Fig. 10.8 and Fig. 10.8 for a visual summary of the other values of ∆ F,α with F P tH, Gu and α P r0, 1s. We highlight in bold the best ranking accuracy, and the fairest algorithm for the relevant constraint. All of the numerical evaluations reported below are evaluations on the held-out test set. and observe that without fairness constraint, men who make less than $50K are much more likely to be mistakenly ranked 10.5. Experiments , positive women now have higher scores overall than positive men, while negative men have higher scores than negative women. These observations illustrate the limitations of AUC-based fairness discussed in Section 10.4. To address them, we use the same ROC constraints as we did in Compas so as to align the distributions of positives and negatives of each group in r0, 1{4s, which is again achieved almost perfectly in the entire interval. While the degradation in ranking performance is more noticeable on this dataset, a clear advantage from ROC-based fairness in both datasets is that the obtained scoring function can be thresholded to obtain fair classifiers at a wide range of thresholds.

Results for the dataset Bank. Recall that for this dataset we consider the AUC constraint Eq. (10.8) to force the same ranking accuracy when comparing the positives of a group with the negatives of another. Fig. 10.8 shows that the score learned without constraint implies a stochastic order between the distributions of the problem that writes

H p1q s ĺ H p0q s ĺ G p0q s ĺ G p1q s
, where h ĺ g means that g is stochastically larger than h. This suggests that the task of distinguishing positives from negatives is much harder for observations of the group Z " 0 than for those of the working age population (Z " 1), which could be a consequence of the heterogeneity of the group Z " 0. On the other hand, the left plot representing ROC H p1q s ,G for the setting without constraint gives an appreciation of the magnitude of those differences. Precisely, it implies that it is much harder to distinguish working age positives (Y " `1, Z " 1) from negatives of group Z " 0 than working age negatives from positives of group Z " 0. The correction induced by the AUC constraint suggests that it was due to the fact that scores for positives of the group (Y " `1, Z " 0) were too small compared to the positives of the working age population (Y " `1, Z " 1). Indeed, learning with the AUC constraint roughly equalizes the scores of the positives across both groups Z " 0 and Z " 1. Additionally, in the left plot for learning with AUC constraints, we can see that ROC are much closer for simultaneously small TPR's and FPR's, which implies that thresholding top scores will yield fair classifiers in FPR and TPR again for a whole range of high thresholds.

Results for the dataset German. Recall that for this credit scoring dataset we consider the AUC-based constraint in Eq. (10.5) to force the probabilities that a positive from a given group is mistakenly ranked higher than a negative to be the same across groups. Despite the blatant issues of generalization due to the very small size of the dataset (see Table 10.3), we see in Fig. 10.8 that the learned score without fairness constraints systematically makes more errors for women with good ground truth credit risk, as can be seen from comparing ROC H p1q s ,G ). On the other hand, the score learned with an AUC constraint makes a similar amount of mistakes for both genders, with only slightly more mistakes made on men than women, and the scores spXq conditioned on the events pY " y, Z " zq with z " 0 and z " 1 are more aligned when considering both y " ´1 and y " `1. Finally, while the score learned with a ROC constraint has a slightly higher discrepancy between the AUC's involved in Eq. (10.5) than the one learned with an AUC constraint, one observes that both pairs of distributions pG p0q s , G p1q s q and pH p0q s , H p1q s q are equal for high thresholds. Consistently with the results on other datasets, this suggests that our score leads to classifiers that are fair in FPR and TPR for a whole range of problems where one selects individuals with very good credit risks by thresholding top scores.

Conclusion

In this chapter, we considered the issue of fairness for scoring functions learned from binary labeled data. We proposed general notions of fairness based on the AUC criterion and the ROC curves, and provided statistical guarantees for scoring functions learned via empirical AUC maximization under such fairness constraints. From a practical perspective, we showed how to implement stochastic gradient descent algorithms to solve these problems and illustrated our concepts and methods via numerical experiments. We point out that our framework can be extended to precision-recall curves (as they are a function of the FPR and TPR Clémençon and Vayatis (2009)) and to similarity ranking, a variant of bipartite ranking covering applications like biometric identification [START_REF] Vogel | A probabilistic theory of supervised similarity learning for pointwise ROC curve optimization[END_REF]. In future work, we plan to investigate how the unrelaxed versions of our fairness constraints can be incorporated to ROC curve optimization algorithms based on recursive partitioning, such as those developed in [START_REF] Clémençon | Adaptive partitioning schemes for bipartite ranking[END_REF]; Clémençon and Vayatis (2010). Chapter 11

Conclusion and Perspectives

This thesis proposed to study important problems in biometrics from the point of view of statistical learning theory. Our work provides new theoretical results, which suggest original approaches to these problems, and also provide security guarantees in the form of generalization results.

In that regard, it is a much-needed answer to the rapidly increasing volume of experimental machine learning literature that biometrics researchers have to follow. Biometric identification, and facial recognition in particular, incarnate many machine learning topics simultaneously, such as pairwise learning, sample bias or ranking. For this reason, we considered stylized versions of those problems, as their simultaneous examination would obscure our discourse, and runs the risk of being disregarded as anecdotal by the machine learning community. The richness of the topics tackled by the thesis is a result of this imperative.

Similarity Ranking Theory (Chapter 5 -Part II). First, we proposed to view similarity learning from the perspective of bipartite ranking on a product space, a problem that we named similarity ranking. From this point of view, we proposed another approach to similarity learning that is better suited to the evaluation of biometric systems. In a context of rapid development of the deep metric learning literature [START_REF] Wang | Deep face recognition: A survey[END_REF], this is a much-needed contribution.

We proposed generalization guarantees that extend known results for bipartite ranking. Our analysis immediately suggest other approaches to similarity ranking, for example with extensions of ranking the best criteria in bipartite ranking (Menon and Williamson, 2016, Section 9). Our empirical illustration of the fast generalization rates for the pointwise ROC optimization problem gives an intuitive interpretation of these rates. Since generalization bounds are formulated as orders of n, they sometimes appear impractical. In that regard, future works could build upon our illustration to convey the meaning of fast learning rates. Finally, our extension of the TreeRank algorithm proposes a first practical approach to similarity ranking, with strong theoretical guarantees. As far as we know, very few papers propose learning a piecewise constant similarity. Future work could derive new methods to learn those and study their practical performance, for example by building on top of the work of Clémençon and Vayatis (2010).

Distributed U -Statistics (Chapter 6 -Part II). While the similarity ranking approach seems natural, it comes with specific challenges, such as the usual computational complexity of pairwise learning. In that regard, we exploited recent results on incomplete U -statistics [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF], to alleviate this problem. Additionally, we derived new estimators for estimating U -statistics in a distributed environment, and compared their variance. Our statistical analysis answers concerns regarding the lack of study of distributed computing frameworks from the perspective of statistical accuracy, deplored for example in [START_REF] Jordan | On statistics, computation and scalability[END_REF]. Finally, we proposed a gradient-descent learning approach with our distributed estimators, which is shown to provide trade-offs between communication costs and expected performance of the final model. While our analyses are well suited to estimation, extending them for our distributed gradient-descent algorithm is an open question. This extension is also a technical challenge, since the optimized functional changes during the training process, which makes the analysis more complicated than that of [START_REF] Papa | SGD algorithms based on incomplete u-statistics: Largescale minimization of empirical risk[END_REF]. Performing the analysis would provide interesting results, as it applies to the optimization of any pairwise loss optimized with batch gradient descent, a problem that arises in many practical settings.

Practical Similarity Ranking (Chapter 7 -Part II). Generalization guarantees justify con-sidering similarity ranking, while sampling techniques alleviate the computational costs associated with similarity ranking. However, both contributions do not address the optimization challenges that arise in that setting. From a biometrics perspective, the most important perspective in this thesis is to realize the potential impact of the methods presented, by providing strong empirical evidence of their relevance in practical settings. Indeed, while the rapid adoption of machine learning techniques by private companies has boosted the growth of the field, it also directed most of the attention to papers that propose unequivocal solutions to specific industrial problems. A notorious example for face recognition is [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. In this direction, we proposed gradient descent-based approaches that solve the pointwise ROC optimization problem, as well as illustrations of the TreeRank algorithm for learning similarities, that we all ran on simple toy examples. In that context, the promotion of this work will require finding and presenting pedagogically large-scale experiments that address precisely practical use-cases, which is a promising direction for future work.

Ranking the Most Likely Labels (Chapter 8 -Part III). Similarity ranking is the theoretical formalization of the flagship problem in biometrics, i.e. 1:1 identification. However, it does not covers all of the realities addressed by biometric systems providers. For example, to identify potential suspects in forensics, the objective is usually to return the most likely labels from a database. More generally, hard classification problemse.g. [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] consider other objectives than simple (top-1) classification to evaluate the performance of the system. Notably, a popular alternative performance measure is top-5 classification. In that context, the optimization customarily optimizes the query system for returning solely the best element, since orderings over labels are derived from class probabilities learned by optimizing for classification. The discrepancy between the risk functional and the evaluated objective suggests that that approach is ill-suited. Our work proposes the first theoretically-guaranteed approach to the prediction of a list of most likely results from classification data, by exploiting recent results in ranking median regression (Clémençon et al., 2018). It relies on the famous One-Versus-One (OVO) strategy for multiclass classification, and a byproduct of our analysis is the first generalization guarantees for OVO. From the point of view of biometrics, we provide statistical guarantees for criteria that are tailored to its specific objectives. Another common problem in the field is the absence of a significant difference in matching score between positive and negative instances, as ROC curves only evaluate order relations between the score of observations. In that regard, future work could consider criteria that explicitly force the score function to discriminate significantly between the two types of instances, i.e. with a large gap in score value. Beyond that example, other settings could be addressed, for example by discussing the 1:N identification problem described in [START_REF] Jain | Introduction to Biometrics[END_REF].

Selection Bias Correction (Chapter 9 -Part III). Another topic of interest in biometrics is the representativeness of training databases with respect to the operating conditions of the systems. [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF] measured practically the effects of biased databases in the context of facial recognition. That report has shown that the predominance of white caucasian people in training datasets makes the end algorithms more accurate on this specific type of population. To correct for representativeness issues, we proposed a reweighting scheme inspired from the idea of importance sampling (Bucklew, 2010, chapter 4), that adjusts for differences between the training distribution and the testing distribution. Our reweighting scheme is practical, in all scenarios where additional information is available on the relationship between the training and testing data. Precisely, our work assumes the knowledge of characteristics of the test distribution that can be summarized by a few finite values, such as class probabilities or the probabilities of specific strata in the population. While additional information is available in many practical scenarios, e.g. for a facial recognition system deployed in an airport that contains known proportions of several nationalities, a few other approches were proposed to relate the training and testing distributions. For example, Sugiyama et al. (2007) proposes to estimate a likelihood ratio with a small sample of the test set as auxiliary information. Additionally, while the absolute continuity assumption is necessary for importance sampling, [START_REF] Laforgue | Statistical learning from biased training samples[END_REF] loosened that assumption by considering several training samples. Therefore, a promising direction for future work is the exploration of new ways to incorporate other types of auxiliary information during training, and relate those to our contribution in a general framework.

Learning Fair Scoring Functions (Chapter 10 -Part III). Although accounting for the representativeness of the databases may correct some of the flaws of a model, the distribution of the training data often contains other type of biases that should be explicitly addressed. On that subject, the fairness in machine learning literature proposes approaches to balance the expected responses of a system between two sensitive groups, e.g. men and women. Our work proposed to learn scoring functions that are fair with respect to criteria based on usual ranking measures, such as the AUC-based ones of [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF], as well as a framework to unify all of the AUC-based fairness criteria. Additionally, our work discussed the limitations of AUC criteria. In reaction to those limitations, we introduce new fairness criteria, based directly on the ROC curve.

The usual approach when dealing with fairness constraints is to relax them, and then integrate them with the performance indicator of the task as a penalty term. As we followed this rationale, we could not explicit a notion of optimal fair score function. In the context of fair regression, [START_REF] Chzhen | Fair Regression via Plug-in Estimator and Recalibration With Statistical Guarantees[END_REF] managed to derive an expression of the optimal fair regressor. In bipartite ranking, overcoming this hurdle paves the way for an extension of the partition-based algorithms algorithms of Clémençon and Vayatis (2009) under fairness constraints. Another possibility concerns the extension of the techniques presented here to the case of similarity ranking. Indeed, that extension gives a framework that matches operational considerations in biometrics very closely, and would be justified by the current interest in methods to explicitly correct biases for facial recognition. The experimental component of that work would be supported by the availability of well-suited face databases [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF].

Conclusion.

In general, each topic discussed in the thesis could be further developed in several natural ways, as presented in the manuscript. They could also be exploited to form at-scale experiments that showcase the effectiveness of the suggested approaches, whereby bolstering their adoption. Another possibility consists in using simultaneously the strategies of several topics, while ensuring that their overlap covers an application of significant importance in the machine learning community. Both of these last points are gateways to bringing more visibility to this work. In conclusion, the richness of the issues that arise in biometrics makes for fertile grounds for new theory and new practices in machine learning. This richness spurred the creation of this thesis and can inspire future research.

Chapter 12

Résumé (Summary in French)

Introduction

La biométrie est la discipline qui consiste à distinguer les individus sur la base de leurs attributs physiques ou comportementaux tels que les empreintes digitales, le visage, l'iris et la voix. Dans le monde moderne, la biométrie a de nombreuses applications essentielles, telles que la surveillance des frontières, le commerce électronique et le versement de prestations sociales. Bien que son usage courant soit récent, la discipline n'est pas nouvelle. En effet, à la fin du XIXe siècle, l'officier de police français Alphonse Bertillon a proposé un système d'identification personnelle basé sur la mesure de parties osseuses du corps [START_REF] Jain | Introduction to Biometrics[END_REF].

Aujourd'hui, la mesure biométrique la plus répandue est l'empreinte digitale, suivie par le visage et l'iris. Toutes reposent sur l'acquisition d'images de parties spécifiques du corps. Ainsi, alors que la biométrie est considérée par de nombreux auteurs comme un domaine distinct de la science, son histoire et son développement sont étroitement liés avec celui de la vision par ordinateur, le domaine scientifique interdisciplinaire qui vise à permettre aux ordinateurs d'acquérir une compréhension de haut niveau des images numériques.

Au début des années 2010, les performances des systèmes de vision par ordinateur ont commencé à s'améliorer [START_REF] Goodfellow | Deep Learning[END_REF], en raison du développement des calculs génériques sur processeur graphique (GPGPU, General-purpose Processing on Graphics Processing Units). Ce développement a permis l'adoption généralisée des réseaux de neurones, des modèles statistiques composés de couches qui résument l'information, car leur entraînement bénéficie fortement d'une multiplication très rapide de matrices. L'entraînement des réseaux de neurones consiste à trouver les paramètres du réseau qui minimisent une fonction de perte avec des algorithmes de descente de gradient. Ceux-ci modifient itérativement les paramètres du réseau, en ajoutant une petite quantité qui est négativement proportionnelle au gradient à chaque étape. L'intérêt croissant pour les réseaux de neurones a engendré un énorme corpus de littérature scientifique. La plupart des articles proposent une meilleure architecture pour le modèle, suggèrent une amélioration de la méthode d'optimisation ou introduisent une meilleure fonction de perte pour un problème particulier.

La littérature récente a proposé de nombreuses fonctions de perte pour la biométrie, basées sur l'intuition qu'une séparation plus stricte des identités dans un espace de représentation entraîne une amélioration des performances [START_REF] Wang | Deep face recognition: A survey[END_REF]. Le problème phare de la biométrie est la vérification 1:1, qui vise à vérifier l'identité d'une personne en comparant une mesure en direct avec des données de référence avec une fonction de similarité. Par exemple, l'entrée d'un individu dans une zone restreinte peut nécessiter la conformité de la mesure avec une carte d'identification personnelle. La performance de la vérification 1:1 est évaluée à l'aide de la courbe ROC, un critère fonctionnel qui résume la qualité d'une fonction de similarité. Pour un ensemble de tests, la courbe ROC donne tous les taux de fausses acceptances et de faux rejet pouvant être atteints en seuillant la fonction de similarité. Il s'agit de la mesure référence pour l'évaluation des fonctions de score. Dans la thèse, nous plaidons pour l'exploitation de la littérature sur l'ordonnancement/scoring bipartite pour concevoir des fonctions de perte pour la vérification 1:1, en la considérant comme l'évaluation d'un score sur des paires d'observations. Bien que la littérature traite indépendamment des problèmes de scoring et d'apprentissage sur paires, leur examen simultané est nouveau et pose des défis particuliers.

Les récentes améliorations spectaculaires de la performance pour de nombreuses applications de l'apprentissage automatique préfigurent l'émergence de nouveaux marchés, issus de la maturation de technologies autrefois très expérimentales. L'un de ces marchés est celui de la reconnaissance faciale, qui a enregistré, et devrait maintenir, une croissance exponentielle. Son développement a suscité une couverture médiatique des éventuelles utilisations abusives et biais systémiques de la technologie, qui s'ajoutent aux préoccupations usuelles en matière de protection de la vie privée. Dans ce contexte, les praticiens et les organismes gouvernementaux ont enregistré des différences de précision entre les ethnies dans la reconnaissance faciale [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. Une explication courante est que les bases de données d'images de visages disponibles pour l'entraînement des systèmes de reconnaissance faciale ne sont pas représentatifs de la population générale. L'écart de performance soulève la question plus large de l'équité, une préoccupation commune dans l'automatisation de décisions, et qui a reçu une attention croissante dans la littérature récente en apprentissage automatique [START_REF] Barocas | Fairness and Machine Learning[END_REF]. Certains observateurs ont même commenté que les algorithmes prédictifs risquent d'être simplement "des opinions ancrées dans les mathématiques". Selon cette idée, les praticiens ne devraient pas se concentrer uniquement sur la performance prédictive, mais aussi s'assurer de la conformité de leur système à un ensemble de valeurs morales. La biométrie est également concernée par l'équité. En effet, même lorsque la représentativité de la base de données en termes de genre est prise en compte, les femmes ont généralement un taux de faux positifs plus élevé, possiblement en raison de normes sociales concernant l'apparence. Cela peut conduire à une discrimination systémique, notamment lorsqu'on considère des systèmes qui signalent des personnes recherchées. Dans la thèse, nous proposons d'abord de corriger les biais en utilisant des poids d'importance, ce qui ajuste la distribution des données d'entraînement. La correction couvre plusieurs cas où les données d'entraînement et de test ne correspondent pas, et suppose l'existence d'informations auxiliaires sur un lien entre ces données. Le niveau de généralité où nous nous plaçons est nouveau, et couvre de nombreuses applications importantes en biométrie. Ensuite, nous proposons de modifier les fonctions de perte afin d'intégrer explicitement des considérations d'équité lors de l'apprentissage d'un score pour l'ordonnancement bipartite. L'équité dans un contexte de classification est l'objet de beaucoup de travaux dans la littérature, mais ce n'est pas le cas pour l'équité en ordonnancement bipartite. Compte tenu de notre perspective de scoring sur le problème de vérification 1:1, ce travail est une étape intermédiaire dans l'intégration explicite de contraintes d'équité dans ce problème.

En général, de nombreuses questions abordées en apprentissage automatique se posent simultanément dans la conception de systèmes biométriques. L'objectif de cette thèse est d'identifier et d'en aborder plusieurs du point de vue de l'apprentissage statistique, afin de: fournir des garanties de sécurité basées sur des hypothèses probabilistes, et de proposer des solutions judicieuses aux fabricants de systèmes biométriques. À cet égard, la littérature habituelle sur l'apprentissage statistique traite de problèmes simples, tels que la classification ou la régression [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF]. Cependant, les problèmes abordés en biométrie impliquent à la fois un apprentissage sur paires et un critère fonctionnel, et nécessitent donc une analyse spécifique.

Plan du chapitre. Le chapitre présent résume les contributions des principales parties II et III de la thèse, et n'aborde pas la partie restante (Partie I). La partie I figure dans le manuscrit pour des raisons de clarté et ne contient que des préliminaires techniques aux résultats théoriques des autres parties. Le chapitre présent est organisé comme suit: premièrement, la section 12.3 couvre les idées présentées dans la brève introduction ci-dessus et donne un aperçu détaillé de la thèse. Deuxièmement, la section 12.4 se concentre sur la partie II et traite de l'ordonnancement par similarité. Troisièmement, la section 12.5 résume les contributions de la partie III sur le vaste sujet de la fiabilité des algorithmes d'apprentissage automatique. Enfin, la section 12.6 détaille les perspectives de la thèse.

La section 12.3 est une introduction développée qui présente les aspects importants de la biométrie qui sont abordés dans la littérature sur l'apprentissage automatique. Plus précisément, elle présente d'abord la relation entre la biométrie et l'apprentissage de métriques, ainsi que l'impact de l'apprentissage approfondi sur les deux domaines. Elle examine ensuite la nature des biais dans la reconnaissance faciale, et détaille les dangers potentiels des biais du point de vue de l'équité des algorithmes. La section 12.4 est un bref résumé de la partie II. Elle se concentre sur l'idée de considérer l'apprentissage de similarité comme un problème de scoring sur un espace produit. Nous appelons ce point de vue ordonnancement par similarité. À cet égard, elle commence par des garanties théoriques pour ce problème. Ensuite, elle propose des stratégies accompagnées de garanties statistiques pour réduire la complexité de calcul en ordonnancement par similarité, et finit avec plusieurs approches pratiques basées sur le gradient. La section 12.5 est un bref résumé de la partie III, qui aborde le sujet de la fiabilité en apprentissage automatique. À ce titre, elle propose d'abord une stratégie pour prévoir une liste ordonnée des classes probables à partir de données de classification multiclasse, au lieu de se concentrer uniquement sur la précision en classification. Ensuite, elle donne des stratégies pour faire face au manque de représentativité des bases de données. Elle se termine par une proposition visant à faire respecter des critères d'équité dans le problème de scoring.

La section 12.6 présente les perspectives de la thèse. Plus précisément, elle souligne l'importance d'illustrations pratiques de ce travail pour les praticiens de la biométrie, et discute plusieurs extensions possibles de nos analyses.

Les notations adoptées tout au long de la thèse sont résumées dans la table 12.1.

Défis récents en biométrie

Les progrès récents dans le domaine de l'apprentissage profond ont entraîné des changements rapides de l'état de l'art en biométrie. À cet égard, la littérature dite d'"apprentissage profond de métriques" a proposé de nombreuses pertes dérivables, mais aucune d'entre elles n'est liée à l'évaluation en termes d'ordonnancement des systèmes biométriques. Simultanément, ces avancées ont suscité un débat public sur les technologies biométriques, et en particulier sur la reconnaissance faciale. Si la plupart des questions concernent l'utilisation des algorithmes de reconnaissance faciale, l'une d'entre elles concerne leur biais racial, mesuré récemment. La communauté de recherche en apprentissage automatique peut proposer des solutions techniques pour ce problème.

N ˚" Nt0u, R `" tx P R | x ě 0u, R ˚" R `zt0u

Introduction à la biométrie

Valeur sociale. La biométrie répond à la nécessité d'établir l'identité d'une personne avec une grande confiance. Elle est devenue cruciale dans le monde moderne, car nous interagissons avec un nombre toujours croissant de personnes. Toutefois, on peut retracer ses origines à la fin du XIXe siècle, avec les premiers enregistrements de l'utilisation d'empreintes digitales pour des fins d'identification (Jain et al., 2011, Section 1.8). Aujourd'hui, la biométrie est largement utilisée dans la police scientifique et dans d'autres applications gouvernementales, ainsi que par diverses industries telles que le secteur bancaire. Un exemple d'application de la biométrie à grande échelle est le projet Aadhaar, géré par l'Unique IDentification Authority of India (UIDAI), qui a assigné des numéros d'identification nationaux et a enregistré les biométries (iris, visages et empreintes digitales) de plus d'un milliard de personnes (Jain et al., 2011, Section 1.6).

Formalisation des objectifs. L'objectif des systèmes biométriques est de comparer deux mesures px, x 1 q dans un espace d'entrée X , par exemple deux empreintes digitales ou deux visages, et de décider si les deux proviennent du même individu. Cela se fait généralement au moyen d'une fonction de similarité sur la paire s : X ˆX Ñ R `, qui quantifie la probabilité que x et x 1 proviennent de la même personne. La décision est prise en fixant un seuil de similarité, ce qui signifie que l'on considère que la paire px, x 1 q est acceptée si spx, x 1 q ą t, où t est un seuil dans R `. Il existe deux problèmes phares en matière de biométrie: la vérification et l'identification (Jain et al., 2011, Section 1.3). Le problème de vérification est également appelé authentification 1:1. Il est illustré par le cas d'utilisation du franchissement des frontières, où un officiel compare un document x 1 avec une mesure en direct x. Ainsi, il consiste à prendre une décision sur une paire x, x 1 . L'identification est illustrée par le cas d'utilisation de la surveillance automatique, où une mesure en direct x est comparée à une base de données. Précisément, il s'agit de trouver l'existence d'un élément correspondant à x dans une base de données de N P N observations D N " tx i u N i"1 Ă X . Si une telle correspondance existe, l'identification doit renvoyer les éléments pertinents dans D N . L'identification est également appelée identification 1:N ou authentification 1:N. Le nombre de personnes enrôlées N peut être important, et peut par exemple se compter en millions. Étapes opérationnelles. Pour comparer une observation x avec une grande base de données D N , il est nécessaire de comparer rapidement des représentations d'éléments dans X ayant des besoins faibles en termes de mémoire. Cela nécessite la dérivation de représentations intermédiaires efficaces des données d'entrée. Les systèmes biométriques peuvent généralement être divisés en trois processus distincts: 1) l'acquisition de données d'entrée, appelée enrôlement, 2) l'extraction de caractéristiques, parfois appelée codage des données, 3) et la comparaison des codages. Voir [START_REF] Jain | Introduction to Biometrics[END_REF], Section 1.2) pour plus de détails. Dans le contexte de la reconnaissance des empreintes digitales, la phase d'enrôlement couvre l'acquisition de la donnée brute, des étapes de post-traitement, ainsi que des vérifications de qualité sur l'image finale. L'extraction de caractéristiques consiste à appliquer des techniques de vision par ordinateur, suivie de techniques spécifiques aux empreintes pour extraire des points précis dans le l'image des empreintes digitales. Par exemple, des filtres de Gabor (Szeliski, 2011, Section 3.4.1) sont utilisés pour obtenir des ridge orientations maps (cartes d'orientation des crêtes) [START_REF] Jain | Introduction to Biometrics[END_REF] [START_REF] Turk | Face recognition using eigenfaces[END_REF], une application de l'Analyse en Composantes Principales (ACP) aux images de visage. Ensuite, cette approche a été remplacée par la combinaison de descripteurs usuels en vision par ordinateur -tels que les motifs binaires locaux (local binary patterns, LBP) et les descripteurs d'images SIFT (Scale-Invariant Feature Transformer: transformation de caractéristiques visuelles invariantes à l'échelle) -avec des techniques de réduction de la dimensionnalité. Enfin, l'extraction de caractéristiques s'appuie désormais sur une approche de bout en bout -qui effectuent la tâche finale en utilisant les données brutes -basée sur des réseaux neuronaux convolutifs profonds, un type de réseau de neurones adapté aux images [START_REF] Wang | Deep face recognition: A survey[END_REF].

Des algorithmes d'apprentissage de métriques servent à entraîner le module d'extraction de caractéristiques. Récemment, l'avènement des algorithmes d'apprentissage profond a poussé les chercheurs en biométrie à remplacer la combinaison de méthodes d'extraction de caractéristiques spécifiques à la tâche et de méthodes d'apprentissage de métriques linéaires, par un apprentissage profond de métrique de bout en bout. Les chercheurs en biométrie doivent donc suivre de près les récents développements en apprentissage profond pour rester compétitifs.

12.3.2 Apprentissage profond de métriques pour la biométrie L'apprentissage de métriques ou l'apprentissage de similarités est un problème d'apprentissage automatique dont le but est d'apprendre à quel point deux objets sont similaires. Nous renvoyons à Bellet et al. (2015a) pour une revue. En biométrie, la supervision de ces algorithmes provient d'une base de données de n images tx i u n i"1 Ă X , chaque image x i ayant une identité y i P t1, . . . , Ku avec K ď n et pn, Kq P N 2 . Apprentissage de métriques linéaires. Les premiers algorithmes d'apprentissage de métriques et une grande partie de la littérature se concentre sur l'apprentissage de métriques linéaires. L'expression se réfère aux distances ou aux fonctions de similarité s : X ˆX Ñ R `qui sont des fonctions linéaires de leurs entrées. Celles-ci reposent principalement sur l'utilisation de distances de Mahalanobis -une distance R d ˆRd Ñ R `pour d P N, paramétrée par une matrice semidéfinie positive M P R dˆd -avec quelques exceptions reposant sur d'autres combinaisons de M et de l'entrée px, x 1 q. La distance de Mahalanobis d M entre les points x et x 1 s'écrit:

d M px, x 1 q " b px ´x1 q J M px ´x1 q.
La factorisation de Cholesky (Petersen and Pedersen, 2008, Section 5.5.1) implique que l'on peut écrire M " LL J , où L est une matrice triangulaire inférieure. Ce résultat justifie le fait de considérer les distances de Mahalanobis comme le calcul d'une simple distance euclidienne sur une transformation des entrées, puisque d M px, x 1 q " Lx ´Lx 1 2 , où ¨ 2 est la distance euclidienne standard. Parmi les approches notables pour l'apprentissage des distances de Mahalanobis, on peut citer l'algorithme Mahalanobis Metric for Clustering (MMC, métrique de Mahalanobis pour le partitionnement) en 2002 [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF], Neighborhood Component Analysis (NCA, analyse des composantes du voisinage) en 2004 [START_REF] Goldberger | Neighbourhood components analysis[END_REF], et Large Margin Nearest Neighbor (LMNN, grande marge pour les plus proches voisins) en 2009 [START_REF] Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF]. Plusieurs auteurs ont envisagé diverses extensions des algorithmes d'apprentissage de métriques linéaires. Par exemple, la kernelisation des méthodes d'apprentissage de la métrique linéaire a été proposée, ainsi que l'utilisation de plusieurs métriques linéaires locales, voir Bellet et al. (2015a) (Section 5). Ces extensions se sont avérées utiles pour les praticiens de la reconnaissance faciale. Par exemple, [START_REF] Bohné | Large margin local metric learning[END_REF] considère l'application de l'algorithme MM-LMNN [START_REF] Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF], qui apprend une métrique linéaire locale, à la reconnaissance faciale.

Apprentissage profond de métriques. En raison du développement du calcul générique sur processeur graphique (GPGPU), l'apprentissage et l'utilisation de réseaux neuronaux très profonds sont devenus raisonnables. Ces réseaux ont eu pour effet d'améliorer considérablement les performances en vision par ordinateur, notamment pour la tâche de classification à grande échelle sur le défi ILSVRC (ImageNet Large Scale Visual Recognition Challenge, challenge à grande échelle de reconnaissance visuelle sur ImageNet). L'avancée la plus marquante sur ce challenge a eu lieu en 2012 et est présentée dans [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. Les résultats obtenus avec les réseaux profonds ont conduit à des avancées importantes en reconnaissance faciale, avec notamment [START_REF] Taigman | Deepface: Closing the gap to human-level performance in face verification[END_REF] en 2014. En apprentissage profond pour la reconnaissance faciale, les données brutes x sont encodées dans un vecteur epxq, où e : X Ñ R d est une fonction non linéaire qui correspond à la sortie d'un réseau de neurones. Ensuite, une simple distance est calculée entre epxq et epx 1 q pour décider si la correspondance entre x et x 1 est acceptée. L'encodage e est optimisé par une descente de gradient qui minimise une fonction de perte. Contrairement à d'autres biométries populaires telles que les empreintes digitales et l'iris, trouver manuellement les traits distinctifs importants d'un visage est difficile. Par conséquent, une approche de bout en bout basée sur l'apprentissage par descente de gradient est très bien adaptée à la reconnaissance faciale. Néanmoins, certains auteurs ont déjà proposé des approches basées sur l'apprentissage profond de métriques pour d'autres biométries [START_REF] Minaee | Fingernet: Pushing the limits of fingerprint recognition using convolutional neural network[END_REF].

Fonctions de perte pour l'apprentissage profond de métriques. L'apprentissage profond de métriques, avec ses modèles de bout en bout, a remplacé le processus composé séquentiellement: d'une sélection par le praticien de caractéristiques importantes suivie d'une étape de réduction de la dimensionnalité, comme l'illustre le très influent article de Schroff et al. (2015). Pour cette raison, de nombreux auteurs se concentrent sur la recherche de meilleures architectures de réseaux de neurones. Simultanément, l'avènement de l'apprentissage par descente de gradient a ouvert la voie à de nombreux articles les fonctions de perte. Les premiers systèmes de reconnaissance faciale utilisaient l'habituelle perte d'entropie croisée softmax (SCE, softmax cross-entropy), une perte de classification usuelle dans l'apprentissage profond qui cherche à séparer les identités (Goodfellow et al., 2016, Section 3.13). Depuis lors, de nombreuses autres fonctions de perte ont été proposées, telles que ArcFace [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF]. Nous renvoyons à Wang and Deng (2018) (Figure 5) pour un aperçu des pertes pour la reconnaissance faciale. Leur objectif est soit: d'augmenter la marge entre les identités pour diminuer la variance inter-classe, regrouper toutes les observations de chaque identité pour diminuer la variance intra-classe, ou combiner les deux approches comme le fait la perte par triplets (triplet loss) [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. Les praticiens ont observé que le fait d'additionner différentes pertes, tout en ajustant la proportion de chaque perte était nécessaire pour optimiser les performances [START_REF] Parkhi | Deep face recognition[END_REF]. Évaluation basée sur l'ordonnancement. La performance des systèmes de reconnaissance faciale est mesurée sur la courbe ROC, comme le montre les évaluations de systèmes commerciaux de reconnaissance faciale par [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. Ces évaluations ont été menées par le National Institute of Standards and Technology (NIST, Institut national des normes et de la technologie) une agence du département du Commerce des États-Unis. La courbe ROC est le standard dans l'évaluation des fonctions de score en ordonnancement bipartite, un problème qui vise à attribuer des scores plus élevés aux éléments associés à un label positif `1 qu'aux éléments avec un label négatif ´1. Nous renvoyons à [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF] pour une revue sur l'ordonnancement bipartite. Dans ce contexte, les fonctions de similarité peuvent être considérées comme des fonctions de score sur un espace produit. Cette observation suggère que l'utilisation du vaste corpus de recherche sur l'ordonnancement bipartite est une approche justifiée pour proposer de meilleures fonctions de perte pour la reconnaissance faciale, ce que nous faisons à la fois dans la partie II, et dans le chapitre 8 contenu dans la partie III.

Récemment, des améliorations rapides en termes de précision ont été observées en reconnaissance faciale, qui ne requiert pas que l'individu soit coopératif. Ainsi, la technologie a récemment attiré l'attention des médias et de l'opinion publique. En plus des préoccupations usuelles en matière de protection de la vie privée, les observateurs ont exprimé des inquiétudes croissantes concernant le manque de fiabilité ou l'injustice éventuelle induite par la reconnaissance faciale.

Fiabilité en biométrie

Les récents progrès en matière de reconnaissance faciale ont confirmé la maturation de la technologie, ce qui préfigure son déploiement et a provoqué un large débat à ce sujet. [START_REF] Gates | Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance[END_REF] a mis en garde contre la tendance du public à extrapoler sur l'omniprésence de ces systèmes, ce qui crée une l'illusion d'une surveillance qui modifie les comportements. Cependant, les obstacles techniques énoncés par [START_REF] Gates | Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance[END_REF] semblent beaucoup plus faibles aujourd'hui. Précisément, des travaux tels que [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] montrent la capacité des systèmes de reconnaissance faciale dans des conditions d'acquisition très peu contrôlées. De plus, certains observateurs ont prévu un taux de croissance annuel composé (TCAC) -c'est-à-dire une progression géométrique moyenne -de 14,5 % par an entre 2020 et 2027 pour le marché mondial de la reconnaissance faciale. Dans ce contexte, les questions relatives au déploiement de la reconnaissance faciale et des autres technologies d'apprentissage automatique appartiennent au domaine du législateur, mais les décisions des modèles sont de la responsabilité du praticien en apprentissage automatique.

Biais dans la reconnaissance faciale. Dans le cas spécifique de la reconnaissance faciale, le NIST a quantifié avec précision une moindre performance pour la reconnaissance de personnes de couleur, avec les caucasiens comme classe de référence. La figure 12.1 illustre leurs travaux et provient de [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. Cette observation a été reprise par de nombreux médias et a été qualifiée de "biais racial" en 2019. La principale justification pour cet écart fut que les bases de données utilisées pour l'entraînement de ces systèmes sont généralement composés de personnalités européennes ou d'Amérique du Nord, qui sont pour la plupart des caucasiens et ne représentent pas la population générale. Les auteurs ont interprété cette observation comme un "other race effect" (littéralement, effet d'autre ethnicité) pour la reconnaissance faciale automatique, une idée introduite dans [START_REF] Furl | Face recognition algorithms and the other-race effect: Computational mechanisms for a developmental contact hypothesis[END_REF][START_REF] Petersen | An other-race effect for face recognition algorithms[END_REF], qui dit que les humains ont généralement du mal à distinguer les individus d'une ethnie différente de la leur. Certains auteurs ont proposé des stratégies pour corriger ce problème explicitement [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF]. Plus généralement, une large littérature sur l'équité en apprentissage automatique peut être invoquée pour aborder ce problème [START_REF] Bolukbasi | Man is to computer programmer as woman is to homemaker? debiasing word embeddings[END_REF][START_REF] Zhao | Men also like shopping: Reducing gender bias amplification using corpus-level constraints[END_REF][START_REF] Hendricks | Women also snowboard: Overcoming bias in captioning models[END_REF][START_REF] Liu | Transfer learning by sample selection bias correction and its application in communication specific emitter identification[END_REF][START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF]. Le chapitre 9 dans la partie III contribue à cet effort, en proposant une méthode générale de repondération qui s'applique aux problèmes de représentativité en biométrie.

Limites de la représentativité des bases de données. Bien qu'il soit important de disposer d'une base de données représentative de la population cible, on ne peut pas s'attendre à ce que cela corrige les biais inhérents aux données d'entraînement. Précisément, même si un groupe social identifié par des attributs protégés tels que la race ou la religion est en moyenne beaucoup plus pauvre qu'un autre groupe social, il peut être considéré comme immoral de refuser un prêt au motif qu'un demandeur appartient au premier groupe. Dans ce contexte, certains observateurs ont brutalement qualifié les algorithmes prédictifs d'"opinions intégrées dans des mathématiques" (O 'Neil, 2016).

Equité algorithmique. Un grand nombre de travaux [START_REF] Agarwal | A reductions approach to fair classification[END_REF][START_REF] Woodworth | Learning non-discriminatory predictors[END_REF]Zafar et al., 2017a[START_REF] Zafar | Fairness constraints: A flexible approach for fair classification[END_REF][START_REF] Menon | The cost of fairness in binary classification[END_REF][START_REF] Bechavod | Learning fair classifiers: A regularization-inspired approach[END_REF] sont apparus sur le thème de l'équité en l'apprentissage automatique -également appelée équité algorithmique -qui est un nouveau domaine d'étude. Ceux-ci cherchent à ajouter des contraintes explicites durant la phase d'entraînement afin que des optimisations brutales de la précision ne conduisent pas à la reproduction systémique d'injustices sociales. Les premiers travaux influents remontent à 2012 [START_REF] Dwork | Fairness through awareness[END_REF]. Plus récemment, des auteurs ont travaillé sur un manuel consacré à ce sujet [START_REF] Barocas | Fairness and Machine Learning[END_REF].

Equité en reconnaissance faciale. Dans la reconnaissance faciale, l'incorporation de contraintes d'équité lors de l'apprentissage des modèles peut corriger le fait que certains groupes sociaux sont plus difficiles à identifier que d'autres. C'est une nécessité dans de nombreux cas pratiques. Par exemple, si un système conçu pour signaler des personnes d'intérêt a un taux plus élevé de fausse acceptance pour une ethnicité spécifique, cela peut être interprété comme du profilage racial automatique.

Littérature scientifique sur l'équité. La littérature scientifique sur l'équité pour le problème de classification est vaste, mais il existe peu de travaux pour des contextes spécifiques, tels que l'ordonnancement ou l'apprentissage de similarités. Parmi les exceptions notables, citons [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF] pour l'ordonnancement, qui utilise une étape de post-traitement pour modifier une fonction de score afin de satisfaire un critère d'équité. Cela laisse entrevoir la possibilité de nouvelles approches pour l'équité spécifiquement adaptées à ces problèmes importants, ce que nous proposons dans le chapitre 10 contenu dans la partie III.

Dans son ensemble, la thèse participe au dialogue entre la communauté de l'apprentissage automatique et celle de la biométrie. Elle propose une vision stylisée et théorique de challenges en biométrie, qui s'appuie sur la littérature récente pour étudier les critères spécifiques -c'est-à-dire fonctionnels et basés sur les paires -qui sont abordés en biométrie. Nous espérons que notre point de vue sur les problèmes biométriques aura des répercussions bénéfiques sur la pratique. De plus, la dérivation de garanties statistiques pour les systèmes biométriques constitue un outil important pour s'assurer de leur sécurité.

Plan de la thèse

La thèse est divisée en trois parties. La partie I de la thèse contient des préliminaires techniques, qui fournissent tous les résultats intermédiaires nécessaires pour prouver les contributions théoriques de la thèse. Elle est comprise dans la thèse pour des raisons de clarté. La partie II et III se concentrent sur nos contributions. La partie II se penche sur l'idée de considérer l'apprentissage de similarité comme un problème de scoring sur un espace produit. La partie III est centrée autour de l'idée générale de la fiabilité des algorithmes d'apprentissage automatique.

La partie I est divisé en trois chapitres. Le premier chapitre (Chapitre 2) est une introduction rapide à la théorie de l'apprentissage statistique. Il présente les résultats nécessaires pour obtenir des garanties de généralisation dans le cas facile de la classification binaire. Il détaille précisément la dérivation de bornes à nombre de données fini sur l'excès de risque du minimiseur empirique. La plupart de nos contributions théoriques peuvent être interprétées comme des extensions de ces résultats, mais nos contributions concernent des problèmes plus complexes. Le deuxième chapitre (Chapitre 3) traite de tous les résultats requis qui concernent l'idée d'ordonnancement en apprentissage automatique. Nos contributions s'appuie sur deux thèmes liés à l'ordonnancement: l'ordonnancement bipartite et l'agrégation d'ordonnancements. En effet, notre travail étend les garanties existantes en ordonnancement bipartite à l'ordonnancement par similarité présenté dans la partie II. L'agrégation d'ordonnancement est utilisée pour le bagging (bootstrap aggregating, une méthode pour l'aggrégation de modèles statistiques) d'arbres d'ordonnancement, et les garanties du premier chapitre de la partie III sont construites sur un modèle paramétrique pour les ordonnancements, qui peut être considéré comme un outil pour l'agrégation d'ordonnancements. Enfin, le troisième et dernier chapitre (Chapitre 4) présente une brève introduction à d'importants résultats sur les U -statistiques, une type de statistique présente dans tous les problèmes d'apprentissage par paires. À ce titre, le chapitre 4 est un pré-requis à nos garanties pour l'ordonnancement par similarité, et intervient lors de l'estimation de fonctions de risque dans le cadre de l'ordonnancement bipartite standard.

La partie II explore l'idée de considérer le problème de vérification biométrique comme l'ordonnancement de paires d'instances. Il est divisé en trois chapitres. Le premier chapitre (Chapitre 5) présente formellement l'idée de voir l'apprentissage de similarité du point de vue de l'optimisation de la courbe ROC. Il propose de nouvelles garanties pour le problème de l'optimisation de la courbe ROC en un point, qui cherche à optimiser le taux de vrais positifs avec une limite supérieure sur le taux de faux positifs. Cette analyse ouvre la voie à une extension des garanties des garanties de l'algorithme TreeRank à l'apprentissage de similarité, que nous fournissons. À l'aide de simulations numériques, nous produisons la première illustration empirique des vitesses d'apprentissage rapide, adaptées ici au cas spécifique de l'optimisation de la courbe ROC en un point pour l'ordonnancement par similarité. En raison du nombre prohibitif de paires impliquées dans les calculs, les propositions du premier chapitre ne sont pas raisonnables pour une application à grande échelle. Pour y remédier, les statisticiens ont proposé des approximations par échantillonnage pour les U -statistiques, une approche utile pour l'ordonnancement par similarité. Le deuxième chapitre (Chapitre 6) étend cette proposition aux contextes où les données sont dans un environnement distribué. Enfin, le troisième chapitre (Chapitre 7) est plutôt prospectif et propose des expériences numériques simples sur l'ordonnancement par similarité, qui abordent la question de l'optimisation pour ce problème. Étendre ces expériences fera l'objet de travaux futurs.

La partie III est la dernière partie de la thèse. Elle s'articule autour de l'idée de fiabilité en apprentissage automatique et est également divisée en trois chapitres. Le premier chapitre (Chapitre 8) donne des garanties d'apprentissage pour la prédiction d'un ordonnancement sur les classes possibles en utilisant seulement des données de classification multiclasse, ce qui repose sur l'utilisation d'une stratégie OVO (One-Versus-One, un-contre-un). Ce problème se pose souvent dans des problèmes bruités ou incertains, c'est-à-dire pour lesquels la classe la plus probable a de bonnes chances d'être un faux positif. Dans ce contexte, on s'intéresse alors aux classes les plus probables, comme c'est souvent le cas dans les enquêtes criminelles. Le deuxième chapitre (Chapitre 9) propose des techniques pour corriger le biais observé entre un échantillon d'entraînement et celui de test, en utilisant des informations auxiliaires concernant la différence entre les deux. Il repose sur l'application du principe bien connu d'échantillonnage préférentiel en statistique. Enfin, le troisième chapitre (Chapitre 10) propose une unification d'une classe de contraintes d'équité, ainsi qu'une nouvelle contrainte d'équité, plus restrictive et plus adaptée aux situations concrètes. Il comporte également des garanties théoriques, ainsi que des approches pratiques basées sur le gradient pour apprendre sous les deux types de contraintes.

Le dernier chapitre de la thèse (Chapitre 11) contient un résumé des contributions de la partie II et III, ainsi qu'un compte rendu détaillé des orientations les plus prometteuses pour de futurs travaux. Il se termine par une conclusion générale sur la thèse.

Les deux sections suivantes de ce chapitre -les sections 12.4 et 12.5 -résument les contributions de la thèse. Chaque section se concentre respectivement sur la première et la deuxième partie de la thèse, et est divisée en sous-sections qui résument chaque chapitre de la partie. Enfin, la section 12.6 résume les perspectives de la thèse.

12.4 Ordonnancement par similarité L'apprentissage de similarité joue un rôle clé dans de nombreux problèmes d'apprentissage automatique comme le partitionnement, la classification ou la réduction de la dimensionnalité. Cette technique est particulièrement importante lorsqu'on considère des problèmes dits "en monde ouvert" -c'est-à-dire pour lesquels un modèle rencontre des classes qui n'étaient pas disponibles pendant l'entraînement au moment du déploiement [START_REF] Chen | Lifelong Machine Learning[END_REF] -ce qui est le cas pour toute application biométrique. Dans cette section, nous considérons l'apprentissage de métrique du point de vue du scoring de paires d'instances, ce qui est cohérent avec l'évaluation 12.4. Ordonnancement par similarité de nombreux systèmes basés sur des techniques d'apprentissage de métrique.

12.4.1 Théorie de l'ordonnancement par similarité L'ordonnancement/scoring bipartite considère un ensemble d'éléments associés à un label binaire, et cherche à classer ceux qui ont le label `1 plus haut que ceux qui ont le label ´1. Pour obtenir un ordre sur un espace d'entrée X , l'ordonnancement bipartite repose généralement sur l'apprentissage d'une fonction de score s : X Ñ R [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF]. D'autre part, l'apprentissage de métrique/similarité (Bellet et al., 2015a) est la tâche d'apprendre une similarité -ou de manière équivalente, une distance -s : X ˆX Ñ R sur l'espace produit X ˆX . Alors que les algorithmes d'apprentissage de métrique ont été évalués à l'origine par rapport à leur pertinence pour une tâche de partitionnement [START_REF] Xing | Distance metric learning with application to clustering with side-information[END_REF], les praticiens utilisent aujourd'hui des indicateurs de performance dérivés de la courbe ROC, la mesure standard pour l'évaluation des fonctions de score en ordonnancement bipartite. Par conséquent, notre travail introduit sous le nom d'ordonnancement par similarité l'idée d'apprendre des similarités pour un objectif d'ordonnancement.

Un critère fonctionnel: la courbe ROC. Dans le cadre de la classification multiclasse, nous introduisons une paire aléatoire pX, Y q P X ˆt1, . . . , Ku, avec K P N le nombre de classes, ainsi qu'une copie indépendante pX 1 , Y 1 q de pX, Y q. Ensuite, nous pouvons définir une variable Z " 2 ¨ItY " Y 1 u ´1 égale à 1 si les deux paires appartiennent à la même classe et à ´1 sinon. La courbe ROC d'une fonction de similarité est alors égale au PP-plot (Probability-Probability plot, représentation utilisée pour comparer deux distributions réelles)

t P R Þ Ñ p s H s ptq, s G s ptqq, où, pour tout t P R: s H s ptq :" PtspX, X 1 q ą t | Z " ´1u et s G s ptq :" PtspX, X 1 q ą t | Z " `1u.
s H s ptq et s G s ptq sont respectivement le taux de faux positifs et de vrais positifs associés à la similarité s. Sous des hypothèses de continuité, la courbe ROC s'écrit comme le tracé de la fonction α P p0, 1q Þ Ñ ROC s pαq " s G s ptq ˝s H ´1 s pαq. Certaines approches en apprentissage de similarité optimisent une version empirique de l'aire sous la courbe ROC (AUC, Area Under the Curve) de la fonction de similarité s [START_REF] Mcfee | Metric learning to rank[END_REF][START_REF] Huo | Cross-modal metric learning for auc optimization[END_REF].

Optimisation en un point de la courbe ROC (pROC). L'AUC est un résumé global de la courbe ROC qui pénalise les erreurs d'ordonnancement, quelle que soit la position des instances concernées dans la liste (Clémençon et al., 2008, Proposition B.2). D'autres critères se concentrent sur le haut de la liste [START_REF] Clémençon | Ranking the best instances[END_REF][START_REF] Huo | Cross-modal metric learning for auc optimization[END_REF], et leur étude est l'objet de la littérature sur l'ordonnancement des meilleures instances (ranking the best instances) (Menon and Williamson, 2016, Section 9). Dans notre travail, nous envisageons d'optimiser le taux de vrais positifs atteint par une similarité sous contrainte d'une limite supérieure α P p0, 1q sur son taux de faux positifs. Ce problème est pertinent dans les applications biométriques, car les garanties de sécurité sont généralement spécifiées par une borne supérieure sur un taux de faux positifs acceptable pour un système. Nous appelons ce problème pointwise ROC optimization (pROC). Nous considérons des risques:

R ´psq :" ErspX, X 1 q | Z " ´1s et R `psq :" ErspX, X 1 q | Z " `1s,
avec S une famille de fonctions de similarités candidates, le problème pROC s'écrit:

max sPS R `psq avec R ´psq ď α. (12.1)
Nous définissons s ˚comme la solution de Eq. (12.1). Notons que Clémençon and Vayatis (2010) ont étudié l'équivalent de Eq. (12.1) dans le cas l'ordonnancement bipartite. Ce problème est analogue à la classification de Neyman-Pearson [START_REF] Scott | A Neyman-Pearson approach to statistical learning[END_REF], et ressemble beaucoup au problème d'apprentissage d'ensembles de volume minimum [START_REF] Scott | Learning minimum volume sets[END_REF]. Lorsque S est la classe de toutes les fonctions mesurables, la solution de Eq. (12.1) s'écrit comme un ensemble de sur-niveau de la probabilité a posteriori η : x, x 1 Þ Ñ PtpX, X 1 q " px, x 1 q | Y " Y 1 u, ce qui est une conséquence du lemme fondamental de Neyman-Pearson [START_REF] Lehmann | Testing Statistical Hypotheses[END_REF], Théorème 3.2.1).

Estimateurs sur paires. L'analyse de Clémençon and Vayatis (2010) repose sur le fait que les estimateurs naturels de R ´psq et de R `psq sont des moyennes empiriques usuelles dans le cas de l'ordonnancement bipartite. Cependant, ce n'est pas le cas pour l'ordonnancement par similarité. Considérons un échantillon D n " tpX i , Y i qu n i"1 composé de n copies i.i.d. de la paire pX, Y q, ainsi que les estimateurs naturels de R ´psq et R `psq basés sur l'échantillon D n : R ń psq :" 1 n ´ÿ iăj ItY i ‰ Y j u ¨spX i , X j q, (12.2)

R ǹ psq :" 1 n `ÿ iăj ItY i " Y j u ¨spX i , X j q, (12.3) où n `:" ř iăj ItY i " Y j u et n
´": npn ´1q{2 ´n`. Les quantités de Eq. (12.2) et Eq. ( 12.3) ne sont pas des sommes de variables aléatoires indépendantes, et l'analyse de Clémençon and Vayatis (2010) ne s'applique alors pas à ce cas. Cependant, il s'agit de ratios de statistiques bien connues, les U -statistiques [START_REF] Lee | U -statistics: Theory and practice[END_REF][START_REF] De La Pena | Decoupling: from Dependence to Independence[END_REF].

Garanties de généralisation pour pROC. La version empirique de pROC (Eq. (12.1)) s'écrit: (12.4) où φ ě 0 est un terme qui tolère les variations de R ń psq autour de son espérance R ´psq. Nous écrivons s n la solution de Eq. (12.4). La généralisation d'inégalités de concentration standard aux U -statistiques nous permet d'étendre directement les garanties uniformes de Clémençon and Vayatis (2010). Précisément, nous garantissons simultanément qu'avec une forte probabilité: R `ps ˚q´R `ps n q est limité par une quantité d'ordre n ´1{2 et R ´ps n q ď α`φ n avec φ n " Opn ´1{2 q.

max sPS p R ǹ psq avec R ń psq ď α `φ,
En résumé, nous montrons que l'excès de risque est borné par la vitesse d'apprentissage usuelle en n ´1{2 en l'absence d'hypothèses sur la distribution des données.

Vitesses de généralisation rapide pour pROC. Dans le cas de la classification binaire et sous une hypothèse de bruit paramétrée par a P p0, 1q sur la distribution des données, [START_REF] Mammen | Asympotical minimax recovery of the sets with smooth boundaries[END_REF] ont montré qu'on obtient une vitesse d'apprentissage rapide en Opn ´1{p2´aq q. La vitesse rapide est la conséquence d'une limite supérieure sur la variance de l'excès de risque, elle-même dérivée de l'hypothèse du bruit. L'analyse de Clémençon and Vayatis (2010) s'est appuyée sur ces idées pour proposer une limite supérieure sur R `ps ˚q ´R`p s n q en Opn ´p2`aq{4 q avec des garanties en Opn ´1{2 q pour R ´, pour pROC en ordonnancement bipartite. Par rapport à la classification binaire, le problème pROC a une vitesse d'apprentissage plus faible, issue de la nature bilatérale de pROC. Notre travail étend les vitesses rapides de Clémençon and Vayatis (2010) pour l'ordonnancement bipartite au cas de l'ordonnancement par similarité. Incidemment, le résultat est vrai sous des hypothèses beaucoup plus faibles. Précisément, il repose sur la seconde décomposition de Hoeffding pour les U -statistiques [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF], qui implique que la variance de l'excès de risque consiste essentiellement en la variance de sa projection de Hájek. Limites de pROC. Alors que le problème de pROC fait écho à des considérations pratiques en biométrie, où les systèmes sont déployés pour fonctionner à un taux fixe de faux positifs α, sa résolution empirique est difficile en pratique. Les rares exceptions reposent sur un partitionnement fixé de l'espace d'entrée [START_REF] Scott | Learning minimum volume sets[END_REF]. Dans de nombreuses situations, le taux de faux positifs α pour un système est inconnu à l'avance, et l'optimisation pour un mauvais α peut ne pas donner de résultats satisfaisants au déploiement.

TreeRank pour l'ordonnancement bipartite. L'algorithme TreeRank pour l'ordonnancement bipartite a été introduit dans Clémençon and Vayatis (2009). TreeRank pour l'ordonnancement par similarité. Notre travail propose une extension de l'algorithme TreeRank pour l'apprentissage de similarité, en considérant des séparations récursives de l'espace produit X ˆX . Afin de s'assurer que la similarité s est symétrique, nous considérons seulement des séparations symétriques par rapport aux deux arguments de l'espace d'entrée X Ă R d , en séparant l'espace sur la simple reparamétrisation suivante de X ˆX :

f :px, x 1 q Þ Ñ ˆ|x ´x1 | x `x.
˙.

En utilisant les mêmes extensions des inégalités de concentration classiques aux cas des Ustatistiques qu'auparavant, nous avons étendu la preuve de Eq. (12.5) à l'ordonnancement par similarité. Notre analyse fournit une approche fondée théoriquement pour l'apprentissage de similarités qui s'approchent de la courbe ROC optimale en norme sup.

Nous ayons prouvé des garanties théoriques pour nos approches d'ordonnancement par similarité, mais les estimateurs des fonctions de risque nécessitent le calcul de sommes comprenant un très grand nombres de termes. Ce coût de calcul rend l'application pratique de ces approches inabordable. Par exemple, le calcul de R ń psq nécessite la somme de n ´termes, une quantité quadratique en n quand K est constant. Dans une application biométrique typique, le nombre d'échantillons par classe est fixe. Ainsi, la proportion de paires négatives n ´sur l'ensemble des paires n 2 est encore plus élevée que dans le cas K constant. La section suivante exploite des analyses récentes concernant l'approximation de U -statistiques afin d'atténuer ce problème.

U -statistiques distribuées

La plupart des applications biométriques nécessitent un apprentissage sur de très grands volumes de données. En reconnaissance faciale, la plus grande base de données mise à la disposition du public contient 8,2 millions d'images [START_REF] Guo | Ms-celeb-1m: A dataset and benchmark for large-scale face recognition[END_REF] et certaines bases de données privées sont bien plus grandes. Cela illustre les problèmes d'échelle présentés à la Section 12.4.1, car le nombre de paires négatives est supérieur à 50 trillions (10 12 ) pour [START_REF] Guo | Ms-celeb-1m: A dataset and benchmark for large-scale face recognition[END_REF]. Outre une limite en termes de nombre d'opérations, ces ensembles de données ne peuvent souvent pas être contenus dans la mémoire vive (RAM) d'une seule machine. Dans notre travail, nous avons proposé une approche pour l'estimation de U -statistiques dans un environnement distribué, qui concerne ces deux limites pratiques simultanément.

U -statistiques incomplètes. L'idée d'alléger la complexité de calcul des statistiques U n'est pas nouvelle, étant donné que [START_REF] Blom | Some properties of incomplete U-statistics[END_REF] proposait déjà l'utilisation un petit échantillon de B paires sélectionnées par tirage aléatoire avec remise dans l'ensemble de toutes les paires pour former des U -statistiques incomplètes en 1976. [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U -statistics[END_REF] propose une borne supérieure, vraie avec une grande probabilité, sur l'écart entre une U -statistique incomplète U B et la statistique complète U n . Dans le cas d'une U -statistique de degré deux à un échantillon -c'est-à-dire une moyenne sur toutes les paires pouvant être formées avec un échantillon -la borne implique qu'un estimateur U B utilisant seulement B " n paires suffit pour retrouver la vitesse d'apprentissage usuelle en n ´1{2 , plutôt que d'additionner toutes les npn ´1q{2 paires pour calculer U n . Ce résultat implique que l'on peut étendre les résultats présentés ci-dessus pour l'ordonnancement par similarité, afin de travailler avec des U -statistiques incomplètes, ce qui rend le problème raisonnable pour tout contexte où l'on a de grands volumes de données.

Environnements distribués. Dans les cas où les données ne tiennent pas sur une seule machine, les récents progrès technologiques en matière de bases de données distribuées et le calcul parallèle ont rendu accessible le déploiement d'algorithmes d'apprentissage automatique dans un environnement distribué, en grande partie grâce au développement de frameworks pour le calcul sur grappes de serveur, comme Apache Spark [START_REF] Zaharia | Spark: Cluster computing with working sets[END_REF] ou Petuum [START_REF] Xing | Petuum: A New Platform for Distributed Machine Learning on Big Data[END_REF] Repartitionnement des données pour les estimateurs distribués. Cette différence de variance entre les estimateurs distribués et U n vient du fait que de nombreuses paires impliquées dans le calcul du second ne sont pas impliquées dans les calculs des premiers. Pour contrebalancer cet effet, nous proposons de faire la moyenne d'estimateurs calculés entre des procédures de repartitionnement des données, qui réattribuent les observations à chacun des groupes aléatoirement, afin que chaque paire impliquée dans U n ait une chance d'être vue. Nous proposons deux estimateurs avec repartitionnement des données: U n,N,T (resp. U n,N,B,T ) qui fait la moyenne de T estimateurs U n,N (resp. U n,N,B ) calculé sur la base de T partitionnements différents des données. À mesure que T augmente, la variance de ces estimateurs se rapproche de la variance de U n par le haut.

Variance relative de nos estimations. Nous fournissons des expressions analytiques pour la variance des estimateurs U n , U B , U n,N , U n,N,B , U n,N,T et U n,N,B,T , pour plusieurs façons de répartir les données sur les serveurs. À cet égard, nous considérons tout d'abord le tirage sans remplacement (SWOR), qui est pertinent lorsque l'on divise toutes les données sur plusieurs machines pour des raisons de contraintes d'espace. Ensuite, nous considérons avec le tirage avec remplacement (SWR), qui est pertinent lors de la sélection d'ensembles de données sur lesquels calculer plusieurs estimations en parallèle, par exemple lors d'une descente de gradient sur des batchs (petits sous-échantillons) de données. Nous supposons pour ces deux types de tirage que chaque serveur contient n{N éléments de l'échantillon D n et m{N éléments de Q m , un réglage que nous appelons prop. L'assouplissement de cette hypothèse implique qu'il y a une probabilité non nulle de ne pas avoir d'éléments de D n ou Q m sur un serveur. Dans ce cas particulier, il faut fournir une valeur par défaut pour l'estimateur. Cela implique que la variance n'a pas de forme analytique simple et interprétable, donc nous fournissons des preuves empiriques que les variances observées sont du même ordre. De plus, nous caractérisons les paramètres h, n, m et les distributions de X 1 , Z 1 pour lesquelles la procédure de répartition est importante.

Apprentissage avec des U -statistiques distribuées. [START_REF] Papa | SGD algorithms based on incomplete u-statistics: Largescale minimization of empirical risk[END_REF] a étudié la descente de gradient stochastique pour les U -statistiques. Bien que nous n'étendons pas leur analyse à nos estimateurs distribués, notre travail considère la minimisation de U -statistiques avec des méthodes de gradient stochastique dans un environnement distribué, en estimant un gradient avec U n,N,B et en repartitionnant les données tous les n r itérations. Nous fournissons des preuves empiriques que la réduction de n r donne une meilleure solution au processus d'optimisation en moyenne. En outre, la variance de la perte de la solution finale est beaucoup plus faible, ce qui montre l'augmentation la robustesse obtenue en repartitionnant les données.

Alors que la section 12.4.1 et cette section rendent compte respectivement de l'aspect de généralisation et de scalabilité pour l'ordonnancement par similarité, la section suivante se concentre sur les questions d'optimisation. Ainsi, la section suivante est plus prospective et donne des stratégies d'optimisation pour le problème d'ordonnancement par similarité sur des exemples-jouets.

Ordonnancement par similarité en pratique

Le développement de la littérature sur l'apprentissage profond de métriques est motivée par un besoin de critères qui correspondent aux exigences de l'identification biométrique, et peuvent être optimisés par descente de gradient. La plupart de ces critères sont basés sur une meilleure séparation des identités avec une heuristique, telle que le fait que toutes les instances associées à une même identité doivent correspondre au même point dans un espace de représentation, voir la center loss [START_REF] Wen | A discriminative feature learning approach for deep face recognition[END_REF]. Bien que ces critères soient raisonnables, ils sont mal reliés à l'évaluation des systèmes biométriques, qui est basée sur la courbe ROC. La courbe ROC est une fonction conçue pour évaluer la capacité d'une fonction de score à distinguer les éléments positifs des négatifs, ou, en biométrie, la capacité d'une fonction de similarité à distinguer les paires correspondantes des paires non correspondantes. Le lien entre l'ordonnancement bipartite et la biométrie nous motive à proposer des approches pratiques pour l'apprentissage de fonctions de similarité qui optimisent une mesure dérivée de la courbe ROC.

Analyses préliminaires de l'ordonnancement par similarité. Considérer l'apprentissage de similarité comme l'ordonnancement bipartite sur les paires est discuté en profondeur dans d'autres parties de la thèse sous le nom d'ordonnancement par similarité. En effet, nous avons fourni des résultats concernant la généralisation de l'optimisation en un point de la courbe ROC (pROC), parfois désigné sous le nom de classification de Neyman-Pearson [START_REF] Scott | Learning minimum volume sets[END_REF]. Le problème pROC consiste à trouver un score s et un seuil t, qui donnent le taux de vrais positif le plus élevé possible tout en satisfaisant une limite supérieure sur le taux de faux positifs. De plus, nous avons fourni une extension de l'analyse et de la méthodologie associées à l'algorithme TreeRank à l'apprentissage de fonctions de similarité, pour lesquelles (Clémençon and Vayatis, 2009) avait prouvé initialement qu'il permettait d'apprendre une fonction de score s qui se rapproche de la courbe ROC optimale en norme sup. Enfin, nous avons motivé théoriquement l'utilisation de telles méthodes, car nous prouvons que les estimateurs gourmands en calcul impliqués peuvent être remplacés par des U -statistiques incomplètes, ce qui corrige les limites computationnelles associées à l'ordonnancement par similarité.

Limites des analyses précédentes. Nos résultats théoriques justifient d'envisager l'optimisation du problème pROC ou l'utilisation de l'algorithme TreeRank. Cependant, elles n'impliquent pas d'approches évidentes pour optimiser le problème pROC, ni ne garantissent la performance empirique de notre extension de l'algorithme TreeRank. Ce chapitre de la thèse fournit des illustrations d'approches possibles, dont nous montrons le fonctionnement sur des données synthétiques simples. L'évaluation empirique approfondie de ces approches est une direction intéressante pour de futurs travaux.

Résoudre pROC pour des similarités linéaires. Bien que de nombreux articles traitent du problème pROC [START_REF] Scott | Learning minimum volume sets[END_REF]Clémençon and Vayatis, 2010;[START_REF] Rigollet | Neyman-pearson classification, convexity and stochastic constraints[END_REF], il subsiste un manque d'approches pratiques pour celui-ci, à l'exception de quelques propositions basées sur du partitionnement récursif, voir par exemple [START_REF] Scott | Learning minimum volume sets[END_REF]. Soit un échantillon D n :" tpx i , y i qu n i"1 Ă X ˆt1, . . . , Ku de n points de données, où y i est l'identité d'une observation, x i P X et X Ă R d . Dans le cas de l'ordonnancement par similarité, pROC au niveau α P p0, 1q pour une classe de fonctions candidates S, avec s : X ˆX Ñ R pour tout s P S, s'écrit:

max sPS 1 n `ÿ iăj
Ity i " y j u ¨spx i , x j q avec 1 n ´ÿ iăj Ity i ‰ y j u ¨spx i , x j q ď α.

(12.8)

Dans le cas général, Eq. (12.8) peut être très difficile à résoudre. Par exemple, si S est composé d'indicateurs d'ensembles dans X ˆX , Eq. (12.8) peut ne pas être différentiable, ni continu. Cependant, si la famille S a une forme particulière, la résolution d'Eq. (12.8) peut être beaucoup plus facile. Par exemple, nous proposons une solution analytique lorsque S est l'ensemble de toutes les similarités bilinéaires bornées px, x 1 q Þ Ñ x J Ax avec A P R dˆd et A F ď 1, où ¨ F est la norme de Frobenius. Bien que nous montrions que cette approche donne des solutions raisonnables pour l'optimisation de la courbe ROC en un point pour des distributions très spécifiques, le cas général nécessite des familles de fonctions plus flexibles.

Résoudre pROC par descente de gradient. Pour traiter pROC avec des familles de fonctions candidates plus complexes, nous proposons une approche basée sur la descente de gradient pour minimiser un analogue de l'excès d'erreur pour l'estimation de l'ensemble de volume minimum de [START_REF] Scott | Learning minimum volume sets[END_REF], adapté à cet effet pour l'ordonnancement bipartite. Bien que nous démontrions son efficacité avec un simple classifieur linéaire et un exemple-jouet, l'approche est suffisamment souple pour s'adapter à des modèles plus complexes, ainsi qu'à une extension à l'ordonnancement par similarité. Toutefois, ses performances devront être démontrées empiriquement sur des exemples plus éloquents. Soit H et G les distributions de X | Y " ´1 et X | Y " `1 respectivement, ainsi que R α la région de rejet optimale pour l'optimisation de la courbe ROC en point, alors notre relaxation de l'excès d'erreur écrit: max pw,bqPR d ˆR ´GpR αq ´p Gpw, bq ¯``´p Hpw, bq ´α¯`t el que w 2 2

`b2 ď 1. (12.9) où p G et p H sont des versions empiriques relaxées de G et H respectivement, et pxq `:" maxpx, 0q pour tout x P R. En minimisant la fonction objectif dans Eq. (12.9) et en projetant les poids sur la balle l'unité, notre méthode retrouve les bonnes régions de rejet par descente de gradient, pour notre exemple-jouet. Bien que la quantité GpR αq soit inconnue, nous supposons que les résultats ne sont pas extrêmement sensibles à cette valeur, et que des approximations raisonnables peuvent être proposées dans la plupart des applications. Le problème pROC est une approche raisonnable pour apprendre une similarité, mais il n'aborde pas la question de l'ordonnancement comme un problème global sur X . Précisément, il se concentre sur la récupération d'un seul ensemble de niveau de la fonction de score, comme le démontre le lemme fondamental de Neyman-Pearson, alors que l'ordonnancement bipartite consiste à récupérer une relation d'ordre entre deux points quelconques de l'espace d'entrée.

TreeRank pour l'apprentissage de similarité. L'algorithme TreeRank de Clémençon and Vayatis (2009) traite l'ordonnancement bipartite comme un problème global. Précisément, il aborde celui-ci par une procédure de séparation récursive, qui résout des problèmes de classification binaire avec des erreurs pondérées asymétriquement entre les observations positives et négatives. Comme présenté précédemment, nos travaux ont étendu l'algorithme à l'apprentissage de similarité, ainsi que les garanties en norme sup pour la distance entre la courbe ROC du score appris et la courbe ROC optimale. Pour illustrer notre variante de TreeRank pour l'ordonnancement par similarité, nous représentons visuellement la forme de nos régions candidates symétriques utilisées pour séparer l'espace.

Considérations pratiques pour TreeRank. Les deux inconvénients de TreeRank sont: 1) sa dépendance aux divisions initiales de l'espace, qui compromettent les performances si la famille de régions candidates est trop limitée, 2) et son caractère discontinu, qui est incompatible avec les hypothèses naturelles de nombreux cas pratiques. Une première réponse à ces limites est d'étendre l'idée de forêts aléatoires proposé dans [START_REF] Breiman | Random forests[END_REF] à TreeRank, ce qui est proposé dans Clémençon et al. (2013). Nous montrons sur des données-jouets avec un rapport de vraisemblance continu dG{dH que les arbres d'ordonnancement moyennés peuvent corriger à la fois la mauvaise spécification initiale des régions candidates et la nature discrète des arbres d'ordonnancement. Précisément, la fonction de score moyennée que nous obtenons donne une courbe ROC presque indiscernable de la courbe ROC optimale, malgré son ensemble de valeurs limité -mais nombreux -et l'inadéquation de chaque arbre relativement au vrai rapport de vraisemblance.

Ces propositions esquissent une voie pour de nouveaux algorithmes résolvant l'ordonnancement par similarité, qui est une approche rationnelle pour traiter le problème d'identification biométrique. Cependant, il ne s'agit que d'illustrations sur des problèmes très simples. Trouver de nouvelles approches pour l'ordonnancement par similarité en pratique, idéalement sur des expériences à grande échelle qui correspondent à des scénarios pratiques en matière de biométrie, est une piste prometteuse pour de futurs travaux.

Fiabilité en apprentissage automatique

Outre l'apprentissage de similarité, la biométrie et en particulier la reconnaissance faciale incarne de nombreuses questions importantes en apprentissage automatique, comme le montrent les rapports du NIST (National Institute of Standards and Technology) sur les évaluations des entreprises de reconnaissance faciale [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF]. Précisément, la reconnaissance faciale est confrontée à des questions concernant la robustesse des prédictions, les biais dans les données d'entraînement, ainsi que sur l'équité algorithmique. Les sous-sections de cette section abordent toutes ces questions séquentiellement. Bien qu'il existe une littérature abondante sur ces sujets en général, notre travail porte sur des situations difficiles et peu étudiées qui s'appliquent directement aux problèmes biométriques. Précisément: 1) concernant la robustesse des prédictions, nous considérons l'apprentissage d'une liste ordonnée d'identités candidates, comme le font certains problèmes d'identification biométrique, 2) pour le biais des données d'entraînement, nous rééquilibrons les instances d'entraînement en utilisant des informations de haut niveau, telle que la nationalité dans le cadre du contrôle aux frontières, 3) pour la question d'équité, nous nous concentrons sur les fonctions de score en tant que passerelle vers les fonctions de similarité, comme dans la partie II.

Ordonner les labels par probabilité

Dans les systèmes biométriques destinés aux enquêtes criminelles, un expert humain considère souvent les suspects les plus probables proposés par un système. En général, les problèmes de classification difficiles se concentrent sur les labels les plus probables, comme le fait par exemple le challenge ILSVRC (ImageNet Large Scale Visual Recognition Challenge), où [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] et les articles suivants considèrent la précision en top-5 à côté de la précision en classification habituelle (en top-1). Dans notre travail, nous proposons une approche pour la prédiction d'une liste ordonnée de labels à partir de données de classification. Nous désignons ce problème par le nom de label ranking (LR, ordonnancement de labels). Précisément, nous proposons d'utiliser la fameuse technique One-versus-One (un contre un) pour la classification multiclasse, et nous dérivons des garanties pour cette approche.

Cadre probabiliste pour l'ordonnancement de labels (LR). Dans le cadre probabiliste de la classification multiclasse, nous considérons une paire aléatoire pX, Y q P X ˆY avec Y " t1, . . . , Ku, ainsi que le risque Lpgq " PtgpXq ‰ Y u associé à un classifieur g : X Ñ Y .

Le classifieur optimal pour Lpgq dans la classe de toutes les fonctions mesurables est le célèbre classifieur de Bayes g ˚, défini comme suit: g ˚pxq :" arg max kPt1,...,Ku η k pxq, où η k : x Þ Ñ PtY " k | X " xu est la probabilité a posteriori de la classe k pour tout k P t1, . . . , Ku. Nous appelons label ranking (LR) la tâche consistant à trouver une liste ordonnée des labels les plus probables. Cela revient à associer à tout x P X une permutation σ x P S K , telle que:

η σ ´1
x p1q ą η σ ´1 x p2q ą ¨¨¨ą η σ ´1 x pKq .

(12.10)

Nous notons σ X la permutation aléatoire associée.

Sur la régression de médiane d'ordonnancements (RMR). Un autre problème bien connu dans la littérature statistique concerne la prédiction d'un ordonnancement sur un ensemble de labels est la régression de médiane d'ordonnancements (RMR, Ranking Median Regression) [START_REF] Tsoumakas | Mining multi-label data[END_REF][START_REF] Vembu | Label ranking algorithms: A survey[END_REF]Clémençon et al., 2018). La RMR considère une paire pX, Σq P X ˆSK et apprend à partir de données une règle d'ordonnancement s : X Ñ S K qui minimise le risque:

Rpsq :" ErdpΣ, spXqqs, (12.11) où d : S K ˆSK Ñ R `est une fonction de perte symétrique. La distance d quantifie une distance entre les ordonnancements. La distance la plus connue est le τ de Kendall d τ , qui est égal au nombre de désaccords sur les paires entre les deux permutations.

Solution optimale de la RMR. Des travaux antérieurs sur la RMR (Clémençon et al., 2018) ont montré que le minimiseur optimal de Eq. (12.11) pour des règles d'ordonnancement mesurables a une formulation analytique simple pour la distance d τ , sous une hypothèse appelée la Transitivité Stochastique Stricte (SST). La SST suppose que les probabilités sur paires p k,l pxq :" PtΣpkq ă Σplq | X " xu ": 1 ´pl,k pxq pour tout 1 ď k ă l ď K satisfont: pour tous x P X et pi, j, kq P t1, . . . , Ku 2 avec i ‰ j, nous avons p i,j pxq ‰ 1{2 et: p i,j pxq ą 1{2 et p j,k pxq ą 1{2 ñ p i,k pxq ą 1{2.

Sous l'hypothèse du SST, la règle d'ordonnancement optimal pour d τ s'écrit:

s X pkq " 1 `ÿ l‰k Itp k,l pXq ă 1{2u. (12.12)

Le LR en tant que RMR avec information partielle. Si la caractérisation de l'élément optimal est une extension bienvenue de la théorie de l'apprentissage usuelle [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF] au problème de RMR, l'ordonnancement sur les labels Σ n'est pas disponible en entier lorsqu'il s'agit de données de classification. Cependant, nos travaux montrent que si l'on considère la permutation aléatoire Σ comme générée par un modèle BLTP conditionnel [START_REF] Korba | Learning from ranking data : theory and methods[END_REF] avec vecteur de préférence ηpXq " pη 1 pXq, . . . , η K pXqq, alors il est possible de construire un Σ qui satisfasse Y " Σ ´1p1q presque sûrement. Sur la base de cette observation, nous proposons de considérer le LR comme un problème de RMR avec l'information partielle Σ ´1p1q sur la permutation aléatoire complète Σ.

Solutions optimales du LR avec la méthode One-versus-One (OVO). Nous pouvons calculer les expressions des probabilités sur paires p k,l pxq sous un modèle BTLP conditionnel avec vecteur de préférence ηpxq. Précisément, nous avons p k,l pxq " η k pxq{pη k pxq `ηl pxqq pour tout x P X et k ă l. Remarquons que p k,l pxq correspond à la probabilité de prédire k contre l pour le problème de classification "One-Versus-One" (OVO, un-contre-un). L'approche OVO a été étudiée en détail [START_REF] Hastie | Classification by pairwise coupling[END_REF][START_REF] Moreira | Improved pairwise coupling classification with correcting classifiers[END_REF][START_REF] Allwein | Reducing multiclass to binary: A unifying approach for margin classifiers[END_REF][START_REF] Fürnkranz | Round robin classification[END_REF] pour la résolution de la classification multiclasse à l'aide d'algorithmes de classification binaire. L'approche OVO consiste à apprendre KpK ´1q{2 fonctions de décision, précisément un classifieur pour chaque classe k contre l avec k ă l, et de prendre le vote majoritaire des KpK ´1q{2 classifieurs. Le classifieur de Bayes pour le problème OVO pk, lq est g k,l : x Þ Ñ 2 ¨Itp k,l pxq ě 1{2u ´1. Ainsi, Eq. (12.12) se réduit à: s X pkq " 1 `ÿ l‰k Itg k,l pXq " ´1u. (12.13) 12.5. Fiabilité en apprentissage automatique Notons que s X correspond à σ X dans Eq. (12.10), dès que tous les η k pXq sont distincts. Nous avons montré qu'une combinaison des solutions optimales de tous les KpK ´1q{2 problèmes OVO implique une règle d'ordonnancement optimale s X . Par conséquent, nous pouvons probablement dériver une bonne solution du problème LR à partir de bonnes solutions de tous les problèmes OVO.

Garanties pour le LR avec la méthode OVO. Nous proposons une solution au LR, qui utilise une combinaison des solutions de tous les problèmes empiriques de classification OVO. Ensuite, nous déduisons des garanties théoriques pour cette solution. Soit un échantillon D n " tpX i , Y i qu n i"1 de n copies i.i.d. de la paire aléatoire pX, Y q, ainsi que la notation Y k,l,i " ItY i " lu ´ItY i " ku pour tout k ă l et n'importe quel i P t1, . . . , nu. Le risque empirique p L k,l de g : X Ñ t´1, `1u pour la classification OVO de k contre l s'écrit: p L k,l pgq :" 1 n k `nl ÿ i:YiPtk,lu ItgpX i q ‰ Y k,l,i u, où n k " ř n i"1 ItY i " ku pour tout k P t1, . . . , Ku. Nous écrivons p g k,l le minimiseur de p L k,l sur la classe de proposition fixe G de classifieurs binaires. Sur le modèle de Eq. (12.13), une solution empirique du LR s'écrit: p s X pkq :" 1 `ÿ k‰l Itp g k,l pXq " ´1u.

Une simple inégalité de Boole implique:

Ptp s X ‰ s X u ď ÿ kăl Ptp g k,l pXq ‰ g k,l pXqu.

(12.14)

Eq. (12.14) montre que la somme des probabilités de ne pas prédire la classe optimale pour chaque problème d'OVO borne la probabilité de ne pas prévoir l'ordonnancement optimal de labels en LR. De plus, une conséquence des hypothèses habituelles pour la dérivation de vitesses de généralisation rapide -présentée pour la première fois dans [START_REF] Mammen | Asympotical minimax recovery of the sets with smooth boundaries[END_REF] et aussi détaillée dans [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] -donne une borne supérieure sur la quantité à droite de Eq. (12.14) par les excès de risque des problèmes de classification k contre l. Sous une hypothèse de bruit standard, nous utilisons les vitesses d'apprentissage rapide usuelles en Opn ´1{p2´aq q pour l'excès d'erreur de chaque problème de classification k contre l, où a P p0, 1q est un paramètre de bruit. Combinés avec Eq. (12.14), ces résultats impliquent une borne de convergence en Opn ´a{p2´aq q pour la quantité Ptp s X ‰ s X u, ce qui est plus lent que la vitesse d'apprentissage usuelle en classification, en raison de la complexité inhérente dans le problème d'ordonnancement des labels.

Implications de l'analyse. Un analogue à l'erreur RMR de l'Eq. (12.11) pour le LR serait le risque suivant:

Rpsq :" ErdpspXq, σ X qs, (12.15) pour une règle d'ordonnancement s : X Ñ S K . Notons que, pour toute distance d bornée:

dpσ, σ 1 q ď Itσ ‰ σ 1 u ˆmax σ0,σ1PS K dpσ 0 , σ 1 q, (12.16) pour tout σ, σ 1 P S K . Eq. (12.16) implique une extension de nos garanties pour Ptp s X ‰ s X u au risque Eq. (12.15) du minimiseur empirique. Incidemment, notre analyse du LR fournit la première garantie de généralisation à nombre d'instances fini pour l'approche OVO en classification multiclasse, en considérant le cas spécifique k " 1 pour les garanties sur la précision en top-k que nous fournissons.

En conclusion, nous avons proposé le nouveau mais naturel problème d'ordonnancement de labels (LR), qui consiste à apprendre la prédiction d'une liste ordonnée des labels les plus probables à partir de données de classification multiclasse. Bien que [START_REF] Korba | A structured prediction approach for label ranking[END_REF] et [START_REF] Brinker | A reduction of label ranking to multiclass classification[END_REF] donnent des approches pratiques à la régression de médiane d'ordonnancements (RMR) avec une information partielle, nos garanties théoriques sont nouvelles. Notre analyse s'inscrit parfaitement dans le cadre usuel de minimisation du risque empirique et exploite des résultats récents sur la RMR. Un sous-produit de notre analyse est la première borne de généralisation à échantillon fini sur l'approche OVO pour la classification multiclasse.

Correction des biais de sélection

Dans certains problèmes d'apprentissage statistique, la distribution P 1 des données d'entraînement Z 1 1 , . . . , Z 1 n peut être différente de celle des données de test P . Cette configuration constitue un cas particulier d'apprentissage par transfert [START_REF] Pan | A survey on transfer learning[END_REF][START_REF] Ben-David | A theory of learning from different domains[END_REF][START_REF] Storkey | When training and test sets are different: characterizing learning transfer[END_REF][START_REF] Redko | Advances in Domain Adaptation Theory[END_REF]. Notamment dans les problèmes de reconnaissance faciale, la population utilisée pour l'entraînement n'est souvent pas représentative de la population test, comme souligné dans [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF]. Des informations auxiliaires sous forme de caractéristiques de haut niveau sont cependant souvent disponibles, comme la nationalité associée à un portrait en reconnaissance faciale. Notre travail porte sur l'apprentissage avec des données biaisées du point de vue de la minimisation du risque empirique (ERM). À cet égard, nous proposons une approche basée sur l'échantillonnage préférentiel qui traite: 1) de problèmes de classification où les probabilités de chaque classe diffèrent entre les phases d'entraînement et de test, 2) de situations où les données proviennent de populations stratifiées représentées différemment entre l'entraînement et le test, 3) de l'apprentissage PU, qui consiste à apprendre avec un échantillon de données positives et non labellisées (du Plessis et al., 2014), 4) et de l'apprentissage avec des données censurées [START_REF] Fleming | Counting processes and survival analysis[END_REF]. Notre analyse est soutenue par des résultats empiriques solides pour la classification sur la base de données ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], pour laquelle nous avons créé une information de strate à partir de concepts de plus haut niveau que les classes prédites.

Minimisation du risque empirique pondéré (WERM). Le but des algorithmes d'apprentissage est généralement de trouver un paramètre θ P Θ, qui minimise l'espérance de risque Rpθq " E P r pθ, Zqs sur les données de test, avec : Θ ˆZ Ñ R `une fonction de perte mesurable. Pour approximer Rpθq, nous proposons un estimateur pondéré r R w,n sur les données d'entraînement: r R w,n pθq :" 1 n n ÿ i"1 w i ¨ pθ, Z 1 i q, où w " pw 1 , . . . , w n q est un vecteur de poids. Lorsque P 1 " P et w " p1, . . . , 1q, alors r R w,n pθq est le risque empirique usuel, ainsi qu'un estimateur non biaisé de l'espérance du risque Rpθq sur les données de test. Lorsque P 1 ‰ P et P est absolument continu par rapport à P 1 , la méthode d'échantillonnage préférentiel (Wasserman, 2010, Section 25.3) suggère de fixer w i :" ΦpZ i q ": w i pour tout i P t1, . . . , nu, où Φpzq :" pdP {dP 1 qpzq pour tout z P Z. Φ est le rapport de vraisemblance entre P 1 et P , donc r R w,n est alors un estimateur non biaisé de R.

Garanties de généralisation pour WERM. Nous dérivons ensuite des garanties de généralisation habituelles en Opn ´1{2 q qui dépendent de la norme sup Φ 8 du rapport de vraisemblance sur l'espace d'entrée Z. Nos garanties montrent que la généralisation est meilleure lorsque les deux distributions P 1 et P sont similaires. Dans le cas général, le rapport de vraisemblance Φ est inconnu, ce qui limite l'applicabilité de cette technique. De plus, Φ est une fonction sur l'espace d'entrée Z, ce qui rend son estimation peu pratique. Notre travail présente des situations pour lesquelles la fonction de vraisemblance Φ a une formulation simple, sous l'hypothèse qu'une information auxiliaire sur une relation entre les distributions P 1 et P est disponible.

WERM pour les probabilités de strates. Dans le contexte de la classification multiclasse, c'est-à-dire quand Z " pX, Y q P X ˆY avec Y " t1, . . . , Ku et K P N est le nombre de classes, Φ a une forme simple lorsque la proportion en espérance p k " P Z"P tY " ku de chaque classe k dans l'ensemble de données du test est connu. Dans ce contexte, les poids optimaux w ˚" pw 1 , . . . , w nq satisfont w i " p Yi {p 1 Yi pour tout i P t1, . . . , nu, où p 1 k " P P 1 tY " ku est la proportion en espérance de la classe k dans les données d'entraînement pour tout k P t1, . . . , Ku. Notons que les p 1 k peuvent être estimés à partir des données d'entraînement. Cependant cette stratégie de repondération ne dépend pas du fait que la prédiction des classes soit l'objectif, mais s'applique dès l'on connaît: les proportions de chaque strate pour une stratification quelconque sur la distribution test, et la strate associée à chaque instance de l'ensemble d'entraînement.

WERM pour l'apprentissage avec des données positives et non labellisées (PU). L'apprentissage avec des données positives et non labellisées (PU) a fait l'objet d'une attention croissante dans la récente littérature en apprentissage statistique (du Plessis and Sugiyama, 2014;du Plessis et al., 2014[START_REF] Du Plessis | Convex formulation for learning from positive and unlabeled data[END_REF][START_REF] Kiryo | Positive-unlabeled learning with nonnegative risk estimator[END_REF][START_REF] Bekker | Beyond the selected completely at random assumption for learning from positive and unlabeled data[END_REF]. L'apprentissage PU considère un problème de classification binaire, c'est-à-dire Z " pX, Y q P X ˆt´1, `1u, et apprend un classifieur avec un échantillon de positifs et d'instances non labellisées. L'échantillon non labellisé est un mélange de données négatives et positives dans des proportions fixes. Notre travail montre que, en repondérant les instances de ces deux échantillons, nous obtenons un estimateur non biaisé du risque R. Nous fournissons des garanties statistiques pour le minimiseur de notre risque estimé.

Expériences sur la base de données ImageNet. Finalement, nous fournissons une illustration numérique convaincante de l'efficacité de WERM sur la base de données ImageNet, une base utilisée pour évaluer les algorithmes de classification à grande échelle en vision par ordinateur [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. Les classes d'ImageNet sont construites à partir de la base de données lexicale WordNet pour l'anglais [START_REF] Fellbaum | WordNet: an electronic lexical database[END_REF]. En tant que telles, ces classes peuvent être regroupées en plusieurs concepts de haut niveau, qui constituent les strates de notre expérience de classification pondérée. Par exemple, la classe "flamant rose" est une sous-classe de la strate "oiseau". L'utilisation d'informations de haut niveau pour re-pondérer les données d'entraînement améliore considérablement les performances sur les données de test, mesuré en termes de précision en classification au top-1 et au top-5.

Bien qu'assurer la représentativité d'une base de données puisse aider à obtenir des prédictions satisfaisant des critères d'équité, il y a un intérêt croissant pour des mécanismes qui corrigent explicitement les biais inhérents aux données d'entraînement.

12.5.3 Équité dans l'apprentissage de fonctions de scoring L'apprentissage d'un classifieur sous des contraintes d'équité a reçu beaucoup d'attention dans la littérature [START_REF] Dwork | Fairness through awareness[END_REF]Zafar et al., 2017a;[START_REF] Donini | Empirical risk minimization under fairness constraints[END_REF][START_REF] Barocas | Fairness and Machine Learning[END_REF][START_REF] Williamson | Fairness risk measures[END_REF][START_REF] Mcnamara | Costs and benefits of fair representation learning[END_REF]. Toutefois, les approches proposées pour assurer une équité en ordonnancement soit corrigent seulement la fonction de score avec une étape de post-traitement [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF][START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF][START_REF] Zehlike | Fa*ir: A fair top-k ranking algorithm[END_REF][START_REF] Celis | Ranking with fairness constraints[END_REF], soit s'attaquent à des notions originales d'équité, comme une équité d'exposition dans la présentation séquentielle d'ordonnancements (Singh andJoachims, 2018, 2019). De nombreux articles sur l'équité pour l'ordonnancement ont proposé différentes contraintes basées sur l'aire sous la courbe ROC (AUC), une mesure standard de la performance en ordonnancement. Notre travail propose tout d'abord: un cadre unifié pour des contraintes d'équité basées sur l'AUC, des garanties de généralisation pour la minimisation d'une perte qui intègre n'importe laquelle de ces contraintes, ainsi qu'une procédure pratique d'optimisation de cette perte basée sur la descente de gradient. Ensuite, nous montrons les limites des contraintes d'équité basées sur l'AUC, et proposons des contraintes plus fortes basées sur la courbe ROC. Pour finir, nous prouvons des garanties de généralisation, ainsi qu'une extension de notre procédure d'optimisation pour l'apprentissage avec un critère d'équité à nos nouvelles contraintes basées sur la ROC.

Cadre probabiliste pour l'équité en ordonnancement bipartite. Le cadre standard d'équité pour la classification binaire considère un triplet de variables aléatoires pX, Y, Zq P X ˆt´1, 1u ˆt0, 1u, où X est la variable aléatoire d'entrée, Y est la variable aléatoire binaire de sortie, et Z encode l'appartenance à un groupe protégé. Dans l'ordonnancement bipartite, nous apprenons une fonction de score s : X Ñ R et nous l'évaluons par rapport à la manière dont elle projette les négatifs Y " ´1 relativement aux positifs Y " `1 sur la droite réelle. Dans le contexte de l'équité, l'influence de Z sur la distribution des scores est important. Pour cela, nous introduisons les distributions conditionnelles suivantes d'un score s donné pour tout z P t0, 1u: Des différences dans la répartition des positifs (ou des négatifs) entre les groupes protégés entraîne des écarts dans les taux d'erreur de ces groupes, observés pour la reconnaissance faciale dans [START_REF] Grother | Face Recognition Vendor Test (FRVT) -Performance of Automated Gender Classification Algorithms[END_REF].

H
Critère unifié d'équité basé sur l'AUC. Pour corriger les écarts de taux d'erreur entre les groupes protégés, de nombreux auteurs -principalement issus de la communauté des systèmes de recommandation -ont proposé des contraintes d'équité basées sur l'AUC [START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF][START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF][START_REF] Kallus | The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric[END_REF]. Avec Dpsq :" pH p0q s , H p1q s , G p0q s , G p1q s q J , ces contraintes s'écrivent: AUC α J Dpsq,β J Dpsq " AUC α 1J Dpsq,β 1J Dpsq , (12.17) pour différentes valeurs de pα, β, α 1 , β 1 q P pPq 4 , où P " tv | v P R 4 `, 1 J v " 1u désigne le 4simplexe. Nos travaux montrent que: si la contrainte d'équité Eq. (12.17) est satisfaite lorsque la distribution X|Y " y, Z " z ne dépend pas de z P t0, 1u pour à la fois y " ´1 et y " `1, alors Eq. (12.17) s'écrit comme la combinaison linéaire Γ J Cpsq " 0 de cinq contraintes d'équité élémentaires Cpsq " pC 1 psq, . . . , C 5 psqq avec Γ P R 5 . Notre définition générale des contraintes d'équité basées sur l'AUC englobe toutes les mesures d'équité basées sur l'AUC proposées, et peut servir à en dériver de nouvelles. Plus important encore, elle ouvre la voie à des approches flexibles pour l'apprentissage de score sous une contrainte d'équité basée sur l'AUC.

Apprentissage sous contraintes d'équité basée sur l'AUC. Nous intégrons la contrainte d'équité basée sur l'AUC dans une pénalisation ajoutée à la fonction objectif, que nous maximisons: max sPS AUC Hs,Gs ´λ Γ J Cpsq , (12.18) sur une famille de fonction de score S. Le paramètre λ règle un compromis entre la précision en ordonnancement et le critère d'équité. Nous fournissons des garanties théoriques sur l'erreur de généralisation de notre critère Eq. (12.18), prouvées avec des inégalités de concentration sur les déviations des U -statistiques. Nous proposons un algorithme basé sur la descente de gradient qui optimise simplement une version relaxée d' Eq. (12.18). Avec la notation r ¨pour indiquer la relaxation d'une estimation empirique de l'AUC par la fonction logistique σ : x Þ Ñ 1{p1 `e´x q, notre relaxation de la perte avec une contrainte particulière basée sur l'AUC s'écrit:

Perspectives

En conclusion, la thèse aborde des problèmes importants en biométrie du point de vue de la théorie de l'apprentissage statistique. Notre travail propose des idées originales pour ces problèmes, et soutient ces idées avec des résultats théoriques. Ces résultats peuvent être interprétés comme des garanties de sécurité qui sont vraies sous des hypothèses probabilistes. Notre travail est une réponse indispensable à l'augmentation rapide du volume de littérature expérimentale en apprentissage automatique que les chercheurs en biométrie doivent suivre. L'identification biométrique, et la reconnaissance faciale en particulier, incarnent simultanément de nombreux sujets en apprentissage automatique, comme l'apprentissage par paires, les biais d'échantillonnage ou l'ordonnancement. Pour cette raison, nous avons envisagé des versions stylisées de ces problèmes, car leur examen simultané obscurcirait notre discours, et court le risque d'être considéré comme anecdotique par la communauté de l'apprentissage automatique. La richesse des sujets abordés par la thèse résulte de cet impératif. L'élargissement de ce spectre pourrait être envisagé, par exemple en considérant l'extension des critères d'ordonnancement des meilleurs [START_REF] Menon | Bipartite ranking: a risk-theoretic perspective[END_REF], Section 9) au problème d'ordonnancement par similarité présenté ci-dessus.

Du point de vue de la biométrie, la perspective la plus importante pour cette thèse est de réaliser l'impact potentiel des méthodes présentées, en fournissant des preuves empiriques solides de leur pertinence dans des contextes pratiques. En effet, si l'adoption rapide des techniques d'apprentissage automatique par les entreprises privées ont stimulé la croissance du domaine, elle a également dirigé l'essentiel de l'attention sur les articles qui proposent des solutions sans équivoque à des problèmes industriels spécifiques. Un exemple notoire en reconnaissance faciale est [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF]. Dans ce contexte, la promotion de nos travaux nécessitera de trouver et de présenter pédagogiquement des expériences à grande échelle qui traitent de cas d'utilisation pratiques précis, ce qui est une orientation prometteuse pour les travaux futurs.

Enfin, nous pourrions étendre les différents sujets abordés dans la thèse. Dans le contexte de l'équité pour l'ordonnancement, une limite de notre analyse provient de l'absence d'une expression analytique pour la meilleure fonction de score sous une condition d'équité. Toutefois, elle est fournie dans le cas de l'équité pour la régression par [START_REF] Chzhen | Fair Regression via Plug-in Estimator and Recalibration With Statistical Guarantees[END_REF] par exemple. Pour l'ordonnancement bipartite, le fait de surmonter cet obstacle ouvre la voie à une extension des algorithmes de Clémençon and Vayatis (2009) -basés sur le partionnement de l'espace d'entrée -à l'apprentissage avec des contraintes d'équité. Une autre possibilité concerne l'extension des techniques présentées ici au cas de l'ordonnancement par similarité. En effet, cette extension donne un cadre qui correspond très étroitement aux considérations opérationnelles en matière de biométrie, et serait justifiée par l'intérêt actuel pour la correction explicite des biais en reconnaissance faciale. Le volet expérimental de ces travaux serait soutenu par la disponibilité de bases de données de visage adaptées [START_REF] Wang | Racial faces in the wild: Reducing racial bias by information maximization adaptation network[END_REF]. Une autre possibilité consiste à adresser les limites de nos travaux sur la minimisation des risques empiriques pondérés, en considérant les cas où la nature de la différence entre les données d'entraînement et de test n'est pas couverte par nos travaux. Par exemple, dans Sugiyama et al. (2007), les auteurs proposent d'estimer le rapport de vraisemblance à l'aide d'un petit échantillon qui suit la distribution de test comme information auxiliaire. Outre les exemples ci-dessus, d'autres extensions de chaque sujet abordé dans la thèse pourraient être envisagées.

En conclusion, la richesse des questions qui se posent en biométrie est un terrain fertile pour le développement à la fois de la théorie et de la pratique en apprentissage automatique. Cette richesse a engendré cette thèse et peut inspirer de futures recherches. In this thesis, we consider those challenges from the angle of statistical learning theory, which leads us to propose or sketch practical solutions. First, we answer to the proliferation of papers on similarity learning for deep neural networks that optimize objective functions that are disconnected with the natural ranking aim sought out in biometrics. Precisely, we introduce the notion of similarity ranking, by highlighting the relationship between bipartite ranking and the requirements for similarities that are well suited to biometric identification. We then extend the theory of bipartite ranking to this new problem, by adapting it to the specificities of pairwise learning, particularly those regarding its computational cost. Usual objective functions optimize for predictive performance, but recent work has underlined the necessity to consider other aspects when training a biometric system, such as dataset bias, prediction robustness or notions of fairness. The thesis tackles all of those three examples by proposing their careful statistical analysis, as well as practical methods that provide the necessary tools to biometric systems manufacturers to address those issues, without jeopardizing the performance of their algorithms.

Institut Polytechnique de Paris 91120 Palaiseau, France

Figure 2 . 1 :

 21 Figure 2.1: Representation of Example 2.1 with a " 1{2. the left-hand side figure represents the different possible parametrizations of P , namely pF, ηq and pp, G, Hq (we omit p " 1{2 here).As F, H, G are all absolute continuous measures, we represented the Radon-Nikodym derivative w.r.t. the Lebesgue measure of those distributions(Shorack, 2000, Section 2, Chapter 4). On the other hand, the two right-hand side figures represent the approximations of G and H that we obtain using histograms of n independent realizations of pX, Y q with n " 200 and n " 1000.

Figure 2 . 3 :

 23 Figure 2.3: Boxplot of the regret over 100 generations of a dataset D n , for three selected values of the parameter a and different values of n, compared to the fast bound derived in Proposition 2.28. See Remark 2.29 for more details.

Figure 3

 3 Figure 3.2: Iterations of the LeafRank algorithm for depths 1, 2, 3 and 10. The top figures represent the split of the space, where greener areas correspond to higher scores. The bottom figures correspond to the ROC curves associated with the score functions.

  Fig. 3.2, we show the ROC curve obtained by TreeRank with coordinate splits Interpolation of ROC (Clémençon and Vayatis, 2009, Section III-D)

Figure 3 . 3 :

 33 Figure 3.3: Description of the ROC ˚interpolation procedure.

Figure 3 . 4 :

 34 Figure 3.4: Description of the TreeRank procedure.

αα

  Figure 5.1: Constraints on the value of α for several values of a and two values of m.

  Figure 5.4: Example distributions and F 1 's for n " 1000 and two values of a.

  Figure 5.5: Boxplot of the 1000 regrets ROC S ˚pαq ´R`p p S n q for each n and several values of a. The line represents the regression on the 0.80-quantile.

Figure 5

 5 Figure 5.8: Symmetric split produced by the Symmetric LeafRank procedure on a bounded two-dimensional space.

  Definition 6.3. (Incomplete generalized U -statistic) Let B ě 1. The incomplete version of the U -statistic (6.1) based on B terms is defined by: where D B is a set of cardinality B built by sampling uniformly with replacement in the set Λ of vectors of tuples ppi p1q 1 , . . . , i p1q d1 q, . . . , pi pKq 1 , . . . , i pKq d K qq, where 1 ď i pkq 1 ă . . . ă i pkq d k ď n k and 1 ď k ď K.

Figure 6 . 4 :

 64 Figure 6.4: Empirical variances as a function of the number of evaluated pairs for SWOR, with n " 100, 000, m " 200 and N " 100, evaluated over 500 runs.

Figure 7 . 1 :

 71 Figure 7.1: Solving pointwise ROC optimization for a toy dataset with a bilinear similarity.

  Ay s.t. xN, Ay ď β and xA, Ay ď 1.

Figure 7

 7 Figure 7.2: Sample and optimal rejection regions visualization with our example distribution.

Figure 7

 7 Figure 7.4: (Left) Relaxation of the indicator used. (Middle) Dynamics of the gradient-based learning process. (Right) Linear separation for the optimal and learned rejection zone.

Figure 7 . 5 :

 75 Figure 7.5: Representation of TreeRank score function with different simple proposed regions. The x-axis corresponds to x 1 while the y-axis corresponds to x 1 1 .

  Figure 7.6: Value of the score learned with a ranking forest on the r0, 1s 2 space.

Figure 8

 8 Figure 8.1: Pseudo-code for 'OVO label ranking'

Figure 8

 8 Figure 8.3, Figure 8.4 and Figure 8.5 represent boxplots obtained with 100 independent estimations on 1000 test points. Precisely, Figure 8.3 represents the number of cycles in predicted permutations, Figure 8.4 the average miss probability Ptp σ X ‰ σ X u for the problem of predicting permutations,

  Distribution for α " 0.2.

Figure 8

 8 Figure 8.2: Representation of the proportion of each class k over r0, 1s.

Figure 9

 9 Figure 9.1: Pdf's and values of the excess risk Epp 1 , pq for different values of α, β.

  Figure 9.2: Comparison of p k 's and p 1 k 's.

Figure 9 . 3 :

 93 Figure 9.3: Dynamics for the class reweighting experiment with MNIST.

Figure 9 Figure 9

 99 Figure 9.4: Distribution of the ImageNet train dataset over the created strata which definitions are given in Table9.5.

Figure 9 . 6 :

 96 Figure 9.6: Dynamics for the linear model for the strata reweighting experiment with ImageNet.

Figure 9 . 7 :

 97 Figure 9.7: Dynamics for the MLP model for the strata reweighting experiment with ImageNet.

  the c.d.f. of the score on the positives (resp. negatives) of group z P t0, 1u, i.e. G pzq s ptq " G pzq pspXq ď tq and H pzq s ptq " H pzq pspXq ď tq, for any t P R. Both[START_REF] Beutel | Fairness in recommendation ranking through pairwise comparisons[END_REF] and[START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] proposed the following fairness constraints:

p0qsFF

  ptq (resp. r F p1q s ptq). Introduce c pkq F as a constant in r´1, `1s and t pkq F as a threshold in R, the following loss pkq F seeks to equalize these two quantities at threshold t q -evaluated on the validation set V m -is not too large, the threshold t pkq F is modified slightly every few iterations so that p F is detailed in Algorithm 3, and we introduce a step ∆t to modify the thresholds t pkq F . The final loss writes: r L Λ,c,t psq :" ´1 ´Ć AUC Hs,Gs ,t by approximating Ć AUC Hs,Gs above by Ć AUC pBq Hs,Gs . The full algorithm is given in Algorithm 3. Algorithm 3 Practical algorithm for learning with ROC-based constraints. Input: training set D n , validation set V m c pkq F Ð 0 for any F P tH, Gu, k P t1, . . . , m F u t pkq F Ð 0 for any F P tH, Gu, k P t1, . . . , m F u for i " 1 to n iter do B N Ð N observations sampled with replacement from D n s Ð updated scoring function using a gradient-based algorithm (e.g. ADAM), using the derivative of r L pBq Λ,c,t psq on B N if pn iter mod n adapt q " 0 then for any F P tH, Gu, k P t1, . . . , m F u do

  Fig. 10.1 plots AUC Hs,Gs and AUC H pzq s ,G pzq s

Figure 10

 10 Figure10.1: Plotting Example 10.11 for q 1 " 17{20. Under the fairness definition Eq. (10.4), a fair solution exists for c " 1{2, but the ranking performance for c ă 1{2 is significantly higher.

Figure 10 LFigure 10 . 7 :

 10107 Figure 10.2: For Example 10.11, L λ ps c q as a function of c P r0, 1s for any λ P t0, 1u, with the parametrization s c pxq " cx 1 `p1 ´cqx 2 , and the values c for the scores obtained by gradient descent with Algorithm 2.

Figure 10 . 8 :

 108 Figure 10.8: ROC curves for the databases Adult, Compas, Bank and German for a score learned without and with fairness constraints. On all plots, dashed and solid lines represent respectively training and test sets. Black curves represent ROC Hs,Gs , and above the curves we report the corresponding ranking performance AUC Hs,Gs .

  , Chapter 2) à partir des images brutes d'empreintes digitales. Des points caractéristiques appelés minuties sont alors extraits de cette représentation intermédiaire. Enfin, la comparaison repose sur l'évaluation d'une distance entre les nuages de points associés aux deux images. Nous renvoyons à Jain et al. (2011) (Section 2) pour plus de détails. Extraction de caractéristiques. Le module qui a reçu le plus d'attention dans la recherche en biométrie est le module d'extraction de caractéristiques. Par exemple, les recherches sur la recon-naissance automatique des empreintes digitales ont consacré une quantité phénoménale de travail pour trouver le plus d'information discriminante possible dans les images. En reconnaissance faciale, l'extraction de caractéristiques a d'abord été basée sur les eigenFaces

Figure 12

 12 Figure 12.1: Ce graphique montre que la fixation d'un seuil pour les faux positifs (appelé taux de fausse acceptance (FAR) en reconnaissance faciale) de α " 10 ´5 pour la population générale peut donner un taux de faux positif bien plus élevé lorsqu'il est utilisé sur un autre type de population. Plus précisément, il donne α " 10 ´2,8 pour une population originaire de Polynésie.

Figure 12

 12 Figure 12.2: Courbes ROC obtenues respectivement: en apprenant un score sans contrainte, avec une contrainte basée sur l'AUC, et avec une contrainte basée sur la ROC. Nous avons choisi les paramètres de la contrainte basée sur la ROC dans le but d'obtenir ROC F p0q s ,F p1q s pαq " α pour tout α P r0, 1{4s et F P tH, Gu.

Titre:

  Ordonnancement par Similarit é pour la Biom étrie: Th éorie et Pratique Mots cl és : Biom étrie, Apprentissage Statistique, Ordonnancement Bipartite, Apprentissage de Similarit é, Biais algorithmique R ésum é : L'augmentation rapide de la population combin ée à la mobilit é croissante des individus a engendr é le besoin de syst èmes de gestion d'identit és sophistiqu és. À cet effet, le terme biom étrie se r éf ère g én éralement aux m éthodes permettant d'identifier les individus en utilisant des caract éristiques biologiques ou comportementales. Les m éthodes les plus populaires, c'est-à-dire la reconnaissance d'empreintes digitales, d'iris ou de visages, se basent toutes sur des m éthodes de vision par ordinateur.L'adoption de r éseaux convolutifs profonds, rendue possible par le calcul g én érique sur processeur graphique, ont port é les r écentes avanc ées en vision par ordinateur. Ces avanc ées ont permis une am élioration drastique des performances des m éthodes conventionnelles en biom étrie, ce qui a acc él ér é leur adoption pour des usages concrets, et a provoqu é un d ébat public sur l'utilisation de ces techniques. Dans ce contexte, les concepteurs de syst èmes biom étriques sont confront és à un grand nombre de challenges dans l'apprentissage de ces r éseaux. Dans cette th èse, nous consid érons ces challenges du point de vue de l'apprentissage statistique th éorique, ce qui nous am ène à proposer ou esquisser des solutions concr ètes. Premi èrement, nous r épondons à une prolif ération de travaux sur l'apprentissage de similarit é pour les r éseaux profonds, qui optimisent des fonctions objectif d étach ées du but naturel d'ordonnancement recherch é en biom étrie. Pr écis ément, nous introduisons la notion d'ordonnancement par similarit é, en mettant en évidence la relation entre l'ordonnancement bipartite et la recherche d'une similarit é adapt ée à l'identification biom étrique. Nous étendons ensuite la th éorie sur l'ordonnancement bipartite à ce nouveau probl ème, tout en l'adaptant aux sp écificit és de l'apprentissage sur paires, notamment concernant son co ût computationnel. Les fonctions objectif usuelles permettent d'optimiser la performance pr édictive, mais de r écents travaux ont mis en évidence la n écessit é de prendre en compte d'autres facteurs lors de l'entraînement d'un syst ème biom étrique, comme les biais pr ésents dans les donn ées, la robustesse des pr édictions ou encore des questions d' équit é. La th èse aborde ces trois exemples, en propose une étude statistique minutieuse, ainsi que des m éthodes pratiques qui donnent les outils n écessaires aux concepteurs de syst èmes biom étriques pour adresser ces probl ématiques, sans compromettre la performance de leurs algorithmes.Titre : Similarity Ranking for Biometrics: Theory and Practice Keywords : Biometrics, Statistical Learning Theory, Ranking, Similarity Learning, Algorithmic biais Abstract : The rapid growth in population, combined with the increased mobility of people has created a need for sophisticated identity management systems. For this purpose, biometrics refers to the identification of individuals using behavioral or biological characteristics. The most popular approaches, i.e. fingerprint, iris or face recognition, are all based on computer vision methods. The adoption of deep convolutional networks, enabled by general purpose computing on graphics processing units, made the recent advances in computer vision possible. These advances have led to drastic improvements for conventional biometric methods, which boosted their adoption in practical settings, and stirred up public debate about these technologies. In this respect, biometric systems providers face many challenges when learning those networks.

  Permutation of t1, . . . , nu A Y B (resp. A X B) Set union (resp. intersection) between the sets A and B A∆B Symmetric difference between the sets A and B

				1.2. Introduction
	Notation	Description
	w.r.t.	With respect to
	s.t.	Subject to
	r.h.s.	Right-hand side
	l.h.s.	Left-hand side
	a.s.	Almost-surely
	r.v.	Random variable
	i.i.d.	Independent and identically distributed
	c.d.f.	Cumulative distribution function
	p.d.f.	Probability density function
	:"	Definition of a variable
	@	Universal quantifier
	X Ñ Y A J	Map of X to Y Transpose of the matrix A
	H	Empty set
	`n k ˘Binomial coefficient
	S n
	#A	Cardinal of the set A
	A c	Complementary of a set A
	PpAq Ă	Set of all parts of a set A Set inclusion
	It¨u	Indicator function
	Impf q	Image of the function f
	sgnp¨q	Sign function, sgnpxq " 2Itx ě 0u	´1
	logp¨q	Natural logarithm function
	Op¨q	"Big O": Asymptotic order of a quantity
	Pr¨s	Probability of event
	Er¨s	Expectation of a random variable
	supppµq	Support of the distribution µ
	X " µ	The r.v. X follows the distribution µ
	µ b ν	Product measure of µ and ν
	δ x	Dirac mass at point x
	s F	Survival function for c.d.f. F , s F " 1	´F
	F ´1

  The general idea of statistical learning is to use a notion of empirical risk to approximate R. Introduce a sequence of n i.i.d. pairs with the same probability distribution as pX, Y q, i.e. D n "

	i"1 tpX i , Y i qu n	i.i.d.

  Section 2.3.1 presents basic concentration inequalities, that are required to study the terms ErZs and Z ´ErZs where Z " E n pGq. The convergence of ErZs results from the limited complexity of the proposed class of functions G, and we present the related mathematical formalization in Section 2.3.2. Finally, Section 2.3.3 exploits the results of Section 2.3.2 and the McDiarmid's inequality of Section 2.3.1 to conclude. It presents a first generalization bound.

  The union bound implies:

	P	" max gPG	|R n pgq ´Rpgq| ą t	*	" 1	´P #	ď
	Applying Corollary 2.6 gives:
		Ptmax

gPG ˆ|R n pgq ´Rpgq| ď t ˙+ , ě 1 ´ÿ gPG P t|R n pgq ´Rpgq| ď tu . gPG |R n pgq ´Rpgq| ď tu ď ÿ gPG Pt|R n pgq ´Rpgq| ď tu, ď 2N e ´2t 2 n , and inverting the bound gives the result.

  , . . . , Z n be n P N ˚i.i.d. real-valued r.v.'s with zero mean, and assume that |Z i | ď c a.s.

					, Corollary
	2.11). Its proof is featured in Boucheron et al. (2013) (Theorem 2.10). We recall the simplest
	formulation of the inequality here.
	Theorem 2.21 (Bernstein's inequality).
	Let Z 1 Let σ 2 :" VarpZ 1 q and s Z n :" p1{nq	ř n i"1 Z i . For any δ ą 0, we have that: w.p. ě 1 ´δ,
	s Z n ď	2c logp1{δq 3n	`c 2σ 2 logp1{δq n	.

  The distribution pF, ηq introduced in Example 2.1, and parameterized by a, satisfies the Mammen-Tsybakov noise condition with parameters B " 1 and a.

	Observe that Mammen's noise condition is void when a " 0. Also, it is implied with a " 1 by
	Massart's noise condition. Both noise assumptions imply a convenient bound on the variance
	of a function f P F. as a consequence of a second moment bound (a bound of the form VarpZq ď ErZ 2 s), combined with the decomposition of the excess risk presented in Eq. (2.1).
	Precisely, since:
		Varpf pX, Y qq ď ErItgpXq ‰ g ˚pX qus,	(2.10)
	and Rpgq ´Rpg ˚q " E r|2ηpXq ´1|ItgpXq ‰ g ˚pX qs, Massart's inequality implies:
	Varpf pX, Y qq ď	1 h	Er|2ηpXq ´1|ItgpXq ‰ g ˚pX qus "	1 h	pRpgq ´Rpg ˚qq .
	Similarly, the Mammen-Tsybakov noise assumption implies the following proposition.
	Proposition 2.26. Assume that Assumption 2.24 is true, then, for any f P F, there exists c that depends on only of B and a, s.t.
			Varpf pX, Y qq ď cpRpgq ´Rpg ˚qq a

  Assume that g ˚belongs to the proposed class, i.e. that g ˚P G. Combining Eq. (2.12) with the implications of the Mammen-Tsybakov noise assumption (Proposition 2.26) for the empirical risk minimizer (g " g n ) implies that: for any δ ą 0, w.p ě 1 ´δ,

	Rpg n q ´Rpg ˚q ď	2c logpN {δq 3n	`pRpg n q ´Rpg ˚qq a{2	c	2c logpN {δq n	,	(2.13)
	which is a fixed point inequality in Rpg n q ´Rpg ˚q. To derive an upper-bound on Rpg n q ´Rpg	˚q
	from Eq. (2.13), we use Lemma 2.27 below. It relies on a generalization of Mignotte (1992)
	(Theorem 4.2), which gives an upper bound on the maximum of the absolute values of the roots
	of a polynomial.
	Lemma 2.27. (Cucker and Smale, 2002, Lemma 7)
					(Section 5.2), but is more
	detailed here. The union bound of applications of Bernstein's inequality on each element of a
	finite family of N elements, gives that: for any δ ą 0, w.p ě 1 ´δ, we have simultaneously, that
	for any g P G that corresponds to f P F,
	Rpgq ´Rpg ˚q ď R n pgq ´Rn pg ˚q	`2c logpN {δq 3n	`c 2Varpf pX, Y qq logpN {δq n	.	(2.12)

  Introduce the family R of super-level sets of S, i.e. R :" tx P X | spxq ą tu ps,tqPSˆR , then Eq. (3.3) is equivalent to:

	classification. max RPR	GpRq s.t. HpRq ď α.
	Formally, it writes:
		max ps,tqPSˆR	s G s ptq s.t. s H s ptq ď α.	(3.3)
	A solution of Eq. (3.3) is written ps α , t α q. It satisfies ROCps α , αq ě ROCps, αq for any s P S and depends on the parameter α.
	While the formulation of Eq. (3.3) involves a scoring function s and threshold t, the quantities
	s G s ptq and s H s ptq are an evaluation of the classifier g : x Þ Ñ 2 ¨Itspxq ą tu ´1. Hence, Eq. (3.3) was
	first introduced as Neyman-Pearson classification due to its relation with the standard hypothesis
	testing framework, as explained below right before Proposition 3.3. We refer to Scott and
	Nowak (2005); Scott (2007); Rigollet and Tong (2011) for a detailed account of Neyman-Pearson

  .

	Then, we can define an estimator of the ROC curve as z ROC Hs,Gs , where:
	z ROC Hs,Gs : r0, 1s Ñ r0, 1s,
			α Þ Ñ 1 ´p H s ˝p G s p1 ´αq.
	Additionally, we introduce an empirical version of the pointwise ROC optimization problem
	introduced in Eq. (3.4), defined as:
	max RPR	p GpRq s.t. p HpRq ď α `φpn, δq,	(3.10)
	and a solution to Eq. (3.10) is written as p R α . Note the presence of a tolerance term φpn, δq ě 0
	in the constraint, whose function is to tolerate the variations of p HpRq around HpRq. Precisely,
	the decomposition p
	AUC Hs,Gs in Eq. (3.2), we Using the same i.i.d. sample of n observations D n as for the estimator z introduce an empirical approximation of the empirical distribution of the negatives and positives
	as p H and p G, respectively. Formally,
		p H :"	1 n ´ÿ Yi"´1	δ Xi and p G :"	1 n `ÿ Yi"`1	δ Xi ,
	where δ x is the Dirac distribution in x, i.e. δ x pAq " 1 if x P A and δ x pAq " 0 otherwise.
	To define an empirical ROC curve, we first introduce an estimator p H s of the distribution H s of
	spXq|Y " ´1 and an estimator p G s of the distribution G s of spXq|Y " `1, such that:
	p H s ptq :" p HpspXq ď tq "	1 n ´ÿ Yi"´1	ItspX i q ď tu,
	p G s ptq :" p GpspXq ď tq "	1 n `ÿ Yi"`1	ItspX i q ď tu.

  Proof. As presented in Chapter 3, precisely Proposition 3.5, the bilateral control of each risk R `, R ´implies guarantees for the constrained problem. Set p "

					ř K k"1 p 2 k and observe that we
	have: @n ą 1,
	sup sPS	p R ǹ psq ´R`p sq ď p ´1 2n ǹpn ´1q	´p `p´1 sup
	A generalization of Hoeffding's inequality for U-statistics (Serfling, 1980, Section 5.6, Theorem
	A), presented as Lemma 4.4 in Chapter 4, gives that, w.p. ě 1 ´δ,
		2n ǹpn ´1q	´p ă	c	logp2{δq n ´1 .

sPS

U ǹ psq ´ErU ǹ psqs .

  (in the preliminaries as Proposition 3.8 of Chapter 3) adapted to the case of similarity ranking. The fast rate bound stated in Theorem 5.5 then follows from the application of Talagrand's inequality (or Bernstein's inequality when S is of finite cardinality), following the steps of[START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (subsection 5.2) or Clémençon and Vayatis (2010) (Theorem 12).Since |Q s | ď 2, Bernstein's inequality combined with the union bound gives that, if S is of cardinality M : w.p. ě 1 ´δ, for all s P S,

	T n psq ď	4 logpM {δq 3n	`c 2Varpq s pX, Y qq logpM {δqq n	.	(5.21)
	Combining Eq. (5.21) and Eq. (5.20) give that: w.p. ě 1 ´δ, for all s P S
	∆ n psq ď	C 1 logp5M {δq n	`c 2Varpq s pX, Y qq logp5M {δqq n	.	(5.22)

  , we propose the following optimization program, with pxq `" maxp0, xq: max pw,bqPR d ˆR ´GpR αq ´p Gpw, bq ¯``´p Hpw, bq

	´α¯`s .t.	w	2 2	`b2 ď 1.	(7.6)

  ItY i ‰ gpX i qu denoting by ItEu the indicator function of any event E, and is taken over a class G of classifier candidates, with controlled complexity (e.g. of finite VC dimension), though supposed rich enough to yield a small bias error inf gPG Lpgq ´L˚, i.e. to include a reasonable approximation of the Bayes classifier g ˚. Theoretical results assessing the statistical performance of empirical risk minimizers are very well documented in the literature, see e.g.[START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF], and a wide collection of algorithmic approaches has been designed in order to solve possibly smoothed/convexified and/or penalized versions of the minimization problem (8.2).

	and is equal to
	L ˚" Lpg ˚q " 1	´E "	max 1ďkďK	η k pXq		.
	As the distribution P is unknown, a classifier must be built from the training dataset and from
	the perspective of statistical learning theory, the Empirical Risk Minimization (ERM) paradigm
	encourages us to replace the risk (8.1) by a statistical estimate p L n pgq, typically the empirical ř n version p1{nq i"1 consider solutions p g n of the optimization problem
	min gPG	p L n pgq,			(8.2)
	where the infimum

It is well-known that the minimum risk is attained by the Bayes classifier g ˚pxq " arg max kPt1, ..., Ku η k pxq,

  B 1´α {pp1 ´αq 1´α α α q. Denoting by F k,l " pp k F k `pl F l q{pp k `pl q the conditional distribution of X given that Y P tk, lu, observe that:

	P p g k,l pXq ‰ g k,l pXq | Y P tk, lu	(	" E X	" dF k,l dF	 ( g k,l pXq ‰ g k,l pXq pXq ˆI p	.	(8.26)
	Under Assumption 8.10, we almost-surely have:
		dF k,l dF	pXq ě	ε p k `pl	ě ε.
	Hence, from (8.25) and Lemma 8.11, we get that
	ε β	P p g k,l pXq ‰ g k,l pXq (	ď `Lk,l pp g k,l q ´Lk	,l ˘α	(8.27)
						ď 2 ˆinf gPG	L k,l pgq ´Lk	,l ˙α `rα n pδq,	(8.28)
	using Minkowski's inequality. Since

t. D n explicitly. Eq. (2.11) in Chapter 2 gives an equivalent formulation of 8.8, that writes as: P p g k,l pXq ‰ g k,l pXq | Y P tk, lu ( ď β `Lk,l pp g k,l q ´Lk ,l ˘α , (8.25) with β "

Table 8 .

 8 1: Top-k performance. The last column is the time to fit the model. test distributions, which is the topic of the next chapter.

	Dataset	Model	Top-1 Top-5 Fit time
	MNIST	LogReg OVO	0.924 0.943	0.995 0.997	50 min 40 min
	Fashion	LogReg	0.857	0.997	35 min
	-MNIST	OVO	0.863	0.997	60 min

  (see Corollary 5 therein) that for any s ˚in S Hs ˚ps ˚pX qq almost-surely. For any s candidate, setting Ψ s pxq " H s {G s pspxqq, one can define:

	˚,
		dF dF ´pX q "	G s
	Φ s px, yq "	p q	Ity " `1u	`1 p1 ´qqp1 ´p `pΨ s pspxqqq ´p	Ity " ´1u.

Table 9

 9

	.1 are plotted in

Table 9 .

 9 1: Optimal parameters θ ˚for different values of α, β.

Table 9

 9 .2.

				Model
		Database	Linear			MLP
		MNIST	7,850		315,625
		ImageNet 2,049,000 4,647,676
	P "	1 2	W	and P "	1 2	W 1	`1 2	W 1 .

.2: Number of parameters for each model. The weight decay or l2 penalization for the linear model and MLP model are written, respectively:

  copies of the random triplet pX, Y, Zq, we denote by n pzq the number of points in group z P t0, 1u, and by n

	pzq `(resp. n	pzq ´) the number of positive
	(resp. negative) points in this group. The empirical counterparts of H s pzq	and G pzq s are then given
	by, respectively:

  Relation with ROC-based fairness. In contrast to AUC-based fairness, ROC-based fairness allows to directly control the points α in Proposition 10.6 at which one obtains fair classifiers as it precisely consists in enforcing equality of ROC G p0q

	s ,G p1q s	and ROC H p0q s ,H	p1q s	at specific points.

  Proposition 10.8. Under Assumption 10.9, if there exists F P tH, Gu s.t. for every k P t1, . . . , m F u, |∆ F,α pkq

		psq| ď , then:
		F
		sup αPr0,1s	ˇˇ∆ F,α psq ˇˇď	`B	`b 2b	max kPt0,...,mu	ˇˇα F pk`1q	´αpkq F ˇˇ,
	with the convention α F " 0 and α p0q F pm F `1q	" 1.
	Proof. For any F P tH, Gu, note that:
		sup αPr0,1s |∆ F,α psq| ď max kPt0,...,mu	xPrα	sup pkq F ,α pk`1q F	s	|∆ F,α psq|.
	ROC F p0q s ,F	p1q s

  α psq is continuously differentiable, then α simultaneously satisfies, with the assumption that |∆ F,α pkq Learning with Pointwise ROC-based Fairness ConstraintsTo impose stricter fairness conditions, the ideal goal is to enforce the equality of the score distributions of the positives (resp. negatives) between the two groups, i.e. G This stronger functional criterion can be expressed in terms of ROC curves. For α P r0, 1s, consider the deviations between the positive (resp. negative) inter-group ROCs and the identity function:

	10.4.2 p0q s " G p1q s (resp.
	H s p0q	" H s ). ∆ G,α psq :" ROC G p0q p1q s ,G	p1q s	pαq	´α	`resp. ∆ H,α psq :" ROC H p0q s ,H	p1q s	pαq	´α˘.
										psq| ď for any k P t1, . . . , Ku:
										F
					˙					˙
		|∆ F,α psq| ď `ˆ1	`B b	α F ´α and |∆ F,α psq| ď `ˆ1 pkq	`B b	α	´αpk`1q F	,
		which implies that |∆ F,α psq| ď `p1 `B{bq α F pk`1q	´αpkq F {2.
		Finally, we have shown that:
		sup αPr0,1s |∆ F,α psq| ď	`B	`b 2b	max kPt0,...,mu	α F pk`1q	´αpkq F .

  and each of the terms is studied independently. The first term is already dealt with for Theorem 10.5, and the second and third terms have the same nature, hence we choose to focus on Dealing with sup s,αPSˆr0,1s |T 1 ps, αq|. Introduce the following functions, for any z P t0, 1u:ItY i " `1, Z i " z, spX i q ď tu and U pzq n,s ptq :" E Dealing with sup s,αPSˆr0,1s |T 2 ps, αq|. We recall that p G

	Since G p0q s is continuous and increasing, the inverse function theorem implies that pG p0q s q ´1 is
	differentiable. It follows that:
		d du	´Gp1q s ˝pG p0q s q ´1puq ¯" ´Gp1q s s	m F Ḡp0q s q ´1puq ¯1 ´pG p0q ¯1 ´pG p0q s q ´1puq ¯ď B b ,
	´LΛ psq ď z AUC Hs,Gs ´AUC Hs,Gs and the mean value inequality implies:	`ÿ F PtF,Gu	ÿ k"1	λ	pkq F	p ∆ F,α k psq ´|∆ F,α k psq| ,
	ď z AUC Hs,Gs ´AUC Hs,Gs |T 1 ps, αq| ď pB{bq ¨sup s,αPSˆr0,1s sup s,αPSˆr0,1s ´p U `ÿ F PtF,Gu p0q n,k p0q s ¯´1 ˜np0q m F ÿ k"1 λ pkq F `α n ¸´α . p ∆ F,α k psq ´∆F,α k psq .
	It follows that: Conditioned upon the Z i 's and Y i 's, the quantity
			˜˜n
	sup sPS	p L Λ psq ´LΛ psq ď sup sPS ? n	z AUC Hs,Gs ´AUC Hs,Gs `s λ H ¨sup s,αPSˆr0,1s n p0q `¸p U p0q s n,k p0q pαq ´α¸,	p ∆ H,α psq ´∆H,α psq
	`s λ G ¨sup is a standard empirical process, and it follows from Shorack and Wellner (1989) (page 86 therein), p ∆ G,α psq ´∆G,α psq , s,αPSˆr0,1s that:
	p ∆ G,α psq ´∆G,α psq. Note that: sup αPr0,1s ´p U p0q n,k p0q s ¯´1 ˜np0q `α n	¸´α " sup αPr0,1s	n n p0q `p U n,k p0q s p0q	pαq ´α .
	p ∆ G,α psq ´∆G,α psq, Similar arguments as those seen in Theorem 10.5 imply:
	" z ROC G p0q s ,G p1q s " " G p1q s ˝´G p0q pαq ´ROC G p0q s ,G s ¯´1 ´Gp1q s ˝´p G p0q p1q s pαq, s ¯´1 looooooooooooooooooooooooooooomooooooooooooooooooooooooooooon  p1 ´αq T1ps,αq sup s,αPSˆr0,1s |T 1 ps, αq| ď pB{bq ¨sup s,αPSˆr0,1s n n p0q p0q n,k p0q s `p U ď B bq 0 p 0 ¨ p0q n ǹ ´q0 p 0 `B bq 0 p 0	`"G p1q s pαq ´α , ˝´p G p0q s loooooooooooooooooooooooooooomoooooooooooooooooooooooooooon ¯´1 ´p G p1q s ´p G p0q ¯´1  p1 ´αq s T2ps,αq ¨sup s,αPSˆr0,1s p U p0q s n,k p0q pαq ´q0 p 0 α ,	.
	Hence: A standard learning bound (see Boucheron et al. (2005), Theorem 3.2 and 3.4 page 326-328
	sup s,αPSˆr0,1s therein) implies that: for any δ ą 0, n ą 0, w.p. ě 1 ´δ, p ∆ G,α psq ´∆G,α psq ď sup s,αPSˆr0,1s |T 1 ps, αq| `sup s,αPSˆr0,1s |T 2 ps, αq|,
	and we study each of these two terms independently. sup s,αPSˆr0,1s p U p0q n,k p0q s pαq ´U p0q n,k p0q s pαq ď C c V n	`c 2 logp2{δq n	,	(10.25)
	where C is an universal constant.
	p U pzq n,s ptq :" A union bound between Eq. (10.25) and a standard Hoeffding inequality for n 1 n n ÿ i"1 δ ą 0, n ą 1, w.p. ě 1 ´δ,	" U pzq p n,s ptq ı `gives: for any p0q ,
	then p G pzq s ptq " pn{n The properties of the generalized inverse of a composition of functions (see van der Vaart (2000), pzq `q ¨p U pzq n,s ptq and G pzq s ptq " p1{q z p z q ¨U pzq n,s ptq for any t P p0, T q. sup sPS |T 1 ps, αq| ď BC bq 0 p 0 c V n `3B bq 0 p 0 c logp4{δq 2n . (10.26)
	Lemma 21.1, page 304 therein) give, for any u P r0, 1s: pzq s ptq " pn{n	pzq `q ¨p U n,s ptq and G pzq	pzq s ptq "
	p1{q z p z q				´p G p0q s	¯´1	puq " ´p U p0q n,s	¯´1 ˜np0q `u n	¸.	(10.23)
	p0q s is increasing. Define k s " G p0q p0q s ˝s, for any t P p0, T q, The assumption on K implies that G we have:
						p U p0q n,s ptq " p U n,k p0q s	´Gp0q s ptq ¯.	(10.24)
	Combining Eq. (10.23) and Eq. (10.24), we have, for any u P r0, 1s:
			´p G p0q s	¯´1	puq " ´Gp0q s	¯´1	˝´p U n,k p0q s p0q	`u n ¯´1 ˜np0q	¸.

  Ity i " ´1, z i " zu ¨σpt ´spx i qq,Ity i " `1, z i " zu ¨σpt ´spx i qq,which can be respectively seen as relaxations of the false positive rate (i.e. s H

			3.3. First,
	we define an approximation of the quantities p H s , p pzq G pzq s on B N , for any z P t0, 1u, as:
	r H pzq s ptq "	1 N pzq ´N ÿ i"1
	r G pzq s ptq "	1 N pzq `N ÿ i"1
			pzq s ptq " 1	´Hpzq s ptq)
	and true positive rate (i.e. s G pzq s ptq " 1	´Gpzq s ptq) at threshold t and conditioned upon Z " z.
	For any F P tH, Gu, k P t1, . . . , m F u, we introduce a loss k F which gradients are meant to enforce
	the constraint | p ∆ F,α pkq F

Table 10 .

 10 2: Results on the test set, averaged over 100 runs (std. dev. are all smaller than 0.02).[START_REF] Donini | Empirical risk minimization under fairness constraints[END_REF], consists in classifying people described by a set of attributes as good or bad credit risks. The sensitive variable is the gender of the individual, i.e. male (Z " 0) or female (Z " 1). It contains 1,000 instances and we retain 30% of those for testing, and the rest for training/validation. • The Adult Income Dataset (Adult), featured in Zafar et al. (2019); Donini et al. (2018), is based on US census data and consists in predicting whether income exceeds $50K a year. The sensitive variable is the gender of the individual, i.e. male (Z " 1) or female (Z " 0). It contains 32.5K observations for training and validation, as well as 16.3K observations for testing. For simplicity, we removed the weights associated to each instance of the dataset. • The Compas Dataset (Compas), featured in Zehlike et al. (2017); Donini et al. (2018), consists in predicting recidivism of convicts in the US. The sensitive variable is the race of the individual, precisely Z " 1 if the individual is categorized as African-American and Z " 0 otherwise. It contains 9.4K observations, and we retain 20% of those for testing, and the rest for training/validation. • The Bank Marketing Dataset (Bank), featured in Zafar et al. (2019), consists in predicting whether a client will subscribe to a term deposit. The sensitive variable is the age of the individual: Z " 1 when the age is between 25 and 60 (which we refer to as "working age population") and Z " 0 otherwise. It contains 45K observations, of which we retain 20% for testing, and the rest for training/validation. For all of the datasets, we used one-hot encoding for any categorical variables. The number of training instances n `m, test instances n test and features d for each dataset is summarized in Table 10.3.

	Method		AUC-based fairness		ROC-based fairness
	Value of λ	λ " 0		λ ą 0		λ	pkq H " λ H ą 0
		AUC ∆AUC AUC ∆AUC	|∆ H,3{4 | AUC ∆AUC	|∆ H,3{4 |
	Example 10.11	0.79	0.28	0.73	0.00	-	-	-	-
	Example 10.12	0.80	0.02	0.80	0.02	0.38	0.75	0.06	0.00
	• The German Credit Dataset (German), featured in Zafar et al. (2019); Zehlike et al. (2017);
	Singh and Joachims (2019);

Table 10 .

 10 4: Parameters selected using the validation set for the runs on real data.Results for the dataset Compas. Compas is a recidivism prediction dataset, where the sensitive variable is Z " 1 if the individual is categorized as African-American and 0 otherwise. As being labeled positive (i.e., recidivist) is a disadvantage, we consider the AUC-based constraint in Eq. (10.6) to force the probabilities that a negative from a given group is mistakenly ranked higher than a positive to be the same across groups. While the scoring function learned without fairness constraint systematically makes more ranking errors for non-recidivist African-Americans, we can see that the AUC-constraint achieves its goal as it makes the area under ROC H p1q s ,Gs and ROC H p0q s ,Gs very similar. We can however see that slightly more of such errors are still made in the first quartile of the scores. As an alternative to AUC-based fairness, we thus configure our ROC-based fairness constraints to align the distributions of positives and negatives across both groups by penalizing solutions with high |∆ G,1{8 psq|, |∆ G,1{4 psq|, |∆ H,1{8 psq| and |∆ H,1{4 psq|.Results for the dataset Adult. We now turn to Adult, an income prediction dataset where Z denotes the gender (Z " 0 for woman) and a positive label indicates that the person makes over $50K/year. For this dataset, we plot ROC H p1q

	Parameters		Constraint
	Dataset Variable None AUC ROC
	German	λ λ reg	0 0.5	0.25 0.5	0.25 0.5
	Adult	λ λ reg	0 0.05	0.25 0.05	0.25 0.05
	Compas	λ λ reg	0 0.05	0.5 0.05	0.25 0.05
	Bank	λ λ reg	0 0.05	0.25 0.05	0.25 0.05

Table 10 .

 10 5: Results on test set. The strength of fairness constraints and regularization is chosen based on a validation set to obtain interesting trade-offs, as detailed in Section 10.5.3. who actually makes more, than the other way around. The learned score thus reproduces a common gender bias. To fix this, the appropriate notion of AUC-based fairness is Eq. (10.8): we can see that it successfully equates the area under ROC H p1q

	Measure			Dataset
	Constraint	Value	German Adult Compas Bank
		AUC		0.76	0.91	0.72	0.94
		∆AUC		0.07	0.16	0.20	0.13
	None	|∆ H,1{8 | |∆ H,1{4 |		0.01 0.20	0.31 0.36	0.26 0.32	0.09 0.18
		|∆ G,1{8 |		0.13	0.02	0.29	0.00
		|∆ G,1{4 |		0.20	0.06	0.29	0.04
		AUC		0.75	0.89	0.71	0.93
		∆AUC		0.05	0.02	0.00	0.05
	AUC-based	|∆ H,1{8 | |∆ H,1{4 |		0.05 0.08	0.09 0.17	0.06 0.03	0.03 0.11
		|∆ G,1{8 |		0.01	0.06	0.02	0.27
		|∆ G,1{4 |		0.02	0.14	0.06	0.37
		AUC		0.75	0.87	0.70	0.91
		∆AUC		0.07	0.07	0.05	0.14
	ROC-based	|∆ H,1{8 | |∆ H,1{4 |		0.03 0.07	0.06 0.01	0.01 0.02	0.03 0.05
		|∆ G,1{8 |		0.04	0.00	0.00	0.06
		|∆ G,1{4 |		0.01	0.02	0.00	0.21
	above a woman s ,G Note however that this comes at the cost of introducing a small bias against men in the top p0q s and ROC H p0q s ,G p1q . s
	scores. As can be seen from ROC H p0q s ,H	p1q s	and ROC G p0q s ,G	p1q s

  's as expected, which is more visible for the dashed lines (i.e. on training data). Finally, the ROC-based constraint induces as expected the equality of G

	well as that of H s p0q	and H s p1q	in the high score regime, as seen on the right plot. It implies
	that ROC H p1q s ,G	p0q s	and ROC H p0q s ,G	p1q s
						H p1q s ,G	p0q s	and ROC H p0q s ,G	p1q s	intersect and
	have similar AUCp0q s and G p1q s as

  . Additionally, the credit score of men with good or bad credit risk is in both cases stochastically larger than that of women of the same credit risk assessment (see ROC G p0q

	ROC H p0q s ,G p1q s	p0q s	and
			s ,G	p1q s	and
	ROC H p0q s ,H	p1q s

  La thèse est issue d'une collaboration entre la grande école Télécom Paris et la société IDEMIA. Le projet s'appuie sur un contrat CIFRE (Convention Industrielle de Formation par la REcherche), un type de contrat introduit en 1981 par le gouvernement français pour renforcer les liens entre les institutions de recherche et les entreprises privées. Le travail de recherche est donc supervisé par les deux parties, ce qui est rendu possible dans notre cas par des interactions fréquentes. ENSTA Paris, ENSAE Paris, Télécom Paris et Télécom SudParis. Sous les auspices de l'Institut, ils partagent leur expertise pour développer des programmes de formation d'excellence et de recherche de pointe. La recherche impliquée dans cette thèse a été effectuée au sein de l'équipe Signal, Statistiques et Apprentissage (S2A) du Laboratoire de Traitement et Communication de l'Information (LTCI). L'équipe de supervision académique était composée de Stephan Clémençon et Anne Sabourin, tous deux membres de l'équipe S2A, ainsi que d'Aurélien Bellet, chercheur à l'INRIA (Institut National de Recherche en Informatique et en Automatique).IDEMIA est une société leader en matière d'identification biométrique et de sécurité numérique. La société est une fusion des sociétés Morpho et Oberthur Technologies, effectuée en 2017. Oberthur technologies a été un acteur dominant dans les solutions de sécurité numérique pour le monde mobile, tandis que Morpho était considéré comme le leader mondial de l'identification biométrique. IDEMIA a pour objectif de faire converger les technologies développées pour le secteur public (par l'ancien Morpho) et pour le secteur privé (par Oberthur Technologies). Dans le secteur privé, les principaux clients de l'entreprise proviennent du secteur bancaire, des télécommunications et des objets connectés. La thèse a commencé en 2017 avec Safran Identité et Sécurité (anciennement Morpho) avant la fusion, lorsque Morpho était une filiale de la grande entreprise d'aéronautique et de défense Safran. Tout au long de la thèse, Stéphane Gentric a assumé la supervision continue de ce projet du côté industriel. Les responsables de l'équipe Advanced Machine Learning à IDEMIA: successivement Julien Bohné, Anouar Mellakh et Vincent Despiegel, ont contribué de manière significative à cette supervision.

	12.1 Contexte

Télécom Paris est l'un des principaux établissements publics français d'enseignement supérieur et de recherche en France, et est membre de l'Institut Mines-Télécom (IMT) et de l'Institut Polytechnique de Paris (IP Paris). L'IP Paris est un établissement public d'enseignement supérieur et de recherche qui regroupe cinq prestigieuses écoles d'ingénieur françaises: l' École Polytechnique,

Table 12 .

 12 1: Résumé des notations.

  Comme la projection de Hájek est une transformation qui réduit la variance d'une U -statistique(van der Vaart, 2000, Section 11), des hypothèses plus faibles impliquent la limite supérieure de la variance nécessaire pour des vitesses d'apprentissage rapides.Clémençon et al. (2008) contient cette utilisation des propriétés des U -statistiques dans l'objectif de dériver des vitesses de convergence rapides, mais ne l'a jamais appliqué à des problèmes comportant une contrainte aléatoire dans le programme d'optimisation.Illustration empirique des vitesses rapides. Notre travail contient également la première illustration expérimentale des vitesses rapides d'apprentissage, sur notre problème spécifique d'ordonnancement par similarité. Cette illustration repose sur la génération de données satisfaisant l'hypothèse de bruit pour différents paramètres de bruit a P p0, 1q, et d'une comparaison de leurs vitesses d'apprentissage empiriques. Nous avons choisi la distribution des données et la famille de propositions S afin que la similarité optimale s ˚soit connue et telle que le minimiseur empirique s n puisse être trouvé exactement. Pour ces raisons, nous avons défini S comme étant un decision stump (arbre de décision de profondeur 1) sur une transformation fixe des données.

  TreeRank apprend une fonction de score constante par morceaux s Dn , construite pour fournir une estimation adaptative et linéaire par morceaux de la courbe ROC optimale. Comme le suggèrent les solutions optimales de Eq. (12.1), la courbe ROC optimale ROC ˚est celle de la probabilité a posteriori η. TreeRank sépare récursivement l'espace d'entrée X et optimise de façon gloutonne l'AUC à chaque séparation, et forme ainsi un arbre binaire de profondeur D n basé sur des partitions imbriquées de l'espace d'entrée X . Sous des hypothèses spécifiques,Clémençon and Vayatis (2009) ont prouvé des bornes uniformes sur la norme sup entre la courbe ROC optimale et celle de s Dn lorsque D n "

	a logpnq, c'est-à-dire qu'avec une grande probabilité:
	sup	ROC s Dn pαq ´ROC ˚pαq ď expp´λ	a logpnqq,	(12.5)
	αPr0,1s
	où λ est une constante spécifiée par l'utilisateur.

  . Ces frameworks ont permis d'abstraire les aspects réseau et communication lors du développement d'algorithmes distribués. Ils ont ainsi facilité le déploiement des algorithmes distribués, mais limité le type d'opérations qui peuvent être réalisées efficacement, généralement afin de respecter certaines propriétés algorithmiques. Simultanément, Jordan (2013) a exhorté les statisticiens de guider les praticiens de l'apprentissage automatique à grande échelle, par l'étude des implications de la distribution sur l'estimation, notamment pour mettre en perspective les gains en temps de calcul avec les pertes potentielles en matière de précision statistique. Nos travaux abordent cette question, en proposant plusieurs estimateurs pour U -statistiques dans un environnement distribué et en comparant leurs variances. Dans ce contexte, nous proposons des compromis entre temps de calcul et variance.Cadre probabiliste. Soit deux échantillons indépendants eti.i.d. D n " tX 1 , . . . , X n u Ă X et Q m " tZ 1 , . . . , Z m u Ă Z contenant respectivement n P N et m P N éléments, et tels que D n et Q m peuventavoir des distributions différentes. La U -statistique complète à deux échantillons de noyau h : X ˆZ Ñ R associée à ces données s'écrit: B est un ensemble de B éléments sélectionnés aléatoirement dans l'ensemble de toutes les paires tpk, lq | pk, lq P t1, . . . , nu ˆt1, . . . , muu. Pour les données à grande échelle, les ensembles de données complets D n et Q m ne peuvent pas être stockés sur une seule machine, ce qui rend le calcul direct de Eq. (12.6) et Eq. (12.7) impossible. Dans ce contexte, l'approche standard consiste à distribuer les données sur un ensemble de N P N serveurs différents. Pour une moyenne standard, le simple calcul d'une moyenne des moyennes locales de chaque serveur donne le même estimateur que dans un contexte centralisé. Ce n'est pas aussi simple pour les U -statistiques, étant donné que sans communication réseau chaque serveur ne peut former des paires qu'avec l'échantillon local.Estimateurs distribués pour les U -statistiques. Nous introduisons tout d'abord deux estimateurs simples dans un contexte distribué qui ne nécessitent pas de communication sur le réseau: la moyenne U n,N de N U -statistiques complètes sur chaque échantillon local, et la moyenne U n,N,B de N U -statistiques incomplètes formées de B paires choisies aléatoirement dans chaque échantillon local. En utilisant la deuxième décomposition de Hoeffding, nous obtenons une expression analytique pour les variances de ces estimateurs, en suivant les mêmes étapes que[START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF] le fait pour U n . Leur expression montre que les statistiques U n,N et U n,N,B ont une précision limitée, c'est-à-dire une variance minimale, qui peut être largement supérieure à celle de U n pour un certain h et des distributions spécifiques de X 1 et Z 1 .

	U n phq :"	1 nm	n k"1 ÿ	m l"1 ÿ	hpX k , Z l q,	(12.6)

avec n " pn, mq. En revanche, la version incomplète de U n phq basée sur B paires s'écrit:

U B phq :" 1 B ÿ pk,lqPD B hpX k , Z l q,

(12.7)

où D

  s ptq :" P tspXq ď t | Y " ´1u et H pzq s ptq :" P tspXq ď t | Y " ´1, Z " zu , G s ptq :" P tspXq ď t | Y " `1u et G pzq s ptq :" P tspXq ď t | Y " `1, Z " zu . Bien que la courbe ROC sert usuellement à évaluer la performance en ordonnancement d'une fonction de score, c'est aussi un outil général pour évaluer les différences entre deux fonctions de répartition h et g sur R. Dans ce contexte, la courbe ROC est connue sous le nom de courbe probability-probability (PP plot) de h et g. L'aire sous la courbe ROC (AUC) est un résumé scalaire de la courbe ROC, et est omniprésent dans la littérature sur l'ordonnancement. L'AUC sert généralement à évaluer les performances des algorithmes d'ordonnancement bipartite (Clémençon et al., 2008). Formellement, les ROC et AUC entre les deux fonctions de répartitions h et g, s'écrivent: ROC h,g : α P r0, 1s Þ Ñ 1 ´g ˝h´1 p1 ´αq et AUC h,g :"

	ż 1
	ROC h,g pαq dα.
	0

  P r´1, 1s est un paramètre qui change au cours du processus d'apprentissage. Le paramètre c est modifié après un nombre fixé de n adapt itérations de l'algorithme du gradient, selon que la différence dans la contrainte d'AUC de l'Eq. (12.19) est évaluée comme positive ou négative sur un ensemble de données de validation.Limites des contraintes basées sur l'AUC. L'égalité entre deux AUC contraint les distributions concernées. Plus précisément, le théorème des accroissements finis montre qu'elle impose un point d'égalité pour les courbes ROC concernées. Cependant, ce point est inconnu a priori. De nombreuses applications -et en particulier la biométrie -se concentrent sur les performances du système pour de petits taux de faux positifs, c'est-à-dire des régions spécifiques de la courbe ROC. Imposer l'égalité des courbes ROC dans ces régions implique que des classifieurs obtenus par le seuillage du score satisfont un critère d'équité dans un contexte de classification. Pour faire en sorte que les courbes ROC soient égales dans une région spécifique, nous pouvons nous concentrer sur quelques points précis, comme usuellement en approximation numérique. Pour ces raisons, nous introduisons des contraintes basées sur les courbes ROC, qui imposent l'égalité de deux courbes ROC à des points précis.Apprentissage sous contrainte d'équité basée sur la ROC. Considérons les courbes ROC entre les négatifs et positifs de chaque groupe sensible, c'est-à-dire ROC H p0q

	r L λ psq :" Ć AUC Hs,Gs ´λ	¨c ´Ć AUC H p0q s ,G p0q s	´Ć AUC H p1q s ,G p1q s	¯,	(12.19)
	où c s ,H leur écart par rapport à la diagonale s'écrit:	p1q s	et ROC G p0q s ,G	p1q s	,
	∆ F,α psq :" ROC F p0q s ,F	p1q s	pαq ´α,

  Nous étendons nos garanties de généralisation aux contraintes d'équité basées sur la courbe ROC, à l'aide de bornes uniformes sur les courbes ROC. Pour prouver ce résultat, nous considérons un processus empirique indexé par la famille de points α P r0, 1s. Nous proposons également une stratégie d'optimisation analogue à celle utilisée pour Eq. (12.19). Notre stratégie comporte des paramètres de seuil, qui sont modifiés d'une manière similaire à c.Résultats expérimentaux. Nous fournissons des preuves empiriques solides de la pertinence notre approche. Précisément, nous présentons dans la figure 12.2 les compromis entre équité et précision obtenus avec nos méthodes.En conclusion, nous avons proposé de nouvelles approches pour aborder la question de l'équité algorithmique pour des problèmes d'ordonnancement. Tout d'abord, nous avons regroupé les contraintes basées sur l'AUC sous une seule définition générale. Ensuite, nous avons proposé des garanties théoriques, et une méthode pratique pour apprendre avec n'importe laquelle de ces contraintes à l'aide d'une descente de gradient. Nous avons souligné les limites des contraintes basées sur l'AUC, et avons proposé une approche basée sur la ROC qui correspond mieux aux conditions opérationnelles. Finalement, nous avons étendu nos garanties de généralisation et notre méthode pratique d'optimisation à notre nouvelle contrainte plus restrictive et plus flexible basée sur la courbe ROC.

A preorder on a set X is any reflexive and transitive binary relationship on X . A preorder is an order if, in addition, it is antisymmetrical.

Alternatively, for scalability purposes, one may instead work with their incomplete counterparts, namely (6.7) and (6.9) respectively.

When H is nonsmooth in θ, a subgradient may be used instead of the gradient.

https://keras.io/applications/

http://www.image-net.org/about-stats

http://image-net.org/download-API

http://www.image-net.org/about-stats

12.4. Ordonnancement par similarité

Remerciements

Télécom Paris is one of the top French public share their expertise to develop training programs of excellence and cuttingedge research. The research involved in this thesis was done within the Signal, Statistics and Learning (S2A) the S2A team, as well as Aurélien Bellet, researcher at

List of Publications

• Learning Fair Scoring Functions:

Fairness Definitions, Algorithms and Generalization Bounds for Bipartite Ranking.

Proof. Our proof shows that we can define a random permutation Σ P S K with the desired properties. To begin with, define Σ ´1p1q as Y . Next, given X and Σ ´1p1q " Y , draw Σ 1 as a BTLP model on the set I " t1, . . . , KuztΣ ´1p1qu with preference parameters η k pXq, k P I. For all r P t1, . . . , K ´1u, set Σ ´1pr `1q " Σ 1´1 prq and invert the permutation pΣ ´1p1q, . . . , Σ ´1pK qq to get the desired random permutation Σ.

The noteworthy fact that the probabilities related to the optimal pairwise comparisons Ptg k,l pXq " `1 | Y P tk, luu " η k pXq{pη k pXq `ηl pXqq are given by a BTLP model has been pointed out in [START_REF] Hastie | Classification by pairwise coupling[END_REF]. With the notations introduced in Lemma 8.5, we have in addition η k,l pXq : " P tΣpkq ă Σplq | Xu , " η k pXq{pη k pXq `ηl pXqq.

Eq. (8.10) can be interpreted as follows: the label ranking problem as defined in subsection 8.

for all k P t1, . . . , Ku. The OVO procedure precisely permits to construct such an empirical version, by replacing the optimal classifier g k,l in Eq. (8.12) by a minimizer of the empirical classification error. As shall be shown by the subsequent analysis, in spite of the very partial nature of the statistical information at disposal, the OVO approach permits to recover the optimal RMR rule σ X with high probability provided that pX, Σq fulfills (a possibly weakened version of) Assumption 8.4, combined with classic complexity conditions.

Remark 8.6. (On the noise condition) Attention should be paid to the fact that, when applied to the random pair pX, Σq defined in Lemma 8.5, Assumption 8.4 simply means that the classic Massart's noise condition is fulfilled for every binary classification subproblem, see [START_REF] Massart | Risk bounds for statistical learning[END_REF].

The OVO Approach to Label Ranking

Let G be a class of decision rules g : R q Ñ t´1, `1u. As stated in subsection 8.2.1, the OVO approach to multiclass classification is implemented as follows. For all k ă l, compute a minimizer p g k,l of the empirical risk: (8.13) over the class G, with Y k,l,i " ItY i " lu ´ItY i " ku for i P t1, . . . , nu and the convention that 0{0 " 0. We set p g l,k " ´p g k,l for k ă l by convention. Equipped with these `K 2 ˘classifiers, for any test (i.e. input and unlabeled) random variable X, the p g k,l pXq's define a complete directed graph G X with the K labels as vertices: @k ă l, l Ñ X k if p g k,l pXq " `1 and k Ñ X l otherwise. The analysis carried out in the next subsection shows that under appropriate noise conditions, with large probability, the random graph G X is acyclic, meaning that the complete binary relation Assumption 8.9. The class G is of finite VC dimension V ă `8. Assumption 8.10. There exists a constant ε ą 0, s.t. for all k ‰ l in t1, . . . , Ku and x P R q , η k pxq `ηl pxq ą ε.

Assumption 8.8 means that Assumption 8.4 is satisfied by the random pair pX, Σq defined in Lemma 8.5 in the case α " 1 (notice incidentally that it is void when α " 0) and reduces to the classic Mammen-Tsybakov noise condition in the binary case K " 2, see [START_REF] Mammen | Smooth discrimination analysis[END_REF]. Under those assumptions, one can derive Lemma 8.11 below, which provides guarantees for the pk, lq OVO classification problem.

Lemma 8.11. Suppose that Assumptions 8.8-8.10 are fulfilled. Let 1 ď k ă l ď K. Then, for all δ P p0, 1q, we have @n ě 1, w.p. ě 1 ´δ:

where, for all n ě n 0 pδ, α, , B, V q and δ P p0, 1q,

where C ą 0 is an universal constant and h " 2´2α {p16βq.

Proof. The result is a slight variant of that proved in Boucheron et al. (2005) (pages 342-346). The sole difference lies in the fact that fact that the empirical risk (and, consequently, its minimizer as well) is built from a random number of training observations (i.e. those with labels in tk, lu).

Here, we detail the proof for completion.

The derivation of fast learning speeds for general classes of functions relies on a sensible use of Talagrand's inequality that exploits the upper bound on the variance of the loss provided by the noise condition, combined with convergence bounds on Rademacher averages, see P. [START_REF] Bartlett | Localized rademacher complexities[END_REF].

To begin with, we define classes of functions that mirror the ones used by [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF].

Those are specifically introduced for the problem of associating elements of the sample to the label k or l, with k ă l, pk, lq P t1, . . . , Ku 2 any pair of labels. Given a label y P Y, it corresponds to solving binary classification for φ k,l pyq " Ity " ku ´Ity " lu for all of the concerned instances, i.e. those with labels k or l. For each binary classifier g in G, we introduce the cost function c k,l and the proportion of concerned instances h k,l , such that, for all x, y P R q ˆt1, . . . , Ku, c k,l px, yq :" Itgpxq ‰ φ k,l pyq, y P tk, luu and h k,l pyq :" Ity P tk, luu.

We denote by F k,l the set that contains the regrets of all functions g P G, formally:

Note that c k,l has an implicit dependence on a classifier g and that F k,l has an implicit dependence on G. With P as the expectation over X, Y and P n as the empirical measure, one can rewrite the risk L k,l and empirical risk p L k,l as:

Unlike c k,l the empirical mean P n h k,l does not depend on an element of g P G, thus minimizing p L k,l is the same problem as minimizing P n c k,l . The rest of the proof consists in using [START_REF] Boucheron | Theory of classification : a survey of some recent advances[END_REF] (section 5.3.5) to derive an upper bound of P f , with f P F k,l . Talagrand's inequality is useful because of an upper-bound on the variance of the elements in F k,l .

Assumption 8.8 induces a control on the variance of the elements of F k,l . [START_REF] Bousquet | Introduction to statistical learning theory[END_REF] (page 202) reviewed equivalent formulations of the noise assumption, in the case of binary classification. One of those formulations is similar to the following equation:

P gpXq ‰ g k,l pXq ( ď βpL k,l pgq ´Lk ,l q α , (8.16)