Hafiz who always showed me the right way in all areas (professional and social), I really say thank you. I thank my sister Chérifa, my second mother, always very close to me and always full of energy to make me feel the greatest joys of the world. To my brother Kemal, I thank you for the good moments of joy and laughter that we shared together, I hope that this thesis will inspire you to do even greater things. I would also like to thank all the aunts and uncles who have supported me during all these years. I dedicate this thesis especially to all this wonderful family. p p i=1 δ λi of the eigenvalues λ1 ≥ . . . ≥ λp of 1 n XX T , converges weakly, with probability one, towards a non-random distribution, known as the Marčenko-Pastur law and denoted by µ c ∞ 0 M P . If c ∞ 0 ∈ (0, 1), µ c ∞ 0 M P has the probability density function:

, then µ M P is a mixture of a point mass 0 and the pdf µ

.1). 0 1 2 3 0 0.5 eigenvalues λ spectral density µ(λ) δ 1 µ Σ µ M P 0 1 2 3 4 0 1 2 3 eigenvalues λ spectral density µ(λ) c ∞ 0 = 0.01

I would like to sincerely thank my thesis director Romain with whom I spent three wonderful years. He passed on to me all his passion for research and especially for the field of random matrix theory. I learned so much from him, whether it was in terms of work methodology, writing scientific articles or oral communication. He was very patient and always knew how to motivate me when almost all hope was lost. I won't forget those late nights or Sundays when Romain was correcting stuff on the article or those moments when a work session in his office could bring up lots of ideas for writing articles. His energy blew me away. He facilitated my professional insertion by introducing me to renowned researchers in machine learning and signal processing, and by inviting me to dinners with colleagues in random matrix theory.

I would also like to thank my co-supervisor Frédéric with whom I was able to work during my third year of thesis. He knew how to listen to me and adapt to my requests and my subjects of interest and I am grateful to him for that.

I would like to thank the reviewers of my thesis Alfred Hero and Mylène Maida. I was so honoured to have such experienced professors in the field of statistics as my reviewers and I was in no way disappointed as their remarks were beneficial for the improvement of the thesis manuscript. I particularly note all the suggestions made for the perspectives of this thesis which have inspired me a lot. I am very grateful to the members of the jury Alexandre Gramfort, Balazs Kegl and Remi Bardenet for their excellent questions and pertinent remarks on the thesis.

A special thanks to the Gipsa Lab where I spent years during my thesis and in particular to the CICS team which later became GAIA. I met some excellent researchers with whom I had book-clubs, interesting meetings and extraordinary collaborations. I would like to thank Florent Bouchard, Guillaume Ginholac, Steeve Zozor, Eric Moisan who have been exceptional collaborators.

To the doctoral students and friends with whom I spent these three years (Hafiz, Zhenyu, Xiaoyi, Mohamed, Lorenzo, Cosme, Charles, Cyprien, Tayeb, Minh Toan, Victor, Hugo, Roza, Yigit, Juliette, Julien, Fateme, Jeanne, Pierre, Imane, Ekkehard, Elaheh, Dawood, Cyril, Sara, Chika and so many others that I will not be able to mention due to lack of space), I would like to thank you all for the wonderful time you gave me. I have been incredibly lucky to be in such a close-knit group that brings joy to the work at all times.

I will finish with my family, who have been a great inspiration to me and on whom I have always been able to rely in moments of joy and pain. My father has always tried to show me the path of excellence, to the point of defending his thesis at the age of 60, to show us the example. My mother, what more can I say than cry, so much she shared my heart. She suffered like me in painful moments and felt the same joy as me in happy moments. She never failed to support me. Thank you Mum and Dad. To my brother 1.1 From curse to blessing of dimensionality

The "curse of dimensionality". Most signal processing and machine learning methods (e.g. statistical tests, parameter estimators, classification, regression, etc) are based on non-trivial functionals of n observed random signal vectors. Under the assumption that the number of available data n is overwhelmingly larger than their dimension p, some theoretical results and insights can be derived because a deterministic behavior sometimes arises when n → ∞, which simplifies the problem as exemplified by the celebrated law of large numbers and the central limit theorem. However, as already shown in the literature [START_REF] Wigner | On the distribution of roots of certain symmetric matrices[END_REF][START_REF] Marcenko | Distributions of eigenvalues for some sets of random matrices[END_REF], some long-held beliefs supported by classical results (and intuitions) break down when n, p are comparably large, a problem often referred to as "curse of dimensionality". Under the current big data paradigm which sees loads of data being produced, exchanged and stored, we constantly face the situation where not only the size n but also the dimension of the data p are large.

On the other hand, the last decade has seen a vast increase both in the diversity of applications to which machine learning is applied (transfer learning, privacy, fairness, etc) and to the practical deployment of these applications. Therefore, machine learning is no longer just the engine behind ad placement and spam filters: it is now used to filter loan applicants [START_REF] Fernández | Artificial intelligence in financial services[END_REF], deploy police officers [START_REF] Rudin | Predictive policing using machine learning to detect patterns of crime[END_REF], make decision in criminal justice [START_REF] Berk | Machine learning forecasts of risk to inform sentencing decisions[END_REF], etc making rapid headway into socio-technical systems. Algorithmic biases and misunderstood algorithms are one of the biggest risks of failure because it compromises the very purpose of machine learning since there has been heightened public concern about the impact of digital technology on society.

Random Matrix Theory as a solution. Understanding the resulting impact of popular statistical learning methods when n and p are both large and comparable is becoming a growing research concern in modern statistic. Despite the expected challenges and difficulties in analyzing statistical methods in this high-dimensional setting, recent evidences, which we will further develop in the present manuscript, suggest that Random Matrix Theory (Random Matrix Theory (RMT)) provides the necessary tools to handle such a framework. The sample covariance matrix, being a dominant object of study in most of the multivariate analyses, is a suitable object to illustrate this "curse of dimensionality" paradigm. Let us consider a p-dimensional random vector x with zeromean E[x] = 0 and covariance Σ = E[xx T ]. Many topics in multivariate analysis (e.g., principal component analysis, factor analysis, multidimensional scaling, etc) deal with the study of functionals (trace, log determinant, etc) or spectral properties (eigenvalues, eigenvectors) of Σ.

Typically, Σ is unknown, and so has to be estimated using a sample of data. Given a sequence of independent random vectors X = [x 1 , . . . , x n ] drawn from the same distribution as x, the usual estimate of the population covariance matrix Σ is the sample covariance matrix Σ defined as:

Σ = 1 n n i=1 x i x T i = 1 n XX T .
For fixed p and n → ∞, the sample covariance matrix is a consistent estimator for the population covariance matrix. If we denote λ 1 ≥ . . . ≥ λ p the eigenvalues of Σ and λ1 ≥ . . . ≥ λp the eigenvalues of Σ, Anderson et al. (1958, Theorem 13.5.1) proves more specifically, that if x ∼ N(0, Σ), for 1 ≤ j ≤ p, the sample eigenvalues λj 's are asymptotically distributed according to:

√ n( λj -λ j ) → N(0, 2λ 2 j ) as n → ∞.

(1.1)

This result shows that when n → ∞, p fixed, the j-th sample eigenvalue λj is a consistent estimator of the j-th population eigenvalue λ j . However, this result doesn't apply when p, n are both large. A typical question would be to study the eigenvalue distribution of Σ in order to quantify its "deviation" from the eigenvalue distribution of the true covariance matrix Σ in the large dimensional setting of n, p both large.

The first result on the spectral behavior of sample covariance matrices is due to the seminal work of Marčenko and Pastur in 1967 [START_REF] Marcenko | Distributions of eigenvalues for some sets of random matrices[END_REF] where they obtained a self-consistent equation for the spectrum of Σ given Σ = I p as p, n go to infinity.

Theorem 1. Suppose that X is a p × n matrix with i.i.d. real-or complex-valued entries with mean 0 and variance 1. Then, as n, p → ∞ such that p/n ≡ c 0 → c ∞ 0 , the empirical spectral measure µ Σ = 1 The heuristic behind this phenomenon is as follows. When the sample size n is very large, each individual coefficient of the population covariance matrix Σ can be estimated with negligible errors. But if p is also large and of the order of n, as is often the case in many situations, the sample estimator Σ becomes inconsistent. More specifically, the large number of simultaneous noisy coefficients of the sample covariance matrix creates important systematic errors in the computation of the eigenvalues of the matrix.

The Marčenko-Pastur result had a tremendous impact on the understanding of the "curse of dimensionality". This observation is the very essence of many applications in principal component analysis [START_REF] Johnstone | On the distribution of the largest eigenvalue in principal components analysis[END_REF], sphericity test [START_REF] Shou-Cheng Yuan | Sphericity and identity test for high-dimensional covariance matrix using random matrix theory[END_REF], statistical inference [START_REF] Mestre | Modified Subspace Algorithms for DoA Estimation With Large Arrays[END_REF], etc.

To illustrate this "curse of dimensionality" paradigm, we present one of the important consequences of this deviation in the study of the sample generalized variance log(| Σ|) appearing in many statistical methods (Quadratic Discriminant Analysis [START_REF] Tharwat | Linear vs. quadratic discriminant analysis classifier: a tutorial[END_REF], hypothesis tests in multivariate statistic [START_REF] Wilbur | An introduction to multivariate statistical analysis[END_REF], differential entropy in probability and information theory (Srivastava & Gupta, 2008), etc). These aspects are closely related to one of the topics covered in this thesis (Chapter 3).

Implications on a hypothesis test problem. An important statistic in multivariate analysis is the sample generalized variance:

Ln = log(| Σ|) = p j=1 log λj .
When the dimension size p is fixed, for Σ = I p , λj → 1 almost surely, as n → ∞ and thus Ln → 0. Further, by taking a Taylor expansion of log(1 + x), one can show from (1.1) that n p Ln → N(0, 2). This suggests the possibility that Ln remains asymptotically Gaussian for large p provided that p = O(n). However, when p/n ≡ c 0 → c ∞ 0 ∈ (0, 1) as n → ∞, using results on the limiting spectral distribution of Theorem 1, we have (see the computation of the integral in [START_REF] Zhidong | Clt for linear spectral statistics of large-dimensional sample covariance matrices[END_REF][Section 5]):

1 p Ln → λ + λ - log(x) 2πıxc ∞ 0 (λ + -x)(λ --x)dx ≡ d(c ∞ 0 ) = c ∞ 0 -1 c ∞ 0 log(1 -c ∞ 0 ) -1 < 0
where λ -= (1 -c ∞ 0 ) 2 and λ + = (1 + c ∞ 0 ) 2 . This shows that almost surely

n p Ln ∼ d(c ∞ 0 ) √ pn → -∞.
Thus the classical estimate n p Ln is biased and will induce serious issues when implemented in hypothesis test problem.

The necessity of a new set of tools. Another problem of the classical regime (p fixed, n → ∞) concerns the intricate nature of the random objects of interest in machine learning and signal processing algorithms which has sometimes limited the capability of learning theory in this regime to explain and predict the properties and the behavior of these algorithms. We illustrate this with the ridge regression problem for classification. Suppose a statistician observes n training examples (x i , y i ) ∈ R p × Y, and wants to find a rule for predicting y on future unlabeled draws x. In other words, the statistician seeks a function h : R p → Y, h(x) = g(ω T x) for which E [ (y, h(x))] is small where (., .) is a loss function; in regression Y = R and in binary classification Y ∈ {-1, 1}. Such prediction problems lie at the heart of several scientific and industrial endeavors. The ridge regression problem consists in taking (y, h(x)) = y -h(x) 2 . Using a linear function g(x) = x and normalizing the data matrix X = [x 1 , . . . , x n ] by √ n1 , the regression parameter vector ω is chosen in order to minimize the residual error in the training dataset X as:

min ω y - X T √ n ω 2 + λ ω 2 (1.2)
where y = [y 1 , . . . , y n ] T ∈ R n and λ is the regularization parameter which controls the complexity of the model. Taking the derivative with respect to ω and setting it to zero, we get 1 n XX T ω -1 √ n Xy + λω = 0.

(1.3) Therefore the optimal regression parameter vector can be computed by

ω = 1 n XX T + λI p -1 X √ n y.
(1.4)

For a new test data x, the decision is based on the sign of the test score :

g(x) = ω T x √ n = 1 n x T 1 n XX T + λI p -1
Xy.

(1.5)

Even under a Gaussian mixture model for the data matrix X, to the best of our knowledge, there exist few theoretical works [START_REF] Tsigler | Benign overfitting in ridge regression[END_REF] in the classical regime that can study the statistical behavior of the random quantity g(x) since it involves complex dependency in X in particular arising in the inverse matrix 1 n XX T + λI p -1 . In Chapter 2, we will see that this object is natural and classical to study under the large dimensional setting of p and n both large. The quantity 1 n XX T + λI p -1 called the resolvent of the matrix 1 n XX T is at the heart of Random Matrix Theory (RMT) tools and will be discussed at length in this thesis.

Summarizing the "curse of dimensionality" paradigm and moving towards the "blessing of dimensionality" through RMT. This section discusses two problems induced by classical asymptotic in multivariate analysis:

1. Tools from Multivariate Analysis in classical statistic are not always sufficient to deal with the increasingly complex random objects appearing in machine learning and signal processing. This has been illustrated through the popular and simple ridge regression problem.

2. For the few methods that can be analyzed, classical asymptotic approximations based on the "fixed dimension, large sample size" regime are inadequate in capturing the effects of dimensionality when the data dimension p is not small compared to the sample size. The spectral behavior of the sample covariance matrix illustrates this phenomenon with consequences shown on the study of the sample generalized variance.

The "curse of dimensionality" initially introduced by Bellman [START_REF] Bellman | Dynamic programming[END_REF] in relation to complications occurring in dynamic programming has now become a common name for issues of theoretical nature arising in high dimensions. The recent advances in large dimensional random matrix theory have raised much interest for problems in statistic and signal processing under the assumption of large but similar population dimension p and sample size n. The "curse of dimensionality" can then be turned into the blessing of dimensionality using Random Matrix Theory as a consequence of the measure concentration phenomena arising for high dimensional data. Therefore, large dimensional setting should not be feared or avoided: it just has to be used properly.

More specifically, we will show that by being a little too "stubborn" to assume n p even when it is not true, we can generate dramatic errors. Furthermore, one may wonder for which value of n, p, we start observing the curse of dimensionality. In fact, the n, p 1 regime already occurs when n, p are rather small. By exploiting both randomness in p and n, the speed of convergence of many functionals discussed in this manuscript can be as high as O(1/ √ pn) (unlike the classical Central Limit Theorem (CLT) which goes in O(1/ √ n)). For example, one could believe that n = 100p with p 1 is a "classical" regime, when in fact the RMT tools explain the phenomena better even in this case, and following the n p path to obtain intuitions when n, p 1 is a bad strategy. The curse of dimensionality remains in fact true: the intuitions coming from n p are simply wrong; one has to wipe the slate clean, start on a radically different approach, which is what RMT offers. A major consequence is that one will get in this thesis new "intuitions" which, at first sight, are in fact rather very counter-intuitive.

In this thesis, we leverage the capability of random matrix theory to overcome the technical difficulties involved in recent machine learning algorithms and to deeply understand limitations and possible corrections of such "large p, large n" systems. Specifically, we will use the opportunity offered by random matrix tools to understand and improve two problems of great interest in machine learning and signal processing (distance between covariance matrices and Multi-Task Learning).

Before delving into the technical details, we next motivate the interest into these two problems starting with the estimation of distances between covariance matrices.

The blessing of dimensionality applied to the estimation of the distance between covariance matrices problem

Motivation of covariance matrix distance estimation problem. Similarities between covariance matrices are objects of interest for many engineering applications, among which machine learning problems (for instance, covariance-based data clustering regularly used in synthetic aperture radar, hyperspectral imaging [START_REF] Chang | Hyperspectral imaging: techniques for spectral detection and classification[END_REF], or EEG datasets [START_REF] Richiardi | Machine learning with brain graphs: predictive modeling approaches for functional imaging in systems neuroscience[END_REF]), dimensionality reduction [START_REF] Michael | Dimensionality reduction on statistical manifolds[END_REF], portfoliooptimization and asset clustering in finance [START_REF] Tola | Cluster analysis for portfolio optimization[END_REF], etc.

State-of-the-art estimation procedure. Depending on context and application, various metrics are available in the literature to compare semi-definite positive matrices (the Frobenius norm, the Fisher Information metric [START_REF] Sueli Ir Costa | Fisher information distance: a geometrical reading[END_REF], the Bhattacharyya distance [START_REF] Bhattacharyya | On a measure of divergence between two statistical populations defined by their probability distributions[END_REF], the Rényi or Kullback-Leibler divergence, the Wasserstein distance, etc.). If we define by Σ 1 and Σ 2 two covariance matrices of size p × p, all these distances, which we will denote generically D(Σ 1 , Σ 2 ), can be expressed as functionals of the eigenvalues of the matrix Σ -1 1 Σ 2 (Fisher distance, Bhattacharyya distance, Kullback-Leibler divergence, or Rényi divergence between centered Gaussians) or eigenvalues of the matrix Σ 1 Σ 2 (Wasserstein distance between two centered Gaussians). Assuming that the number of samples n 1 and n 2 of data having Σ 1 and Σ 2 for covariance is very large compared to p, the law of large numbers guarantees that D( Σ1 , Σ2 ) is a consistent estimator for D(Σ 1 , Σ 2 ) with Σa = 1 Inconsistency of the classical estimate. However, the classical estimator is strongly biased when n 1 , n 2 ∼ p as shown in the middle column of

Σ 2 with [Σ -1 2 1 Σ 2 Σ -1 2 1 ] ij = .3 |i-j| , x (a)
i ∼ N(0, Σ a ); n 1 = 1024 and n 2 = 2048 for different values of p. Averaged over 10 000 trials. random matrix theory, this thesis proposes a general formula for a " universal " distance estimator D(Σ 1 , Σ 2 ) which is consistent within the limit where p, n 1 , n 2 → ∞ with p/n 1 → c 1 > 0 and p/n 2 → c 2 > 0, the estimated outputs of which are displayed in the rightmost column of Table 1.1. These aspects are investigated in Chapter 3.

Covariance matrices are at the heart of most statistical and machine learning methods and therefore knowing how to manage, estimate and understand covariance functions is central in Machine Learning (ML) since it allows for the efficient use of estimators when we have to manage several covariance matrices. These covariance matrix models, one way or another, are indirectly exploited in many machine learning algorithms. Large dimensional statistics have thus naturally provided new results to better understand, analyze and improve these algorithms, yet so far for "bottom of the shelf" algorithms (classical Support Vector Machine (SVM) (Liao & Couillet, 2019), spectral clustering [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], graph-based Semi-Supervised Learning (SSL) [START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF]). The richer and more complex methods such as multi-task learning, transfer learning, learning with fairness, privacy and safety in machine learning, etc which involve multiple biases that are difficult to trace (too many parameters, too much heterogeneity in the data, etc.) are finally those that can gain the most from exploiting the RMT, which will identify these biases. We show that this is indeed the case, that there are multiple biases especially in the case of Multi-Task Learning, but that the RMT cleans them up one by one, while maintaining a remarkable algorithmic simplicity.

The blessing of dimensionality applied to Multi-Task Learning

From single to multiple task learning. The most advanced supervised machine learning algorithms generally require a large number of labeled training samples to achieve high levels of accuracy. Deep learning models are prototypical in their sometimes requiring millions of labeled samples for efficient training. In many (if not most) applications (say, for instance, medical imaging (Abdullah-Al-Zubaer [START_REF] Imran | Semi-supervised multi-task learning with chest x-ray images[END_REF][START_REF] Imran | Partly supervised multitask learning[END_REF]), this is often too demanding, labeled samples being hard to collect. MTL [START_REF] Caruana | Multitask learning[END_REF][START_REF] Zhang | An overview of multi-task learning[END_REF], 2021), an offspring of which is better known as transfer learning, provides a potent workaround by appending the available small training dataset of interest with additional somewhat similar datasets on which similar (classification or regression) tasks can be performed; the additional data possibly being of a different nature, MTL effectively solves multiple tasks in parallel while exploiting task relatedness to enforce collaborative task learning.

How does MTL work? Precisely, multi-task learning simultaneously solves multiple related tasks and introduces shared hyperparameters or feature space, optimized to improve the performance of the individual tasks. The crux of the various multi-task learning algorithms lies in the means to both enforce and, most importantly, evaluate task relatedness: this is in general highly non-trivial as this implies to theoretically understand what common features of the parallel datasets can be adequately exploited by the MTL algorithm -the latter generally deriving from a classical single-task algorithm (such as a mere support vector machine). Several heuristics have been proposed which may be split into two groups: parameter-based versus feature-based MTL. In the parameter-based MTL approach, the tasks are assumed to share some common hyperparameters [START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF]) (e.g., the hyperplanes separating each class in a support vector machine flavor) or that these hyperparameters have a common prior distribution [START_REF] Zhang | A convex formulation for learning task relationships in multi-task learning[END_REF], 2014). Classical learning mechanisms (such as support vector machines (SVM), logistic regression, etc.) can then be appropriately adapted and turned into a multi-task version by enforcing these parameter relatedness assumptions. In this context, [START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF][START_REF] Parameswaran | Large margin multi-task metric learning[END_REF] respectively adapt the SVM, least square-Support Vector Machine (SVM) (LS-SVM), and large margin nearest neighbor (LMNN) methods into a MTL paradigm. In the featurebased MTL approach, the tasks data are instead assumed to share a low-dimensional common representation. In this context, most of the works aim at determining a mapping of the ambient data space into a low-dimensional subspace (through sparse coding, deep neural network embeddings, principal component analysis, etc.) in which the tasks have high similarity [START_REF] Argyriou | Convex multi-task feature learning[END_REF][START_REF] Maurer | Sparse coding for multitask and transfer learning[END_REF][START_REF] Zhang | Deep model based transfer and multi-task learning for biological image analysis[END_REF][START_REF] Sinno Jialin Pan | Domain adaptation via transfer component analysis[END_REF]; other works simply use a feature selection method by merely extracting a subset of the original feature space [START_REF] Obozinski | Multi-task feature selection[END_REF][START_REF] Wang | Safe screening for multi-task feature learning with multiple data matrices[END_REF][START_REF] Gong | Multi-stage multi-task feature learning[END_REF].

The negative transfer plague. The strong underlying limitation of all these methods is their lack of theoretical tractability: as a result, many of the biases inherent to the base methods (SVM, LS-SVM, deep nets) are exacerbated in a multi-task setting. As a striking consequence, many of these heuristic MTL algorithms suffer from negative transfer, which corresponds to scenarios where the multi-task setup performs worse than a single-task approach [START_REF] Michael T Rosenstein | To transfer or not to transfer[END_REF][START_REF] Long | Transfer learning with graph co-regularization[END_REF]; this is particularly the case when task relatedness is weaker than assumed so that the MTL method enforces fictitious similarities, thereby inducing strong biases.

A large dimensional analysis to redesign MTL. In this thesis, we focus on a very elementary (yet, as we shall see, already quite powerful) LS-SVM-based MTL approach and provide a thorough theoretical analysis from which we manage to automatically discard the negative transfer limitation. Specifically, placing ourselves under a large dimensional data setting, we exploit modern tools from large dimensional statistics (here random matrix theory) to theoretically investigate the key components that determine whether tasks interfere constructively or destructively under the MTL framework. This analysis provides powerful insight into the inner workings of the method and allows for a fundamental adaptation of the method which provably avoids all sorts of biases and most importantly discards the problem of negative transfer altogether. Methodologically, in its simplest approach, MTL algorithms can be obtained from a mere extension of support vector machines (SVM), accounting for more than one task. That is, instead of finding the hyperplane (through its normal vector ω) best separating the two classes of a unique dataset, [START_REF] Evgeniou | Regularized multi-task learning[END_REF] proposes to produce best separating hyperplanes (or normal vectors) ω 1 , . . . , ω k for each pair of data classes of k tasks, with the additional constraint that the normal vectors take the form ω i = ω 0 + v i for some common vector w 0 and dedicated vectors v i . The amplitude of the vectors v i is controlled (through an additional hyperparameter) to enforce or relax task relatedness. We study this approach in chapter 4.

The conclusions drawn in this chapter thus allow for an optimal use of MTL LS-SVM with performance-maximizing hyperparameters and strong theoretical guarantees. As such, the work performed in Chapter 4 offers through MTL LS-SVM a viable fullycontrolled (even better performing) alternative to state-of-the-art MTL.

Outline and contributions of the thesis

The classical asymptotic based on "n p" hypothesis is not just "incorrect", it also induces terrible biases that completely destroy the functioning of the algorithms; the thesis proposes to shed light on this by showing the dramatic consequences that these biases can induce. Furthermore, independently of its correctness or not, the n p hypothesis is often not enough to understand the behavior of the algorithms; the thesis shows that, paradoxically, the n, p → ∞ hypothesis gives rise to analytically accessible expressions; the only difficulty is technical: it requires mastering the RMT tools and forgetting our reflexes and small dimensional biases. Particularly, the thesis is based on the fact that when p, n → ∞, the statistics of the functionals of interest of the data (scores, performances, thresholds) depend only on first and second-order statistics (means and covariance matrices); this makes in particular the covariance matrices extremely rich objects that need to be understood.

This thesis first shows how classical estimates can destroy algorithms by introducing biases that are difficult to clean, whereas a consistent RMT estimation of the functionals of interest avoids biases. Furthermore, we illustrate how classical biases induced in "'bottom of the shelf" algorithms such as Support Vector Machine, Semi-Supervised Learning, etc are exacerbated in more involved algorithms like multi-task and transfer learning schemes. To that end:

• In Chapter 2, we introduce the necessary tools from Random Matrix Theory to grasp the technical ideas required for understanding the thesis. To that end, we consider two applications close to Chapters 3 and 4 of this thesis. In particular, we will look at how RMT works through simple examples that are closely related to the specific problems of this thesis. These examples will illustrate very simply the key problems evoked in the introductory chapter, in particular the curse of dimensionality and its cure through the RMT. These examples namely the spectral statistic estimation of the population covariance matrix and the asymptotic of ridge regression in the context of classification require to introduce some basic notions of RMT in particular the generalized Marčenko Pastur law which will begin the chapter.

• After having illustrated by the spectral statistical estimation of the population covariance matrix the limits of the classical asymptotic (n p) and provided a consistent estimation by RMT, we will be able to introduce in the chapter 3, the problem of estimating the distance between covariance matrices while providing consistent estimators for these distances. An application to clustering of covariancebased dataset as well as an application to the covariance matrix estimation problem are provided. This chapter is the result of several contributions that started with the estimation of the most usual covariance matrices distances (Fisher distance, Battacharrya distance, Kullback-Leibler divergence for Gaussian). These distances can be expressed as eigenvalues functionals of the Fisher matrix Σ -1 1 Σ 2 for two population covariance matrices Σ 1 and Σ 2 to compare. This leads to the first contributions of this project: Having a generic framework to consistently estimate all distances between covariance matrices, we develop a framework on the covariance matrix estimation itself based on an optimization problem involving distances between covariance matrices. Concretely, the estimation procedure consists in (i) writing the covariance matrix Σ as the solution to arg min M 0 D(M, Σ) for a wide range of metrics D (Fisher, Batthacharyya, Stein's loss, Wasserstein, etc.), (ii) based on [START_REF] Couillet | Random matriximproved estimation of covariance matrix distances[END_REF], using the fact that D(M, Σ) -D(M, X) → 0 for some consistent estimator D, valid for all deterministic M and samples X = [x 1 , . . . , x n ] ∈ R p×n having zero mean and covariance Σ, and (iii) proceeding to a gradient descent on D rather than on the unknown D itself. This forms the basis of the following contribution and can be seen as an application of the two previous contributions:

Journals: 1. Romain
Conferences: One of the major limitations of the three previous works was that they were only valid when the number of samples was larger than the feature size, which was a real handicap for real applications. We therefore analyzed and understood this important problem and proposed solutions to solve it. This work has been done in the following contribution.

1. Malik

Conferences:

1. Malik • Having a generic framework for consistent estimates of distances between covariance matrices, we show in Chapter 4 how problems as complex as multi-task learning also depend on mean and covariance functionals that can be estimated consistently. But beyond that, the theoretical analysis highlights deep insights into the inner working of these algorithms and opens the way to improvements of the method. The project starts by deriving and analyzing a first simple algorithm based on the ridge regression problem. We provide in the following contribution a first asymptotic behavior of the latter. The main contributions of this work were technical to show that RMT can not only handle simple algorithms like spectral clustering, SVM,

• Chapter 5 lists several perspectives following the work performed in this thesis.

Chapter 2

Basics of Random Matrix Theory The goal of this chapter is to introduce the necessary tools required to tackle the covariance matrix distance estimation problem and the Multi-Task Learning framework. In particular, these make respectively use of linear spectral statistic (discussed in Section 2.2) and of deterministic equivalent (discussed in Section 2.3) of random matrices. But they both rely on a fundamental result in Random Matrix Theory (the generalized Marčenko-Pastur law) introduced in Section 2.1 of this chapter. Two simple examples (estimation of the population generalized variance and ridge regression problem) are used to illustrate the utility of the linear spectral statistic and of the deterministic equivalent.

The interest of looking at two simple and classical problems of statistics is twofold: (i) it allows to illustrate on concrete examples the severe limitations of the classical asymptotic (n p) highlighted in an abstract way in the introduction and to show how RMT allows to understand and correct them; (ii) these problems being simpler, they allow to highlight the main tools which will be used in this thesis. These examples have not been chosen randomly, especially as they are closely related to the problems of interest in Chapters 3 and 4.

Large dimensional spectral behavior of the sample covariance matrix

Setting the stage. Covariance matrices are ubiquitous in many machine learning and signal processing methods. They are used in several classical methods such as Principal Component Analysis (PCA), Canonical Correlation Analysis (CCA), etc. More specifically, they are at the heart of the objects that characterize the asymptotic performance of the algorithms studied in this thesis. The ability to understand the properties of the covariance matrix Σ that is typically unknown depends on the quality of the estimator.

As mentioned in the introduction, the standard maximum likelihood estimator is the sample covariance matrix Σ. A thorough understanding of the behavior and limitations of the methods developed in this thesis requires a good understanding of the spectral properties of Σ.

There is a large body of work concerned with the limiting behavior of the eigenvalues of the sample covariance matrix Σ when p and n both go to ∞; it constitutes an important subset of Random Matrix Theory. The goal of this section is to introduce a fundamental result, the Marčenko-Pastur equation, that relates the asymptotic behavior of the eigenvalues of the sample covariance matrix to that of the population covariance matrix in the "large n, large p" asymptotic setting.

One of the practical uses of the asymptotic results in RMT is their universality with respect to the distribution of the matrix entries. This makes Marčenko-Pastur's result even more interesting for this thesis1 than some results such as the Wishart distribution [START_REF] Wishart | The generalized product moment distribution in samples from a normal multivariate population[END_REF] which characterizes the joint distribution of the eigenvalues of the sample covariance matrix Σ in the case of Gaussian data. Note that most of the methods of interest in this thesis are function of the global behavior of the spectrum of the covariance matrix, therefore at the difference of other results on the spectral behavior of the sample covariance matrix Σ (Wishart distribution (Wishart, 1928) for the joint distribution of the eigenvalues of Σ, Tracy Widow law [START_REF] Tracy | On orthogonal and symplectic matrix ensembles[END_REF] for the behavior of the largest eigenvalue of Σ, etc), the Marčenko Pastur law (describing the global behavior of the limiting eigenvalue distribution) is sufficient for understanding the methods of interest in this thesis. Specifically, we are not interested by the joint distribution of the eigenvalues as well as their local behavior but rather by their behavior as a whole. This makes Marčenko-Pastur's law an indispensable and necessary tools for understanding the techniques in this thesis. However its formulation requires to introduce some concepts and notations.

Going from vectors to measures. One of the first problems to tackle is to find an efficient way to express the limit of a vector (the p eigenvalues λ1 ≥ . . . ≥ λp ) whose size grows to ∞. A natural way to do so is to associate to any vector a probability measure. This leads to introducing the notion of Empirical Spectral Distribution which associates to any vector containing the eigenvalues of a target matrix a probability measure.

Empirical and Limiting Spectral Distribution. The Empirical Spectral Distribution (Empirical Spectral Distribution (ESD)) of a random matrix M ∈ R p×p is defined as

µ M (t) = 1 p p j=1 δ (t -λ j (M )) , (2.1)
where δ is the Dirac delta function and λ j (M )'s denote the p eigenvalues of M , including the multiplicity. The ESD is the normalized counting measure of the eigenvalues of M , i.e., the probability distribution that put mass 1/p at each one of the p eigenvalues of M . In general, the ESD is a probability measure on C; it has support in R (resp. in R + ) if M is Hermitian (resp. non-negative definite Hermitian). In this thesis, we are mainly concerned by covariance matrices and since there are Hermitian and non-negative definite, the corresponding ESD's will have support on R + . In the rest of the thesis, the support of the probability measure µ M will be denoted Supp(µ M ). As p, n → ∞ with constant ratio p/n ≡ c 0 → c ∞ 0 , the eigenvalues of the sequence of sample covariance matrices Σ ∈ R p×p are random variables and the corresponding ESD's µ Σ are random probability measures on R + . A fundamental question in random matrix theory is about whether the sequence µ Σ (sometimes written µ p when there are no ambiguities about the matrix under investigation) has a limit (in probability or almost surely). The Limiting Spectral Distribution (Limiting Spectral Distribution (LSD)) of M denoted µ ∞ is defined as the limit of (2.1) as n, p → ∞ if it exists. In the rest of the thesis, the Empirical Spectral Distribution of the sample covariance matrix will be denoted by µ p and its LSD will be denoted by µ. As for the sample covariance matrix, the ESD and LSD of the population covariance matrix Σ will be denoted respectively ν p and ν. An important area of RMT is concerned with understanding the properties of µ as function of ν. To that end, we do not work directly with the LSD µ but with a tool that is similar in flavor to the characteristic function of a distribution: the "Stieltjes transform" of a measure.

Resolvent and Stieltjes transform. The resolvent of a random matrix

M ∈ R p×p is defined ∀z ∈ C \ Supp(µ M ) as Q(z) = (M -zI p ) -1 .
This quantity displays several interesting properties, making it the relevant object to manipulate. First, it is a continuous function of z and it is easy to differentiate (compared to working directly on the ESD), providing a well-defined tool for mathematical analysis. Furthermore if M is symmetric, it contains the complete information about the eigenvalues λ j 's and the eigenvectors u j 's of the symmetric matrix M since it can be rewritten as:

Q(z) = p j=1 u j u T j λ j -z .
It is easy to see that the number of singularities of the resolvent is equal to the number of eigenvalues of M . While the statistics of the eigenvectors are an interesting and non-trivial subject in itself, we focus for now on the statistics of the eigenvalues through the ESD. For this aim, we define the normalized trace of the resolvent

m µ M (z) = 1 p trQ(z).
We shall skip the index M and replace it by p as soon as there is no confusion about the matrix we are dealing with. As p → ∞,

m µp (z) a.s. -→ m µ (z), m µ (z) = µ(dt) t -z (2.2)
which is known as the Stieltjes transform of µ.

The Stieltjes transform has a lot of interesting properties among which

1. m µ (z) is holomorphic on C \ Supp(µ) 2. If z ∈ C + then m µ (z) ∈ C + 3. If Supp(µ) ⊂ R + and z ∈ C + , then zm µ (z) ∈ C + ,
where we recall that Supp(µ) is the support of the distribution µ and we note C + = {z ∈ C, Im(z) > 0} with Im(z) the imaginary part of the complex number z.

An appealing feature of the Stieltjes transform is its analyticity which implies that its local knowledge leads to its knowledge everywhere else. Furthermore if µ has bounded support, the Stieltjes transform can be equivalently be rewritten as

m µ (z) = - ∞ n=0 1 z n+1 λ n dµ(λ).
Therefore the Stieltjes transform also known as the Cauchy transform is the generating function of the moments of the measure µ, (i.e., it is a power series in 1/z whose coefficients are the moments of µ). The spectral distribution can be recovered from the Stieltjes transform using the inversion formula:

µ({x}) = - 1 π lim →0 + Im [m µ (x + i )] . (2.3)
This important feature states that Stieltjes transform m µ (z) and probability measure µ are one-to-one corresponding to each other.

Of particular interest, the Stieltjes transform can be connected to Cauchy's integral formula.

Theorem 2 (Cauchy's integral formula). For Γ ⊂ C a positively (i.e., counterclockwise) oriented closed curve and a complex function f (z) analytic in a region containing Γ and its inside, then

1 2πı Γ f (z) z-z 0 = f (z 0 ) , if z 0 ∈ C is enclosed by Γ 1 2πı Γ f (z) z-z 0 = 0 , otherwise
Generalized Marčenko-Pastur distribution. In the study of covariance matrices, a fundamental result exists that describes the limiting behavior of the empirical spectral distribution µ, in terms of the limiting behavior of the population spectral distribution ν. The connection between these two measures is made through an equation that links the Stieltjes transform of the empirical spectral distribution to an integral against the population spectral distribution [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF].

Theorem 3. Suppose that the entries of the p×n matrix X are complex random variables that are independent identically distributed which satisfy

E[X 11 ] = 0, E[|X 11 | 2 ] = 1 and E[|X 11 | 4 ] < ∞.
Also, assume that Σ = diag(λ 1 , . . . , λ p ), where λ j ∈ R + and the distribution function of {λ 1 , . . . , λ p } converges almost surely to a probability distribution function ν as p → ∞. Let Σ = 1 n X ΣX. Then as n, p → ∞ such that p n ≡ c 0 → c ∞ 0 , the ESD μp of Σ converges to a non-random distribution μ , where, for any z ∈ C \ Supp(μ) , its Stieltjes transform m = m μ(z) is the unique solution in C + of the equation

z = - 1 m + c ∞ 0 tdν(t) 1 + mt , ( 2
= m µ (z) of µ is the unique solution in C + of m = dν(t) t(1 -c ∞ 0 -c ∞ 0 zm) -z .
(2.5) Equation (2.5) can be conveniently rewritten as:

m ν - 1 m μ(z) = -zm µ (z)m μ(z).
(2.6)

We should note that the assumption of the bounded fourth moment entries of X in the theorem, i.e., E[|X 11 | 4 ] < ∞ ensures from [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF] that the limiting distribution µ has bounded support. Under the assumptions put forth in Theorem 3, the spectral distribution of the sample covariance matrix is asymptotically non-random. Furthermore, it is fully characterized by the true population spectral distribution, through the equation (2.5). A particular case of equation (2.5) is often of interest: the situation when all the population eigenvalues are equal to 1. In this case, ν = δ 1 and we recover the Marčenko-Pastur law introduced in Theorem 1. We should stress that Equation (2.6) is remarkable and extremely powerful. Indeed let's recall due to the inversion formula of the Stieltjes transform that the unique link between the measure µ and its Stieltjes transform m µ (z) allows one to work with either. It turns out that the Stieltjes transform is more convenient and Equation (2.6) thus indirectly links µ and ν through an "explicit" link between m ν and m µ . This is an extremely powerful relationship which plays the role of µ → ν (i.e., λi → λ i ) in the case of the classical regime (n → ∞, p fixed).

In many applications, the population covariance Σ itself may not be the object of central interest. One is often rather interested in scalar functionals of Σ such as linear functionals of its eigenvalues. Based on the global knowledge of the LSD of the sample covariance matrix through the Marčenko-Pastur law, Random-matrix improved estimates of these functionals can be derived. To better introduce the random matrix improved estimate of the distance between covariance matrices in chapter 3, we present in the following section linear spectral statistic of random matrices in particular covariance matrices. An application to the estimation of the logarithm determinant of the population covariance matrix (population generalized variance) already introduced in Chapter 2 will be taken as a preliminary example to tackle the more involved distance between covariance matrices in Chapter 3.

Linear spectral statistic

Setting the stage. We introduce previously the population covariance matrix Σ, the Fisher matrix Σ -1 1 Σ 2 , and the matrix Σ 1 Σ 2 appearing in the distance between two covariance matrices Σ 1 and Σ 2 . Let M be one of these matrices. In one-sample and two-sample multivariate analysis, many statistics are functions of the eigenvalues {λ j } p j=1 of the matrix M of the form

L = 1 p p j=1 f (λ j ) = f (x)dµ M (x) (2.7)
for any function f smooth and bounded over {z ∈ C, R[z] > 0}. Such statistic is called a linear spectral statistic of the matrix M .

The generalized variance: a linear spectral statistic example. For example, the so-called generalized variance introduced previously and discussed also later in this chapter, is

L = 1 p log(|Σ|) = 1 p p j=1 log(λ j ).
(2.8) So this particular L is a linear spectral statistic of the population covariance matrix Σ with function f (x) = log(x). In two-sample multivariate analysis with say covariance matrices Σ 1 and Σ 2 , a lot of distances between Σ 1 and Σ 2 will still be of the previous form in (2.7), where however the eigenvalues λ j will be those of the so-called Fisher matrix Σ -1 1 Σ 2 or Σ 1 Σ 2 depending on the considered metric. Linear spectral statistic of these matrices is at the heart of the statistical tools developed in the first part of this thesis and more generally in modern statistic.

Therefore, understanding the asymptotic properties of eigenvalue statistics such as L above has paramount importance in data analysis when the dimension p is getting large with respect to the sample size n. In order to explain the methodology to deal with such quantity in random matrix theory, we will focus on the spectral statistic of the covariance matrix in this introductory part which has found several applications in signal processing and machine learning. In practice, one often estimates the covariance matrix using the sample covariance matrix Σ = 1 n XX T first and then uses it to compute the log-determinant. This estimate is known as the sample generalized variance.

Consistency of the sample generalized variance. Let λ1 , . . . , λp the eigenvalues of the sample covariance Σ and λ 1 , . . . , λ p the eigenvalues of the population covariance Σ. The sample generalized variance is defined as

Ln = 1 p log(| Σ|) = 1 p p j=1 log λj .
We show in the introductory part that when the dimension size p is fixed, for Σ = I p , λj → 1 almost surely as n → ∞ and thus Ln → 0. Therefore, Ln is a good estimator for the logarithm determinant of the population covariance matrix. However, when p, n → ∞ with p/n ≡ c 0 → c ∞ 0 ∈ (0, 1), since the limiting spectrum of Σ is almost surely bounded away from zero and upper-bounded, we have with probability 1 (see more details in [START_REF] Zhidong | Clt for linear spectral statistics of large-dimensional sample covariance matrices[END_REF]),

Ln → λ + λ - log(x) 2πıxc ∞ 0 (λ + -x)(λ --x)dx = c ∞ 0 -1 c ∞ 0 log(1 -c ∞ 0 ) -1 < 0 (2.9)
where

λ -= (1 -c ∞ 0 ) 2 and λ + = (1 + c ∞ 0 ) 2
. This shows that the classical estimator is biased (except for c ∞ 0 = 0). This is due to the fact that since the eigenvalues of Σ follow asymptotically the Marčenko-Pastur Law discussed previously, the limiting spectrum is spread around 1. Therefore, taking the logarithm of such eigenvalues may lead to dramatic errors.

The solution of the real integral presented in Equation (2.9) has been derived using Poisson's integral formula (see more details in [START_REF] Zhidong | Clt for linear spectral statistics of large-dimensional sample covariance matrices[END_REF][Section 5]). The methodology is not anymore applicable for more general spectral distributions and more complex functionals as will be the case in Chapter 3 of this thesis. Therefore, we present in the following a quite general methodology for recovering the n, p-consistent estimate of 1 p p j=1 f (λ j ).

Design of n, p-consistent estimators. To that end, we recall the definition of the population and empirical eigenvalue distributions:

µ p = 1 p p i=1 δ λ i ( Σ) , ν p = 1 p p i=1 δ λ i (Σ) . As p, n → ∞ with p/n ≡ c 0 → c ∞ 0 ∈ (0, ∞), µ p a.s.
-→ µ and ν p a.s.

-→ ν and we have the convergence of the corresponding Stieltjes transforms (m µp a.s.

-→ m µ and m νp a.s.

-→ m ν ). Instead of using the classical asymptotic result µ p a.s.

-→ ν p (only valid when p is fixed and n → ∞), the main idea of the estimation procedure consists in relating f dν p , to the Stieltjes transform m νp using the Cauchy's integral theorem. Using furthermore the relation between the Stieltjes transforms of the limiting empirical and population spectral distributions m µ and m ν from the generalized Marčenko-Pastur law, f dν p can be expressed as function of the Stieltjes transform of the limiting spectral distribution of the sample covariance matrix m µ . It then remains to use the convergence ν p → ν and m µp a.s.

-→ m µ , along with the fact that the eigenvalues of Σ almost surely do not escape the limiting support µ as p → ∞. This is ensured from [START_REF] Bai | No Eigenvalues Outside the Support of the Limiting Spectral Distribution of Large Dimensional Sample Covariance Matrices[END_REF], by assuming lim p→∞ max( Σ , Σ -1 ) < ∞ with . the operator norm.

The detailed steps are summarized as follows.

1. Express 1 p p j=1 f (λ j ) as function of the Stieltjes transform m νp of the empirical spectral distribution of the matrix Σ using Cauchy's integral formula:

1 p p j=1 f (λ j ) = f (t)dν p (t) = 1 2πı Γν f (ω) ω -t dω dν p (t) = -1 2πı Γν f (ω)m νp (ω)dω
with Γ ν a contour surrounding the limiting spectral distribution of Σ.

2. Relate the Stieltjes transform m ν of the limiting spectrum of the population covariance matrix Σ to the Stieltjes transform m µ of the limiting eigenvalue distribution of the sample covariance matrix Σ using the Marčenko-Pastur law given in equation (2.6) that we recall for convenience ∀z ∈ C \ Supp(µ):

m ν - 1 m μ(z) = -zm µ (z)m μ(z).
3. Deduce the expression of 1 p p j=1 f (λ j ) as function of the Stieltjes transform m µp of the empirical spectral distribution of the matrix Σ. To that end, we need to proceed to the change of variable ω = -1 m μ(z) which ends up to

1 p p j=1 f (λ j ) = - 1 2πı Γµ f - 1 m μ(z) m ν - 1 m μ(z) m μ(z) m μ(z) 2 dz = 1 2πı Γµ f - 1 m μ(z) zm μ(z)m µ (z) m μ(z) m μ(z) 2 dz = 1 2πc ∞ 0 ı Γµ f - 1 m μ(z) zm μ(z)dz.
Letting g(z) = f (1/z) and G(z) such that G (z) = g(z), integration by parts of the above expression along with m μp a.s.

-→ m μ further gives

1 p p j=1 f (λ j ) = 1 2πc 0 ı Γµ G -m μp (z) dz + o(1).
(2.10)

We need furthermore to ensure that the change of variable performed brings any contour Γ µ surrounding the support of µ to a contour Γ ν surrounding only the support of ν (and not additional points such as singular point 0 in order to ensure that Cauchy's integral formula of step (1) still remains valid). We will show that this happens to be true only under the condition c ∞ 0 < 1. In Chapter 3, these aspects are revisited and a solution is proposed to mitigate this important case.

4. Find if possible the solution of the complex integral in closed form using complex integration techniques.

Before going into the details of the integration contour determination, we handle step (4), by introducing some basics of integration in complex analysis.

Background on complex analysis for integration Definition 1. A function f (z) is said to be analytic in a region R of the complex plane if f (z) has a unique derivative at each point of R and if f (z) is single valued.

Note that in Definition 1, the definition of the differentiation of a complex function f (z) at a point z 0 is defined similarly as in the real case as

f (z 0 ) = lim δz→0 f (z 0 + δz) -f (z 0 ) δz .
To be differentiable, it is important that the limit be the same whichever direction we approach. Points at which a function f (z) is not analytic are called singular points or singularities of f (z). There are two different types of singular points:

• if there exists an integer n such that the product (z -a) n f (z) is analytic at z = a, then f (z) has a pole of order n at z = a, if n is the smallest such integer. Example: f (z) = 1/z 2 has a pole of order 2 at z = 0.

• When f (z) is a multivalued function, any point which is not in the region of definition of the single-valued branch of f (z) is a singular branch point. Example:

f (z) = √ z -a, f (z) = log(z -a) have a branch point at z = a.
The set of all branch points are called branch cuts.

Cauchy's residue theorem which is an extension of Cauchy's integral formula allows to compute a complex integral where the contour encloses several poles.

Theorem 4 (Cauchy's residue theorem). Let Γ be a closed path within and on which f is analytic except for m poles ξ 1 , . . . , ξ m . Then

1 2πı Γ f (z)dz = m j=1 Res ξ j f (2.11)
where the residue Res ξ j f of the pole ξ j of order n is defined as Res ξ j f = lim

z→ξ j 1 (n-1)! d n-1 dz n-1 [(z -ξ j ) n f (z)].
In the cases where the contour encloses not only poles but also branch points, it is mandatory to define an auxiliary contour to exclude them from the interior of the initial contour and try to relate the integral on the new contour to the integral on the initial one. The main guidelines to solve complex integrals can be summarized as follows.

• Determine the poles and the branch points of the integrand function.

• In case of branch points, deform the initial contour in order to exclude the branch points from the interior of the contour.

• Express the integral over the new contour as function of the integral over the initial contour and potentially other integrals which are evaluated by either applying residue Cauchy theorem (when surrounding poles) or by computing real integrals.

This procedure is applied to compute the complex integral in (2.10) for f (z) = log(z). For this choice of function, the corresponding function G is defined as G(z) = z (log(z) -1).

The branch points of log(z) are defined as z ∈ R such that R(z) < 0 where R(z) denotes the real part of the complex value z. Therefore, the branch cuts of the integrand are defined as the z's such that m μp (z) ≥ 0.

We recall from the definition of the Stieltjes transform that

m μp (z) = c 0 p p i=1 1 λi -z + 1 -c 0 z .
(2.12)

From equation (2.12), m μp (z) is a rational function, therefore it can be written as

m μp (z) = p i=1 (z -ζ i ) z p i=1 ( λi -z) , (2.13)
with ζ 1 < . . . < ζ p the zeros of m μp (z) and λ1 < . . . < λp the eigenvalues of Σ. Equations (2.12) and (2.13) will be important for future simplifications in the calculation of complex integrals. In particular, we have that:

lim z→ λj ( λj -z)m μp (z) = c 0 p p i=1 log ζ i λi = lim z→0 log -zm μp (z) = log(1 -c 0 )
From equation (2.13), the branch cuts B are defined as λp ] as represented in Figure 2.1. These segments lie inside the integration contour Γ, which needs to be modified for proper integration; the new contour, denoted Γ n is depicted in Figure 2.1. The complex integral defined in (2.10) over the contour Γ n is the sum of several integrals, subdivided into four types:

B = [ζ 1 , λ1 ] ∪ . . . ∪ [ζ p ,
• Integrals I 1 over the circles surrounding {ζ j } p j=1 which, thanks to the variable change z = ζ j + e ıθ , reduce to lim →0 2π-G(-m μp (ζ j + e ıθ )) 2πıc 0 ı e ıθ dθ which leads for G(z) = z(log(z) -1) to:

-1 2πıc 0 lim • Integrals I 2 over the circles surrounding the poles { λj } p j=1 of order 1 which can be computed by using the residue theorem:

I 2 = 1 c 0 p j=1 lim z→ λj G -m μp (z) z -λj = -1 c 0 p j=1 lim z→ λj (z -λj )m μp (z) log -m μp (z) -1 = 1 p p j=1 lim z→ λj log -m μp (z) -1 = -1 p p j=1 log( λj ) + 1 p p i=1 p j=1 lim →0 log λj -ζ i + λi -λj + -1.
• Real integrals I 3 over the segments [ζ j , λj ] which can be computed by remarking that the log function has a discontinuity of 2ıπ at the branch cut.

I 3 = -1 2ıπc 0 p j=1 λj - ζ j + m μp (z) log |m μp (z)| + ıπ -log |m μp (z)| + ıπ dz = -1 c 0 p j=1 λj - ζ j + m μp (z)dz = -1 c 0 p j=1 λj - ζ j + c 0 -1 z + c 0 p p i=1 1 λi -z dz = p j=1 1 -c 0 c 0 log(z) + 1 p p i=1 log( λi -z) λj - ζ j + = 1 -c 0 c 0 p j=1 log( λj ζ j ) + lim →0 1 p p i=1 p j=1 log λj -λi + λi -ζ j + = c 0 -1 c 0 log(1 -c 0 ) - 1 p p i=1 p j=1 lim →0 log λi -ζ j + λj -λi +
where in the last equality we used, among other algebraic simplifications, the fact that p j=1 log

ζ j λj = lim z→0 log -zm μp (z) = log(1 -c 0 ).
• The sought for integral I 4 over Γ for → 0.

Since the contour Γ n doesn't contain any poles or branch points, the sum of the four above integrals reduces to zero. Combining these integrals then yields to the solution of the integral over the contour Γ denoted I 4 :

I 4 = 1 p log(| Σ|) + 1 -c 0 c 0 log(1 -c 0 ) + 1,
where we retrieve the result of [START_REF] Zhidong | Clt for linear spectral statistics of large-dimensional sample covariance matrices[END_REF].

Integration contour determination. As already anticipated in Step (3) of the procedure, we need to ensure that the change of variable performed in Step (3) moves any complex contour closely encircling the support of µ onto a valid contour encircling the support of ν; we will in particular be careful that the resulting contour, in addition to encircling the support of ν, does not encircle additional values possibly bringing undesired residues (such as 0). We will proceed by showing that a contour encircling µ results on a contour encircling ν. These details rely heavily on the works of [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] and follow similar ideas as in e.g., (Couillet et al., 2011).

Let us consider a first contour Γ µ closely around the support of µ (in particular not containing 0). We have to prove that any point z of this contour is mapped to a point ω of a contour Γ ν closely around the support of ν.

The change of variable performed in Step (3) of the proposed methodology reads, for all z ∈ C \ Supp(µ),

ω ≡ ω(z) = -1 m μ(z) .
It therefore remains to show that real z's (outside the support of µ) project onto properly located real ω's (i.e., on either side of the support of ν). This conclusion follows from the seminal work [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] on the spectral analysis of

z- z+ ω- ω+ Supp(µ) Supp(ν) ω z • (ω) Figure 2.2: Variable change ω → z • (ω) = ω + c ∞ 0 tν(dt)
1-t/ω . Supp(θ) is the support of the probability measure θ. sample covariance matrices. The essential idea is to note that, due to (2.4), the relation ω(z) = -1/m μ(z) can be inverted as

z ≡ z(ω) = - 1 m μ + c ∞ 0 tν(dt) 1 + tm μ = ω + c ∞ 0 tν(dt) 1 -t ω .
In [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF], it is proved that the image by z(•) of ω(R \ Supp(ν)) coincides with the increasing sections of the function z • : R \ Supp(ν) → R, ω → z(ω).

The latter being an explicit function, its functional analysis is simple and allows in particular to properly locate the real pairs (z, ω). Details of this analysis are provided in [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF], which shall not be recalled here. The function z • is depicted in Figure 2.2 in the case of c ∞ 0 < 1. In the case of c ∞ 0 > 1, some undesired effects appear; these are discussed more in details in Chapter 3.

Summarizing and introducing Chapter 3. This section shows through the statistical estimation of log(|Σ|), how and why classical estimates fail and how Random Matrix Theory can be used to mitigate this problem. In chapter 3 of this thesis, we will go beyond this simple case. Specifically, we will be interested in computing the distance between two covariance matrices let's say Σ 1 and Σ 2 which can be expressed as a linear spectral statistic of the matrix Σ -1 1 Σ 2 or Σ 1 Σ 2 . Such distances will induce two technical issues compared to the previous case:

• The matrix of interest will not be a single matrix Σ but a more involved matrix Σ -1 1 Σ 2 known as F-matrix or Σ 1 Σ 2 for which asymptotic spectral distribution should be derived.

• The complex integral obtained from Cauchy's integral will involve non-trivial real integrals.

These aspects will be discussed in detail in Chapter 3.

Beyond the eigenvalue functionals of the covariance matrix studied so far, some machine learning methods involve more complex functionals that involve the eigenvectors of the covariance matrix as will be the case in multi-task learning. Thus, it is important to introduce the tools necessary to characterize the general behavior of covariance matrices. Deterministic equivalents are tools that aim at this goal. This tool will be important to explain the behavior of algorithms such as ridge regression, multi-task learning, etc and is introduced next.

Deterministic equivalent of random matrices

Setting the stage. We previously raised in the introduction that most of the quantities involved in signal processing and machine learning applications involve complex dependency which limits the capability of classical learning theory to provide theoretical analysis. The example of ridge regression (close to the Least Square Support Vector Machine Multi-Task Learning tackled in Chapter 4) sounds particularly appealing. Indeed, the decision score for a new test data x is given from 1.4 by

g(x) = 1 √ n ω T x = 1 n x T 1 n XX T + λI p -1 Xy (2.14) which involves a bilinear form of 1 n XX T + λI p -1 .
We should mention here that in the context of classification, the data matrix X will not be assumed anymore to be of zero-mean. We should therefore distinguish between the generalized population covariance matrix defined as

C = E[ 1 n XX T ] from the population covariance matrix Σ = E[ 1 n (X -E[X]) (X -E[X]
) T ] except for zero-mean data. We define similarly the generalized sample covariance matrix Ĉ = 1 n XX T and define its resolvent as

Q(z) = Ĉ -zI p -1 .
Beyond the eigenvalues of Ĉ (which are globally the same as those of Σ) treated for now in the context of linear spectral statistic, our interest will also be on bilinear functionals of Q(z) = Ĉ -zI p -1 therefore involving possibly the eigenvectors of Q(z). As such, beyond studying the trace of the resolvent Q(z), our interest is also on characterizing Q(z) itself.

Precisely, we shall study so-called deterministic equivalents of Q(z), the precise definition of which is given in Definition 2 and that can be described as deterministic matrices Q verifying trA

Q(z) -Q(z) a.s.
-→ 0 when A has unit norm. Note that if we control tr

A(Q(z) -Q(z)) , we also control u T Q(z)v = tr vu T Q(z) for two deterministic vectors u, v ∈ R p with unit norm.
Definition 2 (Deterministic equivalents). A deterministic equivalent, say F ∈ R n×p , of a given random matrix F ∈ R n×p , denoted F ↔ F , is defined by the fact that, for any deterministic linear functional f : R n×p → R, f (F -F ) → 0 almost surely (for instance, for u, v of unit norm, u T (F -F )v → 0 and, for A ∈ R p×n deterministic of bounded operator norm, 1 n trA(F -F ) → 0). In the following lines, for the sake of simplicity and in order to better grasp the main ideas of the derivation, we provide a high-level proof for the deterministic equivalent of the generalized sample covariance matrix Ĉ.

Deterministic equivalent of Q(z). First, let us note that a naive deterministic equivalent for Ĉ -zI p -1 would be (C -zI p ) -1 . This turns out to be incorrect under p, n → ∞. Instead, letting Q(z) = ( C -zI p ) -1 where C is some deterministic matrix to determine. To that end, we may first compute the difference using in particular the identity A -1 -B -1 = A -1 (B -A)B -1 for invertible matrix A and B:

Q -E[Q] = E[Q( 1 n XX T -C) Q] = 1 n n i=1 E[Q(x i x T i -C) Q]
where we recall that x i is the i-th column of X.

Here, to go further, we need to make explicit the dependence between x i and the matrix Q in order to evaluate the expectation of the product Qx i . Let us denote X -i ∈ R p×(n-1) , the matrix X deprived of its i-th column, which leads us to define the matrices Ĉ

-i = 1 n X -i X T -i and Q -i = ( Ĉ-i -zI p ) -1 .
To handle the dependence between x i and Q, we will exploit the classical Schuur identities (Sherman-Morrison identity for example):

Q = Q -i - 1 n Q -i x i x T i Q -i 1 + 1 n x T i Q -i x i
, and

Qx i = Q -i x i 1 + 1 n x T i Q -i x i .
The second inequality allows us to disentangle the relation between Q and x i in the product Qx i with a similar but easier to apprehend product Q -i x i and a factor

1 1+ 1 n x T i Q -i x i
easily controllable term. This leads us to

Q -E[Q] = 1 n n i=1 E Q -i x i x T i 1 + 1 n x T i Q -i x i -C Q + 1 n 2 n i=1 E Q -i x i x T i Q -i C Q 1 + 1 n x T i Q -i x i .
(2.15) Due to the supplementary factor 1 n and under appropriate assumption on the data matrix X, the norm of the rightmost random matrix is negligible (see more details in (Louart & Couillet, 2018)). Thus, if one assumes, say, that the random vectors {x i } n i=1 follow the same law, one would choose naturally

C = C 1+δ , with δ = 1 n E[x T i Q -i x i ] = 1 n tr E[Q -i ]E[x i x T i ] = 1 n tr C Q .
One may establish an implicit equation for δ not involving expectations over Q (or Q -i ). The deterministic equivalent is given by (Louart & Couillet, 2018).

Theorem 5. Let a data matrix X = [x 1 , ..., x n ] ∈ R p×n be distributed as a mixture of concentrated random vectors as per definition 3 in Appendix. Then the resolvent of the generalized sample covariance matrix defined as

Q(z) = 1 n XX T -zI p -1 admits a deterministic equivalent Q(z) given by Q(z) ↔ Q(z) = C 1 + δ(z) -zI p -1
where δ(z) is the unique solution to the fixed point equation defined as

δ(z) = 1 n tr C C 1 + δ(z) -zI p -1
.

As introduced before, the interest of the deterministic equivalent is to allow to compute bilinear forms involving the random matrix Q(z). In particular, 1 p trQ(z)m µ (z) → 0 (where we retrieve the Generalized Marčenko-Pastur distribution) and

a T Q(z)b -a T Q(z)b → 0 for deterministic a, b ∈ R p of bounded Euclidean norm.
Universality. Theorem 5 has been proved in (Louart & Couillet, 2018) under a mixture of concentrated random vectors. The concentrated random vector assumption (introduced rigorously in definition 3 in Appendix) better models realistic datasets by imposing very little structure on the data. Exactly, it only constrains all Lipschitz functionals R p×n → R of X (i.e., its typical observations) to satisfy a concentration inequality; while this may seem demanding, the family of concentrated random vectors in fact contains all Lipschitz generative models (for instance neural networks) fed by Gaussian inputs (such as Generative Adversarial Network (GAN)s [START_REF] Goodfellow | Generative adversarial nets[END_REF]), as well as all further Lipschitz transformations of these vectors (for instance, features extracted by pretrained neural networks). It naturally comes that most derivations performed in Chapter 4 are universal in the sense of its being robust to a broad range of very realistic random data.

Revisiting the ridge regression. Let's consider data distributed in two classes C 1 and C 2 with opposite means and isotropic covariances. Specifically, we consider a data matrix X = [X (1) , X (2) ] with data of class ,

X ( ) = [x ( ) 1 , . . . , x ( ) n ] ∈ R p×n , such that for x ( ) i ∈ C for ∈ {1, 2}, x ( ) i = (-1) µ + ω i , ω i ∼ N(0, I p )
where µ ∈ R p with lim p→∞ µ < ∞. Let's suppose that the data are arranged in classes in the data matrix X and denoting y = [1 n 1 , -1 n 2 ] T . Under this setting, the goal is to study the statistical behavior of the test score g(x) for a new test data x ∼ N(µ x , I p ) independent from X:

g(x) = 1 n x T 1 n XX T + λI p -1 Xy = 1 n x T Q(-λ)Xy.
First, it can easily be proved by conditioning on the training data X that g(x) is asymptotically Gaussian as the scalar product of a deterministic vector with a Gaussian isotropic vector. Moreover, the mean can be computed as follows given that x ∈ C :

E[g(x)] = E X [E x [g(x)|X]] = E X n i=1 (-1) n µ T Q(-λ)x i y i
where we recall that Q(-λ) = 1 n XX T + λI n -1 . We shall write sometimes for simplicity

Q instead of Q(z)
when there are no ambiguities.

The above expectation requires to manage the statistical dependencies between Q and x i , which can be handled using:

Qx i = Q -i x i 1 + 1 n x T i Q -i x i .
(2.16)

As already seen in the derivation of the deterministic equivalent, the quadratic term

1 n x T i Q -i x i a.s.
-→ δ(-λ). Therefore following the same line of reasoning as in (Seddik et al., 2020, Proposition A.3),

E[g(x)] = E 1 n n i=1 (-1) 1 + δ µ T Q -i x i y i + O log p p = 1 n (-1) 1 + δ n i=1 µ T QE[x i ]y i + O log p p (2.17)
where Q is the deterministic equivalent of Q.

As for the mean, the variance σ 2 can be computed as follows:

σ 2 = E (g(x) -E[g(x)]) 2 = 1 n 2 E y T X T 1 n XX T + λI p -2 Xy = 1 n E y T 1 n X T X + λI n -2 X T X n y = 1 n E y T 1 n X T X + λI n -1 y -λy T 1 n X T X + λI n -2 y = 1 n E y T Qy -λy T Q2 y (2.18) where Q = 1 n X T X + λI n -1 .
We therefore need a deterministic equivalent of Q, Q and Q2 given by Corollary 1 as a consequence of Theorem 5.

Corollary 1 (Deterministic equivalents for Q(z), Q(z) and Q(z) 2 ). Under the setting described above, deterministic equivalents of Q(z), Q(z) and Q2 (z) are given by:

Q(z) ↔ m(z)I p - m(z) 2 1 + (c 0 + µ 2 )m(z) µµ T Q(z) ↔ m(z)I n - m(z) 2 µ 2 1 + (1 + µ 2 ) m(z) yy T Q(z) 2 ↔ m (z)I n - 2 m (z) m(z) µ 2 + m (z) m(z) 2 µ 2 (1 + µ 2 ) (1 + (1 + µ 2 ) m(z)) 2 yy T
where

m(z) = 1 -c 0 -z + (1 -c 0 -z) 2 -4zc 0 2zc 0 m(z) = -1 + c 0 -z + (1 -c 0 -z) 2 -4zc 0 2z
and m (z) is the derivative of m(z).

Corollary 1 is a special case of Theorem 5 for C = I p + µµ T (see details in Appendix).

Using the deterministic equivalents in Corollary 1, Theorem 6 unfolds directly.

Theorem 6 (Asymptotics of g(x)). Under the Gaussian mixture model introduced above, for x ∼ N(µ x , I p ) with

µ x = µ , as p, n → ∞ g(x) -G → 0, G ∼ N((-1) m, σ 2 )
in distribution where

m = m(-λ) µ 2 (1 + (c 0 + µ 2 )m(-λ))) σ 2 = (1 + m(-λ)) 2 ( m(-λ) -λ m (-λ)) + m2 (-λ) µ 2 (1 + m(-λ) -λ m (-λ)) (1 + (1 + µ 2 ) m(-λ)) 2 .
Since g(x) has a Gaussian limit centered about ±m, the (asymptotic) standard decision for x to be allocated to Class 1 (x → C 1 ) or Class 2 (x → C 2 ) is obtained by the "averaged-mean" test with

g(x) C 1 ≷ C 2 0 (2.19) the classification error rate ≡ 1 2 P (x → C 2 |x ∈ C 1 ) + 1 2 P (x → C 1 |x ∈ C 2 ) of which is then ≡ P g(x) C 1 ≷ C 2 0 = Q m σ + o(1) (2.
Q(t) = 1 √ 2π
∞ t e -x 2 dx. The close fit between the theoretical and the empirical prediction is illustrated in Figure 2.3.

This section illustrates that Random Matrix Theory provides the tools to predict theoretically the statistical behavior of g(x) and therefore potentially to understand and optimize the ridge regression scheme. In chapter 4, the idea is to exploit these powerful tools to understand the functioning of more complex algorithms, closer to the reality of modern machine learning problems like multi-task learning with the following consequences:

• Multi-Task Learning is of huge interest in machine learning since it helps to leverage scarce labeled sample from similar tasks to help for the generalization performance. Since Transfer Learning phenomenon is prone to negative transfer, the proper understanding of their inner working mechanisms is of particular interest.

• From a technical point of view, in the multi-task learning framework, the data matrix X will be extended to a block diagonal matrix Z and therefore the Marčenko-Pastur law will not be applicable.

Motivation and main findings

Motivation. Evaluating the distance between covariance matrices is at the core of many machine learning and signal processing applications. They are notably used for covariance features-based classification (for instance in brain signal or hyperspectral image classification), as well as for dimensionality reduction and representation of high dimensional points. Denote Σ 1 , Σ 2 ∈ R p×p two large dimensional covariance matrices for which we would like to compute the distance D(Σ 1 , Σ 2 ) based on few (p-dimensional) sample vectors. We assume that D can be written as a linear functional

1 p p i=1 f (λ i ) of the eigenvalue distribution of either Σ -1 1 Σ 2 (λ i = λ i (Σ -1 1 Σ 2 )) (as with the Fisher, Battacharrya, Kullback Leibler, Rényi divergences) or Σ 1 Σ 2 (λ i = λ i (Σ 1 Σ 2 )) (for the Wasserstein distance, Frobenius distance).
Classical estimate. Based on a simple law-of-large-numbers argument, these metrics are commonly estimated from a simple replacement of the genuine p × p-dimensional matrices Σ 1 and Σ 2 by their sample covariance estimates Σ1 and Σ2 . Such estimates, as shall be shown next, are however bound to sometimes extremely severe errors, particularly when the respective numbers n 1 and n 2 of samples to estimate Σ 1 and Σ 2 are not large compared to p. This scenario is however frequently met in practice (short-time brain activity scans with high resolution EEG, large number of shortly-stationary assets in finance, high-resolution hyperspectral imaging, etc.) and therefore induces possibly weak data processing performances.

RMT-consistent estimates and main findings. To tackle these problems, in this work, we provide a random matrix improved consistent estimates for D(Σ 1 , Σ 2 ) for all aforementioned metrics. Technically speaking, our results rely on the following approach already introduced in chapter 2 in the simple case of the linear spectral statistic of the covariance matrix. We express a generic form of the metric under study under the form of a complex integral involving the Stieltjes transform of the (population) eigenvalue distribution of Σ -1 1 Σ 2 or Σ 1 Σ 2 . As the latter distribution is not accessible, we then link it to the (sample) eigenvalue distribution of Σ-1 1 Σ2 (or Σ1 Σ2 ), through a functional equation relating the Stieltjes transforms of population and empirical eigenvalue distribution similarly as in the generalized Marčenko-Pastur law. This results, through an appropriate change of variable, to a complex integral involving only the eigenvalues of Σ-1 1 Σ2 (or Σ1 Σ2 ), which may finally be evaluated using complex analysis techniques. This approach already described in the linear spectral statistic of the covariance matrix in chapter 2 is notably inspired by the seminal work of Mestre [START_REF] Mestre | Improved estimation of eigenvalues of covariance matrices and their associated subspaces using their sample estimates[END_REF] (see also (Couillet et al., 2011)) where functional estimates of the eigenvalue distribution of a single covariance matrix Σ is performed similarly from the corresponding eigenvalue distribution of the sample estimate Σ. Aside from the more involved statistical models Σ-1 1 Σ2 and Σ1 Σ2 , the originality of the present work mostly lies in that the family of metrics involve non-smooth complex functionals (in particular logarithms and square roots) that result in more advanced technical considerations from real and complex analysis than in [START_REF] Mestre | Improved estimation of eigenvalues of covariance matrices and their associated subspaces using their sample estimates[END_REF].

Moreover, although consistent for n 1 , n 2 ∼ p, these improved estimators still demand that n 1 , n 2 > p for all functions f (z) having a singularity at z = 0 (e.g., 1/z, log(z), log 2 (z), √ z). Based on a polynomial approximation of the functions of interest, we furthermore propose to retrieve consistent estimates for the challenging n 2 < p scenario.

Chapter organization. The chapter is organized as follows. Section 3.2 introduces the main model and assumptions, Section 3.3 provides the general idea of the estimation and the problem induced by the challenging case n 1 < p and n 2 < p as well as a solution based on polynomial approximation. In Section 3.4, closed-form and numerically convenient expressions of the proposed estimators are derived for all functions of interest. Section 3.5 proposes a practical application to covariance matrix estimation and covariance-based feature classification.

CHAPTER 3. IMPROVED ESTIMATE OF THE DISTANCE BETWEEN COVARIANCE35

Metrics λ i f (z) d 2 F λ - i log 2 (z) d 2 B λ - i -1 4 log(z) + 1 2 log(1 + z) -1 2 log(2) δ KL λ - i 1 2 z -1 2 log(z) -1 2 δ αR λ - i -1 2(α-1) log(α + (1 -α)z) + 1 2 log(z) d W λ + i -2 √ z d Fro λ + i -2z
Table 3.1: Metrics and associated functions.

Models and Assumptions

For a ∈ {1, 2}, let

X a = [x (a) 1 , . . . , x (a)
na ] be n a independent and identically distributed random vectors with x

(a) i = Σ 1 2 a x(a)
i , where x(a) i ∈ R p has zero-mean, unit variance and finite fourth-order moment entries. This holds in particular for x (a) i ∼ N(0, Σ a ). In order to control the growth rates of n 1 , n 2 , p, we make the following assumption:

Assumption 1 (Growth Rates). As n a → ∞, p/n a ≡ c a → c ∞ a ∈ (0, ∞) and lim sup p max{ Σ -1 a , Σ a } < ∞ for • the operator norm.
We should point out that the hypothesis lim sup p max{ Σ -1 a , Σ a } < ∞ ensures that the limiting spectral distribution of the matrix of interest Σ -1

1 Σ 2 has bounded support away from zero. This will be particularly important to ensure that the limiting distribution of Σ-1 1 Σ2 has bounded support which we will be used in particular to relate the non-asymptotic complex integral to their asymptotic counterpart. We define the sample covariance estimate Σa of Σ a as Σa

≡ 1 n a X a X T a = 1 n a na i=1 x (a) i x (a)T i .
Our objective is to estimate the distance D(Σ 1 , Σ 2 ) between the covariance matrices Σ 1 and Σ 2 of the form:

D(Σ 1 , Σ 2 ) = 1 p p i=1 f (λ i )
where

λ i = λ - i are the eigenvalues of Σ -1 1 Σ 2 or λ i = λ + i are the eigenvalues of Σ 1 Σ 2 . This form comprises, among others, the Fisher d 2 F , Frobenius d Fro , Wasserstein d W and Battacharrya distances d 2 B
, along with the Kullbach-Liebler δ KL and Rényi divergences δ αR as shown in details in table 3.1. The Wasserstein distance d W between two zero-mean Gaussian distributions with covariances Σ 1 and Σ 2 , respectively, assumes the form (Peyré & Cuturi, 2019, Remark 2.31)

: d W = tr(Σ 1 ) + tr(Σ 2 ) -2tr (Σ 1 2 1 Σ 2 Σ 1 2 1 ) 1 2 . It is easily shown that, under Assumption 1, 1 p tr Σa -1 p trΣ a → 0 almost surely. But estimating 1 2 1 Σ 2 Σ 1 2 1 ) 1 2
is more involved and is the focus of this work. This explains why the corresponding function in the table is -2 √ z. The same holds for the Frobenius distance

d Fro = tr Σ 2 1 + Σ 2 2 -2tr (Σ 1 Σ 2 )
for which the corresponding function of interest is -2z since tr Σ 2 i can be estimated consistently (see more details in (Tiomoko & Couillet, 2019b)). The proposed estimate relies on random matrix theory tools developed in chapter 2 and particularly on the Stieltjes transform m θ (z) of a probability distribution θ that we recall for convenience as:

m θ : C \ supp(θ) → C, z → (λ -z) -1 dθ(λ).
The Stieltjes transform is here used to create a link between the population covariance eigenvalue distribution ν p and the sample eigenvalue distribution µ p (similarly as performed in the generalized Marčenko-Pastur law) defined by:

ν p = 1 p p i=1 δ λ i , µ p = 1 p p i=1 δ λi
where λi = λi are the eigenvalues of Σ-1 1 Σ2 or λi = λ+ i are the eigenvalues of Σ1 Σ2 . Similarly,

ν ± p = 1 p p i=1 δ λ ± i , µ ± p = 1 p p i=1
δ λ± i and the corresponding Stieltjes transforms read in particular as

m νp (z) = 1 p p i=1 1 λ i -z , m µp (z) = 1 p p i=1 1 λi -z for µ p = µ ±
p and ν p = ν ± p .

Improved estimate of the distance between covariance

Under the aforementioned setting, Theorem 7 provides a random matrix consistent estimate of these distances in the "easy" regime where lim p/n a < 1.

Theorem 7. Let f : C → C be analytic on a contour Γ µ ⊂ {z ∈ C, R[z] > 0} surrounding µ p . Then under Assumption 1, f dν p - 1 2πi Γµ f ϕ p (z) ψ p (z) ψ p (z) ψ p (z) - ϕ p (z) ϕ p (z) ψ p (z)dz c 2 a.s.
-→ 0 where

ϕ p (z) = z(1 + c 1 zm µp (z)), µ p = µ - p z 1-c 1 -c 1 zmµ p (z) , µ p = µ + p ψ p (z) = 1 -c 2 -c 2 zm µp (z).
The result of Theorem 7 has the strong advantage to be flexible to any smooth function

f over {z ∈ C, R[z] > 0}, so in particular to f (z) = log k (z) or f (z) = log k (1 + αz),
which commonly appear in the distance between covariance matrices and divergences. The constraint c 2 < 1 is however mandatory and cannot be relaxed, unless f is analytic on all C (which fails for logarithm functions). Remark 2 also discusses these aspects (see also Section 6.2.2 for more details) and a solution is furthermore proposed in (Tiomoko & Couillet, 2019a) that we describe briefly in the end of the section.

Before getting to the proof, note that the formulation of Theorem 7 exhibits two important quantities, the functions ϕ p and ψ p , which both relate to the eigenvalue distribution of Σ-1 1 Σ2 or Σ1 Σ2 respectively through c 1 and c 2 ; each function therefore emphasizes the impact of the restricted number of data with respect to the dimension p.

We subsequently provide a sketch of proof of Theorem 7 (here for µ p = µ - p ). The detailed proofs are deferred into Section 6.2.1 in Appendix.

Sketch of Proof. Using the Cauchy integral formula similarly as in

Step (1) of the procedure described in the linear spectral statistic of the population covariance matrix of Chapter 2, we have

D(Σ 1 , Σ 2 ) = 1 p p i=1 f (λ i ) = f (t)ν p (dt) (3.1) = 1 2πı Γν f (z) z -t ν p (dt) = -1 2πı Γν f (z)m νp (z)dz.
Thus, estimating D(Σ 1 , Σ 2 ) is equivalent to relating m νp to m µp . Since X 1 and X 2 are independent, we can condition first on X 2 . By Theorem 3 given in chapter 2, the limiting eigenvalue distribution of Σ 2 Σ-1 1 , denoted ζ, can be written as a function of the limiting eigenvalue distribution of Σ 2 Σ -1 1 , and similarly for the limiting eigenvalue distributions of Σ2

Σ-1 1 and Σ 2 Σ-1 1 . This entails the two equations:

zm µp (z) = ϕ p (z)m ζp (ϕ p (z)) + o p (1) (3.2) m νp (z/ψ p (z)) = m ζp (z)ψ p (z) + o p (1). (3.3)
where we follow the convention to use o p (1) for a sequence of random variables that convergences to zero in probability. Through the changes of variable z → ϕ p (z) and ω → ψ p (ω) applied in Γν f (z)m νp (z)dz, the result follows.

Remark 1 (Known Σ 1 ). For Σ 1 known (which would mean that Σ1 = Σ 1 valid for p fixed and n 1 → ∞), Theorem 7 is particularized by taking the limit c 1 → 0, i.e.,

f dν p - 1 2πı Γµ f z ψ p (z) - 1 z + ψ p (z) ψ p (z) ψ p (z) c 2 dz a.s.
-→ 0

where now m µp (z) = 1 p p i=1 1 λ i (Σ -1 1 Σ2 )-z and m µp (z) = 1 p p i=1 1 (λ i (Σ -1 1 Σ2 )-z) 2 .
Basic algebraic manipulations allow for further simplification, leading up to

f dν p - -1 2πıc 2 Γµ f -1 m μp (z) m μp (z)zdz a.s.
-→ 0

where μp = c 2 µ p + (1 -c 2 )δ 0 is the eigenvalue distribution of 1 n 2 X T 2 Σ -1 1 X 2 (and thus m μp (z) = c 2 m µp (z) -(1 -c 2 )/z). Letting g(z) = f (1/z) and G(z) such that G (z) = g(z)
, integration by parts of the above expression further gives

f dν p - 1 2πıc 2 Γµ G -m μp (z) dz a.s.
-→ 0.

For instance, for

f (z) = log 2 (z), G(z) = z(log 2 (z) -2 log(z) + 2).
We then retrieve the random matrix estimate of the population covariance matrix derived in chapter 2 for Σ 1 = I p .

Remark 2 (On the need for n 2 > p and n 1 > p in Theorem 7). Since distances involving the eigenvalues of Σ -1 1 Σ 2 are estimated from the empirical matrix Σ-1 1 Σ2 , the constraint n 1 > p is inevitable to ensure the existence of Σ-1

1 . The requirement for n 2 > p is less immediate. The two variable changes discussed in the proof of Theorem 7 are only licit if they realize a mapping from a contour Γ µ enclosing the limiting support Supp(µ) of µ p to a valid contour Γ ν enclosing the limiting support Supp(ν) of ν p while enclosing no additional singular points of the function f (z) (otherwise the Cauchy formula used in (3.1) is incorrect). But for n 2 < p, it can be proved (see details in Section 6.2.2 of the appendix) that the pre-image of Γ µ by the variable changes wraps around Supp(ν) and around zero (with leftmost real crossing depending on the ratio n 2 /p). This is a problem for all functions f (z) singular at z = 0. In particular, 1/z, log(z), log 2 (z) and √ z are examples of such invalid functions which, for some, additionally have a branch-cut terminating at zero (that no valid contour may cross). This discussion is most conveniently illustrated in Figure 3.1. In this figure, we represent in blue the support of the distribution of µ (left) and ν (right). The branch cuts of the integrand are represented in red. For proper integration, the integration contour represented in green needs to encircle all the support S µ while avoiding the branch cuts in red. For n 2 > p, this happens to be possible (top figure) while for n 2 < p (bottom figure), this is impossible.

Unfortunately, there seems to be no simple workaround to this situation. We (partially) solve in (Tiomoko & Couillet, 2019a) the problem by introducing entire functions (thus analytical over C) as substitutes for the locally non-analytic functions log(z), log 2 (z) and √ z intervening in the distance D(Σ 1 , Σ 2 ) formulation.

Our approach to extend the work to p ≥ n 2 consists in approximating (arbitrarily closely) the analytic functions f under study that present singularities around zero by entire functions, and particularly degree-N polynomials fN (z) defined by fN (z) = N n=0 a n z n . Our central argument relies in the fact that, since Σ a and Σ -1 a are bounded (as per Assumption 1), the limiting support Supp(ν) of ν p is a compact set strictly away from zero. As such, one needs not approximate f on the whole R + half-line (which would still pose problems in the vicinity of zero) but only on a subset [a, b] ⊂ (0, ∞) over which polynomials are universal approximators.

This gives rise to the extension of Theorem 7 provided in (Tiomoko & Couillet, 2019a). For simplicity for the rest of this chapter, we will mostly focus on the case p < n (Bottom) n 2 < p. For n 2 < p, the left real crossing of Γ ν is necessarily negative (even if the mass at {0} of Supp(µ) were not included in Γ µ ). In case of singularities or branch-cuts (shown in red for the log(z) and √ z functions), the contours are invalid.

p < n 1 . The interested reader can refer to (Tiomoko & Couillet, 2019a) for more details.

We should however stress that (Tiomoko & Couillet, 2019a) only extends the result to p > n 2 . The case p > n 1 is still an open research question.

Special cases

While Theorem 7 holds for all well-behaved f on Γ µ , a numerical complex integral is required in practice to estimate f dν p . It is convenient, when feasible, to assess the approximating complex integral in closed form, which is the objective of this section.

When f is analytic in the inside of Γ µ , the integral can be estimated merely through a residue calculus. This is the case notably of polynomials f (t) = t k . If instead f exhibits singularities in the inside of Γ µ , as for f (t) = log k (t), more advanced contour integration arguments are required. These arguments (branch cuts in particular) were introduced in Chapter 2 and detailed for the generalized variance estimation problem. For the sake of readability, we briefly mention the technical difficulties involved for the current cases of interest and defer all the cumbersome algebraic simplifications in Section 6.2.3 in appendix.

Importantly, Theorem 7 is linear in f . Consequently, the contour integral calculus for elaborate functions f , such as met in most metrics of practical interest, can be reduced to the integral calculus of its linear components.

In the remainder of this section, we focus on the integral calculus for the atomic functions f listed in Table 3.1.

Corollary 2 (Case f (t) = t and µ p = µ - p ). Under the conditions of Theorem 7,

tdν p (t) -(1 -c 1 ) tdµ p (t)
a.s.

-→ 0.

and in the case where c 1 → 0, this is simply tdν p (t) -tdµ p (t)

a.s.

-→ 0.

As such, the classical sample covariance matrix estimator tdµ p (t) needs only be corrected by a product with (1 -c 1 ). This result unfolds from Theorem 7 via a simple residue calculus.

Corollary 3 (Case f (t) = log(t) and µ p = µ - p ). Under the conditions of Theorem 7,

log(t)dν p (t) - log(t)dµ p (t) - 1 -c 1 c 1 log(1 -c 1 ) + 1 -c 2 c 2 log(1 -c 2 ) a.s.
-→ 0.

When

c 1 → 0, log(t)dν p (t) - log(t)dµ p (t) + 1-c 2 c 2 log(1 -c 2 ) + 1 a.s.
-→ 0.

Note interestingly that, for f (t) = log(t) and c 1 = c 2 , the standard estimator is asymptotically p, n 1 , n 2 -consistent. This is no longer true though for c 1 = c 2 but only a fixed bias is induced. This result is less immediate as the complex extension of the logarithm function is multi-form, causing the emergence of branch-cuts inside the contour.

We evaluate the integral here, and in the subsequent corollaries, by means of a careful contour deformation subsequent to a thorough study of the function log(ϕ p (z)/ψ p (z)) and to the identification of its branch-cut locations as performed in Chapter 2.

Corollary 4 (Case f (t) = log(1 + st) and µ p = µ - p ). Under the conditions of Theorem 7, let s > 0 and denote κ 0 the unique negative solution to 1 + s ϕp(x) ψp(x) = 0. Then we have

log(1 + st)dν p (t) - c 1 + c 2 -c 1 c 2 c 1 c 2 log c 1 + c 2 -c 1 c 2 (1 -c 1 )(c 2 -sc 1 κ 0 ) + 1 c 2 log (-sκ 0 (1 -c 1 )) + log 1 - t κ 0 dµ p (t)
a.s.

-→ 0.

In the case where c 1 → 0, this is simply

log(1 + st)dν p (t) - 1 + sκ 0 + log(-sκ 0 ) c 2 + log 1 - t κ 0 dµ p (t)
a.s.

-→ 0.

The proof of Corollary 4 follows closely the proof of Corollary 3, yet with a fundamental variation on the branch-cut locations as the singularities of log(1 + sϕ p (z)/ψ p (z)) differ from those of log(ϕ p (z)/ψ p (z)).

As opposed to the previous scenarios, for the case f (t) = log 2 (t), the exact form of the integral from Theorem 7 is non-trivial and involves dilogarithm functions (see its expression in (6.16) in the Appendix) which originate from numerous real integrals of the form log(x -a)/(x -b)dx appearing in the calculus. This involved expression can nonetheless be significantly simplified using a large-p approximation, resulting in an estimate only involving usual functions, as shown subsequently.

Corollary 5 (Case f (t) = log 2 (t) and µ p = µ - p ). Let 0 < η 1 < . . . < η p be the

eigenvalues of Λ - √ λ√ λT p-n 1 and 0 < ζ 1 < . . . < ζ p the eigenvalues of Λ - √ λ√ λT n 2
, where λ = ( λ1 , . . . , λp ) T , Λ = diag( λ), and λ is understood entry-wise. Then, under the conditions of Theorem 7,

log 2 (t)dν p (t) - 1 p p i=1 log 2 ((1 -c 1 ) λi ) + 2 c 1 + c 2 -c 1 c 2 c 1 c 2 ∆ η ζ T M ∆ η λ + (∆ η λ) T r - 2 p ∆ η ζ T N 1 p -2 1 -c 2 c 2 1 2 log 2 ((1 -c 1 )(1 -c 2 )) + (∆ η ζ ) T r a.s.
-→ 0

where we defined ∆ b a the vector with

(∆ b a ) i = b i -a i and, for i, j ∈ {1, . . . , p}, r i = log((1-c 1 ) λi ) λi
and

M ij =      λi λj -1-log λi λj ( λi -λj ) 2 , i = j 1 2 λ2 i , i = j , N ij =      log λi λj λi -λj , i = j 1 λi , i = j.
In the limit c 1 → 0 (i.e., for Σ 1 known), this becomes

log 2 (t)dν p (t) - 1 p p i=1 log 2 ( λi ) + 2 p p i=1 log( λi ) - 2 p ∆ λ ζ T Q1 p -2 1 -c 2 c 2 1 2 log 2 (1 -c 2 ) + ∆ λ ζ T q a.s.
-→ 0

with

Q ij =      λi log λi λj -( λi -λj ) ( λi -λj ) 2 , i = j 1 2 λi , i = j
, and q i = log( λi ) λi .

Corollary 6 (Case f (t) = (t) and µ p = µ + p ). Let λ1 ≤ . . . ≤ λp , with λi ≡ λ i ( Σ1 Σ2 ), and define {ξ i } p i=1 and {η i } p i=1 the (increasing) eigenvalues of Λ -1

n 1 λ λT and Λ - 1 n 2 λ λT
, respectively, where λ = λ1 , . . . , λp T , Λ = diag( λ) and √ . is understood entry-wise. Then, under Assumption 1,

(t)dν p (t) -D(X 1 , X 2 ; √ •) a.s. -→ 0 where, if n 1 = n 2 , D(X 1 , X 2 ; √ •) = 2 √ n 1 n 2 1 p p j=1 λj + 2n 2 πp p j=1 η j ξ j - ϕ p (x) ψ p (x) ψ p (x)dx
with ϕ p , ψ p defined in Theorem 7 and, if

n 1 = n 2 , D(X 1 , X 2 ; √ •) = 2n 1 p p j=1 λj -ξ j .
While still assuming an integral form (when n 1 = n 2 ), this formulation no longer requires the arbitrary choice of a contour Γ µ and significantly reduces the computational time to estimate

D(Σ 1 , Σ 2 , √ •).
For n 1 = n 2 , the expression is completely explicit and computationally only requires to evaluate the eigenvalues ξ j of Λ -1 n 1 λ λT . The latter being a (negative definite) rank-1 perturbation of Λ, by Weyl's interlacing lemma [START_REF] Joel N Franklin | Matrix theory[END_REF], the ξ j 's are interlaced with the λj 's as

ξ 1 ≤ λ1 ≤ ξ 2 ≤ . . . ≤ ξ p ≤ λp .
As the λj 's are of order O(1) with respect to p, | λj -

ξ j | ≤ | λj -λj-1 | = O(p -1 ), therefore explaining why the expression of D(X 1 , X 2 ; √ •) is of order O(1).

Applications

Confirmation of our results on synthetic data

In this section, we compare the Fisher distance estimate from Corollary 5 to the classical "plug-in" estimator 1 p p i=1 log 2 ( λi ). We first report in Table 3.2 the genuine versus estimated values of the Fisher distance on a synthetic setting (details in caption). A first surprising observation is that the plug-in estimator is extremely unfit to large values of p/n 1 , p/n 2 , bringing up to 500% error for n 1 = 2p; the proposed estimator is instead resilient to large p. Possibly more surprisingly, while Corollary 5 provably holds for asymptotically large p, n 1 , n 2 , our estimator already outperforms the standard approach for p = 2. This may be explained by the fact that the proposed approach essentially exploits randomness both from the size and the number of the dataset, with accuracies provably of order O(1/ √ pn) thereby already reaching accurate values for not too large p (note that this in particular implies central limit theorems and thus convergence speed quadratically faster than in the large-n 1 , n 2 alone setting).

p D F (Σ 1 , Σ 2 ) Classical
(error > 50%) (error > 100%) (error > 500%)
Table 3.2: Proposed versus classical estimator for the Fisher distance between Σ 1 and

Σ 2 with [Σ -1 2 1 Σ 2 Σ -1 2 1 ] ij = .3 |i-j| , x (a)
i ∼ N(0, Σ a ); n 1 = 1024 and n 2 = 2048 for different values of p. Averaged over 10 000 trials.

Application to covariance features-based classification

In this section, we develop a practical application of our theoretical findings to the machine learning context of kernel spectral clustering [START_REF] Von | A tutorial on spectral clustering[END_REF]. In several application domains, such as in brain signal processing [START_REF] Luiz | Multivariate time-series analysis via manifold learning[END_REF] or hyperspectral imaging [START_REF] Chang | Hyperspectral imaging: techniques for spectral detection and classification[END_REF], the relevant discriminative data "features" are the population covariance matrices of the data vectors (p-sensor brain activities, p-frequency spectra). Classification in these contexts is thus performed by comparing the distances between the population covariance matrices for each data-pair. Not being directly accessible, the population covariances are classically substituted by their sample estimates. We will show here that this method has strong limitations that our proposed improved distance estimates overcome.

Specifically, we consider m = 200 data X 1 , . . . , X m to be clustered. Datum X i is a p × n i independent generation of n i independent p-dimensional zero mean Gaussian samples

X i = [x (i) 1 , . . . , x (i) n i ] ∈ R p×n i . Half of the samples have covariance E[x (i) j x (i)T j ] = E[ 1 n i X i X T i ]
= C 1 (they will be said to belong to class 1) and half have covariance C 2 (they belong to class 2). For clarity, let us say that E[x

(i) j x (i)T j ] = C 1 for i ≤ m/2 and E[x (i) j x (i)T j ] = C 2 for i > m/2. We then define the kernel matrix K ∈ R m×m by K ij ≡ exp(- 1 2 DF (X i , X j ) 2 )
with DF either the classical (naive) estimator of D F or our proposed random matriximproved estimator. The purpose of spectral clustering is to retrieve the mapping between data indices and classes. It can be shown (see e.g., [START_REF] Von Luxburg | Consistency of spectral clustering[END_REF] but more fundamentally [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] in this large dimensional context) that, for sufficiently distinct classes (so here covariance matrices), the eigenvectors v 1 and v 2 of K associated to its largest two eigenvalues are structured according to the classes. Thus, the classes can be read out of a two-dimensional display of v 2 versus v 1 . This is our procedure in what follows.

A fundamental, but at first unsettling, outcome arises as we let n 1 = . . . = n m . In this case, for all tested values of p and m, the eigenvectors of K for either choice of DF are extremely similar. Consequently, spectral clustering performs the same, despite the obvious inaccuracies in the estimations of D F . This result can be explained from the following fact: spectral clustering is of utmost interest when C 1 and C 2 are close matrices; in this case, the classical estimator for D F is systematically biased by a constant, almost irrespective of the covariance (since both are essentially the same). This constant bias does affect K but not its dominant eigenvectors.

This observation collapses when the values of the n i 's differ. This is depicted in Figure 3.2. The top display provides a scatter plot of the dominant eigenvectors v 2 versus v 1 of K under the same conditions as above but now with n i chosen uniformly at random in [2p, 4p], with p = 128, m = 200, C 1 = I p and [C 2 ] ij = .05 |i-j| . There, for the classical estimator, we observe a wide spreading of the eigenvector entries and a smaller inter-class spacing. This suggests poor clustering performance. On the opposite, the well-centered eigenvectors achieved by the proposed estimator imply good clustering performances. In a likely more realistic setting in practice, the bottom display considers the case where n 1 = . . . = n m-1 = 512 and n m = 256. This situation emulates a data retrieval failure for one observation (only half of the samples are seen). In this scenario, the classical estimator isolates one entry-pair in (v 1 , v 2 ) (corresponding to their last entries). This is quite expected. However, more surprisingly, the presence of this outlier strongly alters the possibility to resolve the other data. This effect is further exacerbated when adding more outliers (not displayed here). This most likely follows from an adversarial effect between the outliers and the genuine clusters, which all tend to "drive" the dominant eigenvectors.

Application to covariance matrix estimation

In this section we start presenting the covariance matrix estimation procedure for the family of metrics expressed as linear spectral statistic of Σ -1 1 Σ 2 . We then furthermore explain the adaptations needed to handle the Wasserstein distance.

Estimation Method

In summary, our objective is to estimate Σ 2 as:

Σ2 = arg min M h(M ) (3.4) h(M ) = D(M, X 2 ; f (•)) 2 , (3.5)
where D(M, X 2 ; f (•)) 2 is the random-matrix estimate of the distance between any positive definite matrix M and the sought-

for covariance matrix Σ 2 denoted D(M, Σ 2 ) = p j=1 f (λ j (M -1 Σ 2 )).
We solve (3.4) via a gradient descent algorithm on the Riemannian manifold S ++ p of positive definite p × p matrices.

The Riemannian gradient ∇h(M

) of h at M ∈ S ++ p is defined via the directional derivative Dh(M )[ξ] of the functional h : S ++ p → R + , at position M ∈ S ++ p
and in the direction of ξ ∈ S p (the vector space of symmetric p × p matrices), by [START_REF] Absil | Optimization algorithms on matrix manifolds[END_REF])

Dh(M )[ξ] = ∇h(M ), ξ S ++ p M where •, • S ++ p .
is the Riemannian metric defined through η, ξ

S ++ p M = tr M -1 ηM -1 ξ . f (z) G(z) log 2 (z) z log 2 (z) -2 log(z) + 2 log(z) -z log(z) + z log(1 + sz) s log(s + z) + z log s+z z z log(z) f (z) F (z) log 2 (z) z log 2 (z) -2 log(z) + 2 log(z) z log(z) -z log(1 + sz) 1 s + z log(1 + sz) -z z 1 2 z 2
Table 3.3: G(z) and F (z) for "atomic" f (z) functions used in most distances and divergences under study; here s > 0 and z ∈ C.

Differentiating D2 (M, X 2 ) at M in the direction ξ yields:

Dh(M )[ξ] = -D(M, X 2 ) πic 2 Γ g(-m μp (z, M ))Dm μp (z, M ) [ξ]dz.
where Γ is a contour surrounding the support of the almost sure limiting eigenvalue distribution of M -1 Σ2 . By using the fact that

Dm μp (z, M ) [ξ] = c 2 p Dtr M -1 Σ2 -zI p -1 [ξ] = c 2 p tr M -1 Σ2 M -1 Σ2 -zI p -2 M -1 ξ = c 2 p sym Σ2 M -1 Σ2 -zI p -2 , ξ S ++ p M where sym(A) = 1 2 (A + A T ) is the symmetric part of A ∈ R p×p , we retrieve the gradient of h(M ) as -ıπp ∇h(M ) D(M, X 2 ) = Γ g -m μp (z; M ) sym Σ2 (M -1 Σ2 -zI p ) -2 dz (3.6)
(recall that the right-hand side still depends on X 2 implicitly through μp and Σ2 ).

Once ∇h estimated, every gradient descent step in S ++ p corresponds to a small displacement on the geodesic starting at M and towards -∇h(M ), defined as the curve

R + → S ++ p t → M 1 2 exp -tM -1 2 ∇h(M )M -1 2 M 1 2
where, for A = U ΛU T ∈ S ++ p in its spectral decomposition, exp(A) ≡ U exp(Λ)U T (with exp understood here applied entry-wise on the diagonal elements of Λ).

That is, letting M 0 , M 1 , . . . and t 0 , t 1 , . . . be the successive iterates and step sizes of the gradient descent, we have, for some given initialization M 0 ∈ S ++ p ,

M k+1 = M 1 2 k exp -t k M -1 2 k ∇h(M k )M -1 2 k M 1 2 k . (3.7)
Our proposed method is summarized as Algorithm 1.

Algorithm 1 Proposed estimation algorithm.

Require M 0 ∈ S ++ p . Repeat M ← M 1 2 exp -tM -1 2 ∇h(M )M -1 2 M 1 2 with t either fixed or optimized by backtracking line search. Until Convergence 1 Return M . Estimation of Σ -1 2
In our framework, estimating Σ -1 2 rather than Σ 2 can be performed by minimizing

D(M, Σ -1 2 ) instead of D(M, Σ 2 ). In this case, under Assumption 1, D(M, Σ -1 2 ) -Dinv (M, X 2 ) → 0 almost surely, for every deterministic M of bounded operator norm, where Dinv (M, X 2 ) ≡ 1 2πıc 2 Γ F -m μinv p (z; M ) dz for F such that F (z) ≡ f (z)
, where Γ is a contour surrounding the support of the almost sure limiting eigenvalue distribution of M Σ2 and μinv

p = p n 2 µ inv p + (1 -p n 2 )δ 0 , where µ inv p ≡ 1 p p i=1 δ λ i (M Σ2
) . The cost function to minimize under this setting is now given by h inv (M ) ≡ ( Dinv (M, X 2 )) 2 with gradient ∇h inv (M ) satisfying

ıπp ∇h inv (M ) Dinv (M, X 2 ) = Γ f -m μinv p (z; M ) sym M Σ2 (M Σ2 -zI p ) -2 M dz.
With these amendments, Algorithm 1 can be adapted to the estimation of Σ -1 2 . Table 3.3 provides the values of F for the atomic functions f of interest.

Application to Explicit Metrics

Algorithm 1 is very versatile as it merely consists in a gradient descent method for various metrics f through adaptable definitions of the function h(M ) = D(M, X 2 ) 2 and its resulting gradient. Yet, because of the integral form assumed by the gradient (Equation (3.6)), a possibly computationally involved complex integration needs to be numerically performed at each gradient descent step.

In this section, we specify closed-form expressions for the gradient for the atomic f functions of Table 3.3 (which is enough to cover the list of divergences in Table 3.1).

Let us denote

∇h(M ) ≡ 2 D(M, X 2 ) • sym Σ2 • V Λ ∇ V -1
where V are the eigenvectors of M -1 Σ2 and Λ ∇ is to be determined for each f . For readability in the following, let us denote λi ≡ λ i (M -1 Σ2 ), i ∈ {1, . . . , p}, the eigenvalues of the matrix M -1 Σ2 and ξ 1 , . . . , ξ p the eigenvalues of Λ-1 n 2 λ λT with Λ = diag( λ1 , . . . , λp ) and λ = ( λ1 , . . . , λp ) T . Finally, for s > 0, let κ s ∈ (-1/(s(1 -p/n 2 )), 0) be the unique negative number t solution of the equation (see Section 6.2.3 of the appendix for details) m μp (t) = -s.

With these notations at hand, following the derivations in Section 6.2.3 of the appendix, we have the following determinations for Λ ∇ .

Proposition 1 (Case f (t) = t). [Λ ∇ ] kk = - 1 c 2 + 1 p p i=1 1 m μp (ξ i ) λk -ξ i 2 with m μp the derivative of m μp . Proposition 2 (Case f (t) = log(t)). [Λ ∇ ] kk = -1 p λk . Proposition 3 (Case f (t) = log(1 + st)). For s > 0, [Λ ∇ ] kk = -1 p( λk -κ s ) . Proposition 4 (Case f (t) = log 2 (t)). For f (t) = log 2 (t), [Λ ∇ ] kk = 2 p log λk    p i=1 1 λk -ξ i - p i=1 i =k 1 λk -λi - 1 λk    - 2 p p i=1 log(ξ i ) λk -ξ i + 2 p p i=1 i =k log( λi ) λk -λi - 2 -2 log(1 -c 2 )
p λk .

These results unfold from residue calculus for entire functions f or advanced complex integration tools for logarithmic functions which is described in Section 6.2.4.

The Wasserstein case We obtain the relation of the gradient for the Wasserstein distance ( 1p tr(M + Σ2 ) -2 D(M, X 2 ; √ •)) similarly as provided previously.

πıp ∇h(M ) 2 h(M ) = 1 p M 2 + p j=1 λj ξ j 1 m μp (x) sym M Σ2 (M Σ2 -xI p ) -2 M dx
where sym(A) = 1 2 (A + A T ) is the symmetric part of A ∈ R p×p and where the definition of ξ j and λj is given in Corollary 6. We can write the latter as:

∇h(M ) = 2 h(M ) sym V Λ ∇ V -1 + 1 p M 2
where V is the orthogonal matrix of the eigenvectors of M Σ2 and Λ ∇ is the diagonal matrix with

[Λ ∇ ] kk = 1 πp j =k λj ξ j 1 m μp (x) 1 ( λk -x) 2 dx + 1 πp j =k λk ξ k 1 m μp (x) 1 ( λj -x) 2 dx.
Experiments on the Wasserstein distance Figure 3.3 depicts the results of the algorithm. There is displayed the Wasserstein distance D W (Σ 2 , •) between a matrix Σ 2 having four distinct eigenvalues of equal multiplicity (precisely, ν p = 1 4 (δ .1 + δ 3 + δ 4 + δ 5 )) and various estimators of Σ 2 : the sample covariance matrix (SCM), the state-of-the-art "non-linear shrinkage" estimators QuEST1 [START_REF] Ledoit | Spectrum estimation: A unified framework for covariance matrix estimation and pca in large dimensions[END_REF] (based on a Frobenius distance minimization) and QuEST2 [START_REF] Ledoit | Optimal estimation of a large-dimensional covariance matrix under stein's loss[END_REF] (based on a Stein loss minimization), and the result of the gradient descent approach proposed in this section. For fair comparison, the iterative QuEST1, QuEST2 and our proposed method are all initialized at M 0 the linear shrinkage estimator from [START_REF] Ledoit | A well-conditioned estimator for large-dimensional covariance matrices[END_REF]. Note that our proposed choice of Σ 2 is particularly suited to mimick an "optimal transport" problem of displacing the eigenvalues of M 0 to the discrete four positions of the eigenvalues of Σ 2 .

In addition to the computational simplicity of our gradient-descent approach with respect to the QuEST estimators (see the numerical method details in [START_REF] Ledoit | Numerical implementation of the quest function[END_REF]), the figure demonstrates significant gains brought by our proposed approach for large values of p/n 2 , where the SCM particularly fails.

Concluding Remarks

Conclusion. The present study has revealed a strong lack of consistency for the traditional "plug-in" covariance matrix-distance (and divergence) estimators, when the

n 2 Proposed SCM QuEST1 QuEST2 Figure 3.3: Wasserstein distance D W (Σ 2 , •) between Σ 2 with ν p = 1 4 (δ .1 + δ 3 + δ 4 + δ 5
) and (green) our proposed estimator, (blue) the sample covariance matrix, (red) and (light blue) the QuEST estimators proposed in [START_REF] Ledoit | Optimal estimation of a large-dimensional covariance matrix under stein's loss[END_REF][START_REF] Ledoit | Spectrum estimation: A unified framework for covariance matrix estimation and pca in large dimensions[END_REF]; for p = 100 and varying number of samples n 2 averaged over 10 realizations. data dimension p is not small. This is particularly dramatic as p and the number of snapshots n are close. We provided a consistent solution to recover consistency, exploiting random matrix tools.

Importantly, by exploiting both randomness in p and n, our estimator converges as fast as O(1/ √ pn), but a more precise central limit analysis is required to exactly assess confidence intervals, which is yet another avenue of research. But the real strength and robustness of the proposed estimator will only be demonstrated when applied to real (non Gaussian) datasets and more exotic applications. Brain signal processing (or human-machine interaction) and radar imaging (Synthetic Aperture Radar (SAR) or hyperspectral) are both interesting application candidates that shall be investigated in the future.

Introducing the Multi-Task Learning analysis. Many analyses of machine learning algorithms have been conducted recently in high-dimensional statistics, but so far limited to"simple" algorithms (classical support vector machine (SVM) (Liao & Couillet, 2019), spectral clustering [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], semi-supervised graph learning (SSL) [START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF]). These studies have highlighted the prominent and unique role of means and covariance matrices in the understanding of these algorithms. This emphasizes once again the need for a thorough understanding of covariance functionals which has been the focus of the present chapter. However, richer and more complex methods such as multi-task learning, transfer learning, fairness learning, privacy and security in machine learning, etc. that involve multiple biases that are difficult to trace (too many parameters, too much heterogeneity in the data, etc.) are ultimately the ones that can gain the most from leveraging RMT, which will identify these biases. We show that this is indeed the case, that there are multiple biases especially in the case of multi-task learning, but that the RMT cleans them one by one, while keeping a remarkable algorithmic simplicity. This is the focus and goal of the next chapter.

Motivation and main findings

Multi-Task Learning motivation. The methodology for a long time considered in machine learning has consisted in tackling each given (classification, regression, estimation) problem, hereafter referred to as a task, independently. This approach is in general counterproductive as it automatically discards a potentially rich source of data often available to perform more or less similar tasks. Multi-Task Learning (MTL) precisely aims to handle this deficiency by connecting datasets and tasks so to improve the generalization performance of one or several specific target tasks. This framework has recently gained renewed interest [START_REF] Yang | Transfer learning[END_REF][START_REF] Caruana | Multitask learning[END_REF][START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF], given the availability of gigantic datasets (such as huge prelabeled image databases) and costly trained learning machines (such as deep neural nets), which must be useful to help solve learning tasks involving much fewer labeled data. Beyond this resurgence, numerous applications inherently benefit from an MTL approach, of which we may cite a few examples: prediction of student test results for a collection of schools [START_REF] Aitkin | Statistical modelling issues in school effectiveness studies[END_REF], patient survival estimates in different clinics [START_REF] Harutyunyan | Multitask learning and benchmarking with clinical time series data[END_REF][START_REF] Caruana | Using the future to" sort out" the present: Rankprop and multitask learning for medical risk evaluation[END_REF], values of possibly related financial indicators [START_REF] Greg | Marketing models of consumer heterogeneity[END_REF], preference modeling of many individuals in a marketing context [START_REF] William H Greene | Econometric analysis 4th edition[END_REF], etc.

Task relatedness modeling in MTL. Carefully modeling the relatedness between tasks has long been claimed to be the most critical determinant of the MTL algorithm performance. Several such models have been considered in the literature: task relatedness can be modeled by assuming that the parameters relating the tasks lie on a low dimensional manifold [START_REF] Argyriou | Multi-task feature learning[END_REF][START_REF] Agarwal | Learning multiple tasks using manifold regularization[END_REF]; these relating parameters may alternatively be assumed to be close in norm [START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF] or be distributed according to similar priors [START_REF] Xue | Multi-task learning for classification with dirichlet process priors[END_REF][START_REF] Yu | Learning gaussian processes from multiple tasks[END_REF]. However, for all these models, a failure in properly matching the task parameters is often likely to induce possibly severe cases of negative learning, that is occurrences where additional tasks play against rather than in favor of the target task objective. These cases of negative learning are difficult to anticipate as few theoretical works are amenable to prepare the experimenter to these scenarios. In the present work, we adopt a similar strategy as in [START_REF] Evgeniou | Regularized multi-task learning[END_REF], but with a strong theoretical background which will automatically eliminate the risks of negative learning.

A parameter-based modeling approach. In detail, the article [START_REF] Evgeniou | Regularized multi-task learning[END_REF], the spirit of which is followed here, is inspired by the natural extension of support vector machines (SVMs) [START_REF] Vapnik | Universal learning technology: Support vector machines[END_REF] to a multiple, say k, task setting, by paralleling k SVMs but constraining their parameters (specifically, the k separating hyperplane normal vectors ω 1 , . . . , ω k ) to be "close" to each other. This is enforced by simply imposing that ω i = ω 0 + v i for some common hyperplane normal vector ω 0 and dedicated hyperplane normal vectors v i . The norm of the vectors v i is controlled through an additional hyperparameter λ to strengthen or relax task relatedness. This is the approach followed in this chapter, to the noticeable exception that the fully explicit least-square SVM (LS-SVM) [START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF] rather than a margin-based SVM is considered. In addition to only marginally altering the overall behavior of the MTL algorithm of [START_REF] Evgeniou | Regularized multi-task learning[END_REF], the LS-SVM approach entails more explicit, more tractable, as well as more insightful results, let alone numerically cheaper implementations. As a matter of fact, by a now well-established universality argument of large dimensional statistics, it has been shown in closely related works (Mai & Liao, 2019) that quadratic (least-square) cost functions are asymptotically optimal (as the data dimension and number increase) and uniformly outperform alternative costs (such as margin-based methods or logistic approaches) in terms of 0 -1 classification error, even in a classification setting; this argument further motivates to consider first and foremost the least square version of MTL-SVM.

Main findings and chapter contributions. This chapter develops a theoretical framework to exhaustively study the behavior and maximize the performance of a k-task m-class MTL LS-SVM framework, under the regime of numerous (n) and large (p) data, i.e., n, p → ∞ with n/p → c 0 ∈ (0, ∞). The data are here modeled as a mixture of km concentrated random vectors, i.e., for x a data of class j (j ∈ {1, . . . , m}) for Task i

(i ∈ {1, . . . , k}), x ∼ L ij (µ ij , Σ ij )
, where L ij (µ, Σ) is the law of a Lipschitz-concentrated random vector [START_REF] Ledoux | The concentration of measure phenomenon[END_REF] with statistical mean µ ∈ R p and covariance Σ ∈ R p×p . For instance, x = ϕ ij (z) for z ∼ N(0, I q ), ϕ ij : R q → R p a 1-Lipschitz function and lim q/p ∈ (0, ∞). The statistical modeling for the data considered here has been briefly mentioned in chapter 2 as one of the big advantage of RMT tools in the sense that the theoretical insights and conclusions are robust to real data.

The main results and practical consequences of the chapter may be summarized as follows:

• we exhibit sufficient statistics, which concretely enable task comparison in the MTL LS-SVM algorithm; we show that, even when data are of large dimensions (p 1), these statistics remain small dimensional (they only scale with the number k of tasks);

• while it is conventional to manually set labels associated to each dataset within {-1, 1}, we prove that this choice is largely suboptimal and may even cause MTL to severely fail (causing "negative transfer"); we instead provide the optimal values for the labels of each dataset, which depend on the sought-for objective: these optimal values are furthermore easily estimated from very few training data (i.e., no cross-validation is needed);

• for unknown new data x, the MTL LS-SVM algorithm allocates a class based on the comparison of a score g(x) to a threshold ζ, usually set to zero. We prove that, depending on the statistics and number of elements of the training dataset, a bias is naturally induced that makes ζ = 0 a largely suboptimal choice in general. We provide a correction for this bias, which again can be estimated from the training data alone;

• we demonstrate on popular real datasets that our proposed optimized MTL LS-SVM is both resilient to real data and also manages, despite its not being a best-in-class MTL algorithm, to rival and sometimes largely outperform competing state-of-theart algorithms.

In a nutshell, by exploiting random matrix theory tools explained in Section 2.3 (i,e,. deterministic equivalents of random matrices), the work provides a modern vision to multi-task and transfer learning. This vision is here turned into an elementary but costefficient algorithm, which relies on base principles, but which both largely outperforms competing (sometimes complex) methods and provides strong theoretical guarantees. As a side note, we must insist that our present objective is to study and improve "datageneric" multi-task learning mechanisms under no structural assumption on the data; this is quite unlike recent works exploiting convolutive techniques in deep neural nets to perform transfer or multi-task learning mostly for computer vision-oriented tasks, as in e.g., (Zhuang et al., 2020;[START_REF] Tulasi | Deep learning and transfer learning approaches for image classification[END_REF].

Chapter organization. In order to best capture the main intuitions drawn from the large dimensional analysis, after a rigorous introduction of the multi-task learning framework in Section 4.2, a first highlight of our main contributions under the qualitatively more telling setting of binary tasks (m = 2) with data of equal identity covariance (Σ ij = I p ) is proposed in Section 4.3. The technical details under the most generic data modeling setting as well as the most general technical result are then provided in Section 4.4. A broad series of applications is provided in Section 4.5. Extensive simulations are then proposed in Section 4.6, which corroborate our theoretical findings and show their resilience and compatibility to real data settings.

Notation. e

[n]

m ∈ R n is the canonical vector of R n with [e [n] m ] i = δ mi . Moreover, e [2k] ij = e [2k] 2(i-1)+j . Similarly, E [n] ij ∈ R n×n is the matrix with [E [n] ij ] ab = δ ia δ jb .
The notations A ⊗ B and A B for matrices or vectors A, B are respectively the Kronecker and Hadamard products. D x is the diagonal matrix containing on its diagonal the elements of the vector x and A i• is the i-th row of A. The notation Å is used when a centering operation is performed on the matrix or vector A. Uppercase calligraphic letters (A, K, Γ, M, V,...) are used for deterministic small dimensional matrices. Finally, 1 m and I m are respectively the vector of all one's of dimension m and the identity matrix of dimension m × m. The index pair i, j generally refers to Class j in Task i.

The Multi-Task Learning Framework

The deterministic setting

Let X ∈ R p×n be a collection of n independent data vectors of dimension p. The data are divided into k subsets attached to individual "tasks", each task consisting of an m-class classification problem (m being the same for each task). Specifically, letting X = [X 1 , . . . , X k ], Task i is a classification problem from the training samples

X i = [X (1) i , . . . , X (m) i ] ∈ R p×n i with X (j) i = [x (j) i1 , . . . , x (j) in ij ] ∈ R p×n ij the n ij vectors of class C j , j ∈ {1, . . . , m}, for Task i. In particular, n = k i=1 n i and n i = m j=1 n ij for each i ∈ {1, . . . , k}.
To each datum x (j) il ∈ R p of the training set is attached a corresponding output vector (or score) y

(j) il ∈ R m . Correspondingly to the notation X, X i and X (j) i , let Y = [Y T 1 , . . . , Y T k ] T ∈ R n×m be the matrix of the m-dimensional outputs of all data, where Y i = [Y (1)T i , . . . , Y (m)T i ] T ∈ R n i ×m and Y (j) i = [y (j) i1 , . . . , y (j)
in ij ] T ∈ R n ij ×m the matrix of all outputs for Task i.

In the standard MTL learning approach [START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF], one would naturally set y

(j) il = e [m]
j , i.e., all data of class C j are affected a hot-bit in position j. As claimed in the introduction and as we shall see, this hot-bit allocation approach is at the source of deleterious performances, such as negative transfer effects, and we thus voluntarily do not enforce any constraint on the vector y (j) il at this point. Before inserting the data-score pairs (X, Y ) into the MTL LS-SVM framework, it is convenient to "center" the data X to eliminate additional sources of bias. This centering operation could be performed either on the whole dataset X, or task-wise on each X i , or even class-wise on each X (j) i . In [START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF] this centering operation is not performed (which essentially boils down to centering X itself). We choose here to center the data task-wise, and this, for two reasons: (i) centering the whole dataset induces dependencies across tasks so that, even by enforcing the hyperplane controlling factor λ to decorrelate the tasks (i.e., λ → ∞; see next), residual dependence must remain and negative transfer can still appear, (ii) class-wise centering has the double deleterious effect of canceling an important discrimination factor of the classes (i.e., their difference in statistical mean) and of necessitating a complex treatment to classify new (unlabeled) input data. Inappropriate centering choices would induce biases and undesired residual terms in our theoretical derivation, which further justifies our present task-wise centering choice (see e.g., Remark 4). Specifically, the MTL LS-SVM algorithm studied here is based, not on the data X i but on their centered version Xi = X i I n i -

1 n i 1 n i 1 T n i , ∀i ∈ {1, . . . , k},
and we will systematically consider the data-score pair ( X, Y ), where X = [ X1 , . . . , Xk ] rather than (X, Y ).

Having pre-treated the input data, we are in position to introduce the MTL LS-SVM framework. The MTL LS-SVM algorithm aims to predict, relative to each task i, an output score vector y i ∈ R m for any new input vector x ∈ R p . To this end, MTL LS-SVM determines k "hyperplane normal-vector" matrices

W = [W 1 , W 2 , . . . , W k ] ∈ R p×km which take the form W i = W 0 + V i for some common W 0 and individual task-wise matrices V = [V 1 , . . . , V k ] and biases b = [b T 1 , b T 2 , . . . , b T k ] T ∈ R k×m .
These parameters are set to minimize the objective function min

(W 0 ,V,b)∈R p×m ×R p×km ×R k×m J(W 0 , V, b) (4.1)
where

J(W 0 , V, b) ≡ 1 2λ tr W T 0 W 0 + 1 2 k i=1 tr V T i V i γ i + 1 2 k i=1 tr ξ T i ξ i ξ i = Y i -( XT i W i √ kp + 1 n i b T i ), ∀i ∈ {1, . . . , k}.
This is a classical LS-SVM formulation in which the quadratic cost tr(ξ T i ξ i ) replaces the boundary constraint of margin-based SVM and where the costs tr(W T 0 W 0 ) and tr(V T i V i ) are reminiscent of the hyperplane normal-vector norm minimization of classical SVM.

What is specific to the MTL approach is first the hyperparameter λ which enforces or relaxes the relatedness between tasks and the introduction of k extra parameters γ 1 , . . . , γ k which enforce a correct classification of the data in their respective classes. Similarly to [START_REF] Evgeniou | Regularized multi-task learning[END_REF], we place the hyperparameters γ i as a prefactor of tr(V T i V i ), rather than as a prefactor of tr(ξ T i ξ i ); this differs from the normalization scheme proposed in [START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF]. This choice is more flexible in the following sense: for a fixed value of λ, increasing all ratios λ γ i "blurs" the difference between tasks and thus turns the optimization scheme into a single-task SVM (because the optimal V i 's need then be set to zero in the limit); for fixed values of the γ i 's instead, small ratios λ γ i decorrelate the tasks (the optimal W 0 being close to zero). Note however that, unlike in [START_REF] Evgeniou | Regularized multi-task learning[END_REF], we choose to use here one hyperparameter γ i per task instead of a common one. As will be seen next, this choice is more meaningful and of course offers more flexibility.

In passing, remark that the linear common-hyperplane condition W i = W 0 + V i , imposes by definition that all V i 's be of the same size R p×m : this severely constrains (i) the data in each task to be of the same dimension p and (ii) the number of classes per task to be the same (m). Further linear or even non-linear relaxation schemes for W i of the type W i = V i + f i (W 0 ) for some operator f i could be envisioned to relax this constraint. This however goes beyond the scope of the chapter, which seeks to provide insights and optimality into a simplified (yet already non-trivial) form of MTL LS-SVM.

As for the choice of the hyperparameters λ, γ 1 , . . . , γ k , as well as of the score matrix Y which we recall was left open, it is treated independently and is dictated, not by the present optimization scheme, but by an ultimate objective, such as minimizing the misclassification rate for a specific target class. These more applied considerations will be made in Section 4.5.

Remark 3 (LS-SVM classification versus regression).

It may be disputed that the optimization framework (4.1) takes a regression rather than a classification form. It appears that, under a binary-class LS-SVM framework with scores y i ∈ {±1}, the classification constraint (of the form y i (W T x i + b i ) -1 = ξ i ) or the regression constraint (of the form y i -W T x i + b i = ξ i ) are associated to the same losses, thereby leading to the same classification solution and performance. Yet, as will become clear in the following, in addition for the solution of (4.1) to be explicit and theoretically tractable (which is not the case of alternative schemes such as margin-based SVM, logistic regression, Adaboost, etc.), the aforementioned flexibility in the score matrix Y largely outbalances the "failure" of treating a classification problem by means of a regression optimization scheme. Besides, under the large dimensional theoretical framework presently studied, recent works in related problems (Mai & Liao, 2019) forcefully suggest that the square loss is optimal to deal with large dimensional data as it uniformly outperforms all alternative cost functions in terms of 0 -1 classification error.

Being a quadratic cost optimization under linear constraints, (4.1) is easily solved using its dual formulation by introducing Lagrangian parameters α i ∈ R n i ×m for each task i (see details in Section 6.3.1). The solution is explicit and is as follows.

Proposition 5. The solution to (4.1) is given by

W 0 = 1 T k ⊗ λI p Z √ kp α W i = e [k] i T ⊗ I p A Z √ kp α b = (P T QP ) -1 P T QY where Z =   X1 . . . Xk   ∈ R kp×n A = D γ + λ1 k 1 T k ⊗ I p ∈ R kp×kp α = Q(Y -P b), Q = 1 kp Z T AZ + I n -1 ∈ R n×n P =   1 n 1 . . . 1 n k   ∈ R n×k .
Despite the apparent intricate expression of W i , it must be stressed that W i "essentially" takes the form of the standard solution to a ridge regression considered in chapter 2 (or regularized least-square) problem as the term AZQY (in which Q = ( 1 kp Z T AZ + I n ) -1 ) appearing in the expended form of W i confirms. From a technical standpoint, the large dimensional statistical behavior of the matrix Q, known as the resolvent of 1 kp Z T AZ in random matrix theory as defined in chapter 2, plays a central role in the analysis. More specific to the MTL framework, note the interesting isolation of the data subsets Xi in the data matrix Z (it is not possible, to the best of our knowledge, to "linearly" express W i as a function of X itself); the elements Xi are then "mixed" by the term λ1 k 1 T k appearing in matrix A, from which it naturally comes that, in the limit λ → 0, MTL LS-SVM boils down to k independent LS-SVMs with D γ imposing weights γ 1 , . . . , γ k on each data subset.

From Proposition 5, for any new data point x ∈ R p , the classification score vector g i (x) ∈ R m for Task i, is then defined by the linear model considered previously

g i (x) = 1 √ kp W T i x + b i = 1 kp α T Z T A e [k] i ⊗ x + b i (4.2)
where x = x -1 n i X i 1 n i is a centered version of x with respect to the training dataset for Task i.

This formulation, along with the next remark, confirm again the relevance of a task-wise, rather than class-wise, centering of the data X, which allows for a well-defined expression of x.

Remark 4 (Shift invariance of the scores). If the columns of Y i ∈ R n i ×m are shifted by some constant vector P Ȳ for some (small dimensional) matrix Ȳ ∈ R k×m , i.e., if all data of the same task are affected by the same shift of their scores (or labels), then we find that the Lagrangian parameter α shift after the shift is

α shift = Q I n -P (P T QP ) -1 P T Q (Y + P Ȳ) = α.
As such, the matrix

W i = (e [k]T i ⊗ I p )A Z
√ kp α and, consequently, the performance of MTL LS-SVM are insensitive to a simultaneous shift of all the scores of each task.

Statistical modeling and the large dimensional setting

In order to draw insights into the behavior of MTL LS-SVM and evaluate its performance, we propose to first model the dataset X as a mixture of concentrated random vectors and then to assume the dimensions p, n of X to be sufficiently large for deterministic (and predictable) concentration behavior to occur.

Assumption 2 (Distribution of X and x). There exist two constants C, c > 0 (independent of n, p) such that, for any 1-Lipschitz function f : R p×n → R,

∀t > 0, P(|f (X) -E[f (X)]| ≥ t) ≤ Ce -(t/c) 2
We further impose that the columns of X be independent and that the x (j) il , for l ∈ {1, . . . , n ij }, be distributed according to the same law L ij . These conditions guarantee the existence of a mean and covariance for the columns of X and we denote, for all l ∈ {1, . . . , n ij },

µ ij ≡ E[x (j) il ] Σ ij ≡ Cov[x (j) il ].
Furthermore, the dummy variable x ∈ R p used for testing is independent of X, and distributed according to one of the laws L ij .

Assumption 2 notably encompasses the following scenarios: the x (j) il 's are (i) independent Gaussian random vectors N(µ ij , Σ ij ), (ii) independent random vectors uniformly distributed on the R p sphere of radius √ p and, most importantly, (iii) any 1-Lipschitz transformation ϕ ij (z

(j) il ) with z (j)
il itself a concentrated random vector. Scenario (iii) is particularly relevant to model very realistic data by means of advanced non-linear generative models, as recently demonstrated in [START_REF] Seddik | Kernel random matrices of large concentrated data: the example of gan-generated images[END_REF] in the specific example of generative adversarial networks (GANs). As such, Assumption 2 offers the flexibility to assume either synthetic Gaussian mixture models, or very realistic and advanced generative data models. A core result of the chapter consists in showing that, for n, p large, either scenario leads to the same asymptotic performance for MTL LS-SVM (which thus only depends on the statistical means and covariances of the data).

Since all data x (j) il , l ∈ {1, . . . , n ij }, are identically distributed, we will further impose that their associated scores y

(j) il ∈ R m be identical. That is, y (j) i1 = . . . = y (j)
in ij ≡ Y ij within every class j of each task i. The score matrix Y ∈ R n×k may then be reduced under the form

Y = Y 11 1 T n 11 , . . . , Y km 1 T n km T ∈ R n×m for Y = [Y 11 , . . . , Y km ] T ∈ R km×m .
From Remark 4, it is also clear that, the performances of MTL LS-SVM being insensitive to a constant shift in the scores Y i1 , . . . , Y im in every

given task i, the centered version Y = [ Y11 , . . . , Ykm ] T of Y, where Yij ≡ Y ij - m j=1 n ij n i Y ij ,
will naturally appear at the core of the upcoming results.

Although practical data will of course be considered to be of finite dimension p and number n, it will indeed be convenient, for technical reasons, to work under the following large dimensional random matrix assumption.

Assumption 3 (Growth Rate). As n → ∞, n/p → c 0 ∈ (0, ∞) and, for 1

≤ i ≤ k, 1 ≤ j ≤ m, n ij /n → c ij ∈ (0, 1). We further denote c i = k j=m c ij and c = [c 1 , . . . , c k ] T ∈ R k .
With these notations and assumptions in place, we are in position to present the main results of the chapter. Yet, before entering the technical details of the large dimensional analysis of the performance of the MTL LS-SVM framework, the next section first provides a highlight of the main contributions and intuitions drawn by the analysis. To this end, it is convenient to temporarily restrict the setting to binary classes (m = 2) and to an isotropic mixture model for the data X, i.e., Σ ij = I p for each measure L ij . The most general and slightly more technical setting (m ≥ 2 and non-isotropic mixture data modeling) is considered in full in Section 4.4.

Highlights of the main results

To simplify the exposition of our main results, without impacting their core conclusions, in this section, Assumptions 2-3 are further restricted to the binary-classification setting (m = 2) and to measures L ij of equal covariance Σ ij = I p , for all i, j.

The advantage of the isotropic (Σ ij = I p ) condition is that all asymptotic results can be expressed under the form of low-dimensional matrix formulations (of size scaling with k but not with p, n). Adjoined to the m = 2 assumption, the isotropic model further guarantees a simplified form for (i) the (asymptotically) optimal labels Y , (ii) the optimal decision thresholds ζ i , and (iii) the asymptotic performances of MTL LS-SVM, all of which can be estimated consistently as p, n → ∞. Consequently, this simplified setting has the strong benefit to give rise to a first cost-efficient and robust multi-task classification algorithm (Algorithm 2) which, for practical data, makes the approximation that Σ ij ∝ I p .

The binary setting does not a priori alter any of the previously introduced notations which stand with m = 2. Yet, it is particularly convenient in this setting to recast the score vectors y il is here considered as a real score (rather than a binary label) vector, to us this is equivalent to turning a score vector into a scalar score. Matrix Y ∈ R n×m similarly now becomes a score vector y bin ∈ R n , and in particular we define ẙbin = [ẙ bin 11 , . . . ,

ẙbin k2 ] T ∈ R 2k with ẙbin ij ≡ y bin ij - n i1 n i y bin i1 + n i2 n i y bin i2 ∈ R
where y bin ij ≡ y bin i1 (j) = . . . = y bin in ij (j) ∈ R is the common score assigned to the identically distributed data of class j for Task i. Correspondingly, the sought-for (W i , b i ) collection of m hyperplanes of (4.1) becomes a single hyperplane (w bin i , b bin i ) with w bin i ∈ R p and b bin i ∈ R. Yet, our present interest is only on the resulting score vector g i (x) which, replacing Y by y bin in its expression (Equation 4.2), becomes the scalar test score

g bin i (x) ≡ 1 kp (y bin -P b) T QZ T A e [k] i ⊗ x + [(P T QP ) -1 P T Qy bin ] i ∈ R.

Theoretical analysis and large dimensional intuitions

Under the isotropic and binary-class setting, as n, p → ∞ according to Assumption 3, the theoretical performance of MTL LS-SVM explicitly depends on two fundamental and isolated quantities: the data-related matrix M ∈ R 2k×2k and the hyperparameter matrix

A ∈ R k×k : M = k i,i =1 ∆µ T i ∆µ i E [k] ii ⊗ c i c T i A = I k + D -1 2 δ [k] D γ + λ1 k 1 T k -1 D -1 2 δ [k]
-1

where we introduced the shortcut notations

∆µ i ≡ µ i1 -µ i2 , c i ≡ c i1 /c i c i2 /c i c i2 /c i -c i1 /c i and where δ [k] = [δ [k] 1 , . . . , δ [k]
k ] T are the unique positive solutions to the implicit system of k equations

δ [k] i = c i c 0 -A ii , i ∈ {1, . . . , k}. (4.3)
In anticipation of future needs, it is convenient to further introduce the 2k-dimensional variant δ

[2k] = [δ [2k] 11 , . . . , δ [2k] k2 ] T ∈ R 2k where δ [2k] ij = c 0 c ij c i δ [k] i . (4.4)
The asymptotic performances of MTL LS-SVM will be shown to solely depend on X through the matrices M and A, which thus play the role of (asymptotically) sufficient statistics. It is particularly important to stress that, despite the quite generic concentration assumption on X (Assumption 2), when Σ ij = I p , only the k 2 inner products ∆µ T i ∆µ i and the 2k class-wise dimensionality ratios c ij /c i intervene in the expression of M -so in particular none of the higher order moments of X are accounted for, nor the absolute task-wise dimension ratios c i . As for A, it captures instead the information about the impact of the hyperparameters λ, γ 1 , . . . , γ k as well as the task-wise dimensionality ratios c 1 , . . . , c k and the data number-to-dimension ratio c 0 . In the expression of the MTL LS-SVM performance, these two matrices combine into the core matrix ∈ R 2k×2k

Γ = I 2k + A ⊗ 1 2 1 T 2 M -1 (4.5)
where we recall that ' ' is the Hadamard (element-wise) matrix product.

Theorem 8 (Asymptotics of g bin i (x)). Under Assumptions 2-3, with m = 2 and Σ ij = I p , for a test data x with E[x] = µ ij and Cov[x] = I p , as p, n → ∞,

g bin i (x) -G ij a.s. -→ 0, G ij ∼ N(m ij , σ 2 i ) in distribution
, where, letting m = [m 11 , . . . , m k2 ] T and the normalized forms andσ

y bin ≡ D 1 2 δ [2k] y bin , ẙbin = D 1 2 δ [2k] ẙbin , m = D 1 2 δ [2k] m,
2 i = δ [k] i σ 2 i , m = y bin -Γẙ bin σ 2 i = (ẙ bin ) T ΓV i Γẙ bin with V i = D K T i. ⊗1 2 + AD K T i• +e [k] i A ⊗ 1 2 1 T 2 M K = c 0 k [A A] D c - c 0 k [A A] -1
.

Theorem 8 interestingly indicates that the (asymptotic) statistics of the classification scores g bin i (x), for 1 ≤ i ≤ k, reduce to a mere functional of 2k-dimensional deterministic vectors and matrices. In particular, g bin i (x) depends on the data statistical means µ i j , 1 ≤ i ≤ k, 1 ≤ j ≤ 2, and on the hyperparameters λ and γ 1 , . . . , γ k mostly through the 2k-dimensional matrix Γ (and more marginally through V i and K for the variances).

Another non-trivial point to note is that, being in general non-diagonal, Γ acts on the centered scores (labels) ẙbin i j of all classes j and tasks i which, therefore, all influence the performances. It can thus be anticipated that, for the decision on a particular Task i to be successful, not only the scores y bin i1 and y bin i2 , but in fact all scores y bin i j across all classes and tasks, must be appropriately tuned.

Remark also that, in this isotropic (Σ i j = I p ) setting, the variance σ 2 i of the score g bin i (x) with E[x] = µ ij only depends on i, and not on j. This is particularly convenient, as shown next, to devise an optimal decision rule for classification into class 1 or 2 for Task i.

From a more technical standpoint, comparing the exact expression of g i (x) in (4.2) and that of m ij (i.e., the large dimensional approximation of E[g i (x)]), we may interpret the matrix Γ ∈ R 2k×2k as a "condensed" form of Q ∈ R n×n . From the expression (I 2k + A ⊗ 1 2 1 T 2 ) -1 M, observe that: (i) if λ 1, then A is diagonal dominant and thus "filters out" in the Hadamard product all off-diagonal entries of M -that is, all the cross-terms ∆µ T i ∆µ j for i = j -, therefore refusing to exploit the correlation between tasks; (ii) if instead λ ∼ 1, then A may be developed (using the Sherman-Morrison matrix inverse formulas) as the sum of a diagonal matrix, which again filters out the ∆µ T i ∆µ j for i = j, and of a rank-one matrix which instead performs a weighted sum (through the γ i and the δ

[k] i ) of the entries of M; specifically, letting γ -1 = (γ -1 1 , . . . , γ -1 k ) T , we have

D γ + λ1 k 1 T k -1 = D -1 γ - λγ -1 (γ -1 ) T 1 + λ 1 k k i=1 γ -1 i .
As such, letting aside the regularization effect of the δ

[k]

i 's, the off-diagonal ∆µ T i ∆µ j term intervening in the expression of M is weighted by a coefficient (γ i γ j ) -1 : the impact of the γ i 's is thus strongly associated to the relevance of the correlation between tasks, and not only to the individual performances of the k isolated LS-SVM tasks. Section 4.4 provides a more general version (Theorem 9) of Theorem 8 for m ≥ 2 classes per task and generic Σ ij . The technical derivation of these two results, of limited interest at this point, is also deferred to Section 4.4.

Decision threshold and label optimization

Since g bin i (x) has a Gaussian limit centered about m ij and with equal variance for j = 1 and j = 2, the (asymptotically) optimal decision for x to be allocated to class C 1 or class C 2 for Task i, i.e., the decision minimizing the averaged error probability under the prior P(x ∈ C 1 ) = P(x ∈ C 2 ), is obtained by the "averaged-mean" test

g bin i (x) C 1 ≷ C 2 ζ i ≡ 1 2 (m i1 + m i2 ) (4.6)
the associated misclassification rate being

i1 ≡ P g bin i (x) ≥ m i1 + m i2 2 x ∈ C 1 = Q m i1 -m i2 2σ i + o(1) (4.7)
with m ij , σ i as in Theorem 8 and

Q(t) = 1 √ 2π ∞ t e -u 2 2 du.
It is of utmost interest at this point to recall that the asymptotics of g bin i (x) from Theorem 8 (as from the more generic Theorem 9) depend in an elegant and simple manner on the training data scores y bin = D -1 2 δ [2k] y bin . Using again the independence of σ 2 i on the genuine class of x, the vector y bin minimizing the misclassification rate for Task i simply reads:

y bin = arg max y bin ∈R 2k (m i1 -m i2 ) 2 σ 2 i = arg max y bin ∈R 2k (y bin ) T (I 2k -Γ)D -1 2 δ [2k] (e [2k] i1 -e [2k]
i2 ) 2 (y bin ) T ΓV i Γy bin for which the solution is explicitly defined, up to an arbitrarily multiplicative constant (as it maximizes a ratio) and up to an arbitrary additive constant (as per Remark 4), by:

y bin = Γ -1 V -1 i [(A⊗1 2 1 T 2 ) M]D -1 2 δ [2k] (e [2k]
i1 -e

[2k] i2 ). (4.8)
and, for this choice of y bin , the corresponding (asymptotically) optimal classification error i1 defined in (4.7) is then

i1 = Q 1 2 (e [2k]
i1 -e

[2k] i2 ) T G(e [2k]
i1 -e

[2k] i2 ) (4.9) for G = D 1 2 δ [2k] [(A ⊗1 2 1 T 2 ) M]V -1 i [(A ⊗1 2 1 T 2 ) M]D 1 2 δ [2k] . Of course, by symmetry, i2 ≡ P (g bin i (x) ≤ m i1 +m i2 2 |x ∈ C 2 ) has the same limiting optimal value i2 = i1 .
The only non-diagonal matrices in (4.8) are Γ and V i in which M plays the role of a "variance profile" matrix. In particular, assume ∆µ T i ∆µ i = 0 for all i = i, i.e., the differences in statistical means of all tasks are orthogonal to those of Task i. Then the two rows and columns of M associated to Task i are all zero but on the 2 × 2 diagonal block. Therefore, y bin will have all zero entries but on its Task i two elements. All other choices for the null entries of y bin (such as the usual y bin = [1, -1, . . . , 1, -1] T ) would be suboptimal and (possibly severely) detrimental to the classification performance of Task i, not by altering the means m i1 , m i2 but by increasing the variance σ 2 i . This extreme example strongly suggests that, in order to maximize the MTL performance on a targeted Task i, one must impose low absolute scores y bin i j to all Tasks i strongly different from Task i.

The choice y bin = [1, -1, . . . , 1, -1] T can also be very detrimental when ∆µ T i ∆µ i < 0 for some pair i, i : that is, when the mapping of the two classes within each task is reversed (e.g., if class C 1 in Task 1 is closer to class C 2 than class C 1 in Task 2). In this setting, it is easily seen that y bin = [1, -1, . . . , 1, -1] T works against the classification and performs much worse than a single-task LS-SVM.

Another interesting conclusion arises from the simplified setting of equal number of samples per task and per class, i.e., n 11 = . . . = n k2 . In this case, δ

[k2] 11 = . . . = δ [k2]
k2 and, since y bin is defined up to a multiplicative constant, we have

y bin = Γ -1 V -1 i (A ⊗ 1 2 1 T 2 ) M (e [2k]
i1 -e

[2k] i2 )
in which all matrices are organized in 2×2 blocks of equal entries. This immediately implies that y bin i 1 = -y bin i 2 for all i . So in particular, the detection threshold 1 2 (m i1 + m i2 ) of the averaged-mean test (4.6) is zero (as conventionally assumed). In all other settings for the n i j 's, it is very unlikely that y bin i1 = -y bin i2 and the optimal decision threshold must also be estimated. As a matter of fact, following up on Remark 4, the aforementioned optimal value y bin for y bin is not unique and could be shifted by any constant vector. This extra degree of freedom will be of much relevance in the application Section 4.5, as commented in the following remark.

Remark 5 (Setting the decision threshold to zero). As per Remark 4, the addition of a constant term to y bin does not affect the ultimate performance of MTL LS-SVM. Yet, it affects the value of the limiting means m ij of g bin i (x), so in particular the value of the limiting optimal threshold 1 2 (m i1 + m i2 ). Specifically, one may shift all entries of y bin in such a way that 1 2 (m i1 + m i2 ) = 0 and thus recenter the decision threshold to zero. For ȳ ∈ R this constant shift, this boils down to solving in the variable ȳ the equation

0 = 1 2 (m i1 + m i2 ) = 1 2 (y bin + ȳe [k] i ⊗ 1 2 ) T D 1 2 δ [2k] (I 2k -Z e Γ) (e [2k] 2(i-1)+1 + e [2k]
2i

)
where

Z e = I 2k -k i =1 E [k] i i ⊗ c i and c i = 1 2 n i 1 n i n i 2 n i
. Similarly, one may instead impose that m i1 = 0: this will appear to be fundamental to align classifiers in the multiclass "one-versus-all" extension of the present binary classification scheme (see details in Section 4.5.2).

Remark 6 (Tuning the hyperparameters). The previous section provided a high-level interpretation for the impact of the vector parameter γ ∈ R k and the scalar parameter λ ∈ R, the effect of which is to respectively regularize LS-SVM learning and to set the throttle between individual versus collective learning. These hyperparameters intervene deeply inside our theoretical formulas (so far in Theorem 8 but later in Theorem 9) and are not amenable to simple optimization. Yet, as will be confirmed by experiments (see in particular Figure 4.3), the proposed optimization of the input scores y bin partly compensates for suboptimal choices in γ, λ. As such, an "informed guess", based on our previous discussion of the effects of these parameters, is in general sufficient for highly performing MTL LS-SVM. A further gradient descent operation (or local grid search) on the theoretical performance approximation, initialized at the informed guess values, can further improve the overall learning performance.

Practical implementation of improved MTL LS-SVM

As already pointed out, a fundamental aspect of Theorem 8 lies in the performances of the large dimensional (n, p 1) classification problem at hand boiling down to 2k-dimensional statistics. More importantly from a practical perspective, these 2kdimensional "sufficient statistics" are easily amenable to fast and efficient estimation: it indeed only requires a few training data samples to estimate all quantities involved in the theorem (which, as a corollary, lets one envision the possibility of efficient transfer learning methods based on very scarce data samples).

Remark 7 (On the estimation of m ij and σ i ). All quantities defined in Theorem 8 are a priori known, apart from the quantities M ≡ i,i

∆µ T i ∆µ i E [k]
ii ⊗ c i c T i and most specifically the inner products ∆µ T i ∆µ i . For these, define, for j = 1, 2, two sets S ij , S ij ⊂ {1, . . . , n ij } and the corresponding indicator vectors j ij , j ij ∈ R n i with [j ij ] a = δ a∈S ij and [j ij ] a = δ a∈S ij . We further impose that S ij ∩ S ij = ∅. Then, for i = i , the following estimates hold:

∆µ T i ∆µ i - j i1 |S i1 | - j i2 |S i2 | T XT i Xi j i 1 |S i 1 | - j i 2 |S i 2 | = O (p min l∈{1,2} {|S il |, |S i l |}) -1 2 ∆µ T i ∆µ i - j i1 |S i1 | - j i2 |S i2 | T XT i Xi j i1 |S i1 | - j i2 |S i2 | = O (p min l∈{1,2} {|S il |, |S il |}) -1 2 .
Observe in particular that a single sample (two when i = i ) per task and per class (|S il | = 1) is sufficient to obtain a consistent estimate for all quantities, so long that p is large. In a transfer learning setting where some tasks may contain few labeled data, it is thus still possible to optimize the MTL algorithm. Of course, when more data are available, under our assumption that p ∼ n, taking all samples in the averaging, the convergence speed is of order O(1/ √ np) = O(1/n), which is a quadratic increase in the speed of the usual central-limit theorem.

Estimating m ij and σ i not only allows one to anticipate theoretical performances but also enables the actual estimation of the decision threshold1 2 (m i1 + m i2 ) of the test (4.6) and, as shown previously, opens the possibility to largely optimize MTL LS-SVM through an (asymptotically) optimal choice of the training scores y bin .

The series of theoretical and practical results of this section may be synthetized under the form of Algorithm 2.

Algorithm 2 Proposed binary Multi-Task Learning algorithm.

Input: Training samples X = [X 1 , . . . , X k ] with X i = [X (1) i , X (2)
i ] and test data x. Output: Estimated class ĵ ∈ {1, 2} of x for target Task i. Center and normalize data per task: for all i ∈ {1, . . . , k},

• Xi ← X i I n i -1 n i 1 n i 1 T n i • Xi ← Xi / 1 n i p tr( Xi XT i )
Estimate: Matrix M from Remark 7 and δ [k] by solving (4.3).

Create scores y bin = y bin according to (4.8).

Compute the threshold ζ i from (4.6), with m ij defined in Theorem 8 for y bin = y bin .

(Optional) Estimate the theoretical classification error i1 = i1 (λ, γ) from (4.7) and minimize over (λ, γ). 1 Compute classification score g i (x) according to (4.2).

Output: ĵ such that g i (x)

ĵ=1 ≷ ĵ=2 ζ i .

Empirical evidence

This section shortly illustrates the ideas and intuitions developed so far (such as the relevance of an optimal choice of the data labels and decision threshold) through the performances of Algorithm 2 on a transfer learning benchmark application. Sections 4.5-4.6 will cover a much larger spectrum of applications and experiments, under the most general data setting discussed in the subsequent sections.

For optimal comparison, we consider here the standard Office+Caltech256 real image classification benchmark [START_REF] Saenko | Adapting visual category models to new domains[END_REF][START_REF] Griffin | Caltech-256 object category dataset[END_REF], consisting of four tasks and m = 10 categories shared by all tasks. The dataset X consists here of the VGG features of size p = 4096 extracted from these images. We place ourselves under a k = 2 transfer learning setting where Task 1 is the source task and Task 2 is the target task (the performance of which we aim to optimize), taken from two of the four tasks of the dataset (Caltech, Webcam, Amason, dslr). For testing, the samples of the target task are randomly selected from the test dataset of Office+Caltech256 and the classification accuracy is averaged over 20 trials. Table 4.1 reports the accuracy for all possible pairs (4 × 3 = 12 of them) of source and transfer tasks, obtained by Algorithm 2 (Ours) versus the non-optimized LS-SVM of [START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF] (LS-SVM) and versus other state-of-the-art transfer learning algorithms: the max-margin domain transform of [START_REF] Hoffman | Efficient learning of domain-invariant image representations[END_REF] (MMDT) which seeks a linear transform to match the source data to the target data and then applies an SVM on the resulting target domain; the cross-domain landmark selection (CDLS) of (Hubert [START_REF] Tsai | Learning cross-domain landmarks for heterogeneous domain adaptation[END_REF], which learns a feature subspace which matches the cross-domain data distribution and eliminates the domain differences; and the invariant latent space (ILS) of [START_REF] Herath | Learning an invariant hilbert space for domain adaptation[END_REF], which, similar to MMDT, learns an invariant latent space in which the discrepancy between source and target is minimized. As already pointed out in introduction, since we aim to propose an improved classification algorithm independent of the feature representation, it is fair to compare it to methods which use the same data features. The algorithms compared in the table all systematically use VGG features. It would be unfair to compare these against "end to end" MTL learning methods including a (explicit or implicit) step of feature learning like recent deep neural networks methods (Zhuang et al., 2020;[START_REF] Tulasi | Deep learning and transfer learning approaches for image classification[END_REF].

Since m = 10 here, Algorithm 2 cannot rigorously be used as it stands. We apply instead a naive "one-versus-all" extension consisting in running in parallel m = 10 times Algorithm 2 by considering, for each class C j of Task i, 1 ≤ j ≤ m, a binary setting where the fictitious "Class C1 " coincides with C j and the second fictitious "Class C2 " is the union of all C j for j = j. Following up on Remark 5, each classifier ∈ {1, . . . , m} can be set in such a way that E[g bin i (x; )] = 0 when x ∈ C . For a new datum x, of all m classifiers g bin i (x; 1), . . . , g bin i (x; m), the one reaching the greatest score is the selected allocation class for x.

Table 4.1 demonstrates that our proposed improved MTL LS-SVM, despite its simplicity and unlike the competing methods used for comparison, has stable performances and is extremely competitive. It either outperforms all other methods or is second-tobest. But, most importantly, the method comes along with performance predictions and guarantees, which none of the competing works are able to provide.2

These preliminary results are already very conclusive and reveal the strength of our proposed methodology. Yet, the assumptions in place so far are restricted to random concentrated data with identity covariance and to a binary classification setting (which, as already observed, needs be adapted to account for more than two classes per task). The next sections elaborate on the more generic setting of m ≥ 2 classes per task with more realistic data models. The theoretical results no longer reduce to compact expressions as

The General Framework

The results from the previous section are extended here to the more realistic setting where the data arise from a mixture of m ≥ 2 concentrated random vectors with generic covariance Σ ij . New insights, and most importantly, more general and application-driven algorithms will be introduced. In addition, the results are presented here with a sketched development of their main technical arguments, the full proofs being deferred to the appendix.

Main ideas

Taking for the moment for granted the Gaussian limit for g i (x) ∈ R m as p, n → ∞ (for 1 ≤ i ≤ k), the main technical task to obtain our main result (Theorem 9, which generalizes the already introduced Theorem 8) is to evaluate the large dimensional behavior m ij and σ ij of the statistical mean E[g i (x)] = m ij + o(1) and covariance matrix Cov[g i (x)] = σ ij + o(1) of the classification score g i (x) in (4.2) for data vectors x in class C j (i.e., such that E[x] = µ ij and Cov[x] = Σ ij ), respectively given from (4.2) by:

m ij = E 1 kp (Y -P b) T Z T A 1 2 QA 1 2 e [k] i ⊗ µ ij + b i (4.10) σ ij = E 1 (kp) 2 (Y -P b) T Z T A 1 2 QA 1 2 S ij A 1 2 QA 1 2 Z(Y -P b) (4.11) with S ij = e [k] i e [k] i T ⊗ Σ ij and Q = A 1 2 ZZ T A 1 2 kp + I kp -1
.

Our technical approach to evaluate these terms, in the large dimensional regime of Assumption 3 and for data distributed as per Assumption 2, consists in determining deterministic equivalents for the matrices Q, QA

1 2 Z, Z T A 1 2 QA 1 2 S ij A 1 2 QA 1 2
Z which are at the core of the formulation of m ij and σ ij .

Lemma 5, deferred to Section 6.3.2 of the appendix (as the result in itself does not bring any deep insight worth discussing here), provides the necessary deterministic equivalents for these matrices. It is interesting to point out though that, from a technical standpoint, the block structure followed by the core data matrix Z introduced in Proposition 5 makes the large dimensional random matrix analysis more challenging and the result less straightforward than in similar previous works (Mai & Liao, 2019;Liao & Couillet, 2019) and in Section 2.3 for the ridge regression problem. Even in the simplest setting where the x (j) il would be vectors of i.i.d. N(0, 1) entries, the matrix Z is not a matrix of i.i.d. entries (due to precisely located blocks of zeros) and the singular values of Z do not asymptotically follow the popular Marcenko-Pastur distribution, as was the case in Section 2.3.

The main information to be extracted from Lemma 5 (again, see its complete form in the appendix) is the central role played by the deterministic matrices

M = e [k] 1 ⊗ [µ 11 , . . . , µ 1m ], . . . , e [k] k ⊗ [µ k1 , . . . , µ km ] C ij = A 1 2 e [k] i e [k] i T ⊗ (Σ ij + µ ij µ T ij ) A 1 2
which generalize the matrices M and A discussed at length in Section 4.3 when Σ ij = I p . While gaining in genericity, unlike M, the matrices M and C ij preserve their large dimensions: this is the main price paid by the generalization to Σ ij = I p . Yet, the central small dimensional matrix Γ defined in (4.5) remains small and now becomes

Γ = I mk + M T Q0 M -1 Q0 =   k i=1 m j=1 (D γ + λ1 k 1 k ) 1 2 e [k] i e [k] i T (D γ + λ1 k 1 k ) 1 2 ⊗ δ [mk] ij Σ ij + I kp   -1 M = A 1 2 M D 1 2
δ [mk] and the mk scalars δ

[mk] ij are the unique positive solutions of the fixed point equations

δ [mk] ij = c ij c 0 1 + 1 kp tr(C ij Q) Q =   k i=1 m j=1 δ [mk] ij C ij + I kp   -1
.

Here, Q is a deterministic equivalent of Q. Finally, the matrix K appearing in the variance term of Theorem 8 now becomes

K = c 0 T (D c -c 0 T) -1 Tij,i j = δ [mk] ij δ [mk] i j kp tr C i j QA 1 2 S ij A 1 2 Q T ij,i j = δ [mk] ij δ [mk] i j kp tr(C ij QC i j Q)
where Tij,i j is the element at row m(i -1) + j and column m(i -1) + j of T (and similarly for T) and κ ij,. ∈ R mk represents the m(i -1) + j row of matrix κ ∈ R mk×mk . With these technical elements at hand, we are in position to enunciate the main result of the chapter.

Classification score asymptotics

Theorem 9. Under Assumptions 2 and 3, for a test data x with E

[x] = µ ij and Cov[x] = Σ ij , as p, n → ∞, g i (x) -G ij → 0, G ij ∼ N(m ij , σ ij )
in law where, letting m = [m 11 , . . . , m km ] T ∈ R km×m and the normalized forms

Y ≡ D 1 2 δ [mk] Y, Y = D 1 2 δ [mk] Y, m = D 1 2 δ [mk] m, m = Y -Γ Y ∈ R m σ ij = YT ΓV ij Γ Y ∈ R m×m with V ij = D κ ij,. + M T Q0 V ij Q0 M V ij = A 1 2 S ij A 1 2 + k i =1 m j =1 δ [mk] i j κ ij,i j A 1 2 S i j A 1 2 κ ij,i j = K ij,i j δ [mk] ij .
Proof. See Section 6.3.4 of the appendix.

In the particular case of Σ ij = I p and m = 2, Theorem 9 reduces to Theorem 8 (see details in Section 6.3.4 of the appendix) by remarking that

M T Q0 M = A ⊗ 1 k 1 T k M and M T Q0 V ij Q0 M = AD κ i +e [k] i A ⊗ 1 k 1 T k
M which, as already pointed out, have the advantage to be defined as the product of exclusively small dimensional matrices. Still, although more technical, Theorem 9 follows the same structure as Theorem 8.

Before concretely applying the result of Theorem 9 to practical learning problems (multi-task, transfer learning, hypothesis testing), a few comments and immediate corollaries are in order.

Remark 8 (Optimization of y bin for m = 2 and generic Σ ij ). As suggested in Section 4.3, for binary classification (m = 2), it is particularly convenient to recast the score vectors y (j) il ∈ R m into scalar scores y bin(j) il ∈ R (this being irrespective of the nature of Σ ij ). Inspired by Section 4.3, one can trivially extend Theorem 9 to this binary setting. In this case, g i (x) ∈ R m is now turned into a scalar g bin i (x) ∈ R well approximated by N(m ij , σ ij ) where now m ij and σ ij are scalar, obtained by simply replacing y (j) il ∈ R m by y bin(j) il ∈ R in their respective expressions. With these notations, setting the decision threshold of g bin i (x) to ζ ∈ R and assuming equal prior probability for the genuine class of x, the classification error rate for a target task i is

E = 1 2 Q ζ -m i1 √ σ i1 + 1 2 Q ζ -m i2 √ σ i2 .
As in Section 4.3, if Σ i1 = Σ i2 , then σ i1 = σ i2 ≡ σ i and the decision threshold ζ minimizing the classification error is

ζ = m i1 + m i2 2
from which the optimal vector y bin for Task i is computed as

y bin = arg max y bin ∈R 2k (m i1 -m i2 ) 2 σ i = D -1 2 δ [2k] Γ -1 V -1 ij M T Q0 MD -1 2 δ [2k] (e [2k]
i1 -e

[2k] i2 ). (4.12)

It is important to recall here that, while y bin expresses here solely as a function of terms involving the index i, all other statistics of the tasks i = i are in fact "embedded" inside these terms and are thus, of course, accounted for in the optimization.

When σ i1 = σ i2 (which is the case in general), one may minimize E by resorting to numerical optimization techniques. We suggest to use a gradient descent method initialized to the expression obtained in (4.12). So long that Σ i1 and Σ i2 are not drastically different, this approach shows good performances (see our results in Section 4.6).

This said, the specific setting of binary classification may in practice be one of hypotheses testing. Under this scenario, one may not demand that the average error E be minimized (i.e., that data from either class is equally well identified) but rather that the probability of misclassification of a given class (say, a Type-I error) be bounded to some η > 0 while minimizing the error rate for the other class (Type-II error). In this context, if, say, one fixes Q( ζ-m i1 √ σ i1 ) ≡ η, then the classification error for the second class

Q( ζ-m i2 √ σ i2
) is minimized by now choosing

y bin = arg max y bin ∈R 2k ( √ σ i1 Q -1 (η) + m i1 -m i2 ) 2 σ i2
where Q -1 is the inverse function of the Q function. This again can be solved using numerical convex optimization initialized at the value of (4.12). More details on this hypothesis testing setting, along with concrete experiments, are developed in Section 4.6.

Remark 9 (Estimation of m ij and σ ij ). In order to anticipate the performances and best set the decision thresholds for classification, one needs to access all quantities arising in Theorem 9. Yet, as opposed to Remark 7, where the low dimensional quantities of interest (mainly the inner products between statistical means) are easily estimated, the low dimensional quantities involved in Theorem 9 are less convenient to estimate, this being due to the presence of the a priori unknown covariance matrices Σ ij . We propose here two strategies:

1. either make the assumption that Σ ij α ij I p with α ij estimated by 1 pn ij tr X(j) i X(j)T i ; then normalize the data as

X(j) i ← X(j) i / 1 pn ij tr X(j) i X(j)T i
in the spirit of Algorithm 2. This places the experimenter under an isotropic data setting for all classes and tasks, from which the considerations of Section 4.3 (possibly generalized to m > 2) apply.

either estimate Σ ij by means of the sample covariance matrix

1 n ij X(j) i X(j)T i
; this procedure is known to be biased, particularly so if n ij is not large compared to p; yet, as demonstrated in our experiments in Section 4.6, this only marginally (if not at all) alters the performance of our proposed algorithms. 3The choice of strategy mainly depends on the belief from the experimenter that the genuine covariance matrices are "well-conditioned" (i.e., their eigenvalues do not spread much) in which case Option 1 would be favored or "ill-conditioned" (typically when the space spanned by the data is much lower than p) in which case Option 2 would be more appropriate.

Remark 10 (On universality). As pointed out in the introduction, the input data X follows a very generic concentrated random vector assumption (Assumption 2). This choice provides both a technical, but most importantly, a fundamentally practical, advantage:

1. from a technical standpoint, the concentration of measure phenomenon provides efficient and fast mathematical tools (Louart & Couillet, 2018;[START_REF] Ledoux | The concentration of measure phenomenon[END_REF] to analyze the random quantities appearing in the classification test scores g i (x) of MTL LS-SVM (which, in essence, is a mere functional R p×n → R of the random input data X). More specifically, alternative random matrix tools based on Gaussian (Pastur & Shcherbina, 2011) or independent entries assumptions [START_REF] Bai | Spectral Analysis of Large Dimensional Random Matrices[END_REF] of X would both be less general (at least for our machine learning purpose) and more computationally intense;

2. on the practical side, as underlined in Section 4.2.2, the concentrated random vector assumption better models realistic datasets by imposing very little structure on the data. Exactly, it only constrains all Lipschitz functionals R p×n → R of X (i.e., its typical observations) to satisfy a concentration inequality; while this may seem demanding, the family of concentrated random vectors in fact contains all Lipschitz generative models (for instance neural networks) fed by Gaussian inputs (such as GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF]), as well as all further Lipschitz transformations of these vectors (for instance, features extracted by pretrained neural networks). As such, provided that the assumption of a common statistical mean and covariance per class and per task is reasonable, Theorem 9 ensures for instance that the performance of MTL LS-SVM applied to classes of the popular VGG or ResNet representations of GAN images is predictable. From this remark, it naturally comes that the proposed method is universal in the sense of its being robust to a broad range of very realistic random data, and it is not daring to claim that it is equally valid on genuinely real data. This is confirmed by our numerical results of Section 4.6.

With these elements in place, we are in position to apply our findings to a host of applications in statistical learning and to test the resulting algorithms against state-ofthe-art alternatives.

Applications

This section provides various applications and optimizations of the proposed MTL LS-SVM framework based on the findings of the previous sections in the context of multi-class classification.

Having access to the large dimensional behavior of the classification test score in Theorem 9 (i.e., for m ≥ 2 classes per task) is more fundamental than one may think. It indeed allows for a fine-tuning of the hyperparameters to be set to extend the usually considered binary MTL framework of [START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF] to a multiclass-per-task MTL.

Multi-class classification preliminary

The literature [START_REF] Christopher | Pattern recognition and machine learning[END_REF][START_REF] Rocha | Multiclass from binary: Expanding oneversus-all, one-versus-one and ecoc-based approaches[END_REF] describes broad groups of approaches for dealing with m > 2 classes. We focus here on the most common methods, namely one-versus-all, one-versus-one, and one hot encoding. Being so far theoretically intractable, these methods inherently suffer from sometimes severe limitations; these are partly tackled by adapting the theoretical results discussed in Section 4.4:

1. one-versus-all: in this method, focusing on Task i, m individual binary classifiers g bin i ( ) for = 1, . . . , m are trained, each of them separating Class C from the other m -1 classes C , = . Each test sample is then allocated to the class with the highest score among the m classifiers. Although quite used in practice, the approach first suffers a severe data unbalancing effect when using binary (±1) labels as the set of negative labels in each binary classification is on average m -1 times larger than the set of positive labels, and also suffers a centering-scale issue when ultimately comparing the outputs of the m decision functions g bin i (x; ), = 1, . . . , m, whose average locations and ranges may greatly differ; these issues lead to undesirable effects, as reported in (Bishop, 2006, section 7.1.3)).

In Section 4.5.2, these problems are simultaneously addressed: specifically, having access to the theoretical statistics of the classification scores allows us to appropriately center and scale the scores. Moreover, each binary classifier is optimized by appropriately choosing the class labels (no longer binary) so to minimize the resulting classification error (see Figure 4.1 for an illustration of the improvement induced by the proposed approach).

2. one-versus-one: here, 1 2 m(m -1) binary classifiers are trained (one for each pair j, j of classes, solving a binary classification). For each test sample, each binary classifier decides on -or "votes" for -the more relevant class. The test sample is then attributed to the class having the majority of votes. Although the number of binary classifiers is greater than in the one-versus-all approach, the training process for each classifier is faster since the training database is much smaller for each binary classifier. Besides, the method is more robust to class imbalances (since only pairwise comparisons are made) but suffers from an undecidability limitation in the case of equal numbers of majority votes for two or more classes.

In Section 4.5.3, each binary classifier will be optimized according to Algorithm 2 by choosing appropriate labels and appropriate decision thresholds, thereby largely improving over the basal classifier performance.

3. one-hot encoding approach, also known as one-per-class coding: in this approach, each class is encoded using the m-dimensional canonical vector of the class (the code vector for class j has a 1 at position j and 0's elsewhere). When testing an unknown sample x, the index of the encoding output vector g i (x) ∈ R m with maximum value is selected as the class of x.

Exploiting the asymptotic performance of this approach from Theorem 9 allows us to derive a different label (or score) encoding for each class which theoretically minimizes the classification error. This is developed in detail in Section 4.5.4.

In the remainder of the section, each of the three classifiers is studied, optimized and their asymptotic performances are analyzed according to our previous results except for one-versus-one classification which involves difficult combinatorial aspects. While this does not provide a definite and general answer as to which of the three classifiers is best, it however provides an accurate assessment of their asymptotic performances; most importantly, these performances may be evaluated before running the classifier, thereby helping practitioners to anticipate and optimize the method best suited for the application at hand, without resorting to any cross-validation procedure.

Let us finally insist that, for the two multi-class extensions based on binary classifiers (one-versus-one, one-versus-all), each binary classifier will be optimized independently following Remark 8: i.e., by recasting the score vectors y (j) il ∈ R m into scalar scores y bin(j) il ∈ R. As such, from now on, for each binary classifier , g i (x; ) ∈ R m will be systematically turned into a scalar g bin i (x, ) ∈ R well approximated by N(m ij , σ ij ) where now m ij and σ ij are scalar, obtained by simply replacing y (j) il ( ) ∈ R m by y bin(j) il ( ) ∈ R in their respective expressions.

One-versus-all multi-class classification

For every Task i, the one-versus-all approach solves m binary MTL LS-SVM algorithms with target class C , for each ∈ {1, . . . , m}, versus all other classes C , = . Calling g bin i (x; ) the output of the classifier for a new datum x, the class allocation decision is traditionally based on the largest among all scores g bin i (x; 1), . . . , g bin i (x; m). This approach generalizes the naive, yet simpler, method proposed in Algorithm 2 which, despite its good performances (recall Table 4.1), is fundamentally "incorrect" in its assuming that, for each , all classes C ( = ) have the same statistics.

However, this presumes that the distribution of the scores g bin i (x; 1) when x ∈ C 1 , g bin i (x; 2) when x ∈ C 2 , etc., have more or less the same mean and variance. This is not the case in general, as depicted in the first column of Figure 4.1, where data from class C 1 are more likely to be allocated to class C 3 (compare the red curves).

By providing an accurate estimate of the distribution of the scores g bin i (x; ) for all and all genuine classes of x, Theorem 9 allows us to predict the various positions of the Gaussian curves in This first improvement step simplifies the algorithm which still boils down to determining the largest g bin i (x; ), ∈ {1, . . . , m}, output but now limiting the risks induced by the inherent centering and scale issues previously discussed.

This being said, our theoretical analysis further allows to adapt the input scores y bin ( ) in such a way to optimize the expected output. Ideally, assuming x genuinely belongs to class , one may aim at increasing the distance between the output score g bin i (x; ) and the other output scores g bin i (x; ) for = . This however demands to simultaneously adapt all input scores y bin (1), . . . , y bin (m). Instead, we resort to maximizing the distance between the output score g bin i (x; ) for x ∈ C and the scores g bin i (x; ) for x ∈ C . By "mechanically" pushing away all wrong decisions, this ensures that, when x ∈ C , g bin i (x; ) is greater than g bin i (x; ) for = . This is visually seen in the third column of Figure 4.1, where the distances between the rightmost Gaussians and the other two is increased when compared to the second column, and we retrieve the desired behavior.

Specifically, our proposed score optimization consists in solving, for each i ∈ {1, . . . , k} and each ∈ {1, . . . , m} the optimization problems:

y bin ( ) = arg min y bin ( )∈R km max j = Q m i ( ) -m ij ( ) √ σ tj = arg min y bin ( )∈R km max j = Q     y bin ( ) T I mk -D -1 2 δ [mk] ΓD 1 2 δ [mk] (e [mk] m(i-1)+ -e [mk]
m(i-1)+j )

y bin ( ) T D 1 2 δ [mk] ΓV ij ΓD 1 2 δ [mk] y bin ( )     (4.13)
with Q the Gaussian q-function.

Being a non-convex and non-differentiable (due to the max) optimization, Equation 4.13 cannot be solved straightforwardly. An approximated solution consists in relaxing the max operator max(x 1 , . . . , x n ) into the differentiable operator 1 γn log( n j=1 exp(γx j )) for some γ > 0, and use a standard gradient descent optimization scheme here initialized at y bin ( ) ∈ R mk filled with 1's at every m(i -1) + , for i ∈ {1, . . . , m}, and with -1's everywhere else.

In effect, the optimized vector y bin ( ) is evaluated first before the constant shift scalar ȳ (ensuring that E x∈C [g bin i (x; )] is close to zero) is added to y bin ( ). This order of treatment is mandatory as E x∈C [g bin i (x; )] depends explicitly on the value of the input score vector y bin . This global procedure is described in Algorithm 3 below.

Algorithm 3 Proposed one-versus-all multi-task learning algorithm.

Input: Training samples X = [X 1 , . . . , X k ] with X i = [X (1) i , . . . , X (m) i ], X (j) i ∈ R p×n ij
and test data x. Output: Estimated class ˆ ∈ {1, . . . , m} of x for Task i. for = 1 to m do Center and normalize data per task: for all i ∈ {1, . . . , k},

• Xi ← X i I n i -1 n i 1 n i 1 T n i • Xi ← Xi / 1 n i p tr( Xi XT i ).
Estimate: M T Q0 M and V i according to Remark 9.

Create scores y bin ( ) by numerically solving (4.13) (see discussion following (4.13)).

Shift scores y bin ( ) according to Remark 5. Estimate σ i ( ) from Theorem 9 and Remark 9. Compute classification scores g bin i (x; ) according to (4.2). end for Output: ˆ = arg max ∈{1,...,m}

g bin i (x; ) √ σ i ( )
.

As an immediate corollary of Theorem 9, for large dimensional data, the classification accuracy of Algorithm 3 can be precisely estimated, as follows.

Proposition 6. Under the notations of Theorem 9, the probability of correct classification P (j) i (x) for Task i of a test data x ∈ C j is given by

P (j) i (x) = • • • ∞ 0 m-1 1 (2π) m-1 |σ(j)| exp - 1 2 (x -µ(j)) T σ(j) -1 (x -µ(j)) dx + o(1)
where 1) , with Y -j = {y bin (j) Ty bin (j ) T } j =j ∈ R (m-1)×km .

µ(j) = Y -j (I mk -D -1 2 δ ΓD 1 2 δ )e [mk] m(i-1)+j ∈ R m-1 and σ(j) = Y -j D 1 2 δ ΓV ij ΓD 1 2 δ Y T -j ∈ R (m-1)×(m-
Figure 4.1, succinctly introduced above, illustrates the successive improvements of the proposed algorithms. Specifically, it shows the gains of the centering-scaling operation on the input and output scores (2nd column) and of the optimization of the input scores (3rd column) when compared with the standard approach (1st column). Here synthetic data arising from a Gaussian mixture model are considered in a two-task (k = 2) and three-class (m = 3) setting in which x (j) 1l ∼ N(µ 1j , I p ) and x (j) 2l ∼ N(µ 2j , I p ), where

µ 2j = βµ 1j + 1 -β 2 µ ⊥ 1j , with µ 1j = 2e [p] j and µ ⊥ 1j = e [p]
p-j . Here p = 200, β = 0.2, [n 11 , n 12 , n 13 , n 21 , n 22 , n 23 ] = [393,309,394,20,180,480] and the optimization framework used for input score (label) y bin optimization is a standard interior point method [START_REF] Boyd | Convex optimization[END_REF]).4

One-versus-one multi-class classification

For a given Task i, the one-versus-one multi-class method trains 1 2 m(m -1) binary classifiers for each pair j = j ∈ {1, . . . , m}. As intensively discussed in the previous section, as well as in Section 4.3 and Remark 8, each resulting binary classifier g bin i (x; j, j ) can be optimized by choosing optimal input labels y bin (j, j ). This leads to Algorithm 4 described below.

In order to derive the asymptotic correct classification of class based on Algorithm 4, it is necessary to enumerate all scenarios which lead to the prediction of the class . Although this could be done in theory, the combinatorics, already for three classes, are cumbersome and not worth developing here. For the specific one-versus-one setting, we therefore do not provide a theoretical performance analysis.

One-hot encoding approach

For a given Task i in a one-hot encoding approach, using the canonical vector encoding for each class (i.e., Y ij = e

[m] j encodes all training input data x (j) il of class C j ), the class allocated to an unknown test sample x is the index of the output vector g i (x) ∈ R m with maximum value.

We disrupt here from this approach by explicitly not imposing a one-hot encoding for Y ij . Instead we consider a generic encoding Y ∈ R km×m , which will be optimized in such a way to maximize the classification accuracy. Figure 4.1: Test score distribution in a 2-task and 3 classes-per-task setting, using a one-versus-all multi-class classification. Every graph in row depicts the limiting distributions of g i (x; ) for x in different classes. Column 1 (Classical) is the standard implementation of the one-versus-all approach. Column 2 (Scaled scores) is the output for centered and scaled g i (x; ) for x ∈ C . Column 3 (Optimized labels) is the same as Column 2 but with optimized input scores (labels) y bin ( ). Under the "classical" approach, data from C 1 (red curves) will often be misclassified as C 2 . With "optimized labels", the discrimination of scores for x in either class C 2 or C 3 is improved (blue curve in 2nd row further away from blue curve in 1st row; and similarly for green curve in 3rd versus 1st row).

Algorithm 4 Proposed one-versus-one multi-task learning algorithm.

Input: Training samples X = [X 1 , . . . , X k ] with X i = [X (1) i , . . . , X (m) i ], X (j) i
∈ R p×n ij and test data x. Output: Estimated class ˆ ∈ {1, . . . , m} of x for Task i. Center and normalize data per task: for all i ∈ {1, . . . , k},

• Xi ← X i I n i -1 n i 1 n i 1 T n i • Xi ← Xi / 1 n i p tr( Xi XT i ).
for j = 1 to m do for j ∈ {1, . . . , m} \ {j } do Estimate: M T Q0 M and V ij according to Remark 9.

Create optimal scores y bin (j , j) according to Remark 8. Compute classification scores according to (4.2) and deduce the predicted class c(j, j ) = j or c(j, j ) = j based on the decision rule in (4.6). end for end for Output: ĵ = mode j ,j∈{1,...,m} {c(j, j )}. 5 Proposition 7. Under a "one-hot encoding" scheme with generic Y, the probability of correct classification P (j) i (x) for Task i of a test data x ∈ C j is given by

P (j) i (x) = • • • ∞ 0 m-1 1 (2π) m-1 |σ(j)| exp - 1 2 (x -µ(j)) T σ(j) -1 (x -µ(j)) dx, where µ(j) = E j Y T I mk -D 1 2 δ ΓD 1 2 δ e [km] (m(i-1)+j) ∈ R m-1 and σ(j) = E j Y T D 1 2 δ ΓV ij ΓD 1 2 δ YE T j ∈ R (m-1)×(m-1) with E j = {(e (m) j -e (m) j ) T } j =j ∈ R (m-1)×m .
A natural objective is to set Y so to maximize the average correct classification accuracy 1 m m j=1 P (j) i (x) (under assumed uniform prior on x). This form again is not convex in Y but may be approximated by gradient descent starting from the one-hot encoding solution, as described in Algorithm 5.

Experiments

This section has a double objective. The first part (Section 4.6.1) devises numerical experiments on binary classification settings to corroborate the theoretical analyses and conclusions drawn in this previous section. Here, the target is threefold: (i) empirically illustrate the effects of the bias in the threshold decision and in the label optimization Algorithm 5 Proposed "one-hot encoding" multi-task learning algorithm.

Input: Training samples X = [X 1 , . . . , X k ] with X i = [X (1) i , . . . , X (m) i
], X j i ∈ R p×n ij and test data x. Output: Estimated class ˆ ∈ {1, . . . , m} of x for target Task i. Center and normalize data per task: for all i ∈ {1, . . . , k},

• Xi ← X i I n i -1 n i 1 n i 1 T n i • Xi ← Xi / 1 n i p tr( Xi XT i ).
Estimate Matrix M T Q0 M and V ij according to Remark 9. Compute the theoretical score µ(j) and covariance σ(j) and derive the asymptotic

classification accuracy P i (x) = 1 m m j=1 P (j) i (x).
Create optimal scores Y = arg max Y P i (x). Compute classification scores g i (x) according to (4.2). Output: ˆ = arg max ∈{1,...,m} g i (x; ). scheme discussed in Section 4.3, (ii) discuss the impact of numerous tasks (k > 2) in the binary class setting, thereby emphasizing the effects of negative transfer and its correction through input score (label) optimization, and (iii) exemplify the relevance of our theoretical findings to a specific application to hypothesis testing in a multi-task setting.

In a second part (Section 4.6.2), experiments on both synthetic and real data for multi-class classification are realized, which demonstrate, even for real data: (i) the extreme accuracy of the theoretical performance predictions of Propositions 6-7 against empirical data and (ii) the large performance gains induced by the various improvements introduced at length in Section 4.5.

Experiments on binary classification

Effect of input score (label) choice

In the present experiment, the effects of the bias in the decision threshold (in general not centered on zero) and of the input score (label) optimization are demonstrated on both synthetic data and real data.

Specifically, MTL LS-SVM is first applied to the following two-task (k = 2) and two-class (m = 2) setting: for Task 1, x (j) 1l ∼ N((-1) j µ 1 , I p ) (evenly distributed in both classes) and for Task 2, x (j) 2l ∼ N((-1) j µ 2 , I p ) (evenly distributed in both classes), where

µ 2 = βµ 1 + 1 -β 2 µ ⊥
1 and µ ⊥ 1 is any vector orthogonal to µ 1 and β ∈ [0, 1]. This setting allows us to tune, through β, the similarity between tasks. For four different values of β, Figure 4.2 depicts the distribution of the binary output scores g bin i (x) both for the classical MTL LS-SVM (top displays) and for our proposed random matrix improved Histograms drawn from 1 000 test samples of each class. The figure clearly depicts the deviation from 0 of the decision threshold in unbalanced classes and the deleterious effect of "negative transfer" when β is small; these problems are well handled by the proposed optimized scheme. scheme, with optimized input labels (bottom display).

-1 0 1 0 1 2 Binary input y bin β = -1 -1 0 1 0 0.5 1 1.5 β = 0 -2 -1 0 1 0 0.5 1 1.5 β = 0.5 -2 -1 0 1 0 0.5 1 1.5 β = 1 -4 -2 0 2 4 0 0.2 0.4 0.6 0.8 Optimized input y bin -2 0 0 0.5 1 -2 -1 0 1 0 1 2 -2 -1 0 1 0 0.5 1 1.5
As a first remark, note that both theoretical prediction and empirical outputs closely fit for all values of β, thereby corroborating our theoretical findings. In practical terms, the figure supports (i) the importance to estimate the threshold decision which is nontrivial (not always close to zero) and (ii) the relevance of an appropriate choice of the input labels to improve the discrimination performance between both classes, especially when the two tasks are not quite related. In effect, the entries of y bin naturally drop to zero for all unrelated tasks and classes, thereby discarding the undesired use of the latter; the classical binary input labels instead inappropriately exploit these data and induce a negative learning effect, sometimes to such an extent to completely switch the final decision (as here when β = -1).

For experiments on real data, the MNIST dataset [START_REF] Deng | The mnist database of handwritten digit images for machine learning research [best of the web[END_REF] is considered. Specif-ically, the setting is that of a binary classification for two tasks, mimicking a transfer learning setting: there, the "target" Task 2 aims to discriminate Class C 1 and Class C 2 respectively composed of images of digit 1 and digit 4. The "source" Task 1 is here used as a support for classification in the target task, and consists of the classification of other pairs of digits: either (5, 9), (9, 5), (6, 2) or (8, 3) (we recall that the order of the set of digits (X, Y ) is important for the non-optimized MTL LS-SVM since the source and target tasks labels are "paired"; thus (5, 9) or (9, 5) digits for the source task will bring different results). We compare here again the non-optimized MTL LS-SVM with labels y bin = [-1, 1, -1, 1] T to our proposed optimized scheme (as detailed in Remark 8). For both methods, the optimal theoretical threshold decision ζ is used (rather than ζ = 0 for the non-optimized setup) in order to emphasize the influence of input score (label) optimization.

Figure 4.3 depicts the performance for both methods as a function of the hyperparameter λ. We recall that, as λ → 0, the multi-task scheme becomes equivalent to independent single-task classifiers, while as λ → ∞, both source and target tasks are considered together as one task. Figure 4.3 raises the stability of optimal input labeling with respect to λ: this is explained by the fact that y bin is a function of λ and thus adapts to each value of λ, even if suboptimal. Besides, for appropriate values of λ, the proposed improved labeling can largely outperform the non-optimized setting, even here on real data.

Table 4.2 complements the figure by effectively displaying the optimal vectors y bin at the optimal value for λ. The table demonstrates the appropriate adjustment of the labels to the data correlation ∆µ T 1 ∆µ 2 ∆µ 2 2 . Specifically, for a negative correlation between the classes of both tasks, the method naturally "switches" the labels (the input data scores) by opposing the signs of y bin in entries 1, 3 (Class C 1 in each task) and 2, 4 (Class C 2 in each task). For rather orthogonal tasks (here typically (8, 3)), the entries of y bin corresponding to the source task (entries 1 and 2) are almost zero, thereby discarding the source data and avoiding negative transfer. It is also interesting to note that, for moderately correlated tasks (here for the source digits (5, 9)), despite the fact that the source task offers ten times more data (n 1j = 100, n 2j = 10) and is thus deemed trustworthy for classification, the corresponding entries 1 and 2 in y bin are much smaller than the entries 3, 4 corresponding to the target task: the algorithm thus judges the few target data more relevant to target classification than the many related source tasks.

Analysis of increasing number of tasks

This next experiment illustrates the effect of adding more tasks for the transfer learning setting on synthetic and MNIST datasets. For synthetic data, Gaussian classes with mean

µ ij = βµ i1 + 1 -β 2 µ ⊥
i1 and various values of β are successively added. For the MNIST dataset, different classifications of digits are added progressively to help classify the specific pair of digits (1, 4) . Figure 4.4 depicts the classification error after each new task addition, both for a classical binary input label choice and for the proposed optimized input labels. The figure forcefully illustrates that our proposed framework 10 -3 10 -2 10 -1 10 0 10 1 10 2 10 3 A PCA preprocessing is performed on each image to extract their p = 100 principal components; the accuracy is performed over n test = 1 135 test samples. The proposed method shows a low sensitivity to λ.

[Source] (9,5) (5,9) (6,2) (8,3) avoids negative transfer, as the classification error of MTL never increases as the number of tasks grows. This is quite unlike the non-optimized scheme which severely suffers from negative transfer.

∆µ T 1 ∆µ 2 ∆µ 2

Hypothesis testing

The next experiments, on synthetic data, apply the results of MTL LS-SVM to a hypothesis test on a target Task t based on training samples both from a source Task s and the target Task t. For data x in the target task, the test

g bin t (x) H 1 ≷ H 0 ζ is performed
, where H 0 is the null hypothesis (say, Class 2) and H 1 the alternative (say, Class 1) and ζ = ζ(η) is a decision threshold here selected in such a way to enforce the false alarm rate constraint

P (g t (x) ≥ ζ(η) | x ∈ H 0 ) ≤ η, for a given η ∈ (0, 1).
The objective is then to maximize over the input scores y bin the correct detection rate P (g bin t (x) ≥ ζ(η) | x ∈ H 1 ): this induces a different value for the optimal scores y bin than proposed in (4.12), which can be constructed following Remark 8. 10 -6 10 -5 10 -4 10 -3

10 -2 10 -1

10 0 P (x → C 2 |x ∈ C 1 ) P (x → C 1 |x ∈ C 2 )
Non Opt (Sim) Non Opt (Th) Opt (Th) Opt (Sim) The experimental synthetic data is here a two-task (k = 2) setting in which x 1j ∼ N(±µ 11 , I p ) (i.e., µ 12 = -µ 11 ) and x 2j ∼ N(±µ 21 , I p ), where µ 21 = βµ 11 + 1 -β 2 µ ⊥ 11 , µ 11 is a unit-norm vector and µ ⊥ 11 any unit-norm vector orthogonal to µ 11 . We take here β = 0.5, so that both tasks are "slightly" correlated.

Figure 4.5 depicts the algorithm performance through a receiver-operating curve (ROC) for false alarm rates η on synthetic data. Both theoretical (Th) asymptotics (used to set the decision threshold ζ) and actual performances (Sim) are displayed, for the optimal (Opt) choice of y bin (Opt) and for y bin = [-1, 1, -1, 1] T (Non-Opt).

Figure 4.5 confirm, here under the hypothesis testing problem, the large superiority of our proposed optimized MTL LS-SVM over the standard non-optimized alternative. Besides, the theoretical classification error prediction is an accurate fit to the actual empirical performance, even for not so large values of p and the n ij 's, and even for small error values. 6 This remark is here all the more fundamental that, in practice, η can be set a priori, using Theorem 9 with no need for heavy, unreliable, and data-consuming cross-validation procedures.

Experiments on multi-class classification

We here consider the complete setting of a k ≥ 2, m > 2 multi-class learning scenario, first on synthetic and then on real image datasets.

Experiments on synthetic dataset

In the synthetic data experiment, the scenario is a two-task (k = 2) setting in which x (j) 1l ∼ N(µ 1j , I p ) and x

(j) 2l ∼ N(µ 2j , I p ), where µ 2j = βµ 1j + 1 -β 2 µ ⊥ 1j , with µ 1j = 2e [p] j and µ ⊥ 1j = e [p]
p-j , and β varies from 0.1 to 0.8. Table 4.3 provides the empirical classification accuracy achieved by one-versus-all (Algorithm 3), one-versus-one (Algorithm 4) and one-hot (Algorithm 5) learning versus their standard (non-optimized) algorithm equivalent on 10 000 test samples. The table also reports the theoretical classification accuracies predicted by the empirical estimation of the quantities involved in Propositions 6-7 (therefore without any cross-validation) for the one-versus-all and one-hot methods.

(j) 2l ∼ N(µ 2j , I p ), µ 2j = βµ 1j + 1 -β 2 µ ⊥ 1j
, for different values of the data-correlation β > 0 and various multi-class learning algorithms. Theoretical performance predictions are provided in parentheses. Here m = 5, p = 100, c 1j = .16, c 2j = .04, for j ∈ {1, . . . , 5}, λ = 1 and γ = 1 k . The performance gains of the proposed optimal scheme is particularly clear in tasks with low correlation.

β Method one-vs-all one-vs-one one-hot The output performance scores naturally show an improvement using the proposed MTL LS-SVM framework and confirm again the extremely accurate prediction of performance by the theoretical formulas. Most importantly, the table reveals that the gap between the non-optimized and optimized schemes is all the more important that the correlation between task (through the parameter β) is small; this indicates that the optimized MTL LS-SVM learning framework better exploits the (even little) correlation arising between tasks or, alternatively, that the non-optimized scheme suffers from negative learning when "over-emphasizing" the weight of data from the other task (through the binary input labels Y).

β = 0.
As for the comparison of the three classification methods (one-versus-all, one-versusone and one-hot), it shows here an overall superiority of the one-versus-one approach. This result should nonetheless be interpreted with extreme care as no optimization over the hyperparameters γ, λ is conducted in any scenario.

Image classification

Similarly as in Section 4.3, we now turn to the popular Office+Caltech256 multi-task image classification benchmark [START_REF] Saenko | Adapting visual category models to new domains[END_REF][START_REF] Griffin | Caltech-256 object category dataset[END_REF] often exploited for transfer learning. The overall database consists of 10 categories shared by both Office and Caltech256 datasets. As in Table 4.1, we consider in sequence the transfer learning of one out of four possible source tasks, each of which consisting in classifying data from one sub-database (images issued from the Caltech set (c), Webcam images (w), Amazon pictures (a) or dslr images (d)), towards another task; this boils down to 4 × (4 -1) = 12 source-target comparison pairs.)

The results in Table 4.1 using VGG features for the image representations are extremely close to 100%, already for the "naive" approach consisting in a simplified one-versus-all extension of Algorithm 2. Little would be gained (at least not in computational efforts) by running the more involved Algorithm 3 on the same database. For this reason, for the present experiment, we compare the more challenging (since less discriminative) p = 800 SURF-BoW features of the Office+Caltech256 images instead of their VGG features.

Half of the samples of the target task are randomly selected as test data and the accuracy is evaluated over 20 independent trials. For complexity reasons, as in Section 4.3, for each experiment, the naive version of the one-versus-all algorithm is run 10 times, considering a fictitious two-class C1 -versus-C2 setting where, for the classifier focusing on class C , class C1 = C while class C2 is the union of all other classes C , = .

Table 4.4 reports the accuracy obtained by the algorithm (Proposed) versus the nonoptimized MTL LS-SVM from [START_REF] Xu | Multi-output least-squares support vector regression machines[END_REF] (LS-SVM) and state-of-the-art transfer learning algorithms already introduced in Section 4.3. Table 4.4 again demonstrates that our proposed improved MTL LS-SVM, despite its simplicity and unlike the competing methods used for comparison, has stable performances and is highly competitive. Table 4.4: Classification accuracy for transfer learning on the Office+Caltech256 database, against state-of-the-art alternatives. Here with c(Caltech), w(Webcam), a(Amazon), d(dslr) based on SURF-BoW features. Our proposed approach is systematically best or second to best and best on average.

S/T c → w w → c c → a a → c w → a a → d d → a w → d c → d d → c a → w d →

Concluding remarks

Through the example of multi-task learning, as well as its particularization to transfer learning, we demonstrate the ability of random matrix theory to predict the performance of advanced machine learning schemes (here based on an extension of LS-SVM) and most importantly to propose improved learning mechanisms, which are competitive with, if not largely outperforming, elaborate state-of-the-art alternatives. Interestingly, as already reported in recent works [START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF]Mai et al., 2019), the proposed random-matrix-optimized framework is largely counter-intuitive and comes along with novel insights on the overall learning mechanisms of large dimensional data classification. Here specifically, the proposed input score (label) optimization is at odds with the conventional binary input label insights of most machine learning schemes, but is key to optimize the exploitation of other tasks and to discard altogether the long standing problem of negative transfer.

The random-matrix framework also draws a significant advantage in its being universal to data distributions. As shown here, our main results (Theorem 9) are valid for data modeled as mixtures of concentrated random vectors which go quite beyond the usually assumed Gaussian mixtures, as they encompass extremely realistic synthetic data models (such as GAN images). This universality phenomenon, possible surprising at first, in fact holds for a wide range of large dimensional "dense" (as opposed to sparse) data representation vectors, encompassing not only images but also likely other forms of data representations, such as word embeddings in natural language processing, vectors of moments of graphons in statistical graph analysis, etc.

To conclude, we importantly emphasize a fundamental underlying take-away message of the present work: recalling that LS-SVM is nothing but an explicit and computationallycheap linear regression method, the fact that it competes or even outperforms elaborate MTL methods testifies of the possibility, when dealing with large dimensional data, to design highly performing elementary and cost-efficient random-matrix-based learning schemes. This remark is in line with the recent parallel analysis of information-theoretic bounds on the performances of machine learning problems, such as in [START_REF] Lelarge | Asymptotic bayes risk for gaussian mixture in a semisupervised setting[END_REF] for semi-supervised learning (SSL); similar to the present work, in [START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF], the authors propose a random-matrix-based optimization of standard graph SSL learning which they demonstrate to tightly reach the information-theoretic upper bound of [START_REF] Lelarge | Asymptotic bayes risk for gaussian mixture in a semisupervised setting[END_REF]. This strongly suggests the practical relevance of "reinvesting" research efforts in simple, cost-efficient, theoretically tractable, controllable, and usually more stable machine learning schemes, rather than in complex and theoretically intractable techniques.

Chapter 5

Perspectives and future work Contents 5.1 Short-term perspectives related to chapters 3 and 4 . . . . . . . 90 5.2 Towards an efficient, low-cost, controllable and Green AI . . . . 95

Short-term perspectives related to chapters 3 and 4

The analyses carried out throughout this thesis provide a theoretical understanding and corrections of learning schemes involving covariance matrices as well as theoretical and practical insights into Multi-Task and Transfer Learning algorithms. However, some research directions which are discussed next will enable the work to have a significant societal and broader impact.

Application to real data. The random matrix improved estimation of the distance between covariance matrices has been applied to a covariance-based feature classification but the real strength and robustness of the proposed estimator will only be demonstrated when applied to real (non-Gaussian) datasets and more exotic applications (brain signal classification, hyperspectral image classification, etc). Moreover, the random matrix improved estimation scheme proposed can be used in the context of testing the equality of population covariance matrix widely used in signal processing [START_REF] Wj Krzanowski | Between-groups comparison of principal components[END_REF][START_REF] Robert | Common principal components and related multivariate models[END_REF][START_REF] James R Schott | Some tests for common principal component subspaces in several groups[END_REF]. This would however need to access the fluctuations of the random matrix estimator proposed in theorem 7. More concretely, it would be convenient to obtain a result similar to [START_REF] Yao | Fluctuations of an improved population eigenvalue estimator in sample covariance matrix models[END_REF] in the present context, that is a central limit theorem for the fluctuations of the estimator of Theorem 7. This would allow in addition to the applications mentioned previously to access both a consistent estimator for their sought-for matrix distance as well as a confidence margin. This investigation demands even more profound calculi (as can be seen from the detailed derivations of [START_REF] Yao | Fluctuations of an improved population eigenvalue estimator in sample covariance matrix models[END_REF]).

On the other hand, for the metrics we are dealing with, it is crucial that the smallest eigenvalue of the covariance matrices Σ 1 and Σ 2 does not tend to 0 as p → ∞ which leads to singularities of the covariance matrices in the Positive Semi-Definite Riemannian manifold. This happens quite often in over-parametrized settings as in Electroencephalography (EEG) datasets [START_REF] Luiz | Dimensionality reduction for bci classification using riemannian geometry[END_REF]. The randomness involved in the measurements exacerbates the singularities of the sample covariance matrices Σ1 , Σ2 and makes even more challenging the estimation procedure. A first step to tackle this problem may consist to consider instead the regularized population covariance matrices Σ 1 + λ 1 I p and Σ 2 + λ 2 I p for some constants λ 1 , λ 2 > 0. Several questions, therefore, need to be properly addressed. What is the loss incurred by the regularization scheme as function of the strength of λ 1 and λ 2 ? Is there an optimal value of λ 1 and λ 2 as function of the spectrum of Σ 1 and Σ 2 ?

The High Dimension Low Sample Size Regime. We point out the difficulty to handle the high dimension low sample size regime (i.e the regime n i ≤ p). It is important to note that this regime is very common in real-life applications where data collection is often difficult and the dimensions are relatively large. Managing this critically important setting will help to get closer to real-life problems. We propose one solution with polynomial approximation in (Tiomoko & Couillet, 2019a) which only handles partially the case n 2 < p. Random projections and regularization methods can be alternative methods to tackle this scenario, however possibly to the detriment of the estimator consistency. More concretely, relying on the random projection approach one would wonder for a random matrix W ∈ R p×q , what would be the loss induced by projecting the covariance matrices Σ 1 and Σ 2 in the q-dimensional space generated by the columns of W ? In other words, this consists to study the statistical behavior of the random quantity

D(W T Σ 1 W, W T Σ 2 W ) -D(Σ 1 , Σ 2 ) as p, n → ∞.
This would be adequate in the case of p > n 1 , n 2 to choose q < n 1 , n 2 and then apply the estimation framework derived in this thesis and one would furthermore quantify the loss incurred as the ratio q/p decreases. From a technical point of view, this requires choosing an appropriate model for the random matrix W (Haar random matrix, Gaussian matrix,...) and to deal with the asymptotic eigenvalue distribution of (W T Σ 1 W ) -1 W T Σ 2 W which can be quite involved due to the inverse matrix. Free probability theory (Mingo & Speicher, 2017) can be an adequate tool to handle such asymptotic eigenvalue distribution since it allows for computing the asymptotic eigenvalue distribution of rational functions of random matrices. But a first step would rather consider the family metric involving the eigenvalue distribution of

W T Σ 1 W W T Σ 2 W .
Towards an understanding of covariance matrix estimation scheme. We proposed a generic framework to estimate the covariance matrix Σ under different metrics. For the subsequent discussion, we briefly recall the estimation procedure which is based on remarking that Σ ≡ arg min M 0 D(M, Σ), for some distance between any deterministic matrix M and the sought-for covariance matrix Σ. Relying on the proposed improved estimate D(M, X) for D(M, Σ) based on samples X = [x 1 , . . . , x n ] (of zero mean and covariance matrix Σ), the optimization problem introduced above can be approximated as Σ ≡ arg min M 0 h X (M ) with h X (M ) = D(M, X) 2 and solved using a gradient descent algorithm.

Since each distance (Fisher distance, Wasserstein distance, ...) leads to a different estimate, the natural question would be to compare the different estimators in order to give which suits for a specific application. This however requires to understand theoretically the gradient descent step performed which is quite involved and sub-optimal. Indeed, the considered approach suffers a profound limitation: D(M, X) only estimates D(M, Σ) for M independent of X. This poses a formal problem when implemented in a gradient descent. More concretely, in chapter 3, it is precisely shown that, for every deterministic sequence of matrices {M (p) , p = 1, 2, . . .} and {Σ (p) , p = 1, 2, . . .}, with M (p) , Σ (p) ∈ R p×p and max( Σ (p) , M (p) ) < K for some constant K independent of p, we have that, for andz (p) i i.i.d. vectors of i.i.d. zero mean and unit variance entries,

X (p) = [x (p) 1 , . . . , x (p) n ] with x (p) i = Σ (p) 1 2 z (p) i
D(M (p) , Σ (p) ) -D(M (p) , X (p) ) → 0
almost surely as n, p → ∞ and p/n → c 0 ∈ (0, 1). If we denote M k the matrix obtained at the iteration k of the gradient descent step, the proposed methodology supposes that D(M k , X) is a good approximation for the sought-for D(M k , Σ). This, however, only holds true so long that M k is independent of X which clearly does not stand when proceeding to successive gradient descent steps in the direction of ∇h X (M ) which depends explicitly on X. As such, while initializations with, say, M 0 = I p , allow for a close approximation of D(M k , Σ) in the very first steps of the descent, for larger values of k, the descent is likely to drive the optimization in less accurate directions. This needs be tackled: (i) either by estimating the introduced bias so to infer the loss incurred or, better, (ii) by accounting for the dependence to provide a further estimator D(M (X), X) of D(M (X), Σ) for all X-dependent matrices M (X) following a specific form.

Furthermore, if we denote by U and Û respectively the eigenvectors of the population covariance matrix Σ and the sample covariance matrix Σ, we can prove that the set of covariance matrix estimators

H = { Û D Û T | D diagonal} is a stable set of Algorithm 1 in the sense that M k ∈ H ⇒ M k+1 ∈ H.
One may thus wonder if H is also a global attractor: i.e., does every trajectory {M 1 , M 2 , . . .} necessarily converge to H? Extensive simulations initialized randomly (say with M 0 a random Wishart matrix) indeed suggest that, after a few iterates, the eigenvectors of M k do converge to those of Σ. This is, however, not everywhere true. Indeed, in the extreme scenario where M 0 = Σ, 0 = D(Σ, Σ) D(Σ, X) which consistently estimates zero for large n, p (and irrespective of n/p), the gradient descent does not progress much from M 0 and is thus unlikely converging within H (which would mean the existence of a D such that | D(U DU T , X)| < | D(Σ, X)|). These aspects need to be clarified and well-understood. Indeed, in the absence of any a priori knowledge about the structure of Σ, such as sparseness or a factor model, several authors [START_REF] Ledoit | Analytical nonlinear shrinkage of large-dimensional covariance matrices[END_REF][START_REF] Karoui | Spectrum estimation for large dimensional covariance matrices using random matrix theory[END_REF][START_REF] Mestre | On the asymptotic behavior of the sample estimates of eigenvalues and eigenvectors of covariance matrices[END_REF] argue that it is natural to only consider estimators of Σ that are rotation-equivariant (that belong to H). Therefore, one could ask if the estimation scheme proposed does at least better than the rotation-equivariant estimators? If not we should gain more to restrict the estimation procedure to a nonlinear shrinkage of the eigenvalues of the sample covariance matrix, that is, to first consistently estimate D( Û D Û , Σ) for deterministic diagonal matrices D and perform a gradient descent step on D.

Unifying feature-based and parameter-based MTL approaches. MTL approaches are usually divided into parameter-based versus feature-based learning schemes. In the parameter-based MTL approach, the tasks are assumed to share some parameters (e.g., the hyperplanes best separating each class) as discussed in chapter 4 of the present work. In the feature-based MTL approach, the tasks data are instead assumed to share a low-dimensional common representation. In this context, most of the works aim to determine a mapping of the ambient data space into a low-dimensional subspace (through sparse coding, deep neural networks, principal component analysis, etc.) in which the tasks have high similarity [START_REF] Argyriou | Multi-task feature learning[END_REF][START_REF] Maurer | Sparse coding for multitask and transfer learning[END_REF][START_REF] Zhang | Deep model based transfer and multi-task learning for biological image analysis[END_REF][START_REF] Sinno Jialin Pan | Domain adaptation via transfer component analysis[END_REF].

We should note that every year the literature on transfer learning evolves at an incredible speed with new methods that complement the previous long list of existing algorithms. In this context, the practitioner is often confused as to which method to consider. Due to the lack of theoretical understanding, most of these schemes are overparametrized and difficult to tune for practitioners and the optimality of the algorithms is difficult to prove. The work carried out in Chapter 4 is part of this deeper understanding of the methods developed so far in order to identify their similarities, give the user-specific instructions on how to use them since the underlying idea is generally pertinent and should not necessarily be discarded.

Using Random Matrix theory, it will be interesting to completely address MTL learning scheme by studying feature-based methods in order to understand from a statistical point of view the difference between the two approaches, to assess the conditions where one approach is better than the other in order to provide theoretical guidelines for practitioners. In this context, the Principal Component Analysis (PCA), the Transfer Component Analysis (TCA) [START_REF] Sinno Jialin Pan | Domain adaptation via transfer component analysis[END_REF] and the sparse coding may be theoretically investigated using Random Matrix Theory. The ultimate goal is to fully control the dimensionality reduction-based transfer learning problem and to improve them by providing theoretical guidelines for the choice of some hyperparameters (number of the components to extract,...). The idea is to understand the interplay between the generally huge amount of hyperparameters and the sufficient statistics so that to make the method optimal with respect to the theoretic information bounds and at the same time decrease the computational cost by discarding redundant hyperparameters. More concretely, any practitioner would be able to identify which algorithm of transfer learning fits its application based on its optimality or not with respect to a theoretical bound, its computational advantage and be able to use efficiently the chosen algorithm by a proper understanding of its inner working statistics and the guidelines for the hyperparameter tuning.

Task relatedness assumption. Furthermore, as pointed in the introduction, the relatedness assumption is central in the multi-task learning schemes. In chapter 4, we use a relatedness assumption on the separating hyperplane of the SVM. One interesting question could be to try to derive other relatedness assumptions mostly regarding the data. An idea could try to find a projection matrix that should be optimized so that the projected data for source tasks and target tasks have a low discrepancy in the projected space. Concretely, the approach should search for two matrices A and B such that f (A, B) = D(AX 1 , BX 2 ) is as low as possible where D(AX 1 , BX 2 ) refers to a distance function between the data AX 1 and BX 2 with X 1 and X 2 the samples of task 1 and 2. Subsequent learning schemes can then be applied on the optimal low dimensional space. In this setting, one would rather be interested in the theoretical analysis of traditional learning schemes (SVM, Empirical risk minimizers, ...) in the aforementioned optimal subspace and what would be the impact of the dimension of the subspace. To that end, it is of particular interest for tractability to choose a convex function f (A, B). Further constraints on the structure of the matrix A and B can be added to the optimization problem depending on the context.

Towards optimal real-world multi-class algorithms. The improvements performed in Chapter 4 were mainly concerned with the binary classification and the optimality of the algorithm can only be proved for binary classification. This is due to the fact that the support vector machine algorithm was introduced to solve binary classification problems. The treatment of multi-class classification is always tricky and potentially sub-optimal. Generally, we rely on splitting the multi-class classification into several binary classifiers or relying on a one-hot encoding approach. The results obtained by the different extensions as provided in chapter 4 are generally different. Therefore, several questions naturally arise. Are we making optimal use of the improvement achieved in the binary case when building multi-classifiers? To answer this question, one would have to derive theoretical bounds on multi-class classification, compare different multi-class learning algorithms, and choose/design the optimal algorithm or at least quantify the sub-optimality of each multi-class extension. Some attempts to compare existing schemes (one-versus-all, one-versus-one, one-shot coding) have been proposed in the literature, but they are mainly based on experimental studies (see for example [START_REF] Rifkin | In defense of one-vs-all classification[END_REF]). This discussion stems from the overriding importance of multi-class classification in real-world applications compared to binary classification and especially because multi-class classification is by far more difficult than the binary case.

On the other hand, in some real-world problems, data are usually missing and contain outliers that need to be handled properly. More concretely, suppose we have missing data in the dataset inputs and we want to evaluate the capability of the imputation strategies to recover the classes with the least amount of defects.

A possible elementary model to study is the kernel matrix

1 p (X S + A) T (X S + A) with X = [x 1 , . . . , x n ],
S the missing values matrix, and A the imputation matrix (with non-zero values a priori on the same locations as those of S.

An interesting study could try to access the performance of multi-task learning schemes under this model and to choose potentially the optimal imputation strategy. More concretely assuming that S has a simple statistical model (e.g. with Bernoulli i.i.d. inputs), what are the performances (in terms of classification error) obtained by specific simple imputation models for A (e.g., A = α(11 T -S) is constant over the set of non-zero inputs of S, or A ij is chosen as an average of the non-zero inputs of X i,... , etc). A similar study has been performed in [START_REF] Seddik | A random matrix analysis of learning with α-dropout[END_REF] in the single task to characterize the performance in terms of phase transition and can be inspiring for the proposed study.

We conclude this part by pointing out that class imbalance problems have drawn growing interest recently because of their classification difficulty. Class imbalance learning refers to a type of classification problem, where some classes are highly underrepresented compared to other classes. The skewed distribution makes many conventional machine learning algorithms less effective, especially in predicting minority class examples. The learning objective can be generally described as obtaining a classifier that will provide high accuracy for the minority class without severely impacting the accuracy of the majority class. In this case, the least-square loss is generally known to perform poorly. Instead, several adaptations have been proposed ( [START_REF] Tang | Svms modeling for highly imbalanced classification[END_REF][START_REF] Zheng | Oversampling method for imbalanced classification[END_REF]) using classical asymptotic heuristics. Random Matrix Theory analysis can be performed to take this important case into account.

The thesis and the perspectives proposed below are part of an urgent need to adapt to the new challenges of the modern world. We propose in the following section a modern vision of machine learning that we believe is possible and of which this thesis could be one of the founding ideas.

Towards an efficient, low-cost, controllable and Green AI

The traditional vision in machine learning. The classical methodology in machine learning and signal processing is to consider that the number of samples is very large compared to the dimension of the samples. As we have shown in this thesis, this induces many biases that lead to dramatic and unexpected behavior in the algorithms. The traditional explanation for such a failure is often and systematically attributed to the small number of samples. Therefore, the natural solution that follows from this analysis is to increase the number of samples and the model parameters (generally making the model more complex and "black-boxed"), as is generally the case in methods such as neural networks, ensemble learning, etc. The consequences of such approaches can be dramatic in the sense that the price to reach a given performance is excessively high (lots of computational units, lots of data with the carbon footprint, and the dramatic environmental consequences that this induces).

Towards a modern vision of machine learning. For the sake of efficiency, it would make more sense to (i) determine the best performance that would be induced by the a priori knowledge of the problem and (ii) determine algorithms whose performance is predictable and that are demonstrably optimal with respect to this bound. This approach to the problem ensures that the resources are used sparingly, efficiently, and more interestingly that we deeply understand the model parameter behind the problem so as to connect the intrinsic physical model of the problem with the model learned by the algorithm. More concretely, this would allow us to determine the minimum resources needed to achieve a given performance and thus build low-cost algorithms that achieve this bound or at least quantify how far the proposed algorithm is from the optimal one. Indeed, the knowledge of the theoretical bound allows to know in addition the gap between the proposed algorithm and the optimal one, allowing to evaluate the sub-optimality of the designed algorithm versus the gain induced by the reduced computational cost. This reflection is in line with the increasingly urgent need to use data/resources smartly in our daily life.

What can Random Matrix Theory do? Random matrix Theory tools are part of this logic since they allow to analyze and control machine learning algorithms. Although the performance is determined asymptotically, the analyses remain true even for settings that were traditionally considered to be in the classical regime and for finite dimensions. This thesis proposes through random matrix theory a deeper understanding of the functionals of covariance matrices by exploiting randomness both in the sample size and in the dimension size. The improved scheme has important environmental consequences.

To illustrate that, let's assume we want to estimate the Fisher distance between two covariance matrices Σ 1 and Σ 2 denoted

D F (Σ 1 , Σ 2 ) with [Σ -1 2 1 Σ 2 Σ -1 2 1 ] ij = .3 |i-j| based on n a samples x (a)
i ∼ N(0, Σ a ) for p = 64 and a ∈ {1, 2}. Using n 1 = n 2 = 128 samples, a relative error of ∼ 5% could be achieved using the proposed scheme while with 100× samples more (n 1 = n 2 = 12, 800), only ∼ 6% relative error can be achieved using the classical asymptotic. This example illustrates that to achieve the same relative error, one needs to collect more than 100× more samples with the carbon footprint that this induces. Even though experimental, this example illustrates how Random matrix Theory can be part of this change in the field of artificial intelligence to meet the new challenges of the world.

Regarding the Multi-Task Learning Least Square Support Vector Machine analyzed in chapter 4, the classical training cost involved by setting the hyperparameters λ and γ i 's which traditionally relies on cross-validation is time and data consuming since it involves making a grid search over ∼ 1000 hyperparameter candidates. Moreover, to evaluate the pertinence of the different candidates, one needs furthermore to dedicate a specific number of training samples for the validation. This induces training at a small number of samples and therefore achieves low performance. The proposed learning approach instead relies on an explicit expression of the optimal labels which, when optimized, lead to a low sensitivity of the hyperparameters as illustrated in chapter 4. Due to the aforementioned reasons, the carbon footprint of the proposed method is likely lower than the one of the classical method. Furthermore, as will be discussed next, the proposed algorithm has been proven in a preliminary work to be optimal with respect to information-theoretic bound.

The new challenges. Based on this strategy of designing theoretical bounds given a problem, it would be interesting to use Bayesian and Information theory tools to derive theoretical bounds that any algorithm can achieve given the a priori knowledge of the problem (sparsity, feature-based covariance model, etc). This study, already conducted for learning domains such as semi-supervised learning and specific model for the data (Gaussian mixture model) [START_REF] Lelarge | Asymptotic bayes risk for gaussian mixture in a semisupervised setting[END_REF] which paradoxically did not bring much attention, could be applied to transfer learning and compared to the algorithm proposed in this thesis. This work has already been preliminarily tackled (on-going research) and has shown promising results in the sense that it is possible to prove that the optimized method proposed in Chapter 4 of this thesis is close to optimal with respect to this bound. In the same vein of idea, regarding the covariance feature-based algorithm, an optimal bound on the classification algorithm could be designed and compared with the proposed algorithm in Chapter 3. On the other hand, Semi-supervised multi-task learning which has the advantage of using not only unlabeled data (which are easier to obtain than labeled data) but also data belonging to other tasks is another promising research direction. Similar to the information-theoretic bounds provided in [START_REF] Lelarge | Asymptotic bayes risk for gaussian mixture in a semisupervised setting[END_REF] for semi-supervised single-task learning, theoretical bounds of multi-task semi-supervised learning can be derived using statistical physics tools. Furthermore, using the preliminary results of (Mai & Liao, 2019), a graph-based algorithm can be derived which reaches this bound.

However, the information-theoretic bounds using information theory and the algorithms derived using the random matrix theory do not take for now into account the structure of the data (images, text, time series, sparse data, etc). A very interesting approach would be to incorporate the a priori (sparse data, images, text) in the knowledge for deriving the theoretical bounds and design data-dependent algorithms. This will prevent the use of features generally extracted by hand (SIFT features, neural networks, etc). Practically, it would be possible to design a theoretical bound given that the means of the data are sparse (as for MNIST data) or that the covariance matrix presents a specific structure (sparse inverse as in the case of brain applications [START_REF] Cai | Capturing dynamic connectivity from resting state fmri using time-varying graphical lasso[END_REF], etc). The introduction of such a priori will also allow reaching higher performances at lower costs. This study will however require relevant modeling of the data. Several models can be used as starting point as [START_REF] Peyré | Manifold models for signals and images[END_REF] for a manifold model for images, [START_REF] Gerber | Manifold modeling for brain population analysis[END_REF] for a manifold of brain population analysis, [START_REF] George | Markov random field texture models[END_REF] for a random Markov field model for images, [START_REF] Dong | Pm2. 5 concentration prediction using hidden semi-markov model-based times series data mining[END_REF] for a hidden Markov model for time series, etc. But the main technical difficulty will be the theoretical tractability of the introduction of such models into the information-theoretic analysis.

On the other hand, in this learning perspective at lower cost, an idea would be, given a ML question, to determine the minimal information to keep in order to reach a certain level of performance and to derive the optimal feature selection to remove the redundant information through "sparsification", multiplication by random matrices, etc as preliminary proposed in recent works [START_REF] Couillet | Two-way kernel matrix puncturing: towards resource-efficient pca and spectral clustering[END_REF][START_REF] Dall | Nishimori meets bethe: a spectral method for node classification in sparse weighted graphs[END_REF], 2020). These ideas can lead to a modern vision for machine learning integrating the new economical and environmental challenges.

Proof of corollary 1

In this section, we provide the proof of Corollary 1 which finds the deterministic equivalents of Q(z), Q(z) and Q(z) 2 as a consequence of Theorem 5 in the particular case of C = I p + µµ T . We recall from Theorem 5 that the resolvent of the generalized sample covariance matrix defined as

Q(z) = 1 n XX T -zI p -1 admits a deterministic equivalent Q(z) given by Q(z) ↔ Q(z) = C 1 + δ(z) -zI p -1 (6.1)
where δ(z) is the unique solution to the fixed point equation defined as

δ(z) = 1 n tr C C 1 + δ(z) -zI p -1 . (6.2)
Plugging in C = Σ + µµ T in Equation ( 6.1) and applying Sherman Morrison identity matrix i.e., (A + uu T ) -1 = A -1 -A -1 uu T A -1 1+u T A -1 u for any invertible matrix A and any vector u, we obtain

Q(z) = m(z)I p - m(z) 2 1 + δ(z) + m(z) µ 2 where m(z) ≡ 1 1+δ(z) -z -1
. Further, in order to find an explicit expression of δ(z), we rely on the following rank-1 perturbation lemma for the resolvent of a matrix M .

Lemma 1 (Perturbation lemma [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF]). Let A, M ∈ R p×p some symmetric and non negative definite matrices, u ∈ R p , λ > 0 and z < 0, then,

|trA M + λuu T -zI p -1 -trA (M -zI p ) -1 | ≤ A |z| .
Note that the bound in Lemma 1 does not depend on u . In particular denoting

δ(z) = 1 n tr (I p + µµ T ) I p + µµ T 1 + δ(z) -zI p -1 , δ (z) = 1 n tr 1 1 + δ(z) I p -zI p -1 , we obtain ∀z < 0, δ(z) = δ (z) + O(n -1 ).
Therefore as p, n → ∞ with p/n ≡ c 0 → c ∞ 0 , particularizing to C = I p + µµ T (or equivalently to C = I p following Lemma 1), Equation (6.2) has an explicit expression uniquely defined (using the definition of the Stieltjes transform) as:

δ(z) = (c 0 + z -1) 2 -4c 0 z -(c 0 + z -1) 2z
The expression of m(z) is retrieved by using the relation m

(z) = 1 1+δ(z) -z -1
. We conclude by remarking that δ(z) = c 0 m(z).

Similarly as for Q(z), the deterministic equivalent of Q(z) is retrieved from Theorem 5 for C = I n + µ 2 yy T . Finally, the deterministic equivalent of Q(z) 2 follows by remarking that Q(z) 2 = ∂ ∂z Q(z).

Appendix for Chapter 3

We provide here the technical developments for the proof of Theorem 7 as well as all subsequent corollaries (Corollaries 2-5) for the family of metrics expressed as linear spectral statistic of Σ -1 1 Σ 2 for simplicity. The appendix is structured as follows: Appendix 6.2.1 provides the proof of Theorem 7 following the same approach as in (Couillet et al., 2011), relying mostly on the results from [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF][START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF]. Appendix 6.2.2 discusses in detail the question of the position of the complex contours when affected by change of variables. Appendix 6.2.3 then provides the technical details of the calculi behind Corollaries 2-5; this is undertaken through a first thorough characterization of the singular points of ϕ p and ψ p and functionals of these (these singular points are hereafter denoted λi , η i , ζ i and κ i ), allowing for a proper selection of the integration contour, and subsequently through a detailed calculus for all functions f (t) under study. Appendix 6.2.4 provides the details framework for the covariance matrix estimation as well as with the technical arguments.

Integral Form

Relating m ν to m µ We start by noticing that we may equivalently assume the following setting:

• x (1) 1 , . . . , x (1)
n 1 ∈ R p vectors of i.i.d. zero-mean and unit variance entries

• x (2) 1 , . . . , x (2)
n 2 ∈ R p of the form x (2) i = Σ 1 2 x(2) i with x (2)
i ∈ R p a vector of i.i.d. zero-mean and unit variance entries

where Σ ≡ Σ -1 2 1 Σ 2 Σ -1 2 1 . Indeed, with our first notations, Σ-1 1 Σ2 = 1 n 1 Σ -1 2 1 X1 XT 1 Σ -1 2 1 1 n 2 Σ 1 2 2 X2 XT 2 Σ 1 2 2 (here Xa = [x (a) 1 , . . . , x(a) na ]
), the eigenvalue distribution of which is the same as that of the matrix ( 1

n 1 X1 XT 1 )( 1 n 2 Σ -1 2 1 Σ 1 2 2 X2 XT 2 Σ 1 2 2 Σ -1 2
1 ) and we may then consider that the x

i 's actually have covariance I p , while the x

(2) i 's have covariance Σ = Σ -1 2 1 Σ 2 Σ -1 2
1 , without altering the spectra under study. With these new definitions, we first condition with respect to the x (2) i 's, and study the spectrum of Σ-1 1 Σ2 , which is the same as that of

Σ 1 2 2 Σ-1 1 Σ 1 2 2 . A useful remark is the fact that Σ 1 2 2 Σ-1 1 Σ 1 2 2 is the "inverse spectrum" of Σ-1 2 2 Σ1 Σ-1 2 2 , which is itself the same spectrum as that of 1 n 1 X T 1 Σ-1 2 X 1 except for n 1 -p additional zero eigenvalues. Denoting μ-1 p the eigenvalue distribution of 1 n 1 XT 1 Σ-1 2 X1
, we first know from [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF] that, under Assumption 1, as p → ∞, μ-1 p a.s.

-→ μ-1 , where μ-1 is the probability measure with Stieltjes transform m μ-1 defined as the unique (analytical function) solution to

m μ-1 (z) = -z + c ∞ 1 tdξ -1 2 (t) 1 + tm μ-1 (z) -1
with ξ 2 the almost sure limiting spectrum distribution of Σ2 and m ξ 2 its associated Stieltjes transform (note importantly that, from [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF] and Assumption 1, ξ 2 has bounded support and is away from zero). Recognizing a Stieltjes transform from the right-hand side integral, this can be equivalently written

m μ-1 (z) = -z + c ∞ 1 m μ-1 (z) - c ∞ 1 m μ-1 (z) 2 m ξ -1 2 - 1 m μ-1 (z) -1 . (6.3)
Accounting for the aforementioned additional zero eigenvalues, μ-1 is related to µ -1 , the almost sure limiting spectrum distribution of Σ-1/2

2 Σ1 Σ-1/2 2 , through the relation μ-1 = c ∞ 1 µ -1 + (1 -c ∞ 1
)δ 0 with δ x the Dirac measure at x and we have

m μ-1 (z) = c ∞ 1 m µ -1 (z) -(1 -c ∞ 1 ) 1 z .
Plugging this last relation in (6.3) leads then to

m ξ -1 2 z 1 -c ∞ 1 -c ∞ 1 zm µ -1 (z) = m µ -1 (z) 1 -c ∞ 1 -c ∞ 1 zm µ -1 (z) . (6.4)
Now, with the convention that, for a probability measure θ, θ -1 is the measure defined through θ

-1 ([a, b]) = θ [ 1 a , 1 b ] ,
we have the Stieltjes transform relation

m θ -1 (z) = - 1 z - 1 z 2 m θ 1 z .
Using this relation in (6.3), we then deduce

zm µ (z) = (z + c ∞ 1 z 2 m µ (z))m ξ 2 (z + c ∞ 1 z 2 m µ (z)) = ϕ(z)m ξ 2 (ϕ(z)) (6.5)
where we recall that ϕ(z) = z(1 + c ∞ 1 zm µ (z)). It will come in handy in the following to differentiate this expression along z to obtain

m ξ 2 (ϕ(z)) = 1 ϕ(z) m µ (z) + zm µ (z) ϕ (z) -m ξ 2 (ϕ(z))
which might be conveniently rewritten as

m ξ 2 (ϕ(z)) = 1 ϕ(z) - ψ (z) c ∞ 2 ϕ (z) -m ξ 2 (ϕ(z)) . (6.6)
We next determine m ξ 2 as a function of ν. Since Σ2 is itself a sample covariance matrix, we may apply again the results from [START_REF] Silverstein | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF]. Denoting ξ2 the almost sure limiting spectrum distribution of 1 n 2

XT

2 Σ X2 , we first have

m ξ2 (z) = -z + c ∞ 2 tddν(t) 1 + tm ξ2 (z) -1
.

(6.7)

Similar to previously, we have the Stieltjes transform relation

m ξ2 (z) = c ∞ 2 m ξ 2 (z)- (1-c ∞ 2 ) z
which yields, when plugged in (6.7)

m ν - z c ∞ 2 zm ξ 2 (z) -(1 -c ∞ 2 ) = -m ξ 2 (z) (c ∞ 2 zm ξ 2 (z) -(1 -c ∞ 2 )) . (6.8)
The two relations (6.5) and (6.7) will be instrumental to relating f dν to the observation measure µ p , as described in the next section.

Remark 11 (The case c ∞ 2 > 1). The aforementioned reasoning carries over to the case c ∞ 2 > 1. Indeed, since the equation (6.3) is now meaningless (as the support of ξ 2 contains the atom {0}), consider the model Σ-1 1 ( Σ2 + εI p ) = Σ-1 1 Σ2 + ε Σ-1 1 for some small ε > 0. Then (6.5) holds with now ξ 2 the limiting empirical spectral distribution of Σ2 + εI p . Due to ε, Equation (6.7) now holds with m ξ2 (z) replaced by m ξ2 (z + ε). By continuity in the small ε limit, we then have that (6.5) and (6.8) still hold in the small ε limit. Now, since Σ-1 1 ( Σ2 + εI p ) -Σ-1 1 Σ2 = ε Σ-1 1 , the operator norm of which almost surely vanishes as ε → 0 (as per the almost sure boundedness of lim sup p Σ-1 1

), we deduce that µ p → µ defined through (6.5) and (6.8), almost surely, also for c ∞ 2 > 1.

Integral formulation over m ν

With the formulas above, we are now in position to derive the proposed estimator. We start by using Cauchy's integral formula to obtain

f dν = - 1 2πı Γν f (z)m ν (z)dz
for Γ ν a complex contour surrounding the support of ν but containing no singularity of f in its inside. This contour is carefully chosen as the image of the mapping ω

→ z = -ω/(c ∞ 2 ωm ξ 2 (ω) -(1 -c ∞ 2
)) of another contour Γ ξ 2 surrounding the limiting support of ξ 2 ; the details of this (non-trivial) contour change are provided in Appendix 6.2.2 (where it is seen that the assumption c ∞ 2 < 1 is crucially exploited). We shall admit here that this change of variable is licit.

Operating the aforementioned change of variable gives

f dν = 1 2πı Γ ξ 2 f -ω c ∞ 2 ωm ξ 2 (ω)-(1-c ∞ 2 ) m ξ 2 (ω) c ∞ 2 ω 2 m ξ 2 (ω) + (1 -c ∞ 2 ) c ∞ 2 ωm ξ 2 (ω) -(1 -c ∞ 2 )
dω (6.9)

where we used (6.8) to eliminate m ν .

To now eliminate m ξ 2 and obtain an integral form only as a function of m µ , we next proceed to the variable change u → ω = ϕ(u) = u + c ∞ 1 u 2 m µ (u). Again, this involves a change of contour, which is valid as long as Γ ξ 2 is the image by ϕ of a contour Γ µ surrounding the support of µ, which is only possible if c ∞ 1 < 1 (see Appendix 6.2.2 for further details). With this variable change, we can now exploit the relations (6.5) and (6.6) to obtain, after basic algebraic calculus (using in particular the relation

um µ (u) = (-ψ(u) + 1 -c ∞ 2 )/c ∞ 2 ) f dν = 1 2πı Γµ f ϕ(u) ψ(u) ψ(u) c ∞ 2 ϕ (u) ϕ(u) - ψ (u) ψ(u) du - 1 -c ∞ 2 c ∞ 2 1 2πı Γµ f ϕ(u) ψ(u) ϕ (u) ϕ(u) - ψ (u) ψ(u) du.
Performing the variable change backwards (u → z = ϕ(u) ψ(u) ), the rightmost term is

1 2πı Γµ f ϕ(u) ψ(u) ϕ (u) ϕ(u) - ψ (u) ψ(u) du = 1 2πı Γµ f ϕ(u) ψ(u) ψ(u) ϕ(u) ϕ(u) ψ(u) du = 1 2πı Γν f (z) z dz = 0, since Γ ν ⊂ {z ∈ C, R[z] > 0}.
Note that if Γ ν were to contain 0 (which occurs when c ∞ 2 > 1), then an additional residue equal to -

1-c ∞ 2 c ∞ 2 f (
0) would have to be accounted for. We conclude that

f dν = 1 2πı Γµ f ϕ(u) ψ(u) ψ(u) c ∞ 2 ϕ (u) ϕ(u) - ψ (u) ψ(u) du.
To ensure that m µ can be replaced by m µp in the the above expression, one however needs to ensure that dominated convergence on the compact set Γ µ holds. For this, two ingredients are needed: (i) First we need to guarantee that the convergence m µp a.s.

-→ m µ is uniform on Γ µ , which easily follows from the analytic nature of Stieltjes transforms, and most importantly (ii) prove that f

ϕp(u) ψp(u) ψp(u) c 2 ϕ p (u) ϕp(u) - ψ p (u)
ψp(u) is uniformly bounded on Γ µ . This second item follows from (Bai & Silverstein, 1998b) which prove that the eigenvalues of Σ-1 1 Σ2 are asymptotically assembled in contiguous bulks and almost surely do not escape the limiting support µ as p → ∞ under the condition that the fourth moment of the entries of the matrices X1 and X2 is finite and that the limiting spectrum ν of Σ -1

1 Σ 2 has bounded support away from zero (due to the assumption lim sup p max{ Σ -1 a , Σ a } < ∞) along with the analyticity of the involved functions inside the contour. This allows to retrieve Theorem 7 by uniform convergence on the compact contour (see also (Couillet et al., 2011) for a similar detailed derivation).

Remark 12 (Case Σ 1 known). The case where Σ 1 is known is equivalent to setting c 1 → 0 above, leading in particular to m µ = m ξ 2 and to the unique functional equation

m ν z 1 -c ∞ 2 -c ∞ 2 zm µ (z) = m ν (z) (1 -c ∞ 2 -c ∞ 2 zm ν (z)) .
In particular, if Σ 1 = Σ 2 , this reduces to

1 = -m µ (z)(z -ψ(z)) with ψ(z) = 1 -c ∞ 2 -c ∞ 2 zm µ (z)
, which is the functional Stieltjes-tranform equation of the popular Marcenko-Pastur law [START_REF] Marcenko | Distributions of eigenvalues for some sets of random matrices[END_REF].

Integration contour determination

This section details the complex integration steps sketched in Appendix 6.2.1. These details rely heavily on the works of [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] and follow similar ideas as in e.g., (Couillet et al., 2011).

Our objective is to ensure that the successive changes of variables involved in Appendix 6.2.1 move any complex contour closely encircling the support of µ onto a valid contour encircling the support of ν; we will in particular be careful that the resulting contour, in addition to encircling the support of ν, does not encircle additional values possibly bringing undesired residues (such as 0). We will proceed in two steps, first showing that a contour encircling µ results on a contour encircling ξ 2 and a contour encircling ξ 2 results on a contour encircling ν.

Let us consider a first contour Γ ξ 2 closely around the support of ξ 2 (in particular not containing 0). We have to prove that any point ω of this contour is mapped to a point of a contour Γ ν closely around the support of ν.

The change of variable performed in (6.8) reads, for all ω ∈ C \ Supp(ξ 2 ),

z ≡ z(ω) = -ω -(1 -c ∞ 2 ) + c ∞ 2 ωm ξ 2 (ω) = -1 m ξ2 (ω) where we recall that ξ2 = c ∞ 2 ξ 2 + (1 -c ∞ 2 )δ 0 . Since [ω] [m ξ2 (ω)] > 0 for
[ω] = 0, we already have that [z] [ω] > 0 for all non-real ω.

It therefore remains to show that real ω's (outside the support of ξ 2 ) project onto properly located real z's (i.e., on either side of the support of ν). This conclusion follows from the seminal work [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] on the spectral analysis of sample covariance matrices. The essential idea is to note that, due to (6.7), the relation z(ω) = -1/m ξ2 (ω) can be inverted as

ω ≡ ω(z) = - 1 m ξ2 + c ∞ 2 tdν(t) 1 + tm ξ2 = z + c ∞ 2 tdν(t) 1 -t z . ω- ω+ z- z+ S ξ 2 Sν z ω • (z) ω- ω+ z- z+ S ξ 2 Sν z ω • (z) Figure 6.1: Variable change z → ω • (z) = z + c ∞ 2 zdν(t)
z-t for c ∞ 2 < 1 (left) and c ∞ 2 > 1 (right). S θ is the support of the probability measure θ. For 0 < ω -= ω(z -) < inf Supp(ν), the pre-image z -is necessarily negative for c ∞ 2 > 1.

In [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF], it is proved that the image by ω(•) of z(R \ Supp(ξ 2 )) coincides with the increasing sections of the function ω

• : R \ Supp(ν) → R, z → ω(z).
The latter being an explicit function, its functional analysis is simple and allows in particular to properly locate the real pairs (ω, z). Details of this analysis are provided in [START_REF] Silverstein | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] as well as in (Couillet et al., 2011), which shall not be recalled here. The function ω • is depicted in Figure 6.1; we observe and easily prove that, for c ∞ 2 < 1, any two values z -< inf(Supp(ν)) ≤ sup(Supp(ν)) < z + have respectively images ω -and ω + satisfying w -< inf(Supp(ξ 2 )) ≤ sup(Supp(ξ 2 )) < w + as desired. This is however not the case for c ∞ 2 > 1 where {z -, z + } enclose not only Supp(ν) but also 0 and therefore do not bring a valid contour. This essentially follows from the fact that (ϕ p /ψ p ) (0) is positive for c ∞ 2 < 1 and negative for c ∞ 2 > 1. Even though in Figure 6.1 the support µ is considered compact of one component, the analysis and conclusions remains true even for several disjoint supports.

The same reasoning now holds for the second variable change. Indeed, note that here

ω = u(1 + c ∞ 1 um µ (u)) = u 1 -c ∞ 1 - c ∞ 1 u m µ -1 1 u = -m μ-1 1 u .
Exploiting (6.3) provides, as above, a functional inverse given here by

u ≡ u(ω) = 1 ω + c ∞ 1 dξ 2 (t) t -ω -1
the analysis of which follows the same arguments as above (see display in Figure 6.2 of the extension to u

• (ω) = u(ω) for all ω ∈ R \ Supp(ξ 2 )). u- u+ ω- ω+ Sµ S ξ 2 ω u • (ω) u+ u- ω- ω+ Sµ S ξ 2 ω u • (ω) Figure 6.2: Variable change u • (ω) = ( 1 ω + c ∞ 1 1 t-ω dξ 2 (t)) -1 for c ∞ 2 < 1 (left) and c ∞ 2 > 1 (right)
. S θ is the support of the probability measure θ.

Integral Calculus

To compute the complex integral in theorem 7, note first that, depending on f , several types of singularities in the integral may arise. Of utmost interest (but not always exhaustively, as we shall see for f (t) = log(1 + st)) are: (i) the eigenvalues λi of Σ-1 1 Σ2 , (ii) the non-null values η i such that ϕ p (η i ) = 0, (iii) the values ζ i such that ψ p (ζ i ) = 0.

In the following, we first introduce a sequence of intermediary results of interest for most of the integral calculi.

Rational expansion

At the core of the subsequent analysis is the function

ϕ p (z) ϕp(z) - ψ p (z) ψp(z) ψp(z) c 2 .
As this is a rational function, we first obtain the following important partial fraction decomposition, that will be repeatedly used in the sequel:

ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 (6.10) = 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 1 z -λj + 1 -c 2 c 2 1 z + c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 1 z -η j .
This form is obtained by first observing that the λj 's, η j 's and 0 are the poles of the left-hand side expression. Then, pre-multiplying the left-hand side by (z -λj ), z, or (z -η j ) and taking the limit when these terms vanish, we recover the right-hand side, using in particular the following estimates (which easily entail from the definitions of ϕ p and ψ p ):

ϕ p (z) = c 1 p λ2 i λi -z -2c 1 λi p + λi + c 1 p j =i λ2 i λj -λi + O( λi -z) ϕ p (z) = c 1 p λ2 i ( λi -z) 2 + O(1) ψ p (z) = - c 2 p λi λi -z + c 2 p + 1 -c 2 - c 2 p j =i λi λj -λi + O( λi -z) ψ p (z) = - c 2 p λi ( λi -z) 2 + O(1)
in the vicinity of λi , along with

ψ p (η i ) = c 1 +c 2 -c 1 c 2 c 1
and ψ p (0) = 1 -c 2 . To retrieve the constant term of the partial fraction decomposition, we further compute lim

z→∞ ϕ p (z) ϕp(z) - ψ p (z) ψp(z) ψp(z)
c 2 which is shown to be zero by using the expressions of ϕ p (z) and ψ p (z).

From this expression, we have the following immediate corollary.

Remark 13 (Residue for f analytic at λi ). If f • (ϕ p /ψ p ) is analytic in a neighborhood of λi , i.e., if f is analytic in a neighborhood of -(c 1 /c 2 ) λi , then λi is a first-order pole for the integrand, leading to the residue

Res( λi ) = -f - c 1 c 2 λi c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p .
Characterization of η i and ζ i , and ϕ p /ψ p First note that the η i (the zeros of ϕ p (z)) and ζ i (the zeros of ψ p (z)) are all real as one can verify that, for [z] = 0, [ϕ p (z)] [z] > 0 and [ψ p (z)] [z] < 0. Before establishing the properties of ϕ p and ψ p in the vicinity of η i and ζ i , let us first locate these values. A study of the function M p : R → R, x → xm µp (x) (see Figure 6.3) reveals that M p is increasing (since x/( λi -x) = -1 + 1/( λi -x)) and has asymptotes at each λi with lim x↑ λi M p (x) = ∞ and lim x↓ λi M p (x) = -∞. As a consequence, since ϕ p (x) = 0 ⇔ M p (x) = -1 c 1 < -1, there exists exactly one solution to ϕ p (x) = 0 in the set ( λi , λi+1 ). This solution will be subsequently called η i . Since M p (x) → -1 as x → ∞, there exists a last solution to ϕ p (x) = 0 in ( λp , ∞), hereafter referred to as η p . Similarly, ψ p (x) = 0 ⇔ M p (x) = (1 -c 2 )/c 2 > 0 and thus there exists exactly one solution, called ζ i in ( λi-1 , λi ). When x → 0, M p (x) → 0 so that a further solution is found in (0, λ1 ), called ζ 1 . Besides, due to the asymptotes at every λi , we have that

ζ 1 < λi < η 1 < ζ 2 < . . . < η p .
As such, the set Γ defined in Theorem 7 exactly encloses all η i , λi , and ζ i , for i = 1, . . . , p, possibly to the exception of the leftmost ζ 1 and the rightmost η p (as Figure 6.3: Visual representation of x → M p (x) = xm µp (x); here for p = 4, n 1 = 8, n 2 = 16. Solutions to M p (x) = -1/c 1 (i.e., η i 's) and to M p (x) = (1 -c 2 )/c 2 (i.e., ζ i 's) indicated in red crosses. Green solid lines indicate sets of negative ϕ p /ψ p . those are not comprised in a set of the form [ λi+1 , λi ]). To ensure that the latter do asymptotically fall within the interior of Γ, one approach is to exploit Theorem 7 for the elementary function f (t) = 1. There we find that

1 2πı Γν m ν (z)dz - 1 2πı Γ ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz a.s.
-→ 0.

The left integral is easily evaluated by residue calculus and equals -1 (each λ i (Σ -1 1 Σ 2 ), 1 ≤ i ≤ p, is a pole with associated residue -1/p), while the right integral can be computed from (6.10) again by residue calculus and equals -1

+ c 1 +c 2 -c 1 c 2 c 1 c 2 (p -#{η i ∈ Γ • }) with Γ •
the "interior" of Γ. As such, since both integrals are (almost surely) arbitrarily close in the large p limit, we deduce that #{η i ∈ Γ • } = p for all large p and thus, in particular, η p is found in the interior of Γ. To obtain the same result for ζ 1 , note that, from the

relation ψ p (z) = c 1 +c 2 -c 1 c 2 c 1 -c 2 c 1 ϕp(z) z
along with the fact that

ϕ p (z) ϕp(z) - ψ p (z)
ψp(z) is an exact derivative (of log(ϕ p /ψ p )), the aforementioned convergence can be equivalently written

1 2πı Γν m ν (z)dz - -1 2πı Γ ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ϕ p (z) zc 1 dz a.s.
-→ 0.

Reproducing the same line of argument (with an expansion of (

ϕ p (z) ϕp(z) - ψ p (z) ψp(z) ) ϕp(z) zc 1
equivalent to (6.10)), the same conclusion arises and we then proved that both ζ 1 One can also establish that, on its restriction to R + , ϕ p is everywhere positive but on the set ∪ p i=1 ( λi , η i ). Similarly, ψ p is everywhere positive but on the set ∪ p i=1 (ζ i , λi ). As a consequence, the ratio ϕ p /ψ p is everywhere positive on R + but on the set ∪ p i=1 (ζ i , η i ). These observations are synthesized in Figure 6.4.

In terms of monotonicity on their restrictions to the real axis, since ψ p (x) = 1c 2 t t-x dµ p (t), ψ p is decreasing. As for ϕ p , note that

ϕ p (x) = 1 + 2c 1 x t -x dµ p (t) + c 1 x 2 (t -x) 2 dµ p (t) = t 2 -2(1 -c 1 )xt + (1 -c 1 )x 2 (t -x) 2 dµ p (t).
Since c 1 < 1, we have 1 -c 1 > (1 -c 1 ) 2 , and therefore

ϕ p (x) > (t -(1 -c 1 )x) 2 (t -x) 2 dµ p (t) > 0
ensuring that ϕ p is increasing on its restriction to R. Showing that x → ϕ p (x)/ψ p (x) is increasing is important for the study of the case f (t) = log(1 + st) but is far less immediate. This unfolds from the following remark, also of key importance in the following.

Remark 14 (Alternative form of ϕ p and ψ p ). It is interesting to note that, in addition to the zero found at z = 0 for ϕ p , we have enumerated all zeros and poles of the rational functions ϕ p and ψ p (this can be ensured from their definition as rational functions) and it thus comes that

ϕ p (z) = (1 -c 1 )z p j=1 (z -η j ) p j=1 (z -λj ) (6.11) ψ p (z) = p j=1 (z -ζ j ) p j=1 (z -λj ) (6.12)
where the constants 1 -c 1 and 1 are found by observing that, as z = x ∈ R → ∞, ϕ p (x)/x → 1 -c 1 and ψ p (x) → 1. In particular

ϕ p (z) ψ p (z) = (1 -c 1 )z p j=1 (z -η j ) p j=1 (z -ζ j )
. (6.13)

A further useful observation is that the η i 's are the eigenvalues of

Λ - 1 p -n 1 λ λT
where Λ = diag({ λi } p i=1 ) and λ = ( λ1 , . . . , λp ) T . Indeed, these eigenvalues are found by solving These observations allow for the following useful characterization of ϕ p /ψ p :

0 = det   Λ - λ λT p -n 1 -xI p   = det(Λ -xI p ) det   I p -(Λ -xI p ) -1 λ λT p -n 1   = det(Λ -xI p ) 1 - 1 p -n 1 λT (Λ -xI p ) -1 λ = det(Λ -xI p ) 1 - 1 p -n 1
ϕ p (z) ψ p (z) = (1 -c 1 )z det zI p -Λ -1 n 1 -p λ λT det zI p -Λ + 1 n 2 λ λT = (1 -c 1 )z 1 - n 1 + n 2 -p n 2 (n 1 -p) λT zI p -Λ + 1 n 2 λ λT -1 λ
(after factoring out the matrix in denominator from the determinant in the numerator) the derivative of which is, after simplification, for

ϕ p (z) ψ p (z) = (1 -c 1 ) 1 + n 1 + n 2 -p n 2 (n 1 -p) λT Q Λ - 1 n 2 λ λT Q λ .
Q = (zI p -Λ+ 1 n 2 λ λT ) -1 . Since Λ-1 n 2
λ λT is positive definite (its eigenvalues being the ζ i 's), on the real axis the derivative is greater than 1 -c 1 > 0 and the function x → ϕ p (x)/ψ p (x) is therefore increasing. Figure 6.5 displays the behavior of ϕ p /ψ p when restricted to the real axis.

Since we now know that the contour Γ from Theorem 7 encloses exactly all η i 's and ζ i 's, it is sensible to evaluate the residues for these values when f (z) is analytic in their neighborhood.

Remark 15 (Residue for f analytic at η i and ζ i ). If f is analytic with no singularity at zero, then the integral has a residue at η i easily found to be

Res(η i ) = f (0) c 1 + c 2 -c 1 c 2 c 2 .
Similarly, if f (ω) has a well defined limit as |ω| → ∞, then no residue is found at ζ i .

As a consequence of Remarks 13 and 15, we have the following immediate corollary.

Remark 16 (The case f (t) = t). In the case where f (t) = t, a singularity appears at ζ i , which is nonetheless easily treated by noticing that the integrand then reduces to

f ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 = ϕ p (z) c 2 - ψ p (z)ϕ p (z) c 2 ψ p (z)
and thus, with

ψ p (z) = (z -ζ i )ψ p (ζ i ) + O((z -ζ i ) 2
), we easily find the residue

Res {f (t)=t} (ζ i ) = - ϕ p (ζ i ) c 2 = -ζ i c 1 + c 2 -c 1 c 2 c 2 2 .
Together with Remarks 13 and 15, along with the fact that Γ encloses all η i and λi , for i = 1, . . . , p, we then find that

tdν p (t) - c 1 + c 2 -c 1 c 2 c 2 2 p i=1 ( λi -ζ i ) - c 1 c 2 1 p p i=1 λi a.s.
-→ 0.

By then noticing that

i ζ i = tr(Λ -1 n 2 λ λT ) = (1 -c 2 /p) i λi , we retrieve Corol- lary 2. Development for f (t) = log(t)
The case f (t) = log(t) leads to an immediate simplification since, then, log det( Σ-1 1 Σ2 ) = log det( Σ2 )-log det( Σ1 ); one may then use previously established results from the random matrix literature (e.g., the G-estimators in [START_REF] Girko | Introduction to general statistical analysis[END_REF] or more recently [START_REF] Kammoun | Performance of mutual information inference methods under unknown interference[END_REF]) to obtain the sought-for estimate. Nonetheless, the full explicit derivation of the contour integral in this case is quite instructive and, being simpler than the subsequent cases where f (t) = log 2 (t) or f (t) = log(1 + st) that rely on the same key ingredients, we shall here conduct a thorough complex integral calculus.

For z ∈ C, define first f (z) = log(z) where log(z) = log(|z|)e i arg(z) , with arg(z) ∈ (-π, π]. For this definition of the complex argument, since ϕ p (x)/ψ p (x) is everywhere positive but on ∪ p i=1 (ζ i , η i ), we conclude that arg(ϕ p (z)/ψ p (z)) abruptly moves from π to -π as z moves from x + 0 + ı to x + 0 -ı for all x ∈ ∪ p i=1 (ζ i , η i ). This creates a set of p branch cuts [ζ i , η i ], i = 1, . . . , p as displayed in Figure 6.5. This naturally leads to computing the complex integral estimate of f dν based on the contour displayed in Figure 6.6, which avoids the branch cuts.

This contour encloses no singularity of the integrand and therefore has a null integral. With the notations of Figure 6.6, the sought-for integral (over Γ) therefore satisfies

0 = Γ + p i=1 I A i + I B i + I C i + I D i + I E i .
We start by the evaluation of the integrals over I B i and I D i , which can be similarly handled. To this end, note that, since arg( ϕp ψp ) moves from π to -π across the branch cut, we have

1 2πı I B i = 1 2πı λi -ε ζ i +ε log - ϕ p (x) ψ p (x) + ıπ -log - ϕ p (x) ψ p (x) + ıπ × ϕ p (x) ϕ p (x) - ψ p (x) ψ p (x) ψ p (x) c 2 dx Figure 6
.6: Chosen integration contour. The set I B i is the disjoint union of the segments

[ζ i + ε + 0 + ı, λi -ε + 0 + ı] and [ζ i + ε + 0 -ı, λi -ε + 0 -ı]. Similarly the set I D i is the disjoint union of the segments [ λi + ε + 0 + ı, η i -ε + 0 + ı] and [ λi + ε + 0 -ı, η i -ε + 0 -ı].
The sets I A i , I C i and I E i are the disjoint unions of semi-circles (in the upper-or lower-half complex plane) of diameters ε surrounding ζ i , λi and η i respectively. = λi -ε

ζ i +ε ϕ p (x) ϕ p (x) - ψ p (x) ψ p (x) ψ p (x) c 2 dx.
We first exploit the rational form expansion (6.10) of (

ϕ p (z) ϕp(z) - ψ p (z) ψp(z) ) ψp(x) c 2
to obtain the integral over

I B i 1 2πı I B i = 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2   j =i log λi -λj ζ i -λj + log ε ζ i -λi   + 1 -c 2 c 2 log λi ζ i + c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 log λi -η j ζ i -η j + o(ε).
The treatment is similar for the integral over I D i which results, after summation of both integrals, to

1 2πı I B i ∪I D i = 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 log η i -λj ζ i -λj + 1 -c 2 c 2 log η i ζ i + c 1 + c 2 -c 1 c 2 c 1 c 2   j =i log η i -η j ζ i -η j + log ε ζ i -η i   + o(ε).
Note here the asymmetry in the behavior of the integrand in the neighborhood of ζ i (+ε) and η i (-ε); in the former edge, the integral is well defined while in the latter it diverges as log ε which must then be maintained.

Summing now over i ∈ {1, . . . , p}, we recognize a series of identities. In particular, note that from the product form (6.13),

p j=1 p i=1 log η i -λj ζ i -λj = p j=1 log ψp ϕp ( λj ) (1 -c 1 ) λj = log c 1 c 2 (1 -c 1 ) p j=1 log η i ζ i = lim z→0 log   ψp ϕp (z) (1 -c 1 )z   = -log ((1 -c 1 )(1 -c 2 )) p i=1 j =i log η i -η j ζ i -η j + p i=1 log 1 ζ i -η i = lim z→η i p j=1 log ψp ϕp (z) (1 -c 1 )z(z -η j ) = p j=1 log ψp ϕp (η j ) (1 -c 1 )η j .
As such, we now find that 1 2πı

p i=1 I B i ∪I D i = log c 1 c 2 (1 -c 1 ) - 1 -c 2 c 2 log ((1 -c 1 )(1 -c 2 )) - c 1 + c 2 -c 1 c 2 c 1 c 2   p log c 1 c 2 (1 -c 1 ) - p j=1 log ψp ϕp (η j ) (1 -c 1 )η j    + c 1 + c 2 -c 1 c 2 c 1 c 2 p log ε + o(ε).
The diverging term in log ε is compensated by the integral over I E i . Indeed, letting z = η i + εe iθ , we may write

1 2πı I E i log ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ(z) c 2 dz = ε 2πc 2 0 + π + -π 0 - log (1 -c 1 )(η i + εe iθ ) p j=1 (η i -η j + εe iθ ) p j=1 (η i -ζ j + εe iθ ) ×   p j=1 1 p -c 1 +c 2 -c 1 c 2 c 1 c 2 η i + εe ıθ -λj + 1-c 2 c 2 η i + εe ıθ + p j=1 c 1 +c 2 -c 1 c 2 c 1 c 2 η i + εe ıθ -η j   e iθ dθ.
To evaluate the small ε limit of this term, first remark importantly that, for small ε, the term in the logarithm equals

(1 -c 1 ) η i η i -ζ i j =i (η i -η j ) j =i (η i -ζ j ) εe iθ + o(ε)
the argument of which equals that of θ. As such, on the integral over (π, 0), the log term reads log | • | + ıθ + o(ε), while on (0, -π), it reads log | • | -ıθ + o(ε). With this in mind, keeping only the non-vanishing terms in the small ε limit (that is: the term in log ε and the term in 1 ε ) leads to

1 2πı I E i log ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ(z) c 2 dz = c 1 + c 2 -c 1 c 2 c 1 c 2 log ε + c 1 + c 2 -c 1 c 2 c 1 c 2 log ϕ p ψ p (η i ) + o(ε)
where we used the fact that lim ε→0

1 εe ıθ ϕp ψp (η i + εe ıθ ) = ϕp ψp (η i ).
We proceed similarly to handle the integral over

I C i 1 2πı I C i log ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ(z) c 2 dz = ε 2πc 2 0 + π + -π 0 - log (1 -c 1 )( λi + εe iθ ) p j=1 ( λi -η j + εe iθ ) p j=1 ( λi -ζ j + εe iθ ) ×   p j=1 1 p -c 1 +c 2 -c 1 c 2 c 1 c 2 λi + εe ıθ -λj + 1-c 2 c 2 λi + εe ıθ + p j=1 c 1 +c 2 -c 1 c 2 c 1 c 2 λi + εe ıθ -η j   e iθ dθ.
Here, for small ε, the angle of the term in the argument of the logarithm is that of

ϕ p ψ p ( λi ) + ϕ p ψ p ( λi )εe ıθ + o(ε) = - c 1 c 2 λi + εe ıθ c 1 c 2 p c 1 + c 2 -c 1 c 2 c 1 c 2 -1 + o(ε).
That is, for all large p, the argument equals π + o(ε) < π uniformly on θ ∈ (0, π) and -π + o(ε) > -π uniformly on θ ∈ (-π, 0); thus the complex logarithm reads log

| • | + ıθ + o(ε) on (π, 0), while on (0, -π), it reads log | • | -ıθ + o(ε).
Proceeding as previously for the integral over I E i , we then find after calculus that Note that this expression is reminiscent of a "residue" at λi (with negatively oriented contour), according to Remark 13, however for the function log | • | and not for the function log(•), due to the branch cut passing through λi . The final integral over I A i is performed similarly. However, here, it is easily observed that the integral is of order O(ε log ε) in the small ε limit, and thus vanishes.

1 2πı I C i log ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ(z) c 2 dz = c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p log c 1 c 2 λi .
Finally, summing up all contributions, we have

1 2πı Γ = - p i=1 1 2πı I A i + I B i + I C i + I D i + I E i = -log c 1 (1 -c 2 ) c 2 + 1 c 2 log((1 -c 1 )(1 -c 2 )) + 1 p p i=1 log c 1 c 2 λi - c 1 + c 2 -c 1 c 2 c 1 c 2 p log(1 -c 1 ) + p i=1 log λi - p i=1 log((1 -c 1 )η i ) = -log(1 -c 2 ) + 1 c 2 log((1 -c 1 )(1 -c 2 )) + 1 p p i=1 log λi + c 1 + c 2 -c 1 c 2 c 1 c 2 p i=1 log η i λi = 1 p p i=1 log λi + 1 -c 2 c 2 log(1 -c 2 ) - 1 -c 1 c 1 log(1 -c 1 )
where in the last equality we used, among other algebraic simplifications, the fact that

p i=1 log( η i λi ) = lim x→0 log( ψp(x) (1-c 1 )x ) = -log(1 -c 1 )
. This is the sought-for result.

Development for f (t) = log(1 + st)

The development for f (t) = log(1 + st) is quite similar to that of f (t) = log(t), with some noticeable exceptions with respect to the position of singularity points.

A few important remarks are in order to start with this scenario. First note from Figure 6.4 and the previous discussions that the function z → log(1 + sϕ p (z)/ψ p (z)) has a singularity at z = κ i , i = 1, . . . , p, for some κ i ∈ (ζ i , η i ) solution to 1 + sϕ p (x)/ψ p (x) = 0 (indeed, ϕ p (x)/ψ p (x) is increasing on (ζ i , η i ) with opposite asymptotes and thus κ i exists and is uniquely defined). In addition, log(1 + sϕ p (z)/ψ p (z)) has a further singularity satisfying 1 + sϕ p (x)/ψ p (x) = 0 in the interval (-∞, 0) which we shall denote κ 0 .

A few identities regarding κ i are useful. Using the relation between ϕ p and ψ p , we find in particular that

ϕ p (κ i ) = - 1 s c 1 + c 2 -c 1 c 2 c 2 κ i -1 s + c 1 c 2 κ i ψ p (κ i ) = c 1 + c 2 -c 1 c 2 c 2 κ i -1 s + c 1 c 2 κ i (ψ p + sϕ p ) c 2 c 1 s = c 1 + c 2 -c 1 c 2 c 1 .
With the discussions above, we also find that

1 + s ϕ p (z) ψ p (z) = (1 -c 1 )s(z -κ 0 ) p i=1 (z -κ i ) p i=1 (z -ζ i ) (6.14) ψ p (z) + sϕ p (z) = s(1 -c 1 )(z -κ 0 ) p i=1 (z -κ i ) p i=1 (z -λi ) . (6.15) Note now importantly that λi > c 1 c 2 s is equivalent to -c 2 c 1 λi < -1 s which is also ϕ p ( λi )/ψ p ( λi ) < ϕ p (κ i )/ψ p (κ i ); then, as ϕ p /ψ p is increasing, λi > c 1
c 2 s is equivalent to λi < κ i . On the opposite, for λi < c 1 c 2 s , we find λi > κ i . As such, to evaluate the contour integral in this setting, one must isolate two sets of singularities (see Figure 6.7): (i) those for which κ i > λi (which are all the largest indices i for which λi > c 1 c 2 s ) and (ii) those for which κ i < λi . This affects the relative position of the branch cut with respect to λi and therefore demands different treatments. In particular, the integrals over I B i and I D i may be restricted to integrals over shorter (possibly empty) segments. Nonetheless, the calculus ultimately reveals that, since the branch cut does not affect the local behavior of the integral around λi , both cases entail the same result. In particular, in case (i) where λi > κ i , recalling (6.10), one only has to evaluate

κ i -ε ζ i +ε ϕ p (x) ϕ p (x) - ψ p (x) ψ p (x) ψ p (x) c 2 dx = κ i -ε ζ i +ε 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 1 x -λj + 1 -c 2 c 2 1 x + c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 1 x -η j dx = 1 p p j=1 log κ i -λj ζ i -λj + c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 log κ i -η j κ i -λj -log ζ i -η j ζ i -λj + 1 -c 2 c 2 log κ i ζ i + o(ε).
In case (ii), subdividing the integral as λi -ε ζ i +ε + κ i -ε λi +ε brings immediate simplification of the additional terms in λi and thus the result remains the same.

The integral over I C i is slightly more delicate to handle. In case (i), in the limit of small ε,

1 + s ϕ p ψ p ( λi + εe iθ ) = 1 -s c 1 c 2 λi + εs c 1 c 2 p c 1 + c 2 -c 1 c 2 c 1 c 2 -1 e iθ + o(ε)
the angle of which is 0 + o(ε) uniformly on θ ∈ (-π, π] (since 1 -s c 1 c 2 λi > 0). As such, for all small ε, the sum of the integrals over (-π, 0) and (0, π] reduces to the integral over (-π, π], leading up to a mere residue calculus, and

1 2πı I C i log 1 + s ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz = log 1 -s c 1 c 2 λi c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p + o(ε).
In case (ii), 1 -s c 1 c 2 λi < 0 and thus the angle of 1 + s ϕp ψp ( λi + εe iθ ) is close to π; for θ ∈ (0, π), this leads to an argument equal to π + o(ε) < π and for θ ∈ (-π, 0) to an argument equal to -π + o(ε) > -π. All calculus made, we then find that in either case (i) or (ii)

1 2πı I C i log 1 + s ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz = log 1 -s c 1 c 2 λi c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p + o(ε).
As in the case of f (t) = log(t), the integral over I A i is of order o(ε) and vanishes. As a consequence, summing over i ∈ {1, . . . , p}, we find that

1 2πı Γ = - 1 p p i,j=1 log κ i -λj ζ i -λj - 1 -c 2 c 2 p i=1 log κ i ζ i + c 1 + c 2 -c 1 c 2 c 1 c 2 p i,j=1 log ζ i -η j ζ i -λj -log κ i -η j κ i -λj - c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p p i=1 log 1 -s c 1 c 2 λj + o(ε).
Before reaching the final result, note that, from (6.14),

p i=1 1 p p j=1 log |κ i -λj | |ζ i -λj | = 1 p p j=1 log 1 + s ϕ p ( λj ) ψ p ( λj ) 1 λj -κ 0 1 (1 -c 1 )s = 1 p p j=1 log 1 - c 1 c 2 s λi - 1 p p j=1 log( λj -κ 0 ) -log((1 -c 1 )s)
and similarly

p i,j=1 log ζ i -η j ζ i -λj = p i=1 log ϕ p (ζ i ) (1 -c 1 )ζ i = p log c 1 + c 2 -c 1 c 2 c 2 (1 -c 1 ) p i,j=1 log κ i -η j κ i -λj = p i=1 log ϕ p (κ i ) (1 -c 1 )κ i = p i=1 log c 1 + c 2 -c 1 c 2 c 2 (1 -c 1 ) 1 1 -c 1 c 2 s κ i p i=1 log κ i ζ i = log 1 + s ϕp ψp (0) -(1 -c 1 )sκ 0 = -log (-(1 -c 1 )sκ 0 ) .
Using now (6.15), we find that

p i=1 log 1 -s c 1 c 2 λi 1 -s c 1 c 2 κ i = p i=1 log c 2 c 1 s -λi c 2 c 1 s -κ i = log   ψ p c 2 c 1 s + sϕ p c 2 c 1 s s(1 -c 1 ) c 2 c 1 s -κ 0   = log   c 1 + c 2 -c 1 c 2 sc 1 (1 -c 1 ) c 2 c 1 s -κ 0   .
Combining the previous results and remarks then leads to

1 2πı Γ = c 1 + c 2 -c 1 c 2 c 1 c 2 log c 1 + c 2 -c 1 c 2 (1 -c 1 )(c 2 -sc 1 κ 0 ) + 1 -c 2 c 2 log (-sκ 0 (1 -c 1 )) + log((1 -c 1 )s) + 1 p p i=1 log( λi -κ 0 ) = c 1 + c 2 -c 1 c 2 c 1 c 2 log c 1 + c 2 -c 1 c 2 (1 -c 1 )(c 2 -sc 1 κ 0 ) + 1 c 2 log (-sκ 0 (1 -c 1 )) + 1 p p i=1 log 1 - λi κ 0 .
This concludes the proof for the case c 1 > 0. In the limit where c 1 → 0, it suffices to use the Taylor expansion of the leftmost logarithm in the small c 1 limit (i.e., log(c 2 (1c

1 ) + c 1 ) ∼ log(c 2 (1 -c 1 )) + c 1 /(c 2 (1 -c 1 )) and log(c 2 (1 -c 1 ) -sc 1 κ 0 (1 -c 1 )) ∼ log(c 2 (1 -c 1 )) -sc 1 κ 0 /c 2 ). Development for f (t) = log 2 (t)
The function f (t) = log 2 (t) is at the core of the Fisher distance and is thus of prime importance in many applications. The evaluation of the complex integral in Theorem 7 for this case is however quite technical and calls for the important introduction of the dilogarithm function. We proceed with this introduction first and foremost.

The dilogarithm function The (real) dilogarithm is defined as the function Li

2 (x) = - x 0 log(1 -u) u du. for x ∈ (-∞, 1].
The dilogarithm function will intervene in many instances of the evaluation of the contour integral of Theorem 7, through the subsequently defined function F (X, Y ; a). This function assumes different formulations depending on the relative position of X, Y, a on the real axis.

Lemma 2 (Dilogarithm integrals). We have the following results and definition

(X, Y ≥ a > 0) X Y log(x -a) x dx ≡ F (X, Y ; a) = Li 2 a X -Li 2 a Y + 1 2 log 2 (X) -log 2 (Y ) (X, Y > 0 > a) X Y log(x -a) x dx ≡ F (X, Y ; a) = -Li 2 X a + Li 2 Y a + log X Y log(-a) (a > X, Y, 0 & XY > 0) X Y log(a -x) x dx ≡ F (-X, -Y ; -a) = -Li 2 X a + Li 2 Y a + log X Y log(a) (X, Y > 0) X Y log(x) x dx ≡ F (X, Y ; 0) = 1 2 log 2 (X) - 1 2 log 2 (Y ).
Lemma 3 (Properties of Dilogarithm functions [START_REF] Zagier | The dilogarithm function[END_REF], Section I-2)). The following relations hold

(x < 0) Li 2 1 x + Li 2 (x) = - 1 2 log 2 (-x) - π 2 6 (0 < x < 1) Li 2 (1 -x) + Li 2 (x) = -log(x) log(1 -x) + π 2 6 (0 < x < 1) Li 2 (1 -x) + Li 2 1 - 1 x = - 1 2 log 2 (x).
Besides, for x < 1 and ε > 0 small,

Li 2 (x + ε) = Li 2 (x) -ε log(1 -x) x + ε 2 (1 -x) log(1 -x) + x 2(1 -x)x 2 + O(ε 3 ).
Integral evaluation As in the case where f (t) = log(t), we shall evaluate the complex integral based on the contour displayed in Figure 6.6. The main difficulty here arises in evaluating the real integrals over the segments I B i and I D i . Again, we start from the Equation (6.10). In particular, the integral over

I B i reads 1 2πı I B i log 2 ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz = 2 λi -ε ζ i +ε log - ϕ p (x) ψ p (x) ϕ p (x) ϕ p (x) - ψ p (x) ψ p (x) ψ p (x) c 2 dx = 2 λi -ε ζ i +ε log(1 -c 1 ) + log(x) + l<i log(x -η l ) + l>i log(η l -x) + log(η i -x) - l≤i log(x -ζ l ) - l>i log(ζ l -x)   ×   1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 1 x -λj + 1 -c 2 c 2 1 x + c 1 + c 2 -c 1 c 2 c 1 c 2 p j=1 1 x -η j   dx.
Note that above we have specifically chosen to write the logarithms in such a way that every integral is a well-defined real integral.

Using now the fact that

X Y log(x -a) x -b dx = F (X -b, Y -b; a -b), X Y log(a -x) x -b dx = F (b -X, b -Y ; b -a)
that we apply repetitively (and very carefully) to the previous equality, we find that the sum of the integral of I B i and

I D i gives 1 2πı I B i + I D i log 2 ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz = 2 λi -ε ζ i +ε + η i -ε λi +ε log - ϕ p (x) ψ p (x) ϕ p (x) ϕ p (x) - ψ p (x) ψ p (x) ψ p (x) c 2 dx = 2 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2   log(1 -c 1 ) j =i log η i -λj ζ i -λj +F (-ε, ζ i -λi + ε; -λi ) + F (η i -λi -ε, ε; -λi ) + j =i F (η i -λj , ζ i -λj ; -λj ) + l<i j =(i,l) F (η i -λj , ζ i -λj ; η l -λj ) -F (η i -λj , ζ i -λj ; ζ l -λj ) + l<i F (η i -λl , ζ i -λl ; η l -λl ) -F (η i -λl , ζ i -λl ; ζ l -λl ) + l<i F (-ε, ζ i -λi + ε; η l -λi ) + F (η i -λi -ε, ε; η l -λi ) - l<i F (-ε, ζ i -λi + ε; ζ l -λi ) + F (η i -λi -ε, ε; ζ l -λi ) + l>i j =(i,l) F (-η i + λj , -ζ i + λj ; -η l + λj ) -F (-η i + λj , -ζ i + λj ; -ζ l + λj ) + l>i F (-η i + λl , -ζ i + λl ; -η l + λl ) -F (-η i + λl , -ζ i + λl ; -ζ l + λl ) + l>i F (ε, -ζ i + λi -ε; -η l + λi ) + F (-η i + λi + ε, -ε; -η l + λi ) - l>i F (ε, -ζ i + λi -ε; -ζ l + λi ) + F (-η i + λi + ε, -ε; -ζ l + λi ) + j =i F (-η i + λj , -ζ i + λj ; -η i + λj ) -F (η i -λj , ζ i -λj ; ζ i -λj ) +F (ε, λi -ζ i -ε; λi -η i ) + F ( λi -η i + ε, -ε; λi -η i ) -F (-ε, ζ i -λi + ε; ζ i -λi ) -F (η i -λi -ε, ε; ζ i -λi ) + 2 1 -c 2 c 2 log(1 -c 1 ) log η i ζ i + F (η i , ζ i ; 0) + F (-η i , -ζ i ; -η i ) -F (η i , ζ i ; ζ i ) + l<i F (η i , ζ i ; η l ) -F (η i , ζ i ; ζ l ) + l>i F (-η i , -ζ i ; -η l ) -F (-η i , -ζ i ; -ζ l ) + 2 c 1 + c 2 -c 1 c 2 c 1 c 2   log(1 -c 1 ) j =i log η i -η j ζ i -η j + j =i F (η i -η j , ζ i -η j ; -η j ) + j =(i,l) l<i F (η i -η j , ζ i -η j ; η l -η j ) + j =(i,l) l>i F (-η i + η j , -ζ i + η j ; -η l + η j ) + j =i F (-η i + η j , -ζ i + η j ; -η i + η j ) - j =(i,l) l<i F (η i -η j , ζ i -η j ; ζ l -η j ) - j =(i,l) l>i F (-η i + η j , -ζ i + η j ; -ζ l + η j ) - j =i F (η i -η j , ζ i -η j ; ζ i -η j ) + log(1 -c 1 ) log ε η i -ζ i + l<i F (-ε, ζ i -η i + ε; η l -η i ) -F (-ε, ζ i -η i + ε; ζ l -η i ) + l>i F (ε, η i -ζ i -ε; η i -η l ) -F (ε, η i -ζ i -ε; η i -ζ l ) +F (ε, -ζ i + η i -ε, ε) -F (-ε, ζ i -η i + ε, ζ i -η i ) + F (-ε, ζ i -η i + ε, -η i ) + l<i F (η i -η l , ζ i -η l ; 0) + l>i F (-η i + η l , -ζ i + η l ; 0) - l<i F (η i -η l , ζ i -η l ; ζ l -η l ) - l>i F (-η i + η l , -ζ i + η l ; -ζ l + η l ) + 2 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 log(1 -c 1 ) log η i -λi λi -ζ i + o ε (1).
To retrieve the expression above, particular care was taken on the relative positions of the λi,j,l , η i,j,l and ζ i,j,l to obtain the proper form of the F function; besides, to avoid further complications, a small ε approximation was used whenever the F function has a finite limit when ε → 0 (hence the trailing o ε (1) in the formula).

To go further, we now make use of the following additional identities obtained from Lemma 3 (these are easily proved).

Lemma 4 (Properties of the function F ). We have the following properties of the function F :

(X ≥ Y > 0) F (-X, -Y ; -X) -F (X, Y ; Y ) = 1 2 log 2 X Y (Y ≥ X > 0) F (X, Y ; X) -F (-X, -Y ; -Y ) = 1 2 log 2 X Y (X, Y > 0) F (-X + ε, -ε; -X) + F (ε, Y -ε; -X) -F (X -ε, ε; -Y ) -F (-ε, -Y + ε; -Y ) = - π 2 2 + 1 2 log 2 X Y + o ε (1) (T, Z ≥ X, Y > 0) F (-ε, -Y + ε; -T ) + F (X -ε, ε; -T ) -F (-ε, -Y + ε; -Z) -F (X -ε, ε; -Z) + F (T, Z; -X) -F (T, Z; Y ) = log X Y log T Z + o ε (1) (T, Z ≥ X, Y > 0) F (ε, -Y -ε; -T ) + F (-X + ε, -ε; -T ) -F (ε, X -ε; -Z) -F (-Y + ε, -ε; -Z) + F (T, Z; X) -F (T, Z; -Y ) = log X Y log T Z + o ε (1) (X, Y, Z, T > 0) F (X, Y ; T ) + F (T, Z; X) -F (X, Y ; Z) -F (T, Z; Y ) = log X Y log T Z (XY > 0 & ZT > 0) F (X, Y ; -Z) + F (Z, T ; -X) -F (X, Y ; -T ) -F (Z, T ; -Y ) = log X Y log T Z (Y, Z > 0 & εX > 0) F (ε, X; -Y ) + F (ε, X; -Z) -F (Y, Z; -X) = log ε X log Y Z + 1 2 log 2 (Z) - 1 2 log 2 (Y ).
Exploiting the relations from the previous lemma, we have the following first result:

p i=1 F (-ε, ζ i -λi + ε; -λi ) + F (η i -λi -ε, ε; -λi ) = p i=1 Li 2 1 - ζ i λi -Li 2 1 - η i λi + log( λi ) log η i -λi λi -ζ i
The terms involving double or triple sums (over i, l or i, j, l) are more subtle to handle. By observing that i l>i G il = l i<l G il which, up to a switch in the notation (i, l) into (l, i), is the same as i l<i G li , we have that

i l>i G il + i l<i G il = i l>i G il + G li .
Using this observation to gather terms together, we find notably from Lemma 4 that i l<i j =(i,l)

F (η i -λj , ζ i -λj ; η l -λj ) -F (η i -λj , ζ i -λj ; ζ l -λj ) + i l>i j =(i,l) F (-η i + λj , -ζ i + λj ; -η l + λj ) -F (-η i + λj , -ζ i + λj ; -ζ l + λj ) = i l<i j =(i,l) log λj -η i λj -ζ i log λj -η l λj -ζ l = 1 2 i l =i j =(i,l) log λj -η i λj -ζ i log λj -η l λj -ζ l Similarly, i l<i F (-ε, ζ i -λi + ε; η l -λi ) + F (η i -λi -ε, ε; η l -λi ) - i l<i F (-ε, ζ i -λi + ε; ζ l -λi ) + F (η i -λi -ε, ε; ζ l -λi ) + i l>i F (-η i + λl , -ζ i + λl ; -η l + λl ) -F (-η i + λl , -ζ i + λl ; -ζ l + λl ) = i l<i F (-ε, ζ i -λi + ε; η l -λi ) + F (η i -λi -ε, ε; η l -λi ) - i l<i F (-ε, ζ i -λi + ε; ζ l -λi ) + F (η i -λi -ε, ε; ζ l -λi ) + i l<i F (-η l + λi , -ζ l + λi ; -η i + λi ) -F (-η l + λi , -ζ l + λi ; -ζ i + λi ) = i l<i log η i -λi ζ i -λi log λi -η l λi -ζ l + o ε (1)
and, symmetrically,

i l>i F (ε, -ζ i + λi -ε; -η l + λi ) + F (-η i + λi + ε, -ε; -η l + λi ) - i l>i F (ε, -ζ i + λi -ε; -ζ l + λi ) + F (-η i + λi + ε, -ε; -ζ l + λi ) + i l<i F (η i -λl , ζ i -λl ; η l -λl ) -F (η i -λl , ζ i -λl ; ζ l -λl ) = i l>i F (ε, -ζ i + λi -ε; -η l + λi ) + F (-η i + λi + ε, -ε; -η l + λi ) - i l>i F (ε, -ζ i + λi -ε; -ζ l + λi ) + F (-η i + λi + ε, -ε; -ζ l + λi ) + i l>i F (η l -λi , ζ l -λi ; η i -λi ) -F (η l -λi , ζ l -λi ; ζ i -λi ) = i l>i log η i -λi ζ i -λi log λi -η l λi -ζ l + o ε (1).
Also, using Items 1 and 2 of Lemma 4, we find

i j =i F (-η i + λj , -ζ i + λj ; -η i + λj ) -F (η i -λj , ζ i -λj ; ζ i -λj ) = i j =i 1 2 log 2 λj -η i λj -ζ i .
Again from Lemma 4, we also have

F (ε, λi -ζ i -ε; λi -η i ) + F ( λi -η i + ε, -ε; λi -η i ) -F (-ε, ζ i -λi + ε; ζ i -λi ) -F (η i -λi -ε, ε; ζ i -λi ) = - π 2 2 + 1 2 log 2 η i -λi λi -ζ i .
Before going further, remark that the last four established relations can be assembled to reach

j =i F (η i -λj , ζ i -λj ; -λj ) + l<i j =(i,l) F (η i -λj , ζ i -λj ; η l -λj ) -F (η i -λj , ζ i -λj ; ζ l -λj ) + l<i F (η i -λl , ζ i -λl ; η l -λl ) -F (η i -λl , ζ i -λl ; ζ l -λl ) + l<i F (-ε, ζ i -λi + ε; η l -λi ) + F (η i -λi -ε, ε; η l -λi ) - l<i F (-ε, ζ i -λi + ε; ζ l -λi ) + F (η i -λi -ε, ε; ζ l -λi ) + l>i j =(i,l) F (-η i + λj , -ζ i + λj ; -η l + λj ) -F (-η i + λj , -ζ i + λj ; -ζ l + λj ) + l>i F (-η i + λl , -ζ i + λl ; -η l + λl ) -F (-η i + λl , -ζ i + λl ; -ζ l + λl ) + l>i F (ε, -ζ i + λi -ε; -η l + λi ) + F (-η i + λi + ε, -ε; -η l + λi ) - l>i F (ε, -ζ i + λi -ε; -ζ l + λi ) + F (-η i + λi + ε, -ε; -ζ l + λi ) + j =i F (-η i + λj , -ζ i + λj ; -η i + λj ) -F (η i -λj , ζ i -λj ; ζ i -λj ) +F (ε, λi -ζ i -ε; λi -η i ) + F ( λi -η i + ε, -ε; λi -η i ) -F (-ε, ζ i -λi + ε; ζ i -λi ) -F (η i -λi -ε, ε; ζ i -λi ) = 1 2 p log 2 c 1 c 2 (1 -c 1 ) + o ε (1).
Next, we have

F (η i , ζ i ; 0) = 1 2 log 2 (η i ) - 1 2 log 2 (ζ i )
and, again by Lemma 4,

i l =i F (η i , ζ i ; η l ) -F (η i , ζ i ; ζ l ) = i>l F (η i , ζ i ; η l ) + F (η l , ζ l ; η i ) -F (η i , ζ i ; ζ l ) -F (η l , ζ l ; ζ i ) = i>l log η i ζ i log η l ζ l = 1 2 i l =i log η i ζ i log η l ζ l i F (-η i , -ζ i ; -η i ) -F (η i , ζ i ; ζ i ) = 1 2 i log 2 η i ζ i so that i l =i F (η i , ζ i ; η l ) -F (η i , ζ i ; ζ l ) + i F (-η i , -ζ i ; -η i ) -F (η i , ζ i ; ζ i ) = 1 2 i log η i ζ i 2 .
Recall now the already established identity i log(

η i ζ i ) = -log((1 -c 1 )(1 -c 2 )) from which i l =i F (η i , ζ i ; η l ) -F (η i , ζ i ; ζ l ) + i F (-η i , -ζ i ; -η i ) -F (η i , ζ i ; ζ i ) = 1 2 log 2 ((1 -c 1 )(1 -c 2 )).
Continuing, we also have

j =i log η i -λj ζ i -λj = log - c 1 c 2 (1 -c 1 ) -log η i -λi ζ i -λi = log c 1 c 2 (1 -c 1 ) η i -λi λi -ζ i .
By Lemma 4 again, we next find

i l>i F (-η i , -ζ i ; -η l ) -F (-η i , -ζ i ; -ζ l ) + l<i F (η i , ζ i ; η l ) -F (η i , ζ i ; ζ l ) = i l<i log η i ζ i log η l ζ l from which we deduce that i F (-η i , -ζ i , -η i ) -F (η i , ζ i ; ζ i ) + i l>i F (-η i , -ζ i ; -η l ) -F (-η i , -ζ i ; -ζ l ) + l<i F (η i , ζ i ; η l ) -F (η i , ζ i ; ζ l ) = 1 2 i,l log η i ζ i log η l ζ l = 1 2 log 2 ((1 -c 1 )(1 -c 2 )) .
The next term also simplifies through the definition of ϕ p /ψ p :

i j =i log η i -η j ζ i -η j = j log c 1 c 1 + c 2 -c 1 c 2 ϕ p (η j ) (ζ j -η j ) (1 -c 1 )η j .
Still from Lemma 4 and with the same connection to ϕ p /ψ p , we have

j l<i F (η i -η j , ζ i -η j ; η l -η j ) + j l>i F (-η i + η j , -ζ i + η j ; -η l + η j ) + j =i F (-η i + η j , -ζ i + η j ; -η i + η j ) - j l<i F (η i -η j , ζ i -η j ; ζ l -η j ) - j l>i F (-η i + η j , -ζ i + η j ; -ζ l + η j ) - j =i F (η i -η j , ζ i -η j ; ζ i -η j ) = i j =(i,l) l<i log η i -η j ζ i -η j log η l -η j ζ l -η j + 1 2 log 2 η j -η i η j -ζ i = 1 2 i j =(i,l) l =i log η i -η j ζ i -η j log η l -η j ζ l -η j + 1 2 log 2 η j -η i η j -ζ i = 1 2 i l j =(i,l) log η i -η j ζ i -η j log η l -η j ζ l -η j = 1 2 j log 2 c 1 c 1 + c 2 -c 1 c 2 ϕ (η j ) (ζ j -η j ) (1 -c 1 )η j .
Again from Lemma 4, we next have

i l<i F (-ε, ζ i -η i + ε; η l -η i ) -F (-ε, ζ i -η i + ε; ζ l -η i ) - l>i F (-η i + η l , -ζ i + η l ; -ζ l + η l ) = i l<i log ε η i -ζ i log η i -η l η i -ζ l - 1 2 log 2 (η i -η l ) + 1 2 log 2 (η i -ζ l ) + o ε (1).
We also have the following relations

i l>i F (ε, η i -ζ i -ε; η i -η l ) -F (ε, η i -ζ i -ε; η i -ζ l ) - l<i F (η i -η l , ζ i -η l ; ζ l -η l ) = i l>i 1 2 log 2 (ζ l -η i ) - 1 2 log 2 (η l -η i ) + log η l -η i ζ l -η i log ε η i -ζ i + o ε (1) and l<i F (-ε, ζ i -η i + ε; η l -η i ) -F (-ε, ζ i -η i + ε; ζ l -η i ) - l>i F (-η i + η l , -ζ i + η l ; -ζ l + η l ) = i l<i log ε η i -ζ i log η i -η l η i -ζ l - 1 2 log 2 (η i -η l ) + 1 2 log 2 (η i -ζ l ) + o ε (1) which together gives i l>i F (ε, η i -ζ i -ε; η i -η l ) -F (ε, η i -ζ i -ε; η i -ζ l ) - l<i F (η i -η l , ζ i -η l ; ζ l -η l ) + l<i F (-ε, ζ i -η i + ε; η l -η i ) -F (-ε, ζ i -η i + ε; ζ l -η i ) - l>i F (-η i + η l , -ζ i + η l ; -ζ l + η l ) = i l =i log ε η i -ζ i log η i -η l η i -ζ l .
The next term is

F (ε, -ζ i + η i -ε, ε) -F (-ε, ζ i -η i + ε, ζ i -η i ) = 1 2 log 2 ε η i -ζ i and finally the last term gives l<i F (η i -η l , ζ i -η l ; 0) + l>i F (-η i + η l , -ζ i + η l ; 0) = 1 2 i l =i log 2 |η i -η l | -log 2 |ζ i -η l | .
Putting all results above together, we obtain

p i=1 2 λi -ε ζ i +ε + η i -ε λi +ε log - ϕ p (x) ψ p (x) ϕ p (x) ϕ p (x) - ψ p (x) ψ p (x) ψ p (x) c 2 dx = 2 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p log(1 -c 1 ) log c 1 c 2 (1 -c 1 ) + i Li 2 1 - ζ i λi -Li 2 1 - η i λi + log( λi ) log η i -λi λi -ζ i + i j =i F (η i -λj , ζ i -λj ; -λj ) + 1 2 p log 2 c 1 c 2 (1 -c 1 ) - π 2 2 p   + 2 1 -c 2 c 2 -log(1 -c 1 ) log ((1 -c 1 )(1 -c 2 )) + i 1 2 log 2 (η i ) - 1 2 log 2 (ζ i ) + 1 2 log 2 ((1 -c 1 )(1 -c 2 )) + 2 c 1 + c 2 -c 1 c 2 c 1 c 2   log(1 -c 1 ) i j =i log η i -η j ζ i -η j + i j =i F (η i -η j , ζ i -η j ; -η j ) + 1 2 i l j =(i,l) log η i -η j ζ i -η j log η l -η j ζ l -η j + i log(1 -c 1 ) log ε η i -ζ i + i l =i log η l -η i ζ l -η i log ε η i -ζ i + i Li 2 1 - ζ i η i + log ε η i -ζ i log(η i ) + i 1 2 log 2 ε η i -ζ i -p π 2 6 + o ε (1).
The integral over the contour I E i can be computed using the same reasoning as for the f (t) = log(t) function and is easily obtained as

1 2πı I E i = c 1 + c 2 -c 1 c 2 c 1 c 2 -log 2 ϕ p (η i ) c 1 c 1 + c 2 -c 1 c 2 -log 2 (ε) -2 log(ε) ϕ ψ (η i ) - π 2 3 + O(ε log 2 (ε)).
Adding up the "residue" at λi (i.e., the integral over I C i ), we end up with the following expression for the sought-for integral

1 2πı Γ log 2 ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz = -2 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p log(1 -c 1 ) log c 1 c 2 (1 -c 1 ) + i Li 2 1 - ζ i λi -Li 2 1 - η i λi + log( λi ) log η i -λi λi -ζ i + i j =i F (η i -λj , ζ i -λj ; -λj ) + p log 2 c 1 c 2 (1 -c 1 ) - π 2 2 p   -2 1 -c 2 c 2 -log(1 -c 1 ) log ((1 -c 1 )(1 -c 2 )) + i 1 2 log 2 (η i ) - 1 2 log 2 (ζ i ) + 1 2 log 2 ((1 -c 1 )(1 -c 2 )) -2 c 1 + c 2 -c 1 c 2 c 1 c 2   log(1 -c 1 ) j log c 1 c 1 + c 2 -c 1 c 2 ϕ (η j ) (ζ j -η j ) (1 -c 1 )η j + i j =i F (η i -η j , ζ i -η j ; -η j ) 1 2 j log 2 c 1 c 1 + c 2 -c 1 c 2 ϕ (η j ) (ζ j -η j ) (1 -c 1 )η j + i Li 2 1 - ζ i η i + i log ϕ ψ (η i ) log ε η i -ζ i + log(η i -ζ i ) log ε η i -ζ i + i 1 2 log 2 ε η i -ζ i - π 2 6 + o ε (1) + i c 1 + c 2 -c 1 c 2 c 1 c 2 -log 2 ϕ p (η i ) c 1 c 1 + c 2 -c 1 c 2 -log 2 (ε) -2 log(ε) log ϕ ψ (η i ) - π 2 3 + i log 2 c 1 c 2 λi -π 2 c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p + o ε (1).
which, in the limit of small ε, can be simplified as

1 2πı Γ log 2 ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz = -2 1 p - c 1 + c 2 -c 1 c 2 c 1 c 2 p log(1 -c 1 ) log c 1 c 2 (1 -c 1 ) + i Li 2 1 - ζ i λi -Li 2 1 - η i λi + log( λi ) log η i -λi λi -ζ i + i j =i F (η i -λj , ζ i -λj ; -λj ) + p log 2 c 1 c 2 (1 -c 1 )   -2 1 -c 2 c 2 -log(1 -c 1 ) log ((1 -c 1 )(1 -c 2 )) + i 1 2 log 2 (η i ) - 1 2 log 2 (ζ i ) + 1 2 log 2 ((1 -c 1 )(1 -c 2 )) -2 c 1 + c 2 -c 1 c 2 c 1 c 2   log(1 -c 1 ) j log c 1 c 1 + c 2 -c 1 c 2 ϕ (η j ) (ζ j -η j ) (1 -c 1 )η j + i j =i F (η i -η j , ζ i -η j ; -η j ) - i log ϕ ψ (η i ) log (η i -ζ i ) + i Li 2 1 - ζ i η i - 1 2 log 2 (η i -ζ i ) - i c 1 + c 2 -c 1 c 2 c 1 c 2 -log 2 η i -ζ i (1 -c 1 )η i +2 log η i -ζ i (1 -c 1 )η i log c 1 c 1 + c 2 -c 1 c 2 ϕ (η i ) (ζ i -η i ) (1 -c 1 )η i - i log 2 c 1 c 2 λi c 1 + c 2 -c 1 c 2 c 1 c 2 - 1 p .
After further book-keeping and simplifications, we ultimately find:

1 2πı Γ log 2 ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) - ψ p (z) ψ p (z) ψ p (z) c 2 dz (6.16) = c 1 + c 2 -c 1 c 2 c 1 c 2 p i=1 log 2 ((1 -c 1 )η i ) -log 2 (1 -c 1 ) λi +2 1≤i,j≤p Li 2 1 - ζ i λj -Li 2 1 - η i λj + Li 2 1 - η i η j -Li 2 1 - ζ i η j   - 1 -c 2 c 2 log 2 (1 -c 2 ) -log 2 (1 -c 1 ) + p i=1 log 2 (η i ) -log 2 (ζ i ) - 1 p   2 1≤i,j≤p Li 2 1 - ζ i λj -Li 2 1 - η i λj - p i=1 log 2 (1 -c 1 ) λi  
which provides an exact, yet rather impractical (the expression involves the evaluation of O(p 2 ) dilogarithm terms which may be computationally intense for large p), final expression for the integral.

At this point, it is also not easy to fathom why the retrieved expression would remain of order O(1) with respect to p. In order to both simplify the expression and retrieve a visually clear O(1) estimate, we next proceed to a large p Taylor expansion of the above result. In particular, using the last item in Lemma 3, we perform a (second-order) Taylor expansion of all terms of the type Li 2 (1 -X) above in the vicinity of λi / λj . This results in the following two relations

i,j Li 2 1 - ζ i λj -Li 2 1 - η i λj + Li 2 1 - η i η j -Li 2 1 - ζ i η j = (∆ η ζ ) T M (∆ η λ) + o p (1) 1 p i,j Li 2 1 - ζ i λj -Li 2 1 - η i λj = - 1 p (∆ η ζ ) T N 1 p + o p (1)
with ∆ b a , M and N defined in the statement of Corollary 5.

With these developments, we deduce the final approximation 1 2πı Γ log 2 -ϕ p (z) ψ p (z) ϕ p (z) ϕ p (z) -ψ p (z) ψ p (z)

ψ p (z) c 2 dz = 2 c 1 + c 2 -c 1 c 2 c 1 c 2   ∆ η ζ T M ∆ η λ + i log (1 -c 1 ) λi λi η i -λi   - 2 p ∆ η ζ T N 1 p + 1 p i log 2 (1 -c 1 ) λi -2 1 -c 2 c 2 1 2 log 2 (1 -c 2 ) - 1 2 log 2 (1 -c 1 ) + i (η i -ζ i ) log( λi ) λi + o p (1).
For symmetry, it is convenient to finally observe that log(1 -c 1 ) i (η i -ζ i )/ λi ∼ log(1 -c 1 ) i log(η i /ζ i ) = -log 2 (1 -c 1 ); replacing in the last parenthesis provides the result of Corollary 5 for c 1 > 0.

To determine the limit as c 1 → 0, it suffices to remark that in this limit η i = λi + c 1 p λi + o(c 1 ) (this can be established using the functional relation ϕ p (η i ) = 0 in the small c 1 limit). Thus it suffices to replace in the above expression the vector η -ζ by the vector η -λ, the vector c 1 +c 2 -c 1 c 2 c 1 c 2 (η -λ) by the vector 1 p λ, and taking c 1 = 0 in all other instances (where the limits for c 1 → 0 are well defined).

Gradient calculus

The main concern is to find an analytical expression of the gradient defined as: ∇h X (M ) = -D(M, X) πıp Γ g -m μp (z; M ) sym Σ(M -1 Σ -zI p ) -2 dz.

(6.17) Equation (6.17) can be written as:

∇h X (M ) = -D(M, X) πıp sym ΣU Γ g -m μp (z; M ) (Λ -zI p ) -2 dz U T . (6.18)

where M -1 Σ = U ΛU T in its spectral decomposition. Our main focus is on the diagonal matrix A≡ 1 2ıπ Γ g -m μp (z; M ) (Λ -zI p ) -2 dz and particularly on its k-th diagonal element

A kk = 1 2ıπ
Γ g -m μp (z; M ) ( λk -z) 2 dz (6.19) with Λ = diag( λ1 , . . . , λp ).

To solve (6.19), we use elementary properties of the rational function m μp (z; M ) that we recall is defined as

m μp (z; M ) = c p p i=1 1 λi -z + 1 -c z .
Remarking that the poles of m μp (z; M ) are { λi } p i=1 and 0, and that {ξ i } p i=1 are the zeros of m μp (z; M ) (see Section 6.2.3 of the appendix for details), we have :

m μp (z; M ) = p i=1 (z -ξ i ) z p i=1 z -λi .
With these ingredients, we can evaluate A kk for various functions f (recall that g(z) = f (1/z)).

Case f (t) = t For this case, g(t) = 1 t , and thus the integrand of A kk is

I = z p i=1 i =k z -λi (z -λk ) p i=1 (z -ξ i )
.

Under this rational form, A kk is easy to evaluate since it only requires to evaluate the residue for each pole of I:

• the first-order pole λk for which the residue R 1 is given by • and the first-order poles ξ j , j ∈ {1, . . . , p} for which the residue R 2 is given by: Case f (t) = log(t) For this case, g(t) = -log(t) and therefore the integrand of A kk becomes

I = - log p i=1 (z-ξ i ) z p i=1 (z-λi) ( λk -z) 2
Elementary functional analysis allows us to find the discontinuities of this multi-valued function (the z's for which the argument of the logarithm function is negative). This set of points, or branch cuts, are exactly the segments [ξ i , λi ], i = 1, . . . , p. These segments lie inside the integration contour Γ, that needs be modified for proper integration; the new contour, denoted Γ n is depicted in Figure 6.8.

The complex integral over the contour Γ n , is the sum of several integrals, subdivided in four types:

• integrals over the circles surrounding {ξ j } p j=1 which, thanks to the variable change z = ξ j + e ıθ , reduce to inside the integration contour (see Section 6.2.3 of the appendix for details). Therefore, the integrand is written similarly as for the log function as:

I = - log p i=0 (z-κ i ) z p i=1 (z-λi) ( λk -z) 2 .
The integration contour can be deformed as for the log function. Using similar integration techniques, the calculus then yields to the solution derived in Section 3.5.3.

Case f (t) = log 2 (t) Here g(t) = f (t) = log 2 (t) and the integrand for this case is simply

I = - log 2 p i=1 (z-ξ i ) z p i=1 (z-λi) ( λk -z) 2
.

Again, we use here exactly the same line of work performed on the log(t) and log(1 + st) functions. Technical difficulties however arise when addressing the real integrals which involve products of logarithms and rational functions. These difficulties are mostly cumbersome calculus which are addressed similar to 6.2.3.

6.3 Appendix for Chapter 4

Solution of MTL LS-SVM

The Lagrangian of the constrained optimization problem using the relatedness assumption (W i = W 0 + V i ) reads:

L(ω 0 , v i , ξ i , α i , b i ) = 1 2λ tr W T 0 W 0 + 1 2 k i=1 tr V T i V i γ i + 1 2 k i=1 tr ξ T i ξ i + k i=1 tr α T i Y i - XT i W 0 √ kp - XT i V i √ kp -1 n i b T i -ξ i
with α i ∈ R n i ×m the Lagrangian parameter attached to task i.

Differentiating with respect to the unknowns W 0 , V i , ξ i , α i , and b i leads to the following system of equations: (6.24) Plugging the expression of W 0 (Equation (6.20)), V i (Equation (6.21)) and ξ i (Equation (6.22)) into Equation (6.23) leads to:

1 λ W 0 - k i=1 Xi √ kp α i = 0 (6.20) 1 γ i V i - Xi √ kp α i = 0 (6.21) ξ i -α i = 0 (6.22) Y i - XT i W 0 √ kp - XT i V i √ kp -1 n i b T i -ξ i = 0 (6.23) α T i 1 n i = 0.
Y i = (λ + γ i ) XT i Xi kp α i + λ j =i XT i Xj kp α j + 1 n i b T i + α i 1 T n i α i = 0. With Y = [Y T 1 , . . . , Y T k ] T ∈ R n , α = [α T 1 , . . . , α T k ] T ∈ R n , Z = k i=1 e [k]
i e

[k] i T ⊗ Xi ∈ R kp×n and P ∈ R n×k such that the j-th column is P .j = [0 T n 1 +...+n j-1 , 1 T n j , 0 T n j+1 +...+n k ] T , this system of equations can be written under the following compact matrix form: Moreover, using W i = W 0 + V i and Equations (6.20) and (6.21), the expression of W i becomes:

P b + Q -1 α = Y P T α = 0 k with Q = Z T AZ kp + I n -1 ∈ R n×n
W i = e [k] i T ⊗ I p A Z √ kp α.

Calculus of deterministic equivalents

Lemma 5 (Deterministic equivalents). Define, for class C j in Task i, the data deterministic matrices

M = e [k]
1 ⊗ [µ 11 , . . . , µ 1m ], . . . , e

[k]

k ⊗ [µ k1 , . . . , µ km ]

C ij = A 1 2 e [k] i e [k] i T ⊗ (Σ ij + µ ij µ T ij ) A 1 2 .
Then we have the deterministic equivalents of first-order

Q ↔ Q ≡   k i=1 m j=1 δ [mk] ij C ij + I kp   = - 1 kp i E u T QA 1 2 S ij A 1 2 Q-i A 1 2 z i z T i A 1 2 Qv 1 + δ i j + E u T QA 1 2 S ij A 1 2 Q-i C δ Qv - 1 kp E u T QA 1 2 S ij A 1 2 Q-i A 1 2 z i z T i A 1 2 QC δ Qv + O log p p where C δ = ij c ij c 0 C ij 1+δ ij .
Using Assumption 2 and following the work of (Louart & Couillet, 2018),

1 kp E u T QA 1 2 S ij A 1 2 Q-i A 1 2 z i z T i A 1 2 QC δ Qv = O( 1 p ).
Furthermore,

E u T QA 1 2 S ij A 1 2 ( Q -Q)v = - 1 kp i E u T Q-i A 1 2 S ij A 1 2 Q-i A 1 2 z i z T i A 1 2 Qv 1 + δ i j + 1 kp i E u T Q-i A 1 2 z i z T i A 1 2 Q-i A 1 2 S ij A 1 2 Q-i A 1 2 z i z T i A 1 2 Qv kp(1 + δ i j ) 2 + E u T QA 1 2 S ij A 1 2 Q -i C δ Qv + O log p p = 1 kp i E tr C C(i ) QA 1 2 S i j A 1 2 Q (1 + δ C(i ) ) 2 E u T QC C(i ) Qv + O log p p
where -

1 kp i E[ u T Q-i A 1 2 S ij A 1 2 Q-i z i z T i Qv 1+δ i j ] + E[u T Q-i A 1 2 S ij A 1 2 QC δ Qv] = O 1
√ p , following again (Louart & Couillet, 2018).

Let us next denote d ab = n ab kp(1+δ ab ) 2 . We then have the following identity for

E[ QA 1 2 S ij A 1 2 Q]: E[ QA 1 2 S ij A 1 2 Q] = QA 1 2 S ij A 1 2 Q+ k i =1 m j =1 d i j kp E tr C i j QA 1 2 S ij A 1 2 Q QC i
S ij A 1 2 Q ↔ QA 1 2 S ij A 1 2 Q + k i =1 m j =1 d i j T (ij) i j E[ QC i j Q] (6.31)
with T = T (I k -DT) -1 .

Proof of Theorem 9

Proof of the convergence in distribution. Under a Gaussian mixture assumption for the input data X, the convergence in distribution of the statistics of the classification score g i (x) is identical to the central limit theorem derived in (Liao & Couillet, 2019, Appendix B) by writing the classification score g i (x) in polynomial form of a Gaussian vector and by resorting to the Lyapounov central limit theorem [START_REF] Billingsley | Probability and measure[END_REF].

Since conditionally on the training data X, the classification score g(x) is expressed as the projection of the deterministic vector W on the concentrated random vector x, the CLT for concentrated vector unfolds by proving that projections of deterministic vector on concentrated random vector is asymptotically gaussian. This is ensured by the following result.

Theorem 10 (CLT for concentrated vector [START_REF] Klartag | A central limit theorem for convex sets[END_REF][START_REF] Fleury | A stability result for mean width of lp-centroid bodies[END_REF]). If x is a concentrated random vector with E[x] = 0, E[xx T ] = I p with an observable diameter of order O(1) and σ be the uniform measure on the sphere S p-1 ⊂ R p of radius 1, then for any integer k, small compared to p, there exist two constants C, c and a set Θ ⊂ (S p-1 ) k such that σ ⊗ . . . ⊗ σ with G(t) the cumulative distribution function of N(0, 1)

Then the result unfolds naturally.

Statistical mean of the classification scores. Using the definition of the score in (4.2), the average output score g i (x) for x ∈ C j is

E[g i (x)] = E 1 kp e [k] i ⊗ µ ij T A 1 2 QA 1 2 Z(Y -P b) + b i .
Using Lemma 5, this can be further developed as:

E[g i (x)] = 1 kp e [k] i ⊗ µ ij T A 1 2 QA 1 2 M δ J T (Y -P b) + b i + o(1).
(6.32)

Particular Case

In the case of binary classification (m = 2) and for Σ ij = I p , we have the simplification: A ij ∆µ T i ∆µ j e

M = i,j D γ + λ1 k 1 T k 1 2 e [k] i e [k] i T ⊗ δi μij = i D γ + λ1 k 1 T k 1 2 e [k] i e [k] i T ⊗   c i2 √ c i1 c i , - c i1 √ c i2
[k]

i e

[k] j

T ⊗ c i c T j = A ⊗ 1 k 1 T k M with M ≡ i,j ∆µ T i ∆µ j E [k] ij ⊗ c i c T j c i ≡   c i2 c i c i1 c i -c i1 c i c i2 c i   A ≡ I k + D -1 2 δ D γ + λ1 k 1 T k -1 D -1 2 δ -1
.

As for the covariance terms, i e

M T Q0 V ij Q0 M = i,j
[k] i

T + D κ i δ D -1 2 δ A ⊗ 1 k 1 T k M = 1 δ [k] i AD K i +e [k] i A ⊗ 1 k 1 T k M
with K ia = δi κ ia . Using Equation (6.31), after algebraic manipulations, we finally obtain the compact form

K = c 0 k [A A] D c - c 0 k [A A] -1
. (6.33)

Proof of Propositions 6-7

One-versus-all

The probability of correct classification for Task i and for a test data x ∈ C j reads P g bin i (x; j) > max j =j {g bin i (x; j )} = P g bin i (x; j) -max j =j {g bin i (x; j )} > 0 .

Since by definition (Equation (4.2))

g bin i (x; j) = 1 kp ẙ(j) T J T QZ T A e

[k] i ⊗ x + b i , (6.34) we have that g bin i (x, j)1 m-1 -g bin i (x; j ) j =j = 1 kp Y -j J T QZ T A e

[k] k ⊗ x , where Y -j = (ẙ(j) T -ẙ(j ) T j =j ) ∈ R (m-1)×km . Using Theorem 9 with Y replaced by Y -j , g bin i (x, j)1 m-1 -g bin i (x; j ) j =j ∈ R m-1 is asymptotically a multivariate Gaussian random vector with statistics detailed in the theorem statement. Proposition 6 then unfolds trivially by remarking that g bin i (x; j) > max j =j g bin i (x; j ) ⇔ ∀j = j, g bin i (x; j)-g bin i (x; j ) ≥ 0.

One Hot encoding

The proof is similar to the one-versus-all case.

The probability of correct classification for a test data x ∈ C j is P g bin i (x; j) > max j =j {g bin i (x; j )} = P g bin i (x; j) -max j =j {g bin i (x; j )} > 0 where g bin i (x; j) = 1 kp e

[k] j

T YT J T QZ T A e

[k] i ⊗ x + b i . (6.35) Therefore g bin i (x; j)1 m-1 -{(g i (x; j ))} j =j = 1 kp E j YT J T QZ T A(e

[k]
k ⊗ x), with E j = {(e (m) j -e (m) j ) T } j =j ∈ R (m-1)×m . By Theorem 9 with Y replaced by E T j YT , this vector is asymptotically normally distributed and Proposition 7 unfolds immediately using again the fact that g bin i (x; j) > max j =j g bin i (x; j ) ⇔ ∀j = j, g bin i (x; j) -g bin i (x; j j) ≥ 0.

Synthèse de la thèse en français

La plupart des méthodes de traitement du signal et d'apprentissage automatique (par exemple, les tests statistiques, l'estimation de paramètres, la classification, la régression, etc.) sont basées sur des fonctionnelles non triviales de n vecteurs aléatoires de dimension p. Dans l'hypothèse où le nombre n de ces données disponibles est largement supérieur à leur dimension p, certains résultats théoriques peuvent être obtenus, car un comportement déterministe apparaît parfois lorsque n → ∞, ce qui simplifie le problème comme l'illustre par exemple la célèbre loi des grands nombres et le théorème central limite. Cependant, comme déjà montré dans la littérature [START_REF] Wigner | On the distribution of roots of certain symmetric matrices[END_REF][START_REF] Marcenko | Distributions of eigenvalues for some sets of random matrices[END_REF], certains résultats et intuitions basés sur des asymptotiques classiques (n → ∞ et dimension des données p fixe) s'effondrent lorsque la taille des données est comparable à la dimension des données, un problème souvent apparenté à la "malédiction de la dimensionnalité". Dans le cadre du paradigme actuel des données grandissantes, où des masses de données sont produites, échangées et stockées, nous sommes constamment confrontés à une situation où non seulement la taille mais aussi la dimension des données sont importantes. D'un autre côté, la dernière décennie a vu une augmentation considérable de la diversité des applications auxquelles l'apprentissage automatique est appliqué (apprentissage par transfert, confidentialité des données, équité, etc). Par conséquent, l'apprentissage automatique n'est plus seulement utilisé pour le placement des publicités et l'implémentation de filtres antispam : il est désormais utilisé pour filtrer les demandes de prêt [START_REF] Fernández | Artificial intelligence in financial services[END_REF], déployer les forces de police [START_REF] Rudin | Predictive policing using machine learning to detect patterns of crime[END_REF], prendre des décisions dans le domaine de la justice pénale [START_REF] Berk | Machine learning forecasts of risk to inform sentencing decisions[END_REF], etc., faisant ainsi rapidement son entrée dans les systèmes socio-techniques. Les biais algorithmiques et les algorithmes mal compris sont l'un des plus grands risques d'échec car ils compromettent l'objectif même de l'apprentissage automatique depuis que l'opinion publique s'inquiète de l'impact de la technologie sur la société.

Il est donc important de considérer des asymptotiques où le nombre de données n et leur dimension p sont du même ordre de grandeur tout en fournissant un ensemble d'outils adaptés dans ce régime pour analyser, comprendre et améliorer les problèmes de l'apprentissage automatique et de traitement du signal. Cette réflexion a permis l'émergence de nouvelles théories généralement encapsulées sous la dénomination de "statistiques en grande dimension" dont fait partie la théorie des matrices aléatoires. Un ensemble d'outils ont été développés pour analyser les objets statistiques communément rencontrés dans le domaine des télécommunications (Couillet & Debbah, 2011), de la finance [START_REF] Bouchaud | Financial applications of random matrix theory: a short review[END_REF], de la physique [START_REF] Guhr | Random-matrix theories in quantum physics: common concepts[END_REF] pour ne citer que quelques-uns. Seulement récemment, des études utilisant la théorie des matrices aléatoires [START_REF] Hafiz Tiomoko | Nouvelles méthodes pour l'apprentissage non-supervisé en grandes dimensions[END_REF][START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF]Liao & Couillet, 2019;[START_REF] Seddik | A random matrix analysis of learning with α-dropout[END_REF][START_REF] Deng | A model of double descent for high-dimensional binary linear classification[END_REF][START_REF] Elkhalil | Random Matrix Theory: Selected Applications from Statistical Signal Processing and Machine Learning[END_REF] se sont focalisées sur des algorithmes d'apprentissage automatique. De façon spécifique, ces travaux ont permis d'analyser, de comprendre et d'améliorer certains algorithmes simples (machine à vecteurs de support, regroupement spectral, etc). Ces travaux ont mis en particulier en évidence le rôle prépondérant de la moyenne et de la matrice de covariance des données, ce qui en fait des objets qui méritent une compréhension approfondie. Cette thèse s'appuie sur cette importante conclusion pour montrer comment un traitement empirique de la matrice de covariance des données peut avoir des conséquences dramatiques sur des applications de traitement du signal. Nous montrons ensuite comment la moyenne et la matrice de covariance des données interviennent dans la performance d'algorithmes d'apprentissage d'intérêt moderne comme l'apprentissage par transfert et l'apprentissage multi-tâche.

De façon plus détaillée, dans une première partie, la thèse montre en se reposant sur des outils avancés de la théorie des matrices aléatoires comment les estimateurs classiques de distance entre matrices de covariance induisent des biais importants et fournit des estimateurs consistents pour une grande famille de métrique. Ainsi, si on définit par Σ 1 et Σ 2 deux matrices de covariance de taille p × p, on constate que la plupart des distances usuelles, que l'on dénotera génériquement D(Σ 1 , Σ 2 ), s'expriment comme des fonctionnelles des valeurs propres de la matrice Σ -1 1 Σ 2 (distance de Fisher, distance de Bhattacharyya, divergence de Kullback Leibler ou de Rényi entre gaussiennes centrées) ou des valeurs propres de la matrice Σ 1 Σ 2 (distance de Wasserstein entre deux gaussiennes centrées). Dans l'hypothèse où le nombre d'échantillons n 1 et n 2 de données ayant Σ 1 et Σ 2 pour covariance est très grand devant p, la loi des grands nombres garantit que D( Σ1 , Σ2 ) est un estimateur consistent pour D(Σ 1 , Σ 2 ) avec Σa = 1 Cependant, cet estimateur est fortement biaisé lorsque n 1 , n 2 ∼ p. À l'aide d'outils de la théorie des matrices aléatoires, cette thèse propose une formule générale d'un estimateur "universel" des distances D(Σ 1 , Σ 2 ) qui est consistent dans la limite où p, n 1 , n 2 → ∞ avec p/n 1 → c 1 > 0 et p/n 2 → c 2 > 0. Ces résultats s'inspirent des travaux de Mestre Mestre (2008a) sur l'estimation de fonctionnelles des valeurs propres 1 p p i=1 f (λ i (Σ)) de matrices de covariance Σ. La procédure que nous suivons ici est la suivante: (i) la quantité d'intérêt D(Σ 1 , Σ 2 ) est exprimée comme une intégrale complexe faisant intervenir la transformée de Stieltjes de la mesure des valeurs propres de Σ -1 1 Σ 2 (ou Σ 1 Σ 2 ); (ii) cette mesure est reliée asymptotiquement à la mesure limite des valeurs propres de Σ1

-1 Σ2 (ou Σ1 Σ2 ) en utilisant les travaux de Silverstein & Bai (1995) La deuxième partie de la thèse montre encore une fois que les statistiques de premier ordre (moyenne et covariance des données) sont les statistiques suffisantes d'algorithmes plus complexes et d'intérêt plus pratique comme l'apprentissage multi-tâche et par transfert. L'analyse théorique d'un algorithme d'apprentissage multi-tâche basé sur les machines à vecteurs de support révèle d'abord les "statistiques suffisantes" exploitées par l'algorithme et leur interaction. Ces résultats démontrent, que l'approche standard de l'étiquetage des données est largement sous-optimale, peut conduire à de graves effets de transfert négatif, mais que ces déficiences sont facilement corrigées. Ces corrections sont transformées en un algorithme amélioré qui ne fait que bénéficier de données et tâches supplémentaires, et dont la performance théorique est également parfaitement comprise. Comme cela a été démontré et soutenu théoriquement dans de nombreux travaux récents, ces résultats de grande dimension sont robustes à de larges gammes de distributions de données, ce que nos expériences corroborent. Plus précisément, l'étude fait état d'un comportement systématiquement proche entre les performances théoriques et empiriques sur des bases de données populaires, ce qui suggère fortement l'applicabilité de la méthode proposée, soigneusement réglée, à une grande variété de données réelles. Ce réglage fin est entièrement basé sur l'analyse théorique et ne nécessite en particulier aucune procédure de validation croisée. En outre, les performances rapportées sur des ensembles de données réelles surpassent presque systématiquement les méthodes d'apprentissage multi-tâches et de transfert de l'état de l'art, beaucoup plus élaborées et moins intuitives.

Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 2020.

Titre: Méthodes avancées de la théorie des matrices aléatoires pour l'apprentissage automatique.

Mots clés: Théorie des Matrices Aléatoires, Matrices de covariance, Apprentissage par transfert, Apprentissage Multi-tâche, Compréhension théorique de l'apprentissage automatique.

Résumé: L'apprentissage automatique a permis de résoudre de nombreuses applications du monde réel, allant des tâches supervisées à des tâches non supervisées, grâce au développement d'algorithmes puissants (machine à vecteurs de support, réseaux neuronaux profonds, regroupement spectral, etc).

Ces algorithmes sont basés sur des méthodes d'optimisation motivées par des intuitions de petites dimensions qui s'effondrent en grande dimension, un phénomène connu sous le nom de "malédiction de la dimensionnalité". Néanmoins, en supposant que la dimension des données et leur nombre sont à la fois grands et comparables, la théorie des matrices aléatoires fournit une approche systématique pour évaluer le comportement (statistique) de ces grands systèmes d'apprentissage, afin de bien les comprendre et de les améliorer lorsqu'ils sont appliqués à des données de grande dimension.

Les analyses précédentes de la théorie des matrices aléatoires [START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF]Liao & Couillet, 2019;[START_REF] Deng | A model of double descent for high-dimensional binary linear classification[END_REF] Abstract: ML has been quite successful to solve many real-world applications going from supervised to unsupervised tasks due to the development of powerful algorithms (SVM, Deep Neural Network, Spectral Clustering, etc). These algorithms are based on optimization schemes motivated by low dimensional intuitions which collapse in high dimension, a phenomenon known as the "curse of dimensionality". Nonetheless, by assuming the data dimension and their number to be both large and comparable, RMT provides a systematic approach to assess the (statistical) behavior of these large learning systems, to properly understand and improve them when applied to large dimensional data. Previous random matrix analyses [START_REF] Mai | A random matrix analysis and improvement of semisupervised learning for large dimensional data[END_REF]Liao & Couillet, 2019;[START_REF] Deng | A model of double descent for high-dimensional binary linear classification[END_REF] have shown that asymptotic performances of most machine learning and signal processing methods depend only on first and second-order statistics (means and covariance matrices of the data). This makes covariance matrices extremely rich objects that need to be "well treated and understood". The thesis demonstrates first how poorly naive covariance matrix processing can destroy machine learning algorithms by introducing biases that are difficult to clean, whereas consistent random-matrix estimation of the functionals of interest avoids biases. We then exemplify how means and covariance matrix statistics of the data are sufficient (through simple functionals) to handle the statistical behavior of even quite involved algorithms of modern interest, such as multi-task and transfer learning methods. The large dimensional analysis allows furthermore for an improvement of multi-task and transfer learning schemes

  curse to blessing of dimensionality . . . . . . . . . . . . . . 1 1.2 The blessing of dimensionality applied to the estimation of the distance between covariance matrices problem . . . . . . . . . . . 6 1.3 The blessing of dimensionality applied to Multi-Task Learning . 8 1.4 Outline and contributions of the thesis . . . . . . . . . . . . . . . 9

Figure 1

 1 Figure 1.1: (Left) Spectral distribution of the empirical covariance matrix with p = 1000, n = 2000 versus the Marčenko-Pastur law with c ∞ 0 = 0.5. (Right) Marčenko-Pastur law for different values of c ∞ 0 .

  .d. random vectors of zero mean and unit variance entries.

  .4) which gives an explicit inverse for m μ(z). Defining m(z) those of 1 n X ΣX, it can be easily deduced that the ESD µ p of the sample covariance matrix Σ converges to a non-random distribution µ almost surely, where the Stieltjes transform m

m

  μp (ζ j + e ıθ ) log -m μp (ζ j + e ıθ ) -1 ı e ıθ dθ = 0.

Figure

  Figure 2.1: Contour deformation

  Figure 2.3: (Left) Score distribution [empirical histogram vs. theory in solid lines] for x of C 1 (red) or Class C 2 (blue) in a 2-class setting of isotropic balanced Gaussian mixtures. p = 500, n = 600, λ = 1 and µ ∼ 1 10 N(0, I p ) (Right) Theoretical and Empirical classification error averaged over 1000 test samples for the same setting as function of λ.

  Figure3.1: Illustration of the contours maps Γ µ → Γ ν (from right to left) by the variable changes leading up to Theorem 7. (Top) n 2 > p. (Bottom) n 2 < p. For n 2 < p, the left real crossing of Γ ν is necessarily negative (even if the mass at {0} of Supp(µ) were not included in Γ µ ). In case of singularities or branch-cuts (shown in red for the log(z) and √ z functions), the contours are invalid.

Figure 3

 3 Figure 3.2: First and second eigenvectors of K for the traditional estimator (red circles) versus the proposed one (blue crosses); (top) random number of snapshots n i ; (bottom) n 1 = . . . = n m-1 = 512 and n m = 256.

  (j)il ∈ R m into scalar scores y bin(j) il ∈ R. In a standard classification context, this would correspond to turning a two-dimensional hot-bit vector e[2] j into a signed scalar ±1; as we recall that y (j)

  Figure 4.1. In particular, exploiting the theorem along with Remark 5, it is possible, for binary classifier to shift the corresponding input scores y bin ( ) by a constant term ȳ( ) ∈ R in such a way that E x∈C [g bin i (x; )] m i ( ) = 0 and Var x∈C [g bin i (x; )] C i ( ) = 1. This operation prevents the centering and scale problems depicted in the first column of Figure 4.1, the result being visible in the second column of Figure 4.1.

Figure 4 . 2 :

 42 Figure 4.2: Score distribution for new datum x of Class C 1 (red) and Class C 2 (blue) for Task 2 in a 2-task (k = 2) and 2 class-per-task (m = 2) setting of isotropic Gaussian mixtures for: (top) classical MTL LS-SVM with no optimization and a threshold assumed at ζ = 0; (bottom) proposed optimized MTL LS-SVM with estimated threshold ζ; decision thresholds ζ represented in dashed vertical lines; differently related tasks (β = 0 for orthogonal means, β > 0 for positively correlated means and β < 0 for negatively correlated means), p = 100, [c 11 , c 12 , c 21 , c 22 ] = [0.3, 0.4, 0.1, 0.2], γ = 1 2 , λ = 10.Histograms drawn from 1 000 test samples of each class. The figure clearly depicts the deviation from 0 of the decision threshold in unbalanced classes and the deleterious effect of "negative transfer" when β is small; these problems are well handled by the proposed optimized scheme.

Figure 4 . 3 :

 43 Figure 4.3: Classification error of digit pair (1, 4) with different source training pairs for classical LS-SVM and optimized LS-SVM. n 11 = n 12 = 100, n 21 = n 22 = 10 and γ = 1 2 .A PCA preprocessing is performed on each image to extract their p = 100 principal components; the accuracy is performed over n test = 1 135 test samples. The proposed method shows a low sensitivity to λ.

  Figure 4.4: Classification accuracy for increasing number of tasks. (Left) Synthetic data with task correlations β = 1, .9, .5, .2, .8 in this order, p = 100 and c = [.07,.11, .10, .10, .06, .08, .09, .12, .10, .11, .03, .03] T ; accuracy evaluated out of 10 000 test samples. (Right) MNIST dataset with digits (1, 4) as target task, each added task being shown in x-axis; 100 training samples are used for each class of the source tasks and 10 training samples for each class of the target class; HOG features with p = 144 for each image digit; accuracy evaluated out of n test = 1 135 test samples. For both setting, γ = 1 k and λ = 10. The optimized scheme avoids negative transfer by systematically benefiting from additional tasks.

Figure 4

 4 Figure 4.5: ROC curve for proposed optimized versus standard MTL LS-SVM. Synthetic data with p = 128, n 11 = 384, n 12 = 256, n 21 = 64, n 22 = 40, µ 11 = -µ 12 = [1, 0, . . . , 0] T , µ 21 = -µ 22 = [.87, .5, 0, . . . , 0] T . The accuracy of the theoretical anticipation is remarkable and allows for a precise setting of the decision threshold ensuring a desired false alarm rate.

Figure 6 . 4 :

 64 Figure 6.4: Visual representation of the signs of ϕ p and ψ p around singularities.

  x away from the λi (not a solution to ϕ p (x) = 0), reduces to 1 which is exactly equivalent to m µp (x) = -1 c 1 x , i.e., ϕ p (x) = 0. Similarly, the ζ i 's are the eigenvalues of the matrix Λ -

Figure 6 . 5 :

 65 Figure6.5: Example of visual representation of ϕ p /ψ p : R → R, x → ϕ p (x)/ψ p (x); here for p = 4, n 1 = 8, n 2 = 16. In green solid lines are stressed the sets over which ϕ p (x)/ψ p (x) < 0 (which correspond to branch cuts in the study of f (z) = log k (z)). Possible real crossings of the contour Γ are indicated, notably showing that no branch cut is passed through when f (z) = log k (z).

Figure 6 . 7 :

 67 Figure 6.7: Visual representation of the signs of ϕ p and ψ p around singularities for the function f (t) = log(1 + st). Left: case where κ i > λi . Right: case where κ i > λi .

  Figure 6.8: Contour deformation

  ıθ -ξ i) (ξ j + e ıθ ) p i=1 (ξj+ e ıθ -λi)   ( λk -ξ j -e ıθ ) 2ı e ıθ = 0• integrals over the circles surrounding { λi } p i=1 i =k which are null following the same line of reasoning as for ξ i .

  , andA = D γ + λ1 k 1 T k ⊗ I p ∈ R kp×kp .Solving for α and b then gives:α = Q(Y -P b) b = (P T QP ) -1 P T QY.

d

  j Q+O • log p p (6.30) Further introduce the two matrices T and T defined as: Tab,ij = 1 kp tr(C ab QA . These satisfy the following equations (i.e., by right multiplying Equation (6.30) by C i j and taking the trace) ef T ef,ij T i j ,ef , so that T = T (I k -DT) -1 where D = D [d 11 ,...,d km ] T and T ef,i j = 1 kp tr(C ef QC i j Q

k

  (Θ) ≥ 1 -√ pCe -c √ p and ∀θ = (θ 1 , . . . , θ k ) ∈ Θ, ∀a ∈ R k : sup t∈R |P(a T θ T x ≥ t) -G(t)| ≤ Cp -1 4 .

  pour a ∈ {1, 2} la matrice de covariance empirique des n a échantillons centrés x (a) i .

  ; (iii) on obtient alors un estimateur sous forme d'intégrale complexe qu'il s'agit d'évaluer pour chaque fonction f d'intérêt. Cependant, au contraire de Mestre (2008a) qui s'intéresse à des fonctions f simples, les distances D(Σ 1 , Σ 2 ) d'intérêt ici impliquent des logarithmes et racines carrées qui demandent un travail fin d'analyse complexe (notamment un traitement précis des "coupures").

  ont montré que les performances asymptotiques de la plupart des méthodes d'apprentissage automatique et de traitement du signal ne dépendent que des statistiques de premier et de second ordre (moyennes et matrices de covariance des données). Ceci fait des matrices de covariance des objets extrêmement riches qui doivent être "bien traités et compris". La thèse démontre d'abord comment un traitement empirique et naïf de la matrice de covariance peut détruire le comportement d'algorithmes d'apprentissage automatique en introduisant des biais difficiles à supprimer, alors qu'une estimation cohérente des fonctionnelles d'intérêt en utilisant la théorie des matrices aléatoires évite les biais. Nous montrons ensuite comment les moyennes et les matrices de covariance sont suffisantes (par le biais de fonctionnelles simples) pour traiter le comportement d'algorithmes d'intérêt moderne, tels que les méthodes d'apprentissage multi-tâches et par transfert. Title: Advanced Random Matrix Methods for Machine Learning. Keywords: Random Matrix Theory, Covariance matrices, Transfer Learning, Multi-Task Learning, Theory of machine learning.

Table 1

 1

	.1. Using tools from

(error > 50%) (error > 100%) (error > 500%) Table 1.1: Proposed versus classical estimator for the Fisher distance between Σ 1 and

  Couillet, Malik Tiomoko, Steeve Zozor and Eric Moisan, "Random matrix-improved estimation of covariance matrix distances," Journal of Multivariate Analysis, no. 174, pp. 104531, November 2019.Although this family of metrics covered a plethora of distances used in signal processing, we realized that several distances of interest, notably the Wasserstein distance used in optimal transport, did not appear in this first class of distances. A study similar to the first one allowed us to cover this important case of distances that involve eigenvalues functionals of the matrix Σ 1 Σ 2 among which are the Frobenius distance and the Wasserstein distance.

	Conferences:
	1. Malik Tiomoko, Romain Couillet, Steeve Zozor, and Eric Moisan, "Improved
	Estimation of the Distance between Covariance Matrices," in IEEE Interna-
	tional Conference on Acoustics, Speech and Signal Processing (ICASSP'19),
	Brighton, UK, May 2019.

Conferences:

1.

Malik

Tiomoko, and Romain Couillet, "Random Matrix-Improved Estimation of the Wasserstein Distance between two Centered Gaussian Distributions," in European Signal Processing Conference (EUSIPCO'19), A Coruna, Spain, 2019, A Coruna, Spain, 2019, Best Student Paper Award.

  Tiomoko, and Romain Couillet, "Estimation of Covariance Matrix Distances in the High Dimension Low Sample Size Regime," in IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP'19), Guadeloupe, France, 2019.

  Large dimensional spectral behavior of the sample covariance matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Linear spectral statistic . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3 Deterministic equivalent of random matrices . . . . . . . . . . . . 27

	Contents
	2.1

  Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 49

	Chapter 3
	Improved estimation of the
	distance between covariance
	matrices
	Contents
	3.1 Motivation and main findings . . . . . . . . . . . . . . . . . . . . . 33
	3.2 Models and Assumptions . . . . . . . . . . . . . . . . . . . . . . . . 35
	3.3 Improved estimate of the distance between covariance . . . . . . 36
	3.4 Special cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
	3.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
	3.5.1 Confirmation of our results on synthetic data . . . . . . . . . . . .	43
	3.5.2 Application to covariance features-based classification . . . . . . .	43
	3.5.3 Application to covariance matrix estimation . . . . . . . . . . . . .	45
	3.6

  Statistical modeling and the large dimensional setting . . . . . . . 59 4.3 Highlights of the main results . . . . . . . . . . . . . . . . . . . . . 61 4.3.1 Theoretical analysis and large dimensional intuitions . . . . . . . . The General Framework . . . . . . . . . . . . . . . . . . . . . . . . 69 4.4.1 Main ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.6.1 Experiments on binary classification . . . . . . . . . . . . . . . . . 81 4.6.2 Experiments on multi-class classification . . . . . . . . . . . . . . . 87 4.7 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

	Chapter 4
	Large dimensional analysis and
	improvements of Multi-Task
	Learning

Contents 4.1 Motivation and main findings . . . . . . . . . . . . . . . . . . . . . 52 4.2 The Multi-Task Learning Framework . . . . . . . . . . . . . . . . 55 4.2.1 The deterministic setting . . . . . . . . . . . . . . . . . . . . . . . 55 4.2.2 61 4.3.2 Decision threshold and label optimization . . . . . . . . . . . . . . 63 4.3.3 Practical implementation of improved MTL LS-SVM . . . . . . . . 66 4.3.4 Empirical evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.4 69 4.4.2 Classification score asymptotics . . . . . . . . . . . . . . . . . . . . 71 4.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.5.1 Multi-class classification preliminary . . . . . . . . . . . . . . . . . 74 4.5.2 One-versus-all multi-class classification . . . . . . . . . . . . . . . . 76 4.5.3 One-versus-one multi-class classification . . . . . . . . . . . . . . . 78 4.5.4 One-hot encoding approach . . . . . . . . . . . . . . . . . . . . . . 78 4.6

Table 4 .

 4 1: Classification accuracy over Office+Caltech256 database. c(Caltech), w(Webcam), a(Amazon), d(dslr), for different "Source to target" task pairs (S → T ) based on VGG features. Best score in boldface, second-to-best in italic. SVM 96.69 89.90 92.90 90.00 93.80 78.70 93.50 95.00 85.00 90.20 94.70 100 91.70 MMDT 93.90 87.05 90.83 84.40 94.17 86.25 94.58 97.50 86.25 87.23 92.05 97.35 90.96 ILS 77.89 73.55 86.85 76.22 86.22 71.34 74.53 82.80 68.15 63.49 78.98 92.88 77.74 CDLS 97.60 88.30 93.54 88.30 93.54 92.50 93.54 93.75 93.75 88.30 97.35 96.70 93.10 Ours 98.68 89.90 94.40 90.60 94.40 93.80 94.20 100 92.50 89.90 98.70 99.30 94.70 in the previous sections but are easily understood having already delineated the main take-away messages and ideas.

	S/T	c →	w →	c →	a →	w →	a →	d →	w →	c →	d →	a →	d →	Mean
		w	c	a	c	a	d	a	d	d	c	w	w	score
	LS-

Table 4 .

 4 3: Classification accuracy for synthetic data x

	(j) 1l ∼ N(µ 1j , I p ) and x

This normalization is carried out to prevent performance from diverging when n, p tends to infinity (See more details in(Louart & Couillet,

2018))

Since this thesis focuses on universal methods with respect to the data distribution and needs to be robust for real-world applications, a result on the spectral behavior of Σ independently of the data distribution is of particular interest.

convergence reached whether one of the following conditions met: (i)Cost tolerance reached; (ii)Gradient norm tolerance reached; (iii)Maximum user time exceeded; (iv) Maximum iteration count reached

As per Remark 6, this operation involves reevaluating δ[k] and thus y , and thus m for each (λ, γ). It can be performed either on a static grid or by gradient descent until a local minimum is reached.

In the present context of the naive "one-versus-all", this claim should be taken with care: the performance can indeed be predicted provided the binary class model N(µi1, Ip) versus N(µi2, Ip) correctly matches the actual data distribution; this is likely not the case here as the collected fictitious " C2" is rather a mixture of Gaussian rather than a unique Gaussian. In Section 4.5.2, a more elaborate, and theoretically better supported, version of the one-versus-all approach will be discussed.

It must be pointed out that similar random matrix-based studies propose consistent estimates for low dimensional quantities such as those met in Theorem 9; however, these would assume cumbersome forms which, we believe, go against our present request for simple, intuitive but well parameterized algorithms for multi-task and transfer learning.

Here we use the fmincon function implemented in Matlab.

The mode of a set of indices is defined as the most frequent value. When multiple indices occur equally frequently, the smallest of those indices is considered by convention.

Since our main result (Theorem 9) is a central limit theorem, it is not expected to be particularly accurate in the "tails" of the distribution of the output scores g bin i (x); as such, the observed high accuracy for small error values is remarkable.

Acknowledgments

Appendix

Appendix for Chapter 2

Definition 3 (Mixture of Concentrated random vector (Louart & Couillet, 2018)). Let X = [x 1 , . . . , x n ] ∈ M p,n be a data matrix which is constituted of n random vectors distributed on k different classes C 1 , . . . , C k such that the data classes are characterized by the moments, for

In particular the data matrix X satisfy a concentration assumption in the sense that for any 1-Lipschitz function f : M p,n → R with M p,n enrolled by the Frobenius norm F , for q > 0, there exists C, σ > 0 independent of p and n such that

• real integrals over the segments [ξ i , λi ] which can be computed by remarking that the log function has a discontinuity of 2ıπ at the branch cut.

• the integral over the circle surrounding λk computed by remarking that λk is a second-order pole lim

where the second line is obtained remarking that:

Combining these integrals then yields to the solution of the integral:

Case f (t) = log(1 + st) For this case, the integrand of A kk can be derived similarly as in the case of the logarithm by noting that the argument of the logarithm (1 -s/m μp (z)) is a polynomial for which the poles are λi and 0. The zeros are in number p + 1 and denoted κ i , i = 0, . . . , p with κ 0 < 0 < κ 1 < . . . < κ p ; in particular, only κ 1 , . . . , κ p are

and of second-order

in which we defined

i j e

[mk] i j

and the (δ

km ) are the unique positive solutions of

, ∀i, j.

Proof of Lemma 5

First-order deterministic equivalent. A deterministic equivalent for Q is retrieved similarly as provided in (Louart & Couillet, 2018). Our objective is then to find, based on this result, a deterministic equivalent for the random matrix A

To this end, we evaluate the scalar quantity

Furthermore, let us define for convenience the matrix Z -i , which is the matrix Z with a vector of 0 on its i-th column such that

Using the Sherman-Morrison matrix inversion lemma (i.e., (A + uv

Plugging Equation (6.27) into Equation (6.25) leads to

Moreover, following the same line of reasoning as in (Seddik et al., 2020, Proposition A.3), based on Assumption 2 and tools from the concentration of measure theory (see also [START_REF] Ledoux | The concentration of measure phenomenon[END_REF]Louart et al., 2018)), one can show that:

Note that δ ij can be estimated as the solution of the fixed point equation

We then conclude that:

where in the last equality, we used the fact that Q-i is independent from z i . This concludes the proof.

Second-order deterministic equivalent We aim in the following section to prove that

Let us define for convenience C(i) the class of the i-th sample. Similarly as done for the first-order deterministic equivalents, the focus will be on

In order to obtain an estimate of this bilinear form, or equivalently here a deterministic equivalent for

1 2 Z, one must isolate the contribution of the offdiagonal versus diagonal elements of the latter matrix. Starting with the off-diagonal elements, using successively Equation (6.26) and Equation (6.29) on i and j , we have

where the term

is proved to be of order O( 1 √ p ) using (Seddik et al., 2020, Lemma A.2).

As such, the "sub-deterministic equivalent" for the matrix

(note that this matrix estimator of the off-diagonal elements is not zero on the diagonal; however its diagonal elements vanish as n, p → ∞ and may thus be maintained without affecting the final result).

We next need to handle the contribution of the diagonal elements. These are obtained similarly as the off-diagonal elements and lead to the deterministic diagonal matrix equivalent

Put together, the complete deterministic equivalent is then:

i j e

[mk] i j

T

.

This proves that Z

Calculus of B ij . To conclude the proof of Lemma 5, it then remains to find a deterministic equivalent for QA

Q which we denote by B ij . Similar derivations and results are provided in detail in (Louart et al., 2018). For conciseness, we sketch the most important elements of the proof. The interested reader can refer to (Louart et al., 2018, Section 5.2.3)

for any deterministic vector u ∈ R n and v ∈ R n such that u = 1 and v = 1 by using successively Equations (6.29) and (6.26):

1 2 is a finite rank update of Σ ij , one can further use Woodbury identity matrix (i.e., (A + U CV

δ [mk] with δ

. Plugging the expression of Q in Equation ( 6.32), we obtain

with Γ = (I mk + M T Q0 M) -1 and e

[mk] ij is the canonical vector. Finally, to be exhaustive without going into the technical details, 1 let us conclude by remarking that one can show using the deterministic equivalent for Q provided in (Louart & Couillet, 2018) Variance of the classification score. Using Equation (4.2), for x ∈ C j , the covariance of the score g i (x) is given by Cov

1 Due to Remark 4, bi can take any arbitrary value since only the decision threshold but not the performance is sensitive to a shift of Y .

Similarly to the calculus performed for E[g i (x)], using again Q = Q0 -Q0 M(I kp + M T Q0 M) -1 M T Q0 , similar algebraic manipulations lead to:

1 2 and κ ij,. = [κ ij,11 , . . . , κ ij,k2 ] with κ ij,i j = d i j T ij,i j /δ

[mk] i j .