Myriam David

Sylvie Baverel

Huu Hung Vincourt

Stéphanie Vuong

Luc Douesnard

Anne Batalie

Prof Batalie

Gilles Duc

Marielle Lss

Xiaoyi, Cosme, Malik, Lorenzo Hafiz

Mohamed Arun Cyprien

mother Liao Yunxia

Set of real numbers.

C

Set of complex numbers, we denote C + the set {z ∈ C, [z] > 0}. (•) T Transpose operator. tr (•) Trace operator. diag(•) Diagonal operator, for A ∈ R n×n , diag(A) ∈ R n is the vector with entries {A ii } n i=1 ; for a ∈ R n , diag(a) ∈ R n×n is the diagonal matrix taking a as its diagonal.

•

Operator (or spectral) norm of a matrix and Euclidean norm of a vector.

• F

Frobenius norm of a matrix,

Support of a (real or complex) function. dist (•) Distance between elements in a metric space.

P(•)

Probability of an event on the (underlying) probability measure space (Ω, F , P).

E[•]

Expectation operator. Var [•] Variance operator, Var

) 2 if the first two moments of x exist.

a.s.

-→ Almost surely convergence. We say a sequence x n a.s.

-→x if P(lim n→∞ x n = x) = 1.

x e -t 2 2 dt. O(1), o(1) A sequence x n is bounded or converges to zero as n → ∞, respectively.

Vectors and Matrices

x i ∈ R p Input data/feature vector. z i ∈ R p Random vector having i.i.d. zero mean and unit variance entries. X Data/feature matrix having x i as column vectors. Z Random matrix having z i as column vectors. µ Mean vector. C Covariance matrix. K Kernel matrix. w, W Weight vector or matrix (in the context of neural networks). β Regression vector. I n Identity matrix of size n. 1 n Column vector of size n with all entries equal to one. 0 n Column vector of size n with all entries equal to zero. Q A Resolvent of a symmetric real matrix A, see Definition 4.

Introduction (en français)

L'apprentissage automatique étudie la capacité des systèmes d'intelligence artificielle (IA) à acquérir leurs propres connaissances, en extrayant des "modèles" à partir de données. L'introduction de l'apprentissage automatique permet aux ordinateurs d'appréhender le monde réel et de prendre des décisions qui semblent (plus ou moins) subjectives. Les algorithmes d'apprentissage automatique simples, tels que le filtrage collaboratif ou la classification naïve bayésienne, sont utilisés en pratique pour traiter des problèmes aussi divers que la recommandation de films ou la détection de spam dans les courriers électroniques [START_REF] Goodfellow | Deep Learning[END_REF].

La performance des algorithmes d'apprentissage automatique repose énormément sur la représentation des données. La représentation qui (idéalement) contient les informations les plus cruciales pour effectuer la tâche s'appelle les caractéristiques ("features" en anglais) des données et peut varier d'un cas sur l'autre en fonction du problème. Par exemple, les caractéristiques de couleur peuvent jouer un rôle plus important dans la classification des images de chats "noirs versus blancs" que, par exemple, les caractéristiques qui capturent la "forme" des animaux dans les images.

De nombreuses tâches d'intelligence artificielle peuvent être résolues en concevant le bon ensemble de caractéristiques, puis en les transmettant à des algorithmes d'apprentissage simples pour la prise de décision. Cependant, cela est plus facile à dire qu'à faire et pour la plupart des tâches, il est souvent très difficile de savoir quelles caractéristiques doivent être utilisée pour prendre une décision éclairée. À titre d'exemple concret, il est difficile de dire comment chaque pixel d'une image doit peser de sorte que l'image ressemble plus à un chat qu'à un chien. Pendant assez longtemps, trouver ou concevoir les caractéristiques les plus pertinentes avec une expertise humaine, ou "feature engineering", a été considéré comme la clé pour les systèmes d'apprentissage automatique afin d'obtenir de meilleures performances [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF].

Les réseaux de neurones, en particulier les réseaux de neurones profonds, tentent d'extraire des caractéristiques de "haut niveau" (plus abstraites) en introduisant des combinaisons non-linéaires des représentations plus simples (de bas niveau) et ont obtenu des résultats impressionnants au cours de la dernière décennie [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF]. Malgré tous les succès remportés avec ces modèles, nous n'avons qu'une compréhension assez rudimentaire de pourquoi et dans quels contextes ils fonctionnent bien. De nombreuses questions sur la conception de ces réseaux, telles que la détermination du nombre de couches et de la taille de chaque couche, le type de fonction d'activation à utiliser, restent sans réponse.

Il a été observé empiriquement que les réseaux de neurones profonds présentent un avantage crucial lors du traitement de données de grande dimension et nombreuses, autrement dit, lorsque la dimension des données p et leur nombre n sont grands. Par exemple, le jeu de données MNIST [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] contient n = 70 000 d'images de chiffres, de dimension p = 28 × 28 = 784 chacune, réparties en 10 classes (nombres 0 -9). Par conséquent, les systèmes d'apprentissage automatique qui traitent ces grands jeux de ix x CONTENTS données sont également de taille énorme: le nombre de paramètres du modèle N est au moins du même ordre que la dimension p et peuvent parfois même être beaucoup plus nombreux que n.

Plus généralement, les grands systèmes d'apprentissage qui permettent de traiter des jeux de données de très grandes dimensions deviennent de plus en plus importants dans l'apprentissage automatique moderne aujourd'hui. Contrairement à l'apprentissage en petite dimension, les algorithmes d'apprentissage en grande dimension sont sujets à divers phénomènes contre-intuitifs, qui ne se produisent jamais dans des problèmes de petite dimension. Nous montrerons comment les méthodes d'apprentissage automatique, lorsqu'elles sont appliquées à des données de grande dimension, peuvent en effet se comporter totalement différemment des intuitions en petite dimension. Comme nous le verrons avec les exemples de (l'estimation des) matrices de covariance et de matrices de noyau, le fait que n n'est pas beaucoup plus grand que p (disons n ∼ 100p) rend inefficace de nombreux résultats statistiques dans le régime asymptotique "standard", où on suppose n → ∞ avec p fixé. Ce "comportement perturbateur" est en effet l'une des difficultés principales qui interdisent l'utilisation de nos intuitions de petite dimension (de l'expérience quotidienne) dans la compréhension et l'amélioration des systèmes Chapter 1

Introduction

The field studying the capacity of artificial intelligence (AI) systems to acquire their own knowledge, by extracting patterns from raw data is known as machine learning. The introduction of machine learning enables computers to "learn" from the real world and make decisions that appear subjective. Simple machine learning algorithms such as collaborative filtering or naive Bayes are being used in practice to treat as diverse problems as movie recommendation or legitimate e-mails versus spam classification [START_REF] Goodfellow | Deep Learning[END_REF].

The performance of machine learning algorithms relies largely on the representation of the data they need to handle. The representation that (ideally) contains the crucial information to perform the task is known as the data feature, and may vary from case to case depending on the problem at hand. As an example, color features may play a more important role in the classification between black and white cats than, for instance, the features that capture the "look" or the "shape" of the animals.

Many AI tasks can be solved by designing the right set of features and then providing these features to simple machine learning algorithms to make decisions. However, this is easier said than done and for most tasks, it is unclear which feature should be used to make a wise decision. As a concrete example, it is difficult to say how each pixel in a picture should weigh so that the picture looks more like a cat than a dog. For quite a long time, finding or designing the most relevant features with human expertise, or "feature engineering", has been considered the key for machine learning systems to achieve better performance [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF].

Neural networks (NNs), in particular, deep neural networks (DNNs), try to extract high-level features by introducing representations that are expressed in terms of other, simpler representations and have obtained impressive achievements in the last decade [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF]. Yet for all the success won with these models, we have managed only a rudimentary understanding of why and in what contexts they work well. Many questions on the design of these networks, such as how to decide the number of layers or the size of each layer, what kind of activation function should be used, how to train the network more efficiently to avoid overfitting, remain unanswered.

It has been empirically observed that deep neural network models exhibit a major advantage (against "shallow" models) when handling large dimensional and numerous data, i.e., when both the data dimension p and their number n are large. For example, the popular MNIST dataset [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] contains n = 70 000 images of handwritten digits, of dimension p = 28 × 28 = 784 each, from 10 classes (numbers 0 -9). As a consequence, the machine learning systems to "process" these large dimensional data are also huge in their size: the number of system parameters N is at least in the same order of p (so as to map the input data to a scalar output for example) and can sometimes be even much 2 CHAPTER 1. INTRODUCTION larger than n, in the particular case of modern DNNs. 1 More generally, large dimensional data and learning systems are ubiquitous in modern machine learning. As opposed to small dimensional learning, large dimensional machine learning algorithms are prone to various counterintuitive phenomena that never occur in small dimensional problems. In the following Section 1.1, we will show how machine learning methods, when applied to large dimensional data, may indeed strikingly differ from the low dimensional intuitions upon which they are built. As we shall see with the examples of sample covariance matrices and kernel matrices, the fact that n is not much larger than p (say n ≈ 100p) annihilates many results from standard asymptotic statistics that assume n → ∞ alone with p fixed. This "disrupting behavior" is indeed one of the main difficulties that prohibit the use of our low-dimensional intuitions (from everyday experience) in the comprehension and improvement of large-scale machine learning systems.

Nonetheless, by considering the data dimension p and their number n to be both large and comparable and positioning ourselves in the double asymptotic regime where n, p → ∞ with p/n → c ∈ (0, ∞), random matrix theory (RMT) provides a systematic approach to assess the (statistical) behavior of these large learning systems on large dimensional data. As we shall see next, in both examples of sample covariance matrices and kernel matrices, RMT provides direct access to the performance of these objects and thereby allows for a deeper understanding as well as further improvements of these large systems. The major objective of this thesis is to go well beyond these toy examples and to propose a fully-fledged RMT-based framework for more elaborate and more practical machine learning systems: to assess their performance, to better understand their mechanism and to carefully refine them, so as to better handle large dimensional problems in the big data era today. The proposed framework is flexible enough to handle as popular machine learning methods as linear regressions, SVMs, and also to scratch the surface of more involved neural networks.

Motivation: the Pitfalls of Large Dimensional Statistics

In many modern machine learning problems, the dimension p of the observations is as large as -if not much larger than -their number n. For image classification tasks, it is hardly possible to have more than 100p image samples per class: with the MNIST dataset [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] we have approximately n = 6 000 images of size p = 784 for each class; while the popular ImageNet dataset [RDS + 15] contains typically no more than n = 500 000 image samples of dimension p = 256 × 256 = 65 536 in each class.

More generally, in modern signal and data processing tasks we constantly face the situation where n, p are both large and comparable. In genomics, the identification of correlations among hundreds of thousands of genes based on a limited number of independent (and expensive) samples induces an even larger ratio p/n [START_REF] Arnold | Evolutionary formalism for products of positive random matrices[END_REF]. In statistical finance, portfolio optimization relies on the opposite need to invest in a large number p of assets to reduce volatility but at the same time to estimate the current (rather than past) asset statistics from a relatively small number n of "short-time stationary" asset return records [START_REF] Laloux | Random matrix theory and financial correlations[END_REF].

As we shall demonstrate later in this section, the fact that in these problems n, p are 1.1. MOTIVATION: THE PITFALLS OF LARGE DIMENSIONAL STATISTICS 3 both large and in particular, n is not much larger than p inhibits most of the results from standard asymptotic statistics that assume n alone is large [START_REF] Aad | Asymptotic statistics[END_REF]. As a rule of thumb, by much larger we mean here that n must be at least 100 times as large as p for standard asymptotic statistics to be of practical convenience (see our argument on covariance estimation in Section 1.1.1). Many algorithms in statistics, signal processing, and machine learning are precisely derived from this inappropriate n p assumption. A main primary objective of this thesis is to cast a light on the resulting biases and problems incurred, and then to provide a systematic random matrix framework that helps better understand and improve these algorithms.

Perhaps more importantly, we shall see that the low dimensional intuitions which are at the core of many machine learning algorithms (starting with spectral clustering [START_REF] Andrew | On spectral clustering: Analysis and an algorithm[END_REF][START_REF] Von | A tutorial on spectral clustering[END_REF]) often strikingly fail when applied in a simultaneously large n, p setting. A compelling key disrupting property lies in the notion of "distance" between large dimensional data vectors. Most classification methods in machine learning are rooted in the observation that random vectors arising from a mixture distribution (say Gaussian) gather in "groups" of close-by vectors in Euclidean norm. When dealing with large dimensions, concentration phenomena arise that makes Euclidean distances "noninformative", if not counterproductive: vectors of the same Gaussian mixture class may be further away in Euclidean distance than vectors arising from different classes, while, paradoxically, non-trivial classification of the whole set of n ∼ p data may still be doable. This fundamental example of the "curse of dimensionality" phenomenon, as well as its effect on the popular kernel methods and simple nonlinear neural networks, will be discussed at length in Section 1.1.2 and the remainder of the manuscript as well.

Sample covariance matrices in the large n, p regime

Covariance matrices, as a measure of the joint variability between different entries of a random vector, play a significant role in a host of signal processing and machine learning methods. It is particularly efficient in classifying data vectors that are most distinguished through their second order statistics, for instance, in the case of EEG time series [START_REF] De | Graph analysis of functional brain networks: practical issues in translational neuroscience[END_REF] or synthetic aperture radar (SAR) images [START_REF] Chang | Hyperspectral imaging: techniques for spectral detection and classification[END_REF][START_REF] Vasile | Coherency matrix estimation of heterogeneous clutter in high-resolution polarimetric sar images[END_REF].

Let us consider the following illustrating example which shows a first elementary, yet counterintuitive, result: for simultaneously large n, p, sample covariance matrices Ĉ ∈ R p×p based on n samples x i ∼ N (0, C) are jointly entry-wise consistent estimators of the population covariance C ∈ R p×p (in particular, Ĉ -C ∞ → 0 as n, p → ∞ for C ∞ ≡ max ij |C ij |), while overall being extremely poor estimators for a majority of covariance-based methods (i.e., Ĉ -C → 0 with here • the operator norm). This brings forth a first counterintuitive large dimensional observation: matrix norms are not equivalent from a large n, p standpoint.

Let us detail this claim, in the simplest case where C = I p . Consider a data set X = [x 1 , . . . , x n ] ∈ R p×n of n independent and identically distributed (i.i.d.) observations from a p-dimensional Gaussian distribution, i.e., x i ∼ N (0, I p ) for i = 1, . . . , n. We wish to estimate the population covariance matrix C = I p from the n available samples. The maximum likelihood estimator in this zero-mean Gaussian setting is the sample covariance matrix Ĉ defined by

Ĉ = 1 n n ∑ i=1 x i x T i = 1 n XX T . (1.1)
By the strong law of large numbers, for fixed p, Ĉ → I p almost surely as n → ∞, so CHAPTER 1. INTRODUCTION that Ĉ -I p a.s.

-→0 holds for any standard matrix norm and in particular for the operator norm.

One must be more careful when dealing with the regime n, p → ∞ with ratio p/n → c ∈ (0, ∞) (or, from a practical standpoint, n is not much larger than p). First, note that the entry-wise convergence still holds since, invoking the law of large numbers again These results ensure that, as far as spectral properties are concerned, A p can be studied equivalently through B p . We will often use this argument to examine intractable random matrices A p by means of a tractable ersatz B p , which is the main approach to handle the subtle nonlinearity in many random matrix models of interest in machine learning.

The pitfall that consists in assuming that Ĉ is a valid estimator of C since Ĉ -C ∞ a.s.

-→0 may thus have deleterious practical consequences when n is not significantly larger than p.

Resuming on our norm convergence discussion, it is now natural to ask whether Ĉ, which badly estimates C, has a controlled asymptotic behavior. There precisely lay the first theoretical interests of random matrix theory. While Ĉ itself does not converge in any useful way, its eigenvalue distribution does exhibit a traceable limiting behavior [MP67, [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF][START_REF] Bai | Spectral analysis of large dimensional random matrices[END_REF]. The seminal result in this direction, due to Marcenko and Pastur, states that, for C = I p , as n, p → ∞ with p/n → c ∈ (0, ∞), with probability one, the (random) discrete empirical spectral distribution (see also Definition 5)

µ p ≡ 1 p p ∑ i=1 δ λ i ( Ĉ)
converges in law to a non-random smooth limit, today referred to as the "Marcenko-Pastur law" [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF] µ(dx) = (1

-c -1 ) + δ(x) + 1 2πcx (x -a) + (b -x) + dx (1.2)
where a = (1 -√ c) 2 , b = (1 + √ c) 2 and (x) + ≡ max(x, 0). Figure 1.1 compares the empirical spectral distribution of Ĉ to the limiting Marcenko-Pastur law given in (1.2), for p = 500 and n = 50 000. The elementary Marcenko-Pastur result is already quite instructive and insightful.

Remark 1.2 (When is one under the random matrix regime?). Equation (1.2) reveals that the eigenvalues of Ĉ, instead of concentrating at x = 1 as a large-n alone analysis would suggest, spread from (1 -√ c) 2 to (1 + √ c) 2 . As such, the eigenvalues span on a range

(1 + √ c) 2 -(1 - √ c) 2 = 4 √ c CHAPTER 1. INTRODUCTION
which is indeed a slow decaying behavior with respect to c = lim p/n. In particular, for n = 100p, where one would expect a sufficiently large number of samples for Ĉ to properly estimate C = I p , one has 4 √ c = 0.4 which is a large spread around the mean (and true) eigenvalue 1. This is visually confirmed by Figure 1.1 for p = 500 and n = 50 000, where the histogram of the eigenvalues is nowhere near concentrated at x = 1. As such, random matrix results will be largely more accurate than classical asymptotic statistics even when n ∼ 100p. As a telling example, estimating the covariance matrix of each digit from the popular MNIST dataset [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] using the sample covariance, made of no more than 60 000 training samples (and thus about n = 6 000 samples per digit) of size p = 28 × 28 = 784, is likely a hazardous undertaking.

Remark 1.3 (On universality). Although introduced here in the context of Gaussian distributions for x i , the Marcenko-Pastur law applies to much more general cases. Indeed, the result remains valid so long that the x i 's have i.i.d. normalized entries of zero mean and unit variance (and even beyond this setting [A + 11, [START_REF] El | Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond[END_REF][START_REF] Louart | Concentration of measure and large random matrices with an application to sample covariance matrices[END_REF]). Similar to the law of large numbers in standard statistics, this universality phenomenon commonly arises in random matrix theory and high dimensional statistics and helps to justify the wide applicability of the presented theoretical results, even to real datasets.

We have seen in this subsection that the sample covariance matrix Ĉ, despite being a consistent estimator of the population covariance C ∈ R p×p for fixed p as n → ∞, fails to provide a precise prediction on the eigenspectrum of C even with n ∼ 100p. The fact that we are dealing with large dimensional data vectors has an impact not only on the (direct) estimation of the data covariance or correlation but also on various statistics commonly used in machine learning methods. In the following subsection, we will discuss how popular kernel methods behave differently (from our low dimensional intuition) in large dimensional problems, due to the "curse of dimensionality" phenomenon.

Kernel matrices of large dimensional data

Another less known but equally important example of the curse of dimensionality in machine learning involves the loss of relevance of the notion of Euclidean distance between large dimensional data vectors. To be more precise, we will see in this subsection that, under an asymptotically non-trivial classification setting (that is, ensuring that asymptotic classification is neither too simple nor impossible), large and numerous data vectors x 1 , . . . , x n ∈ R p extracted from a few-class mixture model tend to be asymptotically at equal distance from one another, irrespective of their mixture class. Roughly speaking, in this non-trivial setting and under reasonable statistical assumptions on the x i 's, we have

max 1≤i =j≤n 1 p x i -x j 2 -τ → 0 (1.3)
for some constant τ > 0 as n, p → ∞, independently of the classes (same or different) of x i and x j (here the distance normalization by p is used for compliance with the notations in the remainder of the manuscript but has no particular importance). This asymptotic behavior is extremely counterintuitive and conveys the idea that classification by standard methods ought not to be doable in this large n, p regime. Indeed, in the conventional low dimensional intuition that forged many of the leading machine learning algorithms of everyday use (such as spectral clustering [START_REF] Andrew | On spectral clustering: Analysis and an algorithm[END_REF][START_REF] Von | A tutorial on spectral clustering[END_REF]), two data points belong to the same class if they are "close" in Euclidean distance. Here we claim that, when p is large, data pairs are neither close nor far from each other, regardless of their belonging to the same class or not. Despite this troubling loss of individual 1.1. MOTIVATION: THE PITFALLS OF LARGE DIMENSIONAL STATISTICS discriminative power between data pairs, we subsequently show that thanks to a collective behavior of all data belonging to the same (few and thus of large size each) classes, asymptotic data classification or clustering is still achievable. Better, we shall see that, while many conventional methods devised from small dimensional intuitions do fail in the large dimensional regime, some popular approaches (such as the Ng-Jordan-Weiss spectral clustering method [START_REF] Andrew | On spectral clustering: Analysis and an algorithm[END_REF] or the PageRank semi-supervised learning approach [START_REF] Avrachenkov | Generalized optimization framework for graph-based semisupervised learning[END_REF]) still function. But the core reasons for their functioning are strikingly different from the reasons for their initial designs, and they often operate far from optimally.

The non-trivial classification regime 2

To get a clear picture of the source of Equation (1.3), we first need to clarify what we refer to as the "asymptotically non-trivial" classification setting. Consider the simplest setting of a binary Gaussian mixture classification. We give ourselves a training set x 1 , . . . , x n ∈ R p of n samples independently drawn from the two-class (C 1 and C 2 ) Gaussian mixture

C 1 : x ∼ N (µ 1 , C 1 ) C 2 : x ∼ N (µ 2 , C 2 )
each with probability 1/2, for some deterministic µ 1 , µ 2 ∈ R p , positive definite C 1 , C 2 ∈ R p×p , and assume Assumption 1 (Covariance scaling). As p → ∞, we have for a ∈ {1, 2} that max{ C a , C -1 a } = O(1).

In the ideal case where µ 1 , µ 2 and C 1 , C 2 are perfectly known, one can devise a (decision optimal) Neyman-Pearson test. For an unknown x, genuinely belonging to C 1 , the Neyman-Pearson test to decide on the class of x reads

(x -µ 2 ) T C -1 2 (x -µ 2 ) -(x -µ 1 ) T C -1 1 (x -µ 1 ) C 1 ≷ C 2 log det(C 1 ) det(C 2 ) .
(1.4)

Writing x = µ 1 + C 1 2
1 z for z ∼ N (0, I p ), the above test is equivalent to

T(x) ≡ 1 p z T (C 1 2 1 C -1 2 C 1 2 1 -I p )z + 2 p ∆µ T C -1 2 C 1 2 1 z + 1 p ∆µ T C -1 2 ∆µ - 1 p log det(C 1 ) det(C 2 ) C 1 ≷ C 2 0.
(1.5) where we denote ∆µ ≡ µ 1µ 2 that is then normalized by 1/p. Since z ∼ N (0, I p ), U T z follows the same distribution as z for U ∈ R p×p an eigenvector basis of C

1 2 1 C -1 2 C 1 2
1 -I p . As such, the random variable T(x) can be written as the sum of p independent random variables. By Lyapunov's central limit theorem (e.g., [START_REF] Billingsley | Probability and measure[END_REF]Theorem 27.3]), we have, as p → ∞,

V -1 2 T (T(x) -T) d -→N (0, 1) where T ≡ 1 p tr(C 1 C -1 2 ) -1 + CHAPTER 1. INTRODUCTION
As a consequence, the classification performance of x ∈ C 1 is asymptotically non-trivial (i.e., the classification error neither goes to 0 nor 1 as p → ∞) if and only if T is of the same order of √ V T with respect to p. Let us focus on the difference in means ∆µ by considering the worst case scenario where the two classes share the same covariance C 1 = C 2 = C. In this setting, one has from Assumption 1 that

T = 1 p ∆µ T C -1 ∆µ = O( ∆µ 2 p -1 ), V T = 2 p ∆µ T C -1 C -1 2 ∆µ = O( ∆µ p -1 )
so that one must have ∆µ ≥ O(1) with respect to p (indeed, if ∆µ = o(1), the classification of x with C 1 = C 2 = C is asymptotically impossible).

Under the critical constraint ∆µ = O(1) we move on to considering the case of different covariances C 1 = C 2 in such a way that their difference ∆C ≡ C 1 -C 2 satisfies ∆C = o(1). In this situation, by a Taylor expansion of both C -1 2 and det(C 2 ) around

C 1 = C we obtain T = 1 p ∆µ T C -1 ∆µ + 1 2p C -1 ∆C 2 F + o(p -1 ), V T = 4 p 2 ∆µ T C -1 ∆µ + 2 p 2 C -1 ∆C 2 F + o(p -2 ),
which demands ∆C to be at least of order O(p -1 2 ), so that C -1 ∆C 2 F is of order O(1) (as µ ) and can have discriminative power, since (1.6)

C -1 ∆C 2 F ≤ p C -1 ∆C 2 ≤ p ∆C 2 ,
These are the minimal conditions for classification in the case of perfectly known means and covariances in the following sense: i) if none of the inequalities hold (i.e., if means and covariances from both classes are too close), asymptotic classification must fail; and ii) if at least one of the inequalities is not tight (say if ∆µ ≥ O( √ p)), asymptotic classi- fication becomes (asymptotically) trivially easy.

We shall subsequently see that (1.6) precisely induces the asymptotic loss of distance discrimination raised in (1.3) but that in the meantime standard spectral clustering methods based on n ∼ p data remain valid in practice.

Asymptotic loss of pairwise distance discrimination

Under the equality case for the conditions in (1.6), the (normalized) Euclidean distance between two distinct data vectors x i ∈ C a , x j ∈ C b , i = j, is given by

1 p x i -x j 2 = 1 p C 1 2 a z i -C 1 2 b z j 2 - 2 p (µ a -µ b ) T (C 1 2 a z i -C 1 2 b z j ) + 1 p µ a -µ b 2 = 1 p (z T i C a z i + z T j C b z j ) - 2 p z T i C 1 2 a C 1 2 b z j O(p -1/2 ) - 2 p (µ a -µ b ) T (C 1 2 a z i -C 1 2 b z j ) O(p -1 ) + 1 p µ a -µ b 2 O(p -1 )
1.1. MOTIVATION: THE PITFALLS OF LARGE DIMENSIONAL STATISTICS where we see again from Lyapunov's CLT that the second and third terms are of order O(p -1/2 ) and O(p -1 ) respectively, while the last term is of order O(p -1 ), following directly from the critical condition of (1.6). Denote the average covariance of the two classes

C • ≡ 1 2 (C 1 + C 2 ) so that C • = O(1). Note that 1 p E[z T i C a z i ] = 1 p E[tr(C a z i z T i )] = 1 p tr C a = 1 p tr C • + 1 p tr(C a -C • ) ≡ 1 2 τ + 1 p tr C • a
(1.7) where we introduce τ ≡ 2 p tr C • = O(1) and C • a ≡ C a -C • for a ∈ {1, 2}, the operator norm of which is of order O(p -1/2 ) under the critical condition of (1.6).

As such, it is convenient to further write

1 p z T i C a z i = 1 p tr C a + 1 p z T i C a z i - 1 p tr C a ≡ τ 2 + 1 p tr C • a O(p -1/2 ) + ψ i O(p -1/2 ) with ψ i ≡ 1 p z T i C a z i -1 p tr C a = O(p -1/2
) again by the CLT and 1 p tr C • a = O(p -1/2 ) under (1.6). A similar result holds for 1 p z T j C b z j and

1 p x i -x j 2 = τ + 1 p tr(C • a + C • b ) + ψ i + ψ j - 2 p z T i C 1 2 a C 1 2 b z j O(p -1/2 ) - 2 p (µ a -µ b ) T (C 1 2 a z i -C 1 2 b z j ) + 1 p µ a -µ b 2 O(p -1 )
.

This holds regardless of the values taken by a, b. Indeed, one can show with some further refinements that max 1≤i =j≤n 1 p x ix j 2τ → 0 almost surely as n, p → ∞, as previously claimed in (1.3).

To visually confirm the joint convergence of the data distances, Figure 1.2 displays the content of the Gaussian heat kernel matrix K with K ij = exp -1 2p x ix j 2 (which is, therefore, a Euclidean distance-based similarity measure between data points) and the associated second top eigenvector v 2 for a two-class Gaussian mixture x ∼ N (±µ, I p ) with µ = [2; 0 p-1 ]. For a constant n = 500, we take p = 5 in Figure 1.2a and p = 250 in Figure 1.2b. While the "block-structure" in Figure 1.2a agrees with the low dimensional intuition that data vectors from the same class are "closer" to one another, corresponding to diagonal blocks with larger values (since exp(-x/2) decreases with the distance) than in non-diagonal blocks, this intuition collapses when large dimensional data vectors are considered. Indeed, in the large data setting of Figure 1.2b, all entries (but obviously on the diagonal) of K have approximately the same value, which we now know from (1.3) is exp(-1). This is no longer surprising to us. However, what remains surprising at this stage of our analysis is that the eigenvector v 2 of K is not affected by the asymptotic loss of class-wise discrimination of individual distances. Thus spectral clustering seems to work equally well for p = 5 or p = 250, despite the radical and intuitively destructive change in the behavior of K for p = 250. An answer to this question will be given in Section 3.1.1. We will also see that not all kernel choices can reach the same (non-trivial) classification rates, in particular, the popular Gaussian kernel will be shown to be sub-optimal in this respect.

CHAPTER 1. INTRODUCTION K =                                       v 2 =     (a) p = 5, n = 500. K =                                       v 2 =     (b) p = 250, n = 500.

Summary of Section 1.1

In this section, we discussed two simple, yet counterintuitive, examples of common pitfalls in handling large dimensional data.

In the sample covariance matrix example in Section 1.1.1, we made the important remark of the loss of equivalence between matrix norms in the random matrix regime where the data/feature dimension p and their number n are both large and comparable, which is at the source of many intuition errors. We in particular insist that, for matrices

A n , B n ∈ R n×n of large sizes ∀i, j, (A n -B n ) ij → 0 ⇒ A n -B n → 0 (1.8)
in operator norm. We also realized, from a basic reading of the Marcenko-Pastur theorem, that the random matrix regime arises more often than one may think: while n/p ∼ 100 may seem large enough a ratio for classical asymptotic statistics to be accurate, random matrix theory is, in general, a far more appropriate tool (with as much as 20% gain in precision for the estimation of the covariance eigenvalues).

In Section 1.1.2, we gave a concrete machine learning classification example of the message in (1.8) above. We saw that, in the practically most relevant scenario of nontrivial (not too easy, not too hard) large data classification tasks, the distance between any two data vectors "concentrates" around a constant (1.3), regardless of their respective classes. Yet, since again (K n ) ij → τ does not imply that K n -τ1 n 1 T n → 0 in 1.2. RANDOM KERNELS, RANDOM NEURAL NETWORKS AND BEYOND 11 operator norm, we understood that, thanks to a collective effect of the small but similarly "oriented" (informative) fluctuations, kernel spectral clustering remains valid for large dimensional problems.

In a nutshell, the fundamental counterintuitive yet mathematically addressable changes in behavior of large dimensional data have two major consequences to statistics and machine learning: i) most algorithms, originally developed under a low dimensional intuition, are likely to fail or at least to perform inefficiently, yet ii) by benefiting from the extra degrees of freedom offered by large dimensional data, random matrix theory is apt to analyze, improve, and evoke a whole new paradigm for large dimensional learning.

In the following sections of this chapter, we introduce the basic settings under which we work, the machine learning models that we are interested in, as well as the major challenges that we face. In Section 1.2 we discuss three closely related topics: random kernel matrices, random (nonlinear) feature maps, and simple random neural networks. Under a natural and rather general mixture model, we characterize the eigenspectrum of a large kernel matrix and predict the performance of the kernel ridge regressor (which is an explicit function of the kernel function). Random feature maps that are designed to approximate large kernel matrices, take exactly the same form as a single-hiddenlayer NN model with random weights. As a consequence, studying such a network is equivalent to assess the performance of a random feature-based kernel ridge regression, and is thus closely connected to the kernel ridge regression model.

Random Kernels, Random Neural Networks and Beyond

Randomness is intrinsic to machine learning, as the probabilistic approach is one of the most common tools used to study the performances of machine learning methods. Many machine learning algorithms are designed to estimate some unknown probability distribution, to separate the mixture of several different distribution classes, or to generate new samples following some underlying distribution.

Randomness has various sources in machine learning: it may come from the basic statistical assumption that the data are (independently or not) drawn from some probability distribution, from a possibly random search of the hyperparameters of the model (the popular dropout technique in training DNNs [SHK + 14]) or from the stochastic nature of the optimization methods applied (e.g., stochastic gradient descent method and its variants). There are also numerous machine learning methods that are intrinsically random, where randomness is not used to enhance a particular part of the model but is indeed the basis of the model itself, for example in the case of random weights neural networks (as we shall discuss later in Section 1.2.2) as well as many tree-based models such as random forests [START_REF] Breiman | Random forests[END_REF] and extremely randomized trees [START_REF] Geurts | Extremely randomized trees[END_REF].

Let us first focus on the randomness from data by introducing the following multivariate mixture modeling for the input data, which will be considered throughout this manuscript.

Definition 1 (Multivariate mixture model). Let x 1 , . . . , x n ∈ R p be n random vectors drawn independently from a K-class mixture model C 1 , . . . , C K so that each class C a has cardinality n a for a = {1, . . . , K} and ∑ K a=1 n a = n. We say x i ∈ C a if

x i = µ a + C 1 2
a z i for some deterministic µ a ∈ R p , symmetric and positive definite C a ∈ R p×p and some random vector z i with i.i.d. zero mean and unit variance entries. In particular, we say x i follows a K-class Gaussian mixture model (GMM) if z i ∼ N (0, I p ).

The assumption that the data x i are a linear or affine transformation of i.i.d. random vectors z i characterizes the first and second order statistics of the underlying data distribution. Many RMT results such as the popular Marcenko-Pastur law in (1.2), as stated in Remark 1.3, hold universally with respect to the distribution of the independent entries. As such, it often suffices to study the (most simple) Gaussian case to reach a universal conclusion. We will come back to this point with some fundamental RMT results in Section 2.2 and discuss some possible limitations of this "universality" in RMT in Chapter 5.

We insist here that this multivariate mixture model setting for the input data/feature makes the crucial difference between our works and other existing RMT-based analyses of machine learning systems, such as those of L. Zdeborova and F. Krzakala [KMM + 13, SKZ14], or those of J. Pennington [START_REF] Pennington | Nonlinear random matrix theory for deep learning[END_REF][START_REF] Pennington | Nonlinear random matrix theory for deep learning[END_REF][START_REF] Pennington | Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice[END_REF]. In all these contributions, the authors are routinely interested in very homogeneous problems with simple data structures, e.g., x i ∼ N (0, I p ), which makes their analyses more tractable and leads to very simple and more straightforward intuitions or conclusions. On the opposite, we take the more natural choice of mixture model with more involved structures, as in the case of GMM in Definition 1. This enhances the practical applicability of our results, as shall be demonstrated throughout this manuscript: when compared to experiments on commonly used datasets, our theoretical results constantly exhibit an extremely close match to practice. This observation, together with the underlying "universality" argument from RMT, conveys a strong applicative motivation for our works.

As we shall always work in the regime where both n, p are large and comparable, we will, according to the discussions in Section 1.1.2, position ourselves under the following non-trivial regime where the separation of the K-class mixture above is neither too easy nor impossible as n, p → ∞.

Assumption 2 (Non-trivial classification). As n → ∞, for a ∈ {1, . . . , K} 1. p/n = c → c ∈ (0, ∞) and n a /n = c a → ca ∈ (0, 1).

µ

a = O(1) and max{ C a , C -1 a } = O(1) with | tr C • a | = O( √ p), C • a 2 F = O( √ p) for C • a ≡ C a -∑ K i=1 n i n C i .
Assumption 2 is a natural extension of the non-trivial classification condition in (1.6) to the general K-class setting.

Under a GMM for the input data that satisfies Assumption 2, we aim to investigate the (asymptotic) performance of any machine learning method of interest, in the large dimensional regime where n, p → ∞ and p/n → c ∈ (0, ∞). However, in spite of a huge number of well-established RMT results, the "exact" performance (as a function of the data statistics µ, C and the problem dimensionality n, p) still remains technically out of reach, for most machine learning algorithms.

The major technical difficulty that prevents existing RMT results from being applied directly to understand these machine learning systems is the nonlinear and sometimes implicit nature of these models. Powerful machine learning methods are commonly built on top of highly complex and nonlinear features, which make the evaluation of these methods less immediate than linear and explicit methods. As a consequence, we need a proper adaptation of the matrix-based classical RMT results to handle the entry-wise 1.2. RANDOM KERNELS, RANDOM NEURAL NETWORKS AND BEYOND 13 nonlinearity (e.g., the kernel function or the activation function in neural networks) or the implicit solutions arising from optimization problems (of logistic regression for instance).

In this section, in pursuit of a satisfying answer to these aforementioned technical difficulties, we review some recent advances in RMT in this vein, starting from large kernel matrices.

Random matrix consideration of large kernel matrices

Large kernel matrices and their spectral properties

In a broad sense, kernel methods are at the core of many, if not most, machine learning algorithms. For a given data set x 1 , . . . , x n ∈ R p , most learning mechanisms aim to extract the structural information of the data (to perform classification for instance) by assessing the pairwise comparison k(x i , x j ) for some affinity metric k(•, •) to obtain the matrix

K ≡ k(x i , x j ) n i,j=1 .
(1.9)

The "cumulative" effect of these comparisons for numerous (n 1) data is at the heart of a broad range of applications: from supervised learning methods of maximum margin classifier such as support vector machines (SVMs), kernel ridge regression or kernel Fisher discriminant, to kernel spectral clustering and kernel PCA that work in a totally unsupervised manner.

The choice of the affinity function k(•, •) is central to a good performance of the learning method. A typical viewpoint is to assume that data x i and x j are not directly comparable (e.g., not linearly separable) in their ambient space but that there exists a convenient feature mapping φ : R p → H that "projects" the data to some (much higher or even infinite dimensional) feature space H where φ(x i ) and φ(x j ) are more amenable to comparison, for example, can be linearly separable by a hyperplane in H.

While the dimension of the feature space H can be much higher than that of the data (p), with the so-called "kernel trick" [START_REF] Schölkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF] one can avoid the evaluation of φ(•) and instead work only with the associated kernel function k : R p × R p → R that is uniquely determined by k(x i , x j ) = φ(x i ) T φ(x j ) (1.10) via Mercer's theorem [START_REF] Schölkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF]. Some commonly used kernel functions are the radial basis function (RBF, or Gaussian, or heat) kernel k(x i , x j ) = exp -

x i -x j 2 2σ 2
, the sigmoid kernel tanh(x T i x j + c) and polynomial kernels (x T i x j + c) d . These kernel-based learning algorithms have been extensively studied both theoretically and empirically for their feature extraction power in highly nonlinear data manifolds, before the recent "rebirth" of neural networks.

But only very recently has the large dimensional (p ∼ n 1) nature of the data started to be taken into consideration for kernel methods. We have seen in Section 1.1.2 empirical evidence showing that the kernel matrix K behaves dramatically differently from its low dimensional counterpart. In the following, we review some theoretical explanations of this counterintuitive behavior provided by RMT analyses.

To assess the eigenspectrum behavior of K for n, p both large and comparable, the author of [START_REF] El | The spectrum of kernel random matrices[END_REF] considered kernel matrices K with "shift-invariant" type kernel functions k(x i , x j ) = f ( x ix j 2 /p) or "inner-product" kernel f (x T i x j /p) for some nonlinear and locally smooth function f . It was shown that the kernel matrix K is asymptotically CHAPTER 1. INTRODUCTION equivalent to a more tractable random matrix K in the sense of operator norm, as stated in the following theorem.

Theorem 1.1 (Theorem 2.1 in [START_REF] El | The spectrum of kernel random matrices[END_REF]). For inner-product kernel f (x T i x j /p) and x i = C 1 2 z i with positive definite C ∈ R p×p satisfying Assumption 1 and random vector z i ∈ R p with i.i.d. zero mean, unit variance and finite 4 + (absolute) moment entries and f three-times differentiable in a neighborhood of 0, we have

K -K → 0 in probability as n, p → ∞ with p/n → c ∈ (0, ∞), where K = f (0) + f (0) 2p 2 tr(C 2 ) 1 n 1 T n + f (0) p X T X + f τ 2 -f (0) - τ 2 f (0) I n (1.11)
with τ ≡ 2 p tr C > 0. We provide here only the intuition for this result in the inner-product kernel f (x T i x j /p) case with Gaussian z i ∼ N (0, I p ). The shift-invariant case can be treated similarly. To prove the universality with respect to the distribution of entries of z i , a more cumbersome combinatoric approach was adopted in [START_REF] El | The spectrum of kernel random matrices[END_REF], which is likely unavoidable.

Intuition of Theorem 1.1. The proof is based on an entry-wise Taylor expansion of the kernel matrix K, which naturally demands the kernel function f to be locally smooth around zero. More precisely, we obtain, using Taylor expansion of f (x) around x = 0 that, for i = j,

K ij = f (x T i x j /p) = f (0) + f (0) p x T i x j + f (0) p 2 (x T i x j ) 2 + T ij
with some higher order (≥ 3) terms T ij that contain higher order derivatives of f . Note that T ij typically contains terms of the form (x T i x j /p) κ for κ ≥ 3 so that T ij is of order O(p -3/2 ), uniformly on i = j, as a result of x T i x j /p = O(p -1/2 ) by the CLT. As such, the matrix {T ij δ i =j } 1≤i,j≤n (of size n × n and with zeros on its diagonal) can be shown to have a vanishing operator norm as n, p → ∞ since

{T ij δ i =j } 1≤i,j≤n ≤ n {T ij δ i =j } 1≤i,j≤n ∞ = o(1)
as n, p → ∞ with p/n → c ∈ (0, ∞). We then move on to the second order term f (0) p 2 (x T i x j ) 2 , the expectation of which is given by

f (0) p 2 E(x T i x j ) 2 = f (0) p 2 E x i tr x T i E x j [x j x T j ]x i = f (0) p 2 tr(C 2 ) = O(p -1 )
where we use the linearity of the trace operator to push the expectation inside, together with the fact that C = O(1). In matrix form this gives,

f (0) p 2 E[(x T i x j ) 2 δ i =j ] 1≤i,j≤n = f (0) p 2 tr(C 2 )(1 n 1 T n -I n )
which is of operator norm of order O(1), even without the diagonal term f (0) p 2 tr(C 2 )I n (the operator norm of which is of order O(p -1 ) and thus vanishing, also note that we leave the diagonal untreated for the moment). With concentration arguments we can show the fluctuation (around the expectation) of this second order term also has a vanishing operator norm as n, p → ∞ with p/n → c > 0, which, together with easy treatment of the diagonal terms, concludes the proof. This operator norm consistent approximation K -K → 0, as discussed in Remark 1.1, provides a direct access to the limiting spectral measure (if it exists), the isolated eigenvalues and associated eigenvectors that are of central interest in spectral clustering or PCA applications, as well as the regression or label/score vectors, of (functions of) the intractable kernel matrix K, via the study of the more tractable random matrix model K, according to Weyl's inequality (see Lemma 2.10).

The asymptotic equivalent kernel matrix K is the sum of three matrices: i) a rank one matrix proportional to 1 n 1 T n , ii) the identity matrix that makes a constant shift of all eigenvalues and iii) a rescaled version of the standard Gram matrix 1 p X T X. The eigenvalue distribution of the Gram matrix, or equivalently of the sample covariance matrix model 1 p XX T (via Sylvester's identity in Lemma 2.3) has been intensively studied in the random matrix literature [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF], and we shall talk about this model at length in Section 2.2.3. The low rank (rank one here) perturbation of a random matrix model, is known under the name of "spiked model" and has received considerable research attention in the RMT community [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF]; we will come back to this point in more details in Section 2.3.

On closer inspection of (1.11), we see that K is in essence a local linearization of the nonlinear K, in the sense that the nonlinear function f (x) is evaluated solely in the neighborhood of x = 0. This is because by the CLT we have x T i x j /p = O(p -1 2 ) for i = j, and entry-wise speaking, all off-diagonal entries are constantly equal to f (0) to the first order. As such, the nonlinear function f acts only in a "local" manner around 0: all smooth nonlinear f with the same values of f (0), f (0) and f (0) have the same expression of K, up to a constant shift of all eigenvalue due to f (τ/2)I n , and consequently have the same (asymptotic) performance on all kernel-based spectral algorithms.

In [START_REF] El | On Information Plus Noise Kernel Random Matrices[END_REF] the author considered an "information-plus-noise" model for the data, where each observation consists of two parts: one from a "low dimensional" structure y i ∈ R p (e.g., with its 0 norm y i 0 = O(1)) that is considered the "signal" as well as the informative part of the data and the other being high dimensional noise. This modeling indeed assumes that data are randomly drawn from, not exactly a fixed and low dimensional manifold, but somewhere "nearby" such that the resulting observations are perturbed with additive high dimensional noise, more precisely

x i = y i + C 1 2 z i
where y i denotes the random "signal" part of the observation and z i the high dimension noise part that is independent of y i . The proof techniques are basically the same as in [START_REF] El | The spectrum of kernel random matrices[END_REF]. However, by assuming that all x i 's drawn from the same distribution, this result is not sufficient for the understanding of K in a more practical classification context. More recently in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], the authors considered the shift-invariant kernel f ( x ix j 2 /p), under a more involved K-class Gaussian mixture model (GMM, see Definition 1) for the data, that is of more practical interest from a machine learning standpoint, and investigated more precisely the eigenspectrum behavior for a kernel spectral clustering purpose. Not only the kernel matrix K but also the associated (normalized) Laplacian matrix L defined as L ≡ nD -1 2 KD -1 2 are considered in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], where D ≡ diag(K1 n ) denotes the so-called "degree matrix" of K. While the authors followed the same technical approach as in [START_REF] El | The spectrum of kernel random matrices[END_REF], the fact that they considered a Gaussian mixture modeling provides rich insights into the impact of nonlinearity in kernel spectral clustering applications. For the first time, the exact CHAPTER 1. INTRODUCTION stochastic characterization of the isolated eigenpairs are given, as a function of the data statistics (µ, C), problem dimensionality n, p, as well as the nonlinear f in a local manner.

Built upon the investigation of K in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], we evaluate the exact performance of the kernel ridge regression (or least squares support vector machine, LS-SVM) in classifying a two-class GMM, in the following contributions. In a nutshell, we consider the following (soft) output decision function from the LS-SVM formulation

h(x) = β T k(x) + b, β = Q(y -b1 n ), b = 1 T n Qy 1 T n Q1 n (1.12) for k(x) ≡ { f ( x -x i 2 /p)} n i=1 ∈ R n and Q = K + n λ I n -1
the so-called resolvent of the kernel matrix K (see Definition 4). By showing that h(x) can be (asymptotically) well approximated by a normally distributed random variable, we obtain the exact asymptotic classification performance of LS-SVM. The minimum orders of magnitude distinguishable by an LS-SVM classifier are given by

∆µ = O(1), | tr ∆C| = O( √ p), ∆C 2 F = O(p)
for ∆µ = µ 1µ 2 and ∆C = C 1 -C 2 . This is very close to the theoretical optimum in the oracle setting in (1.6) and is the best rate reported in [START_REF] Couillet | Classification asymptotics in the random matrix regime[END_REF]. Perhaps more importantly, when tested on several real-world datasets such as the MNIST [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] and Fashion MNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF] image datasets, a surprisingly close match is observed between theory and practice, thereby conveying a strong applicative motivation for the study of simple GMM in a high dimensional setting. We will continue this discussion in more details and justify that this observation is in fact not that surprising in Section 3.1.1.

All works above are essentially based on a local expansion of the kernel function f , which follows from the "concentration" of the similarity measure x ix j 2 /p or x T i x j /p around a single value of the smooth domain of f . More precisely, due to the independence between two data vectors x i , x j , the diagonal and off-diagonal entries of K behave in a totally different manner. Consider K ij = f (x T i x j /p) with x i ∼ N (0, I p ); we have roughly f (1) on and f (0) off the diagonal of K, so that most entries (of order O(n 2 )) of K evaluate the nonlinear function f around zero. This leads to the "local linearization" of f in the expression of K in Theorem 1.1 and thereby disregards most of the domain of f .

On the other hand, since x T i x j / √ p → N (0, 1) in distribution as p → ∞ (and is thus of order O(1) with high probability), it appears that f (x T i x j / √ p) is a more natural scaling to avoid an asymptotic linearization of the nonlinear K. Nonetheless, in this case, all diagonal entries become x i 2 / √ p = O( √ p) and thus evaluate f asymptotically at infinity, which is again another "improper scaling", but only on the diagonal of K.

RANDOM KERNELS, RANDOM NEURAL NETWORKS AND BEYOND

In the dot product model f (x T i x j / √ p), the entries will not "concentrate" at f (0). Alternative tool is thus needed in the place of the Taylor expansion, so as to handle the nonlinear function f in this case. This more naturally scaling model was studied in [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF], where the authors considered the dot product kernel matrix K, the (i, j) entry of which is given by

K ij = f (x T i x j / √ p)/ √ p for i = j 0 for i = j . (1.13)
The more natural √ p normalization inside f helps, as discussed above, avoid the non- linear K (asymptotically) acting as a linear model X T X, while the √ p scaling outside is simply convenient to ensure the operator norm K is order O(1) for n, p large. Note that the diagonal elements are discarded since they are now "improperly scaled" for the evaluation by f . Again, we are interested in the spectral property of the kernel matrix K defined in (1.13). For x i ∼ N (0, I p ), the empirical spectral measure µ n = 1 n ∑ n i=1 δ λ i (K) (see Definition 5) of K has an asymptotically deterministic behavior as n, p → ∞ with p/n → c ∈ (0, ∞). This limiting spectral measure of K was first characterized in [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF] and then generalized in [START_REF] Do | The spectrum of random kernel matrices: universality results for rough and varying kernels[END_REF] to handle random vectors x i with i.i.d. entries of zero mean, unit variance and finite higher order moments, as summarized in the following theorem.

Theorem 1.2 (Theorem 3.4 in [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF], Theorem 3 in [START_REF] Do | The spectrum of random kernel matrices: universality results for rough and varying kernels[END_REF]). Under some regularity condition for the kernel function f (see Assumption 3 below), the empirical spectral measure of K converges weakly and almost surely, as n, p → ∞ with p/n → c ∈ (0, ∞), to a probability measure µ. The latter is uniquely defined through its Stieltjes transform m :

C + → C + , z → (t -z) -1 µ(dt) (see also Definition 6 in Section 2.2), given as the unique solution in C + of the cubic equation 3 - 1 m(z) = z + a 2 1 m(z) c + a 1 m(z) + ν -a 2 1 c m(z) with a 1 = E[ξ f (ξ)] and ν = Var[ f (ξ)] ≥ a 2 1 for standard Gaussian ξ ∼ N (0, 1).
The technical approach to achieve Theorem 1.2 is rather different from that of Theorem 1.1. Instead of performing a local Taylor expansion of f , the authors of [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF][START_REF] Do | The spectrum of random kernel matrices: universality results for rough and varying kernels[END_REF] rely on the theory of orthogonal polynomials, in particular, of the class of Hermite polynomials defined with respect to the standard Gaussian distribution [START_REF] Abramowitz | Handbook of mathematical functions: with formulas, graphs, and mathematical tables[END_REF][START_REF] George E Andrews | Special functions[END_REF]. This thus allows for a polynomial approximation of the nonlinear function f as long as it is square-integrable, and thus covers naturally non-differentiable kernel functions. Some useful concepts from the theory of orthogonal polynomial are recalled as follows.

For a probability measure µ, we denote the set of orthogonal polynomials with respect to the scalar product f , g = f gdµ as {P l (x), l = 0, 1, . . .}, obtained from the Gram-Schmidt procedure on the monomials {1, x, x 2 , . . .} such that P 0 (x) = 1, P l is of degree l and P l 1 , P l 2 = δ l 1 -l 2 . By the Riesz-Fischer theorem [START_REF] Rudin | Principles of mathematical analysis[END_REF]Theorem 11.43], for any function f ∈ L 2 (µ), the set of squared integrable functions with respect to •, • , one can formally expand f as

f (x) ∼ ∞ ∑ l=0 a l P l (x), a l = f (x)P l (x)dµ(x) (1.14)
where " f ∼ ∑ ∞ l=0 P l " indicates that f -∑ N l=0 P l → 0 as N → ∞ (and f 2 = f , f ). To ensure the polynomial approximation is accurate when truncated at a large but finite degree L, we assume the following assumption holds. Assumption 3. For each p, let ξ p = x T i x j / √ p and let {P l,p (x), l ≥ 0} be the set of orthonormal polynomials with respect to the probability measure µ p of ξ p . For f ∈ L 2 (µ p ) for each p, i.e.,

f (x) ∼ ∞ ∑ l=0 a l,p P l,p (x)
for a l,p defined in (1.14), we demand that

1. ∑ ∞ l=0 a l,p P l,p (x)µ p (dx) converges in L 2 (µ p ) to f (x) uniformly over large p, 2. as p → ∞, ∑ ∞ l=1 |a l,p | 2 → ν ∈ [0, ∞).
Moreover, for l = 0, 1, 2, a l,p converges and we denote a 0 , a 1 and a 2 their limits, respectively.

3. a 0 = 0. Since ξ p → N (0, 1) as p → ∞, the limiting parameters a 0 , a 1 , a 2 and ν are simply (generalized) moments of the standard Gaussian measure involving f . Precisely,

a 0 = E[ f (ξ)], a 1 = E[ξ f (ξ)], a 2 = E[(ξ 2 -1) f (ξ)] √ 2 = E[ξ 2 f (ξ)] -a 0 √ 2 , ν = Var[ f (ξ)]
for ξ ∼ N (0, 1). These parameters are of crucial significance in determining the eigenspectrum behavior of K. For example, the limiting spectral measure is uniquely determined by a 1 and ν as per Theorem 1.2. The last assumption a 0 = 0 is demanded here mainly for technical convenience and will not affect the classification performance in practice, as it adds a non-informative rank one perturbation of the form a 0 (1 n 1 T n -I n )/ √ p to the kernel matrix.

The proof of Theorem 1.2 calls for some standard algebraic and probabilistic manipulations in RMT, that will be introduced in Chapter 2. For self-consistency, we include the intuitive proof for Theorem 1.2 in Section A.1.1 of the appendix. Theorem 1.2 only gives the (limiting) eigenvalue distribution of the kernel matrix K in (1.13) under a null model which, from a machine learning perspective, is not sufficient to decide for example how to choose f with respect to the data/task at hand. Compared to [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], an important piece of the mixture models is still missing, which leads to our following investigation.

(C2) Zhenyu Liao and Romain Couillet. Inner-product kernels are asymptotically equivalent to binary discrete kernels, 2019.

In this contribution we provide, as in Theorem 1.1, a more accessible random matrix K that is asymptotically equivalent to K, under a two-class multivariate mixture model (as in Definition 1) for data with covariance C a = I p + E a that satisfies the non-trivial classification condition of Assumption 2.

Interestingly, K is (again) nothing but the sum of two matrices: i) the null model K N characterized in Theorem 1.2 that depends on f via the two coefficients a 1 and ν and ii) a low rank and informative matrix KI that only depends on a 1 and a 2 . This means that, instead of a local behavior of f as in [START_REF] El | The spectrum of kernel random matrices[END_REF][START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], the classification performance of the properly scaled dot product kernel f (x T i x j / √ p) is related to f in a more "global", yet still simple, way only via the three parameters a 1 , a 2 and ν.

On the downside, it must be pointed out that, the studies performed in [CS13, DV13] (as well as in [START_REF] Liao | Inner-product kernels are asymptotically equivalent to binary discrete kernels[END_REF] above) only cover the case of identity covariance C = I p . By introducing an arbitrary covariance C and considering x i = C 1 2 z i for z i having i.i.d. entries, the limiting spectral measure of K becomes technically more challenging since it breaks most of the orthogonality properties of the orthogonal polynomial approach in the proofs. But it is a needed extension in pursuit of the general multivariate mixture model, and to compare the performance between different scaling strategies, which is of significant interest for future investigation.

Random feature approximation of kernels

From a practical standpoint, the computation of the kernel matrix K in (1.9) requires one to evaluate f for all pairs of (x i , x j ). As such, the computational complexity of K grows quadratically with respect to the number of data points n and can be intense for n large. In this regard, kernel machines are inappropriate to deal with large scale problems. Also, they appear to be less efficient in both online applications [START_REF] Van Vaerenbergh | Kernel recursive least-squares tracker for time-varying regression[END_REF] and applications with privacy concerns [START_REF] Wu | Data mining with big data[END_REF].

To cope with this limitation, a large scale kernel approximation technique was originally introduced in [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF], where the authors proposed to approximate the shift-invariant kernels of the type k(x i , x j ) = f (x ix j ) with "random Fourier features". This idea was then extended to cover other classes of commonly used kernels such as the additive homogeneous kernels [START_REF] Vedaldi | Efficient additive kernels via explicit feature maps[END_REF] and dot-product kernels [START_REF] Kar | Random feature maps for dot product kernels[END_REF]. The core idea of these random feature approximation techniques is based, in a general manner, on the following remark

K ij = φ T (x i )φ(x j ) = k(x i , x j ) = E w [ϕ T w (x i )ϕ w (x j )]
where we shall distinguish the kernel feature map φ : R p → H that determines the kernel function k via Mercer's theorem (and possibly maps to an infinite dimensional space) from the random feature map ϕ w : R p → R d that often maps to low dimensional space (with sometimes d = 1, 2) and depends on the random w. As such, by independently drawing w i from a predefined distribution, one is able to construct an approximation K of K by replacing the expectation E w with an empirical average as

Kij = 1 N N ∑ m=1 ϕ w m (x i ) T ϕ w m (x j ). (1.15)
It is desired to have K close to K in some sense, e.g., in their operator norms K -K → 0 as N → ∞ for spectrum-based methods.

In the popular example of random Fourier features, one focuses on the family of shiftinvariant kernel functions which depends solely on the difference x ix j . In the particular case of the Gaussian kernel, Bochner's theorem [START_REF] Rudin | Fourier analysis on groups[END_REF] guarantees that

K ij = e -1 2 x i -x j 2 = E w [e ıw T x i e -ıw T x j ] 1 N N ∑ m=1 e ıw T m x i e -ıw T m x j ≡ Kij (1.16)
for the random Fourier feature ϕ w (x) = e ıw T x with w ∼ N (0, I p ). We close this subsection with the final remark that, by cascading the Gaussian w m 's as row vectors, we obtain a random Gaussian matrix W ∈ R N×p , the entries of which are independent standard Gaussian random variables, i.e., W ij ∼ N (0, 1). Then, according to (1.16) we have

[ Kij ] = 1 N σ 1 (Wx i ) T σ 1 (Wx j ) + 1 N σ 2 (Wx i ) T σ 2 (Wx j ) (1.17)
for σ 1 (t) = cos(t) and σ 2 (t) = sin(t), which coincides with the Gram matrix of the singlelayer neural network model with random weights W and cosine+sine activation functions, as detailed in the following subsection.

Random neural networks

A fundamental link exists between random feature maps (and thus random kernel matrices) and neural networks with random weights. Neural networks (NNs) with random weights have their roots in the pioneering works on the perceptron [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF] and have then been successively revisited and analyzed in a number of works, both in the feedforward [SKD92, ASD96] and the recurrent case [START_REF] Gelenbe | Learning in the recurrent random neural network[END_REF]. More recently, some of these randomized NN models (the so-called extreme learning machine [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF] as an example for the feed-forward case and the echo state network [START_REF] Jaeger | The "echo state" approach to analysing and training recurrent neural networks-with an erratum note[END_REF] for the recurrent one) have been shown to achieve satisfactory performances in some problems (see examples in the next paragraph), with a relatively short training time and low model complexity.

We refer the readers to [START_REF] Scardapane | Randomness in neural networks: an overview[END_REF] for a complete overview of randomness in NN models.

Random neural networks are more than a "cheap" way to solve less efficiently a difficult problem, given the limited computational resources at hand. On the contrary, they work sufficiently well in many practical scenarios [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF] and can even sometimes achieve remarkable accuracies that are comparable to deep and finely tuned structures, in a large number of applications [LML + 14]. This seemingly counterintuitive fact is perhaps the key to a deeper comprehension of the intrinsic properties of different NN architectures. Taking the example of convolutional neural networks (CNNs), it was observed in [START_REF] Jarrett | What is the best multi-stage architecture for object recognition?[END_REF] that, for small datasets, completely random filters are sufficient to propagate useful information through a rectified linear (ReLU) network. This observation was used to argue for the significance of ReLU nonlinearities and the (convolutional) pooling operation in image classification problems, as also pointed out in [PDDC09, SKC + 11, CP11b, GSB16], and in many other tasks such as image reconstruction [START_REF] He | A powerful generative model using random weights for the deep image representation[END_REF], denoising and super-resolution [START_REF] Ulyanov | Deep image prior[END_REF].

Random NNs have the advantage of being mathematically more tractable, at the same time preserving the "structural" properties of these elaborate learning systems, e.g., the use of appropriate nonlinear functions in DNNs or convolutional filters in CNNs. The study of random NNs can, therefore, shed novel light on the design of more advanced NN models.

In the training procedure of modern NNs, optimization methods such as (stochastic) gradient descent and its variants are always initialized randomly, with the NN weights drawn randomly from a (properly scaled) normal distribution. In this regard, NNs with random weights are essentially the initial states of well-trained NNs, and can provide helpful insight into the understanding of modern DNNs, e.g., to accelerate training, at least at the very initial stage, by examining the eigenspectrum behavior of the inputoutput Jacobian matrix [START_REF] Pennington | Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice[END_REF], or to gain theoretical insights into the geometric properties of DNNs [DLT + 18, AZLS18]. In these works, the network weights, or training data, or both of them, are typically assumed to be normally distributed to facilitate theoretical assessment.

In this manuscript, we focus only on feed-forward NNs. For random recurrent NNs, RMT-based analyses have also been established, but are mainly restricted to the linear case [START_REF] Couillet | The asymptotic performance of linear echo state neural networks[END_REF].

Let us consider the random weights feed-forward NN with a single hidden layer as in Figure 1.3. For a data matrix X = x 1 , . . . , x n ∈ R p×n , we denote Σ ∈ R N×n the output of the middle layer comprising of N neurons in total (also referred to as the random feature matrix) of X by premultiplying some random weight matrix W ∈ R N×p with i.i.d. standard Gaussian entries and then applying entry-wise some nonlinear activation function σ : R → R so that Σ ≡ σ(WX) ∈ R N×n , the column vectors σ(Wx i ) of which are the associated random feature of x i . With Σ, the weights of the second layer β ∈ R N×d are usually "learned" to adapt the feature matrix Σ to the associated target Y = [y 1 , . . . , y n ] ∈ R d×n , for example by minimizing Yβ T Σ 2 F . In the case where β is designed to minimize the regularized mean squared error (MSE)

X ∈ R p×n σ σ σ σ σ Σ ≡ σ(WX) ∈ R N×n β T Σ W ∈ R N×p β ∈ R N×d
L(β) = 1 n ∑ n i=1 y i -β T σ(Wx i ) 2 + λ β 2
F , for some regularization factor λ ≥ 0, we obtain the explicit ridge-regressor

β ≡ 1 n Σ 1 n Σ T Σ + λI n -1 Y T (1.18)
which follows from differentiating L with respect to β to obtain

0 = λβ + 1 n Σ(Σ T β -Y T ) so that ( 1 n ΣΣ T + λI N )β = 1 n ΣY T which, along with ( 1 n ΣΣ T + λI N ) -1 Σ = Σ( 1 n Σ T Σ + λI n ) -1
, gives the result. Note that, similar to the kernel ridge regression investigated in [START_REF] Liao | A large dimensional analysis of least squares support vector machines[END_REF], the resolvent 1 n Σ T Σ + λI n -1 also plays a central role in the performance analysis of this network. The single-hidden-layer random NN model presented above, with the first layer fixed at random and the second layer performing a ridge regression, is sometimes referred to as "extreme learning machines" in the literature [START_REF] Huang | Extreme learning machine for regression and multiclass classification[END_REF]. Note from (1.18) that,

1 n [Σ T Σ] ij = 1 n σ(Wx i ) T σ(Wx j )
which coincides with the approximated kernel matrix from random feature-based techniques in (1.17). From this perspective, the random weights NN model in Figure 1.3 is in

CHAPTER 1. INTRODUCTION
essence equivalent to a random feature-based kernel ridge regression, i.e., by replacing the kernel matrix K with its random feature approximation 1 n Σ T Σ and taking b = 0 in the kernel ridge regression formula in (1.12).

Exploiting the close link between random feature maps (and thus random kernel matrices) and random weights NN models, we evaluate the asymptotic performance of the single-hidden-layer random NN model in Figure 1.3, in the regime where n, p, N → ∞ with p/n → c1 ∈ (0, ∞) and N/n → c2 ∈ (0, ∞), in the following contribution.

(J2) Cosme Louart, Zhenyu Liao, Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1190-1248, 2018.

By focusing on the randomness of the network weights W and considering the datatarget pair (X, Y) deterministic, we show that the asymptotic training MSE defined by

E train = 1 n n ∑ i=1 y i -β T σ(Wx i ) 2 = 1 n Y -β T Σ 2 F (1.19)
has an asymptotically deterministic behavior. A conjecture is also provided for the test MSE on an independent test set ( X, Ŷ) of size n:

E test = 1 n Ŷ -β T σ(W X) 2
F , which is later proved in [START_REF] Louart | A random matrix and concentration inequalities framework for neural networks analysis[END_REF] under additional assumptions on X and X. Both the training and test performances of the network depend on the dimensionality of the problem, and on the data as well as the activation function σ(•) through the "equivalent" kernel matrix

K = E w [σ(X T w)σ(w T X)]
(1.20)

for w ∼ N (0, I p ), which can be explicitly computed via an integration trick [START_REF] Christopher | Computing with infinite networks[END_REF] for most commonly used σ. This again confirms the strong connection between random weights NNs/random feature models and kernel methods.

Moreover, by (additionally) leveraging the stochastic nature of the data, we are able to dive deeper into the comprehension of (the nonlinearity for instance) in random NN models, with the help of our previous knowledge on random kernel matrices discussed Section 1.1.2. This consideration led to the following contribution. In a nutshell, we consider data independently drawn from a GMM (see Definition 1), and focus on the interplay between the nonlinear activation σ(•) and the data statistics (µ a , C a ). Since only the first and second order statistics are considered here, commonly used activations σ are naturally divided into the following three classes:

1. the mean-oriented activations, for which the information in covariance C a (asymptotically) does not appear in the equivalent kernel K in (1.20);

2. the covariance-oriented activations, which exploit only the information in C a with µ a discarded;

1.2. RANDOM KERNELS, RANDOM NEURAL NETWORKS AND BEYOND 3. the "balanced" activations, that utilize both first and second order statistics of the data.

A huge gain in classification performance can be observed by applying the nonlinear activation "adapted" to the data/task. For instance, EEG time series data often contain richer information in their covariance structures than the handwritten digits of MNIST, for which the differences in means between classes are more dominant.

In the case of the random feature-based kernel ridge regression in (1.18), we aim to minimize the square loss between the target and the model output. There exist many other loss functions with solutions significantly differing from one another. The square loss turns out to be a convenient choice since it often leads to closed-form solutions as in (1.18). In many other cases, it is hardly possible to obtain closed-form solutions. Indeed, most commonly used machine learning systems (as simple as logistic regression) arise from generic optimization problems that may only take implicit forms. The performance assessment of such implicit systems is, albeit more challenging, a necessary extension for the proposed RMT-based analysis framework, as shall be discussed in the following subsection.

Beyond the square loss: the empirical risk minimization framework

Other loss functions, rather than square loss, can also be used as optimization metrics. In classification applications, it is commonly considered more appropriate to learn a classifier β ∈ R p using the following cross-entropy loss

L(β) = - 1 n n ∑ i=1 y i log(σ(β T x i )) + (1 -y i ) log(1 -σ(β T x i )) (1.21)
with σ(t) = (1 + e -t ) -1 the logistic sigmoid function, for label y i ∈ {0, 1} and feature vectors x i ∈ R p . This leads to the logistic regression model that, unfortunately, does not admit an explicit solution. The fact that the resulting classifier β arises from an implicit optimization problem makes the performance evaluation technically more challenging, mainly due to the complex dependence between the learned parameter β and the training set {x i , y i } n i=1 : it is thus less direct to assess the statistical behavior of β, and this gets even worse with the logistic sigmoid nonlinearity σ. Despite all the technical difficulties mentioned above, it is nonetheless possible to pursue a stochastic description of β and consequently to evaluate the performance of, not only the logistic regression classifier, but also of any differentiable convex loss L that falls into the general empirical risk minimization framework, as briefly recalled below.

First note that the logistic sigmoid function is symmetric with respect to the labels {0, 1}: σ(t) + σ(-t) = 1. Equation (1.21) can thus be rewritten as the minimization of

1 n ∑ n i=1 L( ỹi β T x i ), for L(t) = log(1 + e -t
) and ỹi = (2y i -1) ∈ {-1, 1}. In fact, logistic regression is a special case of the general empirical risk minimization framework formulated as min

β∈R p 1 n n ∑ i=1 L( ỹi β T x i ) + λ 2 β 2 (1.22)
for some nonnegative convex loss function L : R → R and regularization factor λ ≥ 0. With the logistic loss L(t) = log(1 + e -t ) one gets the logistic regression formulation as in (1.21), while the least squares classifier (or ridge regressor) is obtained with the square loss L(t) = (t -1) 2 . Another popular choice is the exponential loss L(t) = e -t that is widely used in boosting algorithms [START_REF] Freund | A short introduction to boosting[END_REF].

CHAPTER 1. INTRODUCTION

The empirical minimization principle was first proposed in [START_REF] Vapnik | Principles of risk minimization for learning theory[END_REF] based on the following consideration: in a binary classification problem, given a training set of n data x i ∈ R p with associated label ỹi ∈ {-1, 1}, i = 1, . . . , n, one wishes to to "learn" a (soft) linear classifier β ∈ R p from the available training set such that the thresholded output sign(β T x i ) (the "hard" classifier) matches the corresponding label ỹi , i.e., ỹi = sign(β T x i ), for all data-label pair (x i , ỹi ) in the training set as well as for unseen x following the same distribution.

In pursuit of a good classifier β, a first selection for the loss function is the classification error (also known as the 0 -1 loss, in red below) on the given training set. Despite being the most natural choice, the minimization of the non-convex and non-smooth 0 -1 loss is unfortunately proved to be an NP-hard combinatorial optimization problem in the worst case [START_REF] Ben-David | On the difficulty of approximately maximizing agreements[END_REF]. In this regard, many convex surrogate losses are proposed, such as the (non-smooth) hinge loss L(t) = max(1t, 0) used in SVMs (in black), the logistic loss (in green) L(t) = log(1 + e -t ) used in logistic regression and the square loss L(t) = (t -1) 2 (in blue) for linear regression. See Figure 1.4 for an illustration of these different losses for classification.

Figure 1.4: Different loss functions for classification: 0 -1 loss (red), logistic loss (green), exponential loss (purple), square loss (blue) and hinge loss (black).

Within the empirical risk framework, the comparison between different designs of loss functions has been long discussed in the statistical learning literature [Vap92, RVC + 04, MSV09], mostly in the setting where the number of training data n largely exceeds their dimension p. It was shown in [RVC + 04] that, in the limit of large n with p fixed, all convex loss functions mentioned above yield the same Bayes optimal solution β * that minimizes the 0 -1 loss. Meanwhile, they differ in their rates of convergence (with respect to n) and in this respect, the non-smooth hinge loss is as efficient as the logistic loss, which is much better than the square loss.

The case where n is not much larger than p was not taken into consideration until very recently, mainly due to the fact that, instead of converging to its expectation as when n p, the learned parameter β remains random in the (comparably) large n, p limit and makes its statistical properties less tractable. Relying on ideas from random matrix theory, in [EKBB + 13, DM16] the authors managed to capture the stochastic behavior of the M-estimators popularly used in robust statistics that have a similar formulation as in (1.22), but with a loss of the type L(y iβ T x i ).

In a more recent line of works [START_REF] Sur | A modern maximumlikelihood theory for high-dimensional logistic regression[END_REF][START_REF] Emmanuel | The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression[END_REF], the authors considered the logistic regression model for i.i.d. Gaussian feature x i ∼ N (0, I p ) and showed that, in the regime of n, p → ∞ with p/n → c ∈ (0, ∞), the learned β is not only biased, but also has a greater variability than classically predicted [START_REF] Sur | A modern maximumlikelihood theory for high-dimensional logistic regression[END_REF]. Indeed, it was shown in [START_REF] Emmanuel | The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression[END_REF] that, without regularization (λ = 0 in (1.22)), there exists a sharp "phase transition" threshold for the ratio p/n above which the logistic regression solution almost surely does not exist. 4 The results presented above demonstrate the counterintuitive behaviors of not only logistic regression but more generally of the family of empirical risk minimization classifiers in (1.22), where n, p are both large and comparable. As such, a random matrix-based refinement is needed, to gain a better understanding of these statistical learning methods for large dimensional problems, to correct the "bias" induced by the fact that p ∼ n and to determine the (problem-dependent) optimal choice of the loss function.

We go beyond the logistic regression setting and consider any twice differentiable and convex loss function L in the following contributions. Built upon a symmetric two-class GMM (i.e., -µ 1 = µ 2 = µ and C 1 = C 2 = C) for the features, we provide, as in [EKBB + 13, DM16], an asymptotic stochastic description of the learned parameter β (that minimizes (1.22) with λ = 0)

β -β → 0, β ∼ N η 2τ β * , γ nτ I p
with β * = 2C -1 µ the optimal Bayes solution in this setting and constant parameters (η, γ, τ) ∈ R 3 determined by the loss L and the statistics µ, C, via a system of fixed point equations. In particular, we show that in separating two large dimensional Gaussians with the same covariance C and opposite means ±µ, the optimal loss (in the formulation of (1.22) with λ = 0) is the square loss, which always outperforms the maximum likelihood solution (logistic regression in this case), in the regime n, p → ∞ and p/n → c ∈ (0, ∞). This result is achieved by combining RMT techniques with a heuristic "leave-one-out" approach and will be discussed at length in Section 3.3.

Summary of Section 1.2

In this section, under the high dimensional setting n, p → ∞ with p/n → c ∈ (0, ∞) and a multivariate mixture model (Definition 1) for the input data, we discussed three closely related objects: the kernel matrices built from pairwise comparisons of data points, the random feature maps designed to approximate kernel matrices and the resulting random NN models. We examined the performances of these three models in minimizing the square loss and particularly focus on the role played by the nonlinearity therein 4 To make this clear, let us consider the classical example of completely separable training data (x i , ỹi ) of size n, namely, there exists a linear decision boundary b ∈ R p such that for i = 1, . . . , n, one has ỹi b T x i > 0. In this case, taking λ = 0 and L(t) = log(1 + e -t ), the minimization of (1.22) with iterative methods such as gradient descent leads to the solution β = αb with α → ∞, for any given n, p. Hence, the logistic regressor exists only when the data points "overlap". In the large n, p regime, not only the overlap between data points, but also a sufficient large n/p is indispensable for the existence of a logistic regression solution.

(kernel and activation functions). When more general losses are considered, the (extra) nonlinearity from the loss makes these learning systems less tractable and calls for more advanced tools.

We reviewed some recent advances in the understanding of these large dimensional learning systems and discussed very briefly the main technical approaches on which these theoretical results rely. As has been discussed respectively in Section 1.1.1 and 1.1.2, due to the high dimensional nature of the problem when n, p are comparably large, classical asymptotic statistics fail to predict the performance of neither sample covariance matrices, nor random kernel matrices.

Since random feature maps do no more than approximating kernel matrices, and simple random NNs are indeed random feature-based ridge regressions, all these objects of interest are not working in the expected way they were originally designed for. Consequently, we need a careful refinement of these learning methods, if we are not working with n 100p (see again our argument in Remark 1.2), which is almost always the case in practical machine learning applications.

In the topics addressed in this section, we hardly specified the underlying optimization method used to obtain the desired solution. As mentioned in Section 1.2.3, with the cross-entropy loss in (1.21), optimization methods such as gradient descent is needed to minimize the objective function. In the following section, we consider the "optimization" aspect of these learning systems, with a particular emphasis on NN models trained with gradient-based methods.

Training Neural Networks with Gradient Descent

Despite their rapidly growing list of successful applications in as diverse fields as computer vision [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], speech recognition [START_REF] Mohamed | Acoustic modeling using deep belief networks[END_REF] and natural language processing [START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF], our theoretical comprehension of DNNs, is developing at a much more modest pace.

Two salient features of these large neural networks are: i) despite being non-convex optimization problems, training DNN models with simple algorithms such as (stochastic) gradient descent seems always to be able to achieve minimal error and ii) even though they often have far more model parameters than the number of data they are trained on, some models still exhibit remarkably good generalization performance, while others generalize poorly in exactly the same setting [ZBH + 16]. A much-expected explanation of these phenomena would be the key to more powerful and reliable network structures.

In this section, we examine the training of NN models with gradient descent algorithms from a "dynamical" standpoint and consider the temporal evolution of both the network weights and the resulting performance. Starting from the toy model of a singlelayer linear and convex network (with a unique minimum), in Section 1.3.1 we focus on the gradient descent dynamics (GDDs) and provide a precise characterization of both the training and test performances, as a function of the training time, via a random matrixbased analysis.

When deeper networks (with the number of hidden layers H ≥ 1) are considered, the optimization of such networks becomes non-convex and more challenging. In Section 1.3.2 we review some basic notions of optimization (with an emphasis on non-convex problems), and discuss some particularly interesting situations in which the underlying non-convex optimization becomes tractable and for which one can always find an optimal solution with simple gradient-based methods, sometimes even in linear time.

A random matrix viewpoint

In [START_REF] Andrew M Saxe | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[END_REF] the authors proposed to study the "dynamics" of gradient descent algorithms in linear DNN models. Despite the linearity of their input-output map, linear networks have highly nonlinear training error behaviors as a function of the time when trained with gradient descent methods. This "learning dynamic" of linear NNs was shown to be closely connected to the eigenvalues of the input-output covariance/correlation matrix. As further justification for their study, empirical evidence was provided showing that the learning dynamics of nonlinear NN, despite being mathematically less tractable due to the (entry-wise) nonlinear activations, exhibit similar behavior to the linear case.

More recently in [START_REF] Madhu | High-dimensional dynamics of generalization error in neural networks[END_REF], under a teacher-student network setting where a student network tries to "learn" from the noisy samples generated by a predefined teacher network, the authors investigated both the training and test errors, as a function of the gradient descent training time. More precisely, assume a training data matrix X = x 1 , . . . , x n ∈ R p×n with associated target/label y T = y 1 , . . . , y n ∈ R n . With the training pair (X, y), a weight vector w ∈ R p is learned using "full-batch" gradient descent to minimize the regularized square loss

L(w) = 1 2n y -X T w 2 + λ 2 w 2 .
(1.23)

The gradient of L with respect to w is thus given by ∇ w L(w) = -1 n X(y -X T w) + λw so that with a small gradient descent step (or, learning rate) α, we obtain, by performing a continuous-time approximation, the following differential equation

dw(t) dt = -α∇ w L(w) = α n X(y -X T w) -αλw
the solution of which is explicitly given as

w(t) = e -αt( 1 n XX T +λI p ) w 0 + I p -e -αt( 1 n XX T +λI p ) w LS (1.24)
where we denote w LS = ( 1 n XX T + λI p ) -1 1 n Xy the classical ridge regression solution with regularization parameter λ ≥ 0 and w 0 = w(t = 0) the initialization of gradient descent. We recall the definition of the exponential of a symmetric matrix A

e A = ∞ ∑ k=0 1 k! A k = V A e Λ A V T A with the eigendecomposition of A = V A Λ A V T
A and e Λ A the diagonal matrix with elements equal to the exponential of the diagonal elements of Λ A .

Equation (1.24) tells us that the temporal evolution of both the weight vector w(t) and the network performance5 are in fact functionals of the (regularized) sample covariance matrix 1 n XX T + λI p , the evaluation of which is made possible with our proposed resolvent-based RMT techniques and led to the following contribution.

(C5) Zhenyu Liao and Romain Couillet. The dynamics of learning: a random matrix approach. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3072-3081. PMLR, 2018.

CHAPTER 1. INTRODUCTION

Different from all aforementioned works [START_REF] Andrew M Saxe | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[END_REF][START_REF] Madhu | High-dimensional dynamics of generalization error in neural networks[END_REF] where no structure is considered within the data (i.e., data indeed come from the same distribution class), in this contribution, we instead study the GDD of a linear regression model in separating a twoclass GMM (Definition 1) with opposite means ±µ and identity covariance C 1 = C 2 = I p that are assigned labels -1 and 1, respectively. We propose a general RMT-based framework to characterize the learning dynamics of this simple model, in the regime where n, p → ∞ with p/n → c ∈ (0, ∞) and extend the analysis in [START_REF] Madhu | High-dimensional dynamics of generalization error in neural networks[END_REF] to a classification setting for more natural "structural" data. Similar objectives cannot be achieved within the framework presented in [START_REF] Madhu | High-dimensional dynamics of generalization error in neural networks[END_REF], which conveys more practical interest to our results and the proposed analysis framework.

As an important special case, by taking the training time t → ∞ and λ = 0 in (1.24), one obtains the "over-trained" model which is the least-squares solution. In this case, the classification error rate on a new datum x can be consequently shown to be

P(sign( xT w LS ) = ŷ) = Q µ 2 1 -min(c, c -1 ) µ 2 + c
with Q(x) the Gaussian tail function and admits a singularity at c = 1; see Figure 1.5 with µ 2 = 5. This sharp drop in performance around c = 1 can be alleviated, if appropriate regularization techniques such as early stopping are adopted. The fact that the prediction risk blows up around c ≡ lim p/n = 1 in Figure 1.5 indicates the existence of a surprising "double-descent" phenomenon, where one observes, for a given dimension p that the performance of w LS first gets worse as the sample size n grows large (when n < p), which totally collapses to random guess at the point n = p and starts to increases rapidly when n > p. Moreover, in this simple model, the number of model parameters N coincides with the data dimension p, so that for a fixed sample size n, having a more complicated system (with larger N) can lead to a lower prediction risk in the "over-parameterized" regime where N > n. This suggests that one must reconsider the golden rule of the bias-variance tradeoff in classical statistical learning theory [START_REF] Friedman | The elements of statistical learning[END_REF].

A geometric approach

From an optimization viewpoint, understanding DNN models is challenging mainly due to the cascading layers with entry-wise nonlinearity between them. This construction 1.3. TRAINING NEURAL NETWORKS WITH GRADIENT DESCENT 29 naturally gives rise to non-convex optimization problems that are seemingly intractable. In general, finding the global minimum of a non-convex function is, in the worst case, an NP-complete problem [START_REF] Katta | Some NP-complete problems in quadratic and nonlinear programming[END_REF] and it is unfortunately the case for NN models [START_REF] Blum | Training a 3-node neural network is NP-complete[END_REF]. Yet, many non-convex optimization problems including tensor decomposition, matrix completion, dictionary learning, matrix sensing, and phase retrieval are known to have well-behaved optimization landscapes: under some technical conditions, all local minima are also global [START_REF] Ge | No spurious local minima in nonconvex low rank problems: A unified geometric analysis[END_REF][START_REF] Chi | Nonconvex optimization meets low-rank matrix factorization: An overview[END_REF]. This interesting global geometric property, together with some local considerations, e.g., any saddle point has a negative directional curvature (i.e., has at least one strictly negative eigenvalue in its Hessian, which makes it possible to continue to decrease the objective function locally), allows for solving efficiently these problems with basic optimization algorithms such as (stochastic) gradient descent. We will discuss this geometric aspect of optimizing NN models in this subsection.

More generally, consider the following minimization problem (which is a more general form of (1.22) with the possible regularization term included in L)

min θ∈R N 1 n n ∑ i=1 L(h(x i , θ), y i ) (1.25)
for a given training set {x i , y i } n i=1 of size n. The loss function L is often convex (with respect to both h(x i , θ) and y i ), for example variants of those in Figure 1.4. Yet, depending on the application of interest, the system model h(x i , θ) can be highly nonlinear, nonconvex (with respect to the model parameter θ) and even non-smooth, which is more challenging from an optimization standpoint.

Stationary points: a local image. In the case of large-scale problems (N n 1), gradient-based methods are often among the most efficient, if not the only, ways to solve (1.25), despite the existence of more advanced optimization methods in the general differentiable programming context. For instance, some second-order (or Hessian-based) optimization algorithms with Hessian information (such as the Newton's method) is capable of finding second-order stationary points (see definition below) [START_REF] Nesterov | Cubic regularization of newton method and its global performance[END_REF], but they typically require to compute at every iteration the inverse or spectral decomposition of the full Hessian matrix of dimension N × N, which is computationally unaffordable for N large.

Solving non-convex problems without higher order information (e.g., Hessian) is challenging, since gradient-based methods stop moving forward as long as the gradient norm ∇ θ L(•) is small, which can be far away from the desired global minimizer of the non-convex (with respect to the model parameter θ) objective function L(•) in (1.25). To make this more clear, let us introduce the following definitions.

Definition 2 (Stationary points). Denote ∇L(•) and ∇L 2 (•) respectively the gradient and the Hessian of the twice differentiable objective function L with respect to the model parameter θ. We say θ * is a first-order stationary point (or critical point) of L if

∇L(θ * ) = 0 ⇔ ∇L(θ * ) = 0
In practice, the notion of first-order stationarity can be extended to include some numeric tolerance : we say θ * is an -first-order stationary point if ∇L(θ * ) ≤ for some > 0.

First-order stationary points can be divided into the following three categories (see Figure 1.6 for examples of each type), depending on the local behavior of L in the neighborhood of θ * :

1. local minima: we say θ * is a local minimum if there exists > 0 such that for all θ in the -neighborhood of θ * (e.g., θθ * ≤ ) we have L(θ * ) ≤ L(θ), as in Figure 1.6a;

CHAPTER 1. INTRODUCTION 2.
local maxima: we say θ * is a local maximum if there exists > 0 such that for all θ in the -neighborhood of θ * we have L(θ * ) ≥ L(θ), as in Figure 1.6b;

3. saddle points: first-order stationary points that are neither minima nor maxima, as in Figure 1.6c. Whether a given first-order stationary point is a local minimum, a local maximum or a saddle point is closely related to the associated Hessian matrix ∇L 2 (θ * ) ∈ R N×N , more precisely 1. If θ * is a local minimum, the Hessian must be positive semidefinite, i.e., λ min (∇L 2 (θ * )) ≥ 0. Conversely, if the Hessian is positive definite, then θ * must be a local minimum.

2. If θ * is a local maximum, the Hessian must be negative semidefinite, i.e., λ max (∇L 2 (θ * )) ≤ 0. Conversely, if the Hessian is negative definite, then θ * must be a local maximum.

3. Consequently, if the associated Hessian is neither positive nor negative semidefinite, or equivalently, has both positive and negative eigenvalues, θ * must be a saddle point.

As such, by examining the smallest or largest eigenvalue of the associated Hessian matrix, one can sometimes determine weather a first-order stationary point is a local minimum or maximum: this is referred to as the "second partial derivative test". It is worth pointing out that, if the associated Hessian is degenerate, i.e., det(∇L 2 (θ * )) = 0 and thus admits (at least one) zero eigenvalue, the second partial derivative is inconclusive: in the case of the first and second item listed above, having a zero eigenvalue rules out the possibility of both λ min (∇L 2 (θ * )) > 0 and λ max (∇L 2 (θ * )) < 0, but θ * can still be a local extremum or a saddle point. Aiming at minimizing the objective function L, we say a first-order stationary point θ * is a second-order stationary point if

λ min (∇L 2 (θ * )) ≥ 0
which is indeed a necessary but not sufficient condition for a local minimum since it does not rule out the possibility of being a saddle point with degenerate Hessian (such that λ min (∇L 2 (θ * )) = 0). For all objects defined above, their -tolerance versions can be similarly defined by replacing zero with some > 0.

More generally, a stationary point with degenerate Hessian often demands higher (≥ 3) order information to decide to which type it belongs. If all eigenvalues of the Hessian has relatively small amplitude, the local "landscape" of the objective function L is very "flat" and taking a small step in no matter which direction results in a very slight change in the value of L. Therefore, it often takes a long time, if still possible, to escape from this kind of regions.

Let us consider some concrete examples in 2D, i.e., θ ∈ R 2 . In Figure 1.7a we plot the function L(θ 1 , θ 2 ) = (θ 1θ 2 ) 2 , the set of first-order stationary points of which is the line θ 1 = θ 2 , with the corresponding Hessian given by

∇ 2 L(θ 1 , θ 2 ) =   ∂ 2 L ∂θ 2 1 ∂ 2 L ∂θ 1 ∂θ 2 ∂ 2 L ∂θ 2 ∂θ 1 ∂ 2 L ∂θ 2 2   = 2 -2 -2 2
that has eigenvalues (constantly) equal to λ 1 = 0 and λ 2 = 4. In this example, albeit the degeneration of the Hessian (and thus the invariance of the objective function when moving in the direction of the eigenvector6 associated to the zero eigenvalue), all stationary points are indeed local minima. To institute a comparison, we plot in Figure 1.7b a more "classical" saddle point (at (0, 0)) of the objective

L(θ 1 , θ 2 ) = θ 2 1 -θ 2 2
, the Hessian of which admits two eigenvalues λ 1 = -2 and λ 2 = 2, so that the saddle point is "instable" and there is a way "out" locally (to continue to decrease the value of L) due to the existence of the negative eigenvalue Things become more complicated in the example of Figure 1.7c with

λ 1 = -2. -2 0 2 -2 0 2 θ 1 θ 2 (a) L(θ 1 , θ 2 ) = (θ 1 -θ 2 ) 2 . -2 0 2 -2 0 2 θ 1 θ 2 (b) L(θ 1 , θ 2 ) = θ 2 1 -θ 2 2 . 0 1 -1 -0.5 0 θ 1 θ 2 (c) L = (θ 1 + θ 2 )(θ 1 θ 2 + θ 1 θ 2 2 ).
L(θ 1 , θ 2 ) = (θ 1 + θ 2 )(θ 1 θ 2 + θ 1 θ 2
2 ) that admits the following four first-order stationary points z 1 : (0, 0); z 2 : (0, -1); z 3 : (1, -1); z 4 : (3/8, -3/4)

with Hessian eigenvalues respectively given as

z 1 : λ 1 = λ 2 = 0; z 2 : λ 1 = -1, λ 2 = 1; z 3 : λ 1 = -1292/305, λ 2 = 305/1292; z 4 : λ 1 = -1743/1436, λ 2 = -450/1351.
Therefore, z 2 and z 3 are saddle points with both positive and negative eigenvalues, z 4 is a local maximum; while for z 1 the second derivative test is not enough and one must use higher order test to examine the behavior of L at this point.

To avoid the difficulty in the case of degenerate Hessians, in [START_REF] Ge | Escaping from saddle points-online stochastic gradient for tensor decomposition[END_REF] the authors introduced the following "strict saddle" property.

Definition 3 (Strict saddle, Definition 4 in [GHJY15]

). A twice differentiable function is said to have strict saddles, if, apart from the local minima, all its first-order stationary points have at least one negative eigenvalue for the associated Hessian, i.e., λ min (∇ 2 L(θ * )) < 0. A more numerically robust version can be similarly defined as λ min (∇ 2 L(θ * )) <for some > 0.

Having "strict saddles" makes it possible to make some progress to (continue to) decrease the objective function, at least locally. Global guarantee of non-convex optimization. Until now, we have been focusing on the local image of the optimization landscape, which is unfortunately far from our initial goal of achieving a global minimum. Indeed, if the "strict saddle" property is satisfied, it is possible for gradient-based algorithm to (locally) escape from saddle points and continue to decrease the loss function. Nonetheless, it may still get stuck in a local minimum. Different from the convex case in Figure 1.7a, where the convex (but not strictly convex) objective function L(θ 1 , θ 2 ) = (θ 1θ 2 ) 2 has a connected region of local (and global) minima: θ 1 = θ 2 , a non-convex loss function L may have many "isolated" local minimum regions that are separated by higher values of the loss L. Among these regions, some are desired global minima that reach the smallest possible loss L min and some are "bad" local minima that have much greater loss values than L min . There may also exist "good" local minima with a relatively small loss but still slightly larger than L min and can be considered satisfying solutions of the minimization, as illustrated in Figure 1.8.

The isolation between local minimum regions creates many "local valleys" of the loss landscape. Very interestingly, there are both empirical evidence [DVSH18, GIP + 18] and theoretical arguments [START_REF] Daniel | Topology and geometry of halfrectified network optimization[END_REF] stating that in modern DNNs, most of these "local valleys" are essentially "connected": i.e., it is possible to construct continuous paths that connect all these local minimum regions, along which the loss L remains very close to L min , as shown in Figure 1.9 (which could be considered a possible 2D representation of Figure 1.8). Consider now the (more ideal) case that all local minima (connected or isolated) have exactly the same loss value. This means that for the optimization problem under study,

TRAINING NEURAL NETWORKS WITH GRADIENT DESCENT

all local minima are global. This rules out the presence of "bad" local minima which, together with the strict saddle points property in Definition 3, makes it possible for simple gradient-based algorithms (such as GD or SGD) to achieve a global convergence with a guaranteed rate.

To sum up, the key to handle many non-convex optimization problems is to have both 1. a nice local property: for example the "strict saddle" property in Definition 3; 2. some global guarantee: for instance the "all local minima are global" property.

Although seemingly restrictive, both properties are actually satisfied (under some dimensionality condition and at least with high probability) by many non-convex optimization problems of considerable practical interest: many low rank problems including matrix sensing, phase retrieval, matrix completion, robust PCA and many others [START_REF] Ge | No spurious local minima in nonconvex low rank problems: A unified geometric analysis[END_REF][START_REF] Chi | Nonconvex optimization meets low-rank matrix factorization: An overview[END_REF].

In the particular case of neural networks, it was shown in [START_REF] Baldi | Neural networks and principal component analysis: Learning from examples without local minima[END_REF] that for singlehidden-layer linear NNs with square loss and under some dimensionality condition, all local minima are essentially global and there is no local maximum (the rest are all saddle points). This result was then extended to deep linear NNs in [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF] in which the author showed that the same conclusion holds for DNNs with arbitrary H ≥ 1 hidden layers. Perhaps more importantly, a necessary condition (on the rank on the weight matrix product) was given in [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF] to ensure the associated Hessian have at least one negative eigenvalue [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF]Theorem 2.3]. This implies the existence of saddle points with positive semidefinite Hessian, for which higher order derivatives will be indispensable for a thorough understanding of these (first-order) stationary points. More precisely, single-hidden-layer (H = 1) linear NN models are proven to have strict saddles, which is used for instance in [START_REF] Lee | Gradient descent only converges to minimizers[END_REF] to show almost sure (with respect to the initialization in a Lebesgue measure sense) convergence to global minima in this case; while for H ≥ 2 a counterexample was provided in [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF].

We focus in this manuscript on the gradient flow, for linear NN models in the following contribution.

(J4) Yacine Chitour, Zhenyu Liao, and Romain Couillet. A geometric approach of gradient descent algorithms in neural networks. (Submitted to) Journal of Differential Equations, 2019.

Based on a cornerstone "invariance" in the parameter/network weights space induced by the network cascading structure, we prove the existence for all time of trajectories associated with gradient descent algorithms in linear networks with an arbitrary number of layers and the convergence of these trajectories to first order stationary points of the loss function. We also prove the exponential/linear convergence of trajectories under an extra condition on their initializations. We propose alternative proof of the "almost sure convergence to global minima" fact in single-hidden-layer linear NN model, which holds true without the technical and unrealistic assumptions demanded in [START_REF] Lee | Gradient descent only converges to minimizers[END_REF].

Summary of Section 1.3

In this section, we discussed the optimization perspective of large dimensional machine learning problems, and in particular, NN models trained with gradient descent methods. In Section 1.3.1, inspired by recent advances in [SMG13, AS17], we introduced a CHAPTER 1. INTRODUCTION RMT-based analysis framework to investigate the GDDs in a linear convex NN model under a GMM (see Definition 1) for the input feature. We found a huge gap between the training and test performances of the least-square solution when the number of parameters gets close to the number of training data, that can be efficiently reduced with the early stopping strategy. This suggests that the use of gradient descent plays an important role in regularizing the least-square solution, and sheds new light on the training of large dimensional learning systems.

In Section 1.3.2 we recalled the preliminary local descriptions of (first-order) stationary points in the general context of differentiable programming, with a particular focus on non-convex optimization. We saw that, if both conditions of "strict saddles" and "all local are global" are met, simple gradient-based algorithms such as GD or SGD can benefit from a global convergence to global minima, from almost all initializations. It is moreover possible to speed up the convergence, if the underlying non-convex problem is known to have some nice (statistical) structure, by properly initializing (often with spectral methods) the descent algorithm within some locally convex valley: we will come back to this point in Chapter 5.

Outline and Contributions

As stated in the previous sections, this manuscript covers both a "structural" viewpoint of large random kernel matrices and neural networks by considering their eigenspectrum behavior, as well as the "dynamical" optimization aspect of learning in neural networks by means of the corresponding gradient descent dynamics. As a first major contribution, we study the eigenspectrum properties of these large nonlinear learning systems, under a multivariate mixture model (see Definition 1) for the input feature, so as to examine the subtle yet crucial interplay between the nonlinearity (of the kernel function f , the nonlinear activation σ and the loss function L), the feature statistics (limited to first and second here but is envisioned to go well beyond, see our arguments in Remark 1.3 and 2.5 later in Section 2.2.2) as well as the problem dimensionality. The second main contribution is the characterization of the gradient descent dynamics (GDDs) in learning both zero-and single-hidden-layer linear NN models, which opens the door to a more general RMT-based consideration of non-convex optimization problems.

The remainder of the manuscript is organized as follows. In Chapter 2 we introduce the mathematical framework of random matrix theory, with helpful preliminaries on linear algebra, probability theory, and complex analysis. In Chapter 3, we start the discussion of large random kernel matrices with the concrete example of kernel ridge regression, or equivalently least squares support vector machines (LS-SVMs), which serves as the cornerstone of subsequent models such as random feature-based methods and random NNs. We will also briefly mention the additional tool needed in handling implicit (convex) optimization problems (e.g., logistic regression) that are technically more involved. Then in Chapter 4, random matrix techniques are combined with ideas from dynamical systems to study the gradient descent dynamics in training simple NNs, first from a statistical standpoint without hidden layer (i.e., linear regression model) and then with a "geometric" approach to deal with the extra hidden layers in linear networks. In Chapter 5 we provide a quick summary of this manuscript, and perhaps more importantly, discuss some possible limitations of the presented results and eventually some future perspectives within the general context of large dimensional machine learning.

We start with the analysis of the classification performance of kernel ridge regression 1.4. OUTLINE AND CONTRIBUTIONS (or LS-SVM) in Section 3.1.1, which helps understand the counterintuitive behavior of kernel matrices for large dimensional data discussed in Section 1.1.2. Based on a GMM for the input data (see again Definition 1), we show that, for any locally twice differentiable function f , the soft decision function of the kernel ridge regression is asymptotically normally distributed, the mean and variance of which depend on the statistics of mixture model (µ a , C a ), the dimensionality and the function f in a local manner. This theoretical result leads to a novel understanding of the negative impact of imbalanced data in LS-SVM classification and shows an unexpectedly close match when applied on popular image classification datasets. This finding was first reported in The theoretical analysis of LS-SVM presented above are essentially built upon a local expansion of the nonlinearity, which is only possible when the similarity measures under study ( x ix j 2 /p or x T i x j /p) establish a "concentration" around a single point of the nonlinear function f . With a more natural √ p scaling, one results in the inner- product kernel that takes the form

K ij = f (x T i x j / √ p)/ √ p.
Very interestingly, the eigen- spectrum of K is shown to only depend on three scalar parameters of the possibly nondifferentiable f . This finding is discussed at length in Section 3.1.2 and reported in (C2) Zhenyu Liao and Romain Couillet. Inner-product kernels are asymptotically equivalent to binary discrete kernels, 2019.

Section 3.2 begins with the discussion on single-hidden-layer random weights NN, or equivalently the random feature-based ridge regression in Figure 1.3. For comparably large n, p, N, we show that for all dimension-free Lipschitz nonlinear activation σ(•), the random Gram matrix G ≡ 1 n Σ T Σ has an asymptotically tractable behavior, that depends solely on σ via the associated underlying kernel K ≡ E w [σ(X T w)σ(w T X)] for w ∼ N (0, I p ). As a consequence, the training MSE can be shown to have an asymptotically (data-dependent) deterministic behavior, with a conjecture provided for the test MSE (and confirmed later in [START_REF] Louart | A random matrix and concentration inequalities framework for neural networks analysis[END_REF]). This work, discussed in more detail in Section 3.2.1, led to the following publication.

(J2) Cosme Louart, Zhenyu Liao, Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1190-1248, 2018.

In pursuit of a more explicit recognition of the mechanism of different nonlinear activations, we examine the eigenspectrum of the underlying kernel K, by additionally leveraging the stochastic nature of the data (that are assumed to follow a GMM). We show that, depending on whether the first order (µ a ), or second order (C a ) information or both, are conserved after the application of both the random weights W and the entrywise nonlinearity σ(•). As briefly discussed in Section 1.2.2, commonly used activations are divided into three categories of mean-oriented, covariance-oriented and balanced. This result, discussed in Section 3.2.2, led to the following contribution. In Section 3.3 we consider the more generic (and more involved) convex losses (rather than the squares loss treated in Section 3.2.1 that conducts to explicit solutions) in a mixture classification context and focus mainly on the technical ingredients needed to handle implicit and highly dependent learning systems. This work was reported in ICASSP 2019 with a journal version recently submitted to IEEE Transactions on Signal Processing. As regards gradient decent dynamics (GDDs) in NN models, we begin in Section 4.1 with the discussion on a statistical approach of evaluating the GDD in simple neural networks. Based on a toy mixture model for the input feature, we retrieve, in the high dimensional regime, the (asymptotic) training and test risks of a linear regression model, as a function of the GD training time. From a methodological standpoint, we introduce a RMT-based framework (Cauchy's integral formula + deterministic equivalent) to examine GDDs of simple NN models, which can be reexpressed as some functional of the feature sample covariance matrix. As a byproduct, we prove for the first time the existence of a "phase-transition" singularity in Figure 1.5, in a high dimensional classification context. This result was first announced in (C5) Zhenyu Liao and Romain Couillet. The dynamics of learning: a random matrix approach. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3072-3081. PMLR, 2018.

In Section 4.2 we consider a geometric aspect of GDDs in single-hidden-layer linear NN models. By precisely evaluating the (union of) basin of attraction of all saddle points, we show that almost all initializations lead to an (equivalent) global minimum for the simplest gradient descent algorithm. Most importantly, this is true without additional assumptions on these stationary points (e.g., on the gradient norms during descent or their Hessians). Indeed, this "global convergence to global minima" conclusion essentially holds due to a key invariant structure (in the network weight space) during the gradient descent procedure imposed by the network cascading structure. As a practical outcome, we also derive critical initialization schemes with exponential convergence rate. These results are summarized in (J4) Yacine Chitour, Zhenyu Liao, and Romain Couillet. A geometric approach of gradient descent algorithms in neural networks. 2019.

now under review for Journal of Differentiable Equation, which concludes the Chapter 4.

In Chapter 5 we draw a speedy conclusion of the manuscript and mention some possible limitations of the presented theoretical results, which are admittedly below the performance of state-of-the-art models in modern machine learning problems today. We will also discuss some possible future research directions, in the continuation of our preliminary findings presented here.

Chapter 2

Mathematical Background: Random Matrix Theory

Random matrix theory, at its inception, primarily dealt with the eigenvalue distribution (also referred to as spectral measure) of large random matrices. One of the key technical tools to study these measures is the Stieltjes transform, often presented as the central object of the theory [START_REF] Bai | Spectral analysis of large dimensional random matrices[END_REF].

But signal processing and machine learning alike are more fundamentally interested in subspaces and eigenvectors (which often carry the structural data information) than in eigenvalues of random matrices. Subspace or spectral methods, such as principal component analysis (PCA) [START_REF] Wold | Principal component analysis[END_REF], spectral clustering [START_REF] Andrew | On spectral clustering: Analysis and an algorithm[END_REF] and some semi-supervised learning techniques [START_REF] Xiaojin | Semi-supervised learning literature survey[END_REF] are built directly upon the eigenspace spanned by the several top eigenvectors.

Consequently, beyond the Stieltjes transform, a more general object, the resolvent of large random matrices will constitute the cornerstone of this manuscript. The resolvent of a matrix gives access to its spectral measure, to the location of its isolated eigenvalues, to the statistical behavior of their associated eigenvectors when random, and consequently provides an entry-door to the performance analysis of numerous learning methods.

Fundamental Objects

The resolvent

We first introduce the resolvent of a matrix.

Definition 4 (Resolvent). For a symmetric matrix M ∈ R n×n , the resolvent Q M (z) of M is defined, for z ∈ C not an eigenvalue of M, as Q M (z) ≡ (M -zI n ) -1 (2.1)
which is also denoted Q when there is not ambiguity.

Although our focus here will exclusively be on resolvents of (random) matrices, it must be noted that the resolvent operator is in fact a very classical tool in the analysis of linear operators in general Hilbert spaces [START_REF] Ilich | Theory of linear operators in Hilbert space[END_REF] as well as in monotone operator theory of importance to modern convex optimization theory [START_REF] Heinz | Convex analysis and monotone operator theory in Hilbert spaces[END_REF].

38

CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY

Spectral measure and the Stieltjes transform

The first use of the resolvent Q M is in its relation to the empirical spectral measure µ M of a square matrix M, through the Stieltjes transform m µ M , which we all define next. Definition 5 (Empirical spectral measure). For a symmetric matrix M ∈ R n×n , the spectral measure or empirical spectral measure or empirical spectral distribution µ M of M is defined as the normalized counting measure of the eigenvalues λ 1 (M), . . . , λ n (M) of M,

µ M (x) ≡ 1 n n ∑ i=1 δ λ i (M) (x). (2.2)
Since µ M (x) ≥ 0 for all x ∈ R and R µ M (x)dx = 1, the spectral measure µ M of a matrix M ∈ R n×n (random or not) is a probability measure. For (probability) measures, we can define their associated Stieltjes transforms as follows.

Definition 6 (Stieltjes transform). For a real probability measure µ with support supp(µ), the Stieltjes transform m µ (z) is defined, for all z ∈ C \ supp(µ), as

m µ (z) ≡ R 1 t -z dµ(t).
(2.

3)

The Stieltjes transform m µ has numerous interesting properties: it is complex analytic on its domain of definition

C \ supp(µ), it is bounded by |m µ (z)| dist(z, supp(µ)) ≤ 1, it satisfies [z] > 0 ⇒ [m(z)] > 0,

and it is an increasing function on all connected components of its restriction

to R \ supp(µ) (since m µ (x) = R (t -x) -2 dt > 0) with lim x→±∞ m µ (x) = 0 if supp(µ) is bounded.
As a transform, m µ admits an inverse formula to recover µ, as per the following result. (2.4)

Besides, if µ admits a density f at x, i.e., µ(x) is differentiable in a neighborhood of x and

lim →0 (2 ) -1 µ([x -, x + ]) = f (x) then f (x) = 1 π lim y↓0 m µ (x + ıy) . (2.5)
Finally, if µ has an isolated mass at x, then

µ({x}) = - 1 π lim y↓0 ıym µ (x + ıy). (2.6) Proof. Since | y (t-x) 2 +y 2 | ≤ 1 y for y > 0, by Fubini's theorem, 1 π b a m µ (x + ıy) dx = 1 π b a R y (t -x) 2 + y 2 dµ(t) dx = 1 π R b a y (t -x) 2 + y 2 dx dµ(t) = 1 π R arctan b -t y -arctan a -t y dµ(t).
As y ↓ 0, the difference in brackets converges either to ±π or 0 depending on the relative position of a, b, t. By the dominated convergence theorem, limits and integrals can be exchanged, and the limit, as

y ↓ 0, is R 1 [a,b] dµ(t) = µ([a, b]).
The important relation between the spectral measure of M ∈ R n×n , the Stieltjes transform m µ M (z) and the resolvent Q M lies in the fact that

m µ M (z) = 1 n n ∑ i=1 R δ λ i (M) (t) t -z = 1 n n ∑ i=1 1 λ i (M) -z = 1 n tr(Q M ).
(2.7)

Combining inverse Stieltjes transform and the relation above thus provides a link between Q M and the eigenvalue distribution of M. While seemingly contorted, this link is in general the only efficient way to study the spectral measure of large dimensional random matrices M. Remark 2.1 (Resolvent as a matrix-valued Stieltjes transform). As proposed in [START_REF] Hachem | Deterministic equivalents for certain functionals of large random matrices[END_REF], it can be convenient to extrapolate Definition 6 of Stieltjes transforms to n × n matrix-valued positive measures M(dt),1 in which case Equation (2.7) can be generalized as

Q M (z) = R M(dt) t -z = U diag 1 λ i (M) -z n i=1 U T
where we used the spectral decomposition M = U diag{λ i (M)} n i=1 U T . In particular, Q M (z) enjoys similar properties as Stieltjes transforms of real-valued measures:

Q M (z) ≤ dist(z, supp(µ M )) -1 , and x → Q M (x) for x ∈ R \ supp(µ
) is an increasing matrix-valued function with respect to symmetric matrix partial ordering (i.e., A B whenever z T (A -B)z ≥ 0 for all z).

Cauchy's integral, linear eigenvalue functionals, and eigenspaces

Being complex analytic, the resolvent Q M can be manipulated using advanced tools from complex analysis. Of particular interest to spectrum-based machine learning methods is the relation between the resolvent and Cauchy's integral theorem.

Theorem 2.2 (Cauchy's integral formula). For Γ ∈ C a positively (i.e., counterclockwise) oriented simple closed curve and a complex function f (z) analytic in a region containing Γ and its interior, then

1) if z 0 ∈ C is enclosed by Γ, f (z 0 ) = -1 2πı Γ f (z) z 0 -z dz; 2) if not, 1 2πı Γ f (z) z-z 0 dz = 0.
This result provides an immediate link between the linear functionals of the eigenvalues of M and the Stieltjes transform through

1 n n ∑ i=1 f (λ i (M)) = - 1 2πın Γ f (z) tr(Q M (z))dz = - 1 2πı Γ f (z)m µ M (z)dz
for all f complex analytic in a compact neighborhood of supp(µ M ), by choosing the contour Γ to enclose supp(µ M ) (i.e., all the λ i (M)'s). More generally,

1 n ∑ λ i (M)∈Γ • f (λ i (M)) = - 1 2πı Γ f (z)m µ M (z)dz CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY for Γ • the interior of contour Γ.
Another quantity of interest relates to eigenvectors and eigenspaces. Decomposing the symmetric matrix M = UΛU T in its spectral decomposition with U = [u 1 , . . . , u n ] ∈ R n×n and Λ = diag(λ 1 (M), . . . , λ n (M)), we have

Q M (z) = n ∑ i=1 u i u T i λ i (M) -z
and thus the direct access to the i-th eigenvector of M through

u i u T i = - 1 2πı Γ λ i (M) Q M (z)dz for Γ λ i (M) a contour circling around λ i (M) only (if λ i (M) is of unit multiplicity). More generally, U f (Λ; Γ)U T = - 1 2πı Γ f (z)Q M (z)dz
for f analytic in a neighborhood of Γ and its interior and

f (Λ; Γ) = diag({ f (λ i (M))1 λ i (M)∈Γ • } n i=1
). Of interest in this manuscript will be the projection of the individual eigenvectors u i of M onto a deterministic eigenvector v. In particular, from the above,

|v T u i | 2 = - 1 2πı Γ λ i (M) v T Q M (z)vdz.
It is important to note that the resolvent provides access to scalar observations of the eigenstructure of M through linear functionals of the resolvent M, i.e., the scalar observa-

tions 1 n ∑ i f (λ i (M)) and |v T u i | accessible from tr Q M and v T Q M v, respectively.

Deterministic and random equivalents

This manuscript is concerned with the situation where M is a large dimensional random matrix, the eigenvalues and eigenvectors of which need be related to the statistical nature of the model design of M.

In the early days of random matrix theory, the main focus was on the limiting spectral measure of M, that is the characterization of a certain "limit" to the spectral measure µ M of M as the size of M increases. To this purpose, the natural approach is to study the random Stieltjes transform m µ M (z) and to show that is admits a deterministic limit (in probability or almost surely) m(z). However, this method shows strong limitations today: i) it supposes that such a limit does exist, therefore restricting the study to very isotropic models for M and ii) it only quantifies tr Q M (through the Stieltjes transform), thereby discarding all subspace information about M carried in Q M (as a consequence, a further study of the eigenvectors of M requires a complete rework).

To avoid these limitations, modern random matrix theory uses the notion of deterministic equivalents which are non-asymptotic deterministic matrices having (in probability or almost surely) asymptotically the same scalar observations as the random ones.

Definition 7 (Deterministic Equivalent). We say that Q ∈ R n×n is a deterministic equivalent for the symmetric random matrix Q ∈ R n×n if, for (sequences of) deterministic matrix A ∈ R n×n and vectors a, b ∈ R n of unit norms (operator and Euclidean, respectively), we have, as n → ∞,

1 n tr A(Q -Q) → 0, a T (Q -Q)b → 0
where the convergence is either in probability or almost surely.

This definition has the advantage to bring forth the two key elements giving access to spectral information about a random matrix M: traces and bilinear forms (of its resolvent Q M (z) for some z). Deterministic equivalents of resolvents Q M then encode most of the information necessary to statistically quantify a random matrix M.

A practical use of deterministic equivalents is to establish that, for a random matrix M, 1 n tr(Q M (z) -Q(z)) → 0, say almost surely, for all z ∈ C with C ⊂ C some region of C. Denoting mn (z) = 1 n tr Q(z), this convergence implies that the Stieltjes transform of µ M "converges" in the sense that m µ M (z)mn (z) → 0. As we will see, this will indicate that µ M gets increasingly well approximated by a probability measure μn having Stieltjes transform mn (z). Identifying mn (z), which uniquely defines μn , will often be as far as the Stieltjes transform method will lead us. But in some rare cases (such as with the Marcenko-Pastur and the semi-circle laws), μn will be explicitly identifiable.

In the remainder of the manuscript, we will often characterize the large dimensional behavior of random matrix models M through an approximation by deterministic equivalents Q(z) of their associated resolvents Q M (z), as this offers access not only to their asymptotic spectral measure but also to their eigenspaces. We shall therefore often extrapolate some of the traditional results, such as the Marcenko-Pastur law [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF], the sample covariance matrix model [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF], etc., under this more general form.

Remark 2.2 ( Q versus EQ). For Q a deterministic equivalent for Q, the probabilistic convergences

1 n tr A(Q -Q) → 0 and a T (Q -Q)b → 0 will in general unfold from the fact that EQ -Q → 0
and from a control of the variance of 1 n tr(AQ) and a T Qb; this will often be the strategy followed in our proofs. But note importantly that, if the above relation is met, then EQ itself is a deterministic equivalent for Q by Definition 7. However, EQ is often not convenient to work with and a "truly" deterministic matrix Q involving no integration over probability spaces will be systematically preferred.

In some situations, deterministic equivalents may either not exist or, as will often be the case, will only be reachable through the access to an intermediary random matrix. To simplify the readability of the main results and proofs in the remainder of the manuscript, which involve both deterministic and random equivalents, we introduce the following shortcut notation.

Notation 1 (Deterministic and Random Equivalents). For X, Y ∈ R n×n two random or deterministic matrices, we write X ↔ Y if, for all A ∈ R n×n and a, b ∈ R n of unit norms (respectively, operator and Euclidean), we have the simultaneous results

1 n tr A(X -Y) a.s. -→0, a T (X -Y)b a.s. -→0, E[X -Y] → 0.

Foundational Random Matrix Results

In this section we introduce the main historical results of random matrix theory (appropriately updated under a deterministic equivalent form), which will serve as supporting CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY models to most applications to machine learning. For readability and accessibility to the readers new to random matrix theory, we mostly stick to intuitive and short sketches of proofs. Yet, for the readers to have a glimpse on the technical details and modern tools of the field, some of the proof sketches will be appended by a complete exhaustive proof. Both sketches and detailed proofs rely on a set of elementary lemmas and identities will be introduced below in Section 2.2.1. The main difference between sketches and detailed proofs then relies on additional technical probability theory arguments to prove various convergence results. These arguments strongly depend on the underlying random matrix model hypotheses (Gaussian independent, i.i.d., concentrated random vectors, etc.); for readability, we will focus in our proofs on one specific line of proof (that we claim to be the "historical" one) and will introduce some side remarks concerning alternative approaches.

Key lemmas and identities

Resolvent identities

Most results discussed in this section consist in approximating random resolvents Q(z) via deterministic resolvents Q(z), which are both inverse of matrices. The following first identity provides a comparison of inverse matrices.

Lemma 2.1 (Resolvent identity). For invertible matrices A and B, we have

A -1 -B -1 = A -1 (B -A)B -1 .
Proof. This can be easily checked by multiplying both sides on the left by A and on the right by B.

Another useful lemma that helps directly connect the resolvent of BA to that of AB, is given as follows.

Lemma 2.2. For A ∈ R p×n and B ∈ R n×p , we have

A(BA -zI n ) -1 = (AB -zI p ) -1 A
for z ∈ C distinct from 0 and from the eigenvalues of AB.

For AB and BA symmetric, Lemma 2.2 is a special case of the more general rela-

tion A f (BA) = f (AB)A, with f (M) ≡ U f (Λ)U T under the spectral decomposition M = UΛU T and f complex analytic. Since f is analytic, f (BA) = ∑ ∞ i=0 c i (BA) i for some sequence {c i } ∞ i=0 and thus A f (BA) = ∑ ∞ i=0 c i (AB) i A = f (AB)A.
The next lemma, known as Sylvester's identity, similarly relates the resolvents of AB and BA through their determinant.

Lemma 2.3 (Sylvester's identity). For

A ∈ R p×n , B ∈ R n×p and z ∈ C, det AB -zI p = det (BA -zI n ) (-z) p-n .
An immediate consequence of Sylvester's identity is that AB and BA have the same non-zero eigenvalues (those non-zero z's for which both left-and right-hand sides vanish). Thus, say n ≥ p, AB ∈ R p×p and BA ∈ R n×n have the same spectrum, except for the additional np zero eigenvalues of AB. This remark implies the next identity.

Lemma 2.4 (Trace of resolvent and co-resolvent).

Let A ∈ R p×n , B ∈ R n×p , and z ∈ C not an eigenvalue of AB nor zero. Then

tr Q AB (z) = tr Q BA (z) + n -p z .
In particular, if AB and BA are symmetric,

m µ AB (z) = n p m µ BA (z) + n -p pz .

Perturbation identities

Quantifying the asymptotic global (e.g., spectral distribution) or local (e.g., isolated eigenvalues or projection on eigenvector) behavior of random matrices M will systematically involve a perturbation approach. The idea often lies in comparing the behavior of the resolvent Q = Q M to the resolvent Q -i of M -i , with M -i defined as M with either row and column i, or some i-th contribution (e.g.,

M -i = ∑ j =i x j x T j if M = ∑ j x j x T j ), discarded.
A number of so-called perturbation identities are then needed.

The first one involves the segmentation of M under the form of sub-blocks, in general consisting of one large block and three small sub-matrices. The resolvent Q M can correspondingly be segmented in sub-blocks according to the following block inversion lemma.

Lemma 2.5 (Block matrix inversion). For

A ∈ R p×p , B ∈ R p×n , C ∈ R n×p and D ∈ R n×n with D invertible, we have A B C D -1 = S -1 -S -1 BD -1 -D -1 CS -1 D -1 + D -1 CS -1 BD -1
where S ≡ A -BD -1 C is the Schur complement (for the block D) of A B C D .2 As a consequence of Lemma 2.5, we have the following explicit form for all diagonal entries of an invertible matrix A. Lemma 2.6 (Diagonal entries of matrix inverse). For invertible A ∈ R p×p and A -i ∈ R (p-1)×(p-1) , the matrix obtained by removing the i-th row and column from A, i = 1, . . . , p, we have

(A -1 ) ii = 1 A ii -α T i (A -i ) -1 β i for α T i , β i ∈ R p-1
the i-th row and column of A with i-th entries removed, respectively.

The result is a direct consequence of the fact that A -1 = adj(A) det(A) , with adj(A) the adjugate matrix of A, together with the block determinant formula in Lemma 2.5.

Perturbations by addition or subtraction of low-rank matrices to M induce modifications in the resolvent Q M that involve Woodbury's identity as follows.

Lemma 2.7 (Woodbury). For A ∈ R p×p , U, V ∈ R p×n , such that both A and A + UV T are invertible, we have

(A + UV T ) -1 = A -1 -A -1 U(I n + V T AU) -1 V T A -1 .
CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY Note importantly that, while (A + UV T ) -1 is of size p × p, (I n + VAU) -1 is of size n × n. This will turn out useful to relate resolvents of large dimensional matrices to resolvents of more elementary fixed small size matrices. In particular, for n = 1, i.e., UV T = uv T for U = u ∈ R p and V = v ∈ R p , Woodbury's identity specializes to the Sherman-Morrison formula.

Lemma 2.8 (Sherman-Morrison). For A ∈ R p×p invertible and u, v ∈ R p , A + uv T is invertible if and only if 1 + v T A -1 u = 0 and

(A + uv T ) -1 = A -1 - A -1 uv T A -1 1 + v T A -1 u .
Besides,

(A + uv T ) -1 u = A -1 u 1 + v T A -1 u .
Letting A = M -zI p , z ∈ C, and v = τu for τ ∈ R in the previous lemma leads to the following rank-1 perturbation lemma for the resolvent of M. Lemma 2.9 (From Lemma 2.6 in [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF]).

For A, M ∈ R p×p symmetric, u ∈ R p , τ ∈ R and z ∈ C \ R, tr A(M + τuu T -zI p ) -1 -tr A(M -zI p ) -1 ≤ A | (z)| .
Also, for A, M ∈ R p×p symmetric and nonnegative definite, u ∈ R p , τ > 0 and z < 0,

tr A(M + τuu T -zI p ) -1 -tr A(M -zI p ) -1 ≤ A |z| .
It is interesting (and possibly counterintuitive at first) to note that u does not intervene in this inequality. In particular, irrespective of the amplitude of the rank-1 perturbation, under the conditions of the lemma

m µ M+τuu T (z) = m µ M (z) + O(p -1 )
and thus, by the link between spectrum and Stieltjes transform, the spectral measure of M is asymptotically close to that of M + τuu T for any u, in the large p limit. This result can be understood through the following two arguments: i) for large p, the spectrum of M (say M = O(1) without generality restriction) is only non-trivial if the vast majority of the p eigenvalues of M are of order O(1): thus, as p eigenvalues use a space of size O(1), they tend to aggregate; ii) by Weyl's interlacing lemma presented next (Lemma 2.10) for symmetric matrices, the eigenvalues of M and of M + τuu T are interlaced. Both arguments thus indicate that, in the large p limit, the spectral measures are indeed asymptotically the same.

Unlike non-symmetric matrices, symmetric matrices indeed enjoy the nice property of having stable spectra with respect to rank-1 perturbations. For λ ∈ R an eigenvalue of M + τuu T but not of M with, say τ > 0, we indeed have

0 = det(M + τuu T -λI p ) = det(Q M (λ)) det(I p + τQ M (λ)uu T ) = det(Q M (λ)) 1 + τu T Q M (λ)u
where the second equality unfolds from factoring out M -λI p (which is not singular as λ is not an eigenvalue of M) and the third from Sylverster's identity (Lemma 2.3). As a consequence, λ is one of the solutions to

-1 = τu T Q M (λ)u = τ p ∑ i=1 |v T i u| 2 λ i (M) -λ , M = p ∑ i=1 λ i (M)v i v T i
which, seen as a function of λ, has asymptotes at each λ i (M) and is increasing (from -∞ to ∞) on the segments (λ i (M), λ i+1 (M)) (eigenvalues being sorted in increasing order). The eigenvalues of M + τuu T are therefore interlaced with those of M. This idea generalizes to finite rank perturbation as follows.

Lemma 2.10 (Weyl, Theorem 4.3.1 in [START_REF] Roger | Matrix analysis[END_REF]). Let A, B ∈ R p×p be symmetric matrices and let the respective eigenvalues of A, B and A + B arranged in nondecreasing order, i.e.,

λ 1 ≤ λ 2 ≤ . . . λ p-1 ≤ λ p .
Then, for all i ∈ {1, . . . , p},

λ i (A + B) ≤ λ i+j (A) + λ p-j (B), j = 0, 1, . . . , p -i, λ i-j+1 (A) + λ j (B) ≤ λ i (A + B), j = 1, . . . , i,
In particular, taking i = 1 in the first equation and i = p in the second inequation, together with the fact λ j (B) = -λ p+1-j (-B) for j = 1, . . . , p, implies

max 1≤j≤p |λ j (A) -λ j (B)| ≤ A -B .

Probability identities

The results of the previous sections are algebraic identities allowing for handling the resolvent Q M of the deterministic matrix M. The second ingredient of random matrix analysis lies in asymptotic probability approximations as the dimensions of M increase. Quite surprisingly, most results essentially revolve around the convergence of a certain quadratic form, which is often nothing more than a mere extension of the law of large numbers.

Those quadratic form convergence results come under multiple forms. The historical form, due to Bai and Silverstein, sometimes referred to as the "trace lemma", is as follows.

Lemma 2.11 (Quadratic-form-close-to-the-trace, Lemma B.26 in [START_REF] Bai | Spectral analysis of large dimensional random matrices[END_REF]). Let x ∈ R p have i.i.d. entries of zero mean, unit variance and

E[|x i | L ] ≤ ν L for some L ≥ 1. Then for A ∈ R p×p and l ≥ 1 E x T Ax -tr A l ≤ K l ν 4 tr(AA T ) l/2 + ν 2l tr(AA T ) l/2
for some constant K l > 0 independent of p. In particular, if A ≤ 1 and the entries of x have bounded eighth-order moment,

E x T Ax -tr A 4 ≤ Kp 2
for some K independent of p, and consequently, as p → ∞,

1 p x T Ax - 1 p tr A a.s.
-→0.

CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY

This last result is rather intuitive. For A = I p , this is simply an instance of the law of large numbers. For generic A, first note that, by the independence of the entries of x, E[x T Ax] = E[tr(Axx T )] = tr A. Exploiting the fact that Var 1 p x T Ax = O(p -1 ) then ensures that 1 p x T Ax -1 p tr A → 0, but only in probability; since the variance calculus involves exponentiating the entries x i of x to power 4, they need to be of finite fourth power. The almost sure convergence is achieved by showing the faster moment convergence

E[( 1 p x T Ax -1 p tr A) 4 ] = O(p -2
) which is the second statement of the lemma and requires 8-th order exponentiation of the x i 's. The request for A to be of bounded norm with respect to p in this case "stabilizes" the quadratic form x T Ax by maintaining its random concentration properties.

Recalling that Q M (z) ≤ (dist(z, supp(µ M ))) -1 , Lemma 2.11 can be exploited for A = Q M (z) for all z away from the support of µ M and all x independent of Q M (z). The core of the proofs of the main random matrix results is uniquely based on this last remark.

These identities constitute the main technical ingredients needed to understand the proofs of both historical and recent random matrix results. The next section introduces the most fundamental of those which will be called after over and over in the remainder of the manuscript.

The Marcenko-Pastur law

We start by illustrating how the aforementioned tools are used to prove one of the most popular results in random matrix theory: the Marcenko-Pastur law. Another important result is the Wigner semi-circle law, which, despite being popular in many graph-based problems, is less covered in the works addressed in this manuscript and thus omitted.

To simplify the exposition of the results, we will use the notation for deterministic equivalents introduced in Notation 1. That is, for X, Y ∈ R n×n , we will denote X ↔ Y if, for all unit norm A ∈ R n×n and a, b ∈ R n , 1 n tr A(X -Y)

a.s. -→0, a T (X -Y)b a.s. -→0 and E[X -Y] → 0.
Most of the results involve Stieltjes transforms m µ (z) of probability measures with support supp(µ). Since Stieltjes transforms are such that m µ (z) > 0 for z < inf supp(µ), m µ (z) < 0 for z > sup supp(µ) and [z] [m µ (z)] > 0 if z ∈ C \ R, it will be convenient in the following to consider the set

Z (A) = (z, m) ∈ A 2 , ( [z] [m] > 0 if [z] = 0) or (zm < 0 if [z] = 0) .
We present the Marcenko-Pastur law under the slightly modified form of a deterministic equivalent for the resolvent Q(z).

Theorem 2.3 (From [MP67]). Consider the resolvent Q(z) = ( 1 n XX T -zI p ) -1
, for X ∈ R p×n having i.i.d. zero mean and unit variance entries. Then, as n, p → ∞ with p/n → c ∈ (0, ∞), we have

Q(z) ↔ Q(z), Q(z) = m(z)I p (2.8) with (z, m(z)) the unique solution in Z (C \ [(1 - √ c) 2 , (1 + √ c) 2 ]) of zcm 2 (z) -(1 -c -z)m(z) + 1 = 0.
(2.9)

The function m(z) is the Stieltjes transform of the probability measure µ given explicitly by

µ(dx) = (1 -c -1 ) + δ(x) + 1 2πcx (x -a) + (b -x) + dx (2.10) where a = (1 - √ c) 2 , b = (1 + √ c)
2 and (x) + = max(0, x), which is known as the Marcenko-Pastur distribution. In particular, with probability one, the empirical spectral measure µ 1 n XX T converges weakly to µ. Intuitive idea. A first heuristic derivation consists in iteratively "guessing" the form of Q(z) = F(z) -1 for some matrix F(z) -1 . To this end, from Lemma 2.1, it first appears that

Q(z) -Q(z) = Q(z) F(z) + zI p - 1 n XX T Q(z) = Q(z) F(z) + zI p - 1 n n ∑ i=1 x i x T i Q(z)
For Q(z) to be a deterministic equivalent for Q(z), we wish in particular that 1

p tr A(Q(z) - Q(z)) a.s. -→0, for A deterministic with A = 1. That is 1 p tr(F(z) + zI p ) Q(z)AQ(z) - 1 n n ∑ i=1 1 p x T i Q(z)AQ(z)x i a.s.
-→0.

(2.11)

We recognize in 1 p x T i Q(z)AQ(z)x i a quadratic form on which we would like to use Lemma 2.11 to turn it into a trace term independent of x i . Yet, Lemma 2.11 cannot be used as Q(z) depends on x i . To counter the difficulty, we then use Lemma 2.8 to write

Q(z)x i = Q -i (z)x i 1 + 1 n x T i Q -i (z)x i where Q -i (z) = ( 1 n ∑ j =i x j x T j -zI p ) -1 is independent of x i . Now legitimately applying Lemma 2.11, we find that 1 p x T i Q(z)AQ(z)x i = 1 p x T i Q(z)AQ -i (z)x i 1 + 1 n x T i Q -i (z)x i 1 p tr Q(z)AQ -i (z) 1 + 1 n tr Q -i (z)
.

(2.12)

From Lemma 2.9, normalized traces involving Q -i (z) or Q(z) are asymptotically identical and thus this further reads

1 p x T i Q(z)AQ(z)x i 1 p tr Q(z)AQ(z) 1 + 1 n tr Q(z)
.

Getting back to (2.11), we thus end up with the approximation

1 p tr(F(z) + zI p ) Q(z)AQ(z) 1 p tr Q(z)AQ(z) 1 + 1 n tr Q(z)
.

As a consequence, we can now "guess" the form of F(z). Indeed, if it is to exist, F(z) must be of the type

F(z) -z + 1 1 + 1 n tr Q(z) I p
for the approximation above to hold. To close the loop, taking A = I p , 1 n tr Q(z) appearing in this display must be well approximated by m(z) ≡ 1 p tr Q(z) so that

1 p tr Q(z) m(z) = 1 -z + 1 1+ p n 1 p tr Q(z) 1 -z + 1 1+ p n m(z)
(2.13) and we thus have finally

Q(z) = F(z) -1 = m(z)I p
where, in the large n, p limit, m(z) is solution to

m(z) = 1 -z + 1 1+cm(z) or equivalently zcm 2 (z) -(1 -c -z)m(z) + 1 = 0.
This equation has two solutions defined by the two roots of the complex square root function

m(z) = 1 -c -z 2cz + ((1 + √ c) 2 -z)((1 - √ c) 2 -z) 2cz only one of which is such that [z] [m(z)]
√ ((1+ √ c) 2 -x)(x-(1- √ c) 2 ) 2cπx
at point x in the set. The case x = 0 brings a discontinuity in µ with weight equal to

µ({0}) = -lim y↓0 ıym(ıy) = c -1 2c ± c -1 2c
where the sign is established by a second order development of zm(z) in the neighborhood of zero.

Detailed proof.

Having heuristically identified Q(z), we shall now use sound mathematical tools to prove that, indeed, Q(z) is a deterministic equivalent for Q(z) in the sense of the theorem statement. Let us first show that

E[Q(z)] = Q(z) + o • (1)
, where o • (1) denotes a matrix term of vanishing operator norm as n, p → ∞.

Convergence in mean. For mathematical convenience, we will take z < 0 in what follows.

Since Q(z) and Q(z) from the theorem statement are complex analytic functions for z / ∈ R + (matrix-valued Stieltjes transforms are analytic), obtaining the convergence results on R -is equivalent to obtaining the result on all of C \ R + .

We proceed in two steps by first introducing the intermediate deterministic quantities

α(z) ≡ 1 n tr E[Q(z)] and Q(z) ≡ (-z + 1 1+α(z) ) -1 I p . From Lemma 2.1, we have (the argument z in α(z), Q(z) and Q(z) is dropped when confusion is not possible) E[Q -Q] = EQ I p 1 + α - 1 n XX T Q = E[Q] 1 + α Q - 1 n E[QXX T ] Q = E[Q] 1 + α Q - n ∑ i=1 1 n E[Qx i x T i ] Q = E[Q] 1 + α Q - n ∑ i=1 E Q -i 1 n x i x T i 1 + 1 n x T i Q -i x i Q
where we applied Lemma 2.8 to obtain the last equality and denoted Q -i ≡ (∑ j =i 1 n x j x T j -zI p ) -1 as previously.

Since we expect 1 n x T i Q -i x i to be close to α (as a consequence of Lemma 2.11), we rewrite

Q -i 1 n x i x T i 1 + 1 n x T i Q -i x i = Q -i 1 n x i x T i 1 + α - Q -i 1 n x i x T i ( 1 n x T i Q -i x i -α) (1 + α)(1 + 1 n x T i Q -i x i ) so that E[Q -Q] = E[Q] 1 + α Q - n ∑ i=1 E Q -i 1 n x i x T i Q 1 + α + n ∑ i=1 E Q 1 n x i x T i d i Q 1 + α = E[Q] 1 + α Q - n ∑ i=1 E Q -i 1 n x i x T i Q 1 + α + E Q 1 n XDX T Q 1 + α where we introduced D = diag{d i } n i=1 for d i = 1 n x T i Q -i x i -α,
and used again Lemma 2.8 to write

Q -i 1 n x i x T i 1+ 1 n x T i Q -i x i = Q 1 n x i x T i in the first equality. Since E[Q -i x i x T i ] = E[Q -i ], this further reads E[Q -Q] = 1 n n ∑ i=1 (E[Q] -E[Q -i ]) Q 1 + α + E 1 n QXDX T Q 1 + α .
CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY Again from Lemma 2.1 and 2.8,

1 n n ∑ i=1 E[Q -Q -i ] = - 1 n n ∑ i=1 E Q 1 n x i x T i Q -i = - 1 n n ∑ i=1 E Q 1 n x i x T i Q 1 + 1 n x T i Q -i x i = - 1 n E Q 1 n XD 2 X T Q where D 2 = diag 1 + 1 n x T i Q -i x i n i=1
and thus

E[Q -Q] = - 1 n E Q 1 n XD 2 X T Q Q 1 + α + E 1 n QXDX T Q 1 + α . (2.14)
It remains to show that the right-hand side terms vanish in the large p, n limit. For the first term, note that

0 Q 1 n XD 2 X T Q Q 1 n XX T Q max 1≤i≤n [D 2 ] ii in the order of symmetric matrices. Since Q 1 n XX T = I p + zQ which is of bounded op- erator norm, controlling E[Q 1 n XD 2 X T Q] boils down to controlling E[max i [D 2 ] ii ].
This can be established in various ways. From the union bound and the i.i.d. nature of the x i 's,

P max i [D 2 ] ii > t ≤ nP ([D 2 ] 11 > t) .
Now, by Markov's inequality P(X > a) ≤ E[X k ]/a k for every k (for X, a > 0) and the moment inequality in Lemma 2.11 for, say l = 4, P(max i [D 2 ] ii > t) may be bounded by a function decreasing as t -2 , for all t > 1 + α(z), and of order n -1 . Since E[X] = X>0 P(X > t)dt, we then find that E[max i [D 2 ] ii ] is bounded. Alternatively, one may have used a concentration inequality argument to show the same. Consequently, due to the leading 1/n factor in front of the first right-hand side term of (2.14), this term vanishes as n, p → ∞.

To handle the second right-hand side term in (2.14), one needs to control the norm of 1 n QXDX T Q. This is not a symmetric matrix, but E[Q -Q] is. We may thus rewrite (2.14) as the half-sum of itself and its transpose and we are thus left to controlling the operator norm of 1

n QXDX T Q + 1 n QXDX T Q. Using the matrix inequalities AB T + BA T AA T + BB T (from (A -B)(A -B) T 0) and AB T + BA T -AA T -BB T (from (A + B)(A + B) T 0), we are left to bounding the norm of E 1 n √ n QXX T Q + E 1 √ n QXD 2 X T Q
where the division of the 1/n 2 term into 1/(n √ n) and 1/ √ n is essential. The first term above is easily seen to be of order 1/ √ n. As for the second, using as above the moment inequality in Lemma 2.11 along with Markov's inequality, it appears to be also of order 1/ √ n. This can be anticipated by noticing that

d i = 1 n x T i Q -i x i -α fluctuates as 1/
√ n (by a central limit theorem argument) and thus d 2 i is essentially of order 1/n. Gathering the pieces together, we thus conclude that

E[Q] -Q → 0, with Q = 1 1 + α(z) -z -1 I p .

FOUNDATIONAL RANDOM MATRIX RESULTS

51

Since

α(z) = 1 n tr E[Q(z)] = 1 n tr Q(z) + o(1) = c 1 1+α(z) -z + o(1)
by defining m(z) as the unique Stieltjes transform solution with (z

) [m(z)] > 0 of 1 m(z) = 1 1 + cm(z) -z ⇔ zcm 2 (z) -(1 -c -z)m(z) + 1 = 0
(the uniqueness is easily shown by solving the quadratic equation), we finally have cm(z)α(z) → 0, which concludes the proof of (2.8).

Almost sure convergence. To now prove the almost sure convergence 1

p tr A(Q -Q) a.s.
-→0

and a T (Q -Q)b a.s.
-→0, it suffices to show

1 p tr A(Q -EQ) a.s.
-→0, a T (Q -EQ)b a.s.

-→0.

We will only show here the leftmost convergence. This follows from either a moment or a concentration argument. The historical approach, due to Bai and Silverstein (see e.g., in [START_REF] Bai | Spectral analysis of large dimensional random matrices[END_REF]), exploits the following martingale difference inequality.

Lemma 2.12 (Burkholder inequality, Lemma 2.1 in [START_REF] Bai | Spectral analysis of large dimensional random matrices[END_REF]). Let {X i } ∞ i=1 be a martingale difference for the increasing σ-field {F i } and denote E k the expectation with respect to F k . Then, for k ≥ 2,

E   n ∑ i=1 X i k   ≤ C k   E n ∑ i=1 E k-1 [|X i | 2 ] k/2 + n ∑ i=1 E[|X i | k ]   . Remarking that 1 p tr A(Q -EQ) = n ∑ i=1 E i 1 p tr AQ -E i-1 1 p tr AQ = 1 p n ∑ i=1 (E i -E i-1 ) [tr A(Q -Q -i )] (since E i [tr AQ -i ] = E i-1 [tr AQ -i ])
for F i the σ-field generating the columns x i+1 , . . . , x n of X and with the convention E 0 [ f (X)] = f (X), which forms a martingale difference sequence, we fall under the scope of Burkholder's lemma. Now, from the identity Q =

Q -i -1 n Q -i x i x T i Q -i 1+ 1 n x T i Q -i x i (Lemma 2.8), (E i -E i-1 ) 1 p tr A(Q -Q -i ) = -(E i -E i-1 ) 1 pn x T i Q -i AQ -i x i 1 + 1 n x T i Q -i x i
which is bounded by 1/p. As a consequence, from Lemma 2.12, -→0), we then conclude that 1 p tr A(Q -EQ) a.s.

E 1 p tr A(Q -EQ) 4 = O(n -2 ).
-→0

as requested.

Remark 2.3 (Proof by Stein's lemma and Nash-Poincaré inequality). In [START_REF] Andreevich | Eigenvalue distribution of large random matrices[END_REF], Pastur and Scherbina propose an alternative proof to Theorem 2.3, based on a two-fold method: i) a proof for Gaussian X and ii) an interpolation method to non-Gaussian X. The Gaussian case is handled through the powerful Stein's lemma.

Convergence in mean by Stein's lemma.

Lemma 2.13 ([Ste81]

). Let x ∼ N (0, 1) and f : R → R a continuously differentiable function

such that E[ f (x)] < ∞. Then, E[x f (x)] = E[ f (x)].
(2.15)

In particular, for x ∼ N (0, C) with C ∈ R p×p and f : R p → R a continuously differentiable function with derivatives having at most polynomial growth with respect to p,

E[x i f (x)] = p ∑ j=1 C ij E ∂ f (x) ∂x j (2.16)
where ∂/∂x i indicates differentiation with respect to the i-th entry of x.

The lemma, sometimes referred to as the integration-by-parts formula for Gaussian variables, simply follows from

E[x f (x)] = x f (x)e -1 2 x 2 dx = [-f (x)e -1 2 x 2 ] ∞ -∞ + f (x)e -1 2 x 2 dx = E[ f (x)]
by integration by parts u v = [uv]uv for u(x) = -e -1 2 x 2 and v(x) = f (x). To exploit Lemma 2.13, let us thus assume X Gaussian, i.e., X ij ∼ N (0, 1). Observe that

Q = 1 z 1 n XX T Q -1 z I p , so that E[Q ij ] = 1 zn n ∑ k=1 E[X ik [X T Q] kj ] - 1 z δ ij in which E[X ik [X T Q] kj ] = E[x f (x)] for x = X ik and f (x) = [X T Q] kj .
Therefore, from the lemma and the fact that ∂Q = -

1 n Q∂(XX T )Q, E[X ik [X T Q] kj ] = E ∂[X T Q] kj ∂X ik = E[Q ij ] -E 1 n [X T QX] kk Q ij -E 1 n [X T Q] ki [X T Q] kj .
so that, summing over k,

1 z 1 n n ∑ k=1 E[X ik [X T Q] kj ] = 1 z E[Q ij ] - 1 z 1 n 2 E[Q ij tr(QXX T )] - 1 z 1 n 2 E[QXX T Q] ij .

FOUNDATIONAL RANDOM MATRIX RESULTS

53

It is not too difficult to see that the rightmost term has vanishing operator norm (of order O(1/n)) as n, p → ∞. Also recall that tr(QXX T ) = np + zn tr Q. As a result, matrix-wise, we obtain

E[Q] + 1 z I p = E[X •k [X T Q] k• ] = 1 z E[Q] - 1 z 1 n E[Q(p + z tr Q)] + o • (1).
As 1 n tr Q is expected to converge to some m(z), it can be taken out of the expectation in the limit so that, gathering all terms proportional to E[Q] on the left-hand side, we finally have

E[Q](1 -p/n -z -p/nzm(z)) = I p + o • (1)
which, taking the trace to identify m(z), concludes the proof for the Gaussian case.

Almost sure convergence by Nash-Poicaré inequality. To prove the almost sure convergence of traces and bilinear forms of the resolvent in the case of Gaussian X, one may then use the powerful Nash-Poincaré inequality proposed by Pastur.

Lemma 2.14 (Nash-Poincaré inequality from [START_REF] Pastur | A simple approach to the global regime of gaussian ensembles of random matrices[END_REF]). For x ∼ N (0, C) with C ∈ R p×p and f : R p → R continuously differentiable with derivatives having at most polynomial growth with respect to p,

Var[ f (x)] ≤ 2 ∑ 1≤i,j≤n C ij E ∂ f (x) ∂x i ∂ f (x) ∂x j .
In the present case, for Gaussian X with X ij ∼ N (0, 1),

Var 1 p tr AQ ≤ 2 p 2 ∑ 1≤i,j≤n E ∂ tr AQ ∂X ij 2 . Again using ∂Q = -1 n Q∂(XX T )Q, we find ∂ tr AQ ∂X ij = - 1 n [QAQX + QA T QX] ij so that, from (a + b) 2 ≤ 2(a 2 + b 2 ) and A = 1, 2 p 2 ∑ 1≤i,j≤n E ∂ tr AQ ∂X ij 2 ≤ 4 p 2 n 2 tr(QAQXX T QA T Q) + tr(QA T QXX T QAQ) = O(n -2 ).
By Markov's inequality and the Borel Cantelli lemma, we thus have that 1 p tr A(Q -EQ) a.s.

-→0.

When it comes to evaluating the fluctuations of a T (Q -EQ)b with the same approach, it appears that Var[a T (Q -EQ)b] = O(n -1 ) which is enough to ensure convergence in probability (by Markov's inequality) but not almost surely (as the Borel Cantelli lemma cannot be applied). Thus one needs to resort to evaluating a higher moment bound, such as E[|a T (Q -EQ)b| 4 ]. To this end, we may use the fact that

E[|a T (Q -EQ)b| 4 ] = Var[|a T (Q -EQ)b| 2 ] + E |a T (Q -EQ)b| 2 2 = Var[|a T (Q -EQ)b| 2 ] + Var[|a T (Q -EQ)b|] 2 .
Since we know that the rightmost term is of order O(n -2 ), it remains to show, again through Nash-Poincaré inequality, that Var[|a T (Q -EQ)b| 2 ] = O(n -2 ) which is a cumbersome but easily obtained result as well.

Interpolation trick to non-Gaussian X. To "interpolate" these results from Gaussian X to non-Gaussian X, one may then use a generalized version of Stein's lemma to non-Gaussian distributions, for which we have: Lemma 2.15 (Interpolation trick, Corollary 3.1 in [START_REF] Lytova | Central limit theorem for linear eigenvalue statistics of random matrices with independent entries[END_REF]). For x ∈ R a random variable with zero mean and unit variance, y ∼ N (0, 1), and f a k + 2-differentiable function with bounded derivatives,

E[ f (x)] -E[ f (y)] = k ∑ l=2 κ l+1 2l! 1 0 E[ f (l+1) x(t)]t (l-1)/2 dt + k where κ l is the l th cumulant of x, x(t) = √ tx + (1 - √ t)y, and | k | ≤ C k E[|x| k+2 ] sup t | f (k+2) (t)| for some constant C k only dependent on k.
All Gaussian expectations (means and variance) in the proof above can then be expressed as their non-Gaussian form up to a sum of moment control on the derivatives of f . Remark 2.4 (On the convergence rates). In the course of the proofs above, we saw examples of a general concentration trend for linear statistics and quadratic forms of random matrices. We shall indeed typically have for most of the models of random matrices X ∈ R n×n under study that

• linear statistics 1 n ∑ n i=1 f (λ i (X)) for sufficiently well-behaved f (so for instance 1 n tr Q X (z) = 1 n ∑ i (λ i (X) -z) -1 ) converge at a speed O(1/n) (their variance scales as O(1/n 2
). From a central-limit theorem viewpoint, this is as fast as it can get. Indeed, X is maximally composed of p × n = O(n 2 ) "degrees of freedom" and thus, by the central limit theorem, fluctuations are at most at speed O(1/

√ n 2 ) = O(1/n).
• quadratic forms a T f (X)b where f (X) = U diag( f (λ i (X))U T (in the spectral decomposition of X) typically converge at a slower O(1/ √ n) speed.

This remark is particularly interesting as it indicates, from a statistics viewpoint that, for X ∈ R p×n , asymptotic approximations may gain accuracy by doubly exploiting the degrees of freedom in both the sample (n) and feature (p) direction.

Remark 2.5 (On the assumptions on X). The Marcenko-Pastur law has been widely generalized and several times proved using different techniques. For instance [A + 11, O'R12] assume the X ij are "weakly" dependent in the sense that their correlation or higher order cross-moments vanish at a certain speed as n, p → ∞. Alternatively, the works of Bai and Silverstein (see [START_REF] Baik | Eigenvalues of large sample covariance matrices of spiked population models[END_REF]) tend to assume that the entries of X are not necessarily identically distributed; in this case, an additional condition on the tails P(|X ij | > t) of the probability measures of the entries (for instance a uniform bound on some moment higher than 2) is needed. In [START_REF] El | Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond[END_REF], El Karoui provides a first result which assumes the columns x i of X = [x 1 , . . . , x n ] are independent concentrated random vectors. (Very) roughly speaking, concentrated random vectors x ∈ R p can be written as x = ϕ( x) where x has standard i.i.d. entries with either a Gaussian law or a bounded support, and ϕ : R p → R p is any 1-Lipschitz function: this assumption maintains the p degrees of freedom in x (arising from x) while allowing for strong nonlinear correlation between the entries of x. In this case, the Marcenko-Pastur law is indeed still valid if ϕ(x) has zero mean and identity covariance.

Large dimensional sample covariance matrices

The Marcenko-Pastur and semi-circle theorems have long been the gold-standard in both theoretical and applied random matrix theory, in the sense that most mathematical studies and practical results concerned the Wishart and Wigner random matrix models. But

FOUNDATIONAL RANDOM MATRIX RESULTS

55

the assumption of data X with i.i.d., let alone standard Gaussian, entries is often limiting. In statistics where one is interested in the correlation XX T , it is expected that the columns x i ∈ R p of X exhibit a correlation structure and be non-necessarily independent (in particular when they are samples from a time series). This section introduces generalizations of these results, to a level that is convenient to machine learning applications. In particular, in order to model the existence of classes within the data, X will often be subdivided into subblocks that can be identified with each class.

Our first result generalizes the Marcenko-Pastur law to sample covariance matrices and is originally due to a long line of works by Bai and Silverstein [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF].

Theorem 2.4 (Sample covariance matrix, from [SB95]). Let X = C 1 2 Z ∈ R p×n with C ∈ R p×p , Z ∈ R p×n having i.i.d. zero mean and unit variance entries. Then, as n, p → ∞ with p/n → c > 0, letting Q(z) = ( 1 n XX T -zI p ) -1 and Q(z) = ( 1 n X T X -zI n ) -1 , we have Q(z) ↔ Q(z) = - 1 z I p + mp (z)C -1 Q(z) ↔ Q(z) = mp (z)I n where (z, mp (z)) is the unique solution in Z (C \ R + ) of mp (z) = -z + 1 n tr C I p + mp (z)C -1 -1 . In particular, if µ C → ν as p → ∞, then µ 1 n XX T a.s.
-→µ and µ 1 n X T X a.s.

-→ μ as p, n → ∞ where µ, μ are the unique measures having Stieltjes transforms m(z) and m(z) with

m(z) = 1 c m(z) + 1 -c cz , m(z) = -z + c tν(dt) 1 + m(z)t -1 .
A few remarks are in order to better understand the statement of the theorem.

Remark 2.6 (On the implicit statement). As opposed to Theorem 2.3, the statement of the theorem is here implicit in the sense that µ is only defined through m µ (z), itself implicitly defined as the solution of an implicit equation. The main reason for the explicit nature of Theorem 2.3 is that Equation (2.13), that through a perturbation approach provides the connection between m(z) and a function of itself, boils down to a quadratic equation in m(z) which can be solved and from which Theorem 2.1 can be applied. Due to the presence of C, in the present situation, the equivalent to (2.13) will here maintain an implicit form. This will hold true for almost all generalizations of the Marcenko-Pastur theorem to be introduced in this manuscript.

Remark 2.7 (Numerical evaluation of m µ (z)). Due to its implicit nature, determining m(z) for z ∈ C \ R + requires to solve an implicit equation. Using contraction and analyticity arguments, it can be shown that the standard fixed-point algorithm converges, i.e.,

m(z) = lim →∞ m ( ) (z) for m(0) (z) = 0 (say) and, for ≥ 0, m ( ) (z) = 1 c m( ) (z) + 1-c cz , m( +1) (z) = (-z + c tν(dt) 1+ m( ) (z)t ) -1 .
One must be careful here that, since m(z) is not formally defined for z ∈ supp(µ), the above argument does not hold in this set. In practice, trying to solve for m(z) with z ∈ supp(µ) CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY numerically leads to a non-converging m ( ) (z) sequence. This, in passing, can be used to actually determine the support supp(µ) as the set of z's for which the above algorithm does not converge.

Numerically, when evaluating m(z) close to the real axis (say for z = x + ı , | | 1), the convergence can appear to be quite slow for x ∈ supp(µ). A convenient workaround is to sequentially evaluate m(z) for all z's of the form x + ı , starting from some z 0 = x 0 + ı with x 0 / ∈ supp(µ) until reaching the desired value while systematically starting the fixed-point iterations from the value m(z) obtained for the previous z.

Remark 2.8 (Drawing µ). As shown in [START_REF] Jack | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF], the limiting measure µ in Theorem 2.4 admits a density. From the inverse Stieltjes transform formula of Theorem 2.1 and Remark 2.7 above, this can be approximated by solving m(z) for z ∈ ı + R for some > 0 small (say = 10 -6 ) and retrieving the density at x as 1 π [m(x + ı )]. This procedure however only allows for a numerical (rather than theoretical) evaluation of µ and of its support (the latter being approximately the set of x's such that | 1 π [m(x + ı )]| ∼ ). Figure 2.2 depicts the empirical versus limiting measure of µ 1 n XX T for C having three distinct and evenly numerous eigenvalues. In this particular setting, the limiting spectrum is akin to Marcenko-Pastur shaped connected components. For sufficiently distinct eigenvalues of C, these components are disjoint while for close eigenvalues they tend to merge.

Remark 2.9 (Deterministic equivalent for µ 1 n XX T ). The convergence result µ 1 n XX T a.s.

-→µ in Theorem 2.4 imposes that there exists a limit ν to µ C , which may not be practically meaningful. In generalized versions of Theorem 2.4 (see e.g., Theorem 2.5 below), even if the spectral measure of the covariance matrices are to converge, µ 1 n XX T may not even have a limit. One may instead consider the deterministic equivalent µ p for µ 1 n XX T which is a sequence of probability measures, such that dist(µ 1 n XX T , µ p ) a.s.

-→0 for some distance between distributions (for instance, such that µ 1 n XX Tµ p a.s.

-→0 vaguely). Practically speaking, since the data dimension p is in general a fixed quantity and C a given covariance matrix (rather than specific values in a growing sequence of p's and C's), one will always consider that the "effective" limiting measure ν coincides with µ C = 1 p ∑ p i=1 δ λ i (C) . So for instance, if p = q for some fixed q in the data at hand, one may still theoretically consider that n, p → ∞ and apply Theorem 2.4 with ν = µ C for C ∈ R q×q , the actual data covariance matrix of the finite-dimensional model. Sketch of Proof of Theorem 2.4. The proof of Theorem 2.4 generally follows the same line of arguments as that of Theorem 2.3. The main difference is that (2.12) here becomes

1 n x T i QAQx i = 1 n x T i QAQ -i x i 1 + 1 n x T i Q -i x i 1 n tr QAQ -i C 1 + 1 n tr Q -i C
where we used the fact that, denoting x i = C 1 2 z i for z i the i-th column of Z having i.i.d. zero mean and unit variance entries, by Lemma 2.11,

1 n x T i Q -i x i = 1 n z T i C 1 2 Q -i C 1 2 z i 1 n tr Q -i C.
Again with Lemma 2.9 and the fact that 1 n tr

Q -i C ≤ C 1 n tr Q -i ,
C = 1 3 (δ 1 + δ 3 + δ 7 ) (top) and µ C = 1 3 (δ 1 + δ 3 + δ 4 ) (bottom).

58

CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY with F(z) = Q-1 (z) the sought-for deterministic equivalent, which then must admit the form

F(z) C 1 + 1 n tr QC -zI p
for the previous approximation to hold. Ultimately, taking

A = C in 1 n tr A(Q -Q) a.s.
-→0 we deduce

1 n tr CQ 1 n tr C Q 1 n tr C -zI p + C 1 + 1 n tr C Q -1
(2.17)

or equivalently

mp (z) = -z + 1 n tr C I p + mp (z)C -1 -1 if we denote mp (z) = -1 z 1 + 1 n tr C Q(z)
-1 , as requested. Note that we implicitly used here the fact that C is bounded.

With the deterministic equivalent for Q in hand, the deterministic equivalent for Q follows from the direct observation that Q

= 1 z 1 n X T QX -1 z I n so that E[ Q] ij = 1 z 1 n E[x T i Qx j ] - 1 z δ ij = 1 z E 1 n x T i Q -i x j 1 + 1 n x T i Q -i x i - 1 z δ ij - 1 z 1 + 1 n tr C Q -1 δ ij = mp (z)δ ij .
Kernel methods naturally involve matrices of the type K = { 1 p x T i x j } n i,j=1 = 1 p X T X (inner product kernels) or K = { 1 p x ix j 2 } n i,j=1 (distance kernels), where the prefactor 1/p is necessary under our notation framework to ensure that the spectrum of K remains of order O(1) as p, n increase. Assuming the vectors x i arise from a mixture model, the following generalization of Theorem 2.4 will be of practical relevance.

Theorem 2.5 (Sample covariance of mixture models, from [START_REF] Benaych | Spectral analysis of the gram matrix of mixture models[END_REF]).

Let X = [X 1 , . . . , X k ] ∈ R p×n with X a = [x a1 , . . . , x an a ] ∈ R p×n a and x ai = C 1 2
a z ai for z ai a vector with i.i.d. zero mean and unit variance entries. Then, as n 1 , . . . , n k , p → ∞ in such a way that k is fixed and n

a /n → c a > 0, p/n → c 0 > 0, letting Q(z) = ( 1 p X T X -zI n ) -1 and Q(z) = ( 1 p XX T -zI p ) -1 , we have Q(z) ↔ Q(z) = diag{g a (z)1 n a } k a=1 Q(z) ↔ Q(z) = - 1 z I p + k ∑ a=1 c a c 0 g a (z)C a -1 with (z, g a (z)) the unique solutions in Z (C \ R + ) of g a (z) = - 1 z (1 + ga (z)) -1 , ga (z) = - 1 z 1 p tr C a I p + k ∑ b=1 c b c 0 g b (z)C b -1
.

SPIKED MODELS

59

Sketch of proof of Theorem 2.5. Let us begin with the initial guess Q = F -1 . By Lemma 2.1,

Q -Q = Q F + zI p - 1 p k ∑ b=1 n b ∑ i=1 x bi x T bi Q so that, with 1 p tr A( Q -Q) a.s.
-→0, it follows

1 p tr(F + zI p ) QA Q - 1 p k ∑ b=1 n b ∑ i=1 1 p x T bi QA Qx bi a.s.
-→0.

Applying Lemma 2.8 to remove the dependence in Q of x bi , together with Lemma 2.9, we deduce 1

p k ∑ b=1 n b ∑ i=1 1 p x T bi QA Qx bi k ∑ b=1 n b p 1 p tr C b QA Q 1 + 1 p tr QC b
so that F must be written as the following sum over b

F k ∑ b=1 c b c 0 C b 1 + 1 p tr QC b -zI p .
To eventually close the loop, we take A = C a for a = 1, . . . , k to establish

1 p tr Ca Q 1 p tr C a Q ≡ ga (z) 1 p tr C a   -zI p + k ∑ b=1 c b c 0 C b 1 + 1 p tr QC b   -1 ≡ - 1 z 1 p tr C a I p + k ∑ b=1 c b c 0 g b (z)C b
where we denote g a (z) ≡ -

1 z 1 + 1 p tr C a Q -1 = -1 z (1 + ga (z)) -1
, as desired. To derive the deterministic equivalent for Q from that for Q, we use again the fact that

Q = 1 z 1 p X T QX -1
z I n and therefore, indexing the set {1, . . . , n} as {11, . . . , 1n 1 , . . . , 1k, . . . , kn k },

E[Q] ai,bj = 1 z 1 p E[x T ai Qx bj ] - 1 z δ ai,bj - 1 z 1 + 1 p tr C a Q -1
δ ai,bj = g a (z)δ ai,bj .

Spiked Models

In the last section we discussed the popular sample covariance model, by providing a deterministic equivalent for the associated resolvent, and consequently the associated Stieltjes transform that describes the limiting spectral measure as well as a thorough characterization of linear statistics, quadratic forms and the subspaces of interest. Nonetheless, due to the implicit nature of Theorem 2.4, the aforementioned understanding has only a rather limited practical impact.

In this section, we consider a very special, yet practically far reaching, case of sample covariance matrix models for which the limiting spectral measure coincides with CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY the Marcenko-Pastur law. Since the Marcenko-Pastur law assumes an explicit wellunderstood expression (recall Theorem 2.3), the various estimates of interest will be explicit, thus intuitions on their behavior are easily derived.

These special models fundamentally rely on letting the covariance matrix C be a low rank perturbation of the identity matrix I p , i.e., C = I p + L for L ∈ R p×p with rank(L) = k fixed with respect to p, n. Such statistical models corresponding to a low rank update of a classical random matrix model with well-known behavior are generically called spiked models.

Isolated eigenvalues

Let us then consider again the statistical model X = [x 1 , . . . ,

x n ] ∈ R p×n with x i = C 1 2 z i , z i ∈ R p with standard i.i.d.

entries and where

C = I p + L, L = k ∑ i=1 i u i u T i
with k and 1 ≥ . . . ≥ k > 0 fixed with respect to n, p and u i = 1 for all i.

According to Theorem 2.4, µ 1 n XX T has a limiting measure µ defined through the limiting measure ν of µ C . But obviously ν = δ 1 here since

µ C = p -k p δ 1 + 1 p k ∑ i=1 δ 1+ i → δ 1 .
As a consequence, while C is not the identity matrix, the limiting measure µ is the Marcenko-Pastur law introduced in Theorem 2.3. In this case, it is natural to ask, if it is possible to (constantly) observe eigenvalues of 1 n XX T "jumping out" the support of ν, and if yes, can we further "localize" these isolated eigenvalues.

We will precisely show here that, depending on the values of i and c = lim p/n, the i-th largest eigenvalue of 1 n XX T may indeed isolate from supp(µ). As such, since most of the eigenvalues of 1 n XX T congregate but possibly for a few ones (up to k of them), the latter isolated eigenvalues are seen as "spikes" in the histogram of eigenvalues.

The specific result, here due to Baik (not Bai) and Silverstein, is as follows Theorem 2.6 (Spiked models, from [START_REF] Baik | Eigenvalues of large sample covariance matrices of spiked population models[END_REF]). Under the setting of Theorem 2.4 with

E[Z 4 i,j ] < ∞, let C = I p + L with L = ∑ k i=1 i u i u T i in its spectral decomposition,
where k and 1 ≥ . . . ≥ k > 0 are fixed with respect to p, n. Then, denoting λ 1 ≥ . . . ≥ λ p the eigenvalues of 1 n XX T , as p, n → ∞,

λ i a.s. -→ ρ i = 1 + i + c 1+ i i > (1 + √ c) 2 , i > √ c (1 + √ c) 2 , i ≤ √ c.
The theorem thus identifies an abrupt change in the behavior of the i-th dominant eigenvalue λ i of 1 n XX T : if i ≤ √ c, λ i converges to the right-edge (1 + √ c) 2 of the support of µ and thus does not isolate. However, as soon as i > √ c, λ i converges to a limit beyond the right-edge of µ and thus does isolate, from the Marcenko-Pastur support.

With a statistical physics interpretation, this phenomenon is often referred to as the phase transition of the spiked models.

From a statistical viewpoint, the fact that the i-th eigenvalue λ i of the sample covariance matrix "macroscopically" exceeds or not the other eigenvalues according to whether i >

√

c or i ≤ √ c can be interpreted as the fact that the "signal strength" i of the structured data exceeds the minimal detectability threshold √ c: this is achieved if the signal strength i is strong enough, or alternatively if the number of observed independent data n is large enough (so that c = lim p/n is small), as the intuition would suggest. Indeed, if 1 < √ c, the eigenvalues of 1 n XX T are all asymptotically compacted in the support

[(1 - √ c) 2 , (1 + √ c) 2
] and thus it is theoretically (asymptotically) impossible to tell whether C = I p or C is more structured (e.g., C = I p + L) from the observation of the spectral measure.

Proof of Theorem 2.6. When it comes to assessing the eigenvalues of a given matrix A, it first comes to mind to solve the determinant equation det(A -λI) = 0. This approach is not convenient for X of increasing dimensions and we have seen that the Stieltjes transform method is an appropriate substitute in this case. Here, since the low rank matrix L only induces a low rank perturbation of 1 n XX T , the use of Sylverster's identity (Lemma 2.3) will turn the large dimensional determinant equation into a small (fixed) dimensional one, and the method is now valid. This is the approach we pursue here.

Specifically, let us seek for the presence of an eigenvalue λ of 1 n XX T which is asymptotically greater than (1 + √ c) 2 . Our approach is to "isolate" the low rank contribution due to L from the "whitened" sample covariance matrix model with identity covariance. To this end, we use the following sequence of equivalences

0 = det 1 n XX T -λI p = det 1 n (I p + L) 1 2 ZZ T (I p + L) 1 2 -λI p = det I p + L det 1 n ZZ T -λ(I p + L) -1
where we recall X = C 1 2 Z. Obviously, det(I p + L) = 0 so that the first determinant can be discarded. For the second term, first recall from the resolvent identity (Lemma 2.1) that

(I p + L) -1 = I p -(I p + L) -1 L
so that we can isolate the (now well-understood) resolvent of the "whitened" model. That is, letting

Q(z) = ( 1 n ZZ T -λI p ) -1 , we write 0 = det 1 n ZZ T -λI p + λ(I p + L) -1 L = det Q -1 (λ) det I p + λQ(λ)(I p + L) -1 L .
Inverting the matrix 1 n ZZ T -λI p is (almost surely) licit for all large n, p as we demanded

λ > (1 + √ c) 2 . Now, denoting L = UDU T with L = diag( 1 , . . . , k ) and U = [u 1 , . . . , u k ] ∈ R p×k , we further have (I p + L) -1 L = (I p + UDU T ) -1 UDU T = U(I k + D) -1 DU T .
Plugging this expansion into the above equation, this is

0 = det Q -1 (λ) det I p + λQ(λ)U(I k + D) -1 DU T = det Q -1 (λ) det I k + λU T Q(λ)U(I k + D) -1 D
where in the last equality we applied Sylvester's identity. Since the first determinant does not cancel with λ > (1 + √ c) 2 , we finally have for all large n, p,

0 = det I k + λU T Q(λ)U(I k + D) -1 D .
From Theorem 2.3, we now know that

U T Q(λ)U = m(λ)I k + o • (1)
almost surely, for m(z) the Stieltjes transform of the Marcenko-Pastur law µ (the term I k arises from the fact that U T U = I k ). Consequently, by continuity of the determinant (as a polynomial of its entries), we have

0 = det I k + λm(λ)(I k + D) -1 D + o(1)
and thus, if such a λ exists, it must satisfy

λm(λ) = - 1 + i i + o(1).
for some i ∈ {1, . . . , k}. We thus need to understand when this equation has a solution.

To this end, observe that the function R \ supp(µ) → R, x → xm(x) =

x t-x µ(dt) is increasing on its domain of definition and that m(x) → 0 as x → ∞. Solving the expression of m(z) in Theorem 2.3, i.e.,

zcm(z) 2 -(1 -c -z)m(z) + 1 = 0 (2.18)
we further find that lim

x↓(1+ √ c) 2 xm(x) = - 1 + √ c √ c .
Thus, m(x) increases from -1+

- 1 + i i > - 1 + √ c √ c that is, whenever i > √ c.
Assuming this holds, we may then use again (2.18) to obtain, after multiplication by λ, that

c(λm(λ)) 2 -(1 -c -λ)(λm(λ)) + λ = 0 which, after replacement of λm(λ) by -1+ i i finally gives, as expected, that λ → 1 + i + c 1 + i i concluding the proof.
Figure 2.3 depicts the eigenvalues of 1 n XX T versus the Marcenko-Pastur law, in the scenario where C = I p + L with L of rank four, for various ratios p/n. As predicted by Theorem 2.6, the number of visible "spikes" outside the limiting support of 1 n XX T varies with p/n: as the ratio decreases, less spikes are visible.

SPIKED MODELS

Isolated eigenvectors

From a practical standpoint, we have seen that the presence of isolated eigenvalues in the spectrum of the sample covariance 1 n XX T reveals the presence of some "structure" in the population covariance C in the sense that C = I p . We have however also seen that the converse is not true: assuming a spiked model for C, the absence of isolated eigenvalue does not imply C = I p .

More interestingly, whether this "structure" is detected or not, one may wonder whether it can be estimated at all. More specifically, if C = I p + L with L = ∑ k i=1 i u i u T i , are the eigenvectors û1 , . . . , ûk of 1 n XX T associated to the k largest eigenvalues λ 1 , . . . , λ k good estimators for u 1 , . . . , u k ? Not surprisingly, the answer is here again twofold: i) if i ≤ √ c then ûi tends to be orthogonal to u i , while ii) if i > √ c, ûi is to some extent aligned to u i . The following theorem, originally due to Paul, quantifies this "to some extent".

Theorem 2.7 (Eigenvector alignment, from [START_REF] Paul | Asymptotics of sample eigenstructure for a large dimensional spiked covariance model[END_REF]). Under the setting of Theorem 2.6, let û1 , . . . , ûk be the eigenvectors associated with the largest k eigenvalues λ 1 > . . . > λ k of 1 n XX T . Further assume that 1 > . . . > k > 0 are all distinct. Then, for a, b ∈ R k unit norm CHAPTER 2. MATHEMATICAL BACKGROUND: RANDOM MATRIX THEORY deterministic vectors

a T ûi ûT i b -a T u i u T i b 1 -c -2 i 1 + c -1 i 1 i > √ c a.s.
-→0.

In particular,

|u T i ûi | 2 a.s. -→ 1 -c -2 i 1 + c -1 i 1 i > √ c .
Proof. We may first write that, for all large n, p almost surely,

a T ûi ûT i b = - 1 2πı Γ ρ i a T 1 n XX T -zI p -1 bdz
for Γ ρ i a small contour enclosing the almost sure limit ρ i of the eigenvalue λ i of 1 n XX T only.

Isolating 1 n ZZ T from 1 n XX T as in the previous proof, we have from Woodbury's identity, Lemma 2.7,

a T 1 n XX T -zI p -1 b = a T 1 n (I p + L) 1 2 ZZ T (I p + L) 1 2 -zI p -1 b = a T (I p + L) -1 2 1 n ZZ T -zI p -zL -1 (I p + L) -1 2 b = a T (I p + L) -1 2 Q(z)(I p + L) -1 2 b + za T (I p + L) -1 2 Q(z)UL(I k -zU T Q(z)UL) -1 U T Q(z)(I p + L) -1 2 b = a T (I p + L) -1 2 Q(z)(I p + L) -1 2 b + zm(z) 2 a T (I p + L) -1 2 L(I k -zm(z)L) -1 (I p + L) -1 2 b + o(1)
where Q(z) = ( 1 n ZZ T -zI p ) -1 and for the last equality we used U T Q(z)U = m(z)I k + o(1) as per Theorem 2.3. The complex integration of Q(z) on the contour Γ ρ i only brings a positive residue for the second right-hand side term owing to the inverse (I kzm(z)L) -1 which is singular for z = ρ i . We thus finally have

a T ûi ûT i b = - 1 2πı Γ ρ i zm(z) 2 a T (I p + L) -1 2 L(I k -zm(z)L) -1 (I p + L) -1 2 bdz + o(1). Since lim z→ρ i (z -ρ i )(I k -zm(z)L) -1 = -(ρ i m (ρ i ) + m(ρ i )) -1 -1 i u i u T i with m(ρ i ) = -1/(c + i ) and m (ρ i ) = 2 i (c + i ) -2 (-c + i ) -1
(these are obtained from the elements of proof of Theorem 2.6 and from (2.13)), we get that the residue associated to

(I k - zm(z)L) -1 is (-ρ i m (ρ i ) -m(ρ i )) -1 -1 i u i u T i = ( 2 i -c) -1 i u i u T i .
Thus, we finally get after elementary algebra,

a T ûi ûT i b = -a T u i u T i b ρ i m 2 (ρ i ) i 1 + i ( 2 i -c) -1 i = a T u i u T i b 1 -c -2 i 1 + c -1 i .
Figure 2.4 compares, in a single-spike scenario, the theoretical limit of | ûT 1 u 1 | 2 versus its empirical value for different 1 and different p, n with constant ratio p/n. It is important to note that the theoretical asymptotic phase transition phenomenon at 1 = √ c corresponds to a sharp non-differentiable change in the function 1 → | ûT 1 u 1 | 2 ; on real data, this sharp transition is only observed for extremely large values of n, p. This in particular means that, in practice, residual information is present below the phase transition.

SPIKED MODELS

| ûT 1 u 1 | 2 p = 100 p = 200 p = 400 1-c -2 1 1+c -1 1 √ c Figure 2.4: Simulated versus limiting | ûT 1 u 1 | 2 for X = C 1 2 Z, C = I p + 1 u 1 u T 1 , p/n = 1/3, for varying 1 .

Further discussions and other spiked models

As briefly discussed above, the "spiked model" terminology goes beyond sample covariance matrix models with C = I p + L, and L a low rank matrix. In the literature, spiked models loosely are referred to as "low rank perturbative" models in the following sense: there exists an underlying random matrix model X, the spectral measure of which converges to a well-defined measure with compact support and having eigenvalues converging to the support (i.e., no single eigenvalue isolates) which is modified in some way by a low rank matrix L; the resulting matrix has the same limiting spectral measure as that of X but with possibly some spurious eigenvalues.

Baik and Silverstein in [START_REF] Baik | Eigenvalues of large sample covariance matrices of spiked population models[END_REF] were the first to study spiked models, but their approach relied on applying the results on sample covariance matrix models (Theorem 2.4) to the specific case where C = I p + L. This approach requires to have a full understanding of a "more complex" statistical model before particularizing it to a low rank perturbation. The proof of Theorem 2.6 that we proposed follows a second wave of advances in spiked models, mostly triggered by the work of Benaych and Rao [START_REF] Benaych | The singular values and vectors of low rank perturbations of large rectangular random matrices[END_REF] (with a free probability approach), which is rather based on relating the perturbation matrix model to the underlying simple (non perturbed) matrix.

Among the popular spiked models, we have the following cases:

• the information-plus-noise model of the type

1 n (X + L)(X + L) T
with X ∈ R p×n having i.i.d. standard entries (zero mean, unit variance and finite fourth order moment) and L ∈ R p×n deterministic (or at least independent of X) of fixed rank k.

• the additive model of the type

Y + L
where Y ∈ R n×n is either of the type Y = 1 n X T X, X ∈ R p×n with standard i.i.d. entries, of Y = 1 √ n X with X symmetric with standard i.i.d. entries above and on the diagonal and L ∈ R n×n deterministic.

Each of these models has its own phase transition threshold (i.e., the value that eigenvalues of L must exceed for a spike to be observed), dominant eigenvalue limits, and eigenvector projections. These can all be determined with the aforementioned proof approach.

However, we will see later that, in practice, we will be confronted with more general forms of low rank perturbation models that do not fit this conventional "random matrix X and deterministic perturbation L" assumption.

In particular, L will often be a (possibly elaborate) function of X. Also, X itself, which will often stand for the "noisy" part of the data model (while L will in general comprises both the relevant information and some extra noise), may induce its own isolated eigenvalues. For instance, we shall see in Remark 3.7 in Section 3.1.2 that, depending on the ratios p/n and tr C 4 /(tr C 2 ) 2 , the random matrix {[XX T ] 2 ij } p i =j where X = C 1 2 Z and Z with i.i.d. standard entries, may have two isolated eigenvalues although all the eigenvalues of C remain in their limiting support.

Yet, despite these technical differences, the proof approaches of Theorem 2.6 and Theorem 2.7 remain essentially valid. We thus propose here to generalize the notion of "spiked models" to models of the type X + L where X is some reference, well understood, random matrix model (possibly inducing its own spikes) and L is a low rank matrix, possibly depending on X.

With this definition, the aforementioned sample covariance, information-plus-noise and additive models are in fact all equivalent to an additive model. Indeed, we may write

1 n (X + L)(X + L) T = Y + L Y = 1 n XX T , L = 1 n XL T + 1 n LX T + 1 n LL T and 1 n (I p + L) 1 2 XX T (I p + L) 1 2 = Y + L Y = 1 n XX T , L = 1 n XL T + 1 n L X + 1 n L L T
where in the second equation, letting L = UDU T , we denoted L = U((

I k + D) 1 2 - I k )U T .
In the remainder of the manuscript, we shall systematically exploit this generic modeling approach to spiked models.

Chapter 3

Spectral Behavior of Large Kernels Matrices and Neural Nets

Random Kernel Matrices

Kernel ridge regression

Kernel ridge regression, or least squares support vector machine (LS-SVM), as mentioned in Section 1.2.1, is a modification of the standard SVM [START_REF] Johan | Least squares support vector machine classifiers[END_REF] to overcome the drawbacks of SVM related to computational efficiency. In this subsection, we will focus on a twoclass GMM (see Definition 1) classification using LS-SVM as described below.

Given a training set {x i , y i } n i=1 of size n, where data x i ∈ R p and labels y i ∈ {-1, 1}, the objective of LS-SVM is to devise a decision function h(x) that ideally maps all x i in the training set to y i and subsequently all unknown data x to their corresponding y value. Here we denote x i ∈ C 1 if y i = -1 and x i ∈ C 2 if y i = 1 and shall say that x i belongs to class C 1 or class C 2 , respectively. Due to the often nonlinear separability of the data in their input space R p , in most cases, one associates the training data x i to some feature space H through a nonlinear mapping φ : x i → φ(x i ) ∈ H. Optimization methods are then used to define a separating hyperplane in H with direction vector α and a function h(x) = α T φ(x) + b that minimizes the training errors e i = y i -(α T φ(x i ) + b) that yields good generalization performance by minimizing the norm of α [START_REF] Smola | A tutorial on support vector regression[END_REF]. More specifically, the LS-SVM approach consists in minimizing the squared errors e 2 i , thus resulting in

min α,b L(α, e) = α 2 + λ n n ∑ i=1 e 2 i (3.1) s.t. y i = α T φ(x i ) + b + e i , i = 1, . . . , n
where λ > 0 is a penalty factor that weights the structural risk α 2 against the empirical one 1 n ∑ n i=1 e 2 i . The problem can be solved by introducing Lagrange multipliers β i , i = 1, . . . , n with solution α = ∑ n i=1 β i φ(x i ), where, letting y = [y 1 , . . . , where k(x) = φ(x) T φ(x j ) n j=1 ∈ R n . More precisely, x is associated to class C 1 if g(x) takes a small value (below a certain threshold ξ) and to class C 2 otherwise. 1With the "kernel trick" [START_REF] Schölkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF], as shown in (3.2) and (3.3), both in the training and testing steps, one only needs to evaluate the inner product φ(x i ) T φ(x j ) or φ(x) T φ(x j ), and never needs to know explicitly the mapping φ(•). We assume that the kernel is shiftinvariant and focus on kernel functions f : R → R that satisfy φ(x i ) T φ(x j ) = f ( x ix j2 /p) and shall redefine K and k(x) for data point x as

y n ] T , β = [β 1 , . . . , β n ] T , we obtain    β = Q I n -1 n 1 T n Q 1 T n Q1 n y = Q (y -b1 n ) b = 1 T n Qy 1 T n Q1 n (3.2) with Q = K + n λ I n -1 and K ≡ φ(x i ) T φ(x j )
K = f x i -x j 2 /p n i,j=1 , k(x) = f x -x j 2 /p n j=1
(3.4)

We will focus on the performance of LS-SVM, in the large n, p regime, by studying the asymptotic behavior of the decision function g(x) defined in (3.3). We assume that all x i 's are extracted from a two-class GMM (as in Definition 1), which allows for a thorough theoretical analysis. For notational convenience, we rewrite the GMM in Definition 1 as follows:

x i ∼ N (µ a , C a ) ⇔ x i = µ a + √ pω i (3.5)
such that ω i ∼ N (0, C a /p) for a ∈ {1, 2}. In particular, the non-trivial classification condition in Assumption 2 can be adapted to the binary classification setting as follows.

Assumption 4 (Non-trivial binary classification). As n → ∞, we have for a ∈ {1, 2} that 1. p/n = c → c ∈ (0, ∞) and n a /n = c a → ca ∈ (0, 1);

2. µ 1 -µ 2 = O(1) and max{ C a , C -1 a } = O(1) with | tr(C 1 -C 2 )| = O( √ p), C 1 -C 2 2 F = O(p); 3. for C • = n i n C i , 2 p tr C • → τ > 0 as n, p → ∞.
A key observation, also made in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF], is that, as a consequence of Assumption 4, for all pairs i = j,

1 p x i -x j 2 → τ (3.6)
almost surely as n, p → ∞ and the convergence is even uniform across all i = j. This remark is the crux of all subsequent results and can be seen as a manifestation of the "curse of dimensionality" with respect to the Euclidean distance in high-dimensional space, as discussed in Section 1.1.2. Our objective here is to assess the performance of LS-SVM, for all kernel function f that is three-times differentiable in a neighborhood of τ, under the setting of Assumptions 4, by studying the asymptotic behavior of the decision function g(x) defined in (3.3). Following [START_REF] El | The spectrum of kernel random matrices[END_REF][START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] and our introductory demonstrations performed in Section 1.1.2, under our basic settings, the convergence in (3.6) makes it possible to linearize the kernel matrix K around the matrix f (τ)1 n 1 T n , and thus the intractable nonlinear kernel matrix K can be asymptotically linearized in the large n, p regime. As such, since the decision function g(x) is explicitly defined as a function of K (through α and b as defined in (3.2)), one can work out an asymptotic linearization of g(x) as a function of the kernel function f and the statistics of the data. This analysis, presented in detail in Section A.1.2 of the Appendix, allows one to reveal the relationship between the performance of LS-SVM and the kernel function f as well as the given learning task, for Gaussian input data as n, p → ∞.

Mains results. Before going into our main results, a few notations need to be introduced. In the remainder of this section, we shall use the following deterministic and random elements notations:

P = I n - 1 n 1 n 1 T n ∈ R n×n , Ω ≡ [ω 1 , . . . , ω n ] ∈ R p×n ψ ≡ ω i 2 -E ω i 2 n i=1 ∈ R n .
Under Assumptions 4, following up [CBG16] and Section 1.1.2, for three-times differentiable f , one can approximate the kernel matrix K by K in such a way that

K -K a.s. -→0, K = -2 f (τ)(PΩ T ΩP + A) + f (0) -f (τ) + τ f (τ) I n
where K follows a spiked model and consists of i) PΩ T ΩP, a standard Gram/covariancelike random matrix model (of operator norm O(1)) and ii) A a small rank (at most eight here) matrix that is of operator norm O(n), which depends both on P, Ω, ψ and on the class statistics µ 1 , µ 2 and C 1 , C 2 . The same analysis is applied to the vector k(x) by similarly defining the following random variables for a new datum x ∈ C a , a ∈ {1, 2}:

ω x ≡ (x -µ a )/ √ p ∈ R p , ψ x ≡ ω x 2 -E ω x 2 ∈ R.
Based on the (operator norm) approximation K -K → 0, a Taylor expansion is then performed on Q = (K + n λ I) -1 to obtain an asymptotic approximation of Q, and subsequently on β and b which depend explicitly on Q. At last, plugging these results into (3.3), one obtains the following theorem that characterizes the asymptotic behavior of the decision function g(x).

Theorem 3.1 (Asymptotic behavior of g(x)). Let Assumption 4 hold and g(x) be defined by (3.3). Then, for a kernel function f that is three-times differentiable in a neighborhood of τ, we have, as n, p → ∞, n (g(x)g(x))

a.s.

-→0

where

g(x) = c 2 -c 1 + λ R -2c 1 c 2 2 D , if x ∈ C 1 c 2 -c 1 + λ R + 2c 2 1 c 2 D , if x ∈ C 2 (3.7) with R = - 2 f (τ) n y T PΩ T ω x - 4c 1 c 2 f (τ) √ p (µ 2 -µ 1 ) T ω x + 2c 1 c 2 f (τ) 1 p tr(C 2 -C 1 )ψ x (3.8) D = - 2 f (τ) p µ 2 -µ 1 2 + f (τ) p 2 tr 2 (C 2 -C 1 ) + 2 f (τ) p 2 C 2 -C 1 2 F . (3.9)
Leaving the proof to Section A.1.2 in the Appendix, Theorem 3.1 tells us that the decision function g(x) has an asymptotic equivalent g(x) that consists of three parts:

1. the deterministic term c 2c 1 of order O(1) that depends on the number of instances in each class of the training set, which essentially comes from the term 1 T n y/n in b;

70CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS 2. the "noisy" term R of order O(p -1 ) which is a function of the zero mean random variables Ω, ω x and ψ x , thus in particular E[R] = 0; 3. the "informative" term containing D, also of order O(p -1 ), which features the deterministic differences between the two classes.

From Theorem 3.1, under the settings of Assumption 4, for Gaussian data x ∈ C a , a ∈ {1, 2}, we can show that g(x) (and therefore g(x)) converges to a random Gaussian variable the mean and variance of which are given in the following theorem. The proof is deferred to Section A.1.3 in the Appendix. Theorem 3.2 (Asymptotic Gaussian behavior). Under the setting of Theorem 3.1, we have, as n, p → ∞,

n(g(x) -G a ) → 0 in distribution, where G a ∼ N ( Ḡa , V G a ) with Ḡa = c 2 -c 1 -2c 2 • c 1 c 2 λD, a = 1 c 2 -c 1 + 2c 1 • c 1 c 2 λD, a = 2 V G a = 8λ 2 c 2 1 c 2 2 (V a 1 + V a 2 + V a 3 ) and V a 1 = ( f (τ)) 2 p 4 tr 2 (C 2 -C 1 ) C a 2 F V a 2 = 2 ( f (τ)) 2 p 2 (µ 2 -µ 1 ) T C a (µ 2 -µ 1 ) V a 3 = 2 ( f (τ)) 2 np 2 tr C 1 C a c 1 + tr C 2 C a c 2 .
Theorem 3.2 is our main practical result as it allows one to evaluate the large n, p performance of LS-SVM for Gaussian data. Focusing on the implications of Theorem 3.1-3.2, several remarks and discussions are in order. Remark 3.1 (Dominant bias). From Theorem 3.1, under the key Assumption 4, both the random noise R and the deterministic "informative" term D are of order O(n -1 ), which means that g(x) = c 2c 1 + O(n -1 ). This result somehow contradicts the classical decision criterion proposed in [START_REF] Johan | Least squares support vector machine classifiers[END_REF], based on the sign of g(x), i.e., x is associated to class C 1 if g(x) < 0 and to class C 2 otherwise. When c 1 = c 2 , this would lead to an asymptotic classification of all new data x's in the same class as n → ∞. Practically speaking, this means for n, p large that the decision function g(x) of a new datum x lies (sufficiently) away from 0 (0 being the classically considered threshold), so that the sign of g(x) is constantly positive (in the case of c2 > c1 ) or negative (in the case of c2 < c1 ). As such, all new data will be trivially classified into the same class. Instead, a first result of Theorem 3.1 is that the decision threshold ξ should be taken as ξ = ξ n = c 2c 1 + O(n -1 ) for imbalanced classification problems.

The conclusion of Remark 3.1 was in fact already known since the work of [GSL + 02] who reached the same conclusion through a Bayesian inference analysis, for all finite n, p. From their Bayesian perspective, the term c 2c 1 appears in the "bias term" b under the form of prior class probabilities P(y = -1), P(y = 1) and allows for adjusting classification problems with different prior class probabilities in the training and test sets. This idea of a (static) bias term correction has also been applied in [START_REF] Evgeniou | Image representations for object detection using kernel classifiers[END_REF] in order to improve the validation set performance. We visually confirm the problem of imbalanced datasets in Remark 3.1 by Figure 3.1 with c 1 = 1/4 and c 2 = 3/4, where the histograms of g(x) for x ∈ C 1 and C 2 center somewhere close to c 2c 1 = 0.5, thus resulting in a trivial classification by assigning all new data to C 2 if one takes ξ = 0 because P(g(x) < ξ | x ∈ C 1 ) → 0 and P(g(x) > ξ | x ∈ C 2 ) → 1 as n, p → ∞ (the convergence being in fact an equality for finite n, p in this particular figure). 0.49 0.5 0.51

g(x) x∈C 1 g(x) x∈C 2 G 1 G 2 Figure 3.1: Gaussian approximation of g(x) n = 256, p = 512, c 1 = 1/4, c 2 = 3/4, λ = 1, Gaussian kernel with σ 2 = 1, x ∼ N (µ a , C a ) with µ a = [0 a-1 ; 3; 0 p-a ], C 1 = I p and {C 2 } i,j = 0.4 |i-j| (1 + 5/ √ p).
An alternative to alleviate this imbalance issue is to normalize the label vector y. From the proof of Theorem 3.1 in Section A.1.2 we see the term c 2c 1 is due to the fact that in b one has 1 T n y/n = c 2c 1 = 0. Thus, one may normalize the labels y i as y * i = -1/c 1 if x i ∈ C 1 and y * i = 1/c 2 if x i ∈ C 2 , so that 1 T n y * = 0. This formulation is also referred to as the Fisher's targets: {-n/n 1 , n/n 2 } in the context of kernel Fisher discriminant analysis [BA00, MRW + 99]. With these normalized labels y * , we have the following lemma that reveals the connection between the corresponding decision function g * (x) and g(x). Lemma 3.1. Let g(x) be defined by (3.3) and g * (x) be defined as g * (x) = (β * ) T k(x) + b * , with (β * , b * ) given by (3.2) for y * in the place of y, where y

* i = -1/c 1 if x i ∈ C 1 and y * i = 1/c 2 if x i ∈ C 2 . Then, g(x) -(c 2 -c 1 ) = 2c 1 c 2 g * (x).
Proof. From (3.2) and (3.3) we get

g(x) = y T Q - Q1 n 1 T n Q 1 T n Q1 n k(x) + y T Q1 n 1 T n Q1 n = y T v with v = Q -Q1 n 1 T n Q 1 T n Q1 n k(x) + Q1 n 1 T n Q1 n . Besides, note that 1 T n v = 1. We thus have g(x) -(c 2 -c 1 ) = y T v -(c 2 -c 1 )1 T n v = 2c 1 c 2 y -(c 2 -c 1 )1 n 2c 1 c 2 T v = 2c 1 c 2 (y * ) T v = 2c 1 c 2 g * (x)
72CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS which concludes the proof.

As a consequence of Lemma 3.1, instead of Theorem 3.2 for standard labels y i ∈ {-1, 1}, one would have the following corollary for the corresponding Gaussian approximation of g * (x) when normalized labels y * i ∈ {-c -1 1 , c -1 2 } are used.

Corollary 3.1 (Gaussian approximation of g * (x)). Under the setting of Theorem 3.1, and with g * (x) defined in Lemma 3.1, n(g * (x) -G * a ) → 0 in distribution as n, p → ∞, where

G * a ∼ N ( Ḡ * a , V G * a )
with

Ḡ * a = -c 2 λD, a = 1 +c 1 λD, a = 2 V G * a = 2λ 2 (V a 1 + V a 2 + V a 3 )
and D is defined by (3.9), V a 1 , V a 2 and V a 3 as in Theorem 3.2.

(n -1 ) from zero) instead of c 2 -c 1 = 1/2. Still,
even in the case where normalized labels y * are used as observed in Figure 3.2 (where the histograms cross at about -0.004 ≈ 1/n), taking ξ = 0 as a decision threshold may not be an appropriate choice, as Ḡ * 1 = -Ḡ * 2 .

-2 0 2

•10 -2

g * (x) x∈C 1 g * (x) x∈C 2 G * 1 G * 2 Figure 3
.2: Gaussian approximation of g * (x), under the same setting as Figure 3.1.

Remark 3.2 (Insignificance of λ).

As a direct result of Theorem 3.1 and Remark 3.1, note in (3.7) that g(x) -(c 2c 1 ) is proportional to the hyperparameter λ, which indicates that, rather surprisingly, the tuning of λ is (asymptotically) of no importance when n, p → ∞ since it does not alter the classification statistics when one uses the sign of g(x) -(c 2c 1 ) for the decision.

Remark 3.2 is only valid under Assumption 4 and λ = O(1), i.e., λ is considered to remain constant as n, p → ∞. Recall that this is in sharp contrast with [START_REF] Caponnetto | Optimal rates for the regularized least-squares algorithm[END_REF] where

λ = O( √ n) (or O(n)
, depending on the problem) is claimed optimal in the large n only regime. From (1.6), we see here that λ = O(1) is rate-optimal under the present large n, p setting; yet we believe that more elaborate kernels (such as those explored in [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF] and Section 3.1.2) may allow for improved performances (not in the rate but in the constants), possibly for different scales of λ.

Denote Q

(x) = 1 √ 2π ∞
x exp -t 2 /2 dt the Q-function with respect to the standard Gaussian distribution. From Theorem 3.2 and Corollary 3.1, we now have the following immediate corollary for the (asymptotic) classification error rate.

Corollary 3.2 (Asymptotic error rate). Under the setting of Theorem 3.1, for a threshold ξ n possibly depending on n, as n, p → ∞,

P(g(x) > ξ n | x ∈ C 1 ) -Q ξ n -Ḡ1 V G 1 → 0 (3.10) P(g(x) < ξ n | x ∈ C 2 ) -Q Ḡ2 -ξ n V G 2 → 0 (3.11)
with Ḡa and V G a given in Theorem 3.2.

Obviously, Corollary 3.2 is only meaningful when ξ n = c 2c 1 + O(n -1 ) as recalled earlier. Besides, it is clear from Lemma 3.1 and Corollary 3.

1 that P(g(x) > ξ n | x ∈ C a ) = P(2c 1 c 2 g * (x) > ξ n -(c 2 -c 1 ) | x ∈ C a )
, so that Corollary 3.2 extends naturally to g * (x) when normalized labels y * are applied.

Corollary 3.2 allows one to compute the asymptotic misclassification rate as a function of Ḡa and V G a and the threshold ξ n . Combined with Theorem 3.1, one may note the significance of a proper choice of the kernel function f . For instance, if f (τ) = 0, the term µ 2µ 1 vanishes from the mean and variance of G a , meaning that the classification of LS-SVM will not rely (at least asymptotically and under Assumption 4) on the differences in means of the two classes. Figure 3.3 corroborates this finding with the same theoretical Gaussian approximations G 1 and G 2 in subfigures (a) and (b). When µ 2µ 1 2 varies from 0 in (a) to 18 in (b), the distribution of g(x), and in particular, the overlap between two classes, remain almost the same.

-5 0 5

•10 -2 (a) ∆µ = 0

-5 0 5 •10 -2 (b) ∆µ = 3 Figure 3.3: Gaussian approximation of g(x), n = 256, p = 512, c 1 = c 2 = 1/2, λ = 1, polynomial kernel with f (τ) = 4, f (τ) = 0, and f (τ) = 2. x ∼ N (µ a , C a ), with µ a = 0 a-1 ; ∆µ; 0 p-a , C 1 = I p and {C 2 } i,j = .4 |i-j| (1 + 5/ √ p).
In terms of the kernel function f , if f (τ) = 0, the information about the statistical means of the two different classes is lost and will not help perform the classification. Nonetheless, we find that, rather surprisingly, if one further assumes tr C 1 = tr C 2 + o( √ p) (which is beyond the minimum "distance" rate in Assumption 4), using a kernel 74CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS f that satisfies f (τ) = 0 results in V G a = 0 while Ḡa may remain non-zero, thereby ensuring a vanishing misclassification rate (as long as f (τ) = 0). Intuitively speaking, the kernels with f (τ) = 0 play an important role in extracting the covariance "shape" information of both classes, making the classification extremely accurate even in cases that are deemed impossible to classify according to (1.6). This phenomenon was also remarked in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] and deeply investigated in [START_REF] Couillet | Random matrix improved subspace clustering[END_REF]. Figure 3.4 substantiates this finding for µ 1 = µ 2 , C 1 = I p and {C 2 } i,j = .4 |i-j| , for which tr C 1 = tr C 2 = p. We observe a rapid drop of the classification error as f (τ) gets close to 0.

= 1/4, c 1 = c 2 = 1/2, λ = 1, polynomial kernel with f (τ) = 4, f (τ) = 2. x ∼ N (µ a , C a ), with µ 1 = µ 2 = 0 p , C 1 = I p and {C 2 } i,j = .4 |i-j| .
Remark 3.3 (Conditions on the kernel function f ). From Theorem 3.2 and Corollary 3.1, one observes that | Ḡ1 -Ḡ2 | is always proportional to the "informative" term D and should, for fixed V G a , be made as large as possible to avoid the overlap of g(x) for x from different classes. Since V G a does not depend on the signs of f (τ) and f (τ), it is easily deduced that, to achieve optimal classification performance, one needs to choose the kernel function f such that f (τ) > 0, f (τ) < 0 and f (τ) > 0.

Incidentally, the condition in Remark 3.3 is naturally satisfied for the Gaussian kernel f (x) = exp -x/(2σ 2 ) for any σ, meaning that, even without specific tuning of the kernel parameter σ through cross validation or other techniques, LS-SVM is expected to perform rather well with a Gaussian kernel, which is not always the case for polynomial kernels. This especially entails, for a second-order polynomial kernel given by f (x) = a 2 x 2 + a 1 x + a 0 , that attention should be paid to meeting the aforementioned condition when tuning the kernel parameters a 2 , a 1 and a 0 . Figure 3.5 attests of this remark with Gaussian input data. A rapid increase in classification error rate can be observed both in theory and in practice as soon as the condition f (τ) < 0, f (τ) > 0 is no longer satisfied.

In Figure 3.6 we provide a direct visualization of (the "local" behavior of) different kernel functions f with the same or opposite f (τ), f (τ), f (τ). The Gaussian kernel f (x) = exp(-x/2) (in red) and the quadratic kernel (in BLUE) having the same values of f (τ), f (τ) and f (τ) both yield satisfying performance, while by inverting the sign of either f (τ) or f (τ), the misclassification rate increases rapidly.

Clearly, for practical use, one needs to know in advance the value of τ before training

(τ) = 4, f (τ) = -1 Figure 3.5: Performance of LS-SVM, n = 256, p = 512, c 1 = c 2 = 1/2, λ = 1, polynomial kernel. x ∈ N (µ a , C a ), with µ a = [0 a-1 ; 2; 0 p-a ], C 1 = I p and {C 2 } i,j = .4 |i-j| (1 + 4/ √ p).
so that the kernel f can be properly chosen during the training step. The estimation of τ is possible, in the large n, p regime, with the following lemma.

Lemma 3.2. Under Assumption 4, as n → ∞,

2 n n ∑ i=1 1 p x i - 1 n n ∑ i=1 x i 2 a.s.
-→τ.

(3.12)

Proof. First observe that 2 n n ∑ i=1 1 p x i - 1 n n ∑ i=1 x i 2 = 2c 1 c 2 µ 2 -µ 1 2 p + 2 n n ∑ i=1 ω i -ω 2 + κ with κ = 4 n √ p (µ 2 -µ 1 ) T -c 2 ∑ x i ∈C 1 ω i + c 1 ∑ x j ∈C 2 ω j and ω = 1 n ∑ n i=1 ω i . According to Assumption 4 we have 2c 1 c 2 p µ 2 -µ 1 2 = O(n -1
). The term κ is a linear combination of independent zero-mean Gaussian variables and thus κ ∼ N (0,

Var[κ]) with Var[κ] = 16c 1 c 2 np 2 (µ 2 -µ 1 ) T (c 2 C 1 + c 1 C 2 ) (µ 2 -µ 1 ) = O(n -3
). We thus deduce from Chebyshev's inequality and the Borel-Cantelli lemma that κ a.s.

-→0.

We then work on the last term

2 n ∑ n i=1 ω i -ω 2 as 2 n n ∑ i=1 ω i -ω 2 = 2 n n ∑ i=1 ω i 2 -2 ω 2 .
Since ω ∼ N (0, C • /np), we deduce that ω 2 a.s.

-→0. Ultimately by the strong law of large numbers, we have 2 n ∑ n i=1 ω i 2 a.s.

-→τ, which concludes the proof.

Remark 3.4 (Special case: means-dominant). When the difference in means µ 2µ 1 2 is largely dominant over (tr(C 2 -C 1 )/p) 2 and C 2 -C 1 2 F /p, from Theorem 3.2, both Ḡa -(c 2c 1 ) and V G a are (approximately) proportional to f (τ), which eventually makes the choice of the kernel irrelevant (as long as f (τ) = 0). This result also holds true for G * a and V G * a when normalized labels y * are applied, as a result of Lemma 3.1.

76CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS

0 1 2 3 4 5 f (τ) x f (x)
kernel function error Gaussian kernel f (x) = exp(-x/2) 8.6% quadratic kernel with same f (τ), f (τ) and f (τ)

8.8% quadratic kernel with same f (τ), f (τ), while f (τ) of opposite sign 66.4% quadratic kernel with same f (τ), f (τ) while f (τ) of opposite sign 32.9%

= c 2 = 1/2 and λ = 1, with µ a = [0 a-1 ; 2; 0 p-a ], C 1 = I p and {C 2 } i,j = .4 |i-j| (1 + 4/ √ p).
Practical consequences. 2 When the classification performance of real-world datasets is concerned, our theory may be limited by: i) the fact that it is based on an asymptotic results and allows for an estimation error only of order O(p -1/2 ) between theory and practice and ii) the strong Gaussian assumption for the input data. However, when applied to real-world datasets, here to the popular MNIST [LBBH98] and Fashion-MNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF] datasets, our asymptotic results, theoretically only applicable for Gaussian data, show an unexpectedly good predictive behavior. Here we consider a two-class classification problem with a training set of n = 256 vectorized images of size p = 784 randomly selected from the MNIST and Fashion-MNIST datasets (numbers 8 and 9 in both cases as an example). Then a test set of n test = 256 is used to evaluate the classification performance. Means and covariances are empirically obtained from the full set of 11 800 MNIST images (5 851 images of number 8 and 5 949 of number 9) and of 11 800 Fashion-MNIST images (5 851 images of number 8 and 5 949 of number 9), respectively. Despite the obvious non-Gaussianity as well as the clearly different nature of the input data (from the two datasets), the distribution of g(x) is still surprisingly close to its Gaussian approximation computed from Theorem 3.2, as shown in Figure 3.7 for MNIST (left) and Fashion-MNIST (right), respectively.

In Figure 3.8 we plot the misclassification rate as a function of the decision threshold ξ for MNIST and Fashion-MNIST data (number 8 and 9). We observe that the conclusion from Remark 3.1, Lemma 3.1 and Corollary 3.1 that the decision threshold should approximately be c 2c 1 rather than 0 approximately holds true in both cases.

In Figure 3.9 we evaluate the performance of LS-SVM on the MNIST and Fashion-MNIST datasets as a function of the kernel parameter σ of Gaussian kernel f (x) = exp(-x/2σ 2 ). Surprisingly, we face here the situation where there is little difference in the performance of LS-SVM as soon as σ 2 is away from 0, which likely comes from the fact that the difference in means µ 2µ 1 is so large that it becomes predominant over the influence of covariances as mentioned in Remark 3.4. This argument is numerically sustained by Table 3.1. The gap between theory and practice observed as σ 2 → 0 is likely a result of the finite n, p rather than of the Gaussian assumption of the input data, since we continue to observe a similar behavior when one artificiality adds Gaussian white noise to the image data. a "concentration" phenomenon of the normalized Euclidean distance x ix j 2 /p, the classification performance depends on a local behavior of the kernel function f and the nonlinear kernel matrix K is asymptotically close to the linear Gram/covariance matrix model with several additive spikes. Although derived from an unrealistic GMM, our theoretical results showed an unexpected close match to simulations on popular realworld datasets.

Inner-product kernels

The results from the previous section are in essence built upon a local expansion of the nonlinear function f , which we recall follows from the "concentration" of the similarity measures x ix j 2 /p or x T i x j /p around a single value of the smooth domain of f , therefore disregarding most of the domain of f . In this section, following [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF][START_REF] Do | The spectrum of random kernel matrices: universality results for rough and varying kernels[END_REF], we study the inner product kernel f (x T i x j / √ p)/ √ p which avoids the concentration effects with the more natural √ p normalization. With the flexible tool of orthogonal polyno- mials, we are able to prove universal results which solely depend on the first two order moments of the data distribution and allow for nonlinear functions f that need not even be differentiable. As a practical outcome of our theoretical results, we propose an extremely simple piecewise constant function which is spectrally equivalent and thus performs equally well as arbitrarily complex functions f , while inducing enormous gains in both storage and computational complexity.

Let x 1 , . . . , x n ∈ R p be n feature vectors drawn independently from the following two-class (C 1 and C 2 ) mixture model, which is a special case of the general multivariate mixture model in Definition 1 with

C a = I p + E a , a = 1, 2. C 1 : x = µ 1 + (I p + E 1 ) 1 2 z C 2 : x = µ 2 + (I p + E 2 ) 1 2 z (3.13)
each having cardinality n/2 (see for more discussions in Remark 3.6 below), for some deterministic µ a ∈ R p , E a ∈ R p×p , a = 1, 2 and random vector z ∈ R p having i.i.d. entries of zero mean, unit variance and bounded moments. As has been discussed at length in Section 1.1.2, to ensure that the information of µ a , E a is neither (asymptotically) too simple nor impossible to be extracted from the noisy features, we adapt the two-class non-trivial classification condition in Assumption 4 to the covariance setting C a = I p + E a as follows.

Assumption 5 (Non-trivial classification). As n → ∞, we have for a ∈ {1, 2}

1. p/n = c → c ∈ (0, ∞), 2. µ a = O(1), E a = O(p -1/4 ), | tr(E a )| = O( √ p) and E a 2 F = O( √ p).
Remark 3.5 (On the advantage of proper scaling). It is worth noting that, compared to Assumption 4 where C 1 -C 2 2 F = O(p), here with Assumption 5 the mixture separation is performed with a higher precision with E 1 -E 2 2 F = O( √ p). In this respect, the "properly scaled" inner-product kernels investigated here is more powerful than the "improperly scaled" ones of form f (x T i x i /p), in separating the covariance "shapes" of Gaussian mixtures. Following [START_REF] El | On Information Plus Noise Kernel Random Matrices[END_REF][START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF] we consider the following random inner-product kernel matrix

K = δ i =j f (x T i x j / √ p)/ √ p n i,j=1 (3.14)
for f : R → R satisfying Assumption 3. As in [START_REF] El | On Information Plus Noise Kernel Random Matrices[END_REF][START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF], the diagonal elements f (x T i x i / √ p) have been discarded. Indeed, under Assumption 7,

x T i x i / √ p = O( √ p)
which is an "improper scaling" for the evaluation by f (unlike x T i x j / √ p which properly scales as O(1) for all i = j).

In the absence of discriminative information (null model), i.e., if µ a = 0 and E a = 0 for a = 1, 2, we write

K = K N with [K N ] ij = δ i =j f (z T i z j / √ p)/ √ p. (3.15) Letting ξ p ≡ z T i z j / √ p, by the central limit theorem, ξ p d -→N (0, 1) as p → ∞. As such, the [K N ] ij , 1 ≤ i = j ≤ n,
asymptotically behave like a family of dependent standard Gaussian variables to which f is applied. In order to analyze the joint behavior of this family, we exploit some useful concepts of the theory of orthogonal polynomials and, in particular, of the class of Hermite polynomials defined with respect to the standard Gaussian distribution [START_REF] Abramowitz | Handbook of mathematical functions: with formulas, graphs, and mathematical tables[END_REF][START_REF] George E Andrews | Special functions[END_REF].

For a real probability measure µ, we denote the set of orthogonal polynomials with respect to the scalar product f , g = f gdµ as {P l (x), l = 0, 1, . . .}, obtained from the Gram-Schmidt procedure on the monomials {1, x, x 2 , . . .} such that P 0 (x) = 1, P l is of degree l and P l 1 , P l 2 = δ l 1 -l 2 . By the Riesz-Fischer theorem [Rud64, Theorem 11.43], for any function f ∈ L 2 (µ), the set of squared integrable functions with respect to •, • , one can formally expand f as

f (x) ∼ ∞ ∑ l=0 a l P l (x), a l = R f (x)P l (x)dµ(x) (3.16)
where " f ∼ ∑ ∞ l=0 P l " indicates that f -∑ N l=0 P l → 0 as N → ∞ (and f 2 = f , f ). To investigate the asymptotic behavior of K and K N as n, p → ∞, we position ourselves, as [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF][START_REF] Do | The spectrum of random kernel matrices: universality results for rough and varying kernels[END_REF] under Assumption 3, which roughly says that the polynomial approximation of the kernel function f is accurate, for p large, when we truncate the above expansion at a large but finite degree L.

The limiting parameters a 0 , a 1 , a 2 and ν appearing in Assumption 3 are simply (generalized) moments of the standard Gaussian measure involving f . Precisely,

a 0 = E[ f (ξ)], a 1 = E[ξ f (ξ)], √ 2a 2 = E[(ξ 2 -1) f (ξ)] = E[ξ 2 f (ξ)], ν = Var[ f (ξ)] ≥ a 2 1 + a 2 2
for ξ ∼ N (0, 1). These parameters are of crucial significance in determining the eigenspectrum behavior of K. Note that a 0 will not affect the classification performance, as described below.

80CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS Remark 3.6 (On a 0 ). In the present case of balanced mixtures (equal cardinalities for C 1 and C 2 ), a 0 contributes to the polynomial expansion of K N (and K) as a non-informative perturbation of the form a 0 (1 n 1 T n -I n )/ √ p. Since 1 n is orthogonal to the "class-information vector" [1 n/2 , -1 n/2 ],

its presence does not impact the classification performance. If mixtures are unbalanced, the vector 1 n may tend to "pull" eigenvectors aligned to [1 n 1 , -1 n 2 ], with n i the cardinality in C i , so away from purely noisy eigenvectors and thereby impacting the classification performance. See [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] for similar considerations.

It was shown in [CS13, DV13] that, for independent z i 's with independent entries, the empirical spectral measure (see Definition 5) of the null model K N has an asymptotically deterministic behavior as n, p → ∞ with p/n → c ∈ (0, ∞), which depends on the nonlinear f solely via the two parameters (a 1 , ν), as recalled below.

Theorem 3.3 (Theorem 3.4 in [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF], Theorem 3 in [START_REF] Do | The spectrum of random kernel matrices: universality results for rough and varying kernels[END_REF]). Under some regularity condition for the kernel function f (see Assumption 3 below), the empirical spectral measure of K converges weakly and almost surely, as n, p → ∞ with p/n → c ∈ (0, ∞), to a probability measure µ. The latter is uniquely defined through its Stieltjes transform m : C + → C + , z → (tz) -1 µ(dt) (see also Definition 6 in Section 2.2), given as the unique solution in C + of the (cubic) equation 3

- 1 m(z) = z + a 2 1 m(z) c + a 1 m(z) + ν -a 2 1 c m(z) with a 1 = E[ξ f (ξ)] and ν = Var[ f (ξ)] ≥ a 2 1 for standard Gaussian ξ ∼ N (0, 1).
Moreover, Theorem 3.3 is "universal" with respect to the law of the (independent) entries of z i . While universality is classical in random matrix results, with mostly first and second order statistics involved, the present universality result is much less obvious since i) the nonlinear application f (x T i x j / √ p) depends in an intricate manner on all moments of x T i x j and ii) the entries of K N are strongly dependent. In essence, universality still holds here because the convergence speed to Gaussian of x T i x j / √ p is sufficiently fast to compensate the residual impact of higher order moments in the spectrum of K N .

Remark 3.7 (Non-informative spikes). Despite its "universality", Theorem 3.3 only characterizes the limiting spectral measure µ of the kernel matrix K, and thus allows for a "limited" number of isolated eigenvalues to escape from the support of µ. Indeed, as discussed at the end of Section 2.3, depending on the value of p/n, it was shown in [KC17, FM19] that, for continuously differentiable kernel function f , if a 2 = 0, then K has at most two spikes outside supp(µ), while for a 2 = 0 all eigenvalues of K lie in supp(µ) almost surely.

As an illustration, Figure 3.10a compares the empirical spectral measure of K N to the limiting measure obtained from Theorem 3.3.

From a technical viewpoint, our objective here is to go beyond the null model described in Theorem 3.3 by providing a tractable random matrix equivalent K for the kernel matrix K, in the sense that K -K → 0 almost surely in operator norm, as n, p → ∞. This convergence allows one to identify the eigenvalues and isolated eigenvectors (that can be used for spectral clustering purpose) of K by means of those of K via, for instance Lemma 2.10. More importantly, while not visible from the expression of K, the impact of

(x) = sign(x), Gaussian z i , n = 2 048, p = 512, µ 1 = -[3/2; 0 p-1 ] = -µ 2 and E 1 = E 2 = 0. X = [x 1 , . . . , x n ] ∈ R p×n with x 1 , . . . , x n/2 ∈ C 1 and x n/2+1 , . . . , x n ∈ C 2 .
the mixture model (µ 1 , µ 2 , E 1 , E 2 ) on K is readily accessed from K and easily related to the Hermite coefficients ( a 1 , a 2 , ν ) of f . This allows us to further investigate how the choice of f impacts the asymptotic feasibility and efficiency of spectral clustering from the top eigenvectors of K.

Main results. The main idea for the asymptotic analysis of K comes in two steps: first, by an expansion of x T i x j as a function of z i , z j and the statistical mixture model parameters µ, E, we decompose x T i x j (under Assumption 7) into successive orders of magnitudes with respect to p; this, as we will show, further allows for a Taylor expansion of f (x T i x j / √ p) for at least twice differentiable functions f around its dominant term f (z T i z j / √ p). Then, we rely on the orthogonal polynomial approach of [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF] to "lin- earize" the resulting matrix terms { f (x T i x j / √ p)}, { f (x T i x j / √ p)} and { f (x T i x j / √ p)} (all terms corresponding to higher order derivatives asymptotically vanish) and use Assumption 3 to extend the result to arbitrary square-summable f .

Our main conclusion is that K asymptotically behaves like a matrix K following a so-called "spiked random matrix model" in the sense that K = K N + KI is the sum of the full-rank "noise" matrix K N having compact limiting spectrum and a low-rank "information" matrix KI [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF][START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF].

We first show that K can be asymptotically approximated as K N + K I with K N defined in (3.15) and K I an additional (so far full-rank) term containing the statistical information of the mixture model.

As announced, we start by decomposing x T i x j into a sequence of terms of successive orders of magnitude using Assumption 7 and

x i = µ a + (I p + E a ) 1 2 z i , x j = µ b + (I p + E b )
1 2 z j for x i ∈ C a and x j ∈ C b . We have precisely, for i = j,

x T i x j √ p = µ T a µ b √ p + 1 √ p (µ T a (I p + E b ) 1 2 z j + µ T b (I p + E a ) 1 2 z i ) + 1 √ p z T i (I p + E a ) 1 2 (I p + E b ) 1 2 z j = z T i z j √ p O(1) + z T i (E a + E b )z j 2 √ p ≡A ij =O(p -1/4 ) + µ T a µ b + µ T a z j + µ T b z i √ p - z T i (E a -E b ) 2 z j 8 √ p ≡B ij =O(p -1/2 ) +o(p -1/2 ) (3.17) 82CHAPTER 3.

SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS

where in particular we performed a Taylor expansion of (I p + E a )

1 2 (since E a = O(p -1 4 )) around I p , and used the fact that with high probability z T i E a z j = O(p 1/4 ) and z

T i (E a - E b ) 2 z j = O(1).
As a consequence of this expansion, for at least twice differentiable f ∈ L 2 (µ p ), we have

K ij = f (x T i x j / √ p) √ p = f (z T i z j / √ p) √ p + f (z T i z j / √ p) √ p (A ij + B ij ) + f (z T i z j / √ p) 2 √ p A 2 ij + o(p -1 )
where o(p -1 ) is understood with high probability and uniformly over i, j ∈ {1, . . . , n}.

Recall that, this entry-wise expansion up to order o(p -1 ) is sufficient since, matrix-wise, if

A ij = o(p -1 ) uniformly on i, j, from A ≤ p A ∞ = p max i,j |A ij |, we have A = o(1) as n, p → ∞.
In the particular case where f is a monomial of degree k ≥ 2, this implies the following result.

Proposition 3.1 (Monomial f ). Under Assumptions 3 and 5, let f (

x) = x k , k ≥ 2. Then, as n, p → ∞, K -(K N + K I ) → 0 (3.18)
almost surely, with K N defined in (3.15) and

K I = k √ p (Z T Z/ √ p) •(k-1) • (A + B) + k(k -1) 2 √ p (Z T Z/ √ p) •(k-2) • (A) •2 (3.19)
for Z = [z 1 , . . . , z n ] ∈ R p×n and A, B ∈ R n×n defined in (3.17) with A ii = B ii = 0. Here X • Y denotes the Hadamard product between X, Y and X •k the k-th Hadamard power, i.e., [X

•k ] ij = (X ij ) k .
Since f ∈ L 2 (µ) can be decomposed into its Hermite polynomials, Proposition 3.1 along with Theorem 3.3 allows for an asymptotic quantification of K. However, the expression of K I in (3.19) does not so far allow for a thorough understanding of the spectrum of K, due to 2) the delicate Hadamard products between purely random (Z T Z) and informative matrices (A, B) and ii) the fact that K I is full rank (so that the resulting spectral properties of K N + K I remains intractable). We next show that, as n, p → ∞, K I admits a tractable low-rank approximation KI , thereby leading to a spiked-model approximation for K.

Let us consider K I defined in (3.19), the (i, j) entry of which can be written as the sum of terms containing µ a , µ b (treated separately) and random variables of the type

φ = C √ p (x T y/ √ p) α (x T Fy) β
for independent random vectors x, y ∈ R p with i.i.d. zero mean, unit variance and finite moments (uniformly on p) entries, deterministic F ∈ R p×p , C ∈ R, α ∈ N and β ∈ {1, 2}. For Gaussian x, y, the expectation of φ can be explicitly computed via an integral trick [START_REF] Christopher | Computing with infinite networks[END_REF][START_REF] Louart | A random matrix approach to neural networks[END_REF]. For more generic x, y with i.i.d. bounded moment entries, a combinatorial argument controls the higher order moments of the expansion which asymptotically result in (matrix-wise) vanishing terms. See Sections A.1.4 in the appendix. This leads to the following result. where

4 M = [µ 1 , µ 2 ] ∈ R p×2 , T = {tr(E a + E b )/ √ p} 2 a,b=1 , S = {tr(E a E b )/ √ p} 2 a,b=1 ∈ R 2×2 and J = [j 1 , j 2 ] ∈ R n×2 with j a ∈ R n the canonical vector of class C a , i.e., [j a ] i = δ x i ∈C a .
We refer the readers to Section A.1.4 of the Appendix for a detailed exposition of the proof. Proposition 3.2 states that K I is asymptotically equivalent to KI that is of rank at most two. 5 Note that the eigenvectors of KI are linear combinations of the vectors j 1 , j 2 and thus provide the data classes.

From the expression of KI , quite surprisingly, it appears that for f (x) = x k , depending on whether k is odd or even, either only the information in means (M) or only in covariance (T and S) can be (asymptotically) preserved.

By merely combining the results of Propositions 3.1-3.2, the latter can be easily extended to polynomial f . Then, by considering f (x) = P κ (x), the Hermite polynomial of degree κ, it can be shown that, quite surprisingly, one has KI = 0 if κ > 2. As such, using the Hermite polynomial expansion P 0 , P 1 , . . . of an arbitrary f ∈ L 2 (µ) satisfying Assumption 3 leads to a very simple expression of our main result.

Theorem 3.4 (Spiked-model approximation of K). For an arbitrary f ∈ L 2 (µ) with f ∼ ∑ ∞ l=0 a l P l (x), under Assumptions3 and 5,

K -K → 0, K = K N + KI with K N defined in (3.15) and KI = a 1 p (JM T MJ T + JM T Z + Z T MJ T ) + a 2 p J(T + S)J T . (3.21)
Proof. The proof of Theorem 3.4 follows from the fact that the individual coefficients of the Hermite polynomials P κ (x) = ∑ κ l=0 c κ,l x l satisfy the following recurrent relation

[AS65] c κ+1,l = -κc κ-1,l l = 0; c κ,l-1 -κc κ-1,l l ≥ 1; (3.22)
with c 0,0 = 1, c 1,0 = 0 and c 1,1 = 1. As a consequence, by indexing the informative matrix in Proposition 3.2 of the monomial f (x) = x l as KI,l , we have for odd κ ≥ 3,

KI = κ ∑ l=1,3,... c κ,l KI,l = κ ∑ l=1,3,... c κ,l l!!(JM T MJ T + JM T Z + Z T MJ T )/p -diag(•) = 0 with [X -diag(•)] ij = X ij δ i =j .
This follows from the fact that, for κ ≥ 3, we have both ∑ κ l=1,3,... c κ,l l!! = 0 and ∑ κ+1 l=0,2,... c κ+1,l (l + 1)!! = 0. The latter is proved by induction on κ: first, for κ = 3, we have c 3,1 + 3c 3,3 = c 4,0 + 3c 4,2 + 15c 4,4 = 0; then, assuming κ odd, we have ∑ κ l=1,3,... c κ,l l!! = ∑ κ+1 l=0,2,... c κ+1,l (l + 1)!! = 0 so that, together with (3.22

) κ+2 ∑ l=1,3,... c κ+2,l l!! = κ+2 ∑ l=1,... (c κ+1,l-1 -(κ + 1)c κ,l )l!! = κ+2 ∑ l=1,... c κ+1,l-1 l!! = κ+1 ∑ l=0,2,... c κ+1,l (l + 1)!! = 0 as well as κ+3 ∑ l=0,2,... c κ+3,l (l + 1)!! = -(κ + 2)c κ+1,0 + κ+3 ∑ l=2,4,... (c κ+2,l-1 -(κ + 2)c κ+1,l )(l + 1)!! = 0
where we used c κ,l = 0 for l ≥ κ + 1. Similar arguments hold for the case of κ even, which concludes the proof.

-1 0 1 1.5 1 1.5

(a) N : eigs of K for P 1

-1 0 1 1.5 0 1 1.5 (b) N : eigs of K for P 1 -2 -1 0 1 2 2 -1 0 1 2 -2 -1 0 1 (c) N : eigs of K for P 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 (d) N : eigs of K for P 2 -1 0 1 1.5 1 1.5 (e) Stud: eigs of K for P 1 -1 0 1 1.5 0 1 1.5 (f) Stud: eigs of K for P 1 -2 -1 0 1 2 (g) Stud: eigs of K for P 3 -2 -1 0 1 2 (h
(x) = P 1 (x) = x, f (x) = P 2 (x) = (x 2 -1) √ 2, f (x) = P 3 (x) = (x 3 -3x)/ √ 6; n = 2 048, p = 8 192, µ 1 = -[2; 0 p-1 ] = -µ 2 and E 1 = -10I p / √ p = -E 2 .
Figure 3.11 compares the spectra of K and K for random vectors with independent Gaussian or Student-t entries, for the first three (normalized) Hermite polynomials P 1 (x), P 2 (x) and P 3 (x). These numerical evidences validate Theorem 3.4: only for P 1 (x) and P 2 (x) is an isolated eigenvalue observed. Besides, as shown in the bottom display of Figure 3.10c, the corresponding eigenvector is, as expected, a noisy version of linear combinations of j 1 , j 2 . Remark 3.8 (Even and odd f ). While rank( KI ) ≤ 4 (as the sum of two rank-two terms), in Figure 3.11 no more than two isolated eigenvalues are observed (for f = P 1 only one on the right side, for f = P 2 one on each side). This follows from a 2 = 0 when f = P 1 and a 1 = 0 for f = P 2 . More generally, for f odd ( f (-x) =f (x) ), a 2 = 0 and the statistical information on covariances (through E) asymptotically vanishes in K; for f even ( f (-x) = f (x)), a 1 = 0 and the information about the means µ 1 , µ 2 vanishes. Thus, only f neither odd nor even can preserve both first and second order discriminating statistics (e.g., the popular ReLU function f (x) = max(0, x)). This was previously remarked in [START_REF] Liao | On the spectrum of random features maps of high dimensional data[END_REF] based on a local expansion of smooth f in a similar setting.

Practical consequences. As a direct consequence of Theorem 3.4, the performance of spectral clustering for large dimensional mixture models of the type (3.13) only depends on the three parameters of the nonlinear function f :

a 1 = E[ξ f (ξ)], a 2 = E[ξ 2 f (ξ)]/ √ 2 and ν = E[ f 2 (ξ)].
The parameters a 1 , ν determine the limiting spectral measure L of K (since K and K N asymptotically differ by a rank-4 matrix, they share the same limiting spectral measure) while a 2 , a 2 determine the low rank structure within KI .

As an immediate consequence, arbitrary (square-summable) kernel functions f (with a 0 = 0) are asymptotically equivalent to the simple cubic function f (x) = c 3 x 3 + c 2 x 2 + c 1 xc 2 having the same Hermite polynomial coefficients a 1 , a 2 , ν, due to the following relation

a 1 = 3c 3 + c 1 , a 2 = √ 2c 2 , ν = (3c 3 + c 1 ) 2 + 6c 2 3 + 2c 2 2 .
The idea here to design a prototypical family F of functions f having i) universal properties with respect to (a 1 , a 2 , ν), i.e., for each (a 1 , a 2 , ν) there exists f ∈ F with these Hermite coefficients and ii) having numerically advantageous properties. Thus, any arbitrary kernel function f can be mapped, through (a 1 , a 2 , ν), to a function in F with good numerical properties.

One such prototypical family F can be the set of f , parametrized by (t, s -, s + ), and defined as

f (x) =      -rt x ≤ √ 2s - 0 √ 2s -< x ≤ √ 2s + t x > √ 2s + with        a 1 = t √ 2π (e -s 2 + + re -s 2 -) a 2 = t √ 2π (s + e -s 2 + + rs -e -s 2 -) ν = t 2 2 (1 -erf(s + )) (1 + r) (3.23)
where r ≡ 1-erf(s + ) 1+erf(s -) . Figure 3.12 displays f given in (3.23) together with the cubic function c 3 x 3 + c 2 (x 2 -1) + c 1 x sharing the same Hermite coefficients (a 1 , a 2 , ν).

The class of equivalence of kernel functions induced by this mapping is quite unlike that raised in [START_REF] El | The spectrum of kernel random matrices[END_REF] or [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] in the "improper" scaling f (x T i x j /p) regime. While in 86CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS the latter, functions f (x) of the same class of equivalence are those having common f (0) and f (0) values, in the present case, these functions may have no similar local behavior (as shown in the example of Figure 3.12). For the piecewise constant function defined in (3.23) and the associated cubic function having the same (a 1 , a 2 , ν), a close match is observed for both eigenvalues and top eigenvectors of K in Figure 3.13, with gains in both storage size and computational time displayed in Table 3.2.

Conclusion.

In this last section, we evaluated the eigenspectrum of the "properly-scaled" inner-product kernel matrix K = f (X T X/ √ p)/ √ p. Built upon the flexible tool of orthog- onal polynomials, we showed that the spectrum of K depends on the kernel function f via three "global" key parameters (a 1 , a 2 , ν), and this holds true for a large range of nonlinear functions f , including even some non-smooth functions. In this vein, all kernel functions under study are equivalent, in an eigenspectrum sense, to a cubic function, as well as to a piecewise function that takes only three values, as long as they share the same (a 1 , a 2 , ν).

To close this section on random kernel matrices, according to the discussion in Remark 3.5, it is of future interest to extend the current analyses on f (X T X/ √ p)/ √ p to cover data with an arbitrary covariance matrix C (that is not limited to C = I + E with E = o(1)). However, this is technically more involved, since it breaks most of the orthogonality properties of the orthogonal polynomial approach of the proofs, but is a needed extension of the result.

It would also be of interest to properly scale not only the inner-product kernels, but also other popular choice of kernels such as the shift-invariant kernels studied in Section 3.1.1. From this perspective, the kernel matrices of the type f ( x i 2 /p -2x T i x i / √ p +

x j 2 /p) could be a good starting point.

Random Neural Networks

As discussed in Section 1.2.2, the fundamental connection between random kernel matrices and random weights NN models can be built upon random feature maps. In this sub-3.2. RANDOM NEURAL NETWORKS 87 section, we demonstrate how random kernel matrices appear in the performance analysis of single-hidden-layer random NN models, and how this analysis allows us to evaluate the impact on performance of the activation function. In this section, we consider the single-hidden-layer random NN model (or, random feature-based kernel ridge regression) illustrated above (same as Figure 1.3).

Large neural networks with random weights

X ∈ R p×n σ σ σ σ σ Σ ≡ σ(WX) ∈ R N×n β T Σ W ∈ R N×p β ∈ R N×d
For a data matrix X = x 1 , . . . , x n ∈ R p×n with associated targets Y = y 1 , . . . , y n ∈ R d×n , we denote Σ ∈ R N×n the output of the middle layer comprising in total N neurons of X by premultiplying some (random) weight matrix W ∈ R N×p with i.i.d. standard Gaussian entries and then passing through some nonlinear activation function σ : R → R to obtain Σ ≡ σ(WX) ∈ R N×n . In particular, we focus on the case where the second layer weight matrix β ∈ R N×d is designed to minimize the regularized MSE:

L(β) = 1 n ∑ n i=1 y i -β T σ(Wx i ) 2 + λ β 2
F on the given deterministic training set (X, Y), for some regularization factor λ > 0. This givens the following explicit form (as in (1.18)) for β:

β ≡ 1 n Σ 1 n Σ T Σ + λI n -1 Y T . (3.24)
We are interested in the asymptotic behavior of the training and test MSE of the network (as in (1.19)) defined by

E train = 1 n n ∑ i=1 y i -β T σ(Wx i ) 2 = 1 n Y -β T Σ 2 F , E test = 1 n Ŷ -β T Σ 2 F (3.25)
on a (deterministic) test set ( X, Ŷ) of size n, i.e., X ∈ R p× n, Ŷ ∈ R d× n, where we similarly denote

Σ ≡ σ(W X) ∈ R N× n.
Let us start with the training error E train . Note that, by defining the resolvent (see Definition 4) of the Gram matrix 1 n Σ T Σ as

Q(z = -λ) = Q ≡ 1 n Σ T Σ + λI n -1 ∈ R n×n
the training MSE of interest can be rewritten as

E train = 1 n Y -β T Σ 2 F = λ 2 n tr(YQ 2 Y T ) = - λ 2 n ∂ tr(YQY T ) ∂λ (3.26)
and is therefore wished to establish an asymptotically deterministic behavior as the sum of a T Qb for deterministic vector a, b ∈ R n of bounded Euclidean norm (with the normalization n -1 ). if we could find a deterministic equivalent (see Definition 7) for Q.

To properly state our main results, the following assumptions are needed.

88CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS Assumption 6 (Lipschitz σ). The function σ is Lipschitz continuous with parameter independent of n, p.

Assumption 7 (Growth rate in random NN). As n → ∞,

1. p/n → c1 ∈ (0, ∞) and N/n → c2 ∈ (0, ∞);
2. X has a bounded operator norm, i.e., X = O(1);

3. Y has bounded entries, i.e., Y ∞ = O(1).

Note that, for the nonlinear Gram matrix 1 n Σ T Σ, with σ(t) = t one merely gets the sample covariance/Gram matrix 1 n X T W T WX, the spectrum of which is known to have an asymptotic deterministic behavior (recall Section 2.2.3) with X = O(1). Intuitively speaking, Assumption 6 ensures that the nonlinear activation σ "varies" in a "controlled" manner.

Also, by demanding Y to have bounded entries as n, p, N → ∞, we indeed ask the rows of Y, together with the n -1/2 normalization, to have bounded Euclidean norm, so as to apply our deterministic equivalent approach.

Under Assumptions 6 and 7, with a we are now ready to prove the concentration of the resolvent Q (in the sense that Q -E W [Q] → 0) as n, p, N → ∞, and consequently we can provide an exact characterization of the training and test MSE in (1.19).

Main results.

Before going into our main results, let us first re-introduce the following "equivalent" kernel matrix (as in (1.20))

K = E w [σ(X T w)σ(w T X)] (3.27)
for w ∼ N (0, I p ). Again, the appearance of kernel matrix K here is rather natural since the Gram matrix 1 n Σ T Σ at the heart of our analysis is indeed an approximation of K under the random feature framework, for instance with ψ w (x) = σ(w T x) in (1.15).

To show the concentration of the resolvent Q (as well as the eigenspectrum) of a Gram/sample covariance-like random matrix Z T Z, a concentration of quadratic forms based on the row vectors of Z is necessary. For instance, recall from Section 2.2.2 that, to show the (almost sure) convergence of 1 n tr A 1 n Z T Z -zI n -1 for Z ∈ R p×n having i.i.d. standard Gaussian entries, we exploited the concentration of quadratic form result

1 n z T i Az i -1 n tr A a.s.
-→0 in Lemma 2.11, to bound the difference between 1 n Z T Z -zI n -1

and 1 n Z T Z -1 n z i z T i -zI n -1 , for z ∈ R n the i-th row of Z so that z i ∼ N (0, I n ).
In general, such concentration of quadratic form results are obtained by exploiting the independence (or linear dependence) in the vector entries (e.g., z i ∼ N (0, I n ) in the Marcenko-Pastur case in Section 2.2.2). For the nonlinear model under consideration

1 n Σ T Σ = 1 n σ(WX) T σ(WX)
the desired independence unfortunately does not hold, since the entries of the vector σ(X T w) are in general not independent. To overcome this technical difficulty, we resort to a concentration of measure approach, as advocated in [START_REF] El | Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond[END_REF]. The following lemma provides this concentration result.

Lemma 3.3 (Concentration of quadratic forms). Let Assumptions 6 and 7 hold and A ∈ R n×n such that A ≤ 1. For w ∼ N (0, I p ), denote the random vector σ ≡ σ(X T w) ∈ R n . Then,

P 1 n σ T Aσ - 1 n tr(KA) > t ≤ Ce -cn min(t,t 2 )
for some C, c > 0 and K defined in (3.27).

Lemma 3.3 states that, with exponentially high probability, the quadratic form 1 n σ T Aσ is asymptotically close to its expectation (with respect to the random w)

1 n E w [σ T Aσ] = 1 n tr(E w [σσ T ]A) = 1 n tr E w σ(X T w)σ(w T X) A = 1 n tr(KA).
This can be seen as an extension of the crucial Lemma 2.11 for random vectors of linearly correlated entries to the nonlinear setting under investigation. Roughly speaking, with w ∼ N (0, I p ) we have the following normal concentration [Led05, Proposition 1.10] result for all Lipschitz applications,

E f -f dµ k ≤ (Cλ f ) k
holds for some C > 0 and all k ≥ 1, f : R p → R a λ f -Lipschitz function and dµ(z) = (2π) -p 2 e -1 2 z 2 the standard Gaussian measure on R p . Since σ : R → R is Lipschitz (Assumption 6), the normal concentration of w transfers to σ and Σ, which further implying that all Lipschitz functionals of σ or Σ are also concentrated. Nonetheless, note that the quadratic form 1 n σ T Aσ is "quadratic" with respect to w and thus not Lipschitz. AS such, to prove Lemma 3.3 we choose to first provide an exponentially high probability O(1) bound on n -1/2 σ such that, conditioned on this event, the mapping w → 1 n σ T Aσ can be shown to be O(1)-Lipschitz.

With the above result in place, we can follow the same idea as in Section 2.2.2 and apply Lemma 3.3 instead of Lemma 2.11, to show subsequently the concentrations of 1 n σ T AQ -i Bσ as well as a T Qb for independent A, B ∈ R n×n and a, b ∈ R n of bounded norm. This eventually leads to the following result on the deterministic equivalent (see Definition 7) for the resolvent Q of interest, the proof of which is deferred to Section A.1.5 of the Appendix.

Theorem 3.5 (Asymptotic equivalent for E[Q]). Let Assumptions 6 and 7 hold and define

Q as Q ≡ Ǩ -zI n -1 , Ǩ ≡ N n K 1 + δ where δ is implicitly defined as the unique solution of δ = 1 n tr(K Q). Then, as n, p, N → ∞ one has E[Q] -Q → 0.
With Theorem 3.5 at hand, we now present the following corollary on the training and test MSE, which follows directly from our observation in (3.26).

Theorem 3.6 (Asymptotic training and test MSE). Let Assumptions 6 and 7 hold. For E train , E test given by (3.25), Q defined as in Theorem 3.5, denote

Ētrain = λ 2 n tr Y Q 1 N tr Q Ǩ Q 1 -1 N tr Ǩ Q Ǩ Q Ǩ + I n QY T Ētest = 1 n Ŷ -Ŷ Q ǨX X 2 F + 1 N tr Y Q Ǩ QY T 1 -1 N tr Ǩ Q Ǩ Q 1 n tr Ǩ X X - 1 n tr I n + N n λ Q ǨX X Ǩ XX Q . 90CHAPTER 3.

SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS

Then, as n, p, N → ∞ E train -Ētrain → 0, E test -Ētest → 0 almost surely, with

K AB ≡ E w σ(w T A) T σ(w T B) , ǨAB ≡ N n K AB 1 + δ , K ≡ K XX , Ǩ ≡ ǨXX .
We also mention here that, while the above finding on the (asymptotic) training MSE always holds true under Assumptions 6 and 7, the test MSE presented is only a conjecture which not be proved under a general situation, for example with only X = O(1) and Ŷ ∞ = O(1). Under some statistical assumptions for ( X, Ŷ) (e.g., GMM as in Definition 1) the conjecture is valid [START_REF] Louart | A random matrix and concentration inequalities framework for neural networks analysis[END_REF]. Further empirical evidences on real-world datasets also show an extremely close match. This leads to the fundamental question on the minimal (statistical) assumption on the data to establish an asymptotically deterministic behavior for neural networks performances. A first promising answer is given in [START_REF] Louart | A random matrix and concentration inequalities framework for neural networks analysis[END_REF] by assuming the data to be concentrated random vectors, e.g., Lipschitz maps of standard Gaussian random vector or of vectors with i.i.d. entries. Intuitively speaking, this means that the high dimensional data are random, but in a "concentrated" manner such that, instead of "spreading" all over their large ambient space, the data live within a rather low dimensional layer. As a consequence, each scalar observation of these data, even with complicated function such as regressor or classifier, tends to have an almost deterministic and predictable behavior, that depends only on the first several order of statistics of the data. Practical consequences. Theorem 3.6 allows to assess the training and test performance of the network, via the equivalent kernel matrix K (and K) built from the (deterministic) data. It thus remains to compute this kernel matrix K, which is given under the form of an expectation/integral with respect to the standard Gaussian distribution (of dimension p). For most commonly used nonlinear activations σ(•), the generic form K(a, b) = E w [σ(w T a)σ(w T b)] can be computed explicitly via an integral trick [START_REF] Christopher | Computing with infinite networks[END_REF], for arbitrary vector a, b ∈ R p . We list the results for commonly used functions in Table 3.3, with computational details provided in Section A.1.6 of the Appendix.

Table 3.3: K(a, b) for different σ(•), ∠(a, b) ≡ a T b a b . σ(t) K(a, b) t a T b max(t, 0) 1 2π a b ∠(a, b) arccos (-∠(a, b)) + 1 -∠(a, b) 2 |t| 2 π a b ∠(a, b) arcsin (∠(a, b)) + 1 -∠(a, b) 2 1 t>0 1 2 -1 2π arccos (∠(a, b)) sign(t) 2 π arcsin (∠(a, b)) ς 2 t 2 + ς 1 t + ς 0 ς 2 2 2 a T b 2 + a 2 b 2 + ς 2 1 a T b + ς 2 ς 0 a 2 + b 2 + ς 2 0 cos(t) exp -1 2 a 2 + b 2 cosh(a T b) sin(t) exp -1 2 a 2 + b 2 sinh(a T b) erf(t) 2 π arcsin 2a T b √ (1+2 a 2 )(1+2 b 2 ) exp(-t 2 /2) 1 √ (1+ a 2 )(1+ b 2 )-(a T b) 2
To corroborate the findings in Theorem 3.6 we consider the task of classifying the MNIST database [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] with a single-hidden-layer random weight neural network composed of N = 512 hidden-units. Similar to Section 3.1.1 we represent each image as a p = 784-size vector; 1 024 images of sevens and 1 024 images of nines were extracted from the database and were evenly split in 512 training and test images, respectively, so that n = n = 1 024. The database images were jointly centered and scaled so to fall close to the setting of Assumption 7 on X and X. The columns of the output values Y and Ŷ were taken as unidimensional (d = 1) with Y i , Ŷi ∈ {-1, 1} depending on the image class. Figure 3.15 displays the simulated (averaged over 100 realizations of W) versus theoretical values of E train and E test for three choices of Lipschitz continuous functions σ(•), as a function of the regularization factor λ. We observe an almost perfect match between the simulations and our theoretical results from Theorem 3.6 for not so large n, p, N.

10 -4 10 -3 10 -2 10 -1 10 0 10 1 10 2 10 -1 10 0 σ(t) = max(t, 0)

σ(t) = erf(t) σ(t) = t λ MSE E train (Theory) E test (Theory) E train (Simulation) E test (Simulation)

Conclusion.

In this section, we characterized the resolvent Q of 1 n Σ T Σ with Σ = σ(WX) the output of a random NN with input X and showed it establishes an asymptotically deterministic behavior for Lipschitz activations σ(•). As a direct consequence, we gave the training and test MSE of the random NN model in Figure 1.3 that depends on the data and the activation function σ(•) via the key "equivalent" kernel matrix K ≡ E w [σ(X T w)σ(w T X)]. For most commonly used σ, the expression of K is given in Table 3.3.

Nonetheless, the expressions of K in Table 3.3 are still in very barely interpretable and provide little understanding of the role played by the nonlinear activations. For instance, we are not able to say which σ(•) should be used for a given task. To further exploit the interplay between the data and the activation function, we next investigate in the following section the eigenspectrum of K by (additionally) considering a GMM for the input data. 92CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS

Random feature maps and the equivalent kernels

We saw in the previous section that the single-layer random NN performance depends on the deterministic data X and the nonlinear activation σ(•) via the following "equivalent" kernel matrix (as in (1.20))

K ≡ E w [σ(X T w)σ(w T X)]
the expression of which is explicitly given in Table 3.3 for most commonly used σ with w ∼ N (0, I p ). In this section we develop a deeper understanding of, the eigenspectrum of this kernel matrix K, for structural statistical models for X. In particular, we focus on the interplay between σ and the data statistics µ, C of X under a GMM as detailed in Definition 1.

In the context of random feature maps, 1 n Σ T Σ is indeed the Gram matrix of the random features σ(Wx i ), that approximates an underlying kernel matrix as discussed at the end of Section 1.2.1. From Theorem 3.5, we know that the resolvent of 1 n Σ T Σ and thus its limiting spectral measure, if exists, are both closely related to K. In this vein, the kernel matrix K above is again at the heart of all random feature-based spectral algorithms.

Consider x 1 , . . . , x n ∈ R p independent data vectors, each belonging to one of K distribution classes C 1 , . . . , C K . Class C a has cardinality n a , for all a ∈ {1, . . . , K}. The data vector x i that belongs to C a is assumed to be drawn from the following GMM

x i = 1 √ p µ a + ω i
with ω i ∼ N (0, C a /p) for some mean µ a ∈ R p and covariance C a ∈ R p×p that satisfy the non-trivial classification in Assumption 2. Note that we normalize the data by 1/ √ p, together with Assumption 2 to ensure x i = O(1) with high probability, which is in consistent with Assumption 7 in the presented random NN model. Taking a = x i and b = x j , we see that the expressions of K in Table 3.3 are indeed nonlinear functions of the "concentrated" measures of the data, e.g., x i or x T i x j , as investigated in Section 3.1.1 under the binary setting K = 2. Therefore, the spectral analysis of K for GMM under Assumption 2 follows exactly the same line of arguments as in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] and discussed at length in Section 3.1.1, that we briefly recall next.

From a random feature map standpoint, the Gram matrix 1 n Σ T Σ describes the correlation of data in the feature space. It is thus natural to recenter 1 n Σ T Σ, hence K, by pre-and post-multiplying with the projection matrix P ≡ I n -1 n 1 n 1 T n . In the case of K, we get

K c ≡ PKP.
This centering operation technically eliminates, from a technical standpoint, the noninformative isolated eigenvalues that may go to infinity as n, p → ∞, with corresponding non-informative eigen-direction 1 n .

Main results.

Let us now introduce the key steps of our present analysis. Under Assumption 2, observe that for x i ∈ C a and x j ∈ C b , i = j,

x T i x j = ω T i ω j O(p -1/2 ) + µ T a µ b /p + µ T a ω j / √ p + µ T b ω i / √ p O(p -1 )

RANDOM NEURAL NETWORKS

93

which allows one to perform a Taylor expansion around 0 as p, T → ∞, to give a reasonable approximation of nonlinear functions of x T i x j , such as those appearing in K ij (see again Table 3.3). For i = j, one has instead

x i 2 = ω i 2 O(1) + µ a 2 /p + 2µ T a ω i / √ p O(p -1 )
.

From

E ω i [ ω i 2 ] = tr(C a )/p it is convenient to further write ω i 2 = tr(C a )/p + ω i 2 -tr(C a )/p
where tr(C a )/p = O(1) and ω i 2tr(C a )/p = O(n -1/2 ). By definition6 τ ≡ tr(C • )/p that is assumed to converge in (0, ∞) as in the two-class case in Assumption 4. Exploiting again Assumption 2, one results in,

x i 2 = τ O(1) + tr(C • a )/p + ω i 2 -tr(C a )/p O(p -1/2 ) + µ a 2 /p + 2µ T a ω i / √ p O(p -1 )
which allows for a Taylor expansion of nonlinear functions of x i 2 around τ, as done for x T i x j . From Table 3.3, it appears that, for every listed σ(•), K(x i , x j ) is a smooth function of x T i x j and x i , x j , despite their possible discontinuities (for example, the ReLU function and σ(t) = |t|). The above results therefore allow for an entry-wise Taylor expansion of the matrix K in the large n, p limit. A critical aspect of the analysis where random matrix theory comes into play now consists in developing K as a sum of matrices arising from the Taylor expansion and ignoring terms that give rise to a vanishing operator norm, so as to find an asymptotic equivalent matrix K such that K -K → 0 as n, p → ∞, as described in detail below. This analysis provides a simplified asymptotically equivalent expression for K with all nonlinearities removed.

To present our main theoretical result, we define, similarly to Section 3.1.1 and 3.1.2 the following notations for random elements

Ω ≡ ω 1 , . . . , ω n ∈ R p×n , φ ≡ ω i 2 -E ω i 2 n i=1 ∈ R n
as well as for deterministic elements7 ,

M ≡ µ 1 , . . . , µ K ∈ R p×K , J ≡ j 1 , . . . , j K ∈ R n×K t ≡ 1 √ p tr C • a K a=1 ∈ R K , S ≡ 1 p tr(C a C b ) K a,b=1 ∈ R K×K
where j a ∈ R n denotes, as usual, the canonical vector of class C a such that (j a ) i = δ x i ∈C a .

Theorem 3.7 (Asymptotic equivalent of K c ). Let Assumption 2 hold and K c be defined as K c ≡ PKP for K given in (1.20). Then, as n → ∞, for all σ(•) given in Table 3.3,

K c -Kc → 0 94CHAPTER 3.

SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS

almost surely, with Kc = P KP and

K ≡ d 1 Ω + M J T √ p T Ω + M J T √ p + d 2 UBU T + d 0 I n ≡ K N + KI + d 0 I n
where we recall that P ≡ I n -1 n 1 n 1 T n and

U ≡ J √ p , φ , B ≡ tt T + 2S t t T 1 , K N ≡ d 1 Ω T Ω + d 2 φφ T KI ≡ d 1 p (JM T MJ T + √ pΩ T MJ T + √ pJM T Ω) + d 2 p J(tt T + 2S)J T + √ pφt T J T + √ pJtφ T with the coefficients d 0 , d 1 , d 2 given in Table 3.4. Table 3.4: Coefficients d i in Kc for different σ(•). σ(t) d 0 d 1 d 2 t 0 1 0 ReLU(t) ≡ max(t, 0) 1 4 -1 2π τ 1 4 1 8πτ |t| 1 -2 π τ 0 1 2πτ ς + max(t, 0) + ς -max(-t, 0) ≡ LReLU(t) π-2 4π (ς + + ς -) 2 τ 1 4 (ς + -ς -) 2 1 8τπ (ς + + ς -) 2 1 t>0 1 4 -1 2π 1 2πτ 0 sign(t) 1 -2 π 2 πτ 0 ς 2 t 2 + ς 1 t + ς 0 2τ 2 ς 2 2 ς 2 1 ς 2 2 cos(t) 1 2 + e -2τ 2 -e -τ 0 e -τ 4 sin(t) 1 2 -e -2τ 2 -τe -τ e -τ 0 erf(t) 2 π arccos 2τ 2τ+1 -2τ 2τ+1 4 π 1 2τ+1 0 exp(-t 2 /2) 1 √ 2τ+1 -1 τ+1 0 1 4(τ+1) 3
Theorem 3.7 tells us as a corollary (from for example Lemma 2.10) that the maximal difference between the eigenvalues of K c and Kc vanishes asymptotically as n, p → ∞. Similarly the distance between the "isolated eigenvectors" also vanishes. This is of tremendous importance as the determination of the leading eigenvalues and eigenvectors of K c (that contain crucial information for clustering, for example) can be studied from the equivalent problem performed on Kc and becomes mathematically more tractable.

It is also of interest to remark from Theorem 3.7 that, albeit derived from different data models and kernel types, Theorem 3.7 provides the same intuition as Theorem 3.4: the first order information (M) always goes with one coefficient (d 1 and a 1 , respectively), while the second order information (t, S or T, S, respectively) is always multiplied by another coefficient (d 2 and a 2 , respectively).

To make this clear, the major differences between these two results are summarized as follows:

1. Theorem 3.4 characterizes the spectrum of the "properly scaled" inner-product kernel matrix model f (x T i x j / √ p), with zero on its diagonal, regardless of the underly- ing distribution; it holds for any square-summable kernel function f (that satisfies Assumption 3); the coefficients a 1 , a 2 and ν are known to be the (generalized) moments of f (with respect to standard Gaussian measure). Nonetheless, due to the lack of more advanced technical tools, only the special case C a = I p + E a is covered in Theorem 3.4.

2. On the other hand, Theorem 3.7 treats the kernel matrices arising from random feature maps (or random NNs), in the Gaussian case (i.e., for GMM data); these kernels are "improperly scaled" (as those studied in Section 3.1.1) with the form f (x T i x j /p) in Table 3.3, and the analysis is based on a local expansion of f , which imperatively demands f to be locally (at least three-times) differentiable near 0 and τ. Also, Theorem 3.7 holds only for the nonlinear activation listed in Table 3.3 for which we are capable of explicitly computing the expression of K. However, Theorem 3.7 holds for arbitrary covariance C a .

Practical consequences. On closer inspection of Theorem 3.7, the matrix K is expressed as the sum of three terms, weighted respectively by the three coefficients d 0 , d 1 and d 2 , that depend on the nonlinear function σ(•) via Table 3.4. Note that the statistical structure of the data {x i } n i=1 (namely the means in M and the covariances in t and S) is perturbed by random fluctuations (Ω and φ). In this perspective, we see that d 1 and d 2 (asymptotically) control the "expression" of the first (M) and second order (t and S) statistical information in K, respectively. In particular, there exists a balance between the means and covariances, that provides some instructions in the appropriate choice of the nonlinearity. From Table 3.4, the functions σ(•) can be divided into the following three groups:

• mean-oriented, where d 1 = 0 while d 2 = 0: this is the case of the functions t, 1 t>0 , sign(t), sin(t) and erf(t), which asymptotically track only the difference in means (i.e., t and S disappear from the expression of Kc );

• covariance-oriented, where d 1 = 0 while d 2 = 0: this concerns the functions |t|, cos(t) and exp(-t 2 /2), which asymptotically track only the difference in covariances;

• balanced, where both d 1 , d 2 = 0: here for the ReLU function max(t, 0) and the quadratic function ς 2 t 2 + ς 1 t + ς 0 .

To corroborate the above classification of different nonlinearity, we perform kernel spectral clustering in Figure 3.16 on four classes of Gaussian data: N (µ 1 , C 1 ), N (µ 1 , C 2 ), N (µ 2 , C 1 ) and N (µ 2 , C 2 ) with the LReLU function that takes different values for ς + and ς -. For a = 1, 2, µ a = 0 a-1 ; 5; 0 p-a and C a = (1 + 15(a -1)/ √ p)I p . By choosing

ς + = -ς -= 1 (equivalent to σ(t) = |t|) and ς + = ς -= 1 (equivalent to the linear map σ(t) = t)
, with the leading two eigenvectors we always recover two classes instead of four, as each setting of parameters only allows for a part of the statistical information of the data to be used for clustering. However, by taking ς + = 1, ς -= 0 (the ReLU function) we distinguish all four classes in the leading two eigenvectors, to which the k-means method can then be applied for final classification, as shown in Figure 3.17.

We complete this section by showing that our theoretical results in Theorem 3.7, derived from GMMs, show an unexpected close match in practice when applied to realworld datasets. We consider two different types of classification tasks: one on the MNIST [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF] database (number 6 and 8), and the other on epileptic EEG time series data [ALM + 01] (set B and E). These two datasets are typical examples of means-dominant 96CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS (handwritten digits recognition) and covariances-dominant (EEG times series classification) tasks. This is numerically confirmed in Table 3.5 (see also Table 3.1 from Section 3.1.1).

C 1 C 2 C 3 C 4 (a) σ(t) = |t|: eigenvector 1 C 1 C 2 C 3 C 4 (b) σ(t) = |t|: eigenvector 2 C 1 C 2 C 3 C 4 (c) σ(t) = t: eigenvector 1 C 1 C 2 C 3 C 4 (d) σ(t) = t: eigenvector 2
We perform random feature-based spectral clustering on data matrices that consist of n = 32, 64 and 128 randomly selected vectorized images of size p = 784 from the MNIST dataset. Means and covariances are empirically obtained from the full set of 11 769 MNIST images (5 918 images of number 6 and 5 851 of number 8). Comparing the matrix K c built from the data and the theoretically equivalent Kc obtained as if the data were Gaussian with the (empirically) computed means and covariances, we observe an extremely close fit in the behavior of the eigenvalues and the leading eigenvector in Figure 3.18. The k-means method is then applied to the leading two eigenvectors of the Gram matrix 1 n Σ T Σ that consists of N = 32 random features to perform unsupervised classification, with resulting accuracies (averaged over 50 runs) reported in Table 3.6. As seen from Table 3.5, the mean-oriented σ(t) functions are expected to outperform the covariance-oriented functions in this task, which is consistent with the results in Table 3.6.

RANDOM NEURAL NETWORKS

97

Eigenvector 1

C 1 C 2 C 3 C 4 Eigenvector 2 C 1 C 2 C 3 C 4
Eigenvector 1

Eigenvector 2

Figure 3.17: Leading two eigenvectors of K c (top) for the LReLU function with ς + = 1, ς -= 0 (equivalent to ReLU(t)) and two dimensional representation of these eigenvectors (bottom), in the same setting as in Figure 3.16.

= c 2 = 1/2, j 1 = [1 n 1 ; 0 n 2 ] and j 2 = [0 n 1 ; 1 n 2 ].
ically estimated from the full set (4 097 segments of set B and 4 097 segments of set E). Similar behavior of eigenpairs as for Gaussian mixture models is once more observed in Figure 3.19. After k-means classification on the leading two eigenvectors of the (centered) Gram matrix composed of N = 32 random features, the accuracies (averaged over 50 runs) are reported in Table 3.7.

As opposed to the MNIST image recognition task, from Table 3.7 it is easy to check that the covariance-oriented functions (i.e., σ(t) = |t|, cos(t) and exp(-t 2 /2)) far outperform any other with almost perfect classification accuracies. It is particularly interesting to note that the popular ReLU function is suboptimal in both tasks, but never performs very badly, thereby offering a good risk-performance tradeoff.

= c 2 = 1/2, j 1 = [1 n 1 ; 0 n 2 ] and j 2 = [0 n 1 ; 1 n 2 ].

Conclusion.

In this section, by leveraging the randomness of the data, we dived deep into the comprehension of the equivalent kernel matrix K = E w [σ(X T w)σ(w X )] that appears in the random NN context (in Section 3.2.1) as well as in random feature-based techniques. The same technical approach as in Section 3.1.1 was applied to handle the

Empirical Risk Minimization of Convex Loss

In Section 3.1 and 3.2 we discussed both large random kernel matrices (and kernel ridge regression build on top of it) as well as random NN/feature-based models that are closely connected to one another. However, all aforementioned objects of interest are in these machine learning methods assume closed forms, as they come in essence from the minimization of (regularized) square losses. More generally, almost all machine learning algorithms are given in form of (solutions of) optimization problems, with most of them expressed only in an implicit manner. An important example is the popular logistic regression, where one aims to find an optimal decision vector β ∈ R p by minimizing the logistic loss 1 n ∑ n i=1 log(1 + e -ỹi β T x i ) over the training set {(x i , ỹi )} n i=1 with labels ỹi ∈ {-1, +1}. Different from the linear regression solution that arises from the minimization of square loss, the logistic regression solution β takes an implicit form and it is less direct to understand how β depends on the data X so as to investigate its statistical behavior. The technical challenge from implicit optimization problems appears not only in the analysis of logistic regression, but also in many other machine learning algorithms in daily use.

In this regard, it is of crucial importance to adapt the proposed random matrix-based analysis framework to assess the performance of optimization-based learning methods, we evaluate here the classifier obtained by minimizing an arbitrary convex and differentiable loss, with a major emphasis on the "leave-one-out" technical approach to "decouple" the learning system that contains complicated dependences.

In this section, we consider the following (regularized) empirical risk minimization problem (as in (1.22)) min

β∈R p 1 n n ∑ i=1 L( ỹi β T x i ) + λ 2 β 2 (3.28)
100CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS under a binary symmetric GMM for the input data, i.e.,

C 1 :

x i = -µ + C 1 2 z i , y i = -1 C 2 : x i = +µ + C 1 2 z i , y i = +1 (3.29)
each with a class prior of 1/2, for z ∼ N (0, I p ), µ ∈ R p and positive definite C ∈ R p×p .

Let us first consider the case where λ > 0 and the existence of the minimizer β is guaranteed. Similar to the non-trivial classification condition in Assumption 2, we shall position ourselves under the following assumption.

Assumption 8 (Non-trivial classification). As n → ∞, we have p/n → c ∈ (0, ∞), µ = O(1) and max{ C , C -1 } = O(1) with respect to p.

Note that the GMM in (3.29) satisfies the logistic regression model in the sense that the conditional class probability is given by

P(y i = 1 | x i ∈ C 2 ) = 1 1 + exp(-2µ T C -1 x i ) ≡ σ(β T * x i )
with σ(t) = (1 + e -t ) -1 the logistic sigmoid function and the optimal Bayes solution β * = 2C -1 µ. With the GMM in (3.29) and labels y i ∈ {-1, +1}, it is convenient to denote the shortcut xi ≡ ỹi x i so that xi ∼ N (µ, C) regardless of the class to which x i belongs.

To investigate the asymptotic performance of the empirical risk-based classifier in (3.28), it is of crucial importance to understand the statistical properties of β that minimizes (3.28). The main technical difficulty of this analysis lies in the fact that β, as the solution of a convex optimization problem, does not have an explicit form. Nonetheless, by canceling the loss function derivative with respect to β we obtain the following implicit relation

λβ = 1 n n ∑ i=1 -L (β T xi ) xi (3.30)
where we assume the loss function L is convex and at least three-times differentiable. In this section, in addition to RMT techniques, in order to handle the implicit nature of β, we mainly focus on the additional "leave-one-out" tool to handle the complex dependences in the optimization problem.

Main results. From (3.30), β can be seen as a linear combination of all xi 's, weighted by the coefficient -L (β T xi ). The idea is to understand how xi (and its statistical properties) affects the corresponding coefficient -L (β T xi ). However, as a solution of (3.30), β depends on all xi 's in an intricate manner. We handle this correlation by establishing a "leave-one-out" version of β, denoted β -i , that is asymptotically close to β and independent of xi , by solving (3.28) for all data xj different from xi , i.e., for j = 1, . . . , i -1, i + 1, . . . , n.

In details, since by definition

λβ -i = -1 n ∑ j =i L (β T -i xj ) xj , the difference λ(β -β -i ) is given by λ(β -β -i ) = 1 n ∑ j =i L (β T -i xj ) -L (β T xj ) xj - 1 n L (β T xi ) xi (3.31)

EMPIRICAL RISK MINIMIZATION OF CONVEX LOSS 101

Intuitively speaking, under Assumption 8, the difference ββ -i should be small, in the sense that it goes to 0 as n, p → ∞. As such, by a Taylor expansion of L (x) around x = β T -i xj , j = i, we obtain

L (β T xj ) = L (β T -i xj ) + L (β T xj )(β -β -i ) T xj + O( β -β -i 2 ).
This approximation, together with (3.31), leads to the following relation on β -

β -i λ(β -β -i ) - 1 n ∑ j =i L (β T -i xj ) xj xT j (β -β -i ) - 1 n L (β T xi ) xi or in matrix form 1 n X-i D -i XT -i + λI p (β -β -i ) - 1 n L (β T xi ) xi with D -i ≡ diag{L (β T -i xj )} n j =i ∈ R (n-1)×(n-1
) and X -i ∈ R p×(n-1) the data matrix without x i .

Note here that by convexity of L, L (t) ≥ 0 for all t, so that with λ > 0, the matrix 1 n X-i D XT -i + λI p is invertible and we get

β -β -i - 1 n L (β T xi ) 1 n X-i D -i XT -i + λI p -1
xi so that its projection on xi gives

(β -β -i ) T xi - 1 n L (β T xi ) xT i 1 n X-i D -i XT -i + λI p -1
xi .

Note that 1 n X-i D XT -i + λI p -1 is of bounded operator norm and independent of xi .

Following the idea of Lemma 2.11 we deduce the following approximation.

Lemma 3.4 (Asymptotic approximation of quadratic forms). Let Assumption 8 holds. Then as n, p → ∞,

1 n xT i 1 n X-i D -i XT -i + λI p -1 xi -κ a.s.
-→0

where κ is the unique positive solution of κ = 1 n tr( QC) with

Q ≡ E L (β T x) 1 + κL (β T x) C + λI p -1
.

As a consequence of Lemma 3.4, one has immediately

(β -β -i ) T xi -L (β T xi )κ
and therefore

β T xi = β T -i xi + (β -β -i ) T xi β T -i xi -L (β T xi
)κ where we observe the quantity β T xi on both sides, which is given by the explicit equation

β T xi prox κL (β T -i xi ) 102CHAPTER
(x) + 1 2 (β T -i xi -x) 2 .
The uniqueness of the minimization problem above is guaranteed by the convexity of the loss L. Moreover, the proximal operator can be defined for all lower semi-continuous convex functions, and thus covers non-smooth loss L(t) such as the hinge losses L(t) = max(1t, 0). In this vein, our results are envisioned to extend to non-smooth losses, for which more refinements (to replace the gradient with the subgradient in the sense of Clarke [START_REF] Clarke | Optimization and Nonsmooth Analysis[END_REF] and to rework on Lemma 3.4) are needed.

Since β -i is independent of xi , we have β T -i xi ∼ N (m, σ 2 ) in the large p limit for some unknown deterministic m and σ 2 , as summarized in the following lemma.

Lemma 3.5. Under Assumption 8, there exist two positive constants m, σ 2 such that, as n, p → ∞,

β T -i xi -r d -→0, r ∼ N (m, σ 2 ) and L (β T xi ) -f (r) d -→0, f (r) ≡ L (prox κL (r)) = r -prox κL (r)
κ with the proximal operator prox κL (r) defined as the unique solution of the following minimization problem

min x∈R κL(x) + 1 2 (x -r) 2
for some κ > 0 determined by the fixed point equation in Lemma 3.4.

With Lemma 3.5, to characterize the stochastic behavior of β T -i xi and β T xi (and subsequently β), it remains to determine the unknown constants m and σ 2 . This can be done by taking the expectation on both sides of (3.30) as

λE[β] = 1 n n ∑ i=1 -E f (β T -i xi ) xi
which helps connect the statistics of β to m and σ 2 . Nonetheless, the right-hand-side expectation in the above equation is non-trivial, due to the dependence between the random vector xi and its projection xT i β -i onto the independent β -i . Fortunately, in the case of Gaussian xi ∼ N (µ, C), this dependence can be separated and treated explicitly. By writing xi = µ + √ pω i with ω i ∼ N (0, C/p) we can decompose ω i as the following sum

ω i = β T -i ω i β T -i Cβ -i Cβ -i + ω ⊥ i β T -i ω i tr CE[β -i β T -i ] Cβ -i + ω ⊥ i β T -i ω i σ 2 Cβ -i + ω ⊥ i
for σ 2 defined in Lemma 3.4 and such that ω T i β -i is independent of ω ⊥ i , i.e.,

E ω i ω ⊥ i ω T i β -i = E ω i ω i - β T -i ω i β T -i Cβ -i Cβ -i ω T i β -i = 0
and in particular β T -i ω ⊥ i = 0. As a consequence, (3.30) can be reduced to

λβ 1 n n ∑ i=1 -f (β T -i xi ) µ + √ pβ T -i ω i σ 2 Cβ -i + √ pω ⊥ i 3.3. EMPIRICAL RISK MINIMIZATION OF CONVEX LOSS 103 which, since β -i β, further leads to 1 n n ∑ i=1 f (β T -i xi )( xi -µ) σ 2 C + λI p β 1 n n ∑ i=1 -f (β T -i xi ) µ + u for the random vector u ≡ 1 n ∑ n i=1 -f (β T -i xi ) √ pω ⊥ i such that E[u] = 0, E[uu T ] = E[ f 2 (β T -i xi )] n C - 1 σ 2 CE[ββ T ]C .
This allows us to conclude that

E[ f (r)(r -m)] σ 2 C + λI p β E[-f (r)]µ + u or β (τC + λI p ) -1 µ + (τC + λI p ) -1 u where we denote η ≡ E[-f (r)], γ ≡ E[ f 2 (r)] and τ ≡ E[ f (r)(r -m)/σ 2 ] for r ∼ N (m, σ 2
) as defined in Lemma 3.4. Lastly, with

E[ββ T ] η 2 (τC + λI p ) -1 µµ T (τC + λI p ) -1 + (τC + λI p ) -1 E[uu T ](τC + λI p ) -1
we deduce

σ 2 = tr CE[ββ T ] η 2 µ T (τC + λI p ) -1 C(τC + λI p ) -1 µ + γ n (τC + λI p ) -1 C 2 F
and finally reach the following theorem.

Theorem 3.8 (Asymptotic behavior of β). Let Assumption 8 hold. Then, under the notations of Lemma 3.4, we have, as n, p → ∞ β -β → 0, (τC + λI p ) -1 β ∼ N (ηµ, γC/n) with (η, γ, τ) ∈ R 3 the unique solution of

η ≡ E[-f (r)], γ ≡ E[ f 2 (r)], τ ≡ E[ f (r)(r -m)/σ 2 ] = E[-f (r)]
for f (r) ≡ L (prox κL (r)) defined in Lemma 3.5, r ∼ N (m, σ 2 ) with m = ηµ T (τC

+ λI p ) -1 µ, σ 2 = η 2 µ T (τC + λI p ) -1 C(τC + λI p ) -1 µ + γ n (τC + λI p ) -1 C 2 F .
As a direct consequence of Theorem 3.8 and Lemma 3.5, we obtain the following corollary on the (asymptotic) training and test performance.

Corollary 3.3. Under the conditions and notations of Theorem 3.8, the asymptotic test classification error rate is given by

P(β T -i xi < 0) -Q m σ → 0
where we recall that Q(t) ≡ 1

√ 2π

∞ t e -u 2 /2 du. Similarly, the training classification error is given by P(β T xi < 0) -P (prox κL (r) < 0) → 0 for r ∼ N (m, σ 2 ) as in Lemma 3.5. Practical consequences. To validate Theorem 3.8 and Lemma 3.5 for n, p of reasonable sizes, we compare in Figure 3.20 the empirical distribution of β T -i xi and the Gaussian distribution N (m, σ 2 ) from the system of fixed point equations in Theorem 3.8. Our theoretical results fit the simulation almost perfectly, already for p = 256 and n = 512.

To interpret Theorem 3.8 we first restrict ourselves to the unregularized case where λ = 0 and assume the minimization problem (3.28) admits a solution β such that β = O(1), at least with high probability. Note that depending on the problem (µ, C and the loss L) and the dimensionality (n, p), such a solution may not exist with λ = 0 (for instance as pointed out in [START_REF] Emmanuel | The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression[END_REF] for the logistic regression L(t) = log(1 + e -t )).

With λ = 0, the notations in Theorem 3.8 are reduced to

β ∼ N η τ C -1 µ, γ nτ I p with m = η τ µ T C -1 µ and σ 2 = η 2 τ 2 µ T C -1 µ + γ τ 2 p n .
Several remarks are in order: 1. E[ β] is aligned to the optimal Bayes solution β * = 2C -1 µ. However, unlike in the regime where n → ∞ and p fixed, β contains an additional zero mean Gaussian noise of covariance γ nτ I p , which results in a larger variability of the soft output β T x than classical asymptotics predict.

2. The scaling of β depends on the interplay between the data statistics (µ, C), the problem dimensionality (n, p) and the loss function L. In general, E[ β] differs from the optimal Bayes solution β * by a multiplicative (constant) factor, i.e., E[ β] = η 2τ β * . In Figure 3.21 we examine the empirical mean (so as to estimate the expectation) of β, the rescaled version 2τ η β and the optimal Bayes solution β * . We see that by rescaling the obtained β by the factor 2τ/η, one gets (in expectation) the optimal β * . Nonetheless, note that for classification applications, one has sign(β T x) = sign(aβ T x) for a > 0, meaning that a positive rescaling of β does not affect the classification performance.

3. As a consequence of Corollary 3.3, in pursuit of an optimal design of the loss function L, one must find (η, γ, τ) that maximizes the ratio m 2 /σ 2 , or equivalently arg max

(η,γ,τ)∈R 3 m 2 σ 2 = arg max (η,γ,τ)∈R 3 (µ T C -1 µ) 2 µ T C -1 µ + γ η 2 p n = arg max (η,
m σ ) = Q µ T C -1 µ √ µ T C -1 µ+p/n
for the model under consideration. On the other hand, it is of interest to note that, by the law of large numbers, the empirical average 1 n ∑ n i=1 -L (β T xi ) converges to its expectation η as n → ∞, and similarly for γ. As such, one may alternatively wish to maximize the following empirical version max

(η,γ,τ)∈R 3 L (β T X)1 n L (β T X)L ( XT β)
where the function L is applied entry-wise on the vector β T X ∈ R n . Under the above form, note from (3.30) that in the unregularized case with λ = 0 one must has XL ( XT β) = 0 so that by considering a singular value decomposition of

X = UΣV T = U S 0 V T 1 V T 2 for X ∈ R p×n with n > p such that S ∈ R p×p , V 1 ∈ R n×p and V 2 ∈ R n×(n-p) , one must have V T 1 L ( XT β) = 0,
or equivalently, L ( XT β) lies on the subspace spanned by the columns vectors of V 2 such that there exists a ∈ R n-p for which

L ( XT β) = V 2 a.
With the formulation, one aims to find arg max

(η,γ,τ)∈R 3 a T V T 2 1 n a
which attains the maximum if and only if a is aligned to V T 2 1 n a = aV T 2 1 n for some a > 0. This optimality condition is met for instance with the square loss L(t) = (t -1) 2 .

Conclusion.

In this section, we considered the separation of a symmetric binary GMM with opposite means ±µ and identical covariance C, by minimizing the empirical convex and differentiable risk L on a given training set. In this scenario, the statistical of the resulting classifier β depends on the chosen loss metric via the generalized moment of a nonlinear function f (x) ≡ L (prox κL (x)) with respect to the standard Gaussian measure, in an implicit manner via the proximal operator and the constant κ > 0. As a consequence of the analysis, we saw that the in the regime n, p → ∞ with p/n → c ∈ (0, ∞), β is the sum of a rescaled of β * , the optimal Bayes solution, together with an additional homogeneous Gaussian noise.

Summary of Chapter 3

In this chapter, we focused on the eigenspectrum of large dimensional random kernel matrices and neural networks. Built upon a simple Gaussian mixture modeling of the input data, we provided a tractable characterization of the spectral behavior of kernel matrices and nonlinear Gram matrices, from which we further deduced the (asymptotic) performance of various machine learning methods, as a function of the data statistics, the dimensionality, and the hyperparameters of the algorithm. The presented theoretical findings shed novel light on the understanding, as well as a better designing of the aforementioned machine learning methods.

To further address more involved learning algorithms, for example, those (implicitly) defined by convex optimization problems (e.g., logistic regression), we resort to a "leave-one-out" approach and assume that the optimality conditions are met. By working on these conditions, we deduced the (asymptotic) performance reached by the optimal points of the these optimization problems.

In practice, optimization methods such a gradient descent are regularly used to reach these optimal points. The solution of the optimization problem, and therefore its performance, naturally depends on the optimization method applied. The impact of the optimization method is more significant in non-convex problems, where depending on the initialization, totally different solutions can be reached. In the following chapter, we focus precisely on the gradient descent algorithm and study its behavior in convex and non-convex machine learning problems.

where we denote the resolvent Q(z) ≡ 1 n ZZ T -zI p -1 of 1 n ZZ T , a deterministic equivalent of which is given by

Q(z) ↔ Q(z) ≡ m(
m(z) = 1 -c -z 2cz + (1 -c -z) 2 -4cz 2cz (4.1)
where the branch of the square root is selected in such a way that (z) • m(z) > 0, i.e., for a given z there exists a unique corresponding m(z).

Substituting Q(z) by the simple form deterministic equivalent m(z)I p , we are able to estimate the random variable µ T w(t) with a contour integral of some deterministic quantities as n, p → ∞. Similar arguments also hold for w(t) T w(t), together leading to the following theorem, the proof of which is deferred to Section A.2.1 in the Appendix.

Theorem 4.1 (Test performance). Let Assumptions 9 hold. As n, p → ∞ with p/n → c ∈ (0, ∞), with probability one

P(w(t) T x < 0) -Q E √ V → 0 where E ≡ - 1 2πi γ 1 -f t (z) z µ 2 m(z) dz ( µ 2 + c) m(z) + 1 V ≡ 1 2πi γ 1 z 2 (1 -f t (z)) 2 ( µ 2 + c) m(z) + 1 -σ 2 f 2 t (z)m(z) dz
with γ a closed positively oriented contour that surrounds the support of Marčenko-Pastur law, the origin (0, 0) and the point (λ s , 0) with λ s = c + 1 + µ 2 + c µ 2 , f t (z) ≡ exp(-αtz) and m(z) given by Equation (4.1).

To compare test versus training performances, we now evaluate the training error by considering

1 n 1 T n XT w(t) = - 1 2πi γ f t (z, t) 1 n 1 T n XT 1 n X XT -zI p -1 w 0 dz - 1 2πi γ 1 -f t (z) z 1 n 1 T n XT 1 n X XT -zI p -1 1 n X1 n dz
which yields the following results. The proofs of Theorem 4.1 and 4.2 are provided in Section A.2.1 of the Appendix.

P(w(t) T xi < 0) -Q E * √ V * → 0 almost surely, with E * ≡ 1 2πi γ 1 -f t (z) z dz ( µ 2 + c) m(z) + 1 V * ≡ 1 2πi γ 1 z (1 -f t (z)) 2 ( µ 2 + c) m(z) + 1 -σ 2 f 2 t (z)
In Figure 4.1 we compare finite dimensional simulations with theoretical results obtained from Theorem 4.1 and 4.2 and observe a very close match, already for not too large n, p. As t grows large, the test error first drops rapidly with the training error, then increases, although slightly, while the training error continues to decrease to zero. This is because the classifier eventually over-fits the training data X and performs badly on unseen data. To avoid over-fitting, one effectual approach is to apply regularization strategies [START_REF] Christopher | Pattern Recognition and Machine Learning[END_REF]; for example, to "early stop" (at t = 100 for instance in the setting of Figure 4.1) in the training process. This introduces new hyperparameters such as the optimal stopping time t opt that is of crucial importance for the network performance and is often tuned through cross-validation in practice. Theorem 4.1 and 4.2 tell us that the training and test performances, although random themselves, have asymptotically deterministic behaviors described by (E * , V * ) and (E, V), respectively, which allows for a deeper understanding on the choice of t opt , since E, V are in fact functions of t via f t (z) ≡ exp(-αt(z + λ)).

Nonetheless, the expressions in Theorem 4.1 and 4.2 of contour integrations are not easily analyzable nor interpretable. To gain more insight, we shall rewrite (E, V) and (E * , V * ) in a more readable way. First, note from Figure 4.2 that the matrix 1 n XX T has (possibly) two types of eigenvalues: those inside the main bulk (between

λ -≡ (1 - √ c) 2 and λ + ≡ (1 + √ c) 2 ) of the Marčenko-Pastur distribution ν(dx) = (x -λ -) + (λ + -x) + 2πcx dx + (1 -c -1 ) + δ(x) (4.2)
and a (possibly) isolated one lying away from [λ -, λ + ], that will be treated separately. We rewrite the path γ (that contains all eigenvalues of 1 n XX T ) as the sum of two paths γ b and γ s , that circle around the main bulk and the isolated eigenvalue (if any), respectively. To handle the first integral of γ b , we use the fact that for any nonzero λ ∈ R, the limit lim z∈C→λ m(z) ≡ m(λ) exists [START_REF] Jack | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] and follow the idea in [START_REF] Zhidong | Clt for linear spectral statistics of large-dimensional sample covariance matrices[END_REF] by choosing the contour γ b to be a rectangle with sides parallel to the axes, intersecting the real axis at 0 and λ + and the horizontal sides being at a distance ε → 0 from the real axis, to split the contour integral into four real line integrals. The second integral circling around γ s can be computed by residue calculus. This together leads to the expressions of (E, V) and (E * , V * ) as follows.

E = 1 -f t (x) x µ(dx) (4.3) V = µ 2 + c µ 2 (1 -f t (x)) 2 µ(dx) x 2 + σ 2 f 2 t (x)ν(dx) (4.4) E * = µ 2 + c µ 2 1 -f t (x) x µ(dx) (4.5) V * = µ 2 + c µ 2 (1 -f t (x)) 2 µ(dx) x + σ 2 x f 2 t (x)ν(dx) (4.6)
where we recall f t (x) = exp(-αtx), ν(x) given by (4.2) and denote the measure

µ(dx) ≡ (x -λ -) + (λ + -x) + 2π(λ s -x) dx + ( µ 4 -c) + µ 2 δ λ s (x) (4.7)
as well as

λ s = c + 1 + µ 2 + c µ 2 ≥ ( √ c + 1) 2 (4.8)
with equality if and only if µ 2 = √ c. A first remark on the expressions of (4.3)-(4.6) is that E * differs from E only by a factor µ 2 +c µ 2 . Also, both V and V * are the sum of two parts: the first part that strongly depends on µ and the second one that is independent of µ. One thus deduces for µ → 0 that E → 0 and

V → (1 -f t (x)) 2 x 2 ρ(dx) + σ 2 f 2 t (x)ν(dx) > 0 with ρ(dx) ≡ √ (x-λ -) + (λ + -x) + 2π(c+1)
dx. Therefore the test performance goes to Q(0) = 0.5. On the other hand, for µ → ∞, one has E √ V → ∞ and hence the classifier makes perfect predictions.

In a more general context (i.e., for Gaussian mixture models with generic means and covariances as investigated in [START_REF] Benaych | Spectral analysis of the gram matrix of mixture models[END_REF], and obviously for practical datasets), there may be more than one eigenvalue of 1 n XX T lying outside the main bulk, which may not be limited to the interval [λ -, λ + ]. In this case, the expression of m(z), instead of being explicitly given by (4.1), may be determined through more elaborate formulations as in the sample covariance model in Section 2.2.3. While handling more generic models is technically reachable within the present analysis scheme, the results are much less intuitive. Similar objectives cannot be achieved within the framework presented in [START_REF] Madhu | High-dimensional dynamics of generalization error in neural networks[END_REF]; this conveys more practical interest to our results and the proposed analysis framework.

Practical consequences. With a careful inspection of (4.3) and (4.4), we now discuss several different aspects of their practical implications.

• On the test performance. First of all, recall that the test performance is simply given by Q

µ T w(t) w(t)
, with the term µ T w(t) w(t) describing the alignment between w(t) and µ. Therefore the best possible test performance is simply Q( µ ). Nonetheless, this "best" performance can never be achieved as long as p/n → c > 0, as described in the following remark. Remark 4.1 (Optimal test performance). Note that, with Cauchy-Schwarz inequality and the fact that µ(dx) = µ 2 from (4.7), one has

E 2 ≤ (1 -f t (x)) 2 x 2 dµ(x) • dµ(x) ≤ µ 4 µ 2 + c V
with equality in the right-most inequality if and only if the variance σ 2 = 0. One thus concludes that E/ √ V ≤ µ 2 / µ 2 + c and the best test performance (lowest misclassification rate) is Q( µ 2 / µ 2 + c) which can be reached only when σ 2 = 0. Note in particular that the optimal (test) classification performance agrees with the theoretical optimum derived under the empirical risk minimization framework in Section 3.3. This remark is of particular interest because, for a given task (thus for p, µ fixed) it gives access to the minimum number of training data number n needed to fulfill a certain classification accuracy.

As a side remark, note that in the expression of E/ √ V the initialization variance σ 2 only appears in V, meaning that random initializations impair the test performance of the network. As such, one should initialize with σ 2 very close, but not equal, to zero, to obtain symmetry breaking between hidden units [START_REF] Goodfellow | Deep Learning[END_REF] as well as to mitigate the drop of performance due to large σ 2 . In Figure 4.3 we plot the optimal test performance with the corresponding optimal stopping time as a function of σ 2 , showing that small initializations help training both in terms of accuracy and efficiency.

• Approximation for t close to 0. Although the integrals in (4.3) and (4.4) do not have nice closed forms, note that, for t close to 0, with a Taylor expansion of f t (x) ≡ exp(-αtx) around αtx = 0, one gets more interpretable forms of E and V without integrals, as presented next.

Taking t = 0, one has f t (x) = 1 and therefore E = 0, V = σ 2 ν(dx) = σ 2 , with ν(dx) the Marčenko-Pastur distribution given in (4.2). As a consequence, at the beginning stage of training, the test performance is Q(0) = 0.5 for σ 2 = 0 and the classifier makes random guesses.

For t not equal but close to 0, the Taylor expansion of f t (x) ≡ exp(-αtx) around αtx = 0 gives

f t (x) ≡ exp(-αtx) ≈ 1 -αtx + O(α 2 t 2 x 2 ).
Making the substitution x = 1 + c -2 √ c cos θ and with the fact that π 0 sin 2 θ p+q cos θ dθ = pπ q 2 1 -1q 2 /p 2 (see for example 3.644-5 in [START_REF] Solomonovich | Table of integrals, series, and products[END_REF]), one gets

E = Ẽ + O(α 2 t 2 ) and V = Ṽ + O(α 2 t 2 ), where Ẽ ≡ αt 2 g(µ, c) + ( µ 4 -c) + µ 2 αt = µ 2 αt Ṽ ≡ µ 2 + c µ 2 ( µ 4 -c) + µ 2 α 2 t 2 + µ 2 + c µ 2 α 2 t 2 2 g(µ, c) + σ 2 (1 + c)α 2 t 2 -2σ 2 αt + 1 - 1 c + σ 2 + σ 2 2c 1 + c -(1 + √ c)|1 - √ c| = ( µ 2 + c + cσ 2 )α 2 t 2 + σ 2 (αt -1) 2 with g(µ, c) ≡ µ 2 + c µ 2 -µ + √ c µ µ - √ c µ and consequently 1 2 g(µ, c) + ( µ 4 -c) + µ 2 = µ 2 .
It is interesting to note from the above calculation that, although E and V seem to have different behaviors for µ 2 > √ c or c > 1, it is in fact not the case as the extra term involving µ 2 > √ c (or c > 1) exactly compensates the singularity of the integral. As such, the test performance of the classifier is a smooth function of both µ 2 and c, and there is indeed no "phase transition" as far as performance is concerned.

Taking the derivative of Ẽ Yet, the approximation error arising from Taylor expansion can be large for t away from 0, e.g., at t = 1/α the difference E -Ẽ is of order O(1) and thus cannot be neglected.

• As t → ∞: the least-squares solution. As t → ∞, one has f t (x) → 0 which results in the least-square solution w LS = (XX T ) -1 Xy and consequently

µ T w LS w LS = µ 2 µ 2 + c 1 -min c, 1 c . ( 4
.9)

Comparing (4.9) with the expression in Remark 4.1, one observes that when t → ∞ the network becomes "over-trained" and the performance drops by a factor 1min(c, c -1 ). This becomes worse when c gets close to 1, which is consistent with the empirical findings in [START_REF] Madhu | High-dimensional dynamics of generalization error in neural networks[END_REF]. However, the point c = 1 is a singularity for (4.9), but not for E √ V as in (4.3) and (4.4). One may thus expect to have a smooth and reliable behavior of the well-trained network for c close to 1, which is a noticeable advantage of gradient-based training compared to the simple least-squares CHAPTER 4. GRADIENT DESCENT DYNAMICS IN NEURAL NETWORKS method. This coincides with the conclusion of [START_REF] Yao | On early stopping in gradient descent learning[END_REF] in which the asymptotic behavior in the limit n → ∞ with p fixed is considered.

In Figure 4.4 we plot the test performance from simulation (blue line), the approximation from Taylor expansion of f t (x) as described above (red dashed line), together with the performance of w LS (cyan dashed line). One observes a close match between the result from Taylor expansion and the true performance for t small, with the former being optimal at t = 100 and the latter slowly approaching the performance of w LS as t goes to infinity.

In Figure 4.5 we underline the case c = 1 by taking p = n = 512 with all other parameters unchanged from Figure 4.4. One observes that the simulation curve (blue line) increases much faster compared to Figure 4.4 and is supposed to end up at 0.5, which is the performance of w LS (cyan dashed line). This confirms a serious degradation of performance for c close to 1 of the classical least-squares solution. ±µ and identity covariance C = I p . Based on the random matrix-based analysis, both the training and test performance of the network establish asymptotically tractable behaviors as n, p → ∞ with p/n → c > 0. These performances can be expressed generally in the form of contour integrations, which can further be reduced to real integrals for the simple model here with C = I p . We obtain several interesting yet counterintuitive conclusions: one should prohibit the use of large random initialization, as it not only harms the network performance but also slows down the convergence rate (i.e., having a larger optimal stopping time); also, different from the initial stage of training, when the network is over-trained as t goes large, the performance drops by a factor 1min(c, c -1 ), and the misclassification error rate blows up when n ≈ p, i.e., the number of training sample approximates the number of network parameters (and also the data dimension in this case).

A Geometric Approach to GDD of Linear NNs

In this section, we move forward to consider linear DNN models with H ≥ 1 hidden layers as in [START_REF] Andrew M Saxe | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[END_REF][START_REF] Kawaguchi | Deep learning without poor local minima[END_REF] and to evaluate the dynamics of the associated gradient system in a "continuous" manner. We prove that, in the case H = 1, for almost every choice of training data-target pair (X, Y) and almost every initialization for the weight matrices W 1 , W 2 (see network configuration in Figure 4.6 below), the corresponding trajectory of the gradient system exists for all t ≥ 0 and converges to a global minimizer of the square loss function. Based on a key "invariant" structure in the network weight space induced by the network cascading structure, we further propose a generic framework for the geometric understanding of linear deep neural networks, including a critical initialization scheme that ensures exponential convergence rate, a detailed description of the (firstorder) stationary points, as well as the associated Hessians and basins of attraction of these stationary points. For a deep linear neural network with H hidden layers as illustrated in Figure 4.6, we introduce the following notations and basic settings.

Denote (X, Y) the training data and associated targets, with X ≡ x 1 , . . . , x n ∈ R d x ×n and Y ≡ [y 1 , . . . , y n ] ∈ R d y ×n , where n denotes the number of instances in the training set and d x , d y the dimensions of data and targets, respectively. We denote the weight matrix

x ∈ R d x h 1 ≡ W 1 x ∈ R d 1 h H ≡ W H h H-1 ŷ ≡ W H+1 h H W 1 ∈ R d 1 ×d x . . . W H+1 ∈ R d y ×d H
W i ∈ R d i ×d i-1
that connects h i-1 to h i for i = 1, . . . H + 1 and set h 0 = x, h H+1 = ŷ as in Figure 4.6. The network output is thus given by Ŷ = W H+1 . . . W 1 X. We denote W the (H + 1)-tuple of (W 1 , . . . , W H+1 ) for simplicity and work on the square loss L(W) given by the Frobenius norm below,

L(W) = 1 2 Y -Ŷ 2 F = 1 2 Y -W H+1 . . . W 1 X 2 F . (4.10)
We position ourselves under the following assumptions:

Assumption 10 (Dimension condition). n ≥ d x ≥ max(d 1 , . . . , d H ) ≥ min(d 1 , . . . , d H ) > d y .
In particular in the case H = 1 this condition yields n ≥ d

x ≥ d 1 > d y .
Assumption 11 (Full rank data and targets). The matrices X and Y are of full (row) rank, i.e., of rank d x and d y , respectively, accordingly with Assumption 10.

Assumption 10 and 11 on the dimension and rank of the training data are realistic and practically easy to satisfy, as discussed in previous works [START_REF] Baldi | Neural networks and principal component analysis: Learning from examples without local minima[END_REF][START_REF] Kawaguchi | Deep learning without poor local minima[END_REF]. Indeed, Assumption 10 is demanded here for convenience and our results can be extended to handle more elaborate dimension settings. Similarly, when the training data is rank deficient, the learning problem can be reduced to a lower-dimensional one by removing these non-informative data in such a way that Assumption 11 holds.

Under Assumptions 10 and 11, with the singular value decomposition on X

= U X Σ X V T X with V X = V 1 X V 2 X , V 1 X ∈ R n×d x and then on YV 1 X ≡ Ȳ = U Y Σ Y V T Y , together with a change of variable, we get L(W) = L(W) + 1 2 YV 2 X 2 F with L(W) ≡ 1 2 Σ Y -W H+1 W H . . . W 2 W 1 2 F (4.11)
where Σ X ≡ S X 0 ∈ R d x ×n , Σ Y ∈ R d y ×d x and we denote, with a slight abuse of notations that

W H+1 ≡ U T Y W H+1 ∈ R d y ×d H and W 1 ≡ W 1 U X S X V Y ∈ R d 1 ×d x . Therefore the state space 1 of W ≡ (W H+1 , . . . , W 1 ) is equal to X = R d y ×d H × . . . × R d 1 ×d x . In particular, for H = 1 we have d H = d 1 and X has dimension d 1 (d x + d y ).
With the above notations, we demand also the following assumption on the target Y.

Assumption 12 (Distinct singular values). The target Y has d y distinct singular values.

The objective of this section is to study the gradient descent [START_REF] Boyd | Convex optimization[END_REF] dynamics (GDD) defined as follows.

In particular, in the case of H = 1 we get W

T 2 W 2 -W 1 W T 1 = C 0 ≡ (W T 2 W 2 -W 1 W T 1 ) t=0 . Proof. Simply check that d dt W T j+1 W j+1 -W j W T j = 0.
Note in particular that, as a result of Lemma 4.1, there exists constant c j , 1 ≤ j ≤ H such that for all t ≥ 0,

W j+1 (t) 2 F -W j (t) 2 F = c j (4.14)
i.e., the difference between the (Frobenius) norms of successive weight matrices W j are preserved along with the gradient descent process. With the key Lemma 4.1 and the remark above, to prove the boundedness of W j (t) for all j = 1, . . . , H + 1, it suffices to bound any single one of them, say for j = H + 1. To this end, first note that

W H+1 . . . W 1 2 F = tr(W T H+1 W H+1 . . . W 2 W 1 W T 1 W T 2 . . . W T H ) = tr(W T H+1 W H+1 . . . (W 2 W T 2 ) 2 . . . W T H ) -tr(W T H+1 W H+1 . . . W 2 C 0 1 W T 2 . . . W T H )
together with the fact that for symmetric positive semi-definite A we have λ min (B) tr(A) ≤ tr(AB) ≤ λ max (B) tr(A) so that

| tr(W T H+1 W H+1 . . . W 2 C 0 1 W T 2 . . . W T H )| ≤ C 0 1 W H+1 . . . W 2 2 F
so that by successively applying Lemma 4.1 one deduces the following bound

P l (W H+1 ) ≤ W H+1 . . . W 1 2 F ≤ P u (W H+1 )
for some polynomial P l and P u of degree H + 1 with positive leading coefficients. As a consequence, by considering the time derivative of W H+1 2 F , one can similarly get

d W H+1 2 F dt ≤ -2C 0 W H+1 2(H+1) F + C 1 (1 + W H+1 2H F )
for some positive constants C 0 and C 1 , so that

d W H+1 2 F dt < 0 for some W H+1 2
F large enough and W H+1 remains bounded for t ≥ 0 and this holds for all W j , j = 1, . . . , H + 1 via (4.14). As such, by Lojasiewicz's theorem one achieves the global convergence to stationary points, stated as below.

Proposition 4.1 (Global convergence to stationary points). Let (X, Y) be a data-target pair satisfying Assumptions 10 and 11. Then, every GDD trajectory defined in Definition 8 converges to a (first-order) stationary point as t → ∞, at rate at least t -α , for some fixed α > 0 only depending on the problem.

As another important byproduct of Lemma 4.1, we have the following corollary that ensures an exponential convergence with a critical initialization scheme, and perhaps more importantly, to convergence to the global minima with L = 0.

C 0 j ≡ W j+1 (0) T W j+1 (0) -W j (0)W j (0) T ∈ R d j ×d j
has at least d j+1 positive eigenvalues, i.e., λ d j+1 (C 0 j ) > 0 for j = 1, . . . , H. Then, every trajectory of the GDD converges to a global minimum at least at the rate of exp(-2αt), for some α > 0.

Proof. Recall the definition M = Σ Y -W H+1 . . . W 1 and consider its time derivative as

dM dt = - H+1 ∑ j=1 W H+1 . . . W j+1 dW j dt W j-1 . . . W 1 = - H+1 ∑ j=1 W H+1 . . . W j+1 W T j+1 . . . W T H+1 MW T 1 . . . W T j-1 W j-1 . . . W 1 so that dL dt = d M 2 F dt = -2 H+1 ∑ j=1 tr M T W H+1 . . . W j+1 W T j+1 . . . W T H+1 MW T 1 . . . W T j-1 W j-1 . . . W 1 ≤ -2 H+1 ∑ j=1 j-1 ∏ k=1 λ min (W T k W k ) tr M T W H+1 . . . W j+1 W T j+1 . . . W T H+1 M ≤ -2 H+1 ∑ j=1 j-1 ∏ k=1 λ min (W T k W k ) H+1 ∏ l=j+1 λ min (W l W T l ) M 2 F
where we constantly use the fact that for symmetric and positive semi-definite A we have | tr(AB)| ≥ λ min (B) tr(A). Therefore, if there exists at least 1

≤ j ≤ H + 1 such that H+1 ∏ l=j+1 λ min (W l W T l ) j-1 ∏ k=1 λ min (W T k W k ) > 0 (4.15)
then we obtain

d M 2 F dt ≤ -C M 2
F for some C > 0 and thus the conclusion. To this end, we derive, from Lemma 4.1 and Weyl's inequality (see Lemma 2.10) that, for j = 1, . . . , H

λ i (W T j+1 W j+1 ) ≥ λ i (C 0 j ) + λ min (W j W T j ) ≥ λ i (C 0 j ), i = 1, . . . , d j
with λ i (C 0 j ) the i-th eigenvalue of C 0 j arranged in nondecreasing order so that λ 1 (C 0 j ) = λ min (C 0 j ). Then since d 1 ≥ . . . ≥ d j ≥ d j+1 ≥ . . . ≥ d H , the matrix W T j+1 W j+1 ∈ R d j ×d j is of rank maximum d j+1 and thus admits at least d jd j+1 zero eigenvalues so that

λ i (W T j+1 W j+1 ) = 0, λ i (C 0 j ) ≤ 0 for i = 1, . . . , d j -d j+1 .
Moreover, since for i = 1, . . . , d j+1 we also have,

λ i+d j -d j+1 (W T j+1 W j+1 ) = λ i (W j+1 W T j+1 )
we further deduce that for i = 1, . . . , d j+1 ,

λ i (W j+1 W T j+1 ) = λ i+d j -d j+1 (W T j+1 W j+1 ) ≥ λ i+d j -d j+1 (C 0 j )
so that by taking j = 1 in (4.15) we result in

H ∏ l=1 λ min (W l+1 W T l+1 ) ≥ H ∏ l=1 λ d l -d l+1 +1 (C 0 l ) ≡ α > 0
which concludes the proof.

CHAPTER 4. GRADIENT DESCENT DYNAMICS IN NEURAL NETWORKS

Remark 4.1 entails that, for a linear network, although in general only polynomial convergence rate can be established, it is possible to wisely initialize the gradient descent algorithm to achieve exponential convergence.

Apart from the critical initialization scheme proposed in Corollary 4.1 that leads to a global convergence to global minima, Proposition 4.1 itself does not guarantee the "quality" of stationary point to which the trajectory converges.

In the following, we move on to consider the general situation that we initialize the gradient descent algorithm from a arbitrary point in the network weight space X and focus on the set of the first-order stationary points S. We consider for simplicity the case H = 1 and comment on H > 1.

Proposition 4.1 ensures, for all initializations, the convergence of the gradient descent to a stationary point, i.e., a point W in the state space X verifying ∇ W L(W) = 0. Nonetheless, the information on the "quality" (e.g., the loss value L) of the solution achieved is still unknown. To obtain a clearer picture, we now consider the case H = 1 and focus on the set of all stationary points by further decomposing the loss L with

Σ Y ≡ S Y 0 for diagonal S Y ∈ R d y ×d y with [S Y ] ii > 0 as L(W 1 , W 2 ) = 1 2 Σ Y -W 2 W 1 2 F = 1 2 S Y -CA 2 F + 1 2 CB 2 F (4.16) with C ≡ W 2 ∈ R d y ×d 1 , A ∈ R d 1 ×d y and B ∈ R d 1 ×(d x -d y ) such that A B ≡ W 1 .
Under the notations above, we further expand L(W + w) to obtain its higher order variation as

L(A + a, B + b, C + c) ≡ L(W + w) = L(W) + D W (w) + H W (w) + O w 3 with M ≡ S Y -CA, L(W) = 1 2 M 2 F + 1 2 CB 2 F and D W (w) ≡ -tr M T (Ca + cA) + tr B T C T (Cb + cB) = O( w ) H W (w) ≡ -tr(M T ca) + 1 2 Ca + cA 2 F + tr(B T C T cb) + 1 2 Cb + cB 2 F = O( w 2 )
that give the differential and the Hessian of L, respectively. Recall that S(L) ≡ {W | D W (w) = 0} and denote M ≡ S Y -CA, so that, by Definition 8,

     dA dt = C T M = 0 dB dt = -C T CB = 0 dC dt = MA T -CBB T = 0 ⇔      C T S Y = C T CA CB = 0 AS Y = AA T C T .
(4.17)

Observing the symmetric structure of A, C in (4.17) we have the following lemma. Moreover, denote r the common rank of A and C with 0 ≤ r ≤ d y . Then there exists some orthogonal matrix U ∈ R d y ×d y such that (A,B,C).

       AU = Ā 0 d 1 ×(d y -r) C T U = CT 0 d 1 ×(d y -r) U -1 S Y U = S Y (4.
For W ∈ S(L) with A, C of rank r with 0 ≤ r ≤ d y , rewriting S Y in two blocks r) . With Lemma 4.2, we then simplify (4.17) as

S Y = D Y 0 0 E Y , with D Y ∈ R r×r and E Y ∈ R (d y -r)×(d y -
C A = D Y CB = 0 , U T MU = 0 0 0 E Y (4.19)
with the fact that C T , A are both of full rank (equal to r). The loss L(W) (at stationary points) can thus be simplified as

L(W) = 1 2 E Y 2
F where E Y measures the "quality" of each stationary points.

For any W ∈ S(L), with Lemma 4.2 we are allowed to "extract" the full rank (sub-)structures of A, C with S Y unchanged, via a simple change of basis. For 0 ≤ r ≤ d y , let S r (L) be the subset of S(L) such that the rank of A and of C is equal to r. Then, one has the following disjoint union S(L) = ∪ d y r=0 S r (L). Proposition 4.2 (Landscape of single-hidden-layer linear NN model). Under Assumptions 10-12, the loss function L(W) has the following properties: 1. The set of possible limits of L along the GDD given by (4.12) is equal to the finite set made of the sum of the squares of any subset of the singular values of Ȳ.

2. The set S d y (L) is in fact the set of local (and global) minima, with L = 0 and M = 0.

3. Every first-order stationary point W ∈ S r (L) with 0 ≤ r ≤ d y -1 is a saddle point such that the Hessian has at least one negative eigenvalue. In particular, the set of saddle points is an algebraic variety of positive dimension, i.e., (up to a unitary matrix) the zero set of the polynomial functions given in (4.19), with E Y = 0.

For more general DNNs with H > 1, similar conclusion holds, except from the negative eigenvalue of the Hessian for all saddle points. As a matter of fact, for H > 1, there exists saddle points with positive semidefinite Hessian and we show that a sufficient condition to ensure at least one negative eigenvalue is to have rank(W H . . . W 2 ) > rank(W H+1 . . . W 2 ).

The fact that all local minima are equivalently global minima and all critical points that are not global minima are saddle points with at least one negative eigenvalue for the 122 CHAPTER 4. GRADIENT DESCENT DYNAMICS IN NEURAL NETWORKS Hessian is in fact already known for single-hidden-layer linear networks [START_REF] Baldi | Neural networks and principal component analysis: Learning from examples without local minima[END_REF][START_REF] Kawaguchi | Deep learning without poor local minima[END_REF]. Here, we improve the condition given in [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF], with an alternative and shorter proof based only on the quadratic form involving the Hessian, for deep linear networks.

With Proposition 4.2, and assume in addition that the loss function L admits two by two distinct values over two by two distinct subsets made of (the sum of the squares of) the singular values of the target Ȳ, one is able to provide a precise "local" description of all saddle points. Based on this description, one is able to further construct the local stable and unstable manifolds, with the help of the notation of normally hyperbolic [HPS06, Theorem 3.5]. Ultimately, with the fact that the loss value L is decreasing along the gradient descent trajectory and by contradiction, one reaches the convergence of the GDD trajectory to a global minima, from almost all initialization (in the sense of Lebesgue measure).

Conclusion.

The essential difference between the case H = 1 and deeper NN with H > 1 is the existence of saddle points with positive semidefinite Hessian matrices. In this case, one has to evaluate higher order information (≥ 3) to understand more precisely the basins of attraction of these undesired stationary points, or otherwise establish tighter bounds for the dimension of the parameter space X that is occupied by these bad saddle points and their basins of attraction. Alternatively, one may also wish to provide initialization scheme, in this H > 1 case, that can avoid, at least with high probability say, these bad saddle points, so that we can again apply the arguments as in the case H = 1.

Chapter 5

Conclusions and Perspectives

Conclusions

Under a simple GMM for the input data model, we investigated, in Section 3.1.1-3.2.2 the performance of three closely connected objects: large random kernel matrices, large dimensional random feature maps, and large simple neural networks random weights. In the setting where the number of data n and their dimension p are both large and comparable, the MSE of random neural networks, the misclassification error rate of kernel ridge regression, as well as the performance of kernel/random feature-based spectral clustering techniques, all depend heavily on the eigenspectrum or on a functional of a particular random kernel/nonlinear Gram matrix. The major technical challenge to theoretically grasp these kernel matrices, so as to assess the performance of the aforementioned algorithms, lies in the presence of nonlinearities, e.g., the kernel function f and the neural network activation function σ. To handle this difficulty, we placed ourselves under the critical regime of asymptotically non-trivial classification (namely, Assumption 2 in Section 1.2). In this regime, a "concentration" phenomenon emerges, for the similarly measures x ix j 2 /p or x T i x j /p, which then helps (asymptotically) "linearize" (or Taylor-expand to first orders) the nonlinear objects of interest. The associated eigenspectrum, therefore, behaves like (a scaled and shifted version of) the classical sample covariance/Gram matrix, perturbed by low rank matrices, and only depends on the local behavior of the nonlinear function at the "concentration" point.

This "concentration" effect, if theoretically convenient, however, hinders the full discriminative power of kernels (such as the discrimination of covariance matrices in different classes). To work this limitation around, a proper scaling as x T i x j / √ p is necessary, which we studied in the second part of the thesis. In this situation, a "concentration" cannot be established so that, in place of Taylor expansion arguments, we exploited the method of orthogonal polynomials to decompose the nonlinear f into the sum of a linear and a purely nonlinear parts, that both contribute to the kernel eigenspectrum, but in a very different way. A surprising outcome is that the kernel matrix eigenspectrum depends on f solely via its first three Hermite coefficients, suggesting that, even without concentration, the space of kernel functions can be mapped to a low dimensional subspace. Similar behavior is also observed for the empirical risk-based classifier in Section 3.3, where the performance only depends on the loss function L through the same three coefficients of the implicit proximal function. However, the performance description in Section 3.3 is more complicated and can only be expressed as the fixed point of a system of nonlinear equations. RMT analyses allow for direct access to the (asymptotic) performance of the aforementioned machine learning methods, so long as their solutions 123 are explicit or implicitly determined by (convex) optimization problems.

Aiming to study the impact of optimization algorithm in convex or non-convex problems, we considered in Section 4.1 the temporal evolution of a single-layer linear neural network, learned by the gradient descent method, still under a GMM for the input data. Despite being a linear classifier, the training and test performance of this linear regression model is a highly nonlinear function of the training time and involves the sum of a nonlinear function of all the eigenvalues of the input covariance matrix. Based on the proposed RMT analysis, both the training and test performance of the model can be fully characterized in the large n, p limit. This result provides a first precise description of large network performances when trained with gradient descent and sheds new light on many common questions in neural networks, such as overfitting, early stopping and the initialization of training, in the large dimensional regime where n, p → ∞ with p/n → c > 0.

When the number of hidden layers is more than one, the underlying optimization problem to train the neural network becomes non-convex and less tractable. In Section 4.2 we took a geometric and, so far, only deterministic approach to understand the convergence properties of simple gradient descent algorithms in this problem. We improved the existing analysis of the gradient descent dynamics in single-hidden-layer linear neural networks by considering the corresponding dynamical system. Based on a cornerstone "invariance" structure in the network weight space, we provided alternative and simpler proofs and removed some unnecessary assumptions from previous contributions to achieve the "almost sure" convergence to global minima.

Limitations and Perspectives

Yet, these findings are limited to the simple but seemingly restrictive Gaussian mixture modeling of the data. In the following, we discuss some possible future research directions, in the continuation of our preliminary findings.

Universality of RMT results in machine learning applications.

As discussed in the proof of the Marcenko-Pastur law in Section 2.2.2, as well as in the "properly scaled" inner product kernel matrix in Section 3.1.2, by positioning ourselves under the large dimensional setting where n, p are both large and comparable, many random matrix-based analyses, for linear or nonlinear models, lead to universal results, in the sense that they hold as long as the (original) random matrix Z ∈ R p×n has i.i.d. zero mean and unit variance entries, and are independent of the higher order statistics. Since real data generally do not comply with this stringent i.i.d. assumption, it is of interest to investigate the limitation of this universality phenomenon in machine learning applications. It is worth pointing out that the universality has already been observed to collapse in many nonlinear models, for instance in the "improperly scaled" shift-invariant kernel f ( x ix j 2 /p) investigated in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] where, by considering non-Gaussian Z with a non-zero excess kurtosis κ = E[Z 4 ij ] -3 = 0, the random variable ψ i ≡ ω i 2 -E[ ω i 2 ] yields a different (statistical) behavior, which nonetheless only moderately impacts the performances of kernel spectral clustering and kernel ridge regression methods. In [START_REF] Louart | A random matrix approach to neural networks[END_REF][START_REF] Benigni | Eigenvalue distribution of nonlinear models of random matrices[END_REF] where one considers the Gram matrix σ(WX) T σ(WX) in a random neural network context, it has been observed that the higher order moments of the distribution of (the i.i.d. entries of) the weight matrix W has an important impact on the eigenspectrum of the Gram matrix and consequently on the performance of the resulting neural network model. In this respect, one may wonder on the possibility to identify machine learning methods that 5.2. LIMITATIONS AND PERSPECTIVES 125 are "universal" and depend only on the first several moments of the data distribution from those that depend on the higher order moments of the data modeling. From a practical standpoint, it is perhaps of more importance to empirically investigate which tasks can be sufficiently well fulfilled by only considering the first and second order information from the data. For non-universal models, a possible idea is to perform preliminary studies on the influence of higher order information, starting from a qualitative study, so as to understand the (systematic) advantage of certain machine learning methods in some specific (e.g., kurtosis-sensitive) applications to provide theoretical guarantees of the existing algorithms as well as new insights in designing more efficient techniques.

RMT-based analyses of optimization problems. The random matrix analyses of optimization problems developed in this thesis are twofold:

1. As in Section 3.3, by considering the optimality condition of convex or non-convex optimization problems (e.g., first-or second-order stationary points condition), RMT analyses provide statistical descriptions of the local or global optima, which further lead to a theoretical understanding of the algorithm performance as well as a better hyperparameter tuning or "unbiasing" for these methods, as discussed at the end of Section 3.3. This scheme mainly works for convex (or strongly convex) optimization problems as in Section 3.3, where the uniqueness of the optimal solution is naturally guaranteed.

2. For non-convex optimization problems as the single-hidden-layer neural networks studied in Section 4.2, there may exist in general a large or even infinite number of first or higher order stationary points. In this case, even if one manages to establish an almost sure global convergence to global minima, the rate of convergence can be extremely slow due to the presence of (possibly numerous) saddle points [DJL + 17].

A wisely chosen initialization (as proposed in Corollary 4.1) within a locally convex structure helps accelerate the convergence rate, up to linear/exponential. This is where a RMT analysis may serve practical purposes: by leveraging the random nature of many non-convex optimization problems (e.g., with sensing matrices having i.i.d. standard Gaussian entries [START_REF] Emmanuel | Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements[END_REF] or with random measurement of random signals in phase retrieval [START_REF] Chen | Solving random quadratic systems of equations is nearly as easy as solving linear systems[END_REF]), it is possible to propose the following two-step algorithms:

• initialization, that makes a RMT-based initial guess so as to start the gradient descent away from any saddle points and within the "local valleys" of local (and global) minima;

• iterative refinement based on the gradient method, without leaving the initial valley.

In this case, the convergence rate relies heavily on a proper initialization, that can be achieved by means of spectral methods, e.g., to initialize with the top eigenvectors of a carefully designed matrix, as in the case of phase retrieval, matrix sensing/completion and many others [START_REF] Chi | Nonconvex optimization meets low-rank matrix factorization: An overview[END_REF]. In many cases, by considering a statistical modeling of the problem with repeated and informative patterns, it is possible to construct a data-dependent (random) matrix M that follows a spiked model as discussed at length in Section 2.3. The RMT analysis can then be applied to capture the statistical behavior of the top eigenvectors (that correspond to the isolated eigenvalues) of M, so as to measure the "distance" between the proposed (spectral-based) initialization and the desired global minima. In this regard, RMT offers not only a powerful initialization scheme by proposing an optimal design for M, but also a precise description of the conditions under which the proposed initialization works, by establishing "phase transition" conditions as discussed at the end of Section 2.3.

As a closing remark, in the regime where n, p, N are all large and comparable, with N the number of model parameters, many counterintuitive phenomena arise (recall from Figure 1.5 that for N > n the test error decreases as the ratio N/n increases), for which we need a deeper theoretical understanding. Indeed, modern machine learning systems are often highly over-parameterized and it is an everyday occurrence in modern DNNs to have N > max(n, p). As a consequence, random matrix-based approaches are a promising tool to investigate these modern over-parameterized learning systems that are of greater importance today.

where we recall that the diagonal of both K and K -i contain only zero entries. Since we are interested in the Stieltjes transform m(z) = 1 n ∑ n i=1 Q ii (z), the key object is the (nonlinear) quadratic form 1 p f (x T i X -i / √ p)Q -i f (X T -1 x i / √ p).

To handle the nonlinear random vector f (X T -i x i / √ p), we follow [START_REF] Cheng | The spectrum of random inner-product kernel matrices[END_REF] and perform the following orthogonal decomposition of x j : for all j = i,

x j = α j x i x i + x ⊥ j (A.2)
where x i x i is the unit vector in the direction of x i and x ⊥ j lies in the (p -1)-dimensional subspace orthogonal to x i . By orthogonality between x ⊥ j and x i we have

x T i x j / √ p = α j x i / √ p ⇔ α j = x T i x j x i
for j = i. Since x i and x j are independent standard Gaussian vectors, we have, in the large p limit that x T j x i / √ p ∼ N (0, 1) and x i √ p. Moreover, α j ∼ N (0, 1), x ⊥ j ∼ N (0, I p-1 ) and both are decorrelated thus independent.

Indeed,

E x j [α i x ⊥ j ] = E x j x T i x j x i x j - x i x T i x j x i 2 = E x j x j x T j x i x i -E x j x i x T i x j x T j x i x i 3 = 0.
As such, we have for k = j, k = i that

x T j x k = α j α k + (x ⊥ j ) T x ⊥ k ≡ α j α k + Φ ⊥ jk (A.3)
where the cross terms disappear again by orthogonality. Note from (A.2) that with high probability, both x j and x ⊥ j are of (Euclidean) norm O( √ p) while α j = O(1), Similarly, in (A.3), both x T j x k and Φ ⊥ jk are of order O( √ p), while α j α k = O(1). In this sense, Φ ⊥ jk is asymptotically close to the inner product x T j x k , with only the contribution from x i excluded and explicitly given by α j α k . We further denote α -i = α 1 , . . . , α i-1 , α i+1 , . . . , α n T ∈ R n-1 and K ⊥ -i ∈ R (n-1)×(n-1) with (j, k) entry given by

[K ⊥ -i ] jk ≡ δ j =k f (x ⊥ j ) T x ⊥ k / √ p / √ p = δ j =k f (Φ ⊥ jk / √ p)/ √ p (A.4) so that f (X T -i x i / √ p) f (α -i ) in the sense that f (X T -i x i / √ p) -f (α -i ) / √ p → 0 as p → ∞.
It is worth remarking here that, intuitively speaking, the random vector α -i is merely a standard Gaussian random vector α -i ∼ N (0, I n-1 ) in the large n, p limit in the sense that its each entry is "asymptotically" Gaussian and uncorrelated with each other.

The advantage of introducing Φ ⊥ (as well as K ⊥ -i ) is that α -i is "essentially" asymptotically independent of Φ ⊥ in the sense that the expectation E[Φ ⊥ α -i ] asymptotically vanishes. Note that this is in particular not the case for E[K -i α -i ] as an instance.

Since the study of K -i boils down to that of K ⊥ -i , we will need in the remainder its resolvent Q ⊥ -i ≡ K ⊥ -i -zI n-1 -1

the resolvent of K ⊥ -i that is therefore also independent of α -i .

Nonlinear quadratic forms. We first focus on K -i ∈ R (n-1)×(n-1) . By (A.3), the (k, l)entry of K -i ≡ f (X T -i X -i / √ p)/ √ p is given by

[K -i ] jk = 1 √ p f 1 √ p α j α k + 1 √ p Φ ⊥ jk
where we recall that Φ ⊥ jk / √ p = O(1), α j α k / √ p = O(p -1/2 ) and they are independent.

As a consequence, with a Taylor expansion of f 1 √ p α j α k + 1 √ p Φ ⊥ jk around the dominant Φ ⊥ jk / √ p we obtain 1

f 1 √ p α j α k + 1 √ p Φ ⊥ jk = f 1 √ p Φ ⊥ jk + f 1 √ p Φ ⊥ jk 1 √ p α j α k + O(p -1 )
so that in matrix form

[K -i ] jk = 1 √ p f 1 √ p α j α k + 1 √ p Φ ⊥ jk = 1 √ p f 1 √ p Φ ⊥ jk + 1 p a 1 α j α k + 1 p g 1 √ p Φ ⊥ jk α j α k + O(p -3/2 ) = [K ⊥ -i ] jk + 1 p a 1 α -i α T -i -diag(α 2 -i ) jk + 1 p (diag(α -i )G diag(α -i )) jk + O(p -3/2 )
where we denote the shortcut g(x) = f (x)a 1 , for K ⊥ -i given by (A.4), [α 2 -i ] j = α 2 j and G ≡ g(Φ ⊥ / √ p) ∈ R (n-1)×(n-1) .

The linear part a 1 x of the nonlinear function f (x) is treated separately since, intuitively speaking, taking the derivative of f (x) with a 0 = 0 (see the last item of Assumption 3), results in E[ f (x)] = a 1 = 0. The fact that f (x) is not centered (with respect to the Gaussian measure), together with α -i = O( √ p), implies that a 1 p α -i α T -i that has non-vanishing operator norm as n, p → ∞. By subtracting a 1 from f (x), one obtains

[G/

√ p] jk = δ j =k g (x ⊥ j ) T x ⊥ k / √ p / √ p.

Since the (original) kernel matrix K is of bounded operator norm for all f with a 0 = E[ f (ξ)] = 0 and ξ ∼ N (0, 1) (see for a proof in [FM19, Theorem 1.7]), G/ √ p, which can be seen as another inner-product kernel matrix with centered kernel function g (with E[g(ξ)] = 0), has asymptotically bounded operator norm. Further noting that diag(α -i ) = O • (1), diag(α -i ) = O • (1), we conclude that

K -i = K ⊥ -i + a 1 p α -i α T -i + o • (1) (A.5)
where o • (1) denotes a matrix with vanishing operator norm as n, p → ∞. Here we use the fact that for A, B ∈ R n×n , we have AB ≤ A B and A ≤ n A ∞ .

We now move on to the quadratic form 1 p f (x T i X -i / √ p)Q -i f (X T -1 x i / √ p), for Q -i ≡ (K -i -zI n-1 ) -1 . As a consequence of (A.5) we deduce

Q -i K ⊥ -i + 1 p a 1 α -i α T -i -zI n-1 -1 = Q ⊥ -i - a 1 Q ⊥ -i 1 p α -i α T -i Q ⊥ -i 1 + a 1 p α T -i Q ⊥ -i α -i Q ⊥ -i - a 1 Q ⊥ -i 1 p α -i α T -i Q ⊥ -i 1 + a 1 p tr Q ⊥ -i Q ⊥ -i - a 1 Q ⊥ -i 1 p α -i α T -i Q ⊥ -i 1 + a -1 n p m(z)
where we recall Q ⊥ -i ≡ K ⊥ -i -zI n-1 -1 the resolvent of K ⊥ -i that is independent of α -i . Here we use Lemma 2.8 for the equality and Lemma 2.11 for the approximation in the second line.

With the approximation above and (A.2), the quadratic form of crucial interest can be expanded as

1 p f (x T i X -i / √ p)Q -i f (X T -1 x i / √ p) 1 p f (α -i ) T Q ⊥ -i f (α -i ) -a 1 1 p α T -i Q ⊥ -i f (α -i ) 2 1 + a 1 n p m(z) a 2 1 p α T -i Q ⊥ -i α -i + 1 p f >1 (α -i )Q ⊥ -i f >1 (α -i ) -a 1 a 1 p α T -i Q ⊥ -i α -i 2 1 + a 1 n p m(z)
where we write the Hermite polynomial f (x) as the sum of its linear part a 1 x and the purely nonlinear part f >1 (x) = f (x)a 1 x that is orthogonal to x in the sense that E ξ [ξ f >1 (ξ)] = 0 for ξ ∼ N (0, 1). Again by orthogonality, the cross terms of α -i and f >1 (α -i ) vanish. Also, since

1 p f >1 (α -i )Q ⊥ -i f >1 (α -i ) (ν -a 2 1 ) n p m(z)
one obtains the following approximation for the nonlinear quadratic form of interest

1 p f (x T i X -i / √ p)Q -i f (X T -1 x i / √ p) a 2 1 n p m(z) 1 + a 1 n p m(z) + (ν -a 2 1 ) n p m(z).
Ultimately with (A.1) one has . Before going into the detailed proof, as we will frequently deal with random variables evolving as n, p grow large, we will use the extension of the O(•) notation introduced in [CBG16]: for a random variable x ≡ x n and u n ≥ 0, we write x = O(u n ) if for any η > 0 and D > 0, we have n D P(x ≥ n η u n ) → 0. Note that under Assumption 4 it is equivalent to use either O(u n ) or O(u p ) since n, p scale linearly. In the following we shall use constantly O(u n ) for simplicity.

m(z) = 1 n n ∑ i=1 1 -z -1 p f (x T i X -i / √ p)Q -i f (X T -1 x i / √ p) 1 -z -
When multidimensional objects are concerned, v = O(u n ) means the maximum entry of a vector (or a diagonal matrix) v in absolute value is of order O(u n ) and M = O(u n ) means that the operator norm of M is of order O(u n ). We refer the reader to [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] for more discussions on these practical definitions. We start with the resolvent Q. The terms of leading order in K, i.e.,-2 f (τ)A n and n γ I are both of operator norm O(n). Therefore a Taylor expansion can be performed as

Q = K + n γ I -1 = 1 n L -1 - 2 f (τ) n A √ n + A 1 + PΩ T ΩP + βI n + O(n -3 2 ) -1 = L n + 2 f (τ) n 2 LA √ n L + L B - β n 2 I L + O(n -5 2 ) with L = f (τ) 1 n 1 n 1 T n + I γ -1
of order O(1) and B = 2 f (τ)

n 2 A 1 + PΩ T ΩP + 2 f (τ) n A √ n LA √ n .
With the Sherman-Morrison formula we are able to compute explicitly L as

L = f (τ) 1 n 1 n 1 T n + I γ -1 = γ I - γ f (τ) 1 + γ f (τ) 1 n 1 n 1 T n = γ 1 + γ f (τ) I + γ 2 f (τ) 1 + γ f (τ) P = O(1). (A.8)
Writing L as a linear combination of I and P is useful when computing L1 n or 1 T n L, because by the definition of P = I -1 n 1 n 1 T n , we have 1 T n P = P1 n = 0. We shall start with the term 1 T n Q, since it is the basis of several other terms appearing in β and b,

1 T n Q = γ1 T n 1 + γ f (τ) I n + 2 f (τ) n 2 A √ n L + B - β n 2 I L + O(n - 3

A.1.4 Proof of Proposition 3.2

We aim to prove, for f (x) = x k with k ≥ 2, the informative kernel matrix K I in Proposition 3.2 defined as Define by L the matrix with L ij ≡ [ 1 p (JM T MJ T + JM T Z + Z T MJ T )] ij for i = j and L ii = 0. Then K I can be written as

K I = k √ p (Z T Z/ √ p)
K I = k(Z T Z/ √ p) •(k-1) • L + Φ, Φ ij ≡ k p (z T i z j / √ p) k-1 z T i 1 2 (E a + E b ) - 1 8 (E a -E b ) 2 z j + k(k -1) 8p (z T i z j / √ p) k-2 1 √ p (z T i (E a + E b )z j ) 2
for i = j and Φ ii = 0. With this expression, the proof of Proposition 3.2 can be divided into three steps. for independent random vectors x, y ∼ N (0, I p ) in the Gaussian case. As such, we resort to computing, as in [Wil97, LLC18] (or Section A.1.6), the integral

E z [(z T a) k 1 (z T b) k 2 ] = (2π) -p/2 R p (z T a) k 1 (z T b) k 2 e -z 2 /2 dz = 1 2π R 2 ( z1 ã1 ) k 1 ( z1 b1 + z2 b2 ) k 2 e -( z2 1 + z2 2 )/2 d z1 d z2 = 1 2π R 2 ( zT ã) k 1 ( zT b) k 2 e -
E z i (z T i z j / √ p) k (z T i b) = E z i (z T i z j / √ p) k-1 (z T i b) 2 = 0; E z i (z T i z j / √ p) k-1 (z T i b) = (k -1)!!( z j / √ p) k-2 (z T j b)/ √ p; E z i (z T i z j / √ p) k (z T i b) 2 = (k -1)!! k( z j / √ p) k-2 (z T j b/ √ p) 2 + ( z j / √ p) k b 2 ;
where k!! is the double factorial of an integral k defined by k!! = k(k -2)(k -4) x i α y i α p ∑ j 1 ,j 2 =1 F j 1 ,j 2 x j 1 y j 2 (A.24) with x i and y i the i-th entry of x and y, respectively, so that i) x i is independent of y j for all i, j and ii) x i is independent of x j for i = j with E[x i ] = 0, E[x 2 i ] = 1 and E[|x i | k ] ≤ C k for some C k independent of p (and similarly for y).

At this point, note that to ensure E[K I ] has non-vanishing operator norm as n, p → ∞, we need E[φ] ≥ O(p -1 ) since A ≤ p A ∞ for A ∈ R p×p . Also, note that (as β = 1), all terms in the sum ∑ p j 1 ,j 2 =1 F j 1 ,j 2 x j 1 y j 2 with j 1 = j 2 must be zero since in other terms x i always appears together with y i , so that all terms with j 1 = j 2 give rise to zero in expectation. Hence, the p 2 terms of the sum only contain p nonzero terms in expectation (those with j 1 = j 2 ). The arbitrary (absolute) moments of x and y being finite, the first αp terms must be divided into α /2 groups of size two (containing O(p) terms) so that, with the normalization by p -1 for each group of size two, the associated expectation does not vanish. We thus discuss the following two cases:

1. α even: the α terms in the sum form α/2 groups with different indices each and also different from j 1 = j 2 . Therefore we have E x j [φ] = 0.

2. α odd: the α terms in the sum form (α -1)/2 groups with indices different from each other and the remaining one goes with the last term containing F and one has E[φ] = Cα!! p tr(F) by a combinatorial argument.

The case β = 2 follows exactly the same line of arguments except that j 1 may not equal j 2 to give rise to non-vanishing terms.

and therefore

E γ b = - 1 2πi γ b 1 -f t (z) z µ 2 m(z) 1 + ( µ 2 + c)m(z) dz = - µ 2 πi λ + λ - 1 -f t (x) x m -i m 1 + ( µ 2 + c)( m -i m) dx = - µ 2 πi λ + λ - 1 -f t (x) x m + µ 2 +c cx -i m 1 + 2( µ 2 + c) m + ( µ 2 +c) 2 cx dx
with z = x ± iε and in the limit ε → 0 (on different sides of the real axis) and with ( m) 2 + ( m) 2 = 1 cx . We then take the imaginary part in the right-hand side, which results in

E γ b = µ 2 π λ + λ - 1 -f t (x) x m 1 + 2( µ 2 + c) m + ( µ 2 +c) 2 cx dx = 1 2π λ + λ - 1 -f t (x) x 4cx -(1 -c -x) 2 λ s -x dx (A.26)
where we recall the definition λ s ≡ c + 1 + µ 2 + c µ 2 . Ultimately we assemble (A.25) and (A.26) to get the expression in (4.3). The derivations of (4.4)-(4.6) follow the same arguments and are thus omitted. Ces études pr éliminaires fournissent une premi ère compr éhension quantitative des algorithmes d'apprentissage pour le traitement de donn ées en grandes dimensions, ce qui permet de proposer de meilleurs crit ères de conception pour les grands syst èmes d'apprentissage et, par cons équent, d'avoir un gain de performance remarquable lorsqu'il est appliqu é à des jeux de donn ées r éels. Profond ément ancr é dans l'id ée d'exploiter des donn ées de grandes dimensions avec des informations r ép ét ées à un niveau "global" plut ôt qu' à un niveau "local", ce sch éma d'analyse RMT permet une compr éhension renouvel ée et la possibilit é de contr ôler et d'am éliorer une famille beaucoup plus large de m éthodes d'apprentissage automatique, ouvrant ainsi la porte à un nouveau sch éma d'apprentissage automatique pour l'intelligence artificielle.

Title : A random matrix framework for large dimensional machine learning and neural networks Keywords : Machine learning, random matrix theory, neural networks Abstract : Large dimensional data and learning systems are ubiquitous in modern machine learning. As opposed to small dimensional learning, large dimensional machine learning algorithms are prone to various counterintuitive phenomena and behave strikingly differently from the low dimensional intuitions upon which they are built. Nonetheless, by assuming the data dimension and their number to be both large and comparable, random matrix theory (RMT) provides a systematic approach to assess the (statistical) behavior of these large learning systems, when applied on large dimensional data. The major objective of this thesis is to propose a full-fledged RMT-based framework for various machine learning systems: to assess their performance, to properly understand and to carefully refine them, so as to better handle large dimensional problems that are increasingly needed in artificial intelligence applications. Precisely, we exploit the close connection between kernel matrices, random feature maps, and singlehidden-layer random neural networks. Under a simple Gaussian mixture modeling for the input data, we provide a precise characterization of the performance of these large dimensional learning systems as a function of the data statistics, the dimensionality, and most importantly the hyperparameters (e.g., the choice of the kernel function or activation function) of the pro-blem. Further addressing more involved learning algorithms, we extend the present RMT analysis framework to access large learning systems that are implicitly defined by convex optimization problems (e.g., logistic regression), when optimal points are assumed reachable. To find these optimal points, optimization methods such as gradient descent are regularly used. Aiming to have a better theoretical grasp of the inner mechanism of optimization methods and their impact on the resulting learning model, we further evaluate the gradient descent dynamics in training convex and non-convex objects.

These preliminary studies provide a first quantitative understanding of the aforementioned learning algorithms when large dimensional data are processed, which further helps propose better design criteria for large learning systems that result in remarkable gains in performance when applied on real-world datasets. Deeply rooted in the idea of mining large dimensional data with repeated patterns at a global rather than a local level, the proposed RMT analysis framework allows for a renewed understanding and the possibility to control and improve a much larger range of machine learning approaches, and thereby opening the door to a renewed machine learning framework for artificial intelligence.

Universit é Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

Figure 1 . 1 :

 11 Figure 1.1: Histogram of the empirical spectral distribution of Ĉ versus the Marcenko-Pastur law, for p = 500 and n = 50 000.

  with equality if and only if both C and ∆C are proportional to identity, i.e., C = 1 I p and ∆C = 2 I p . Also, by the Cauchy-Schwarz inequality, we have | tr ∆C| ≤ tr(∆C 2 ) • tr I p = O( √ p), with equality (again) if and only if ∆C = I p , and we must therefore have | tr ∆C| ≥ O( √ p). This allows us to conclude on the following non-trivial classification conditions ∆µ ≥ O(1), ∆C ≥ O(p -1/2 ), | tr ∆C| ≥ O( √ p), ∆C 2 F ≥ O(1).

Figure 1

 1 Figure 1.2: Kernel matrices K and the second top eigenvectors v 2 for small and large dimensional data X = [x 1 , . . . , x n ] ∈ R p×n with x 1 , . . . , x n/2 ∈ C 1 and x n/2+1 , . . . , x n ∈ C 2 .

( C1 )

 C1 Zhenyu Liao and Romain Couillet. Random matrices meet machine learning: a large dimensional analysis of LS-SVM. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2397-2401, 2017. (J1) Zhenyu Liao and Romain Couillet. A large dimensional analysis of least squares support vector machines. IEEE Transactions on Signal Processing, 67(4):1065-1074, 2019.

Figure 1 . 3 :

 13 Figure 1.3: Illustration of a single-hidden-layer random NN

( C3 )

 C3 Zhenyu Liao and Romain Couillet. On the spectrum of random features maps of high dimensional data. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3063-3071. PMLR, 2018.

c

  Figure 1.5: Classification error rate as a function of c, µ 2 = 5.

Figure 1

 1 Figure 1.6: Illustration of three types of stationary points in one dimension.

Figure 1 . 7 :

 17 Figure 1.7: Examples of stationary points in two-dimensional case.

Figure 1 . 8 :

 18 Figure 1.8: Examples of different local minima.

Figure 1 . 9 :

 19 Figure 1.9: Minimum energy path (left) and associated loss value (right) in [DVSH18, Figure 1].

( C1 )

 C1 Zhenyu Liao and Romain Couillet. Random matrices meet machine learning: a large dimensional analysis of LS-SVM. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2397-2401. 2017. and then with more discussions and detailed proofs in (J1) Zhenyu Liao and Romain Couillet. A large dimensional analysis of least squares support vector machines. IEEE Transactions on Signal Processing, 67(4):1065-1074, 2019.

( C3 )

 C3 Zhenyu Liao and Romain Couillet. On the spectrum of random features maps of high dimensional data. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3063-3071. PMLR, 2018.

(

  C4) Xiaoyi Mai, Zhenyu Liao, and Romain Couillet. A large scale analysis of logistic regression: Asymptotic performance and new insights. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3357-3361. IEEE, 2019. (J3) Xiaoyi Mai and Zhenyu Liao. High dimensional classification via empirical risk minimization: Improvements and optimality. 2019.

Theorem 2. 1 (

 1 Inverse Stieltjes transform). For a, b continuity points of the probability measure µ, we have µ([a, b]) = 1 π lim y↓0 b a m µ (x + ıy) dx.

Figure 2 .Figure 2

 22 Figure 2.1 depicts the density of the Marcenko-Pastur distribution for different values of c.For a fixed dimension p, the ratio c decreases as the number of samples n grows large, so that the eigenvalues of the sample covariance matrix become more "concentrated" (their spread is given by the length of the support [(1 -√ c) 2 , (1 + √ c) 2 ]) around the unique population covariance matrix eigenvalue equal to 1.

  > 0 as imposed by the definition of Stieltjes transforms. Now, from the inverse Stieltjes transform theorem, Theorem 2.1, we find that m(z) is the Stieltjes transform of the measure µ with µ([a, b]) = 1 π lim ↓0 b a [m(x + ı )]dx for all continuity points a, b ∈ R of µ. The term under the square root in m(z) being negative only in the set [(1 -√ c) 2 , (1 + √ c) 2 ], the latter defines the support of the continuous part of the measure µ with density

c

  to 0 on the interval ((1 + √ c) 2 , ∞). The equation λm(λ) = -1+ i i thus only has a solution if and only if

Figure 2 . 3 :

 23 Figure 2.3: Eigenvalues of 1 n XX T (BLUE crosses) and the Marcenko-Pastur law (red line) for X = C 1 2 X, C = I p + L with µ L = p-4 p δ 0 + 1 p (δ 1 + δ 2 + δ 3 + δ 4 ), for p = 500 and different values of n.

  n i,j=1 referred to as the kernel matrix [SV99]. 68CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS Given α and b, a new datum x is then classified into class C 1 or C 2 depending on the value of the decision function g(x) = β T k(x) + b (3.3)

Figure 3 .

 3 Figure 3.2 illustrates this result in the same settings as Figure 3.1. Compared to Figure 3.1, one can observe that both histograms are now centered close to 0 (at distanceO(n -1 ) from zero) instead of c 2c 1 = 1/2. Still,even in the case where normalized labels y * are used as observed in Figure3.2 (where the histograms cross at about -0.004 ≈ 1/n), taking ξ = 0 as a decision threshold may not be an appropriate choice, as Ḡ * 1 = -Ḡ * 2 .

  Figure 3.4: Performance of LS-SVM, c 0= 1/4, c 1 = c 2 = 1/2, λ = 1, polynomial kernel with f (τ) = 4, f (τ) = 2. x ∼ N (µ a , C a ), with µ 1 = µ 2 = 0 p , C 1 = I p and {C 2 } i,j = .4 |i-j| .

Figure 3 . 6 :

 36 Figure 3.6: Classification error rate for different kernel functionsf , n = 256, p = 512, c 1 = c 2 = 1/2 and λ = 1, with µ a = [0 a-1 ; 2; 0 p-a ], C 1 = I p and {C 2 } i,j = .4 |i-j| (1 + 4/ √ p).

Figure 3 . 7 :Figure 3

 373 Figure 3.7: Gaussian approximation of g(x), n = 256, p = 784, c 1 = c 2 = 1/2, λ = 1, Gaussian kernel with σ = 1, with MNIST (left) and Fashion-MNIST data (right), numbers 8 and 9.

3

  C + ≡ {z ∈ C, [z] > 0}. We also recall that, for m(z) the Stieltjes transform of a measure µ, µ can be obtained from m(z) via µ([a, b]) = lim ↓0 1 π b a[m(x + ı )]dx for all a < b continuity points of µ. See Section 2.1.2 for more details.

  Figure 3.10: Eigenvalue distribution and top eigenvector of K N and K, together with the limiting spectral measure from Theorem 3.3 (in red); f(x) = sign(x), Gaussian z i , n = 2 048, p = 512, µ 1 = -[3/2; 0 p-1 ] = -µ 2 and E 1 = E 2 = 0. X = [x 1 , . . . , x n ] ∈ R p×n with x 1 , . . . , x n/2 ∈ C 1 and x n/2+1 , . . . , x n ∈ C 2 .

Proposition 3. 2 (

 2 Low rank asymptotics of K I ). Under Assumptions 3 and 5, for f (x) = x k , k ≥ 2, K I -KI → 0 almost surely as n, p → ∞, for K I defined in (3.19) and KI = k!! p (JM T MJ T + JM T Z + Z T MJ T ), for k odd k(k-1)!! 2pJ (T + S) J T , for k even (3.20)

  Figure 3.13: Eigenvalue distribution and top eigenvectors of K for the piecewise constant function (in green) and the associated cubic function (in blue) with the same (a 1 , a 2 , ν), performed on Bernoulli distribution with zero mean and unit variance, in the setting of Figure 3.11.

Figure 3 . 14 :

 314 Figure 3.14: Illustration of a single-hidden-layer random NN

Figure 3 . 15 :

 315 Figure 3.15: Performance of the network for Lipschitz σ, as a function of λ, for MNIST data (number 7 and 9), N = 512, n = n = 1024, p = 784.

Figure 3 . 16 :

 316 Figure 3.16: Leading two eigenvectors of K c for the LReLU function with ς + = -ς -= 1 (top) and ς + = ς -= 1 (bottom), performed on four classes Gaussian mixture data with p = 512, n = 256, c a = 1/4 and j a = [0 n a-1 ; 1 n a ; 0 n-n a ], for a = 1, 2, 3, 4. Expectation estimated by averaging over 500 realizations of W.

C 2 Figure 3

 23 Figure 3.19: Eigenvalue distribution of K c and Kc for the epileptic EEG data (left) and leading eigenvector of K c for the EEG and Gaussian mixture data (right) with a width of ±1 standard deviations (generated from 500 trials). With the ReLU function, p = 100, n = 128 and c 1= c 2 = 1/2, j 1 = [1 n 1 ; 0 n 2 ] and j 2 = [0 n 1 ; 1 n 2 ].

104CHAPTER 3 .

 3 Figure 3.20: Comparison between the histogram of β T -i xi and the Gaussian distribution N (m, σ 2 ) defined in Theorem 3.8 with µ = [2, 0 p-1 ], C p = I p , for λ = 1, p = 256 and n = 512.

106CHAPTER 3 .

 3 Figure 3.21: Comparison of the empirical mean of β (denoted β) the solution of (1.22) for logistic loss L(t) = log(1 + e -t ), the optimal Bayes solution β * and the rescaled solution 2τ η β for (η, τ) derived from Theorem 3.8 with µ = [1 p/3 , -1 p/3 , 3 4 1 p/3 ]/ √ p, C = 2I p , for p = 60, n = 300. Empirical means obtained by averaging over 500 realizations.

  z)I p with m(z) determined by the popular Marčenko-Pastur equation[START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF]

Theorem 4. 2 (

 2 Training performance). Under the assumptions and notations of Theorem 4.1, as n, p → ∞ with p/n → c ∈ (0, ∞),

Figure 4 . 1 :

 41 Figure 4.1: Training and test performance for µ = [2; 0 p-1 ], p = 256, n = 512, σ 2 = 0.1, α = 0.01 and c 1 = c 2 = 1/2. Results obtained by averaging over 50 runs.

Figure 4 . 2 :

 42 Figure 4.2: Eigenvalue distribution of 1 n XX T for µ = [3/2; 0 p-1 ], p = 512, n = 1 024 and c 1 = c 2 = 1/2.

√ µ 2

 2 +c+cσ 2 and can be attained with t = 1/α. Moreover, taking t = 0 in the above equation one gets∂ ∂t Ẽ √ Ṽ t=0 = α σ .Therefore, large σ is harmful to the training efficiency, which coincides with the conclusion from Remark 4.1.

Figure 4 . 3 :

 43 Figure 4.3: Optimal performance and corresponding stopping time as functions of σ 2 , with c = 1/2, µ 2 = 4 and α = 0.01.

Figure 4 . 4 :Figure 4 . 5 :

 4445 Figure 4.4: Generalization performance for µ = [2; 0 p-1 ], p = 256, n = 512, c 1 = c 2 = 1/2, σ 2 = 0.1 and α = 0.01. Simulation results obtained by averaging over 50 runs.

Figure 4 . 6 :

 46 Figure 4.6: Illustration of the H-hidden-layer linear neural network

Corollary 4. 1 (

 1 Exponential convergence with critical initialization). Let Assumptions 10 and 11 hold. If we have in addition that d 1 ≥ d 2 ≥ . . . ≥ d H and the initialization

Lemma 4. 2 (

 2 Same kernel for A and C T ). Let Assumptions 10 and 11 hold. Then for all W ∈ S(L), ker A = ker C T , with ker A ≡ {x, Ax = 0}.

  Our interest here is on the decision function of LS-SVM: g(x) = β T k(x) + b with (β, b)

  T + 4 tr(C a C b )

I.

  Concentration of Φ. We first show that Φ -E[Φ] → 0 almost surely, as n, p → ∞. This follows from the observation that Φ is a p -1/4 rescaling (since E a = O(p -1/4 )) of the null model K N , which concentrates around its expectation in the sense thatK N -E[K N ] = O(1) for E[K N ] = O( √ p) if a 0 = 0 (see Remark 3.6). Indeed, itis shown in [FM19, Theorem 1.7] that, the leading eigenvalue of order O( √ p) discarded (arising from E[K N ]), K N is of bounded operator norm for all large n, p with probability one; this, together with the fact that E[Φ] = O(1) that will be shown subsequently (and independently), allows us to conclude that Φ -E[Φ] → 0 as n, p → ∞. II.I Computation of E[Φ]: Gaussian case. Recall that the entries of Φ are the sum of random variables of the type φ = C √ p (x T y/ √ p) α (x T Fy) β

  z 2 /2 d z where we apply the Gram-Schmidt procedure to project z onto the two-dimensional space 2 spanned by a, b with ã1 = a , b1 = a T b a , b2 = b 2 -(a T b) 2 a 2 and denote 138 APPENDIX A. MATHEMATICAL PROOFS z = [ z1 ; z2 ], ã = [ ã1 ; 0] and b = [ b1 ; b2 ]. As a consequence, we obtain, for k even, E (z T i z j / √ p) k = E[ξ k ] = (k -1)!!;

Titre:

  Th éorie des matrices al éatoires pour l'apprentissage automatique en grande dimension et les r éseaux de neuronesMots cl és : Apprentissage automatique, th éorie des matrices al éatoires, r éseaux de neurones R ésum é : Le "Big Data" et les grands syst èmes d'apprentissage sont omnipr ésents dans les probl èmes d'apprentissage automatique aujourd'hui. Contrairement à l'apprentissage de petite dimension, les algorithmes d'apprentissage en grande dimension sont sujets à divers ph énom ènes contre-intuitifs et se comportent de mani ère tr ès diff érente des intuitions de petite dimension sur lesquelles ils sont construits. Cependant, en supposant que la dimension et le nombre des donn ées sont à la fois grands et comparables, la th éorie des matrices al éatoires (RMT) fournit une approche syst ématique pour évaluer le comportement statistique de ces grands syst èmes d'apprentissage, lorsqu'ils sont appliqu és à des donn ées de grande dimension. L'objectif principal de cette th èse est de proposer un sch éma d'analyse bas é sur la RMT, pour une grande famille de syst èmes d'apprentissage automatique: d' évaluer leurs performances, de mieux les comprendre et finalement les am éliorer, afin de mieux g érer les probl èmes de grandes dimensions aujourd'hui.Pr écis ément, nous commenc ¸ons par exploiter la connexion entre les grandes matrices à noyau, les projection al éatoires non-lin éaires et les r éseaux de neurones al éatoires simples. En consid érant que les donn ées sont tir ées ind épendamment d'un mod èle de m élange gaussien, nous fournissons une caract érisation pr écise des performances de ces syst èmes d'apprentissage en grande dimension, exprim ée en fonction des statistiques de donn ées, de la dimensionnalit é et, surtout, des hyper-param ètres du probl ème. Lorsque des algorithmes d'apprentissage plus complexes sont consid ér és, ce sch éma d'analyse peut être étendu pour acc éder à de syst èmes d'apprentissage qui sont d éfinis (implicitement) par des probl èmes d'optimisation convexes, lorsque des points optimaux sont atteints. Pour trouver ces points, des m éthodes d'optimisation telles que la descente de gradient sont r éguli èrement utilis ées. À cet égard, dans le but d'avoir une meilleur compr éhension th éorique des m écanismes internes de ces m éthodes d'optimisation et, en particulier, leur impact sur le mod èle d'apprentissage, nous évaluons aussi la dynamique de descente de gradient dans les probl èmes d'optimisation convexes et non convexes.

  From Markov's inequality (i.e., P(|X| > t) ≤ E[|X| k ]/t k ) and Borel-Cantelli's lemma (i.e., P(|X n | > t) = O(n -) for some > 1 for all t > 0 implies X n a.s.

  Histogram of the eigenvalues of 1 n XX T , p = 300, n = 3000, for C having spectral measure µ

	2.2. FOUNDATIONAL RANDOM MATRIX RESULTS	57
			Empirical spectrum
	0.6		Limiting law (from Theorem 2.4)
	0.4
	0.2
	0	1	3	7
			Empirical spectrum
	0.6		Limiting law (from Theorem 2.4)
	0.4
	0.2
	0	1	3 4
	Figure 2.2:
					we obtain the ap-
	proximation
	1 n	tr(F + zI p ) QAQ	1 n tr C QAQ 1 + 1 n tr QC

Table 3

 3 Classification performance of LS-SVM, n = 256, p = 784, c 1 = c 2 = 1/2, λ = 1, Gaussian kernel, MNIST and Fashion-MNIST data (numbers 8 and 9).

	.1: Empirical estimation of differences in means and covariances of MNIST and
	Fashion-MNIST data (numbers 8 and 9)
	MNIST data Fashion-MNIST data
	2 tr 2 (C 2 -C 1 )/p µ 2 -µ 2	251 19	483 89
	C 2 -C 1 ) 2 2 F /p	30	86

Conclusion.

In this section, we investigated the asymptotic performance of kernel ridge regression (or LS-SVM) in separating a two-class Gaussian mixture. We saw that, due to 78CHAPTER 3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETS

  ) Stud: eigs of K for P 3

	Figure 3.11: Eigenvalue distributions of K and K from Theorem 3.4 (blue) and L from
	Theorem 3.3 (red), for z i with Gaussian (top) or Student-t with degree of freedom 7 (bot-
	tom) entries; functions f

Table 3 .

 3 2: Storage size and top eigenvector running time of K for piecewise constant and cubic f , in the setting of Figure3.11 and 3.13.

	f	Size (Mb) Running time (s)
	Piecewise 4.15	0.2390	Figure 3.12: Piecewise constant (green)
	Cubic	16.75	0.4244	versus cubic (BLUE) function with equal (a 1 , a 2 , ν).

Table 3

 3

	.5: Empirical estimation of (normalized) differences in means and covariances of
	the MNIST (Figure 3.18) and epileptic EEG (Figure 3.19) datasets.
		M T M	tt T + 2S
	MNIST data	172.4	86.0
	EEG data	1.2	182.7

Table 3

 3 ALM + 01]. The dataset consists of five subsets (denoted A-E), each containing 100 single-channel EEG segments of 23.6-sec duration. Sets A and B were collected from surface EEG recordings of five healthy volunteers, while sets C, D and E were collected from the EEG records of the pre-surgical diagnosis of five epileptic patients. Here we perform random feature-based spectral clustering on n = 32, 64 and 128 randomly picked EEG segments of length p = 100 from the dataset. Means and covariances are empir-8 http://www.meb.unibonn.de/epileptologie/science/physik/eegdata.html. Eigenvalue distribution of K c and Kc for the MNIST data (left) and leading eigenvector of K c for the MNIST and Gaussian mixture data (right) with a width of ±1 standard deviations (generated from 500 trials). With the ReLU function, p = 784, n = 128 and c 1

	.6: Classification accuracies for random feature-based spectral clustering with dif-
	ferent σ(t) on the MNIST dataset.
		σ(t)	n = 32 n = 64 n = 128
		t	85.31% 88.94% 87.30%
	mean-oriented	1 t>0 sign(t) sin(t)	86.00% 82.94% 85.56% 81.94% 83.34% 85.22% 85.31% 87.81% 87.50%
		erf(t)	86.50% 87.28% 86.59%
	cov-oriented	|t| cos(t) exp(-t 2 /2) 64.00% 60.44% 58.67% 62.81% 60.41% 57.81% 62.50% 59.56% 57.72%
	balanced	ReLU(t)	82.87% 85.72% 82.27%
	The epileptic EEG dataset 8 , developed by the University of Bonn, Germany, is de-
	scribed in [

Table 3 .

 3 7: Classification accuracies for random feature-based spectral clustering with different σ(t) on the epileptic EEG dataset. Despite the difference in data models and technical approaches, we reached a similar conclusion on the classification of different σ as in Section 3.1.2, i.e., two key parameters (d 1 , d 2 ) control separately the statistical information in means and covariance. Experiments on typical real-world datasets were performed to validate our theoretical arguments.

		σ(t)	n = 32 n = 64 n = 128
		t	71.81% 70.31% 69.58%
	mean-oriented	1 t>0 sign(t) sin(t)	65.19% 65.87% 63.47% 67.13% 64.63% 63.03% 71.94% 70.34% 68.22%
		erf(t)	69.44% 70.59% 67.70%
	cov-oriented	|t| cos(t) exp(-t 2 /2) 99.81% 99.81% 99.77% 99.69% 99.69% 99.50% 99.00% 99.38% 99.36%
	balanced	ReLU(t)	84.50% 87.91% 90.97%
	nonlinearity arising from the activation function σ(•).

  3. SPECTRAL BEHAVIOR OF LARGE KERNELS MATRICES AND NEURAL NETSwith prox κL (t) the proximal operator[START_REF] Heinz | Convex analysis and monotone operator theory in Hilbert spaces[END_REF] defined as the unique solution of the following minimization problem min

x∈R κL

  zm(z) dz.

	110	CHAPTER 4. GRADIENT DESCENT DYNAMICS IN NEURAL NETWORKS 0.1 0.2 0.3 0.4 0.5 0 Misclassification rate Simulation: training performance Theory: training performance Simulation: test performance Theory: test performance 0.1 0.2 0.3 0.4 Theory: training perform Simulation: test performa Misclassification rate Theory: test performance
		0	0	50	0	100	150	200 50	250	300 100	150	200
						Training time (t)		Training time (t)

  CT ∈ R d 1 ×r . Moreover, if S Y has distinct eigenvalues (i.e., Y has d y distinct singular values, as demanded in Assumption 12), then U is a diagonal matrix made of 1 and -1. of proof of Lemma 4.2. It can be shown with basic algebraic manipulations that the eigenvectors of S 2 Y (thus of S Y ) form a basis of both ker A and ker C T . Therefore ker A = ker C T and in particular dim ker A = dim ker C T . We denote this dimension d yr and A, C are thus both of rank r. Choose U 2 from ker A and U 1 ⊥ ker A; we deduce U = U 1 U 2 so that (4.18) holds.In the general case H ≥ 1, similar conclusion as in Lemma 4.2 holds for the matrix product (W H+1 . . . W 2 ) T and W H+1 . . .W 2 A for W 1 ≡ A B .Remark from (4.18) in Lemma 4.2 that, for arbitrary S Y , there are infinitely many possibilities on the choice of U with the risk of occupying too much of the state space X , since, with the change of variable in Lemma 4.2 the state variable now becomes the tuple (A, B, C, U). Using Assumption 12, U only takes a finite number of values for a given W ∈ S r (L), hence the state variable essentially becomes the tuple

	4.2. A GEOMETRIC APPROACH TO GDD OF LINEAR NNS	121
	with Ā, Sketch
		18)

2 )

 2 Since Z i 1 n i ∼ N (0, n i I n i ), by applying the trace lemma (Lemma 2.11), we get We start with the variable g(x)|E and check that Lyapunov's condition for ri = r i -E[r i ], As a consequence, we have the CLT for the random variable g(x)|E. Thus Ḡas n → exp(-u 2 /2).With the help of Lévy's continuity theorem, we thus prove the CLT of the variable n g(x)-Ḡa s n. Since s 2 n → V G a , with Slutsky's theorem, we have the CLT for n g(x)-Ḡa √

		y T PΩ T C a p	ΩPy -	4nc 2 1 c 2 2 p 2	tr C 1 C a c 1	+	tr C 2 C a c 2	a.s. -→0.	(A.20)
	Consider now the events
						E = y T PΩ T C a p	ΩPy -ρ <
							Ē = y T PΩ T C a p	ΩPy -ρ >
	for any fixed with ρ =	4nc 2 1 c 2 2 p 2	tr C 1 C a c 1	+ tr C 2 C a c 2	and write
	E exp iun	g(x) -Ḡa s n	= E exp iun	g(x) -Ḡa s n	E P(E)
										+ E exp iun	g(x) -Ḡa s n	Ē P( Ē)	(A.21)
		lim n→∞	1 s 4 n	n ∑ i=1	E[|r i | 4 ] = lim n→∞	n ∑ i=1	60λ 4 i + 12λ 2 i bi n s 4	2 + 3 bi	4	= 0
	since both λ E exp iun	g(x) -Ḡa s n	E → exp(-u 2 /2).
										g(x)-Ḡa
										s n
										V Ga	and
	eventually for n	g(x)-Ḡa

i and bi are of order O(n -3/2 ).

Next, we see that the second term in (A.21) goes to zero because E exp iun Ē ≤ 1 and P( Ē) → 0 from (A.20). We eventually deduce E exp iun g(x) -

  ij ≡ δ i =j z T i (E a +E b )z j 2 √ p and B ij ≡ δ i =j (JM T MJ T + JM T Z + Z T MJ T ), for k odd;

			µ T a µ b +µ T a z j +µ T b z i √ p	-	z T i (E a -E b ) 2 z j 8 √ p	, has a tractable low-
	rank approximation K given by
		k!!
	KI =	k(k-1)!! 2p	J (T + S) J T ,		for k even.	(A.23)

•(k-1) • (A + B) + k(k -1) 2 √ p (Z T Z/ √ p) •(k-2) • (A) •2 (A.22)

for A p

  • • •. This further leads, in the Gaussian case, to the expression of KI in Proposition 3.2. We then show that, for random vectors z with zero mean, unit variance and bounded moments entries, the expression of E[Φ] coincides with the Gaussian case. To this end, recall that the entries of Φ are the sum of random variables of the type R p with i.i.d. zero mean, unit variance and finite moments (uniformly on p) entries, deterministic F ∈ R p×p , C ∈ R, α ∈ N and β ∈ {1, 2}. Let us start with the case β = 1 and expand φ as

	φ = √ p) φ = C √ p (x T y/ C √ p 1 √ p p ∑ i 1 =1 x i 1 y i 1 . . . 1 √ p

II.II Computation of E[Φ]

: beyond the Gaussian case. α (x T Fy) β for independent random vectors x, y ∈ p ∑ i α =1

This subsection is extracted from our contribution[START_REF] Couillet | Classification asymptotics in the random matrix regime[END_REF].

C + ≡ {z ∈ C, [z] > 0}. We also recall that, for m(z) the Stieltjes transform of a measure µ, µ can be obtained from m(z) via µ([a, b]) = lim ↓0 1 π b a[m(x + ı )]dx for all a < b continuity points of µ. See Section 2.1.2 for more details.

e.g., the prediction risk E test (t) = E ( x, ŷ) ŷw(t) T x 2 under a regression setting on a test set ( x, ŷ) or the correct classification rate P(sign( xT w(t)) = ŷ) for an unseen new datum x (of underlying label ŷ).

Which happens to be again the line θ 1 = θ 2 in Figure1.7a.

Defined by the fact that µ(dt; z) = z T M(dt)z = ∑ ij z i z j M ij (dt) is a positive real-valued measure for all z. See[START_REF] Anatol'evich Rozanov | Stationary random processes[END_REF] for an introduction.

The Schur complement S = A -BD -1 C is particularly known for its providing the block determinant formula det A B C D = det(D) det(S).

Since data from C 1 are labeled -1 while data from C

are labeled 1.

Reproducibility: visit https://github.com/Zhenyu-LIAO/RMT4LSSVM for the Python

codes to reproduce the results in this subsection.

For mental reminder, M stands for means, T accounts for the difference in traces of covariance matrices and S for the "shapes" of the covariances.

Note that, as defined, KI has non-zero diagonal elements, while [K I ] ii = 0. This is not contradictory as the diagonal matrix diag( KI ) has vanishing norm and can thus be added without altering the approximation K I -KI → 0; it however appears convenient as it ensures that KI is low rank (while without its diagonal, KI is full rank).

Note that the notation τ defined here differs from that in Section 1.1.2 and 3.1.1 by a factor 2.

Similarly to Section 3.1.2, M stands for means, t accounts for (difference in) traces while S for the "shapes" of covariances. Note in particular that the definition of S is different from that in Section 3.1.2, since only the special case C a = I p + E a is considered in Section 3.1.2.

The network (weight) parameters W evolve through time and are considered to be state variables of the dynamical system, while the pair (X, Y) is fixed and thus referred as the "parameters" of the given system.

Here we consider for the moment f to be a Hermite polynomial, and then extend to square-summable f with Assumption 3.

By assuming first that a, b are linearly independent before extending by continuity to a, b proportional.

Acknowledgments vii

Acknowledgments

Firstly, I would like to express my sincere gratitude to my advisor Prof. Romain Couillet and co-advisor Yacine Chitour, for their continuous support of my Ph.D study and related research, for their great patience, incredible motivation, and immense knowledge. Their guidance helped me in all the three years of my Ph.D. research and in writing of this thesis. I could not imagine having better advisors and mentors than them.

Besides my advisors, I would like to thank the two rapporteurs: Prof. Julien Mairal and Prof. Stéphane Mallat, for their helpful and insightful comments on my manuscript. I would like to thank all jury members for their time and effort in coming to my defense and in evaluating my work.

Contents

List of Symbols v

Chapter 4

Gradient Descent Dynamics in Neural Networks

In Chapter 3 we discussed either kernel-based methods that are of explicit forms (in Section 3.1.1-3.2.2), or the statistical behavior of the unique minimum from a convex optimization problem (in Section 3.3), without considering by which means can we reach this optimum. In this chapter, we discuss the possibly most widely used gradient descent method, and in particular, the temporal evolution of (the performance of) the learning system when trained with gradient descent, starting from the simple and convex linear regression model in the section below.

A Random Matrix Approach to GDD

In the section we consider the gradient descent dynamics (GDDs) for the training of a ridge regression, under a binary symmetric GMM (see Definition 1) with identity covariance for the input pattern, i.e.,

C 1 :

for z i ∼ N (0, I p ) and µ = O(1) as n, p → ∞ with p/n = c → c ∈ (0, ∞).

Let us detail our basic settings: for a given training data matrix X = x 1 , . . . , x n ∈ R p×n with associated labels y = y 1 , . . . , y n ∈ R n , a weight vector w ∈ R p is learned using gradient descent to minimize the square loss

The gradient of L with respect to w is thus given by ∇ w L(w) = -1 n X(y -X T w) so that with small gradient descent step (or, learning rate) α, we obtain, by performing a continuous-time approximation, the following differential equation

the solution of which is given explicitly by (as in (1. where we denote w LS = ( 1 n XX T ) -1 1 n Xy ∈ R p the classical ridge regression solution with regularization parameter λ ≥ 0 and w 0 = w(t = 0) the initialization of gradient descent. We recall that the exponential of a symmetric matrix A is given by e A = ∑ ∞ k=0 1

A the spectral decomposition of A. We apply, as proposed in [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF], the following random initialization for gradient descent.

Assumption 9 (Random initialization). Let w 0 ≡ w(t = 0) be a random vector with i.i.d. entries of zero mean, variance σ 2 /p for some σ > 0 and finite fourth moment.

Given the expression of w(t) above, we are interested in the training and test misclassification error rate, i.e.,

Due to the symmetry of the GMM under consideration, we denote, similar to Section 3.3, xi ≡ y i x i so that xi ∼ N (µ, C) and use, with a slight abuse of notations for test data x = x. As such, we further write Xy = X1 n , XX T = X XT so that the classification error rates become

From a random matrix perspective, all aforementioned objects are functionals of the eigenvalues of the sample covariance matrix 1 n X XT and can thus be evaluated with the resolvent-based technique discussed in Chapter 2.

Main results.

Let us start with the test misclassification error rate. Since the new datum x is independent of w(t), conditioned on w(t), w(t) T x is a Gaussian random variable of mean w(t) T µ and variance w(t) 2 . The above probabilities can therefore be given via the Q-function of Gaussian distribution. We thus resort to the computation of w(t) T µ as well as w(t) T w(t) to evaluate the aforementioned classification error.

For µ T w(t), with Cauchy's integral formula we have

with f t (z) ≡ exp(-αtz) and γ a positive closed path circling around all eigenvalues of 1 n X XT . Note that the (unified) data matrix X can be rewritten as

To isolate the deterministic vectors µ from the random Z in the expression of µ T w(t), we exploit Woodbury's identity to obtain

Definition 8 (GDD). The gradient descent dynamic of L is the dynamical system defined on X by dW dt = -∇ W L(W) (4.12)

where ∇ W L(W) denotes the gradient of the loss function L with respect to W. A point W ∈ X is a (first-order) stationary point of L if and only if ∇ W L(W) = 0 and we denote S(L) the set of (first-order) stationary points.

In the following, we work directly on the equivalent equation (4.11) and start by evaluating the gradient of L. With the previous notations, for w ≡ (w H+1 , . . . , w 1 ), we expand the variation of L(W + w) as

F and the differential term given by D W (w) ≡ -∑ H+1 j=1 tr M T W H+1 . . . W j+1 w j W j-1 . . . W 1 . We thus derive from Definition 8 the dynamics of L, for j = 1, . . . , H + 1, as

Main results. We start with the global convergence to stationary points of all gradient descent trajectories. While one expects the gradient descent algorithm to converge to stationary points, this may not always be the case. Two possible (undesirable) situations are i) a trajectory is unbounded or ii) it oscillates "around" several stationary points without convergence, i.e., along an ω-limit set made of a continuum of stationary points. The property of an iterative algorithm (like gradient descent) to converge to a stationary point for any initialization is referred to as "global convergence" [START_REF] Willard I Zangwill | Convergence conditions for nonlinear programming algorithms[END_REF]. However, it is very important to stress the fact that it does not imply (contrary to what the name might suggest) convergence to a global (or good) minimum for all initializations.

To answer the convergence question, we resort to Lojasiewicz's theorem for the convergence of a gradient descent flow as (4.13) with real analytic loss L, as recalled below.

Theorem 4.3 (Lojasiewicz,from [Loj82]). Let L be a real analytic function and W(•) be a solution trajectory of the gradient system given by Definition 8 such that W(t) remains bounded for all t ≥ 0. Then W(•) converges to a stationary point of L as t → ∞, with the convergence rate determined by the associated Lojasiewicz component [START_REF] Didier | Explicit bounds for the łojasiewicz exponent in the gradient inequality for polynomials[END_REF].

Since the loss function L(W) is a polynomial of degree (H + 1) 2 in the component of W, Lojasiewicz's theorem ensures that a bounded trajectory of the gradient descent flow must converge to a stationary point with a guaranteed rate of convergence. In particular, the aforementioned phenomenon of "oscillation" cannot occur and we are left to ensure the absence of unbounded trajectories. The following lemma characterizes the "invariants" along trajectories of GDD, inspired by [START_REF] Andrew M Saxe | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[END_REF] which essentially considered the case where all dimensions are equal to one. These invariants will be used at several stages: to prove convergence to stationary points, to ensure an exponential convergence rate, as well as to help understand the basin of attractions of the (undesired) saddle points.

Lemma 4.1 (Invariant in GDD). Consider any trajectory of the gradient system given by (4.13). Then, for j = 1, . . . , H, the value of W T j+1 W j+1 -W j W T j remains constant, i.e.,

Appendix A

Mathematical Proofs

A.1 Proofs in Chapter 3

A.1.1 Intuitive calculus for Theorem 1.2

In this section we consider the empirical spectral measure of the following inner-product kernel matrix as in (1.13)

for independent random vector x i ∼ N (0, I p ) and in particular, the associated Stieltjes transform (see Definition 6) given by

Our objective here is to prove that m(z) is the (unique) solution of the following cubic equation

Basic settings and notations. Following the ideas of the sample covariance model in Section 2.2.3, we remove the i-th row and the i-th column of the symmetric matrix K such that

with zero on the diagonal of K -i and X -i ∈ R p×(n-1) the data matrix without x i . As such, we have that K -i is independent of x i . We similarly define the resolvent Q -i of K -i as

so that under the above notations, the (i, i)-th (diagonal) entry of Q is given by

(A.11) The inverse of 1 T n Q1 n can consequently be computed using a Taylor expansion around its leading order, allowing an error term of O(n -3 2 ) as

(A.12) Combing (A.9) with (A.12) we deduce

and similarly the following approximation of b as

where

which gives the asymptotic approximation of b.

1 n 1 n 1 T n = γP, and we can thus rewrite:

Here again, we use

+O(n -5 2 ).

(A.15) Note here the absence of a term of order O(n -3/2 ) in the expression of β since PA √ n P = 0 from (A.6).

We now work on the vector k(x) for a new datum x, following the same analysis as in [START_REF] Couillet | Kernel spectral clustering of large dimensional data[END_REF] for the kernel matrix K, assuming that x ∼ N (µ a , C a ) and recalling the random variables definitions,

We show that the j-th entry of k(x) can be written as

Combining (A.15) and (A.16), we deduce

At this point, note that the term of order O(n -1 2 ) in the final object g(x) = β T k(x) + b disappears because in both (A.14) and (A.17) the term of order O(n -1/2 ) is 2γ

√ p c 1 c 2 f (τ)(t 2t 1 ) but of opposite signs. Also, we see that the leading term c 2c 1 in b will remain in g(x) as stated in Remark 3.1.

The development of y T P k(x) induces many simplifications, since i) P1 n = 0 and ii) random variables as ω x and ψ in k(x), once multiplied by y T P, thanks to probabilistic averaging of independent zero-mean terms, are of smaller order and thus become negligible. We thus get

This result, together with (A.17), completes the analysis of the term β T k(x). Combining (A.17)-(A.19) with (A.14) we conclude the proof of Theorem 3.1.

A.1.3 Proof of Theorem 3.2

This section is dedicated to the proof of the central limit theorem for

with the shortcut c x = -2c 1 c 2 2 for x ∈ C 1 and c x = 2c 2 1 c 2 for x ∈ C 2 , and R, D as defined in (3.8) and (3.9).

Our objective is to show that, for a ∈ {1, 2}, n( g(x) -G a ) → 0 in distribution with

where Ḡa and V G a are given in Theorem 3.2. We recall that x = µ a + √ pω x with ω x ∼ N (0, C a /p).

Letting z x such that ω x = C 1/2 a z x / √ p, we have z x ∼ N (0, I) and we can rewrite g(x)

in the following quadratic form (of z x ) as

Since z x is standard Gaussian and has the same distribution as Uz x for any orthogonal matrix U (i.e., such that U T U = UU T = I), we choose U that diagonalizes A such that A = UΛU T , with Λ diagonal so that g(x) and g(x) have the same distribution where

and b = U T b, λ i the diagonal elements of Λ and z i the elements of z x .

Conditioning on Ω, this results in the sum of independent but not identically distributed random variables r i = z 2 i λ i + z i bi + c n . We then resort to the Lyapunov CLT [Bil12, Theorem 27.3].

We begin by estimating the expectation and the variance

We shall rewrite Ω into two blocks as:

where Z 1 ∈ R p×n 1 and Z 2 ∈ R p×n 2 with i.i.d. Gaussian entries with zero mean and unit variance.

Then

and with Py = y -(c 2c 1 )1 n , we deduce

III. Concentration of Hadamard product. It now remains to treat k(Z

and show it also has an asymptotically deterministic behavior (as Φ). It can be shown

To prove this, note that, depending on the parameter a 0 = E[ f (ξ)], the operator norm of f (Z T Z/ √ p)/ √ p is either of order O( √ p) for a 0 = 0 or O(1) for a 0 = 0. In particular, for monomials f (x) = x k under study here, we have a 0 = E[ξ k ] = 0 for k odd and a 0 = E[ξ k ] = (k -1)!! = 0 for k even, ξ ∼ N (0, 1). To control the operator norm of the Hadamard product between matrices, we introduce the following lemma.

Proof of Lemma A.1. Let e 1 , . . . , e p be the canonical basis vectors of R p , then for all 1

As a consequence, for any v = ∑ p i=1 v i e i , we obtain

which, by Cauchy-Schwarz inequality further yields ∑

A.1.5 Proof of Theorem 3.5

We aim to prove Theorem 3.5, which states that the resolvent

admits the following deterministic equivalent (see Definition 7)

with δ the unique solution of δ = 1 n tr(K Q). The proof of Theorem 3.5 generally follows that of the Marcenko-Pastur law in Section 2.2.2. We provide only the main steps as follows.

Denote the random (column) vector σ i ≡ σ(X T w i ) ∈ R n for X = O(1) and w the i-th row of W such that w ∼ N (0, I p ). By rewriting 1 n Σ T Σ as

we obtain, from the resolvent identity (Lemma 2.1) that

Working on the expectation inside the second term, we further get, with Sherman-Morrison identity (Lemma 2.8), that

where we denote

Intuitively, the second expectation above should asymptotically vanish, as a result of Lemma 3.3. To show this, let us rewrite again with Sherman-Morrison identity (Lemma 2.8)

where we denote D the diagonal matrix with its i-th entry equal to 1 n tr(

) from Lemma 3.3, imply that the matrix has asymptotically vanishing operator norm. This gives

Following similar arguments as above, we have, from the rank one update Q -

A.1.6 Computation details for Table 3.3

Here we only provide the case of the popular ReLU nonlinearity σ(t) = ReLU(t) ≡ max(t, 0). Other functions σ(•) in Table 3.3 can be treated similarly.

We first write the expectation as an integral on R p , which is then reduced to an integral on R 2 with the classical Gram-Schmidt process:

With a simple geometric representation we observe

. Therefore with a polar coordinate change of variable we deduce, for σ(t) = ReLU(t), that

with ∠(a, b) ≡ a T b a b as Table 3.3.

A.2 Proofs in Chapter 4

A.2.1 Proofs of Theorem 4.1 and 4.2

We start with the proof of Theorem 4.1 which characterizes the (asymptotic) test performance of the classifier w(t) given by

on an unseen test datum x ∼ N (µ, I p ). To this end, we shall evaluate subsequently the random variable w(t) T µ and w(t) 2 as below. Since

We thus resort to the computation of the bilinear form a T Q(z)b. By plugging in the deterministic equivalent of Q(z) ↔ Q(z) = m(z)I p we obtain the following estimations

for m(z) the unique solution of the Marčenko-Pastur equation (4.1). The above relation is a direct consequence of the fact that Z T Z and ZZ T have the same eigenvalues, except for the additional zeros eigenvalues for the larger matrix. We thus get

and the term

where we use the relations m(z) = cm(z) -1 z (1c) and (zm(z) + 1)(cm(z) + 1) = m(z) from (4.1), while the term µ T 1 n X XT -zI p -1 w 0 = O(n -1/2 ) due to the independence of w 0 with respect to Z.

Following the same arguments, we have

We now wish to replace the terms in µ T w(t) and w(t) T w(t) by their asymptotic approximations to reach an almost sure convergence of the right-hand side contour integration. Note that, both µ T w(t) and w(t) T w(t) are functionals of the resolvent 1 n XX T -zI p -1

that is only well defined for z not an eigenvalue of 1 n XX T . More generally, the aforementioned approximations can be summarized by the fact that, for a generic analytic h(z), we have, as n → ∞, h(z) -h(z) → 0

A.2. PROOFS IN CHAPTER 4

143 almost surely for all z not an eigenvalue of 1 n XX T . To reach a convergence result for γ h(z)dzγ h(z)dz → 0, we first show that there exists a probability one set Ω z on which h(z) is uniformly bounded for all large n, with a bound independent of z. Then by the "no eigenvalues outside the support" theorem (see for example [START_REF] Paul | No eigenvalues outside the support of the limiting empirical spectral distribution of a separable covariance matrix[END_REF]) we know that, with probability one, for all n, p large, no eigenvalue of 1 n ZZ T appears outside the interval

As such, the intersection set Ω = ∩ z i Ω z i for a finitely many z i , is still a probability one set. Finally by Lebesgue's dominant convergence theorem [START_REF] Billingsley | Probability and measure[END_REF], together with the analyticity of the function under consideration, we conclude the proof of Theorem 4.1. The proof of Theorem 4.2 follows exactly the same line of arguments and is thus omitted here.

A.2.2 Detailed Derivation of (4.3)-(4.6)

Our objective here is to further simplify the contour integrations obtained above, by carefully choosing the path γ. First note that, 1 n X XT , as a low rank perturbation of 1 n ZZ T such that

may have (asymptotically) one isolated eigenvalue that jumps out the support of the (limiting) Marčenko-Pastur distribution.

We first determine the location of the isolated eigenvalue λ. More concretely, we wish to find λ an eigenvalue of 1 n XX T that lies outside the support of the Marčenko-Pastur distribution (more precisely, not an eigenvalue of 1 n ZZ T ). Solving the following equation

where we recall that Q(λ) ≡ 1 n ZZ T -λI p -1 and we used the fact that det(AB) = det(A) det(B) as well as the Sylvester's determinant identity det(I p + AB) = det(I n + BA) for A, B of appropriate dimensions (Lemma 2.3). Together with (4.1) we deduce the (empirical) isolated eigenvalue λ = λ s + o(1) with

µ 2 which gives the asymptotic location of the isolated eigenvalue as n → ∞. In the following, we may thus use λ s instead of λ throughout the computation. By splitting the path γ into γ b + γ s that circles respectively around the main bulk between [λ -≡ (1 -√ c) 2 , λ + ≡ (1 + √ c) 2 ] and the isolated eigenvalue λ s , we deduce, with the residual theorem that E = E γ b + E γ s where

with m (z) the derivative of m(z) with respect to z and is obtained by taking the derivative of (4.1). We now move on to the contour integration γ b in the computation of E γ b . We follow the idea in [START_REF] Zhidong | Clt for linear spectral statistics of large-dimensional sample covariance matrices[END_REF] and choose γ b to be a rectangle with sides parallel to the axes, intersecting the real axis at 0 and λ + (in fact atand λ + + so that the functions under consideration remain analytic) and the horizontal sides being a distance ε → 0 away from the real axis. Since for nonzero x ∈ R, the limit lim z∈Z→x m(z) ≡ m(x) exists [START_REF] Jack | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF] and is given by m

with the branch of ± is determined by the imaginary part of z such that (z)