Afonso Bandeira

Alice Guionnet

Marylou Laura

Jonathan Benjamin

Stefano Bruno

Stéphane, Cédric, Hugo Federica

Jean Luca

Nico Jack

For this dissertation -First of all, I would like to thank all members of the jury

Foreword

Throughout the thesis, the works to which I contributed are cited in orange while other references are cited in green. For instance I contributed to [Mai21] but not to [START_REF] Borel | Introduction géométrique à quelques théories physiques[END_REF].

This thesis presents some of the work I did during my time as a Ph.D. student in École Normale Supérieure de Paris, between September 2018 and September 2021, under the supervision of Florent Krzakala (and very often also Lenka Zdeborová). It is divided into three parts, each taking a distinctive view on high-dimensional inference and learning problems.

Organization of the manuscript and summary of contributions

In Part I we introduce a large set of tools, mostly originated from the statistical physics of disordered systems, that are designed to study the fundamental properties of inference problems. Chapter 1 is quite introductory and pedagogical. While it does not present any novel results, it gives a detailed introduction to the statistical physics of inference problems as a whole, and introduces many concepts and definitions that will be used throughout the thesis. It also recalls some important historical connections between statistical physics, probability, learning and random matrix theory. Chapters 2 and 3 present the main results of two publications: [MFK + 21] Towards exact solution of extensive-rank matrix factorization. Antoine Maillard, Laura Foini, Florent Krzakala, Marc Mézard and Lenka Zdeborová. In preparation (2021).

The approach of these chapters is in essence an "old-school" strategy of statistical physics for disordered systems, namely the derivation of the TAP equations via high-temperature expansions.

In Chapter 2 we show how such expansions are related to modern approximations and algorithms in inference problems, while in Chapter 3 we use them to derive systematic corrections to previous incorrect approximations that aimed to solve extensive-rank matrix factorization.

Part II is divided intro three chapters, each presenting a detailed study of the fundamental limits (both information-theoretic and algorithmic) of different high-dimensional inference and learning problems, leveraging many of the tools introduced in Part I. Beyond the heuristic techniques of statistical physics, we also provide proofs of their predictions, and detailed algorithmic studies. In particular we discuss the existence of computational-to-statistical gaps (also known as hard phases) in which a problem can be solved information-theoretically but not with any polynomialtime algorithm. In Chapter 4 we discuss learning in a model of two-layers neural network known as the committee machine. It is based on the following publication:

[AMB + 19] The committee machine: Computational to statistical gaps in learning a two-layers neural network. Benjamin Aubin, Antoine Maillard, Jean Barbier, Florent Krzakala, Nicolas Macris and Lenka Zdeborová. Journal of Statistical Mechanics: Theory and Experiment 2019 (12), 124023.

iii

In Chapter 5, we show how the knowledge of data structure can be used to enhance recovery of signals in a generic inference problem known as spiked matrix estimation. It gives a detailed account of the main results of:

[ALM + 20] The spiked matrix model with generative priors. Benjamin Aubin, Bruno Loureiro, Antoine Maillard, Florent Krzakala and Lenka Zdeborová. IEEE Transactions on Information Theory, 2020.

Finally, Chapter 6 focuses on phase retrieval, a particularly relevant problem for inference and optimization, and which possesses many applications across different scientific fields. We add to our study of the fundamental limits of phase retrieval a constructive derivation of a class of optimal spectral methods. This chapter allows to put many previous results of the literature into a common framework, and its conclusions are published in the following references.

[MLKZ20] Phase retrieval in high dimensions: Statistical and computational phase transitions. Antoine Maillard, Bruno Loureiro, Florent Krzakala and Lenka Zdeborová. Advances in Neural Information Processing Systems, 33 (2020).

[MKLZ21] Construction of optimal spectral methods in phase retrieval. Antoine Maillard, Florent Krzakala, Yue M. Lu and Lenka Zdeborová. Mathematical and Scientific Machine Learning, 2021.

Part III, the last one of the thesis, takes a somehow more direct approach to the problem of learning and optimization in high dimensions. It develops a framework based on the Kac-Rice formula, an important tool of random differential geometry, to understand the topology of the high-dimensional landscapes that are optimized by learning algorithms. Chapter 7 presents a work done in collaboration with Profs. Ben Arous (NYU) and Biroli (ENS) in which we derive explicit formulas for the complexity of these landscapes, i.e. the statistics of their number of critical points.

[MBAB20] Landscape complexity for the empirical risk of generalized linear models. Antoine Maillard, Gérard Ben Arous and Giulio Biroli. Mathematical and Scientific Machine Learning 287-327, 2020.

We end this thesis in Chapter 8 with a more direct excursion into the realm of random matrix theory. We study the large deviations of the extreme eigenvalues of a large class of random matrices, with direct applications to the topological approach to high-dimensional landscapes described above. It is based on the letter: [Mai21] Large deviations of extreme eigenvalues of generalized sample covariance matrices. Antoine Maillard. EPL (Europhysics Letters) 133 (2), 20005, 2021.

Topics not covered in this dissertation

The present manuscript does not cover one of my PhD publications:

[BMMK18] The mutual information in random linear estimation beyond iid matrices. Jean Barbier, Nicolas Macris, Antoine Maillard and Florent Krzakala. 2018 IEEE International Symposium on Information Theory (ISIT), 1390-1394.

In this contribution we study linear estimation problems. We go beyond the restrictive i.i.d. matrix assumption and discuss the formula proposed by [START_REF] Takeda | Analysis of CDMA systems that are characterized by eigenvalue spectrum[END_REF] and later by [START_REF] Tulino | Support recovery with sparsely sampled free random matrices[END_REF] who used the heuristic replica method of statistical physics (that we shall encounter several times in this thesis). Using an adaptive interpolation method and random matrix theory, we prove this formula for a relevant large sub-class of rotationally invariant matrices. The techniques used in this work share similarities with the ones of [AMB + 19, MLKZ20], and we introduce most of them in Chapters 4 and 6. For this reason, their application to random linear estimation is not presented in this thesis.

Notations and abbreviations

Index of abbreviations

Mathematical notations

An important remark -Following standard notation in random matrix theory, we will often define a parameter β = 1 for real variables (and then denote K = R) and β = 2 for complex variables (with K = C). This β parameter must not be confused with the inverse temperature in statistical physics notations1 , for which we will rather use the notation η. When we consider real variables (i.e. β = 1) without any possible ambiguity, we will often remove the β and K indications.

x, x, Φ Scalar, vector, matrix. x • y or x y Dot product between x and y. x

l 2 norm of x ∈ K n , x 2 = n i=1 |x i | 2 . Q F Frobenius norm of the matrix Q, Q 2 F = i,j |Q ij | 2 . R + , R +
Set of non-negative and strictly positive reals. C + Complex numbers with strictly positive imaginary part. x = Θ(y)

Two variables of the same order, i.e. x = O(y) and y = O(x).

I n

The identity matrix of size n.

1 n
The vector in R n with all components equal to 1.

E

Expectation with respect to all involved random variables. E X,Y

Expectation with respect to X, Y only.

•

Expectation with respect to the Gibbs-Boltzmann measure. S n-1 β (R) Sphere in K n of radius x = R. N β (µ, σ 2 ) Gaussian distribution on K such that Ez = µ and E|z -µ| 2 = σ 2 . If β = 2 (complex variables), we also impose E(z -µ) 2 = 0. M + 1 (E) Set of probability measures on E. D KL (µ|ν) Kullback-Leibler divergence (or relative entropy) of µ, ν ∈ M + 1 (E). H n (K) or S n (K) Set of symmetric (K = R) or Hermitian (K = C) matrices of size n. H + n (K) or S + n (K) Positive symmetric (K = R) or Hermitian (K = C) matrices of size n. U β (n)

The (compact) group of orthogonal/unitary matrices. S ν , R ν , • • • Transforms of the probability measure ν (e.g. S ν (x) ≡ ν(dt)/(t -x)).

S Φ , R Φ , • • • Transforms of the ESD of Φ ∈ H n (K). ReLU
Rectified Linear Unit, i.e. ReLU(x) = max(0, x).

Introduction An unexpected journey

This thesis is an incursion between the theory of computation and theoretical physics. Computation theory aims at understanding the capabilities and limitations of computers and algorithms, not through the analysis of the performance of actual machines, but in the fundamental sense: it studies the mathematical structure of problems, in order to discern which problems are solvable and which are not, and to design algorithms capable of solving them in the most efficient possible way. The ever-growing impact of computation on our modern world has led computation theory, and theoretical computer science as a whole, to become a field of scientific research that can not be overlooked. Historically, the first well-known example of an actual algorithmic procedure is perhaps the one described by Euclid in Book VII of his Elements, which is still taught in elementary mathematics as the canonical way to find the greatest common divisor of two integers. The performance of Euclid's algorithm is elementary to compute for relatively small numbers, see Fig. 1, but understanding it for random or very large numbers is much more demanding. Understanding the fundamental limits of computational problems (e.g. "is it possible to find the greatest common divisor of two integers ?" -the answer is obviously positive here with Euclid's algorithm) and of the algorithms themselves (e.g. "what is the best algorithm to do so, and what is its performance?"1 ) are the two questions that guide this thesis, in the context of the inference problems described below.

Physics, on the other hand, seeks to grasp some bribes of the laws of nature, from the cosmological evolution to the structure of subatomic particles. In particular, statistical physics, the branch of physics to which this thesis belongs (and perhaps one of the least-known to the general audience), deals with the fundamental properties of systems composed of a huge number of simple individual elements, whose interaction can give rise to fascinating collective behavior.

Initially motivated by the study of gases, the field was developed in the 19 th century by the combined genius of Boltzmann, Maxwell, and Gibbs [START_REF] Clerk | Illustrations of the dynamical theory of gases[END_REF][START_REF] Boltzmann | Lectures on gas theory[END_REF][START_REF] Gibbs | Elementary principles in statistical mechanics[END_REF]. For a non-specialist audience, a statistical physicist studying computation theory might look somehow out-of-place: what could the physics of gases have to do with the fundamental limits of computational problems? This brief introduction aims at providing an intuitive description, accessible to a general audience, of this surprising association.

Entropy -A first intuition on the existence of this connection can be grasped by considering the notion of entropy. It is indeed foundational in both fields, albeit being formalized by Boltzmann in the 1870s in statistical mechanics and by Shannon in 1948 in theoretical computer science [START_REF] Shannon | A mathematical theory of communication[END_REF]. From the statistical mechanics point of view, since the numbers of elements is too large to follow the individual evolution of each one (e.g. molecules in the example of a gas), we rather describe the probability of a global configuration of the system. Denoting C the ensemble of all possible configurations, each configuration i is assigned with a probability p i of appearance.

The entropy of the system is then defined as:

S = - i∈C p i ln p i . ( 1
)
As shown by Boltzmann, the entropy represents the amount of uncertainty present in the system. To picture it, let's imagine a single molecule which has two possible configurations A and B.

If the molecule is certain to be in one of the two configurations (let's say A) then p A = 1 and p B = 0, which leads to S = 0. On the other hand, if both configurations are equiprobable, S = -(2/2) ln 1/2 = ln 2 > 0: the entropy is higher because the system is less "ordered"!

In the first half of the 20 th century, Shannon realized that eq. ( 1) was also suited to quantify the notion of information in computational problems. Taking the example of data communication, he wanted to understand if a receiver could exactly identify a message X from an observation Y that was possibly corrupted, or truncated. If possible, this allows for instance to send compressed messages much smaller than the originals without loosing any information, or to "clean" messages that were corrupted by noise. Because of the noise or compression, the observation Y obtained by the receiver can take many values {Y i }, each with probability p i . The entropy of eq. ( 1), now applied to these probabilities, is known as the Shannon entropy2 . Leveraging this definition, Shannon showed a foundational theorem: he proved that the entropy is related to the absolute mathematical limit of how much one can compress a message without loosing any information. Starting from a physical concept, we somehow described the fundamental limits of a computational problem! Based on this fundamental result, Shannon's work laid the foundations of the field of information theory.

Statistical physics of inference and learning -

The connection between statistical physics and information theory, which inspired much of this thesis, did not end at the definition of entropy. It also inherits from diverse perspectives, whose main lines can be understood from the point of view of Bayesian inference. In this general framework, the observer wishes to estimate a set of parameters x = (x 1 , • • • , x n ) from the observations of some data Y = (Y 1 , • • • , Y m ) (which might contain noise), while having some prior knowledge on the parameters (for instance we may know that they must correspond to a word). The Bayesian statistician then asks the question: "Given the data Y that I observed, what is the probability that it was generated from some given parameters (x 1 , • • • , x n )"? A key realization is that, when stated this way, many problems of Bayesian inference can be seen as statistical physics models! In this transposition, the parameters (x 1 , • • • , x n ) play the role of particles, which are interacting with each other. These interactions of the physical model are shaped by the knowledge acquired through the observation of the data Y in the original inference task. Moreover, like in statistical mechanics, the important quantities are macroscopic: when the number of parameters is very large, we care about the fraction of the message that we can recover, rather than the recovery of every single parameter x i . When stating the problem in this way, we can harness much of the power of statistical physics to tackle Bayesian inference.

Looking closer, Bayesian inference problems have a somehow unusual feature compared to classical statistical physics models. As we mentioned, the observations Y play the role of interactions between the particles x; however in practical inference procedures these observations are usually affected by some random noise: translated into our statistical physics model, this would imply a stochastic interaction between particles! While this does not make much sense in terms of gases, such models of random interactions between particles are called spin glasses in physics, and they arise in a variety of situations (we will discuss further the origin of these models and their physical description in Section 1.2).

A canonical example of this representation of a statistical estimation procedure as a spin glass model was described by Gardner and Derrida in the late 1980s [START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF], for an elementary algorithm known as the perceptron. After physicists realized that the theory of spin glassesthat was rapidly developing since the 1970s -could be leveraged to tackle many open problems in information theory, the way was open for decades of fruitful collaborations between communities of theoretical physicists, computer scientists, and probabilists. For instance, spin glasses provide an intuitive understanding of the possible hardness of inference problems. Indeed, in many cases solving an inference task translates into minimizing the energy of the corresponding spin glass model. However, because of the random interactions between particles the energy landscape of a spin glass is generally very rugged, see Fig. 2. Unfortunately, practical algorithms used to minimize the energy are local: they generally start at a random point, and then try to guess the position of the minimum, while they can only see the portion of the landscape infinitely close to their current position. A natural strategy would be to go down for as long as possible: but given the form of the landscape this surely means that the algorithm will end up in a small well, very far from the actual minimum that solves the estimation problem. Worse, the algorithm will then have no way of knowing if it found the lowest point or not! With this picture in mind, one can get an idea of why solving the original estimation problem might be computationally hard. This is just a tiny glimpse of the incredible amount of intuition offered on inference problems by the physics of spin glasses, these "unicorns" that the prolific physicist Sam Edwards was chasing 3 .

Generalization error

Computational gap

Information-theoretic optimum Algorithmic optimum Figure 3: A computational gap (or hard phase) in learning in a two-layers neural network. The full line is the theoretical optimal error, while the dashed line is the error achieved by the best-known algorithm. When the number of data divided by the number of parameters exceeds a threshold (around 7.5) all algorithms are incapable of reaching the information-theoretic optimum in reasonable (i.e. polynomial) time (see Chapter 4).

From the physics of spin glasses and their connections to theoretical computer science, we are now reaching the scientific questions that drive this thesis. Namely, this dissertation will discuss the following general question through the prism of the statistical physics of spin glasses: number of internal parameters (organized in what is called a neural network in the case of deep learning) using the gigantic amounts of data provided by our evermore inter-connected world, and are currently achieving state-of-the-art (and often super-human) performance for tasks as diverse as image classification, natural language processing, or speech recognition. In this thesis, we will be harnessing theoretical tools developed in spin glass theory to understand the fundamental limitations of a variety of learning and inference procedures in this high-dimensional limit, and we show an example of such results in Fig. 3. In a time in which the theory of learning and inference is an increasingly active and debated area of research, the variety of approaches developed in the spin glass and statistics literature allows to explore very diverse directions, and the three parts of this dissertation reflects some of these distinct perspectives. Some further reading -The book of C. Moore and S. Mertens [START_REF] Moore | The nature of computation[END_REF] proposes an amazing scientific journey through the theory of computation in general, and is a must-read for the audience interested in this field. The curious reader can also refer to the introduction of [START_REF] Abbara | Statistical mechanics of learning with correlated patterns[END_REF] (for a short introduction accessible to a general public) or the general reviews [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF][START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF] (for more technical and completes descriptions), which are all great presentations of the foundations of the field.

Part I

Statistical physics approach to inference models and neural networks

Chapter 1

The statistical physics toolbox "[. . . ] briefly, and in its most concrete form, the object of statistical methods is the reduction of data. A quantity of data, which usually by its mere bulk is incapable of entering the mind, is to be replaced by relatively few quantities which shall adequately represent the whole, or which, in other words, shall contain as much as possible, ideally the whole, of the relevant information contained in the original data." R. A. Fisher, On the Mathematical Foundations of Theoretical Statistics (1922).

Disclaimer -This first chapter defines important notions of the theory of statistical estimation.

Most importantly, we introduce several theoretical tools and concepts that mainly originated in the statistical physics literature, starting from the 1970s with the seminal works on disordered systems of Edwards & Anderson [START_REF] Samuel | Theory of spin glasses[END_REF] and Sherrington & Kirkpatrick [START_REF] Sherrington | Solvable model of a spin-glass[END_REF]. It will provide us with a diversified toolbox to tackle the fundamental properties of inference problems in high dimension. The last Section 1.5 is more mathematical in nature, and presents several results on random matrices and large deviations theory that will prove useful throughout this dissertation.

Bibliographical note -Let us start by mentioning a few important references on the statistical physics approach to disordered systems and high-dimensional inference, for the curious reader who might finish this introductory chapter with more questions than answers. [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF] is an ageless work that focuses on models of disordered systems known as spin glasses and introduces several methods used in this thesis, e.g. the replica theory or the Thouless-Anderson-Palmer equations. Several important works followed and focused on the links between the physics of spin glasses and estimation models. Let us mention a few particularly impacting ones: [START_REF] Nishimori | Statistical physics of spin glasses and information processing: an introduction[END_REF], which presents a detailed derivation of the methods, [START_REF] Mézard | Information, physics, and computation[END_REF] with a point of view perhaps more adapted to theoretical computer scientists and information theorists, or [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF] with a specific focus on the physics approach to Bayesian inference. Finally, [START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF] is a recent review of the theoretical understanding of learning in the context of neural networks, using a variety of "mean-field" techniques that originated in the statistical physics literature.

Elements of Bayesian statistical inference 1.A motivating example: classifying data

Before diving into more technical details, we first motivate the theoretical study of inference by the important example of binary classification of data.

Imagine a medical student in neuroradiology, which has seen during her long studies thousands of brain MRI images, belonging both to patients with a brain tumor and to healthy ones. One Y = + 1 Y = 1 ?

Figure 1.1: Classifying data points in R 2 among two sets, represented as red and blue points.

(Left) Linearly separable data, we draw a possible decision boundary learned by the perceptron.

(Middle) We add a yet unseen point, for which the algorithm now predicts the label Y = -1.

(Right) Non-linearly separable data, impossible to fit perfectly with the perceptron algorithm.

day she is for the first time confronted to the brain image of an undiagnosed patient, and must deduce by herself alone1 if he is affected by a tumor. How can she minimize the risk of a mistake? This problem enters the category of "binary classification", and arises in several areas of statistics and computer science. A natural and systematic idea would be to classify all the images she has seen before into two categories -cancerous and non-cancerous -and to compare the new image to all these known diagnostics. After having looked through all of them, she finds the past image which is the most similar to the new one, and then takes a decision based on the category to which this known image belongs to. While very naive, this decision process is a simple instance of the k-nearest neighbors algorithm [START_REF] Cover | Nearest neighbor pattern classification[END_REF], which solves the binary classification problem by assigning a new data point to the label of its closest known examples.

Unfortunately the nearest-neighbor approach is quite limited, especially as it is very sensitive to noisy data or to outliers: if a few cancerous pictures are very atypical they could greatly compromise the output of the procedure! To circumvent some of these limitations we describe in the following two strategies for binary classification based on models of neural networks.

The perceptron -One of the historically most important techniques used to perform binary classification of data is the perceptron algorithm, introduced by Rosenblatt in 1957 [START_REF] Rosenblatt | The perceptron, a perceiving and recognizing automaton[END_REF]. Given a set of weights W ∈ R n and a bias b ∈ R, it predicts the classification Y ∈ {-1, +1} of a data point z ∈ R n as follows:

Y = sign n i=1 W i z i + b . (1.1)
In mathematical terms, the perceptron cuts the space R n in two regions at the decision boundary, i.e. the hyperplane characterized by the equation n i=1 W i z i + b = 0. The perceptron algorithm then consists in using labeled points (i.e. a set of m values {z µ , Y µ }) to train the algorithm, that is to learn W, b such that eq. (1.1) matches "as best as possible" the vectors z µ to the labels Y µ . The hope is that it will then allow to predict the class of a new point previously unseen in the training procedure, as the algorithm will have "learned" how to differentiate the two types of data points. Importantly, the perceptron algorithm can only perfectly fit a training set {z µ , Y µ } which is linearly separable, i.e. that can be separated in two by a decision hyperplane. We summarize this discussion in Fig. 1.1. Note that the perceptron algorithm can also be adapted to solve regression tasks, in which the labels are not discrete like in the present example, but continuous: one simply replaces the sign function in eq. (1.1) by a smooth function ϕ.

Multi-layer neural networks -In order to classify data of increasing complexity, such as natural images (which will surely not be linearly separable), a variety of subsequent algorithms Figure 1.2: Classification of cats and dogs images by a fully-connected neural network. We take the training images randomly in a large set of labeled pictures (left). Note that it contains a very small number of misclassified examples (labeled in red), which we will discuss in Section 1.1.2. We use it to train a fully-connected neural network of the form of eq. (1.2) (middle). We then use eq. (1.2) with the trained weights to predict the label of an unseen image (right).

were developed. Fully-connected neural networks, also called multi-layer perceptrons, are among them. In these models, the input data z is propagated among a number L ≥ 1 of layers, adding a non-linearity ϕ l component-wise in each layer, which results in the observation:

Y = ϕ L (W L ϕ L-1 (• • • ϕ 1 (W 1 z))). (1.2)
For classifying data in two categories we simply have ϕ L : R → {-1, 1}. By use of training algorithms (discussed at a later point of this chapter) on samples of data {z µ , Y µ }, we infer the value of the weight vectors {W l }, so that eq. (1.2) is able to accurately predict the category of an unseen data point. The procedure is illustrated in Fig. 1.2, and we will study a neural network of this category in Chapter 4. While the introduction of the multi-layer perceptron dates back to the late 60s [START_REF] Minsky | Perceptron: an introduction to computational geometry[END_REF], it took many more years for a practical training algorithm, called backpropagation, to be developed [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF]. Even then, the available computing power was not sufficient to train these networks, and it was especially during the past decade that the dramatic increase of available computing power allowed to finally train networks with many layers and nodes, a strategy known as deep learning. Actual modern neural networks are actually much more refined than multi-layer perceptrons, and the growing range of their applications has led to a whole zoology of deep neural nets, such as convolutional neural networks (CNNs) which are particularly used in computer vision, or recurrent neural networks (RNNs) with many applications in natural language processing [START_REF] Lecun | Deep learning[END_REF]. A general theory of the learning procedure in deep neural networks however still appears beyond reach [START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF][START_REF] Zdeborová | Understanding deep learning is also a job for physicists[END_REF].

Fundamental limits of inference and learning -These sophisticated neural networks are just a particular class of inference procedures, which consist in extracting information (e.g. the values of the weights {W l } in a neural network, cf. eq. (1.2)) from the analysis of data samples. Such inference procedures appear all over the scientific spectrum, as they can be applied to artificial intelligence methods, to medicine (e.g. recognizing tumors by medical imaging analysis as we discussed), self-driving systems or quantitative finance, among many. In this thesis, we consider the fundamental limits of inference procedures. Our main interest will therefore be to study the following question, which essentially sums up the task given to our medical student:

How much information is it possible to extract from a given set of data ?

This question of optimal performance is in a sense two-dimensional: one can consider both a statistical version ("How much information can we extract in principle ?") and an algorithmic one ("How to extract this information? Can it be done efficiently?"), and this dissertation is concerned with both aspects. Importantly, we will consider these questions in the framework of the current "big data" age, in which computing systems have access to effectively unlimited data material, and use it to estimate a gigantic number of internal parameters.

Inference problems in high dimension and the Bayes-optimal setting

Worst-case and typical inference A large part of statistical learning theory has focused on obtaining uniform bounds for the performance achieved by various estimation methods, using tools such as the Rademacher complexity or the Vapnik-Chervonenkis dimension [START_REF] Vapnik | The nature of statistical learning theory[END_REF][START_REF] Abbara | Rademacher complexity and spin glasses: a link between the replica and statistical theories of learning[END_REF]. Although very powerful and generic, such uniform bounds can be vacuous in the presence of very rare events, or structured data distributions. Indeed one would ideally want to understand the optimal achievable performances for typical realizations of the data, which are more likely to accurately describe real-world applications. The notion of typicality is natural, as one generally has access to data generated from an unknown underlying probability distribution: atypical events would then correspond to very rare realizations of the data, which we can (quite naively) illustrate by the red labels in Fig. 1.2. This has motivated another approach, that aimed at understanding the performance of said methods in the typical case, and which will be a key setting of this dissertation.

Bayes' theorem

To study typical inference, we will take the point of view of Bayesian estimation, which originates from the works of Bayes and Laplace in the 18 th century [START_REF] Bayes | An essay towards solving a problem in the doctrine of chances[END_REF][START_REF] Simon | Mémoire sur la probabilité de causes par les évenements[END_REF]. This approach to inference is centered around Bayes' rule: when trying to infer a vector x from a set of data Y we can write P(x|Y) = P(Y|x) P(Y) P(x).

(1.

3)

The left-hand side of this equation is called the posterior distribution of x given Y. The probability P(x) is known as the prior: it quantifies the knowledge we have of x before receiving any data. Finally, P(Y|x) is the channel distribution: it describes how the data is generated given the parameters x.

Denoising a vector -To illustrate these concepts, imagine that we observe a real vector Y = X * + √ ∆Z with Z ∼ N (0, I n ): we wish to "denoise" the vector and recover X * . Our prior knowledge on X * (independently of the observation of any data) is modeled by the prior distribution P(X * ). Given a set of observations Y generated by an unknown vector X * , the probability that it was actually generated by x is:

P(x|Y) = 1 P(Y) exp -1 2∆ Y -x 2 (2π∆) n/2 P(x).
Bayesian inference consists in leveraging P(x|Y) to recover information about the unknown "ground-truth" vector X * . In the Bayesian point of view, the notion of typicality discussed above is therefore very natural: we will describe the performance of inference procedures with high probability under the distributions of X * and Y, effectively discarding the very rare events that might influence a worst-case analysis.

Bayes-optimal setting and the Nishimori identity

An important concept that we will sometimes use to simplify the theoretical analysis is Bayesoptimality:

Definition 1.1 (Bayes-optimal setting, informal)

Let us consider an inference problem in which the observer wishes to recover a ground-truth vector X * ∈ K n through a set of observations {Y µ } m µ=1 ∈ R m which depend on X * . Such a problem is said to be Bayes-optimal if she/he knows the probabilistic models used to generate the vector X * (called the prior distribution, P(x) in eq. (1.3)), and to generate the observations {Y µ } given X * (called the channel distribution, P(Y|x) in eq. (1.3)).

Therefore, in the Bayes-optimal setting the observer knows the density of the posterior distribution P(x|Y) by eq. (1.3)2 . We will see that this knowledge allows to simplify a lot the theoretical analysis of estimation procedures. Although it may not seem a realistic assumption, we will see in the analyses of Part II that many results derived under a Bayes-optimality hypothesis transfer to more realistic setups in which the underlying distribution of the data is unknown.

In the Bayes-optimal setting, one can use a very important property called the Nishimori identity. Indeed, at fixed data samples Y we will often consider independent samples x 1 , • • • , x k drawn from the posterior distribution P(x|Y) of eq. (1.3): such samples are called replicas. Physically speaking, the Nishimori identity shows that the planted solution X behaves like another replica of the system.

Proposition 1.1 (Nishimori identity)

Let (X, Y ) ∈ R n 1 × R n 2 be a couple of random variables. Let k ≥ 1 and let X (1) , . . . , X (k) be k i.i.d. samples (given Y ) from the conditional distribution P(X = • |Y ). Let us denotethe expectation operator w.r.t. P(X = • |Y ) and E the expectation w.r.t. (X, Y ). Then, for all continuous bounded function g we have E g(Y, X (1) , . . . , X (k) ) = E g(Y, X (1) , . . . , X (k-1) , X) .

Proof of Proposition 1.1 -This is a simple consequence of Bayes' formula. It is equivalent to sample the couple (X, Y ) according to its joint distribution or to sample first Y according to its marginal distribution and then to sample X conditionally to Y from its conditional distribution P(X = • |Y ). Thus the (k+1)-tuple (Y, X (1) , . . . , X (k) ) is equal in law to (Y, X (1) , . . . , X (k-1) , X). This proves the proposition.

Inference in high dimension

Modern algorithms which rule increasing parts of our lives, notably thanks to progress in deep learning, are mainly data-driven: they manage to learn a great number of parameters using comparably large numbers of data points. For instance, a classical neural network called GoogLeNet, introduced in 2015 for image classification [SLJ + 15] and which achieved at the time state-of-theart performance, possesses around 5 millions of internal parameters, and is trained on a dataset of more than one million images! Our theory will focus on such inference procedures, in which the dimension of the internal parameters learned by algorithms is very large, while the number of data points available to the algorithm is also going to infinity, and both numbers are comparable. Anticipating on the connection with statistical physics, we call this setting the thermodynamic limit: Definition 1.2 (Thermodynamic limit in inference models) Consider an inference (or learning) problem, in which one has to optimize over a set of parameters x ∈ K n . In order to do so, we are given a number m of observations, or data samples in the context of learning. We will generically study a high-dimensional limit that we call thermodynamic, in which n and m both go to ∞, but their ratio remains finite m/n → α > 0.

Generalized Linear Models

Generalized Linear Models (GLMs) are a particularly important class of supervised learning models [START_REF] Ashworth | Generalized linear models[END_REF][START_REF] Mccullagh | Generalized linear models[END_REF] that we will often take as example in the rest of Chapter 1, and that can be seen as broad generalizations of the perceptron algorithm of eq. (1.1). Their precise definition is the following: Model 1.1 (Generalized Linear Model (GLM))

Let m, n ≥ 1. We are given a data, or sensing, matrix F ∈ K m×n . Given F, data samples {Y µ } are generated as:

∀µ ∈ {1, • • • , m}, Y µ ∼ P out • 1 √ n (FX * ) µ ,
in which X * ∈ K n is the vector we will try to recover. X is drawn with i.i.d. coordinates from a prior P X , and P out is a fixed probabilistic channel. Compressive sensing corresponds to a Gaussian channel distribution P out with zero mean and variance ∆ > 0.

In this sense, GLMs generalize the usual linear regression by allowing the output function to be non-linear and possibly stochastic. They arise in many different areas of statistics, such as compressed sensing, phase retrieval [START_REF] James R Fienup | Phase retrieval algorithms: a comparison[END_REF], or logistic regression. GLMs can also be thought of as the building blocks of fully-connected neural networks [START_REF] Lecun | Deep learning[END_REF], and we refer to [BKM + 19] for a review of their numerous applications.

Gibbs-Boltzmann measure and the free entropy

Gibbs measure and posterior distribution -Let us briefly forget about inference and recall some basic elements of statistical mechanics. Consider a model of n particles x i with i.i.d. distribution P X , which interact via an energy function -or Hamiltonian -H(x) at inverse temperature η = T -1 > 0. The probability of a configuration x under the Gibbs distribution is proportional to the Boltzmann weight:

P(x) = 1 Z n exp{-ηH(x)} n i=1 P X (x i ).
(1.4)

As we will see in a more mathematical way in Section 1.5, the Gibbs-Boltzmann measure described in eq. (1.4) arises naturally as the maximal-entropy distribution at a given temperature. The factor that ensures the proper normalization of the distribution is called the partition function, while its normalized logarithm is called the free entropy3 :

Z n ≡ dx n i=1 P X (x i ) exp{-ηH(x)}, f n ≡ 1 n ln Z n .
(1.5)

Very interestingly, all these notions are naturally transposed to the realm of inference models under a Bayesian point of view. To illustrate it, let us consider a GLM in the Bayes-optimal setting (cf. Definition 1.1 and Model 1.1), so that P out and P X are known. By Bayes' rule, the posterior distribution of x reads: P(x|Y, F) = P(x)P(Y|x, F) P(Y|F)

= 1 Z n (Y, F) n i=1 P X (x i ) m µ=1 P out Y µ 1 √ n (Fx) µ . (1.6)
It is clear that the distribution of eq. (1.6) can be easily mapped to the Boltzmann distribution of eq. (1.4), with η = 1 and energy given by:

H(x) ≡ - m µ=1 ln P out Y µ 1 √ n (Fx) µ
The partition function is given by Z n (Y, F) = P(Y|F), and it will be instrumental in our asymptotic description of this problem. Note that in inference models the Gibbs distribution is a conditional distribution: it is parametrized by the observations Y (which implicitly contain the distribution of X * ) and the input data (or sensing matrix) F.

Notations -In general, we will denote • the average with respect to the posterior distribution P(x|Y), or the Gibbs distribution in statistical physics, while the symbol E will be used for average with respect to the other variables, e.g. Y, F, X * in the GLM.

Free energy and mutual information -The average free entropy of an estimation model in the random setting is closely related to the mutual information between the observations and the signal, an important notion of information theory. Let us again illustrate it in the case of the GLM:

1 n I(X * ; Y|F) ≡ 1 n E Y,X * ,F ln P(Y, X * |F) P(X * )P(Y|F) = 1 n E ln P(Y|F, X * ) Z n (Y, F) , = - 1 n E ln Z n (Y, F) +
1 n E ln P(Y|F, X * ).

(1.7)

Therefore the mutual information and the free energy are equal up to an additive constant.

Energy-entropy competition -The Gibbs measure and the free entropy quantify the competition between entropy and energy in high-dimensional models, which is parametrized by the temperature parameter. Indeed, even if the minimizers of H(x) have very low energy, their mass under the "entropic" contribution to the measure (e.g. the term i P X (x i ) in eq. (1.4)) might be extremely small. In particular, at high temperature η -1 → ∞, we expect for this reason the entropic contribution to dominate the physical state of the system. On the other hand, as T = η -1 ↓ 0 the energetic contribution prevails, and the Gibbs measure concentrates on the actual minima of the Hamiltonian H(x).

Estimators

In order to gauge the performance of a procedure, we need a definition of the quantity we try to optimize. Let us describe two common estimators, again in the case of the GLM:

• MMSE estimator -The Minimal Mean Squared Error estimator XMMSE minimizes the L 2 distance between the estimator and the ground-truth signal. As we do not have access to X * , this distance is estimated using the posterior distribution, so that (recall that ρ ≡ E P 0 [X 2 ]):

XMMSE ≡ arg min x 1 nρ E dx P(x |Y, F) x -x 2 .
(1.8)

As the MSE is a quadratic function, it is easy to see from eq. (1.8) that XMMSE = E[x|Y, F] = x , that is the marginal under the posterior distribution. The achieved MMSE is: MMSE = 1 nρ E X * -x 2 = ρ -q, (1.9) with q ≡ E{n -1 n i=1 X * i x i }. Eq. (1.9) follows from the Nishimori identity (Proposition 1.1) which gives E{ x 2 } = E{X * • x } = nq.

• MAP estimator -The Maximum A Posteriori estimator maximizes directly the posterior distribution:

XMAP ≡ arg max

x P(x|Y, F).

(1.10)

In this thesis, we will usually favor the MMSE estimator over the MAP one. A first reason is that the MMSE provides directly a way to gauge the statistical significance of an estimate.

Other reasons are the presence of overfitting in the MAP estimator in high dimensions, and the possibility that maximizing the posterior might be a very non-convex optimization problem, while the Bayes-optimal MMSE estimation is often easier to access with the tools we will develop.

For more details and the introduction of other estimators in our framework we refer to [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF].

Intuitions from the physics of spin glasses 1.2.1 Why spin glasses?

What is a spin glass ? -Spin glasses are a very peculiar class of physical states that were first observed in dilute magnetic alloys [START_REF] Daniel L Stein | Spin glasses[END_REF]. While classical magnetic materials can be classified into two main categories known as ferromagnetic (when all magnetic moments align in the same direction) or paramagnetic (when all magnetic moments rapidly evolve with no global order), spin glasses are characterized by frozen disorder. In this phase the orientation of the magnetic spins shows no global order, but it evolves very slowly so that it is effectively frozen on short time scales. In 1975, Edwards and Anderson [START_REF] Samuel | Theory of spin glasses[END_REF] introduced a statistical model of spins on the ndimensional regular lattice which was found to exhibit such a glassy phase at low temperatures, paving the way for a theory of spin glasses. This model is very similar to the celebrated Ising model, and its interactions are described by the Hamiltonian:

H EA (σ) = (i,j) J ij σ i σ j , σ i = ±1 (1.11)
where the sum runs over nearest-neighbors sites on a regular square lattice with n sites. In order to model the frozen disorder exhibited by spin glasses, Edwards and Anderson assumed that the interactions between sites were random, and took them to be Gaussian random variables

J ij i.i.d.
∼ N (J 0 /n, J 2 /n). Depending on the values of J 0 , J and the temperature T , they found that the system exhibited either paramagnetic, ferromagnetic, or glassy behavior, see Fig. 1.3.

It is important to stress that the randomness of the interactions is the critical feature that allows such models to exhibit glassy phases. In this dissertation, following a long line of theoretical work [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF], we thus usually call spin glasses statistical models with random disordered interactions.

Spin glasses and inference models -

The correspondence between estimation problems and disordered systems in high dimensions is actually very natural. To illustrate it, let us consider a flagship problem of statistics called spiked matrix estimation. The goal is to recover a signal vector X * generated uniformly in {±1} n from the observation of the "spiked" matrix

Y ≡ W + √ λX * (X * ) , (1.12)
in which W is a random "noise" matrix, which we will consider here to be a standard Gaussian i.i.d. symmetric matrix, with E W 2 ij = 1 + δ ij . The posterior probability of x is P(x = σ|Y) = P(Y|x = σ)P(x = σ) P(Y)

= 1 2 n P(Y) exp - 1 4 Tr Y - √ λσσ 2 , ∝ exp{ √ λ 2 i,j Y ij σ i σ j }.
(1.13)

The probability of eq. (1.13) is easily seen to be the Gibbs measure of a very simple disordered system at temperature T = λ -1/2 with Hamiltonian:

H n (σ) ≡ 1 2 i,j Y ij σ i σ j .
As the interactions are given by the random matrix Y, this Hamiltonian indeed describes a disordered model (note its similarity with eq. (1.11)). Therefore, studying the Gibbs measure of this long-range spin glass model with interaction matrix given by eq. (1.12) is equivalent to solving the spiked matrix estimation problem! More generally, many inference and learning tasks can be formulated as a statistical physics problem: the random interactions in spin glasses are mapped to the random quantities that parametrize the inference model (e.g. the noise matrix W in eq. (1.12) or the data matrix F in Model 1.1).

The wide extent of applications of spin glasses across the physical sciences was foreseen in [START_REF] Philip W Anderson | Spin glass vi: Spin glass as cornucopia[END_REF], an amazing two-page letter written in 1989 in which Anderson anticipated the pertinence of the rough high-dimensional landscapes of spin glasses in computer science and complexity theory. In fact, some of his interrogations on finding the ground state of the Sherrington-Kirkpatrick model (cf. Model 1.2) were only very recently answered [START_REF] Montanari | Optimization of the Sherrington-Kirkpatrick Hamiltonian[END_REF]. The general connection of disordered systems with inference and optimization problems was first made explicit by Hopfield in [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF] for a simple model of neural networks, and by E. Gardner and coauthors for the perceptron algorithm [START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF]. Since then, it has been an extremely fruitful line of research (see e.g. [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF] for a review), and is an important guideline of this thesis. While the reader will find some basics of spin glass theory in this dissertation, she/he should refer e.g. to [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF] for more details, or to [START_REF] Castellani | Spin-glass theory for pedestrians[END_REF] for a classical introduction to spin glasses aimed at newcomers to the field.

Important concepts of spin glass theory

Annealed versus quenched, and self-averaging

An important concept in spin glass theory (and in high-dimensional probability) is the distinction between annealed and quenched averages4 . It formalizes the difference between typical and average quantities for highly-fluctuating random variables. Let us illustrate this distinction first on a very simple example.

A toy example -Let X be a real random variable taking value e 3n with probability e -n , and e n with probability 1 -e -n . Because lim n→∞ P[X = e n ] = 1, the "typical" value of X is e n . Since X is in the exponential scale, it is natural to rather consider the random variable n -1 ln X.

Its expectation is n -1 E ln X = 2e -n + 1 -e -n = 1 + O n (1): this average therefore describes well the typical behavior of X in the n → ∞ limit, and it is trivial to check that the random variable n -1 ln X indeed concentrates on its mean. We call this limit the quenched average of X.

One could be tempted however to rather use EX to describe the typical behavior of X, as is common for non-fluctuating random variables. It is also exponentially large, and its leading order is given by n -1 ln EX = n -1 ln(e 3n-n + e n -1) = 2 + O n (1). This is what we call the annealed average of X, and it turns out here to be very different from the quenched one.

Where does this disparity come from? Actually, as the fluctuations of X are both very rare and very large, they dominate the naive average EX! Annealed and quenched averages are here different because the annealed average is influenced by exponentially rare events to which the quenched average is immune.

Concentration: the blessing of dimensionality -In principle, the properties of a system depend on the realization of the disorder: e.g. the free entropy n -1 ln Z n (Y, F) of the GLM in eq. (1.6) depends on Y, F. As this partition function Z n is in general a highly-fluctuating random variable which scales exponentially in n, we define the annealed and quenched free entropies as:

       f annealed ≡ lim n→∞ 1 n ln EZ n , f quenched ≡ lim n→∞ 1 n E ln Z n .
(1.14) However, in high dimensions the fluctuations of intensive quantities often vanish, so that said quantities concentrate around their mean: this phenomenon is called self-averaging in statistical physics, and is particularly relevant for spin glasses. This is in particular the case of the free entropy n -1 ln Z n of many disordered systems, which is an intensive quantity that will selfaverage around its mean. This motivates our interest in f quenched rather than f annealed , as the first one characterizes the typical behavior of the system by the concentration of the free entropy: Annealed approximation at high temperature -Note that by Jensen's inequality the annealed average of a random variable is always an upper bound of its quenched counterpart:

E ln X ≤ ln EX.

(1. 16) At high temperature the partition function Z n should not be dominated by rare events which are "smoothed out" by temperature, so that we expect the annealed average to be an informative approximation of the quenched free entropy in this regime. However it is very hard to control the accuracy of eq. (1. 16) in general, and to gauge the significance of annealed results. Their main interest lies in the relative facility of their computation: as opposed to the quenched averages, they are often exactly computable by straightforward calculations.

"Quenched" variables -Besides the "quenched-annealed" terminology introduced above for asymptotic quantities, we will also call the variables on which we average in eq. (1.14) quenched variables: these are the "frozen" variables which parametrize the Gibbs distribution, e.g. the random interactions in a spin glass model. In the context of statistical estimation, the quenched variables will encompass several quantities, such as the planted solution X , the observations Y, and the sensing matrix F in the case of the GLM, cf. Model 1.1.

Order parameters and phase transitions

The whole field of statistical physics is built around the idea that a complex system of infinitely many particles interacting with each other can be efficiently described on the macroscopic scale by a few simple quantities. Such quantities are called order parameters, and they can take many forms. For the sake of the presentation, let us focus on a non-disordered model called the Curie-Weiss model, at inverse temperature η > 0, and with a small external field h > 0 [Nis01]:

P η (σ) ≡ 1 Z n (η) exp - η n i<j σ i σ j -h n i=1 σ i .
(1.17)

In the Curie-Weiss model, as in non-disordered magnetic systems, the order parameter describing the macroscopic state of the system is the average magnetization m ≡ n -1 n i=1 σ i . Note that the Hamiltonian of eq. (1.17) favors aligned spins: the minimal energy is reached when all σ i = 1. In Fig. 1.4 we show the evolution of m with the temperature T = η -1 , clearly describing the transition between a paramagnetic (m = 0) phase at high temperature, in which the entropic contribution to the free entropy dominates, and a ferromagnetic (m > 0, cf. Fig. 1.3a) phase, in which the spins are aligned to minimize the energy. The change between the two, called a phase transition, happens at the critical temperature T c = 1.

In systems with a random disorder, such as spin glasses, the magnetization is however not a good order parameter in general. This can be intuitively understood by Fig. 1.3b: while all the local spins are "frozen" in specific directions -due to the randomness of the interactions -there is no global orientation of the spins, and the total magnetization will always be equal to zero. In this class of models, one must turn to another order parameter: the overlap5 , also called the Edwards-Anderson order parameter [START_REF] Samuel | Theory of spin glasses[END_REF]:

q EA ≡ 1 n n i=1 E[ σ i 2 ].
(1.18)

The overlap will be non-zero as soon as all local magnetizations are non-zero, even if they do not share a global direction. It will be instrumental in describing phase transitions arising in spin glasses and inference models in this thesis.

Some classical spin glass models

To conclude this rapid presentation of spin glasses let us define three important models that will be of interest in several parts in this manuscript. In order to define the first one we anticipate a bit on our introduction of random matrices, and describe a first random matrix ensemble known as the Gaussian Orthogonal (or Unitary) Ensemble.

Definition 1.3 (Gaussian Orthogonal/Unitary Ensemble)

We say that a matrix J ∈ H n (K) is drawn from the Gaussian Orthogonal Ensemble

(if K = R) or Gaussian Unitary Ensemble (if K = C), and we write J ∼ GOE(n) (resp. J ∼ GUE(n)), if we have J ij ∼ N β (0, (1 + δ ij )/n) for i ≤ j, and all {J ij } i≤j are independent.
This allows to define a spin glass model, introduced by Sherrington and Kirkpatrick in [START_REF] Sherrington | Solvable model of a spin-glass[END_REF], which played a cardinal role in the history of the field.

Model 1.2 (Sherrington-Kirkpatrick)

Let n ≥ 1 and J ∼ GOE(n). The Sherrington-Kirkpatrick Hamiltonian is defined as:

H n,J (σ) ≡ i<j J ij σ i σ j , σ ∈ {±1} n . (1.19)
Another important class of spin glasses that has received tremendous attention since the 1990s is the class of p-spin models. The simplest instance of this class is known as the pure p-spin, and we define it here on the high-dimensional sphere.

Model 1.3 (Pure spherical p-spin)

Let n ≥ 1 and p ≥ 2. The pure p-spin model Hamiltonian is defined as:

H n,p (σ) ≡ 1 n p-1 2 i 1 ,••• ,ip J i 1 ,••• ,ip σ i 1 • • • σ ip , σ ∈ S n-1 ( √ n), (1.20) with J i 1 ,••• ,ip i.i.d.
∼ N (0, 1). H n,p is therefore a Gaussian random field with zero mean and covariance

E[H n,p (σ)H n,p (σ )] = n(σ • σ /n) p .
In mathematical terms, the spherical pure p-spin corresponds to the simplest example of a random function: a homogeneous polynomial of degree p with Gaussian coefficients. We can generalize the model further, to what is known as the mixed p-spin.

Model 1.4 (Mixed spherical p-spin)

Let n ≥ 1 and a real sequence (c p ) p≥2 with p≥2 c 2 p < ∞. The mixed p-spin model Hamiltonian is defined as:

H n (σ) ≡ ∞ p=2 c p H n,p (σ), σ ∈ S n-1 ( √ n). (1.21)
Equivalently, H n is a Gaussian random field with zero mean and covariance

E[H n (σ)H n (σ )] = nξ(σ • σ /n), with the mixture function ξ(t) ≡ ∞ p=2 c 2 p t p .
By a classical result of Schoenberg [START_REF] Ij Schoenberg | Positive definite functions on spheres[END_REF], the mixed p-spin models span all stationary isotropic Gaussian random fields on the sphere S n-1 ( √ n). Note that Models 1.3 and 1.4 can easily be generalized to a variety of prior distributions different from the spherical one, e.g. one can consider Ising spins on the hypercube σ ∈ {±1} n , as in the SK model 1.2.

Static approximations to the free energy

Replica theory and replica symmetry breaking

As we emphasized in Section 1.2, one of our most important tasks will be to compute quenched averages, which (as opposed to annealed ones) are representative of the typical behavior of a high-dimensional system. Unfortunately, quenched quantities are in general much harder to compute than their annealed counterparts, and the replica method was developed precisely to tackle this difficulty. Its first application to disordered systems goes back to Edwards & Anderson [START_REF] Samuel | Theory of spin glasses[END_REF], and since then it has achieved tremendous success in the study of spin glasses, but also of inference problems. More precise introductions to the beautiful field of replica theory can be found in [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF][START_REF] Nishimori | Statistical physics of spin glasses and information processing: an introduction[END_REF][START_REF] Castellani | Spin-glass theory for pedestrians[END_REF][START_REF] Mézard | Information, physics, and computation[END_REF].

For the sake of the presentation, let us focus on the Sherrington-Kirkpatrick Hamiltonian (Model 1.2). In the context of inference models the calculations are similar, and we refer the reader to Appendix B which precisely details two replica computations in high-dimensional estimation. As we detailed above, in order to characterize the typical behavior of the model, our goal will be to compute the quenched free energy at inverse temperature η > 0:

Φ(η) ≡ lim n→∞ 1 n E J ln Z n , with Z n ≡ x∈{±1} n exp - η √ n i,j J ij x i x j . (1.22)
Note that we rescaled the interaction variables J ij by a factor √ n so that

EJ 2 ij = 1 + δ ij .

The replica trick to compute quenched averages

The difficulty in computing the quenched average in eq. (1.22) is the expectation of the logarithm of a highly-fluctuating quantity. In [START_REF] Samuel | Theory of spin glasses[END_REF], Edwards & Anderson proposed an informal way, now known as the replica trick, to compute such averages. It is based on the following identity:

E J ln Z n = lim p↓0 1 p ln E J [Z p n ]. (1.23)
While such an identity is a priori correct, the hope of the replica method is to compute the RHS of eq. (1.23) for integer p (i.e. the moments of Z n ) and to analytically expand this expression for arbitrary p > 0, before taking the limit p ↓ 0. The actual replica method relies on an additional heuristic inversion of the p ↓ 0 and n → ∞ limits, so that we will refine eq. (1.23) by

Φ(η) ≡ lim n→∞ 1 n E J ln Z n = lim p↓0 lim n→∞ 1 np ln E J [Z p n ]. (1.24)
Let us apply this identity to the SK model. We obtain by direct Gaussian integration:

1 n ln E J [Z p n ] = 1 n ln i≤j dJ ij e -1 4 i,j J 2 ij (2π) n(n-1) 4 (4π) n/2 p a=1 x a ∈{±1} n exp - η 2 √ n p a=1 i,j J ij x a i x a j , = 1 n ln p a=1 x a ∈{±1} n exp n η 2 4 1≤a,b≤p 1 n n i=1 x a i x b i 2 .
(1.25)

Let us denote Q ∈ S p (R) the overlap matrix that appears in eq. (1.25):

Q ab ≡ 1 n n i=1 x a i x b i . (1.26)
Note in particular that Q aa = 1. Introducing Q in eq. (1.25) and using the exponential representation of the delta function we reach:

1 n ln E J [Z p n ] = (1.27) 1 n ln 1≤a,b≤p dQ ab d Qab e n 2 a,b η 2 2 (Q ab ) 2 +Q ab Qab x 1 ,••• ,x p =±1 e -1 2 a,b Qab x a x b n + O n (1).

Replica symmetric assumption

When looking at eq. (1.27), one can make wo important remarks:

• In eq. (1.27) we can apply, as n → ∞, Laplace's method on the variables {Q ab , Qab }. Indeed, the number of these variables is O n (1).

• The application of Laplace's method tells us that the overlaps Q ab are critical in characterizing the thermodynamical state of the system. For this reason these variables are called order parameters, see Section 1.2.2.

Nevertheless, Laplace's principle in eq. (1.27) is quite hard to formulate. Recall indeed that, in order to apply the replica method, we need the LHS of eq. (1.27) as an analytical function of p! This leads us to rely on physical arguments to deduce the correct form of Q, Q at the solution of Laplace's method. When confronted with this problem the first solution that comes to mind is very natural: as all replicas should be equivalent, the matrix Q at the solution should have a replica-symmetric form

Q ab = q 0 + (1 -q 0 )δ ab .
(1.28)

We assume a similar form of Qab = -q 0 (1 -δ ab ). This was the assumption of Sherrington & Kirkpatrick [START_REF] Sherrington | Solvable model of a spin-glass[END_REF] who proposed a first solution to this model. After some straightforward calculations 6 we obtain from eq. (1.27) an analytic expression in p:

1 n ln E J [Z p n ] = η 2 p 4 + η 2 p(p -1) 4 q 2 0 - p(p -1) 4 q 0 q0 - pq 0 2 + Dξ 2 cosh( q0 ξ) p + O n (1).
6 Using in particular the identity e x 2 /2 = Dξ e ξx .

Recall that Dξ is the standard Gaussian law N (0, 1). Applying then the replica method in the form of eq. (1.24) we reach finally the replica-symmetric free entropy for the SK model:

Φ = η 2 (1 -q 2 0 ) 4 + q 0 q0 4 - q0 2 + Dξ ln 2 cosh( q0 ξ) . (1.29)
In this equation one should extremize with respect to the set of variables (q 0 , q0 ). We can make several remarks from eq. (1.29) (details can be found in [START_REF] Sherrington | Solvable model of a spin-glass[END_REF] or [START_REF] Nishimori | Statistical physics of spin glasses and information processing: an introduction[END_REF]):

• There exists a transition point η c = 1. For η ≤ η c , the solution that maximizes eq. (1.29) is q 0 = 0, while q 0 > 0 for η > η c . Recall that from eq. (1.26) one can easily see that q 0 = n -1 i x i 2 is the Edwards-Anderson order parameter. This implies that at T c = η -1 c = 1 there is a transition from a paramagnetic phase (in which q 0 = 0) to a spin glass phase (in which q 0 > 0). Moreover, in both phases the average magnetization m ≡ n -1 i x i = 0.

• However very puzzling behaviors happen when T = η -1 approaches zero. Let us cite [START_REF] Sherrington | Solvable model of a spin-glass[END_REF]: "The entropy S [...] goes to a negative limit at T = 0. We speculate that this unphysical behavior has its origin in the interchange of limits n → ∞ and p ↓ 0, but that the consequences are confined to low temperatures."

This second remark implies that one of our assumptions breaks down at sufficiently low temperatures. However the intuition of [START_REF] Sherrington | Solvable model of a spin-glass[END_REF] was not really exact: what happens actually is that at T < T c the replica-symmetric solution of eq. (1.28) is not stable in overlap space when applying Laplace's method in eq. (1.27), so that the actual maximum is reached in a point that breaks the symmetry between the replicas...

A word on replica symmetry breaking

In a series of beautiful papers [Par79, Par80a, Par80b], Giorgio Parisi proposed with a spark of genius the solution to the issue mentioned above. What he understood was that the overlap q between two replicas of the system with the same interactions, i.e. the order parameter of the system, could have a very non-trivial behavior when n → ∞. More precisely, its probability distribution P (q) is in general involved, and can not be characterized by a single q 0 as we assumed in the replica-symmetric ansatz. Without diving into details (which the reader will find e.g. in [START_REF] Castellani | Spin-glass theory for pedestrians[END_REF] or [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF]), one brilliant realization was the mapping of a generic P (q) to a point in replica space, i.e. a Q ab that generalized the replica-symmetric form of eq. (1.28).

First of all, note that the replica-symmetric ansatz corresponds to a distribution P (q) = δ(q-q 0 ): in probabilistic terms, replica symmetry is equivalent to the concentration (or self-averaging) of the overlap, an important intuition to keep in mind at several points of this thesis. In the SK model detailed above the overlap is not concentrating for T < T c , and the RS solution has to be discarded 7 . The next step is to consider a P (q) made of two delta peaks. One can show that in the limit p ↓ 0, it corresponds to the following overlap matrix:

P (q) = xδ(q -q 0 ) + (1 -x)δ(q -q 1 ) ⇐⇒ Q =              1 q 1 q 1 q 1 1 q 1 • • • q 0 • • • q 1 q 1 1 . . . 1 q 1 q 1 • • • q 0 • • • q 1 1 q 1 q 1 q 1 1              . (1.30) 0 1 q P(q) (q q 0 ) RS 0 1 q k a = 0 p a (q q a ) k-RSB 0 1 q P(q)
Full RSB Figure 1.5: The different forms of the Parisi measure P (q) corresponding to replica symmetry (left), k-th level of replica symmetry breaking (center), and full replica symmetry breaking (right).

The inner blocks with off-diagonal term q 1 have size x ∈ {1, • • • , p} (here we show x = 3). However (another oddness of the replica method which contradicts all usual mathematical intuitions) since we eventually take the limit p ↓ 0, the parameter x will lie in the interval [0, 1] so that the parameterization of P (q) above is well-defined. Going further, one can iterate the hierarchical structure of the overlap matrix in eq. (1.30) to create an arbitrary number of delta peaks in P (q), called the replica symmetry breaking (RSB) level (e.g. the distribution of eq. (1.30) is 1-RSB). When k → ∞, we say that the system is in a full replica symmetry breaking (FRSB) phase, and the support of P (q) is continuous. We illustrate the different possible shapes of P (q) depending on the level of RSB in Fig. 1.5. In the SK model we considered above, it was shown by Parisi [START_REF] Parisi | Infinite number of order parameters for spin-glasses[END_REF] that for η ≥ η c the system is in a FRSB phase, and that the overlap distribution P (q) (also called the Parisi measure) has a continuous support [0, q * (η)]. The FRSB picture solved the negative-entropy problem, and after decades of mathematical works the Parisi solution for the SK model was eventually proven by Talagrand [START_REF] Talagrand | The Parisi formula[END_REF], putting on firm ground one of the most important predictions of the physics of disordered systems.

RSB and the form of the Gibbs measure -Roughly speaking, 1-RSB corresponds to a space of local minima of the free energy which is organized into exponentially many clusters. Inside each cluster two solutions typically have overlap q 1 , while solutions belonging to two different clusters have a typical overlap q 0 . This hierarchy can be iterated inside each cluster, which gives rise to the 2-RSB structure. Iterating even further, the level of RSB corresponds to the depth of this hierarchical structure of clusters, which is known as an ultrametric structure. Ultrametricity and RSB is an incredibly beautiful mathematical representation of the free energy landscape of spin glass models, which also allows to create efficient algorithms, something which was only understood very recently [AMS20, AM20, Sub21, Mon21]. Unfortunately, a precise description would be beyond the scope of this dissertation (and would surely require an additional chapter), and we refer instead to [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF] for more details on this subject.

Final remarks

Our discussion of replica theory was very brief, as our main goal was to introduce the ideas behind the method, without diving too much into its physical consequences, which are extensively discussed in the literature mentioned above. To conclude this part, as we focused a lot on spin glasses, let us make two remarks on the application to estimation models.

Application to inference problems -In high-dimensional Bayesian inference (e.g. in the Generalized Linear Model, cf. Section 1.1), the role of the quenched interactions J ij is played by the data samples (the matrix F in the GLM). Leveraging on this analogy we can apply the replica method to these models to compute the quenched free entropy, characteristic of typical realizations of the data. As in spin glasses, the order parameter governing the state of the system in the high-dimensional limit is the overlap q between two replicas, to which we must add the overlap m between a replica and the planted solution X . As we saw in Section 1.1 this overlap is related to the asymptotic MMSE. Historically, the first applications of the replica method to the optimal performance in inference models date back to studies of the perceptron [START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF][START_REF] Györgyi | First-order transition to perfect generalization in a neural network with binary synapses[END_REF][START_REF] Sompolinsky | Learning from examples in large neural networks[END_REF]. The reader interested in more concrete applications of the replica method to inference problems should refer to Appendix B in which we detail two replica calculations, in a model of neural networks and in phase retrieval.

No replica symmetry breaking in Bayes-optimal problems -An important property that we will use several times throughout this thesis is that in Bayes-optimal inference (see Section 1.1), replica symmetry is never broken. This was shown in several ways and can intuitively be understood as follows. By Bayes-optimality the planted signal X * can just be seen as an additional replica (that we will index as x 0 = X ), which, combined with the Nishimori identity, implies that the couple of replicas (0, a) and (a, b) should be equivalent for all a, b, i.e. the system is replica-symmetric. For more details on this argument, we refer the reader to [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF].

Thouless-Anderson-Palmer approach

Pure states and TAP free energy

We present here a derivation, first performed by Thouless, Anderson and Palmer [START_REF] David J Thouless | Solution of 'solvable model of a spin glass[END_REF] for the SK model, of what is now known as the TAP free energy. Note that the replica method described in Section 1.3.1 only allows to access the average free energy. However at finite temperature we would also like to access properties of a specific instance of the system: we will achieve this feat by characterizing the position of what is known as pure states.

Indeed, Thouless Anderson and Palmer [START_REF] David J Thouless | Solution of 'solvable model of a spin glass[END_REF] understood that at a given η ≥ 0 the Gibbs measure typically concentrates all its mass on a large number of small regions of the landscape. These regions are characterized by their barycenter (also called magnetization) m ∈ R n , and they are known as pure states. When the temperature approaches zero, these pure states approach the global minima of the original Hamiltonian. To summarize, we can decompose the mean of every observable O at inverse temperature η as:

O = α w α O α , (1.31)
in which • α is the average over a single pure state, and w α are the weights of each pure state under the Gibbs measure. We illustrate this in an informal way in Fig. 1.6. Note that this description has very recently partially been put on rigorous ground for spin glass models in the mathematics literature [START_REF] Zhou Fan | TAP free energy, spin glasses and variational inference[END_REF][START_REF] Chen | The generalized TAP free energy II[END_REF].

We wish to build a free entropy corresponding to the decomposition into pure states, focusing on the SK model for introducing the method. Stated differently, what we want is a function of the local magnetization m, whose local maxima are the pure states of the system. In order to achieve this, we will tilt the original Gibbs measure of the system (see eq. (1.22)) in order to constraint

x i = m i .
The idea is that the free entropy associated to this constrained Gibbs measure will be maximal when m belongs to the pure states on which the Gibbs measure decomposes. This function is what we call the Thouless-Anderson-Palmer (TAP) free entropy. Introducing Lagrange multipliers {λ i } to enforce the constraints x i = m i , it reads:

Φ TAP (η, m) ≡ 1 n n i=1 λ i m i + 1 n ln x∈{±1} n exp - n i=1 λ i x i - η 2 √ n i,j J ij x i x j , (1.32)
in which we implicitly extremize over the {λ i }. In particular, when m is a global maximum of Φ TAP , Φ TAP (m) should be the actual free entropy of the system, that we can compute e.g. with the replica method, cf. Section 1.3.1. The behavior of the TAP free energy landscape gave For z high enough (high energy, or high temperature by the microcanonical-canonical equivalence in the thermodynamic limit) the slices are well-defined by a single "pure state". As the energy decreases the slices get disconnected, and for low enough energy they are concentrated around the location of the deep minima of f (x, y). This low-dimensional view is merely an intuition and should not be considered as anything more.

rise to a rich literature, both from the physics and mathematics perspective, and we refer to Chapter 2 for more details on the TAP approach.

The TAP free entropy of eq. (1.32) can be computed in different ways: the perhaps simplest route is through a low-η expansion, introduced by Plefka [Ple82] and later made systematic in [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF]. We will make extensive use of these expansions in Chapters 2 and 3, so that we do not detail their application to the SK model and leave the derivation of the following equation to the reader:

Φ TAP (η, m) = - 1 n n i=1 1 + m i 2 ln 1 + m i 2 + 1 -m i 2 ln 1 -m i 2 + η 2n 3/2 i,j J ij m i m j + η 2 4n 2 i,j J 2 ij (1 -m 2 i )(1 -m 2 j ) + O n (1). (1.33)
Let us briefly discuss the different terms of eq. (1.33) and their physical consequences.

First order: naive mean-field

Truncating the series of eq. (1.33) at order 1 in η yields what is known as the mean-field approximation:

Φ MF (η, m) = - 1 n n i=1 1 + m i 2 ln 1 + m i 2 + 1 -m i 2 ln 1 -m i 2 + η 2n 3/2 i,j J ij m i m j .
(1.34)

It is instructive to consider the maximization equations of this mean-field free entropy. They read

m i = tanh η n j=1 J ij m j / √ n , that is x i = tanh η n j=1 J ij x j / √ n .
However, the actual exact equations that one can easily derive from eq. (1.32) read x i = tanh η n j=1 J ij x j / √ n . We therefore see quite clearly the approximation: it neglects the fluctuations of the effective field j J ij x j felt by the spin x i , and replaces it with its mean (hence the name "mean-field"). Mean-field as a variational approximation -The mean-field approximation can be recovered by a different avenue, namely by a variational principle. Indeed, let us denote µ η the Gibbs measure at inverse temperature η ≥ 0. Then minimizing the Kullback-Leibler divergence D KL (ν|µ η ) can be tractable provided that we minimize it on an appropriate set of measures. The mean-field approximation neglects correlations between variables, and therefore corresponds to minimizing over the set of factorized measures, parametrized by their mean m i :

dν dx = n i=1 m i δ(x -1) + (1 -m i )δ(x + 1) .
For the detailed variational derivation of the mean-field approximation, and how it yields back eq. (1.34), one can refer to [START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF].

Higher orders and Onsager correction

The naive mean-field approximation can be exact provided the interactions are long-ranged and weak enough, e.g. in the Curie-Weiss model. While the SK model 1.2 is long-ranged, its interactions are not weak enough, so that the actual TAP free entropy of eq. (1.33) contains an additional term with respect to the mean-field, which is of order η 2 and is called the Onsager reaction term. Let us write the complete maximization equations of eq. (1.33) (called the TAP equations), that one has to solve in order to find the pure states of the system. For use in a further argument, we also add an external field i h i m i to the Hamiltonian, which does not affect the free entropy of eq. (1.33) beyond the mean-field term. In the end, the TAP equations read:

m i = tanh η √ n n j=1 J ij m j + ηh i - η 2 n n j=1 J 2 ij m 2 j m i . (1.35)
The last term in the RHS of eq. (1.35) is a reaction term: indeed, we saw that the site i affects its neighboring sites j by an effective field J ij m j . This effective field therefore modifies the value of m j by δm j = χ jj J ij m i , in which χ jj ≡ ∂m j /∂h j is the susceptibility of the site j. By eq. (1.35) we have χ jj = η(1 -m 2 j ), so that m j is modified by an amount δm j = ηJ ij m i (1 -m 2 j ). The Onsager reaction term then arises to cancel the retro-action of this modified value of m j on m i . Indeed, by our arguments, the field sent by δm j in the direction of m i is equal to

J ij δm j = ηJ 2 ij (1 -m j ) 2 m i .
Summing these corrections over all the neighbors of i yields the Onsager reaction term of eq. (1.35). Roughly speaking, this term amounts to remove the effect of the retro-action of m i on itself, which was wrongfully taken into account in the mean-field approximation.

Orders k ≥ 3 -Note that in the TAP free entropy of the SK model, cf. eq. (1.33), there is no contributions of terms of order η k with k ≥ 3. However, in spin glass or inference models with non-Gaussian interactions one needs in general to compute the whole perturbation series at any order in η, which complicates drastically the reaction terms. We will analyze such models in Chapter 2, and in particular we will show how to compute these reaction terms using the free cumulants of the interaction matrix, an important object of random matrix theory defined in the following Section 1.5.

From physics to algorithms

In Section 1.3 we focused on different approaches to compute the free energy (or free entropy), or to characterize the pure states of disordered systems. This leaves unanswered many important questions: if we are given a practical instance of the problem, are there efficient ways to find an approximate global minimum of the Hamiltonian? Can we sample in polynomial time from the Gibbs-Boltzmann distribution? Can we compute the asymptotic free entropy? All these considerations fall within the scope of algorithmic studies.

In this part we present a class of algorithms, known under the umbrella of message-passing, and which will allow us to study optimal performances in a variety of high-dimensional estimation problems.

A note on gradient-descent algorithms -While they will not be investigated directly here, let us mention another popular class of procedures that are also related to the physics point of view. The perhaps most natural approach to design an algorithm minimizing an energy function is to consider a local gradient-descent algorithm (or one of its refinements -e.g. stochastic gradient descent), a strategy which is naturally transposed to inference problems. For instance, in supervised learning a classical procedure is empirical risk minimization (ERM): we wish to learn a predictor function h(x) : X → Y, and we are given a set of input-output data points

{x µ , y µ } m µ=1 ∈ (X × Y) m .
In ERM one chooses a loss function l : Y × Y → R and minimizes the empirical risk over the given space of predictors h:

Rm (h, {x µ , y µ }) ≡ 1 m m µ=1 l(y µ , h(x µ )).
(1.36)

Empirical risk minimization with local gradient-descent algorithms has known tremendous success in the past decade, so that these methods are now widely used in the field of machine learning [START_REF] Bottou | Stochastic learning[END_REF], and they are a major ingredient of the current success of deep neural networks [START_REF] Zdeborová | Understanding deep learning is also a job for physicists[END_REF]. From the physics point of view, such methods are related to the celebrated Langevin dynamics, whose performances have been analyzed for spherical spin glasses [START_REF] Leticia | Analytical solution of the offequilibrium dynamics of a long-range spin-glass model[END_REF][START_REF] Ben Arous | Cugliandolo-Kurchan equations for dynamics of spin-glasses[END_REF]. This analysis has recently been generalized to the context of inference, for a class of spiked matrix-tensor spherical models [SMBC + 19, SMKUZ19, SMBC + 20b], and the construction of a theoretical framework for tracking the performance of gradient-based algorithms in learning is a very dynamic line of research [GAS + 20, SMBC + 20a, MKUZ20, MUZ21]. Finally, gradient-based optimization will motivate the topological analysis undertook in Part III of this dissertation.

The Generalized Linear Model -For the sake of the presentation, we will focus in Sections 1.4.1 and 1.4.2 on the Generalized Linear Model (Model 1.1), in which the sensing matrix F is generated i.i.d. from the Gaussian distribution N β (0, 1). For simplicity we will also consider the real case β = 1, while the generalization to the complex case is straightforward, see e.g. [START_REF] Schülke | Statistical physics of linear and bilinear inference problems[END_REF]. The derivation of BP and AMP in this context is quite classic, and the reader can also find them e.g. in [KMS + 12, Gab20].

Belief propagation (BP)

Assume we have access to a high-dimensional probability distribution P (x) with x ∈ X n . We would like to compute some important quantities related to this distribution (e.g. the free entropy, as we did in Section 1.3), or to be able to sample in a reasonable computational time. A naive approach of exhausting configurations would require an exponential number of operations |X | n , which makes it intractable in practice. Belief propagation (BP) was introduced precisely to answer these questions in a tractable time -i.e. polynomial in the dimension n. Before diving into the details, let us mention that Chapter D of [START_REF] Mézard | Information, physics, and computation[END_REF] is a very complete introduction to the belief propagation equations that the reader interested in these topics should read.

Factor graph representation

A particularly useful tool to represent high-dimensional probability distributions is factor graphs: they are undirected bipartite graphs with two types of nodes, variable and factor nodes. To fix the ideas, let us consider such a graph G with a set V of variable nodes, F of factor nodes, and let E be the set of edges. Each variable node i ∈ V represents one of the variables to infer, while each factor node a ∈ F represents one of the interactions present in the probability distribution.

x 1 P 0 (x 1 )

x 2 P 0 (x 2 ) • • • • • • x n P 0 (x n ) P out [Y µ |(Fx) µ / √ n] µ = 1, • • • , m (a) Factor graph of a Generalized
Linear Model, cf. eq. (1.6).

x i a b ∈ ∂i\{a} As the graph is bipartite, edges only connect variable and factor nodes, so that the neighbors ∂a ⊆ V of the factor node a are all the variable nodes arising in the interaction term represented by a. All in all, if we denote ψ a (x ∂a ) the interaction represented by a factor node a, a generic factor graph represents the probability distribution

mt b→i (x i ) m t+1 i→a (x i ) (b) Update of a message mi→a. x i ψ a (x ∂a ) m t j→a (x j ) mt a→i (x i ) j ∈ ∂a\{i} (c) Update of a message ma→i.
P (x) = 1 Z n a∈F ψ a (x ∂a ).
(1.37)

Notation -Throughout this description we will generically use the notations x A for A ⊆ V to designate the subset of variables {x i } i∈A .

Conversely, when given an arbitrary probability distribution, one can write it in the form of eq. (1.37), which allows then to draw the corresponding factor graph. As an example, the factor graph representation of the posterior distribution of the GLM (eq. (1.6)) is given in Fig. 1.7a.

Belief propagation equations

Let us consider G a tree factor graph (i.e. with no loops of any size). The Belief Propagation (BP) algorithm is an inference procedure that aims as computing marginals of the underlying distribution P (x), cf. eq (1.37). As we saw, these marginals characterize the MMSE estimator in Bayes-optimal inference problems, see Section 1.1. The BP algorithm uses a set of auxiliary distributions (called messages), one for each edge of the graph, that are propagated using precise update rules. Roughly speaking, a message (e.g. ma→i (x i ), as depicted in Fig. 1.7) communicates the belief (hence the name of the procedure) that a variable node i takes value x i , based on all the nodes already visited along the tree. More precisely, the belief propagation update rule is the following8 :

           mt a→i (x i ) = 1 Z a→i x ∂a\i ψ a (x ∂a ) j∈∂a\i m t j→a (x j ), m t+1 i→a (x i ) = 1 Z i→a b∈∂j\a mt b→i (x i ).
(1.38)

i = x * i } in the factor graph ; U ← U ∪ {i} end Return x ≡ (x * 1 , • • • , x * n );
The BP updates of eq. (1.38) are represented in Fig. 1.7b and 1.7c. Note that the sum over configurations naturally becomes an integral in the continuous setting. On tree factor graphs, the BP equations provably describe the probability distribution P (x):

Theorem 1.2 (Exactness of BP on trees [Pea82, MM09])

If the factor graph G is a tree with maximal distance t * between two variable nodes, then the BP iterations (1.38) provably converge to a stationary point in at most t * iterations, whatever the initialization. This stationary point is unique, and is an exact representation of the probability distribution P (x) (e.g. for sampling, marginalization and free entropy computation, see below).

The idea behind Theorem 1.2 is that in the updates of eq. (1.38), all the incoming messages are independent from each other (i.e. all the nodes on the top of Figs. 1.7b,c are not connected to each other). In particular, if we remove the node a in Fig. 1.7c, then all the {x j } j∈∂a become independent under the new factor graph.

Application to the GLM -Even though the factor graph of Fig. 1.7a is clearly not a tree, let us forget this for a moment and apply the BP update (1.38) to the GLM. It yields the following iterative equations:

             mt µ→i (x i ) = 1 Z µ→i j( =i) dx j m t j→µ (x j ) P out y µ 1 √ n n k=1 F µk x k , m t+1 i→µ (x i ) = 1 Z i→µ P 0 (x i ) ν( =µ) mt ν→i (x i ).
(1.39)

Three applications of the BP algorithm

The belief propagation updates of eq. (1.38) define an iterative algorithm able to solve three important tasks for tree graphical models.

(i) Marginalization -At their converging point, the BP iterations allow to compute efficiently the marginal distributions of each variable x i as:

m i (x i ) = 1 Z i a∈∂i ma→i (x i ).
(1.40)

(ii) Sampling -The BP algorithm also allows to sample from the probability distribution of eq. (1.37), leveraging the computation of the marginals for a fixed set of variables x U with U ⊆ V . This is summarized in Algorithm 1.

(iii) The Bethe free entropy -Finally, the BP messages can also be used to compute the free entropy of the model (i.e. the average of the log-normalization factor in eq. (1.37)), as a function of the messages. This function is called the Bethe free entropy and is expressed as (see [START_REF] Mézard | Information, physics, and computation[END_REF] for the derivation):

Φ Bethe ≡ i∈V ln x i a∈∂i ma→i (x i ) + a∈F ln x ∂a ψ a (x ∂a ) i∈∂a m i→a (x i ) - (i,µ)∈E ln x i m i→a (x i ) ma→i (x i ) .
(1.41)

Loopy Belief Propagation

As we saw, the BP iterations provably provide access to many important features of the distribution P (x), in polynomial time. However, a priori this only stands for tree factor graphs, which limits a lot the range of applicability of this algorithm. As we saw, this limitation arises as the incoming messages in the updates (cf. Fig. 1.7) must be independent. All the rigorous mathematical analysis of BP falls down when the factor graph contains loops, e.g. in Fig. 1.7a representing the GLM.

However, one can still iterate eqs. (1.38), despite the lack of theoretical guarantees. This strategy is generically known as loopy belief propagation, and a general theory of it is still missing. Nevertheless, various studies demonstrated that loopy BP could provide very good approximations to the marginals and the free entropy in a variety of loopy factor graphs. For instance, if all loops in the graph are long, the graph will effectively be locally tree-like, and the BP iteration then provide accurate approximations. Loopy BP has also been used in infinite-range inference models [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF], and we will focus on this kind of models in the following. Moreover, the BP fixed points can be shown to be related to the stationary points of the Bethe free entropy of eq. (1.41) for any type of factor graph [START_REF] Mézard | Information, physics, and computation[END_REF].

In particular, applying BP to the GLM, the updates of eq. (1.39) are an attractive tool to e.g. sample from the posterior distribution of eq. (1.6), or to compute the asymptotic free entropy via eq. (1.41). As we will see, in the large-n limit these updates will turn out to yield the optimal performance achievable in polynomial time in GLMs, even though the graph of Fig. 1.7a is extremely far from a tree!

Approximate Message Passing (AMP): derivation and consequences

Although easy to write, the BP equations of eq. (1.38) are still computationally very heavy, since one needs to compute the messages for any value of the variables, and that they are Θ(n 2 ) messages in long-range models such as the GLM. In [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF], the authors proposed a novel algorithm, that they named Approximate Message Passing (AMP), for compressed sensing with Gaussian matrices. This algorithm is derived from BP, and it can be seen as a way to make the BP equations tractable, using two features:

• The high-dimensional limit n → ∞, which we will allow us to use "central limit theorem"-type results.

• The projection of the BP messages on a parametrized family of distributions (here Gaussians), reducing the computation of each message to the one of a few scalar quantities. This projection is justified using the high-dimensional limit mentioned above.

This algorithm was later generalized to the GLM of Model 1.1 with Gaussian matrices and arbitrary outputs [START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF], and later to more generic ensembles of matrices [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF][START_REF] Sundeep Rangan | Vector approximate message passing[END_REF]. We follow here a very classical route to derive the AMP algorithm from the BP equations, see also the presentations of [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF][START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF] 9 .

Relaxed belief propagation

The first step of the procedure is to write what is called relaxed belief propagation equations, which were derived in [START_REF] Rangan | Estimation with random linear mixing, belief propagation and compressed sensing[END_REF]. Let us focus on the GLM, that is eq. (1.39). We start by the first equation. Under the statistics of j( =i) dx j m j→µ (x j ), and conditioned on the value of x i , the variable k( =i) F µk x k is approximately Gaussian in the large n limit, with mean and variance given by:

ω t µ→i ≡ 1 √ n j( =i) F µj a t-1 j→µ and V t µ→i ≡ 1 n j( =i) F 2 µj v t-1 j→µ , (1.42)
in which a j→µ , v j→µ are the mean and variance of x j under the message m j→µ . The first equation of eq. (1.39) can then be simplified into

mt µ→i (x i ) ∝ dz µ P out (y µ |z µ ) exp - (z µ -ω t µ→i -F µi x i / √ n) 2 2V t µ→i .
(1.43)

Expanding the exponential in eq. (1.43) as n → ∞, we reach that mt µ→i (x i ) is approximately Gaussian in this limit, with variance 1/A t µ→i and mean B t µ→i /A t µ→i given by:

B t µ→i = 1 √ n F µi g out (y µ , ω t µ→i , V t µi ) and A t µ→i = - 1 n F 2 µi ∂ ω g out (y µ , ω t µ→i , V t µi ), (1.44)
in which we defined:

g out (y, ω, V ) ≡ 1 V dz (z -ω) e -(z-ω) 2 2V P out (y|z) dz e -(z-ω) 2 2V P out (y|z)
.

(1.45)

Finally, we focus on the second equation of eq. (1.39). Using the approximate Gaussianity of the messages yields that this equation reduces to:

a t i→µ = f x ν( =µ) A t ν→i , ν( =µ) B t ν→i and v t i→µ = ∂ B f x ν( =µ) A t ν→i , ν( =µ) B t ν→i , (1.46)
with the auxiliary function:

f x (B, A) ≡ P 0 (dx) x e -A 2 x 2 +Bx P 0 (dx) e -A 2 x 2 +Bx
.

(1.47)

Eqs. (1.42),(1.44),(1.46) define the relaxed BP updates.

Algorithm 2: GAMP for the Generalized Linear Model [START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF].

Result:

The estimator x Input: Observations Y ∈ R m and sensing matrix F ∈ R m×n ; Initialize x, ĉ, ω, V randomly; while not converging do

• Estimation of the mean and variance of z ≡ Fx/ √ n;

ω t µ = 1 √ n µ,i F µi xt i - 1 n g out (y µ , ω t-1 µ , V t-1 µ ) i F 2 µi c t i and V t µ = 1 n µ,i F 2 µi c t i ;
• Mean and variance of x estimated from the channel observations;

A t i = - 1 n µ F 2 µi ∂ ω g out (y µ , ω t µ , V t µ ) and B t i = A t i xt i + 1 √ n µ F µi g out (y µ , ω t µ , V t µ ) ;
• Update of the estimated marginals with the prior information;

xt+1 i = f x (B t i , A t i ) and ĉt+1 i = ∂ B f x (B t i , A t i ); t = t + 1; end

From relaxed BP to Approximate Message-Passing

The relaxed BP equations can be simplified further in the n → ∞ limit. This is detailed in [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF] and essentially reduces to show that all eqs. (1.42),(1.44),(1.46) can be written as a function of the averages over the "output" nodes (e.g.

ω µ ≡ n -1 i ω µ→i and A i ≡ m -1 µ A i→µ )
. One must however be careful when replacing the messages with these averages, to take properly into account all terms of leading order in n. Carrying out this procedure yields Algorithm 2.

Note that there are corrections in Algorithm 2 with what a "naive" average of the r-BP equations (1.42),(1.44),(1.46) would give. These corrections are similar to the Onsager reaction terms arising the TAP approach in Section 1.3.2, and we will see actually that they are completely equivalent to them. Algorithm 2 was first described in this form by Sundeep Rangan in [START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF], and is a generalization of the original AMP algorithm written by Donoho, Maleki and Montanari for compressed sensing in [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF].

Connection with the TAP equations -Let us examine the stationary limit of Algorithm 2. It yields that any fixed point of GAMP must satisfy:

                 ω µ = 1 √ n µ,i F µi xi - 1 n g out (y µ , ω µ , V µ ) i F 2 µi c i and V µ = 1 n µ,i F 2 µi c i , A i = - 1 n µ F 2 µi ∂ ω g out (y µ , ω µ , V µ ) and B i = A i xi + 1 √ n µ F µi g out (y µ , ω µ , V µ ), xi = f x (B i , A i ) and ĉi = ∂ B f x (B i , A i ).
(1.48)

In fact, eq. (1.48) are exactly the TAP equations for the GLM, that we introduced in Section 1.3.2 for the SK model. Generalizing the TAP picture from the SK model to the GLM, and subsequently obtaining eq. (1.48), is not very involved and we will detail it in a much more general context in Chapter 2, so that we leave this computation aside.

On a historical note, while the TAP equations have been used by physicists since the 1970s [START_REF] David J Thouless | Solution of 'solvable model of a spin glass[END_REF], the AMP algorithm (because it is derived from BP) provides an explicit (and nontrivial!) iteration scheme of these equations. This is of the utmost importance: the iteration of the TAP equations in disordered systems is a long-lasting challenge of theoretical physics (see e.g. [START_REF] Bolthausen | An iterative construction of solutions of the TAP equations for the Sherrington-Kirkpatrick model[END_REF][START_REF] Opper | A theory of solving TAP equations for Ising models with general invariant random matrices[END_REF] for discussions on the SK model and on more generic Ising models), and has led to a significant literature [TAP77, Méz89, OW96, BBBZ07, Kab08a]10 . This issue was solved for a large class of inference models by AMP algorithms. The Bethe free entropy (1.41), applied at the fixed point of the AMP algorithm, can also be shown to be equal to the TAP free entropy introduced in Section 1.3.2, and we will detail this correspondence further in Chapter 2.

State evolution and connection to the replica method

An extremely important characteristic of AMP algorithms is that they can be analyzed in the high-dimensional limit. More precisely, we can compute their asymptotic performance via State Evolution (SE) equations (as named by [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF] for compressed sensing). The most natural way to derive them is to start from the relaxed-BP equations (1.42),(1.44),(1.46). Using the assumption of independence of incoming messages in the BP approach, one can use the central limit theorem to reduce the equations to the evolution of the mean and variances of the sums of many random variables. Tracking the iterations of these means and variances yields the state evolution equations, and a detailed derivation following these lines can be found in [KMS + 12].

In the context of the GLM with arbitrary output channel, the SE equations were proven in [START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling[END_REF], so that we present them here as a theorem. In order to precisely state them, we first describe the results of the replica theory (cf. Section 1.3.1) for the GLM with Gaussian matrices.

Replica formula for the free entropy -As we mentioned in Section 1.3.1, the replica method, initially developed for disordered systems, has proven to be a very powerful tool as well for the analysis of inference models. In particular, the replica-symmetric formula for Bayes-optimal generalized linear models with Gaussian sensing matrices builds on early physics analysis of the perceptron [GD89, Gyö90, STS90], and was derived in [KMS + 12]. The formula was eventually proven in [BKM + 19], confirming the physics conjecture, so that we refer to this work for the following theorem.

Theorem 1.3 (RS formula for Gaussian GLMs, informal [BKM + 19])

Assume that:

• The signal X * ∈ R n is generated from an i.i.d. prior distribution P 0 , with zero mean and variance E P 0 [X 2 ] = ρ > 0.

• The matrix F has i.i.d. elements from N (0, 1).

• We are in the Bayes-optimal setting, i.e. the channel and prior used in the posterior distribution of eq. (1.6) are the ones used to generate the data Y µ and the signal X * .

Then the free entropy of the posterior distribution of eq. (1.6) converges to:

Φ ≡ lim n→∞ 1 n E ln Z n (Y, F) = sup q∈[0,ρ] inf q≥0 f RS (q, q) = sup q,q f RS (q, q), (1.49)
with the replica-symmetric potential defined as 11 :

f RS (q, q) ≡ ψ 0 (q) + αΨ out (q) - qq 2 , (1.50)        ψ 0 (q) ≡ DZP 0 (dX * ) exp - q 2 (X * ) 2 + qX * Z ln P 0 (dx) exp - q 2 x 2 + qxZ , Ψ out (q) ≡ dY DV DW * P out (Y | √ qV + √ ρ -qW * ) ln DwP out (Y | √ qV + √ ρ -qw). 0.0 0.2 0.4 0.6 0.8 1.0 q f RS (q)
Replica-symmetric potential < 0.5 0.5 < < 1 1 < < 1.12 > 1.12 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Real phase (sign) retrieval IT optimal error GAMP asymptotic error GAMP simulations

Figure 1.8: (Left) Replica-symmetric potential f RS (q) for ρ = 1 and different values of the sampling ratio α. We plot the global minimum as a circle and the local minimum closest to q = 0 (if different) as a square. The replica potential picture is merely a representation and not an accurate picture. (Right) Mean squared error ρ -q achieved in the sign retrieval problem with Gaussian matrices. We compare the global maximum of f RS (information-theoretic MMSE) to the local maximum closest to q = 0 (GAMP asymptotic error). We confirm our predictions with finite-size simulations of GAMP. The qualitative behavior of the replica-symmetric potential in each of the four colored areas is given by the curve of the left figure with the corresponding color.

State evolution of AMP -

We can now state the SE equations for the GAMP algorithm.

Theorem 1.4 (Bayes-optimal State Evolution of GAMP, informal [BM11, JM13])

Let xt be the estimator of GAMP (Algorithm 2), and X * the solution to infer. We define two asymptotic quantities known as overlaps

q t AMP ≡ lim n→∞ 1 n n i=1 (x t i ) 2 and m t AMP ≡ lim n→∞ 1 n n i=1 xt i X * i .
(1.51)

Then these two quantities are well-defined, and moreover by the Nishimori identity (Proposition 1.1) one has m t AMP = q t AMP along the trajectory of GAMP. We introduce another variable qAMP ≡ 2αΨ out (q AMP ). Then at all iteration times t ≥ 0, (q AMP , qAMP ) is a solution of the State Evolution (SE) equations: qt AMP = 2αΨ out (q t AMP ) and q t+1 AMP = 2ψ 0 (q t AMP ).

(1.52)

Moreover, for a random initialization of GAMP we have q t=0 AMP = qt=0 AMP = 0.

Optimality of AMP -Theorem 1.4 clarifies the link between AMP and the replica method. It shows precisely that, in the Bayes-optimal setting, the replica potential f RS describes both the information-theoretic optimal performance (via its global maximum, cf. Theorem 1.3) and the asymptotic performance of AMP, via the SE equations (1.52). Note that these equations correspond to a local optimization algorithm of f RS starting from the point q = q = 0. Basically, combining the replica method and message-passing algorithms allowed us to reduce the study of the fundamental limits of a high-dimensional inference problem to the investigation of the landscape of a simple scalar potential, as illustrated in Fig. 1.8 (see Chapter 6 for more details on the specific problem of phase/sign retrieval illustrated in this figure). Importantly, the link between the state evolution of AMP and the replica-symmetric potential is actually very general, and has several crucial consequences:

• When f RS has a unique maximum in a Bayes-optimal inference model, AMP (here GAMP) achieves the information-theoretic optimal error. However when it is not the case (e.g. the green region in Fig. 1.8-right), Theorem 1.4 leads to the conjecture that AMP should achieve the optimal performance (i.e. the optimal overlap, or MSE) over all "local" polynomial-time algorithms. Indeed, any such algorithm should start at the un-informative point q = 0, and would not be able to cross the "barrier" in the replica-symmetric potential in polynomial time.

Of course this argument is very far from rigorous and such a wild statement should not be taken as is. There indeed exists problems for which this statement is not correct, notably the famous XOR-SAT problem for which all local search algorithms (including AMP) fail, while the problem can be solved in polynomial time by Gaussian elimination using a well-designed mapping [RTWZ01, BHL + 02, JMS04]. In an effort to put the conjectured optimality of AMP on more precise grounds, the authors of [START_REF] Celentano | The estimation error of general first order methods[END_REF] have shown that AMP is optimal in a very large class of inference problems among all general first order methods, which includes e.g. all gradient-based optimization methods. This result covers in particular the GLM, and it proves that Fig. 1.8 indeed describes the fundamental statistical and algorithmic performances in real Gaussian phase retrieval among general first order methods.

• To summarize, the replica formula (here Theorem 1.3) gives access to the information-theoretic optimal error, while the class of AMP algorithms provides the (conjectured) algorithmic optimal error, and this error is also characterized by the replica formula (see here Theorem 1.4). This allows to study computational-to-statistical gaps (also called hard phases) merely by analyzing the replica-symmetric potential, a simple function of a few scalars! Leveraging this statistical physics toolbox to study computational gaps in inference is an important line of research, see e.g. [KMS + 12, DJM13, DM15, ZK16, BPW18, BKM + 19] (without any aim of exhaustivity), and a significant fraction of Part II of the present thesis will be devoted to the investigation of such gaps.

Final remarks on AMP algorithms

AMP for spin glass optimization -AMP algorithms have recently been adapted as optimization algorithms in a variety of spin glass models with full replica symmetry breaking (see Section 1.3.1). In this context they were called Incremental AMP, and they have solved longlasting open problems on the optimization of the SK model and a wide class of FRSB spin glass models satisfying a "no-overlap-gap" property [AMS20, AM20, Sub21, Mon21, Sel21].

The cavity method -In our presentation of methods which originated in statistical physics, we did not detail a very important technique known as the cavity method. It was introduced by Mézard and coauthors in [START_REF] Mézard | SK model: The replica solution without replicas[END_REF] as a way to obtain the "replica solution without replicas" for the SK model. The basis of the cavity method for generic graphical models described by eq. (1.37) is to study the original model along with a modified version in which one removed a single variable node: as these two systems must be equivalent in the thermodynamic limit this yields self-consistent equations on the order parameters of the system. For the case of the GLM, these self-consistent equations are exactly the state evolution of Theorem 1.4! The cavity method has actually been shown generically to be equivalent to the replica computations12 , and can be generalized to an arbitrary number of replica symmetry breaking levels. As we will not discuss the cavity computations in this thesis, the reader interested in learning more should refer to [START_REF] Mézard | Information, physics, and computation[END_REF][START_REF] Mézard | Cavity method: message-passing from a physics perspective[END_REF].

The mismatched case -The BP equations (1.39) and the GAMP algorithm 2 do not use the knowledge of the true prior and channels, so that they are easily generalized to the non-Bayesoptimal (or "mismatched") case. The state evolution (Theorem 1.4) can also be generalized to this more general case, and is again equivalent to the replica-symmetric computation. However, out of Bayes-optimality, the replica-symmetric computation generally does not yield the correct asymptotic free entropy of the system, so that we have no guarantee of any kind of optimality of AMP algorithms. However as we will illustrate in Chapter 6, these algorithms can still achieve very good performance in the mismatched case. As we will most often need the Bayes-optimal version of the AMP algorithm in this dissertation we do not present here the mismatched state evolution, which the reader can find e.g. in [KMS + 12] or [START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF].

Three approximations for non-Gaussian inference problems

In this section, inspired by [MFC + 19], we introduce three different approximation schemes for disordered systems that can also be applied to many inference problems, when the underlying interactions are more structured than simply Gaussian. Let us briefly recap their history:

• The adaptive TAP (adaTAP) scheme was developed and presented in 2001 in [START_REF] Opper | Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach[END_REF][START_REF] Opper | Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling[END_REF][START_REF] Opper | Advanced mean field methods: Theory and practice[END_REF], for systems close to the SK model.

• The same year, Thomas Minka's Expectation Propagation (EP) approach was presented [START_REF] Thomas P Minka | Expectation propagation for approximate Bayesian inference[END_REF]. Opper and Winther used an alternative view of local-consistency approximations of the EP-type which they call Expectation Consistent (EC) approximations in [START_REF] Opper | Expectation consistent approximate inference[END_REF][START_REF] Opper | Expectation consistent free energies for approximate inference[END_REF], effectively re-deriving their adaTAP scheme from this new point of view.

• The Vector Approximate Message-Passing (VAMP) approach is more recent [SRF16, RSF17], and is again another EP approach, for Generalized Linear Models (GLMs). Compared with other EP-like approaches it has the advantage that it leads to both a practical converging algorithm, and a rigorous treatment of its time evolution.

The connection between these different approaches was hinted several times for SK-like problems, see e.g. [START_REF] Opper | A theory of solving TAP equations for Ising models with general invariant random matrices[END_REF][START_REF] Çakmak | Convergent dynamics for solving the TAP equations of ising models with arbitrary rotation invariant coupling matrices[END_REF], and we establish it clearly in the remaining of Section 1.4.3, along with a detailed presentation of these three approximations.

Expectation Consistency approximation

Consider a model in which the probability distribution of a vector x ∈ R n is of the generic form:

P (x) = 1 Z n P 0 (x)P J (x).
(1.53)

As we saw above, such distributions typically appear in Bayesian approaches to inference problems, and as emphasized in Section 1.1, our goal will be to compute ln Z n for large values of n.

We will use the Bayesian language and denote P 0 as a prior distribution on x, which will be typically factorized (i.e. all the components of x are independent under P 0 ). The distribution P J is responsible for the interactions between the {x i }: we will often be interested in pairwise interactions, i.e. when ln P J is a quadratic form in the {x i } variables. An example of such a model is the infinite-range Ising model at inverse temperature η ≥ 0, with external field h (i.e. a SK model with external field and non-Gaussian couplings):

           P 0 (x) = n i=1 1 2 cosh(ηh i ) δ(x i -1)e -ηh i + δ(x i + 1)e ηh i , P J (x) = exp η 2 i,j J ij x i x j .
(1.54) Each of the two distributions P 0 and P J allows for tractable computations of physical quantities (e.g. averages), but the difficulty arises when considering their product. The idea behind EC is to simultaneously approximate P 0 and P J by a tractable family of distributions. For the sake of the presentation we will consider Gaussian probability distributions, although this can be generalized to different families, as detailed in [START_REF] Opper | Expectation consistent approximate inference[END_REF]. We define the first approximation as:

µ 0 (x) ≡ 1 Z 0 (Γ 0 , λ 0 ) P 0 (x)e -1 2 x Γ 0 x+λ 0 x ,
with Γ 0 ∈ S + n (R) and λ 0 ∈ R n . We will denote • 0 the averages with respect to µ 0 . We can write the trivial identity:

Z n = Z n × Z 0 (Γ 0 , λ 0 ) Z 0 (Γ 0 , λ 0 ) = Z 0 (Γ 0 , λ 0 ) P J (x) e 1 2 x Γ 0 x-λ 0 x 0 .
When computing the average P J (x) e 1 2 x Γ 0 x-λ 0 x 0 , we can replace the distribution µ 0 by an approximate Gaussian distribution:

µ S (x) ≡ 1 Z S e -1 2 x (Γ J +Γ 0 )x+(λ 0 +λ J ) x .
This yields the Expectation-Consistency (EC) approximation to the free entropy:

ln Z EC (Γ 0 , Γ J , λ 0 , λ J ) = ln dxP 0 (x)e -1 2 x Γ 0 x+λ 0 x + ln dxP J (x)e -1 2 x Γ J x+λ J x -ln dx e -1 2 x (Γ 0 +Γ J )x+(λ 0 +λ J ) x .
(1.55)

Note that all three parts of this free entropy are tractable. In order to symmetrize the result we can define a third measure µ J with average • J :

µ J (x) ≡ 1 Z J (Γ J , λ J ) P J (x)e -1 2 x Γ J x+λ J x .
The final free entropy should not depend on the values of the parameters, so we expect that the optimal values for Γ 0 , Γ J , λ 0 , λ J make Z EC stationary. This is a strong hypothesis, and the reader can refer to [START_REF] Opper | Expectation consistent approximate inference[END_REF] for more details and justifications. These stationarity equations yield the Expectation Consistency (EC) conditions, giving their name to the procedure:

x i 0 = x i J = x i S , x i x j 0 = x i x j J = x i x j S .
(1.56)

Adaptive TAP approximation

The adaptive TAP (adaTAP) approximation [START_REF] Opper | Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling[END_REF][START_REF] Opper | Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach[END_REF] provides an equivalent way to derive the free entropy of eq. (1.55) for models with pairwise interactions. Let us briefly sketch its derivation and the main arguments behind it. Again, for the sake of the presentation we consider the infinite-range Ising model of eq. (1.54). As we saw in Section 1.3.2, the free entropy Φ J ≡ n -1 ln Z n at fixed values of the magnetizations m i = x i and v ij = x i x j can be written using Lagrange multipliers (λ, Γ):

Φ J (η, m, v) = extr λ∈R n Γ∈S + n (R) -λ m + i,j Γ ij 2 (v ij + m i m j ) + ln dx P 0 (x) e η 2 x Jx-1 2 x Γx+λ x .
The adaTAP approximation consists in writing:

nΦ J (η, m, v) = Φ J (0, m, v) + η 0 dl ∂Φ J (l, m, v) ∂l , Φ J (0, m, v) + η 0 dl ∂Φ (Gauss.) J (l, m, v) ∂l , Φ J (0, m, v) + Φ (Gauss.) J (η, m, v) -Φ (Gauss.) J (0, m, v).
(1.57)

In this expression Φ (Gauss.) J (η, m, v) denotes the free entropy of the same system, but in which the spins have Gaussian statistics. The idea behind the adaTAP approximation is that in

∂ l Φ J (l, m, v) = (2n) -1 ij J ij x i x j
it is reasonable to assume that the underlying variables were Gaussian, as we consider the expectation of a sum of a large number of variables. The assumptions of adaTAP, although reasonable, are a priori hard to justify more rigorously and systematically. Note that the free entropy (1.57) of adaTAP is equivalent to the one derived using Expectation Consistency in eq. (1.55). Indeed, using additional Lagrange multipliers we can write the three terms of eq. (1.57) as:

nΦ adaTAP J (η, m, v) = extr λ 0 ,Γ 0 ln dxP 0 (x) e -1 2 x Γ 0 x+λ 0 x -λ 0 m + 1 2 i,j (Γ 0 ) ij (v ij + m i m j ) + extr λ J ,Γ J ln dxP J (x) e -1 2 x Γ J x+λ J x -λ J m + 1 2 i,j (Γ J ) ij (v ij + m i m j ) -extr λ S ,Γ S ln dx e -1 2 x Γ S x+λ S x -λ S m + 1 2 i,j (Γ S ) ij (v ij + m i m j ) .
Once written in this form, the extremization over m and v of the free entropy implies that Γ S = Γ 0 + Γ J and λ S = λ 0 + λ J . It is then clear that we found back ln Z EC of eq. (1.55).

Vector Approximate Message Passing approximation

The Vector Approximate Message Passing (VAMP) algorithm [RSF17, SRF16] extends previous message-passing approaches, such as the AMP algorithm that we saw in Section 1.4.2 for Gaussian interactions, to a class of correlated interaction matrices that satisfy a right-rotation invariance property, that we will define more precisely in Section 1.5. The algorithm itself can be derived in several ways: here we will use a method based on the belief propagation equations on a "duplicated" factor graph, and their Gaussian projection. As we shall see, the Bethe free entropy of this model is then equivalent to the Expectation-Consistency free entropy. For the sake of the presentation we consider again the infinite-range Ising model of eq. (1.54). The idea behind VAMP is to "duplicate" the model as follows:

Φ J ≡ 1 n ln dx P 0 (x)P J (x) = 1 n ln dx 1 dx 2 P 0 (x 1 )P J (x 2 ) δ(x 1 -x 2 ).
This partition function can be represented as a duplicated factor graph involving two vector nodes, as shown in Fig. 1.9. One then writes the BP equations for this problem, as we saw in Section 1.4.1, in terms of two "messages" m 0 (x 2 ) and m J (x 1 ). These equations can be obtained by the stationarity conditions of the Bethe free entropy defined in eq. (1.41), and which reads here:

Φ Bethe ≡ ln dx P 0 (x) m J (x) + ln dx P J (x) m 0 (x) -ln dx m 0 (x) m J (x).

(1.58)

x 1 x 2 δ x 1 ,x 2 P 0 P J m 0 m J Figure 1
.9: Duplicated factor graph for the VAMP approximation. Circles represent vector nodes and squares factor nodes. We represent the two messages m 0 and m J in terms of which we write the full BP equations.

As the factor graph of Fig. 1.9 is a tree, the BP equations are an exact representation of the original problem, but they are in general intractable. In order to make the computation possible one can make a Gaussian approximation, which is at the core of the VAMP algorithm: the messages m 0 and m J are assumed to be Gaussian, and thus are fully characterized by their first two moments:

m 0 (x) ∝ e -1 2 x Γ 0 x+λ 0 x , m J (x) ∝ e -1 2 x Γ J x+λ J x .
(1.59)

Writing the BP update rule with this assumption yields the VAMP iterations:

                 Γ t 0 = xx t-1 0 -x t-1 0 x t-1 0 -Γ t-1 J , λ t 0 = (Γ t 0 + Γ t-1 J ) x t-1 µ 0 -λ t-1 J , Γ t J = xx t J -x t J x t J -Γ t 0 , λ t J = (Γ t 0 + Γ t J ) x t µ J -λ t 0 ,
(1.60)

with the measures µ 0 (x) ∝ P 0 (x)m J (x) and µ J (x) ∝ P J (x)m 0 (x). Note that plugging the ansatz of eq. (1.59) into eq. (1.58) immediately gives back the EC free entropy of eq. (1.55). Moreover, the stationary limit of the BP equations of eq. (1.60) (i.e. when removing the time indices) is identical to the Expectation-Consistency conditions of eq. (1.56).

Conclusion

We introduced three approximation schemes to compute the free entropy of disordered systems, that have natural extensions in inference problems: Expectation Consistency, adaptive TAP, and Vector Approximate Message Passing. As we detailed, they all rely on the same underlying Gaussian approximations, and therefore yield equivalent expressions for the free entropy. As a final note, an important advantage of the VAMP approach is that it naturally provides an iterative scheme to solve the fixed point equations. These iterations form a very efficient algorithm [START_REF] Sundeep Rangan | Vector approximate message passing[END_REF], which is intuitive as they were derived directly from the belief propagation equations.

Some rudiments of probability and random matrix theory

Theoretical physics and random matrix theory share a long history that dates back to Wigner [START_REF] Wigner | Characteristic vectors of bordered matrices with infinite dimensions[END_REF], and that powered progress in various areas ranging from disordered systems [START_REF] Samuel | Theory of spin glasses[END_REF][START_REF] Sherrington | Solvable model of a spin-glass[END_REF] to quantum chaos [START_REF] Bohigas | Characterization of chaotic quantum spectra and universality of level fluctuation laws[END_REF], quantum chromodynamics [START_REF] Verbaarschot | Random matrix theory and chiral symmetry in qcd[END_REF], or superconductivity [START_REF] Safi | Random matrix model for superconductors in a magnetic field[END_REF]. The growing interplay of physics and statistics [ZK16, Gab20, Zde20], which is the subject of this thesis, further strengthened this connection. In this section, the last one of Chapter 1, we review some important techniques and models from random matrix theory, as well as a few probabilistic tools that will be useful for the rest of the thesis. Apart from some specific arguments, it is written in a fashion closer to mathematical standards, as a large part of its content originated in the mathematics literature. For more details and advanced results in random matrix theory, one can refer for instance to [START_REF] Madan | Random matrices[END_REF] and [START_REF] Greg W Anderson | An introduction to random matrices[END_REF] for classical references of theoretical physics and mathematics. On a less advanced level, [START_REF] Livan | Introduction to random matrices: theory and practice[END_REF] provides a comprehensive introduction to techniques of random matrix theory used in theoretical physics.

Random matrix ensembles and asymptotic spectra

Recall that M + 1 (R) is the set of probability measures on R. Spectral distributions -Consider a (random) symmetric matrix M ∈ H n (K), with (real) eigenvalues {λ i } n i=1 . The Empirical Spectral Distribution (ESD) of M is defined as:

ν n ≡ 1 n n i=1 δ λ i .
(1.61)

For many random matrices considered in this thesis, the (random) probability measure ν n will converge almost surely and in the weak sense to a deterministic probability measure ν ∈ M + 1 (R) as n → ∞. In this case, we will call ν the Limiting Spectral Distribution (LSD) of M.

A few random matrix ensembles

Wigner matrices -In the 1950s, Wigner introduced a class of random matrices to study the nuclei of heavy atoms [START_REF] Wigner | Characteristic vectors of bordered matrices with infinite dimensions[END_REF]. This led to what is now known as Wigner matrices, which are Hermitian/symmetric random matrices J ∈ H n (K) such that:

(i) All {J ij } i<j are i.i.d
W ≡ 1 m m µ=1 x µ x µ ∈ S n .
At fixed n, when the number m of data samples gets large, W approaches the covariance E Px [xx ] of the distribution, by the law of large numbers. However the situation is much more complex in a high-dimensional regime, in which both n and m gets large in the same scale, that is m/n → α > 0, a limit which we referred to as the thermodynamic limit, cf. Definition 1.2. Let us assume that P x is a Gaussian distribution with covariance matrix Σ ∈ S ++ n (R). As

x µ d = √ Σz µ , with z µ i.i.d.
∼ N (0, I n ), we reach

W d = √ Σ 1 m m µ=1 z µ z µ √ Σ.
(1.62)

Eq. (1.62) defines the random matrix ensemble of Wishart matrices. Under suitable assumptions on Σ their asymptotic spectra was first studied by Marchenko and Pastur in [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF], see Theorem 1.7. In Chapter 8 we will consider an extension of Wishart matrices, that we call generalized sample covariance matrices.

Rotationally-invariant matrices -Roughly speaking, when we refer to rotational invariance, we mean a complete delocalization of eigenvectors: they are distributed according to the Haar measure on the orthogonal group U β (n), with no privileged direction. Throughout the manuscript, we will use two assumptions that we will both refer to as rotation invariance.

Model S (Hermitian/Symmetric rotationally invariant matrix)

Let n ≥ 1. The random matrix J ∈ H n (K) is said to be rotationally invariant if:

(i) For any O ∈ U β (n), J d = OJO † .
(ii) The empirical spectral distribution (ESD) of J:

ρ n ≡ 1 n n i=1 δ λ i (J)
converges (almost surely and in the weak sense) as n → ∞ to a probability distribution ρ with compact support, called the Limiting Spectral Distribution (LSD) of J.

(iii) The smallest and largest eigenvalue of J converge (almost surely) to the infimum and supremum of the support of ρ.

Model R (Rectangular rotationally invariant matrix)

Let n ≥ 1, and m = m(n) ≥ 1 such that m/n → α > 0 as n → ∞.

• The random matrix L ∈ R m×n is said to be right-rotationally invariant if J ≡ L † L is rotationally invariant, according to Model S.

• In the same way, L is left-rotationally invariant if J ≡ LL † is a symmetric rotationally invariant matrix.

Note that in both models we added additional hypotheses to the complete delocalization of eigenvectors, more precisely on the asymptotic behavior of the spectral distribution and on the existence of outliers in the spectrum. These hypotheses have been added for convenience as we will use them repeatedly in this manuscript. Note that a strong form of eigenvectors delocalization can also be proven for non-rotationally invariant random matrices, e.g. the Wigner matrices with generic i.i.d. distribution [START_REF] Erdős | Semicircle law on short scales and delocalization of eigenvectors for Wigner random matrices[END_REF].

Examples of rotationally-invariant ensembles -

• The GOE/GUE (cf. Definition 1.3) satisfies Model S, with LSD given by the Wigner semicircle law as detailed in Theorem 1.6.

• The Wishart ensemble of eq. (1.62) with Σ = I n and ratio α > 0. This ensemble satisfies Model S, with LSD given by the Marchenko-Pastur law as shown in Theorem 1.7.

• Standard Gaussian i.i.d. matrices L ∈ K m×n satisfy Model R and are left and right-rotationally invariant.

• Matrices generated via a potential V (x), i.e. J ∈ H n (K) generated with a probability density proportional to exp{-n Tr[V (J)]}. Given some natural hypotheses on the behavior of the potential V (x), such matrices satisfy Model S.

Useful transforms of probability measures

We now introduce a few transforms of probability measures that are especially useful in the context of random matrix theory. Let C + = {z ∈ C, Im z > 0}. We let ν ∈ M + 1 (R), and we denote supp(ν) ⊆ R its support. We define (t min , t max ) as the infimum and supremum of supp(ν). For any z such that z ∈ C + or z ∈ R\supp(ν), we define the Stieltjes transform of ν (or resolvent) as13 :

S ν (z) ≡ ν(dt) 1 t -z .
(1.63)

Note that S ν (z) is a one-to-one mapping of C + on itself. The Stieltjes transform has proven to be a very useful tool in particular in random matrix theory [START_REF] Greg W Anderson | An introduction to random matrices[END_REF]. Importantly, the knowledge of the Stieltjes transform of a probability measure above the real line allows to characterize it completely, via the Stieltjes-Perron inversion formula (see Theorem X.6.1 of [START_REF] Dunford | Linear operators. 2. Spectral theory: self adjoint operators in Hilbert Space[END_REF]):

Theorem 1.5 (Stieltjes-Perron inversion formula)

Let ν ∈ M + 1 (R).
Then for all a < b, we have

ν((a, b)) = lim δ↓0 lim ↓0 1 2iπ b-δ a+δ [S ν (x + i ) -S ν (x -i )]dx.
In particular, if ν has a continuous density with respect to the Lebesgue measure then:

∀x ∈ R, dν dx = lim ↓0 1 π Im S ν (x + i ).
Let us now assume that t max < ∞, so that the support of ν is bounded from above. On (t max , +∞), S ν induces a strictly increasing C ∞ diffeomorphism S ν : (t max , ∞) → (-∞, 0), and we denote its inverse S -1 ν . It is easy to see that the same property holds as well on (-∞, t min ). One can then introduce the R-transform of ν as: R ν (s) ≡ S -1 ν (-s) -

1 s .
(1.64) R ν (s) is defined for -s ∈ S ν [(t min , t max ) c ], and one can show that it admits an analytical expansion around s = 0, see e.g. [START_REF] Antonia | Random matrix theory and wireless communications[END_REF]. We can write this expansion as:

R ν (s) = ∞ k=0 c k+1 (ν) s k . (1.65)
The elements {c k (ν)} k≥1 are called the free cumulants of ν. They are analogous to the usual cumulants since the R-transform is analogous to the cumulant generating function in the context of random matrix theory. This relation was shown in detail in [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF] and we will detail many results of this work in the following. In particular, one verifies that c

1 (ν) = E ν [X] and c 2 (ν) = E ν [X 2 ] -(E ν X) 2 .
The free cumulants can be recursively computed from the moments of the measure using the so-called free cumulant equation:

∀k ∈ N * , E ν [X k ] = k m=1 c m (ν) {k i } i∈[|1,m|] s.t i k i =k m i=1 E ν [X k i -1 ].
(1.66)

The free cumulants naturally arise from the theory of free probability, which is the study of non-commutative random variables [START_REF] Voiculescu | Free random variables[END_REF]. This field has very strong connections with random matrix theory, and many of the results we will mention can be understood from the free probability point of view.

Limit spectra of random matrices

As we mentioned, one of the first object of studies of random matrix theory was the limit spectra of different random matrix ensembles. Naturally, the first one to be studied was the Wigner ensemble that we defined above. Its LSD was shown in [START_REF] Wigner | Characteristic vectors of bordered matrices with infinite dimensions[END_REF], using a moments method, to be equal to the now-celebrated semicircle distribution.

Theorem 1.6 (Wigner's semicircle law, [Wig55])

Let J ∈ H n (K) be a Wigner matrix (see the definition above). Then as n → ∞, the ESD of J/ √ n a.s. tends (in the weak sense) to the semicircle law σ s.c. with density

σ s.c. (x) = √ 4 -x 2 2π 1{|x| ≤ 2}.
(1.67)

The proof of Theorem 1.6 was performed in [START_REF] Wigner | Characteristic vectors of bordered matrices with infinite dimensions[END_REF] using the calculation of all moments of a Wigner matrix. This proof was then much simplified by use of the Stieltjes transform of eq. (1.63), using arguments very similar to the cavity method of statistical physics. These different approaches are detailed in [START_REF] Greg W Anderson | An introduction to random matrices[END_REF], and we will apply the Stieltjes transform method to other kinds of random matrices in Chapter 5 of this thesis. Finally, we will also present in Section 1.5.2 a third way to prove Theorem 1.6, using a Coulomb Gas representation of the eigenvalues. As we will see, this method grants access not only to the convergence of the ESD but also to its large deviations away from its expected value.

Semicircle law and free probability -Intuitively, the semicircle law plays in free probability a role analogous to the one of the normal distribution in classical probability. Indeed its free cumulants c k (cf. eq (1.65)) satisfy c k = 0 for k ≥ 3, much like the cumulants of the Gaussian distribution are zero for orders greater or equal to 3.

A second very natural class of random matrices to study are the Wishart matrices of eq. (1.62).

As we mentioned, this was done first in [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF], while some generalizations and precisions were brought after, e.g. in [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF]. Note first that the matrix of eq. (1.62) has the same spectrum, up to zeros, as Z † ΣZ/m (with Z ∈ R n×m an i.i.d. Gaussian matrix) so that we will study this matrix in the following. We state here the main result of [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF] in the form of Theorem 1.1 of [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF], naming it the Marchenko-Pastur equation.

Theorem 1.7 (Marchenko-Pastur equation [MP67])

Let p, k → ∞ with p/k → α > 0. Let W ∈ K p×k a matrix whose elements are drawn i.i.d. from N β (0, 1). Let T p ∈ H p (K) be a random Hermitian matrix, independent of W, such that the ESD of T p converges weakly (and a.s.) to a measure ν T . Then, almost surely, the ESD of B k ≡ W † T p W/k converges in law to µ B ∈ M + 1 (R), whose Stieltjes transform satisfies, for every z ∈ C + :

S µ B (z) = -z -α ν T (dt) t 1 + tS µ B (z) -1
.

(1.68)

Moreover, for every z ∈ C + , there is a unique solution to eq. (1.68) such that S µ B (z) ∈ C + . This equation thus characterizes uniquely the measure µ B .

Theorem 1.7 is proven by means of the cavity method. We refer the reader to the random matrix proofs of Chapter 5, where we will re-derive Theorem 1.7. Note that eq. (1.68) can be written as an explicit formula in terms of the R-transform of eq. (1.64):

R µ B (s) = α ν T (dt) t 1 -ts .
The Marchenko-Pastur equation has been generalized further, e.g. by considering non-linear transformations of the elements of a Wishart matrix [BP19, PW19, PS21]. To conclude, let us mention the most celebrated application case of Theorem 1.7, when T p = I p . This corresponds to Wishart matrices with identity covariance, and in this case the LSD is called the Marchenko-Pastur distribution µ B = µ MP,α . Its density ρ MP,α is given by:

ρ MP,α (x) = max(0, 1 -α)δ(x) + (λ + (α) -x)(x -λ -(α)) 2πx 1{λ -(α) < x < λ + (α)}, (1.69) with λ ± (α) ≡ (1 ± √ α) 2 .

Large deviations

Large deviations theory is a set of probabilistic techniques, heavily used in mathematical physics, that focuses on the study of extremely rare events. A large part of statistical mechanics can actually be understood as a consequence of large deviations properties.

References -Our description of large deviations theory and its results will be extremely far from complete, as we simply aim at introducing a few important definitions and theorems. We refer the reader to e.g. [START_REF] Dembo | Large Deviations Techniques and Applications[END_REF][START_REF] Klenke | Probability theory: a comprehensive course[END_REF] for more complete mathematical discussions.

To introduce large deviation principles, let us consider a sequence (X i ) n i=1 of i.i.d. random variables, with common mean m and variance σ 2 . By the strong law of large numbers we know

S n ≡ 1 n n i=1 X i a.s. → n→∞ m.
(1.70)

The goal of large deviations theory is to understand the speed of this convergence, which is equivalent to understand large deviations of S n from its limit value. Note that while the central limit theorem gives that √ n(S n -m) weakly → n→∞ N (0, σ 2 ), large deviations theory focuses on very rare events in which the deviation S n -m is of order O(1) rather than in the scale n -1/2 . For all x ≥ m and t ≥ 0 we can write using Chernoff's inequality that

1 n ln P[S n ≥ x] ≤ -tx + 1 n ln E e n i=1 tX i ≤ -tx + ln E e tX .
So we reach easily that Another seminal large deviations result, that will prove useful in Chapter 7 of this thesis, is the LDP for the empirical measure of a set of i.i.d. variables, known as Sanov's theorem: Theorem 1.9 (Sanov's theorem, [START_REF] Ivan | On the probability of large deviations of random variables[END_REF]) Let (X n ) n≥1 be a sequence of i.i.d. real random variables with common law µ. We let ν n ≡ n -1 n i=1 δ X i the (random) empirical measures of the sequence. Then the law of ν n satisfies, as n → ∞, a large deviation principle in the scale n, with rate function given by the Kullback-Leibler divergence 15 :

1 n ln P[S n ≥ x] ≤ -sup t∈R tx -ln E e tX . ( 1
I(ν) ≡ D KL (ν|µ) =      dν ln dν dµ if ν µ, +∞ otherwise.
One of the most important applications of large deviation principles is the ability to use them to compute the exponential limit of some high-dimensional integrals, by use of Varadhan's lemma (named after one of the most prolific mathematicians in large deviations theory, who obtained in 2007 the Abel Prize for its contributions to the field):

Lemma 1.10 (Varadhan's lemma, [DZ98])

Let (X n ) n≥1 be a sequence of real random variables that satisfies a large deviation principle with rate function I(x). Let ϕ : R → R be a continuous function, and assume that one of the two following conditions hold:

       lim A→∞ lim sup n→∞ 1 n ln E exp{n ϕ(X n )}1[φ(X n ) ≥ A] = -∞, or lim sup n→∞ 1 n ln E exp{n γ ϕ(X n )} < ∞ for some γ > 1.
Then

lim n→∞ 1 n ln E exp{n ϕ(X n )} = sup x∈R [ϕ(x) -I(x)].
(1.75)

How physicists say it -Note that theoretical physicists usually do not refer to Varadhan's lemma. As the large deviation principle is stated in the form of eq. (1.74), we can write:

1 n ln dxP n (x) e nϕ(x) 1 n ln dx e n(ϕ(x)-I(x)) sup x∈R [ϕ(x) -I(x)], (1.76)
which is usually mentioned as an application of Laplace's method.

Application I: the Gibbs-Boltzmann measure and the canonical ensemble -Sanov's theorem 1.9 is intimately related to asymptotic quantities described in statistical mechanics. It formalizes one of the most important physical properties of statistical systems: when given a set of thermodynamical constraints, the distribution of a system will maximize the entropy while satisfying these constraints. Indeed, let us consider a set of n i.i.d. real random variables {X i } (representing e.g. the position or velocity of particles), with a common law µ. Directly applying Sanov's theorem then tells us about the microcanonical description of the ensemble: as n → ∞, the distribution of the particles will concentrate on the distribution ν * that maximizes the entropy, i.e. that minimizes D KL (ν|µ). Trivially, this distribution is ν * = µ, and this result is intimately related to the fundamental postulate of statistical physics: in an isolated system at equilibrium, all microstates are equally probable.

Let us now assume to have access to an (intensive) energy function E({X i }) = n -1 n i=1 E(X i ), which is thus a function of the empirical measure ν n ≡ n -1 n i=1 δ X i (e.g. the kinetic energy E kin = (m/2) i v 2 i if the variables X i represent the velocity of particles). We want to study the distribution of the {X i } for a fixed value E of the total intensive energy. By Sanov's theorem 1.9, we know that the asymptotic distribution of {X i } will concentrate as n → ∞ to the maximumentropy distribution satisfying the constraint:

ν(E) ≡ arg min ν∈M + 1 (R) s.t. ν(dx)E(x)=E [D KL (ν|µ)].
(1.77)

The solution to this variational problem is quite easy. We introduce a Lagrange multiplier η (our choice of notation will become clear very soon) to enforce the constraint, and we reach that

ν(E) = ν η (E) ≡ arg min ν∈M + 1 (R) [η ν(dx)E(x) + D KL (ν|µ)], (1.78)
with η = η(E) chosen such that ν η (E)(dx)E(x) = E. It is straightforward to solve eq. (1.78) and we reach:

ν(E)(dx) = µ(dx)e -ηE(x) Z , (1.79)
with Z chosen to ensure normalization of the distribution. Eq. (1.79) is precisely the Gibbs-Boltzmann measure of statistical physics we introduced in Section 1.1.4: it is the distribution that maximizes entropy at a fixed energy level. In statistical physics, the canonical ensemble consists in studying the dual problem: we fix a η > 0 (usually called inverse temperature), and we study the distribution of eq. (1.79). This description is dual (and thus equivalent) to the path we took here, with the energy given by E = Z -1 µ(dx) E(x) e -ηE(x) .

Application II: large deviations and the semicircle law -Large deviations are wellsuited to describe asymptotic phenomena that frequently appear in the physics literature. A good example is the derivation of Theorem 1.6 when the entries of the Wigner matrix are i.i.d. Gaussian (real or complex). Indeed, let us consider a Gaussian Wigner matrix J ∈ H n (K).

In this case, the eigenvalues {λ i } n i=1 of J are independent of the eigenvectors (which are Haar distributed). It is then easy to see by a change of variables of the original Gaussian matrix measure to the eigenvalues that (see e.g. Proposition 4.1.1 of [START_REF] Greg W Anderson | An introduction to random matrices[END_REF]):

P(dλ 1 , • • • , dλ n ) = 1 Z n i<j |λ i -λ j | β e -βn 4 n i=1 λ 2 i n i=1 dλ i . (1.80)
As we saw in Section 1.5.1, a particularly important quantity to study in random matrix theory is the empirical spectral distribution (ESD) ν n ≡ n -1 n i=1 δ λ i . Eq. (1.80) can be interpreted as the Gibbs-Boltzmann measure (at temperature T = 1) of an interacting gas of particles, subject to a logarithmic repulsion potential and a confining parabolic potential, resulting in the energy:

1 n 2 E({λ i }) ≡ β 4 ν n (dx) x 2 - β 2 x =y ν n (dx)ν n (dy) ln |x -y|.
Because of the logarithmic repulsion potential, this representation is usually called in the physics literature a Coulomb gas. Let us wear a physicist's hat for a moment to state an informal argument that will allow us to derive the large deviations of ν n from the previous description. As we know from Sanov's theorem 1.9, we can write, with ρ n (x) the density of ν n :

n i=1 dλ i δ ρ n (x) - 1 n n i=1 δ(x -λ i ) exp -n dx ρ n (x) ln ρ n (x) .
(1.81) Therefore we reach from eqs. (1.80) and (1.81):

P [ν n ] 1 Z n e -n 2 β 4 dx ρn(x) x 2 -β 2 x =y dx dy ρn(x)ρn(y) ln |x-y| -n dx ρn(x) ln ρn(x)
.

(1.82)

Note that as the large deviations described by Sanov's theorem 1.9 are in the scale n while the energy E({λ i }) scales as n 2 , this latter term will be the only contribution to the large deviations at leading order! In physics terms, the free energy of this Coulomb gas is dominated solely by the energetic component at leading order, while we can discard the entropic contribution. This informal argument derived from eq. (1.82) leads to the conjecture thaw the law of ν n satisfies a large deviation principle, in the scale n 2 , with rate function:

I(ν) ≡ - β 4 ν(dx) x 2 + β 2 ν(dx)ν(dy) ln |x -y| -C, (1.83)
with C chosen such that inf ν∈M + 1 (R) I(ν) = 0 16 . This statement is indeed rigorous, and its proof is done in [START_REF] Ben | Large deviations for Wigner's law and Voiculescu's non-commutative entropy[END_REF]. By a straightforward computation left to the reader (done e.g. in [START_REF] Livan | Introduction to random matrices: theory and practice[END_REF], or in [START_REF] Ben | Large deviations for Wigner's law and Voiculescu's non-commutative entropy[END_REF]), one can check that σ s.c. = arg min ν∈M + 1 (R) I(ν), and I(σ s.c. ) = 0, (1.84)

and that this minimum is unique. Therefore we have re-derived Theorem 1.6 as a simple corollary of our large deviations result! Our result is actually much stronger, as we showed that the convergence of the spectral measure happens in the scale e Θ(n 2 ) .

High-dimensional "spherical" integrals

In this section we introduce quantities known as spherical integrals, which can be seen as the equivalent of Laplace/Fourier transforms in the context of random matrices. Precisely, we consider integrals of the type (DO is the Haar measure on the compact group U β (n)) Closed formula at any n -For the unitary group, Harish-Chandra [HC57], followed by Itzykson and Zuber [START_REF] Itzykson | The planar approximation[END_REF], derived explicit formulas for these integrals, valid for any dimension 17 . For this reason we will generally refer to integrals of the type of eq. (1.85) as Harish-Chandra-Itzykson-Zuber (HCIZ) integrals. However these formulas are quite involved functions of determinants, and can not easily be used to compute the high-dimensional asymptotics.

I n (A, B) ≡ U β (n) DO exp{nTr[AOBO † ]}, ( 1
We now present results on the high-dimensional limit of different classes of HCIZ integrals. We give references to the literature in which these statements were derived or proven.

Rank-one HCIZ integrals

We start by the simplest case in which one of the two matrices A, B of eq. (1.85) has rank one. This class of spherical integrals will be important in different approaches taken in this thesis, especially in Chapters 2 and 8.

Theorem 1.11 (Rank-one HCIZ integral [GM05])

Let J ∈ H n (K) be a matrix such that the Empirical Spectral Distribution (ESD) of J: ρ n ≡ n -1 n i=1 δ λ i (J) converges a.s. (in the weak sense) as n → ∞ to a distribution ρ with compact support. We also assume that the largest eigenvalue λ max (J) converges a.s. as n → ∞ to x ∈ R. Let θ ≥ 0, and µ n be the uniform measure on S n-1 β . Then

I(θ) ≡ lim n→∞ 1 n ln µ n (de) exp βθn 2 e † Je = β 2 inf γ≥θx γ -ρ(dλ) ln(γ -θλ) - β 2 . (1.86)
Proof of Theorem 1.11 -We give here a simple heuristic derivation of the theorem. Its mathematical proof follows essentially the same lines, and can be found in [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF]. Let us denote I n (θ) the LHS of eq. (1.86) before taking the limit n → ∞. We renormalize it as:

I n (θ) = 1 n ln µ n (de) exp βθn 2 e † Je = 1 n ln de δ( e 2 -n) exp βθ 2 e † Je de δ( e 2 -n) .
We can then introduce a Lagrange multiplier γ to enforce the condition e 2 = n. At leading order, one obtains:

I n (θ) = 1 n ln dγ K n dx e βθ 2 x † Jx+ βγ 2 (n-i |x i | 2 ) 1 n ln dγ K n dx e βγ 2 (n-i |x i | 2 ) = inf γ 1 n ln dx e βθ 2 x † Jx+ βγ 2 (n-i |x i | 2 ) exp{ βn 2 (1 + ln 2π))} , = inf γ βγ 2 - β 2n ln det γI n -θJ - β 2 .
(1.87) Indeed, one checks easily that the extremum over the Lagrange multiplier γ actually corresponds to an infimum, and this infimum is made over all γ such that γI n -θJ is positive definite, i.e. γ > θλ max (J). Since λ max (J) → x by hypothesis, we obtain from the n → ∞ limit of eq. (1.87):

I(θ) ≡ lim n→∞ I n (θ) = inf γ>θx βγ 2 - β 2 ρ(dλ) ln(γ -θλ) - β 2 .
Recall indeed that ρ is the LSD of J. This ends the proof.

Phase transition in rank-one HCIZ integrals

Let us make a few remarks on the form of Theorem 1.11, which will be particularly useful in Chapter 8 when applying these results to derive large deviations of the eigenvalues of random matrices. We denote γ(θ) the solution to the infimum in eq. (1.86). If this infimum arises for γ(θ) > θx, then it satisfies the saddle-point equation

ρ(dλ) γ(θ) -θλ = 1.
(1.88)

Eq. (1.88) then has the solution γ(θ) = θS -1 ρ (-θ), as long as -S ρ (x) ≥ θ, where S ρ is the Stieltjes transform of ρ, introduced in Section 1.5.1 18 . If rather θ > -S ρ (x) then γ "sticks" to the solution γ = θx. In the end, we can compute I n (θ) in the small-θ (or high-temperature in the physics language) phase θ ≤ θ c (x) ≡ -S ρ (x):

I(θ) = βθ 2 R ρ (θ) - β 2 ρ(dλ) ln[1 + θR ρ (θ) -θλ],
in which R ρ (x) ≡ S -1 ρ (-x) -x -1 is the R-transform of ρ, cf. Section 1.5.1. By taking the derivative of this last expression with respect to θ it is easy to show that it simplifies to:

I(θ) = β 2 θ 0 R ρ (u)du.
(1.89)

In the "low-temperature" phase θ ≥ θ c (x) = -S ρ (x) one rather has

I(θ) = β 2 -1 + θx -ln θ -ρ(dλ) ln(x -λ) .
(1.90)

Note that in both phases it is clear that I(θ) is concentrating (or "self-averaging" in the physics language), as it only depends on J via its ESD and its largest eigenvalue, which we assumed converge a.s. Moreover its series expansion around θ = 0 is related to the free cumulants of ρ, cf. eq. (1.65):

I(θ) = β 2 ∞ p=1 c p (ρ) p θ p , (1.91)
which is valid in the high-temperature phase θ ≤ θ c (x).

Finite-rank k ≥ 1

Let us now assume that one of the two matrices A, B in eq. (1.85) has finite rank k ≥ 1 as n → ∞. Then one can state very natural generalizations of the previous results to this setting.

Informally, the integral in this case behaves at leading order as k decoupled rank-one integrals described by Theorem 1.11, which is shown in detail in the following theorem.

Theorem 1.12 (Finite-rank HCIZ integral [GM05, CŚ07])

Let J ∈ H n (K) as in Theorem 1.11. Let θ 1 , • • • , θ k ≥ 0,
I k (θ 1 , • • • , θ k ) ≡ lim n→∞ 1 nk ln E exp k a=1 βθ a n 2 e † a Je a = 1 k k a=1 I 1 (θ a ), (1.92)
in which the expectation is done over {e 1 , • • • , e k } and I 1 is the asymptotic of the rank-one case given in Theorem 1.11.

Rectangular spherical integrals

Interestingly, the asymptotic results we described so far on spherical integrals can be generalized to a class of "rectangular" spherical integrals. We state this generalization for rank-one integrals, but it can be written for finite-rank integrals along the same lines we described for usual spherical integrals. These rectangular HCIZ integrals will be useful at several points in this thesis, in very different contexts, e.g. in Chapters 2, 6 and 8.

Theorem 1.13 (Rank-one "rectangular" HCIZ integral [Kab08a, Kab08b, BG11])

Let L ∈ K m×n , such that J ≡ L † L satisfies hypothesis (ii) of Model S. We assume that n, m → ∞ with m/n → α > 0. Assume moreover that λ max (J) converges a.s. as n → ∞ to x ≥ 0. Let θ ≥ 0, and µ n be the uniform measure on S n-1 β . Then

I rect. (θ) ≡ lim n→∞ 1 n ln µ m (de)µ n (df) exp β √ αθn(e † Lf) = β 2 inf γe,γ f >0 γeγ f ≥θ 2 x γ f + αγ e -(α -1) ln γ e -ρ(dλ) ln(γ e γ f -θ 2 λ) - 1 + α 2 .
The function I rect. (θ) is closely related to the rectangular R-transform introduced in [BG11]. While we do not detail the definitions of this function, in analogy with eq. (1.91) we define a set of coefficients Γ p (α, ρ) (analogous to the free cumulants in this rectangular context) by the analytical expansion around θ = 0:

I rect. (θ) = β 2 ∞ p=1 Γ p (α, ρ) p θ 2p . (1.93)
As in the symmetric case, the function I rect. (θ) might admit phase transitions corresponding to a "saturation" of the Lagrange multipliers γ e , γ f at the largest eigenvalue, i.e. when γ e γ f = θ 2 x. Such transitions will be derived and used in Chapter 8.

Diverging rank k → ∞

If one of the two matrices A, B has extensive rank in n while the other one has a diverging rank k = k(n) → ∞ as n → ∞, the computations become a priori more complicated. While we will not directly use these results in this dissertation, solutions have been found in different regimes of k(n) and the resulting picture is quite simple. Indeed, for k = O(n), the HCIZ integral behaves as in the finite-rank case: its limit is given by k decoupled rank-one integrals. This was first shown in [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF] for k(n) = O(n 1/2-) (for arbitrary > 0), and later generalized in [START_REF] Collins | New scaling of Itzykson-Zuber integrals[END_REF] to any k(n) = O(n).

Theorem 1.14 (Slowly-diverging-rank HCIZ integral [GM05, CŚ07])

Let J ∈ H n (K) as in Theorem 1.11 and

k = k(n) such that lim n→∞ k(n) = +∞ and k(n) = O(n). Let θ 1 , • • • , θ k ≥ 0 such that k -1 k a=1 δ θa converges weakly to µ ∈ M + 1 (R)
, and we assume that there exists C > 0 that uniformly bounds θ a ≤ C. We consider e 1 , • • • , e k the first k columns of a Haar-distributed matrix O ∈ U β (n). Then

I(µ) ≡ lim n→∞ 1 nk(n) ln E exp k(n) a=1 βθ a n 2 e † a Je a = µ(dθ)I 1 (θ), (1.94)
in which the expectation is done over {e 1 , • • • , e k } and I 1 is the asymptotic of the rank-one integral given in Theorem 1.11.

On the other hand the case k(n) = Θ(n) is much more involved and can not be reduced to the rank-one case. It was first studied by Matytsin [START_REF] Matytsin | On the large-N limit of the Itzykson-Zuber integral[END_REF] and later proven in [START_REF] Guionnet | Large deviations asymptotics for spherical integrals[END_REF].

Theorem 1.15 (Extensive-rank HCIZ integral [Mat94, GZ02])

Let n ≥ 1, and A, B ∈ H n (K). We assume that the ESDs of A and B both converge a.s. as n → ∞ (in the weak sense) to probability measures µ A , µ B ∈ M + 1 (R). Then:

lim n→∞ 1 n 2 ln U β (n) DO exp βn 2 Tr[AOBO † ] = β 2 - 3 4 + J(µ A ) + J(µ B ) - 1 2 inf ρ,m 1 0 dt dx m t (x) 2 ρ t (x) + π 2 3 ρ t (x) 3
Let us denote the measure-valued process µ t (dx) ≡ ρ t (x)dx. The infimum in the above equation is done over m, ρ satisfying the Euler continuity equation ∂ t ρ t (x) + ∂ x m t (x) = 0, and the boundary conditions µ 0 = µ A and µ 1 = µ B . This result assumes finally that J(µ A ) and J(µ B ) are finite, where for any µ ∈ M + 1 (R) we define:

J(µ) ≡ 1 2 µ(dx) x 2 - 1 2 µ(dx)µ(dy) ln |x -y|.
Chapter 2

Revisiting high-temperature expansions

"One of the principal objects of theoretical research in my department of knowledge is to find the point of view from which the subject appears in its greatest simplicity." J. Willard Gibbs, Proceedings of the American Academy of Arts and Sciences (1881).

Disclaimer -In this chapter, we revisit one of the oldest theoretical tools of the statistical physics of disordered systems: high-temperature expansions, initiated by Plefka [START_REF] Plefka | Convergence condition of the TAP equation for the infiniteranged ising spin glass model[END_REF] and refined by Georges and Yedidia [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF]. This chapter is based on the published work [MFC + 19], and serves both as a presentation of its results and as a pedagogical introduction to high-temperature expansions, which will prove useful in particular to tackle the involved problem of extensive-rank matrix factorization in Chapter 3.

Organization of the chapter and main results

Beyond i.i.d. couplings -As we saw in Chapter 1, the statistical physics approach, combining the TAP free energy and message-passing algorithms, is especially powerful when the coupling constants in the underlying statistical model are distributed as i.i.d. variables. This is, of course, a strong limitation and many strategies have been designed to improve on it, and we introduced some of them in Section 1.4.3: the adaptive TAP (adaTAP) method [START_REF] Opper | Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling[END_REF][START_REF] Opper | Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach[END_REF], approximation schemes such as Expectation Consistency (EC) [START_REF] Thomas P Minka | Expectation propagation for approximate Bayesian inference[END_REF][START_REF] Opper | Expectation consistent approximate inference[END_REF] and the recent improvements of AMP such as Vector Approximate Message Passing (VAMP) and its many variants [SRF16, OCW16, COFW16, MP17, RSF17]. Given these different strategies, one may wonder when they actually lead to asymptotically exact inference. In this chapter, we address this question using high-temperature expansions.

A short history of high-temperature expansions -High-temperature expansions at fixed order parameters, denoted in this thesis as Plefka or Plefka-Georges-Yedidia (PGY) expansions, have historically been an important tool of the study of disordered systems. In the context of spin glass models they have been introduced by Plefka [START_REF] Plefka | Convergence condition of the TAP equation for the infiniteranged ising spin glass model[END_REF] for the Sherrington-Kirkpatrick (SK) model, and have been subsequently generalized by Georges and Yedidia [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF]. This latter paper provides a systematic way to compute high-temperature (or high-dimensional) expansions of the TAP free entropy, which is defined for a fixed value of the order parameters.

Our goal -One important aim of the present chapter is to apply this method to a general class of inference problems with pairwise interactions, in which the coupling constants are not i.i.d., but rather possess correlations, satisfying a rotational invariance property. This encapsulates many widely studied models, such as Restricted Boltzmann Machines (RBMs) or Generalized Linear Models (GLMs) with correlated data matrices, introduced in Section 1.1. In particular, we generalize earlier and inspirational work by Parisi and Potters [START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF], who computed the self-consistent equations for the marginals in Ising models with orthogonal couplings via a resummation of the infinite series given by the high-temperature expansion. In our general setting, we shall show that a similar resummation yields the EC, adaTAP and VAMP formalisms: they are all equivalent as we saw in Section 1.4.3, and we conjecture that they are exact in the thermodynamic limit in the replica symmetric phases. On the way to these conclusions we will uncover diagrammatical results in connection with free probability and random matrix theory.

Organization of the chapter and main results -

• Spherical models with rotationally-invariant couplings -We first provide a pedagogical introduction to "high-temperature" PGY expansions, inspired by the work of Georges-Yedidia [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF] for Ising models, adding some new results on the diagrammatics of the expansions. This yields a general framework that encapsulates many known properties of systems sharing this pairwise structure. In the corresponding section 2.2, we focus on spherical models and we generalize the seminal works of [START_REF] Marinari | Replica field theory for deterministic models: I. binary sequences with low autocorrelation[END_REF][START_REF] Marinari | Replica field theory for deterministic models. II. a non-random spin glass with glassy behaviour[END_REF][START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF]. While these works studied Ising models with orthogonal couplings, we consider spherical models with more general rotationally invariant couplings. We examine two types of models: "symmetric" models with an interaction of the type x Jx , in which J follows Model S, and "bipartite" models with interactions of the type h Fx, in which F follows Model R. This encapsulates orthogonal couplings, but can also be applied to other random matrix ensembles such as the Gaussian Orthogonal Ensemble (GOE), the Wishart ensemble, and many others. Using diagrammatic results that we derive with random matrix theory, we conjecture a resummation of the PGY expansion giving the Gibbs free entropy in these models. Our results are in particular consistent with the findings of classical works for Gaussian couplings [START_REF] Plefka | Convergence condition of the TAP equation for the infiniteranged ising spin glass model[END_REF] and orthogonal couplings [START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF]. • PGY expansion for statistical models with correlated couplings -Section 2.3.1 is devoted to the description of the Plefka expansion for different statistical models and inference problems which possess a coupling or data matrix that has rotation invariance properties. We consider e.g. models similar to the spherical models of Section 2.2, but with generic prior distributions on the underlying variables. Our main conjecture for this part can be stated as the following:

Conjecture 2.1 (Informal)

For statistical models of symmetric or bipartite interactions with coupling matrices that satisfy respectively Model S or Model R (left and right rotation-invariance), the three equivalent approximations introduced in Section 1.4.3: Expectation Consistency, adaptive TAP and Vector Approximate Message Passing (generically referred to as EC approximations), are exact in the large size limit in the high temperature phase.

We believe that the validity of the above conjecture extends beyond the high temperature phase. In particular that it is correct for inference problems in the Bayes-optimal setting, and more generally anytime the system is in a replica symmetric phase as defined in Chapter 1.

The approximation behind EC approximations can be checked order by order using our PGY expansion and its resummation. Using diagrammatic results, we show that the EC approximations are exact for these models in the large size limit. We then apply our generic results to different situations, e.g. the Hopfield model [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF], compressive sensing, and a very broad class of bipartite models, which includes the Generalized Linear Models (GLMs) with correlated data matrices. We emphasize that we are able to derive the free entropy of all these models using very generic arguments relying only on the rotational invariance of the problem. • The TAP equations and message-passing algorithms -In Section 2.3.2, we show that the TAP equations that we derived by PGY expansion in rotationally invariant models can be understood as the fixed point equations of message-passing algorithms. In the converse way, many message-passing algorithms can be seen as an iteration scheme of these TAP equations. While this was known in several models in which the underlying data matrix was assumed to be i.i.d. (cf. Chapter 1), using our diagrammatic results we are able to generalize these correspondences to correlated models. We argue that the stationary limit of the Vector Approximate Message Passing (VAMP) algorithm [START_REF] Sundeep Rangan | Vector approximate message passing[END_REF] for compressed sensing with correlated matrices gives back our TAP equations derived via PGY expansion. Even more generally, the Generalized Vector Approximate Passing (G-VAMP) algorithm [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF], defined for the very broad class of Generalized Linear Models with correlated matrices, yields fixed point equations that are equivalent to our PGY-expanded TAP equations. • Diagrammatics of the expansion -Our results are largely based on a precise control on the diagrammatics of the PGY expansion for rotationally invariant matrices, which are presented in Section 2.4. In particular, we leverage mathematical results on HCIZ integrals (cf. Section 1.5.3) to argue that only a very specific class of diagrams contributes to the expansion.

Some technicalities or generalizations will be deferred to Appendix A.

Additional notation -In this chapter we will sometimes write A n B n to denote that

1 n ln A n = 1 n ln B n + O n (1).

Plefka-Georges-Yedidia expansion step-by-step

Pedagogical derivation for a spherical SK-like model

In this section n ≥ 1, σ > 0, and we define the following pairwise interaction Hamiltonian on S n-1 (σ √ n), the n-th dimensional sphere of radius σ √ n:

H J (x) ≡ - 1 2 x Jx = - 1 2 1≤i,j≤n J ij x i x j , x ∈ S n-1 (σ √ n).
(2.1)

The coupling matrix J is a n×n symmetric random matrix, assumed to be rotationally invariant in the sense of Model S. The interest of this simple "toy" model (which is a spherical "SK-like" model) is that, as we will see, its free energy can be computed exactly so that we can easily control the steps of the expansion by comparison with the exact solution.

Direct free entropy computation

The Gibbs measure associated to the energy of eq. (2.1) at inverse temperature η ≥ 0, and the corresponding free entropy, are 1 :

G η,J (dx) ≡ 1 Z η,J e η 2 i,j J ij x i x j δ( x 2 -nσ 2 )dx, Φ J (η) ≡ lim n→∞ 1 n ln Z η,J . (2.2)
The partition function of the model can directly be written as a function of the spherical HCIZ integrals we studied in Section 1.5.3. We use more precisely Theorem 1.12, and the expression of the spherical integral in the two phases, eqs. (1.89) and (1.90). We reach that in the high temperature phase η ≤ η c ≡ -σ -2 S ρ (λ max ) the free entropy can be expressed as:

Φ J (η) = 1 + ln 2πσ 2 2 + 1 2 ησ 2 0 R ρ (x)dx, in which R ρ (x) ≡ S -1 ρ (-x)-x -1
is the R-transform of ρ, cf. Section 1.5. In the low temperature phase η ≥ η c = -σ -2 S ρ (λ max ) one rather has

Φ J (η) = 1 2 ln 2π + λ max ησ 2 -ln η -ρ(dλ) ln(λ max -λ) .
Note that in both phases the free entropy can formally be expressed as:

Φ J (η) = 1 2 ln 2π + 1 2 inf γ≥ηλmax γσ 2 -ρ(dλ) ln(γ -ηλ) , (2.3)
a formulation which is more compact and easier to implement algorithmically.

Plefka expansion and the Georges-Yedidia formalism

A more generic way to compute the free entropy is to follow the formalism of [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF] to perform a high-temperature Plefka expansion [START_REF] Plefka | Convergence condition of the TAP equation for the infiniteranged ising spin glass model[END_REF]. The principle is simple: expand the TAP free entropy of the Gibbs measure of eq. (2.2) at low η, in the high temperature phase. More precisely we will compute the free entropy imposing constraints on the first two moments

2 x i η = m i and x 2 i η = v i + m 2 i .
A set of parameters {m i , v i } will thus determine a free entropy value, and the comparison with the direct calculation will be made by maximizing the free entropy with respect to {m i , v i }. The spherical constraint x 2 = nσ 2 thus becomes:

σ 2 = 1 n n i=1 [v i + m 2 i ].
(2.4)

All-in-all, we wish to expand at low η the free entropy at a given realization of the disorder (also called single-graph free entropy)

Φ J (η, m, v) ≡ 1 n n i=1 λ i m i + γ i 2 (v i + m 2 i ) + 1 n ln e -ηH J (x)- n i=1 λ i x i + γ i 2 x 2 i dx, (2.5)
in which we implicitly extremize over the Lagrange multipliers {λ i } and {γ i } (for lightness of the notations we will keep their dependency on η explicit only when needed).

The Georges-Yedidia operator -In order to perform the expansion we introduce the operator U , defined in [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF]:

U (η, J) ≡ H J -H J η + n i=1 ∂ η λ i (η)(x i -m i ) + 1 2 n i=1 ∂ η γ i (η)[x 2 i -v i -m 2 i ]. (2.6)
Note that by definition we have U η = 0. One can easily check the following relation for any observable O, which is the main property of the operator U :

∂ O η ∂η = ∂O ∂η η -OU η .
2 The notation • η indicates an average over the Gibbs measure at inverse temperature η.

Therefore since the magnetizations {m i } and variances {v i } do not depend on η this implies:

0 = ∂ η x i η = -x i U η = -(x i -m i )U η , 0 = ∂ η x 2 i η = -x 2 i U η = -(x 2 i -v i -m 2 i )U η .
Considering the previous results one can easily compute the derivative of U :

∂U ∂η = U 2 η + n i=1 ∂ 2 η λ i (η)(x i -m i ) + 1 2 n i=1 ∂ 2 η γ i (η)[x 2 i -v i -m 2 i ].
Equipped with all the previous relations we can compute the successive derivatives of the TAP free entropy Φ J (η, m, v). The reader can easily check that for any η ≥ 0:

                 n∂ η Φ J = -H J η , (2.7a) n∂ 2 η Φ J = H J U η = U 2 η , (2.7b) n∂ 3 η Φ J = -U 3 η , (2.7c) n∂ 4 η Φ J = U 4 η -3 U 2 2 η -3 n i=1 ∂ 2 η λ i U 2 (x i -m i ) η + ∂ 2 η γ i 2 U 2 [x 2 i -v i -m 2 i ] η . (2.7d)
In principle one can push this strategy up to arbitrary order p ≥ 1. Unfortunately, as already noticed in [GY91], we do not know of any closed expression for the order-p derivative. Nevertheless, we are now ready to compute the first orders of the high-temperature (or small-η) expansion of the free entropy. Note that in this expansion the Lagrange parameters {λ i (η), γ i (η)} will always be considered at η = 0.

First orders of the expansion -First of all, taking η = 0 one easily reaches from eq. (2.5):

Φ J (η = 0) = 1 2 ln 2π + 1 n n i=1 γ i 2 (v i + m 2 i ) - 1 2 ln γ i + λ i m i + λ 2 i 2γ i .
After extremization over the Lagrange multipliers {λ i , γ i } this yields:

Φ J (η = 0) = 1 + ln 2π 2 + 1 2n n i=1 ln v i .
At order 1 we have from eq. (2.7):

∂ η Φ J η=0 = - 1 n H J η=0 = 1 2n i,j J ij m i m j + 1 2n n i=1 J ii v i . (2.8)
For any η, we can write the Maxwell-type relations:

γ i (η) = n ∂ v i Φ J (η), m i γ i (η) + λ i (η) = n ∂ m i Φ J (η).
(2.9)

These relations plugged in eq. (2.8) lead to

∂ η γ i (η = 0) = J ii and ∂ η λ i (η = 0) = j( =i) J ij m j .
We then obtain the U operator at η = 0 from eq. (2.6):

U (η = 0, J) = - 1 2 i =j J ij (x i -m i )(x j -m j ).
(2.10) We can then directly apply eq. (2.7) to reach the order 2:

• • • • • • • • (a) A simple cycle of order p = 8. • • • • • • • • • (b) A diagram at order p = 12.
1 2 ∂ 2 η Φ J η=0 = 1 2n U 2 η=0 = 1 4n i =j J 2 ij v i v j .
(2.11)

At order 3 we obtain:

1 3! ∂ 3 η Φ J η=0 = - 1 6n U 3 η=0 = 1 6n i,j,k J ij J jk J ki v i v j v k + O n (1),
in which the sum is made over pairwise distinct i, j, k indices. At order 4 we reach3

1 4! ∂ 4 η Φ J η=0 = 1 8n i,j,k,l J ij J jk J kl J li v i v j v k v l + O n (1),
where again, i, j, k, l are pairwise distinct indices.

Larger orders -As we mentioned, the PGY expansion can not give analytic results for arbitrary perturbation orders. Nevertheless, our results up to order 4 lead to the natural conjecture:

Φ J (η) = 1 + ln 2π 2 + 1 2n n i=1 ln v i + η 2n i =j J ij m i m j (2.12) + 1 n ∞ p=1 η p 2p i 1 ,••• ,ip pairwise distinct J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 p α=1 v iα .
Note that in order to obtain this formula, we took the n → ∞ limit at every perturbation order in η, which is part of the implicit assumptions of the PGY expansion. The terms of this perturbative expansion can be represented diagrammatically as simple cycles of order p, see Fig. 2.1a.

Elementary diagrammatics -Generically, at any order in the expansion one can construct a diagrammatic representation of the contributing terms, as shown in Fig. 2.1. While we will extensively discuss these diagrammatics in Section 2.4, let us note that the only remaining terms in eq. (2.12) correspond to simple cycles, as depicted in Fig. 2.1a. For the case of orthogonal couplings, this dominance of simple cycles was already noted in [START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF]. Note that many other diagrams may not be negligible in the limit n → ∞ (e.g. the "cactus" diagram of Fig. 2.1b), however as we will detail in Section 2.4, they will cancel out in the free entropy calculation4 .

Comparison with the exact solution -In the high temperature phase, the solution to the maximization of eq. (2.12) under m is the paramagnetic solution m = 0. Furthermore, we expect that the {v i } that maximize the free entropy of eq. (2.12) are homogeneous, that is ∀i, v i = v. The spherical constraint of eq. (2.4) thus gives v = σ 2 . We can compare the result of the resummation of simple cycles, eq. (2.12) with the exact results of eq. (2.3) in the paramagnetic phase. For these two results to agree, we need the generating function for simple cycles to be related to the R-transform of ρ by:

E 1 n ∞ p=1 η p σ 2p 2p i 1 ,••• ,ip pairwise distinct J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 = 1 2 ησ 2 0 R ρ (x)dx + O n (1), (2.13)
in which the outer expectation is with respect to the distribution of J. In particular, an orderby-order comparison yields that the free cumulants {c p (ρ)} p∈N , defined in Section 1.5, satisfy:

∀p ∈ N , c p (ρ) = lim n→∞ E 1 n i 1 ,••• ,ip pairwise distinct J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 .
(2.14)

Using rigorous results of [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF], we can prove a stronger version of eq. (2.14), namely convergence in L 2 norm, that we give as a theorem:

Theorem 2.2 (Simple cycles and free cumulants)

For any matrix J ∈ S n generated by Model S and any p ∈ N , one has:

lim n→∞ E 1 n i 1 ,••• ,ip pairwise distinct J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 -c p (ρ) 2 = 0.
We postpone the proof of this result, along with a much more detailed analysis of the diagrammatics, to Section 2.4. Assuming that we can invert the summation over p and the n → ∞ limit, Theorem 2.2 implies we have a stronger version eq. (2.13) with L 2 convergence, for small enough η:

1 n ∞ p=1 η p σ 2p 2p i 1 ,••• ,ip pairwise distinct J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 L 2 --→ n→∞ 1 2 ησ 2 0 R ρ (x)dx. (2.15)
This identity is a "resummation" of the single-graph free entropy of eq. (2.12)! As a final note we can use eq. ( 2.3) to write the resummation in an alternative form (dropping O n (1) terms):

1 n ∞ p=1 η p σ 2p 2p i 1 ,••• ,ip pairwise distinct J i 1 i 2 • • • J ipi 1 = 1 2 inf γ≥ηλmax γσ 2 -ρ(dλ) ln(γ -ηλ) - 1 + ln σ 2 2 . (2.16)
Note that this is true a priori only in the high temperature phase η < η c , so that both sides are analytic functions of η. In the next paragraph, we will investigate this discontinuity and show that it coincides with the instability of the paramagnetic solution in the TAP free entropy.

Stability of the paramagnetic phase

We can check the stability of the paramagnetic solution for η ≤ η c ≡ -σ -2 S ρ (λ max ). Recall that we must satisfy the norm constraint

v = σ 2 -(1/n) i m 2 i .
Solely as a function of m the free entropy therefore reads, up to O n (1) terms:

Φ J (η, m) = 1 + ln 2π 2 + 1 2 ln σ 2 - 1 n n i=1 m 2 i + η 2n i =j J ij m i m j + G ρ η σ 2 - 1 n n i=1 m 2 i ,
in which G ρ is the integrated R-transform of ρ, defined for all 0 ≤ x ≤ -S ρ (λ max ):

G ρ (x) ≡ 1 2 x 0 du R ρ (u).
(2.17)

The Hessian of the extensive free entropy nΦ J at the paramagnetic solution m = 0 is thus:

Hess n (η) ≡ n ∂ 2 Φ J ∂m i ∂m j m=0 = - δ ij σ 2 [1 + ησ 2 R ρ (ησ 2 )] + ηJ ij + O n (1).
The paramagnetic solution is stable as long as the Hessian is negative. This is true as long as η < η c , at which point the spectrum of Hess n (η c ) touches zero by definition of η c . Our PGY expansion allows thus to compute the free entropy in the high temperature phase (which is paramagnetic), coherently with the direct computation results. More generically, as shown by Plefka [START_REF] Plefka | Convergence condition of the TAP equation for the infiniteranged ising spin glass model[END_REF] in the SK model, Hess n (η) is related to the inverse susceptibility matrix of the system, and thus the singularity of the Hessian implies a singularity of the free entropy (i.e. a phase transition).

Validity of the expansion and stability of the replica symmetric solution -

Beyond the paramagnetic regime, it is an open question to relate the range of validity η c of the Plefka expansion and the de Almeida-Thouless condition that characterizes the local stability of the replica symmetric solution (see [START_REF] Jrl | Stability of the Sherrington-Kirkpatrick solution of a spin glass model[END_REF] for its original derivation, and [START_REF] Kabashima | Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF] for examples of its applications in inference problems). The equivalence of these two conditions was shown in the seminal paper of Plefka [START_REF] Plefka | Convergence condition of the TAP equation for the infiniteranged ising spin glass model[END_REF] in the Sherrington-Kirpatrick model. It is tedious but straightforward to generalize this conclusion to a model with Ising spins x i = ±1 and a Hamiltonian given by eq. (2.1) with a rotationally-invariant coupling matrix J. However, investigating the relation between these two conditions in general models appears to be an open problem, and does not enter the scope of this thesis.

The free cumulant series -We performed an expansion of Φ J (η) close to η = 0, which implies that this expansion is thus valid in the region (0, η c ), in which η c = -σ -2 S ρ (λ max ) is the first non-analyticity of Φ J (η), see eq. ( 2.3) and the discussion above. However there exists spectrums for which the function R ρ (x) can be analytically extended beyond x c ≡ -S ρ (λ max ), e.g. Wigner's semicircle for which R s.c. (x) = x and x c = 1. Yet, one has to be careful that this does not imply that the free entropy Φ J (η) is analytic beyond η c , and even in this case our Plefka expansion is a priori only valid up to η = η c .

Generalization to a bipartite model

In this section n, m ≥ 1, σ x , σ h > 0, and we will consider the thermodynamic limit in which n, m → ∞ with m/n → α > 0. We define the following bipartite Hamiltonian:

H L (h, x) ≡ -h Lx = - m µ=1 n i=1 L µi h µ x i , h ∈ S m-1 (σ h √ m), x ∈ S n-1 (σ x √ n). (2.18)
The coupling matrix L ∈ R m×n is assumed to be rotationally-invariant in the sense of Model R.

Direct free entropy computation

The calculation for this bipartite case is very similar to the direct calculation of Section 2.2.1, and we introduced in Section 1.5.3 the corresponding "rectangular" HCIZ integrals, cf. in particular Theorem 1.13. We obtain that for all values of η the free entropy can be expressed as:

Φ L (η) ≡ lim n→∞ 1 n ln µ m (dh) µ n (dx) e ηh Lx , (2.19) = 1 + α 2 ln 2π + 1 2 inf γ h ,γx>0 γ h γx≥η 2 λmax αγ h σ 2 h + γ x σ 2 x -(α -1) ln γ h -ρ(dλ) ln(γ x γ h -η 2 λ) ,
where ρ is the asymptotic eigenvalue distribution of L L (see Model R).

Plefka-Georges-Yedidia expansion

The PGY expansion for this model is very similar to the symmetric model we just studied. We constraint the first and second moments as

h µ = m h µ , x i = m x i , h 2 µ = v h µ + (m h µ ) 2 and x 2 i = v x i + (m x i ) 2 . The spherical constraints impose σ 2 h = (1/m) m µ=1 [v h µ + (m h µ ) 2 ] and σ 2 x = (1/n) n i=1 [v x i + (m x i ) 2 ].
As in Section 2.2.1 one can study the diagrams that appear in the Plefka expansion. We show again the L 2 concentration of the simple cycles, and the negligibility of all other diagrams in the expansion. We state in more details these results for the bipartite case in Section 2.4.5. We obtain the following result, a counterpart to eq. (2.12) for bipartite models:

Φ L (η) = 1 + α 2 [1 + ln 2π] + α 2m m µ=1 ln v h µ + 1 2n n i=1 ln v x i + η n m µ=1 n i=1 L µi m h µ m x i (2.20) + 1 n ∞ p=1 η 2p 2p µ 1 ,••• ,µp pairwise distinct i 1 ,••• ,ip pairwise distinct L µ 1 i 1 L µ 1 i 2 L µ 2 i 2 • • • L µpip L µpi 1 p α=1 v h µα v x iα + O n (1),
in which indices {µ l } run from 1 to m and indices {i l } run from 1 to n. Assuming uniform variances at the maximum:

v h µ = v h , v x i = v x
, and comparing to eq. (2.19) in the paramagnetic phase, we obtain the correspondence, in the high temperature phase:

α ln σ 2 h + ln σ 2 x 2 + 1 n ∞ p=1 η 2p σ 2p h σ 2p x 2p µ 1 ,••• ,µp pairwise distinct i 1 ,••• ,ip pairwise distinct L µ 1 i 1 L µ 1 i 2 L µ 2 i 2 • • • L µpip L µpi 1 = - 1 + α 2 + 1 2 inf γ h ,γx αγ h σ 2 h + γ x σ 2 x -(α -1) ln γ h -ρ(dλ) ln(γ h γ x -η 2 λ) . (2.21)
Again, the n → ∞ limit is implicit, and the equality then holds in the sense of L 2 convergence.

PGY expansion for inference models

In this section we perform Plefka (or PGY) expansions for generic models of pairwise interactions, both symmetric and bipartite, and discuss the iterations of the resulting fixed point equations. First, let us recall the important different approximation schemes studied in Section 1.4.3, namely Expectation-Consistency (EC), adaptive TAP (adaTAP) and Vector Approximate Message Passing (VAMP). These three schemes (that we showed to be equivalent) allow to compute the free entropy of the inference models we will consider, and will be crucial in our discussion. Indeed, one goal of this chapter is to provide a precise justification for the exactness of these schemes by leveraging random matrix theory. Let us briefly describe our strategy:

• In Section 2.3.1 we generalize the PGY expansion of Section 2.2 to inference models and highlight the main differences and assumptions of our method. This yields a precise and systematic justification of the TAP equations for rotationally invariant models. We apply these results to retrieve the TAP free entropy of the Hopfield model, compressed sensing, as well as Generalized Linear Models (GLMs), flagship high-dimensional inference models that we defined in Section 1.1.

• One then needs to maximize this free entropy, which yields fixed point equations: the celebrated TAP equations. Iterating them is in itself a challenge since different choices for the iteration scheme can lead to drastically different convergence properties. In Section 2.3.2 we relate message-passing algorithms, which have proven to be very successful both numerically and for theoretical studies, to the approximation schemes mentioned above, and to the PGY expansion. More precisely, we show that the stationary limit of the message-passing equations yields the TAP equations obtained by the PGY expansion.

PGY expansion in generic models of pairwise interactions

Models of symmetric pairwise interactions

A symmetric model with generic priors -Let us first consider a slight generalization of the model of eq. (2.1): the vector x is now drawn with independent components, each x i with distribution P i . They interact via a pairwise interaction (with a rotationally-invariant coupling matrix J), and are subject to an external field h. At a given inverse temperature η ≥ 0 and a fixed realization of J, the Gibbs-Boltzmann distribution of the spins is given as:

P η,J (dx) ≡ 1 Z J (η) n i=1 P i (dx i ) exp η 2 i,j J ij x i x j -η i h i x i . (2.22)
As before, we compute the large n limit of the free entropy Φ J (η) ≡ n -1 ln Z J (η) at fixed values of the magnetizations m i = x i and variances v i = (x i -m i ) 2 , using a PGY expansion. We impose these constraints with Lagrange multipliers {λ i } and {γ i }. Clearly the zero-th order term in η is different from the spherical case, and is given by:

Φ J (0) = 1 n i λ i m i + 1 2n i γ i (v i + m 2 i ) + 1 n ln i P i (dx i ) e -1 2 i γ i x 2 i -i λ i x i . (2.23)
At order 1 in η we obtain at leading order:

∂Φ J ∂η η=0 = 1 2n i,j J ij m i m j + 1 2n i J ii v i - 1 n i h i m i .
The operator U of Georges-Yedidia, defined in eq. (2.6), is the same as in the spherical case, that is eq. (2.10). This implies that many of the results obtained for the spherical case will transfer here directly. For instance the second order term is identical and given in eq. (2.11). At third order we obtain:

1 3! ∂ 3 Φ J ∂η 3 η=0 = 1 6n i 1 ,i 2 ,i 3 pairwise distincts J i 1 i 2 J i 2 i 3 J i 3 i 1 v i 1 v i 2 v i 3 + 1 6n i =j J 3 ij κ (3) i κ (3) j .
(2.24)

In this equation, we denoted κ (p) i the cumulant of order p of x i at η = 0. By the rotation invariance of Model S, the term i =j J 3 ij gives a negligible contribution to the free entropy. We shall therefore assume that the second part of the RHS of eq. (2.24) is negligible as n → ∞. This is correct provided that the possible correlations of the third order cumulants κ

(3) i with J do not drastically change the scaling of this term to make it thermodynamically relevant. This will be the case at all orders, as argued in Section 2.4: the cumulants of order p ≥ 3 will not contribute to the asymptotic free entropy! The first term of eq. (2.24) corresponds to a simple cycle of order 3 and is the same term that appeared in the spherical case.

We carry on the computation of the derivatives ∂ p η Φ J (η = 0). Given our remark above on highorder cumulants, we conjecture that the higher-order terms are different from the spherical model only in terms which are sub-leading in n. We will give more detail on the precise hypothesis behind this conjecture in Section 2.4.4. In the end, we reach the following conjecture for Φ J :

Φ J (η) = Φ J (0) + η 2n i,j J ij m i m j - η n i h i m i + 1 n ∞ p=1 η p 2p i 1 ,••• ,ip pairwise distincts J i 1 i 2 • • • J ipi 1 p α=1 v iα . (2.25)
For the remainder of this section we assume that the maximum of the free entropy of eq. (2.25) is attained for variables {v i } such that v i = v. This hypothesis can be argued as reasonable for many models, and we postpone this argumentation to the specific models we will consider. We obtain a resummation of the Plefka free entropy using the correspondence of eq. (2.15):

Φ J (η) = Φ J (0) + η 2n i,j J ij m i m j - η n i h i m i + 1 2 ηv 0 R ρ (u) du. (2.26)
Recall that Φ J (0) is given in eq. (2.23). As discussed in the spherical case, we expect this expansion of the free entropy to hold for η < η c , in which η c ≡ -v -1 S ρ (λ max ).

The PGY expansion to justify EC approximations -

The result of the PGY expansion in eq. (2.25) provides a systematic way to show the exactness of the EC approximations (cf. Section 1.4.3) for rotationally invariant models. For instance, as we saw in eq. (1.57), adaTAP amounts to assuming that at every order p ≥ 1 of perturbation in η, one can perform the calculation as if the statistics of the variables were Gaussian. This statement, which generalizes the Parisi-Potters result [START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF], is exactly what we argued to obtain eq. (2.25), using the diagrammatic analysis of Section 2.4. As such, our analysis provides a clear meaning to the EC approximations, by detailing which diagrams are negligible in the n → ∞ limit. This also justifies that the EC approximations are actually exact asymptotically for rotationally invariant models in the high temperature phase, which we summarized in Conjecture 2.1. We believe that this asymptotic exactness extends beyond the high temperature phase to any model in the replica symmetric phase, and the diagrammatic analysis provides a route to proving this statement.

Application to the Hopfield model -As a first application of our framework, we consider the Hopfield model [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF]. We let binary spins x ∈ {±1} n and the coupling matrix J is constructed out of p patterns, which are spin configurations ξ l ∈ {±1} n , for l ∈ {1, • • • , p}:

J ij = 1 n p l=1 ξ l i ξ l j (i = j), J ii = 0.
(2.27)

We assume that the {ξ l i } are i.i.d. variables with equal probability in {±1}, so that EJ ij = 0 and EJ 2 ij = p/n 2 . We study this system in the limit in which both p, n → ∞ with a fixed ratio p/n → α. The derivation of the TAP free energy for these models has been performed in [START_REF] Nakanishi | Mean-field theory for a spin-glass model of neural networks: TAP free energy and the paramagnetic to spin-glass transition[END_REF][START_REF] Mézard | Mean-field message-passing equations in the Hopfield model and its generalizations[END_REF] via the Plefka expansion, and via the cavity method in [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF]. If the random matrix ensemble of eq. (2.27) is a priori not rotationally invariant, one can show that since the variables {ξ l i } are i.i.d., only the first and second moment of their distributions will contribute to the thermodynamic limit of the free entropy, so that we can assume that they are standard centered Gaussian variables without changing the free entropy. The ensemble of eq. (2.27) is thus for our purposes essentially a Wishart matrix model in which the diagonal has been removed. From Section 1.5, we know that its R-transform reads:

R J (x) = α 1 -x -α = αx 1 -x .
(2.28)

Note that because x i ∈ ±1 the variance is given by v = 1 -n -1 i m 2 i = 1 -q, with q the spin glass order parameter. From eq. (2.26) and eq. (2.28), we reach:

Φ J (η) = - 1 n i 1 + m i 2 ln 1 + m i 2 + 1 -m i 2 ln 1 -m i 2 + η 2 i,j J ij m i m j - αη(1 -q) 2 - α 2 ln[1 -η(1 -q)] + O n (1).
Maximizing it over the magnetizations {m i } yields the TAP equations for the Hopfield model:

η -1 tanh -1 (m i ) = j( =i) J ij m j -αη 1 -q 1 -η(1 -q) m i .
This is in agreement with the findings of [MPV87, NT97, Méz17]. However, our framework and results allowed us to treat this kind of model in a very fast and generic way.

Models of bipartite pairwise interactions

A bipartite model with generic priors -As in the symmetric setting, we extend the bipartite model of eq. (2.18) by considering a generic prior on the variables rather than a spherical constraint. More precisely, the fields h and x are assumed to follow prior distributions P X (dx) = i P i (dx i ) and P H (dh) = µ P µ (dx µ ). For a fixed η ≥ 0 and a realization of the coupling matrix L we define the Gibbs-Boltzmann distribution:

P η,L (h, x) ≡ 1 Z η,L m µ=1 P µ (dh µ ) n i=1 P i (dx i ) exp η µ,i L µi h µ x i .
As in the symmetric case we compute the large n limit of the free entropy Φ L (η) ≡ n -1 ln Z η,L . Recall that we take the limit n, m → ∞ with m/n → α > 0. We constraint the first and second moments of {x i } and {h µ } under the Gibbs measure to be

x i = m x i , h µ = m h µ , (x i -m x i ) 2 = v x i , (h µ -m h µ ) 2 = v h µ .
The Lagrange multipliers introduced to enforce these conditions are λ x i , λ h µ , γ x µ , γ h µ . At order 0, we obtain easily:

Φ L (η = 0) = 1 n n i=1 λ x i m x i + 1 2 γ x i (v x i + (m x i ) 2 ) + 1 n m µ=1 λ h µ m h µ + 1 2 γ h µ (v h µ + (m h µ ) 2 ) + 1 n ln µ P µ (dh µ ) i P i (dx i ) e -i [λ x i x i + 1 2 γ x i x 2 i ]-µ [λ h µ hµ+ 1 2 γ h µ h 2 µ ] .
(2.29)

The calculations at the first orders are very similar to the ones we already performed. We obtain:

                                   ∂Φ L ∂η η=0 = 1 n µ,i L µi m h µ m x i , 1 2 ∂ 2 Φ L ∂η 2 η=0 = 1 2n µ,i L 2 µi v h µ v x i , 1 3! ∂ 3 Φ L ∂η 3 η=0 = 1 6n µ,i L 3 µi κ (3,h) µ κ (3,x) i , 1 4! ∂ 4 Φ L ∂η 4 η=0 = 1 8n µ 1 =µ 2 i 1 =i 2 L µ 1 i 1 L µ 1 i 2 L µ 2 i 2 L µ 2 i 1 v h µ 1 v h µ 2 v x i 1 v x i 2 + O n (1).
(2.30)

Recall that κ

(p,x) i , κ (p,h) µ
are the p-th order cumulant of x i , h µ at η = 0. The discussion on the higher orders in perturbation is very similar to the one we just made in the symmetric case: we only retain the simple cycles made of matrix elements {L µi }. We will detail more the extension of the diagrammatics to the bipartite case and the handling of the higher-order cumulants in Section 2.4.5. We obtain the free entropy at leading order in n:

Φ L (η) = Φ L (0) + η n µ,i L µi m h µ m x i + 1 n ∞ p=1 η 2p 2p µ 1 ,••• ,µp i 1 ,••• ,ip L µ 1 i 1 L µ 1 i 2 • • • L µpip L µpi 1 p α=1 v h µα v x iα . (2.31)
In the summation above all indices µ 1 , • • • , µ p are pairwise distinct, and so are i 1 , • • • , i p . As in the symmetric case, we now assume that the maximum of the free entropy of eq. (2.31) is attained for variables

{v h µ , v x i } such that v h µ = v h and v x i = v x .
Using eq. (2.21), the free entropy can be resummed:

Φ L (η) =Φ L (0) + η n µ,i L µi m h µ m x i - 1 + ln v x 2 -α 1 + ln v h 2 + 1 2 inf ζ x ,ζ h αζ h v h + ζ x v x -(α -1) ln ζ h -ρ(dλ) ln(ζ x ζ h -η 2 λ) , (2.32)
where ρ is the LSD of L L, and recall that Φ L (0) is given by eq. (2.29). As in the symmetric case, the diagrammatic study (performed in Section 2.4.5) plays a decisive role in our analysis.

Compressed Sensing: a first inference problem -Compressed sensing [START_REF] David | Compressed sensing[END_REF] is a textbook inference problem with numerous applications, which can be seen as a particular case of GLMs. It can be formulated as the inference of the vector X, generated from a prior P X , from:

Y µ = n i=1 F µi X i + √ ∆z µ . (2.33)
In this equation we modeled the noise by z µ ∼ N (0, 1), and the noise value is ∆ > 0. We follow [KMS + 12], in which the authors considered i.i.d. matrices F. Defining J = -F F and using Bayes' rule we can write the posterior distribution of x as:

P F (x|Y) = 1 Z Y,F P X (x) exp 1 2∆ i,j J ij x i x j + 1 ∆ µ,i F µi Y µ x i .
Defining η ≡ ∆ -1 this distribution can be matched to the one of eq. (2.22) with

h i ≡ -µ F µi Y µ .
Using this correspondence we can use eq. (2.32) to directly obtain:

Φ Y,F = Φ Y,F (∆ = ∞) - 1 2∆n i,j (F F) ij m i m j + 1 n∆ µ,i F µi Y µ m i + 1 2 v∆ -1 0 R J (u)du. (2.34)
We postpone the analysis of the corresponding fixed point equations to our algorithmic discussion in Section 2.3.2.

Generalized Linear Models with correlated matrices -GLMs are of primary importance in a very wide variety of scientific and engineering fields, as we underlined in Section 1.1. Consider m, n ≥ 1 both going to infinity with a ratio m/n → α > 0. We are given a left and right rotationally invariant measurement matrix F ∈ R m×n , as defined in Model R. Given F, data samples {Y µ } are generated as:

∀µ ∈ {1, • • • , m}, Y µ ∼ P out • (FX) µ , (2.35)
in which X ∈ R n is the vector we try to recover (drawn with i.i.d. coordinates from a prior P X ), and P out is a fixed probabilistic channel. Compressed sensing (2.33) corresponds to a Gaussian channel distribution with zero mean and variance ∆. Recall that we consider the Bayes-optimal setting: P out and P X are known, so we can use them in the posterior distribution

P (x|Y) = 1 Z(Y, F) n i=1 P X (x i ) m µ=1 P out Y µ (Fx) µ .
While in compressed sensing η = ∆ -1 played naturally the role of an inverse temperature, in the general setting of eq. (2.35) there is a priori no way to perform a Plefka expansion. As it turns out, there is a way to introduce an auxiliary parameter in terms of which we will perform the expansion, similarly to what is done in [START_REF] Altieri | The jamming transition in high dimension: an analytical study of the TAP equations and the effective thermodynamic potential[END_REF][START_REF] Altieri | Higher-order corrections to the effective potential close to the jamming transition in the perceptron model[END_REF]. Introducing the usual Lagrange parameters to fix the means and variances of {x i }, we obtain the free entropy:

Φ Y,F ≡ 1 n i λ i m i + 1 2N i γ i (v i + m 2 i ) + 1 n ln R n dx e -S[x] ,
in which we introduced an action S[x]:

S[x] ≡ i λ i x i + 1 2 i γ i x 2 i - µ ln P out Y µ i F µi x i - i ln P X (x i ).
Letting h ≡ Fx and using the Fourier transform of the Dirac distribution we reach:

Φ Y,F = -α ln 2π + 1 n i λ i m i + 1 2n i γ i (v i + m 2 i ) + 1 n ln dx dh d h e -S eff [x,h, h] ,
with a new effective action S eff :

S eff ≡ i [λ i x i + 1 2 γ i x 2 i -ln P X (x i )] - µ [ln P out (Y µ |h µ ) + h µ (i hµ )] + µ,i (i hµ )F µi x i . (2.36)
The key idea is to treat x and i h as two independent non-Gaussian fields that interact via the last (quadratic) term of eq. (2.36) and to perform a PGY expansion in terms of this effective Hamiltonian, which is exactly the bipartite Hamiltonian of eq. (2.18). We again denote η the inverse temperature, that is in eq. (2.36) we substitute:

µ,i F µi x i (i hµ ) → η µ,i F µi x i (i hµ ),
and at the end of the expansion we will set η = 1. Similarly as for the field x we will fix the first and second moments of the field i h as i hµ η = g µ and (i hµ ) 2 η = -r µ + g 2 µ , conditions that will be enforced by new Lagrange parameters {ω µ , b µ }. Although a bit tedious, this is straightforward, and we obtain a free entropy in which we will perform a low-η expansion:

Φ Y,F (η) = -α ln 2π+ 1 n i λ i m i + 1 2n i γ i (v i + m 2 i ) + 1 n µ ω µ g µ - 1 2n µ b µ (-r µ + g 2 µ ) + 1 n ln dx dh d h e -S eff [x,h, h] .
The effective action and Hamiltonian are expressed as follows:

S eff [x, h, h] ≡ i λ i x i + 1 2 i γ i x 2 i + µ ω µ (i hµ ) - 1 2 µ b µ (i hµ ) 2 - i ln P X (x i ) - µ ln P out (Y µ |h µ ) - µ h µ (i hµ ) + η H eff [x, h] µ,i F µi x i (i hµ ) . (2.37)
From this equation it is clear that the priors on the variables {x i } and {(i hµ )} decouple. The prior on x i is P X (x i ), while the prior distribution on (i h) µ is related to the Fourier transform of the channel distribution:

P H (i hµ ) ≡ dh 2π e ih hµ P out (Y µ |h).
Using our previous result (2.31) on the PGY expansion for bipartite systems, we conjecture:

Φ Y,F (η) = Φ Y,F (0) - η n µi F µi g µ m i (2.38) + 1 n ∞ p=1 (-1) p η 2p 2p µ 1 ,••• ,µp pairwise distincts i 1 ,••• ,ip pairwise distincts F µ 1 i 1 F µ 2 i 1 • • • F µpip F µ 1 ip p α=1 r µα v iα + O n (1).
We again assume that the maximum of the free entropy of eq. (2.38) will be attained for variance variables {v i , r µ } that are homogeneous: they satisfy v i = v and r µ = r. Using the resummation of eq. (2.32), this leads to a simplified expression:

Φ Y,F (η) = Φ Y,F (0) - 1 n µi F µi g µ m i -α 1 + ln r 2 - 1 + ln v 2 (2.39) + 1 2 inf ζ,ζ αζr + ζ v -(α -1) ln ζ - 1 n ln det[ζζ I n + F F] + O n (1).
We will study the fixed point equations corresponding to the free entropy of eq. (2.39) in the following Section 2.3.2.

High-temperature expansions and message-passing algorithms

Several iterations schemes for the TAP equations have been studied in the past literature, and the reader can refer for instance to [START_REF] Opper | A theory of solving TAP equations for Ising models with general invariant random matrices[END_REF][START_REF] Cakmak | Selfaveraging Expectation Propagation[END_REF]. On a parallel point of view, messagepassing algorithms have been extensively studied in the statistical physics literature, as we mentioned in Section 1.4. For i.i.d. matrices it is well understood that the stationary limit of the message-passing iterations is directly related to the fixed point equations of the TAP free entropy. In this section, we extend this correspondence to generic rotationally-invariant matrices and the G-VAMP algorithm [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF]. We focus on Generalized Linear Models (GLMs) with matrices F that satisfy right-rotation invariance, cf. Model R. We first derive the TAP equations from the PGY expansion we performed in Section 2.3.1, before comparing them to the stationary limit of the G-VAMP algorithm.

The TAP equations from the PGY expansion

Following the assumptions of the VAMP and G-VAMP algorithms [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF][START_REF] Sundeep Rangan | Vector approximate message passing[END_REF] we assume that the variances {v i , r µ } are homogeneous, that is r µ = r and v i = v. We can then use the resummed expression of the Plefka free entropy expressed in eq. (2.39). We first extremize this expression with respect to the Lagrange multipliers {λ i , γ i , ω µ , b µ }:

             m i = E P X (λ i ,γ) [x], v i = E P X (λ i ,γ) [(x -m i ) 2 ], g µ = -g out (y µ , ω µ , b), r = -1 m m µ=1 ∂ ω g out (y µ , ω µ , b).
(2.40)

We defined P X and g out as:

         P X (λ i , γ i )(x) ∝ P X (x)e -1 2 γ i x 2 -λ i x , g out (y, ω, b) ≡ 1 b dz P out (y|z) (z -ω) e -(z-ω) 2 2b dz P out (y|z) e -(z-ω) 2 2b .
(2.41)

The remaining equations are obtained by maximizing eq. (2.39) with respect to the physical parameters. We make use of the Jacobi formula for a symmetric positive definite matrix J:

∂ J ij [ln det J] = (J -1 ) ij . We reach:                                        λ i = -γm i + µ F µi g µ , ( 2.42a
)

ω µ = g µ b + i F µi m i , (2.42b) ζS F F (-ζζ ) = v, (2.42c) ζ S F F (-ζζ ) = αr - α -1 ζ , (2.42d) γ = 1 v -ζ , (2.42e) b = 1 r -ζ. (2.42f)
Additive Gaussian channel -In the case of an additive Gaussian channel with variance ∆ we find r = (∆ + b) -1 , which gives ζ = ∆ and γ = R F F/∆ (-v). One can check from this that we recover the TAP equations for the compressed sensing problem obtained from eq. (2.34), even though these equations were derived with a "naive" PGY expansion in η ≡ ∆ -1 .

The G-VAMP algorithm

With a similar reasoning that we used to derive the VAMP algorithm for a symmetric pairwise model in Section 1.4.3, we can write a VAMP algorithm for a bipartite model. We do not describe its full derivation here, and we simply report the G-VAMP algorithm for the GLM as stated in [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF]. We define a set of functions:

                                     Fm (r, γ) ≡ dxP X (x) x e -1 2 γ(x-r) 2 dx P X (x) e -1 2 γ(x-r) 2 , (2.43a) Fv (r, γ) ≡ dxP X (x) x 2 e -1 2 γ(x-r) 2 dx P X (x) e -1 2 γ(x-r) 2 -( Fm (r, γ)) 2 , (2.43b) Fz (ω, τ ) ≡ dzP out (y|z) z e -1 2 τ (z-ω) 2 dzP out (y|z) e -1 2 τ (z-ω) 2 = g out (y, ω, τ -1 )τ -1 + ω, (2.43c) Fκ (ω, τ ) ≡ dzP out (y|z) z 2 e -1 2 τ (z-ω) 2 dzP out (y|z) e -1 2 τ (z-ω) 2 -( Fz (ω, τ )) 2 = ∂ ω g out (y, ω, τ -1 )τ -2 + τ -1 . (2.43d)
The full algorithm then amounts to iterate the following equations:

m t 1i = Fm ((r t J ) i , γ t J ), v t 1 = 1 n i Fv ((r t J ) i , γ t J ), (2.44a) r t 0 = (m t -γ t J v t r t J ) (1 -γ t J v t ) , γ t 0 = 1 v t -γ t J , (2.44b)
z t 1µ = Fz ((ω t J ) µ , τ t J ), κ t 1 = 1 n µ Fκ ((ω t J ) µ , τ t J ), (2.44c
)

ω t 0 = (z t -τ t J κ t ω t J ) (1 -τ t J κ t ) , τ t+1 0 = 1 κ t 1 -τ t J ,
(2.44d)

m t 2 = 1 τ t+1 0 F F + γ t 0 (γ t 0 r t 0 + F τ t 0 ω t 0 ), v t 2 = 1 n Tr 1 τ t 0 F F + γ t 0 , (2.44e) r t+1 J = (m t 2 -γ t 0 v t 2 r t 0 ) (1 -γ t 0 v t 2 ) , γ t+1 J = 1 v t 2 -γ t 0 , (2.44f) z t 2 = F 1 τ t 0 F F + γ t 0 (γ t 0 r t 0 + F τ t 0 ω t 0 ), κ t 2 = 1 n Tr F F 1 τ t 0 F F + γ t 0 , ( 2.44g
)

ω t+1 J = (z t 2 -τ t 0 κ t 2 ω t 0 ) (1 -τ t 0 κ t 2 ) , τ t+1 J = 1 κ t 2 -τ t 0 .
(2.44h)

TAP equations and fixed points of G-VAMP

We want to see if the stationary limit of G-VAMP, i.e. eq. (2.44) without time indices, is related to the TAP equations we derived with a PGY expansion. At the fixed points of the G-VAMP algorithm, one has in particular the following:

m 1 = m 2 = m, z 1 = z 2 = z, v 1 = v 2 = v and κ 1 = κ 2 = κ.
We start from the TAP equations (2.40) and eq. (2.42), and we map them to eq. (2.44):

• From eq. (2.42f) and eq. (2.43d) we can write

1 b = 1 κ - 1 ζ , (2.45)
which can be identified with eq. (2.44h), with b = τ -1 J and ζ = τ -1 0 .

• Using eq. (2.43d) we write eq. (2.42d) as

r ζ = - κ ζ b 2 + 1 ζ b = 1 n Tr 1 ζζ + F F + 1 -α -1 ζ ζ .
Finally from eq. (2.45) we obtain

κ ζ = 1 α - 1 m Tr ζζ ζζ + F F ,
which is compatible with the second part of eq. (2.44g), with ζ = τ -1 0 and ζ = γ 0 .

• Eq. (2.42c) and eq. (2.42e) are equivalent to the second parts of eq. (2.44e) and eq. (2.44f), with

ζ = γ 0 , ζ = τ -1 0 and γ = γ J .
• We write eq. (2.44e) as

(τ 0 F F + γ 0 )m = (γ 0 r 0 + F τ 0 ω 0 ),
and using that z = Fm, as well as eq. (2.44b) and eq. (2.44d), we arrive at

γ J r J = γ J m + τ J F (z -ω J ),
which is exactly eq. (2.42a) with ω = ω J , λ = -r J γ J and τ J = b -1 .

• Finally we note that eq. (2.42b) at the fixed point is nothing but z = Fm, which gives eq. (2.44g).

This shows in detail the equivalence between the stationary limit of the G-VAMP algorithm of [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF] and the TAP maximization equations that we derived with our PGY expansion!

Diagrammatics and free cumulants

The goal of this section is to precise how the different diagrams arising in our Plefka expansions can be computed. Recall that for symmetric random matrices J we construct diagrams as described in Fig. 2.1. For instance the diagram depicted in Fig. 2.1b is:

• • • • • • • • • = 1 n i 1 ,••• ,i 9 pairwise distinct (J i 1 i 2 J i 2 i 3 J i 3 i 4 J i 4 i 1 )(J i 3 i 5 J i 5 i 6 J i 6 i 7 J i 7 i 3 )J 2 i 6 i 8 J 2 i 2 i 9 .
The perturbation order of any diagram is equal to its number of edges, since each of them represents a factor J ij . The structure of the section is the following:

• In Section 2.4.1 we prove a first rigorous result on the 'simple cycles' arising in the Plefka expansion: we study these diagrams in expectation over J and show a weaker version of Theorem 2.2.

• In Section 2.4.2 we extend this study to all possible diagrams, in expectation over J.

• In Section 2.4.3 we show how the results of Section 2.4.1 and Section 2.4.2 can be extended to study the second moments of these diagrams, and use it to show concentration results. This will in particular imply the full statement of Theorem 2.2.

• In Section 2.4.4 we explain how to handle the higher-order moments that can appear as additional factors in these diagrams for the statistical models studied in Section 2.3.

• In Section 2.4.5 we explain how to generalize all these techniques and results to diagrams made of rectangular matrices, that arise in the Plefka expansion for bipartite models.

• Finally, in Section 2.4.6 we show that if one considers an i.i.d. coupling matrix, all the diagrams of order greater than 3 will not contribute in the thermodynamic limit and that one can effectively consider the distribution of the matrix elements to be Gaussian.

Some technicalities, as well as side results and generalizations of these diagrammatics for Hermitian matrices and diverging-size diagrams, which are not directly useful for our expansions, are given in Appendix A.

Expectation of simple cycles and free cumulants

In the following, J ∈ S n is a rotationally-invariant random matrix, cf. Model S. Recall that we defined the free cumulants c p (ρ) of a distribution ρ in Section 1.5. We first show a weaker version of Theorem 2.2:

Theorem 2.

(Expectation of simple cycles and free cumulants)

For any p ≥ 1, and any set of pairwise

distinct indices i 1 , • • • , i p ∈ N p : lim n→∞ E[n p-1 J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 ] = c p (ρ). (2.46)
Actually, we only need to average over O to obtain the result:

n p-1 O(n) DO OJO i 1 i 2 OJO i 2 i 3 • • • OJO ipi 1 a.s. -→ n→∞ c p (ρ).
(2.47)

Writing J = ODO with D diagonal, the expectation over O of the simple loops considered in the Plefka expansion is an immediate consequence of Theorem 2.3:

∀p ∈ N , E O 1 n i 1 ,••• ,ip pairwise distinct J i 1 i 2 J i 2 i 3 • • • J i p-1 ip J ipi 1 a.s. -→ n→∞ c p (ρ).
Proof of Theorem 2.3 -A first pedestrian way to show eq. (2.47) for small values of p is to use explicit integration of polynomials over the Haar measure of the orthogonal or unitary group. This was first studied by Weingarten [START_REF] Weingarten | Asymptotic behavior of group integrals in the limit of infinite rank[END_REF] and later greatly extended, see for instance [START_REF] Collins | Integration with respect to the Haar measure on unitary, orthogonal and symplectic group[END_REF]. Performing asymptotic expansions of the resulting so-called Weingarten functions could allow to prove Theorem 2.3, and perhaps even to extend it further, e.g. by a precise description of the rate of the convergence of the simple cycles to the free cumulants. We choose here a different (and arguably simpler) path, leveraging the finite-rank "HCIZ" integrals [HC57, IZ80, GM05] analyzed in Section 1.5.3. Let:

L (n) p ≡ n p-1 O(n) DO[(OJO ) i 1 i 2 (ODO ) i 2 i 3 • • • (ODO ) ipi 1 ]. (2.48)
To simplify the calculation, we assume that (i

1 , • • • , i p ) = (1, • • • , p): by rotation invariance L (n) p
does not depend on the particular choice of indices, so this does not remove any generality. The case p ∈ {1, 2} is trivial, so we will assume p ≥ 3 in the following. One can rewrite eq. (2.48) as:

L (n) p = 1 n p l=1 ∂ ∂b l O(n) DO e n 2 Tr [M(b)OJO ] b=0 , in which b ≡ (b 1 , • • • , b p )
and M(b) ∈ S n is the following symmetric block matrix of rank p:

M(b) ≡ M 1 (b) (0) (0) (0) , with M 1 (b) ≡           0 b 1 0 • • • 0 b p b 1 0 b 2 • • • 0 0 0 b 2 0 • • • 0 0 . . . . . . . . . . . . . . . . . . 0 0 0 • • • 0 b p-1 b p 0 0 • • • b p-1 0          
.

We now apply Theorem 1.12. and we reach (limits are taken a.s. with respect to the law of J):

lim n→∞ L (n) p = lim n→∞ 1 n p l=1 ∂ ∂b l {Z(b)} b=0 , with Z(b) ≡ exp n 2 ∞ k=1 c k (ρ) k Tr [M(b) k ] . (2.49)
Note that differentiating Z(b) with respect to b 1 yields by cyclicity of the trace:

1 Z(b) ∂ ∂b 1 Z(b) = n 2 ∞ k=1 c k (ρ)Tr [E 12 M(b) k-1 ],
(2.50)

with elementary symmetric matrices (E ab ) ll ≡ δ l,a δ l ,b +δ l ,a δ l,b . These matrices satisfy

E ab E cd = 0 if {c, d} ∩ {a, b} = ∅.
The only way to obtain a matrix of non-zero trace with a product of matrices {E ab } is thus to have a cycle structure in the indices of the matrices. For instance:

Tr

[E 2 12 E 13 E 23 E 12 ] = Tr [E 12 E 21 E 13 E 32 E 21 ] = 0, while Tr [E 2 12 E 24 E 23 E 12 ] = 0.
Using this along with M(0) = 0, it is easy to see that the only term that will survive after taking all the successive derivatives and letting b = 0 will be the derivatives of the right-hand-side of eq. (2.50), and not other derivatives of Z(b). Let us analyze what differentiating this term yields. As we saw, differentiating with respect to b 1 yields a matrix E 12 . When differentiating with respect to b 2 this yields a matrix E 23 . Note that a priori, one has:

∂ ∂b 2 Tr[E 12 M (b) k-1 ] = Tr [E 12 k-2 l=0 M (b) l E 23 M (b) k-2-l ].
(2.51)

However, the following differentiations with respect to b 3 , • • • , b p will never yield a matrix of the type E 2a . Therefore in eq. (2.51) only two terms, the term l = 0 and l = k -2, will yield a non-zero contribution. In the end, after taking all the p successive derivatives, only two terms will remain, which correspond to the two possible orientations of the simple cycle:

lim n→∞ L (n) p = 1 2 ∞ k=p c k (ρ)Tr[(E 12 E 23 • • • E p1 + E 1p E pp-1 • • • E 32 E 21 ) M(0) k-p ] = c p (ρ),
since M(0) = 0. This finishes the proof.

The expectation of generic diagrams

Following [GY91, PP95] we can define three disjoint categories of connected diagrams:

T.1 Non-Eulerian diagrams -By definition, a diagram is Eulerian if one can construct a cyclic path in the graph that goes through each edge exactly once. It is a classic result of graph theory (the Euler-Hierholzer theorem [START_REF] Euler | Solutio problematis ad geometriam situs pertinentis[END_REF][START_REF] Hierholzer | Über die möglichkeit, einen linienzug ohne wiederholung und ohne unterbrechung zu umfahren[END_REF]) that these graphs are exactly the connected graphs with even degree in each vertex. For instance, the graph depicted in Fig. 2.2a is not Eulerian, whereas the one of Fig. 2.2b is Eulerian.

• • • • (a) A non-Eulerian diagram. • • • • (b) An Eulerian strongly irreducible diagram. • • • • • • • • • (c) A cactus diagram.
T.2 Eulerian diagrams that are strongly irreducible but not simple cycles -By strongly irreducible [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF], we mean that one can not make it disconnected by removing any single vertex. E.g. Fig. 2.2b is strongly irreducible, whereas Fig. 2.2c is not.

T.3

Cactus diagrams -These diagrams, like the one of Fig. 2.2c, are trees made of simple cycles joining at their vertices. Among them are of course the simple cycles.

As generically argued in [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF], only strongly irreducible diagrams will appear in the PGY expansions. This is an important hypothesis of the Plefka expansion, somehow a bit hidden by the formalism. We give precise descriptions of the large n limit of the expectation of all these diagrams in the following5 . More precisely, we will show:

(i) All non-Eulerian diagrams (type T.1) have vanishing expectation in the n → ∞ limit.

(ii) All diagrams of type T.2 also have vanishing expectation in the n → ∞ limit.

(iii) By Theorem 2.3, the expectation of a simple cycle of size p converges to the p-th free cumulant of ρ. We show that the expectation of a cactus diagram converges to the product of the expectations of all its constituent simple cycles. For instance, for the diagram C of Fig. 2.2c we obtain that its expectation converges to:

lim n→∞ E C = c 2 (ρ) 2 c 4 (ρ) 2 .
(2.52)

In the remaining of Section 2.4.2, we justify all these claims.

Eulerian diagrams, strongly irreducible diagrams and simple cycles

Let us consider a connected diagram G with V vertices and E edges. We show here that:

• If G is not Eulerian, then E G → n→∞ 0
• If G is Eulerian and strongly irreducible but not a simple cycle, then E G → n→∞ 0 as well.

As we average over orthogonal matrices the permutation invariance of the indices allows to write: in which the ll are positive integers such that l<l ll = E. We can now use the results of [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF], as we did in Section 2.4.1, to write this diagram as:

E G = n V -1 [1 + O n (1)] O(n) DO 1≤l<l ≤V (ODO ) ll ll , • • ; • • • • • • • ; • • •
E G = n V -E-1 [1 + O n (1)] l<l ∂ ll ∂b ll ll exp n 2 ∞ k=1 c k (ρ) k Tr [M(b) k ] b=0 .
(2.53)

Here M ll (b) = M l l (b) ≡ b ll for l < l , and M ll (b) = 0. Exactly as in Section 2.4.1, the elementary matrices {E ll } will appear in eq. (2.53) by successive derivatives of the exponential, using the fact that ∂ b ll M(b) = E ll and that M(0) = 0. As we explained in Section 2.4.1, a trace of the products of the {E ll } matrices will only be non-zero if and only if the indices in the products form a cycle. Moreover, as is clear in eq. (2.53), the terms corresponding to the decomposition of E G into the maximum number of such cycles will dominate in the large n limit, as each derivation of the exponential term adds a multiplicative factor n6 . These two facts together imply that:

• If G is not Eulerian (e.g. Fig. 2.2a
) its expectation will be 0 in the limit n → ∞, as by definition it is not possible to decompose it into disjoint cycles.

• If G is Eulerian, strongly irreducible, but not a simple cycle, the dominant contribution to E G in eq. (2.53) will arise from decomposing G into simple cycles, as this decomposition maximizes the number of cycles, and we already showed that each simple cycle has a non-negligible contribution. For the graph of Fig. 2.2b, we show two such possible decompositions in Fig. 2.3.

Given the remarks above we assume now that G is Eulerian and strongly irreducible. Let us denote P the maximal number of simple cycles in such a decomposition of the graph G. Then one can easily see that the scaling of eq. (2.53) will be:

E G ∼ n V +P -E-1 .
By elementary graph theory, for a strongly irreducible diagram G we have V + P -E -1 ≤ 0, and we have equality iff G is a simple cycle. This implies that all strongly irreducible diagrams that are not simple cycles will not contribute in the n → ∞ limit. This fully justifies claims (i) and (ii) above.

Cactus diagrams

As a side result, although it's not directly useful for our PGY expansion, we show that we can compute the large n limit of any "cactus" [START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF] diagram (like the one of Fig. 2.2c) as a function of the free cumulants of ρ. The argument is straightforward and uses the same technique as in the previous paragraph. Consider a cactus diagram G with V vertices and E edges: one can write exactly eq. (2.53). it is easy to see that there is only one maximal decomposition of EG, which corresponds to its natural decomposition into its constituent simple cycles, and that the number of such cycles is P = E + V -1. Let us denote {r 1 , • • • , r P } the number of vertices in each of these P simple cycles. The dominant contribution corresponds to differentiating P times inside the exponential of eq. (2.53). Using exactly the argument of Section 2.4.1 for each of the P simple cycles we finally obtain the claim (iii) above:

E G = n P +V -E-1 P α=1 c rα (ρ) + O n (1) = P α=1 c rα (ρ) + O n (1).

Concentration of the diagrams: a second moment analysis

Using the limits of the first moments derived in Sections 2.4.1,2.4.2, we will show:

(A) If C p is the simple cycle of order p, then C p L 2 → n→∞ c p (ρ), which ends the proof of Theorem 2.2.
Moreover, if G is a cactus diagram then it converges in the L 2 sense to the products of the free cumulants corresponding to its constituent simple cycles.

(B) If G is of the type T.1 or T.2 then E G 2 → n→∞ 0: G will be negligible in the n → ∞ limit.
As we already mentioned, only strongly irreducible diagrams will appear in the PGY expansion.

Together with point (B) this justifies why only the simple cycles contribute in our PGY expansion, as we noticed in Section 2.2.1. In order to show (A) and (B) we will first establish the following fact. Consider a diagram G with V vertices and E edges, of any of the types T.1, T.2, or T.3. Then one has:

E G 2 = (EG) 2 + 1 n α E C α + O n (1). ( 2

Given our previous computations of the first moments this implies results (A) and (B).

To conclude our argument, we now show eq. (2.54). One can write any diagram G as:

G = 1 n i 1 ,••• ,i V pairwise distinct 1≤l<l ≤V J ll i l i l ,
in which the integers ll satisfy l<l ll = E. Thus one has:

E G 2 = E 1 n 2 i 1 ,••• ,i V pairwise distinct j 1 ,••• ,j V pairwise distinct 1≤l<l ≤V J ll i l i l J ll j l j l .
In this expression, one can see that two types of terms have to be taken into account:

• Terms for which all indices {i 1 , • • • , i V , j 1 , • • • , j V } are pairwise distinct.
Diagrammatically, this corresponds to a graph with two disconnected components that are identical and equal to G. Using the exact same arguments as in Section 2.4.2 and since all the indices are distinct, the decomposition of this diagram into the maximum number of simple cycles will be two copies of the maximal decomposition of G. This shows that the total contribution of these terms behaves asymptotically as

(E G) 2 . E • • • 2 = E • • • 2 + 1 n 9 E • • • • • + 18 E • • • • + 6 E • • • Figure 2
.4: Second moment decomposition of the simple cycle of order 3. We detail the combinatorial factors.

• Terms for which there is at least one equality of the type i l = j l for 1 ≤ l, l ≤ V . Such a term thus corresponds to a diagram with a single connected component and constructed by "gluing" some of the vertices of two identical copies of G. Since these diagrams have a single connected component, they carry a single n -1 factor, which explains the term n -1 α E C α in eq. (2.54), denoting denote C α the "glued" diagrams.

This ends our justification of eq. (2.54). We give a schematic representation of this decomposition for G a simple cycle in Fig. 2.4.

Higher-order moments in the diagrammatics

All the diagrammatic results we just derived were valid for diagrams solely made out of the matrix elements {J ij }, without any additional factors. However in the PGY expansion for non-spherical models there are possible factors that are the cumulants (or the moments) of the variables at η = 0, as we pointed out in Section 2.3. As an example, let us consider the simple symmetric model of eq. (2.1), but assuming that x i are independent variables (rather than spherical), with κ (p) i the cumulant of order p of x i . For instances, two possible contributions to the free entropy at order 6 would be:

     n -1 i 1 ,i 2 ,i 3 pairwise distinct J i 1 i 2 J i 2 i 3 J i 3 i 4 J i 4 i 1 J 2 i 1 i 3 v 2 i 1 v i 2 v 2 i 3 v i 4 , n -1 i 1 ,i 2 ,i 3 pairwise distinct J i 1 i 2 J i 2 i 3 J i 3 i 4 J i 4 i 1 J 2 i 1 i 3 κ (4) i 1 v i 2 κ (4) i 3 v i 4 .
(2.55)

Both these contributions are represented by the diagram of Fig. 2.2b! One can see that in order to apply our diagrammatic results to the PGY expansion, we need some additional assumptions:

A.1 By construction of the diagrams, odd cumulants of order greater or equal to 3 only appear in non-Eulerian graphs. By the results of Section 2.4.3 we know that such diagrams, without the moments or cumulants as factors, are negligible. We assume that the possible correlations of the cumulants of x i with the matrix elements {J ij } are not strong enough to yield thermodynamically relevant corrections to the free entropy.

A.2

We showed that Eulerian strongly irreducible diagrams that are not simple cycles are negligible. We assume that the higher order moments that appear as additional factors do not change their scaling, so that they remain negligible in the thermodynamic limit.

For instance, A.2 implies that the contributions of both terms in eq. (2.55) are negligible in the n → ∞ limit, as the diagram of Fig. 2.2b is strongly irreducible but is not a simple cycle.

Extension to bipartite models

We detail here how we can treat the diagrams that arise in the Plefka expansion of bipartite models with pairwise interactions. The structure of this section is the following:

• We show first how we can generalize all the techniques and results we developed in the symmetric setting to diagrams constructed from a rotationally-invariant rectangular matrix L satisfying Model R.

• • • • • • Figure 2.5: A diagram constructed from F.
Each blue vertex is an index µ, each red vertex an index i. Each edge is a factor F µi , and each connected component of the diagram carries a global factor n -1 . Note that there can only be edges between red and blue vertices.

• We then transpose the assumptions of Section 2.4.4 to this bipartite case, to deal with the higher-order moments of the fields that can arise in the PGY expansion.

Generalization of the previous results to rectangular matrices

Consider a left and right rotationally-invariant random matrix F ∈ R m×n , i.e. satisfying Model R. We are interested in the limit m, n → ∞ with a finite ratio m/n → α > 0. In the PGY expansions performed for bipartite models (e.g. in Section 2.2.2) there appears quantities that we represent as diagrams, as explained in Fig. 2.5. The diagram depicted in this figure represents the quantity:

1 n µ 1 ,µ 2 ,µ 3 pairwise distinct i 1 ,i 2 ,i 3 pairwise distinct F µ 1 i 1 F µ 1 i 2 F µ 2 i 2 F µ 2 i 3 F µ 3 i 3 F µ 3 i 1 F 2 µ 1 i 3 .
Recall the rectangular transforms and spherical integrals of Section 1.5, in particular the coefficients Γ p (α, ρ) of eq. (1.93). We let F = UΣV , with U, V ∈ O(m) × O(n) two matrices drawn from the Haar measure of their respective group. We can state the counterpart of all our previous results in this rectangular setting:

R.1 Consider a simple cycle of size 2p. Then it converges in L 2 to Γ p (α, ρ) as n → ∞:

E U,V 1 n µ 1 ,••• ,µp pairwise distinct i 1 ,••• ,ip pairwise distinct F µ 1 i 1 F µ 1 i 2 • • • F µpip F µpi 1 -Γ p (α, ρ)

R.3

Any strongly irreducible diagram G (i.e. it can not be disconnected by removing a single vertex) that is not a simple cycle (e.g. Fig. 2.5) also has vanishing first and second moments.

R.4

If G is a cactus, i.e. a tree made of r simple cycles of sizes (2p 1 , • • • , 2p r ) joining at vertices:

lim n→∞ E G - r l=1 Γ p l (α, ρ) 2 = 0.
Since every argument to show R.1 to R.4 is straightforwardly given by slightly modifying what we did in the symmetric case, we will focus on R.1, and leave the remaining points for the reader.

Justifying R.1 -As in Section 2.4.1, we begin by a first-moment analysis and we show:

E 1 n µ 1 ,••• ,µp pairwise distinct i 1 ,••• ,ip pairwise distinct F µ 1 i 1 F µ 1 i 2 F µ 2 i 2 • • • F µpip F µpi 1 = Γ p (α, ρ) + O n (1).
(2.57) Indeed, by rotation invariance we can remove the summations in the LHS of eq. (2.57), and we obtain at leading order in n:

α p n 2p-1 E[F 11 F 12 F 22 • • • F pp F p1 ] = 1 n ∂ 2p ∂b 1 • • • ∂b p ∂c 1 • • • ∂c p DUDVe √ αnTr [M(b,c) UΣV ] b,c=0
, with M(b, c) a block matrix of rank p defined as:

M(b, c) = M 1 (b, c) (0) (0) (0) , with M 1 (b, c) ≡           b 1 c 1 0 • • • 0 0 0 b 2 c 2 • • • 0 0 0 0 b 3 • • • 0 0 . . . . . . . . . . . . . . . . . . 0 0 0 • • • b p-1 c p-1 c p 0 0 • • • 0 b p           .
Using Theorem 1.13 we obtain:

α p n 2p-1 E[F 11 F 12 F 22 • • • F pp F p1 ] = (2.58) 1 n ∂ 2p ∂b 1 • • • ∂b p ∂c 1 • • • ∂c p exp n ∞ k=1 Γ k (α, ρ) 2k Tr [(M(b, c) M(b, c)) k ] b,c=0 + O n (1),
Recall the symmetric elementary matrices (E ab ) ij = δ ai δ bj + δ bi δ aj . As in the symmetric case, the dominant terms in eq. (2.58) will be given by the maximum number of differentiations of the exponential term. Moreover, one easily shows

∂ 2 b 1 ,c 1 [M(b, c) M(b, c)] = E 12 .
Note that the exponential can only be differentiated once: since M(0, 0) = 0, one would need to create cycles with the matrices E ab , and such a cycle can only appear if one derives a single time the exponential term. As in Section 2.4.1, there are two cycles that are created by the successive derivatives:

E 12 E 23 • • • E p1 and E 21 E 1p • • • E 32 .
These two cycles yield the dominant contribution:

α p n 2p-1 E[F 11 F 12 F 22 • • • F pp F p1 ] = ∞ k=p Γ k (α, ρ) 2 Tr [(E 12 E 23 • • • E p1 + E 21 E 1p • • • E 32 )(M(0, 0) M(0, 0)) k-p ] + O n (1), = 1 2 Γ p (α, ρ)Tr [E 12 E 23 • • • E p1 + E 21 E 1p • • • E 32 ] + O n (1) = Γ p (α, ρ) + O n (1).
This shows eq. (2.57). The exact same arguments as the ones used in Section 2.4.3 show that we have L 2 concentration, which concludes our justification of R.1.

The higher order moments and their influence on the diagrammatics

The assumptions we need to make in order to deal with the higher-order moments for rectangular non-spherical models are very similar to A.1 and A.2 (for the symmetric case), and we state them here for completeness:

B.1 From the construction of the diagrams, odd moments of order greater or equal to 3, like κ (3,x) , only appear in non-Eulerian graphs. By R.2 we know that these diagrams (without the moments as factors) are negligible. We assume that the possible correlations of the higher order moments with the matrix elements {F µi } are not strong enough to yield thermodynamically relevant corrections to the free entropy.

B.2

Eulerian strongly irreducible diagrams that are not simple cycles are negligible by R.3. We assume that the higher order moments that appear as additional factors do not change their scaling, so that they remain negligible in the thermodynamic limit.

A note on i.i.d. matrices

We finish this section by a comment on i.i.d. rectangular matrices. We consider a random matrix F ∈ R m×n whose elements {F µi } are taken i.i.d., such that √ nF µi is drawn from a given probability measure ρ, with zero mean and finite moments of all orders. These matrices appear e.g. in the GAMP algorithm in Section 1.4. If ρ is not a Gaussian probability measure, the matrix F does not satisfy the rotation invariance of Model R. However, one can still derive strong results on its diagrammatics. Keeping assumptions B.1 and B.2, it is easy to see that because the {F µi } are uncorrelated, all diagrams with order p ≥ 3 are negligible in the n → ∞ limit. The only diagram that does contribute is:

• • = 1 n µ,i F 2 µi v h µ v x i .
(2.59)

In particular, all our PGY expansion formalism remains valid in this case.

Conclusion of Chapter 2

This chapter presented a step-by-step application of the formalism of Georges-Yedidia [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF] to a wide class of inference problems. It can be seen as a classical example of how "old" methods from statistical mechanics (the TAP formalism and the Plefka expansion were introduced in the 70s and 80s [TAP77, Ple82]) can often provide surprisingly deep and modern results. Our main result is arguably Conjecture 2.1, which we derived in a precise and systematic way by the means of PGY expansions. The diagrammatic results of Section 2.4, as well as the precise forms of the TAP free entropies of Section 2.3, are also of interest, as to the best of our knowledge we did not see in the previous literature such an analysis relating the free cumulants of rotationallyinvariant matrices to the diagrams we presented. In the following chapter, we will leverage all the formalism of PGY expansions introduced here to tackle the very involved, and yet open, problem of extensive-rank matrix factorization.

A significant part of the results of [MFC + 19] were not developed here, and the interested reader might find there for instance an adaptation of all the techniques of the PGY expansion to study a replicated system, i.e. several instances of the system that are correlated through an overlap matrix. This allows to relate the results of the PGY expansion to the ones of the replica method, and to explicit how these two different approaches yield equivalent results. We also describe the connection between AMP equations and the Plefka expansion in the context of generalized linear models with i.i.d. matrices, retrieving the GAMP algorithm [START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF] and the statistical mechanics analysis of [KMS + 12].

Finally, we underline in [MFC + 19] a possible important limitation of EC approximations and the PGY expansion with respect to the VAMP approach. Namely, the VAMP approach differs from other EC approximations in the sense that it provides a natural iteration scheme for the equations. This is an important benefit, as iterating these equations is in general a very involved task. For instance, when considering compressive sensing, we showed in Section 2.3 that we could recover the stationary limit of VAMP by a PGY expansion in terms of the inverse noise ∆ -1 . However, we show in [MFC + 19] that there is no clear iteration scheme of the TAP equations that gives back the VAMP algorithm: as physicists have known since forty years, iterating the TAP equations is often itself a hard problem.

Chapter 3

Towards exact solution of extensive-rank matrix factorization "But there's no sense crying over every mistake, You just keep on trying till you run out of cake."

GLaDOS, Portal (2007).

Disclaimer -This chapter is devoted to an exciting and involved problem: factorization of extensive-rank matrices. It plays a peculiar role in this thesis, as it is the only chapter solely based on yet unpublished work. For this reason, we will only focus on a very small subset of results, which are on firm enough ground to be presented. While this will leave many open unanswered questions, many of them will be addressed in the upcoming [MFK + 21]. We nevertheless provide important results, notably strong arguments disproving several previous approaches to this problem, either based on the replica method [KKM + 16] or message-passing algorithms [PSC14a, PSC14b, ZZY20], and we lay out a potential path to correct them.

Introduction

Definition of extensive-rank matrix factorization

In this chapter we consider the matrix factorization problem in the extensive-rank setting, a problem sometimes referred to as dictionary learning. Our approach has similarities with the one of [KKM + 16], and an important result of this chapter is a correction to the predictions of this paper. More precisely, we will study the following inference problem:

Model FX (Extensive-rank matrix factorization)

Consider n, m, p ≥ 1. Extensive-rank matrix factorization is defined as the inference of the matrices F ∈ R m×n and X ∈ R n×p from the observation of Y ∈ R m×p , generated as:

Y µl ∼ P out • 1 √ n n i=1 F µi X il , 1 ≤ µ ≤ m, 1 ≤ l ≤ p.
We also assume that the matrix elements of F and X are both generated as i.i.d. random variables, with respective prior distributions P F and P X , both having zero mean and finite moments of all order.

In order to estimate F and X the statistician can use the posterior distribution, also referred to as Gibbs measure (see Section 1.1 for reminders on these notions)1 :

P Y,n (dF, dX) ≡ 1 Z Y,n µ,i P F (dF µi ) i,l P X (dX il ) µ,l P out Y µl 1 √ n i F µi X il . (3.1)
We assume that she/he has access to the distributions P out , P F , P X : this is known as the Bayesoptimal setting: it is then known that the mean under the posterior distribution of eq. (3.1) is the information-theoretically optimal estimator.

We consider this inference problem in the high-dimensional (or thermodynamic) limit, i.e. we assume n, m, p → ∞ with finite limit ratios m/n → α > 0 and p/n → ψ > 0. In this limit, we can define the single-graph free entropy Φ Y,n2 of the system as:

Φ Y,n ≡ 1 n(m + p) ln µ,i P F (dF µi ) i,l P X (dX il ) µ,l P out Y µl 1 √ n i F µi X il . (3.2)
The averaged limit free entropy is denoted Φ ≡ lim n→∞ E Y Φ Y,n . We shall also consider a symmetric version of this problem:

Model XX (Symmetric extensive-rank matrix factorization)

Consider n, m ≥ 1. Symmetric extensive-rank matrix factorization is defined as the inference of the matrix X ∈ R m×n from the observation of Y generated as:

Y µν ∼ P out • 1 √ n n i=1 X µi X νi , 1 ≤ µ < ν ≤ m.
We also assume that the elements of X are generated i.i.d. from a prior distribution P X .

As in the non-symmetric case, we can define both the posterior distribution of X and the free entropy of the system as3 :

           P Y,n (dX) ≡ 1 Z Y,n µ,i P X (dX µi ) µ<ν P out Y µν 1 √ n i X µi X νi , (3.3a) Φ Y,n ≡ 1 nm ln µ,i P X (dX µi ) µ<ν P out Y µν 1 √ n i X µi X νi . (3.3b)
The Gaussian setting -In this chapter, we denote Gaussian setting the specific models (both symmetric and non-symmetric) in which all the prior distributions and the channel distributions are zero-mean Gaussians, with variances to be precised.

Possible symmetries of the problem -Assuming that P x is a zero-mean gaussian distribution, Model XX has a huge symmetry group: namely one can never distinguish X from XO, for any O ∈ O(n). A similar symmetry can also arise in Model FX provided that both P F and P X are zero-mean Gaussians. This symmetry is present as well in the finite-rank case, which was analyzed in [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF]. Note that the zero-mean Gaussian is the only prior to satisfy such a property, being the only i.i.d. rotationally invariant distribution in R m×n . This implies that specifically for this prior, the set of solutions to the TAP equations should possess the same type of symmetry group.

Organization of the chapter and summary of the results

Let us now describe the structure of the rest of Chapter 3.

• We begin in Section 3.2 by a critical analysis of previous approaches to this problem, in the Bayes-optimal setting. In particular, we show that the approach taken in [KKM + 16] (as well as many subsequent works) is neglecting some crucial correlations of the variables. We provide direct evidence for the influence of these correlations in both the replica calculation and the algorithmic (message-passing) approaches. This implies that adapting these methods (and especially the replica method) for this model is still an open problem.

• In Section 3.3, we tackle the problem using the high-temperature expansion techniques described in Chapter 2. This allows for a systematic computation of the corrections to the results of [KKM + 16]. While we restrict here to presenting clear evidence of the existence of such corrections, a final formula for the TAP free entropy of extensive-rank matrix factorization is beyond the scope of this chapter, and will be addressed in [MFK + 21].

Critical treatment of previous approaches

In this section, we briefly sketch the computation of [KKM + 16], and detail which approximations used in this work are actually not valid in the thermodynamic limit. The authors derived the free entropy of this problem, in the Bayes-optimal setting, in two different ways, namely via the replica method and via belief propagation (BP) equations. We believe that both their approaches are based on incorrect assumptions.

These wrong hypotheses are also at present in the derivation of the BiGAMP (and BiG-VAMP) algorithm (cf. [PSC14a, PSC14b, ZZY20] among others). We therefore believe that these algorithms are also not able to give exact asymptotic computation of the marginal probabilities in this problem.

Let us now describe both approaches taken in [KKM + 16], and explain how the assumptions behind them fail. We focus primarily on the replica analysis, and briefly describe the messagepassing approach.

The replica analysis

We focus first on the replica analysis performed in Section V.B of [KKM + 16]. As we have introduced in Section 1.3.1, the main idea behind the replica method is to compute the quenched free entropy from the evaluation of the moments of the partition function, using the relation:

lim n→∞ 1 n 2 E Y ln Z Y,n = ∂ ∂r lim n→∞ 1 n 2 ln E Y Z r Y,n r=0
.

The computation of the quenched free entropy thus reduces to the evaluation of the integer moments of the partition function. When expanding E Y Z r Y,n , there naturally appears (r + 1) replicas of the system, that interact via the channel distribution term, as represented in the following equation:

E Y Z r Y,n = dY r a=0 P F (dF a )P X (dX a ) r a=0 µ,l P out Y µl 1 √ n i F a µi X a il .
A key step in the calculation of [KKM + 16] is the assumption that

Z a µl ≡ 1 √ n i F a µi X a il (3.4)
are multivariate Gaussian random variables. However, although {F a } and {X a } follow statistically independent distributions, with zero mean and finite variances, this is not enough to guarantee the gaussianity of the Z a µl variables in the high-dimensional limit. Indeed, there is a number O(n 2 ) of variables Z a µl , and therefore classical central limit results can not conclude on the asymptotic gaussianity of the joint distribution of such variables. In general, this gaussianity is actually false, as can be seen e.g. by considering the following quantity, for a single replica:

L 4 ≡ lim n→∞ E 1 n 3 µ 1 =µ 2 l 1 =l 2 Z µ 1 l 1 Z µ 1 l 2 Z µ 2 l 2 Z µ 1 l 2 .
Let us assume that both P F and P X are standard Gaussian distributions for the simplicity of the argument. The computation of L 4 is then straightforward and yields L 4 = α 2 ψ 2 . However, should the joint distribution of the {Z µl } converge to a multivariate (zero-mean) Gaussian distribution, such a distribution would satisfy by definition

E[Z µ 1 l 1 Z µ 2 l 2 ] = 1 n i,j E[F µ 1 i F µ 2 j ]E[X l 1 i X l 2 j ] = δ µ 1 µ 2 δ l 1 l 2 .
And using the diagrammatic results of Section 2.4, we know that then L 4 = 0 by eq. (2.56) for Gaussian matrices. There is therefore an inconsistency: this argument illustrates the error made (we believe) in [KKM + 16] (and many subsequent works) when considering the joint distribution of {Z a µl }. To conclude on the replica analysis, we did not find a way to correct the calculation, as many of the usual tricks and tools used in the replica method do not transfer here. In particular, the nature of the physical order parameter governing the problem is not clear yet.

The message-passing approach

Another approach to the problem are the Belief Propagation (BP) equations4 , or the cavity method. The goal of BP is to compute the marginal distributions of each variable in the system, by solving iterative equations involving probability distributions over each single variable. These probability distributions are called messages in the BP language, and the fixed point of the iterative equations yields an estimate of the marginal distributions. A detailed treatment of the BP derivation of [KKM + 16] is beyond the scope of this chapter, however we believe that the same hidden assumption of Gaussianity is also present in the BP approach, as it neglects the structure of the correlations of some variables.

We will see very precisely in the following Section 3.3 which terms are (wrongly) neglected in the message-passing approach of [KKM + 16], and how this is related to neglecting the correlation structure of the variables of eq. (3.4), as in the replica approach.

TAP equations and PGY expansion

In this section, we detail the Plefka-Georges-Yedidia (PGY) expansion applied to extensiverank matrix factorization. We mainly focus on detailing the method and its results for the non-symmetric Model FX. In Sections 3.3.1 to 3.3.3 we describe the PGY expansion approach to deriving the TAP equations, and we discuss their relation with the previous approaches described in Section 3.2. We end this chapter by Section 3.3.4, in which we generalize our findings to the symmetric Model XX .

Sketch of the computation

Rather than the replica method, a more pedestrian way to compute the free entropy of this problem is to perform a perturbative PGY expansion, following the formalism we described in Chapter 2. Let us consider the most general Model FX. In order to place our problem in a formalism suited for high-temperature expansions, we introduce an auxiliary field h ≡ FX/ √ n in eq. (3.2), and use the Fourier transform of the resulting delta functions5 . Introducing delta functions in eq. (3.2), we reach:

e n(m+p)Φ Y,n = µ,i P F (dF µi ) i,l P X (dX il ) µ,l d Ĥµl P out (Y µl | Ĥµl )δ Ĥµl - 1 √ n i F µi X il .
Using the Fourier transform of the delta function δ(x) = (2π) -1 dh e ihx , we reach an effective free entropy in terms of three fields F, X, H, with H of size m × p:

Φ Y,n ≡ 1 n(m + p) ln µ,l P µl H (dH µl ) µ,i P F (dF µi ) i,l P X (dX il )e -H eff [F,X,H] , (3.5)
in which we introduced an effective Hamiltonian H eff [F, X, H] and un-normalized probability distributions {P µl H } defined as:

H eff [F, X, H] ≡ 1 √ n µ,i,l (iH) µl F µi X il , P µl H [dH] ≡ d H 2π e iH H P out (Y µl | H). (3.6)
As we have seen in Chapter 2, the formalism of [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF] allows to compute the free entropy of the system, constraining the means and variances of each variable {F µi , X il , H µl } by "tilting" the Gibbs measure of eq. (3.5). More formally, we impose:

       F µi = m F µi , X il = m X il , F 2 µi = v F µi + (m F µi ) 2 , X 2 il = v X il + (m X il ) 2 , (iH) µl = -g µl , (iH) 2 µl = -r µl + g 2 µl .
(3.7)

Recall that • denotes an average over the (now tilted) Gibbs measure. The resulting free entropy is a function of these means and variances {m F , m X , v F , v X , g, r}, on which we will then have to maximize. The conditions of eq. (3.7) will be imposed via Lagrange multipliers, which we denote respectively by {λ F , γ F , λ X , γ X , ω, b}. The free entropy can now be expressed as a function of all these parameters (we still denote it Φ Y,n , with a slight abuse of notations):

n(m + p)Φ Y,n = µ,i λ F µi m F µi + γ F µi 2 v F µi + (m F µi ) 2 + i,l λ X il m X il + γ X il 2 v X il + (m X il ) 2 (3.8) + µ,l -ω µl g µl - b µl 2 -r µl + g 2 µl + ln P H (dH)P F (dF)P X (dX) e -S eff [F,X,H] ,
in which we introduced an effective action S eff :

S eff [F, X, H] ≡ µ,i λ F µi F µi + γ F µi 2 F 2 µi + i,l λ X il X il + γ X il 2 X 2 il (3.9) + µ,l ω µl (iH) µl - b µl 2 (iH) 2 µl + H eff [F, X, H].
Similarly as what we did in Section 2.3.1 for Generalized Linear Models, the idea of the expansion is to introduce a parameter η > 0 in front of the interaction Hamiltonian, i.e. to replace H eff → ηH eff in eq. (3.9), and then take η = 1 after performing a small-η expansion. We denote the corresponding free entropy Φ Y,n (η). Note that at η = 0, all the fields {F µi , X il , H µν } are independent. In order to make the PGY expansion tractable in this model, we introduce the following hypotheses on the scaling of the parameters:

H.1 At their physical value, the variables {m F µi , m X il } should be uncorrelated, in coherence with the fact that the elements of F and X are drawn i.i.d. Importantly, this is not true for H: although the elements H µl are independent by eq. (3.6), their distribution depends on µ, l and therefore their statistics might be correlated, e.g. the variables g µl might be correlated, through the correlation of the variables Y µl .

H.2

Recall that g µl is the average of (iH) µl , the conjugate variable to Z µl ≡ i F µi X il / √ n. We therefore assume that the correlations of the elements {g µl } scale as the ones of {Z µl }, that is the correlations of the g = {g µl } scale similarly as the product of two independent matrices that have i.i.d. zero-mean components 6 . Assuming H.1 and H.2, we derive in Appendix A.3 the following result 7 :

Result 3.1 (First orders of the PGY expansion for Model FX)

We have, at leading order as n → ∞:

n(m + p)[Φ Y,n (η) -Φ Y,n (0)] = η √ n µ,i,l g µl m F µi m X il - η 2 2n µ,i,l r µl [v F µi (m X il ) 2 + (m F µi ) 2 v X il ] + η 2 2n µ,ν,i (g 2 µl -r µl )v F µi v X il + η 4 4n 2 µ 1 ,µ 2 l 1 ,l 2 g µ 1 l 1 g µ 2 l 1 g µ 2 l 2 g µ 1 l 2 v F µ 1 i v F µ 2 i v x il 1 v x il 2 + O(η 5 ).
Recall moreover that Φ Y,n (0) is given by

n(m + p)Φ Y,n (0) = µ,i λ F µi m F µi + γ F µi 2 v F µi + (m F µi ) 2 + ln df P F (f ) e - γ F µi 2 f 2 -λ F µi f (3.10) + i,l λ X il m X il + γ X il 2 v X il + (m X il ) 2 + ln dx P X (x) e - γ X il 2 x 2 -λ X il x + µ,l -ω µl g µl - b µl 2 -r µl + g 2 µl + ln dz e -1 2b µl (z-ω µl ) 2 2πb µl P out (Y µl |z) ,
in which one has to extremize with respect to all Lagrange multipliers.

It is enlightening to consider the fixed point equations that result from the extremization of the free entropy of Result 3.1. Note that the Lagrange multipliers only appear in the η = 0 part of the free entropy (this is a general consequence of the formalism of [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF], that we noticed already in Chapter 2), so we can easily write the maximization over these parameters:

       m F µi = E P F (λ F µi ,γ F µi ) [F ], v F µi = E P F (λ F µi ,γ F µi ) [(F -m F µi ) 2 ], m X il = E P X (λ X il ,γ X il ) [X], v X il = E P X (λ X il ,γ X il ) [(X -m X il ) 2 ], g µl = g out (Y µl , ω µl , b µl ), r µl = -∂ ω g out (Y µl , ω µl , b µl ).
(3.11) 6 Accounting for such correlations was missing in the previous computations we mentioned in Section 3.2. 7 We make extensive use of the technicalities of the PGY expansion that we derived in Chapter 2.

We defined P F (λ, γ), P X (λ, γ) and g out in eq. (2.41) in the previous chapter. We now focus on the equations obtained from the maximization over the "physical" parameters.

The series at order 2 and the approximation of [KKM + 16]

We first examine the very first orders of the perturbation series of Result 3.1. E.g. at order 2 in η, the free entropy is:

n(m + p)[Φ Y,n (η) -Φ Y,n (0)] = η √ n µ,i,l g µl m F µi m X il - η 2 2n µil r µl [v F µi v X il + v F µi (m X il ) 2 + (m F µi ) 2 v X il ] + η 2 2n µ,i,l g 2 µl v F µi v X il + O(η 3 ). (3.12)
Since they are taken at η = 0, the fixed point equations of eq. (3.11) are unchanged. The maximization over the physical parameters can be done and yields (we indicate on the right of the equation the corresponding parameter over which we maximized):

                         b µl = η 2 n i [v F µi v X il + v F µi (m X il ) 2 + (m F µi ) 2 v X il ], (r µl ) ω µl = η √ n i m F µi m X il -g µl [b µl -η 2 n i v F µi v X il ], (g µl ) γ F µi = η 2 n l [r µl v X il + r µl (m X il ) 2 -g 2 µl v X il ], (v F µi ) λ F µi = -η √ n l g µl m X il + m F µi [-γ F µi + η 2 n l r µl v X il ], (m F µi ) γ X il = η 2 n µ [r µl v F µi + r µl (m F µi ) 2 -g 2 µl v F µi ], (v X il ) λ X il = -η √ n µ g µl m F µi + m X il [-γ X il + η 2 n µ r µl v F µi ]. (m X il ) (3.13)
The reader can check easily that the combined equations (3.11) and (3.13) are actually completely equivalent to the GAMP equations derived in [KKM + 16], taking η = 1 and replacing the notations of the variables following the dictionary:

b µl → V µl ω µl → ω µl g µl → g out (Y µl , ω µl , V µl ) r µl → -∂ ω g out (Y µl , ω µl , V µl ) γ F µi → Z -1 µi λ F µi → - W µi Z µi m F µi → fµi v F µi → s µi γ X il → Σ -1 il λ X il → - T il Σ il m X il → xil v X il → c il
In conclusion, our PGY expansion truncated to order η 2 gives back exactly the stationary limit of the GAMP algorithm of [KKM + 16]! However, as we will see below, the higher order corrections of order η 4 (and beyond) cannot be neglected: this shows explicitly how the GAMP equations of [KKM + 16] (and we believe as well as the BiGAMP equations of e.g. [PSC14a, PSC14b, ZZY20], which are based on the same approximation) are missing important correlations of the problem.

Going to higher orders: open directions

While the order η 2 truncation of Result 3.1 yields back the approximation of [KKM + 16], we have computed via our PGY expansions the order η 4 , which we recall:

1 4!n(m + p) ∂ 4 Φ Y,n ∂η 4 (η = 0) = 1 4n 2 i µ 1 ,µ 2 l 1 ,l 2 g µ 1 l 1 g µ 2 l 1 g µ 2 l 2 g µ 1 l 2 v F µ 1 i v F µ 2 i v x il 1 v x il 2 + O n (1) (3.14)
A crucial observation is that in general, under the natural hypothesis H.2, the term of eq. (3.14) is not negligible as n → ∞! This can be easily seen using our diagrammatic results of Chapter 2, more precisely eq. (2.56). Indeed, assuming e.g. that the variances are all equal v F µi = v F , v X il = v X , we have:

1 4!n(m + p) ∂ 4 Φ Y,n ∂η 4 (η = 0) = ψ α + ψ (v F ) 2 (v X ) 2 4 Γ 2 α ψ , g g n + O n (1).
in which recall that the coefficients Γ p are functions of the spectrum of the matrix, and were introduced in Section 1.5. They play a role very similar to the free cumulants for symmetric random matrices. Therefore, assuming that the bulk of eigenvalues of g g/n stays of order 1 as n → ∞ (which is a natural scaling given Hypothesis H.2), the order 4 term gives a nonnegligible contribution to the free entropy! This shows in detail how the approximation of [KKM + 16] breaks down in this case.

However, our computation does not give all the orders of perturbation, as Result 3.1 is limited to order η 4 . As we saw in Chapter 2 this is an intrinsic limitation of the PGY method: there is no generic expression for an arbitrary order of perturbation. However, one can use the first orders to conjecture the form of the higher-order terms. The precise analysis of all these terms is still under investigation, and will be a subject of [MFK + 21]. We end this discussion by a few remarks on the results presented here:

• Similarity with finite-rank problems -The watchful reader would have noticed that the first orders of Result 3.1 are very similar the the TAP free entropy of GLMs that we derived in Chapter 2, more precisely eq. (2.38). A crucial difference is that here the role of the sensing matrix is played by -g, which is itself a parameter of the TAP free entropy. While we do not yet have an expression for the higher orders in Result 3.1, this similarity is already striking. Importantly, the resummation of eq. (2.39) then suggests that the fully-expanded free entropy might be expressed solely in terms of the singular value distribution of g/ √ n.

• Nature of the order parameter -In the Gaussian setting, one can perform exact calculations that leverage extensive-rank HCIZ integrals (cf. Section 1.5.3). While these calculations are still under investigation on our part, and are therefore not presented in this thesis, they suggest that the order parameter governing the state of the model is a spectral density. This corroborates our intuition (cf. the previous remark) that the full TAP free entropy might be expressed in terms of the singular value distribution of g/ √ n. This is an important difference from the finite-rank case: for rank-k matrix factorization (or in general for rank-k recovery problems), the state of the system is governed by a k × k overlap matrix. For instance, such a matrix order parameter will be crucial to understand simple neural networks in Chapter 4.

• Iterating the equations -As we discussed in Section 2.3, the TAP equations are in general not sufficient to obtain an algorithm with good convergence properties. A classical example is given by the Generalized Vector Approximate Passing algorithm (G-VAMP) [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF], for which the corresponding TAP equations are derived in Section 2.3.1. As highlighted there, the TAP equations correspond to the stationary limit of G-VAMP, however there is no obvious iterative resolution scheme of the TAP equations that gives the GVAMP algorithm. This indicates that even if one obtains all the orders of perturbations in Result 3.1, turning them into an efficient algorithm will pose a serious challenge. We will discuss further these points in [MFK + 21].

Symmetric matrix factorization

Performing the Plefka-Georges-Yedidia expansion for Model XX is extremely similar to the calculation done in previous sections for Model FX: one can introduce a field h = XX / √ n, and then performs the same calculations via the Fourier transform of the delta function. In the following of Section 3.3.4, we give the results of our derivation, while more details are given in Appendix A.4. We adopt similar notations to the ones of Section 3.3.1, removing the X, F subscripts. More precisely, we impose the first and second moment constraints, for µ < ν and i = 1, • • • , n:

X µi = m µi , X 2 µi = v µi + m 2 µi , (iH) µν = -g µν , (iH) 2 µν = -r µν + g 2 µν .
(3.15)

Note that we will sometimes symmetrize the quantities involved, e.g. we write g µν ≡ g νµ for µ > ν, and moreover we adopt the convention g µµ = r µµ = 0.

First orders of the PGY expansion

Similarly to Result 3.1, we obtain the first orders of the free entropy as:

Result 3.2 (First orders of the PGY expansion for Model XX )

At leading order as n, m → ∞:

nm[Φ Y,n (η) -Φ Y,n (0)] = η √ n µ<ν i g µν m µi m νi - η 2 2n µ<ν i r µν [v µi v νi + v µi m 2 νi + m 2 µi v νi ] + η 2 4n µ,ν,i g 2 µν v µi v νi + η 3 6n 3/2 i µ 1 ,µ 2 ,µ 3 pairwise distinct g µ 1 µ 2 g µ 2 µ 3 g µ 3 µ 1 v µ 1 i v µ 2 i v µ 3 i + η 4 8n 2 i µ 1 ,µ 2 ,µ 3 ,µ 4 pairwise distinct g µ 1 µ 2 g µ 2 µ 3 g µ 3 µ 4 g µ 4 µ 1 v µ 1 i v µ 2 i v µ 3 i v µ 4 i + O(η 5 ).
Recall that the term Φ Y,n (0) contains the dependency on the channel and priors contributions, as well as the Lagrange multipliers introduced to enforce the conditions of eq. (3.15). Its precise form can be easily deduced from its counterpart in the non-symmetric case, eq. (3.10).

Higher-order terms, and breakdown of previous approximations

One can check that the approximation of [KKM + 16, PSC14a, PSC14b, ZZY20] can be adapted easily to Model XX , and as in Section 3.2, this approximation amounts to truncating the perturbation series of Result 3.2 at order η 2 . However, as in Section 3.3.3, the higher-order terms in Result 3.2 are in general non-negligible. Under a similar hypothesis as H.2, they are actually related to the free cumulants of g/ √ n. This is a consequence of Theorem 2.2, so that assuming e.g. v µi = v, we have at η = 0:

                 1 3!nm ∂ 3 η Φ Y,n = v 3 6n 3/2 m µ 1 ,µ 2 ,µ 3 pairwise distinct g µ 1 µ 2 g µ 2 µ 3 g µ 3 µ 1 = v 3 6 c 3 g √ n + O n (1), 1 4!nm ∂ 4 η Φ Y,n = v 4 8n 2 m µ 1 ,µ 2 ,µ 3 ,µ 4 pairwise distinct g µ 1 µ 2 g µ 2 µ 3 g µ 3 µ 4 g µ 4 µ 1 = v 4 8 c 4 g √ n + O n (1). (3.16)
Again, in general these free cumulants are non-negligible in the limit n → ∞: our PGY expansion allowed us to precisely point out the breakdown of previous approximations.

Conclusion of Chapter 3

In this chapter, we presented an application of the very generic formalism of high-temperature PGY expansions (introduced in Chapter 2) to the problem of extensive-rank matrix factorization. Our main finding is that the previous statistical-physics-based approaches to this problem [KKM + 16, PSC14a, PSC14b, ZZY20] rely on an approximation that fails in this setting. Via the PGY expansions, we are able to derive systematic corrections to this approximation. While the present chapter does not provide a complete investigation of our corrected equations, these results already raise several important possible generalizations or extensions:

• Of course, the most natural open question is to generalize Results 3.1,3.2 to arbitrary orders of perturbation. Let us focus on Model XX for the sake of this argument. One could be tempted to generalize eq. (3.16) to any order in perturbation by conjecturing that the order k is related to the k-th free cumulant of g/ √ n. The resulting expansion is then very similar to the finite-rank series of eq. (2.12), with g playing the role of the coupling matrix. However, justifying such a wild conjecture (provided it is correct, which is far from obvious) is quite involved, and will be one of the main subjects of [MFK + 21].

• In the Gaussian setting, using the results of Matytsin [START_REF] Matytsin | On the large-N limit of the Itzykson-Zuber integral[END_REF] presented in Section 1.5.3, one can perform exact calculations of the free entropy. Although these exact calculations are here very involved, they would be an important check of our PGY expansion. We have already advanced significantly on this line of work, and it will be an important part of [MFK + 21].

• It would also be interesting to perform a perturbative calculation (for instance in the symmetric model XX ), with a slowly-diverging rank r = n/m (expanding the free entropy in powers of r). Similarly to the PGY expansion, this could provide another systematic correction to the results of [KKM + 16], with a different point of view than the one presented in this chapter. Such a calculation is related to an important open problem in random matrix theory, that is a sharp description of the transition between the low-rank spherical HCIZ integral of Theorem 1.14 and the extensive-rank case described in Theorem 1.15. Solving this later problem would help to describe the transition between low-rank and extensive-rank results in the PGY expansion.

• Another interesting open problem is to relate the PGY-expanded free entropy described in Section 3.3 to other approximation schemes. Indeed, as we discussed in Chapter 2, in finite-rank problems these expansions are equivalent to other techniques, e.g. the Expectation Consistency or adaptive TAP approaches. This equivalence does not seem to easily transfer to the extensive-rank case: understanding how to apply these approaches here is another interesting direction of research.

Part II

Physics joins probability: all you need for optimal estimation

A selection of high-dimensional problems Chapter 4

The physics of learning in a two-layers neural network "Do you remember the question that caused the creators to attack us, Tali'Zorah? Does this unit have a soul?"

Legion, Mass Effect 3 (2012).

Disclaimer -In this chapter, we begin our tour of high-dimensional problems by a simple model of neural networks. More precisely, we will unveil and rigorously prove phase transitions in the algorithmic and information-theoretic optimal performances in a two-layers neural network known as the committee machine in the statistical physics literature. We leverage the replica method, and we put the replica results on rigorous grounds using sophisticated probabilistic interpolation techniques. We will also apply our message-passing toolbox to design an AMP algorithm for this problem, that achieves the optimal performance among a large class of iterative algorithms. All in all, this chapter can be considered as a textbook application of a large part of the statistical physics machinery that we introduced in Chapter 1. It is based on the published work [AMB + 19].

Introduction: the committee machine

While the traditional approach to learning and generalization follows the Vapnik-Chervonenkis [START_REF] Vapnik | Statistical learning theory[END_REF] and Rademacher [BM02, AAKZ20] worst-case type bounds, there has been a considerable body of theoretical work on calculating the generalization ability of neural networks for data arising from a probabilistic model. Such settings fit well within the framework of statistical mechanics, and the 1990s saw a burst of such studies applied to neural networks [SST92, WRB93, MZ95a, MZ95b, EVdB01]. As the modern success of neural networks has only increased the need to understand their effectiveness [ZBH + 16], it is of interest to revisit the results that have emerged thanks to the physics perspective. This direction has been experiencing an important revival in the past years, see e.g. [BJSG + 18, BKM + 19, CCS + 19, Gab20], and this thesis as a whole inscribes itself in this line of work.

As we discussed in Section 1.1, the physics approach is particularly suited to study these models in the so-called teacher-student setting. Precisely, labels are generated by feeding i.i.d. random samples to a neural network architecture (the teacher) and are then presented to another neural network (the student) that is trained using these data. Early studies computed the information theoretic limitations of the supervised learning abilities of the teacher weights by a student who is given m independent n-dimensional examples with α ≡ m/n = Θ(1) and n → ∞ [SST92, [START_REF] Timothy Lh Watkin | The statistical mechanics of learning a rule[END_REF][START_REF] Engel | Statistical mechanics of learning[END_REF]. These works relied on non-rigorous heuristic approaches, such as the replica and cavity methods, cf. Sections 1.3.1 and 1.4. Additionally no provably efficient algorithm was provided to achieve the predicted learning abilities, and it was thus difficult to test those predictions, or to assess the computational difficulty.

Recent developments in statistical estimation and information theory -in particular of messagepassing algorithms (cf. [DMM09, Ran11, BM11, JM13] and Section 1.4), and a rigorous proof of the replica formula for the free entropy [BKM + 19]-allowed to settle these two missing points for single-layer neural networks (i.e. without any hidden variables).

In the present chapter we leverage many of these works, and provide rigorous asymptotic predictions and corresponding message-passing algorithm for a wide class of two-layers networks. While our results hold for a rather large class of non-linear activation functions, we illustrate our findings on a case considered most commonly in the early literature: the committee machine. This is possibly the simplest version of a two-layers neural network where all the weights in the second layer are fixed to unity, and we illustrate it in Fig. 4.1. Denoting Y µ the label associated with a n-dimensional sample X µ , and W * il the weight connecting the i-th coordinate of the input to the l-th node of the hidden layer, it is defined by:

Y µ = sign K l=1 sign n i=1 X µi W * il . (4.1) {(X µi ) n i=1 } m µ=1 m samples in R n W * il ∈ R n×K Y µ W (2) ∈ R K f (1) f (1) f (1) f (2)
K hidden units output The committee machine is one of the simplest models of two-layers neural networks. Both activation functions are sign functions f (1) , f (2) = sign and the second-layer weights W (2) are all fixed to 1. Here we represent a generic two-layers neural network with K = 3.

In our teacher-student scenario, the teacher generates i.i.d. data samples X µi ∼ N (0, 1), then she/he generates the associated labels Y µ using a committee machine as in eq. (4.1), with weights W * il which are unknown to the student. The goal of the student is to learn the weights W * il from the available examples (X µ , Y µ ) m µ=1 in order to reach the smallest possible generalization error (i.e. to be able to predict the label the teacher would generate for a new sample not present in the training set).

Classical physics predictions

There have been several studies of this model within the non-rigorous statistical physics approach in the limit where α ≡ m/n = Θ(1), K = Θ(1) and n → ∞. In particular, the committee machine attracted a lot of interest during the 1990s, and the reader may refer e.g. to the following series of works: [Sch93, SH92, SH93, MP92, MZ95b, EVdB01]. A particularly interesting result in the teacher-student setting is the specialization of hidden neurons (see Sec. 12.6 of [START_REF] Engel | Statistical mechanics of learning[END_REF], or [START_REF] Saad | Online learning in soft committee machines[END_REF] in the context of online learning). These works predicted that for α < α spec (where α spec is a certain critical value of the sample complexity), the permutational symmetry between hidden neurons remains conserved even after an optimal learning, and the learned weights of each of the hidden neurons are identical. For α > α spec , however, this symmetry gets broken as each of the hidden units correlates strongly with one of the hidden units of the teacher. Another remarkable result obtained in this early literature is the calculation of the optimal generalization error as a function of α.

Main contributions of this chapter

In this chapter we will go beyond the mentioned literature in two main directions. Our first contribution consists in a proof of the replica-symmetric formula for the free entropy, which was conjectured in the statistical physics literature. This proof uses an adaptive interpolation method [BM19a, BKM + 19], that allows to put several of these results on a rigorous basis. Secondly, we design an Approximate-Message-Passing-type algorithm (recall that we introduced such algorithms in Section 1.4) that is able to achieve the optimal generalization error for a wide range of parameters. The study of AMP -that is widely believed to be optimal among all polynomial-time algorithms in the above setting [DJM13, DM15, ZK16, BPW18]-unveils, in the case of the committee machine with a large number of hidden neurons, the existence a large hard phase. In this phase, learning is information-theoretically possible, leading to a generalization error decaying asymptotically as O(K/α) (in the α = Θ(K) regime), but where AMP fails and provides only a poor generalization that does not go to zero when increasing α. This strongly suggests that no efficient algorithm exists in this hard region and therefore that there is a computational gap in learning in such neural networks. In other problems where a hard phase was identified its study boosted the development of algorithms that are able to match the predicted thresholds, and we hope that this study will contribute in this regard.

Main theoretical results

General probabilistic model

Our theoretical analysis is performed for a class of models much broader than the specific committee machine of eq. (4.1). Namely we consider

Model 4.1 ("Generalized" committee machine, Bayes-optimal setting)

The observer is given m input samples {(X µi ) n i=1 } m µ=1 i.i.d.

∼ N (0, 1) and a probabilistic output channel P out (y|z). Our aim is to recover a set of teacher weights W ∈ R n×K , generated i.i.d. from a prior distribution P 0 , from the observations

Y µ ∼ P out • 1 √ n n i=1 X µi W * il K l=1 .
We assume that P 0 has zero mean, and we denote ρ ≡ E[W 0 W 0 ] its covariance matrix. As we know, in the Bayes-optimal setting, the best strategy for the student is to learn W * from the data (X µ , Y µ ) m µ=1 by computing the marginal means of the posterior probability distribution. A word on notations -In this chapter, we will often manipulate quantities that live either in R K or in R K×K . To lighten the notations, we do not use bold symbols for vectors and matrices of size K, while we keep bold symbols for quantities that have diverging size with n.

Alternative view -Another equivalent way to generate the observations is via

Y µ = ϕ out 1 √ n n i=1 X µi W * il K l=1 , A µ , (4.2)
where ϕ out : R K × R → R is a generic functions, and (A µ ) m µ=1 are i.i.d. real valued random variables with known distribution P A , that form the probabilistic part of the model, generally accounting for noise.

Different scenarii fit into the general framework of Model 4.1. Among those, the committee machine of eq. (4.1) is obtained when choosing ϕ out (h) = sign( K l=1 sign(h l )). Another renowned model is the parity machine, which corresponds to ϕ out (h) = K l=1 sign(h l ), see e.g. [START_REF] Engel | Statistical mechanics of learning[END_REF]. We will not investigate this last model in this chapter, but the interested reader can refer to [AMB + 19] in which we conducted this analysis. A number of layers beyond two has also been considered in specific regimes in [START_REF] Mato | Generalization properties of multilayered neural networks[END_REF]. Using other activation functions ϕ out , one can describe many more problems, e.g. compressed pooling [EARK + 18, ARK + 19] or multi-vector compressed sensing [START_REF] Zhu | Performance limits for noisy multimeasurement vector problems[END_REF].

Picking from the toolbox

Now that we fixed our model of interest, we can pick our favorite tool from the statistical physics toolbox to study optimal learning. In this chapter, we leave aside the TAP approach (the reader interested in the latter can refer to Chapters 2 and 3) and we focus instead on two other methods.

• First, we leverage the replica method that we introduced in Section 1.3.1. Recall that the aim of this method is the compute the large n limit of the free entropy f n of Model 4.1, i.e. the log-normalization of the posterior that is written in eq. (4.5). It allows to obtain an explicit (conjectural) expression of f n in the high-dimensional limit n, m → ∞ with α = m/n fixed, called the replica-symmetric (RS) formula. One can then naturally ask: can we prove said conjecture? While proving the replica method itself seems out of reach, in this chapter we manage to prove its prediction by use of probabilistic methods: this is summarized by Theorem 4.1, which is the main theoretical result of this chapter. For this reason, while we first derived the results of Theorem 4.1 with the replica method, we focus in this chapter on its probabilistic proof, while the replica computation is detailed in Appendix B.1.

• The second tool, complementary to the first, is the use of message-passing algorithms to assess the optimal algorithmic (among a large class of first-order methods, see Section 1.4.2) performance. We introduced these approaches in Section 1.4, and we will extend them to the present setting in Section 4.3.

We hope that this chapter will illustrate to the reader that by leveraging our toolbox one can gain a precise understanding of the computational and statistical optimal performances in a broad class of inference models, and that it can be instrumental in gaining a deeper comprehension of learning in neural networks.

Main theorem: the replica-symmetric formula

Some notation -Recall that S + K is the set of semi-definite positive real symmetric K × K matrices. For any M ∈ S + K , we can uniquely define its square root

√ M = M 1/2 . Let us define for N ∈ S + K (R), the set S + K (N ) ≡ {M ∈ S + K s.t. N -M ∈ S + K }. Note that S + K (N ) is both convex and compact.

Two auxiliary inference problems

Stating the RS formula requires introducing two simpler K-dimensional estimation problems:

• The first one is a function of the prior P 0 of Model 4.1. It consists in retrieving a K-dimensional

vector W 0 ∼ P 0 from the K-dimensional observations Y 0 = r 1/2 W 0 + Z 0 , with Z 0 ∼ N (0, I K )
and the "channel gain" matrix r ∈ S + K . The posterior distribution on w is given by

P(w|Y 0 ) = 1 Z P 0 P 0 (w)e Y 0 r 1/2 w-1 2 w rw , (4.3)
and the associated free entropy is given by ψ P 0 (r) ≡ E ln Z P 0 .

• The second problem is a function of the channel P out of Model 4.1. We consider K-dimensional i.i.d. vectors V, U * ∼ N (0, I K ) where V is considered to be known and one has to retrieve U * from the observation of

Y 0 ∼ P out ( • |q 1/2 V + (ρ -q) 1/2 U * ),
with q ∈ S + K (ρ) known as the overlap matrix, for reasons which will become clear later on. The associated posterior is

P(u| Y 0 , V ) = 1 Z Pout e -1 2 u u (2π) K/2 P out Y 0 |q 1/2 V + (ρ -q) 1/2 u , (4.4)
and the free entropy reads this time Ψ out (q; ρ) ≡ E ln Z Pout .

In Appendix C.2 (more precisely Lemmas C.2 and C.3), we prove regularity and convexity properties of ψ P 0 and Ψ out which will be useful for our analysis.

The free entropy

As we know from Section 1.1, the central object to study optimal performances is the posterior distribution of the weights:

P(w|X, Y) = 1 Z n n i=1 P 0 (w i ) m µ=1 P out Y µ 1 √ n n i=1 X µi w il K l=1 , (4.5)
The expected free entropy is by definition

f n ≡ (1/n)E ln Z n .
Recall that we consider the thermodynamic limit n, m → ∞ with m/n → α > 0. The main theoretical result of this chapter is a rigorous proof of the free entropy formula predicted by the replica method, using a probabilistic interpolation method developed in [START_REF] Guerra | Broken replica symmetry bounds in the mean field spin glass model[END_REF][START_REF] Talagrand | Spin glasses: a challenge for mathematicians: cavity and mean field models[END_REF][START_REF] Barbier | The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference[END_REF].

We define the replica symmetric (RS) potential as

f RS (q, r) ≡ ψ P 0 (r) + αΨ out (q; ρ) - 1 2 Tr(rq), (4.6)
where α ≡ m/n, and Ψ out (q; ρ) and ψ P 0 (r) are the free entropies of the two simpler Kdimensional estimation problems (4.3) and (4.4). We moreover assume the following:

H.1 The prior P 0 has bounded support in R K . Recall that ρ = E P 0 [W 0 W 0 ].

H.2

The activation ϕ out : R K × R → R is a bounded C 2 function with bounded first and second derivatives w.r.t. its first argument.

H.3 X µi i.i.d.
∼ N (0, 1).

Theorem 4.1 (Replica formula)

Suppose H.1, H.2 and H.3. The asymptotic free entropy is:

lim n→∞ f n ≡ lim n→∞ 1 n E ln Z n = sup r∈S + K inf q∈S + K (ρ)
f RS (q, r) .

Theorem 4.1 is an extension of the results of [BKM + 19] which studied usual generalized linear models (GLMs), i.e. the case K = 1 of this chapter. Our proof is based on an adaptive interpolation method developed in [START_REF] Barbier | The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference[END_REF], and is outlined in Section 4.4. We conclude the presentation of the theorem by two remarks.

Remark 4.1 (Regularization)

Theorem 4.1 actually stands under the addition of an (arbitrarily small) Gaussian regularization noise to the model (4.2), which thus becomes (with ∆ > 0, Z µ ∼ N (0, 1))

Y µ = ϕ out 1 √ n n i=1 X µi W * il K l=1 , A µ + √ ∆Z µ .
In general this regularizing noise is needed for the free entropy limit to exist in noiseless scenarios. Some exceptions exist, such as the case of discrete outputs (e.g. in the committee machine of eq. (4.1)), for which we can safely take ∆ = 0. The reader can refer to [BKM + 19] for more details on this technical point.

Remark 4.2 (Relaxing the hypotheses)

Following similar approximation arguments as in [BKM + 19], H.1 can be relaxed to the existence of the second moment of the prior, thus covering e.g. the Gaussian prior case. H.2 can moreover be dropped (so that our results include the committee machine (4.1)), and H.3 can be extended to X having i.i.d. entries of zero mean, unit variance and finite third moment.

Optimal generalization error

As we know, in the Bayes-optimal setting the estimator Ŵ that minimizes the mean-square error with the ground-truth W * is given by the expected mean of the posterior distribution of eq. (4.5). Denoting q * the extremizer in the replica formula of Theorem 4.1, we expect from the replica method that with high probability over the quenched variables, Ŵ W * /n p → q * . This phenomenon is known as overlap concentration, and will be discussed in the proof of Theorem 4.1.

From the overlap matrix q * , one can compute the Bayes-optimal generalization error when the student tries to classify a new, yet unseen, sample X new ∈ R n . Indeed, the estimator of the new label Ŷnew that minimizes the mean-squared error with the true label is given by the posterior mean of ϕ out (X new w). The Bayes-optimal generalization error is then

1 2 E X,W * E w|X,Y ϕ out (X new w) -ϕ out (X new W * ) 2 ---→ n→∞ g (q * ), (4.7)
where w is distributed according to the posterior measure of eq. (4.5) 1 . In particular, when the distribution of X is rotationally invariant, the asymptotic generalization error only depends on the overlap matrix w W * /n p → q * . In specific cases (e.g. when K = 2 or in the limit K → ∞ as we will see), one can derive explicit formulas for the function g (q ). In Appendix D.1 we give more details on possible definitions of the generalization error, and derive its expression for K = 2. The analytical treatment of the limit K → ∞ in g (q * ) is detailed in Appendix D.2, which is devoted to this limit.

Investigating computational-to-statistical gaps

Approximate Message-Passing

In order to investigate optimal algorithmic performances in this model, we pick the appropriate tool from our toolbox. As we discussed in Section 1.4, the natural candidate to design an algorithm that is optimal among a very wide class of polynomial-time algorithms is Approximate Message-Passing (AMP). While AMP was initially developed to solve random instances of generalized linear models with Gaussian data matrices [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF][START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF], here we adapt the algorithm for the committee machine, and more generically for models described by eq. (4.2). As in generalized linear models, in the Bayes-optimal setting AMP is strongly conjectured to perform the best among all polynomial algorithms (and shown to be optimal among the large class of general first order methods [START_REF] Celentano | The estimation error of general first order methods[END_REF]), in terms of achieved overlap. It thus gives us a tool to evaluate both the intrinsic algorithmic hardness of learning and the performance of existing algorithms with respect to the optimal one. Algorithm 3: Approximate Message-Passing iterations for the committee machine

Input : A vector Y ∈ R m , a data matrix X ∈ R m×n Output: An estimator Ŵ ∈ R n×K Initialize variables randomly; while Ŵt - Ŵt-1 > do Update of the mean ω µ ∈ R K and covariance V µ ∈ S + K : ω t µ = n i=1 X µi √ n Ŵ t i - X 2 µi n Σ t-1 i -1 Ĉt i Σ t-1 i g t-1 out,µ | V t µ = n i=1 X 2 µi n Ĉt i ; Update of g out,µ ∈ R K and ∂ ω g out,µ ∈ S + K : g t out,µ = g out (ω t µ , Y µ , V t µ ) | ∂ ω g t out,µ = ∂ ω g out (ω t µ , Y µ , V t µ ); Update of the mean T i ∈ R K and covariance Σ i ∈ S + K : T t i = Σ t i m µ=1 X µi √ n g t out,µ - X 2 µi n ∂ ω g t out,µ Ŵ t i | Σ t i = - m µ=1 X 2 µi n ∂ ω g t out,µ -1 ;
Update of the estimated marginals Ŵi ∈ R K and Ĉi ∈ S + K :

Ŵ t+1 i = f w (Σ t i , T t i ) | Ĉt+1 i = f c (Σ t i , T t i ); t = t + 1; end
The AMP algorithm for generalized committee machines is summarized in Algorithm 3. The update functions used in this algorithm are defined as:

• The functions f w (Σ, T ) and f c (Σ, T ) are respectively the mean and variance under the posterior distribution of the auxiliary model of eq. (4.3), when r → Σ -1 and Y 0 → Σ 1/2 T .

• g out (ω µ , Y µ , V µ ) is related to the mean of posterior of the second auxiliary model, i.e. eq. (4.4). More precisely one can define it as:

g out (ω, y, V ) ≡ 1 V R K (z -ω)e -1 2 (z-ω) V -1 (z-ω) P out (y|z)dz R K e -1 2 (z-ω) V -1 (z-ω) P out (y|z)dz . (4.8)
The detailed derivation of Algorithm 3 from loopy belief propagation is quite tedious. While we do not detail it in this thesis, the interested reader may refer to [AMB + 19]. After convergence, Ŵ estimates the weights of the teacher neural network. A demonstration code of the algorithm is available on GitHub [AMB + 18].

Let us briefly recall some properties of AMP that we highlighted in Section 1.4. A major strength of AMP is that its performance can be tracked rigorously in the asymptotic limit n → ∞, via a procedure known as State Evolution (SE) [START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling[END_REF]. In the present model, SE allows to track the value of the overlap between the teacher weights W * and the AMP estimate Ŵt , defined as q t AMP ≡ lim n→∞ ( Ŵt ) W * /n, via the iteration of the following equations:

q t+1 AMP = 2∇ψ P 0 (r t AMP ) , r t+1 AMP = 2α∇Ψ out (q t AMP ; ρ) . (4.9)
A crucial property of State Evolution (that is trivial to derive from definition) is that its fixed points exactly correspond to the critical points of the replica-symmetric potential of eq. (4.6). This allows to treat both the information-theoretic optimal performance (via Theorem 4.1) and the AMP performance (via eq. (4.9)) in a single framework. We will illustrate the power of this correspondence in the following Section 4.3.2.

On the convergence of the algorithm -In the large n limit, and if all functions are computed without errors, the algorithm is guaranteed to converge. This is a consequence of the Bayes-optimal setting, as detailed in [AMB + 19]. In practice, of course, n is finite and Kdimensional integrals need to be approximated. In that case convergence is not guaranteed, but is robustly achieved in all the cases presented in this chapter. We also expect (by experience with the single layer case) that if the input-data matrix X has more structure than simply i.i.d. then the AMP described here would encounter convergence issues. Such issues could however be fixed by moving to refined variants of the algorithm, such as Vector Approximate Message-Passing (VAMP) [START_REF] Sundeep Rangan | Vector approximate message passing[END_REF], cf. Section 1.4 for an introduction to the VAMP algorithm.

Studying a "VAMP"-like algorithm in the context of the committee machine with structured data is however not in the scope of this chapter.

From two to more hidden neurons, and the specialization transition

A network with two hidden neurons

Let us now discuss how the above results (i.e. Theorem 4.1 and the AMP algorithm) can be used to study optimal learning in a simple (yet non-trivial) two-layers neural network with two hidden neurons. Concretely, we consider the following special case of Model 4.12 :

Y µ = sign sign n i=1 X µi W * i1 + sign n i=1 X µi W * i2 .
In Fig. 4.2 we illustrate our results for this model. In the left panel the weights are Gaussian (i.e. P 0 = N (0, 1)), while in the right panel they are binary (or Ising/Rademacher, i.e. P 0 = (δ -1 + δ 1 )/2). We plot in Fig. 4.2 several quantities:

• In black we show the generalization error. Full lines are obtained as global maximizers of the replica potential of eq. (4.6), while dashed lines are obtained from the fixed point of the State Evolution (SE) of the AMP algorithm (i.e. eq. (4.9)). Note that it corresponds to the local maximizer of the replica-symmetric potential of eq. (4.6), when iterating the SE equations starting at q = 0. Dots are finite-size simulations of the AMP algorithm. Overlap q AMP q 00 AMP q 01 SE AMP q 00 SE AMP q 01

Bayes-optimal g (α)

SE AMP g (α)
AMP g (α)

Specialization

Figure 4.2: Generalization error and overlap matrix elements for a committee machine with two hidden neurons (K = 2) with Gaussian weights (left) and binary/Rademacher weights (right), as a function of the sampling ratio α = m/n. q 00 and q 01 denote diagonal and off-diagonal overlaps, and their values are given by the labels on the far-right of the figure.

• In blue and orange we show the diagonal and off-diagonal element of the matrix overlap q achieved by AMP. Solid lines are obtained again using the SE equations (4.9), while dots are simulations of AMP. Finally, in green dashed-dot line we show the specialization point at which the diagonal and off-diagonal overlaps start differing.

We ran the AMP algorithm with n = 10 4 until convergence, and averaged over 10 instances with different random initial conditions. As expected by our theory we observe excellent agreement between the SE and AMP. Let us now comment further on our results.

Specialization -In both panels of Fig. 4.2 we observe the so-called specialization phase transition. Indeed eq. (4.9) has two types of fixed points: a non-specialized fixed point where every matrix element of the K × K order parameter q is the same (so that both hidden neurons learn the same function), and a specialized fixed point where the diagonal elements of the order parameter are different from the non-diagonal ones. We checked for other types of fixed points for K = 2 (one where the two diagonal elements are not the same), but have not found any.

In terms of weight-learning, this means for the non-specialized fixed point that the estimators for both W 1 and W 2 are the same, whereas in the specialized fixed point the estimators of the weights corresponding to the two hidden neurons are different: the network "figured out" that the data is better described by a model that is not linearly separable. The specialized fixed point is associated with lower error than the non-specialized one (as one can see in Fig. 4.2). The existence of such a phase transition was previously discussed in the statistical physics literature, see e.g. [START_REF] Schwarze | Generalization in a large committee machine[END_REF][START_REF] Saad | Online learning in soft committee machines[END_REF]. Interestingly, one can show that specialization is absent (for arbitrary K) when the activations are linear, as detailed in [AMB + 19]. The non-linearity of the activation functions is therefore an essential ingredient in order to observe a specialization transition.

Gaussian weights -For Gaussian weights (Fig. 4.2 left), the specialization phase transition arises continuously at α G spec (K = 2) 2.04. This means that for α < α G spec (K = 2) the number of samples is too small, and the student network is not able to learn that two different teachervectors W 1 and W 2 were used to generate the observed labels. For α > α G spec (K = 2), however, it is able to distinguish the two different weight-vectors and the generalization error decreases fast to low values.

Binary weights -

The right part of Fig. 4.2 depicts the case of binary weights. We observe two phase transitions in the performance of AMP in this case: (a) the specialization phase transition at α B spec (K = 2) 1.58, and for slightly larger sample complexity a transition towards perfect generalization (beyond which the generalization error is asymptotically zero) at α B perf (K = 2) 1.99. The binary case with K = 2 differs from the Gaussian one in the fact that perfect generalization is achievable at finite α. While the specialization transition is continuous here, the error has a discontinuity at the transition of perfect generalization. This discontinuity is associated with a 1 st order phase transition (in the physics nomenclature), leading to a gap between algorithmic performance and information-theoretically optimal performance. To quantify said optimal performance we need to evaluate the global maximum of the replica free entropy (not the local one reached by the state evolution). In doing so that we discover that informationtheoretically there is a single discontinuous phase transition towards perfect generalization at α B IT (K = 2) 1.54. Computational-to-statistical gap -While the information-theoretic and specialization phase transitions were partially discussed in the physics literature on the committee machine [SH92, SH93, SST92, WRB93], the gap between the information-theoretic performance and the performance of AMP -that is conjectured to be optimal among polynomial-time algorithms-was not previously discussed for such neural networks. Indeed, even its understanding in simpler models than those discussed here, such as the single layer case, is quite recent [DMM09, DJM13, ZK16]. The existence of similar gaps (or their absence) will also be among the core results of Chapters 5 and 6.

K

1: when more is different

It becomes increasingly difficult to study the replica formula for larger values of K as it involves (at least) K-dimensional integrals. Quite interestingly, it is however possible to work out the solution of the replica formula in the large K limit (taken after the large n limit). Indeed, when the activations functions of all hidden units are identical, it is natural to look for solutions of the replica formula of the form q = q d I K + (q a /K)1 K 1 K , with the unit vector 1 K = (1) K l=1 . Such solutions are what we call committee symmetric. To simplify the argument, we assume that the prior P 0 has covariance ρ = I K . Since both q and ρ -q are positive, the committee-symmetry assumption implies that 0 ≤ q d ≤ 1 and 0 ≤ q a + q d ≤ 1 in the large-K limit.

In Appendix D.2, we detail the corresponding large K expansion of the free entropy for the teacher-student scenario with Gaussian weights, and sign activation functions (the model of eq. (4.1)). While the information-theoretically optimal generalization error was previously computed [START_REF] Schwarze | Learning a rule in a multilayer neural network[END_REF], we concentrated on the analysis of the performance of AMP by tracking the state evolution equations. In doing so, we unveil a large computational gap.

Description of our results -Our results at large K are presented in Fig. 4.3. In the right panel we show the fixed point values of the two overlaps q 00 = q d +q a /K and q 01 = q a /K. The resulting generalization error is plotted in the left panel. As shown in Appendix D.2 it can be written as g = arccos[2(q a + arcsin q d )/π]/π. The information-theoretic specialization transition arises for α = Θ(K), so we define α ≡ α/K. Contrary to the case K = 2, specialization is here a 1 st order phase transition, meaning that the specialization fixed point first appears at α G spinodal 7.17 but the free entropy global extremizer remains the one of the non-specialized fixed point until α G spec 7.65. This has interesting implications for the optimal generalization error that gets towards a plateau of value ε plateau 0.28 for α < α G spec and then jumps discontinuously down. For large α, the Bayes-optimal error decays asymptotically as 1/ α, i.e. IT g ( α) = Θ(1/ α). A large computational gap -As we have already discussed throughout this thesis, AMP is conjectured to be optimal among a wide class of polynomial-time algorithms [START_REF] Celentano | The estimation error of general first order methods[END_REF], and thus analyzing its state evolution sheds light on possible computational gaps, that come hand in hand with 1 st order phase transitions. In the regime α = O(K 2 ) for large K, we find in Appendix D.2 Overlap q * Bayes-optimal g ( α)

AMP g ( α)
Spinodal transition 1 st order specialization Bayes-optimal q 00 Bayes-optimal q 01 Figure 4.3: (Left) Bayes-optimal and AMP generalization errors and (right) diagonal and off-diagonal overlaps q 00 , q 01 for a committee machine with a large number of hidden neurons K and Gaussian weights, as a function of the rescaled parameter α = α/K. Solutions corresponding to global and local maxima of the replica free entropy are respectively represented with full and dashed lines. The dotted line marks the spinodal transition at α G spinodal 7.17, i.e. the apparition of a local minimum in the replica free entropy, associated to a solution with specialized hidden units. The dotted-dashed line shows the first order specialization transition at α G spec 7.65, at which the specialized fixed point becomes the global minimum. For α < α G spec , AMP reaches the Bayes-optimal generalization error and overlaps, corresponding to a non-specialized solution. However, for α > α G spec , the AMP algorithm does not follow the optimal specialized solution and is stuck in the non-specialized solution plateau, represented with dashed lines. Hence it unveils a large computational gap (yellow area).

that the non-specialized fixed point is always stable, implying that AMP will not be able to give a lower generalization error than ε plateau corresponding to a non-specialized solution. However, information-theoretically a specialization transition arises in the scale α = Θ(K): this implies a large gap between the performance that can be reached information-theoretically and the one reachable tractably, see the yellow area in Fig. 4.3. Such large computational gaps have been previously identified in a range of inference problems -most famously in the planted clique problem [DM15]-but the committee machine is the first model of a multi-layer neural network with realistic non-linearity that presents such a large gap.

Proof of the replica formula: adaptive interpolation

In this section, we sketch the proof of Theorem 4.1 by doing a small excursion closer to the realm of probability. We use an adaptive interpolation method, based on an original idea of Guerra [START_REF] Guerra | Broken replica symmetry bounds in the mean field spin glass model[END_REF], and refined by Barbier and collaborators, see [START_REF] Barbier | The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference[END_REF] for a review on this technique. In this regard, the interested reader can also consider this section as an introduction to this powerful technique, in a quite involved context3 . Details of proof will often be postponed to Appendix C, as we focus on describing the method.

All along this section we assume H.1, H.2 and H.3, and all our statements will implicitly assume these hypotheses. We also define a few quantities for the remaining of this section:

• For µ = 1, . . . m, let V µ , U * µ be two vectors drawn from N (0, I K ).

• Let s n ∈ (0, 1/2] an arbitrary sequence s.t. s n → n→∞ 0.

• Let M be the compact subset of positive definite matrices in S K with all eigenvalues in the interval [1, 2]. In particular, for all M ∈ (s n M), we have (2s n I K -M ) ∈ S + K .

Interpolating estimation problem

Let = ( 1 , 2 ) ∈ (s n M) 2 . Let q : [0, 1] → S + K (ρ)
and r : [0, 1] → S + K be two "interpolation functions" (that will eventually depend on ), and

R 1 (t) ≡ 1 + t 0 r(v)dv , R 2 (t) ≡ 2 + t 0 q(v)dv . (4.10)
For t ∈ [0, 1], define the K-dimensional vector (using the matrix square root):

S t,µ ≡ 1 -t n n i=1 X µi W * i + R 2 (t) V µ + tρ -R 2 (t) + 2s n I K U * µ . (4.11)
We will interpolate between our original problem, and auxiliary problems related to the ones of eqs. (4.3),(4.4). More precisely, the interpolating estimation problem is given by the following observation model, with two types of t-dependent observations:

Y t,µ ∼ P out ( • | S t,µ ), 1 ≤ µ ≤ m, Y t,i = R 1 (t) W * i + Z i ∈ R K , 1 ≤ i ≤ n, (4.12)
where for each i, Z i ∼ N (0, I K ). Recall that in our notation the " * -variables" have to be retrieved, while other random variables are fixed (they are quenched variables in the statistical physics language). Define now s t,µ by a similar expression to eq. (4.11):

s t,µ ≡ 1 -t n n i=1 X µi w i + R 2 (t) V µ + tρ -R 2 (t) + 2s n I K u µ . (4.13)
The posterior of the interpolating problem is given by:

P t, (w, u|Y t , Y t , X, V) = 1 Z n, (t) n i=1 P 0 (w i )e - Y t,i - √ R 1 (t)w i 2 2 m µ=1 e -uµ 2 2 (2π) K/2 P out (Y t,µ |s t,µ ). (4.14)
The free entropy "at time t" is by definition (recall that D is the standard Gaussian measure)

f n, (t) ≡ 1 n E ln Du n i=1 dw i P 0 (w i )e -1 2 Y t,i - √ R 1 (t)w i 2 m µ=1 P out (Y t,µ |s t,µ ), (4.15)
At t = 0, the free entropy of eq. (4.15) is very close to the free entropy f n of the original problem (cf. the statement of Theorem 4.1), a difference arising from the presence of the small "perturbation" . We first show that this perturbation does not change the asymptotic free entropy:

Lemma 4.2 (Perturbation of the free entropy)

There exists a constant C > 0 such that for all ∈ (s n M) 2 we have

|f n, (0) -f n, =(0,0) (0)| ≤ Cs n .
Proof of Lemma 4.2 -One can compute easily4 :

∇ 1 f n, (0) = - 1 2 ρ -E Q n,0, ,
where the overlap matrix Q ∈ S K is defined below by eq. (4.18). Therefore, ∇ 1 f n, (0) F is bounded (using H.1). We define u y (x) ≡ ln P out (y|x). We can also compute (by calculations very similar to the ones used in the proof of the following Proposition 4.3, so that we leave them for the reader):

∇ 2 f n, (0) = 1 2n m µ=1 E ∇u Yt,µ (S t,µ ) ∇u Yt,µ (s t,µ ) n,0, .
Note that the r.h.s. of the above equation is symmetric by the Nishimori Proposition 1.1, and it is bounded by Hypothesis H.2. By the mean value theorem we obtain then directly that (for some constants

C 1 and C 2 ) |f n, (0)-f n, =(0,0) (0)| ≤ ∇ 1 f n, (0) F 1 F + ∇ 2 f n, (0) F 2 F ≤ C 1 max i i ≤ C 2 s n .
Let us now precisely relate the extremal values {f n, (0), f n, (1)} to our original problem. Using Lemma 4.2 , and continuity and boundedness properties of ψ P 0 and Ψ out stated in Lemmas C.2 and C.3, one can easily verify from the very definition of f n, (t):

       f n, (0) = f n - K 2 + O n (1) f n, (1) = ψ P 0 1 0 r(t)dt + αΨ out 1 0 q(t)dt; ρ - 1 2 1 0 Tr[ρ r(t)]dt - K 2 + O n (1) .
(4.16)

Here O n (1) is meant uniformly in t, q, r, .

Overlap concentration and fundamental sum rule

Notice from (4.16) that at t = 1 the interpolating estimation problem constructs part of the RS potential (4.6), while at t = 0 it is the free entropy of the original model (4.2) (up to a constant).

In order to compare these boundary values, we use the identity

f n = f n, (0) + K 2 + O n (1) = f n, (1) - 1 0 df n, (t) dt dt + K 2 + O n (1). (4.17)
The next obvious step is therefore to compute the free entropy variation along the interpolation path. This is summarized in the following Lemma, which is proven in Appendix C.4:

Proposition 4.3 (Free entropy variation)

Denote byn,t, the (Gibbs) expectation w.r.t. the posterior P t, given by (4.14). Set u y (x) ≡ ln P out (y|x). Then for all t ∈ [0, 1] we have

df n, (t) dt = - 1 2 E Tr 1 n m µ=1 ∇u Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) -r(t) Q -q(t) n,t, + 1 2 Tr[r(t)(q(t) -ρ)] + O n (1).
Again O n (1) → 0 is meant uniformly in t, q, r, . The K × K overlap matrix Q is defined as

Q ll ≡ 1 n n i=1 W * il w il . (4.18)
We will then plug the expression of Proposition 4.3 into eq. (4.17). In order to simplify it we need the following crucial proposition, which states that the overlap concentrates. As we have discussed in Chapter 1, this property is equivalent to what we know as replica symmetry.

Proposition 4.4 (Overlap concentration -Replica symmetry)

Assume that for any t ∈ (0, 1) the transformation

∈ (s n M) 2 → (R 1 (t, ), R 2 (t, )
) is a C 1 diffeomorphism with a Jacobian determinant greater or equal to 1. Then one can find a sequence s n going to 0 slowly enough such that there exists a constant C > 0 depending only on ϕ out , P 0 , K and α, and a constant γ > 0 such that:

1 Vol(s n M) 2 (snM) 2 d 1 0 dt E Q -E Q n,t, 2 F n,t, ≤ C n γ .
Proof of Proposition 4.4 -The proof of this concentration result can be directly taken from [START_REF] Barbier | Overlap matrix concentration in optimal Bayesian inference. Information and Inference[END_REF], which is entirely devoted to proving overlap matrix concentration. Using the results of [START_REF] Barbier | Overlap matrix concentration in optimal Bayesian inference. Information and Inference[END_REF] is straightforward assuming that → R(t, ) is a C 1 diffeomorphism with a Jacobian determinant greater or equal to 1. The reader can also refer to [AMB + 19] for more details.

Combining eq. (4.17) with Propositions 4.3 and 4.4, we can deduce the following fundamental sum rule which is at the core of the proof: Proposition 4.5 (Fundamental sum rule)

Assume that the interpolation functions r and q are such that the map = (

1 , 2 ) → R(t, ) = (R 1 (t, ), R 2 (t, ))
given by (4.10) is a C 1 diffeomorphism whose Jacobian determinant is greater or equal to 1. Assume that for all t ∈ [0, 1] and ∈ (s n M) 2 we have q(t) = q(t,

) = E Q n,t, ∈ S + K (ρ).
Then

f n = 1 Vol(s n M) 2 (snM) 2 d ψ P 0 1 0 r(t)dt + αΨ out 1 0 q(t, )dt; ρ - 1 2 1 0 Tr[q(t, )r(t)]dt + O n (1) .
In Proposition 4.5 we already see what we mean by adaptive interpolation: indeed, in order to apply this identity, we need q(t), which is a parameter of the model, to be equal to the (averaged) overlap of said model: this will create a self-consistent condition on q(t).

Proof of Proposition 4.5 -Let us denote V n ≡ Vol(s n M) 2 . In the following, the integral over is always over (s n M) 2 . Consider the first term, i.e. the Gibbs bracket, in the free entropy derivative given by Proposition 4.3. By the Cauchy-Schwarz inequality we have

E Tr 1 n m µ=1 ∇u Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) -r(t) Q -q(t) n,t, 2 ≤ 1 V n d 1 0 dt E 1 n m µ=1 ∇u Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) -r(t) 2 F n,t, × 1 V n d 1 0 dt E Q -q(t) 2 F n,t, .
The second term of this product is bounded by Cn -γ by Proposition 4.4, since we assumed that for all ∈ (s n M) 2 and all t ∈ [0, 1] we have q(t) = q(t, ) = E Q n,t, . The first term can itself be shown to be bounded by some constant C that only depend on ϕ out and α. This is shown in Lemma C.5 in Appendix C.5. In the end, applying Proposition 4.3 we obtain

1 V n d 1 0 df n, (t) dt dt = 1 2V n d 1 0 Tr q(t, )r(t) -r(t)ρ dt + O n (1). (4.19)
Again, the term O n (1) is uniform w.r.t. to the choice of q and r. When replacing eq. (4.19) in eq. (4.17) and combining it with eq. (4.16) we reach the claimed identity.

A technical lemma and an assumption

We give here a technical lemma used in the rest of the proof, and which allows us to detail the assumption on which we rely to prove Theorem 4.1.

Lemma 4.6 (Technical lemma)

Recall that E Q n,t, is a function of (n, t, R(t, )). Let F

(1)

n (t, R(t, )) ≡ 2α∇Ψ out (E Q n,t, ) and F (2) n (t, R(t, )) ≡ E Q n,t, . F n ≡ (F (1) n , F (2)
n ) is defined on the set:

D n = (t, r 1 , r 2 ) ∈ [0, 1] × S + K × S + K (ρt -r 2 + 2s n I K ) ∈ S + K .
F n is a continuous function from D n to S + K × S + K (ρ). Moreover, F n admits partial derivatives with respect to R 1 and R 2 on the interior of D n . For every (t, R(t, )) for which they are defined, they satisfy:

K l≤l ∂(F (1) n ) ll ∂(R 1 ) ll ≥ 0. (4.20)
Proof of Lemma 4.6 -The fact that the image domain of

F n is S + K × S + K (ρ) is a consequence of Lemma C.1.
The continuity and differentiability of F n follow from standard theorems of continuity and derivation under the integral sign (recall that we are working at finite n). Indeed, the domination hypotheses are easily satisfied since we work under H.1 and H.2. Let us now prove (4.20). We write the formal differential of F

(1)

n with respect to R 1 as D R 1 F (1)
n , which is a 4-tensor, and our goal is to prove that Tr

[D R 1 F (1) n ] ≥ 0, the trace of a 4-tensor A (ij)(kl) over S K being Tr[A] ≡ i≤j A (ij)(ij) . Then one can write Tr[D R 1 F (1) n ] = 2αTr[∇∇ Ψ out (E Q n,t, ) × ∇ R 1 E Q n,t, ]. We know from Lemmas C.1 and C.3 that ∇∇ Ψ out (E Q n,t,
) is a positive linear operator over S K . Moreover, it is a known result that the derivative ∇ R 1 E Q n,t, is also positive, since R 1 is the "matrix snr" of a linear channel (see e.g. [START_REF] Reeves | Mutual information as a function of matrix SNR for linear gaussian channels[END_REF]). Since the product of two symmetric positive matrices (here linear operators on S K ) has always positive trace, this shows that Tr

[D R 1 F (1) n ] ≥ 0.
We can now state a technical assumption on which we rely, and which essentially allows us to derive that the map → R(t, ) is a C 1 diffeomorphism with a Jacobian determinant greater or equal to 1 as will become clear in the next section: Hypothesis 4.1 (Technical assumption) With the notations of Lemma 4.6,

K l≤l ∂(F (2) n ) ll ∂(R 2 ) ll ≥ 0.
An unnecessary hypothesis -Note that when [AMB + 19] first came out, the proof was relying on the unproven Hypothesis 4.1. Since then, the work of Barbier & Reeves [START_REF] Barbier | Information-theoretic limits of a multiview lowrank symmetric spiked matrix model[END_REF] has shown that this hypothesis is not necessary if one uses a slightly more involved interpolation path than the one of eq. (4.12), so that Theorem 4.1 stands without the need for Hypothesis 4.1. However, since this technicality significantly lengthens our proof without adding any new physical insight, in this thesis we chose to detail our original approach, assuming Hypothesis 4.1.

Matching bounds: adapting the interpolation path

We can now turn to proving the final result. We will use two different adaptive interpolation paths, to obtain respectively lower and upper bounds on the free entropy.

Proposition 4.7 (Lower bound)

Under Assumption 4.1, the free entropy associated to the posterior of eq. (4.5) verifies

lim inf n→∞ f n ≥ sup r∈S + K inf q∈S + K (ρ)
f RS (q, r).

Proof of Proposition 4.7 -Choose first r(t) = r ∈ S + K a fixed matrix. Then R(t) = (R 1 (t), R 2 (t)) can be fixed as the solution to the first order Cauchy problem:

d dt R 1 (t) = r , d dt R 2 (t) = E Q n,t, , and
R(0) = . (4.21)
We denote this (unique) solution R(t, ) = (rt + 1 , t 0 q(v, ; r)dv + 2 ). It is possible to check that this ODE satisfies the hypotheses of the parametric Cauchy-Lipschitz theorem, and that by the Liouville formula the determinant J n, (t) of the Jacobian of → R(t, ) satisfies (see Lemma C.4 for details on the Liouville formula)

J n, (t) = exp t 0 K l≥l ∂E Q ll n,s, ∂(R 2 ) ll (s, R(s, )) ds (a) ≥ 1, (4.22)
in which (a) is a consequence of Assumption 4.1. Moreover by eq. (4.21), q(t, ; r) = E Q n,t, , which is in S + K by Lemma C.1. The fact that the map → R(t, ) is a C 1 diffeomorphism is easily verified by its bijectivity (from the positivity of J n, (t)) combined with the local inversion theorem. All the assumptions of Proposition 4.5 are verified which then implies (recall the RS potential of eq. (4.6)):

f n = 1 Vol(s n M) 2 (snM) 2 d f RS 1 0 q(v, ; r)dv, r + O n (1).
Since this is true for any r ∈ S + K , this easily implies the lower bound.

Proposition 4.8 (Upper bound)

Under Assumption 4.1, the free entropy associated to the posterior of eq. (4.5) verifies lim sup

n→∞ f n ≤ sup r∈S + K inf q∈S + K (ρ)
f RS (q, r).

Proof of Proposition 4.8 -We now fix R(t) = (R 1 (t), R 2 (t)) as the solution R(t, ) = ( t 0 r(v, )dv + 1 , t 0 q(v, )dv + 2 ) to the following Cauchy problem:

d dt R 1 (t) = 2α∇Ψ Pout (E Q n,t, ) , d dt R 2 (t) = E Q n,t, , and
R(0) = .
We denote this equation as

∂ t R(t) = F n (t, R(t)), R(0) = . It is then possible to verify that F n (R(t), t) is a bounded C 1 function of R(t)
, and thus a direct application of the Cauchy-Lipschitz theorem implies that R(t, ) is a C 1 function of t and . The Liouville formula (Lemma C.4) for the Jacobian determinant of the map ∈ (s n M) 2 → R(t, ) gives this time

J n, (t) = exp t 0 K l≥l ∂(F n,1 ) ll ∂(R 1 ) ll (s, R(s, )) + ∂(F n,2 ) ll ∂(R 2 ) ll (s, R(s, )) ds (a) ≥ 1 . (4.23)
Equality (a) follows again from the positivity of this sum of partials, see Lemma 4.6 and Assumption 4.1. Eq. (4.23) implies the bijectivity of → R(t, ) which, combined with the local inversion theorem, makes it a diffeomorphism. Since E Q n,t, and ρ -E Q n,t, are positive matrices (see Lemma C.1) we also have that q(t, ) ∈ S + K (ρ) and since by the differential equation we have r(t, ) = 2α∇Ψ out (q(t, )), we also have r(t, ) ∈ S + K by Lemma C.3. We have everything needed to apply Proposition 4.5 again which gives in this case

f n = 1 Vol(s n M) 2 d ψ P 0 1 0 r(v, )dv + αΨ out 1 0 q(v, )dv; ρ - 1 2 Tr 1 0 q(v, )r(v, )dv + O n (1).
Then by convexity of ψ P 0 and Ψ out (see Lemmas C.2 and C.3) and Jensen's inequality:

f n ≤ 1 Vol(s n M) 2 d 1 0 dv ψ P 0 (r(v, )dv) + αΨ out (q(v, ); ρ) - 1 2 Tr[q(v, )r(v, )] + O n (1), = 1 Vol(s n M) 2 d 1 0 dv f RS (q(v, ), r(v, )) + O n (1) .
We now remark that

f RS [q(v, ), r(v, )] = inf q∈S + K (ρ) f RS [q, r(v, )].
Indeed, for every r ∈ S + K , the function g r : q ∈ S + K (ρ) → f RS (q, r) is convex (by Lemma C.3), and moreover ∇g r (q) = α∇Ψ out (q) -r/2. Thus ∇g r(v, ) (q(v, )) = 0 by definition of r(v, ). Since S + K (ρ) is convex, the minimum of g r(v, ) (q) is necessarily achieved at q = q(v, ). Therefore:

f n ≤ 1 Vol(s n M) 2 (snM) 2 d 1 0 dv inf q∈S + K (ρ) f RS (q, r(v, )) + O n (1) ≤ sup r∈S + K inf q∈S + K (ρ) f RS (q, r) + O n (1),
which concludes the proof of Proposition 4.8.

Combining the two matching bounds of Propositions 4.7 and 4.8 ends the proof of Theorem 4.1.

Conclusion of Chapter 4

This chapter aimed at presenting how the different elements of the toolbox introduced in Chapter 1 can be combined to study optimal learning in a large class of two-layers neural networks.

One of its main contributions is the design of an AMP-type algorithm that is able to achieve Bayes-optimal learning in the high-dimensional limit. For the committee machine with a large number of hidden neurons we uncover regimes in which a low generalization error is informationtheoretically achievable while the AMP algorithm fails to deliver it; strongly suggesting that no efficient algorithm exists for those cases, and unveiling a large computational gap. These hard phases are associated to first-order phase transitions in the solution of the model.

Importantly, we studied the Bayes-optimal setting in which the student network has the same architecture as the teacher. However, the replica method can be generalized to a case in which the teacher and students have different architectures, as was done for compressive sensing, see e.g. [KMS + 12]. It is an interesting subject for future work to leverage the replica method to study how the hard phase evolves under e.g. over-parametrization. Note that since the writing of [AMB + 19], works have studied the influence of under and over-parametrization in the committee machine on the performance of stochastic gradient descent, see e.g. [GAS + 20].

Even though we focused in this chapter on a two-layers neural network, the analysis and algorithm can be extended to a multi-layer setting, following e.g. the lines of [START_REF] Mato | Generalization properties of multilayered neural networks[END_REF], as long as the number of layers as well as the number of hidden neurons in each layer is Θ n (1), and as long as one only learns the first-layer weights. Indeed, such models still belong to the class of Model 4.1, so that Theorem 4.1 applies. The numerical evaluation of the phase diagram would be more challenging than the cases presented in this chapter, as multiple integrals would appear in the corresponding formulas. Going even further, one could study the case in which the weights of all layers are learned. A possible strategy to tackle this setting (still assuming a number Θ n (1) of hidden neurons) would be to combine our AMP algorithm for the first layer with Expectation-Maximization procedures to learn the remaining layers. Similar ideas were already implemented in simpler settings than neural networks, see e.g. [KMS + 12, KRFU14].

Let us make a final comment that will shed light on some of our motivation for the upcoming Part III. There exists a complex line of work that studies the loss-function landscape of neural networks. While a range of works show under various assumptions that spurious local minima are absent in neural networks, others show under different conditions that they do exist, see e.g. [START_REF] Safran | Spurious local minima are common in two-layer relu neural networks[END_REF]. In the models we studied here, the regime of parameters that is hard for AMP must have spurious local minima, but the converse is not true in general5 : this motivates a precise study of the loss landscape of neural networks, that will be the subject of Part III.

Chapter 5

Generative models, or how to exploit the structure in the data

"Data! Data! Data! . .

. I can't make bricks without clay!"

Sir Arthur Conan Doyle, Adventures of Sherlock Holmes (1892).

Disclaimer -While Chapter 4 illustrated well the power of our statistical physics toolbox to study optimal learning, it was restricted to i.i.d. data. In this chapter, we go further in this regard by investigating a crucial element to understand learning in practice: the structure in the data. In order to study theoretically such structure, we leverage generative models: more precisely, the data will be taken as the output of a (random) neural network. This will allow to model synthetic data sets that possess very non-trivial correlations arising from the structure of the generative model1 , and to understand how such structure can affect learning. We will focus on a rather simple inference problem to recover said data, namely spiked matrix estimation. This chapter is mainly based on the published work [ALM + 20], and will illustrate the flexibility of the toolbox of Chapter 1, as we will adapt it to study the influence of data structure on the feasibility of learning. It also contains a detailed random matrix analysis in Section 5.4, which is inspired by the famous "BBP" transition [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF].

Generative models for spiked matrix estimation

Introduction: exploiting data structure

Taking advantage of specific structures to enhance signal reconstruction is a central endeavour in modern signal processing. Notable technological advances -such as e.g. JPEG and MP3 compression -stem from the fact that images and sound admit a sparse representation in wavelet and Fourier bases. In a series of seminal works, see e.g. [START_REF] Emmanuel | Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information[END_REF], Candès, Romberg and Tao have shown that underparametrized linear systems can be inverted if the signal is assumed to be sparse. This opened the door for novel sub-Nyquist sampling strategies leveraged by sparsity which are at the heart of compressed sensing [START_REF] David | Compressed sensing[END_REF]. But interest in sparse representations reaches far beyond compressed sensing, and similar results have been derived for other signal processing tasks, such as sparse coding and sparse principal component analysis (PCA). Despite the remarkable success of these results, they broadly assume the latent sparse representation is given, thus relying on expert knowledge for signal pre-processing.

On the other hand, recent progress in deep learning has witnessed a surge of interest in neural network-based generative models. Opposed to sparsity, generative networks are trained to learn a latent representation of the structured signal. The expressiveness of neural networks allied with the capacity to capture hierarchical representations led to impressive results in signal modelling, the most notable perhaps being Generative Adversarial Networks (GANs), which can be trained to generate realistic images of human faces [GPAM + 14]. An important and natural question to ask is whether signals from generative models enjoy the same aforementioned interesting properties as sparse signals in reconstruction tasks. A series of recent results indeed suggest that the latent structure in generative models can be leveraged to improve signal reconstruction, see e.g. [TMC + 16, BJPD17, HV18, HLV18, GMKZ20] among many other works. These suggest indeed that, in the words of [START_REF] Villar | Generative models are the new sparsity[END_REF], "Generative models are the new sparsity".

This chapter is a further step in this direction: we analyze a class of random-neural generative priors in an unsupervised task: rank-one matrix factorization. Given a "data" matrix Y ∈ R n×p , the problem consists in finding two vectors (known as spikes) u ∈ R n , v ∈ R p such that Y can be factorized as Y = uv + √ ∆ξ, where ξ is an i.i.d. noise matrix of unit variance. This model is widely studied as a prototype for principal component analysis (PCA), since for small noise (∆ < 1) and Gaussian spikes u, v, the optimal estimator is given by the leading principal component of Y [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF]. Optimality relies on the assumption of unstructured spikes, and no longer holds if one of the spikes is sparse. In a similar spirit to compressed sensing, the investigation of sparse spikes in this model resulted into bespoke algorithms widely studied under the umbrella of sparse-PCA, see e.g. [START_REF] Jenatton | Structured sparse principal component analysis[END_REF].

An important conclusion of the aforementioned works is the existence of an algorithmic gap for sparse signal reconstruction: even if signal reconstruction is information-theoretically possible, no polynomial-time algorithm is known. For spiked-matrix factorization, this means that even though the best known sparse-PCA algorithm performs better than "vanilla" PCA, it doesn't reach the optimal threshold set by the theoretical Bayesian estimator. As we will show, this is in sharp contract to the class of neural generative models we study, for which we provide a polynomial time algorithm reaching the optimal theoretical performance. Our line of work has been continued by collaborators to study generative models in the phase retrieval problem, see [ALB + 20].

Inference model: spiked matrix estimation

We will focus on the following two models, which are widely studied in the sparse-PCA literature [RF12, DM14a, LKZ15, PWBM16, BDM + 16, Mio17, LM19] (note that they are as well very natural random matrix theory problems):

Model 5.1 (Spiked Wigner model vv )

Consider an unknown vector (the spike) v ∈ R p drawn from a distribution P v on R p ; we observe a matrix Y ∈ R p×p with a symmetric noise term ξ ∈ S p and ∆ > 0:

Y = 1 √ p v v + √ ∆ξ, (5.1)
where ξ/ √ p ∼ GOE(p), i.e. ξ ij i.i.d.

∼ N (0, 1 + δ ij ) for i ≤ j. The aim is to find back the spike v from Y (up to a global sign).

Model 5.2 (Spiked Wishart/covariance model uv )

Consider two unknown vectors u ∈ R n and v ∈ R p drawn from distributions P u and P v and let ξ ∈ R n×p with ξ µi i.i.d.

∼ N (0, 1), and ∆ > 0. We observe

Y = 1 √ p u v + √ ∆ξ, (5.2)
and the goal is to find back the spikes u and v from Y ∈ R n×p .

The high-dimensional limit that we consider (the thermodynamic limit) is p, n → ∞ while τ ≡ n/p = Θ(1), and the noise variance is ∆ = Θ(1). The prior P v is representing the spike v ∈ R p via a k-dimensional parametrization with α ≡ p/k = Θ(1). Note that in sparse estimation, k is the number of non-zeros components of v , while in generative models k is the number of latent variables. As in Chapter 4, we assume a Bayes-optimal setting (introduced in Section 1.1): the priors and the noise ∆ > 0 are known to the observer.

Generative models for the data

Multivariate Gaussian prior

The simplest non-separable prior P v that one can consider is the Gaussian model with a covariance matrix Σ, that is P v = N (0, Σ). While this prior is not compressive, it captures some structure and can be simply estimated from data via the empirical covariance. This prior is very elementary, and we introduce it here as we will use it later to produce Fig. 5.5.

Multi-layer generative prior

To exploit the practically observed power of generative models, one would ideally consider models trained on datasets of examples of possible spikes, e.g. GANs, variational auto-encoders, or restricted Boltzmann machines. Such training, however, leads to strong correlations between the weights of the underlying neural networks, for which the theoretical part of the present chapter can not be applied readily. To allow for tractable theoretical analysis we focus on multilayer generative models where all the weight matrices W (l) (1 ≤ l ≤ L) are fixed, independent, and generated i.i.d. from N (0, 1). Let v ∈ R p be the output of such a generative model:

v = ϕ (L) 1 √ k L W (L) . . . ϕ (1) 1 √ k W (1) z , ( 5.3)
with z ∈ R k a latent variable drawn from separable distribution P z , with zero mean and variance ρ z . The ϕ (l) are element-wise activation functions that can be either deterministic or stochastic.

It will be useful to define the hidden variables h (l) ∈ R k l obtained from the output of layer l -1:

h (l+1) = ϕ (l) 1 √ k l W (l) h (l) ⇔ h (l+1) ∼ P (l) out • 1 √ k l W (l) h (l) .
We naturally let h (1) ≡ z and h (L+1) ≡ v. We wrote in the last equation an equivalent parametrization in terms of channel densities P (l) out over R k l+1 , that are also applied element-wise, and that parametrize the input/output relationship at each layer of the generative network. For instance, a deterministic layer l with non-linearity ϕ (l) is fully characterized by the scalar density

P (l) out (x|z) = δ(x -ϕ (l) (z)).
Recall that the observer has access to the generative prior P v , i.e. she/he knows the matrices W (l) ∈ R k l+1 ×k l and non-linearities ϕ (l) . In the case of Model 5.2, she/he has similarly access to the details of P u . The spike v * is generated using a ground-truth value of the latent variable z * .

The spike is then estimated from the knowledge of the data matrix Y, and the known form of the spiked-matrix model and of the generative prior. Only the spike v * (and u * for Model 5.2) and the latent vector z * are unknown, and are to be inferred.

For concreteness and simplicity, the generative model that will be analyzed in the majority of examples given in this chapter is the single-layer case, i.e. eq. (5.3) with L = 1. In this case we define the total compression ratio of the generative prior as α ≡ p/k. In what follows we will illustrate our results for ϕ being linear, sign and ReLU functions.

Summary of main results

Before diving into the details let us briefly summarize the main results of this chapter.

• We first conduct an asymptotic analysis for the performance of the optimal estimators (both information-theoretic and algorithmic) for the spiked-matrix Models 5.1 and 5.2. This analysis is based on a rigorous expression for the mutual information between the matrix Y and a general spike v drawn from an arbitrary distribution P v in R p . As in Chapter 4, this expression can be obtained by use of the replica method, but we will focus in this chapter on its rigorous derivation. Evaluating this expression on the generative priors discussed in Section 5.1.3, we obtain the threshold ∆ c ≡ ∆ Algo below which the spike v can be partially (or weakly) reconstructed in polynomial time.

• In order to understand the algorithmic limits of this problem, we derive an approximate message-passing (AMP) algorithm for Model 5.1 and 5.2. For all the generative architectures we analyze, we show that is attains the information-theoretic optimal performance.

• Finally, we propose a simple spectral method derived from our AMP algorithm, and we argue that it reaches the same weak-recovery threshold ∆ c . Remarkably, in certain cases we are able to rigorously analyze this spectral method independently of AMP using only random matrix theory tools.

Our main findings are in stark contrast to the known results for sparse PCA, and we therefore emphasize two important conclusions of our analysis:

(i) No algorithmic gap with generative priors -Sharp and detailed results are known in the thermodynamic limit (as defined above) when the spike v ∈ R p is sampled from a separable distribution P v . A detailed account of several examples can be found in [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF]. The main finding for sparse priors P v with k non-zero components is that when the sparsity ρ = k/p = 1/α is large enough then there exist optimal algorithms [DM14a], while for small enough ρ there is a striking gap between statistically optimal performance and the one of best known algorithms [START_REF] Thibault Lesieur | Phase transitions in sparse PCA[END_REF]. These conclusions are consistent with the well-known results for exact recovery of the support of v [AW09a, BR13], which is one of the best-known cases in which gaps between statistical and best-known algorithmic performance were described.

Our analysis of the spiked-matrix models with generative priors reveals that in this setting, iterative algorithms are able to obtain asymptotically optimal performance even when the compression factor is important, i.e. α 1. It therefore suggests that generative priors are "better than sparsity" in the sense that they lead to algorithmically easier problems2 .

(ii) Spectral algorithms reaching statistical threshold -Arguably the most basic algorithm used to solve a spiked-matrix model is based on the leading singular vectors of the matrix Y. We will generically refer to this strategy as PCA. Previous works on spikedmatrix models [START_REF] Perry | Optimality and sub-optimality of PCA for spiked random matrices and synchronization[END_REF][START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF] established that for separable priors, PCA reaches the optimal (among polynomial-time algorithms) weak-recovery threshold ∆ c , below which it is able to provide positive correlation between its estimator and the ground-truth spike. While for sparse priors positive correlation is statistically reachable for values ∆ > ∆ c , no efficient algorithm beating the PCA threshold is known3 .

In the case of generative priors we find that well-chosen spectral methods can improve on canonical PCA. We design a spectral method, called LAMP, that reaches the statistically optimal threshold, meaning that for larger values of noise variance no other (even exponential) algorithm is able to reach positive correlation with the spike. This is another striking difference with sparse separable priors, making the generative priors algorithmically more attractive. We moreover demonstrate the performance of LAMP on real data, and show considerable improvement over canonical PCA, even though real data does not arise from a random generative prior.

Section 5.2 is dedicated to the analysis of information-theoretic and algorithmic optimal estimation, while Section 5.3 focuses on the spectral methods and our random matrix theory analysis. Section 5.4 is devoted to the proofs of the random matrix analysis.

Analysis of optimal estimation

Mutual information: the replica method rigorous once again

For conciseness, the following information-theoretic results are given for the Wigner model 5.1. They can be fully generalized to the Wishart case, and we refer the reader to [ALM + 20] for more details on this regard.

From an optimization perspective, the problem we want to solve is to find the estimator v that minimizes the mean squared error (MSE)

mse(∆) ≡ E||v -v || 2 2 .
(5.4)

Since the information about the generative model P v of the spike is given, we know from Section 1.1 that the estimator minimizing eq. (5.4) is given by the mean of the posterior distribution of the spike P(v|Y), which we can write from Bayes' rule as

P(v|Y) = 1 P(Y ) P v (v) 1≤i<j≤p 1 √ 2π∆ e -1 2∆ Y ij - v i v j √ p 2 p i=1 1 √ 4π∆ e -1 4∆ Y ii - v 2 i √ p 2 .
(5.5)

The expression above is written in full generality, and for the time being we have not assumed anything about P v . The naive approach of estimating vopt from exact sampling of the posterior is intractable numerically, especially in the large-dimensional limit p → ∞ of interest. However, it is still possible to track the performance of the optimal estimator without direct sampling, through the I-MMSE theorem connecting the minimal mean squared error (MMSE) to a derivative of the mutual information between the signal and the data [START_REF] Guo | Mutual information and minimum mean-square error in Gaussian channels[END_REF]. Note that this is completely equivalent to studying the free entropy in the statistical physics language.

Following this rationale, our first main result is a rigorous expression for the mutual information between the ground-truth spike v and the observation Y, valid in the thermodynamic limit p → ∞. We state it informally, while a full technical statement can be found in [ALM + 20]:

Theorem 5.1 (Mutual information for Model 5.1 with structured spike, informal)

Assume that the spike v comes from a sequence (of growing dimension p) of structured priors P v on R p . We define the mutual information as

I(Y; v ) ≡ D KL (P (v ,Y) |P v P Y ). Then lim p→∞ i p ≡ lim p→∞ I(Y ; v ) p = inf qv∈(0,ρv) i RS (∆, q v ), with i RS (∆, q v ) ≡ (ρ v -q v ) 2 4∆ + lim p→∞ I(v; v + ∆ qv w) p .
(5.6)

Here, w ∼ N (0, I p ), and

ρ v ≡ lim p→∞ E Pv [v v]/p.
The proof for this theorem can be found in [ALM + 20]. It uses interpolation techniques, but as opposed to the proof of the replica prediction in Chapter 4, interpolation is here not adaptive, and the arguments are therefore significantly simpler. The subscript of i RS denotes replica symmetry, acknowledging that this prediction is first obtained by the replica method before being put on rigorous ground.

Let us first draw the consequences of Theorem 5.1. It connects the asymptotic mutual information of the spiked model with generative prior P v to the mutual information between v taken from P v and its noisy version, I(v; v + ∆/q v w). As mentioned before, the mutual information is connected to the performance of the optimal estimator, and one can prove that for the spiked-matrix model (cf. [START_REF] Krzakala | Estimation in the spiked Wigner model: A short proof of the replica formula[END_REF]) the MMSE on the spike v is asymptotically given by:

MMSE v = ρ v -q v , (5.7)
where q v is the minimizer of i RS (∆, q v ) that appears in Theorem 5.1. Computing i RS is itself a high-dimensional task, hard in full generality, but it can be done for a range of non-trivial P v :

• The simplest tractable case is when the prior P v is separable, then it yields back exactly the previously known formula from [KXZ16, BDM + 16, LM19].

• For the correlated Gaussian generative model P v = N (0, Σ), one can easily compute I(v; v + ∆/q v w) = Tr[ln(I p + q v Σ/∆)]/2.

• More interestingly, let us consider the multi-layer generative prior with random weights from eq. (5.3). The single-layer formula (i.e. when L = 1) for i RS has been derived and proven in [BKM + 19]. For the multi-layer L ≥ 1 case the mutual information formula has been derived in [START_REF] Manoel | Multilayer generalized linear estimation[END_REF][START_REF] Reeves | Additivity of information in multilayer networks via additive gaussian noise transforms[END_REF] and proven for the case of two layers in [GML + 19]. In [ALM + 20], we showed that these previous works yield the following formula for the spiked Wigner Model 5.1 with multi-layer generative prior given by eq. (5.3):

i RS (∆, q v ) = ρ 2 v + q 2 v 4∆ + 1 α extr {q l ,q l } 1 2 L l=1 α l ql q l - L l=1 α l+1 Ψ (l)
out (q l+1 , q l ) -Ψ z (q z ) .

(5.8)

where α l = k l /k (in particular α 1 = 1 and α L+1 = α). We also defined qL+1 ≡ q v /∆, q z = q 1 , qz = q1 , and the functions Ψ z , Ψ out are defined by

   Ψ z (x) ≡ E ξ [Z z (x 1/2 ξ, x) ln(Z z (x 1/2 ξ, x))], Ψ (l) out (x, y) ≡ E ξ,η [Z (l) out (x 1/2 ξ, x, y 1/2 η, ρ l -y) × ln(Z (l) out (x 1/2 ξ, x, y 1/2 η, ρ l -y))],
with ξ, η ∼ N (0, 1), ρ l is the second moment of the hidden variable h l and Z z , Z

out are the normalizations of the following two denoising scalar distributions:

             Q z (z; γ, Λ) ≡ P z (z)e -Λ 2 z 2 +γz Z z (γ, Λ) , Q (l) out (v, x; B, A, ω, V ) ≡ P (l) out (v|x)e -A 2 v 2 +Bv Z (l) out (B, A, ω, V ) e -(x-ω) 2 2V √ 2πV .
(5.9)

Optimal performance and statistical thresholds: phase diagrams

As many results of the replica method, Theorem 5.1 combined with eq. (5.8) is remarkable in that it connects the asymptotic mutual information of a high-dimensional model with a simple scalar formula that can be easily evaluated. Moreover, it fully characterizes the statistical and algorithmic performance of the optimal estimators, allowing us to readily identify the thresholds separating the region between possible and impossible inference of the spike. Let us now draw some of its consequences for common activations. Taking the extremization over q v and (q l , q l ) 1≤l≤L in eq. (5.8), we obtain the following system of coupled equations:

                           q v = 2∂ x Ψ (L) out (q v /∆, q L ) q L = 2∂ x Ψ (L-1) out (q L , q L-1 ) . . . q l = ∂ x Ψ (l-1)
out (q l , q l-1 ) . . .

q z = 2Ψ z (q z )                            qL = 2 α α L ∂ y Ψ (L) out (q v /∆, q L ) qL-1 = 2 α L α L-1 αL-1 ∂ y Ψ (L-1) out (q L , q L-1 ) . . . ql = 2 α l+1 α l ∂ y Ψ (l) out (q l+1 , q l ) . . . qz = 2 α 2 α 1 ∂ y Ψ (1)
out (q 2 , q z ) , (5.10)

As previously discussed, the set of solutions of these equations provides all the information about the performance of the Bayes-optimal estimator through eq. (5.7), and of the optimal algorithmic estimator as we will see. We shall refer to them (with a bit of anticipation) as the state evolution (SE) equations.

Weak-recovery threshold

An important first question that can be answered from eq. (5.10) is weak recovery, i.e. when is it possible to perform better than a random guess from the prior distribution P v . For instance, we intuitively expect that when the prior is not biased towards a particular direction in R p and for very high noise ∆ 1 weak recovery is impossible. In terms of fixed points of eq. (5.10), this situation corresponds to the existence of the non-informative fixed point q v = 0 (i.e. maximum MSE v = ρ v , or zero overlap with the spike). Evaluating the right-hand side of eqs. (5.10) at q v = 0, we can see that q v = 0 is a fixed point iff

E Pz [z] = 0 and E Q (l),0 out [v] = 0, (5.11)
where

Q (l),0 out (v, x) ≡ Q (l)
out (v, x; 0, 0, 0, ρ l ) from eq. (5.9). Note that for multi-layer networks with deterministic channels and ϕ (l) ≡ ϕ for all l, the second condition is equivalent to ϕ being an odd function.

When the condition of eq. (5.11) holds, (q v , q L , qL , . . . , qz , q z ) = (0, 0, 0, . . . , 0, 0) is a fixed point of eq. (5.10). In [ALM + 20] we perform a detailed linear stability analysis of this fixed point. This gives a generic way to compute the weak-recovery threshold ∆ c . Note that as we use a local analysis of the uninformative fixed point, ∆ c corresponds to the algorithmic weak-recovery threshold: indeed, the performance of message-passing algorithms will be given by a local descent on i RS starting from q = 0, as we will see in Section 5.2.34 , while there might exists another global minimum of i RS which does not arise from local instability of the q = 0 point. However, in all cases we investigated this situation did not occur, so that ∆ c corresponded both to the information-theoretic and the algorithmic weak-recovery threshold.

Single-layer generative prior

We first consider a single-layer generative prior L = 1. Fix P z = N (0, 1) and P

(1) out (v|x) = δ(vϕ(x)), for ϕ ∈ {linear, sign, ReLU}. The first two choices of nonlinearity are odd, and therefore in these cases we expect a sharp weak-recovery transition as discussed above. This transition can be computed from the Jacobian (details of this computation are given in [ALM + 20]) as we mentioned. We obtain ∆ c = 1 + α for linear activation and ∆ c = 1 + 4α/π 2 for sign activation. In both cases, since α > 0, it is clear that knowledge of the generative prior improves reconstruction, as the weak-recovery threshold of canonical PCA is ∆ PCA = 1. Moreover, the larger α (i.e. the smaller the latent dimension with respect to the signal dimension), the better the reconstruction. Fig. 5.1 summarizes this discussion. We numerically solve eq. (5.10), and plot the MMSE obtained from the solution in a heat map, for the linear, sign and ReLU activations. The white dashed line marks the threshold ∆ c obtained analytically as mentioned above. The property that we find the most striking is that in these three evaluated cases, for all values of ∆ and α that we analyzed, we always found that eq. (5.10) has a unique stable solution. Thus we have not identified, in the physics terminology, any first order phase transition. Figure 5.2 shows examples of numerical MMSE curves for three activations discussed, and different values of α. The fixed point equations were solved iteratively from uncorrelated initial condition, and from initial condition corresponding to the ground truth signal, and we found that both lead to the same solution, illustrating the absence of computational gap.

Deeper generative priors

This observation generalizes to deeper L > 1 generative priors. Consider P z = N (0, 1) and layer-wise constant activation P (l) out (v|x) = δ(v -ϕ(x)). For the previous odd activation functions (5.12) Note in particular that for L = 1 and in the limit α = 0 we recover the transition of PCA ∆ c = 1 known from the case with separable prior [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF]. For α > 0, we have ∆ c > 1 meaning the spike can be estimated more efficiently when its structure is accounted for. In particular, the deeper the generative network for the spike, the easier estimation becomes.

• Sign activation -For ϕ(x) = sign(x) the leading eigenvalue of the Jacobian becomes one at

∆ c = 1 + L l=1 4 π 2 l α α l .
(5.13)

As in the linear case, we recover the threshold of canonical PCA for α = 0, while for α > 0 we can estimate the spike for larger noise values than in the separable case, and depth also improves estimation.

Fig. 5.3 illustrates our results for the multi-layer case. Note that we also didn't observe first order transitions for deeper networks, at least in the first-to-come-in-mind cases that we have investigated, i.e. deterministic deep networks with ϕ (l) ≡ ϕ ∈ {linear, sign, ReLU}. However, we do not expect this behavior to be completely general neither. One can engineer a situation, for Algorithm 4: AMP for the spiked Wishart model 5.2 with single-layer generative prior.

Result:

The estimators û, v, ẑ Input: Observation Y ∈ R n×p and weight matrix W ∈ R p×k ; Initialize to zero:

(g, û, v, B v , A v , B u , A u ) t=0 ; Initialize: ût=1 , vt=1 , ẑt=1 ∼ N (0, σ 2 ), ĉt=1 u = 1 n , ĉt=1 v = 1 p , ĉt=1 z = 1 k ; while not converging do
Spiked layer;

B t u = 1 ∆ Y √ p vt -1 ∆ 1 p ĉt v p I n ût-1 and A t u = 1 ∆ vt 2 2 p ; B t v = 1 ∆ Y √ p ût -1 ∆ 1 n ĉt u p I p vt-1 and A t v = 1 ∆ ût 2 2 p ; Generative layer; V t = 1 k (1 k ĉt z ) and ω t = 1 √ k Wẑ t -V t g t-1 and g t = f out (B t v , A t v , ω t , V t ) ; Λ t = 1 k g t 2 2 and γ t = 1 √ k W g t + Λ t ẑt ; Update of the estimated marginals; ût+1 = f u (B t u , A t u ) and ĉt+1 u = ∂ B f u (B t u , A t u ); vt+1 = f v (B t v , A t v , ω t , V t ) and ĉt+1 v = ∂ B f v (B t v , A t v , ω t , V t ); ẑt+1 = f z (γ t , Λ t ) and ĉt+1 z = ∂ γ f z (γ t , Λ t ) ; t = t + 1; end
instance with a very shifted ReLU on the last layer, and a very large intermediate layer, so that the spike v becomes effectively sparse with weakly correlated, almost independent, components, thus recovering the classical algorithmic gap [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF].

So far we have only discussed the performance of the information theoretic optimal estimator -averting the question of estimating the signal itself. In the next section we close this gap by mean of an approximate message-passing (AMP) algorithm, again originating in our statistical physics toolbox.

Algorithmic optimal estimation

In this section we state and analyze an AMP algorithm tailored for spiked matrix estimation with generative priors. Its derivation from the belief propagation equations is fairly technical, and detailed in [ALM + 20]. As we have discussed already in this thesis, AMP has two great virtues in this type of Bayes-optimal models: firstly, it achieves the best MSE among a wide class of general first order methods [START_REF] Celentano | The estimation error of general first order methods[END_REF], and secondly we can derive state evolution (SE) equations to track the MSE of AMP in the thermodynamic limit. As we will see, this MSE coincides with the optimal performance discussed in Section 5.2, even for large α. This result is particularly interesting when compared to the known performance of message-passing algorithms for sparse-PCA, for which AMP is not able to reach optimal statistical performance in the small sparsity regime [LML + 17]. + 20], one can combine two AMPs for independent inference problems into a single one for a structured problem mixing the two. In particular, this applies to spiked-matrix estimation with single-layer generative prior, which can be seen as the combination of a rank-one matrix factorization problem [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF] with a generalized linear model (GLM, cf. Section 1.1). Note that the multi-layer case can be derived by iterating this procedure, and the interested reader can refer to [ALM + 20] for more details on this regard. We give the AMP algorithm for the spiked Wishart model 5.2 in Algorithm 4, with a single-layer generative prior.

As underlined in [ALM

We defined some auxiliary functions:

• f u and f z are the means of the distributions Q u and Q z , defined as

Q u (u; B, A) ≡ P u (u)e -1 2 Au 2 +Bu Z u (B, A) and Q z (z; γ, Λ) ≡ P z (z)e -Λ 2 z 2 +γz Z z (γ, Λ) .
(5.14)

• f v is the mean of v, and f out is the mean of V -1 (x -ω), both with respect to

Q out (v, x; B, A, ω, V ) ≡ e -1 2 Av 2 +Bv P out (v|x) Z out (B, A, ω, V ) e -(x-ω) 2 2V .
(5.15)

The AMP algorithm for the spiked Wigner model 5.1 is very similar and can be readily obtained by imposing at each time step (û t , ĉt u ) = (v t , ĉt v ) and removing the redundant equations in Algorithm 4.

State evolution equations

Perhaps the most important virtue of AMP-type algorithms is that their asymptotic performance (here the MSE) can be tracked exactly via a set of scalar equations called state evolution (SE). This has been proven for a range of models including the spiked matrix models with separable priors in [START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling[END_REF], and with non-separable priors in [START_REF] Berthier | State evolution for approximate message passing with non-separable functions[END_REF]. Adapting the steps of these works, we now derive the state evolution equations for our structured model. As for the algorithm, we state the equations for the Wishart model 5.2, from which the Bayes-optimal state evolution equations for the Wigner model 5.1 can be readily obtained.

We simply state the SE equations obtained by an asymptotic analysis of AMP. The SE gives the evolution of the following order parameters under Algorithm 4:

                   q t u ≡ E u lim n→∞ (û t ) ût n = E u lim n→∞ (û t ) u n ≡ m t u , q t v ≡ E v lim p→∞ (v t ) vt p = E v lim p→∞ (v t ) v p ≡ m t v , q t z ≡ E z lim k→∞ (ẑ t ) ẑt k = E z lim k→∞ (ẑ t ) z k ≡ m t z .
(5.16)

Note that we used the Nishimori proposition 1.1, itself a consequence of Bayes-optimality, to deduce the equality between q and m. From this definition, it is clear that computing q u , q v , q z is enough to access the asymptotic MSE achieved by the AMP algorithm. We can now state the closed set of AMP state evolution equations for the Wishart model 5.2:

                               q t+1 u = E u ,ξ f u q t v ∆ u + q t v ∆ ξ, q t v ∆ 2 ,
(5.17a)

q t+1 v = E v ,ξ,η f v τ q t u ∆ v + τ q t u ∆ ξ, τ q t u ∆ , q t z η, ρ z -q t z 2 , (5.17b) qt z = αE v ,ξ,η f out τ q t u ∆ v + τ q t u ∆ ξ, τ q t u ∆ , q t z η, ρ z -q t z 2 ,
(5.17c)

q t+1 z = E z ,ξ [f z (q t z z + qt z ξ, ρ z -q t z ) 2 ].
(5.17d)

Note that we introduced an additional auxiliary variable qz , and recall that τ ≡ n/p, α = p/k and the definitions of the auxiliary functions in eqs. (5.14),(5.15). In the expectations, the variables ξ, η are taken independently from N (0, 1), and u * , v * , z * are drawn from P u , P v , P z . For a detailed derivation of eq. (5.17 5.21) at α = 2 with informative leading eigenvector with eigenvalue equal to 1 out of the bulk for ∆ ≤ 1 + α. We show the bulk spectral density µ(α, ∆) and the two leading eigenvalues (inset).

State evolution equations in the Wigner model

The state evolution equations for the Wigner model 5.1 are obtained as a particular case of the above by simply restricting q t u = q t v and τ = 1. In the end (after performing a change of variables) this leaves us with only three coupled equations:

                   q t+1 z = E ξ [{Z z × f 2 z }( qt z ξ, qt z )] = 2∂ qz Ψ z (q t z ), (5.18a)
qt z = αE ξ,η {Z out × f 2 out } q t v ∆ ξ, q t v ∆ , q t z η, ρ z -q t z = 2α∂ qz Ψ out q t v ∆
, q t z , (5.18b)

q t+1 v = E ξ,η {Z out × f 2 v } q t v ∆ ξ, q t v ∆ , q t z η, ρ z -q t z = 2∂ qv Ψ out q t v ∆ , q t z .
(5.18c)

We initialize these iterations as q t=0 v = ε, q t=0 z = ε, with a small ε > 05 . We notice immediately that eq. (5.18) is identical to the fixed point equations related to the Bayes-optimal estimation, eq. (5.10) with specific time-indices and initialization, but crucially the same fixed points. Thus the analysis of fixed points in Section 5.2.2 applies straightforwardly here. In particular, since in all cases analyzed we found the stable fixed point of eq. (5.10) to be unique, we conclude that our AMP algorithm reaches asymptotically optimal performance in these cases.

We further check our results by numerically comparing the runs of AMP on finite size instances with the state evolution curves already presented in Fig. 5.2, also giving an idea of the amplitude of the finite size effects (which are found to be fairly small). This experiment is illustrated in Fig. 5.4, together with a curve for PCA and for LAMP, a spectral method we derive from AMP in the next section. A code for reproducing this experiment is provided in a GitHub repository [ALM + 19].

LAMP: a spectral algorithm for generative priors

Spectral algorithms are arguably the most popular and simplest methods for solving the spiked matrix estimation problem. A seminal result from Baik, Ben Arous and Péché (BBP) [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF] shows that the leading eigenvector of Y (i.e. PCA) is correlated with the signal if and only if the signal-to-noise ratio satisfies ρ 2 v /∆ > 1. For sparse separable priors (with

ρ 2 v = Θ(1)), ∆ PCA = ρ 2
v is also the threshold for AMP and it is conjectured that no polynomial algorithm can improve upon it [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: Phase transitions, approximate message passing and applications[END_REF]. In contrast, in Section 5.2 we have developed an AMP algorithm (Algorithm 4) that has consistently better performance than PCA for structured priors, and in particular achieves the optimal weak recovery threshold. Despite all its virtues, AMP is unarguably a convoluted algorithm. It would be desirable to design a simple spectral algorithm that still takes into account the structured nature of the prior. In this section we design such a spectral algorithm, hereafter named L-AMP, and we show that it matches the optimal weak-recovery threshold. Our derivation follows a strategy pioneered in [KMM + 13], consisting on analyzing the AMP equations linearized around the noninformative fixed point. In this section, the discussion is framed on the Wigner model, the Wishart case being a straightforward generalization, given in [ALM + 20]. We also focus on the single-layer generative prior, and we will describe how multi-layer generalizations can be made.

We discussed in Section 5.2 the existence condition of the uninformative fixed point in the Bayesoptimal SE equations. Not surprisingly, since the SE of AMP is identical to the Bayes-optimal one, the same conditions can be obtained independently from the AMP equations by analyzing when v = 0, and we find back eq. (5.11):

(v, ẑ) = (0, 0) is a fixed point of AMP iff {E Q 0 out [v] = 0 and E Pv [z] = 0}.
(5.19)

Linearizing the AMP equations

The linearization of Algorithm 4 around the uninformative fixed point under eq. (5. 19) is fairly straightforward, and the interested reader can find it in [ALM + 20]. In the end, dropping time indices, we obtain an equation v = Γ vv p v where the LAMP operator Γ vv p is given by

Γ vv p ≡ 1 ∆ (a -b)I p + b WW k + c 1 p 1 k k W √ k Y √ p -aI p , ( 5.20)
where the parameters are simply the moments of distributions P z and Q

0 out a ≡ E Q 0 out [v 2 ] = ρ v , b ≡ ρ -1 z E Q 0 out [vx] 2 , c ≡ 1 2 ρ -3 z E Pz [z 3 ]E Q 0 out [vx 2 ]E Q 0 out [vx].
Note that in most of the cases we studied, the parameter c, taking into account the skewness of the variable z, is zero, simplifying considerably the structured matrix. Moreover, for the specific examples already discussed in Section 5.2, the LAMP operator Γ vv p is very simple. For instance, for Gaussian z and P out (v|x) = δ(v -sign(x)), we have (a, b, c) = (1, 2/π, 0). Instead, for linear activation we get (a, b, c) = (1, 1, 0). In this latter case, the LAMP operator can be written as

Γ vv p = 1 ∆ K p [ Y √ p -I p ] with K p = [WW ] k Σ ≡ 1 n α v α (v α ) , (5.21)
or, in other words, K p is the covariance matrix of the structured spike v. The same observation holds for the sign activation function. Interestingly, the covariance matrix Σ can be empirically estimated directly from samples of spikes, without the knowledge of the generative model (ϕ, W ) itself, suggesting a simple practical implementation of LAMP. With this in mind, we use a more A note on normalization -Note that the spectral methods and AMP use different normalizations. In order to compare them fairly in Fig. 5.4, we renormalized the spectral estimators to match the AMP estimator norm. We detail the reasons behind this renormalization in Appendix D.4.

Random matrix perspective on the spectral methods

Remarkably, the performance of the spectral method described in Algorithm 5 can be investigated independently of AMP using solely random matrix theory, for a linear generative prior. An analysis of the random matrix of eq. (5.21) shows that a spectral phase transition takes place as ∆ c = 1 + α (as for AMP). This transition is analogous to the well-known BBP transition [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF], but for the non-GOE random matrix of eq. (5.21).

For the spiked Wigner models with linear generative prior we prove two detailed theorems describing the behavior of the supremum of the bulk spectral density, the transition of the largest eigenvalue and the correlation of the corresponding eigenvector. The interested reader can find natural extensions of these theorems for spiked Wishart models in our work [ALM + 20]. We assume in the following that ρ v = 1 to simplify the analysis (without any loss of generality).

Recall that we have α = p/k and

Γ vv p ≡ 1 k WW 1 √ ∆p ξ + 1 ∆ vv p - 1 ∆ I p . (5.22) Here ξ ij i.i.d.
∼ N (0, 1 + δ ij ). Note that v also depends on W via the generative prior, so that the spike in eq. (5.22) is correlated with the unspiked matrix, as opposed to the classical case of [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF]. We first describe the behavior of the bulk of Γ vv p .

Theorem 5.2 (Bulk of the spectral density of Γ vv p ) For any α, ∆ > 0, as p → +∞, the spectral measure of Γ vv p converges almost surely and in the weak sense to a well-defined and compactly supported probability measure µ(α, ∆), and we denote supp µ its support. We separate two cases:

(i) If ∆ ≤ 1/4, then supp µ ⊆ R -.
(ii) Assume now ∆ > 1/4 and denote z 1 (∆) ≡ -∆ -1 + 2∆ -1/2 > 0. Let ρ ∆ be the probability measure on R with density

ρ ∆ (dt) = √ ∆ 2π 4 -∆ t + 1 ∆ 2 × 1 |t + 1 ∆ | ≤ 2 √ ∆ dt.
(5.23)

The following equation admits a unique solution for s ∈ (-z 1 (∆) -1 , 0):

α ρ ∆ (dt) st 1 + st 2 = 1.
(5.24)

We denote this solution as s edge (α, ∆) (or simply s edge ). The supremum of the support of µ(α, ∆) is denoted λ max (α, ∆) (or simply λ max ). It is given by:

λ max =          α ρ ∆ (dt)t 1 + s edge t - 1 s edge if α ≤ 1, max 0, α ρ ∆ (dt)t 1 + s edge t - 1 s edge if α > 1.
As a function of ∆, λ max has a unique global maximum, reached exactly at the point ∆ = ∆ c (α) = 1 + α. Moreover, λ max (α, ∆ c (α)) = 1.

We turn now to the description of the transition in the spectrum:

Theorem 5.3 (Transition of the largest eigenvalue of Γ vv p ) Let α > 0. We denote λ 1 ≥ λ 2 the first and second eigenvalues of Γ vv p .

• If ∆ ≥ ∆ c (α), then as p → ∞ we have a.s. λ 1 → λ max and λ 2 → λ max .

• If ∆ ≤ ∆ c (α), then as p → ∞ we have a.s. λ 1 → 1 and λ 2 → λ max .

Further, denoting ṽ a normalized ( ṽ 2 = p) eigenvector of Γ vv p with eigenvalue λ 1 , then a.s. |ṽ v | 2 /p 2 → (∆) a.s., where (∆) = 0 for all ∆ ≥ ∆ c (α), (∆) > 0 for all ∆ < ∆ c (α) and lim ∆↓0 (∆) = 1. Theorems 5.2 and 5.3 are illustrated in the right panel of Fig. 5.4. As we will see, the proof gives the value of (∆), which coherently leads to the MSE of Fig. 5.4 in the linear case. The proofs of Theorems 5.2 and 5.3 are given in Sections 5.4.1 and 5.4.2. The method of proof of Theorem 5.3 is inspired by [START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF] 6 , and allows us to compute the squared correlation (∆). It is given, for all ∆ < ∆ c (α), as

(∆) = 1 α [S (2) (1)] 2 S (1,2) (1) .
The functions S (1,2) and S (2) are defined in the following Lemma 5.5, and formulas are also given that allow to compute them numerically. A Mathematica demonstration notebook is provided in a GitHub repository [ALM + 19].

Non-linear activation -In the non-linear case the random matrix analysis is harder to perform. Indeed, in the matrix Γ vv p , the Wishart matrix WW /k is replaced by aI+bWW /k with a, b ≥ 0. It is thus not possible to relate the spectrum of Γ vv p to the one of a symmetric matrix of the type WZW with W a gaussian i.i.d. matrix (a generalized sample covariance matrix as defined in Section 1.5). Techniques from free probability allow for a detailed treatment of the spectral transition in this case, but such an analysis is not performed in this thesis.

Application to real data recovery

Remarkably, the LAMP operator in eq. (5.21) only depends on the generative prior through its covariance. An interesting exercise is therefore to apply LAMP to recover real data by simply using the empirical covariance for n samples of the spikes {v α } n α=1 . As an illustration we perform the following experiment: the spikes v are drawn from the Fashion-MNIST dataset [START_REF] Xiao | Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms[END_REF], and are used to generate the spiked matrix Y according to eq. (5.1). We then apply LAMP (Algorithm 5) to reconstruct the spikes, repeating this experiment for different noise values ∆. In Fig. 5.5 we compare the reconstruction by LAMP with standard PCA over Y. In principle, we have no theoretical guarantees about the performance of LAMP, since the Fashion-MNIST images are not drawn from the generative class studied above. Nevertheless, it is striking to observe that LAMP greatly outperforms PCA. A demonstration notebook illustrating this experiment is provided in a GitHub repository [ALM + 19].

Random matrix analysis of the transition

In this section we present the proofs of Theorems 5.2 and 5.3.

The bulk of eigenvalues: proof of Theorem 5.2

Proof of Theorem 5.2 (ii) -We begin by treating the more involved case (ii), that is we assume ∆ > 1/4. Note first that by basic linear algebra, the spectrum of Γ vv p is, up to 0 eigenvalues, the same as the spectrum of the following matrix Γ vv k :

Γ vv k ≡ 1 k W [ 1 √ ∆p ξ + 1 ∆ vv p - 1 ∆ I p ]W ∈ S k .
(5.25) More precisely, if p ≥ k we have Sp (Γ vv p ) = Sp (Γ vv k ) ∪ {0} p-k , and conversely if k > p. These additional zero eigenvalues in the case α > 1 explain the max(0, •) term in the expression of λ max in Theorem 5.2.

For the remainder of the proof we consider Γ vv k instead of Γ vv p given the remark above. Moreover, for simplicity we will drop the vv exponent in those matrices, and just denote them Γ k , Γ p .

The bulk of Γ k can be studied using standard random matrix theory results. Indeed, by the celebrated results of Wigner [START_REF] Wigner | Characteristic vectors of bordered matrices with infinite dimensions[END_REF], the spectral distribution of the matrix ξ/ √ ∆p -I p /∆ converges in law (and almost surely) as p → ∞ to ρ ∆ , given by eq. (5.23). Moreover, Γ k is precisely a generalized sample covariance matrix that we introduced in Section 1.5. Let us define ν(α, ∆) as the LSD of Γ k , and g ν (z) its Stieltjes transform. The main quantity of interest to us is z edge , defined as the supremum of the support of ν(α, ∆). It is easy to see that g ν induces a strictly increasing diffeomorphism g ν : (z edge , +∞) → (lim z ↓ z edge g ν (z), 0), so that we can define its inverse g -1 ν and from Theorem 1.7, it satisfies for every s ∈ (lim z ↓ z edge g µ (z), 0):

g -1 ν (s) = - 1 s + α ρ ∆ (dt) t 1 + st .
(5.26)

From the remarks above, µ(α, ∆) and ν(α, ∆) only differ by the addition of a delta distribution.

If z edge ≥ 0, then it will also be the supremum of the support of µ(α, ∆), and thus equal to λ max .

In order to compute z edge from eq. (5.26), we use a result of Section 4 of [START_REF] Jack | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF], also stated for instance in [START_REF] Oon | Tracy-Widom distribution for the largest eigenvalue of real sample covariance matrices with general population[END_REF], that describes the form of the support of ν(α, ∆). It can be stated in the following way. Recall that since ∆ > 1/4, z 1 (∆) > 0 is the maximum of the support of ρ ∆ . Let s edge be the unique solution in (-z 1 (∆) -1 , 0) of the equation (g -1 ν ) (s) = 0, that is by eq. (5.26):

α ρ ∆ (dt) st 1 + st 2 = 1.
(5.27) Indeed, it is straightforward to show that the left-hand side of eq. (5.27) tends to 0 as s ↑ 0, to +∞ as s ↓ -z 1 (∆) -1 , and is a strictly decreasing and continuous function of s. Then (see for instance eqs. (2.13) and (2.14) of [START_REF] Oon | Tracy-Widom distribution for the largest eigenvalue of real sample covariance matrices with general population[END_REF]) z edge is given by

z edge = lim s↓s edge g -1 ν (s) = - 1 s edge + α ρ ∆ (dt) t 1 + s edge t .
This ends the proof of (ii).

Let us make a final remark that will be useful in our future analysis. Note that z 1 (∆) > 1 for all ∆ > 1. Moreover, for all ∆ > 1, we have by an explicit computation:

α ρ ∆ (dt) t 1 -t 2 = α ∆ -1 .
By the argument above, this yields the following result, that we state as a lemma:

Lemma 5.4 Assume ∆ > 1. If ∆ < ∆ c (α), then s edge > -1. Conversely, if ∆ > ∆ c (α), then s edge < -1.
And moreover for ∆ = ∆ c (α), s edge = -1.

Proof of Theorem 5.2 (i) -Assume now ∆ ≤ 1/4, so that supp(ρ ∆ ) ⊂ R -. Since 0 ∈ R -, we can use again the remark we made in the proof of (ii) to study Γ k instead of Γ p . Moreover, Theorem 1.7 still applies here so that we have the Marchenko-Pastur equation (5.26). By the Stieltjes-Perron inversion formula (Theorem 1.5) it is enough to check that for every z > 0, there exists a unique s < 0 such that g -1 ν (s) = z. Indeed, this will yield that s = g ν (z) ∈ R, so that lim ↓0 Im g ν (z + i ) = 0. As this holds for every z > 0, it implies supp(ν) ⊆ R -and thus supp(µ) ⊆ R -. From eq. (5.26) and the fact that supp(ρ ∆ ) ⊆ R -, we easily obtain lim s→-∞ g -1 ν (s) = 0, and lim

s→0 - g -1 ν (s) = +∞.
Moreover, g -1 ν (s) is a strictly increasing continuous function of s, so that the existence and uniqueness of s = g ν (z) < 0 for any z > 0 is immediate, which ends the proof.

We now prove the final statement of Theorem 5.2, on the behavior of λ max with ∆, at fixed α. Proof of the behavior of λ max with ∆ -Recall that z edge = -(1/s edge ) + α ρ ∆ (dt)t/(1 + s edge t). Then we know that λ max = z edge if α ≤ 1 and λ max = max(0, z edge ) if α > 1. Let us make a few remarks:

• We already showed that, if ∆ ≤ 1/4 then z edge ≤ 0.

• It is trivial by the form of Γ k that, as ∆ → +∞, z edge → 0.

It is easy to see that z edge is a continuous and differentiable function of ∆, so that if we show the two following facts for any ∆ ≥ 1/4:

dz edge d∆ = 0 ⇔ ∆ = ∆ c (α) = 1 + α,
(5.28)

z edge (∆ c (α)) = 1, (5.29)
this would end the proof as z edge would then have a unique global maximum, located in ∆ = ∆ c (α), in which we have λ max = 1. We thus prove eq. (5.28) and eq. (5.29) in the following.

Proof of eq.

(5.28) -By the chain rule:

dz edge d∆ = ∂z edge ∂∆ + ∂s edge ∂∆ ∂z edge ∂s edge = ∂z edge ∂∆ .
Indeed one can check ∂z edge /∂s edge = 0 from eq. (5.27) and the Marchenko-Pastur equation.

Given the explicit form of ρ ∆ , one can compute easily:

∂z edge ∂∆ = - α 2s 3 edge s edge + 2s 2 edge -∆ s 2 edge -2s edge (1 + 2s edge )∆ + ∆ 2 + 1 .
It is then simple analysis to see that since s edge < 0, ∂z edge /∂∆ = 0 iff s edge = -1 and ∆ > 1. By Lemma 5.4, this is equivalent to ∆ = ∆ c (α) = 1 + α.

Proof of eq. (5.29) -By Lemma 5.4, we know that for ∆ = ∆ c (α) we have s edge = -1. Given the definition of ρ ∆ in eq. (5.23), it is straightforward to compute:

z edge (∆ c (α)) = -1 + α ρ ∆c(α) (dt) t 1 -t = 1.

BBP-like transition: proof of Theorem 5.3

Transition of the largest eigenvalue and corresponding eigenvector -This part is a detailed outline of the proof. Some parts of the calculation are not fully rigorous, however they can be justified more precisely by following exactly the lines of [START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF] and [START_REF] Jack | Analysis of the limiting spectral distribution of large dimensional random matrices[END_REF]. We will emphasize when such refinements have to be made. Recall that we have by eq. (5.22) the following decomposition of Γ vv p (denoted Γ p for simplicity):

Γ p = 1 k WW 1 √ ∆p ξ - 1 ∆ I p Γ (0) p + 1 ∆ WW k vv p rank-1 perturbation .
(5.30) Theorem 5.2, along with its proof, already describes in great detail the LSD of Γ (0) p . Note that for any λ ∈ R that is not an eigenvalue of Γ (0) p one can write:

det(λI p -Γ p ) = det I p -(λI p -Γ (0) p ) -1 1 ∆ WW k vv p det(λI p -Γ (0) p ).
In particular, this implies immediately that λ is an eigenvalue of Γ p and not an eigenvalue of Γ (0) p if and only if 1 is an eigenvalue of (λI p -Γ (0) p ) -1 WW vv /(∆kp). Since this is a rank-one matrix, its only non-zero eigenvalue is equal to its trace, so it is equivalent to:

1 = Tr [(λI p -Γ (0) p ) -1 1 ∆ WW k vv p ].
(5.31)

Recall that by definition, v is constructed as v = Wz/ √ k, with z ∼ N (0, I k ). For any matrix A, we have the classical concentration z Az/k = TrA/k with high probability as k → ∞. In eq. (5.31), this yields at leading order as p → ∞:

∆ = 1 p Tr (λI p -Γ (0) p ) -1 WW k 2 .
(5.32)

For practicality, we will prefer to use k × k matrices. We use the simple linear algebra identity, for any p × p symmetric matrix A, and any integer q ≥ 1:

Tr λI p - WW k A -1 WW k q = Tr λI k - 1 k W AW -1 W W k q .
This can be derived for instance by expanding both sides in powers of λ -1 and using the cyclicity of the trace. Using this along with eq. (5.32) we can state that the eigenvalues of Γ p that are outside of the spectrum of Γ (0) p must satisfy, as k → ∞:

α∆ = 1 k Tr (λI k -Γ (0) k ) -1 W W k 2 , (5.33) with Γ (0) k ≡ 1 k W 1 √ ∆p ξ - 1 ∆ I p W.
We will now make use of two important lemmas, at the core of our analysis. They will also prove to be useful in the eigenvector correlation analysis.

Lemma 5.5 (Hierarchy of S (r) k functions)

Recall that ν is the limit eigenvalue distribution of Γ (0) k , that the supremum of its support is λ max , and its Stieltjes transform is g ν . For every integer r ≥ 0, we define:

S (r) k (λ) ≡ 1 k Tr (Γ (0) k -λI k ) -1 W W k r .
For r ∈ {0, 1, 2, 3} 7 and every λ > λ max , as k → ∞, S (r) k (λ) converges almost surely to a well defined limit S (r) (λ). This limit is given by:

                 S (0) (λ) = g ν (λ), S (1) (λ) = g ν (λ)[α -(1 + λg ν (λ))], S (2) (λ) = g ν (λ)[α(1 + α) -(1 + 2α)(1 + λg ν (λ)) + (1 + λg ν (λ)) 2 ], S (3) (λ) = g ν (λ)[(α + 3α 2 + α 3 ) + (2 + 3α)(1 + λg ν (λ)) 2 -(1 + 5α + 3α 2 )(1 + λg ν (λ)) -(1 + λg ν (λ)) 3 ].
We define similarly for every integers r, q ≥ 0:

S (r,q) k (λ) ≡ 1 k Tr (Γ (0) k -λI k ) -1 W W k r (Γ (0) k -λI k ) -1 W W k q . Note that S (r,q) k = S (q,r) k and that S (r,0) k (λ) = ∂ z S (r) k (λ). For every λ > λ max , S (1,1) k (λ) and S
(1,2) k (λ) converge a.s. as k → ∞ to well-defined limits, that satisfy the following equations:

S (1,1) (λ) = g ν (λ)S (2) (λ) -[1 + λg ν (λ)]∂ λ S (1) (λ) + αg ν (λ)[g ν (λ) + S (1) (λ)] ρ ∆ (dt)t (1 + tg ν (λ)) 2 [t ∂ λ S (1) (λ) -g ν (λ)], S (1,2) (λ) = -[1 + λg ν (λ)][S (1,1) (λ) + (1 + α)∂ λ S (1) (λ)] + g ν (λ)S (3) (λ) + αg ν (λ)[(1 + α)g ν (λ) + S (1) (λ) + S (2) (λ)] ρ ∆ (dt)t (1 + tg ν (λ)) 2 [t ∂ λ S (1) (λ) -g ν (λ)].

Lemma 5.6 (Properties of S (r) )

Let α, ∆ > 0. We focus mainly on S (2) (λ). We have:

(i) For every r, S (r) (λ) is a strictly increasing function of λ, and lim λ→∞ S (r) (λ) = 0.

(ii) For every λ > λ max , S (2) (λ) = -α∆ iff ∆ ≤ ∆ c (α) and λ = 1. (iii) For every ∆ > ∆ c (α), lim λ↓λmax S (2) (λ) ∈ (-α∆, 0).
Let us first see how item (ii) of Lemma 5.6 and eq. (5.33) end the proof of the eigenvalue transition. First, note that by the celebrated Weyl's interlacing inequalities [START_REF] Weyl | Inequalities between the two kinds of eigenvalues of a linear transformation[END_REF], we have:

lim inf p→∞ λ 1 ≥ λ max and lim sup p→∞ λ 2 ≤ λ max .
This implies that because the perturbation of the matrix is of rank one, at most one outlier eigenvalue will exist in the limit p → ∞. By eq. (5.33), this outlier λ 1 exists if and only if it 7 The almost sure convergence can be extended to all r ∈ N but we will only use these values of r in the following.

satisfies, in the large p → ∞ limit, the equation S (2) (λ 1 ) = -α∆. By item (ii) of Lemma 5.6, this is the case only for λ 1 = 1 and ∆ ≤ ∆ c (α), which ends the proof.

A completely rigorous treatment of the previous arguments requires to state more precisely concentration results. Such a treatment has been made in [START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF] in a close case (from which all the arguments transpose), and we refer to it for more details. Lemma 5.5, which is at the core of our proof, is proven in the following. We postpone the proof of Lemma 5.6 to Appendix D.3.1.

Proof of Lemma 5.5 -The essence of the computation originates from the derivation of Theorem 1.7 in [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF], and is in essence a cavity computation, see Section 1.4. Note that S (0) k (λ) converges a.s. to the Stieltjes transform g ν (λ) as k → ∞ by Theorem 1.7. For every 1 ≤ i ≤ p, w i denotes the i-th row of W. We denote y = ξ/ √ ∆p-I p /∆. Since W is independent of y, we can denote y 1 , • • • , y p the eigenvalues of y, and their empirical distribution converges a.s. to ρ ∆ as we know. We have in distribution:

Γ (0) k = 1 k W y W d = α p p i=1 y i w i w i .
For every i, we denote Γ (0)

k,i ≡ (α/p) p j( =i) y j w j w j . Note that Γ (0)
k,i is independent of w i . We start from the (trivial) decomposition, for every λ:

- 1 λ = (Γ (0) k -λI k ) -1 - 1 λ W y W k (Γ (0) k -λI k ) -1 .
(5.34)

We will make use of the Sherman-Morrison formula that gives the inverse of a matrix perturbed by a rank-one change:

(B + tωω ) -1 = B -1 - t 1 + tω B -1 ω B -1 ωω B -1 . (5.35)
Using it in eq. (5.34) yields:

-

1 λ = - α λ 1 p p i=1 y i w i w i (Γ (0) k,i -λI k ) -1 1 + y i k w i (Γ (0) k,i -λI k ) -1 w i + (Γ (0) k -λI k ) -1 .
(5.36)

Taking the trace of eq. (5.36), using the independence of w i and Γ (0) k,i , and the concentration (1/k)w i Aw i = (1/k)TrA with high probability for large k, we obtain the following equation:

- 1 λ = g ν (λ) -g ν (λ) α λ ρ ∆ (dt) t 1 + tg ν (λ)
.

(5.37) This is exactly Theorem 1.7! In the following, we will use very similar identities. A completely rigorous derivation of these would, however, require many technicalities to ensure in particular the concentration of all the involved quantities. It would exactly follow the proof of [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF], and thus we do not repeat all the technicalities here. We can multiply eq. (5.36) by W W/k, and take the trace:

- 1 λ 1 k Tr WW k = S (1) k (λ) - α λ 1 p i y i w i √ k (Γ (0) k,i -λI k ) -1 ( 1 k j w j w j ) w i √ k 1 + y i k w i (Γ (0) k,i -λI k ) -1 w i .
This implies that S

(1) k (λ) converges as p, k → ∞ to a limit S (1) (λ), which satisfies:

- α λ = S (1) (λ) - α λ ρ ∆ (dt) t 1 + tg ν (λ) (g ν (λ) + S (1) (λ))
Using finally eq. (5.37), it is equivalent to:

S (1) (λ) = g ν (λ)[α -(1 + λg ν (λ))].
Multiplying eq. (5.36) by (W W/k) 2 or (W W/k) 3 yields, by the same analysis:

S (2) (λ) = [α(1 + α) -(1 + 2α)(1 + λg ν (λ)) + (1 + λg ν (λ)) 2 ]g ν (λ), S (3) (λ) = [(α + 3α 2 + α 3 ) -(1 + 5α + 3α 2 )(1 + λg ν (λ)) + (2 + 3α)(1 + λg ν (λ)) 2 -(1 + λg ν (λ)) 3 ]g ν (λ).
The convergence of S

(1,1) k (λ) and S

(1,2) k (λ) follows from the same analysis, as well as the equations they satisfy. We detail the derivation of the equation on S (1,1) (λ) and leave the derivation of the second equation for the reader. We multiply eq. (5.36) by W W/k. To simplify the calculations, we make use of concentrations, and denote F i ≡ (W Ww i w i )/k, which is independent of w i . We obtain at leading order as p → ∞:

- W W kλ = (Γ (0) k -λI k ) -1 W W k - α λ 1 p p i=1 y i g ν (λ)w i w i 1 + y i g ν (λ) - α λ 1 p p i=1 y i 1 + y i g ν (λ) w i w i (Γ (0) k,i -λI k ) -1 F i .
We multiply this equation by (Γ

k -λI k ) -1 and use eq. (5.35) in the form:

(Γ (0) k -λI k ) -1 = (Γ (0) k,i -λI k ) -1 -(Γ (0) k,i -λI k ) -1 y i w i w i 1 + y i g ν (λ) (Γ (0) k,i -λI k ) -1 .
Using again simple concentration, this yields at leading order:

- W W kλ (Γ (0) k -λI k ) -1 = (Γ (0) k -λI k ) -1 W W k (Γ (0) k -λI k ) -1 - α λ 1 p p i=1 y i w i w i (Γ (0) k,i -λI k ) -1 F i (Γ (0) k,i -λI k ) -1 1 + y i g ν (λ) + ∂ λ S (1) (λ) λ α p p i=1 y 2 i w i w i (Γ (0) k,i -λI k ) -1 (1 + y i g ν (λ)) 2 - α λ 1 p p i=1 y i g ν (λ)w i w i (Γ (0) k,i -λI k ) -1 (1 + y i g ν (λ)) 2 .
We finally multiply this equation by W W/k and take the trace. Using again usual concentrations we reach:

- S (2) (λ) λ = S (11) (λ) - α λp p i=1 y i [S (11) (λ) + ∂ λ S (1) (λ)] 1 + y i g ν (λ) (5.38) + ∂ λ S (1) (λ) λ α p p i=1 y 2 i [g ν (λ) + S (1) (λ)] (1 + y i g ν (λ)) 2 - α λ 1 p p i=1 y i g ν (λ) (1 + y i g ν (λ)) 2 [g ν (λ) + S (1) (λ)].
We now take the limit p → ∞ and use Theorem 1.7 in the form:

α λ ρ ∆ (dt) t 1 + tg ν (λ) = 1 + 1 λg ν (λ)
.

Inserting this into eq. (5.38) along with some trivial algebra yields the sought identity:

S (1,1) (λ) = g ν (λ)S (2) (λ) -[1 + λg ν (λ)]∂ λ S (1) (λ) + αg ν (λ)[g ν (λ) + S (1) (λ)] ρ ∆ (dt)t (1 + tg ν (λ)) 2 [t ∂ λ S (1) (λ) -g ν (λ)],
which is what we aimed to show. Performing the same analysis for S (1,2) (λ) ends the proof.

The proof of correlation of the leading eigenvector is derived on the same principles, and is given in Appendix D.3.2. All together, this ends the proof of Theorem 5.3.

On the nature of the transition -As was already noticed in previous works (see e.g. a related remark in [START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF]), the existence of a transition in the largest eigenvalue and the corresponding eigenvector for a large matrix of the type M + θP (with P of finite rank and θ > 0) depends on the decay of the asymptotic spectral density of M at the right edge of its bulk. For a power-law decay, there can be either no transition, a transition in the largest eigenvalue and the corresponding eigenvector, or a transition in the largest eigenvalue but not in the corresponding eigenvector. The situation in our setting is somewhat more involved, as both the bulk and the spike depend on the parameter ∆, and they are not independent (they are correlated via the matrix W). However, this intuition remains true: if we do not show and use it explicitly, the decay of the density of µ(α, ∆) at the right edge is of the type (λ max -λ) 1/2 , which is the hidden feature that is responsible for a transition both in the largest eigenvalue and the corresponding eigenvector that what we show in Theorem 5.3.

Perspectives on Chapter 5

This chapter presented a detailed analysis of the influence of data structure on optimal learning in the spiked-matrix model. We modeled the data structure by generative priors, and were motivated by comparing our findings to the case of sparse data, for which the underlying structure induces computational gaps. We detailed the two main conclusions of our analysis in Section 5.1.4: first, in contrast to the sparse case, there is no algorithmic gap with random generative priors. And secondly, we have designed a spectral algorithm (LAMP, Algorithm 5) that reaches the optimal weak-recovery threshold and that we can analyze both using the asymptotic analysis of AMP and random matrix theory. + 20], part of the theoretical machine learning community has very recently gained interest in understanding the role of data structure in learning, especially in neural networks. As highlighted in [START_REF] Zdeborová | Understanding deep learning is also a job for physicists[END_REF], data structure is one of the three key elements needed to build a theory of neural network learning, along with the network architecture and the optimization algorithms. It is therefore quite natural to see more and more recent works in the same general line as this chapter. To name a few (from collaborators), the reader can refer to [ALB + 20] for the phase retrieval problem, or [START_REF] Goldt | Modeling the influence of data structure on learning in neural networks: The hidden manifold model[END_REF] which introduced a similar model called hidden manifold, that allowed to study the influence of structure in a very flexible manner. While the majority of these works rely on synthetic (i.e. random) datasets, recent theoretical and empirical studies are indicating that a large part of their results remains valid for trained generative priors [GRM + 20]. All these contributed to create an exciting and rapidly growing line of work, and one aim of this chapter was to show that many tools of statistical physics applied to inference are particularly suited to build the premises of a theory of the role of data structure in learning.

While this chapter is based mainly on [ALM

Chapter 6

Phase retrieval: theoretical transitions and efficient algorithms "Quand les physiciens nous demandent la solution d'un problème, ce n'est pas une corvée qu'ils nous imposent, c'est nous au contraire qui leur devons des remerciements."

Henri Poincaré, La Valeur de la Science -Chapitre V: L'Analyse et la Physique (1911).

Disclaimer -We end our tour of high-dimensional inference problems by one of the most renowned models in this class: phase retrieval. This dense chapter merges items from a large part of our toolbox: the replica method and the means to prove its results, message-passing algorithms to study hardness, but also the TAP picture, and the results of Chapter 2, to understand spectral algorithms to solve phase retrieval. It is mainly based on two published works [MLKZ20, MKLZ21], which focused respectively on the fundamental limits of phase retrieval, and on the design of optimal spectral methods to solve it.

Recall our mathematical notation: we let β ∈ {1, 2} for respectively real (K = R) and complex (K = C) variables.

The phase retrieval problem

Consider the reconstruction problem of a signal X * ∈ K n from m observations of its modulus

Y µ = 1 √ n n i=1 Φ µi X * i , µ = 1, • • • , m, (6.1)
where the m × n sensing matrix Φ ∈ K m×n is known. More generally, measurements can be a noisy function of the modulus, for example by an additive Gaussian noise, or by a Poisson noise channel. This inverse problem, known in the literature under the umbrella of phase retrieval arises in a large set of problems ranging from signal processing [Fie82, UE88, DLM + 15] to statistical estimation [CLS15b, JEH15], optics, X-ray crystallography, astronomy or microscopy [SEC + 15], where detectors can often only measure information about the amplitude of signals, and lose all information about its phase, cf. Fig. 6.1. Phase retrieval is also a textbook example of a learning problem with a highly non-convex loss landscape [NJS15, SQW18, HLV18]. It moreover falls into the large category of generalized linear models (GLMs), that we introduced in Section 1.1.

Phase retrieval and generalized linear models -Let us first precise the class of models studied in this chapter. We assume that the (random) sensing matrix Φ satisfies right-rotation invariance, in the sense of Model R. The observations are then generated by a Bayes-optimal GLM similar to the ones we encountered in the previous chapters. Precisely, the signal X * to recover is drawn using a factorized prior distribution P 0 with zero mean and variance ρ ≡

E P 0 [|x| 2 ] > 0.
The observations are generated as:

Y µ ∼ P out • 1 √ n n i=1 Φ µi X i , 1 ≤ µ ≤ m, (6.2)
and we assume that the prior P 0 and the "channel" distribution P out (y|z) are known. Eq. (6.2) defines the very general class of Generalized Linear Models (GLMs), cf. Section 1.1. The present chapters focuses on phase retrieval problems: Definition 6.1 (Phase retrieval)

We generically denote as phase retrieval GLMs of the type of eq. (6.2) in which we assume that P out (y|z) is a function of |z|, and in which the prior distribution P 0 is also symmetric: P 0 (x) = P 0 (|x|).

For instance, for Gaussian additive noise one has P out (y|z) = N 1 (y; |z| 2 , ∆), while the noiseless case corresponds to the limit ∆ ↓ 0 : P out (y|z) = δ(y -|z| 2 ). Finally, we consider a now-standard high-dimensional (thermodynamic) limit, in which n, m → ∞ with m/n → α > 0. All in all, the model we consider is very generic, and encompasses e.g. phase retrieval with unitary matrices (and in some extent Fourier phase retrieval as we will see), or Gaussian phase retrieval.

Related literature -A great amount of work is present both in the statistical physics and in the information theory literature for different assumptions on the matrix Φ. The asymptotic optimal performances (both information-theoretic and algorithmic) of generic GLMs with rightrotation invariance sensing matrices were conjectured using the non-rigorous replica method of statistical physics in [START_REF] Kabashima | Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Takahashi | Macroscopic analysis of vector approximate message passing in a model mismatch setting[END_REF], and these results were proven for Gaussian matrices in [BKM + 19]. This analysis was later non-rigorously extended to the case of non-separable prior distributions [ALB + 20]. Specifically for the phase retrieval problem, the limits of weak-recovery were analyzed for Gaussian matrices Φ in [MM19, LAL19, LL20], and for column-unitary Φ in [DMM20, DBMM20, MDX + 21]1 . As we know, the Bayes-optimal estimation can be summarized in the study of the posterior probability of x given the observations Y and the matrix Φ:

P(x|Y, Φ) ≡ 1 Z n (Y) n i=1 P 0 (x i ) m µ=1 P out Y µ 1 √ n n i=1 Φ µi x i . (6.3)
We will generically denote by • the average with respect to the posterior probability (6.3).

On the algorithmic side, the (generalized) vector approximate message-passing (G-VAMP) algorithm [SRF16, RSF17] (cf. Section 1.4 and Chapter 2) has been conjectured to achieve the optimal polynomial-time recovery for rotationally (unitarily) invariant matrices2 . Despite its properties, G-VAMP is a numerically very cumbersome algorithms. It is therefore of great interest to investigate more robust and computationally cheaper algorithms that could achieve similar performances. A natural class of such algorithms are spectral methods, i.e. methods based on the leading eigenvector of a matrix constructed from the observations. Their output can be used as informative initializations for local gradient-based optimization algorithms, and can induce a jump in the accuracy achieved at a reasonable computational cost. Such techniques have already been applied e.g. in optical systems [START_REF] Valzania | Accelerating ptychographic reconstructions using spectral initializations[END_REF]. In the context of phase retrieval, the performance of these spectral methods has been rigorously analyzed for Gaussian [MM19, LAL19, MTV20, LL20] and unitary [DBMM20, DB20, MDX + 21] sensing matrices.

Organization of the chapter and main contributions -

The present chapter extends all these past results by considering arbitrary matrices with orthogonal or unitary invariance properties, encapsulating all the cases described above into a single framework. Our main contributions are twofold:

(I) First, we derive sharp asymptotics for the lowest possible estimation error achievable statistically and algorithmically, locate the phase transitions for weak-and full-recovery as a function of the singular values of the matrix Φ and also discuss the existence of a statisticalto-algorithmic gap. Our main contributions on this matter are:

-In Section 6.2, we use the replica method to derive a unified single-letter formula for the performance of the Bayes-optimal estimator for generic real or complex GLMs defined by eq. ( 6.2) in the thermodynamic limit, and for right-rotationally invariant Φ. We then prove this conjecture in two particular cases. First, when the distribution P 0 is Gaussian and Φ = WB is the product of a Gaussian matrix W with an "arbitrary" matrix B. Second, for a Gaussian matrix Φ (real or complex) with any separable distribution P 0 . These are non-trivial extensions of the the proofs of [BKM + 19, BM19a] and [START_REF] Barbier | The mutual information in random linear estimation beyond iid matrices[END_REF]. We also argue that message-passing algorithms, here G-VAMP, achieves the optimal performance reachable in polynomial time among a large class of algorithms, and describe its iterations.

-In Section 6.3, we analytically characterize (as a function of the singular values distribution of Φ) the algorithmic weak-recovery threshold α WR,Algo above which better-than-random inference reconstruction of X * is possible in polynomial time. We also derive an explicit formula for the information-theoretic full recovery threshold α FR,IT above which full reconstruction of X * (i.e. perfect recovery up to the possible rank deficiency of Φ) is statistically possible, as a function of the singular values distribution of Φ.

Our findings for the statistical and algorithmic thresholds are summarized in Table 6.1, for different real and complex ensembles of Φ. Entries in bold emphasize new results obtained in this chapter, filling a gap between the different previous works in the phase retrieval literature.

(II) Secondly, in Section 6.4 we design (conjecturally) optimal spectral methods for the phase retrieval problem in the aforementioned limit, for the very generic class of right-rotationally invariant sensing matrices. Most importantly, in contrast to previous works our approach is completely automated, in the sense that our spectral methods are conjectured to be optimal, and their derivation does not require optimization over any hyperparameter. The constructiveness of our approach gives more weight to our optimality conjecture, as we do not restrict

1 [CT06] 1.12 [BKM + 19] Complex Gaussian Φ (β = 2) 1 [MM19, LAL19] 2 2.027 Real column-orthogonal Φ (β = 1) 1.5 1 [CT06] 1.584 Complex column-unitary Φ (β = 2) 2 [MP17, MDX + 21] 2 2.265 Φ = W 1 W 2 (β = 1, aspect ratio γ) γ/(2(1 + γ)) [ALB + 20] min(1, γ) [CT06] Thm. 6.2 [ALB + 20] Φ = W 1 W 2 (β = 2, aspect ratio γ) γ/(1 + γ) min(2, 2γ) Thm. 6.2 Φ, β ∈ {1, 2}, rk[Φ † Φ]/n = r
Eq. (6.13) βr Conj. 6.1 Gauss. Φ, β ∈ {1, 2}, symm. P 0 , P out Eq. (6.12) [START_REF] Mondelli | Fundamental limits of weak recovery with applications to phase retrieval[END_REF][START_REF] Luo | Optimal spectral initialization for signal recovery with applications to phase retrieval[END_REF] Thm. 6.2 Thm. 6.2 Φ, β ∈ {1, 2}, symm. P 0 , P out Eq. (6.11) Conj. 6.1 Conj. 6.1 Table 6.1: Values of the algorithmic weak recovery, information-theoretic full recovery, and algorithmic full recovery thresholds for several random matrix ensembles. When the ensemble of Φ is not specified, we consider any right-orthogonally (unitarily) invariant ensemble in the sens of Model R. The last two lines are given for any symmetric (cf Def. 6.1) prior P 0 and channel P out , while all other results are for Gaussian P 0 and a noiseless phase retrieval channel. We reference results of this chapter when the value is not given by a closed-form expression, but can be computed from the formulas herein. In some particular ensembles, we have numerically analyzed these equations in Section 6.5. The new results of this chapter are written in bold blue, and we give references to papers in which the previously known thresholds were computed.

to a specific family of spectral methods. In designing these methods, we unify three different approaches: a "pedestrian" optimization of the preprocessing function (the approach of the aforementioned previous works), the linearization of message-passing algorithms, and a Bethe Hessian analysis.

Finally, in Section 6.5, we numerically investigate our two contributions. In particular, we uncover interesting transition phenomena in the eigenvalues of the spectral methods, and we numerically establish the existence or absence of a statistical-to-algorithmic gap for many ensembles of Φ in noiseless phase retrieval, for which such an analysis was lacking.

Important notation -Recall that for m ≥ n, a matrix A ∈ K m×n is said to be columnorthogonal (unitary) if A † A = I n . For x, y ∈ K, we define a 'dot product' as

x • y ≡ xy if K = R and x • y ≡ Re[xy] if K = C. In particular x • x = |x| 2 .
The Gaussian measure N β (0, 1) is defined as D β z ≡ (2π/β) -β/2 exp(-β|z| 2 /2) dz. ν will denote the asymptotic spectral density of Φ † Φ/n and we designate f (λ) ν ≡ ν(dλ)f (λ) the linear statistics of ν.

Optimal estimation in GLMs with structured data

Replica free entropy and how to prove it

Recall that we placed ourselves in a setting known as Bayes-optimal: the statistically optimal estimator X minimizing the mean-squared error mse( X) ≡ || X -X || 2 2 is therefore simply given by the posterior mean Xopt = x , where the posterior distribution is given in eq. (6.3). As we know, while exact sampling from the posterior is intractable for large values of n, m ∈ N * , the replica method allows us to access important asymptotic quantities, such as the free entropy3 . It fully characterizes the asymptotic performance of the Bayes-optimal estimator Xopt in high dimensions via the I-MMSE theorem [START_REF] Guo | Mutual information and minimum mean-square error in Gaussian channels[END_REF].

The first result of this chapter is a single-letter formula for the asymptotic free entropy for rightrotationally invariant sensing (or data) matrices Φ4 , in the limit n, m → ∞ with m/n → α > 0.

Conjecture 6.1 (Replica-symmetric conjecture for generic GLMs)

Under the assumptions above, the asymptotic free entropy for the posterior distribution defined in eq. ( 6.3) with right-orthogonally (unitarily) invariant sensing matrix Φ is:

lim n→∞ 1 n E Y,Φ ln Z n (Y) = sup qx∈[0,ρ] sup qz∈[0,Qz]
[I 0 (q x ) + αI out (q z ) + I int (q x , q z )], (6.4)

where

                                           I 0 (q x ) ≡ inf qx≥0 - β qx q x 2 + E ξ Z 0 ( qx ξ, qx ) ln Z 0 ( qx ξ, qx ) , I out (q z ) ≡ inf qz≥0 - β qz q z 2 - β 2 ln( Qz + qz ) + β qz 2 Qz +E ξ R dy Z out y; qz Qz ( Qz + qz ) ξ, 1 Qz + qz ln Z out y; qz Qz ( Qz + qz ) ξ, 1 Qz + qz , I int (q x , q z ) ≡ inf γx,γz≥0 β 2 (ρ -q x )γ x + αβ 2 (Q z -q z )γ z - β 2 ln(ρ -1 + γ x + λγ z ) ν - β 2 ln(ρ -q x ) - βq x 2ρ - αβ 2 ln(Q z -q z ) - αβq z 2Q z .
We defined Q z ≡ ρ λ ν /α and Qz ≡ 1/Q z , ξ ∼ N β (0, 1) and the following auxiliary functions:

Z 0 (b, a) ≡ K dz P 0 (z)e -β 2 a|z| 2 +βb•z , Z out (y; ω, v) ≡ E z [P out (y| √ vz + ω)], (6.5)
with z ∼ N β (0, 1). Moreover, the asymptotic MMSE achieved by the Bayes-optimal estimator is equal to ρ -q * x , with q * x the solution of the above extremization problem:

lim n→∞ MMSE = lim n→∞ 1 n E X * -Xopt 2 = ρ -q * x . (6.6)
This formula is derived in Appendix B.2 using the heuristic (hence the conjecture) replica method that we introduced in Section 1.3.1. Given its generality, the calculation of Appendix B.2 also serves in this thesis as a textbook example of application of the replica method, along with the one for the committee machine presented in Appendix B.1. It extends the formula from [START_REF] Takahashi | Macroscopic analysis of vector approximate message passing in a model mismatch setting[END_REF] to complex signals X * and sensing matrices Φ. In particular, it also holds in the case of complex matrices Φ and real signal X , by adding a constraint on the imaginary part of X in P 0 . It also encompasses the case of sparse signals, which is of wide interest in the compressive sensing literature [Don06, DMM09, KMS + 12, SR14, KMTZ14].

Proving Conjecture 6.1 is a challenging open problem. We provide a significant step by doing so for a broad class of likelihoods P out and in two settings: a restricted signal distribution P 0 and a broad class of real and complex likelihoods and sensing matrices Φ, or a broad class of prior distribution P 0 and (real or complex) Gaussian Φ. To state the theorem, we rewrite the model of eq. (6.2) as:

Y µ = ϕ out 1 √ n n i=1 Φ µi X i , A µ , 1 ≤ µ ≤ m, (6.7)
where (A µ ) m µ=1 ∈ K m are i.i.d. random variables with (known) distribution P A accounting for a possible noise, ϕ out is the observation channel and Φ is a random matrix with elements in K. P out (•|z) is then the PDF associated to the stochastic function ϕ out (z, A).

Theorem 6.2 (Replica-symmetric formula)

Let us define a set of hypotheses:

(H0) ϕ out : K 2 → R is C 2 , and (z, a) → (ϕ out (z, a), ∂ z ϕ out (z, a), ∂ 2 z ϕ out (z, a)) is bounded.
(h1) P 0 is a centered Gaussian distribution, without loss of generality P 0 = N β (0, 1).

(h2) Φ is distributed as Φ d = WB/ √ p, with W ∈ K m×p an i.i.d. standard Gaussian matrix, and B ∈ K p×n an arbitrary matrix (random or deterministic), independent of W. Moreover, as n → ∞, p/n → δ > 0.

(h3) The ESD of B † B/n weakly converges (a.s.) to a compactly-supported measure ν B = δ 0 . Moreover, there exists λ max > 0 such that a.s. λ max (B † B/n) → n→∞ λ max .

(h 1) P 0 has a finite second moment, and Φ µi

i.i.d ∼ N β (0, 1).
Assume that all (H0),(h1),(h2),(h3) or that all (H0),(h 1) stand. Then Conjecture 6.1 holds with ν the LSD of Φ † Φ/n5 .

The proof is based on the adaptive interpolation method6 . Its main strategy is sketched in the following, and shares similarities with the committee machine that we studied in Chapter 4. In particular, Theorem 6.2 allows to rigorously compute the asymptotic minimum mean-squared error (MMSE) achieved by the Bayes-optimal estimator. Remark 6.1 (Relaxing the hypotheses)

Following arguments of [BKM + 19] and Chapter 4, (H0) can be relaxed to continuity a.e. and the existence of moments of ϕ out , so that our theorem also covers noiseless phase retrieval.

As with the previous high-dimensional models we studied in Part II, the replica formula reduces the high-dimensional computation of the MMSE to a simple low-dimensional extremization problem. The MMSE as a function of the sample complexity α can be readily computed from eqs. (6.4) and (6.6) for a given signal distribution P 0 (determining I 0 ), likelihood P out (determining I out ) and spectral density ν (determining I int ).

Proof strategy for Theorem 6.2

Let us work under (H0),(h1),(h2),(h3). We will simply sketch the main steps of the proof strategy for the free energy. More details (e.g. going from the free entropy to the MMSE characterization), as well as the extension to hypotheses (H0),(h 1), can be found in [MLKZ20].

First, we simplify the conjectured expression of the free entropy of Conjecture 6.1 using the particular form of the prior P 0 and of the sensing matrix Φ. Finally, using (h1),(h2),(h3) and a proof similar to the one of [BKM + 19, AMB + 19], we give a rigorous derivation of this simplified expression. Note that with respect to the analysis of [BKM + 19, AMB + 19], there are two main novelties in our setting:

(i) The sensing matrix Φ is not i.i.d. but has a well-controlled structure, see (h2) and (h3).

(ii) The variables can be complex numbers. We will argue that the arguments generalize to this case. The physical reason of this generalization is that even in the complex setting, the overlap will concentrate on a real positive number, as a consequence of Bayes-optimality.

First, we note that we can simplify the replica conjecture under the considered hypotheses:

Proposition 6.3 (Simplified replica-symmetric formula)

Under (H0),(h1),(h2),(h3), the replica conjecture 6.1 for the free entropy

f n ≡ n -1 E ln Z n (Y) is equivalent to: lim n→∞ f n = sup q≥0 inf q∈[0,Qz] β q 2 (E ν B [X] -δq) - β 2 E ν B ln(1 + qX) + αΨ out (q) , (6.8) with Q z = E ν B [X]
/δ and Ψ out defined in terms of the auxiliary functions of eq. (6.5):

Ψ out (q) ≡ K D β ξ R dy Z out (y; √ qξ, Q z -q) ln Z out (y; √ qξ, Q z -q).
The proof of Proposition 6.3 is quite technical and is detailed in [MLKZ20]. Proposition 6.3 presents the replica-symmetric formula in a manner closer to its usual statement for Gaussian matrices (e.g. Theorem 4.1 for the committee machine), as a single supinf involving an order parameter q playing the role of the overlap. To finish the proof of the free entropy statement of Theorem 6.2, we therefore just need to show: Lemma 6.4

Under the assumptions of Proposition 6.3, the limit of the free entropy

f n ≡ n -1 E ln Z n (Y)
is given by eq. (6.8).

Let us now briefly describe our strategy to prove Lemma 6.4. The main idea is to reduce the problem to a Generalized Linear Model with a Gaussian sensing matrix, but a non-i.i.d. prior. We make use of the "SVD" decomposition of B/ √ n = USV † , with U ∈ U β (p), V ∈ U β (n), and S ∈ R p×n a pseudo-diagonal matrix with positive elements. Leveraging on the gaussianity of P 0 (hypothesis (h1)), and that W is an i.i.d. Gaussian matrix independent of B, one can see that our estimation problem is formally equivalent to an usual Generalized Linear Model with m measurements, a signal of dimension p, and a Gaussian i.i.d. sensing matrix. A key feature is that here the prior distribution on the data Z * ∈ K p is defined as

• If δ ≤ 1, for every k ∈ {1, • • • , p}, Z * k is distributed as S k X * k with X * k i.i.d. ∼ P 0 . • If δ ≥ 1, for every k ∈ {1, • • • , n}, Z * k is distributed as S k X * k with X * k i.i.d.
∼ P 0 , while for every

k ∈ {n + 1, • • • , p}, Z * k is almost surely 0.
More precisely, we can define rigorously the prior P (S) 0 described above by its linear statistics. For any continuous bounded function g : K p → R, one has:

K p P (S) 0 (dz)g(z) ≡ K n n i=1 P 0 (dx i ) g({I[k ≤ n]S k x k } p k=1
for t = 1, • • • , T do (Denoising x) (Denoising z ≡ 1 √ n Φx) xt 1 = g x1 (T t 1 , γ t 1 ) ẑt 1 = g z1 (R t 1 , τ t 1 ) v t 1 = 1 βn n i=1 ∂ T i g x1 (T t 1 , γ t 1 ) c t 1 = 1 βm m µ=1 ∂ Rµ g z1 (R t 1 , τ t 1 ) T t 2 = 1 v t 1 xt 1 -T t 1 R t 2 = 1 c t 1 ẑt 1 -R t 1 γ t 2 = 1 v t 1 -γ t 1 τ t 2 = 1 c t 1 -τ t 1 (Estimation of x) (Estimation of z) xt 2 = g x2 (T t 2 , R t 2 , γ t 2 , τ t 2 ) ẑt 2 = g z2 (T t 2 , R t 2 , γ t 2 , τ t 2 ) v t 2 = 1 τ t 2 λ+γ t 2 ν c t 2 = 1 α λ τ t 2 λ+γ t 2 ν T t+1 1 = 1 v t 2 xt 2 -T t 2 R t+1 1 = 1 c t 2 ẑt 2 -R t 2 γ t+1 1 = 1 v t 2 -γ t 2 τ t+1 1 = 1 c t 2 -τ t 2 return xT 1 ;
Armed with this equivalent representation of our problem, we can then apply an adaptive interpolation strategy to prove the replica-symmetric formula of Proposition 6.3. The strategy is very similar to the detailed proof we presented in Chapter 4: we design a simpler interpolating estimation problem whose free energy is precisely the one of Proposition 6.3, and that is parameterized by the two parameters q, q. These parameters are then well-chosen along the interpolation path, which allows to deduce the replica-symmetric formula. Since this proof is very similar to the one of the committee machine presented in Chapter 4 we do not reproduce it in this thesis, but the interested reader will find it in detail in [MLKZ20].

Algorithmic point of view: the G-VAMP algorithm

The majority of algorithms developed to solve phase retrieval are based either on semi-definite programming relaxations [START_REF] Emmanuel | Phase retrieval from coded diffraction patterns[END_REF][START_REF] Waldspurger | Phase retrieval with random gaussian sensing vectors by alternating projections[END_REF][START_REF] Goldstein | Phasemax: Convex phase retrieval via basis pursuit[END_REF] or on more direct non-convex optimization procedures, e.g. Wirtinger flow [START_REF] Emmanuel | Phase retrieval via Wirtinger flow: Theory and algorithms[END_REF]. The class of approximate message-passing algorithms has also been quite successful for specific instances of phase retrieval [START_REF] Schniter | Compressive phase retrieval via generalized approximate message passing[END_REF][START_REF] Mondelli | Approximate message passing with spectral initialization for generalized linear models[END_REF]. Leveraging our usual toolbox, we describe here the most generic algorithm of this class, that allows to tackle any right-rotationally invariant matrix Φ. It is denoted G-VAMP, and its general form was first written in [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF]. Recall that we already studied G-VAMP in relation with the TAP free entropy in Chapter 2. As for many similar problems, G-VAMP is the best-known polynomial time algorithm for this problem in terms of achieved MSE. It makes use of the SVD decomposition of Φ, that we write as Φ/ √ n = USV † . The full iterations of the algorithm are detailed in Algorithm 6. We used some auxiliary functions, defined below: .

g x1 (T, γ) i ≡ E P 0 (γ,-T i ) [x], g x2 (T, R, γ, τ ) ≡ T γ + VS γ τ + SS -1 U † R τ - SV † T γ , g z1 (R, τ ) µ ≡ E Pout yµ, Rµ τ , 1 τ [z], g z2 (T, R, γ, τ ) ≡ USV † g x2 (T, R, γ, τ
Statistical and algorithmic performance -Conjecture 6.1 and Theorem 6.2 show that the global maximum of the potential in eq. (6.4) describes the performance of the statistically optimal estimator Xopt . Moreover eq. (6.4) also contains rich information about the algorithmic aspects of this problem. Indeed, as for other message-passing algorithms analyzed in the previous chapters, it has been shown that the asymptotic performance of G-VAMP corresponds to the MSE achieved by running gradient descent on the potential in eq. (6.4) from the trivial initial condition q x = q z = 0 [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF]. This allows to derive the thresholds characterizing the statistical and algorithmic limitations of signal estimation solely from eq. (6.4).

State Evolution -

The extrema of the potential in eq. ( 6.4) can be characterized by the solutions of the following State Evolution (SE) equations, obtained by looking at the zerogradient points: ξ, 1 Qz+qz respectively. Note in particular that the two equations in eq. (6.10c) have to be solved over (γ x , γ z ) in order to be iterated.

               q x = E ξ Z 0 |f 0 | 2 , q z = 1 Qz+qz qz Qz + E ξ dy Z out |f out | 2 , (6.10a) qx = q x ρ(ρ -q x ) -γ x , qz = qz Qz(Qz-qz) -γ z , (6.10b) ρ -q x = 1 ρ -1 + γ x + λγ z ν , α(Q z -q z ) = λ ρ -1 +γx+λγz ν . ( 6

Weak and perfect recovery transitions

Notation -We adopt the subscript IT for the thresholds related to the Bayes-optimal estimator and Algo for the G-VAMP ones.

Weak recovery: beating a random guess

A natural question to ask is: what is the minimum sample complexity α WR,Algo ≥ 0 such that for all α ≥ α WR,Algo we can algorithmically reconstruct X better than a trivial random draw from the known signal distribution P 0 ? Also known as the algorithmic weak-recovery threshold, α WR,Algo can also be characterized in terms of the MSE achieved by G-VAMP:

α WR,Algo ≡ arg min α≥0 {MSE GVAMP (α) < ρ}.
Note that we already encountered such weak-recovery transitions (in terms of noise level) in the spiked matrix model of Chapter 5. Since the algorithmic performance is characterized by precisely maximizing eq. (6.4) starting from the trivial point, the threshold α WR,Algo can be analytically computed from a local stability analysis of this point. On a more general note, an important algorithmic question is to characterize the class of polynomial-time algorithms that can achieve weak recovery directly above the optimal threshold. Beyond G-VAMP, in Section 6.4 we will design simple spectral methods that achieve this feat.

Existence and location of the weak-recovery threshold -It is easy to verify that the state evolution equations (6.10) admit a trivial fixed point in which q x = q z = qx = qz = γ x = γ z = 0 when P 0 and P out are symmetric in the sense of Definition 6.1. When it exists, the trivial extremizer q x = q z = 0 correspond to either a linearly stable or unstable trivial fixed point of the state evolution equations. The weak-recovery threshold can therefore be determined by looking at the Jacobian around the trivial fixed point. The details of the stability analysis are given in Appendix D.5.1. The result is that a linear instability of the trivial fixed point appears at α = α WR,Algo satisfying the equation:

α WR,Algo = λ 2 ν λ 2 ν 1 + R dy K D β z (|z| 2 -1) P out y ρ λ ν α WR,Algo z 2 K D β z P out y ρ λ ν α WR,Algo z -1 . (6.11)
Note that the integrand and the averages • ν depend on α WR,Algo , so that this is an implicit equation on α WR,Algo . Eq. (6.11) is the most generic formula for the weak recovery threshold for any rotationally-invariant data matrix Φ and phase retrieval channel P out . As emphasized in the following examples, it generalizes in particular several previously known formulas for different channels and random matrix ensembles.

Gaussian matrix -For a Gaussian i.i.d. matrix, λ ν = α and λ 2 ν = α 2 + α, so that

α WR,Algo = R dy | K D β z (|z| 2 -1) P out (y| √ ρz)| 2 K D β z P out (y| √ ρz) -1
, (6.12) a result which was previously derived in [START_REF] Mondelli | Fundamental limits of weak recovery with applications to phase retrieval[END_REF] in the real and complex cases.

Noiseless phase retrieval -In noiseless phase retrieval one has P out (y|z) = δ(y -|z| 2 ). In particular one can easily check that this implies:

α WR,Algo = 1 + β 2 λ 2 ν λ 2 ν . (6.13)
This last formula allows to retrieve and generalize many results previously derived in the literature. For instance, for a Gaussian i.i.d. matrix, we find α WR,Algo = β/2 , which was derived in [START_REF] Mondelli | Fundamental limits of weak recovery with applications to phase retrieval[END_REF][START_REF] Luo | Optimal spectral initialization for signal recovery with applications to phase retrieval[END_REF]. For an orthogonal or unitary column matrix, α WR,Algo = 1 + (β/2), which was already known for β = 2[MM19] (but not for β = 1). For the product of p i.i.d. Gaussian matrices with sizes k 0 , • • • , k p , with k 0 = m and k p = n, and γ l ≡ n/k l for 0 ≤ l < p, we have α WR,Algo = (β/2)[1 + p l=1 γ l ] -1 , which generalizes the previously-known real case [ALB + 20]. Eq. (6.13) encapsulates all these results and goes beyond by considering an arbitrary spectrum for the sensing matrix, while eq. (6.11) also considers arbitrary channels P out .

The weak-recovery IT transition -We only considered the algorithmic weak-recovery threshold. Extending our analysis to the information-theoretic threshold α WR,IT is an interesting open direction, which requires understanding the appearance of a global maximum in the replica-symmetric potential of eq. (6.4), but not necessarily continuously from the q x = q z = 0 solution. At the moment, we are not able to carry such an analysis.

Perfect recovery for Gaussian signals in noiseless phase retrieval

We consider here noiseless phase retrieval P out (y|z) = δ(y -|z| 2 ) and a prior P 0 = N β (0, 1)7 .

For high number of samples α 1, we expect the MMSE to plateau at a minimum achievable reconstruction error MMSE 0 ≡ inf α MMSE(α), which is a function of the statistics of Φ. We thus look for the information-theoretical full-recovery threshold α FR,IT as the smallest sample complexity such that MMSE 0 is attained. In Appendix D.5.2 we show that the full-recovery can be perfect (MMSE 0 = 0) or partial (MMSE 0 > 0) depending on the rank of Φ and that α FR,IT ≡ β(1 -ν({0})).

(6.14)

Informally, ν({0}), the fraction of zeros in the spectrum of Φ † Φ/n, is the fraction of the signal "lost" by the sensing matrix. The effect of rank deficiency is illustrated in Fig. 6.4-left, with Φ given by a product of two Gaussian matrices. We emphasize that α FR,IT is in general not well-defined for an arbitrary channel, which is why we restricted here to the noiseless case. Note that in [START_REF] Afonso S Bandeira | Saving phase: Injectivity and stability for phase retrieval[END_REF] it was conjectured that for column-unitary matrices, α = 4 measurements were necessary to recover an arbitrary signal X . Here we have shown that a typical X can be perfectly recovered as soon as α = 2: such possible gaps between worst-case and typical performances are important to keep in mind in a large part of this thesis.

Surprising consequences and open questions

We list here some interesting (and often surprising) consequences of our analysis of the transitions. Since our rigorous results concern a subclass of orthogonally invariant matrices, proving and/or interpreting these statements more generally is an interesting future direction.

• One sees from eq. (6.11) that maximizing α WR,Algo implies maximizing λ 2 ν / λ 2 ν . The highest ratio is reached when ν is a delta distribution: for any symmetric channel and prior the ensemble that maximizes α WR,Algo is thus the one of uniformly-sampled column-orthogonal (β = 1) or column-unitary (β = 2) matrices. Conversely, α WR,Algo can be made arbitrarily small using a product of many Gaussian matrices, both in the real and complex cases.

• In complex noiseless phase retrieval the information-theoretic weak-recovery threshold for column-unitary matrices is located at α WR,IT = 2 [MDX + 21]. Our results (Table 6.1) imply that this corresponds to an "all-or-nothing" transition located precisely at α = 2. Moreover, our characterization of α WR,Algo and α FR,IT shows that for any complex matrix

α WR,Algo = 2 λ 2 ν / λ 2 ν ≤ α FR,IT = 2(1 -ν({0}))
, with the equality only being attained for ν a delta distribution. Uniformly sampled column-unitary matrices are thus the only right-unitarily invariant complex matrices which present an "all-or-nothing" transition in complex noiseless phase retrieval (for a Gaussian prior). To the best of our knowledge, this is a first establishment of such a transition in a "dense" problem (as opposed to a sparse setting [START_REF] Gamarnik | High dimensional linear regression with binary coefficients: Mean squared error and a phase transition[END_REF][START_REF] Reeves | All-or-nothing phenomena: From single-letter to high dimensions[END_REF]).

• Consider again noiseless phase retrieval with Gaussian prior. For real orthogonal matrices, one has α WR,Algo -α FR,IT = 1.5 -1 > 0: information-theoretic perfect recovery is achieved before algorithmic weak recovery! Since α WR,Algo is a smooth function of ν, we expect that the inequality holds for many real random matrix ensembles. However, in the complex case, by our previous point, α WR,Algo ≤ α FR,IT for all matrices: the gap thus only occurs in the real setting.

Efficient algorithms: constructing optimal spectral methods

Universality of the optimal method

In general, the different optimization methods used to solve phase retrieval (cf. Section 6.2.2) require an "informed" initialization X, i.e. that is positively correlated with the signal X * . The privileged class of algorithms to obtain such initializations in a computationally cheap manner is spectral methods, i.e. estimates given by the principal eigenvector of an appropriate matrix constructed from the sensing matrix Φ and the observations {y µ }.

In most previous approaches [MM19, LAL19, LL20, MDX + 21], the design of spectral methods for the phase retrieval problem was restricted to matrices of the type:

M(T ) ≡ 1 n m µ=1 T (y µ )Φ µi Φ µj . (6.15)
These matrices are functions of a (bounded) preprocessing function T . It was previously shown for Gaussian i.i.d. matrices Φ [LAL19, LL20] and for random column-unitary matrices Φ [DBMM20, MDX + 21] that the optimal transition and reconstruction errors in the class of spectral methods described by eq. (6.15) is attained by the following functions:

T * Gaussian (y) ≡ ∂ ω g out (y µ , 0, ρ) 1 + ρ∂ ω g out (y µ , 0, ρ) , T * Unitary (y) ≡ ∂ ω g out (y µ , 0, ρ/α) 1 + ρ α ∂ ω g out (y µ , 0, ρ/α) . (6.16)
In eq. ( 6. 16) we introduced the function g out , defined as8 :

g out (y µ , ω, σ 2 ) ≡ 1 σ 2 K dx e -β 2σ 2 |x-ω| 2 (x -ω) P out (y µ |x) K dx e -β 2σ 2 |x-ω| 2 P out (y µ |x) . (6.17)
The main result of this section is a conjecture that generalizes the above results and gives the optimal spectral method for any phase retrieval problem of the type of eq. ( 6.2) with rightrotation invariant Φ9 . Conjecture 6.5 (Optimal spectral method in phase retrieval)

For any right-rotationally (or unitarily) invariant matrix Φ (cf. Model R), the optimal (in terms of both weak-recovery transition and achieved reconstruction error) spectral method belongs to the class of eq. (6.15), and is attained by:

T * (y) ≡ ∂ ω g out (y µ , 0, ρ λ ν /α) 1 + ρ λ ν α ∂ ω g out (y µ , 0, ρ λ ν /α) .
Before detailing further our results, let us explicit two important consequences of Conjecture 6.5:

• Note that one can always assume the global scaling Tr[Φ † Φ]/n 2 → λ ν = α, as it can be absorbed into the channel P out10 . The optimal spectral method (in terms of weakrecovery threshold and achieved correlation) is then given by T * (y) = ∂ ω g out (y µ , 0, ρ)/(1 + ρ∂ ω g out (y µ , 0, ρ)). Remarkably, this optimal function does not depend on the spectrum of the sensing matrix Φ, nor on the sampling ratio α. The universality of the method is striking when one compares it to the replica result of Conjecture 6.1, which is heavily dependent on the spectrum of Φ and on the sampling ratio α. Universality also has deep consequences for phase retrieval practitioners: when using a spectral initialization, she/he does not have to take into account the details of the correlations in Φ to construct an optimal spectral method! Although this universality requires in theory right-rotation invariance, this assumption can be loosened as numerically explored in Section 6.5.

• Conjecture 6.5 claims optimality of our method among all spectral methods that one can construct from the data Φ and the observations {y µ }. It turns out that this optimal method belongs to the class of eq. (6.15), but our derivation is fully constructive and did not assume anything on the form of the spectral method. This is an important improvement with respect to the previous analyses we mentioned, which always assumed the method to be in the class of eq. (6.15). In this sense, this chapter also confirms the validity of this restriction.

Let us now briefly outline the remaining of this section. Its main purpose is to derive and unify three different approaches to construct optimal spectral methods for the phase retrieval problem. The first one, based on linearizing G-VAMP (Algorithm 6) is studied in Section 6.4.2. In Section 6.4.3 we consider a second approach, based on the Bethe Hessian and intimately connected with the TAP picture we derived in Chapter 2. A remarkable property of these two spectral methods is that their derivation is automated. Nevertheless, as we show in Section 6.4.4, for any channel distribution and sensing matrix Φ, the Bethe Hessian method coincides exactly with a third approach, which consists in simply generalizing a spectral method that has been proven to be optimal for Gaussian [START_REF] Luo | Optimal spectral initialization for signal recovery with applications to phase retrieval[END_REF] and unitary [START_REF] Dudeja | Information theoretic limits for phase retrieval with subsampled Haar sensing matrices[END_REF] sensing matrices, see eq. (6.16).

We moreover relate the performance of these three different approaches, and show that they allow to conjecture the optimal spectral method, summarized in Conjecture 6.5.

Linearized vector approximate message passing

This first approach arises as a linearization of G-VAMP (Algorithm 6). It is similar in essence to the technique we used in Chapter 5 to develop spectral methods that leveraged the structure of the signal. Here we apply this method to real and complex phase retrieval with a rightrotationally invariant sensing matrix.

The trivial fixed point

In Algorithm 6, one can use the Nishimori identity to derive the following relation (see for instance eq. ( 107) of [KKM + 16]):

1 m m µ=1 E Pout yµ, (R t 1 )µ τ t 1 , 1 τ t 1 z - (R t 1 ) µ τ t 1 2 = 1 τ t 1 . (6.18)
Informally, eq. (6.18) expresses that the estimated variance of z, defined as τ t 1 , is equal to the mean square difference between z and the estimation of z (being R t 1 /τ t 1 ) under the estimated posterior. Using eq. (6.18) along with the symmetry assumptions of Def. 6.1, it is easy to see that Algorithm 6 admits the following fixed point, that we call "trivial" as it is completely uninformative:

     γ 1 = 0, γ 2 = ρ -1 , v 1 = ρ, v 2 = ρ x1 = T 1 = 0, x2 = T 2 = 0, τ 1 = α/(ρ λ ν ), τ 2 = 0 c 1 = ρ λ ν /α, c 2 = ρ λ ν /α, ẑ1 = R 1 = 0, ẑ2 = R 2 = 0. (6.19)

Linearization around the fixed point

We can now linearize Algorithm 6 around the fixed point given by eq. (6. 19). One can easily check (see [START_REF] Maillard | Construction of optimal spectral methods in phase retrieval[END_REF] for details) that the first order variations of all the variances and inverse variances parameters are negligible. This will greatly simplify our linearization around the trivial fixed point, as we can focus solely on the vector parameters. For clarity, we restrict here to the real case β = 1, while the publication [MKLZ21] also considered the complex case. We write the linearization of Algorithm 6 as (all derivatives are taken at the fixed point of eq. (6.19)):

δx t 1 = ∇ T g x1 δT t 1 , δẑ t 1 = ∇ R g z1 δR t 1 , δT t 2 = 1 ρ δx t 1 -δT t 1 , δx t 2 = ∇ T g x2 δT t 2 + ∇ R g x2 δR t 2 , δẑ t 2 = ∇ T g z2 δT t 2 + ∇ R g z2 δR t 2 , δR t 2 = α ρ λ ν δẑ t 1 -δR t 1 , δT t+1 1 = 1 ρ δx t 2 -δT t 2 , δR t+1 1 = α ρ λ ν δẑ t 2 -δR t 2 .
(6.20)

The derivatives of the auxiliary functions of G-VAMP at the trivial fixed point are:

   ∂ T j [(g x1 ) i ] = ρ δ ij , ∂ T j [(g x2 ) i ] = ρ δ ij , ∂ Rν [(g z1 ) µ ] = δ µν E Pout yµ,0, ρ λ ν α [z 2 ], ∂ Rµ [(g x2 ) i ] = ρ (Φ † ) iµ √ n , ∂ T i [(g z2 ) µ ] = ρ Φ µi √ n , ∂ Rν [(g z2 ) µ ] = ρ (ΦΦ † )µν n . (6.21)
Plugging eq. (6.21) in eq. (6.20) yields, with v(y µ ) ≡ E

Pout yµ,0,

ρ λ ν α [z 2 ]:        δx t 1 = ρδT t 1 , δẑ t 1 = Diag({v(y µ )})δR t 1 , δT t 2 = 1 ρ δx t 1 -δT t 1 , δR t 2 = α ρ λ ν δẑ t 1 -δR t 1 , δx t 2 = ρδT t 2 + ρ Φ † √ n δR t 2 , δẑ t 2 = ρ Φ √ n δT t 2 + ρ ΦΦ † n δR t 2 , δT t+1 1 = 1 ρ δx t 2 -δT t 2 , δR t+1 1 = α ρ λ ν δẑ t 2 -δR t 2 .
(6.22)

In particular, these equations imply δT t 2 = 0. The equations can then simply be closed on δR t 1 :

δR t+1 1 = α λ ν ΦΦ † n -I m α ρ λ ν Diag({v(y µ )}) -I m δR t 1 .
(6.23)

In the complex case, one obtains the same linearized equation. Interestingly, v(y µ ) can be linked to the function ∂ ω g out of eq. (6.17):

∂ ω g out (y µ , 0, σ 2 ) = -σ -2 + σ -4 v(y µ ).

The LAMP spectral method

The Linearized-AMP (LAMP) spectral method is based on eq. (6.23), and consists in taking the dominant eigenvalue and corresponding eigenvector of the m × m matrix:

M (LAMP) ≡ ρ λ ν α α λ ν ΦΦ † n -I m Diag(∂ ω g out (y µ , 0, ρ λ ν /α)).
Note that M (LAMP) is not a Hermitian matrix, so "dominant" eigenvalue means here eigenvalue of largest real part. If û is the eigenvector of M (LAMP) associated to this dominant eigenvalue, then one can construct a corresponding estimate x using the relations of eq. (6.22), as:

x ≡

Φ † α ρ λ ν Diag({v(y µ )}) -I m û Φ † α ρ λ ν Diag({v(y µ )}) -I m û √ nρ. (6.24)
Surprisingly, and as we will see in more details in the following, this spectral method achieves the optimal weak-recovery threshold but only sub-optimal performance compared to M(T * )! There is, however, a way to recover the optimal performance from M (LAMP) by considering an eigenvalue "hidden" inside the bulk, as we will see in Proposition 6.6.

The Bethe Hessian: TAP revisited

Our second approach leverages the Thouless-Anderson-Palmer (TAP) formalism of statistical physics [START_REF] David J Thouless | Solution of 'solvable model of a spin glass[END_REF], and in particular the results of Chapter 2.

Let us first recall a few facts on the TAP formalism. It consists in studying a modified posterior distribution with respect to eq. (6.3), which is tilted so that the first and second moments of all x i are fixed. These moments become then variables of the TAP free energy associated with this modified posterior distribution. When weak recovery of the signal is impossible, this free energy possesses a global minimum in the completely uninformative point in which the estimator is the vector m = 0. On the other hand, when weak recovery is possible, the optimal estimator corresponds to the global minimum of the TAP free energy with m = 0. However as we will see the point m = 0 always remains a stationary point of the TAP free energy. Moreover, a spectral method used for initializing a non-convex optimization algorithm can be based solely on the observations (i.e. on Φ and {y µ }), and therefore can not exploit any physical information other than the one present in the uninformative point. When this point is locally stable, we therefore expect all polynomial-time algorithms not to be able to achieve weak recovery. On the other hand, linear instability of the m = 0 point implies that there should exist a minimum of the TAP free entropy with positive correlation with the signal. With this picture in mind, it is natural to conjecture that the optimal spectral estimator is the dominant unstable direction of the uninformative fixed point, i.e. the smallest eigenvalue of the Hessian at the uninformative fixed point, also denoted Bethe Hessian 11 .

The TAP free entropy

We denote f TAP (Y, Φ, m, σ) the TAP free entropy (or negative free energy) of the model of eq. (6.2). Of particular interest are its maxima, corresponding to pure states in the statistical physics language. Recall that we showed in Chapter 2 that these pure states are in exact correspondence with the fixed points of the G-VAMP algorithm. Let us now use the results of Chapter 2 to compute f TAP , more specifically eq. (2.39), which yields up to O n (1) terms:

f TAP (m) = sup σ≥0 sup g∈K m r≥0 extr ω∈K m b≥0 extr λ∈K n γ≥0 β n n i=1 λ i • m i + βγ 2n nσ 2 + n i=1 |m i | 2 - β n m µ=1 ω µ • g µ - βb 2n m µ=1 |g µ | 2 -αnr + 1 n n i=1 ln K P 0 (dx)e -βγ 2 |x| 2 -βλ i •x (6.25) + α m m µ=1 ln K dh 2πb β β/2 P out (y µ |h)e -β|h-ωµ| 2 2b + β n n i=1 m µ=1 g µ • Φ µi √ n m i + βF (σ 2 , r) .
The function F is defined as:

F (x, y) ≡ inf ζx,ζy>0 ζ x x 2 + αζ y y 2 - α -1 2 ln ζ y - 1 2 ln(ζ x ζ y + λ) ν - 1 2 ln x - α 2 ln y - 1 + α 2 .
One can write the saddle-point equations associated to eq. (6.25), called the TAP equations:

m i = E P 0 (γ,λ i ) [x], σ 2 = 1 n n i=1 E P 0 (γ,λ i ) [|x -m i | 2 ], g µ = g out (y µ , ω µ , b), r = 1 m m µ=1 |g µ | 2 + 1 b -E Pout(yµ,ωµ,b) h-ωµ b 2 , ω µ + bg µ = n i=1 Φ µi √ n m i , b = -2 α ∂ r F (σ 2 , r), γ = -2∂ σ 2 F (σ 2 , r).
(6.26)

The trivial stationary point

It is easy to see that the TAP equations (6.26) admit a trivial fixed point at m = 0 (corresponding to a stationary point of f TAP ). At this point, the parameters are

σ 2 = ρ, g = ω = λ = 0, γ = r = 0, b = ρ λ ν /α.
The derivation of the fixed point also uses the behavior of F (σ 2 , r) at small r, computed in Appendix D.6.1:

F (σ 2 , r) = - λ ν rσ 2 2 + σ 4 r 2 4α [α λ 2 ν -(1 + α) λ 2 ν ] + σ 6 r 3 G(rσ 2 ), (6.27)
with G(x) a continuous bounded function in x = 0.

The spectral method

As we argued, a natural way to design the optimal spectral method for this inference problem is to consider the Hessian of -f TAP at this trivial fixed point, as we expect a descending informative direction to appear in its spectrum at the weak recovery threshold. Computing the Hessian from eq. (6.25) is quite lengthy, but poses no technical difficulties. For this reason we do not reproduce its derivation here (it can be found in [START_REF] Maillard | Construction of optimal spectral methods in phase retrieval[END_REF]). In the end this procedure leads to consider the n × n matrix:

M (TAP) ≡ - n β ∇ 2 f TAP (0) = - 1 ρ I n + 1 n m µ=1 ∂ ω g out (y µ , 0, ρ λ ν /α) 1 + ρ λ ν α ∂ ω g out (y µ , 0, ρ λ ν /α)
Φ µi Φ µj . (6.28)

Unifying the approaches

We detail here our main analytical results concerning the spectral methods we just derived.

The optimal spectral method and the Bethe Hessian

Very importantly, as opposed to previous approaches, our derivation is constructive: we start from the fully-explicit expression of the TAP free entropy given in eq. ( 6.25) and simply compute its Hessian at the trivial fixed point. As we argued, we expect from the statistical physics literature that the optimal spectral method will be given by the largest eigenvalue (and associated eigenvector) of this Hessian, given in eq. (6.28). Importantly, this implies that the optimal spectral method that can be built from the data Φ and the observations {y µ } belongs to the class of methods given by eq. (6.15). Our conjecture therefore also gives weight to many previous analysis of spectral methods for phase retrieval, which only studied spectral methods of the type of eq. (6.15) [MM19, LAL19, LL20, MDX + 21].

Relating linearized-AMP and the Bethe Hessian

Our derivation of M (LAMP) is constructive as well, and in this sense fundamentally differs from the L-AMP algorithms designed in [MDX + 21] to assess the performance of other spectral methods. We can moreover relate the eigenpairs of our two methods:

Proposition 6.6 (Relating LAMP to the Bethe Hessian)

Without loss of generality, we assume λ ν = α. Let z µ ≡ ∂ ω g out (y µ , 0, ρ λ ν /α), and (λ LAMP , v) be an eigenpair of M (LAMP) . Assume that λ LAMP + ρz µ = 0 for all µ = 1, • • • , m.

Then Φ † Diag(z µ )v = 0, and we let x ∝ Φ † Diag(z µ )v with x 2 = n. Moreover:

1 m m µ=1 ρz µ λ LAMP + ρz µ Φ µ Φ † µ x = x.
Conversely, let x be an eigenvector of M (TAP) with norm x 2 = n, with associated eigenvalue λ TAP . We define u ≡ Diag[(1 + ρz µ ) -1 ]Φx/ √ n. Then one has:

M (LAMP) u = u + ρλ TAP Diag(1 + ρ∂ ω g out (y µ , 0, ρ))u.
Moreover, if λ TAP = 0, eq. (6.24) applied to u yields the same performance as the TAP estimator.

This proposition is proven in Appendix D.6.2. By considering λ LAMP = 1 and λ TAP = 0, one immediately deduces two important consequences of Proposition 6.6:

• The appearance of an unstable direction, in the spectrum of M (TAP) (i.e. a positive eigenvalue) and of M (LAMP) (i.e. an eigenvalue with real part greater than 1), occurs at a common threshold, the weak-recovery threshold, given by eq. (6.11).

• An eigenvalue 0 appears in the spectrum of M (TAP) if and only if an eigenvalue 1 appears in the spectrum of M (LAMP) . These two eigenvalues therefore correspond to marginal stability of the linear dynamics. Moreover, the two estimators associated to these eigenvalues are identical, i.e. M (LAMP) contains the optimal estimator. However, this estimator is different from the dominant eigenvector of M (LAMP) , which reaches only suboptimal performance as we will see in Section 6.5.

Statistical and algorithmic analysis: numerical experiments

This section is devoted to a numerical investigation of all the theoretical claims of the previous sections. In Section 6.5.1 we solve the state evolution equations (6.10) for different real and complex ensembles of sensing matrix Φ, and compare it to numerical simulations of G-VAMP. This allows to uncover the existence or absence of hardness depending on the structure of the sensing matrix. In the following Section 6.5.2 we give the performance of the spectral methods we derived in Section 6.4, for noiseless and Poisson-noise observations. Finally, in Section 6.5.3 we apply both message-passing and spectral algorithms to the recovery of real images, illustrating the relevance of our analysis for practical phase retrieval.

We also show in all these cases that the algorithms we present perform very well even by allowing more structure in the sensing matrix than assumed in Model R, by considering for example randomly subsampled DFT, Hadamard or DCT matrices12 .

Optimal algorithms and computational gaps

While our results hold for any phase retrieval problem (in the sense of Def. 6.1), we focus for the analysis of computational gaps on noiseless phase retrieval. We fix P out (y|z) = δ(y -|z| 2 ) and take P 0 = N β (0, 1). We can indeed consider ρ = 1, as the scaling is irrelevant under a noiseless channel. The numerical code used to generate all simulations of the G-VAMP algorithm is available in a Github repository [MLKZ20].

Real case

In Fig. 6.2, we illustrate the case of real Gaussian and real column-orthogonal sensing matrix Φ, the latter not having been investigated previously in the literature. We compute the MMSE by solving the State Evolution equations starting from an informed solution (close to full recovery).

The minimal mean-squared error achievable with the G-VAMP algorithm is computed using the State Evolution equations starting from the uninformed q z = 0 solution. We compare these predictions with numerical simulations of the G-VAMP algorithm on Gaussian matrices and uniformly sampled orthogonal matrices, as well as randomly subsampled Hadamard matrices. The simulations are in very good agreement with the prediction, and our results on Hadamard matrices suggest that the curves of Fig. 6.5-right are valid for more general ensembles than uniformly sampled orthogonal matrices, and that one can allow some controlled structure in the matrix without harming the performance of the algorithm.

Complex case

Previous works on complex signals X * ∈ C n have mainly focused on the study of the weak recovery threshold α WR , which was located for Gaussian matrices [MM19, LAL19] and uniformly sampled column-unitary matrices [START_REF] Ma | Orthogonal AMP[END_REF][START_REF] Dudeja | Analysis of spectral methods for phase retrieval with random orthogonal matrices[END_REF]. We begin by extending the aforementioned results by identifying the full recovery threshold α FR,IT in these cases, and comparing the performance of the G-VAMP algorithm to the Bayes-optimal solution. Fig. 6.3 illustrates our results for these two ensembles. The algorithmic (i.e. G-VAMP) full-recovery threshold α FR,Algo is found numerically from the state evolution equations and is in good agreement with finite size simulations. The existence of a statistical-to-algorithmic gap ∆ = α FR,Algo -α FR,IT ≥ 0 reflects the hardness of phase retrieval in the real and complex case. However, it is interesting to note that even though full-recovery in the complex case requires more data than in the real case, the size of the statistical-to-algorithmic gap in the complex ensembles is smaller than in their real counterparts. W 1 W 2 , with W 1 ∈ C m×p and W 2 ∈ C p×n for different aspect ratios γ ≡ p/n. We can identify the presence of a threshold α WR,Algo = γ/(1 + γ) (computed in Section 6.3.1) that delimits the possibility of weak recovery both information-theoretically and in polynomial time.

The information-theoretic full-recovery is achieved at α FR,IT = min(2, 2γ), in agreement with eq. (6.14). Note that the full recovery algorithmic threshold is very close to the informationtheoretic one, and precisely equal for γ = 1, although the gap is too small to be visible in the left and right parts of Fig. 6.4. Therefore, the performance of G-VAMP is exactly given by the Bayes-optimal estimator, apart for γ = 1 in a very small range (α FR,IT , α FR,Algo ), whose size is of order 10 -3 for γ ∈ {0.5, 1.5}. As γ → ∞, one recovers the statistical-to-algorithmic gap present in the complex Gaussian case, which is again very small (around 0.027, cf Table 6.1).

Although this hard phase is very small, our results show its existence for all γ = 1.

Spectral methods: cheap and efficient

In this section, we numerically assess our predictions and compare the performances of the spectral methods on various problems. The numerical code used to generate all figures related to the spectral methods is available in a Github repository [START_REF] Maillard | Demonstration codes and notebooks[END_REF].

Another spectral method -In the figures, we sometimes consider another spectral method, denoted M (MM) . It is obtained by naively considering the preprocessing function of [START_REF] Mondelli | Fundamental limits of weak recovery with applications to phase retrieval[END_REF], which was shown to achieve the optimal transition for Gaussian sensing matrices. More precisely, we have (assuming ρ = 1 and λ ν = α):

T MM (y) ≡ ∂ ω g out (y, 0, 1) 2α β + ∂ ω g out (y, 0 , 1)
.

In particular note that at α = β/2, we have T MM = T * , so that T MM indeed achieves the optimal weak-recovery transition for Gaussian matrices, for which α WR,Algo = β/2.

Performance of the spectral methods -We show the performance of the spectral methods to recover a random signal in three different cases, that we briefly describe:

• In Fig. 6.5, we consider noiseless real phase retrieval (i.e. sign retrieval), with uniformly sampled column-unitary sensing matrices. We also show that our conclusions transfer to randomly subsampled Hadamard matrices, validating the conclusions of [START_REF] Dudeja | Universality of linearized message passing for phase retrieval with structured sensing matrices[END_REF]. • In Fig. 6.6a we consider noiseless real phase retrieval when the sensing matrix is a product of two Gaussian i.i.d. matrices. This setup can for instance be interpreted as Gaussian phase retrieval in which the signal is drawn from a known generative prior, similarly to what we analyzed in Chapter 5. Importantly, it is not covered by any previous analysis of the spectral methods, emphasizing the generality of Conjecture 6.5.

• In Fig. 6.6b, we compare our results in noiseless and noisy settings. More precisely, we consider complex phase retrieval with a Gaussian sensing matrix, and either a noiseless channel or a Poisson observation channel with intensity Λ > 0:

P out (y|z) = e -Λ|z| 2 ∞ k=0 δ(y -k) Λ k |z| 2k k! .
This latter channel is particularly relevant for optical applications, in which the detector can be modeled as being affected by a Poisson noise. In both cases, we find that all our conclusions on the optimality of the M (TAP) , and on the link between M (LAMP) and M (TAP) , still hold.

Transition phenomena in the spectra -We illustrate the weak-recovery transition in the spectra of the different methods. Precisely, we confirm the following claims of Section 6.4.4:

• Both M (LAMP) and M (TAP) have a dominant eigenvalue that detaches from the bulk for α > α WR,Algo , given by eq. (6.11).

• In the regime in which weak-recovery is possible, the largest eigenvalue of M (TAP) approaches 0 as n → ∞. The associated eigenvector achieves optimal correlation with the signal (among spectral methods) as n → ∞.

• M (LAMP) gives two estimators that are positively correlated with the signal for α > α WR,Algo . The first one corresponds to its largest eigenvalue in real part, and achieves worse correlation than the largest eigenvector of M (TAP) . The second one corresponds to an eigenvalue inside the bulk (but isolated from the other eigenvalues) that approaches 1 as n → ∞, and achieves the same optimal performance as the estimator given by M (TAP) .

We verify these claims for different values of α, below and above the weak-recovery threshold, in complex Gaussian phase retrieval with Poisson-noise, in Fig. 6.7. This analysis is extended in [MKLZ21], which contains similar results for other observation channels and sensing matrices.

Computational cost of the spectral methods -When weak recovery is possible the largest eigenvalue of M (TAP) concentrates on 0 as we noticed. However, the spectrum of M (TAP) also contains many very large negative eigenvalues. In practice, we use an inverse iteration method to quickly estimate the associated eigenvector. We use a similar approach for M (LAMP) , using inverse iterations to estimate the eigenvector with eigenvalue 1, and usual power iterations for the largest eigenvalue.

A puzzling open question -As we already said, the optimal estimator is always associated with marginal stability, both in M (LAMP) and M (TAP) . A clear understanding of this marginal stability is still lacking. Moreover, the dominant eigenvector of the matrix M (LAMP) is associated to an unstable direction, thus dominating the dynamics of the linearized-AMP. However its achieved correlation is smaller than the one achieved by the marginally stable, optimal, eigenvector. We also noticed that the eigenvectors of M (TAP) do not contain any information about this suboptimal estimator13 . This blindness of M (TAP) to the principal eigenvector of M (LAMP) is very puzzling from a theoretical point of view. Indeed, as we showed in Chapter 2, the stationary limit of G-VAMP (Algorithm 6) is in exact correspondence with the stationary point equations of the TAP free entropy. One would therefore expect the two spectral methods M (LAMP) and M (TAP) to contain the same physical information on the system. Moreover, the different qualitative behaviors of the two dynamics (instability of M (LAMP) a opposed to marginal stability of M (TAP) ) only deepens this puzzle. Original image = 2.3, MSE = 0.0 = 2.2, MSE = 0.639 = 2.1, MSE = 0.81 = 2.0, MSE = 0.999 Figure 6.8: Performance of the G-VAMP algorithm for noiseless phase retrieval. We wish to recover a 77x102 image (on the left), and we use a complex Gaussian prior to infer the signal. The data matrix Φ is a randomly subsampled DFT matrix.

Real image reconstruction

Importantly, while the knowledge of the distribution of the true signal is required for our theoretical analysis, the G-VAMP algorithm and our spectral methods are also well-defined beyond this scope, e.g. they can be used to infer natural images with Fourier matrices. Using a Gaussian prior to infer the image can then actually be seen as the minimal assumption on the underlying signal, as it amounts to simply fix its norm: in particular, our theory can thus predict the performance of G-VAMP for any signal, structured or not.

Performance of G-VAMP

We conducted a simple experiment on a natural image with a randomly subsampled DFT matrix Φ, described in Fig. 6.8. Although we are far from a Bayes-optimal setting, the achieved MSE is very close to values of Fig. 6.3, for all values of α. In particular, we achieve perfect recovery for α ≥ 2.3, just above α FR,Algo 2.27 which was derived for random unitary matrices, i.i.d. data and in the Bayes-optimal setting!

Spectral initialization on real images

As a final analysis, we numerically investigate our spectral methods for the reconstruction of a natural image. For comparability, we consider the image of The Birth of Venus already used in [MM19, MDX + 21]. Although this signal is not i.i.d., we will see that all the conclusions that we numerically investigated in Section 6.5.2 transfer to this case. We consider a noiseless Figure 6.9: MSE achieved by the different spectral methods for the recovery of a natural image in noiseless phase retrieval. We consider column-unitary matrices Φ (both uniformly sampled and partial DFT matrices, left) and the product of two complex Gaussian matrices with aspect ratio γ = 1 (right). We reduced each dimension of the original 1280 × 820 image by a factor 20 (left) or 10 (right), and we average the MSE over 5 instances and the 3 RGB channels.

phase retrieval channel and different sensing matrices Φ: multiple ensembles of column-unitary matrices (which partly reproduces the analysis of [MDX + 21]) and a product of two complex Gaussian matrices with aspect ratio γ = 1. In Fig. 6.9, we give the MSE obtained by the different spectral methods and these two matrix ensembles. We also give examples of the images recovered by the algorithms. Moreover, despite the fact that the signal (and possibly the matrix as well) is structured, we still observe the same transition phenomena in the spectra of M (TAP) and M (LAMP) . Namely, we still observe that the optimal estimator is associated with marginal stability of both spectral methods, while the dominant eigenvalue of M (LAMP) is associated to a non-optimal estimator.

Let us finally illustrate how the optimal spectral method of Conjecture 6.5 can be combined with a subsequent local optimization algorithm. We use the spectral estimator as the initialization point to running vanilla gradient descent on the square loss

L(x) ≡ 1 2m m µ=1 (Φx) µ √ n 2 - (ΦX * ) µ √ n 2 2
. This allows to already obtain a perfect recovery of the image for α = 4, as shown in Fig. 6.10. In [START_REF] Maillard | Construction of optimal spectral methods in phase retrieval[END_REF] we describe the obtained MSE curves in more details. In particular, we confirm that combining the gradient descent with the spectral initialization allows to reach perfect recovery at finite α, which is not possible with the "vanilla" spectral methods.

Conclusion of Chapter 6 and perspectives on Part II

In this last well-filled chapter on the statistical physics approach to optimal learning in inference models we considered phase retrieval, one of the flagship models of theoretical computer science, with a wide variety of applications across scientific fields (notably through optical experiments).

Our main findings can be briefly summarized as:

• As in Chapters 4 and 5, we leverage the replica method to derive a single-letter formula for the asymptotic free entropy (or mutual information) of the system, which we then prove under assumptions on the structure of the data matrix. Combined with message-passing algorithms, which are known to be optimal among a large class of general first order methods [CMW20], we uncovered the existence or absence of statistical-to-computational gaps in phase retrieval depending on the structure of the sensing matrix. In particular, for the case of Fourier phase retrieval, which is of particular interest to the practitioner, we uncover a significant hard phase, while the IT performance shows an "all-or-nothing" transition, see Fig. 6.3. We also conducted a precise theoretical analysis of the weak and full recovery thresholds, as a function of the sampling ratio, in the wide class of phase retrieval problems characterized by Def. 6.1.

Recall that these results are summarized in Table 6.1.

• We presented an automated derivation of a generic spectral method, which we conjecture to be optimal for the wide class of phase retrieval problems that satisfy Def. 6.1, cf. Conjecture 6.5. Our derivation leverages the Bethe Hessian of the TAP free energy computed in Chapter 2, and in particular, we show that our method achieves the optimal weak-recovery threshold, and give numerical evidence for its optimality in terms of MSE.

• Finally, we gave numerical evidence for two generalizations of our results, regarding both optimal learning and the spectral methods: first, the i.i.d. nature of the signal is not required for our results to hold, as we show by recovering real structured images, see Figs. 6.8 and 6.9. Second, one can allow more structure in the data matrix than rotational invariance, as illustrated in Figs. 6.2, 6.3, 6.8 and 6.9. In particular, we can consider randomly subsampled Fourier matrices without harming any of our conclusions. Note that this random subsampling is necessary, as we found that our conclusions did not transfer to "vanilla" Fourier matrices.

Proofs of replica formulas for inference: beyond Gaussianity -As we emphasized, the adaptive interpolation method used at several points in Part II, while being highly versatile and applied to a wide range of problems [BM19a, BM19b, BKM + 19, Bar19, BR20, BMDK20], [START_REF] Barbier | The mutual information in random linear estimation beyond iid matrices[END_REF], deeply relies on some Gaussianity in the problem, either in the sensing matrix or the prior. Recent works by collaborators have taken a different path, leveraging known results on the asymptotic performance of approximate message-passing algorithms, which allows to indirectly prove the replica predictions [START_REF] Gerbelot | Asymptotic errors for highdimensional convex penalized linear regression beyond Gaussian matrices[END_REF][START_REF] Gerbelot | Asymptotic errors for teacher-student convex generalized linear models (or: How to prove Kabashima's replica formula[END_REF]. This has allowed to put on rigorous ground formulas for matrices without any Gaussianity (i.e. the generic rotationallyinvariant matrices described by Models S and R). Still these approaches also have limitations, e.g. requiring convexity of the loss, and an improvement in this regard, or a way to loosen the Gaussianity assumptions in adaptive interpolation, would both be significant advancements in putting the replica predictions in Bayes-optimal problems on rigorous grounds.

Additional references -Let us conclude by naming some references for the reader interested in the ideas behind the whole Part II, and who wishes to know more. First and foremost, [START_REF] Mézard | Information, physics, and computation[END_REF] is an important introduction to the field of statistical physics applied to theoretical computer science. It details many of the concepts that we introduced in this thesis, e.g. message-passing algorithms, the cavity method or the appearance of replica symmetry breaking in inference or constraint satisfaction problems. With a point of view more focused on statistical gaps in inference problems, and close to the one we took in this thesis, [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF] gives a very detailed description of how to apply the statistical physics toolbox to learning. The authors focus on important examples particularly relevant to the theoretical computer science community, such as compressed sensing and community detection. In the same vein, [START_REF] Afonso S Bandeira | Notes on computationalto-statistical gaps: Predictions using statistical physics[END_REF] reviews some of the statistical physics prediction on the existence of hard phases, and offers a very good introduction to the field for the mathematics and theoretical computer science audience. Finally, closer to the physics perspective, [START_REF] Gabrié | Mean-field inference methods for neural networks[END_REF] is a recent review on the mean-field approaches for learning in neural networks, and highlights many very recent advances and perspectives. Beyond these reviews, the reader will find many recent published works taking similar approaches (e.g. [START_REF] Barbier | The mutual information in random linear estimation beyond iid matrices[END_REF] by the author), that one can regroup under the broad category of "statistical physics of learning".

Chapter 7

The complexity of high-dimensional landscapes "As the light grew a little he saw to his surprise that what from a distance had seemed wide and featureless flats were in fact all broken and tumbled. Indeed the whole surface of the plains of Gorgoroth was pocked with great holes, as if, while it was still a waste of soft mud, it had been smitten with a shower of bolts and huge slingstones. The largest of these holes were rimmed with ridges of broken rock, and broad fissures ran out from them in all directions." J.R.R. Tolkien, The Lord of the Rings -Book VI (1955).

Disclaimer -In this chapter we present a method to obtain the average and the typical value of the number of critical points of the empirical risk landscape for generalized linear estimation problems and variants. This represents a substantial extension of previous computations which restricted to Gaussian random functions. The new results in this chapter are mainly based on [START_REF] Maillard | Landscape complexity for the empirical risk of generalized linear models[END_REF], and the introductory Section 7.1 builds on a presentation given at the Kavli Institute for Theoretical Physics during the winter of 2019 [START_REF] Maillard | An introduction to the Kac-Rice formula[END_REF]. Some proofs and details are given in Appendix E. On a general note, the majority of this chapter is written in a style closer to the mathematics literature than the other chapters in this thesis. It aims however at being accessible to any reader from the statistical physics community.

Counting complexity: the Kac-Rice formula

How to "count" the complexity of a landscape?

Characterizing the landscape of the empirical risk is a key issue in several contexts. Indeed, many current machine learning problems (such as the ones we encountered in the previous chapters) are both non-convex and high-dimensional. In these cases the analysis of local optimization algorithms, such as gradient descent and its stochastic variants, represents a very hard feat. In recent years, there has been a series of works that developed a landscape-based approach to tackle this challenge. The key idea is to study the statistical properties of the empirical risk landscape, and to use these findings to obtain results on the performance of algorithms. Without the aim of being exhaustive this research avenue includes analysis of the landscape of neural networks, matrix completion, tensor factorization and tensor principal component analysis (PCA) [Fyo04, FN12, Kaw16, SC16, GLM16, FB17, BNS16, PKCS17, DLT + 18, GM17, GJZ17, LK17, LXB19, BAMMN19, RBABC19, SMKUZ19, SMBC + 19, BCRT20, BAGJ20]. The majority of these works identifies the region of parameters where the landscape is "easy", i.e. it focuses on the regime where there shouldn't be any bad local minima and it proves that indeed there are none. However, gradient descent and other landscape-based algorithms are used extensively in machine learning, and they are often observed to work even very far from the region described above where the landscape can be proved mathematically to be "easy". A possible reason is that the bounds obtained rigorously are not tight enough. Another, more interesting, is that the landscape is "hard", i.e. spurious minima are present, but their basins of attraction are small and the dynamics is able to avoid them [SMKUZ19, SMBC + 19].

In Chapters 4 and 6, we uncovered large hard phases in the learning of two-layers neural networks and in the phase retrieval problem. This led us to the conclusion that, in the loss landscapes of such models, there must exist a large number of spurious local minima (i.e. local minima with loss value far from the global minimum), so that local optimization procedures (as well as all polynomial-time algorithms) fail whenever the system is in a hard phase. The existence of many such local minima is what we will generically describe as the complexity of a high-dimensional landscape.

In Fig. 7.1 we describe the intuitive view of complexity of a landscape. Crucially, one must keep in mind that such a picture can be very misleading, as almost all our usual low-dimensional intuitions become incorrect when entering the realm of high dimensions1 .

Counting critical points -In this chapter we develop a general method that allows to study the "hard" regime, where the loss landscape may display a huge number of bad minima. Our aim is to obtain explicit formulas for the number of critical points of the empirical risk landscape, and to characterize the Hessian associated to them. For a given problem, this will allow to identify the topological transition where the landscape becomes "easy", and to analyze very precisely the "hard" regime by investigating the topology of its sublevel sets. In recent years, there has been remarkable progress on this subject in the field of spin glasses and probability theory [FSW07, FW07, BD07, ABA13, ABAČ13, Sub17a, Sub17b, SZ17, BAMMN19, RBABC19, BASZ20] (this list being far from exhaustive, and we will refer to many other works adopting similar or complementary approaches in specific paragraphs of this chapter). This line of research has allowed to put on a firm ground results previously obtained in the physics literature [BM80, Kur91, CS95, CGG99, Fyo04], and it has unveiled important relationships with random matrix theory. Its main domain of application has been the study of the landscapes associated to Gaussian random functions, and we will describe one of these analyses in Section 7.1.4. Its extension to tackle non-Gaussian high-dimensional random functions is an open problem -one that is crucial to address in order to characterize the landscape of the empirical risk.

Annealed and quenched complexity -One of the earliest and most important insights of the physics analysis of spin glasses is that for random smooth functions in high dimensions, we should expect the number of critical points to typically be in the scale e Θ(n) [START_REF] Bray | Broken replica symmetry and metastable states in spin glasses[END_REF][START_REF] Monasson | Structural glass transition and the entropy of the metastable states[END_REF].

Therefore one must be particularly careful as how to consider its asymptotic value, as the number of critical points may be highly fluctuating. In Section 1.1 we introduced two generic asymptotic limits for such random variables, that we denoted as annealed and quenched, borrowing from the physics jargon. We refer the reader to this section for a more precise description of these two limits. We therefore define the annealed and quenched complexity of a random function f n as:

       Σ annealed ≡ lim n→∞ 1 n ln E Crit(f n ) Σ quenched ≡ lim n→∞ 1 n E ln Crit(f n ) , ( 7.1)
Crit(f n ) being the number of critical points of f n . In the following we will generalize this definition, to count e.g. the local minima, or critical points with given value of the function.

Usually the annealed complexity is easier to compute than its quenched counterpart. However, since Crit(f n ) is in general exponentially large and strongly fluctuating, the quenched complexity really describes the typical value of the number of critical points, and is truly representative of the typical properties of the landscape, while the annealed complexity might be dominated by rare instances, as e.g. in tensor PCA [START_REF] Ben Arous | The landscape of the spiked tensor model[END_REF][START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF]. Annealed and quenched complexities are therefore usually different, with very few exceptions [START_REF] Crisanti | Thouless-Anderson-Palmer approach to the spherical p-spin spin glass model[END_REF][START_REF] Subag | The complexity of spherical p-spin models-a second moment approach[END_REF], one of them being the pure p-spin model that we will see in detail in Section 7.1.4.

Our main tool to count the critical points of random functions will be the Kac-Rice formula [START_REF] Stephen O Rice | Mathematical analysis of random noise[END_REF]. In the following Sections 7.1.2 and 7.1.3 we describe its derivation and the intuition behind it, before describing a step-by-step first application to a spin glass model in Section 7.1.4.

The area formula

The Kac-Rice formula is intuitively not derived from any involved probabilistic tool, but rather is a consequence of a purely geometric result, called the area formula (itself a consequence of the co-area formula), described in great generality by Federer [START_REF] Federer | Curvature measures[END_REF], and stated for instance in [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF]. This formula is the generalization of the following non-rigorous intuition: for a smooth function f : R → R, and T ⊆ R, denoting N f (u, T ) the number of solutions to the equation f (t) = u with t ∈ T , one would want to write informally:

N f (u, T ) = f (T ) δ(v -u)dv = T δ(f (t) -u)|f (t)|dt. (7.2)
The area formula generalizes and makes rigorous this intuition, by showing this last equality in the weak sense. We follow here the statement of [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF].

Proposition 7.1 (Area formula, from [AW09b])

Let f : U → R d be a C 1 function defined on an open subset U of R d .
Assume that the set of critical values2 of f has zero Lebesgue measure, and denote N f (u, T ) the number of solutions to the equation f (t) = u with t ∈ T . Then, for any Borel set T ⊆ R d , and any g : R d → R continuous and bounded:

R d g(u)N f (u, T ) = T | det f (t)| g(f (t)) dt.
Note that in a large part of the theoretical physics literature, the area formula (and subsequently the Kac-Rice formula) is directly written in the form of eq. (7.2).

The Kac-Rice formula

Consider a smooth compact manifold M of dimension n (a practical example is the n-dimensional unit sphere S n-1 ), equipped with a Riemannian metric, and an associated volume measure µ M . We are given a random smooth function f : M → R and we want to use the area formula (Prop. 7.1) to estimate the moments of the number of critical points of f , in order to compute the complexity defined by eq. (7.1).

Given the assumptions of Proposition 7.1, a reasonable hypothesis is to assume that f is almost surely a Morse function [START_REF] Willard Milnor | Morse theory[END_REF], i.e. that all its critical points are non-degenerate. Since M is compact, one easily deduces that the number of critical points of a Morse function is finite3 . For any k ∈ N and Borel set B ⊆ R, we define Crit f,k (B) to be the number of critical points x ∈ M of f such that f (x) ∈ B and such that the index of Hess f (x) (that is the number of strictly negative eigenvalues of the Hessian) is equal to k. The informal area formula of eq. (7.2) applied to grad f reads:

Crit f,k (B) = M dµ M (x)δ(grad f (x))| det Hess f (x)| 1[f (x) ∈ B, i(Hess f (x)) = k]
Taking the expectation of this equality, one directly obtains the Kac-Rice formula:

Proposition 7.2 (Kac-Rice formula, informal)

Let M be a smooth compact Riemannian manifold of dimension n, with volume measure µ M . Let f : M → R be a random function that is almost surely Morse, and that satisfies some regularity properties (see the remark below). Denote ϕ(0) the density of grad f (x) with respect to the Lebesgue measure on R n-1 , taken at 0. Then:

E Crit f,k (B) = M dµ M (x)E | det Hess f (x)| 1{f (x) ∈ B, i Hess f (x) = k} grad f (x) = 0 ϕ(0).
Let us make a few important remarks on this formula:

1. The rigorous derivation of the Kac-Rice formula is quite involved, as one has to start from the weak equality of Proposition 7.1 and to use continuity arguments in order to obtain a strong equality at u = 0. This proof is detailed e.g. in [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF]. This is the first reason for which the formula has mostly been used for Gaussian random fields, since then continuity arguments can be justified using simple hypotheses on the covariance of the random field. The second reason is that in general, conditional expectations of non-Gaussian random variables are intractable, making the Kac-Rice formula effectively useless since one has to know the law of the Hessian conditioned by the gradient being zero. Under many heavy technical conditions, one can however derive rigorous non-Gaussian versions of the Kac-Rice formula (see for instance [START_REF] Adler | Random Fields and Geometry[END_REF][START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF]). We will apply such sophisticated results in Section 7.3.

2. The Kac-Rice formula transforms a random differential geometry problem into a random matrix theory problem. The main difficulty in evaluating the Kac-Rice formula comes from the distribution of the Hessian conditioned by the gradient being zero: even for Gaussian random fields, this is in general a heavily correlated Gaussian random matrix, for which very few results exist.

3. The Kac-Rice formula can be generalized to compute higher moments of Crit f,k (B) as well (cf. Theorem 6.3 of [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF]). Via Morse's theory, it can also be used to compute the moments of the Euler characteristic of the level sets of f , see [START_REF] Auffinger | Complexity of random smooth functions on the high-dimensional sphere[END_REF] for an example.

The complexity of the pure spherical p-spin model

To conclude our introduction to the Kac-Rice formula, we will describe its application in a flagship model of disordered systems: the (pure) spherical p-spin, introduced in [CS92], and that we defined in Model 1.3. This calculation was first performed by physicists [CS95, CGG99, Fyo04, FSW07, FW07], and made rigorous in [START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF] (and subsequent papers). We will follow the derivation of the latter, and we will often refer to it for technicalities: the goal of this section is not to present novel results, but rather to give the reader an intuition of the mechanisms of the Kac-Rice method.

Statement of the problem

Consider n ≥ 1, p ≥ 3, and the following function f n,p on the unit sphere S n-1 :

f n,p (σ) ≡ 1≤i 1 ,••• ,ip≤n J i 1 ,•••ip σ i 1 • • • σ ip (σ ∈ S n-1 ), (7.3) in which J i 1 ,•••ip i.i.d.
∼ N (0, 1). In the physics language, the Hamiltonian H n,p of the pure spherical p-spin defined in Model 1.3 is related to f n,p by f n,p (σ) = n -1/2 H n,p ( √ nσ). For any B ⊆ R, we want to compute the large n limit of the expectation of the number of local minima σ of f n,p , such that f n,p (σ) ∈ √ nB, that we denote Crit 0 n,p (B). One can apply the Kac-Rice formula 7.24 :

E Crit 0 n,p (B) = S n-1 dµ(σ)ϕ grad fn,p(σ) (0) (7.4) E | det Hess f n,p (σ)| 1 f n,p (σ) ∈ √ nB, Hess f n,p (σ) ≥ 0 grad f n,p (σ) = 0 ,
in which µ is the usual surface measure on S n-1 . Note that here grad and Hess stand for the Riemannian gradient and Hessian on the sphere, while we will denote ∇, ∇ 2 the Euclidean gradient and Hessian.

The joint distribution of (f n,p (σ), grad f n,p (σ), Hess f n,p (σ))

Deriving the joint law of (f n,p (σ), grad f n,p (σ), Hess f n,p (σ)) is a necessary step in the Kac-Rice method, as these three random variables appear in the conditional expectation.

We fix σ ∈ S n-1 . Note that (f n,p (σ), grad f n,p (σ), Hess f n,p (σ)) is a Gaussian centered random variable since f is a Gaussian random field. We thus simply need to compute its correlations to fully characterize the joint distribution. We will naturally identify the tangent space T σ (S n-1 ) with R n-1 . If we denote P ⊥ σ the orthogonal projector on {σ} ⊥ , one has:

grad f n,p (σ) = P ⊥ σ ∇f n,p (σ), Hess f n,p (σ) = P ⊥ σ ∇ 2 f n,p (σ)P ⊥ σ -σ, ∇f n,p (σ) P ⊥ σ .
For instance one can compute the covariance of the gradient:

E[grad f n,p (σ)grad f n,p (σ) ] = P ⊥ σ E[∇f n,p (σ)∇f n,p (σ) ]P ⊥ σ = pP ⊥ σ .
Using the same kind of simple algebraic calculations, one obtains the joint distribution:

       f n,p (σ) d = Z, grad f n,p (σ) d = √ pg, Hess f n,p (σ) d = (n -1)p(p -1)M n-1 -pZ Id n-1 ,
in which Z ∼ N (0, 1), g ∼ N (0, I n-1 ), and M n-1 ∼ GOE(n -1) (recall that we defined the GOE in Section 1.5) and the variables (Z, g, M n-1 ) are pairwise independent. Let us make the following important remarks:

• The joint distribution of (f n,p (σ), grad f n,p (σ), Hess f n,p (σ)) is independent of σ.

• The variables (f n,p (σ), Hess f n,p (σ)) are independent from grad f n,p (σ).

• From the gradient distribution, one easily obtains its density evaluated in 0:

ϕ grad fn,p(σ) (0) = e -n-1 2 ln(2πp) .
Recalling that the volume of the unit sphere is V (S n-1 ) = 2π n/2 /Γ(n/2), one deduces from the Kac-Rice formula (7.4) and the remarks above:

E Crit 0 n,p (B) = 2π n/2 Γ(n/2) e n-1 2 ln (n-1)(p-1) 2π E[| det H n-1 |1(H n-1 ≥ 0, z ∈ √ nB)], (7.5)
in which

H n-1 ≡ M n-1 - p (n -1)(p -1) zI n ,
with z ∼ N (0, 1) and M n-1 ∼ GOE(n -1). It is now completely explicit that we reduced our original random differential geometry problem (counting the number of critical points of a random function) to a random matrix theory problem.

Simplification of the formula

It is clear from eq. (7.5) that the following lemma (from [START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF]), which computes the average absolute value of the determinant of shifted GOE matrices, will be crucial.

Lemma 7.3 (Expectation of the absolute determinant of a shifted GOE matrix)

Let G ⊆ R a Borel set, X ∼ N (0, t 2 ) (for a t > 0) and M n-1 ∼ GOE(n -1). Then:

E[| det(M n-1 -XI n-1 )|1((M n-1 -XI n-1 ) ≥ 0, X ∈ G)] = 2 n 2 Γ(n/2)(n -1) -n/2 √ πt 2 E GOE(n) e - nλ 2 0 4 2 (n-1)t 2 -1 1 λ 0 ∈ n -1 n G .
Here λ 0 is the smallest eigenvalue of a random matrix from the GOE(n) ensemble.

Lemma 7.3 is a result obtained purely from random matrix theory, and is a particular case of Lemma 3.3 of [START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF]. We refer to this work for its proof.

Let us make a very important remark here. While the explicit calculation of Lemma 7.3 is possible because of the GOE structure of the matrix, the appearance of the smallest eigenvalue should not come as a surprise. We indeed wrote the Kac-Rice formula for the number of local minima of the function f n,p , i.e. critical points with positive Hessian matrix. In order to condition

-1 0 1 3 5 λ 0.0 0.2 0.4
Bulk density σ(λ)

∆ ∼ n -2/3 ∆ ∼ O(1) "Bulk" ∆ Figure 7
.2: Generic illustration of a large deviations event in which the smallest eigenvalue of a random matrix (here a shifted GOE matrix) is macroscopically far from its expected value (here 1). ∆ is the shift of λ max from its expected value, at the left edge of the bulk. We emphasize that while the smallest eigenvalue typically fluctuates in the scale n -2/3 , as shown in the grey area (these fluctuations are connected to the Tracy-Widom law [START_REF] Craig | Level-spacing distributions and the airy kernel[END_REF]), large deviations instead correspond to macroscopic fluctuations, which are exponentially rare in n (cyan area).

on this positivity, we have to understand the law of this smallest eigenvalue. More precisely, we will see that we must understand rare events in which the smallest eigenvalue can be very atypical, a regime called large deviations that we defined in Section 1.5.2 and that we illustrate in Fig. 7.2.

Applying Lemma 7.3 to eq. (7.5) with t 2 = p/[(n -1)(p -1)] and G = (t √ n)B yields:

E Crit 0 n,p (B) = 2 2 p (p -1) n 2 E GOE(n) e -n p-2 4p λ 2 0 1 λ 0 ∈ p p -1 B .
(7.6)

The large n limit

We are interested here in the annealed complexity Σ 0 p (B) ≡ lim n→∞ (1/n) ln E Crit 0 n,p (B), as a first step towards understanding the landscape's topology. As we noticed above, this requires understanding the large deviations of the smallest eigenvalue of a GOE(n) matrix, cf. Fig. 7.2. This is precisely given by Theorem A.1 of [ABAČ13]:

Lemma 7.4 (LDP of the smallest eigenvalue of a GOE matrix)

Let λ 0 be the smallest eigenvalue of a GOE(n) matrix. The law of λ 0 satisfies a large deviation principle in the scale n, with rate function I(x):

I(x) ≡        +∞ if x ≥ -2, -x 2 dz z 2 4 -1 otherwise.
.

In Chapter 8 we will specifically focus on the large deviations of the smallest eigenvalue of random matrices, from an ensemble much larger than only the GOE. Using Lemma 7.4 alongside eq (7.6) and Varadhan's lemma 1.10 yields:

Σ 0 p (B) = 1 2 ln(p -1) + sup x∈ p p-1 B - p -2 4p x 2 -I(x) . ( 7
k (u) E 0 E 1 E 2 E 10 E 100 p = 3 k = 0 k = 1 k = 2 k = 10 k = 100 E 1.
E 0 E 1 E 2 E 10 E 100 p = 4

Description of the results

Considering B = (-∞, u) (for u ∈ R) in eq. (7.7) amounts to count the local minima of the pure p-spin Hamiltonian of extensive energy smaller than nu. In this case, the supremum in eq. (7.7) can be analytically performed. Moreover, the calculation we sketched for local minima can be generalized for critical points of any fixed index k ∈ N, and if we define:

Θ k (u) ≡ lim n→∞ 1 n ln E Crit k n,p ((-∞, u)),
there exists analytic expressions for all Θ k (u) functions, given in eq (2.16) of [START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF]. We plot these functions for p = 3, 4 in Fig. 7.3. In particular, all these functions agree for u ≥ -E ∞ , which is often referred to as the threshold energy

E ∞ ≡ 2 p -1 p . (7.8)
We also denote -E k the value at which the function Θ k (u) becomes positive: Θ k (-E k ) = 0, cf Fig. 7.3. We end this description by a series of remarks that one can make from the figure:

(i) The local minima always dominate the complexity for all energies below -nE ∞ . For u ≥ -E ∞ , one can show that the complexity is dominated instead by critical points of diverging index.

(ii) The value -E 0 is the lowest energy of critical points, and corresponds to the ground state of the original function. It can be indeed shown (cf. [START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF]) that the global minimum energy concentrates to a deterministic value as n → ∞, and that this value is precisely -E 0 ! (iii) One can perform similar calculations for the complexity of critical points whose indices diverge with n, see [START_REF] Auffinger | Complexity of random smooth functions on the high-dimensional sphere[END_REF] for the rigorous derivation.

To conclude our analysis, let us mention a few generalizations and extensions of the Kac-Rice calculation for p-spin models that we described:

• Beyond the annealed complexity, one can also use the Kac-Rice formula for higher-order moments, in order to compute e.g. the second moment of the number of critical points. For the pure spherical p-spin this is done in [START_REF] Subag | The complexity of spherical p-spin models-a second moment approach[END_REF], which then shows by a second-moment method that the number of local minima concentrates. The annealed and quenched complexities of local minima are therefore identical for a pure spherical p-spin, which increases a lot the physical relevance of our previous discussion: we were describing the actual quenched complexity! This allows to describe very detailed properties of the Gibbs measure at low temperatures, see e.g. [Sub17b, SZ17, BASZ20].

• Going further, one can even compute all the integer moments: this allows to perform heuristic replica calculations to obtain the quenched complexity, as performed for Gaussian models in [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF].

• All the calculations we mentioned can be generalized to mixed spherical p-spins (cf. Model 1.4), which was done on [ABA13, RBABC19], or to bipartite spherical models (both pure and mixed) [START_REF] Mckenna | Complexity of bipartite spherical spin glasses[END_REF].

• Our arguments relied quite strongly on the isotropy of the random field we are describing. Recent works in the physics and mathematics literature derive important extensions of the Kac-Rice approach to non-isotropic Gaussian model (e.g. the elastic manifold) [START_REF] Yan | Manifolds in a high-dimensional random landscape: Complexity of stationary points and depinning[END_REF][START_REF] Yan | Manifolds pinned by a high-dimensional random landscape: Hessian at the global energy minimum[END_REF][START_REF] Ben Arous | Landscape complexity beyond invariance and the elastic manifold[END_REF].

Let us emphasize once again that the Kac-Rice method we presented transforms our original counting problem into a random matrix theory problem. Moreover, as we saw, counting critical points of specific index requires knowing the large deviations of the corresponding eigenvalues of the Hessian matrix. This last remark is important, as it will be one of the motivations behind Chapter 8.

Kac-Rice for inference models: main results

In the rest of this chapter, based on [MBAB20], we present an extension of the Kac-Rice method to compute the number of critical points of the empirical risk arising in generalized linear estimation problems, i.e. the GLMs that we already encountered several times in this thesis. We obtain a rigorous explicit variational formula for the annealed complexity (Theorems 7.5 and 7.6), that we then simplify and extend using a heuristic Kac-Rice replicated method, which originated in theoretical physics. In this way we find an explicit variational formula for the quenched complexity (Results 7.1 and 7.2), which allows to obtain the number of critical points for typical instances up to exponential accuracy.

Let us first mention a recent line of work that generalized the Kac-Rice calculations of Section 7.1 to a class of inference models known as spiked matrix-tensor models [SMKUZ19, SMBC + 19]. The authors leveraged the Kac-Rice formula and a precise random matrix analysis of the Hessian to gain precise insight into the landscape and the behavior of local optimization algorithms. Crucially, the loss function considered is still a Gaussian random function (although much more involved than the pure p-spin of eq. (7.3)), so that the random matrix problem resulting from the application of Kac-Rice is essentially the analysis of a (spiked) GOE matrix. We refer to [START_REF] Sarao | Thresholds of descending algorithms in inference problems[END_REF] for a review on this approach.

In this chapter we aim at going beyond this Gaussian setting, and we instead consider two classes of high-dimensional random functions.

• The first one is a kind of energy that arises in a simple model of neural networks (the perceptron, cf. [START_REF] Engel | Statistical mechanics of learning[END_REF]) and in mean-field glass models [START_REF] Franz | The simplest model of jamming[END_REF]:

L 1 (x) ≡ 1 m m µ=1 φ(ξ µ • x), (7.9)
where φ : R → R is a "smooth" (the precise sense will be given later) activation function, x ∈ S n-1 (the unit sphere in n dimensions) and ξ µ i.i.d.

∼ N (0, I n ).

• The second class of functions we will consider is related to the loss functions of generalized linear models (GLMs), that we introduced in Section 1.1 and that we already investigated many times in this thesis. In the current setting, an observer has to infer a hidden vector x ∈ S n-1 from the observation of the m-dimensional output vector Y = {φ(ξ µ •x )} m µ=1 5 . The data (or measurement) matrix ξ is taken random, with an i.i.d. standard Gaussian distribution, and we assume that the function φ and the data matrix ξ are given to the observer. This naturally leads to the mean square loss L 2 :

L 2 (x) ≡ 1 2m m µ=1 φ(ξ µ • x ) -φ(ξ µ • x) 2 . (7.10)
As we discussed in Section 7.1, we are interested in the statistics of the number of critical points of the functions of eqs. (7.9),(7.10). To introduce the notations we focus on L 2 . For any open intervals B ⊆ R + and Q ⊆ (-1, 1), we consider the (random) number Crit n,L 2 (B, Q) of critical points of the function L 2 with loss value in B and overlap with the signal q ≡ x • x in Q:

Crit n,L 2 (B, Q) ≡ x: grad L 2 (x)=0 1{L 2 (x) ∈ B, x • x ∈ Q}. (7.11)
Here grad is the Riemannian gradient on S n-1 . For L 1 we define the similar quantity Crit n,L 1 (B), dropping the notion of overlap.

Our main results consist in explicit formulas for the annealed and quenched 6 complexities for L 1 and L 2 . The annealed formula is obtained rigorously using the Kac-Rice formula, whereas the quenched one is derived combining the Kac-Rice formula with the replica method, one of the classical tools of our statistical physics toolbox, cf Section 1.3.1. The quenched calculation leverages in particular important ideas of [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF].

We consider the thermodynamic limit n, m → ∞ with m/n → α > 1. The condition α > 1 is essential, as can be seen e.g. in eq. (7.9): if m < n, for each realization of {ξ µ }, the function L 1 has an infinite number of critical points in the set of vectors x orthogonal to all the {ξ µ }, and counting the critical points in this case is meaningless (or one would have to quotient the space to lift the degeneracy).

Our results hold for many classical activation functions φ, such as e.g. the hyperbolic tangent, the arctangent, the sigmoid, or a smoothed and leaky version of the ReLU activation function 7 .

For two probability measures µ, ν recall that we define the relative entropy (or Kullback-Leibler divergence) as D KL (µ|ν) ≡ ln(dµ/dν)dµ if µ is absolutely continuous with respect to ν, and +∞ otherwise. Finally, µ G is a generic notation for the standard Gaussian measure on any R k .

The annealed complexity

We can now present our main results for the annealed complexity.

In general GLMs, the output function is stochastic. Here, we restrict to deterministic outputs. Recall that we discussed the difference between annealed and quenched complexities in Section 7.1, cf eq. (7.1). The precise hypotheses on the activation function φ are precised in Section 7.3.

Theorem 7.5 (Annealed complexity of L 1 )

Let B ⊆ R a non-empty open interval and denote M φ (B) the set of probability measures ν on R such that ν(dt)φ(t) ∈ B. We define:

• E φ (ν) ≡ ln ν(dx)φ (x) 2 , • t φ (ν) ≡ ν(dx)xφ (x),
• Let z ∈ R n×m an i.i.d. standard Gaussian matrix, and y ∈ R m a vector with components taken i.i.d. from a probability measure ν. Let D (ν) the diagonal matrix of size m with elements D (ν) µ = φ (y µ ). We define µ α,φ [ν] as the LSD of zD (ν) z /m.

• κ α,φ (ν, C) ≡ µ α,φ [ν](dx) ln |x -C|,
Under a technical assumption (see the remark below), one has 8 :

lim n→∞ 1 n ln E Crit n,L 1 (B) = 1 + ln α 2 + sup ν∈M φ (B) - 1 2 E φ (ν) + κ α,φ (ν, t φ (ν)) -αD KL (ν|µ G ) .
A note on free probability -Interestingly, the measure µ α,φ [ν] can be interpreted as the free multiplicative convolution of the Marchenko-Pastur law (at ratio α) and the asymptotic spectral distribution of the matrix D (ν) , cf. e.g. [Voi87, AGZ10] 9 . We describe in Section 7.4 how to explicitly compute the density of µ α,φ [ν], or its linear spectral statistics (as e.g. κ α,φ (ν, C)), via the computation of its Stieltjes transform.

We turn to our second (quite heavy) annealed result:

8 A fully rigorous statement would imply a lower and an upper bound given by a supremum over the adherence and the interior of M φ (B). For reasons of lightness and clarity of the presentation we write it in the simpler presented form. 9 Free multiplication is usually defined for positively-supported measures, however one can generalize it here by explicitly separating the positive and negative parts of φ (we can show freeness of the resulting two random matrices).

Theorem 7.6 (The annealed complexity of L 2 )

Let B ⊆ R + and Q ⊆ (-1, 1) two non-empty open intervals. For q ∈ (-1, 1) we denote M φ (B, q) the set of probability measures ν on R 2 such that

ν(dx, dy) y φ (x)[φ(qx + 1 -q 2 y) -φ(x)] = 0, ν(dx, dy)[φ(qx + 1 -q 2 y) -φ(x)] 2 ∈ B.
Given the following definitions:

• E φ (q, ν) ≡ ln ν(dx, dy) φ (x) 2 φ(qx + 1 -q 2 y) -φ(x) 2 , • t φ (q, ν) ≡ ν(dx, dy) x φ (x) φ(x) -φ(qx + 1 -q 2 y) , • f q (x, y) ≡ φ (x) 2 -φ (x) φ(qx + 1 -q 2 y) -φ(x) ,
• Let z ∈ R n×m an i.i.d. standard Gaussian matrix, and Y ∈ R m×2 with components taken i.i.d. from ν ∈ M + 1 (R 2 ). Let D (ν,q) the diagonal matrix of size m with elements D (ν,q) µ = f q (Y µ ). We define µ α,φ [q, ν] as the LSD of zD (ν,q) z /m.

• κ α,φ (q, ν) ≡ µ α,φ [q, ν](dx) ln |x -t φ (q, ν)|.
Then one has 10 :

lim n→∞ 1 n ln E Crit n,L 2 (B, Q) = 1 + ln α 2 + sup q∈Q sup ν∈M φ (B,q) 1 2 ln(1 -q 2 ) - 1 2 E φ (q, ν) + κ α,φ (q, ν) -αD KL (ν|µ G ) .
The proof of Theorem 7.5 will be the main subject of Section 7.3.

Technical assumption -Our proof relies on an assumption given in eq. (E.16). Under Definition 7.1 on the behavior of φ(x), we are currently working on proving this assumption by leveraging the recent work of close collaborators [START_REF] Ben Arous | Exponential growth of random determinants beyond invariance[END_REF]. However, note that eq. (E.16) can also be considered as an hypothesis on φ(x), i.e. that there is no accumulation of eigenvalues around 0 in the spectrum of the Hessian of L 1 (x). This assumption is quite natural, and we found it to be satisfied for all relevant functions φ(x) we investigated, so that it does not limit the reach of our results.

The proof of Theorem 7.6 is a straightforward generalization, and we will sketch it briefly in Appendix E.2.5.

The variational problems in Theorems 7.5 and 7.6 are challenging, as they imply an optimization on a set of measures, and they involve transforms of this measure that are very hard to access numerically. In Section 7.4 we present a drastic simplification: a heuristic calculation that allows one to reduce the supremum over the probability measure ν to a much simpler optimization over a relatively small number of parameters.

The quenched complexity

As we have already stressed, the annealed complexity, although interesting in itself, is generically not representative of the landscape corresponding to a given typical instance of the empirical risk. In order to obtain the value of the quenched complexity we use the replicated Kac-Rice method, which is an extension to non-Gaussian functions of the one developed in [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF]. As we know, although the replica method is non-rigorous it has been proven to give exact results for both spin glasses and inference problems [Tal06, BKM + 19] (and proving the replica conjectures was one of the main aims of Part II of this thesis.). We have obtained an explicit formula11 for the quenched complexity of L 1 and L 2 at fixed values of the empirical risk, and overlap with the solution (in the L 2 case).

For L 1 , using the notations of Theorem 7.5 we have:

Result 7.1 (Quenched complexity of L 1 )
Let B ⊆ R an open interval. Then:

lim n→∞ 1 n E ln Crit n,L1 (B) = ln α -α ln 2π 2 + sup ν∈M φ (B) q∈(0,1) extr κ α,φ (ν, C) + 1 -α 2 ln(1 -q) + 1 -αq 2(1 -q) -ν(dλ)g(λ) - A Â -aâ 2 + C(qĉ -Ĉ) - 1 2 ln[A -a] - a 2(A -a) + α R 4
Dξ ln I(ξ) .

Here ξ ≡ (ξ q , ξ a , ξ c , ξ c ) and Dξ is the standard Gaussian probability measure on R 4 . extr denotes extremization with respect to all variables (A, Â, a, â, C, Ĉ, ĉ, {g(λ)}). We defined

I(ξ) ≡ R dλ e -λ 2 2(1-q) + g(λ) α + Â-â 2α φ (λ) 2 + Ĉ-ĉ α φ (λ)λ+ √ q 1-q ξqλ+ â α ξaφ (λ)+ ĉ 2α [φ (λ)(ξc+iξ c )+λ(ξc-iξ c )] .
On the extremization -In this formula, the notation "extr" denotes that one should set the partial derivatives with respect to the involved variables to zero. This notation arises from the replica calculation, which mixes saddle-point computations with Lagrange multipliers associated to certain constraints, and the precise meaning of this extremization (as a supremum or infimum) would have to be clarified by a more rigorous method. On a numerical point of view, one would have to solve the associated saddle-point equations, so that this precise meaning is not crucial for applications.

We can state a very similar result for L 2 :

Result 7.2 (Quenched complexity of L 2 ) Let B ⊆ R, Q ⊆ (-1, 1
) two open intervals and define:

• For m ∈ (-1, 1), M φ (B, m) is the set of ν ∈ M + 1 (R 2 ) such that: 1 2 ν(dλ 0 , dλ)[φ(λ) -φ(λ 0 )] 2 ∈ B, ν(dλ 0 , dλ)φ (λ)[φ(λ) -φ(λ 0 )](λ 0 -mλ) = 0. • Let f (x, y) ≡ φ (y)[φ(y) -φ(x)] + φ (y) 2 . Let z ∈ R n×m an i.i.d. standard Gaussian matrix,
and Y ∈ R m×2 with components taken i.i.d. from ν ∈ M + 1 (R 2 ). Let D (ν) the diagonal matrix of size m with elements D

(ν) µ = f (Y µ ). We define µ α,φ [ν] as the asymptotic spectral measure of zD (ν) z /m. • χ α,φ (ν, C) ≡ µ α,φ [ν](dx) ln |x -C|.
One has:

lim n→∞ 1 n E ln Crit n,L 2 (B, Q) = sup m∈Q q∈(0,1) sup ν∈M φ (B,m) extr ln α -α ln 2π 2 + χ α,φ (ν, C) + 1 -αq -m 2 2(1 -q) + 1 -α 2 ln(1 -q) - 1 2 ln(A -a) - a 2(A -a) - A Â 2 + aâ 2 -C 0 Ĉ0 -C Ĉ + cĉ -ν(dλ 0 , dλ)g(λ 0 , λ) + α R 4 ×R Dξ Dλ 0 ln I(λ 0 , ξ) .
The extremum is made over all variables (A, a, C 0 , C, c, Â, â, Ĉ, ĉ, Ĉ0 , {g(λ 0 , λ)}). D is the standard Gaussian measure, and the variables C 0 , c, C are related by the additional constraint

-m(1 -q)C 0 -(q -m 2 )C + (1 -m 2 )c = 0.
I(λ 0 , ξ) is defined as (with ξ ≡ (ξ q , ξ a , ξ c , ξ c )):

I(λ 0 , ξ) ≡ R dλ e m 1-q λ 0 λ-λ 2 2(1-q) + √ q-m 2 1-q ξqλ+ g(λ 0 ,λ) α + Ĉ0 α φ (λ)[φ(λ)-φ(λ 0 )]λ 0 + Ĉ-ĉ α λφ (λ)[φ(λ)-φ(λ 0 )] e Â-â 2α φ (λ) 2 [φ(λ)-φ(λ 0 )] 2 + â α ξaφ (λ)[φ(λ)-φ(λ 0 )]+ ĉ 2α φ (λ)[φ(λ)-φ(λ 0 )](ξc+iξ c )+λ(ξc-iξ c ) .
The derivation of Result 7.1 is given in Section 7.5. Similarly to the annealed case, Result 7.2 can be derived by a simple generalization of this computation, see Appendix E.2.5.

Tackling the variational problems

• As a sanity check of our results, the reader can analytically check by explicit solution that for a linear activation function, the annealed complexity of L 1 is null in Theorem 7.5. It is again a tedious but straightforward computation to check that the annealed complexity of L 1 with a quadratic activation φ(x) = x 2 is also null, as the number of critical points in this case is linear with n. In both these cases, the measure µ α,φ [ν] is indeed independent of ν, which simplifies the problem drastically. Note that for L 2 however, even the case of a linear activation is non trivial (because of the spherical constraint), as shown in the recent analysis of [START_REF] Yv Fyodorov And R Tublin | Counting stationary points of the loss function in the simplest constrained least-square optimization[END_REF].

• However, for more generic activation functions, solving the variational problem of the previous annealed and quenched calculations is a very involved task. We explain in Section 7.4 a route to make tractable theses variational problems, by reducing them to optimization problems over a relatively small number of parameters.

• Apart from the simple cases described above, it is not clear to deduce from our results if the quenched and annealed complexities are equal for given values of the loss function, or for a given activation function. Such an equality is equivalent to the concentration of the number of critical points, and its occurrence (or lack thereof) has deep physical consequences on our understanding of these landscapes. Note that while this happens notably in the pure p-spin, as shown by a second moment analysis in [START_REF] Subag | The complexity of spherical p-spin models-a second moment approach[END_REF], it remains a very peculiar situation, not satisfied e.g. by generic spherical p-spin "mixtures", and in the context of generalized linear models it would have to be investigated through an extensive numerical analysis of our results, which is not presented in this thesis.

Proof of the annealed complexity

In this section we prove Theorem 7.5. Our technique leverages the Kac-Rice formula and Sanov's theorem 1.9 on the large deviations of the empirical measure of i.i.d. variables. First we precise our hypotheses on φ, that we will take in the following set of "well-behaved" activation functions:

Definition 7.1 (Well-behaved activation function)

φ : R → R is "well-behaved" if it is of class C 3 and if, for y ∼ N (0, 1), the random variable a = φ (y) admits a continuous probability density in a neighborhood of a = 0.

Applying the Kac-Rice formula

The first step is to apply the Kac-Rice formula to the random function L 1 :

Lemma 7.7 (Kac-Rice formula for L 1 )

For any x ∈ S n-1 , denote grad L 1 (x) and Hess L 1 (x) the (Riemannian) gradient and Hessian of L 1 at the point x. Then grad L 1 (x) has a well defined density (on the tangent space T x S n-1 R n-1 ) in a neighborhood of zero, that we denote ϕ grad L 1 (x) . Denote µ S the usual surface measure on S n-1 . One has:

E Crit n,L 1 (B) = S n-1 ϕ grad L 1 (x) (0)E 1{L 1 (x) ∈ B}| det Hess L 1 (x)| grad L 1 (x) = 0 µ S (dx).
This lemma is a direct application of Proposition 7.2 and uses necessary conditions for a random function of the type of L 1 to be a.s. Morse that are stated in [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF]. The details of the proof are given in Appendix E.2.1.

The complexity at finite n

E Crit n,L 1 (B) = C n e n 1+ln α 2 E y 1 1 m µ φ(yµ)∈B e -n-1 2 ln 1 m µ φ (yµ) 2 E z | det H Λ n (y)| ,
in which C n is exponentially trivial, i.e. (1/n) ln C n = O n (1). The variable y ∈ R m follows N (0, I m ), and z ∈ R (n-1)×m has i.i.d. standard Gaussian matrix elements, independent of y. H Λ n (y) is a square matrix of size (n -1) with the following distribution :

H Λ n (y) d = 1 n zΛ(y)z - 1 m m µ=1 y µ φ (y µ ) I n-1 . (7.13)
The rest of Section 7.3.2 is devoted to the proof of Lemma 7.8. First, the following proposition specifies the joint distribution of (L 1 (x), grad L 1 (x), Hess L 1 (x)), which will be useful to apply Lemma 7.7. Proposition 7.9 (Distribution of the gradient and Hessian)

Let x ∈ S n-1 . Then (L 1 (x), grad L 1 (x), Hess L 1 (x)) follows the following joint distribution: ∼ N (0, I n-1 ), and all {y µ , z ν } are independent. We identified in these equations the tangent space T x S n-1 with R n-1 .

                         L 1 (x) d = 1 m m µ=1 φ(y µ ), (7.14a
Proof of Proposition 7.9 -Denote P ⊥

x the orthogonal projection on {x} ⊥ . For a smooth function f : S n-1 → R, ∇f and ∇ 2 f denote its Euclidean gradient and Hessian. The Riemannian structure on S n-1 induces the gradient and Hessian of f as grad f (x) = P ⊥

x ∇f and Hess f (x) = P ⊥

x ∇ 2 f P ⊥ x -(x • ∇f (x))P ⊥ x . Applying these formulas yields:

             grad L 1 (x) = 1 m m µ=1 (P ⊥ x ξ µ )φ (ξ µ • x), (7.15a) Hess L 1 (x) = 1 m m µ=1 φ (ξ µ • x)(P ⊥ x ξ µ )(P ⊥ x ξ µ ) - 1 m m µ=1 (ξ µ • x)φ (ξ µ • x) P ⊥ x .(7.15b)
Letting y µ ≡ ξ µ • x and z µ ≡ P ⊥ x ξ µ (identified to an element of R n-1 ) yields the result.

The joint distribution of eq. (7.14) does not depend on x, thus we can chose x to be the North pole x = e n = (δ i,n ) n i=1 . With ω n ≡ 2π n/2 /Γ(n/2) the volume of S n-1 , we obtain from Lemma 7.7:

E Crit n,L 1 (B) = ω n ϕ grad L 1 (en) (0)E | det Hess L 1 (e n )|1 L 1 (en)∈B grad L 1 (e n ) = 0 . (7.16)
Removing the e n indication and conditioning on the distribution of y, we reach:

E Crit n,L 1 (B) = ω n E y 1 1 m µ φ(yµ)∈B ϕ gradL 1 |y (0) E z | det Hess L 1 | grad L 1 = 0, y .
Once conditioned on y, eq. (7.14b) describes a Gaussian density so we can directly compute:

ω n ϕ gradL 1 |y (0) = 2π n/2 Γ(n/2) exp - n -1 2 ln 2π m 2 m µ=1 φ (y µ ) 2 , = C n exp n 2 + n 2 ln m n - n -1 2 ln 1 m m µ=1 φ (y µ ) 2 , ( 7.17)
in which ln C n = O n (n) (using Stirling's formula). The conditioning of Hess L 1 by grad L 1 = 0 at fixed y reduces to a linear conditioning on z. One thus obtains by classical Gaussian conditioning:

E z | det HessL 1 | grad L 1 = 0, y = E z | det H Λ n (y)| , (7.18)
in which H Λ n (y) is defined by eq. (7.13). This ends the proof of Lemma 7.8.

Concentration and large deviations

This section is devoted to the end of the proof of Theorem 7.5. We denote ν m y ≡ m -1 m µ=1 δ yµ the empirical distribution of y, and take the notations of Theorem 7.5 and Lemma 7.8. We first state an important lemma on the concentration of

E z | det H Λ n (y)| 12 : Lemma 7.

(Concentration of the log-determinant)

There exists η > 0 such that for all t > 0:

lim n→∞ 1 n 1+η ln P 1 n ln E z | det H Λ n (y)| -κ α,φ (ν m y , t φ (ν m y )) ≥ t = -∞.
The proof of Lemma 7.10 is detailed in Appendix E.2.2, and is the section of our proof that requires to assume eq. (E.16). On a general note, we expect this result to be valid in the whole range η ∈ (0, 1), as the large deviations of the spectral distribution of random matrices is typically on the n 2 scale [START_REF] Ben | Large deviations for Wigner's law and Voiculescu's non-commutative entropy[END_REF][START_REF] Hiai | Eigenvalue density of the Wishart matrix and large deviations[END_REF]. Note that very similar results on the concentration of determinants of very generic classes of random matrices have recently been analyzed in [START_REF] Ben Arous | Exponential growth of random determinants beyond invariance[END_REF] (see as well the companion papers [START_REF] Ben Arous | Landscape complexity beyond invariance and the elastic manifold[END_REF][START_REF] Mckenna | Complexity of bipartite spherical spin glasses[END_REF]). The following moment condition, proven in Appendix E.2.3, will be important to apply Varadhan's lemma 1.10:

Lemma 7.11

For every γ ∈ (1, α) we have:

               lim sup n→∞ 1 n ln E y e γn -1 2 ln 1 m µ φ (yµ) 2 +κ α,φ ν m y ,t φ (ν m y ) < +∞, (7.19a) lim sup n→∞ 1 n ln E y e γn -1 2 ln 1 m µ φ (yµ) 2 + 1 n ln Ez | det H Λ n (y)| < +∞. (7.19b)
We then conclude from Lemmas 7.8, 7.10 and 7.11, using Sanov's principle (Theorem 1.9) and Varadhan's lemma 1.10. We finally reach the statement of Theorem 7.5:

lim n→∞ 1 n ln E Crit n,L 1 (B) = sup ν∈M φ (B) 1 + ln α 2 - E φ (ν) 2 + κ α,φ (ν, t φ (ν)) -αD KL (ν|µ G ) . (7.20)
The detailed proof of eq. (7.20) from the lemmas above is given in Appendix E.2.4.

Towards a numerical solution?

7.4.1 The logarithmic potential of µ α,φ [ν] Let ν ∈ M + 1 (R). As we have shown in the introduction to this thesis, cf. Theorem 1.7, the Stieltjes transform g(z) ≡ µ(dt)(t -z) -1 of µ α,φ [ν] is given by the unique solution in C + to the implicit Marchenko-Pastur equation:

∀z ∈ C + , g(z) = -z -α φ (t) α + φ (t)g(z) ν(dt) -1 . (7.21)
For any µ ∈ M + 1 (R) and t ∈ R we define the logarithmic potential as U [µ](t) ≡ µ(dx) ln |x -t|. It is well defined with values in R∪{±∞}, and the reader can refer to [START_REF] Faraut | Logarithmic potential theory, orthogonal polynomials, and random matrices[END_REF] for a review on this subject. To numerically evaluate Theorem 7.5, we have to compute U [µ](t) for µ = µ α,φ [ν] and an arbitrary t ∈ R. For clarity, we will write µ = µ α,φ [ν] for the remainder of this section. Let us define G(z) ≡ µ(dx) ln(z -x) for any z ∈ C + . One sees directly that G(z) is holomorphic on C + . Moreover, from Chapter II of [START_REF] Faraut | Logarithmic potential theory, orthogonal polynomials, and random matrices[END_REF], we know that

U [µ](t) = lim ↓0 Re G(t + i ).
It is then clear that a way to compute the logarithmic potential is to evaluate G(z) for z ∈ C + . Define, for z, g ∈ C +13 :

F (z, g) ≡ -ln(-g) -zg + α ν(dλ) ln(α + φ (λ)g) -1 -α ln α. (7.22)
At any fixed z, F (z, g) is an holomorphic function of g on C + . Its Wirtinger derivative is:

∂F ∂g (z, g) = - 1 g -z + α ν(dλ) φ (λ) α + φ (λ)g .
Thus g(z) (the Stieltjes transform of µ, cf. eq. (7.21)) is the only g ∈ C + such that ∂ g F (z, g) = 0. Moreover, by definition g(z) is an holomorphic function on C + , with values in C + . We can thus apply the usual composition of derivatives and obtain:

dF dz (z, g(z)) = -g(z).
Furthermore by definition dG/dz = -g(z). Computing the remaining constant by investigating the limit Re[z] → ∞, the reader can easily check that G(z) = F (z, g(z)) for every z ∈ C + . We thus have the crucial relation:

∀t ∈ R, U [µ α,φ [ν]](t) = lim ↓0 Re F [t + i , g(t + i )]. (7.23)
This formula allows for an efficient numerical derivation of the logarithmic potential of µ α,φ [ν], for any value of t (including possibly inside the bulk of µ α,φ [ν]), as we will detail below.

Heuristic derivation of simplified fixed point equations

We present here an heuristic derivation of scalar fixed point equations for the numerical resolution of Theorem 7.5. This technique can be easily extended to Theorem 7.6 as well as the quenched calculations presented afterwards, and we restrict to this simpler case for the sake of the presentation.

Expressing κ α,φ (ν, t)

From eq. (7.23) we know that for every t ∈ R:

κ α,φ (ν, t) = lim ↓0 Re -ln(-g(t + i )) -(t + i )g(t + i ) + α ν(dλ) ln[α + φ (λ)g(t + i )] -1 -α ln α .
Recall that g(t + i ) is the only solution in C + to the partial derivative of the previous equation:

- 1 g -(t + i ) + α ν(dλ) φ (λ) α + φ (λ)g = 0. (7.24)
So heuristically we can write for small enough :

κ α,φ (ν, t) = extr g∈C + -ln |g| -tg r + g i + α ν(dλ) ln |α + φ (λ)g| -1 -α ln α , (7.25)
with g = g r + ig i (in practice one considers the two variables g r and g i to find the extremum).

Heuristic solution to Theorem 7.5

We start from the result of Theorem 7.5. For a function f , we write E ν [f (X)] ≡ ν(dt)f (t). We introduce Lagrange multipliers to fix the conditions E ν [φ(X)] ∈ B, and we fix the values of E ν [φ (X) 2 ] and E ν [Xφ (X)]. We obtain:

lim n→∞ 1 n ln E Crit n,L 1 (B) = sup l∈B ν∈M + 1 (R) extr λ 0 ,λ 1 ,λ 2 sup A,t 1 + ln α 2 - 1 2 ln A + λ 0 l + λ 1 A + λ 2 t + κ α,ν (ν, t) -αD KL (ν|µ G ) -λ 0 E ν [φ(X)] -λ 1 E ν [φ (X) 2 ] -λ 2 E ν [Xφ (X)] .
The supremum over ν is now unconstrained over the set M + 1 (R) of probability distributions. We now make use of eq. (7.25) to write, with 2K(α) ≡ -1 + ln α -2α ln α and a small > 0:

lim n→∞ 1 n ln E Crit n,L 1 (B) = sup l∈B ν∈M + 1 (R) extr {λ i },A,t g∈C + K(α) - 1 2 ln A + λ 0 l + λ 1 A + λ 2 t -ln |g| -tRe[g] + Im[g] + α ν(dλ) ln |α + φ (λ)g| -αD KL (ν|µ G ) -λ 0 E ν [φ(X)] -λ 1 E ν [φ (X) 2 ] -λ 2 E ν [Xφ (X)] .
For any function F : R → R, the maximum sup ν {E ν [F (X)] -αD KL (ν|µ G )} is attained in ν * with density proportional to e -x 2 /2+F (x)/α , which is exactly the Gibbs measure of statistical physics, see Section 1.5. This gives (D is the standard Gaussian measure on R):

sup ν∈M + 1 (R) {E ν [F (X)] -αD KL (ν|µ G )} = α ln R Dx e F (x)/α .
Plugging this into our previous equation for the annealed complexity yields:

lim n→∞ 1 n ln E Crit n,L 1 (B) = sup l∈B extr {λ i },A,t g∈C + K(α) + λ 0 l + λ 1 A + λ 2 t - ln A 2 -ln |g| -tRe[g] + Im[g] + α ln R Dx exp - λ 0 φ(x) + λ 1 φ (x) 2 + λ 2 xφ (x) α + ln |α + φ (x)g| .
This can be further simplified, as the extrema over A, t are trivially solved and give the value of λ 2 = Re[g] and λ 1 = (2A) -1 . Thus we obtain:

lim n→∞ 1 n ln E Crit n,L 1 (B) = sup l∈B extr {λ 0 ,λ 1 } g∈C + K(α) + λ 0 l + 1 + ln 2 2 + 1 2 ln λ 1 -ln |g| + Im[g] + α ln R Dx exp - λ 0 φ(x) + λ 1 φ (x) 2 + Re[g]xφ (x) α + ln |α + φ (x)g| . (7.26)
Let us now denote the measure:

• • • λ 0 ,λ 1 ,g ≡ R Dx(• • • ) exp{-α -1 [λ 0 φ(x) + λ 1 φ (x) 2 + Re[g]xφ (x)] + ln |α + φ (x)g|} R Dx exp{-α -1 [λ 0 φ(x) + λ 1 φ (x) 2 + Re[g]xφ (x)] + ln |α + φ (x)g|}
, then the fixed point equations corresponding to the formula of eq. (7.26) can be written as:

                         l = φ(x) λ 0 ,λ 1 ,g , 1 2λ 1 = φ (x) 2 λ 0 ,λ 1 ,g , - Re[g] |g| 2 = xφ (x) - αφ (x)(α + φ (x)Re[g]) |α + φ (x)g| 2 λ 0 ,λ 1 ,g , - Im[g] |g| 2 = - αφ (x) 2 Im[g] |α + φ (x)g| 2 λ 0 ,λ 1 ,g .
(7.27)

These equations are to be iterated over λ 0 , λ 1 , g, and l (while enforcing the constraint l ∈ B).

From experience, the best procedure is to start from the solution of the unconstrained problem (without any constraint on the loss value), before smoothly following the solution while adding the constraint. In the case of L 2 (x), one would follow a similar procedure.

The quenched complexity and the replica method

In this section we detail the principle of the quenched calculation that gives rise to Results 7.1 and 7.2. For the sake of the presentation we restrict to Result 7.1, while Result 7.2 will be discussed in Appendix E.2.5. As the very basis of this calculation is non-rigorous we present this calculation in a fashion closer to theoretical physics standards, differently from Section 7.3 which was written in a more mathematical convention. Many technicalities will be postponed to Appendix E.1.

Computing the p-th moment

If needed, the reader can refer to Section 1.3.1 for an introduction to the replica method, one of the most important heuristic tools of theoretical statistical physics, that we already used extensively throughout this thesis. The first step of the method is to compute the integer moments of the observable, here the number of critical points. Let B ⊆ R an open interval. The Kac-Rice formula can then be stated for the p-th moment of the complexity [AW09b, AT09]:

E Crit n,L 1 (B) p = p a=1 S n-1 µ S (dx a ) 1[{L 1 (x a ) ∈ B} p a=1 ] ϕ {grad L 1 (x a )} p a=1 (0) × E p a=1 det Hess L 1 (x a )| {grad L 1 (x a )} p a=1 = 0 .
Here, ϕ {grad L 1 (x a )} p a=1 (0) represents the joint density of the p gradients, taken at 0. Note that the functions {L 1 (x a )} p a=1 only depend on the parameters y a µ ≡ ξ µ • x a , so we will abusively write L 1 (y a ) ≡ L 1 (x a ). Proceeding as in the annealed case, we can rewrite the expectations by conditioning over {y a } p a=1 :

E Crit n,L 1 (B) p = p a=1 µ S (dx a ) E {y a } 1[{L 1 (y a ) ∈ B} p a=1 ] ϕ {grad L 1 (x a )} p a=1 {y a } (0) × E p a=1 | det Hess L 1 (x a )| {grad L 1 (x a ) = 0, y a } p a=1 .
(7.28)

The gradient and Hessian at x a live in the tangent plane to the sphere at x a , identified with R n-1 . Note that the {y a µ } are Gaussian variables with zero mean and covariance E[y a µ y b ν ] = δ µν q ab , with q ab ≡ x a • x b the "overlap" between replicas a and b. We introduce the variables {q ab } via delta functions in eq. (7.28): Since we fixed the {q ab }, the distribution of the {y a } is fixed, as well as the joint distribution of the loss, gradients and Hessians, as we will explicit in the following. As the number of overlap variables is p(p -1)/2 = O n (1), we can apply Laplace's method over the variables {q ab } in the thermodynamic limit. As we explained in Section 1.3.1, we can make a replica-symmetric ansatz: It amounts to assume that, once Laplace's method is performed, the extremizing {q ab } satisfy q aa = 1, q ab = q for a = b. Assuming this structure of the overlap matrix allows to extend the expression of the moments to arbitrary non-integer p, and then to take the p ↓ 0 limit as needed in the replica method. Note that while the RS structure is correct in many cases, it is also a very good approximation in others where replica symmetry is broken, which gives relevance to our RS calculation even in this case. We now detail the computation of the three factors of eq. (7.29), and their limit as p ↓ 0 and n → ∞.

E Crit n,L 1 (B) p =

The phase volume factor

Let us first compute the phase space factor in eq. (7.29). More precisely, the term:

p a=1 µ S (dx a ) a<b δ(q ab -x a • x b ) = n -p(p-1) 2 p a=1 R n dx a a≤b δ(nq ab -nx a • x b ),
in which we defined q aa = 1. As we detail in Appendix E.1.1 we reach, when p ↓ 0 and n → ∞:

1 np ln p a=1 R n dx a a≤b δ(nq ab -nx a • x b ) 1 2 ln 2π n + 1 2 1 1 -q + ln(1 -q) . (7.30)
The joint density of the gradients

We will now compute the joint density of the gradients at {x a }, conditioned on the values of {y a }. This calculation is an extension of Sections V.C and V.E of [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF]. We consider two vectors x a and x b of overlap q ab = q. It is easy to see that E[grad L(x a )|{y b } p b=1 ] = 0 from eq. (7.14b), so we will focus on the covariance matrix E[grad L(x a )grad L(x b ) |{y c } p c=1 ]. After some calculations detailed in Appendix E.1.2 we get the gradient density at leading exponential order:

ϕ {grad L 1 (x a )} p a=1 |{y a } (0) a =b δ 1 m m µ=1 φ (y a µ ) z p (q)y a µ + f 0 p (q)y b µ + f p (q) c( =a,b) y c µ × exp np 2 ln m 2π - n 2 ln det 1 m m µ=1 φ (y a µ )φ (y b µ ) 1≤a,b≤p , (7.31)
in which the auxiliary functions (z p (q), f p (q), f 0 p (q)) are explicit, and defined in eq. (E.3).

Factorization of the mean product of determinants

The argument of this section is very close to Section V.F of [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF]. We consider the term:

E p a=1
| det Hess L 1 (x a )| {grad L 1 (x a ) = 0, y a } p a=1 .

(7.32)

We make two important remarks, which are straightforward transpositions of the arguments of [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF] to our problem, and we refer to this work for more extensive physical justifications.

• The conditioning over the gradients being zero, similarly to what we showed in the annealed calculation, only gives a finite-rank change to the Hessians Hess L 1 (x a ) and thus does not modify the behavior of the determinant in the scale e Θ(n) . In this scale, the statistics of the p matrices {Hess L 1 (x a )} p a=1 are identical.

• For each a, the spectral measure of Hess L 1 (x a ) concentrates at a rate at least n 1+ for a small enough > 0 (we expect that the actual rate is n 2 ). This argument is very similar to what we used to prove Lemma 7.10: this implies that at the order e Θ(n) the expectation value factorizes over the replicas, and we can assume all the Hessians to be independent.

Before stating the consequences of such remarks, we give some definitions:

• µ G,q is the Gaussian probability measure on R p with zero mean and covariance E[X a X b ] = (1 -q)δ ab + q. Note that {y µ } m µ=1 are i.i.d. variables distributed according to µ G,q .

• We define ν y as the empirical measure of (y 1 , • • • , y m ), that is ν y ≡ (1/m) µ δ y µ . For every a, we denote ν a y its marginal distribution: ν a y (dλ a ) ≡ b( =a) ν y (dλ). Then ν a y is also the empirical distribution of (y a µ ) m µ=1 .

Our remarks show that we can use the results of the annealed calculation, and that the expectation of the determinants factorizes at leading exponential order:

E p a=1
| det Hess L 1 (x a )| {grad L 1 (x a )} p a=1 = 0, {y a } e n p a=1 κ α,φ ν a y ,t φ (ν a y ) .

(7.33)

As for the annealed case, the concentration behind eq. (7.33) has recently been shown for a very large of class of random matrices by collaborators [START_REF] Ben Arous | Exponential growth of random determinants beyond invariance[END_REF] (eq. (1.2) of this work is precisely equivalent to our eq. (7.33)).

Decoupling replicas and the p ↓ 0 limit

We can then apply Sanov's theorem 1.9 to the empirical measure ν y ∈ M + 1 (R p ). Recall that we have constraints on this measure by the density of the gradient and the fixation of the energy level. More precisely, we denote M (p) φ (q, B) the set of probability measures on R p that satisfy the following:

         ν(dλ) φ(λ a ) ∈ B ∀1 ≤ a ≤ p, (7.34a) ν(dλ) φ (λ a ) z p (q)λ a + f 0 p (q)λ b + f p (q) c( =a,b) λ c = 0 ∀1 ≤ a = b ≤ p. (7.34b)
Recall that the functions (z p (q), f 0 p (q), f p (q)) are defined in eq. (E.3). Leveraging from the results of eqs. (7.30), (7.31) and (7.33), we obtain from Sanov's theorem 1.9 and Varadhan's lemma 1.10:

lim n→∞ 1 n ln E [Crit n,L 1 (B) p ] = p 2 ln α + sup q∈(0,1) sup ν∈M (p) φ (q,B) p 2 1 1 -q + ln(1 -q) (7.35) - 1 2 ln det ν(dλ)φ (λ a )φ (λ b ) 1≤a,b≤p + p a=1 κ α,φ ν a , t φ (ν a ) -αD KL (ν|µ G,q ) .
Recall that ν a is the marginal distribution of ν for the variable λ a . We can then decouple the replicas under an assumption on the measure ν that amounts for replica symmetry. We stress that this replica symmetric assumption in the Kac-Rice calculation actually corresponds to a 1-step replica symmetry breaking (1RSB) structure of the zero-temperature Gibbs measure, that is an exponential number of single-point metastable states that all have the same two-point overlap. While possibly not exact, this assumption should already yield a good approximation to the landscape, and could be analytically checked by studying the stability of the replicasymmetric ansatz within replica theory. This allows to take subsequently the p ↓ 0 limit, and after some simplifications, we reach from eq. (7.35) the expression of Result 7.1. These steps are fairly technical, and are postponed to Appendix E.1.3.

Discussion on Chapter 7

In this chapter, we obtained analytical results for the annealed and quenched complexities of statistical models with non-Gaussian loss functions arising in generalized linear estimation and simple models of glasses and neural networks. Our method is versatile and can be easily extended to other cases, as we will discuss below.

Our results allow for a complete characterization of the empirical loss landscapes of generalized linear models. The main issue ahead is determining for which class of functions φ and in which regimes (e.g. values of α), the annealed and quenched complexities become positive, i.e. when the associated landscape is rough, and if they differ, i.e. if the number of critical points is concentrating as the dimension grows large. These investigations will surely require a systematic and extensive numerical evaluation of our results, and will allow to study the connection between landscape properties and dynamics induced by local algorithms. In particular, it will shed light on the relationship between the roughness of the empirical loss landscape and the existence of "hard" phases in the learning of generalized linear models [BKM + 19]. It will also provide an interesting benchmark for obtaining the algorithmic thresholds of gradient descent (and variants) only through the knowledge of the landscape properties [SMBC + 19, SMBC + 20b, BAGJ21].

Based on ongoing works, we can for instance conjecture the existence of a rough landscape for small enough α in phase retrieval [START_REF] Lucibello | Generalized approximate survey propagation for high-dimensional estimation[END_REF] and retarded learning [START_REF] Engel | Statistical mechanics of learning[END_REF]. Addressing these questions is an exciting direction of research, and a natural follow-up to this chapter that is under investigation.

Counting the minima -Another important extension of this chapter consists in counting the critical points of a fixed index (i.e. with a fixed number of negative directions in the spectrum of the Hessian). This would provide additional interesting information, in particular it would allow to differentiate local minima from the other critical points of the landscape, as we did for the p-spin model in Section 7.1. As we mentioned in this spin glass calculation, such a counting would require to understand the large deviations of the eigenvalues of the Hessian arising for generalized linear models, i.e. random matrices of the type of eq. (7.13). Such a random matrix problem is hard, but in Chapter 8 we present a technique that builds on recent developments and that allows to compute the rate function of the large deviations of the smallest eigenvalue. This allows then to use our Kac-Rice calculation to count solely minima of the loss! Generalization to other models -Our calculations, both annealed and quenched, can be generalized straightforwardly to many other loss functions and models. As is clear for instance in the annealed computation of Section 7.3, the key features that must be present are:

1. A Gaussian distribution of the data ξ µ .

2. A loss function L(x) that only depends on the data samples ξ µ via their projection over a few vectors (e.g. x for L 1 (x) and x, x for L 2 (x)).

We give thereafter three examples of models, that can be found in [START_REF] Engel | Statistical mechanics of learning[END_REF][START_REF] Mei | The landscape of empirical risk for nonconvex losses[END_REF], and in which our calculations can be easily performed: ∼ N (0, I n ) . We want to learn the vector θ 0 ∈ S n-1 by minimizing the following loss function:

L(θ) ≡ 1 2m m µ=1 [y µ -σ(θ • x µ )] 2 , θ ∈ S n-1 . (7.36)

Model 7.2 (Mixture of two Gaussians)

Consider n, m ≥ 1 such that m/n → α > 1. We are given m samples y µ ∈ R n , generated as

y µ i.i.d.
∼ 2 a=1 p a N (θ 0 a , I n ). The proportions p 1 , p 2 are known, and we wish to recover θ 0 1 and θ 0 2 by minimizing the maximum-likelihood estimator:

L(θ 1 , θ 2 ) ≡ - 1 m m µ=1 ln 1 2 a=1,2 1 √ 2π n exp - 1 2 y µ -θ a 2 .
(7.37)

Model 7.3 (Simple unsupervised learning model)

Consider n, m ≥ 1 such that m/n → α > 1. Let φ : R → R a smooth activation function, V : R → R + a "potential", and x 0 ∈ S n-1 a fixed vector. We assume that we are given i.i.d. data samples {ξ µ } m µ=1 ∈ R n distributed such that their projection on x 0 has a probability density

P (ξ µ • x = h) ∝ e -1 2 h 2 -V (h)
, and the other coordinates of ξ µ are i.i.d. standard Gaussian variables. We wish to recover the vector x 0 by minimizing:

L(x) ≡ 1 m m µ=1 φ(ξ µ • x), x ∈ S n-1 . (7.38)
For each of these three models one can easily replicate the annealed and quenched calculations of Sections 7.3 and 7.5, under suitable technical hypotheses. As a final note, it is however an open problem to generalize our methods to neural network models with many nodes and hidden layers; the random matrix analysis of the Hessian in this case is a particularly exciting challenge.

A note on non-spherical priors -It is clear from the calculation of Appendix E.1 (particularly Section E.1.1) that we can also generalize our techniques (at least on the heuristic level) to nonspherical prior distributions on the vectors x. The most natural hypothesis that allows the computation to be generalized is that the prior distribution takes the decoupled form P (dx) = i P (dx i ). Going further: discrete systems -In a series of recent fascinating papers [Sub21, Mon21, AMS20, AM20, Sel21], the authors developed polynomial-time algorithms that can provably optimize a large class of discrete and continuous disordered systems (such as the p-spin on the hypercube). However, the very notion of "critical points" is generally not well-defined in discrete problems, thus preventing from using the Kac-Rice method as we did in this chapter. Looking for topological invariants of the landscape to characterize the optimal algorithmic performance in discrete models is therefore an open and exciting research direction. First explorations have been performed using the Kac-Rice formula on the corresponding continuous TAP landscape (cf. Section 1.3.2), cf. e.g. for Ising spins [START_REF] Chen | The generalized TAP free energy II[END_REF][START_REF] Zhou Fan | TAP free energy, spin glasses and variational inference[END_REF].

Erratum to the proof -Finally, note that the proof of Theorem 7.5 that we presented here has a minor modification with respect to the one published in [START_REF] Maillard | Landscape complexity for the empirical risk of generalized linear models[END_REF], as we uncovered a small missing step in the proof. This addition will soon be present in the published version of this work as well.

Chapter 8

An excursion to large deviations in random matrix theory "On voit par cet Essai, que la théorie des probabilités n'est au fond, que le bon sens réduit au calcul : elle fait apprécier avec exactitude ce que les esprits justes sentent par une sorte d'instinct, sans qu'ils puissent souvent s'en rendre compte." Pierre-Simon de Laplace, Essai philosophique sur les probabilités (1814).

Disclaimer -In this chapter we present an analytical technique to compute the probability of rare events in which the largest eigenvalue of a large class of random matrices, known as generalized sample covariance matrices, is atypically large (i.e. the right tail of its large deviations). The results also transfer to the left tail of the large deviations of the smallest eigenvalue. In particular, these include the Hessian of the loss of the empirical risks L 1 and L 2 of eqs. (7.9),(7.10). As we mentioned in Section 7.1, the calculation of these large deviations is primarily motivated by the ability to compute the complexity of local minima in the class of inference models studied in Chapter 7. Moreover we will detail below other strong motivations arising from theoretical statistics to study these large deviations. This chapter is based on the published work [Mai21]. While it revolves around subjects very close to Chapter 7, it is written in a style much closer to the theoretical physics literature; in particular, we will often not use theorems to designate our results. However, given the nature of the technique we use, which originated in the mathematics literature, we expect that a rigorous proof would follow exactly the steps taken here.

Why the large deviations of the eigenvalues?

The landscape of generalized linear models and variants

The first motivation behind our study naturally comes from our results of Chapter 7, and the Kac-Rice formalism developed there. Recall that this chapter focused on the study of the complexity of the empirical risk landscape of generalized linear models. One of the most important extensions of this chapter would be to restrict the counting of critical points to local minima, which are more representative of the actual roughness of the landscape from the point of view of local optimization algorithms.

As the Kac-Rice formula turns the counting of the complexity into the random matrix analysis of the Hessian, we saw that conditioning a critical point to be a local minimum naturally requires to understanding very rare events in which the smallest eigenvalue of the Hessian has extremely atypical value, i.e. its large deviations (as defined in Section 1.5.2). Exploring precisely these inference landscapes is an important open problem for the disordered systems and statistical learning communities, as the traditional methods have been limited to simpler Gaussian models (see e.g. [MBM18, SMBC + 19, SMKUZ19, RBABC19] and many other references given in Chapter 7), and the large deviations results of the present chapter are an important step in this exciting direction.

PCA for correlated data

On the other hand, a textbook example of the interplay between theoretical physics and statistics (that fuels this thesis) is principal components analysis (PCA), a statistical estimation method based on random matrix theory, and applied in fields as diverse as image compression [DF07, ZDZS10, AW10, Fuk13], neurosciences [START_REF] Viktor K Jirsa | A theoretical model of phase transitions in the human brain[END_REF][START_REF] Brenner | Adaptive rescaling maximizes information transmission[END_REF], genetics [START_REF] Reich | Principal component analysis of genetic data[END_REF], or finance [START_REF] Bouchaud | Theory of financial risks[END_REF]. In Chapter 5 we studied variants of this problem through the prism of generative models.

To fix our ideas, let X ∈ R m×n be the data matrix, whose columns {x i } n i=1 are observations independently drawn from a Gaussian distribution N (0, Γ). PCA aims at discovering a "principal component" eigenspace of the covariance matrix Γ by studying the largest eigenvalue of the sample covariance matrix C n ≡ i x i x i /n: indeed, a strong outlier eigenvalue in Γ typically induces a corresponding outlier in C n [EJ76, BBAP05, BGN11] (see as well Chapter 5).

Pioneering physics works addressed the general question "How good is PCA ?" [START_REF] David | Large deviations of extreme eigenvalues of random matrices[END_REF][START_REF] Satya | Large deviations of the maximum eigenvalue for Wishart and Gaussian random matrices[END_REF]. Precisely, they wished to understand if an outlier can appear in C n even if there is no structure to uncover in Γ: this "null hypothesis" provides a way to gauge the significance of results obtained on a real-world dataset. As we defined in Section 1.5.2, such atypical events are known as large deviations, and the mentioned works, as well as subsequent ones, had to restrict to uncorrelated data, in which Γ is the identity matrix (or a finite-rank perturbation of it) [DM06, Mai07, VMB07, MV09, MS14, BG20]. Realistic data (e.g. a natural image) indeed contain non-trivial correlations that the Coulomb gas analysis used in [START_REF] David | Large deviations of extreme eigenvalues of random matrices[END_REF][START_REF] Satya | Large deviations of the maximum eigenvalue for Wishart and Gaussian random matrices[END_REF] is not equipped to handle. While data structure is a key ingredient of learning and inference (see [START_REF] Zdeborová | Understanding deep learning is also a job for physicists[END_REF] and our discussion in Chapter 5), probing the statistical significance of PCA on correlated data remained an open question. In this regard, the present chapter addresses and solves this long-lasting problem for arbitrary Γ, i.e. PCA with correlated data.

Organization of the chapter

We begin in Section 8.2 by defining precisely the class of matrices we consider, and we recall some known results on the behavior of its asymptotic spectrum. The main contribution of this chapter is Result 8.1, which gives the rate function of the large deviations of the largest eigenvalue of the class of matrices we consider. Our derivation is based on a tilting method, developed in a series of recent mathematical works [BG20, GH20, Hus20, BGH20, [START_REF] Augeri | Large deviations for the largest eigenvalue of sub-Gaussian matrices[END_REF][START_REF] Mckenna | Large deviations for extreme eigenvalues of deformed Wigner random matrices[END_REF]. This technique is more adaptable than a more traditional Coulomb gas analysis, as it does not require the joint probability of the eigenvalues of the matrix, which is unknown here. Moreover, the calculation does not rely on any heuristics, and we therefore expect it to be adaptable into mathematically rigorous statements. In Section 8.3 we numerically probe our results using an importance sampling Monte-Carlo approach, effectively simulating events with probability as small as 10 -100 . Finally, Section 8.4 is devoted to the detailed derivation of the rate function, i.e. of Result 8.1.

Large deviations of extreme eigenvalues of generalized sample covariance matrices

Some formal definitions and assumptions

To state our main result, we need first to define some important mathematical concepts. Bulk density σ(λ)

0 4 t 0 ρ(t) ρ(t) d max σ(λ) s max x c (ρ) G σ (x) G σ (x) 0.2 0.4 0.6 Functions G, G ∆ ∼ n -2/3 ∆ ∼ O(1)
"Bulk" ∆ Figure 8.1: The bulk σ(λ), and the functions G σ , G σ for α = 2 and ρ(t) the Marchenko-Pastur law with ratio 1. In the box, we plot ρ(t) and the right edge d max of its support. The black arrow is an outlier in the spectrum of H n , and ∆ is the gap between this outlier and the bulk σ(λ). As we will see, the rate function of the large deviations of λ max (H n ) is directly proportional to the area between G σ and G σ .

Bulk density and Marchenko-Pastur equation

Letting x i = √ Γz i with z i ∼ N (0, I n ), one can see that C n has the same eigenvalues (up to possible zeros and a scaling factor) as H n ≡ Z ΓZ/m. The "bulk" of H n -i.e. the large n limit of its eigenvalue density, or LSD -is denoted σ(λ):

dλ σ(λ) f (λ) = lim n→∞ 1 n n i=1 f [λ i (H n )].
The LSD of Γ is in turn denoted ρ(t). We illustrate these different quantities in Fig. 8.1, when ρ(t) is the Marchenko-Pastur law with ratio one. The other functions pictured in Fig. 8.1 will be introduced later on.

Importantly, σ(λ) can be analytically derived using the Stieltjes transform of random matrix theory, that we introduced in Section 1.5. Here we adopt a slightly different convention than in the rest of this thesis, namely we will consider:

G σ (x) ≡ -S σ (x) = dλ σ(λ) x -λ .
Assuming that m/n → α, the Marchenko-Pastur equation [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF] (Theorem 1.7) gives the inverse of G σ :

G -1 σ (ω) = 1 ω + α dtρ(t) t α -tω . (8.1)
The bulk density σ(λ) is then determined via the Stieltjes-Perron inversion formula (Theorem 1.5). In particular, the support of σ(λ) and its right edge s max can be computed (analytically or numerically) from eq. (8.1).

Generalized sample covariance matrices

By rotation invariance of Z, one can diagonalize Γ, i.e. assume Γ = Diag({d µ }), with all d µ ≥ 0, which implies that:

H n ≡ 1 m m µ=1 d µ z µ z † µ , (8.2)
in which z µ i.i.d.

∼ N (0, I n ). This form leads us to further extend the random matrix model we consider. More precisely, given a set of variables d µ ∈ R (not necessarily positive), we call matrices of the type of eq. (8.2) generalized sample covariance matrices. We can furthermore allow the z µ to be real or complex Gaussian variables, and recall that we consider the limit n, m → ∞ with m/n → α > 0. The limit of the empirical distribution of {d µ } m µ=1 is denoted ρ(t), and we define d max ∈ R ∪ {+∞} the right edge of its support.

Importantly, the positivity (or negativity) of the matrix H n is equivalent to the positivity (or negativity) of all d µ . Coherently with our motivation, these matrices (with arbitrary d µ ∈ R) contain, up to a shift, the Hessian matrix of eq. (7.13) arising in the computation of the complexity of the empirical risk of generalized linear models via the Kac-Rice formula.

Finally, we note that generalized sample covariance matrices are precisely the class of matrices covered by the generic form of the Marchenko-Pastur equation (Theorem 1.7).

The functions G σ , G σ

By monotonicity arguments, it can easily be seen that the equation G -1 σ (ω) = x, with G -1 σ given by eq. (8.1), can have more than one solution! More precisely, it can have exactly one other solution ω = G σ (x), sometimes referred to as the "second branch" of the Marchenko-Pastur equation. In Fig. 8.1, we show an example of functions (G σ , G σ ). These functions satisfy the following properties 1 :

• G σ is decreasing, while G σ is non-decreasing. • G σ (x) ∼ x→∞ x -1 . • lim x↓smax G σ (x) = lim x↓smax G σ (x).
• For d max ∈ (0, +∞) (and therefore nonnegative H n ), let:

x c (ρ) ≡ d 2 max G ρ (d max ) + (α -1 -1)d max ∈ (s max , +∞]. (8.3)
Here, G ρ (z) ≡ -S ρ (z) = dt ρ(t)/(z -t) is the (negative of the) Stieltjes transform of ρ(t).

If x c (ρ) < +∞, then the equation G -1 σ (ω) = x has a single solution for x > x c (ρ), which is G σ (x). In this case, we define G σ (x) ≡ α/d max for x ≥ x c (ρ). One can check that this ensures that G σ (x) is continuous in x c (ρ): we will say that G σ saturates at the point x c (ρ). An example of this saturation point is pictured in Fig. 8.1.

• If H n is negative (i.e. d max ≤ 0), then lim x↑0 G σ (x) = +∞.
In this case, we set G σ (x) = +∞ for x ≥ 0.

1 The derivation of these properties is straightforward and left to the reader.

Main result

From now on we will restrict to the study of λ max (H n ). Since we can always consider d µ = -d µ , our analysis also applies to λ min (H n ), so that we do not lose any generality. As we know, the large deviations regime corresponds to macroscopic changes in λ max (H n ), which are exponentially rare, as opposed to the typical fluctuations, which are generically in the scale n -2/3 for such random matrices [TW94, Joh01, DY20]: these two regimes are shown as cyan and grey regions in Fig. 8.1. Crucially, we assume Hypothesis 8.1 ("No outliers hypothesis")

Recall that we denote ρ(t) the empirical distribution of {d µ } m µ=1 , and d max the right edge of the support of ρ(t). We assume that d max < ∞, and moreover:

lim m→∞ max 1≤µ≤m d µ = d max
In other words, there is no outlier in the list {d µ }.

Importantly, Hypothesis 8.1 ensures that λ max (H n ) converges to the right edge s max of the bulk σ(λ). This implies that the set of {z µ } such that the spectrum of H n has an outlier is very atypical under the Gaussian distribution. Let us now state our main result.

We adopt a standard notation, used e.g. in Chapters 2,6: we let β ∈ {1, 2} for respectively real and complex z µ , with the convention E|z| 2 = 1 for a Gaussian standard random variable. We state the following result under all aforementioned hypotheses.

Result 8.1 (Large deviations of the largest eigenvalue of H n )

Recall that the variables {d µ } are given. Under the randomness of {z µ } m µ=1 , the law of λ max (H n ) satisfies a large deviation principle, in the scale n, with rate function I(x) given by:

I(x) =      +∞ if x < s max β 2 x smax [G σ (u) -G σ (u)]du if x ≥ s max .
Result 8.1 is the main result of this chapter, and will be derived in Section 8.4. In Fig. 8.2, we show analytical computations of the rate function I(x) for different α and ρ(t). As we mentioned in the introduction of this chapter, we state this LDP as a result rather than a theorem, as its publication was made in a physics journal [Mai21]. As we will see however, we expect all arguments to transfer into a rigorous proof without significant changes. Let us now draw some first consequences of Result 8.1.

Consistency with previous results

Importantly, in the white Wishart case -i.e. ρ(t) = δ(t -1) -such large deviations were already analyzed in e.g. [START_REF] Satya | Large deviations of the maximum eigenvalue for Wishart and Gaussian random matrices[END_REF][START_REF] Biroli | Large deviations for the largest eigenvalues and eigenvectors of spiked Gaussian random matrices[END_REF]. Our result should therefore be consistent with their findings. As detailed in Appendix E.3, Result 8.1 indeed reduces in this case to the previously-known expression:

I(x) = αβ 2 x λ + (α) (u -λ + (α))(u -λ -(α)) u du,
with λ + (α) ≡ (1 + α -1/2 ) 2 , and x ≥ λ + (α).

A phase transition in the rate function

Let us describe a first notable consequence of Result 8.1. We assume that d max > 0 and that x c (ρ) < +∞. For instance, this is true if ρ is the Marchenko-Pastur law, as shown in Figs. 8.1 and 8.2.

We saw that in this case G σ (x) saturates at α/d max for x ≥ x c (ρ). It is in general not smooth at x = x c (ρ) and this singularity induces a phase transition in the rate function I(x). The order of the transition -i.e. the order of the first discontinuous derivative of I(x) -can be computed assuming the right tail of ρ(t) behaves as ρ(t) ∼ t↑dmax (d max -t) η , with η > 0, so that x c (ρ) < ∞. When η ≥ 1 and 1/2 ≤ η < 1 (e.g. the Marchenko-Pastur law, for which η = 1/2) we show that the transition is respectively of second and third order2 . The details are given in Appendix E.4, and we conjecture generically the order of the transition to be k + 1 if 1/k ≤ η < 1/(k -1).

Monte-Carlo simulations

While Result 8.1 is in essence high-dimensional, this section is devoted to numerically probe its predictions. As we will see, even at moderately large n we are able to recover the rate function we computed analytically! While this regime is not directly relevant to the Kac-Rice computations of Chapter 7 (which are analytical results in the large n limit), it is the appropriate regime to investigate correlated data in PCA, which is our second main motivation for this study.

Importance sampling

Since we need (1/m) µ δ(t -d µ ) to be very close to ρ(t) (and thus large enough m), we can not perform simple histograms of λ max (H n ), as e.g. in [START_REF] Satya | Large deviations of the maximum eigenvalue for Wishart and Gaussian random matrices[END_REF], since the large deviations probability decays exponentially in n. Instead, we will modify (or "tilt") the law of z so that it favors large deviations, a technique which is known as importance sampling [START_REF] Bucklew | Introduction to rare event simulation[END_REF]. This powerful Monte-Carlo method allows to numerically access the tails of a given high-dimensional probability distribution and has been successfully applied to various problems across the physical sciences, from random graphs [START_REF] Engel | On large deviation properties of Erdös-Rényi random graphs[END_REF] to simulations of the height distribution in the Kardar-Parisi-Zhang equation [HLDM + 18], and random matrices [START_REF] Tobin | Searching for rare growth factors using multicanonical monte carlo methods[END_REF][START_REF] Saito | Multicanonical sampling of rare events in random matrices[END_REF], as in this chapter. For a more exhaustive description of the applications of importance sampling in physics, we refer the reader to [HLDM + 18].

Tilting the measure

Let us now detail our approach in detail. For the purpose of the presentation we will restrict to the real case β = 1, while all subsequent arguments can straightforwardly be adapted to the complex case. Recall that we denote Dz ≡ dz e -||z|| 2 /2 /(2π) n/2 the standard Gaussian law. We will tilt this distribution by explicitly giving more weight to configurations having a larger λ max (H n ), making them more probable. More precisely, we aim at sampling from the distribution P t (z)dz ∝ Dz e ntλmax (Hn) .

(8.4)

For a given t ≥ 0, Result 8.1 and Laplace's method imply that when sampling z under P t , the largest eigenvalue of H n concentrates on

x (t) ≡ arg min x≥smax [tx -I(x)]. (8.5)
One sees clearly now that sampling from the tilted distribution of eq. (8.4) gives information about the Legendre transform of the large deviations function I(x).

The Monte-Carlo algorithm

We implement a classical Metropolis-Hastings algorithm in order to sample from P t . The physical (given) parameters are n, m, ρ, t, and we generate i.i.d. samples {d µ } m µ=1 from ρ. We pick two hyperparameters ∆, β d > 0 (we will later detail how to fine-tune them). We initialize {z µ } m µ=1 as standard Gaussian vectors, and we sample from the move proposal distribution g(z |z) as follows:

(i) Pick a random index µ ∈ {1, • • • , m} with probability P (µ) ∝ e β d dµ .

(ii) Draw a uniform vector e ∈ S n-1 ( √ n), and draw L ≥ 0 from a truncated Gaussian distribution centered in 1 and with variance ∆ > 0. Let z µ = √ Le.

(iii) The new state is given by changing z µ → z µ .

We impose the detailed balance condition with stationary distribution P t (z) and move proposal distribution g(z |z) in the MCMC. We measure the largest eigenvalue λ max (H n ), which we then compare to x (t). The parameters (β d , ∆) are found to reduce greatly the equilibration time of the Markov chain, and are adapted during a warmup phase to obtain an acceptance ratio in the range [0.2, 0.3]. Physically, β d can be seen as favoring changes close to the right edge of the bulk, while ∆ favors large norms of z µ , more likely to induce macroscopic changes in the largest eigenvalue.

Results of the experiments

The code is available in a public Github repository [Mai21] and the results of the simulations are given in Fig. 8 whether H n is negative, positive, or neither. Even though the variability of the results naturally increases with t, we are able to access very large values of x (t), beyond the transition point x c (ρ) when it exists (cf. cases (ii) and (iii)). For example, in (iii) we are able to sample up to x (t) 8. Comparing with Fig. 8.2, this implies that our simulations reach events which have probability of order e -0.5n ∼ 10 -109 under a naive sampling! We were therefore able to probe Result 8.1 deep into the large deviations regime.

Derivation of the rate function

In this section, we derive Result 8.1. We will focus on the real case β = 1, and briefly describe at the end how to generalize our arguments to the complex case.

General idea behind the method

Let x ≥ s max , and let us denote P n (x) the PDF of λ max (H n ). Our aim is to compute the asymptotics of n -1 ln P n (x), i.e. the probability of exponentially rare events in which λ max (H n ) is close to x rather than to its typical value s max .

The main idea of the method is to tilt the probability measure of H n so that having λ max (H n ) x becomes a typical event, rather than an exponentially rare one. This new tilted law will be parametrized by a number θ ≥ 0: for each θ, the largest eigenvalue will typically be close to a value x(θ) as n gets large (very similarly to the importance sampling strategy used for Monte-Carlo simulations, cf. eq. (8.4)). Conversely, each x ≥ s max will be associated to a θ x ≥ 0, and a tilting parametrized by θ x will typically induce the largest eigenvalue to be close to x.

As we will see, these functions {x(θ), θ x } contain all the information about the large deviations we want to compute. To put it roughly, studying how much tilt we need to apply in order to push the largest eigenvalue from s max to x will give us information on what was the probability that this eigenvalue was close to x in the first place. On a general note, the tilting strategy is a privileged approach to prove many results in large deviations theory, e.g. Cramer's theorem 1.8 [START_REF] Dembo | Large Deviations Techniques and Applications[END_REF]. However, its adaptation to the random matrix context, using the spherical integrals we introduced in Section 1.5.3, is very recent and was introduced by the series of mathematical works mentioned above.

Tilting the measure: a first attempt

The tilted law of H n

We start with a first simple use of the tilting method using spherical integrals3 . The simplest possible tilting of the measure, inspired by the aforementioned mathematical works, is an exponential tilting. More precisely, we define the tilted distribution of z as4 : P θ,e (z)dz ∝ Dz exp θn 2 e H n e , (8.6) for a given vector e ∈ S n-1 and a parameter θ ≥ 0. As we will see, this tilting induces a macroscopic move of the largest eigenvalue. Moreover, by rotation invariance of the distribution of H n , this move only depends on θ, and not on the specific direction of e.

Let us now detail the distribution of H n under the tilted law of eq. (8.6). Computing the normalization factor, we reach:

I n - θ α d µ ee -1/2 = I n + (1 -θd µ /α) -1/2 -1 ee . (8.8)
Changing variables to z µ ≡ I n -θd µ ee /α 1/2 z µ in eq. (8.7) and using eq. (8.8) yields that H n is distributed under P θ,e as:

H (e,θ) n d = 1 m m µ=1 d µ [I n + κ θ (d µ )ee ]z µ z µ [I n + κ θ (d µ )ee ], (8.9)
with κ θ (t) ≡ (1 -α -1 θt) -1/2 -1, and in which z µ are again i.i.d. standard Gaussian vectors.

Since H (e,θ) n is a finite-rank change of H n , its largest eigenvalue can be typically larger than s max . Moreover, we see from the expression of H (e,θ) n that as θ → α/d max , some of the coefficients κ θ (d µ ) will grow very large: we thus expect that for sufficiently large θ, an outlier eigenvalue will pop out from the right edge of the "bulk", as pictured as a black arrow in Fig. 8.1. We denote x(θ) ≥ s max the typical value of this outlier eigenvalue (i.e. of λ max (H n )), as n → ∞. As we mentioned, x(θ) does not depend on e by rotation invariance.

Relating the tilting and the original problem

Let us now see how to relate the PDF P n of λ max (H n ) to this tilted distribution. For any θ, we can write the trivial identity: We introduced the spherical integral

P n (x(θ)) = Dz δ(λ max (H n ) -x(θ)) = Dz δ(λ max (H n ) -x(θ)) e 2 =1
J n (H n , θ) ≡ 1 n ln e 2 =1
de e nθ 2 e Hne .

(8.11)

In eq. (8.10), we almost see the tilted probability distribution of eq. (8.6) appearing. However this is not exactly the case, as the term e nJn(Hn,θ) depends on z via H n . If this term would not depend on z at leading exponential order in n, the law of z would be the tilted law of eq. (8.6), and we could remove the δ term in eq. (8.10): indeed, the constraint λ max (H n ) x(θ) would already be satisfied by the very definition of x(θ)!

The spherical integrals

Therefore, we study first J n (H n , θ). Let us introduce J 1 (θ, x), defined as the limit of J n (H n , θ), assuming λ max (H n ) → x as n → ∞ (which we can safely assume because of the constraint in eq. (8.10)). Adopting the language of statistical physics, we call J 1 a quenched spherical integral. More precisely, J 1 belongs to a class of high-dimensional integrals known as Harish-Chandra-Itzykson-Zuber (HCIZ) integrals [START_REF] Harish-Chandra | Differential operators on a semisimple Lie algebra[END_REF][START_REF] Itzykson | The planar approximation[END_REF], that we already introduced and studied in Section 1.5.3. In particular, Theorem 1.12 yields:

J 1 (θ, x) = inf γ>θx γ 2 - 1 2 du σ(u) ln(γ -θu) - 1 2 . (8.12)
Let us come back to eq. (8.10). We have at leading exponential order:

P n (x(θ)) e 2 =1
de Dz δ(λ max (H n ) -x(θ)) e θn 2 e Hne e nJ 1 (θ,x(θ)) , e -nJ 1 (θ,x(θ))

e 2 =1
de Dz e θn 2 e Hne . (8.13)

As already argued, we removed the δ constraint in eq. (8.13) by definition of x(θ): under the tilted law P θ,e , the largest eigenvalue λ max (H n ) typically concentrates on x(θ), so this constraint is superfluous. The expression of eq. (8.13) involves another integral, that we call annealed and denote F n (θ), borrowing again from the statistical physics jargon:

F n (θ) ≡ 1 n ln Dz e 2 =1
de e θn 2 e Hne .

Similarly to J n , we denote by F 1 (θ) the limit of F n (θ). If d max > 0, we also impose θ < α/d max so that F n (θ) is well-defined. We compute it by direct integration on z:

F 1 (θ) = - α 2 dtρ(t) ln(1 -α -1 θt). (8.14)
Combined with eq. (8.13), this implies

P n (x(θ)) exp{-n[J 1 (θ, x(θ)) -F 1 (θ)]}. (8.15)
Note that we imposed θ < θ max , with

θ max ≡ α/d max if d max > 0, +∞ otherwise.
Conversely, this implies that eq. (8.15) can only be applied for s max ≤ x < x max ≡ x(θ max ). This creates a possibly important limitation of the tilting we used, if x max is finite: in this case, the method does not give access to the large deviations for x ≥ x max ! We will precisely characterize when such a limitation occurs in the following, relating it to the phase transition phenomenon described in Section 8.2, and we will develop a second tilting to circumvent this issue.

Simplifying the rate function

First, let us focus on x < x max and show that we find Result 8.1. We can rewrite eq. (8.15) as:

1 n ln P n (x) -[J 1 (θ x , x) -F 1 (θ x )]. (8.16)
Recall that θ x is chosen exactly to be able to remove the delta constraint in eq. (8.10). However, for any θ ≥ 0, we can always write an equivalent to eq. (8.13), keeping the delta constraint:

P n (x) e 2 =1 de Dz δ(λ max (H n ) -x) e θ n
2 e Hne e nJ 1 (θ ,x) .

From here, we can upper bound P n (x) by discarding the delta constraint in this equation, which gives at leading exponential order:

1 n ln P n (x) -[J 1 (θ , x) -F 1 (θ )].
Combining this with eq. (8. 16), we see we can write the rate function I(x) n -1 ln P n (x) as:

I(x) = sup θ∈[0,θmax) [J 1 (θ, x) -F 1 (θ)]. (8.17)
We focus now on simplifying the rate function of eq. (8.17), to obtain Result 8.1. We need to study the behavior of the quenched integral J 1 of eq. (8.12). We recall here known results on J 1 (θ, x), as we already studied the behavior of such a function, and the possible transitions in it, in Section 1.5.3.

Cancelling the derivative with respect to γ in eq. (8.12) yields γ = γ ≡ θG -1 σ (θ). Plugging back this solution in eq. (8.12), and using the Marchenko-Pastur equation (8.1) yields that J 1 (θ, x) = F 1 (θ). We detail this derivation in Appendix E.5.1. However, note that γ is constrained to be smaller than θx. Therefore, the infimum in eq. (8.12) is reached in γ only for θ ≤ θ c (x), with θ c (x) ≡ G σ (x). At θ = θ c (x), J 1 (θ, x) undergoes a transition, as γ "saturates" at its limit value θx for θ ≥ θ c (x). All in all, we reach:

J 1 (θ, x) =        F 1 (θ) = - α 2 dtρ(t) ln(1 -α -1 θt) if θ ≤ G σ (x), θx -1 -ln θ 2 - 1 2 du σ(u) ln(x -u) if θ ≥ G σ (x).
(8.18)

Using eq. (8.18) in the result of eq. (8.17), it is then simple algebra to see that the maximum of J 1 (θ, x) -F 1 (θ) is reached in θ x = G σ (x). Differentiating the resulting expression yields

I (x) = G σ (x) -G σ (x) 2 ,
which justifies Result 8.1 in this case. The whole computation we just described is detailed in Appendix E.6.1.

Limitations of the tilting

As we mentioned, the tilting method we presented is not capable of predicting the large deviations for x ≥ x max = x(θ max ). As we showed that θ x = G σ (x), we can separate two cases:

• If d max ≤ 0, then θ max = +∞ by definition, and therefore x max = 0 since lim x↑0 G σ (x) = +∞. For x ≥ 0, G σ (x) = +∞ and so Result 8.1 is valid (indeed I(x) = +∞ since H n is negative).

In the end, our tilting allowed to compute the large deviations rate function I(x) for any x ≥ s max in this case.

• If d max > 0, then θ max = α/d max . Since G σ (x c (ρ)) = α/d max , this yields that x max = x c (ρ),
given by eq. (8.3). Therefore, we see that in this case, the condition for the tilting to be able to induce arbitrarily large outliers is x c (ρ) = +∞, i.e. G ρ (d max ) = +∞. As we saw, the finiteness of G ρ (d max ) is exactly the existence condition of a phase transition in I(x), which prevents the tilting from capturing all the large deviations.

Beyond the transition: a second tilting

Here, we briefly outline the method we use to go beyond the phase transition when d max > 0, to circumvent the limitation described above. As the method is extremely similar to the one we just described in detail, we will focus primarily on the main steps and quantities, while leaving some details to the reader. We change the tilt of eq. (8.6) to:

P θ,e,f (z) dz ∝ Dz exp θn √ m i,µ d µ e i z µi f µ , ( 8.19)
with i e 2 i = µ f 2 µ = 1. When d µ ≤ 0, we define d µ ≡ i -d µ so that the tilt is possibly complex-valued. Eq. (8. 19) corresponds to a simple additive shift of z µ , and the law of H n under the tilt of eq. (8. 19) is:

H (θ,e,f) n ≡ 1 m m µ=1 d µ z µ z µ + θ 2 m α 2 d 2 µ f 2 µ ee + θ √ m α 1 {dµ≥0} d 3/2 µ f µ (ez µ + z µ e ) .
Let us give an intuitive view of the reasons why this new tilting manages to induce the largest eigenvalue to be typically close to x, for any x ≥ s max :

• When θ = 0 the largest eigenvalue of the unspiked matrix naturally concentrates on s max .

• As θ 1, a spike proportional to θ 2 will push the largest eigenvalue of H (θ,e,f) n to +∞.

By continuously varying θ, we see that the tilt should be able to induce any outlier x ≥ s max in the spectrum. The annealed and quenched "HCIZ" integrals corresponding to this tilting are:

F 2 (θ) = 1 n ln Dz e 2 =1 de f 2 =1 df exp θn √ m i,µ d µ e i z µi f µ , J 2 (θ, x) = 1 n ln e 2 =1 de f 2 =1 df exp θn √ m i,µ d µ e i z µi f µ .
In J 2 (θ, x), we assume that λ max (H n ) converges a.s. to x as n → ∞. Introducing Lagrange multipliers in the spherical integrals, we find:

F 2 (θ) = α 2 inf γ≥dmax γθ 2 α 2 -dtρ(t) ln(γ -t) -1 -ln θ 2 α 2 .
Similarly to our previous analysis of J 1 , we show that there is a transition in J 2 : for θ ≤ θ c (x), J 2 (θ, x) = F 2 (θ), while for θ ≥ θ c (x) one reaches:

J 2 (θ, x) =              F 2 (θ) if θ ≤ θ c (x), α -1 2 ln 1 -α + (α -1) 2 + 4xθ 2 2x - 1 + α 2 - α 2 ln θ 2 α + 1 2 (α -1) 2 + 4xθ 2 - 1 2 dλ σ(λ) ln(x -λ) if θ ≥ θ c (x), with θ c (x) ≡ xG σ (x) 2 + (α -1)G σ (x).
The details of the derivations of F 2 and J 2 are given in Appendix E.5.2. Importantly, the very existence of the transition in J 2 (θ, x) relies on the positivity of x, so that this tilting fails for negative matrices. This notably implies that the first tilt of eq. (8.6) is still crucial to handle the case d max ≤ 0.

We deduce from the tilting method, in the exact same way as in the first tilting, that

P n (x) exp{-n sup θ≥0 [J 2 (θ, x) -F 2 (θ)]}.
Using eq. (8.1) and the explicit expressions of F 2 and J 2 we derived, one shows that for all x ≥ s max the supremum is attained in θ

x ≡ [xG σ (x) 2 + (α -1)G σ (x)] 1/2 . We compute then I (x) = [G σ (x)-G σ (x)]/2,

Going further: the complex case and the left tail of the large deviations

A remark on the complex case

We give an intuitive remark on how the factor β = 2 in the complex case arises in Result 8.1. As we showed above, the method allows to write the large deviations rate function in the form

I(x) = sup θ [J(θ, x) -F (θ)],
with F and J annealed and quenched spherical integrals. This result straightforwardly transfers to the complex setting, however the integrals F and J are now defined over unit vectors on the complex unit sphere, i.e. they satisfy i |e i | 2 = 1. It is known that the asymptotic behavior of real and complex spherical integrals only differ by a factor 2 (i.e. the complex integral is twice the real one), a property known as "Zuber's 1/2-rule" [START_REF] Zinn-Justin | On some integrals over the U(N ) unitary group and their large-N limit[END_REF]: this explains the origin of the β factor in Result 8.1.

The left tail of the large deviations

Importantly, we did not consider large deviations at the left of s max . Such an event requires moving the whole bulk of eigenvalues, i.e. a number O(n) of eigenvalues, an event which has probability in the scale exp{-n 2 } [DM06, VMB07, MV09]. Whether the method applied here could be extended to study this left tail is an interesting open question. As we saw, the core of the method is to create a tilt of the measure such that the largest eigenvalue is shifted in a controllable manner: in this case, the tilting would need to induce a shift of the whole spectrum.

The perhaps most natural extension of the tilting of eq. (8.6) to this setting would be to consider an extensive-rank change in the covariance of the z µ :

Dz → Dz e n 2 Tr[MnOHnO ] ,
with O an orthogonal matrix and M n an arbitrary matrix (with extensive rank) that will parametrize the tilting, similarly to the parameter θ in the calculation we performed. Provided the mechanisms of the method we presented transfer to this case, this would give the large deviations function in terms of involved extensive-rank "HCIZ" spherical integrals. The study of these extensive-rank HCIZ integrals in the high-dimensional limit was conducted in [START_REF] Matytsin | On the large-N limit of the Itzykson-Zuber integral[END_REF], and rigorously proven in [START_REF] Guionnet | Large deviations asymptotics for spherical integrals[END_REF], and we stated their main result in Theorem 1.15. The resulting formulas are very tedious as they involve hydrodynamical PDEs, however in a very recent analysis [START_REF] Belinschi | Large deviation principles via spherical integrals[END_REF] the authors managed to leverage this approach to prove large deviations in several contexts, e.g. for the empirical measure of the sum of two freely independent random matrices. This indicates that these extensive-rank spherical integrals might indeed be the most natural path to analyse the left tail of the large deviations, a line of work that is however beyond the scope of this chapter.

Discussion on Chapter 8

In this chapter we presented a generic technique to derive the right tail of the large deviations of the largest eigenvalue of the large class of generalized sample covariance matrices. By symmetry, this also transfers to the left tail of the large deviations of the smallest eigenvalue. Our main result 8.1 significantly improves over the seminal works of [START_REF] David | Large deviations of extreme eigenvalues of random matrices[END_REF][START_REF] Satya | Large deviations of the maximum eigenvalue for Wishart and Gaussian random matrices[END_REF] for sample covariance matrices with identity covariance, and leverages a recent technique developed in a series of mathematical works [BG20, GH20, Hus20, BGH20, AGH21, McK21b]. Thanks to the relative simplicity of our main result, we will further investigate its consequences in particular for PCA on real-world datasets. We also proposed importance sampling simulations, that allow to probe our result and its consequences, e.g. the existence of a phase transition in the rate function depending on the spectral density of a diagonal matrix involved in the definition of our model.

The Hessian matrix of inference landscapes -Recall that we motivated Result 8.1 in particular by its application to the Kac-Rice formalism developed in Chapter 7. Our analytical large deviations computation therefore paves the way toward a direct understanding of the topology of the local minima in the landscape of complex inference models, since H n is directly related to the Hessian matrix of the models studied in Chapter 7! We are currently investigating the derivation of a Kac-Rice formula similar to Theorem 7.5 but restricted to local minima, which will contain the rate function of Result 8.1. Afterword "Look at me still talking when there's science to do" GLaDOS, Portal (2007).

Universality of the large deviations -

This dissertation aimed at leveraging diverse tools of statistical physics, statistics, information theory and probability theory to tackle different questions on the fundamental limits of estimation problems. In Part I we revisited high-temperature expansions, historically one of the first theoretical methods developed to deal with disordered systems, to unify and justify different algorithmic approaches for inference problems with rotationally-invariant data distributions. We also used it to provide first hints at an exact solution to the extensive-rank matrix factorization problem, which puts into the light the limitations of current theories, as we will further discuss below. Part II places itself in the general line of work reviewed in [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF], and which combines the heuristic replica method and message-passing algorithms to assess the optimal performances in a wide class of high-dimensional estimation problems. We complete this approach with probabilistic methods, notably an adaptive interpolation developed in [START_REF] Barbier | The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference[END_REF][START_REF] Barbier | The adaptive interpolation method for proving replica formulas. applications to the Curie-Weiss and Wigner spike models[END_REF], which allows to put the replica predictions on rigorous grounds in quite general settings. This general venue is applied to two-layers neural networks (Chapter 4), to study the influence of data structure on the optimal performances (Chapter 5), and to analyze the phase retrieval problem with generic right-rotationally invariant sensing matrices (Chapter 6). Finally, Part III considers a topological point of view on the general question of high-dimensional optimization, and provides an important step towards a mathematical characterization of the ruggedness of the landscapes of empirical risk minimization for simple estimation models, with important connections to random matrix theory.

Towards a global theory of learning? -Nonetheless, this dissertation touches upon important limitations of the statistical physics approach to high-dimensional estimation. Crucially, as the rest of "classical" approaches to machine learning, it fails to provide a clear theory of modern machine learning procedures based on deep neural networks, namely to answer the following problems that remain open:

• Why do models with very large number of parameters (such as deep nets), often larger than the number of data, not overfit? How do they manage to generalize well?

• Why do the optimization algorithms used in such models not get stuck on poor local minima?

These questions defy our statistical intuition, and as emphasized in [START_REF] Zdeborová | Understanding deep learning is also a job for physicists[END_REF], answering them can not be done by overlooking any of the three following key ingredients: the architecture of the problem, the structure of the data, and the learning algorithms. In this thesis we were confronted with each of these features, and an important long-term goal is to unify what statistical physics taught us on each of them to obtain answers.

We can illustrate this strategy on the analysis of data structure: indeed, the consideration of i.i.d. data samples has historically been one of the important limitations of the statistical physics approach, but recent progress has been made on going beyond this restriction, either by rotationally-invariant models or generative models as shown in Parts I and II, and in particular generative models of data are better and better understood using statistical physics techniques [GRM + 20, GMKZ20]. This is also the case of optimization algorithms: while physicists have described the Langevin dynamics in continuous disordered models very precisely [START_REF] Leticia | Analytical solution of the offequilibrium dynamics of a long-range spin-glass model[END_REF], this description is not adapted to study actual optimization algorithms (e.g. stochastic gradient descent) in machine learning procedures1 . However, recent works in the same community strive to create an analytical description of realistic optimization algorithms using statistical physics approaches (let us mention e.g. [START_REF] Mignacco | Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification[END_REF]), and combining these diverse ideas is an exciting prospect.

The extensive-rank challenge -Let us conclude our discussion by what we believe is a crucial challenge ahead for the statistical physics community interested in inference and learning: the analysis of extensive-rank problems, to which we gave a brief introduction in Chapter 3. Such models are especially instrumental in building a theory of actual deep neural networks with both many layers and many nodes in each layer, and the tremendous attention received by simple learning regimes in infinitely-wide networks [START_REF] Jacot | Neural tangent kernel: convergence and generalization in neural networks[END_REF][START_REF] Ghorbani | Linearized two-layers neural networks in high dimension[END_REF] demonstrates the importance of these models to the community. However, the tools that have been developed for fifty years by theoretical physicists and statisticians, such as the replica theory, are often inadequate to tackle extensive-rank problems as we explained in Chapter 3, and new ideas must be developed to expand them or to develop new ones. Finally, even though we focused in the aforementioned chapter on the matrix factorization problem with i.i.d. Gaussian weights, this model is an instrumental building block to move on to structured data and the design of efficient algorithms, much like the perceptron with Gaussian weights in finite-rank problems. While this very exciting research avenue is surely a very long-term goal, it is also a thrilling opportunity to rethink many of our intuitions and to gather again various scientific communities in physics, computer science, statistics, and probability.

Appendix A

Technicalities of the Plefka-Georges-Yedidia expansion A.1 Order 4 of the expansion for a spherical model

We start from eq. (2.7d), that we consider at η = 0 :

n∂ 4 η Φ J = U 4 0 -3 U 2 2 0 -3 n i=1 ∂ 2 η λ i U 2 (x i -m i ) 0 - 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -m 2 i -v i ) 0 . (A.1)
For simplicity we will denote xi ≡ (x i -m i ), so that at η = 0 the { xi } variables are Gaussian variables with mean xi = 0 and covariance xi xj = δ ij v i . In particular eq. (2.10) becomes:

U (η = 0, J) = - 1 2 i =j J ij xi xj .
From our calculation at order 2 we obtain the following relation that we can represent diagrammatically:

-3 U 2 2 0 = - 3 4 i =j J 2 ij v i v j 2 = - 3n 4 • • 2 . (A.2)
We now turn to the next term:

- 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -m 2 i -v i ) 0 -3 n i=1 ∂ 2 η λ i U 2 (x i -m i ) 0 = - 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -v i ) 0 -3 n i=1 U 2 xi ∂ 2 η λ i + m i ∂ 2 η γ i 0 , (a) = - 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -v i ) 0 -3n n i=1 U 2 xi ∂ ∂ 2 η Φ J ∂m i 0 (b) = - 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -v i ) 0 + O n (1).
In (a) we used the Maxwell equations (2.9), while in (b) we made use of the fact that the order 2 of the free entropy does not depend on the m i variables. We obtain for the remaining term:

- 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -v i ) 0 = 3 2 i =j J 2 ij v i v j 2 -3 i =j J 2 ij v j U 2 x2 i 0 ,
in which we used the Maxwell relations (2.9) to compute ∂ 2 η γ i . To compute U 2 x2 i 0 , we expand: We can then use Wick's theorem to simplify the average. There are two types of contractions (or pairings) that appear:

U 2 x2 i 0 = 1 4 i 1 =j 1 i 2 =j 2 J i 1 j 1 J i 2 j 2 x2 i xi 1 xj 1 xi 2 xj 2 0 . • • • • • • • • (a) • • • • • • • • (b)
• Contractions that do not mix the indices i 1 , j 1 , i 2 , j 2 with i. There are 2 such possible pairings and they give rise to the diagram n • • 2 .

• Contractions that mix these indices with i. There are all equivalent and there are 8 of them, each giving the diagram n • • • .

In the end, we reach:

U 2 x2 i 0 = v i 2 k =l J 2 kl v k v l + 2 k( =i) J 2 ik v 2 i v k .
We can finally compute the term we were seeking:

- 3 2 n i=1 ∂ 2 η γ i U 2 (x 2 i -v i ) 0 = -6n i,j,k pairwise distincts J 2 ij J 2 ik v 2 i v j v k = -6 • • • . (A.3)
Note that in this last equation we could add the hypothesis that j = k. Indeed the term j = k would give rise to the diagram n • • , which is negligible, since for every i = j one has

J ij = O(n -1/2
) as a consequence of rotational invariance (cf. Model S). We finally turn to the computation of U 4 0 :

U 4 0 = 1 16 3 α=0 iα =jα J iαjα 3 α=0 xiα xjα 0 .
The possible contractions arising from Wick's theorem yield several contributions, that we can also represent by diagrams. Note that these diagrams are very different from the diagrams that we described for instance in Fig.

U 4 0 = 3 4 i =j J 2 ij v i v j 2 + 6 i,j,k pairwise distincts J 2 ij J 2 ik v 2 i v j v k + 3 i 0 ,i 1 ,i 2 ,i 3 pairwise distincts J i 0 i 1 J i 1 i 2 J i 2 i 3 J i 3 i 0 v i 0 v i 1 v i 2 v i 3 , = 3 4 n • • 2 + 6n • • • . + 3n • • • • . (A.4)
(ii) Consider a simple cycle C p with p vertices. Then this diagram converges as n → ∞, a.s. and in L 2 norm, to the free cumulant c p (ρ), as in the real case. More precisely:

lim n→∞ E U 1 n i 1 ,••• ,ip pairwise distincts (UDU † ) i 1 i 2 (UDU † ) i 2 i 3 • • • (UDU † ) ipi 1 -c p (ρ) 2 = 0.
(iii) Any Eulerian strongly irreducible diagram that is not a simple cycle will be negligible in the n → ∞ limit (in L 2 norm).

(iv) Any Eulerian cactus diagram (like in Fig. A.2c) will converge in L 2 to the products of the free cumulants of ρ corresponding to each one of its constituent simple cycles.

These results are straightforward generalizations of the ones obtained for real matrices in Sec. 2.4. For completeness, we describe how to show a weaker version of (ii), and leave the other statements to the reader. As before, by unitary invariance we can assume that

(i 1 , • • • , i p ) = (1, , • • • , p
), and we can apply the results of [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF] to obtain a similar equation to eq. (2.49):

L p ≡ lim n→∞ n p-1 E U [(UDU † ) 12 • • • (UDU † ) p1 ], = lim n→∞ 1 n p l=1 ∂ ∂b i + i ∂ ∂c i exp n ∞ n=1 c n (ρ) n Tr [M(b, c) n ] b,c=0 , (A.5)
with now the matrix M(b, c) defined as:

M(b, c) ≡ 1 2           0 b 1 + ic 1 0 • • • 0 b p -ic p b 1 -ic 1 0 b 2 + ic 2 • • • 0 0 0 b 2 -ic 2 0 • • • 0 0 . . . . . . . . . . . . . . . . . . 0 0 0 • • • 0 b p-1 + ic p-1 b p + ic p 0 0 • • • b p-1 -ic p-1 0           . We have [∂ b i + i∂ c i ]M(b, c) = F i+1,i , in which (F a,b
) ll ≡ δ al δ bl are elementary non-symmetric matrices. In the exact same way as in Sec. 2.4.1, the dominant contribution in eq. (A.5) will be given by differentiating a single time the exponential term, and creating a cycle with the matrices F i+1,i . Note that contrary to the symmetric case of Sec. 2.4.1, here only the directed cycle will contribute, whereas both possible directions of the cycle contributed in eq. (2.49). Indeed, the cycles in terms of the matrices {F a,b } have to be directed in order to yield a nonzero contribution:

Tr [F 1,2 F 2,1 F 1,3 F 3,2 F 2,1 ] = 0, while Tr [F 1,2 F 2,1 F 2,3 F 3,2 F 2,1 ] = 0.
Thus we reach:

L p = ∞ k=p c k (ρ)Tr [(F 1,p F p,p-1 • • • F 2,1 ) M(0, 0) n-p ] = c p (ρ).
In order to get L 2 concentration of the simple cycle on the free cumulant, one can then exactly repeat the arguments of Sec. 2.4.3.

A.2.2 A note on the expectation of diagrams of diverging size

Although it is not directly useful in our PGY expansions, another side question one can ask on the behavior of these diagrams is: how do diagrams that have a number of edges that diverges with n behave in the n → ∞ limit ? In all of Sec. 2.4 we only considered diagrams of finite size. The behavior of HCIZ-type integrals with a matrix with diverging rank (as opposed to the finite-rank case) has first been analyzed rigorously in [START_REF] Guionnet | A Fourier view on the R-transform and related asymptotics of spherical integrals[END_REF] for a rank O(n 1/2-), and then generalized in [START_REF] Collins | New scaling of Itzykson-Zuber integrals[END_REF] for ranks O(n). We recall the main result of [START_REF] Collins | New scaling of Itzykson-Zuber integrals[END_REF]:

Theorem A.1 (Collins-Śniadyc) Let A n , B n be diagonal real matrices of size n. Assume that the rank m(n) of A n is such that m = m(n) = O(n), and denote a 1,n ≥ • • • ≥ a m,n the eigenvalues of A n .
Assume that the spectral measure of B n converges a.s. and in the weak sense to a probability measure ρ B , and that all elements of A n are bounded by a constant independent of n. Then one has:

1 nm ln U (n) DU e nTr[AnUBnU † ] = 2 m Tr[G ρ B (A n )] + O n (1).
A similar result holds for real orthogonal matrices:

1 nm ln O(n) DO e n 2 Tr[AnOBnO ] = 1 m Tr[G ρ B (A n )] + O n (1).
Recall that we defined the G ρ function in Sec. 1.5 in a variational form, and showed that it can be related to the integral of the R-transform. Using Theorem A.1, the techniques of Sec. 2.4 generalize to this case.

To illustrate this last claim, we consider real symmetric matrices under Model S. We say that a sequence {p(n)} satisfies the bounded free cumulant property if there exists C > 0 such that for all n, |c p(n) (ρ)| < C. We state two of the results of Sec. 2.4 that can be easily generalized to the diverging size case without changing any of the arguments:

(a) Consider a sequence p = p(n) = O(n) that satisfies the bounded free cumulant property. Then one obtains the generalization of eq. (2.47):

n p(n)-1 O(n) DO[(ODO ) 12 (ODO ) 23 • • • (ODO ) p(n)1 ] = c p(n) (ρ D ) + O n (1). (c) Consider a cactus diagram G composed of p(n) simple cycles of size (r 1 (n), • • • , r p(n) (n)), joining at vertices. Assume that p(n) i=1 r i (n) = O n (n)
and that all the sequences r i (n) satisfy the bounded free cumulant property. Then one has:

E G = p(n) i=1 c r i (n) (ρ) (1 + O n (1)).
Other results obtained in Sec. 2.4 for finite-size diagrams might also be applicable to the diverging size case, but they are not investigated in this thesis.

A.3 PGY for extensive-rank matrix factorization

We describe here in more details the formalism we used to derive Result 3.1. Many parts of the derivation are very similar to what we described in Chapter 2, and we will refer to it when necessary. We start from eqs. (3.8) and (3.9):

n(m + p)Φ Y,n (η) = µ,i λ F µi m F µi + γ F µi 2 (v F µi + (m F µi ) 2 ) + i,l λ X il m X il + γ X il 2 (v X il + (m X il ) 2 ) , + µ,l -ω µl g µl - b µl 2 (-r µl + g 2 µl ) + ln P H (dH) P F (dF) P X (dX) e -S eff,η [F,X,H] , S eff,η [F, X, H] ≡ µ,i λ F µi F µi + γ F µi 2 F 2 µi + i,l λ X il X il + γ X il 2 X 2 il + µ,l ω µl (iH) µl - b µl 2 (iH) 2 µl + η √ n µ,i,l (iH) µl F µi X il , H eff [F, X, H] ≡ 1 √ n µ,i,l (iH) µl F µi X il .

A.3.1 Orders 1 and 2 in η

At order 1, we have directly:

∂Φ Y,n ∂η η=0 = - 1 n(m + p) H eff 0 = 1 n 3/2 (m + p) µ,i,l g µl m F µi m X il . (A.6)
As in Chapter 2, we can then use "Maxwell" relations to compute the derivatives of the Lagrange parameters at η = 0. For instance, for λ F µi and γ F µi they read:

λ F µi + m F µi γ F µi = n(m + p) ∂Φ Y,n ∂m F µi , γ F µi = 2n(m + p) ∂Φ Y,n ∂v F µi . (A.7)
From it we reach easily ∂ η γ F µi (η = 0) = 0. Applying this technique to all Lagrange parameters, we can compute the operator U Y,0 (defined in eq. (2.6)). For clarity of the notation, we will denote by lowercase letters centered variables, for instance x µi ≡ X µi -m X µi . We obtain:

U Y,0 = 1 √ n µ,i,l [(ih) µl f µi x il -g µl f µi x il + (ih) µl m F µi x il + (ih) µl f µi m X il ]. (A.8)
One can then compute:

1 2 ∂ 2 Φ Y,n ∂η 2 η=0 = 1 2n(m + p) U 2 0 , = 1 2n 2 (m + p) µ,i,l [-r µl v F µi v X il + g 2 µl v F µi v X il -r µl (m F µi ) 2 v X il -r µl v F µi (m X il ) 2 ]. (A.9)

A.3.2 Order 3 in η

To describe the results at order 3, we need to introduce the third cumulants of the variables {iH µl , F µi , X il } at η = 0 (all these variables are thus independent). These cumulants are denoted {κ

(3,H) µl , κ (3,F ) µi , κ (3,X) il }.
Using the order 3 formula of eq. (2.7), one has:

1 3! ∂ 3 Φ Y,n ∂η 3 η=0 = - 1 6n(m + p) U 3
3 and 4 of the variables ih µl , f µi and x il . One can check that the combination of the terms containing these moments would not give a thermodynamically relevant contribution. From the equations (3.12) at order 2 and the Maxwell relations (e.g. eq. (A.7)) we obtain the derivatives of the Lagrange multipliers, at leading order and at η = 0:

                       ∂ 2 η ω µl = 2 n g µl i [(m F µi ) 2 v X il + v F µi (m X il ) 2 ], ∂ 2 η b µl = -2 n i [v F µi v X il + (m F µi ) 2 v X il + v F µi (m X il ) 2 ], ∂ 2 η λ F µi = 2 n m F µi l [r µl (m X il ) 2 -g 2 µl v X il ], ∂ 2 η γ F µi = -2 n l [r µl v X il + r µl (m X il ) 2 -g 2 µl v X il ], ∂ 2 η λ X il = 2 n m X il µ [r µl (m F µi ) 2 -g 2 µl v F µi ], ∂ 2 η γ X il = -2 n µ [r µl v F µi + r µl (m F µi ) 2 -g 2 µl v F µi ]. (A.15)
We can compute the following averages:

U 2 x il 0 = - 2 n m X il v X il µ r µl v F µi , U 2 (x 2 il + 2m X il x il -v X il ) 0 = 2(v X il ) 2 n µ [-r µl v F µi + g 2 µl v F µi -r µl (m F µi ) 2 ] - 4 n v X il (m X il ) 2 µ r µl v F µi .
From this and eq. (A.15), one can obtain the term involving the derivatives of the Lagrange parameters λ X and γ X in eq. (2.7d):

-3 i,l ∂ 2 η λ X il U 2 x il 0 + ∂ 2 η γ X il 2 U 2 (x 2 il + 2m X il x il -v X il ) 0 = (A.16) - 12 n 2 i,l v X il (m X il ) 2 µ r µl v F µi 2 - 6 n 2 i,l (v X il ) 2 µ [r µl v F µi + r µl (m F µi ) 2 -g 2 µl v F µi ] 2 .
Doing similarly for (ih) µl and f µi , we obtain all the terms involving derivatives of the Lagrange multipliers:

-3

µ,i ∂ 2 η λ F µi U 2 f µi 0 + ∂ 2 η γ F µi 2 U 2 (f 2 µi + 2m F µi f µi -v F µi ) 0 = (A.17) - 12 n 2 µ,i v F µi (m F µi ) 2 l r µl v X il 2 - 6 n 2 µ,i (v F µi ) 2 l [r µl v X il + r µl (m X il ) 2 -g 2 µl v X il ] 2 , -3 µ,l ∂ 2 η ω µl U 2 (ih) µl 0 + ∂ 2 η b µl 2 U 2 ((ih) 2 µl -2g µl (ih) µl + r µl ) 0 = (A.18) 12 n 2 µ,l r µl g 2 µl i v F µi v X il 2 - 6 n 2 µ,l r 2 µl i [v F µi v X il + (m F µi ) 2 v X il + v F µi (m X il ) 2 ]
• Terms involving strongly irreducible diagrams of m F or m X . For instance, the term:

µ i =i l =l m X il m X il m X i l m X i l r µl r µl v F µi v F µi .
By H.1, the variables m F , m X behave like uncorrelated variables, so that all these diagrams (including the simple loops) will be negligible. This is precisely the sort of terms that is not negligible when involving g µl , as a consequence of H.2.

We can now sum all eqs. (A.16) to (A.22), simplifying the terms that are negligible by the arguments above, and checking that almost all non-negligible terms are cancelling each other. This is a lengthy but straightforwards calculation, and we reach:

1 4! ∂ 4 Φ Y,n ∂η 4 0 = 1 4n 3 (m + p) i µ 1 =µ 2 l 1 =l 2 g µ 1 l 1 g µ 1 l 2 g µ 2 l 2 g µ 2 l 1 v F µ 1 i v F µ 2 i v X il 1 v X il 2 + O n (1).

A.4 The expansion for symmetric extensive-rank matrix factorization

For the symmetric Model XX , we can perform a PGY expansion in a very similar way to the case of Model FX that we just described in Section A.3. The majority of the calculation is straightforwardly transposed, and we briefly outline its main steps here. Recall the free entropy of eq. (3.3b):

Φ Y,n ≡ 1 nm ln µ,i P X (dX µi ) µ<ν P out Y µν 1 √ n i X µi X νi .
Introducing a field h ≡ XX / √ n, its Fourier conjugate H, and Lagrange multipliers to fix the first and second moments of both X and H, we reach a very similar form to eq. (3.8):

nmΦ Y,n = µ,i λ µi m µi + γ µi 2 (v µi + (m µi ) 2 ) + µ<ν -ω µν g µν - b µν 2 (-r µν + g 2 µν ) (A.23) + ln P H (dH) P X (dX) e -S eff [X,H] ,
in which we introduced the effective action:

S eff [X, H] = µ,i λ µi X µi + γ µi 2 X 2 µi + µ<ν ω µν (iH) µν - b µν 2 (iH) 2 µν (A.24) + 1 √ n µ<ν i (iH) µν X µi X νi H eff .
The (un-normalized) distribution over H µν is given by the Fourier transform of the channel:

P µν H [dH] ≡ d H 2π e iH H P out (Y µν | H).
Again, we introduce a factor η in front of H eff in eq. (A.24), and we expand the free entropy as a function of η. Computing the first order of perturbation and the operator U of Georges-Yedidia . By Fourier expanding the delta functions in I prior and performing a saddle-point one obtains:

lim n→∞ 1 n ln E[Z p n ] = extr Q, Q [H(Q, Q)], (B.1)
in which (recall m/n → α > 0) :

H(Q, Q) ≡ 1 2 p a=0 l,l Q al al Qal al - 1 2 a =b l,l Q al bl Qal bl + ln I + α ln J, (B.2)
with the two functions:

I ≡ p a=0 R K dw a P 0 (w a ) exp - 1 2 p a=0 l,l Qal al w a l w a l + 1 2 a =b l,l Qal bl w a l w b l , J ≡ R dy p a=0 R K dZ a (2π) K(p+1)/2 P out (y|Z a ) √ det Σ exp - 1 2 p a,b=0 K l,l =1 Z a l Z b l (Σ -1 ) al bl .
Our goal is to express H(Q, Q) as an analytical function of p, in order to perform the replica trick. To do so, we will assume that the extremum of H is attained at a point in Q, Q space such that a replica symmetry property is verified. More concretely, we assume:

∃Q 0 ∈ S K s.t ∀a ∈ [|0, p|] ∀(l, l ) ∈ [|1, K|] 2 Q al al = Q 0 ll , ∃q ∈ S K s.t ∀(a < b) ∈ [|0, p|] 2 ∀(l, l ) ∈ [|1, K|] 2 Q al bl = q ll , (B.3)
and similarly for Q. Under the ansatz (B.3), we obtain from eq. (B.2):

H(Q 0 , Q0 , q, q) = p + 1 2 Tr[Q 0 Q0 ] - p(p + 1) 2 Tr[q q] + ln I + α ln J. (B.4)
Remains now to compute an expression for I and J that is analytical in p, in order to take the limit p ↓ 0. This can be done using the identity, for any M ∈ S + K and x ∈ R K :

exp 1 2 x M x = R K Dξ exp ξ M 1/2 x ,
in which Dξ is the standard Gaussian measure on R K . We obtain, after a tedious algebraic calculation that is left to the reader:

       I = R K Dξ R K dw P 0 (w) exp - 1 2 w ( Q0 + q)w + ξ q1/2 w p+1 , J = R dy R K Dξ R K dZP out y|(Q 0 -q) 1/2 Z + q 1/2 ξ p+1 . (B.5)
We now notice that the value of Q 0 is actually constrained by the problem and the replica symmetry. Indeed, our calculation must be consistent in the sense that if taking p = 0 in eq. (B.1), we must find extr Q, Q[H (Q, Q)] = 0 at p = 0.

In the p ↓ 0 limit, one easily gets J = 1 and I = R K dw P 0 (w) exp[-w Q0 w 0 /2]. Extremizing over Q 0 , Q0 implies that they satisfy Q0 = 0 and Q 0 = ρ (the covariance matrix of P 0 ). We can now plug in these values in eq. (B.5), and then take the p ↓ 0 limit of H(q, q)/p in eq. (B.4).

In the end, by eq. (B.1) and the famous "replica trick" we obtain the final formula for the free entropy:

lim n→∞ 1 n E ln Z n = extr q,q - 1 2 Tr[q q] + I P + αI C , (B.6)
wih the two functions:

I P ≡ R K Dξdw 0 P 0 (w 0 )e -1 2 (w 0 ) qw 0 +ξ q1/2 w 0 ln R K dwP 0 (w)e -1 2 w qw+ξ q1/2 w , I C ≡ dy R K Dξ DZ 0 P out {y|(Q 0 -q) 1/2 Z 0 + q 1/2 ξ} ln R K DZ P out {y|(Q 0 -q) 1/2 Z + q 1/2 ξ}.
A known ambiguity of the replica method is that its result is given as an extremum, here over the set S + K (ρ). Assuming that this extr is realized as a sup q inf q , one can easily see that eq. (B.6) yields back the claim of Theorem 4.1. Our rigorous proof, sketched in Section 4.4, will allow to lift the ambiguity of the extremum.

B.2 Replica computation for generic GLMs

In this section, we perform the step-by-step replica calculation that gives Conjecture 6.1.

B.2.1 Setting and strategy

We let n, m → ∞ with m/n → α > 0. Recall that we are interested in the partition function:

Z n (Y) ≡ K n n i=1 P 0 (dx i ) m µ=1 P out Y µ 1 √ n n i=1 Φ µi x i .
Here Φ is a matrix that is left and right-orthogonally (unitarily) invariant, meaning that for all

O, U ∈ U β (m) × U β (n), Φ d = OΦU.
Compared to Conjecture 6.1, we added a left-invariance hypothesis. However the analysis of G-VAMP [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF][START_REF] Sundeep Rangan | Vector approximate message passing[END_REF] shows that this left invariance is actually not needed for the result, and thus we state Conjecture 6.1 for matrices that are only right-invariant, but we use the left invariance to simplify the following (heuristic) calculation. Moreover, we assume that the LSD ν of Φ † Φ/n is well-defined, and that the eigenvalue distribution of Φ † Φ/n has large deviations in a scale at least n 1+η for an η > 0.

Recall that the replica trick introduced in Section 1.3.1 consists in computing the p-th moment of the partition function for arbitrary integer p, before extending this expression analytically to any p > 0 and using the replica trick:

lim n→∞ 1 n E Φ,Y ln Z n (Y) = lim p↓0 lim n→∞ 1 np ln E Φ,Y [Z n (Y) p ].
This method is obviously heuristic given the inversion of limits p ↓ 0 and n → ∞, as well as the analytic continuation to arbitrary p > 0 of the p-th moment.

assume a replica symmetric form of Q x , Qx at the point at which the extremum is reached:

Q x =       Q x q x • • • q x q x Q x • • • q x . . . . . . . . . . . . q x q x • • • Q x       , Qx =       Qx -q x • • • -q x -q x Qx • • • -q x . . . . . . . . . . . . -q x -q x • • • Qx       . (B.8)
Note that we have Q x , q x , Qx , qx ∈ R. After a simple Gaussian transformation of the squared term using the general identity exp(β|x| 2 /2) = K D β ξ exp(βx • ξ) we reach the final expression:

I 0 (p, Q x , q x ) = (B.9) inf Qx,qx β(p + 1) 2 Q x Qx - βp(p + 1) 2 q x qx + ln K D β ξ K P 0 (dx)e -β( Qx+qx) 2 |x| 2 +β √ qxx•ξ p+1 . The channel term I out (p, Q z )
This term is very similar to the prior term we just detailed. We use completely similar replica symmetric assumptions for the overlaps Q z to the ones on Q x described in eq. (B.8). We reach:

I out (p, Q z , q z ) = inf Qz,qz β(p + 1) 2 Q z Qz - βp(p + 1) 2 q z qz + β(p + 1) 2 ln(2π/(β Qz )) (B.10) + ln R dy K D β ξ K dz 2π β Qz -β/2 P out (y|z) e -β Qz +qz 2 |z| 2 +β √ qzz•ξ p+1
.

We normalized the integrals so that in the limit p ↓ 0, the term inside the logarithm goes to 1, which will be a useful remark.

The "delta" term

I int (p, Q x , Q z )
We now turn to the computation of the "delta" term:

I int (p, Q x , Q z ) ≡ lim n→∞ 1 n ln E Φ p a=0 δ z a - 1 √ n Φx a , (B.11) assuming that Q x , Q z are known.
Computing this term is central in this replica calculation. We use, as is done in [TK20], the identity:

1 n ln E Φ p a=0 δ z a - 1 √ n Φx a = lim ↓0 1 n ln E Φ exp -β 2 a z a -1 √ n Φx a 2 (2π /β) βm(p+1) 2 , (B.12)
and we invert the n → ∞ and the ↓ 0 limit. Let us rewrite the RHS of eq. (B.12). As Φ is orthogonally (unitarily) invariant, we can write it as:

E Φ exp -β 2 a z a -1 √ n Φx a 2 (2π /β) βm(p+1) 2 = E Φ,O,U exp -β 2 a Oz a -1 √ n ΦUx a 2 (2π /β) βm(p+1) 2 , (B.13) in which (O, U) is uniformly sampled over U β (m) × U β (n).
Note that when U is uniformly distributed over U β (n), the set of vectors {Ux a } p a=0 is uniformly distributed over the set of (p + 1) vectors in K n with overlap matrix Q x . There is a completely similar result for z as well. The consequence is that we can replace in eq. (B.13) the average over O, U by an average over the vectors satisfying this constraint:

I int (p, Q x , Q z ) (B.14) 1 n ln E Φ K a dx a dz a a≤b δ(nQ x ab -(x a ) † x b )δ(mQ z ab -(z a ) † z b ) e - β 2 a z a -1 √ n Φx a 2 (2π /β) βm(p+1)/2 K a dx a dz a a≤b δ(nQ x ab -(x a ) † x b )δ(mQ z ab -(z a ) † z b )
.

The numerator and the denominator correspond to two terms, that we denote

I int (p, Q x , Q z ) = I (n) int (p, Q x , Q z ) -I (d) int (p, Q x , Q z ).
We can introduce the Fourier-transform of the delta distribution to compute both terms, as in the previous sections. Let us start with the denominator. It reduces after Fourier-transformation to a Gaussian integral involving a block-diagonal matrix:

I (d) int (p, Q x , Q z ) β 2 inf Γ x ,Γ z Tr[Q x Γ x ] + αTr[Q z Γ z ] -ln det βΓ x 2π -α ln det βΓ z 2π ,
with symmetric (Hermitian) positive matrices Γ x , Γ z of size (p + 1). The infimum is readily solved by Γ x = (Q x ) -1 and Γ z = (Q z ) -1 , which yields:

I (d) int (p, Q x , Q z ) β(α + 1)(p + 1) 2 (1 + ln 2π β ) + β 2 ln det Q x + αβ 2 ln det Q z . (B.15)
Let us now compute the numerator with the same technique. We obtain:

I (n) int (p, Q x , Q z ) β(p + 1) 2 ln 2π β α + β 2 inf Γ x ,Γ z Tr[Q x Γ x ] + αTr[Q z Γ z ] - 1 n ln det M n , (B.16)
with a Hermitian block-matrix M n that we write here in the tensor product form:

M n ≡   (Γ z + 1 I p+1 ) ⊗ I m 1 I p+1 ⊗ Φ √ n 1 I p+1 ⊗ Φ † √ n Γ x ⊗ I n + 1 I p+1 ⊗ Φ † Φ n   .
Using the block-matrix determinant calculation

det A B C D = det A × det(D -CA -1 B),
we reach:

1 n ln det M n = α ln det Γ z + 1 I p+1 + 1 n ln det Γ x ⊗ I n + 1 I p+1 ⊗ Φ † Φ n - 1 2 Γ z + 1 I p+1 -1 ⊗ Φ † Φ n , = (α -1) ln det Γ z + 1 I p+1 + 1 n ln det Γ x Γ z ⊗ I n + 1 Γ x ⊗ I n + 1 Γ z ⊗ Φ † Φ n , = (α -1) ln det Γ z + 1 I p+1 + ln det Γ x Γ z + 1 (Γ x + λΓ z ) ν + O n (1),
with λ distributed according to ν, the LSD of Φ † Φ/n. This allows to write I (n) int from eq. (B.16) and to take the ↓ 0 limit, keeping the terms that do not vanish:

I (n) int (p, Q x , Q z ) β 2 inf Γ x ,Γ z [Tr[Q x Γ x ] + αTr[Q z Γ z ] -ln det(Γ x + λΓ z ) ν ]. (B.17)
Finally, we again consider a replica-symmetric assumption for Γ x , Γ z , in the form:

Γ x =       Γ x -γ x • • • -γ x -γ x Γ x • • • -γ x . . . . . . . . . . . . -γ x -γ x • • • Γ x       , Γ z =       Γ z -γ z • • • -γ z -γ z Γ z • • • -γ z . . . . . . . . . . . . -γ z -γ z • • • Γ z       .
Again, by hermiticity, we have γ x , γ z ∈ R. Combining eqs. (B.15) and (B.17) and using the replica symmetric assumption, we finally obtain the cumbersome expression:

2 β I int (p, Q x , Q z ) = inf Γx,γx,Γz,γz [(p + 1)Q x Γ x -p(p + 1)q x γ x + α(p + 1)Q z Γ z -αp(p + 1)q z γ z -p ln(Γ x + γ x + λΓ z + λγ z ) ν -ln[Γ x -pγ x + λ(Γ z -pγ z )] ν ] -(α + 1)(p + 1) ln 2πe/β + (p + 1) ln 2π β -p ln(Q x -q x ) -ln(Q x + pq x ) -αp ln(Q z -q z ) -α ln(Q z + pq z ). (B.18)
A note on quenched and annealed averages -Importantly, we did not consider the average over Φ to compute I int . Indeed, the result only depends on the eigenvalue distribution of Φ † Φ/n, which (by hypothesis) has large deviations in a scale at least n 1+η with η > 0. Since we are looking at a scale exponential in n, we can thus consider that this eigenvalue distribution is equal to its limit value ν. However, one must be careful that this argument breaks down if our result starts to be sensitive to the extremal eigenvalues of Φ † Φ/n. Since these variables typically have large deviations in the scale n (for instance for Wigner or Wishart matrices [START_REF] David | Large deviations of extreme eigenvalues of random matrices[END_REF]), this could invalidate our calculation. This phenomenon is well-known in the study of "HCIZ" spherical integrals, and we gave an example of it in Section 1.5.3. We argue in Section B.2.4 that this possible issue, not discussed in [START_REF] Takahashi | Macroscopic analysis of vector approximate message passing in a model mismatch setting[END_REF], never arises for physical values of the overlaps.

Expressing the p-th moment

Combining the three previous results we finally obtain:

lim n→∞ 1 n ln E Z n (Y) p = sup Qx,qx Qz,qz [I 0 (p, Q x , q x ) + αI out (p, Q z , q z ) + I int (p, Q x , q x , Q z , q z )], (B.19)
in which the three terms are given by eqs. (B.9),(B.10),(B.18).

B.2.3 The p ↓ 0 limit

One can easily see that the RHS of eq. (B. 19) is analytic in p. The next step of the replica method is to analytically extend this expression to arbitrary p > 0, before considering the limit p ↓ 0.

Consistency of the limit

One must be careful that, when extending our expression to arbitrarily small p > 0, we satisfy the trivial condition lim p↓0 ln EZ p n = 0. As we will see, this will yield constraints on the diagonals of the overlap matrices. Taking the limit p ↓ 0 in the three terms of eq. (B.19) yields:

                             I 0 (0, Q x , q x ) = inf Qx β 2 Q x Qx + ln K P 0 (dx)e -β Qx 2 |x| 2 , I out (0, Q z , q z ) = inf Qz β 2 Q z Qz + β 2 ln 2π β Qz , I int (0, Q x , q x , Q z , q z ) = inf Γx,Γz β 2 Q x Γ x + αβ 2 Q z Γ z - β 2 ln[Γ x + λΓ z ] ν - β(α + 1) 2 (1 + ln 2π β ) + β 2 ln 2π β - β 2 ln Q x - αβ 2 ln Q z .
One can easily solve the saddle point equations on Q z , Qz , they give Γ z = 0 and Qz = 1/Q z . One can then find all the remaining variables easily:

Q x = ρ, Qx = 0, Γ x = ρ -1 , Q z = ρ λ ν /α, Qz = 1/Q z , Γ z = 0.
This yields (we drop the vacuous dependency on q x , q z ):

               I 0 (0, Q x = ρ) = 0, I out 0, Q z = ρ λ ν α = β 2 + β 2 ln 2πρ λ ν βα , I int 0, Q x = ρ, Q z = ρ λ ν α = - βα 2 1 + ln 2π β - αβ 2 ln ρ λ ν α . (B.20)
Recall that we have

lim p↓0 lim n→∞ 1 n ln EZ n (Y) p = I 0 + αI out + I int ,
so that we obtain from eq. (B.20) that indeed the limit is consistent.

The replica-symmetric result

Using eq. (B.19) for the p-th moment and the consistency conditions we just derived, we obtain after using the replica trick:

lim n→∞ 1 n E ln Z n (Y) = sup qx,qz
[I 0 (q x ) + αI out (q z ) + I int (q x , q z )], (B.21) with the auxiliary functions:

I 0 (q x ) ≡ inf qx≥0 - β qx q x 2 + K D β ξP 0 (dx)e -β qx 2 |x| 2 +β √ qxx•ξ ln K P 0 (dx)e -β qx 2 |x| 2 +β √ qxx•ξ , I out (q z ) ≡ inf qz≥0 - β qz q z 2 - β 2 ln( Qz + qz ) + β qz 2 Qz + dyD β ξ J(q z , y, ξ) ln J(q z , y, ξ) , I int (q x , q z ) ≡ inf γx,γz≥0 β 2 (ρ -q x )γ x + αβ 2 (Q z -q z )γ z - β 2 ln(ρ -1 + γ x + λγ z ) ν - β 2 ln(ρ -q x ) - βq x 2ρ - αβ 2 ln(Q z -q z ) - αβq z 2Q z ,
with Q z ≡ ρ λ ν /α and Qz = 1/Q z . Moreover, the domain of the supremum is q x ∈ [0, ρ] and q z ∈ [0, Q z ]. The function J(q z , y, ξ) appearing in the expression of I out is defined as:

J(q z , y, ξ) ≡ K D β zP out y z Qz + qz + qz Qz ( Qz + qz ) ξ .
Note that compared to the calculation presented in the previous sections, we moved a term (βα/2)(1 + ln 2π/β) between I out and I int , and we also made a few straightforward change of variables in the expression of I out . This is exactly the result given in Conjecture 6.1!

B.2.4 Concentration of the spectrum of Φ † Φ/n and the absence of saturation

As emphasized during the computation of I int , our calculation assumed that the extremization equations on (γ x , γ z ) always admitted a solution. Moreover, we assumed that this solution is not sensitive to the extremal eigenvalues of Φ † Φ/n. If this assumption is indeed true, the rate of the large deviations of the spectrum of Φ † Φ/n was assumed to be large enough to justify our calculation. This important condition can be phrased by saying that for all physical values of (q x , q z ), we must not "touch" the edge of the spectrum in the variational expression of I int :

1 ρ + γ x + γ z λ min (ν) > 0. (B.22)
We justify here eq. (B.22) for all physical values of (q x , q z ). We will combine three arguments:

(i) In the computation of I int the matrix Γ z is assumed to be Hermitian positive in the p ↓ 0 limit. Since Γ z = 0, this implies that we must have λ z ≥ 0.

(ii) The saddle point equation on q x yields1 :

qx = q x ρ(ρ -q x ) -γ x . (B .23)
(iii) Finally, we will derive a lower bound on q x . As one can see from the computation of I 0 , q x is the optimal overlap achievable in the following scalar inference problem [BKM + 19]:

Y 0 = qx X + Z, (B.24)
in which one observes Y 0 , and the noise Z is distributed according to N β (0, 1). The optimal estimator is given by the average of x under the posterior distribution, whose density is proportional to P 0 (x)e -β 2 |y-√ qxx| 2 . If this is untractable for generic P 0 , we can consider a suboptimal estimation by using a Gaussian prior with variance ρ in the estimation procedure (so that the problem is mismatched). This yields the bound:

q x ≥ D β ξ K P 0 (dx) x e -β qx 2 |x| 2 +β √ qxx•ξ • K dx x e - β|x| 2 2ρ e -β qx 2 |x| 2 +β √ qxx•ξ K dx e - β|x| 2 2ρ e -β qx 2 |x| 2 +β √ qxx•ξ .
This can easily be simplified by performing the Gaussian integral, and yields the bound:

q x ≥ ρ 2 qx 1 + ρq x . (B.25)
Combining (ii) and (iii) gives:

q x ≥ ρ - ρ -q x 1 -γ x (ρ -q x )
.

Since q x ∈ [0, ρ], this implies in particular that γ x ≥ 0. Using this along with (i), we reach:

1 ρ + γ x + γ z λ min (ν) ≥ 1 ρ > 0,
which is what we wanted to show.

Finally we will use the matrices w

∈ R n×K , u ∈ R m×K , Y t ∈ R m , Y t ∈ R n×K , X ∈ R m×n , V ∈ R m×K , W * ∈ R n×K and U * ∈ R m×K .
It is convenient to reformulate the expression of the free entropy f n, (t) in the Hamiltonian language. We introduce an interpolating Hamiltonian:

H t (w, u; Y t , Y t , X, V) ≡ - m µ=1 u Yt,µ (s t,µ ) + 1 2 n i=1 Y t,i -R 1 (t) 1/2 w i 2 2 , (C.2)
where we recall that s t,µ is defined in eq. (4.13). The expression of H t (W * , U * ; Y t , Y t , X, V) is similar to eq. (C.2), but with w replaced by W * and s t,µ replaced by S t,µ given by eq. (4.11). The average free entropy at time t (cf eq. (4.15)) then reads

f n, (t) ≡ 1 n E ln R n×K dw P 0 (w) R m×K
Du e -Ht(w,u;Yt,Y t ,X,V) .

Note that to develop the calculations it is sometimes fruitful to represent the expectations over W * , U explicitly as:

f n, (t) = 1 n E X,V,Yt,Y y dP 0 (W * )
DU * e -Ht(W * ,U;Yt,Y t ,X,V) ln dP 0 (w)Du e -Ht(w,u;Yt,Y t ,X,V) .

C.4 Free entropy variation: Proof of Proposition 4.3

The proof provided here follows very closely the one in [BKM + 19] for the case K = 1, so that we will sometimes refer to this paper for more details. We first prove that for all t ∈ (0, 1):

df n, (t) dt = - 1 2 E Tr 1 n m µ=1 ∇u Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) -r(t) 1 n n i=1 W * i w i -q(t) n,t, + 1 2 Tr[r(t)(q(t) -ρ)] - A n 2 , (C.3)
where

A n ≡ E Tr 1 √ n m µ=1 ∇∇ P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) 1 √ n n i=1 (W * i (W * i ) -ρ) 1 n ln Z n, (t) .
Once we will have proven eq. (C.3), we show that A n goes to 0 as n → ∞ uniformly in t ∈ [0, 1] in order to conclude the proof of Proposition 4.3. Recall that the Hamiltonian was defined in eq. (C.2). Its t-derivative evaluated at the ground-truth matrices is given by Proof of Lemma C.4 -The existence and uniqueness of the solution of (C.12) follows from the classical Cauchy-Lipschitz Theorem. The solution is indeed defined on all the segment [0, 1] because F is bounded. Theorem 3.1 from Chapter 5 in [START_REF] Hartman | Ordinary Differential Equations: Second Edition[END_REF] gives that y admits continuous partial derivatives ∂ i y for i = 1, . . . , d, and Corollary 3.1 from Chapter 5 in the same reference states the Liouville formula (C.13). It remains to show that z t is a C 1 diffeomorphism. By the Cauchy-Lipschitz theorem, two solutions of y (t) = F t, y(t) that are equal at some t ∈ [0, 1] are equal everywhere. This implies that the mapping z t : → y(t, ) is injective, for all t ∈ [0, 1]. Since y admits continuous partial derivatives in i , i = 1, . . . , d, we obtain that z t is of class C 1 on (0, +∞) d . Now, the equation (C.13) gives that J(z t )( ) > 0 for all ∈ (0, +∞) d . The local inversion theorem gives then that z t is a C 1 diffeomorphism.

dH t dt (W * , U * ; Y t , Y t , X, V) = - m µ=1 ∇ u Yt,µ (S t,µ ) dS t,µ dt - n i=1 dR 1 (t) 1/2 dt W * i (Y t,i -R 1 (t) 1/2 W * i ), = - m µ=1 Tr dS t,µ dt ∇ u Yt,µ (S t,µ ) - n i=1 Tr dR 1 (t) 1/2 dt (Y t,i -R 1 (t) 1/2 W * i )W * i . (C.4)

Lemma C.5 (Boundedness of an overlap fluctuation)

Recall that for the sake of the proof we added (cf. Remark 4.1) a small Gaussian noise of variance ∆ > 0 to the observations. Under hypothesis H.2 one can then find a constant C(ϕ, K, ∆) < +∞ (independent of n, t, ) such that for any R n ∈ S where C(ϕ, K, ∆) < +∞ is a finite constant depending only on ϕ, K, and ∆. This is easily seen by expanding all squares and using that m/n → α. This ends the proof of Lemma C.5.

For concision, we denote here sign(x) = σ(x). One obtains from eq. (D.2):

1 2 -2 Bayes,K=2 g = R 4
Dx σ[σ(x 1 ) + σ(x 2 )] (D.3) × σ σ ( q a 2 + q d )x 1 + q a 2 x 2 + x 3 1 -q 2 a 2 -q a q d -q 2 d + σ q a 2 x 1 + ( q a 2 + q d )x 2 -x 3 q a (q d + qa 2 ) 1 -q 2 a 2 -q a q d -q 2 d + x 4

(1 -q 2 d )(1 -(q a + q d ) 2 ) 1 -q 2 a 2 -q a q d -q 2 d . These integrals were then computed using Monte-Carlo methods to obtain the generalization error in Fig. 4.2.

D.2 Large K limit in the committee machine

We consider the large K limit1 for a sign activation function, and for different priors on the weights. We also consider a noiseless channel. We assume a committee symmetric solution, i.e. the matrices q and q (q and r in the notations of Theorem 4.1) are of the type q = q d I K + q a 1 K 1 K /K, with the unit vector 1 K ≡ (1) K l=1 , and similarly for q. In the large K limit, this scaling of the order parameters is natural. Indeed, assume that the covariance of the prior is ρ = I K . Since both q and (ρ -q) are assumed to be positive matrices, it is easily shown to imply that 0 ≤ q d ≤ 1 and 0 ≤ q a + q d ≤ 1, so that these variables stay of order O(1).

In the following of this section we take notations arising from the replica calculation of Section B.1. In particular, we note Q 0 = ρ, and we will take eq. (B.6) as the expression of the free entropy (recall that it is equivalent to Theorem 4.1, so that this expression is mathematically rigorous).

D.2.1 Limit of the channel integral

In the following, we consider Q 0 = σ 2 I K . We are interested here in computing the leading order term in the "channel" integral I C of eq. (B.6). Note that replacing σ 2 by 1 in this equation only amounts to replacing q by q/σ 2 , so we can assume σ 2 = 1 without loss of generality. We write I C as I C = y=±1 R K Dξ I C (y, ξ) ln I C (y, ξ), with the definition

I C (y, ξ) ≡ R K DZP out {y|(Q 0 -q) 1/2 Z + q 1/2 ξ}.
Recall that we consider a sign activation function and no noise. Moreover, we assume committee symmetry (see the remark above). Note that this implies q 1/2 = √ q d I K + ( √ q a + q d -√ q d )1 K 1 K /K and (Q 0 -q) 1/2 = √ 1 -q d I K + ( √ 1 -q a -q d -√ 1 -q d )1 K 1 K /K. All together, this yields the following expression for I C (y, ξ) :

I C (y, ξ) = R K DZ δ y -sign 1 √ K K l=1
sign 1 -q d Z l + ( 1 -q a -q d -1 -q d ) 1 K Z K + (q 1/2 ξ) l .

We introduce a new variable w ≡ 1 K Z/ √ K as well as another variable u being the argument of the outer sign function in the previous equations. Using the Fourier transform of the Dirac distribution one obtains:

I C (y, ξ) = R dwd ŵ 2π
dudû 2π e iw ŵ+iuû δ y,sign(u)

× K l=1 R Dze -i ŵ z √ K e -iû √ K sign z+ 1-qa-q d 1-q d -1 w √ K + 1 √ 1-q d
(q 1/2 ξ) l .

We denote

λ l (w, ξ) ≡ 1 -q a -q d 1 -q d -1 w √ K + 1 √ 1 -q d (q 1/2 ξ) l .
For 1 ≤ l ≤ K, one can rewrite the factorized integral in I C (y, ξ) as: By our committee-symmetry assumption, we have λ l (w, ξ) = λ l,0 (ξ) + λ 1 (w, ξ)/ √ K, with λ l,0 and λ 1 typically of order 1 when K → ∞ and given by:

I C (y, ξ) =
           λ l,0 (ξ) ≡ q d 1 -q d ξ l , λ 1 (w, ξ) ≡ 1 -q a -q d 1 -q d -1 w + q a + q d 1 -q d - q d 1 -q d 1 K ξ √ K .
(D.7)

Expanding J(λ l , ŵ, û) as K → ∞, we obtain using the asymptotic development of erf(x): + O(K -3/2 ) .

J(λ l , ŵ, û) = e -1 2K ŵ2 1 - û2 2K -i Ĥ[λ l,0 (ξ)] û √ K -i û[λ 1 (w, ξ) -i ŵ] K 2 π e -
This yields:

K l=1 J[λ l (w, ξ), ŵ, û)] = e -1 2 ŵ2 - û2 2 -iûS 1 -i 2 π û(λ 1 -i ŵ)Γ 0 + 1 2 û2 S 2 +O(K -1/2 ) , (D.8) 1 -S 2 -2 π Γ 2 0 qa 1-q d .
Introducing once again delta functions and their Fourier transforms for w ξ , Γ 0 , S 1 , S 2 , we write, starting from eq. (D.9): × ln G(y, w ξ , Γ 0 , S 1 , S 2 )

I C = y=±1 R K
R K
Dξe -i ŵw ξ (ξ)-i Γ0 Γ 0 (ξ)-i Ŝ1 S 1 (ξ)-i Ŝ2 S 2 (ξ) + O(K -1 ). (D.10)

The integral over ξ in eq. (D.10) can be computed in the limit K → ∞:

Λ ≡ R K
Dξ e -i ŵw ξ (ξ)-i Γ0 Γ 0 (ξ)-i Ŝ1 S 1 (ξ)-i Ŝ2 S 2 (ξ)

= R Dξ exp -i ŵξ √ K -i Γ0 e - q d 2(1-q d ) ξ 2 K -i Ŝ1 Ĥ q d 1-q d ξ √ K -i Ŝ2 Ĥ q d 1-q d ξ 2 K K
The large K expansion of this expression yields

Λ = exp - 1 2 ŵ2 -i Γ 1 -q d -Ŝ1 ŵE ξ Ĥ q d 1 -q d ξ - Ŝ2 1 + i Ŝ2 2 E Ĥ q d 1 -q d ξ 2 + O(K -1 ).
The expectations are taken with respect to a real variable ξ ∼ N (0, 1). They are known properties of the error function:

E Ĥ q d 1 -q d ξ 2 = 2 π
arcsin q d , and E ξ Ĥ q d 1 -q d ξ = 2q d π .

One can now compute the integrals over the "hat" variables in eq. (D.10). Denote Γ f 0 ≡ 2(1-q d ) π , and S f 2 ≡ 2 π arcsin q d . This yields:

I C = R 2 DwDS 1 G y, w, Γ f 0 , 2(arcsin q d -q d ) π S 1 + w 2q d π , S f 2 × ln G y, w, Γ f 0 , 2(arcsin q d -q d ) π S 1 + w 2q d π , S f 2 . (D.11) Note that G y, w, Γ f 0 , 2(arcsin q d -q d ) π S 1 + w 2q d π , S f 2 = H -y 2 π √ arcsin q d -q d S 1 + w √ q d + q a 1 -2 π (q a + arcsin q d )
.

Making the change of variable S new 1 = S 1 + w √ q d +qa √ arcsin q d -q d in eq. (D.11), and defining γ ≡ 2 π (q a + arcsin q d ), one reaches:

I C = y=±1 R DxH yx γ 1 -γ ln H yx γ 1 -γ + O(K -1 ).
The two values y = ±1 contribute in the same way, which finally yields:

I C = 2 R DxH x γ 1 -γ ln H x γ 1 -γ + O(K -1
). (D.12)

Note that the parameter γ is naturally bounded to the interval [0, 1] by the conditions 0 ≤ q d ≤ 1 and 0 ≤ q a + q d ≤ 1.

D.2.2 Limit of the prior integral

The prior part I P of the free entropy of eq. (B.6) is very easy to evaluate in the Gaussian prior setting. Recall that we consider a prior with covariance matrix Q 0 = I K . Performing the Gaussian integration in I P yields:

I P = K 2 qd + 1 2 qa - K -

D.2.3 Limit of the State Evolution

From the definition of the free entropy in eq. (B.6) and the expansions for I P and I C obtained in (D.12) and (D.13), one obtains the fixed point equations after having extremized over qd and qa (recall that α ≡ m/n):

∂ qa [I G (q d , q a ) + αI C (q d , q a )] = 0, ∂ q d [I G (q d , q a ) + αI C (q d , q a )] = 0, (D.14) with I G , I c defined as:

I G (q d , q a ) ≡ 1 2 [q a + Kq d ] + K -1 2 ln[1 -q d ] + 1 2 ln[1 -q a -q d ], I C (q d , q a ) ≡ 2 R DxH x γ 1 -γ ln H x γ 1 -γ ,
and recall that γ ≡ 2 π (q a + arcsin q d ). The fixed point equations (D.14) have different behavior depending on the scaling of α with the hidden layer size K. We detail them in the following paragraphs.

Regime α = O K→∞ (K)

In this regime (which in particular contains the case in which α stays of order 1 when K → ∞), the fixed point equations (D.14) can be simplified as:

q d = 0, q a = 2α(1 -q a ) ∂I C ∂qa .
(D.15)

Regime α = Θ K→∞ (K)
In this regime, we naturally define αK ≡ α/K, such that α will remain of order 1. One can show that the solutions of the fixed point equations (D.14) must satisfy the following scaling: q a + q d = 1 -χ/K, with χ ≥ 0 reaching a finite value when K → ∞. The fixed point equations in terms of χ and q d read: Stability of the q d = 0 non-specialized solution

   q d = 2(1 -q d ) 1 √ 1-q 2 d -1 α ∂I C ∂qa , χ -1 = 2 α
It is easy to show that eq. (D. 16) always admits what we call a non-specialized solution, i.e. a solution with q d = 0. This solution stops to be globally optimal in term of the free entropy at a finite α spec 7.65. However, one can show that this solution will remain linearly stable for every α. We can actually show that it is linearly stable in the much broader regime α = O(K 2 ). Let us now justify this statement. Going back to the initial formulation of the fixed point equations (D.14), and adding the correct time indices to iterate them, one obtains:

q t+1 d =
F (q t d , q t a ) 1 + F (q t d , q t a )

, (D.17) q t+1 a = G(q t d , q t a ) (1 + F (q t d , q t a ))(1 + F (q t d , q t a )G(q t d , q t a )) , (D.18)

with F and G defined as:

F (q d , q a ) ≡ 2α K -1 [∂ q d I C -∂ qa I C ], (D.19) G(q d , q a ) ≡ 2αK K -1 [∂ qa I C - 1 K ∂ q d I C ]. (D.20)
We focus on the behavior of (D.17) around q d = 0. Given our previous expansion of I C in the K → ∞ limit (cf. eq. (D.12)), and eq. (D. 19), one easily sees that for α = O K→∞ (K 2 ), then ∂ q d F (q d = 0) → K→∞ 0, which means the q d = 0 solution always remains linearly stable. However, assume now that α = Θ(K 2 ). Performing a similar calculation to the one described in Sec. D.2.1, one can show the following expansion:

I C (q d , q a ) = I (0)
C (q d , q a ) + 1 K I

(1)

C (q d , q a ) + O 1 K 2 .
The term of ∂ q d F (q d = 0) arising from I

(1)

C will thus have a possibly non-zero contribution in the K → ∞ limit as soon as α/K 2 is no longer negligible, as seen from eq. (D.19).

To summarize, the non-specialized solution always remains linearly stable in the large K limit at least for α K 2 . This implies that in this regime, Approximate Message Passing can not escape the non-specialized fixed point, as seen in Fig. 4.3. For α of order larger than K 2 , one would have to explicitly compute

I (1)
C in order to check that ∂ q d F (q d = 0) = 0, to show that the non-specialized solution is indeed linearly unstable. This verification of actual instability for α ∼ K 2 is not done in this thesis, and remains to be conducted.

D.2.4 The generalization error at large K

Recall the definition of the generalization error in eq. (D.1). Having this definition in mind, and recalling that this generalization error only depends on the asymptotic overlap, one can compute it at large K by applying the same techniques used to compute the channel integral I C in Sec. D.2.1. One obtains after a tedious (but straightforward) calculation:

k ) -1 W W k (zI k -Γ (0) k ) -1 W W k 2 = kS (1,2) k (z),
where in (a) we used the cyclicity of the trace. Given the last statement of Theorem 5.2, we know lim inf p→∞ λ 1 ≥ λ max , so we can use the above calculation to write from eq. (D.25):

(∆) = lim As in the eigenvalue transition proof, to make this fully rigorous one would need to use more precisely concentration results, and would follow exactly the lines of [START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF]. We now use the (already proven) transition of the leading eigenvalue, that gives us the value of λ 1 .

• For ∆ < ∆ c (α), we know that λ 1 converges almost surely to 1. Consequently: 2) (1)] 2 S (1,2) (1) .

(∆) = 1 α [S (
By Lemma 5.6, we know that S (2) (1) = -α∆. Moreover λ max < 1 by Theorem 5.2. This implies that S (1,2) (1) is finite and strictly positive, from its very definition. Therefore for every ∆ < ∆ c (α), (∆) > 0.

• For ∆ = ∆ c (α), we have λ max = 1 and lim λ→1 S (2) (λ) = -α∆ as we have shown. For every r, q, let us define the functions T (r) and T (r,q) by S (r) (λ) = T (r) [g ν (λ)] and S (r,q) (λ) = T (r,q) [g ν (λ)]. By Lemma 5.5 and the chain rule, we have for all s(s edge , 0): Recall that g -1 ν (s) is explicit by eq. (5.26) and s edge = lim λ↓λmax g ν (λ). It moreover satisfies (cf. Theorem 5.2) ∂ s g -1 ν (s edge ) = 0. For ∆ = ∆ c (α), by Lemma 5.4 we have g ν (1) = -1 = s edge . It is then only trivial algebra to verify from eq. (D.27) and the remaining relations of Lemma 5.5 that T (1,2) (-1) = +∞, which implies (∆ c (α)) = 0.

T (1,2) (s) = sT (3) (s) -[1 + sg -1 ν (s)] T (1,
• We investigate here the ∆ → 0 limit. In this limit, we know from eq. (D.26) and the analysis in the case ∆ < ∆ c (α) above that lim ∆→0 (∆) = lim ∆→0 α∆ 2 S (1,2) (1) .

It is again heavy but straightforward algebra to verify from eq. (D.27) and the remaining relations of Lemma 5.5 that as ∆ → 0 and for any s ∈ (s edge , 0):

T (1,2) (s) = α∆ 2 + O(∆ 3 ).
This yields lim ∆→0 (∆) = 1.

• Finally, we consider ∆ > ∆ c (α). By eq. (D.26) and item (iii) of Lemma 5.6, to obtain (∆) = 0 we only need to prove that lim λ→λmax S (1,2) (λ) = +∞. Equivalently, we must show lim s→s edge T (1,2) (s) = +∞. Recall that ∂ s g -1 ν (s edge ) = 0 and that since s edge is finite, all T (r) (s edge ) for r = 0, 1, 2, 3 are finite as well by Lemma 5.5. It thus only remains to check that lim s→s edge T (1,2) (s)∂ s g -1 ν (s) > 0. This would imply that lim s→s edge T (1,2) (s) = +∞. We put this statement as a lemma, actually stronger than what we need: Lemma D.1 (Lower bound on T (1,2) )

For every α > 0 and ∆ > 1, we have lim inf s→s edge T (1,2) (s)∂ s g -1 ν (s) > 0.

We prove this for every ∆ > 1, while only the case ∆ > ∆ c = 1 + α is needed in our analysis.

As already argued, this lemma ends the proof of the eigenvector correlation in Theorem 5.3.

Proof of Lemma D.1 -The idea is to lower bound S (1,2) (λ) by ∂ λ g ν (λ), for every λ > λ max . We separate three cases:

• First, assume α > 1. Then W W/k has full rank. In particular, by the classical results of [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF], its lowest eigenvalue, denoted ζ min , converges almost surely to (1-α -1/2 ) 2 . Moreover, for any two symmetric positive square matrices A and B, we know that Tr [AB] ≥ 0 2 . This implies immediately that if a 0 is the smallest eigenvalue of A, then Tr Taking the limit k → ∞ in this last inequality, we obtain:

S (1,2) (λ) ≥ (1 -α -1/2 ) 6 ∂ λ g ν (λ).
Taking the limit λ → λ max (or equivalently s → s edge ) yields the sought result: lim inf

s→s edge T (1,2) (s)∂ s g -1 ν (s) ≥ (1 -α -1/2 ) 6 > 0.
(D.28)

• Assume now α < 1. We do the same reasoning, as WW /k has now full rank, and its smallest eigenvalue, also denoted ζ min , converges a.s. as k → ∞ to (1 -√ α) 2 . We know that we can rewrite S in which the last inequality comes from Tr [AA ] ≥ Tr [A 2 ] for any positive square matrix A. Once again, taking the limit k → ∞, and then the limit λ → λ max , this yields lim inf s→s edge T (1,2) (s)∂ s g -1 ν (s) ≥ (1 -α 1/2 ) 6 > 0. (D.29)

• Finally, we treat the α = 1 case. In this case, we can not use easy bounds as in the two previous cases, since the support of the Marchenko-Pastur distribution touches 0. However, recall that everything is explicit here : ρ ∆ is given by eq. (5.23), g -1 ν (s) is given by eq. (5.26) and Lemma 5.5 gives all the T (r) and T (r,q) in terms of g -1 ν and ρ ∆ . We can moreover use what we proved in Theorem 5.2:

∂ s g -1 ν (s edge ) = 1 s 2 -α ρ ∆ (dt) t 2 (1 + ts edge ) 2 = 0.
This can be used to simplify the term ∂ s T (1) (s) and the term ρ ∆ (dt)[t/(1 + ts)] 2 . Some heavy but straightforward algebra yields from these relations that the following limit is finite, and is given by: lim s→s edge T (1,2) (s) ∂ s g -1 ν (s) = h(s edge ), with h(s) = h 1 (s) 2 × h 2 (s) 4s 6 , and h 1 (s) = -∆ + ∆ 2 + s 2 -2∆(2s + 1)s + s, h 2 (s) = 3∆ -3 ∆ 2 + s 2 -2∆(2s + 1)s + s(4s -3).

It is then very simple algebra (solving quadratic equations and using ∆ > 1) to see that there is no real negative solution to h(s) = 0, and that h(s) > 0 for all s ∈ (-∞, 0). This implies that h(s edge ) > 0, which ends the proof.

D.4 State evolution of spectral methods with generative prior

As we have already mentioned in Section 5.2.4, one of the greatest virtues of AMP is being able to track its asymptotic performance through a set of simple scalar state evolution equations. Note that for the noiseless linear channel P out (v|x) = δ(v -x), Algorithm 4 is already linear! As a consequence, the state evolution equations associated to the spectral method are simply dictated by the set of AMP state evolution equations eq. (5.18).

However, it is worth stressing that as LAMP returns a normalized estimator, the LAMP MSE is not given by the AMP mean squared error. We now compute the overlaps and mean squared error performed by this spectral algorithm.

MSE achieved by LAMP -

Recall that m v and q v are the parameters defined in eq. (5.16), respectively measuring the overlaps between the ground truth v and the estimator v, and the norm of the estimator. In the general case, the MSE of eq. (5.7) becomes:

MSE v = ρ v + E v lim p→∞ 1 p v 2 2 -2E v lim p→∞ 1 p v v = ρ v + q v -2m v , (D.30)
Letting q x = q z = qx = qz = γ x = γ z = 0, it is clear that the equations are satisfied if the signal distribution P 0 and the likelihood P out satisfy the symmetry conditions of Def. 6.1. Assuming these conditions hold, we are interested in studying the linear stability of this local maximum.

Recalling that Q z = ρ λ ν /α, the first, third and fourth equations of (D.33) can be linearized: Finally, it remains to compute the infinitesimal variation for δγ x , δγ z :

δq x =
         δγ x = λ 2 ν ρ 2 [ λ 2 ν -λ 2 ν ] δq x - α λ ν ρ 2 [ λ 2 ν -λ 2 ν ] δq z , δγ z = - λ ν ρ 2 [ λ 2 ν -λ 2 ν ] δq x + α ρ 2 [ λ 2 ν -λ 2 ν ]
δq z .

(D.36)

Combining eqs. (D.34),(D.35),(D.36), we can simplify the system to a closed set equations over only (δq x , δ qx , δq z , δ qz ). Given the usual heuristics of the replica method and its link with message-passing algorithms [ZK16, TK20], one can easily check that the following time iteration of these equations corresponds to the state evolution of the G-VAMP algorithm:

                              
δq t x + α λ ν ρ 2 [ λ 2 ν -λ 2 ν ] δq t z , δ qz t = λ ν ρ 2 [ λ 2 ν -λ 2 ν ] δq t x + [ α 2 ρ 2 λ 2 ν - α ρ 2 [ λ 2 ν -λ 2 ν ]
]δq t z .

(D.37)

From these equations, one can easily see that a linear instability of the trivial fixed points appears at α = α WR,Algo satisfying eq. (6.13). Indeed at α = α WR,Algo , the modulus of all the eigenvalues of the size-4 matrix of the linear system (D.37) cross 1.

D.5.2 Perfect recovery

In this section, we assume a Gaussian standard prior P 0 = N β (0, 1) and a noiseless phase retrieval channel, and we show that information-theoretic full recovery is achieved exactly at α = α FR,IT ≡ β(1 -ν({0})). We can assume without loss of generality that λ ν = α, as this amounts to a simple rescaling of Φ, irrelevant under the noiseless channel. This implies in particular that Q z = Qz = 1.

have by Gaussian integration by parts and using the specific form of P out : ∼ 4c.

Note that in the complex case the derivative P out is here taken over z considered as an element of R 2 : it is not the Wirtinger derivative of complex analysis. This yields 1-q z = ∆c(1-4c)+O ∆ (1). Combining this result with eq. (D.41), we have

c(1 -4c) = c α -1 + ν({0}) α .
This implies c = (1 -ν({0}))/(4α), and we finally obtain the leading order asymptotics of q z , qz , γ z as ∆ ↓ 0:

                 qz = 4α (1 -ν({0}))∆ + O ∆ (∆ -1 ), 1 -q z = (1 -ν({0})(α -1 + ν({0})) 4α 2 ∆ + O ∆ (∆), γ z = 4α ∆(α -1 + ν({0}))
+ O ∆ (∆ -1 ).

(D.42)

We then compute the asymptotics of the three auxiliary functions of Conjecture 6.1. Using eq. (D.42) and the specific form of the channel, we reach: I 0 (q x ) + I int (q x , q z ) ∼ -β(α -1 + ν({0})) 2 ln ∆, I out (q z ) ∼ (β -1) 2 ln ∆.

Therefore when considering the total free entropy we have I 0 (q x ) + I int (q x , q z ) + αI out (q z ) ∼ β(1 -ν({0})) -α 2 ln ∆.

This implies that the full recovery point has a free entropy of -∞ for α < α FR,IT ≡ β(1-ν({0})), and +∞ for α > α FR,IT . Thus this point is always the global maximum of the free entropy for α > α FR,IT , while it is never the case for α < α FR,IT , which ends our argument.

D.6 Details of the spectral methods analysis for phase retrieval D.6.1 The expansion of F (x, y) around y = 0

We describe here the behavior of F (x, y) defined in eq. (6.25) as x > 0 and y ↓ 0. Let us write the equations satisfied by ζ x , ζ y :

ζ y ζ x ζ y + λ ν = x, α -1 ζ y + ζ x ζ x ζ y + λ ν = αy. (D.43)
As y ↓ 0, this implies necessarily that ζ y → +∞, and one finds easily that ζ y ∼ 1/y, ζ x ∼ 1/x. We now turn to the next order variations, that we write as:

ζ y = y -1 + c 1 + O(y), ζ x = 1 x + c 2 y + O(y 2 ).
We use eq. (D.43) to compute c 1 = -x λ ν /α and c 2 = -λ ν . We can then develop: This yields for F (x, y):

F (x, y) = - λ ν 2 xy + x 2 4α α λ 2 ν -(1 + α) λ 2 ν y 2 + O(y 3 ),
which concludes our analysis.

D.6.2 Proof of Proposition 6.6

Let us recall the two spectral methods M (TAP) , M (LAMP) . Without loss of generality, we assume λ ν = α. Recall that we defined z µ ≡ ∂ ω g out (y µ , 0, ρ). We let Z ≡ Diag(z µ ). We can thus write:

M (LAMP) = ρ ΦΦ † n -I m Z and M (TAP) = - 1 ρ I n + 1 n Φ † Z I m + ρZ Φ.
We start by the first claim. By definition of (λ LAMP , v), we have ρ ΦΦ † n Zv = (ρZ + λ LAMP )v. (D.45)

Since we assumed λ LAMP + ρz µ = 0 for all µ, this implies that Φ † Zv = 0, and we thus let

x ≡ Φ † Zv Φ † Zv √ n.
Multiplying eq. (D.45) by Φ † Z(λ LAMP +ρZ) -1 on both sides, we directly reach the sought result:

1 n Φ † ρZ λ LAMP + ρZ Φ x = x.
We move on to the second claim. Let x ∈ K n be an eigenvector of M (TAP) with norm x 2 = n, with associated eigenvalue λ TAP . We let:

u ≡ I m I m + ρZ Φ √ n x.
And we can then easily compute: At α = α WR,Algo , the largest eigenvalue of M (TAP) concentrates on 0, which corresponds to the onset of marginal instability of the trivial local maximum. As one can see from eq. (D.46), this implies that M (LAMP) also possesses an eigenvalue equal to 1 at α = α WR,Algo , indicating marginal instability as well. To put it shortly, the two spectral methods have the same weak recovery threshold. Moreover, eq. (D.46) implies that for any α ≥ α WR,Algo , if M (TAP) has en eigenvalue that concentrates on 0 as n → ∞, then M (LAMP) has a corresponding eigenvalue concentrating on 1, and with the same performance. Indeed, as described in eq. (6.24), the estimator associated to M (LAMP) will be given by:

M (LAMP) u = ρ ΦΦ † n -I m Z I m + ρZ Φ √ n x = ρΦ √ n M (
xLAMP ∝ Φ † √ n Zu = Φ † √ n Z I m + ρZ Φ √ n xTAP ,
in which xTAP is an eigenvector of M (TAP) with eigenvalue 0. Therefore, we reach that xLAMP ∝ xTAP , and these two vectors are thus equal as they are both normalized.

• For every a ≤ b, we fix the linear statistics ν(dλ) φ (λ a )φ (λ b ) = A ab , with Lagrange multipliers Âab . Note that by replica symmetry, we can assume that A ab = a for a = b and A aa = A (and similarly for the Lagrange multipliers).

• For all a, b we fix the linear statistics ν(dλ) φ (λ a )λ b = B ab , with Lagrange multipliers Bab . By replica symmetry, we assume that B aa = B and B ab = b (and similarly for Bab ).

Combining these remarks, we reach that the ν-dependent term of eq. (7. Note that here we did not always explicit the replica-symmetry assumption on all the variables to obtain more compact expressions. The supremum over B, b is moreover constrained by the following condition of eq. (7.34b): ∀a = b, z p (q)B aa + f 0 p (q)B ab + f p (q) c( =a,b) B ac = 0. Under the replica symmetric assumption, this becomes: z p (q)B + f 0 p (q)b + f p (q)(p -2)b = 0. (E.8) Again, we introduce Lagrange multipliers C ab to fix these conditions, that reduce to C ab = C because of replica symmetry. Finally, in order to fix the marginal distributions of ν, we will have to introduce "functional" Lagrange multipliers g a (λ a ). Again, by replica symmetry, we expect all of them to be equal to g(λ a ). In the end, we obtain from eq. (E. We can now solve exactly the supremum over ν. By a classical Gibbs measure calculation that we already detailed in Section 7.3 we obtain (recall that Q ∈ S p is the overlap matrix): (i) {Y (t), t ∈ U } is a Gaussian random field with values in R k , C 1 paths, and such that for every t ∈ U , the distribution of Y (t) is non-degenerate.

sup ν∈M(R n ) ν(dλ)
(ii) H : R k → R d is a C 1 function.

(iii) For all t ∈ U , Z(t) has a density ϕ Z(t) (x) which is a continuous function of (t, x) ∈ U ×R d .

(iv) P[∃ t ∈ U s.t. Z(t) = 0 and det ∇Z(t) = 0] = 0.

For every compact set B ⊆ U , we let N (Z, B) the (finite) number of zeros of Z in B. Then:

E[N (Z, B)] = B E | det ∇Z(t)| Z(t) = 0 ϕ Z(t) (0)dt.
We wish to apply this theorem to Z(x) = grad L 1 (x). Verifying its hypotheses will end the proof of Lemma 7.7. We denote ξ ∈ R n×m the matrix {ξ iµ } = {(ξ µ ) i }, ∇L 1 the Euclidean gradient of L 1 , and P ⊥ x the orthogonal projection on T x S n-1 . Since grad L 1 (x) = P ⊥ x ∇L 1 (x) we have: . Since Y (x) is just ξ written in an orthonormal basis of R n whose last vector is x, its distribution is non-degenerate for every x. H : R n×m → R n-1 is defined as:

∀1 ≤ i < n, H(Y ) i ≡ 1 m m µ=1 Y i,µ φ (Y n,µ ), (Y ∈ R n×m ).
Since φ is C 2 , H is C 1 . This verifies (i) and (ii). We turn our attention to verifying (iii). One can write the distribution of the gradient of eq. (E.13) as grad L 1 (x) ∼ N (0, I n-1 ), and all {y µ , z ν } are independent. Since the distribution of grad L 1 (x) does not depend on x, it is enough to check that its density exists and is a continuous function. To do so, we will show that its characteristic function φgradL 1 (x) ∈ L 1 (R n-1 ). We denote φa the characteristic function of the random variable a ≡ φ (y), and one obtains:

φgradL 1 (x) 1 = R n-1 dt E z∼N (0,I n-1 ) φa t • z m m = R n-1 dt E z∼N (0,1) φa t z m m , = 2π n-1 2 m n-1 Γ n-1 2 ∞ 0 dq q n-2 |E z φa (qz) m .
Since α > 1, if qE z φa (qz) = O q→∞ (1) we can conclude that φgradL 1 (x) 1 < ∞. And:

q E z φa (qz) = R dz √ 2π e -z 2 2q 2 φa (z) = R dz √ 2π E e -z 2
2q 2 e iaz = 1 q E e -q 2 a 2 2 ,

We have for all y and t > 0 ( Ā(n)

Concentration of the logarithmic potential with a cut-off

We show here that discarding the eigenvalues of the Hessian that are close to 0 using a cut-off n ≡ n -δ , we have concentration of the logarithmic potential.

Proposition E.8 (Concentration of the logarithmic potential)

Let us fix δ < 1/2 and recall that n = n -δ . Then:

∀t > 0, lim sup n→∞ sup y∈R m 1 n 2(1-δ) ln P z 1 n ln n | det H Λ n (y)| -E z ln n | det H Λ n (y)| ≥ t < 0.
Proof of Proposition E.8 -We will try to use traditional Lipschitz concentration bounds (cf e.g. [START_REF] Greg W Anderson | An introduction to random matrices[END_REF]). We will study under which conditions the function G(z) ≡ (1/n) ln n | det HΛ n (y)| is a Lipschitz function of z ∈ R (n-1)×m (for fixed y). We will do it by bounding ∇ z G ∞ . Let f n (x) ≡ ln n |x| for x ∈ R. We have: We give here a sketch of the generalization of our annealed and quenched calculations to L 2 , yielding Theorem 7.6 and Result 7.2. We restrict here to the annealed calculation (the generalization of the quenched calculation is completely similar). The majority of the arguments being identical to the L 1 case, we will only highlight the main differences and give the important intermediary results.

In the Kac-Rice formula, one has to integrate over the overlap q ≡ x • x as well. Moreover, we condition over the joint values of a µ ≡ ξ µ • x and b µ ≡ (1 -q 2 ) -1/2 [(ξ µ • x ) -qa µ ], rather than just ξ µ • x (as we did for L 1 ). Note that (a µ , b µ ) follows a joint standard Gaussian distribution. Using these definitions we can obtain the counterpart of Lemma 7.8 for L 2 : in which C n is exponentially trivial, and we defined:

E Crit n,L2 (B, Q) = C n Q dq e
                                   P n (a, b) ≡ 1 m m µ=1 b µ φ (a µ )[φ(qa µ + 1 -q 2 b µ ) -φ(a µ )], E n (a, b) ≡ 1 2 ln m µ=1 φ (a µ ) 2 [φ(qa µ + 1 -q 2 b µ ) -φ(a µ )] 2 , H n (a, b) ≡ 1 m m µ=1 φ (a µ ) 2 -θ (a µ )[φ( 1 -q 2 b µ + qa µ ) -φ(a µ )] z µ z µ - 1 m m µ=1 a µ φ (a µ )[φ(a µ ) -φ(qa µ + 1 -q 2 b µ )] I n-2 .
Here z ∈ R (n-2)×m is an i.i.d. standard Gaussian matrix. The condition P q (a, b) = 0 arises from the conditioning on the nullity of the gradient in the linear subspace of {x} ⊥ spanned by x , and E n (a, b) from the density of the gradient in the subspace orthogonal to {x, x }. A crucial feature of this equation is that the joint distribution of (L 2 (x), grad L 2 (x), Hess L 2 (x)) only depends on x via the overlap q = x • x with the "true" solution. Once conditioned over the values of q, it thus becomes clear why the calculations made for L 1 will generalize here.

As in Section 7.3.3, one can then show the concentration of the empirical logarithmic potential on the functional κ α,φ (q, ν m a,b ), in which ν m a,b ∈ M + 1 (R 2 ) is now the empirical measure of {a µ , b µ } m µ=1 . We obtain the counterpart of Lemma 7.10: there exists η > 0 such that for all t > 0: Thanks to this result, we apply then Laplace's method on the overlap q and the empirical measure ν ∈ M(R 2 ), using Sanov's theorem 1.9 and Varadhan's lemma 1.10. This yields the result of Theorem 7.6.

As a final note, there exists similar results to the one presented in Section 7.4 that allow to compute the density (and the logarithmic potential) of µ α,φ [q, ν], via the computation of its Stieltjes transform.

If η ≥ 1, we have G ρ (d max ) < ∞, so that F σ (α/d max ) < ∞, and thus G σ (x) → 1/F σ (α/d max ) > 0 as x ↑ x c (ρ). The transition in I(x) is thus of second order in this case, as G σ (x) is discontinuous.

If we now assume that η < 1, we have G ρ (d max ) = +∞. By eq. (E.37), this implies that G σ (x) → 0 as x ↑ x c (ρ). Thus in this case both G σ and G σ are continuous in x = x c (ρ). We can differentiate the relation F σ (G σ (x)) = x once more, and we find easily:

G σ (x) = - F σ (G σ (x)) F σ (G σ (x)) 3 . (E.38)
From eqs. (E.37), (E.38), one can show that G σ (x) → 0 as x ↑ x c (ρ) if and only if η < 1/2. In particular, for any 1/2 ≤ η < 1, the transition in I(x) is of third order.

Differentiating three times, one can show in a similar way that the transition is of fourth order if and only if η ∈ [1/3, 1/2). Generalizing this to any order, we conjecture that I(x) is smooth at any point x = x c (ρ), and that the first discontinuous derivative of the rate function at x = x c (ρ) is I (k+1) (x), with η ∈ [1/k, 1/(k -1)) (with the convention 1/0 = +∞).

E.5 Technicalities on spherical integrals

E.5.1 Simplifying J 1 (θ, x)

In this section, we simplify the expression of J 1 (θ, x) when θ ≤ θ c (x) ≡ G σ (x). We start from eq. (8.12):

J 1 (θ, x) = inf γ>θx γ 2 - 1 2 du σ(u) ln(γ -θu) - 1 2 . (E.39)
When θ ≤ G σ (x) the infimum is reached in γ = θG -1 σ (θ). This implies

J 1 (θ, x) = θG -1 σ (θ) 2 - 1 2 ln θ - 1 2 du σ(u) ln(G -1 σ (θ) -u) - 1 2 .
Let us differentiate this expression with respect to θ:

∂ θ J 1 (θ, x) = G -1 σ (θ) 2 + θ 2G σ (G -1 σ (θ)) - 1 2θ - G σ (G -1 σ (θ)) 2G σ (G -1 σ (θ)) = G -1 σ (θ) 2 - 1 2θ .
Using now the Marchenko-Pastur equation (8.1) we can simplify this into:

∂ θ J 1 (θ, x) = α dt ρ(t) t α -tθ = F 1 (θ).
Since J 1 (0, x) = F 1 (0) = 0, this implies that for every θ ≤ θ c (x) we have J 1 (θ, x) = F 1 (θ), which justifies the claim made in the main text.

E.5.2 Derivations of F 2 (θ) and J 2 (θ, x)

The derivation of F 2 (θ)

We start from the definition of F 2 (θ): We introduce a Lagrange multiplier γ to fix the norm of f. This yields (recall α = m/n):

F 2 (θ) = inf γ≥θ 2 dmax/α 2 αγ 2 - 1 2n ln det γI m - θ 2 α 2 D m - α 2 + O n (1).
The condition γ ≥ θ 2 d max /α 2 arises as the matrix inside the determinant must be positive. Changing variables by letting γ = θ 2 γ /α 2 we arrive at:

F 2 (θ) = α 2 inf γ≥dmax θ 2 γ α 2 -dt ρ(t) ln(γ -t) - α 2 ln θ 2 α 2 - α 2 .
This ends the derivation of the expression of F 2 (θ) given in the main text.

Computing J 2 (θ, x)

The goal of this section is to compute J 2 (θ, x). More precisely, we will first show eq. (E.42), which will then be simplified, precisely showing the transition phenomenon described in the main text.

We start from the definition of J 2 (θ, x) (we omit the writing of the n → ∞ limit): We introduce two Lagrange multipliers to fix the norms of e and f. Let us start with the computation of the denominator:

1 n ln de df δ(||e|| 2 -n)δ(||f 2 || -m) inf Λe,Λ f ≥0 Λ e 2 + αΛ f 2 - 1 2 ln Λ e - α 2 ln Λ f + (1 + α) 2 ln 2π .
The positivity constraint on Λ e , Λ f arises naturally for the Gaussian integral to be well-defined. This is easily solved by Λ e = Λ f = 1, and we arrive at:

1
Λ e + αΛ f -(α -1) ln Λ f -dλ σ(λ) ln Λ e Λ f - θ 2 α λ - 1 + α 2 .
The transition in J 2 (θ, x)

We start from the expression of J 2 (θ, x) of eq. (E.42). The variational parameters Λ e , Λ f can saturate, which is associated to a phase transition. At this point J 2 (θ, x) will become sensitive to the largest eigenvalue of H n (assumed to be equal to x).

Given the infimum in eq. (E.42), this phase transition occurs for θ = θ c (x) such that the corresponding values of Λ e , Λ f satisfy αΛ e Λ f = θ c (x) 2 x. From this equation and the equations on Λ e , Λ f obtained by making the derivative inside the infimum equal to 0 (which is valid for θ ≤ θ c (x)), it is easy to obtain θ c (x) = xG σ (x) 2 + (α -1)G σ (x).

The case θ ≤ θ c (x) -In this case J 2 (θ, x) is not sensitive to the value of x, and we can use a very useful expression derived in Chapter 7 (more precisely eq. (7.23)) for the log-potential of σ(λ). For any x ≥ s max : dλ σ(λ) ln(x -λ) = inf 0<g<Gσ(smax)

ln g + zg + α dt ρ(t) ln(α -tg) -1 -α ln α.

This infimum is attained at g = G σ (x), as it is the unique zero of the derivative of the expression above in the interval (0, G σ (s max )), by eq. (8.1). We can then write eq. (E.42) as: Since we are in the "no-saturation" regime, we can use the zero-gradient equations on Λ e , Λ f : Λ f = θ 2 /(αg), Λ e = θ 2 /g -(α -1).

J 2 (θ, x) = - α(1 -ln α) 2 - 1 2 ln θ 2 α + 1 2 inf 0<g<Gσ(smax)
Plugging this back into eq. (E.43) we obtain:

J 2 (θ, x) = 1 2 inf 0<g<Gσ(smax)
θ 2 g + α ln g -α ln θ 2 α -α dt ρ(t) ln(α -tg) + α(ln α -1) .

Changing variables γ = α/g, we reach:

J 2 (θ, x) = α 2 inf γ≥α/Gσ(smax)
γθ 2 α 2 -dtρ(t) ln(γ -t) -

α 2 1 + ln θ 2 α 2 . (E.44)
In order to map J 2 (θ, x) to F 2 (θ), we must only show that the Lagrange multiplier γ in eq. (E.44) does not "saturate" for θ ≤ θ c (x). This is easily shown using the Marchenko-Pastur equation (8.1). Indeed since θ ≤ θ c (x) we have θ ≤ θ c (s max ), and thus:

θ 2 ≤ s max G σ (s max ) 2 + (α -1)G σ (s max ) ≤ αG σ (s max ) 1 + G σ (s max ) s max α - 1 αG σ (s max ) , ≤ αG σ (s max ) 1 + G σ (s max ) dt ρ(t) t α -tG σ (s max ) ≤ α 2 dt ρ(t) α/G σ (s max ) -t .
This precisely means that the infimum in eq. (E.44) will be attained for a point γ which is a critical point of the functional inside the infimum:

θ 2 α 2 = dt ρ(t) 1 γ -t ,
i.e. there is no saturation in both J 2 and F 2 , and therefore we have J 2 (θ, x) = F 2 (θ) in this case.

The case θ ≥ θ c (x) -In this case, we have a "saturation" in the infimum of eq. (E.42). More precisely, the Λ e , Λ f attaining the infimum satisfy αΛ e Λ f = θ 2 x. One can solve the infimum over Λ e , Λ f constrained by this equality. Introducing a Lagrange parameter ρ, we reach:

J 2 (θ, x) = - 1 + α 2 - 1 2 ln θ 2 α - 1 2 dλ σ(λ) ln(x -λ) + 1 2 inf Λe,Λ f ≥0 extr ρ Λ e + αΛ f -(α -1) ln Λ f -ρ Λ e Λ f - θ 2 x α .
The extr notation denotes solving the associated zero-gradient equation, as is standard with Lagrange multipliers. One can now solve the infimum over Λ e , Λ f easily, and we reach:

J 2 (θ, x) = 1 2 extr ρ α ρ + (α -1) ln ρ + ρθ 2 x α -ln θ 2 α -dλ σ(λ) ln(x -λ) - 1 + α 2 .
This can also be solved easily, and finally we have, for θ ≥ θ c (x):

J 2 (θ, x) = 1 2 -(1 + α) -α ln θ 2 α -(α -1) ln(2x) + (α -1) 2 + 4xθ 2 + (α -1) ln 1 -α + (α -1) 2 + 4xθ 2 -dλ σ(λ) ln(x -λ) .

This ends the argument by justifying all the expressions given for J 2 (θ, x) in the main text.

A remark on the case d max ≤ 0 -In this case s max ≤ 0, and the transition we described does not take place, as Λ e , Λ f ≥ 0 can not satisfy αΛ e Λ f = θ 2 x < 0. The difference between the quenched and annealed integrals in this case has, as far as we know, not been investigated before, and it remains an open question. Importantly, in this setting the first tilting allowed to derive the large deviations, as emphasized in the main text, so that solving this question is not crucial for our purpose.

E.6 Simplifying the rate function E.6.1 When x < x max , in the first tilting

The goal of this section is to show, for all x < x max :

I(x) = sup θ∈(0,θmax) [J 1 (θ, x) -F 1 (θ)] = 1 2 x smax [G σ (u) -G σ (u)]du,
and that the maximum in θ is reached in θ x = G σ (x). Recall eq. (8.18):

J 1 (θ, x) =        F 1 (θ) = - α 2 dtρ(t) ln(1 -α -1 θt) if θ ≤ G σ (x), θx -1 -ln θ 2 - 1 2 dλ σ(λ) ln(x -λ) if θ > G σ (x).
Differentiating with respect to θ, we reach:

∂ θ [J 1 (θ, x) -F 1 (θ)] =      0 if θ ≤ G σ (x), θx -1 2θ - α 2 dt ρ(t) t α -θt if θ > G σ (x).
So the supremum sup θ∈(0,θmax) [J 1 (θ, x) -F 1 (θ)] is attained for θ = θ x > G σ (x) that satisfies:

x = 1 θ + α dtρ(t) t α -θt .
Note that this is exactly the Marchenko-Pastur equation (8.1), so that θ x is precisely the second "branch" θ x = G σ (x). Moreover, we know that I(s max ) = 0, and we conclude by noticing that:

I (x) = ∂ x [J 1 (θ x , x) -F 1 (θ x )] = θ x 2 - 1 2 dλ σ(λ) x -λ = 1 2 G σ (x) -G σ (x) .

E.6.2 The second tilting

Our objective is to show, for all x ≥ s max :

I(x) = sup

[MFC + 19 ]

 19 High-temperature expansions and message-passing algorithms. Antoine Maillard, Laura Foini, Alejandro Lage Castellanos, Florent Krzakala, Marc Mézard and Lenka Zdeborová. Journal of Statistical Mechanics: Theory and Experiment 2019 (11), 113301.

→β

  Shortcut for a and b having same probability distribution. Limit in the weak, probability, almost sure sense. S n-1Unit sphere in K n .

Figure 1 :

 1 Figure 1: Number of operations needed by Euclid's algorithm to compute the greatest common divisor of two integers.

Figure 2 :

 2 Figure 2: Example of a rugged landscape in dimension two. Picture taken from [LXT + 18].

  (a) A low-temperature ferromagnetic state in which all spins are aligned in a common direction.(b) A spin glass state: the spins are frozen in random directions due to the randomness of the interactions.

Figure 1 . 3 :

 13 Figure 1.3: Schematic representation of a ferromagnetic and a spin glass state in a 3-dimensional model with sites on a cube.

Figure 1 . 4 :

 14 Figure 1.4: Average magnetization as a function of the temperature T = η -1 for the Curie-Weiss model of eq.(1.17) with infinitely small external field h ↓ 0.

Figure 1

 1 Figure 1.6: (Left) A generic "rough" landscape f (x, y) with several deep minima in dimension 2.(Right) Slices z = f (x, y), shown as planes in the left picture. For z high enough (high energy, or high temperature by the microcanonical-canonical equivalence in the thermodynamic limit) the slices are well-defined by a single "pure state". As the energy decreases the slices get disconnected, and for low enough energy they are concentrated around the location of the deep minima of f (x, y). This low-dimensional view is merely an intuition and should not be considered as anything more.

Figure 1 . 7 :

 17 Figure 1.7: Representation of a factor graph (Fig. 1.7a) and of the updates of the generic BP equations (1.38) (Figs. 1.7b and 1.7c).

Figure 2 . 1 :

 21 Figure 2.1: Diagrammatic representation of the perturbative terms. Each vertex represents an index i α and carries a factor v iα , while each edge is a factor J ij . Each connected component of the diagram carries a global factor n -1 .

Figure 2 . 2 :

 22 Figure 2.2: Cactus and non-cactus connected diagrams. Each vertex represents an index i over which we sum, and each edge is a factor J ij . Each diagram carries a global factor n -1 .

Figure 2 . 3 :

 23 Figure 2.3: Two possible decompositions of the diagram of Fig. 2.2b into simple cycles.

  .54) In this formula, the sum α C α represents all the possible diagrams C α that one can obtain by 'gluing' together two replicas of the diagram G. Let us first see why it implies (A) and (B): all diagrams C α have a negligible expectation by the first moment analysis we performed in Sections 2.4.1,2.4.2. So for every kind of diagram we considered we have E G 2 = (E G) 2 + O n (1).

  Any non-Eulerian diagram G has vanishing first and second moments: lim n→∞ E G 2 = 0.

Figure 4

 4 Figure 4.1:The committee machine is one of the simplest models of two-layers neural networks. Both activation functions are sign functions f (1) , f (2) = sign and the second-layer weights W(2) are all fixed to 1. Here we represent a generic two-layers neural network with K = 3.

Figure 5

 5 Figure 5.1: (Spiked Wigner model) MMSE v as a function of noise to signal ratio ∆/ρ 2 v , and single-layer generative prior with compression ratio α for linear (left), sign (center), and ReLU (right) activations. Dashed white lines mark the transition ∆ c for odd activation. Dotted white lines mark the weak-recovery transition of canonical PCA.

Figure 5 Figure 5

 55 Figure 5.2: (Spiked Wigner model) MMSE v as a function of noise ∆ for L = 1 and a wide range of compression ratios α ∈ {0, 1, 10, 100, 1000}, for linear (left), sign (center), and ReLU (right) activations. For the linear and sign functions, we show in dotted lines the weak-recovery transition ∆ c . In all cases we find a unique solution for eq. (5.10).

  ) the reader should refer to [ALM+ 20].

Figure 5 . 4 :

 54 Figure5.4: Comparison between PCA, LAMP and AMP for linear (left) and sign (center) activations, at compression ratio α = 2. Full lines correspond to the asymptotic performances obtained from the SE equations, while dots correspond to simulations. We use k = 10 4 for LAMP, k = 5.10 3 for AMP. (Right) Illustration of the spectral phase transition in the matrix Γ vv p of eq. (5.21) at α = 2 with informative leading eigenvector with eigenvalue equal to 1 out of the bulk for ∆ ≤ 1 + α. We show the bulk spectral density µ(α, ∆) and the two leading eigenvalues (inset).

Algorithm 5 :

 5 LAMP (Linearized AMP) spectral algorithmInput: Observed matrix Y ∈ S p , priorP v on v ∈ R p ; Take the leading eigenvector v ∈ R p of Γ vv p ≡ K p [ Y √ p -I p ] with K p = E Pv [vv ];generic definition for LAMP as Algorithm 5. LAMP can therefore be interpreted as a refined version of PCA that takes into account the structure of the prior by incorporating the non-trivial correlations through K p into the spectral estimation. In particular taking P v = N (0, I p ), we obtain Γ vv p = [Y/ √ p -I p ]/∆ and recognize the vanilla PCA operator (up to a shift).State evolution for LAMP in the linear case -Analogously to the state evolution for AMP, the asymptotic performance of both PCA and LAMP can be evaluated in a closed form for the spiked Wigner model with single-layer generative prior with linear activation. The corresponding expressions are derived in Appendix D.4 and plotted in the left panel of Fig.5.4.

Figure 5 . 5 :

 55 Figure 5.5: Illustration of canonical PCA (top line) and LAMP (bottom line) spectral methods on the spiked Wigner model. The covariance Σ is estimated empirically from the FashionMNIST database. The estimation of the spike is shown for two images from FashionMNIST, with (from left to right), noise variance ∆ = 0.01, 0.1, 1, 2, 10.

Figure 6 . 1 :

 61 Figure 6.1: Schematic of a Coherent Modulation Imaging set-up, a typical optical experiment in which one needs to retrieve measurements from the observation of the modulus of their projections (here by a modulator), which can be modeled by eq. (6.2). Picture from [ZCM + 16].

  .10c) where f 0 (b, a) = ∂ b ln Z 0 (b, a) and f out (y; ω, v) = ∂ ω ln Z out (y; ω, v) are evaluated at (b, a) = ( √ qx ξ, qx ) and (ω, v) = qz Qz( Qz+qz)

Figure 6 . 2 :

 62 Figure 6.2: Comparison of the Bayes-optimal MSE and the G-VAMP algorithm, for an i.i.d. real Gaussian (left) and a column-orthogonal (right) sensing matrix Φ (i.e. Φ Φ/n = I n ), with a realGaussian prior. Dots correspond to finite size simulations of G-VAMP (the mean and std are taken over 5 instances, with n = 8000 in the Gaussian case and m = 8192 in the orthogonal case), while full lines are obtained from the state evolution. The vertical grey dashed lines denote the algorithmic weak recovery threshold α WR,Algo . Note the presence of a statistical-to-algorithmic gap in both ensembles, and that for column-orthogonal matrices α WR,Algo > α FR,IT .

Figure 6 . 3 :

 63 Figure6.3: We show quantities identical to Fig.6.2, but for an i.i.d. complex Gaussian (left) and a column-unitary (right) sensing matrix Φ (i.e. Φ † Φ/n = I n ), with a complex Gaussian prior. We used n = 5000 in the simulations of G-VAMP.

Figure 6 . 4 :

 64 Figure 6.4: Mean squared error as a function of the measurement rate α, for a sensing matrix Φ = W 1 W 2 a product of two complex i.i.d. standard Gaussian matrices W 1 ∈ C m×p , W 2 ∈ C p×n with aspect ratios γ = p/n ∈ {0.5, 1.0, 1.5}. Red curves denote the recovery on ΦX / √ n and blue curves on X . Cyan dashed lines denote the full reconstruction threshold α FR,IT . The G-VAMP experiments were performed with n = 5000, and the mean and std are taken over 5 instances.

Figure 6 . 5 :

 65 Figure 6.5: MSE achieved by our spectral methods and a naive version of the spectral method of [MM19] for real column-orthogonal sensing matrices and a noiseless channel. We give the performance on uniformly sampled column-orthogonal matrices as well as randomly subsampled Hadamard matrices. The simulations were done using m = 8192, and the error bars are taken over 10 instances.

( a )

 a Product of two real i.i.d. Gaussian sensing matrices with a size ratio γ ∈ {0.5, 1.0, 2.0}. The simulations were done using m = 10000, and error bars are taken over 10 instances.(b) Complex Gaussian matrix, in noiseless phase retrieval and in Poisson-noise phase retrieval with Λ = 1. The simulations were done using m = 10000 (noiseless case), 12000 (Poisson case), and the error bars are taken over 10 (noiseless case), 5 (Poisson case) instances.

Figure 6 . 6 :

 66 Figure 6.6: MSE achieved by the different spectral methods in two different settings.

Figure 6 . 7 :

 67 Figure 6.7: Transition in the spectra of M (LAMP) (left) and M (TAP) (right) for a complex Gaussian Φ and a Poisson channel with Λ = 1. For α > α WR,Algo = 2, we indicate the approximate overlap q corresponding to the the relevant eigenvalues.

Figure 6 .

 6 Figure6.10: Reconstruction of a real image in noiseless phase retrieval with partial DFT matrices. We reduce the image size from 1280 × 820 to 128 × 82. We compare, for three different values of α, the estimators of M(T * ) (top line) and the estimator obtained by running a gradient descent procedure starting from the estimator of M(T * ) (bottom line). We recover the 3 RGB channels with independent instances of the sensing matrix.

  (a) A "rough" optimization landscape, with a great number of spurious local minima and saddle points.(b) A smooth landscape, intuitively easier to optimize via local algorithms.

Figure 7 . 1 :

 71 Figure 7.1: "Naive" representation of a rough (left) and smooth (right) optimization landscape. Both landscapes have a global minimum with the same value. Pictures taken from [LXT + 18].

Figure 7 . 3 :

 73 Figure 7.3: The functions Θ k for k ∈ 0, 1, 2, 10, 100, for two values of p.

For y ∈

 ∈ R m , let Λ(y) ∈ S m be Λ(y) ≡ I m -φ (y)φ (y) φ (y) 2 D(y) I m -φ (y)φ (y) φ (y) 2 , (7.12) in which we denote φ (y) ≡ (φ (y µ )) m µ=1 , and D(y) ∈ R m×m the diagonal matrix with elements D(y) µ = D(y µ ) = nφ (y µ )/m. We aim at proving the following lemma: Lemma 7.8 (Complexity at finite n)

φ

  (y µ )z µ z µ -1 m m µ=1 y µ φ (y µ ) I n-1 , (7.14c) in which y = (y µ ) m µ=1 ∼ N (0, I m ), (z µ ) m µ=1 i.i.d.

µ|.

  S (dx a ) a<b dq ab δ(q ab -x a • x b ) E {y a } 1[{L 1 (y a ) ∈ B} p a=1 ] ϕ {grad L 1 (x a )} p a=1 {y a } det Hess L 1 (x a )| {grad L 1 (x a ) = 0, y a } p a=1

Model 7. 1 (

 1 Binary linear classification) Consider n, m ≥ 1 such that m/n → α > 1. Let σ : R → [0, 1] a smooth threshold function. We are given m samples (y µ , x µ ) m µ=1 with y µ ∈ {0, 1} and x µ ∈ R n . The elements of (y µ ) m µ=1 are generated according to P(Y µ = 1|X µ = x) = σ(θ 0 • x), and x µ i.i.d.

Figure 8 . 2 :

 82 Figure8.2: The rate function I(x) for different values of α and two different distributions ρ, in the real case. Full dots depict the right edge s max of the bulk, while empty dots (when present) correspond to the transition x c (ρ). We draw the dotted line I(x) = 0.5 in the top plot for later use.

Figure 8 . 3 :

 83 Figure 8.3: The function x (t) for ρ(t) = : (i) two peaks (δ 1 + δ -1 )/2, (ii) Wigner's semicircle law, (iii) the Marchenko-Pastur law with ratio 1, (iv) the uniform distribution in [-2, -1]. In all cases α = 2 except for (ii), in which α = 1. Solid lines are analytical predictions. The different Monte-Carlo runs (n = 500) are shown in green with their respective noise. The mean of the green points is depicted as a red dot.

de

  Dz δ(λ max (H n ) -x(θ)) eθn 2 e Hne e nJn(Hn,θ) .(8.10)

Figure A. 1 :

 1 Figure A.1: Different types of diagrams of indices appearing in U 4 0 .

  2.1, and are merely a way to visualize the contractions in Wick's theorem. The first column contains the i α indices and the second contains the j α . Note that we always have i α = j α . The two different types of contractions are represented as Fig. A.1a and Fig. A.1b. They are 12 possible contractions of the type of Fig. A.1a and 48 of Fig. A.1b. We also take into account that in the pairings of Fig. A.1b indices are not all necessarily pairwise distinct. Discarding terms that are O(n), we finally reach:

  with the two auxiliary integrals:I prior ({Q ar br }) ≡ p a=0dw a P 0 (w a ){(a,l);(b,l )}

  ŵ+iuû δ y,sign(u) K l=1 J(λ l (w, ξ), ŵ, û), (D.4) in which we defined J(λ l (w, ξ), ŵ, û) ≡ e -) dropped the dependency of λ l on (w, ξ). Note the following identity: R Dz e αz+iβ sign(z) = e α 2 /2 [cos β + i sin β Ĥ(α)], (D.6) with Ĥ(x) = erf(x/ √ 2). Using eq. (D.6) in eq. (D.5) we obtain:J(λ l , ŵ, û) = e -1 2K ŵ2 cos û √ K -i sin û √ K Ĥ λ l -i ŵ √ K .

e iw ŵ+iΓ 0

 0 DξI C (y, ξ) ln I C (y, ξ), Γ0 +iS 1 Ŝ1 +iS 2 Ŝ2 G(y, w ξ , Γ 0 , S 1 , S 2 )

16 )

 16 Note that the State Evolution (SE) computation of Figure 4.2 was performed by solving the fixed point equations (D.15) and (D.16) (depending on the regime of α).

  + arcsin q d ) + O(K -1 ). (D.21)This expression is the one used in the computation of the generalization error in the left panel of Fig.4.3. matrix M that is independent of W. For any z large enough, we can expand: Tr (zI p -(Γ (0) p ) ) -1 (zI p -Γ (0) p )

k

  [AB] ≥ a 0 Tr [B], as A -a 0 I is positive. We can use this to write, for any λ > λ max : -λI k ) -2 ].

  as the trace of a p × p matrix:S (1,2) k (λ) = 1 k Tr [((Γ (0) p ) -λI p ) -1 (Γ (0) p -λI p ) -1 WW k Tr [((Γ (0) p ) -λI p ) -1 (Γ (0) p -λI p ) -1 ] ≥ ζ 3 min 1 k Tr [(Γ (0) p -λI p ) -2 ],2 Indeed, there exists a positive square root of A, and Tr [AB] = Tr[A 1/2 BA 1/2 ] ≥ 0.

ρ 2 δ qx , δ qx = δq x ρ 2 -δγ x , δ qz = α 2 δq z ρ 2 λ 2 ν- 2 K

 22 δγ z . (D.34)The second equation in (D.33) can be linearized to give:δq z = ρ 2 λ 2 ν α 2 δ qz 1 + R dy K D β z (|z| 2 -1) P out y ρ λ ν α z D β z P out y ρ λ ν α z .(D.35)

2 K

 2 z (|z| 2 -1) P out y ρ λ ν α z D β z P out y ρ λ ν λ 2 ν -λ 2 ν ]

  x ζ y + λ) ν = -1 2 ln y -1 2 ln x -x 2α λ ν y + O(y 2 ).Developing the other terms involved in F (x, y) is straightforward and yields:F (x, y) = -xy 2 λ ν + O(y 2 ). (D.44)One can push this analysis to the next order, and finds in the exact same way, from eq. (D.43): ν -(1 + α) λ 2 ν y 2 + O(y 3 ).

  TAP) + 1 ρ I n x -ρZu, = ρλ TAP Φ √ n x + Φ √ n x -ρZu = u + ρλ TAP (I m + ρZ)u. (D.46)

  n ) s.t. {ν a =µ} pκ α,φ [µ, t φ (µ)] -1 2 ln det[{A ab }]dλ)φ (λ a )φ (λ b ) + Bab ν(dλ)φ (λ a )λ b -αD KL (ν|µ G,q ) . (E.7)

  ,{g(λ)} pκ α,φ [µ, t φ (µ)] -1 2 ln det[{A ab }]ab + Bab B ab (E.9) -p µ(dλ)g(λ) + p(p -1)C Bz p (q) + b{f 0 p (q) + (p -2)f p (q)} -αD KL (ν|µ G,q ) + a,b Âab 2 ν(dλ)φ (λ a )φ (λ b ) + Bab ν(dλ)φ (λ a )λ b + a ν(dλ)g(λ a ) .

2 2=

 2 a )φ (λ b ) + Bab φ (λ a )λ b + a g(λ a ) -αD KL (ν|µ G,q ) -1 ) ab λ a λ b + Âab 2α φ (λ a )φ (λ b )+ Bab α φ (λ a )λ b + a g(λ a ) α . (E.10)To completely decouple the replicas, we will make use of two classical identities, for any x, y:e x Dξe ξx , e xy = DξDξ e Let k, d ∈ N . Let Z : U → R d be a random field, in which U is an open subset of R d . Assume that for every t ∈ U , we can write Z(t) = H[Y (t)], such that:

grad L 1 (

 1 x ξ µ )φ (ξ µ • x). (E.13) We will apply Theorem E.1 with d = n -1 and k = m × n. The Gaussian random field Y (x) ∈ R n×m is defined as Y (x) ≡ P ⊥ x ξ 1 • • • P ⊥ x ξ m ξ 1 • x • • • ξ m • x

d=

  (1/m) m µ=1 φ (y µ )z µ , in which y µ i.i.d.∼ N (0, 1), z µ i.i.d.

t

  ] 1/γ ≥ e -n(δ ln(n)+C) 1 -e n(t-K) , (δ ln(n)+C) [1 -e n(t-K) ] γ (b) = -∞,in which we used the Markov inequality in (a) and eq. (E.16a) in (b). For simplicity we will often abusively denote in the following ln n | det H Λ n (y)| ≡ n-1 i=1 ln n |λ i | and ln n E| det H Λ n (y)| ≡ ln E e n i=1 ln n |λ i | .

  Annealed and quenched calculations for L 2

  n(1+ln α+ln(1-q 2 )) 2 E a,b [δ(P n (a, b))1 L2(a,b)∈B e -nEn(a,b) E z | det H n (a, b)|],

lim n→∞ 1 n

 1 1+η ln P1 n ln E z | det H n (a, b)| -κ α,φ (q, ν m a,b ) ≥ t = -∞. (E.33)

F

  d µ e i z µi f µ , = 1 n ln de df δ(||e|| 2 -n) δ(||f 2 || -m) exp θ √ n m i,µ d µ e i z µi f µ de df δ(||e|| 2 -n) δ(||f 2 || -m) .

inf

  Λe,Λ f ≥0 (αΛeΛ f ≥θ 2 x) Λ e + αΛ f (E.43) -(α -1) ln Λ f + ln g -αΛ e Λ f θ 2 g -α dt ρ(t) ln(α -tg) .

θ≥0[J 2

 2 (θ, x) -F 2 (θ)] (u) -G σ (u)]du. (E.45)

(Generalized) Wishart matrices -Wishart matrices

  . with zero mean and E|J ij | 2 = 1.

	(ii) All {J ii } n i=1 are i.i.d. with zero mean, independent of {J ij } i<j , and E|J ii | 2 = 2.
	Note that if the distribution of the elements is Gaussian, then Wigner matrices are (up to a
	rescaling) the GOE/GUE matrices of Definition 1.3. The LSD of Wigner matrices is given in
	Theorem 1.6 and was first investigated in [Wig55]: it can be considered as the beginning of the
	field of random matrix theory.
	, introduced in [Wis28], are a model
	of sample covariance matrices. More precisely, let x 1 , • • • , x m ∈ R n be i.i.d. vectors drawn from
	a zero-mean distribution P x ∈ M + 1 (R n ). Their sample covariance matrix is

Definition 1.4 (Large deviations of real variables)

  Let {µ n } n≥1 be a sequence of Borel probability measures on R and {a n } n≥1 a sequence of positive reals such that lim n→∞ a n = +∞. The sequence {µ n } n≥1 satisfies a large deviation principle (LDP) with speed {a n } if for any Borel measurable set E ⊆ R 14 :

	Eq. (1.72) exactly describes what we call a large deviation principle, here in the scale n: it
	details the exponential tail of the probability of a random variable. It motivates the following
	general definition, stated here in a precise mathematical language.
	-inf x∈E o I(x) ≤ lim inf n→∞	1 a n	ln µ n (E) ≤ lim sup n→∞	1 a n	x∈E ln µ n (E) ≤ -inf	I(x),	(1.73)
	with a lower semi-continuous I : R → R + ∪ {+∞} called the rate function.
	Note in particular that if the random variables concentrate, as n → ∞, to a limit value m (as
	in eq. (1.70)), then I(m) = 0 is the global minimum of the rate function.
	The physics language -In theoretical physics, LDPs are usually stated in a much simpler,
	informal, way. We consider a sequence P n of PDFs, and we state the LDP of Definition 1.4 as
	1 n	ln P n (x) -I(x), or P n (x) e -nI(x) .	(1.74)
	Eq. (1.72) can then be turned into what is known as Cramér's theorem:
								.71)
	Actually, as first shown by Cramér [Cra38], the bound of eq. (1.71) is tight as n → ∞. The
	perhaps easiest way to prove this is to use a tilting of the measure, a technique that we will use
	again in Chapter 8 in a random matrix context, and the reader can refer to e.g. [Kle13] for a
	detailed proof. In the end, we reach:
	lim n→∞	1 n	t∈R ln P[S n ≥ x] = -sup	tx -ln E e tX .	(1.72)

Theorem 1.8 (Cramér's theorem, [Cra38])

	Let (X i ) n i=1 be a sequence of i.i.d. real random variables such that Λ(t) ≡ ln E[exp(tX)] is finite for all t ∈ R. Let Λ * (u) ≡ sup t∈R [tu -Λ(t)] its Legendre transform. Then the law of the random variables S n ≡ (1/n) n i=1 X i satisfies a large deviation principle, in the scale n, with rate function Λ

* .

  and e 1 , • • • , e k be the first k columns of a Haar-distributed matrix O ∈ U β (n). Then

  USV † , outputs {y µ } m µ=1 , a number of iterations T . Result: An estimate x of X * . Randomly initialize all variables;

	Algorithm 6: Generalized Vector Approximate Message Passing Data: Sensing matrix Φ/ √ n =
	).	(6.9)

  The vertical grey dashed lines denote the algorithmic weak recovery threshold α WR,Algo . Note the presence of a statistical-to-algorithmic gap in both ensembles, and that for column-orthogonal matrices α WR,Algo > α FR,IT .

				Complex Gaussian matrices					Column-unitary matrices
		1.0
		0.8
	Mean Squared Error	0.0 0.2 0.4 0.6	1.9 Informed State Evolution 2.0 State Evolution G-VAMP	2.1					Informed State Evolution State Evolution G-VAMP, uniform unitary matrices G-VAMP, partial DFT matrices G-VAMP, partial DFT matrices with no mask
		0.8	1.0	1.2	1.4	1.6	1.8	2.0	2.2	2.0	2.1	2.2	2.3	2.4

  .3 for four different choices of ρ(t). The agreement with Result 8.1 is excellent,

	4	(i)			(iii)			10
	3							8
								6
	2
		0.2	0.4	0.6	0.05	0.10	0.15	0.20 4
	6	(ii)			(iv)
								-0.05
	4							-0.10
		0.050 0.075 0.100 0.125 0.150	0	5	10	15
			t			t
		x (t)	s max		x c (ρ)	MC (mean)	MC

  Very interestingly the tilting method used in this chapter does not fundamentally rely on the Gaussianity of the variables z µi , and several works have used this technique to investigate universality properties of the large deviations of the extreme eigenvalues, e.g. for Wigner matrices [GH20, Hus20, AGH21]. This hints towards a possible universality of the large deviations described in Result 8.1 as long as the variables z µi are independent, centered, have unit variance, and satisfy some properties, e.g. sub-Gaussianity or if they are Rademacher/Ising variables. Proving or refuting this intuition in detail would be an interesting follow-up to this chapter. An interesting result, which should be accessible, is to generalize the large deviations of Result 8.1 to the k-th eigenvalue (not necessarily the smallest), with fixed k as n → ∞. In terms of the spherical integrals, which are at the heart of the tilting techniques (cf. Section 8.4), this requires results on the asymptotics of rank-k HCIZ spherical integrals, which are well-known, see Section 1.5.3. This venue has recently been investigated for Wigner and Wishart matrices[START_REF] Guionnet | Asymptotics of k-dimensional spherical integrals and applications[END_REF], and generalizing to the present context should follow on the same lines. In the Kac-Rice formalism of Chapter 7, this would then allow to count the critical points of any finite index (i.e. saddle points with a finite number of descending directions). As a final note, the generalizations mentioned in the conclusion to Chapter 7 would naturally create new random matrix challenges, e.g. understanding the large deviations of the extremal eigenvalues of the Hessian of the loss for a neural network with multiple layers and nodes, a random matrix problem that, to the best of our knowledge, is still open at the moment.

	Other possible generalizations -

few technical lemmas Lemma C.4 (Cauchy-Lipschitz Theorem and Liouville Formula)

  The t-derivative of f n, (t) thus reads, for t ∈ (0, 1):We now compute T 1 . We start from eq. (C.4) and we consider for the moment the first term (recall also the expression (4.11) for S t,µ ):Yt,µ (S t,µ ) ln Z n, (t) = E Tr -R 2 (t) + 2s n I K U * µ ∇ u Yt,µ (S t,µ ) ln Z n, (t) . (C.6)We then compute the first term of the right-hand side of eq. (C.6). By Gaussian integration by parts w.r.t. X µi (recall hypothesis H.3), and using eq. (C.1), we find after some algebra For the remaining terms of the right hand side of (C.6), we use again Gaussian integrations by parts but this time w.r.t. V µ , U * µ which have i.i.d. N (0, 1) entries. This calculation has to be done carefully using the cyclicity and linearity of the trace, and with the help of the identity (t) ∈ S + K . Applying eq. (C.8) to t 0 q(s)ds and t 0 (ρ -q(s))ds, as well as the identity of eq. (C.1), we reach after some algebraR 2 (t) + 2s n I K U * µ ∇ u Yµ (S µ,t ) ln Z n, (t) = E Tr ρ ∇∇ P out (Y t,µ |S µ,t ) P out (Y t,µ |S µ,t ) ln Z n, (t) + E Tr q(t)∇u Yt,µ (S µ,t )∇ u Yt,µ (s µ,t )This completes the computation of the terms of eq. (C.6). It now remains to compute the second term of the right hand side of eq. (C.4). Recall that Y t,i -R 1 (t)W * i = Z i ∼ N (0, I K ). Using Gaussian integration by parts as well as eq. (C.8) one obtains Finally the term T 1 is obtained by putting together eqs. (C.6), (C.7), (C.9) and (C.10). This ends the derivation of eq. (C.3). It now remains to check that A n → 0 as n → +∞ uniformly in t ∈ [0, 1]. The proof from [BKM+ 19] can easily be adapted so we give here just a few indications for the ease of the reader. First one notices thatE ∇∇ P out (Y t,µ |S t,µ ) P out (Y µ |S t,µ ) W * , {S t,µ } m µ=1 = dY µ ∇∇ P out (Y t,µ |S t,µ ) = 0,so that by the tower property of the conditional expectation one gets Next, one shows by standard second moment methods that E[(ln Z n, (t)/n -f n, (t)) 2 ] → 0 as n → +∞ uniformly in t ∈ [0, 1] (see [BKM + 19] for the proof at K = 1, that generalizes straightforwardly for any finite K). Then, using this last fact together with (C.11), and under hypotheses H.1, H.2, H.3, an easy application of the Cauchy-Schwarz inequality implies A n → 0 as n → +∞ uniformly in t ∈ [0, 1]. This ends the proof of Proposition 4.3. ) is a diffeomorphism of class C 1 , from (0, +∞) d to z t (0, +∞) d . Moreover the determinant J(z t )( ) of the Jacobian of z t at verifies

	df n, (t) dt	=								(C.5)
	-	1 n	E	dH t dt	(W T 1	-	1 n	E	dH t dt	T 2 (w, u; Y t , Y t , X, V)	n,t,	.
	First, we note that T 2 = 0 by the Nishimori identity, Proposition 1.1:
	T 2 = (W E Tr 1 n E dH t dt (w, u; Y t , Y t , X, V) n,t, = 1 n E dH t dt dS t,µ dt ∇ u n i=1 X µi W * i 2 n(1 -t) + d dt R 2 (t)V µ + d dt tρ --1 2 n(1 -t) E Tr = -1 2 E Tr 1 n n i=1 W * i W -1 2 E Tr 1 n n i=1 W n,t, d dt M (t) = M (t) d M (t) dt + d M (t) dt M (t) for any M E Tr d dt R 2 (t)V µ + d dt tρ -n,t, . E Tr d R 1 (t) dt 1 √ n m µ=1 ∇∇ P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) n 1 √ n i=1 (W * i (W * i ) -ρ) = 0. Let F : [0, 1] × (0, +∞) J(z t )( ) = det ∂y i ∂ j i,j = exp t 0 d i=1 ∂F i ∂z i s, y(s, ) ds . (Y E Tr Thus, in particular, if in addition d i=1 (∂F	(C.7) (C.8) . (C.9) (C.11) (C.13)

* , U * ; Y t , Y t , X, V) ln Z n, (t) * , U * ; Y t , Y t , X, V) = 0 . n i=1 X µi W * i ∇ u Yt,µ (S t,µ ) ln Z n, (t) i ∇∇ P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) ln Z n, (t) * i w i ∇u Yt,µ (S t,µ )∇ u Yt,µ (s t,µ ) t,i -R 1 (t)W * i )W * i ln Z n, (t) = Tr R 1 (t) E W * i w i n,t, -ρ . (C.10) C.5 A d → [0, +∞) d (t, z) → F (t, z)

be a continuous bounded function. Assume that F admits continuous partial derivatives ∂F/∂z i (i = 1, . . . , d) on its domain of definition. Then, for all ∈ (0, +∞) d , the Cauchy problem y(0) = and y (t) = F t, y(t) (C.12) admits a unique solution t → y(t, ). For all t ∈ [0, 1], the mapping z t : → y(t, i /∂z i ) ≥ 0 then J(z t )( ) ≥ 1 for all .

  We note that the constant remains bounded as ∆ → 0, and diverges as K → +∞. It is easy to see that for symmetric matrices A, B we haveTr(A-B) 2 ≤ 2(TrA 2 + TrB 2 ). Therefore Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) -R n Yt,µ (s t,µ )∇u Yt,µ (S t,µ )In the rest of the argument we bound the second term of the right hand side of this last inequality. Using the triangle inequality and then Cauchy-Schwarz we obtain Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) F Yt,µ (s t,µ ) 2 ∇u Yt,µ (S t,µ ) 2 where ∇ϕ is the K-dimensional gradient w.r.t. the first argument x ∈ R K . From the observation model we get |Y t,µ | ≤ sup |ϕ| + √ ∆|Z µ |, where the supremum is taken over both arguments of ϕ, and thus we immediately obtain for all s ∈ R K ∇u Yt,µ (x) ≤ (2 sup |ϕ| + √ ∆|Z µ |) sup ∇ϕ . (C.16) From eqs. (C.15) and (C.16) we see that it suffices to check that m 2 n 2 E (2 sup |ϕ| + |Z µ |) 2 (sup ∇ϕ ) 2 2 ≤ C(ϕ, K, ∆),

									+ K we have
		E	1 n	m µ=1	∇u Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) -R n	2 F n,t,	≤ 2Tr(R 2 n ) + α 2 C(ϕ, K, ∆).	(C.14)
	Proof of Lemma C.5 -E 1 m ∇u 2 n µ=1 F n,t,
						≤ 2Tr(R 2 n ) + 2E Tr	1 n	µ=1 m	∇u 2	n,t,	.
	E	1 n					2 F n,t,	≤ E	1 n 2	m µ=1	∇u 2	n,t,
			≤ E	1 n 2	µ=1 m	∇u 2	n,t,	.	(C.15)
	Recall the random representation of the transition kernel:
									µ )	1 2π∆ √	e -1 2∆ (Yt,µ-ϕ(x,aµ)) 2 ,
	and thus
				∇u Yt,µ (x) =	dP

m µ=1 ∇u Yt,µ (s t,µ )∇u Yt,µ (S t,µ ) u Yt,µ (s) = ln P out (Y t,µ |x) = ln dP A (a A (a µ )(Y t,µ -ϕ(x, a µ ))∇ϕ(x, a µ )e -1 2∆ (Yt,µ-ϕ(x,aµ)) 2 dP A (a µ )e -1 2∆ (Yt,µ-ϕ(x,aµ)) 2 ,

  We know ln n |x| ≥ -δ ln(n). By Lemma E.5, for all γ > 0:e γ n-1 i=1 ln n |λ i | < +∞.Fixing γ > 1 and using Hölder's inequality, there exists therefore C > 0 such that for all K > 0 and t ∈ (0, K):

												(n) t ):
	1 n	ln E z e	n-1 i=1 ln |λ i | ≥	1 n	ln E z e	n-1 i=1 ln |λ i | 1[	Ā(n) t ] ≥ -t +	1 n	ln E z e	n-1 i=1 ln n |λ i | 1[	Ā(n)
	So that (using ln n (x) ≥ ln(x) for all x > 0):
	0 ≤	1 n	ln E e	n-1 i=1 ln n |λ i | -	1 n	ln E| det H Λ n (y)| ≤ t -	1 n	ln 1 -	E z e E z e n-1 i=1 ln n |λ i | 1[A n-1 i=1 ln n |λ i |	(n) t ]	.
					lim sup n→∞ ln E lim sup sup y∈R m 1 n n→∞ 1 n 1+η ln P y 1 n ln E z | det H Λ n (y)| -	n 1	ln E z e	n-1

t being the complementary event to

A t ] . z i=1 ln n |λ i | ≥ K ≤ lim sup n→∞ 1 n 1+η ln P y P z [A

  We can compute the determinant of the block matrix easily, and we arrive at:1 n ln de df δ(||e|| 2 -n) δ(||f 2 || -m) exp θ √ n m i,µ d µ e i z µi f µNote that the matrix inside the log-det must be positive, which constrains Λ e Λ f ≥ θ 2 x/α, as we assumed λ max (H n ) x. All in all, we have, taking n → ∞:1 n ln de df δ(||e|| 2 -n) δ(||f 2 || -m)exp θ Combining eqs. (E.40) and (E.41) yields the general formula for J 2 : Λe,Λ f ≥0 s.t. αΛeΛ f ≥θ 2 x

		inf Λe,Λ f ≥0	Λ e 2	+	αΛ f 2	-	α -1 2	ln Λ f -	1 2n	ln det Λ e Λ f I n -	θ 2 α	H n +	(1 + α) 2	ln 2π .
														√ m i,µ n	d µ e i z µi f µ	(E.41)
	inf Λe,Λ f ≥0			Λ e 2	+	αΛ f 2	-	α -1 2	ln Λ f -	1 2	dλ σ(λ) ln Λ e Λ f -	θ 2 α	λ +	(1 + α) 2	ln 2π .
	s.t. αΛeΛ f ≥θ 2 x
	J 2 (θ, x)												(E.42)
	=	1 2	inf
				n	ln de df δ(||e|| 2 -n) δ(||f 2 || -m)	(1 + α) 2	(1 + ln 2π).	(E.40)
	We use the same method to compute the numerator:
	1 n	ln de df δ(||e|| Λe,Λ f ≥0 Λ e 2 + αΛ f 2	-	1 2n	ln det	θ √ α	Λ e I n √ D m	z √ m	θ √ α	z √ m Λ f I m √ D m	+	(1 + α) 2	ln 2π .

2 -n) δ(||f 2 || -m) exp θ √ n m i,µ d µ e i z µi f µ inf

We will always clarify possible ambiguities between these concepts.

The answer to this question is much harder, but the original algorithm of Euclid is not optimal in terms of computational time.

Shannon named it entropy after receiving a famous advice from von Neumann: "You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage."

(a hopefully unrealistic situation)

Of course sampling from this high-dimensional distribution might still be very hard even when knowing its density.

In this thesis we will mainly use this convention, while many physics works rather consider the negative of the free entropy, usually called free energy.

This nomenclature comes from the name of metallurgic treatments.

It is called overlap as it corresponds to the scalar product of the magnetization vectors of two independent "replicas" of the system which share the same quenched disorder.

Eq. (1.38) is sometimes called the sum-product update rule, in contrast with the (equivalent) max-sum formalism sometimes used, which corresponds in essence to a zero-temperature limit of the BP equations, see[START_REF] Mézard | Information, physics, and computation[END_REF].

This algorithm is sometimes called Bayes-AMP to distinguish it from a broader class of methods also called AMP algorithms. Bayes-AMP has been shown to be optimal in this category of algorithms[START_REF] Celentano | The estimation error of general first order methods[END_REF].

Note that an iteration scheme very close to the AMP algorithm, and derived as well from the BP equations, was first proposed in[START_REF] Kabashima | A BP-based algorithm for performing Bayesian inference in large perceptron-type networks[END_REF].

However cavity computations are often easier to prove mathematically than their replica counterpart, as they rely on less heuristic methods.

At different points in this thesis we will use the different notations gν (z) = Sν (z) and Gν (z) = -Sν (z).

Recall that E o and E are the interior and adherence of E.

Recall that ν µ means absolute continuity of ν with respect to µ, i.e. for any measurable set A we have µ(A) = 0 ⇒ ν(A) = 0. By the Radon-Nikodym theorem[START_REF] Nikodym | Sur une généralisation des intégrales de MJ Radon[END_REF], we know that ν µ is equivalent to the existence of the Radon-Nikodym derivative dν/dµ such that ν(A) = A (dν/dµ)dµ.

This constant comes from the normalization factor Zn and ensures that the probability of the limit spectral density approaches 1 as n → ∞.

Note that such an explicit formula does not exist in the real orthogonal case.

We used physicists' convention, introducing a delta constraint on x

, while mathematicians usually prefer employing the uniform measure on the sphere S n-1 ( √ σn), cf. e.g. Theorem 1.12. These conventions are equivalent and only differ by an irrelevant global factor in the partition function.

For pedagogical purposes we detail this calculation in Appendix A.1.

At order 4, this was shown explicitly in Appendix A.1.

By "expectation" we mean here expectation over O in J = ODO , by rotation invariance.

There might be a confusion, so we emphasize that this "decomposition" of E G is a decomposition of the graph representing E G.

In this chapter, we will generically use greek indices µ, ν for indices between 1 and m, while latin i, j, k indices will run from 1 to n, and the l index between 1 and p.

We normalize the free entropy by the total number n(m + p) of variables to infer, while the normalization of [KKM + 16] is n 2 .

Note that one can include the diagonal terms µ = ν in these equations without changing the large n limit of the free entropy.

Recall that we discussed the BP equations on a general factor graph, and their simplifications into approximate message-passing algorithms and their relation with the replica method, in Section 1.4.

Note that we used the same representation for generalized linear models in Section 2.3.

The Bayes-optimal error differs from the Gibbs generalization error by a factor

2, as detailed in Appendix D.1.

We adopt the convention sign(0) = 0.

For applications of this method to simpler models, one can refer to[START_REF] Barbier | The adaptive interpolation method for proving replica formulas. applications to the Curie-Weiss and Wigner spike models[END_REF] for e.g. the spiked Wigner model, or to [BKM+ 19] for GLMs.

or directly obtain by the I-MMSE formula for vector channels[START_REF] Reeves | Mutual information as a function of matrix SNR for linear gaussian channels[END_REF]

Indeed it might be that there are spurious local minima, yet the AMP approach succeeds.

Understanding the influence of data structure in learning through synthetic generative models has recently received a burst of attention from some closes collaborators, see e.g. [GMKZ20, ALB +

20, GLK + 20, GRM+ 20].

Analogous conclusions about the lack of algorithmic gaps for the problem of phase retrieval under a generative prior can be found in [HLV18, ALB+ 20].

This result holds for sparsity ρ = Θ(1). A line of work shows that when the sparsity k scales as O(p), there exists algorithm that can outperform PCA in terms of weak recovery[START_REF] Arash | High-dimensional analysis of semidefinite relaxations for sparse principal components[END_REF][START_REF] Deshpande | Sparse PCA via covariance thresholding[END_REF].

This is an important and generic property of AMP algorithms and their state evolution, that is also discussed in Chapters 4 and 6.

This is necessary as, by symmetry of the problem, the point qv = qz = 0 is always a fixed point of the SE.

Note that while all the calculations are justified, refinements would be needed in order to be completely rigorous. These refinements would follow exactly some proofs of[START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF] and[START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF], so we will refer precisely to them when necessary.

All these works (and this chapter) consider random instances of phase retrieval, while there is another important literature that studies injectivity of phase retrieval. For instance, the perfect recovery of an arbitrary (rather than typical) signal in complex phase retrieval has been studied in[START_REF] Afonso S Bandeira | Saving phase: Injectivity and stability for phase retrieval[END_REF], and the thresholds are very different.

To test the performance of the G-VAMP algorithm, we used the TrAMP library[START_REF] Baker | TRAMP: Compositional inference with TRee Approximate Message Passing[END_REF] that provides an open-source implementation.

It is directly related to the mutual information between X * and Y, see Chapter 5 in which we adopted the mutual-information convention

With respect to Model R we assume the following, which is (slightly) stronger: the large deviations of the spectral density of Φ † Φ/n should happen in the scale n 1+ for an > 0. This condition was not precised in the replica calculation of[START_REF] Takahashi | Macroscopic analysis of vector approximate message passing in a model mismatch setting[END_REF] for real matrices. In practice, in classical orthogonally (unitarily)-invariant random matrix ensembles, we often have = 1.

The rigorous statement on the limit of the MMSE requires adding a side information channel with arbitrarily small signal, as is detailed in[MLKZ20].

In Theorem 6.2, we rely on some Gaussianity, either in the prior or in the data matrix. This is a not specific to our setting, but rather a fundamental limitation of the adaptive interpolation method used for the proof.

We can take ρ = 1, as the scaling is irrelevant under a noiseless channel.

In the complex case, ∂ωgout is the "Wirtinger" derivative ∂zf (z) ≡ (∂xf (z) -i∂yf (z))/2.

Note that Conjecture 6.5 is compatible with the results of eq. (6.16). Indeed, for Gaussian i.i.d. matrices, one has λ ν = α, while for random column-unitary matrices, λ ν = 1.

This scaling is chosen to match the one of Gaussian i.i.d. matrices.

The universality of linearized approximate message passing algorithms for a Gaussian prior and different ensembles of column-orthogonal matrices was analyzed recently in[START_REF] Dudeja | Universality of linearized message passing for phase retrieval with structured sensing matrices[END_REF].

In particular, this is an important distinction between our L-AMP constructive derivation and the L-AMP algorithms of [MDX + 21], which are designed to match the spectral methods of the type M(T ): in the latter, it was shown that the L-AMP estimator always matched the one of the spectral method.

For instance, picture the unit sphere S n-1 with a "North pole" en. As n → ∞ almost all its mass is concentrated on the equator orthogonal to en, which is not intuitive when representing it in 3 dimensions!

i.e. points t for which f (t) is singular.

The numbers of critical points of different indices (i.e. the number of strictly negative Hessian eigenvalues) of a Morse function are constrained by the topology of M by the Morse inequalities[START_REF] Willard Milnor | Morse theory[END_REF].

The proof that fn,p is almost surely a Morse function can be found in[START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF]

We used a replica symmetric (RS) structure, which is correct in many cases, and a very good approximation in others were replica symmetry has to be broken. See Section 1.3.1 for more discussion on the RS ansatz.

In the proofs of this section we assume that xφ (x) and φ (x) are bounded. As one can always smoothly truncate the largest values of φ without affecting the complexity, this does not remove any generality to our results.

g ∈ C+, and (since α > 1) α + φ (λ)g ∈ C\(-∞, 1], thus we can use the principal determination of the logarithm.

Note that a similar argument based on the vanishing exponent of the density was already used in the literature, in the context of multi-critical matrix models[START_REF] Satya | Top eigenvalue of a random matrix: large deviations and third order phase transition[END_REF].

The analysis of[START_REF] Biroli | Large deviations for the largest eigenvalues and eigenvectors of spiked Gaussian random matrices[END_REF][START_REF] Belinschi | Large deviation principles via spherical integrals[END_REF] suggests a tilting which is function of An, with Hn = AnA n . However for arbitrary dµ, An is not defined so that we use this simpler tilting. We shall later come back to this idea by adapting the method to allow for complex-valued An.

Recall that Dz ≡ dz e -||z|| 2 /2 /(2π) n/2 is the standard Gaussian law.

This illustrates the arguably greatest peril of our interdisciplinary approach: to perform careless transposition of results from one scientific field to the other.

This relation is valid even if λx would "saturate" to a constant value that does not depend on (qx, qz).

Note that a similar limit has been derived in the context of coding with sparse superposition codes[START_REF] Barbier | Approximate message-passing decoder and capacity achieving sparse superposition codes[END_REF].

√ q d +qa-√ q d √ 1-q d

Note that this result is uniform over y, and thus stronger than what is needed to show Proposition E.11.

Acknowledgements

Part III

Towards a topological approach to high-dimensional optimization Finally, combining eqs. (A.2),(A.3),(A.4) to plug them into eq. (A.1), we reach:

which is what we wanted to show !

A.2 Generalizations of the diagrammatics

We detail here some extensions of the results of Section 2.4. First, in Sec. A.2.1, we explain how to transpose these results to Hermitian matrix models, and in Sec. A.2.2 we show how to extend some of them to diagrams of diverging size (as n → ∞).

A.2.1 Hermitian matrix model

We can generalize the results of Sec. 2.4 to the Hermitian matrix model described by Model S, i.e. when β = 2. Note that the diagrams for Hermitian matrices are directed, as J ij = J ji .

We describe examples of such diagrams in

In the complex case, an Eulerian graph is defined as a graph in which one can construct a cyclic path (following the directions of the edges) that visits each edge exactly once. Note that a simple cycle is defined such that the arrows on its edges themselves form a cycle, like the constituent cycles of Fig. A.2c. We describe the main results we get, using the same kind of techniques as used in Sec. 2.4:

(i) Only Eulerian diagrams contribute in the n → ∞ limit.

Recall the form of the operator U at η = 0, i.e. eq. (A.8):

To compute U 3 0 , we decompose U = A + B H + B F + B X as in the equation above. Recall that all the variables in the equation above are centered, so that we get easily:

Using again the centering of the variables and the decomposition above, we get that the only non-zero terms of the type X 2 Y 0 with X, Y ∈ {A, B H , B F , B X } yield the contribution:

Finally, the last contribution to U 3 0 comes from the term:

Summing the contributions from eqs. (A.11),(A.12),(A.13) yields U 3 0 , which gives:

Since all involved terms inside the sum are of order O n (1), and given the global scaling factor, it is clear that the third order is subdominant:

1). (A.14)

On higher order moments -An important remark that one can already conjecture by generalizing from these arguments is that at any given order of perturbation in η, only the first two moments of the fields H, F, X will appear at dominant order. This conjecture arises as a consequence of a scaling argument: in the terms remaining in the Plefka expansion, the higher-order moments constraint too many indices, and thus the terms involving them can be neglected. Note that we showed this property for a large class of inference models in Chapter 2.

A.3.3 Order 4 in η

This section describes the calculation of the order 4 perturbation of the free entropy. It is particularly tedious and lengthy, but the techniques involved are not conceptually complicated. We start with eq. (2.7d). For simplicity, we will not consider terms involving cumulants of order .

The calculation at order 2 already gave:

We finally have to compute U 4 0 , whose calculation is very tedious, but not conceptually difficult. We again make use of the decomposition of eq. (A.10). A first simplification arises since the variables are centered and have negligible moments of odd order. This implies:

We now compute the three terms I 1 , I 2 , I 3 independently. An explicit calculation gives

Importantly, here the indices are not supposed to be pairwise distinct unless explicitly stated so. The terms I 1 , I 2 can be explicitly computed as well, and are very lengthy:

.

Many simplifications occur in the terms of eqs. (A.16) to (A.22). Two type of terms are for instance negligible:

• Terms of the type n -4 µil A µil , with A µil typically of order 1. These terms are negligible by a simple scaling argument. is readily done exactly as in Section A.3.1, and we reach:

in which the lowercase variables again designate the centered variables. From the expression of U we can obtain the following orders of perturbation:

Here we symmetrized g νµ ≡ g µν , and we adopted the convention g µµ = 0.

Appendix B

Details of replica computations B.1 Replica calculation for the committee machine

Our goal here is to provide a heuristic derivation of the formula of Theorem 4.1 using the replica method, a powerful non-rigorous tool from statistical physics of disordered systems. To the best of our knowledge, this computation was first performed for a committee machine in [START_REF] Schwarze | Learning a rule in a multilayer neural network[END_REF]. Recall that we introduced the method in Section 1.3.1, and the reader can come back to it to freshen her/his memory if necessary. As we know, in order to compute the asymptotic free entropy with the replica method we need the moments of the partition function (given by eq. (4.5)), for integer p:

The outer expectation is done over X µi ∼ N (0, 1), W and Y. Writing W as w 0 we have:

To perform the average over X we notice that the variables {Z a µl }, defined by

follow a multivariate Gaussian distribution with zero mean and covariance tensor:

For every a, b, Q a b ∈ S K is the overlap matrix between replicas a and b. Introducing δ functions to fix Q we arrive at :

B.2.2 Computing the p-th moment of the partition function

Thanks to Bayes-optimality, we can easily write the average of Z n (Y) p as an average over p + 1 replicas of the system, by considering X as the replica of index 0. We obtain for any p ≥ 1:

The first step is to decompose eq. (B.7) into three terms, corresponding to the prior P 0 , the channel P out , and the "delta" term. Note that the matrix Φ only appears in the last "delta" term. By left and right invariance of Φ, the quantity

is completely determined by the overlaps

, which are positive Hermitian matrices. As is standard in such replica calculations, we will constraint the terms in eq. (B.7) by the value of these overlaps, before performing a Laplace method on the resulting function of the overlaps. By A n B n , we will mean (ln A n )/n = (ln B n )/n + O n (1). We introduce in eq. (B.7) the term:

We can use a Fourier transformation of the delta terms, which allows in the end to transform eq. (B.7) into the product of three independent terms. Performing then Laplace's method on Q x , Q z we obtain:

in which the supremum is made over positive symmetric (Hermitian) matrices, and I 0 , I out and I int are functions whose calculation will be detailed below.

The prior term

We have after Fourier transformation of the delta terms:

The infimum is again over positive symmetric (Hermitian) matrices. We also made use of the fact that the prior P 0 is i.i.d. over the elements of x.

A very important assumption that one can then use is replica symmetry: it amounts to assume that all the (p + 1) replicas are equivalent, and that this symmetry is not broken by the system at the solution of the variational principle on Q, Q. It has been shown that for an inference problem in the Bayes-optimal setting (as is the present case), replica symmetry is never broken [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF]. For more details on replica symmetry, the reader can refer to Section 1.3.1. We therefore

Appendix C

Proving the replica formula: details in the committee machine

C.1 Positivity of some matrices

We first prove the following lemma which provides basic properties of the overlap matrix:

Lemma C.1 (Positivity of some matrices)

The matrices ρ, E Q and ρ -E Q are all positive semi-definite, i.e. in S + K .

Proof of Lemma C.1 -The statement for ρ follows from its very definition. Note for further use that we have

, by the Nishimori identity (Proposition 1.1), we have

which is obviously in S + K . Finally we note that

where the last equality follows again from the Nishimori identity. This last expression is obviously in

C.2 Properties of the auxiliary channels

Lemma C.2 (Properties of ψ P 0 )

Recall that ψ P 0 is the free entropy of the auxiliary channel of eq. (4.3). More precisely we have, for any r ∈ S + K :

Then ψ P 0 is convex and differentiable on S + K , and moreover ∇ψ P 0 (r) ∈ S + K for any r ∈ S + K .

Proof of Lemma C.2 -One can easily compute (either directly, or by the I-MMSE theorem for vector channels [START_REF] Reeves | Mutual information as a function of matrix SNR for linear gaussian channels[END_REF]) ∇ψ P 0 (r) = (ρ -E[ w w ])/2. Using the Nishimori Proposition 1.1, we can write it as ∇ψ P 0 (r) = E[(w -w )(w -w ) ]/2, which is clearly a positive matrix. By a very similar computation (see for instance Lemma 4 of [START_REF] Reeves | Mutual information as a function of matrix SNR for linear gaussian channels[END_REF]), one can check that ∇ 2 ψ P 0 is a positive operator on S + K × S + K , so that ψ P 0 is convex, which ends the proof.

Lemma C.3 (Properties of Ψ out )

Recall that Ψ out is the free entropy of the auxiliary channel of eq. (4.4). More precisely, for q ∈ S + K (ρ), we have:

Then Ψ out is continuous and convex on S + K (ρ), and twice differentiable inside S + K (ρ). Moreover, for all q ∈ S + K (ρ), one has ∇Ψ out (q) ∈ S + K .

Proof of Lemma C.3 -The continuity and differentiability of Ψ out is easy, and the reader can simply follow the proof of Proposition 18 of [BKM + 19]. It follows from H.2 which allows to use continuity and differentiation under the expectation, as all usual domination hypotheses are then easily verified.

One can compute the gradient and Hessian matrix of Ψ out (q), for q inside S + K (ρ), using Gaussian integration by parts and the Nishimori identity 1.1. The calculation is tedious and essentially follows the steps of Proposition 11 of [BKM + 19]. Recall that u Y 0 (x) ≡ ln P out ( Y 0 |x). We define the averagesc (where sc stands for "scalar channel") as

Using this definition, one arrives at:

Note that this gradient is actually a symmetric matrix of size K, as q is itself a matrix of size K. The Hessian ∇∇ Ψ out with respect to q is thus a 4-tensor that can be computed similarly:

In this expression, ⊗2 means the "tensorized square" of a matrix, i.e. for any matrix M of size

From this expression, it is clear that the Hessian of Ψ out is always positive, when seen as a matrix with rows and columns in S K , and thus Ψ out is convex, which ends the proof.

C.3 Setting in the Hamiltonian language

In this section, we set up some notations which will be useful in the following Section C.4. Let u y (x) ≡ ln P out (y|x). Here x ∈ R K and y ∈ R. We will denote by ∇u y (x) the Kdimensional gradient w.r.t. x, and ∇∇ u y (x) the K × K Hessian w.r.t. x. Moreover ∇P out (y|x) and ∇∇ P out (y|x) also denote the K-dimensional gradient and Hessian w.r.t. x. We will also use the matrix identity

Appendix D

Technical results of Part II D.1 Generalization error in the committee machine D.1.1 Bayes-optimal and Gibbs generalization error

We detail here two different possible definitions of the generalization error, and how they are related. Recall that we wish to estimate W * from the observation of ϕ out (XW * ). In the following, we denote E for the average over the (quenched) teacher weights W * and the data X, andfor the Gibbs average over the posterior distribution of W. we can define two notions of generalization error, namely the Gibbs generalization error and the Bayes-optimal generalization error (which is the one considered in eq. (4.7)) as:

Using the Nishimori identity 1.1, one can show that:

Using again the Nishimori identity one can write:

, and relates the two definitions of the generalization error.

D.1.2 The generalization error at K = 2 with committee symmetry

In this subsection alone, we consider the K = 2 case. From the definition of the generalization error that we saw in Section D.1.1, one can obtain an explicit expression in the K = 2 case. Recall that we assume committee symmetry, which here reads

in which we defined the following quantities, that only depend on ξ (recall eq. (D.7))

A detailed calculation actually shows that the previous expansion of eq. (D.8) is valid up to O(K -1 ), and not only O(K -1/2 ). Recall also eq. (D.4), in which one can now readily perform the integration over all variables w, ŵ, u, û to obtain (dropping the ξ dependency in w ξ , Γ 0 , S 1 , S 2 ):

2)]/2. Note that all quantities w ξ , Γ 0 , S 1 , S 2 only depend on ξ via its empirical measure: this remark is what will make the integration over ξ ∈ R K tractable. We compute it in the following, using theoretical physics methods. We denote the quantity that appears in eq. (D.9) as a function of w ξ , Γ 0 , S 1 , S 2 :

D.3 RMT analysis of the spiked matrix model D.3.1 Proof of Lemma 5.6

Point (i) is trivial by definition of S (r) k (λ) and the result of Lemma 5.5. We turn to points (ii) and (iii). Let us denote the following function:

By Lemma 5.5, we have

The condition s > s edge arises naturally as the counterpart of z ≥ λ max . Recall that by Theorem 5.2, we have λ max ≤ 1 for all ∆. As g -1 ν (s) is here completely explicit by eq. (5.26), and recalling the form of ρ ∆ in eq. (5.23), it is easy to show by an explicit computation that:

It is then easy to see that the only possible solution to T (s) = -α∆ with s ∈ (s edge , 0) is s = g ν (1), if g ν (1) = -1. However, by Lemma 5.4, for any ∆ > ∆ c (α) we have s edge < -1. Moreover, in this case, one computes very easily (all expressions are explicit) g -1 ν (-1) = 1. Given the identity above, there is therefore no solution to T (2) (s) = -α∆ in (s edge , 0). By continuity of T (2) (s), and since lim s→0 T (2) (s) = 0, this implies T (2) (s) > -α∆ for s ∈ (s edge , 0), which proves point (iii).

Assume now ∆ ≤ ∆ c (α). Note that the case ∆ = ∆ c (α) is easy, as s edge = -1 is the unique solution to T (2) (s) = -α(1 + α). For ∆ < ∆ c (α), by Lemma 5.4 we obtain -1 < s edge . In particular, g ν (1) > s edge > -1, and we thus have that s = g ν (1) is a solution (and the only one) to T (2) (s) = -α∆ by the identity shown above. This shows (ii) and ends the proof of Lemma 5.6.

D.3.2 Proof of correlation of the leading eigenvector

We now turn to the study of the leading eigenvector in the claim of Theorem 5.3. Let ṽ be an eigenvector associated with the largest eigenvalue λ 1 , normalized such that ṽ 2 = p. We have:

By normalization of ṽ, we obtain:

Therefore:

k and the concentration of z Mz/k on Tr M/k, we reach as p, k → ∞:

The numerator is equal to [α -1 S

(2)

k is defined in Lemma 5.5). Let us compute the denominator. Recall that we can write Γ (0) p = WW M/k, with a symmetric However the LAMP spectral method computes the normalized leading eigenvector of the structured matrix Γ vv p . Hence the norm of the LAMP estimator is v 2 LAMP = q v,LAMP = 1, while the Bayes-optimal AMP estimator is not normalized, and satisfies v 2 AMP = q v,AMP = m v,AMP = 1, with q vAMP solution of eq. (5.18). As the non-normalized LAMP estimator follows the AMP state evolution in the linear case, the overlap with the ground truth is thus given by:

Therefore the mean squared error performed by the LAMP method is easily obtained from the optimal overlap reached by the AMP algorithm and yields

MSE achieved by PCA -The respective result for PCA can be obtained from the observation that for the linear case, the α = 0 LAMP operator reduces exactly to the matrix Y. Therefore we can simply state that the mean squared error performed by PCA is computed using the optimal overlap reached by AMP at α = 0:

Comparing AMP with spectral methods -In order to fairly compare PCA, LAMP and AMP in Fig. 5.4, instead of showing the MSE corresponding to the normalized PCA and LAMP estimators (i.e. eqs. (D.31) and (D.32)), we rescale these spectral estimators by the optimal normalization (q v,AMP ) 1/2 (obtained from AMP). This is the convention used in Fig. 5.4, both in the linear and non-linear cases.

D.5 Derivation of thresholds in phase retrieval

D.5.1 Weak recovery

We detail here the derivation of the algorithmic weak-recovery threshold α WR,Algo . As discussed in Section 6.3.1, the weak-recovery threshold can be identified as the sample complexity for which the trivial fixed point q x = q z = qx = qz = γ x = γ z = 0 of the state evolution equations becomes linearly unstable. We repeat here the detailed state evolution for convenience:

(D.33)

The state evolution equations

Since we assumed a Gaussian prior we have, with P out (y|z) = δ(y -|z| 2 ):

Comparing these equations to Conjecture 6.1, one can see that in particular we have γ x = 0, a straightforward consequence of the Gaussian prior.

Noisy phase retrieval with small variance

We wish to show that the free entropy of the full recovery solution is the global maximum of the free entropy potential for α > α IT , while it is never the case for α < α IT . However, under a noiseless channel, the free entropy potential might diverge in this point, which indicates towards a regularization procedure. Therefore we consider a noisy Gaussian channel with noise ∆ > 0, i.e. P out (y|z) = (2π∆) -1/2 exp{-(y -|z| 2 ) 2 /(2∆)}. We will compute the limit, as ∆ ↓ 0, of the free entropy of the "almost perfect" recovery fixed point. We look for a solution close to the point which corresponds to the best possible recovery, that is q z = 1 and q x = 1-ν({0}). Indeed it is easy to see that q x ≤ 1 -ν({0}) since rk[Φ † Φ] ∼ n(1 -ν({0})). We are thus looking for a fixed point of the state evolution equations (D.38) that satisfies:

Let us now precise the asymptotics of these quantities as ∆ ↓ 0. By eq. (D.38d), we find easily:

Then from eq. (D.38c), we also have:

Note that if α ≤ 1, then necessarily ν({0}) ≥ 1 -α, so that the quantity in the numerator is always positive. We now turn to eq. (D.38a). We assume the scaling q-1 z = c∆ + O ∆ (∆). We

Appendix E

Details of the topological approach E.1 The quenched complexity calculation

E.1.1 The phase volume factor

Introducing the Fourier transform of the deltas, we reach at leading exponential order in n:

q ab qabln det q .

The replica symmetric assumption can also be made on the variables q that achieve this supremum : qaa = q0 and qab = -q for a = b. This leads to det q = (q 0 + q) p-1 (q 0 -(p -1)q), and after taking the p ↓ 0 limit, we reach:

q0 + q qln(q 0 + q) + q q0 + q .

The diverging term -(ln n)/2 will be canceled out by the joint density of the gradients as we will see later. The solution of the supremum is easy to carry out, and we finally reach eq. (7.30).

E.1.2 The joint density of the gradients

We denote S = Span ({x a } p a=1 ) ⊂ R n . Following [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF], for every 1 ≤ a ≤ p we can construct an orthonormal basis of S, denoted (e a b ) 1≤b≤p for which x a is the first vector, that is e a a = x a . This basis is convenient, since {x a } ⊥ ∩ S = Span ({e a b } b( =a) ). We can also chose an arbitrary orthonormal basis (e p+1 , • • • , e n ) of S ⊥ . With this choice of basis, we can see that the gradient grad L(x a ) is identified with the vector in R n-1 with components:

Let us make a few remarks:

• For every a, the Gram matrix of the basis (e a b ) p b=1 is only a function of the values of the overlaps {q ab }.

• We consider the joint density of the gradients conditioned by the value of {y a }. In particular, this means that for every a = b, ∇L(x a ) • e a b is fixed by the values of {y c } p c=1 and the overlaps q ab . In particular, the first (p-1) components of eq. (E.1) are deterministic, thus their density will yield delta functions that are constraints on {y a } and {q ab }.

• The last n-p components of eq. (E.1) are (at fixed {y a }) zero mean Gaussian random variables with covariance given by E[gradL(x a ) i gradL(x b ) j ] = (δ ij /m 2 ) µ φ (y a µ ) φ (y b µ ). Their joint density taken at 0 is thus at leading exponential order in n:

Given these remarks and eq. (E.2), in order to complete the calculation of the joint gradient density we need to compute the quantities (∇ L(x a )•e a b ) for every a = b as a function of {y a µ } and {q ab }. In order to simplify the calculation, we will already make use of the replica-symmetric assumption on q, that is we assume q aa = 1 and q ab = q for a = b. Let us now describe a possible construction for the basis (e a b ) p b=1 . We introduce three auxiliary quantities that are functions of q and p:

Using these definitions, we can consider:

It is straightforward to check from eq. (E.4) that we have for all a, b, c that e a b • e a c = δ bc . We can now see that the delta term of the joint density of the gradients taken at 0 is:

The product of eq. (E.2) and eq. (E.5) gives eq. (7.31).

E.1.3 Decoupling replicas, and the p ↓ 0 limit

Replica symmetry and decoupling

In order to apply the replica method, we need to be able to take the p ↓ 0 limit, by analytically extending eq. (7.35) to all p > 0. The main idea is that we expect replica symmetry to influence the measure ν that solves the supremum in eq. (7.35). More precisely, we expect that for all permutation π of {1, • • • , p}, we have ν(dλ → sup

In this last expression, the replica symmetric assumption leads us in particular to assume that µ a = µ for all a. In order to make the remaining calculation tractable we will also need to fix some linear statistics of ν via Lagrange multipliers: Thanks to replica symmetry, we can compute Q -1 and det Q as:

We define

Using all the above, we can now simplify eq. (E.10):

DξI p (ξ) p , (E.11) in which we defined ξ ≡ (ξ q , ξ a , ξ b , ξ b ) and

Although the involved expressions are very cumbersome, we have successfully decoupled the replicas.

The p ↓ 0 limit, and the final result

We begin by a remark on eq. (E.11). Note that lim p↓0 (1/p) ln Dξ I p (ξ) p = Dξ ln I 0 (ξ). Thus, after multiplication by (1/p), the p ↓ 0 limit of eq. (E.11) will yield:

+ α Dξ ln I(ξ), (E.12) in which I(ξ) is defined in Result 7.1. We can wrap up the calculation. We make two remarks. First the condition of eq. (E.8) reduces, in the p ↓ 0 limit, to b = qB, so that we will be able to simplify the terms involving the Lagrange multiplier C. Secondly, the variable B is equal to t φ (µ), defined in Theorem 7.5. We combine now eqs. (7.34a),(7.35),(E.9) and (E.12) with the two remarks above. Changing notations from µ to ν and B to C, we obtain finally the conclusion of Result 7.1.

E.2 Details of proof for the annealed complexity E.2.1 Proof of Lemma 7.7

We will apply the Kac-Rice machinery in the form of the remark made in Paragraph 6.1.4 of [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF]. We recall it as a theorem: by Fubini's theorem. Therefore q E z φa (qz) → q→∞ ϕ a (0) by continuity of ϕ a around a = 0 (Definition 7.1), so φgradL 1 (x) 1 < ∞. Thus gradL 1 (x) admits the following probability density:

which is a continuous function of u, since φgradL 1 (x) ∈ L 1 (R n-1 ). This shows (iii). In order to show (iv), we will use Proposition 6.5 of [START_REF] Azaïs | Level sets and extrema of random processes and fields[END_REF], that we recall here: Since φ is assumed to be of class C 3 , hypothesis (a) is verified for Z = grad L 1 . Notice then that we can fix C > 0 such that |E z∼N (0,1) φa (qz)| ≤ C/(1 + q) for all q ≥ 0. Starting from eq. (E.14):

with C c , D n constants depending only on n, using that m ≥ n (α > 1). This shows (b), so by Lemma E.2, hypothesis (iv) of Theorem E.1 follows. This ends the proof of Lemma 7.7.

E.2.2 Proof of Lemma 7.10

The proof is done in several parts, and is inspired by arguments of [Sil95, SB95, SC95, BS10].

Technicalities on the Hessian

We begin by a technical lemma on Λ(y), defined in eq. (7.12).

Lemma E.3 (Low-rank perturbation)

Since the distributions of z and y are independent, by rotation invariance we can assume that Λ(y) is a diagonal matrix with elements Λ µ (y). There exists a constant, denoted ||D|| ∞ , such that for all n, y, |D(y)| ≤ ||D|| ∞ . Then we have:

(ii) Let Z ∈ R (n-1)×m be i.i.d. variables with zero mean and unit variance. We denote µ

Λ the ESDs of ZD(y)Z /n and ZΛ(y)Z /n respectively. Then for all η ∈ (0, 1),

in which we used the uniform boundedness of |D(y µ )|, and the Cauchy-Schwarz inequality. This proves (i). We note that zΛ(y)z /n and zD(y)z /n differ by a rank-2 matrix. (ii) is thus an immediate application of the following result (Lemma 2.5 of [START_REF] Bordenave | Lecture notes on random matrix theory[END_REF]):

Lemma E.4 (Low-rank perturbation and empirical spectral distribution)

Let n ≥ 1, and A, B two symmetric matrices of size n, such that the rank of A -B is r. Denote F A (resp. F B ) the CDF of the empirical spectral distribution of A (resp. B). Then

This ends the proof of Lemma E.3.

Proof of Lemma E.4 -We note λ

We have some control of the boundedness of the Hessian, summarized in two subsequent lemmas:

Lemma E.5 (Moment bound)

For all γ > 0, one has:

Proof of Lemma E.5 -We begin by bounding the extremal eigenvalues λ min , λ max of H Λ n (y) (denoted H for lightness):

We used (i) of Lemma E.3. Note that this bound is independent of y. In the same way we can bound λ min , and we reach:

Using this identity, we have the bound:

It is then a very classical result of random matrix theory that the largest eigenvalue of a Wishart matrix zz /n satisfies a large deviation principle in the scale n. This is stated e.g. in Theorem 2.4 of [START_REF] Biroli | Large deviations for the largest eigenvalues and eigenvectors of spiked Gaussian random matrices[END_REF], or as a particular case of Chapter 8 of the present thesis (Result 8.1), which gives moreover the behavior of the rate function I(x). We recall some of its properties:

• I(x) : [s max , +∞) → R + is continuous and increasing.

In particular, by Varadhan's lemma 1.10 it implies that for any C, D, γ > 0:

Combining this inequality with eq. (E.15) ends the proof of Lemma E.5.

Lemma E.6 (Properties of µ α,φ )

Denote ρ n (y) the spectral radius of H Λ n (y). There exists C > 0 such that:

The cut-off and the logarithmic potential

For any > 0, define ln : x ∈ R + → ln(max(x, )), then x → ln |x| is a -1 -Lipschitz function on R. Let δ ∈ (0, 1). In this section, we show that a cut-off n = n -δ on the eigenvalues closest to 0 does not perturb the logarithmic potential at the thermodynamical scale. As we mentioned in Section 7.2 we rely on a technical assumption on φ(x). Precisely, for any δ ∈ (0, 1), we assume that there exists η > 0 such that for all t > 0:

Physically, this makes explicit that, with large probability, there should not be enough eigenvalues of H Λ n (y) around zero so that they contribute macroscopically to the logarithmic potential. This is a consequence the natural fluctuations and repulsion of the eigenvalues of H Λ n (y), and we are working to prove it under Definition 7.1 by adapting the arguments of [START_REF] Ben Arous | Exponential growth of random determinants beyond invariance[END_REF]. Denote {λ i } n-1 i=1 the (sorted) eigenvalues of H Λ n (y). We can now state:

Lemma E.7 (Effect of the cut-off on the expected determinant)

There exists η > 0 such that for all K > 0:

Proof of Lemma E.7 -We consider η given by eq. (E.16a). Let t > 0. We denote

in which ∆ iµ ∈ R (n-1)×(n-1) with (∆ iµ ) jk ≡ Λ µ (y)(δ ij z kµ + δ ik z jµ ). So one shows easily:

Let us recall the Hoffman-Wielandt inequality [START_REF] Hoffman | The variation of the spectrum of a normal matrix[END_REF]:

Lemma E.9 (Hoffman-Wielandt inequality for the L 2 norm)

Let k ∈ N , and A, B ∈ S k (R) be two symmetric matrices with respective eigenvalues

In particular if A and B are positive matrices one has Tr[AB] ≤ i λ i (A)λ i (B). We use this in eq. (E.17) along with the n δ -Lipschizity of f n :

in which we used Lemma E.3. We denote A the event

It is a classical concentration result (cf e.g. Chapter 3.1 of [START_REF] Vershynin | High-dimensional probability: An introduction with applications in data science[END_REF]) that there exists c > 0 such that:

In particular, this implies

Let us now show that it suffices to prove the bound of Proposition E.8 assuming that A does not occur. Indeed, n -2(1-δ) ln P z [A] ≤ -cn 2δ for a constant c > 0, and:

≤ lim sup

We used Markov's inequality in (a). The inequality (b) can be obtained by very similar arguments than the one used to prove Lemma E.5: by rescaling z by z , it is easy to see that the event A will not change the scaling of the large deviations of the largest eigenvalue of the Wishart matrix zz /n, so that the bound of Lemma E.5 will also apply when conditioning by the event

Using the law of total expectation, we can safely ignore the occurrence of A when showing Proposition E.8. Assuming that A is not occurring yields:

Recall the Lipschitz concentration of independent variables with laws satisfying the logarithmic Sobolev inequality with a uniform constant c (see for instance [START_REF] Greg W Anderson | An introduction to random matrices[END_REF] for a proof and an introduction to the logarithmic Sobolev inequalities):

Lemma E.10 (Herbst)

Let n ∈ N and P be a probability distribution on R n satisfying the Logarithmic Sobolev Inequality with constant c > 0. Let G be a Lipschitz function on R n with Lipschitz constant G L . Then for all t > 0,

It is easy to check that the Gaussian standard law of z, conditioned by the (extremely probable) event A, satisfies the Logarithmic Sobolev Inequality with constant c = 1 + O n (1). Applying Lemma E.10 alongside eq. (E.20) finishes the proof.

The logarithmic potential of the asymptotic measure

In this part we relate the expected logarithmic potential to the logarithmic potential of the measure µ α,φ [ν m y ], cf Theorem 7.5.

Proposition E.11 (Limit of the expected logarithmic potential)

There exists η > 0 such that for all t > 0:

Proof of Proposition E.11 -The proof goes in two parts. First, we show that there exists η 1 > 0 such that 1 :

We will then conclude by showing that there exists η 2 > 0 such that for all t > 0:

We begin by eq. (E.22). We take η 2 given by eq. (E.16b). We have, since ln (x) ≥ ln(x):

and using eq. (E.16b), we reach eq. (E.22). Let us show eq. (E.21). Its proof is based on the following lemma, a consequence of the analysis of [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF][START_REF] Bai | Spectral analysis of large dimensional random matrices[END_REF]:

Lemma E.12 (Convergence of the Stieltjes transform)

Denote g n (z) the Stieltjes transform of zΛ(y)z /n, and g α,φ [ν m y ](z) the one of µ α,φ [ν m y ], for z ∈ C + . Then there exists η ∈ (0, 1) such that for all z ∈ C + :

The proof follows quite closely [START_REF] Jack | On the empirical distribution of eigenvalues of a class of large dimensional random matrices[END_REF], using cavity method arguments for random matrices that we used as well in Chapter 5. The interested reader can refer to the Appendix of [START_REF] Maillard | Landscape complexity for the empirical risk of generalized linear models[END_REF] for the detailed proof. Let us fix η given by Lemma E.12. As stated for instance in Theorem 2.4.4 of [START_REF] Greg W Anderson | An introduction to random matrices[END_REF], a consequence of the Stieltjes-Perron inversion (Theorem 1.5) is that for every Borel set E ⊆ R:

in which µ n is the ESD of zΛ(y)z /n. Fix η 1 < η. We have, uniformly over y:

Let us fix C > 0 given by item (i) of Lemma E.6. We can bound t φ (ν m y ) by ||xφ || ∞ . This gives that for n large enough the RHS of eq. (E.25) is bounded (uniformly over y) by:

Since η 1 < η we can use eq. (E.24), which shows eq. (E.21).

Conclusion of the proof

Let us conclude the proof of Lemma 7.10 from all our results of Section E.2.2. We fix δ > 0 such that δ < 1/2. Note that Proposition E.8 is a uniform result on y, much stronger than what we required. Since it shows that the concentration (as a function of z) of the log-determinant is in a scale n 1+ with > 0, it implies that there exists η > 0 such that for all t > 0:

Combining this identity with Lemma E.7 and Proposition E.11, we reach the conclusion of Lemma 7.10.

E.2.3 Proof of Lemma 7.11

Let γ ∈ (1, α). We fix C > 0 given by Lemma E.6. Then for all y ∈ R m :

By

By continuity of the involved functions, in order to conclude from eq. (E.28) we just need to be able to show that (i) : lim sup S↑A (-Λ (S)) < ∞ and (ii) : lim sup S↓0 [-γ 2 ln S -αΛ (S)] < ∞. Point (i) is trivial since Λ (S) ≥ 0 for all S ∈ (0, A) (it is a rate function). To show (ii), we use the fact that for all S ∈ (0, v) and θ ≥ 0 we have Λ (S) ≥ -θS -ln E[e -θφ (y) 2 ]. In particular, for θ = S -1 we have Λ (S) ≥ -1 -ln E[e -S -1 φ (y) 2 ]. Since a = φ (y) has a density ϕ a continuous around 0 (Def. 7.1), we fix a 0 > 0 such that ϕ a is continuous in [-a 0 , a 0 ]. For every θ > 0:

ln E[e -θφ (y) 2 ] ≤ ln E(e -θa 2 1 |a|≤a 0 ) + e -θa 2 0 ≤ ln ( sup

and thus ln E[e -θφ (y) 2 ] ≤ C -(1/2) ln θ with a constant C > 0. Using this bound and the remark before we reach

Since α -γ > 0, we have lim S↓0 [-γ 2 ln S -αΛ (S)] = -∞, which obviously implies point (ii), which in turn shows eq. (E.27).

E.2.4 Proof of eq. (7.20)

Let t > 0, and fix η > 0 given by Lemma 7.10. We define

.

(E.29)

A n is related to the complexity by Lemma 7.8, and by Lemma 7.11 we can apply Varadhan's lemma 1.10 to B n , which yields

The factor α in front of the relative entropy arises as we consider the empirical distribution of m i.i.d. variables. For all t > 0, we have by definition of A n , B n :

.

(E.31) Using Hölder's inequality and Lemma 7.11, there exists therefore γ > 1 and a constant C > 0 such that:

However, by eq. (E.32) and Lemma 7.10, we

n ] → -∞. So we showed that lim B n = -∞ ⇒ lim A n = -∞, which shows eq. (7.20) in this case.

Let us now assume that lim B n > -∞. Using the left inequality of eq. (E.32) and Lemma 7.10, we reach in the same way that lim inf[A n -B n ] ≥ -t, which implies that lim inf A n > -∞. Thus we can use the right inequality of eq. (E.32) to show similarly that lim sup[A n -B n ] ≤ t. Taking the t → 0 limit finishes the proof of eq. (7.20).

E.3 The large deviations in the white Wishart case

In the white Wishart case, we have ρ(t) = δ(t -1), and the density σ(λ) is explicitly known, it is the Marchenko-Pastur distribution [START_REF] Vladimir | Distribution of eigenvalues for some sets of random matrices[END_REF]:

One can also explicitly solve the Marchenko-Pastur equation (8.1) (it is just a quadratic equation in this case) and obtains for x ≥ λ + (α):

By Result 8.1, this implies that the rate function I(x) satisfies for every x ≥ λ + (α):

On the other hand, direct calculations using the joint law of eigenvalues of a Wishart matrix give the following expression of the rate function (see e.g. Theorem 2.4 of [START_REF] Biroli | Large deviations for the largest eigenvalues and eigenvectors of spiked Gaussian random matrices[END_REF]) for x ≥ λ + (α):

The logarithmic potential of the Marchenko-Pastur law is known analytically, as stated in Proposition II.1.5 of [START_REF] Faraut | Logarithmic potential theory, orthogonal polynomials, and random matrices[END_REF]. More precisely, we have for all x ≥ λ + (α):

It is then immediate to see that eq. (E.35) and eq. (E.36) are equivalent, validating Result 8.1 in this simple (yet important) case.

E.4 The phase transition in the rate function

In this section, we investigate possible discontinuities in the derivatives of the rate function I(x), when d max > 0 and x c (ρ) is finite. In this case, the function G σ (x) is constant and equal to α/d max for x ≥ x c (ρ). Recall that if s max ≤ x ≤ x c (ρ), G σ (x) is the second branch to the Marchenko-Pastur equation (8.1). This equation can be written as

Let us assume that ρ(t) ∼ (d max -t) η with η > 0 and t close to d max .

Recall the functions J 2 and F 2 (with θ c (x) = xG σ (x) 2 + (α -1)G σ (x)):

We perform the change of variable θ(τ, x) 2 ≡ xτ 2 + (α -1)τ . At the critical value θ c (x), we have τ c (x) = G σ (x). We obtain the expression of the rate function as I(x) = sup τ ≥Gσ(x) I(x, τ ), with I(x, τ ) = J(τ, x) -F (τ, x), in which we naturally defined:

Similarly, we have the following expression for F (τ, x):

Using these expressions for J and F , we then compute

] is again the Marchenko-Pastur equation (8.1), with ω = τ . Since τ x > G σ (x), it is easy to check from eq. (E.46) that the supremum must be attained in τ x = G σ (x). This is true even if x > x c (ρ), as then the maximum is attained in τ = α/d max = G σ (x), again from eq. (E.46). Moreover we can compute (from the first equality in eq. (E.45)):

Since I(s max ) = 0 and τ x = G σ (x), this implies eq. (E.45).

RÉSUMÉ

Le déluge croissant de données qui a rythmé la dernière décennie a donné naissance à des techniques modernes dans le domaine de l'intelligence artificielle. Ces méthodes sont basées sur l'optimisation d'un très grand nombre de paramètres par l'exploitation d'une quantité gargantuesque de données, et ces algorithmes sont désormais l'état de l'art pour des tâches aussi diverses que la classification d'images, le traitement automatique des langues, ou la reconnaissance vocale, et leurs performances excèdent régulièrement les capacités humaines. En conséquence, de nombreuses recherches se sont concentrées sur la construction d'une théorie mathématique qui pourrait expliquer l'efficacité de ces algorithmes, créant un fort gain d'intérêt pour les statistiques en haute dimension, où la quantité de données et le nombre de paramètres sont tous deux très grands. Nous analysons ici quelques pièces de cet immense puzzle à travers le prisme de la physique statistique, en empruntant également aux probabilités et à la théorie des matrices aléatoires. Ces outils nous permettent de proposer trois approches au problème de l'apprentissage statistique en haute dimension. Dans la première nous revisitons un classique de la physique statistique, les expansions de haute température. Nous expliquons comment cette méthode est liée à des algorithmes modernes, et nous l'utilisons pour proposer les prémices d'une théorie exacte de la factorisation de matrices à rang extensif. Pour cela nous exploitons la connection forte qui relie la physique des systèmes désordonnés et les statistiques en grande dimension, un sujet de recherche qui suscite un intérêt croissant depuis les années 1990. Dans une seconde partie nous poussons cette correspondance plus loin et utilisons des outils heuristiques de physique théorique, comme la méthode des répliques, associés à des outils probabilistes et des algorithmes de passage de message, pour décrire les limites fondamentales d'une grande catégorie de problèmes d'apprentissage. Nous appliquons cette analyse à des réseaux de neurones, à l'extraction de phase, ainsi que pour étudier l'influence de la structure des données sur les procédures d'inférence. Enfin nous proposons une direction alternative, une approche topologique au problème d'inférence en haute dimension: en utilisant des outils de géométrie différentielle stochastique et de matrices aléatoires, nous prouvons des formules exactes décrivant la structure des paysages d'énergie optimisés par les algorithmes d'apprentissage.

MOTS CLÉS

Physique statistique, Apprentissage automatique, Statistiques en haute dimension, Théorie de l'information, Théorie des matrices aléatoires, Optimisation non convexe.

ABSTRACT

The past decade saw an intensification of the deluge of data available to learning algorithms, which allowed for the development of modern artificial intelligence techniques. These methods rely on the optimization of a very large number of internal parameters using gigantic amounts of data, and now provide state-of-the-art algorithms for tasks as diverse as image classification, natural language processing, or speech recognition, and regularly achieve super-human performances. This exacerbated research efforts to build a mathematically sound theory of data science able to explain the extraordinary efficiency of these procedures, and has led to a surge of interest for high-dimensional statistics (i.e. when the amount of data and the number of parameters are both very large). In this dissertation we analyze a few pieces of this immense puzzle through the prism of statistical physics, borrowing also often from probability and random matrix theory, and we propose three approaches to the high-dimensional learning problem. In the first one we revisit high-temperature expansions, an archetypal method of statistical physics. We show how this classical approach is related to modern algorithms, and use it to pave the way towards an exact theory of extensive-rank matrix factorization. Our theory leverages the intimate relation between the statistical physics of disordered systems and high-dimensional statistics, a connection which has been a growing subject of research since the 1990s. Our second approach pushes further this correspondence as we leverage heuristic tools of theoretical physics such as the replica method, along with modern probabilistic methods and message-passing algorithms, to describe the fundamental limits of a wide class of high-dimensional learning problems. We apply our analysis to neural networks, phase retrieval, and to study the influence of data structure on the optimal learning procedures. In a third part we take an alternative route and consider a topological approach to the problem of learning in high dimension. Using tools of random differential geometry and random matrix theory we prove exact formulas describing the structure of the high-dimensional landscapes optimized by learning algorithms.

KEYWORDS

Statistical physics, Machine learning, High-dimensional statistics, Information theory, Random matrix theory, Non-convex optimization.