Merci À Adrien

Cédric, Florent, Frédéric, Guilhem, Lauriane, Lenka Maxence Benjamin

Christian, Thibault Alaa Jean

Ale- Jandro Laura

Antoine Benjamin

Antoine Bruno

Frédéric, Benoît, Gabriel Dimitri

Pour cela je remercie aussi

My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, "You should call it entropy, for two reasons. In

Contents

Acknowledgements v

Foreword vii

Index of notations and abbreviations xi

I Background

Foreword

With the continuous improvement of storage techniques, the amount of available data is currently growing exponentially. While it is not humanly feasible to treat all the data created, machine learning is one possible response. It is a class of algorithms that allows to automatically infer structure in large data sets. In particular, deep learning methods, based on neural networks, have drastically improved performances in key fields of artificial intelligence such as image processing, speech recognition or text mining. A good review of the first successes of this technology published in 2015 is [START_REF] Lecun | Deep learning[END_REF]. A few years later, the current state-of-the-art of this very active line of research is difficult to envision globally. However, the complexity of deep neural networks remains an obstacle to the understanding of their great efficiency. Made of many layers, each of which constituted of many neurons with a collection of parameters, the set of variables describing completely a typical neural network is impossible to only visualize. Instead we must consider aggregated quantities to characterize these models and hopefully help and explain the learning process. The first open challenge is therefore to identify the relevant observables to focus on. Often enough, what seems interesting is also what is hard to calculate. In the high-dimensional regime we need to consider, exact analytical solutions are unknown most of the time and numerical computations are ruled out. Hence we need to find ways of approximations that are simultaneously simple enough to be tractable and fine enough to retain interesting features.

In the context where dimensionality is an issue, physicists have experimented that macroscopic behaviors are typically well described by the theoretical limit of infinitely large systems. Under this thermodynamic limit, the statistical physics of disordered systems offers powerful frameworks of approximation called mean-field theories. Nevertheless, the models studied by physicists usually include assumptions of randomness and (statistical) homogeneities. Whereas in machine learning, interactions between neurons, that are chosen by training algorithms with respect to specific data sets, have no reason a priori to be as regular as mean-field theories assume. Thus, it is not straightforward that methods suited in theoretical physics can be applied to deep learning. In this dissertation we discuss cases where the bridge between these two disciplines can be crossed. Thanks to this approach we propose new learning procedures and contribute to the theoretical characterization of other popular, yet poorly understood, training algorithms.

Organization of the manuscript

The first part of this dissertation covers the basic theoretical concepts that are relevant for the presentation of our original contributions.

In Chapter 1 we wish to clarify the context and motivation of our work by giving a minimal introduction to methods and challenges in deep learning that will be of particular interest to the physicist reader. This Chapter is also the occasion to recall some of the historical connections between learning and mean-field theories. In Chapter 2, we introduce the conceptual frameworks of statistical inference and statistical physics that are underlying our approach. Finally, we present mean-field methods in Chapter 3 with several goals in mind. First, we wish to introduce the different techniques of derivation that will be used in the second part of the dissertation. Our second aim is to provide intuitions on where the approximation lies in mean-field computations, or in other words on what they neglect. Third, we attempt to clarify how the different methods are related, when they are equivalent and when they complement. Lastly, we present some recent advances on which the contributions presented in this dissertation are building.

The second part is divided into three Chapters, presenting different directions of research in the training and analysis of neural networks.

In Chapter 4, we present our publications on the unsupervised learning of Boltzmann machines with hidden units: vii Index of notations and abbreviations [N] Set of integers from 1 to N δ(•) Dirac distribution σ(x) = (1 + e -x ) -1 Sigmoid relu(x) = max(0, x) Rectified Linear Unit This chapter provides the fundamental concepts in machine learning that will be used throughout this thesis. A comprehensive reference is [START_REF] Goodfellow | Deep Learning[END_REF]. We also take this chapter as an opportunity to introduce the current challenges in the understanding of deep learning. With these first elements of context, we briefly review the historical connections between mean-field methods coming from statistical physics and neural networks used for machine learning.

Neural networks for machine learning

Machine learning is traditionally divided into three classes of problems: supervised, unsupervised and reinforcement learning. For all of them, the advent of deep learning techniques, relying on deep neural networks, has brought great leaps forward in terms of performance and opened the way to new applications. Nevertheless, the utterly efficient machinery of these algorithms remains full of theoretical puzzles. In this Section we quickly cover key concepts in supervised and unsupervised machine learning that will be useful to understand the context of this thesis and motivate its contributions.

Supervised learning

Learning under supervision

Supervised learning aims at discovering systematic input to output mappings from examples. For instance, classification is a typical supervised learning problem. For example, from a set of pictures of cats and dogs labelled accordingly, the goal is to find a function able to predict in any new picture the species of the displayed pet.

In practice, the training set is a collection of P example pairs D = {x (k) , y (k) } P k=1 from an input data space X ⊆ R N and an output data space Y ⊆ R M . Formally, they are assumed to be i.i.d. samples from a joint distribution p(x, y). The predictor h is chosen by a training algorithm from a hypothesis class, a set of functions from X to Y, so as to minimize the error on the training set. This error is formalized as the empirical risk

R(h, , D) = 1 P P k=1 (y (k) , h(x (k) )), (1.1)
where the definition involves a loss function : Y × Y → R measuring differences in the output space. This learning objective nevertheless does not guarantee generalization, i.e. the ability of the predictor h to be accurate on inputs x that are not in the training set. It is a surrogate for the ideal, but unavailable, population risk

R(h, ) = E x,y (y, h(x)) = X ,Y
dx dy p(x, y) (y, h(x)), (1.2) expressed as an expectation over the joint distribution p(y, x). The different choices of hypothesis classes and training algorithms yield the now crowded zoo of supervised learning algorithms.

Representation ability of deep neural networks

In the context of supervised learning, deep neural networks enter the picture in the quality of parametrized hypothesis class. Let us first quickly recall the simplest network, the perceptron. It Let's assume we wish to classify data points x ∈ R 2 with labels y = ±1. We choose as an hypothesis class the perceptron sketched on the left with sign activation. For given weight vector w and bias b the plane is divided by a decision boundary assigning labels. If the training data are linearly separable, then it is possible to perfectly predict all the labels with the perceptron, otherwise it is impossible. corresponds to single neuron, so a function from R N to Y ⊂ R applying an activation function f to a weighted sum of its inputs shifted by a bias b ∈ R,

ŷ = h w,b (x) = f (w x + b) (1.3)
where the weights are collected in the vector w ∈ R N . From a practical standpoint this very simple model can only solve the classification of linearly separable groups (see figure 1.1). Yet from the point of view of learning theory, it has been the starting point of a rich statistical physics literature as will be discussed in section 1.2.

Combining several neurons into networks defines more complex functions. The universal approximation theorem [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF][START_REF] Hornik | Approximation Capabilities of Multilayer Neural Network[END_REF] proves that the following two-layer network architecture can approximate any well-behaved function with a finite number of neurons, ŷ = h θ (x) = w (2) f (W (1) x + b) = M α=1 w (2) α f (w (1) α

x + b α ), θ = {w (2) , W (1) , b} (1.4) for f a bounded, non-constant, continuous scalar function, acting component-wise. In the language of deep learning this network has one hidden layer of M units. Input weight vectors w [START_REF] Achille | Emergence of Invariance and Disentangling in Deep Representations[END_REF] α ∈ R N are collected in a weight matrix W (1) ∈ R M ×N . Here, and in the following, the notation θ is used as short for the collection of adjustable parameters. The universal approximation theorem is a strong result in terms of representative power of neural networks but it is not constructive. It does not tell us what is the minimal M to approximate a given function, nor what are the optimal values of the parameters w (2) , W (1) and b. These questions remain unsolved to this date and practice is widely led by empirical considerations.

In applications, neural networks with multiple hidden layers were found more effective. A generic neural network of depth L is the function ŷ = h θ (x) = f (W (L) f (W (L-1) • • • f (W (1) x + b (1)

) • • • + b (L-1) ) + b (L) ), (1.5
)

θ = {W (l) ∈ R N l ×N l-1 , b (l) ∈ R N l ; l = 1 • • • L}, ( 1.6)
where N 0 = N is the dimension and the input and N L = M is the dimension of the output. The architecture is fixed by specifying the number of neurons, or width, of the hidden layers {N l } L-1 l=1 . The latter can be denoted t (l) ∈ R N l and follow the recursion t (1) = f (W (1) x + b (1) ) ,

(1.7)

t (l) = f (W (l) t (l-1) + b (l) ) , l = 2 • • • L -1 , ( 1.8)
ŷ = f (W (L) t (L-1) + b (L) ) .

(1.9)

Fixing the activation functions and the architecture of a neural network defines an hypothesis class.

It is crucial that activations introduce non-linearities; the most common are the hyperbolic tangent tanh and the rectified linear unit defined as relu(x) = max(0, x). Note that it is also possible, albeit uncommon in supervised learning applications, to define stochastic neural networks by using noisy activation functions. An originally proposed intuition for the advantage of depth is that it enables to treat the information in a hierarchical manner; either looking at different scales in different layers, or learning more and more abstract representations [START_REF] Bengio | Representation Learning: A Review and New Perspectives[END_REF]. Nevertheless, there is no theory to this day clarifying why in practice 'the deeper the better'.

Neural network training

Given an architecture defining h θ , the supervised learning objective is to minimize the empirical risk R with respect to the parameters θ. This optimization problems lives in the dimension of the number of parameters which can range from tens to millions. The idea underlying the majority of training algorithms is to perform a gradient descent (GD) from a random initialization of the parameters:

θ 0 ∼ p θ0 (θ 0 )
(1.10)

θ t+1 ← θ t -η∇ θ R = θ t -η 1 P P k=1 ∇ θ y (k) , h θt x (k) . (1.11)
The parameter η is the learning rate, controlling the size of the step in the direction of decreasing gradient per iteration. The computation of the gradients can be performed in time scaling linearly with depth by applying the derivative chain-rule leading to the back-propagation algorithm [START_REF] Goodfellow | Deep Learning[END_REF]. A popular alternative to gradient descent is stochastic gradient descent (SGD) where the sum over the gradients for the entire training set is replaced by the sum over a small number of samples, randomly selected at each step [START_REF] Robbins | A stochastic approximation method[END_REF][START_REF] Bottou | Large-Scale Machine Learning with Stochastic Gradient Descent[END_REF].

During the training iterations, one typically monitors the training error (another name for the empirical risk given a training data set) and the validation error. The latter corresponds to the empirical risk computed on a set of points held-out from the training set, the validation set, to assess the generalization ability of the model either along the training or in order to select hyperparameters of training such as the value of the learning rate. A posteriori, the performance of the model is judged from the generalization error, which is evaluated on the never seen test set. While two different training algorithms (e.g. GD vs SGD) may achieve zero training error, they may differ in the level of generalization they typically reach.

Open questions and challenges

Building on the fundamental concepts presented in the previous paragraphs, practitioners managed to bring deep learning to unanticipated performances in the automatic processing of images, speech and text (see [START_REF] Lecun | Deep learning[END_REF] for a few years old review). Still, many of the greatest successes in the field of neural network where obtained using ingenious tricks while many fundamental theoretical questions at a very basic level remain unresolved.

Regarding the optimization first, (S)GD training generally discovers parameters close to zero risk. Yet gradient descent is guaranteed to converge to the neighborhood of a global minimum only for a convex function and is otherwise expected to get stuck in a local minimum. Therefore efficiency of gradient based optimization is a priori a paradox given the empirical risk R is nonconvex in the parameters θ. Second, the representation capacity of deep learning models is also poorly understood. Results in the literature that relate the size and architecture of a network to a measure of its ability to learn are too far from realistic settings to guide choices of practitioners. On the one hand, traditional bounds in statistics, considering worst cases, appear overly pessimistic. On the other hand, historical statistical physics analyses of learning, briefly reviewed in Section 1.2, only concern simple architectures and synthetic data. This lack of theory results in potentially important waste: in terms of time lost by engineers in trial and error to optimize their solution, and in terms of electrical resources used to train and re-train possibly oversized networks and store potentially un-necessarily large training data sets. Lastly, the generalization ability of deep neural networks trained by gradient descent is still unaccounted for. The size of training data sets is limited by the cost of human labelling. Thus training a deep and wide network amounts in practice to fitting a model of millions of degrees of freedom against a somehow relatively small amount of data points. Nevertheless it does not systematically lead to overfitting. The resulting neural networks can have good predictions both on inputs seen during training and on new inputs.

The success of deep learning, beyond these apparent theoretical puzzles,certainly lies in the interplay of advantageous properties of training algorithms, the neural network hypothesis class and structures in typical data (e.g. real images, conversations). Disentangling the role of the different ingredients is a very active line of research (see [START_REF] Vidal | Mathematics of Deep Learning[END_REF] for a review) among which this thesis is a modest contribution to. In particular, Chapter 5 touches the question of the generalization capability of neural networks.

Unsupervised learning

Density estimation and generative modelling

The goal of unsupervised learning is to directly extract structure from data. Compared to the supervised learning setting the training data set is made of a set of example inputs D = {x (k) } P k=1 without corresponding outputs. A simple example of unsupervised learning is clustering, consisting in the discovery of unlabelled subgroups in the training data. Most unsupervised learning algorithms either implicitly or explicitly adopt a probabilistic viewpoint and implement density estimation. The idea is to approximate the true density p(x) from which the training data was sampled by the closest (in various senses) element among a family of parametrized distributions over the input space {p θ (.), θ ∈ R N θ }. The selected p θ is then a model of the data. Structural properties of the data can sometimes be inferred from it, such as dependencies according to its factorization (see Section 2.1.1). If the model p θ is easy to sample, it can also be used to generate new inputs comparable to the training data points -which leads to the terminology of generative models. In this context, unsupervised deep learning exploits the representational power of deep neural networks to create sophisticated candidate p θ .

A common formalization of the learning objective is to maximize the likelihood, defined as the logarithm of the probability of i. (1.12)

The second logarithmic additive formulation is generally preferred. It can be interpreted as the minimization of the Kullback-Leibler divergence between the empirical distribution p D (x) = although considering the divergence with the discrete empirical measure is slightly abusive. The detail of the optimization algorithm here depends on the specification of p θ . As we will see, the likelihood in itself is often intractable and learning consists in a gradient ascent on at best a lower bound, otherwise an approximation, of the likelihood.

A few years ago, an alternative strategy called adversarial training was introduced [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] leading to a remarkable quality of samples produced by generative models. However, the training of models appearing in this thesis, presented in the following paragraphs, will fall under the classical maximum-likelihood paradigm.

Deep Variational Autoencoders

A variational autoencoder (VAE) [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF][START_REF] D J Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] defines a density p θ obtained by propagating a simple distribution through a deep neural network. It can be formalized by introducing a latent variable z ∈ R N and a deep neural network h θ similar to (1.5) of input dimension N : z ∼ p z (z) (1.14) x ∼ p θ (x|z) = p out (x|h θ (z)), (1.15) where p z is typically a factorized distribution on R N easy to sample (e.g. a standard normal distribution), and p out (.|h θ (z)) is for instance a multivariate Gaussian distribution with mean and covariance that are function of h θ (z). The computation of the likelihood of one training sample x (k) requires then the marginalization over the latent variable z, p θ (x) = dz p out (x|h θ (z))p z (z). (1.16) This multidimensional integral cannot be performed analytically in the general case. It is also hard to evaluate numerically as it does not factorize over the dimensions of z which are mixed by the neural network h θ . Yet a lower bound on the log-likelihood can be defined by introducing a tractable conditional distribution q(z|x) that will play the role of an approximation of the intractable posterior distribution p θ (z|x) implicitly defined by the model: log p θ (x) = dz q(z|x) log p θ (x) (1.17)

= dz q(z|x) log p θ (x, z) p θ (z|x)

= dz q(z|x) log p θ (x, z) p θ (z|x) × q(z|x) q(z|x)

= dz q(z|x) -log q(z|x) + log p θ (x, z) + KL(q(z|x)||p θ (z|x))

≥ dz q(z|x) -log q(z|x) + log p θ (x, z) = LB(q, θ, x) (1.18) where the last inequality comes from the fact that a KL-divergence is always positive; it is replaced by an equality if and only if q(z|x) = p θ (z|x). Provided q has a tractable expression and is easy to sample, the remaining lower bound LB(q, θ, x) can be approximated by a Monte Carlo method. Maximum likelihood learning is then approached by the maximization of the lower bound LB(q, θ, x), which requires in practice to parametrize the tractable posterior q = q φ as well, typically with a neural network. In order to adjust the parameters θ and φ through gradient descent a last reformulation is nevertheless necessary. The expectation over q φ estimated via Monte Carlo indeed involves the φ parameters. To disentangle this dependence z ∼ q φ (z|x) must be rewritten as z = g( , φ, x) with ∼ p ( ) independent of φ. This so-called re-parametrization trick allows to compute the gradients of LB(q φ , θ, x) with respect to θ and φ.

In this thesis we will mention variational autoencoders in the perspectives of Chapter 4 and Chapter 6. We will discuss a different proposition to approximate the likelihood and the alternative Bayesian learning objective involving prior knowledge on the parameters θ.

Restricted and Deep Boltzmann Machines

Models described in the preceding paragraphs comprised only feed forward neural networks. In feed forward neural networks, the state or value of successive layers is determined following the recursion (1.7) -(1.9), in one pass from inputs to outputs. Boltzmann machines instead involve undirected neural networks which consist of stochastic neurons with symmetric interactions. The probability associated to a neuron state is a function of neighboring neurons, themselves reciprocally function of the first neuron. Sampling a configuration therefore requires an equilibration in the place of a simple forward pass.

A Restricted Boltzmann Machine (RBM) [START_REF] David H Ackley | A Learning Algorithm for Boltzmann Machine[END_REF][START_REF] Smolensky | Information Processing in Dynamical Systems: Foundations of Harmony Theory[END_REF] with M hidden neurons defines a joint distribution over an input (or visible) layer x ∈ {0, 1} N and a hidden layer

t ∈ {0, 1} M p θ (x, t) = 1 Z e a x+b t+x W t , θ = {W , a, b} , (1.19)
where Z is the normalization factor, similar to the partition function of statistical physics. The parametric density model over inputs is then the marginal p θ (x) = t∈{0,1} M p θ (x, t). Identically to VAEs, RBMs can represent sophisticated distributions at the cost of an intractable likelihood.

Indeed the summation over 2 M +N terms in the partition function cannot be simplified by an analytical trick and is only realistically doable for small models. This intractibility remains a priori an issue for the learning by gradient ascent of the loglikelihood. Given an input training point x (k) , the derivatives of log p(x (k) ) with respect to trainable parameters are

∇ a log p(x (k) ) = x (k) -x , (1.20a) ∇ b log p(x (k) ) = t x (k) -t , (1.20b) ∇ W log p(x (k) ) = t x x (k) -t x , (1.20c)
where we introduce the notation . for the average over the RBM measure (1.19) and . x (k) for the average over the RBM measure conditioned on x = x (k) , also called the clamped RBM measure.

To approximate the gradients, sampling over the clamped measure is actually easy as the hidden neurons are independent when conditioned on the inputs; in other words p(t|x) is factorized. Yet sampling over the complete RBM measure is more involved and requires for example the equilibration of a Markov Chain. It is a costly operation that would need to be repeated at each parameter update in the gradient ascent. Instead it is commonly approximated by contrastive divergence (CD) [START_REF] Hinton | Training products of experts by minimizing Contrastive divergence[END_REF], which approximates the . in (1.20) by the final state of a Monte Carlo Markov chain initialized in x (k) and updated for a small number of steps. RBMs were the first effective generative models using neural networks. They found applications in various domains including dimensionality reduction [START_REF] Hinton | Reducing the Dimensionality of Data with Neural Networks[END_REF], classification [START_REF] Larochelle | Classification using discriminative restricted Boltzmann machines[END_REF], collaborative filtering [START_REF] Salakhutdinov | Restricted Boltzmann machines for collaborative filtering[END_REF], feature learning [START_REF] Coates | An analysis of single-layer networks in unsupervised feature learning[END_REF], and topic modeling [START_REF] Hinton | Replicated softmax: an undirected topic model[END_REF]. Used for an unsupervised pre-training of deep neural networks layer by layer [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Greedy Layer-Wise Training of Deep Networks[END_REF], they also played a crucial role in the take-off of supervised deep learning.

Furthermore, they can be generalized to Deep Boltzmann Machines (DBMs) [START_REF] Salakhutdinov | Deep Boltzmann Machines[END_REF], where several hidden layers are stacked on top of each other. The measure defined by a DBM of depth L is

p θ (x, t) = 1 Z e a x+x W (1) t+b (L) t (L) + L-1 l=1 t (l) W (l+1) t (l+1) +b (l) t (l) (1.21) θ = {a , b (l) , W (l) ; l = 1 • • • L}. (1.22)
Similarly to (1.20), the log-likelihood gradient ascent with respect to the parameters involves averages over the DBM measure and the corresponding clamped measure. In this case both of them involve an equilibration to be approximated by a Markov Chain. The contrastive divergence algorithm can be generalized to circumvent this difficulty, but the approximation of the gradients appear all the more less controlled.

Open questions and challenges

Generative models involving neural networks such as VAE, GANs and RBMs have great expressive powers at the cost of not being amenable to exact treatment. Their training, and sometimes even their sampling requires approximations. From a practical standpoint, whether these approximations can be either made more accurate or less costly is an open direction of research. Another important related question is the evaluation of the performance of generative models [START_REF] Mehdi | Assessing Generative Models via Precision and Recall[END_REF]. To start with the objective function of training is very often itself intractable (e.g. the likelihood of a VAE or a RBM), and beyond this objective, the unsupervised setting does not define a priori a test task for the performance of the model. Additionally, unsupervised deep learning inherits some of the theoretical puzzles already discussed in the supervised learning section. In particular, assessing the difficulty to represent a distribution and select a sufficient minimal model and/or training data set seems today far out of reach of theory. In this thesis, unsupervised learning models will be discussed in particular in Chapter 4. A novel deterministic framework for RBMs and DBMs with a tractable learning objective is derived, allowing to exploit efficiently their representative power.

A brief history of mean-field methods for neural networks

The mathematical description of learning implies to deal with a large set of correlated random variables: the data, the network parameters, and the resulting neuron activations. The major hurdle in our theoretical understanding of algorithms based on deep neural networks is the difficulty in performing computations involving these high-dimensional multivariate distributions. This is one aspect of the so-called curse of dimensionality. Here, exact computations are almost never possible and numerical treatments are unrealistically costly. Finding good approximations is therefore crucial. Mean-field methods can be understood as one approximation strategy. Originally developed in the statistical physics literature, their applications to machine learning, and in particular to neural networks, already have a long history and significant contributions to their records.

In the 80s and 90s, a series of works pioneered the analysis of learning with neural networks through the statistical physics lense. By focusing on simple models with simple data distributions, and relying on the mean-field method of replicas, these papers managed to predict quantitatively important properties such as capacities -the number of training data point that could be memorized by a model -or learning curves -the generalization error (or population risk) of a model as a function of the size of the training set. This effort was initiated by the study of the Hopfield model [START_REF] Daniel | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF], an undirected neural network providing associative memory [START_REF] Hopfield | Neural networks and physical systems with emergent collective computational abilities[END_REF]. The analysis of feed forward networks with simple architectures followed (among which [START_REF] Gardner | Maximum Storage Capacity in Neural Networks[END_REF][START_REF] Gardner | The space of interactions in neural network models[END_REF][START_REF] Opper | Calculation of the learning curve of Bayes optimal classification algorithm for learning a perceptron with noise[END_REF][START_REF] Timothy | The statistical mechanics of learning a rule[END_REF][START_REF] Monasson | Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Networks[END_REF][START_REF] Opper | Mean field approach to bayes learning in feed-forward neural networks[END_REF][START_REF] Engel | Statistical Mechanics of Learning[END_REF][START_REF] Monasson | Learning and Generalization Theories of Large Committee-Machines[END_REF][START_REF] Aubin | The committee machine: Computational to statistical gaps in learning a two-layers neural network[END_REF]). Physicists, accustomed to studying natural phenomena, fruitfully brought the tradition of modelling to their investigation of learning. Their approach was in contrast to the focus of the machine learning theorists on 'worst case' guarantees specific to an hypothesis class and that must hold for any data distribution (e.g. Vapnik-Chervonenkis dimension and Rademacher complexity). The originality of their approach, along with the heuristic character of the derivations of mean-field approximations, may nonetheless explain the minor impact of their theoretical contributions in the machine learning community at the time.

Before the deep learning era, mean-field methods probably had a greater influence in the practice of unsupervised learning through density estimation, where we saw that approximations are almost always necessary. In particular the simplest method of naive mean-field, that will also be our first example in Chapter 3, was easily adopted and even extended by statisticians (see e.g. [START_REF] Wainwright | Graphical Models, Exponential Families, and Variational Inference[END_REF] for a recent textbook and [START_REF] Blei | Variational Inference: A Review for Statisticians[END_REF] for a recent review). The belief propagation algorithm is another example of a well known mean-field methods by machine learners -it was actually discovered in both communities. Yet, these applications rarely involved neural networks and rather relied on simple probabilistic models such as mixtures of elementary distributions. They also did not take full advantage of the latest, simultaneous developments in statistical physics of the mean-field theory of disordered systems.

In this context, the inverse Ising problem has been a notable exception. The underlying question, rooted in theoretical statistical physics, is to infer the parameters of an Ising model given a set of equilibrium configurations. This corresponds to the unsupervised learning of the parameters of a Boltzmann machine (without hidden units) in the machine learning jargon. The corresponding Boltzmann distribution, with pairwise interactions, is remarkable, not only to physicists, as it is the least biased model under the assumption of fixed first and second moments; in the sense that it maximizes the entropy. For this problem, physicists lead an efficient cross-fertilization and proposed dedicated developments of advanced mean-field methods for applications in other fields, and in particular in bio-physics (see [START_REF] Nguyen | Inverse statistical problems: from the inverse Ising problem to data science[END_REF] for a recent review). Nevertheless, these works almost never considered the case of Boltzmann machines with hidden units, more common in the machine learning community, especially at the first hours of deep learning.

The language barrier between communities is undoubtedly a significant hurdle delaying the transfer of developments in one field to the other. In machine learning, the potential of the most recent progress of mean-field approximations was already advocated for in a pioneering workshop mixing communities in 1999 [START_REF] Opper | Advanced mean field methods: Theory and practice[END_REF]. Yet their first widely-used application is possibly the Approximate Message Passing (AMP) algorithm for compressed sensing in 2009 [START_REF] Donoho | Message-passing algorithms for compressed sensing[END_REF] -related to the belief propagation algorithm and even more closely to the TAP equations known in the physics of disordered systems. Meanwhile, there have been much tighter connections between developments in statistical physics and algorithmic solutions in the context of Constraint Satisfaction Problems (CSPs). The very first popular application of advanced mean-field methods outside of physics, beyond naive mean-field and belief propagation, is probably the survey propagation algorithm [START_REF] Mézard | Analytic and Algorithmic Solution of Random Satisfiability Problems[END_REF] in 2002. It borrows from the 1RSB cavity method (that will not be mentioned in this dissertation) to solve efficiently certain types of CSPs.

The great leap forward in the performance of machine learning with neural networks brought by deep learning algorithms, along with the multitude of theoretical and practical challenges it has opened, re-ignited the interest of physicists. While the power of advanced mean-field theories, was mostly unexploited in this context at the time this thesis started, this strategy has yielded over the last couple of years a number of contributions that would be difficult to list exhaustively (some of them were recently reviewed in [START_REF] Carleo | Machine learning and the physical sciences[END_REF]). In these works, applying mean-field ideas to deep neural networks has served different objectives and required different simplifying assumptions such as considering an infinite size limit, random (instead of learned) weights, or vanishing learning rates. Hence, there is no such a thing as one mean-field theory of deep neural networks and these contributions are rather complementary pieces of solving a great puzzle. In this thesis in particular, we investigated mean-field insights in models being explicitely trained, either on real or synthetic data sets. To this end we notably exploited some recent progress in the statistical physics treatment of disordered systems. In the following Chapters of this First Part we formally introduce the fundamental frameworks and mean-field tools underlying the research presented in the Second Part.

Statistical inference and statistical physics (Fundamental theoretical frameworks)

To study neural networks we will be led to manipulate high-dimensional probability distributions. Statistical inference consists in extracting useful information from these complicated objects. By adopting a probabilistic interpretation of natural systems composed of many elementary components statistical physics is interested in similar questions. In particular the theory of disordered systems, considering different sources of randomness, appears especially relevant. In this Chapter we introduce these important theoretical frameworks underlying the approach adopted in this thesis.

Statistical inference

Joint probability distributions will be the starting point of the different problems that will interest us in the following, as they arise from the definition of machine learning models or potentially from the choice of a statistical model introduced for the purpose of theory. Inference questions will aim at understanding the properties of these distributions. For instance we may ask what is the expected value of one of the random variable of the model, or whether the realization of a random variable can be recovered from the observations of others.

Note that the practical problem of training neural networks from real data discussed in Chapter 1 does not a priori fit in this perspective as there is a priori no need to assume for a joint probability distribution neither for the training data nor for the learned parameters. However in Section 2.1.2 we will discuss how the learning problem can be cast as inference. Yet it will not be our only concern and we shall consider other inference questions arising naturally from the definition of neural network models.

Statistical models representations

Graphical representations have been developed to represent and efficiently exploit (in)dependencies between random variables encoded in joint probability distributions. In this thesis they will be useful tools to concisely present the model under scrutiny, as well as direct supports for some derivations of inference procedures. Let us briefly present two types of graphical representations.

Probabilistic graphical models

Formally, a probabilistic graphical model is a graph G = (V, E) with nodes V representing random variables and edges E representing direct interactions between random variables. In many statistical models of interest, it is not necessary to keep track of all the possible combinations of realizations of the variables as the joint probability distribution can be broken up into factors involving only subsets of the variables. The structure of the connections E reflects this factorization.

There are two main types of probabilistic graphical models: directed graphical models (or Bayesian networks) and undirected graphical models (or Markov Random Fields). They allow to represent different independencies and factorizations. In the next paragraphs we provide intuitions and review some useful properties of graphical models, a good reference to understand all the facets of this powerful tool is [START_REF] Koller | Probabilistic Graphical Models Principles and Techniques[END_REF].

Undirected graphical models

In undirected graphical models the direct interaction between a subset of variables C ⊂ V is represented by undirected edges interconnecting each pair in C. This fully connected subgraph is called a clique and associated with a real positive potential functions ψ C over the variable x C = {x i } i∈C carried by C. The joint distribution over all the variables carried by V , x V is the normalized product of all potentials Example (i): the Restricted Boltzmann Machine,

p(x V ) = 1 Z C∈C ψ C (x C ). (2.1) t1 t2 pt(tα) ∀α = 1 • • • M x1 x2 x3 px(xi) ∀i = 1 • • • N ψiα(xi,
p(x, t) = 1 Z e x W t p x (x)p t (t) (2.2)
with factorized p x and p t is handily represented unsing an undirected graphical model depicted in

C = {{i}, {α}, {i, α} ; i = 1 • • • N, α = 1 • • • M } , ψ iα (x i , t α ) = e xiWiαtα , ( 2.3)
p(x, t) = 1 Z {i,α}∈C ψ iα (x i , t α ) N i=1 p x (x i ) M α=1 p t (t α ). (2.4)
It belongs to the subclass of pairwise undirected graphical models for which the size of the cliques is at most two.

Undirected graphical models handily encode conditional independencies. Let A, B, S ⊂ V be three disjoint subsets of nodes of G. A and B are said to be independent given S if p(A, B|S) = p(A|S)p(B|S). In the graph representation it corresponds to cases where S separates A and B: there is no path between any node in A and any node in B that is not going through S.

Example (i):

In the RBM, hidden units are independent given the inputs, and conversely:

p(t|x) = M α=1 p(t α |x), p(x|t) = M i=1 p(x i |t). (2.5)
This property is easily spotted by noticing that the graphical model (Figure 2.1a) is bipartite.

Directed graphical model A directed graphical model uses a Directed Acyclic Graph (DAG), specifying directed edges E between the random variables V . It induces an ordering of the random variables and in particular the notion of parent nodes π i ⊂ V of any given vertex i ∈ V : the set of vertices j such that j → i ∈ E. The overall joint probability distribution factorizes as

p(x) = i∈V p(x i |x πi ). (2.6)
Example (ii): The stochastic single layer feed forward network y = g(W x; ), where g(•; ) is a function applied component-wise including a stochastic noise that is equivalent to a conditional distribution p out (y|W x), and where inputs and weights are respectively drawn from distributions p x (x) and p W (W ), has a joint probability distribution

p(y, x, W ) = p out (y|W x)p x (x)p W (W ), (2.7)
precisely following such a factorization. It can be represented with a three-node DAG as in Figure 2.1b. Here we applied the definition at the level of vector/matrix valued random variables. By further assuming that p out , p W and p x factorize over their components, we keep a factorization compatible with a DAG representation

p(y, x, W ) = N i=1 p x (x i ) M µ=1 p out (y µ | N i=1 W µi x i ) µ,i p W (W µi ). (2.8)
For the purpose of reasoning it may be sometimes necessary to get to the finest level of decomposition, while sometimes the coarse grained level is sufficient.

While a statistical physicist may have never been introduced to the formal definitions of graphical models, she inevitably already has drawn a few -for instance when considering the Ising model. She also certainly found them useful to guide physical intuitions. The following second form of graphical representation is probably newer to her.

Factor graph representations

Alternatively, high-dimensional joint distributions can be represented with factor graphs, that are undirected bipartite graphs G = (V, F, E) with two subsets of nodes. The variables nodes V representing the random variables as in the previous section (circles in the representation) and the factor nodes F representing the interactions (squares in the representation) associated with potentials. The edge (iµ) between a variable node i and a factor node µ exists if the variable i participates in the interaction µ. We note ∂i the set of factor nodes in which variable i is involved, they are the neighbors of i in G. Equivalently we note ∂µ the neighbors of factor µ in G, they carry the arguments {x i } i∈∂µ , shortened as x ∂µ , of the potential ψ µ . The overall distributions is recovered in the factorized form:

p(x) = 1 Z M µ=1 ψ µ (x ∂µ ).
(2.9)

Compared to an undirected graphical models, the cliques are represented by the introduction of factor nodes.

Examples: The factor-graph representation of the RBM (i) is not much more informative than the pairwise undirected graphical (see Figure 2.1a). For the feed forward neural networks (ii) we draw the factor graph of p(y, x|W ) (see Figure 2.1b).

Some inference questions in neural networks for machine learning

To fix the ideas, we present what kind of inference questions arising in the context of deep learning will be of particular interest to us.

Inference in generative models

Generative models used for unsupervised learning are precisely statistical models defining highdimensional distributions with complex dependencies. As we have seen in Section 1.1.2, the most common training objective in unsupervised learning is the maximization of the log-likelihood -the log of the probability assigned by the generative model to the training set {x (k) } P k=1 . Computing the probability of observing a given sample x (k) is an inference question. It requires to marginalize over all the hidden representations of the problem. For instance in the RBM (i) assuming binary hidden units,

p(x (k) ) = 1 Z t∈{0,1} M e x (k) W t p x (x (k) )p t (t).
(2.10)

While the numerator will be easy to evaluate, the computation of the partition function is troublesome. Inference in generative models will be discussed in particular in Chapter 4.

Mutual information and reconstruction in stochastic deep neural networks

Recently, it has been proposed to analyse supervised learning with deep feed forward neural networks with information theory [START_REF] Tishby | Deep Learning and the Information Bottleneck Principle[END_REF][START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF]. A quantity of particular interest in this setting is the mutual information between the input x and output y of a neural network for a given value of the weights. The mutual information is a functional of the joint probability distribution: .11) It measures the dependence between two random variables. Expressed as a KL-divergence it is zero if and only if p(x, y) = p(x)p(y), so if the random variables are independent. Otherwise it is always positive. In fact, the mutual information between input and output even diverges to +∞ for a deterministic map. We shall consider networks with stochastic activations to ensure that the mutual information I(x, y) is well-defined.

I(x, y) = KL(p(x, y)||p(x)p(y)) = dx dy p(x, y) log p(x, y) p(x)p(y) . ( 2
A related inference question is whether from the observed output y of a feed forward neural network, the input x at its origin can be reconstructed. If the two variables are largely independent, due for instance to a high level of noise, their mutual information is low while the reconstruction is impossible. In our toy example (ii), the posterior distribution p(x|W , y) is deriving from the statistical model following Bayes rule p(x|W , y) = p(y|W , x)p x (x) p(y|W ) = p(y|W , x)p x (x) dx p(y|W , x)p x (x) .

(2.12)

In this context p x (x) is usually called the prior, it biases the reconstruction towards values compatible with the knowledge on x provided by the statistical model before any observation was considered. The difficulty in the corresponding inference problem is the marginalization necessary to compute the denominator p(y|W ), sometimes called the evidence. These questions will be the main concern of Chapter 5.

Learning as statistical inference: teacher-student scenario

Assuming a statistical model for the data and the trainable parameters, the learning problem can be formulated as an inference in the teacher-student setting. Let us use as an illustration the case of the supervised learning of the single-layer model (ii). We draw a weight matrix W 0 -the teacher -from the prior distribution p W , along with a set of P inputs {x (k) } P k=1 i.i.d from p x , and the corresponding outputs y (k) from p out (.|W 0 x (k) ). The inference question is: can we recover the value W 0 from the observations of the P pairs {x (k) , y (k) }? In other words, can we learn the parameters W of a similar single-layer model -the student -using the observations as a training set so as to reproduce the ground-truth rule defined by W 0 ?

Note that the terminology applies for a generic inference problem of reconstruction: the statistical model used to generate the data along with the realization of the unknown signal is called the teacher; the statistical model assumed to perform the reconstruction is called the student. They may be identical as in the example above, in which case they are said to be matched. But they can also be different: a bit (e.g. wrong assumptions of prior distributions) or entirely (e.g. different architecture of graphical model), in which case they are said to be mismatched.

Of course in practical applications, data do not follow a known distribution and there is no mathematical ground truth rule underlying the input-output mapping. Yet the teacher-student setting offers interesting possibilities of analysis and we shall resort to it in Chapter 5 and Chapter 6.

Challenges in inference

Handling dimensions Interesting inference questions in the line of the ones mentioned in the previous section have no tractable closed-form solution. Exact computations of averages and marginals requires summing over a number of terms scaling exponentially with the size of the system. Hence the computational hardness comes from the astronomically large number of configurations to be tracked to describe a high-dimensional joint probability distribution. It is therefore necessary for inference to design strategies of approximations that can be implemented as algorithms running in finite, typically polynomial, time.

Finding good estimators While some inference questions are clearly posed -as for instance the computation of the likelihood of the RBM -some can accept multiple answers -in particular reconstruction problems. Let us consider the following setting: we are interested in recovering an unknown signal x 0 from the knowledge of the prior distribution p x , downstream observations y and the channel p(y|x) also called the likelihood in this context. The distribution of x knowing y, called the posterior distribution, is given by Bayes rule

p(x|y) = p(y|x)p x (x) p(y) . ( 2

.13)

As soon as x has more than a few components this distribution is difficult to handle. Instead of keeping all the information it encodes, it is preferable to build directly an estimator x of the unknown signal. We can either choose

• the maximum a posteriori estimator xMAP = arg max x p(x|y). It corresponds to the most probable value of x 0 given the prior and the observations. It minimizes the probability of errors defined as xi = x 0,i .

• the minimum mean-square error estimator xMMSE = dx x p(x|y). It is the mean of the posterior distribution and it minimizes the mean-square error under the posterior distribution MSE(x) = dx p(x|y) x -x 2 .

• the maximum likelihood estimator xMLE = arg max x p(y|x). It discards the information of the prior.

The MAP and MMSE estimators corresponds to the Bayesian inference setting incorporating information about a prior. We have implicitly assumed above that the prior and the channel are perfectly known, i.e. a matched teacher-student scenario. This situation is also called the Bayes optimal setting. It is of particular interest to assess theoretical optimal performances of reconstruction. Still, Bayesian inference can also be used when only guesses of the prior and the channel are available. For instance a sparse analytical prior can be assumed for the reconstruction of real sparse signals. In the case where no prior on the data is known, the maximum likelihood estimator can be preferred to the arbitrariness of choosing a prior. This is the strategy of most unsupervised learning algorithm to find good parameters. One or the other of these estimators can be preferable according to the situation at hand.

Performance analysis in estimation

The perspective of statistical inference and the teacherstudent scenario allow to formalize an analysis of the difficulty of a learning problem. Intuitively, the definition of a ground-truth rule with a fixed number of degrees of freedom sets the minimum information necessary to extract from the observations, or training data, in order to achieve perfect reconstruction. This is an information-theoretic limit. The assumption of an underlying statistical model further enables the measurement of performance of different learning algorithms over the class of corresponding problems from an average view point. The traditional approach of computer science in studying the difficulty of a class of problem is to examine the worst case. This conservative strategy yields strong guarantees of success, yet it may be overly pessimistic compared to the experience of practitioners. By defining a distribution over the possible problems (or teachers), one can compute instead averaged performances, sometimes more informative of behaviors for typical instances. For deep learning, this approach may prove particularly interesting as the traditional bounds appear extremely loose when compared to practical examples.

Statistical physics of disordered systems

To solve and study inference questions we will resort to concepts and methods originally developed by physicists. In a nutshell we remind some key concepts in statistical physics.

Boltzmann distribution and free energy

Statistical physics aims at explaining the behaviors of systems composed of a large number of degrees of freedom. Although the laws of classical physics governing microscopic interactions are deterministic, it is unrealistic to keep track of the evolution of each of the elementary component to account for the behavior of the whole. Remarkably, macroscopic quantities of interest to the physicist are well described by an averaging over the elementary states according to the Boltzmann distribution. For a system with N degrees of freedom noted x ∈ X N and an energy functional E(x), we have

p(x) = e -βE(x) Z N , Z N = x∈X N e -βE(x) , β = 1/k B T, ( 2.14)
where we defined the partition function Z. While the adoption of the probabilistic framework is already a great simplification to the complete microscopic description, it takes us back to the difficult task of handling high-dimensional joint probability distribution discussed in Section 2.1.

Let us discuss key conceptual steps further taken by physicists to understand large interacting systems.

To characterize macroscopic systems careful attention should be given to the free energy

F N = -log Z N /β = - 1 β log x∈X N e -βE(x) (2.15) = - 1 β    x∈X N -βE(x)e -βE(x) /Z N + x∈X N βE(x) + log Z N e -βE(x) /Z N    (2.16) = E(x) + log p(x) /β = U N -H N /β (2.17)
where • stands for the average over the Boltzmann distribution, U N is the average energy and H N is the entropy. Note that the entropy of statistical physics and the entropy of information theory [START_REF] Shannon | A mathematical theory of communication[END_REF] have the same definition relatively to a distribution.

The thermodynamic limit

The number of available configurations X N grows exponentially with N , yet considering the thermodynamic limit N → ∞ can simplify computations due to concentrations. Let e N = E/N be the energy per degree of freedom, Ω(E) = x∈X N δ E -E(x) = e N h N be the number of configurations at energy E = N e N , and h N (e N ) the entropy per degree of freedom. Then the partition function can be re-written as a sum over the configurations of a given energy e N

Z N = E x∈X N δ E -E(x) e -N βe N = e N e -N (βe N -h N (e N )) = e N e -N βf N (e N ) , (2.18)
where we define f N (e N ) the free energy density of states of energy e N . This rewriting implies that at large N the states of energy minimizing the free energy are exponentially more likely than any other states. In the thermodynamic limit, the statistics are dominated by these states. Using the definitions of the limits

e = lim N →∞ e N , h(e) = lim N →∞ h N (e N ), βf (e) = lim N →∞ βe N -h N (e N ), (2.19)
we have

f = min e f (e) = lim N →∞ F N /N, ( 2.20)
Z = lim N →∞ Z N = e -βf .
(2.21)

Statistical physics of disordered systems

The existence of the previous limits is the condition to the application of the thermodynamic limit.

Disordered systems

To consider the thermodynamic limit, the reader might have imagined a translationally invariant system, readily extendable. Remarkably, the statistical physics framework can be applied more generally to systems with quenched disorder. Here the interactions in the system are function of the realization of random variables so that the energy functional is itself function of a global random variable, noted W ∈ W for instance. The energy of a state of the system x is then E(x; W ). In principle the system properties depend on a given realization of the disorder W -quite certainly the average value x i x j W for instance. Yet some aggregated properties are expected to be self-averaging in the thermodynamic limit, meaning that they concentrate on their mean with respect to the disorder W as the fluctuations are averaged out. It is the case of the free energy, which formally verifies:

lim N →∞ F N ;W /N = lim N →∞ E W [F N ;W /N ] = f. (2.22)
Thus the typical behavior of an ensemble of complicated systems can be studied in the statistical physics framework by averaging over the disorder.

Statistical physics of inference problems

Statistical inference questions are mapped to statistical physics systems by interpreting general joint probability distributions as Boltzmann distributions (2.14). Turning back to our simple inference examples:

(i) The RBM directly borrows its definition from statistical physics with

E(x, t; W ) = -x W t -log p x (x) -log p t (t). (2.23)
The inverse temperature parameter can either be considered equal to 1 or as a scaling factor of the weight matrix W ← βW and log-priors. The weights play the role of the disorder.

Here the computational hardness in estimating the log-likelihood comes from the estimation of the log-partition function, which is precisely the free energy.

(ii) The posterior in the estimation of the input from the output of a network is mapped to a Boltzmann distribution by setting E(x; y) = -log p(y|x)p x (x). The disorder is here materialized by the observations.

In this Section we introduced the key conceptual preliminary steps taken by physicists to handle high-dimensional joint distributions. These are the starting point to design approximate inference methods. Before turning to an introduction to mean-field approximations, we stress the originality of the statistical physics approach. Relying on the thermodynamic limit, these methods will provide asymptotic results. Nevertheless, experience shows that the behavior of large finite-size systems are often well explained by the infinite-size limit. Furthermore, the disorder average transposed in the language of statistical inference will equate to studying the typical rather than worst case scenario, with the advantages and limitations discussed in 2.1.3. Finally, we must emphasize that previous and following derivations are not mathematically rigorous. They are based on 'correct' assumptions allowing to push further the understanding of the problems at hand, while a formal proof of the assumptions is possibly much harder to obtain.

Selected overview of mean-field treatments:

Free energies and algorithms

(Techniques)
Mean-field methods are a set of techniques enabling to approximate marginalized quantities of joint probability distributions by exploiting knowledge on the dependencies between random variables. They are usually said to be analytical -as opposed to numerical Monte Carlo methods. In practice they usually replace a summation exponentially large in the size of the system by an analytical formula involving a set of parameters, themselves solution of a closed set of non-linear equations.

Finding the values of these parameters typically requires only a polynomial number of operations.

In this Chapter, we will give a selected overview of mean-field methods as they were introduced in the statistical physics and/or signal processing literature. A key take away of the following sections is that closely related results can be obtained from different heuristics of derivation. We will start by deriving the simplest and historically first mean-field method. We will then introduce the important broad techniques that are used in the following Chapters, high-temperature expansions, message-passing algorithms and the replica method. In a closing section we will cover the most recent extensions of mean-field methods that were stepping stones for the contributions of this thesis.

Naive mean-field

Variational derivation

The naive mean-field method consists in approximating the joint probability distribution of interest by a fully factorized distribution. Therefore, it ignores correlations between random variables. Among multiple methods of derivation, we present here the variational method: it is the best known methods across fields and it readily shows that, for any joint probability distribution interpreted as a Boltzmann distribution, the rather crude naive mean-field approximation yields an upper bound on the free energy. For the purpose of demonstration we consider an Ising model with spins variables x = (x 1 , • • • , x N ) ∈ X = {0, 1} N , and energy function

E(x) = - N i=1 b i x i - (ij) W ij x i x j = -b x - 1 2 x W x , b ∈ R N , W ∈ R N ×N , ( 3.1)
where the notation (ij) stands for pairs of connected spin-variables, and the weight matrix W is symmetric. The choices for {0, 1} rather than {-1, +1} for the variable values, the notations W for weights (instead of couplings), b for biases (instead of local fields), as well as the vector notation, are leaning towards the machine learning conventions and the interpretation of the model as a Boltzmann machine (without hidden units). We denote by q m a fully factorized distribution on {0, 1} N , which is a multivariate Bernoulli distribution parametrized by the mean values m

= (m 1 , • • • , m N ) ∈ [0, 1] N of
the marginals (denoted by q mi ):

q m (x) = N i=1 q mi (x i ) = N i=1 m i δ(x i -1) + (1 -m i )δ(x i ). (3.2)
We look for the optimal q m distribution to approximate the Boltzmann distribution p(x) = e -βE(x) /Z by minimizing the KL-divergence

min m KL(q m ||p) = min m x∈X q m (x) log q m (x) p(x) (3.3) = min m x∈X q m (x) log q m (x) + β x∈X q m (x)(E(x)) + log Z (3.4) = min m βG(q m ) -βF ≥ 0, (3.5)
where the last inequality comes from the positivity of the KL-divergence. For a generic distribution q, G(q) is the Gibbs free energy for the energy E(x),

G(q) = x∈X q(x)(E(x)) + 1 β x∈X q(x) log q(x) = U (q) -H(q)/β ≥ F, ( 3.6)
involving the average energy U (q) and the entropy H(q). It is greater than the true free energy F except when q = p, in which case they are equal. Note that this fact also means that the Boltzmann distribution minimizes the Gibbs free energy. Restricting to factorized q m distributions, we obtain the naive mean-field approximations for the mean value of the variables (or magnetizations) and the free energy:

m * = arg min m G(m) = x q m * (3.7
)

F NMF = G(m * ) ≥ F. (3.8)
The choice of a very simple family of distributions q m limits the quality of the approximation but allows for tractable computations of other observables, for instance the two-spins correlations

x i x j q * = m * i m * j or variance of one spin x 2 i q * -x i 2 q * = m * i -m * i 2 .
In our example of the Boltzmann machine it is easy to compute the Gibbs free energy for the factorized ansatz,

U NMF (m) = E(x) qm = -b m - 1 2 m W m , (3.9)
H NMF (m) = -log q(x) qm = - N i=1 m i log m i + (1 -m i ) log(1 -m i ) , (3.10)
G NMF (m) = U NMF (m) -H NMF (m)/β. (3.11)
Looking for stationary points we find a closed set of non linear equations for the m * i ,

∂G ∂m i m * = 0 ⇒ m * i = σ(βb i + j∈∂i βW ij m * j ) ∀i , ( 3.12)
where σ(x) = (1 + e -x ) -1 . The solutions can be computed by iterating these relations until a fixed point is reached. To understand the implication of the restriction to factorized distributions, it is instructive to compare this naive mean-field equation with the exact identity

x i p = σ(βb i + j∈∂i βW ij x j ) p . (3.13)
Under the Boltzmann distribution p(x) = e -βE(x) /Z, these averages are difficult to compute. The naive mean-field method is neglecting the fluctuations of the effective field felt by the variable x i : j∈∂i W ij x j , keeping only its mean j∈∂i W ij m j . This incidentally justifies the name of mean-field methods.

When does naive mean-field hold true?

The previous derivation shows that the naive mean-field approximation allows to bound the important free energy. While this bound is expected to be rough in general, the approximation is reliable when the fluctuations of the local effective fields j∈∂i W ij x j are small. This happens in particular at the thermodynamic limit N → ∞ in infinite range models, that is when weights or couplings are not only local but distributed in the entire system, or if each variable interacts directly with a non-vanishing fraction of the whole set of variables. The influence of the rest of the system can then be treated as an average background. Provided the couplings are weak enough, the naive mean-field methods even becomes asymptotically exact. This is the case of the Curie-Weiss model, which is the fully connected version of the model (3.1) with all W ij = 1/N . The sum of weakly dependent variables then concentrates on its mean by the central limit theorem. We

Thouless Anderson and Palmer equations

stress that it means that for finite dimensional models (more representative of a physical system, where for instance variables are assumed to be attached to the vertices of a lattice with nearest neighbors interactions), mean-field methods are expected to be quite poor. By contrast, infinite range models are thus traditionally called mean-field models by physicists.

In the next Section we will recover the naive mean-field equations through a different method.

The following derivation will also allow to compute corrections to the rather crude approximation we just discussed by taking into account some of the correlations it neglects.

Thouless Anderson and Palmer equations

The TAP mean-field equations [START_REF] Thouless | Solution of 'Solvable model of a spin glass[END_REF] were originally derived as an exact mean-field theory for the Sherrington-Kirkpatrick (SK) model [START_REF] Sherrington | Solvable Model of a Spin-Glass[END_REF]. This emblematic spin glass model corresponds to a fully connected Ising model with energy (3.1) and disordered couplings W ij drawn independently from a Gaussian distribution with zero mean and variance W 0 /N . The derivation followed from arguments specific to the SK model. Later, it was showed that the approximation could be recovered from a second order Taylor expansion at high temperature by Plefka [START_REF] Plefka | Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model[END_REF] and that it could be further corrected by the systematic computation of higher orders by Georges and Yedidia [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF]. We will briefly present this last derivation, having again in mind the example of the generic Boltzmann machine (3.1).

Outline of the derivation

Going back to the variational formulation (3.5), we now perform a minimization in two steps. We start by considering the family of distributions q m enforcing x qm = m for a fixed vector of magnetizations m but without any factorization constraint. We consider the corresponding Gibbs free energy G(q m ) = U (q m ) -H(q m )/β, (3.14) and perform a first minimization at fixed m over the q m to define another auxiliary free energy

G(m) = min qm G(q m ). (3.15)
A second minimization over m would recover the overall unconstrained minimum of the variational problem (3.5) which is the exact free energy

F = -log Z/β = min m G(m). (3.16)
Yet the actual value of G(m) turns out as complicated to compute as F itself. Fortunately, βG(m) can be easily approximated by a Taylor expansion around β = 0 due to interactions vanishing at high temperature. After expanding, the minimization over G(m) yields a set a self consistent equations on the magnetizations m -called the TAP equations -reminiscent of the naive mean-field equations (3.12), again typically solved by iterations. Plugging the solutions m * back into the expanded expression yields the TAP free energy F TAP = G(m * ). Note that ultimately the approximation lies in the truncation of the expansion. At first order the naive mean-field approximation is recovered. At second order, this derivation justifies the original TAP approximation for SK model [START_REF] Thouless | Solution of 'Solvable model of a spin glass[END_REF]. In Section 4.1 of the next Chapter, we detail the different steps here outlined for a generalization of the Boltzmann machine (3.1).

Illustration on the Boltzmann machine and important remarks

For the Boltzmann machine (3.1), the TAP equations and TAP free energy (truncated at second order) are [START_REF] Thouless | Solution of 'Solvable model of a spin glass[END_REF],

m * i = σ   βb i + j∈∂i βW ij m * j -β 2 W 2 ij (m * j - 1 2 )(m * i -m * i 2 )   ∀i (3.17) βG(m * ) = -H NMF (m * ) -β N i=1 b i m * i -β (ij) m * i W ij m j (3.18) - β 2 2 (ij) W 2 ij (m * i -m * i 2 )(m j -m * j 2 ) ,
where naive mean-field entropy was defined in (3.10). For this model, albeit with {+1, -1} variables instead of {0, 1}, several references present the details of the derivation sketched in the previous paragraph, in particular [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF][START_REF] Opper | Advanced mean field methods: Theory and practice[END_REF][START_REF] Zamponi | Mean field theory of spin glasses[END_REF].

Onsager reaction term Compared to the naive mean-field approximation the TAP equations include a correction to the effective field called the Onsager reaction term. The idea is that, in the effective field at variable i, we should consider corrected magnetizations of neighboring spins j ∈ ∂i, that would correspond to the absence of variable i. This intuition echoes at two other derivations of the TAP approximation: the cavity method [90] that we will not cover and the message passing which will be discussed in the next section.

As far as the SK model is concerned, this second order correction is enough in the thermodynamic limit as the statistics of the weights imply that higher orders will typically be subleading. In general, the correct TAP equations for a given model will depend on the statistics of interactions and there is no guarantee that there exists an order of truncation leading to an exact mean-field theory.

Single instance

Although the selection of the correct TAP approximation relies on the statistics of the weights, the derivation of the expansion outlined above does not require to average over them, i.e. it does not require an average over the disorder. Consequently, the approximation method is well defined for a single instance of the random disordered model. The TAP free energy and magnetizations can be computed for a given (realization of the) set of weights {W ij } (ij) . Note that this fact further allowed the designing of the adaptive TAP approximation [START_REF] Opper | Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling[END_REF][START_REF] Opper | Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach[END_REF][START_REF] Opper | Expectation consistent free energies for approximate inference[END_REF], that we will discuss in Section 3.5.3, requiring no prior knowledge on the statistics of the weights. The Onsager correction is here computed directly for a given W .

Finding solutions

The self-consistent equations on the magnetizations (3.17) are usually solved by turning them into an iteration scheme and looking for fixed points. This generic recipe leaves nonetheless room for interpretation: which exact form should be iterated? How should the updates for the different equations be scheduled? Which time indexing should be used? While the following scheme may seem natural,

m i (t+1) ← σ   βb i + j∈∂i βW ij m j (t) -W 2 ij m j (t) - 1 2 m i (t) -m i (t) 2   (3.19)
it typically has more convergence issues than the following alternative scheme including the time index t -1,

m i (t+1) ← σ   βb i + j∈∂i βW ij m j (t) -W 2 ij m j (t) - 1 2 m i (t-1) -m i (t-1) 2   . (3.20)
This issue was first discussed by [START_REF] Bolthausen | An iterative construction of solutions of the tap equations for the Sherrington-Kirkpatrick model[END_REF]. Remarkably, this last scheme, or algorithm, is actually the one obtained by the approximate message passing derivation that will be discussed in the next Section.

Solutions of the TAP equations

The TAP equations can admit multiple solutions with either identical or different TAP free energy. While the true free energy F corresponds to the minimum of the Gibbs free energy, reached for the Boltzmann distribution, the TAP derivation consists in performing an effectively unconstrained minimization in two steps, but with an approximation through a Taylor expansion in between. The truncation of the expansion therefore breaks the correspondence between the discovered minimizer and the unique Boltzmann distribution. For the SK model for instance, the number of solutions of the TAP equations increases rapidly as β grows [START_REF] Mézard | Spin Glass Theory and Beyond[END_REF]. The different solutions must be weighted according to their free energy density and averaged to recover the thermodynamics predicted by the replica computation [START_REF] De | Weighted averages and order parameters for the infinite range Ising spin glass[END_REF], another mean-field approximation discussed in Section 3.4.

Generalizing the Georges-Yedidia expansion

In the derivation outlined above for binary variables, x i = 0 or 1, the mean of each variable m i was fixed. This is enough to parametrize the corresponding marginal distribution q mi (x i ). Yet the expansion can actually be generalized to Potts variables (taking multiple discrete values) or real variables by introducing appropriate parameters for the marginals. For real variables, another level of approximation is introduced by restricting the set of marginal distributions tested to a parametrized family of distributions. By choosing a Gaussian parametrization, one recovers TAP equations equivalent to the approximate message passing algorithm that will be discussed in the next Section. In Section 4.1, we will present a derivation for real-valued Boltzmann machines with a Gaussian parametrization.

Belief propagation and approximate message passing

Another route to rediscover the TAP equations is through the approximation of message passing algorithms. Variations of the latter were discovered multiple times in different fields. In physics they were written in a restricted version as soon as 1935 by Bethe [START_REF] Bethe | Statistical Theory of Superlattices[END_REF]. In statistics, they were developed by Pearl as methods for probabilistic inference [START_REF] Pearl | Probabilistic Reasoning in Intelligent Systems[END_REF]. In this section we will start by introducing a case-study of interest, the Generalized Linear Model. We will then proceed by steps to outline the derivation of the Approximate Message Passing (AMP) algorithm from the Belief Propagation (BP) equations.

Generalized linear model

Definition We introduce the Generalized Linear Model (GLM) which is a fairly simple model to illustrate message passing algorithms and which is also an elementary brick of the inference questions that will interest us in the following. It falls under the teacher-student set up, a student model is used to reconstruct a signal from a teacher model producing indirect observations. In the GLM, the product of an unknown signal x 0 ∈ R N and a known weight matrix W ∈ R N ×M is observed through a noisy channel p out,0 ,

     W ∼ p W (W ) x 0 ∼ p x0 (x 0 ) = N i=1 p x0 (x 0,i ) ⇒ y ∼ p out,0 (y|W x 0 ) = M µ=1 p out,0 (y µ |w µ x 0 ). (3.21)
The probabilistic graphical model corresponding to this teacher is represented on Figure 3.1. The prior over the signal p x0 is supposed to be factorized, and the channel p out,0 likewise. The inference problem is to produce an estimator x for the unknown signal x 0 from the observations y. Given the prior p x and the channel p out of the student not necessarily matching the teacher, the posterior distribution is

p(x|y, W ) = 1 Z(y, W ) p out (y|x, W )p x (x) , Z(y, W ) = dx p(y|x, W )p x (x), (3.22)
represented as a factor graph also on Figure 3.1. The difficulty of the task is controlled by the measurement ratio α = M/N and the amplitude of the noise possibly present in the channel.

(Techniques)

y1 y2 x0,1 x0,2 x0,3 x0,4 pout,0(yµ|w µ X 0 ) ∀µ = 1 • • • M p x 0 (x 0 i ) ∀i = 1 • • • N Teacher x1 x2 x3 x4 pout(yµ|w µ X) ∀µ = 1 • • • M px(xi) ∀i = 1 • • • N Student x1 x2 x3 x4 m (t) i →µ (xi ) m(t) µ→i (xi) x1 x2 x3 x4 m (t) i→µ (xi) m(t+1) µ→i (xi)

Applications

The generic GLM underlies a number of applications. In the context of neural networks of particular interest in this thesis, the channel p out generating observations y ∈ R M can equivalently be seen as a stochastic activation function g(•, ) incorporating a noise ∈ R M component-wise to the output. The inference of the teacher signal in a GLM has then two possible interpretations. On the one hand, it can be interpreted as the reconstruction of the input of a stochastic single-layer neural network from its output. On the other hand, the same question can also corresponds to the Bayesian learning of a single-layer neural network with a single outputthe perceptron -where this time {W , y} are interpreted as the collection of training input-output pairs and x 0 plays the role of the unknown weight vector of the teacher. However, one of the most important application of the GLM, Compressed Sensing (CS) [START_REF] Donoho | Compressed sensing[END_REF], does not involve neural networks. It consist in recovering a ρ-sparse signal x 0 , i.e. with ρN (< N ) non zero entries, from M noisy linear measurements gathered in y. At zero noise, the information theoretic limit for the reconstruction is M > ρN , or equivalently α > ρ. Yet at this threshold the reconstruction requires the knowledge of the position of the non-zeros entries, or an exponential number of operations to test all possibilities. Hence the problem is non-trivial as soon as M < N . Studying the performance of algorithms in between the impossible and trivial regime is of primal practical and theoretical interest.

Statistical physics treatment and scalings

From the statistical physics perspective, the effective energy functional is read from the posterior (3.22) seen as a Boltzmann distribution with energy

E(x) = -log p out (y|x, W )p x (x) = - M µ=1 log p out (y µ | N i=1 W µi x i ) - N i=1 p x (x i ).
(3.23)

The inverse temperature β has here no formal equivalent and can be thought as being equal to 1.

The energy is therefore a function of the random realizations of W and y, playing here the role of the disorder. The weight matrix is assumed to have i.i.d. Gaussian entries with zero mean and variance 1/N . The prior of the signal is chosen so as to ensure that the x i (and consequently the y µ ) remain of order 1. Finally, the thermodynamic limit N → ∞ is taken for a fixed measurement ratio α = M/N .

Belief Propagation

The Belief Propagation algorithm is an exact inference algorithm for joint probability distributions with acyclic (or tree) underlying factor graph. It allows to compute marginals of the random variables using a set of auxiliary functions called messages, attached to the edges of the factor graph. Starting at a leaf, the messages communicate beliefs of a given node variable taking a given value based on the nodes and factors already visited along the tree.

General formulation For a generic joint probability distribution, BP can be written considering the underlying factor graph. Using notations from Section 2.1.1, the sum-product version of BP (as opposed to the max-sum, see e.g. [START_REF] Mézard | Information, Physics, and Computation[END_REF]) consists in the update equations

m(t) µ→i (x i ) = 1 Z µ→i i ∈∂µ\i dx i ψ µ (x ∂µ ) i ∈∂µ\i m (t) i →µ (x i ), (3.24) m (t+1) i→µ (x i ) = 1 Z i→µ p x (x i ) µ ∈∂i\µ m(t) µ →i (x i ) (3.25)
where the i-s index the variable nodes, the µ-s index the factor nodes and the ψ µ are the potential functions. The notation ∂µ \ i designate the set of neighbor variables of the factor µ except the variable i (and reciprocally for ∂i \ µ ). The partition functions Z i→µ and Z µ→i are normalization factors ensuring that the messages can be interpreted as probabilities. At convergence of the iterations the posterior marginals can be computed as

m i (x i ) = 1 Z i p x (x i ) µ∈∂i mµ→i (x i ) (3.26)
and the Bethe approximation of the free energy is given by

F Bethe = - i∈V log Z i - µ∈F log Z µ + (iµ)∈E log Z µi , ( 3.27)
with

Z i = dx i p x (x i ) µ∂i mµ→i (x i ) , (3.28)
Z µ = i∈∂µ dx i ψ(x ∂µ ) i∈∂µ m i→µ (x i ) , ( 3.29)
Z µi = dx i mµ→i (x i )m i→µ (x i ) . (3.30)
These marginals, as well as the Bethe free energy, will only be exact if the underlying factor graph is a tree. Nonetheless, the algorithm (3.24)-(3.25), occasionally then called loopy-BP, can sometimes be converged on graphs with cycles and in some cases will provide high quality approximations. For instance, graphs with no short loops are locally tree like and well treated by BP provided correlations decay with distance. BP will also appear principled for some infinite range mean-field models previously discussed; an example of which being our case-study the GLM. Note that for graphs with cycles the Bethe free energy is a priori not a bound of the exact free energy.

Belief propagation for the GLM The writing of the BP-equations for our case-study is schematized on the right of Figure 3.1. There are N × M updates:

m(t) µ→i (x i ) = 1 Z µ→i i =i dx i p out (y µ |w µ x) i =i m (t) i →µ (x i ), (3.31) m (t+1) i→µ (x i ) = 1 Z i→µ p x (x i ) µ =µ m(t) µ →i (x i ). (3.32)
Despite a relatively concise formulation, running BP in practice turns out intractable since for a signal x taking continuous values it would entail keeping track of distributions on continuous variables. In this case, BP is approximated by the (G)AMP algorithm presented in the next section.

(Generalized) approximate message passing

The name of approximate message passing (AMP) was fixed by Donoho, Maleki and Montanari [START_REF] Donoho | Message-passing algorithms for compressed sensing[END_REF] who derived the algorithm in the context of Compressed Sensing. Several works from statistical physics had nevertheless already proposed related algorithmic procedures and made connections with the TAP equations for different systems. The algorithm was derived systematically for any channel of the GLM by Rangan [START_REF] Rangan | Generalized Approximate Message Passing for Estimation with Random Linear Mixing[END_REF] and became Generalized-AMP (GAMP).

The systematic procedure to write AMP for a given joint probability distribution consists in first writing BP on the factor graph, second project the messages on a parametrized family of function to obtain the corresponding relaxed-BP and third close the equations on a reduced set of parameters by keeping only leading terms in the thermodynamic limit. We will quickly review these steps for the GLM.

Relaxed Belief Propagation

In the thermodynamic limit M, N → +∞, on can show that the scaling 1/ √ N of the W ij and the extensive connectivity of the underlying factor graph imply that messages are approximately Gaussian. The algorithm then only needs to keep track of the two first moments. In the derivation of r-BP different means and variances are in fact introduced to parametrize messages as follows

m(t) µ→i (x i ) ∝ e B (t) µ→i xi+ 1 2 A (t) µ→i x 2 i ∝ dz µ p out (y µ |z µ )e - (zµ-W µi x i -ω (t) µ→i ) 2 2V (t) µ→i , (3.33) m (t+1) i→µ (x i ) ∝ e - (x (t+1) i→µ -x i ) 2 2C x (t+1) i→µ ∝ p x (x i )e - (λ (t) i→µ -x i ) 2 2σ (t) i→µ , ( 3.34)
where we introduced the dummy variable z = W x, an important intermediate step in the reconstruction of x from y, and the notation ∝ that omits the normalization factor for distributions.

The BP algorithm projected on this parametrization makes up the r-BP inference algorithm with O(M × N ) quantities to track. An approximation of the marginals is recovered from the projection on the parametrization of (3.26). A detailed derivation and developed algorithm of r-BP for the GLM can be found for example in [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF]. Also, in Chapter 6 Section 6.1.1, we review the derivation in a slightly more general setting where the variables x i and y µ are possibly vectors instead of scalars. Nonetheless, r-BP is scarcely used as such as the computational cost can be readily reduced with little more approximation. In the thermodynamic limit, the messages are closely related to the marginals as the contribution of the missing message between (3.25) and (3.26) is to a certain extent negligible. Careful book keeping of the order of contributions leads to a set of closed equations on parameters of the marginals, i.e. O(N ) variables, corresponding to the (G)AMP algorithm.

Approximate message passing

The GAMP algorithm with respect to marginals parameters, analogous to the messages parameters introduced in (3.33)-(3.34), is given in Algorithm (1). While steps 1) and 3) are general for any GLM with a random Gaussian weight matrix, steps 2) and 4) involve update functions. These special functions are defined respectively to choices of a prior on the signal x and an output channel producing the observations y.

Input update functions relative to the prior

Z x = dx p x (x)e -(x-λ) 2 2σ (3.35) f x 1 (λ, σ) = 1 Z x dx x p x (x)e -(x-λ) 2 2σ (3.36) f x 2 (λ, σ) = 1 Z x dx x 2 p x (x)e -(x-λ) 2 2σ -f x 1 (λ, σ) 2 (3.37)
Output update functions relative to the channel

Z out (y, ω, V ) = dz p out (y|z)N (z; ω, V ) (3.38) g out (y, ω, V ) = 1 Z out dz (z -ω) V p out (y|z)N (z; ω, V ) (3.39) ∂ ω g out (y, ω, V ) = 1 Z out dz (z -ω) 2 V 2 p out (y|z)N (z; ω, V ) - 1 V -g out (y, ω, V ) 2 (3.40)
Algorithm 1 Approximate Message Passing

Input: matrix y ∈ R M and matrix W ∈ R M ×N : Initialize: xi , C x i
∀i and g outµ , ∂ ω g outµ ∀µ repeat 1) Estimate mean and variance of z given current x

V (t) µ = N i=1 W 2 µi C x i (t) (3.41
)

ω (t) µ = N i=1 W µi x(t) i - N i=1 W 2 µi C x (t) i g out (t-1) µ (3.42)
2) Estimate mean and variance of the gap between optimal z and ω given y

∂ ω g out (t) µ = ∂ ω g out (y µ , ω (t) µ , V (t) µ ) (3.43) g out (t) µ = g out (y µ , ω (t) µ , V (t) µ ) (3.44)
3) Estimate mean and variance of x given current optimal z σ

(t) i =   - M µ=1 W 2 µi ∂ ω g out (t) µ   -1 (3.45) λ (t) i = x(t) i + σ (t) i   M µ=1 W µi g out (t) µ   (3.46)
4) Estimate of mean and variance of x augmented of the information about the prior

C x i (t+1) = f x 2 (λ (t) i , σ (t) i ) (3.47) x(t+1) i = f x 1 (λ (t) i , σ (t) i ) (3.48) until convergence
Relation to TAP equations Historically the main difference between the AMP algorithm and the TAP equations is that the latter was first derived for binary variables with 2-body interactions while the former was proposed for continuous random variables with N -body interactions. The details of the derivation, not described here, rely on the knowledge of the statistics of the disordered variable W but do not require a disorder average, as in the Georges-Yedidia expansion yielding the TAP equation. By focusing on the GLM with a random Gaussian weight matrix scaling as O(1/ √ N ) (similar to the couplings of the SK model) we naturally obtained TAP equations at second order, with an Onsager term in the update (3.42) of ω µ . The TAP free energy can also be recovered by plugging-in the approximate parametrized messages in the definition of the Bethe free energy (3.27). Yet an advantage of the AMP derivation from BP is that it explicitly provides 'correct' time indices in the iteration scheme to solve the self consistent equations.

Reconstruction with AMP

The AMP algorithm is therefore a practical reconstruction algorithm which can be run on a single instance and estimate an unknown signal x 0 . Note that the prior p x and channel p out used in the algorithm correspond to the student statistical model and they may be different from the true underlying teacher model that generates x 0 and y. In other words, the AMP algorithm may be used either in the Bayes Optimal or in the mismatched setting defined in Section 2.1.3. Remarkably, it is also possible to now consider a disorder average in the thermodynamic limit to study the average case computational hardness of the GLM inference problem in either of these configurations.

State Evolution

The statistical analysis of the AMP equations in the average case leads to another closed set equations in the thermodynamic limit N → ∞ that was called State Evolution (SE) in [START_REF] Donoho | Message-passing algorithms for compressed sensing[END_REF]. Its derivation starts from the r-BP equations and relies on the assumption of independent incoming messages to invoke the Central Limit Theorem. It is therefore only necessary to follow the evolution of a set of means and variances. When the different variables and factors are statistically equivalent, as it is the case of the GLM, the State Evolution reduces to a few scalar equations. The interested reader should refer to Section 6.1.2 for a detailed derivation in a more general setting.

Mismatched setting

In the general mismatched setting we need to carefully differentiate the teacher and the student. We note p x0 the prior used by the teacher. We also rewrite its channel p out,0 as the explicit function g 0 (•, ) assuming the noise to be distributed according to the standard normal distribution. The tracked quantities are the overlaps,

q = lim N →∞ 1 N N i=1 x2 i , m = lim N →∞ 1 N N i=1 xi x 0,i , q 0 = lim N →∞ 1 N N i=1 x 2 0,i = E px 0 [x 2 0 ], (3.49)
along with auxiliary V , q, m and χ:

q(t) = d p 0 ( ) dω dz N (z, ω; 0, Q (t) )g out (ω, g 0 (z; ) , V (t) ) 2 (3.50) m(t) = d p 0 ( ) dω dz N (z, ω; 0, Q (t) )∂ z g out (ω, g 0 (z; ) , V (t) ) (3.51) χ(t) = -d p 0 ( ) ddωdz N (z, ω; 0, Q (t) )∂ ω g out (ω, g 0 z; µ , V (t) ) (3.52) q (t+1) = dx 0 p x0 (x 0 ) Dξ f x 1 (α χ(t) ) -1 αq (t) ξ + α m(t) x 0 ; (α χ(t) ) -1 2 (3.53) m (t+1) = dx 0 p x0 (x 0 ) Dξ x 0 f x 1 (α χ(t) ) -1 αq (t) ξ + α m(t) x 0 ; (α χ(t) ) -1 (3.54) V (t+1) = dx 0 p x0 (x 0 ) Dξ f x 2 (α χ(t) ) -1 αq (t) ξ + α m(t) x 0 ; (α χ(t) ) -1 (3.55)
where we use the notation N (•; •, •) for the normal distribution, Dξ for the standard normal measure and the covariance matrix Q (t) is given at each time step by

Q (t) =    q 0 m (t) m (t) q (t)    .
Due to the self-averaging property, the performance of the reconstruction by the AMP algorithm on an instance of size N can be tracked along the iterations given

MSE(x) = 1 N N i=1 (x i -x 0,i ) 2 = q -2m + q 0 , ( 3.56)
with only minor differences coming from finite-size effects. State Evolution also provides an efficient procedure to study from the theoretical perspective the AMP fixed points for a generic model, such as the GLM, as a function of some control parameters. It reports the average results for running the complete AMP algorithm on O(N ) variables using a few scalar equations. Furthermore, the State Evolution equations simplify further in the Bayes optimal setting.

Bayes optimal setting When the prior and channel are identical for the student and the teacher, the true unknown signal x 0 is in some sense statistically equivalent to the estimate x coming from the posterior. More precisely one can prove the Nishimori identities [START_REF] Opper | Calculation of the learning curve of Bayes optimal classification algorithm for learning a perceptron with noise[END_REF][START_REF] Iba | The Nishimori line and Bayesian statistics[END_REF][START_REF] Nishimori | Statistical Physics of Spin Glasses and Information Processing: An Introduction[END_REF] (or [START_REF] Kabashima | Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization[END_REF] for a concise demonstration and discussion) implying that q = m, V = q 0 -m and q = m = χ. Only two equations are then necessary to track the performance of the reconstruction:

q(t) = d p 0 ( ) dω dz N (z, ω; 0, Q (t) )g out (ω, g 0 (z; ) , V (t) ) 2 (3.57) q (t+1) = dx 0 p x0 (x 0 ) Dξ f x 1 (α χ(t) ) -1 αq (t) ξ + α m(t) x 0 ; (α χ(t) ) -1 2 .
(3.58)

Replica method

Another powerful technique from the statistical physics of disordered systems to examine models with infinite range interactions is the replica method. It enables an analytical computation of the quenched free energy via non-rigorous mathematical manipulations. More developed introductions to the method can be found in [START_REF] Nishimori | Statistical Physics of Spin Glasses and Information Processing: An Introduction[END_REF][START_REF] Castellani | Spin-glass theory for pedestrians[END_REF].

Steps of a replica computation

The basic idea of the replica computation is to apply the average over the disorder to the identity log Z = lim n→0 (Z n -1)/n, while at first considering n ∈ N, before taking the n → 0 limit -thus taking advantage of the fact that the average of a power of Z is easier to compute than the average of a logarithm. We illustrate the key steps for the calculation of the partition function of the GLM (3.22).

Disorder average for the replicated system: coupling of the replicas The average of Z n for n ∈ N can be seen as the partition function of a system with n + 1 non interacting replicas of x indexed by a ∈ {0, • • • , n}, where the first replica a = 0 is representative of the teacher and the n other replicas are identically distributed as the student:

E W ,y,x 0 [Z n ] = E W dy dx 0 p out,0 (y|W x 0 )p x 0 (x 0 ) dx p out (y|W x)p x (x) n (3.59) = E W   dy n a=0 dx a p out,a (y|W x a )p xa (x a )   (3.60) = E W   dy n a=0 dx a dz a δ(z a -W x a )p out,a (y|W x a )p xa (x a )   . (3.61)
To perform the average over the disordered interactions W we consider the statistics of z a = W x a .

Recall that W µi ∼ N (W µi ; 0, 1/N ), independently for all µ and i. Consequently, the z a are jointly Gaussian in the thermodynamic limit with means and covariances

E W [z a,µ ] = E W   N i=1 W µi x a,i   = 0 , E W z a,µ z b,ν = N i=1 x a,i x b,i /N = q ab . (3.62)
The overlaps, that we already introduced in the SE formalism, naturally re-appear. We introduce the notation q for the (n + 1) × (n + 1) overlap matrix. Note that integrating out the disorder W shared by the n + 1 replicas will therefore leave us with an effective system of now coupled replicas.

Change of variable for the overlaps: decoupling of the variables

Considering the overlaps as variables, and considering the Fourier representation of the Dirac distribution fixing the consistency between overlaps and replicas, we multiply the replicated average by

1 = a,b dN q ab δ(N q ab - N i=1 x a,i x b,i ) = a,b dN q ab a,b
dN qab e qab (N q ab -N i=1 xa,ix b,i ) .

(3.63)

As a result

E W ,y,x 0 [Z n ] = a,b dN q ab a,b dN qab exp (N qab q ab ) (3.64) dy n a=0 dz a p out,a (y|z a ) exp   - 1 2 M µ=1 a,b z a,µ z b,µ (q -1 ) ab -M C(q, n)   n a=1 dx a p xa (x a ) exp   -q ab N i=1 x a,i x b,i  
where C(q, n) is related to the normalization of the Gaussian distributions over the z a variables, and the integrals can be factorized over the i-s and µ-s. Thus we obtain

E W ,y,x 0 [Z n ] = a,b dN q ab a,b
dN qab e N qab q ab e M log Îz(q) e N log Îx(q) , (

Îz (q) = dy n a=0 dz a p out,a (y|z a ) exp   - 1 2 a,b z a z b (q ab ) -1 -C(q, n)   , 3.65)
Îx (q) = n a=1 dx a p xa (x a ) exp (-q ab x a x b ) , ( (3.66)
where we introduce the notation q for the auxiliary overlap matrix with entries (q) ab = qab . The decoupling of the x i and the z µ of the infinite range system yields pre-factors N and M in the exponential arguments and consequently an integral for the replicated average that is easily computed in the thermodynamic limit by the saddle point method:

E W ,y,x 0 [Z n ] e N extr qq φ(q,q) , φ(q, q) = a,b qab q ab + α Îz (q) + Îx (q), (3.68)
where we defined the replica potential φ.

Exchange of limits: back to the quenched average

The thermodynamic average of the log-partition is recovered through an a priori risky mathematical manipulation: (i) perform an analytical continuation from n ∈ N to n → 0

1 N E W ,y,x 0 [log Z] = lim n→0 1 nN E W ,y,x 0 [Z n -1] = lim n→0 1 nN log E W ,y,x 0 [Z n ] (3.69)
and (ii) exchange limits

-f = lim n→0 1 n lim N →∞ 1 N log E W ,y,x 0 [Z n ] = lim n→0 1 n extr qq φ(q, q) . (3.70)
Despite the apparent lack of rigour in taking these last steps, the replica method has been proven to yield exact predictions in the thermodynamic limit for different problems and in particular for the GLM [START_REF] Reeves | The replica-symmetric prediction for compressed sensing with Gaussian matrices is exact[END_REF][START_REF] Barbier | Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models[END_REF].

Saddle point solution: choice of a replica ansatz

At this point, we are still left with the problem of computing the extrema of φ(q, q). To solve this optimization problem over q and q, a natural assumption is that replicas, that are a pure artefact of the calculation, are equivalent. This is reflected in a special structure for overlap matrices between replicas that only depend on three parameters each,

q =    
q 0 m m m m q q 12 q 12 m q 12 q q 12 m q 12 q 12 q

    , q =     q0 m m m m q q12 q12 m q12 q q12 m q12 q12 q     , (3.71)
here given as an example for n = 3 replicas. Plugging this replica symmetric (RS) ansatz in the expression of φ(q, q), taking the limit n → 0 and looking for the stationary points as a function of the parameters q, m, q 12 and m, q, q12 recovers a set of equations equivalent to SE (1.7), albeit without time indices. Hence the two a priori different heuristics of BP and the replica method are remarkably consistent under the RS assumption.

Nevertheless, the replica symmetry can be spontaneously broken in the large N limit and the dominating saddle point do not necessarily correspond to the RS overlap matrix. This replica symmetry breaking (RSB) corresponds to substantial changes in the structure of the examined Boltzmann distribution. It is among the great strength of the replica formalism to naturally capture it. We will not investigate here this direction further. By considering the Bayes optimal setting, we will avoid the necessity of considering the breaking of the replica symmetry [START_REF] Nishimori | Statistical Physics of Spin Glasses and Information Processing: An Introduction[END_REF][START_REF] Castellani | Spin-glass theory for pedestrians[END_REF][START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF] in the problems discussed in Chapters 5 and 6.

Bayes optimal setting

As in SE the equations simplify in the matched setting, where the first replica corresponding to the teacher becomes equivalent to all the others. The replica free energy of the GLM is then given as the extremum of a potential over two scalar variables:

-f = extr q q -1 2 q q + I x (q) + αI z (q 0 , q) (3.72)

I x (q) = Dξ dx p x (x)e -q x 2 2 + √ qξx log dx p x (x )e -q x 2 2 + √ qξx (3.73)
I z (q, q 0 ) = Dξ dy dz p out (y|z)N (z; √ qξ, q 0 -q) log dz p out (y|z )N (z ; √ qξ, q 0 -q) .

(3.74)

The saddle point equations corresponding to the extremization (3.72), fixing the values of q and q, would again be found equivalent to the Bayes optimal SE (3.57) -(3.58).

Assumptions and relation to other mean-field methods

A crucial point in the above derivation of the replica formula is the extensivity of the interactions of the infinite range model that allowed the factorization of the N scaling of the argument of the exponential integrand. The statistics of the disorder W and in particular the independence of all the W µi was also necessary. This is an important assumption for the technique to go through, although it can be possible to relax it for some types of correlation statistics, as we will see in Section 3.5.3. Note that the replica method directly enforces the disorder averaging and does not provide a prediction at the level of the single instance. Therefore it cannot be turned into a practical algorithm of reconstruction. Nonetheless, we have seen that the saddle point equations of the replica derivation, under the RS assumption, matches the SE equations derived from BP. This is sufficient to theoretically study inference questions under a teacher-student scenario in the Bayes optimal setting.

In the mismatched setting however, the predictions of the replica method under the RS assumption and the equivalent BP conclusions can be wrong. By introducing the symmetry breaking between replicas, the method can sometimes be corrected. It is an important endeavor of the replica formalism to grasp the importance of the overlaps and connect the form of the replica ansatz to the properties of the joint probability distribution examined. When BP fails on loopygraph, correlations between variables are not decaying with distance, which manifests into an RSB phase. Subsequently, a message passing formalism taking into account (one step of) replica symmetry breaking was proposed [START_REF] Mézard | The Bethe lattice spin glass revisited[END_REF][START_REF] Mézard | Information, Physics, and Computation[END_REF]; it maintains the consistency of the replica method with the corresponding ansatz.

Extensions of interest for this thesis

In the previous Section we saw how frameworks and procedures of mean-field approximations rely on structural and statistical properties of the model under scrutiny. While we focused on the simple, and original, examples of the SK-Boltzmann machine and the GLM with Gaussian i.i.d weight matrices as examples, the span of applicability of mean-field methods now goes far beyond. We present here some of the recent applications of these concepts in more complex cases that are of particular interest for the rest of this thesis.

Streaming AMP for online learning

In learning applications, it is sometimes advantageous for speed or generalization concerns to only treat a subset of examples at the time -making for instance the SGD algorithm the most popular training algorithm in deep learning. Sometimes also, the size of the current data sets may exceed the available memory. Methods implementing a step-by-step learning as the data arrives are referred as online, streaming or mini-batch learning, as opposed to offline or batch learning.

In [START_REF] Manoel | Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing[END_REF], a mini-batch version of the AMP algorithm is proposed. On the example of the GLM, one imagines receiving at each iteration a subset of the components of y to reconstruct x. We denote by y (k) these successive mini-batches. Bayes formula gives the posterior distribution over x after seeing k mini-batches

p(x|y (k) , {y (k-1) , • • • y (1) }) = p(y (k) |x)p(x|{y (k-1) , • • • y (1) }) dx p(y (k) |x)p(x|{y (k-1) , • • • y (1) }) . (3.75)
This formula suggests the iterative scheme of using as a prior on x at iteration k the posterior obtained at iteration k -1. This idea can be implemented in different approximate inference algorithms, as proposed by [START_REF] Broderick | Streaming Variational Bayes[END_REF] using a variational method. In the regular version of AMP an effective factorized posterior is given at convergence by the input update functions (3.35)-(3.37):

p(x|y, W ) N i=1 1 Z x (λ i , σ i ) p x (x i )e - (λ i -x i ) 2 2σ i . (3.76)
Plugging this posterior approximation in the iterative scheme yields the mini-AMP algorithm using the converged values of λ ( ) i and σ ( ) i at each anterior mini-batch < k to compute the prior

p (k) x (x) = p(x|{y (k-1) , • • • y (1) }, W ) N i=1 1 Z x,i p x (x i ) e - k-1 =1 (λ ( ),i -x i ) 2 2σ ( ),i , ( 3.77)
where the Z x,i normalize each marginal factor. Compared to a naive mean-field variational approximation of the posterior, AMP takes into account more correlations and is indeed found to perform better in experiments reported by [START_REF] Manoel | Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing[END_REF]. An other advantage of the AMP based online inference is that it is amenable to theoretical analysis by a corresponding State Evolution. In Chapter 6, we follow the lines of [START_REF] Manoel | Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing[END_REF] to derive a streaming version of the Cal-AMP algorithm (see next section), for the training of minimal models of neural networks.

A special special case of GAMP: Cal-AMP

The calibration AMP algorithm was introduced in [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF], relatively to the problem of blind calibration. In the generic setting of the inference of an unknown signal from a set of observations, one can imagine a situation where the channel is not perfectly known and needs to be calibrated.

For the GLM problem, a possible formalization of this uncertainty is the addition of unobserved calibration variables, noted s ∈ R M here, in the channel likelihood:

p s out (y|W x, s) = M µ=1 p s out (y µ |w T µ x, s µ ) ⇐⇒ y µ = g(w T µ x; µ , s µ ), µ ∼ p ( µ ) ∀µ. (3.78)
The recovery of the value of s will only be feasible if multiple observations {y (k) } are available. If a set of complementary pairs {x (k) , y (k) } is furthermore available a priori, then one can imagine a supervised procedure to calibrate s before dealing with the original problem of reconstruction of x. Otherwise, a blind calibration has to be attempted. A Bayesian inference approach was proposed in [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF], using P observations {y (k) } P k=1 to compute simultaneously, in the Bayes-optimal setting, the MMSE estimators of the input signals {x (k) } P k=1 and the calibration s. The posterior distribution examined including prior distributions is

p({x (k) } P k=1 , s|y, W ) = 1 Z(W , {y (k) } P k=1 ) P,M k,µ=1 p s out (y (k) µ |w µ x (k) , s µ ) P,N k,i=1 p x (x (k) i ) M µ=1 p s (s µ ).
(3.79)

The Cal-AMP algorithm was derived following the steps of Section 3.3 (see [START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF] for details of this derivation). It is closely related to the GAMP for the reconstruction of the {x (k) } P k=1 , with two main differences. First it considers P observations and P signals to reconstruct simultaneously where AMP only has one, thereby multiplying the number of update equations. Second, the update output functions (3.38)- (3.40) are modified so as to take into account the effective output channel with uncertain calibration variables p out (y|z) = ds p s out (y|z, s)p s (s).

(3.80)

The proposed strategy is numerically tested in [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF] for a series of sub-problems, and succeeds at the reconstruction task when M and/or P are large enough.

In Chapter 6, we re-derive the algorithm using a slightly less factorized version of the posterior (recovering the algorithm of [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF] as a particular case). This is the starting point towards the analysis of Bayesian learning in multi-layer neural networks, an ongoing work presented in this last Chapter.

Algorithms and free energies beyond i.i.d. matrices

The derivations outlined in the previous Sections of the equivalent replica, TAP and AMP equations required the weight matrices to have Gaussian i.i.d. entries. Different weight statistics are a priori feasible if one finds a way to perform the corresponding disorder average in the replica computation or to evaluate the corresponding Onsager term in the TAP equations. For AMP however one considers a high connectivity limit of the BP equations relying on the independence of incoming messages at each variable or factor node. This high connectivity limit, albeit introducing short loops, turns out correct in the examined models with i.i.d. weights as suggests the consistency between AMP, TAP and replica equations. Rigorous proof were also given for the SK model [START_REF] Talagrand | The Parisi formula[END_REF] and the GLM [START_REF] Barbier | Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models[END_REF]. Nevertheless, this derivation may appear incorrect for correlated weight matrices that are promoting dependencies between messages.

Efforts to broaden in practice the class of matrices amenable to such mean-field treatments lead to a series of works in statistical physics and signal processing with related propositions. Parisi and Potters pioneered this direction by deriving mean-field equations for orthogonal weight matrices using a high-temperature expansion [START_REF] Parisi | Mean-field equations for spin models with orthogonal interaction matrices[END_REF]. The adaptive TAP approach proposed by Opper and Winther [START_REF] Opper | Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling[END_REF][START_REF] Opper | Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach[END_REF] further allowed for inference in densely connected graphical models without prior knowledge on the weight statistics. The Onsager term of these TAP equations was evaluated using the cavity method for a given weight sample. The resulting equations were then understood to be a particular case of the Expectation Propagation (EP) [START_REF] Thomas P Minka | A family of algorithms for approximate Bayesian inference[END_REF] -belonging to the class of message passing algorithms for approximate inference -yet applied in densely connected models [START_REF] Opper | Expectation consistent free energies for approximate inference[END_REF]. An associated approximation of the free energy called Expectation Consistently (EC) was additionally derived from the EP messages. Subsequently, Kabashima and collaborators [START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF][START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF] focused on the perceptron and the GLM to propose TAP equations and a replica derivation of the free energy for the ensemble of orthogonally invariant random weight matrices. In the singular value decomposition of such weight matrices W = U S V ∈ R M ×N the orthogonal basis matrices U and V are drawn uniformly at random from respectively O(M ) and O(N ), while the diagonal matrix of singular values S has an arbitrary spectrum. The consistency between the EC free energy and the replica derivation for orthogonally invariant matrices was verified by [START_REF] Kabashima | Signal recovery using expectation consistent approximation for linear observations[END_REF] for signal recovery from linear measurements (the GLM without G). From the algorithmic perspective, Fletcher, Rangan and Schniter [START_REF] Sundeep Rangan | Vector approximate message passing[END_REF][START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF] applied the EP to the GLM to obtain the (Generalized) Vector-Approximate Message Passing (G-VAMP) algorithm. Remarkably, these authors proved that the behavior of the algorithm could be characterized in the thermodynamic limit, provided the weight matrix is drawn from the orthogonally invariant ensemble, by a set of scalar State Evolution equations similarly to the AMP algorithm. These equations are again related to the saddle point equations of the replica free energy. Concurrently, Opper, Cakmak and Winther proposed an alternative procedure for solving TAP equations with orthogonally invariant weight matrices in Ising spin systems relying on an analysis of iterative algorithms [START_REF] Opper | A theory of solving TAP equations for Ising models with general invariant random matrices[END_REF].

Below we present the aforementioned free energy as proposed by [START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF][START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF], and the G-VAMP algorithm of [START_REF] Schniter | Vector approximate message passing for the generalized linear model[END_REF].

Replica free energy for the GLM in the Bayes Optimal setting

Consider the ensemble of orthogonally invariant weight matrices with spectral density N i=1 δ(λλ i )/N of their 'square' W W converging in the thermodynamic limit N → +∞ to ρ λ (λ). The quenched free energy of the GLM in the Bayes optimal setting derived in [START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF] writes -f = extr q q -1 2 q q + I x (q) + J z (q 0 , q, α, ρ λ ) (3.81)

J z (q 0 , q, α, ρ λ ) = extr uû F ρ λ ,α (q 0 -q, û/λ 0 ) + ûq 0 2 -αûu 2λ 0 + αI z (q 0 λ 0 /α, u) , (3.82) where I x and I z were defined as (3.73)-(3.74) and the spectral density ρ λ (λ) appears via its mean λ 0 = E λ [λ] and in the definition of

F ρ λ ,α (q, u) = 1 2 extr Λq,Λu -(α -1) log Λ u -E λ log(Λ u Λ q + λ) + Λ q q + αΛ u u - 1 2 (log q + 1) + α 2 (log u + 1). (3.83)
Gaussian random matrices are a particular case of the considered ensemble. Their singular values are characterized asymptotically by the Marcenko-Pastur distribution [START_REF] Marčenko | DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES[END_REF]. In this case, one can check that the above expression reduces to (3.72). More generally, note that J z generalizes I z .

Vector Approximate Message Passing for the GLM

The VAMP algorithm consists in writing EP [START_REF] Thomas P Minka | A family of algorithms for approximate Bayesian inference[END_REF] with Gaussian messages on the factor graph representation of the GLM posterior distribution given on Figure 3.2. The estimation of the signal x is decomposed onto four variables, two duplicates of x itself and two duplicates of the linear transformation z = W x. The potential functions ψ x and ψ z of factors connecting copies of the same variable are Dirac distributions enforcing their equality. The factor node linking z (2) and x (2) is assumed Gaussian with variance going to zero. The procedure of derivation, equivalent to the projection of the BP algorithm on Gaussian messages, is recalled in Appendix A and leads to Algorithm 2. Like for AMP, the algorithm features some auxiliary variables introduced along the derivation. At convergence the duplicated x1 , x2 (and ẑ1 , ẑ2 ) are equal and either can be returned by the algorithm as an estimator. For readability, we omitted the time indices in the iterations that here simply follow the indicated update.

z (1) z (2)
x (1) x (2) p x (x (1) )

p out (y|z)

ψ x (x (1) , x (2) ) = δ(x (1) -x (2) )
ψ z (z (1) , z (2) ) = δ(z (1) z (2) ) lim ∆→0 N (z (2) ; W x (2) , ∆)

Initialize: A x 1 , B x 1 , A z 1 , B z 1 repeat x1 = f x 1 (B x 1 , A x 1 ) , C x 1 = f x 2 (B x 1 , A x 1 ) (3.84) A x 2 = C x 1 -1 -A x 1 , B x 2 = C x 1 -1 x2 -B x 1 (3.85) ẑ1 = f z 1 (B z 1 , A z 1 ) , C z 1 = f z 2 (B z 1 , A z 1 ) (3.86) A z 2 = C x 1 -1 -A z 1 , B z 2 = C x 1 -1 x2 -B z 1 (3.87) x2 = g x 1 (B x 2 , A x 2 , B z 2 , A z 2 ) , C x 2 = g x 2 (B x 2 , A x 2 , B z 2 , A z 2 ) (3.88) A x 1 = C x 2 -1 -A x 2 , B x 1 = C x 2 -1 x1 -B x 2 (3.89) ẑ2 = g z 1 (B x 2 , A x 2 , B z 2 , A z 2 ) , C z 2 = g z 2 (B x 2 , A x 2 , B z 2 , A z 2 ) (3.90) A z 1 = C x 2 -1 -A z 2 , B z 1 = C x 2 -1 x1 -B z 2 (3.91) until convergence Output: signal estimate x1 ∈ R N , and estimated covariance C x 1 ∈ R N ×N
For a given instance of the GLM inference problem, i.e. a given weight matrix W , one can always launch either the AMP algorithm or the VAMP algorithm to attempt the reconstruction. If the weight matrix has i.i.d. zero mean Gaussian entries, the two strategies are conjectured to be equivalent and GAMP can be provably convergent for certain settings [START_REF] Sundeep Rangan | On the convergence of approximate message passing with arbitrary matrices[END_REF]. If the weight matrix is not Gaussian but orthogonally invariant, then only VAMP is expected to always converge. More generally, even in cases where none of these assumptions are verified, VAMP has been observed to have less convergence issues than AMP.

Like for AMP, a State Evolution can also be derived for VAMP (which was actually directly proposed for the multi-layer GLM [START_REF] Fletcher | Inference in Deep Networks in High Dimensions[END_REF]). It rigorously characterizes the behavior of the algorithm when W is orthogonally invariant. One can also verify that the SE fixed points can be mapped to the solutions of the saddle point equations of the replica free energy (3.81) (see Section 1 of Supplementary Material of [START_REF] Gabrié | Entropy and mutual information in models of deep neural networks[END_REF]); so that the robust algorithmic procedure can advantageously be used to compute the fixed points to be plugged in the replica potential to approximate the free energy.

Model composition

Another recent and ongoing direction of extension of mean-field methods is the combination of solutions to elementary models to tackle more sophisticated inference questions. The graphical representations introduced in Section 2.1.1 are here of great help. In a complicated joint probability distribution, it is sometimes possible to identify well-known sub-models, such as the GLM or the RBM already discussed in this thesis. Understanding how and when it is justified to plug-in different solutions is of course non-trivial and a very promising direction of research. For instance, in Chapter 4, we perform an ad hoc combination of the RBM and GLM model to design a practical algorithm operating on arbitrary (real) data sets, albeit not yet analyzable. An instead theoretically grounded extension in this direction is the treatment of multi-layer GLMs, or in other words multilayer neural networks. In [START_REF] Manoel | Multi-layer generalized linear estimation[END_REF] a multi-layer version of AMP is derived, assuming Gaussian i.i.d weight matrices, along with a State Evolution and a free energy. In [START_REF] Fletcher | Inference in Deep Networks in High Dimensions[END_REF], the multi-layer version of the VAMP algorithm is derived with the corresponding State Evolution for orthogonally invariant weight matrices. In Chapter 5, we derive the corresponding replica free energy. Yet another example, in Chapter 6, we derive AMP and SE while combining a calibration problem with a GLM. The derivation presented there will follow the lines of ML-AMP although in a slightly more general case.

Part II

Contributions 4 Mean-field inference for (deep) unsupervised learning

Unsupervised learning often consists in fitting a parametric probabilistic model to a collection of empirical samples. To this end, neural networks are a precious tool. They can define joint probability distributions with complex correlations able to reproduce data from the real world such as images or sounds. As exact inference on large neural networks is almost never possible, it is interesting to consider mean-field methods in this context. They are precisely designed to exploit structures and properties of interactions in large graphs. Yet, they also rely on certain assumptions of randomness and homogeneity which can be broken when parameters are chosen by a training algorithm. In this Chapter, we show that mean-field methods can indeed be helpful in the unsupervised training and downstream application of a class of neural network models called Boltzmann machines. In a first Section, we present the derivation of mean-field equations for approximate inference in Boltzmann machines. In particular, we generalize the usual case of binary variables to arbitrary real-valued variables. In a second Section, we leverage the formalism derived in the first Section to propose a new training algorithm for Restricted Boltzmann Machines. We verify numerically that the mean-field strategy achieves state-of-art performance. These two first sections are based on the publications [42] and [150]. In a third Section, we discuss an application where the representative power of RBMs can be exploited through mean-field inference for a reconstruction task. The designed method successfully integrates unsupervised learning to improve Bayesian reconstruction. This project was led by E. W. Tramel and published in [151]. Finally, as perspectives, we hint at further directions leveraging mean-field methods for the nowadays more common feed forward models used for unsupervised learning.

Mean-field inference in Boltzmann machines

We start by presenting mean-field approximations for Boltzmann machines with an arbitrary architecture before turning to the specific case of the Restricted and Deep Boltzmann machines -RBMs and DBMs -that are of special interest in machine learning. To derive TAP free energies, we choose here the formalism of Georges-Yedidia [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF] which allows to compute systematically corrections to the naive mean-field approximation by the addition of orders to the expansion.

Georges-Yedidia expansion for binary Boltzmann machines

We consider the binary vector x ∈ {0, 1} N following the distribution defined by a Boltzmann machine with symmetric weight matrix W ∈ R N ×N and bias vector b ∈ R N ,

E(x) = -b x - 1 2 x W x , p(x) = 1 Z e -βE(x) . (4.1)
This model is closely related to the pairwise Ising model for which the TAP equations [START_REF] Thouless | Solution of 'Solvable model of a spin glass[END_REF], the Plefka expansion [START_REF] Plefka | Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model[END_REF] and the Georges-Yedidia expansion [START_REF] Georges | How to expand around mean-field theory using high-temperature expansions[END_REF] were originally derived. The only difference consists in assuming x i = 0 or 1 instead of x i = ±1. The derivation already outlined in Section 3.2.1 can be found in detail for this original case in different references including [START_REF] Opper | Advanced mean field methods: Theory and practice[END_REF] and [START_REF] Zamponi | Mean field theory of spin glasses[END_REF]. Following these steps we obtain the Georges-Yedidia expansion for the binary Boltzmann machine for x i ∈ {0, 1} up to the third order as a function of the marginals (or magnetizations),

-βG(m) = - N i=1 m i log m i + (1 -m i ) log(1 -m i ) + β N i=1 b i m i + β (ij) W ij m i m j (4.2) + β 2 2 (ij) W 2 ij (m i -m i 2 )(m j -m j 2 ) + 2β 3 3 (ij) W 3 ij (m i -m i 2 )(1/2 -m i )(m j -m j 2 )(1/2 -m j ) + β 3 (ijk) W ij W jk W ki (m i -m i 2 )(m j -m j 2 )(m k -m k 2 ) + • • •
where the summation over (ij) runs over distinct pairs of connected variables and similarly (ijk) runs over distinct triplets of interconnected variables. Note that in the above energy definition (4.1) we assumed a fully connected graph of connections, yet any other architecture can be recovered by setting some entries of W to zero. To recover an approximation of the free energy, we must fix the values of the marginals by finding the stationary points

m * i of G, ∂G(m) ∂m i m * i = 0 ∀i = 1 • • • N ⇒ G(m * ). (4.3)
These conditions can be formulated as a set of self-consistent equations verified by the m * i (noted simply m i in the following),

m i = σ βb i + j∈∂i βW ij m j -β 2 W 2 ij (m j -1/2)(m i -m i 2 ) (4.4) + 2β 3 3 j∈∂i W 3 ij 2(1/2 -m i ) 2 -(m i -m i 2 ) (m j -m j 2 )(1/2 -m j ) + β 3 2 (jk)∈∂i W ij W jk W ki (1/2 -m i )(m j -m j 2 )(m k -m k 2 ) + • • •
where σ(x) = (1+e -x ) -1 is the sigmoid function, and (jk) ∈ ∂i corresponds to the set of connected pairs of variables that are also connected to variable i. A truncation of the expansion (4.2) at a given order is reflected by a truncation of the sigmoid argument in (4.4). We refer to the resulting set of equations as the TAP equations of a given order, allowing to compute the corresponding TAP free energy of a given order. In particular, the naive mean-field approximation discussed in Section 3.1 is recovered by keeping only the first order.

Georges-Yedidia expansion for generalized Boltzmann machines

The binary variable case presented above can be recovered from a more general setting considering the x i -s to be a priori real-valued. Formally we consider now x ∈ R N governed by the energy function and parametrized distribution

E(x) = - (ij) W ij x i x j - 1 β log p x (x i ; θ i ) , p(x) = 1 Z e β 2 x W x N i=1 p x (x i ; θ i ), (4.5)
where p x (x i ; θ i ) is an arbitrary prior distribution with parameter θ i . For Bernoulli prior with parameter σ(βb i ) we recover the measure of binary Boltzmann machine. However we choose here a prior that does not depend on the temperature a priori. We now derive the Georges-Yedidia expansion for this general case following the outline discussed in 3.2.1, and highlighting the differences with the binary case. We note that inference in the generalized fully connected Boltzmann machine is somehow related to the symmetric rank-1 matrix factorization problem, which also features pairwise interactions.

Similarly, inference for the bi-partite RBM maps to the asymmetric rank-1 matrix factorization. However, conversely to the Boltzmann inference, these factorizations are reconstruction problems. The mean-field techniques, derived in [START_REF] Thibault Lesieur | MMSE of probabilistic lowrank matrix estimation: Universality with respect to the output channel[END_REF][START_REF] Thibault Lesieur | Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications[END_REF], allow there to compute the MMSE estimator of unknown signals from approximate marginals. Here our main goal is to evaluate the free energy.

Minimization for fixed marginals

While fixing the value of the first moment is sufficient for binary variables, we now need more than one constraint in order to minimize the Gibbs free energy at a given value of the marginals. In the same spirit of the AMP algorithm we assume a Gaussian parametrization of the marginals. We note a the first moment of x and c its variance. We wish to compute the constrained minimum over the distributions

q on R N G(a, c) = min q E(x) q -H(q)/β | x q = a , x 2 q = a 2 + c , (4.6)
where the notation of squared vectors corresponds here and below to the vectors of squared entries.

It is equivalent to an unconstrained problem with Lagrange multipliers λ(a, c, β) and ξ(a, c, β)

G(a, c) = min q E(x) q -H(q)/β -λ ( x q -a)/β -ξ( x 2 q -a 2 -c)/β . (4.7)
The terms depending on the distribution q in the the functional to minimize above can be interpreted as a Gibbs free energy for the effective energy functional

Ẽ(x) = E(x) -λ x/β -ξ x 2 /β. (4.8)
The solution of the minimization problem (4.10) is therefore the corresponding Boltzmann distribution

q a,c (x) = e -β Ẽ(x) Z = 1 Z e -βE(x)+λ(a,c,β) x+ξ(a,c,β) x 2 (4.9)
and the minimum G(a, c) is

-βG(a, c) = -λ a -ξ (a 2 + c) + log dx e -βE(x)+λ x+ξ x 2
= log dx e -βE(x)+λ (x-a)+ξ (x 2 -a 2 -c) , (4.10)

where the Lagrange multipliers λ(a, c, β) and ξ(a, c, β) enforcing the constraints are still implicit. Defining a functional G for arbitrary vectors λ ∈ R N and ξ ∈ R N , -β G(a, c, λ, ξ) = log dx e -βE(x)+ λ (x-a)+ ξ (x 2 -a 2 -c) , (

we have

a i = x i qa,c ⇒ -β ∂ G ∂ λi λ,ξ = 0, -β ∂ 2 G ∂ λ2 i λ,ξ = x 2 i qa,c -a 2 i > 0, (4.12
)

c i + a 2 i = x 2 i qa,c ⇒ -β ∂ G ∂ ξi λ,ξ = 0, -β ∂ 2 G ∂ ξ2 i λ,ξ = (x 2 i ) 2 qa,c -(c i + a 2 i ) 2 > 0. (4.13)
Hence the Lagrange multipliers are identified as minimizers of -β G and

-βG(a, c) = -β G(a, c, λ(a, c, β), ξ(a, c, β)) = min λ, ξ -β G(a, c, λ, ξ). (4.14)
The true free energy F = -log Z/β would eventually be recovered by minimizing the constrained minimum G(a, c) with respect to its arguments. Nevertheless, the computation of G and G involves an integration over x ∈ R N and remains intractable. The following step of the Georges-Yedidia derivation consists in approximating these functionals by a Taylor expansion at infinite temperature where interactions are neutralized.

Expansion around β = 0

To perform the expansion we introduce the notation A(β, a, c) = -βG(a, c). We also define the auxiliary operator

U (x; β) = - 1 2 x W x + 1 2 x W x qa,c - N i=1 ∂λ i ∂β (x i -a i ) - N i=1 ∂ξ i ∂β (x 2 i -a 2 i -c i ), (4.15)
that allows to write concisely for any observable O the derivative of its average with respect to β,

∂ O(x; β) qa,c ∂β = ∂O(x; β) ∂β qa,c -U (x; β)O(x; β) qa,c . (4.16)
To compute the derivatives of λ and ξ with respect to β we note that

∂A ∂a i = -β ∂ G ∂a i = -λ i (β, a, c) -2a i ξ i (β, a, c) , ( 4.17
)

∂A ∂c i = -β ∂ G ∂c i = -ξ i (β, a, c), ( 4.18)
where we used that ∂ G/∂ λi = 0 and ∂ G/∂ ξi = 0 when evaluated for λ(a, c, β) and ξ(a, c, β). Consequently,

∂ξ i ∂β = - ∂ ∂c i ∂A ∂β , ∂λ i ∂β = - ∂ ∂a i ∂A ∂β + 2a i ∂ξ i ∂β . ( 4.19)
We can now proceed to compute the first terms of the expansion that will be performed for the functional A. dx i e λi(0,a,c)xi+ξi(0,a,c)x 2 i p x (x i ; θ i ). (

Zeroth order

At infinite temperature the interaction terms of the energy do not contribute so that the integral in Z0 factorizes and can be evaluated numerically in the event that it does not have a closed-form.

First order

We compute the derivative of A with respect to β. We use again that λ(a, c, β) and ξ(a, c, β) are stationary points of G to write

∂A ∂β = -β ∂ G ∂β = ∂ ∂β log dx e -βE(x)+λ(a,c,β) (x-a)+ξ(a,c,β) (x 2 -a 2 -c) (4.23) = ∂ ∂β (-βE(x)) + ∂λ ∂β (x -a) + ∂ξ ∂β (x 2 -a 2 -c) qa,c (4.24) = 1 2 x W x qa,c . (4.25)
At infinite temperature the average over the product of variables becomes a product of averages so that we have

∂A ∂β β=0 = 1 2 a W a = (ij) W ij a i a j . ( 4

.26)

Second order Using the first order derivative of A we can compute the derivatives of the Lagrange parameters (4.19) and the auxillary operator at infinite temperature,

∂ξ i ∂β β=0 = 0 , ∂λ i ∂β β=0 = - j∈∂i W ij a j , U (x; 0) = - (ij) W ij (x i -a i )(x j -a j ).
The second order derivative is then easily computed at infinite temperature

∂ 2 A ∂β 2 β=0 = 1 2 ∂ ∂β x W x qa,c β=0 = - 1 2 U (x; 0)(x W x) β=0 qa,c (4.27) = (ij) W 2 ij (x i -a i )x i (x j -a j ) β=0 qa,c = (ij) W 2 ij c i c j . (4.28)

TAP free energy for the generalized Boltzmann machine

Stopping at the second order of the systematic expansion, and gathering the different terms derived above we have

-βG(a, c) = -λ(0, a, c) a -ξ(0, a, c) (a 2 + c) + log Z0 (λ(0, a, c), ξ(0, a, c)) (4.29) + β (ij) W ij a i a j + β 2 2 (ij) W 2 ij c i c j ,
where the values of the parameters λ(0, a, c) and ξ(0, a, c) are implicitly defined through the stationary conditions (4.12)-(4.13). The TAP approximation of the free energy also requires to consider the stationary points of the expanded expression as a function of a and c. This second condition yields the relations

-2ξ i (0, a, c) = -β 2 j∈∂i W 2 ij c j = A i (4.30) λ i (0, a, c) = A i a i + β j∈∂i W ij a j = B i (4.31)
where we define new variables A i and B i . While the extremization with respect to the Lagrange multipliers gives

a i = 1 Z x i dx i x i p x (x i ; θ i )e -A i 2 x 2 i +Bixi = f x 1 (B i , A i ; θ i ), (4.32
)

c i = 1 Z x i dx i x 2 i p x (x i ; θ i )e -A i 2 x 2 i +Bixi -a 2 i = f x 2 (B i , A i ; θ i ), (4.33)
where we introduce update functions f x 1 and f x 2 with respect to the partition function

Z x i (B i , A i ; θ i ) = dx i p x (x i ; θ i )e -A i 2 x 2 i +Bixi . (4.34)
Finally we can rewrite the TAP free energy as

-βG(a, c) = -B a + A (a 2 + c)/2 + N i=1 log Zi x (B i , A i ; θ i ) (4.35) + β (ij) W ij a i a j + β 2 2 (ij) W 2 ij c i c j ,
with the values of the parameters set by the self-consistency conditions (4.30), (4.31), (4.32) and (4.33), which are the TAP equations of the generalized Boltzmann machine at second order. Note that the naive mean-field equations are recovered by ignoring the second order terms in β 2 .

Relation to message passing

The TAP equations obtained above must correspond to the fixed points of the Approximate Message Passing (AMP) following the derivation from Belief Propagation (BP) that was presented in Section 3.3.3. In the Appendix B of [150] we derive the relaxed-BP equations for the generalized Boltzmann machine:

B (t) i→j = k∈∂i\j βW ik a (t) k→i , A (t) i→j = - k∈∂i\j β 2 W 2 ik c (t)
k→i , (4.36)

a (t) i→j = f x 1 (B (t-1) i→j , A (t-1) i→j ; θ i ), c (t) i→j = f x 2 (B (t-1) i→j , A (t-1) i→j ; θ i ). (4.37)
To recover the TAP equations for them we define

B (t) i = k∈∂i βW ik a (t) k→i , A (t) i = - k∈∂i β 2 W 2 ik c (t)
k→i , (4.38)

a (t) i = f x 1 (B (t-1) i , A (t-1) i ; θ i ), c (t) i = f x 2 (B (t-1) i , A (t-1) i ; θ i ). (4.39) As B (t) i = B (t) i→j + βW ij a (t)
j→i and A

(t) i = A (t) i→j -β 2 W 2 ij c (t)
j→i we have by developing

f x 2 that c (t) i = c (t) i→j + O(β) so that A (t) i = -β 2 j∈∂i W 2 ij c (t) j + o(β 2 ). (4.40)
By developing f x 1 we also have

a (t) k = f x 1 (B (t-1) k→j + βW kj a (t-1) j→i , A (t-1) k→j -β 2 W 2 kj c (t-1) j→k ; θ i ) (4.41) = a (t) k→j + ∂f x 1 ∂B k βW kj a (t-1) j→k + O(β 2 ), (4.42)
with

∂f x 1 ∂B k (B (t-1) k , A (t-1) k ; θ k ) = c (t)
k . Finally, by replacing in the definition of B i the messages we obtain

B (t) i = k∈∂i βW ik a (t) k→i = k∈∂i βW ik a (t) k -βW ki c (t) k a (t-1)
i→k .

(4.43)

As a

(t-1) i→k = a (t-1) i
+ O(β) and using the definition of A (t) i , we finally recover

B (t) i = k∈∂i βW ik a (t) k + A (t) i a (t-1) i . ( 4.44)
Hence we indeed recover the TAP equations as the AMP fixed points in (4.39), (4.40) and (4.44). Beyond the possibility to cross-check our results, the message passing derivation also specifies a scheme of updates to solve the self-consistency equations obtained by the Georges-Yedidia expansion. In the applications we consider below we should resort to this time indexing with good convergence properties [START_REF] Bolthausen | An iterative construction of solutions of the tap equations for the Sherrington-Kirkpatrick model[END_REF].

Solutions of the TAP equations

As already discussed in Section 3.2, the TAP equations do not necessarily admit a single solution. In practice, different fixed points are reached when considering different initializations of the iteration of the self-consistent equations. In the applications of the TAP formalism discussed in the remaining of this Chapter we will discuss the strategy we adopt to take into account this multiplicity.

Application to RBMs and DBMs

Generalized Restricted Botlzmann Machines

In machine learning applications the bipartite architecture of RBMs, that we recall on Figure 4.1, is of particular interest. Traditionally they have binary units, yet they can be generalized to x (1) x (2) p x (x (1) ) The corresponding measure for a model with N input (or visible) units and M hidden units is

t 1 t 2 x 1 x 2 x 3 RBM x 1 x 2 x 3 t (1) 1 t (1) 2 t (2) 1 t (2) 2 t (2) 3 t (3) 1 t (3) 2 DBM
ψ x (x (1) , x (2) ) = δ(x (1) -x (2) ) ψ out (x (2) ) = e x (2) W x (2) +b x (2)
p(x, t; θ) = N i=1 p x (x i ; θ x i ) M α=1 p h (t α ; θ h α )e x W t , ( 4.45)
where the weight matrix W has dimension N × M and the temperature is set as β = 1. Note that the expansion at small β can equivalently be interpreted as a small weight expansion here. We use the notation θ for the collection of all the parameters of the model including

θ x = (θ x 1 , • • • θ x N ), θ h = (θ h 1 , • • • θ h M )
and W . The TAP free energy (4.35) is rewritten as

-G RBM (a x , c x , a h , c h ) = N i=1 -B x i a x i + A x i (a x i 2 + c x i )/2 + log Z x i (B x i , A x i ; θ x i )) (4.46)
+ M α=1 -B h α a h α + A h α (a h α 2 + c h α )/2 + log Z h α (B h α , A h α ; θ h α )) + (iα) W iα a x i a h α + (iα) W 2 iα c x i c h α ,
where the sums over (iα) runs over all i-s and α-s and

Z x i (B x i , A x i ; θ x i ) = dx p x (x; θ x i ) e -A x i x 2 /2+B x i x , (4.47)
Z h α (B h α , A h α ; θ h α ) = dt p h (t; θ h α ) e -A h α t 2 /2+B h α t . (4.48)
The values of the parameters are computed by running Algorithm 3 corresponding to AMP, where the differentiated visible and hidden update functions, f x 1 , f x 2 and f h 1 , f h 2 , are following the usual definition with respect to the partitions Z x i and Z h α .

Deep Botlzmann Machines

It is also possible to define deep models of Boltzmann machines by considering several stacked hidden layers. A Deep Boltzmann Machines (DBM) [START_REF] Salakhutdinov | Deep Boltzmann Machines[END_REF] with L hidden layers defines the distribution p(x, t (1) , .., t (L) ; θ) = 1 Z e

x W t (1)

+ L-1 =1 t ( ) W ( ) t (l+1) N i=1 p x (x i ; θ x i ) L l=1 M l α=1 p ( ) α (t ( ) α ; θ h α ( ) ). (4.49)
Here again the specific architecture considered can be interpreted as a particular case of the general fully connected Boltzmann machine. The corresponding TAP free energy writes

-G DBM (a x , c x , {a h, , c h, } L =1 ) = N i=1 -B x i a x i + A x i (a x i 2 + c x i )/2 + log Z0,x i (B x i , A x i ; θ x i )) (4.50) + L =1 M α=1 -B h, α a h, α + A h, α (a h, α 2 + c h, α )/2 + log Z0,h, α (B h, α , A h, α ; θ h, α )) + (iα) W (1) iα a x i a h,1 α + (iα) W (1) iα 2 c x i c h,1 α , + L-1 =1   (αβ) W ( ) αβ a h, α a h, +1 β + (αβ) W ( ) αβ 2 c h, α c h, +1 β   ,
and the corresponding AMP algorithm straightforwardly follows from Algorithm 3. The DBM is also bipartite, with connections only between layers of even depth and odd depth. Therefore the TAP parameters of even and odd layers can be updated sequentially, as for the hidden and input parameters of the RBM.

Algorithm 3 TAP solutions for GRBMs

Input: W , θ x , θ h , a x,(0) , c x,(0) , τ Initialize: t = 0 repeat Hidden Side Updates A h,(t+1) α = -i W 2 iα c x,(t) i B h,(t+1) α = A h,(t+1) α a h,(t) α + i W iα a x,(t) i a h,(t+1) α = f h 1 (B h,(t+1) α , A h,(t+1) α ; θ h α ) c h,(t+1) α = f h 2 (B h,(t+1) α , A h,(t+1) α ; θ h α ) Visible Side Updates A x,(t+1) i = -α W 2 iα c h,(t+1) α B x,(t+1) i = A x,(t+1) i a x,(t) i + α W iα a h,(t+1) α a x,(t+1) i = f x 1 (B x,(t+1) i , A
x,(t+1) i

;

θ x i ) c x,(t+1) i = f x 2 (B x,(t+1) i , A x,(t+1) i ; θ x i ) t = t + 1 until Convergence or t = τ

Algorithm 4 AdaTAP for binary BMs

Input: J, b, τ

Initialize: t = 0, A (0) 1 , B (0) 1 repeat Prior Updates a (t+1) 1,i = σ(B (t) 1,i -A (t) 1,i /2) c (t+1) 1,i = a (t+1) 1,i (1 -a (t+1) 1,i ) A (t+1) 2,i = 1/c (t+1) 1,i -A (t+1) 1,i B (t+1) 2,i = 1/(1 -a (t+1) 1,i ) -B (t+1) 1,i Interaction Updates C (t+1) 2 = (A (t+1) 2 -J) -1 a (t+1) 2,i = j C (t+1) 2 ij B (t+1) 2,j + b j A (t+1) 1,i = 1/ C (t+1) 2 ii -A (t+1) 2,i B (t+1) 1,i = a (t+1) 2,i / C (t+1) 2 ii -B (t+1) 2,i t = t + 1 until Convergence or t = τ

Adaptive-TAP fixed points for binary BM

As already discussed in Sections 3.1 and 3.2, for certain weight statistics, there exists an order of truncation of the Georges-Yedidia expansion which results in exact inference in the thermodynamic limit. In the general case nevertheless, the truncation is ad-hoc and adding orders allows to improve the accuracy of the approximation. Alternatively, one can prefer the adaptive TAP (adaTAP) strategy [START_REF] Opper | Tractable approximations for probabilistic models: The adaptive Thouless-Anderson-Palmer mean field approach[END_REF][START_REF] Opper | Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling[END_REF][START_REF] Opper | Expectation consistent free energies for approximate inference[END_REF] which computes an Onsager correction to the naive mean-field adaptively to a given weight matrix and without any assumption on the underlying statistics of its entries.

The adaTAP equations originally derived by the cavity method [START_REF] Mézard | Spin Glass Theory and Beyond[END_REF] can also be recovered via different equivalent variants of message passing. We choose here the conventions of the VAMP [START_REF] Sundeep Rangan | Vector approximate message passing[END_REF] procedure, originally applied to the GLM. We focus on the case of the fully connected Boltzmann machine with binary units x ∈ {0, 1} N and weight matrix J ∈ R N ×N and biases b ∈ R N . Using the factor graph of Figure 4.2, we follow the steps described in Appendix A to obtain Algorithm 4, where the input and output update functions were directly replaced, following the usual definitions:

Input functions a 1,i = f x 1 (B 1,i , A 1,i ) = 1 Z x i dx x 1 2 δ(x) + 1 2 δ(x -1) e -A1,ix 2 /2+B1,ix (4.51) c 1,i = f x 2 (B 1,i , A 1,i ) = 1 Z x i dx x 2 1 2 δ(x) + 1 2 δ(x -1) e -A1,ix 2 /2+B1,ix (4.52)
Output/Interaction functions

a 2,i = g x 1 (B 2 , A 2 ) = 1 Z out dx x i ψ out (x)e -x A 2 x/2+B 2 x (4.53) C 2 = g x 2 (B 2 , A 2 ) = 1 Z out dx xx ψ out (x)e -x A 2 x/2+B 2 x (4.54)
with the interaction potential ψ out (x) = e x W x+b x and A 2 = diag(A 2 ) the diagonal matrix with entries equal to the A 2,i . Compared to the VAMP Algorithm 2, we restrict the estimated covariances to be diagonal except for C 2 , a common simplification.

The bottleneck in the execution time of Algorithm 4 is the evaluation of the matrix inverse in the update of C 2 which has a typical complexity of O(N 3 ), and must be repeated at each iteration. Hence the price to pay for the adaTAP/VAMP algorithm is a significant increase of the computation time.

Applications to Boltzmann machine learning with hidden units

Boltzmann machines are used as parametric probabilistic models for unsupervised learning. The idea is to fit the weights and biases (or more generally local parameters for non-binary BMs) to a set of training data points so that the BM measure mimics the empirical data distribution. To achieve this goal the common objective of learning is to maximize the log-likelihood, yet an exact implementation of this approach is computationally intractable for all but the smallest models.

In the statistical physics community, Boltzmann machines typically do not feature any hidden units. This fully-visible Boltzmann architecture is justified as it corresponds to the model of maximum entropy under the constraint of fixed first and second moments. In fact the process of learning weights from a set of configurations at equilibrium is referred to as the inverse Ising problem in the physics literature. Learning algorithms based on sophisticated extensions of meanfield methods have been developed for these fully-visible Boltzmann machines(see [START_REF] Nguyen | Inverse statistical problems: from the inverse Ising problem to data science[END_REF] for a recent review). However, since they only include couplings between pairs of variables such models cannot successfully capture higher-order correlations. Conversely, in the RBM, the hidden layer mediates interactions between all the visible units and drastically increases the representative power of the model.

In the machine learning community fast approximate Monte Carlo methods, specifically contrastive divergence (CD) [START_REF] Hinton | Training products of experts by minimizing Contrastive divergence[END_REF] and persistent CD (PCD) [START_REF] Radford | Connectionist learning of belief networks[END_REF][START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF], have made large-scale training of RBMs possible. These rather crude methods have popularized RBMs even though the reason of their efficiency remains unclear. We propose to examine an alternative strategy based on meanfield approximations. In the 80s and 90s, this direction was already investigated in a number of previous works [START_REF] Peterson | A mean field theory learning algorithm for neural networks[END_REF][START_REF] Hinton | Deterministic Boltzmann learning performs steepest descent in weightspace[END_REF][START_REF] Conrad | The limitations of deterministic Boltzmann machine learning[END_REF][START_REF] Hilbert | Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction[END_REF], albeit in small models with binary units and for artificial data sets very different from modern machine learning benchmarks. More recently, a deterministic training based on naive mean-field [START_REF] Welling | A new learning algorithm for mean field Boltzmann machines[END_REF][START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF] was tested in the large-scale and found to provide poor performance when compared to both CD and PCD. Here we show that going beyond naive mean-field allows to bridge the gap in efficiency with the approximate Monte Carlo methods. Additionally the deterministic mean-field framework offers a tractable way of evaluating the learning success by exploiting the mean-field observables.

Mean-field likelihood and deterministic training

Mean-field likelihood

In RBMs, the log-likelihood of a training set P k=1 log p(x (k) ; θ)/P, follows from the marginalization over the hidden units of the definition (4.45),

log p(x; θ) = N i=1 log p x (x i ; θ x i ) + M α=1 dt α log p h (t α ; θ h α )e i Wiαtαxi -log Z(θ). (4.55)
The log-partition it involves is nothing else but the intractable free energy of the RBM discussed in the beginning of this Chapter. It can be estimated using advanced, yet unfortunately costly, Monte Carlo methods such as Annealed Importance Sampling (AIS) [START_REF] Salakhutdinov | On the quantitative analysis of Deep Belief Networks[END_REF]. Practitioners often prefer to monitor the surrogate pseudo-likelihood. 1 Instead, we propose to leverage mean-field inference to estimate this objective of learning and among other applications, help comparing different training algorithm quantitatively.

To compute the TAP free energy introduced in the previous Section, one needs to solve the TAP equations using Algorithm 3 and plug the solutions into the definition (4.46). We consider the TAP solutions resulting from initializations of the inference algorithm in data points of the training (or testing) set, a x,(0) = x (k) and c x,(0) = 0 and average them uniformly. This choice follows from that the trained RBM will be used in a space of configurations neighboring the real data. Below, we investigate numerically the properties of the TAP solutions in learning experiments, and in particular find that their number grows as the weight magnitude (playing the role of an inverse temperature here) increases along the training.

The distribution of the weight entries after learning usually results from successive parameter updates and is a priori unknown. Therefore, the order of truncation of the small weight expansion is inevitably ad-hoc and the accuracy of the approximation hard to estimate. Alternatively, the AdaTAP free energy could be considered, yet at a higher computational cost as discussed in Section 4.1.4. In the numerical experiments presented in the following, we compare different orders and find that, albeit heuristic, the second order truncation yields a practical effective scoring of the learning process.

Mean-field training algorithm

The usual strategy for training RBMs consists in an approximate gradient ascent of the loglikelihood. The popular CD and PCD algorithms [START_REF] Hinton | Training products of experts by minimizing Contrastive divergence[END_REF][START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF] use fast Monte Carlo approximations of the gradients of log Z(θ). Alternatively we consider the gradients of the TAP free energy truncated at second order (4.46), which yields

∆θ x i = ∂ ∂θ x i log p x (x i ; θ x i ) x - ∂ ∂θ x i log Z x i (B x i , A x i ; θ x i ) , (4.56) ∆θ h α = ∂ ∂θ h α log p(t α |x; θ) x - ∂ ∂θ h α log Z h α (B h α , A h α ; θ h α ) , ( 4.57)
∆W iα = x i f h 1 ( N i=1 W iα x i , 0; θ h α ) - (iα) a x i a h α + W iα c x i c h α . (4.58)
These directions of increment can be seen as approximations of the exact likelihood gradients or exact gradients of the mean-field likelihood. The latter can be regarded in itself as an objective of learning. The first term of the three gradients depend on the data point x and can be evaluated exactly. In practice we consider training with mini-batches over which these data-dependent terms are averaged. The second term in the gradient requires to compute TAP solutions, and corresponds to the part evaluated by Monte Carlo in CD. We compare multiple procedures:

• TAP-τ . We initialize the TAP inference in the data points of the mini-batch considered at the given parameter update and average the gradients over the TAP solutions retrieved after a given number τ of iterations of Algorithm 3. We speed up the gradient evaluation by an early stopping after τ steps, instead of converging the inference at each parameter update.

• Persistent-TAP-τ (P-TAP-τ ). We initialize the TAP inference at the very beginning of training in a collection of randomly selected training data point (typically as many as the size of a mini-batch), and iterate the TAP inference for a few steps at each parameter update, always conserving the TAP solutions of the previous update as the initialization of the next one. Since the model parameters are only slightly changing between updates, TAP fixed points of neighboring steps of the gradient ascent are expected to be closely related, and the convergence from one to the next within a few iterations.

These strategies are comparable to CD-τ [START_REF] Hinton | Training products of experts by minimizing Contrastive divergence[END_REF] and P-CD [START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF] where iterations of the inference algorithm are replaced by Gibbs sampling on the RBM.

In Algorithm 5, we summarize the mean-field training. The learning rate is noted γ. We include an 2 penalty with a coefficient , also called weight decay, promoting small weights; it is a common regularization of training for generalization that is all the more interesting here since the TAP likelihood relies on a small weight expansion. We also consider a momentum term involving the previous weight updates with a parameter η, known to speed up the training.

Algorithm 5 GRBM Mean-field training

Input: {x (k) } P k=1 Initialize: W (0) iα ∼ N (0, σ), θ x,(0) , θ h,(0) , t = 0 repeat for All mini-batches {x (k) } K k=1 do if Persistent and t > 0 then a (t+1) (k) , c (t+1) (k) , B (t+1) (k) , A (t+1) (k)
← Alg. 3(W (t) , θ (t) , a

x,(t-1) (k)

, c

x,(t-1) (k)

, τ ) ∀k else a TAP log-likelihood < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 s a U j h a t x y t X N q X H l

(t+1) (k) , c (t+1) (k) , B (t+1) (k) , A (t+1) (k) ← Alg. 3(W (t) , θ (t) , x (k) , 0, τ ) ∀k end if W (t+1) iα ← W (t) iα + γ∆W (t) iα + γ W 2 iα (t) + η∆W (t-1) iα θ h α ,(t+1) ← θ h α ,(t) + γ∆θ h α ,(t) θ x i ,(t+1) ← θ x i ,(t) + γ∆θ x i ,(t) t = t +
P c C R S G F d H I = " > A A A B + 3 i c b V D L S g M x F L 3 j s 9 b X W J d u g k V w Y 5 m p g i 4 r b l x W 6 A v a o W Q y a R u a S Y Y k I 5 a h v + L G h S J u / R F 3 / o 1 p O w t t P X D h c M 6 9 y b 0 n T D j T x v O + n b X 1 j c 2 t 7 c J O c X d v / + D Q P S q 1 t E w V o U 0 i u V S d E G v K m a B N w w y n n U R R H I e c t s P x 3 c x v P 1 K l m R Q N M 0 l o E O O h Y A N G s L F S 3 y 0 1 b u u I y + E F Z 2 P 7 y E j K q O + W v Y o 3 B 1 o l f k 7 K k K P e d 7 9 6 k S R p T I U h H G v d 9 b 3 E B B l W h h F O p 8 V e q m m C y R g P a d d S g W O q g 2 y + + x S d W S V C A 6 l s C Y P m 6 u + J D M d a T + L Q d s b Y j P S y N x P / 8 7 q p G d w E G R N J a q g g i 4 8 G K U d G o l k Q K G K K E s M n l m C i m N 0 V k R F W m B g b V 9 G G 4 C + f v E p a 1 Y p / W a k + V M u 1 q z y O A p z A K Z y D D 9 d Q g 3 u o Q x M I P M E z v M K b M 3 V e n H f n Y 9 G 6 5 u Q z x / A
+ t R Y n c R D v 9 x E = " > A A A B / n i c b V D L S g M x F L 3 j s 9 b X q L h y E y y C G 8 t M F X R Z c O O y g n 1 A O 5 R M J t O G Z p I h y Q h l K P g r b l w o 4 t b v c O f f m L a z 0 N Y D F w 7 n 3 J v c e 8 K U M 2 0 8 7 9 t Z W V 1 b 3 9 g s b Z W 3 d 3 b 3 9 t 2 D w 5 a W m S K 0 S S S X q h N i T T k T t G m Y 4 b S T K o q T k N N 2 O L q d + u 1 H q j S T 4 s G M U x o k e C B Y z A g 2 V u q 7 x w 1 N s 0 g i L g c X n I 3 s O 0 M p o 7 5 b 8 a r e D G i Z + A W p Q I F G 3 / 3 q R Z J k C R W G c K x 1 1 / d S E + R Y G U Y 4 n Z R 7 m a Y p J i M 8 o F 1 L B U 6 o D v L Z + h N 0 Z p U I x V L Z E g b N 1 N 8 T O U 6 0 H i e h 7 U y w G e p F b y r + 5 3 U z E 9 8 E O R N p Z q g g 8 4 / i j C M j 0 T Q L F D F F i e F j S z B R z O 6 K y B A r T I x N r G x D 8 B d P X i a t W t W / r N b u a 5 X 6 V R F H C U 7 g F M 7 B h 2 u o w x 0 0 o A k E c n i G V 3 h z n
G d w E G R N J a q g g i 4 8 G K U d G o l k Q K G K K E s M n l m C i m N 0 V k R F W m B g b V 9 G G 4 C + f v E p a 1 Y p / W a k + V M u 1 q z y O A p z A K Z y D D 9 d Q g 3 u o Q x M I P M E z v M K b M 3 V e n H f n Y 9 G 6 5 u Q z x /
N i T T k T t G m Y 4 b S T K o q T k N N 2 O L q d + u 1 H q j S T 4 s G M U x o k e C B Y z A g 2 V u q 7 x w 1 N s 0 g i L g c X n I 3 s O 0 M p o 7 5 b 8 a r e D G i Z + A W p Q I F G 3 / 3 q R Z J k C R W G c K x 1 1 / d S E + R Y G U Y 4 n Z R 7 m a Y p J i M 8 o F 1 L B U 6 o D v L Z + h N 0 Z p U I x V L Z E g b N 1 N 8 T O U 6

Numerical experiments

We reproduce here some of the numerical experiments published in [42] and [150].

Comparing orders of truncations and different gradient strategies on binary RBMs

In a first set of experiments we focus on binary RBMs to validate the proposed mean-field deterministic training, for which we provide a public Julia package [START_REF] Tramel | Boltzmann.jl[END_REF]. We use two different data sets. The MNIST data set2 contains handwritten digit images labeled from 0 to 9. Each image is comprised of 28 × 28 pixels taking values in the range [0, 255]. The MNIST data set was binarized by setting all non-zero pixels to 1. The 28 × 28 pixel version of the Caltech 101 Silhouette data set3 consists of black regions of a primary foreground scene objects on a white background, labeled among 101 object classes according to the object in the original picture. Following previous studies evaluating RBMs on these data sets, we fix the number of RBM hidden units to 500. To compare RBM training approaches we do not include momentum nor adaptive learning rates here. When comparing learning procedures on the same plot, all free parameters of the training were set identically as reported in Table 4.1.

We test performances for mean-field trainings comparing the first-order (MF), second-order (TAP2), and third-order (TAP3) small weight expansions, with either persistent (prefix P) or reinitialized inference at each update. The inference algorithm is run for τ = 3 iterations for all mean-field trainings expect for one of them (P-TAP2-30) where we wait for τ = 30 iterations each time. For benchmark, we also train RBM models using CD-1, following the prescriptions of [START_REF] Hinton | A Practical Guide to Training Restricted Boltzmann Machines A Practical Guide to Training Restricted Boltzmann Machines[END_REF], and PCD, as implemented in [START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF]. On Figure 4.3a and 4.3b we report the evolution of the pseudo log-likelihood (left panels) and the second order mean-field likelihood (right panels). The ascent of the log-likelihood surrogates over training epochs is very similar across all methods. The mean-field trainings do maximize the pseudo-likelihood and, reciprocally, the approximate Monte Carlo trainings do maximize the TAP-likelihood. On MNIST, we compare the different orders of truncation of the mean-field expansion. As expected by design, the persistent TAP2 algorithm with 30 iterations of the inference (P-TAP2-30) achieves the best maximization of the TAP loglikelihood. However, P-TAP2, with only 3 iterations, achieves a very close performance, making it preferable for faster training. Moreover, we note that P-TAP2 demonstrates improvements in terms of pseudo-likelihood with respect to P-MF, in agreement with the previously noted disappointment of naive mean-field [START_REF] Welling | A new learning algorithm for mean field Boltzmann machines[END_REF][START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF] and the improvements brought by the consideration of the second order observed in toy examples [START_REF] Conrad | The limitations of deterministic Boltzmann machine learning[END_REF][START_REF] Hilbert | Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction[END_REF]. However, the P-TAP3 does not yield significantly better pseudo-likelihood results than P-TAP2. The third order term of the small weight expansion, given by (4.2) for fully connected binary BMs, does not feature the triangular diagrams for the bipartite RBM. As a result it sums over as many terms as the second order, but at a higher order in the small W iα -s, and is finally expected to be sub-leading. In the following we mainly focus on the second order expansion.

On Figure 4.4 we examine further the quality of the different trainings. The persistent chains in Figure 4.4a gives an idea of the performance of the RBM as a generative model. The PCD samples and the P-TAP2 marginal means are recognizable digits. While the chains extracted from a P-MF training are of poorer quality, with half of them featuring non-identifiable digits. Here we further confirm the necessity to include the second order of the expansion to achieve competitive training. On Figure 4.4b we also evaluate these RBM training algorithms through the prism of a downstream task. We consider the supervised classification, via logistic regression, of the two data sets from the learned hidden marginals a h (x) = p(t|x; θ), interpreted as learned features. The MNIST classification accuracy of the RBMs trained with the P-TAP2 algorithms is roughly equivalent with that obtained when using PCD training, while CD-1 training yields markedly poorer classification accuracy. The slight decrease in performance of CD-1 and TAP2 along the training might be emblematic of over-fitting by the non-persistent algorithms, although no decrease in the TAP-likelihood of the test set was observed. For the Caltech 101 Silhouettes data set, the classification task is a priori much more difficult. The persistent algorithms do not yield better results on this task. However, we observe that the performance of deterministic mean-field RBM trainings is at least comparable with both CD-1 and PCD.

Lastly, we note the computation times for each of these approaches. For a Julia implementation of the tested RBM training techniques running on a 3.2 GHz Intel i5 processor, the average wall times for fitting a single 100-sample batch normalized against the model complexity are 14.10 ± 0.97 µs/batch/unit for PCD, which uses only a single sampling step, 21.25±0.22 µs/batch/unit for PMF, 37.22 ± 0.34 µs/batch/unit for P-TAP2, and 64.88 ± 0.45 µs/batch/unit for P-TAP3, each of which use 3 inference iterations. Run times of the P-MF and P-TAP2 approaches are therefore comparable with PCD.

Annealed importance sampling and AdaTAP, pushing precision further?

On Figure 4.5a we compare the evolution for the previously considered likelihood surrogates: the TAP log-likelihood and the pseudo-likelihood, to the Annealed Importance Sampling (AIS) [START_REF] Radford | Annealed importance sampling[END_REF] estimation of the likelihood as implemented in [START_REF] Salakhutdinov | On the quantitative analysis of Deep Belief Networks[END_REF]. By considering Markov Chains at multiple temperatures, AIS produces an accurate, albeit costly, Monte Carlo approximation of partition functions. We find that during the considered mean-field training all three measures have the The AIS results are averaged over 100 runs. For each run, 14 500 intermediate distributions are generated between the initial and target distribution using the same schedule as in [START_REF] Salakhutdinov | On the quantitative analysis of Deep Belief Networks[END_REF]. The pseudo-likelihood is computed using a stochastic approximation over 100 of the 784 pixels. Hyperparameters of training are given in Table 4.1. (b) Left: Evolution of the pseudo-likelihood along the training of a binary RBM with 784 visible units and 500 hidden units, on the 5000 first images of binary-MNIST, with a learning rate of 0.001 and batches of size 100. Different number of iterations τ of the TAP inference (Algorithm 3) and adaTAP inference (Algorithm 4) are compared. In all cases, a damping of 0.5 is used. Right: Computation time for one iteration of the inference algorithm, as a function of the batch size. Time is reported in seconds for identical experimental settings.

same qualitative behavior as a function of epochs, indicating that they are consistent in evaluating or guiding the training.

In the experiments presented above we found no improvement in mean-field training with the inclusion of the third order of the expansion. We investigate here whether replacing the inference based on the truncated expansion by the theoretically more principled adaTAP makes a difference. On Figure 4.5b the different curves correspond to different strategies of estimation of the likelihood gradients, with either TAP (Algorithm 3) or adaTAP (Algorithm 4) inference, again for a binary RBM training. All methods yield comparable results in terms of training performance, except for adaTAP with only three iterations of inference, which shows a poorer performance. Meanwhile, we also compare computational costs of the inference procedures. The need for a matrix inversion for each batch element makes adaTAP 3 orders of magnitude slower than TAP. Unfortunately the greater cost of the more accurate adaTAP is not compensated here by gains in training quality. Yet we simply replaced the TAP solutions by the AdaTAP fixed points in the mean-field gradients derived from the TAP free energy. It would be interesting to test whether using the gradients of the adaTAP free energy would yield any different outcome in terms of performance. Still, the computational time of the method would remain an issue.

Efforts to increase the accuracy beyond the heuristic application of the TAP approximation are therefore not paid by great improvements. We thus conclude that, as far as proposing a tractable and efficient training algorithm for RBMs, the TAP inference is well indicated.

Beyond the binary RBMs

We now consider the mean-field training (now fixed at the second order) of real-valued RBMs. This extension allows to go beyond toy examples and consider the unsupervised learning of realvalued data sets. We use the MNIST data set without binarization. We keep binary hidden units and choose truncated Gauss-Bernoulli real-valued units to account for the image sparsity. We also consider the CBCL face database4 . It consists of both face and non-face 8-bit gray-scale 19 × 19 pixel images. For our experiments, we utilize only the face images, on which we train an RBM with binary hidden units and truncated Gaussian visible units. The detail of the corresponding update functions for mean-field inference can be found in Appendix C of [150]. The real-valued case necessitates to pay careful attention to a few numerical issues also discussed in Appendix B and C of [150]. In particular, they are the reason we consider distributions with bounded support. On the top row of Figure 4.6, we report the normalized TAP likelihood over successful trainings on both real-valued data sets.

In Appendix E of [150], we also demonstrate how the mean-field inference can be leveraged to train Deep Boltzmann Machines and report experiments for binary units trained on binary MNIST. The difference between DBMs and RBMs lies in the fact that beyond the log-partition, the data dependent terms in the likelihood and its gradients become intractable while they had closed-forms in the RBMs (4.55) and (4.56)- (4.58). This data-dependent terms are commonly approximated using the naive mean-field (i.e. first order) inference [START_REF] Salakhutdinov | Deep Boltzmann Machines[END_REF]. Instead we propose to consistently approximate the log-partition and the data-dependent term with the more powerful second-order TAP expansion. We report successful TAP likelihood maximization by the proposed algorithm. Yet DBMs are famously hard to train [START_REF] Salakhutdinov | On the quantitative analysis of Deep Belief Networks[END_REF][START_REF] Salakhutdinov | An Efficient Learning Procedure for Deep Boltzmann Machines[END_REF][START_REF] Montavon | Deep Boltzmann machines and the centering trick[END_REF][START_REF] Kyung | Gaussian-Bernoulli deep Boltzmann machine[END_REF], they require a layer-wise pre-training phase or the implementation of sophisticated tricks of weight tying. The proposed algorithm does not escape these limitations. Consequently, feed forward neural networks, VAEs [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF][START_REF] D J Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] and GANs [START_REF] Goodfellow | Generative Adversarial Networks[END_REF], are currently preferred to exploit the advantage of deep architectures in unsupervised learning.

Probing the TAP solutions

The evaluation of unsupervised learning models is an open direction of research [START_REF] Mehdi | Assessing Generative Models via Precision and Recall[END_REF]. The deterministic mean-field framework provides a tractable estimate of the likelihood, but also allows to further characterize the learning state of BMs by looking at the TAP solutions. These attractors of the TAP inference gives an intuition on the representational power of a GRBM with a given set of parameters.

Previously, we examined the visual quality of the learned marginal means at the end of training (Figure 4.4a). On of training there are very few unique fixed points for the different initializations. The RBMs are not able to grasp the variety in the training data and has learned some kind of average. As the training progresses there are more and more TAP solutions that populate more evenly the space spanned by the training data points. The GRBM learns different representations rendering the complexity of the data. This observation is consistent with the intuition that the weight magnitude growing during the training is playing the role of an effectively decreasing temperature, inducing a multiplication of the TAP solutions as observed for the SK model [START_REF] Mézard | Spin Glass Theory and Beyond[END_REF]. Although we do not enter such a phase in our experiment, we could imagine the case where each initialization will lead to a different fixed point. This regime can be interpreted as an overtraining where the GRBM is merely memorizing the training data.

On the bottom row of Figure 4.6 we provide a quantitative analysis. We report the number of TAP solutions and their corresponding TAP free energies for the training of the GRBMs on real-MNIST and CBCL. We observe a fast growth in the number of fixed points at the beginning of training, followed by a relative stabilization. Moreover we note that the free energies of the different TAP solutions are concentrated compared to the overall evolution of the uniform average along the training. This implies that a weighted average would not result in a very different estimation of the overall free energy. We have confirmed in numerical experiments that the mean-field free energy at second order allows to design an efficient training algorithm for BMs with hidden units. Moreover, the deterministic framework enables to track the progress of learning by choosing as an objective the tractable mean-field likelihood and by offering the TAP solutions as probes of the learned representations. There are no straightforward equivalent of these analyses using sampling. In the next Section, we show that mean-field BMs can also be advantageous in following applications, here as priors for Bayesian inference. Middle: Approximate posterior after mean-field inference in the RBM leading to an effective prior as seen by the outer loop of the algorithm. Right: Effective RBM factor graph incorporating information about the observations using the approximate posterior coming from the message passing on the CS problem.

Application to Bayesian reconstruction

We return here to the general problem of estimating an unknown signal x 0 from observations y through a channel p(y|x) discussed in Section 2.1.3. Under the Bayesian paradigm, a prior distribution on the signal p x (x) that incorporates available information about the typical signal is used to help the reconstruction. Given the recent progress of unsupervised learning, an idea is to take advantage of a set of samples of the signal and learn a generative model to serve as a prior in Bayesian reconstruction. A potential hurdle to the approach remains that the joint distribution defined by the prior may be very informative only at the cost of being intractable and difficult to articulate in the Bayesian inference. Here we demonstrate on the example of an RBM prior for Compressed Sensing (CS) how mean-field methods, here envisaged through Approximate Message Passing, allow the implementation of such a program. This project was led by E. W. Tramel, so that we only review the main concepts and results here, and refer to the publication [151] for further details. In [150], we also apply this strategy in a denoising problem.

Combining CS and RBM inference

Compressed sensing consists in the reconstruction of a sparse signal x 0 ∈ R N from noisy linear measurements y ∈ R M . It is a special case of the GLM introduced in Section 3.3.1, with a Gaussian channel

p out (y|F x) = M µ=1 N (y µ ; f µ x, ∆) , ( 4.59)
where we note F the measurement matrix (with rows f µ ) with i.d.d. Gaussian entries of zero mean and variance 1/N . Compared to the usual setting, we add here an RBM to model the correlations between the components of x resulting in the posterior

p(x|y) = 1 Z(θ) p x (x i )p out (y|F x) dt p t (t) e x W t (4.60)
corresponding to the factor graph on the left of Figure 4.8. Considering both problems separately the mean-field inference of the CS problem is possible following the AMP Algorithm 1, while the mean-field inference of the RBM is realized by Algorithm 3. Both of these iterative procedures output factorized approximate marginals of x. As a result they can heuristically be nested to design Algorithm 6, taking into account alternatively the observations y and the RBM prior information as schematized on the middle and right of Figure 4.8. In the Algorithm we have replaced the channel output functions following definitions 3.39-3.40,

g out (y µ , ω µ , V µ ) = (y µ -ω µ )/(V µ + ∆) , ∂ ω g out (y µ , ω µ , V µ ) = 1/(V µ + ∆). ( 4
Update V (t) µ = N i=1 F 2 µi C x i (t) , ω (t) µ = N i=1 F µi x(t) i - N i=1 F 2 µi C x (t) i (y µ -ω (t-1) µ )/(V (t-1) µ +∆) Update σ (t) i -1 = - M µ=1 F 2 µi /(V (t) µ + ∆) , λ (t) i = x(t) i + σ (t) i M µ=1 F µi (y µ -ω (t) µ )/(V (t) µ + ∆) run (inner loop) Initialize c x,(0) i = f x 2 (λ (t) i , σ (t) i , 0, 0) , a x,(0) i = f x 1 (λ (t) i , σ (t) i , 0, 0) Converge a x i , c x i ← Alg. 3(W , θ x , θ h , a x,(0) , c x,(0) ) . Update x(t+1) i = γ x(t) i + a x i , C x i (t+1) = γC x i (t) + c x i t = t + 1 until Convergence Output:
x

The input update functions used to initialize and run Algorithm 3 take as argument the mean-field parameters coming from the RBM and the CS inference. They are given by,

Z x i (λ i , σ i , A x i , B x i ) = dx p x (x) e -(x-λi) 2 /2σi e -A x i x 2 i /2+B x i xi , (4.62)
f x 1 (λ i , σ i , A x i , B x i ) = 1 Z x i dx x p x (x) e -(x-λi) 2 /2σi e -A x i x 2 i /2+B x i xi , (4.63) f x 2 (λ i , σ i , A x i , B x i ) = 1 Z x i dx x 2 p x (x) e -(x-λi) 2 /2σi e -A x i x 2 i /2+B x i xi . (4.64)
In the last line of the outer loop, the update of the means and variance xi and C x i involve a damping parameter γ between 0 and 1, which is found to stabilize the reconstruction in experiments. In the next section we review the performance of the proposed Algorithm for the reconstruction of MNIST digits.

Finally, we note that the construction of Algorithm 6 is heuristic and one could have considered for instance parallel updates rather than nested loops. The theoretical analysis of such an algorithm remains an open direction of research. In particular, future works should clarify whether, under the teacher-student scenario, a State Evolution can be derived and whether it is consistent with a replica computation.

Numerical experiments

To test the performances of the proposed strategy we focus on the MNIST data set featuring sparse images of handwritten digits. A vanilla AMP implementation of CS consists in assuming a sparse prior on the images and run the GLM Algorithm 1. We consider here such an implementation, where we use a Gaussian-Bernoulli prior with sparsity level ρ i computed for each of the 784 pixels by an empirical average over the training data. In this sense, the prior is not identical across components of the signal but it does not take into account correlations. This method will serve as a reference case for the following two other implementations with RBM priors.

The idea to exploit RBMs in CS applications was pioneered by [START_REF] Dremeau | Boltzmann Machine and Mean-Field Approximation for Structured Sparse Decompositions[END_REF] and [148], who trained binary RBMs (BRBM) using Contrastive Divergence to locate the support of the non-zeros entries of sparse signals, and combined it with an AMP reconstruction. The knowledge of the location of the non-zeros entries indeed allows to solve the CS problem as soon as their number K is matched by the number of measurements M , which is an information theoretic threshold. We consider this strategy albeit using the deterministic mean-field training for RBMs discussed at the beginning of this Chapter. Finally, we consider the strategy of combination derived in the previous Section with a GRBM with truncated Gauss-Bernoulli visible units. The hyperparameters of training can be found in [151].

In the reconstruction experiments we consider the testing samples of MNIST, which were not seen by the RBMs at learning time. We recall that the measurement matrix has i.i.d Gaussian entries with zero mean and variance 1/N . We choose a channel noise of ∆ = 10 -8 . We compare the performances of three approaches on Figure 4.9, as a function of the sparsity level ρ = K/N and the measurement ratio α = M/N . We observe an improvement of the reconstruction as the priors become more informative. In particular, for the GRBM, a near perfect reconstruction can be reached even below the 'information theoretic threshold'. Not only the support is correct, but the values of the pixels as well are well recovered below the oracle K = M . The parameters of the RBM learned on the training set of MNIST are pointing towards a reconstructed signal very close to the ground-truth signal even when the information available is strictly insufficient to determine it exactly. On the right of the Figure, we also provide images for the reconstruction of a given MNIST digit with the three different approaches as a function of the measurement ratio. Both reconstruction algorithms involving RBMs manage to clearly reconstruct a '6' for α ≈ 0.25ρ. The GRBM further offers a reconstructed signal visually better than the BRBM.

Here we took advantage of the mean-field inference on real-valued BMs to derive a practical algorithm incorporating a learned prior in a Bayesian reconstruction problem. When a set of training examples of the signal is available, such a strategy should enable great economies in terms of the new information to acquire to reconstruct new samples, pushing further the idea of compressed sensing. There is a growing body of works investigating such scenarios. In the last paragraph of the following perspectives we discuss why we expect mean-field approaches to be especially relevant here.

Perspectives

Statistical physics theory of Restricted Boltzmann Machines

We presented in this Chapter contributions leveraging mean-field approximations to exploit the representational power of Boltzmann machines with hidden units in practical machine learning applications. Meanwhile, the recent craze for deep learning also motivated theoretical studies of RBMs following the statistical physics tradition and using similar tools.

Studying an ensemble of RBMs with random parameters, Tubiana and Monasson [START_REF] Tubiana | Emergence of Compositional Representations in Restricted Boltzmann Machines[END_REF] evidenced different regimes of typical pattern of activations in the hidden units controlled by parameters as the sparsity of the weights, their strength (playing the role of an effective temperature) or the type of prior for the hidden layer. Their study contributes to the understanding of the conditions under which the RBMs can represent high-order correlations between input units, albeit without including data and learning in their model. Barra and collaborators [10,[START_REF] Barra | Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors[END_REF], exploited the connections between the Hopfield model and RBMs to characterize RBM learning understood as an associative memory. They characterize the retrieval phase of RBMs. Mézard [START_REF] Mézard | Mean-field message-passing equations in the Hopfield model and its generalizations[END_REF] also re-examined retrieval in the Hopfield model using its RBM representation, showing in particular that the addition of correlations between memorized patterns could still allow for a mean-field treatment at the price of a supplementary hidden layer in the BMs representation. This result remarkably draws a theoretical link between correlations in the training data and the necessity of depth in neural network models.

While the above results do not include characterization of the learning, a few others were able to discuss the dynamics of training. Huang [START_REF] Huang | Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses[END_REF] studied the Bayesian leaning of a RBM with a single hidden unit and binary weights. Decelle and collaborators [START_REF] Decelle | Spectral dynamics of learning in restricted Boltzmann machines[END_REF][START_REF] Decelle | Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics[END_REF] introduced an ensemble of RBMs characterized by the spectral properties of the weight matrix and derived the typical dynamics of the corresponding order parameters during learning driven by data. Finally, Barra and collaborators [START_REF] Barra | Phase transitions in restricted Boltzmann machines with generic priors[END_REF] empirically study a teacher-student scenario of unsupervised learning by maximum likelihood on samples of an Hopfield model which they can compare to their theoretical characterization of the retrieval phase.

Feed forward generative models

While BMs are a natural interest of statistical physicists given its relations to inverse Ising problems and the Hopfield model, current state-of-the-art in unsupervised learning relies on feed forward neural networks used as VAEs [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF][START_REF] D J Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] or GANs [START_REF] Goodfellow | Generative Adversarial Networks[END_REF]. These probabilistic models also define powerful but intractable probability distributions. Recent advances in mean-field theory and message-passing on multi-layer networks offer an interesting toolbox to design practical algorithms or develop a theoretical understanding of these models. Below, we give one example of practical application we started to investigate and comment on one direction of research of particular interest in our views.

An attempt at VAE training

Similarly to RBMs, feed forward generative models can be trained for density modelling by maximum likelihood. Again as for RBMs, this objective is nevertheless intractable and in practice approximated. In VAEs, a variational lower bound of the likelihood is parametrized by an auxiliary neural network, itself optimized in parallel of the training of the generative model. Instead the likelihood and its gradients could be approximated by mean-field methods allowing for a learning procedure similar to the RBM training algorithm presented in Section 4.2. Indeed the likelihood of the feed forward architecture is directly related to the Bethe free energy corresponding to the recently derived multi-layer AMP of [START_REF] Manoel | Multi-layer generalized linear estimation[END_REF]. Nevertheless, running preliminary experiments in this direction on a single layer VAE, we encountered numerical problems. After a few epochs of learning the AMP algorithm had convergence issues, probably due to building correlations in the weights [START_REF] Sundeep Rangan | On the convergence of approximate message passing with arbitrary matrices[END_REF]. Although not obvious at the time, it seems that the ML-VAMP algorithm [START_REF] Fletcher | Inference in Deep Networks in High Dimensions[END_REF] could alternatively be used to compute the TAP solutions bypassing this difficulty. In [START_REF] Pandit | Asymptotics of MAP Inference in Deep Networks[END_REF], the authors demonstrate a successful inference for an in-painting experiment in a deep generative model trained on MNIST as a VAE.

GANs and VAEs as learned priors for pratice and theory

In the line of the application presented in Section 4.3, several works have also investigated using feed forward generative models for inference tasks, and in particular for compressed sensing [START_REF] Bora | Compressed Sensing using Generative Models[END_REF][START_REF] Hand | Phase Retrieval Under a Generative Prior[END_REF][START_REF] Hand | Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk[END_REF][START_REF] Mixon | SUNLayer: Stable denoising with generative networks[END_REF]. In these works the inference is performed via gradient descent using back-propagation to compute gradients with respect to the input of the deep generative models. Using again the recently derived multi-layer version of AMP and VAMP, it would be interesting to replicate these propositions in the mean-field framework and compare empirical performances of the different algorithms.

A relevant property of these algorithms is that they can be characterized by State Evolution equations, paving the way to a theoretical teacher-student analysis of the reconstruction in these kind of settings, in the lines of the analysis performed for vanilla CS [START_REF] Krzakala | Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices[END_REF][START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF]. More generally, these multi-layer models offer indeed interesting possibilities to build scenarios with non-trivial priors remaining tractable in the mean-field framework. In the next Chapter, we put in application this observation to design model experiments of deep learning trainings. Recently, [START_REF] Aubin | The spiked matrix model with generative priors[END_REF] also took advantage of the method to extend the mean-field theory of low-rank matrix factorization to structured priors.

Mean-field inference for information theory in deep supervised learning

In networks made of thousands of neurons, we must find aggregated quantities to describe and hopefully explain the learning process. In particular, their generalization ability, remains a great puzzle [START_REF] Zhang | Understanding Deep Learning Requires ReThinking Generalization[END_REF]. Recently, it was proposed to consider the mutual information between random variables defined as the layers of a neural network [START_REF] Tishby | Deep Learning and the Information Bottleneck Principle[END_REF]. The idea stems from the Information Bottleneck principle [START_REF] Tishby | The information bottleneck method[END_REF] proposing an information based objective for learning useful representations in probabilistic graphical models. Yet the verification of its relevance in practice requires the computation of mutual information for high-dimensional variables, a notoriously hard inference problem. Pioneering works in this direction focused either on small network models with discrete (continuous, eventually binned) activations [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF], or on linear networks [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF]. Using mean-field inference we are able to approximate mutual informations for a class of deep neural networks trained in the teacher-student scenario. This strategy allows us to capture some of the key properties of the deep learning setting that are usually difficult to include in tractable frameworks: non-linearities, arbitrary large width and depth, correlations in the input data and trainable weights.

In the first Section of this Chapter we present the extension to multi-layer networks of the mean-field approximation for entropies of [START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF]. In the following Section we describe how the resulting formula can be leveraged to study learning in a teacher-student scenario, and examine numerical experiments. The ideas presented in these two first parts were published in [START_REF] Gabrié | Entropy and mutual information in models of deep neural networks[END_REF]. In a last Section we discuss more generally the potentialities and difficulties surrounding the information theoretic approach to deep learning theory, through presenting original numerical results and summarizing the debate that took place in the literature. This last part is un-published and results from a collaboration with Léon Bottou.

Mean-field entropy for multi-layer models

Extension of the replica formula to multi-layer networks

Multi-layer model

We consider the following statistical model involving a stochastic neural network of arbitrary depth ,

W (k) ∼ p (k) W (W (k) ) ∀k = 1, • • • (5.1)
t (0) = x ∼ p x (x) = N0 i=1 p x (x i ) (5.2) t (k) ∼ p (k) out (t (k) |W (k) t (l-1) ) = N l µ=1 p (k) out (t (k) µ |w (k) µ t (l-1) ) ∀k = 1, • • • (5.3)
where the weight matrices are independent samples from the orthogonal invariant ensemble. This means that in their singular value decomposition

W (k) = U (k) S (k) V (k) ∈ R N l ×N l-1
, the basis matrices are Haar distributed: drawn uniformly from the set of orthogonal matrices O(N l ) and O(N l-1 ). We further assume the ratios αk = N k /N 0 to be fixed when N 0 is varied. In the following we consider a fixed realization of the weight matrices, playing the role of quenched disorder. The model then defines a Markov Chain following the graphical model represented on Figure 5.1a. The distribution of a given variable t (k) is intractable since it requires to marginalize over the inputs and the preceding layers in the Markov Chain.

Replica formula

One-layer For a single layer, the described model corresponds to a GLM (see 3.3.1) of output t (1) with orthogonally invariant weight matrix, for which the posterior p(x|W (1) , t (1) ) given by Bayes t ( 1) (1) , W (2) , W (3) , t (3) ).

W (2) t (2) W ( ) t ( ) W (k) = U (k) S (k) V (k) (a) p (3) out (t (3) µ |w (3) t (2) ) µ = 1 • • • N 3 t (2) 1 t (2) 2 t (2) 3 p (2) out (t (2) µ |w (2) t (1) ) µ = 1 • • • N 2 t (1) 1 t (1) 2 p (1) out (t (1) µ |w (1) x) µ = 1 • • • N 1 x 1 x 2 x 3 x 4 p x (x i ) i = 1 • • • N (b)
rule is explicit up to an intractable partition function Z(t (1) , W (1) ). The corresponding quenched free energy, averaged over the realizations of the weight matrices, can however be approximated using a replica computation in the thermodynamic limit N 0 → +∞ following [START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF] as already discussed in Section 3.5.3. In the Bayes optimal setting, where the model generating the observations and the model used for reconstruction are identical, this free energy is in fact equal to the entropy of the output averaged over the disorder: Z(t (1) , W (1) ) = dx p out (t (1) |W (1) x)p x (x) = p(t (1) |W (1) ),

(5.4)

E W (1)
,t (1) log Z(t (1) , W (1) ) = E W (1) dt (1) p(t (1) |W (1) ) log p(t (1) |W (1) )

= -E W (1) H(t (1) |W (1) ) .

(5.5)

In the infinite-size limit the self-averaging property of the free energy ensures that the entropy for a given realization of the weight concentrates, this remains approximately true for large models. Hence the replica symmetric formula of [START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF] enables an asymptotic evaluation of H(t (1) |W (1) ). Note that there is no replica symmetry breaking in a Bayes optimal setting. Compare to [START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Shinzato | Perceptron capacity revisited: classification ability for correlated patterns[END_REF][START_REF] Shinzato | Learning from correlated patterns by simple perceptrons[END_REF] and Section 3.5.3 we start by rewriting the single layer free energy (3.81) as

f = lim N0→∞ 1 N 0 H(t (1) |W (1) ) (5.6) = -extr q,q,u,û - 1 2 q q + ûq 0 2 - αûu 2λ 0 + F ρ λ ,α (q 0 -q, û/λ 0 ) + I x (q) + αI z (q 0 λ 0 /α, u) (5.7) = extr A,V, Ã, Ṽ φ(A, V, Ã, Ṽ ) 1 (5.8) with φ(A, V, Ã, Ṽ ) = 1 2 Ã(q 0 -V ) -αA Ṽ + FW (AV ) -I x ( Ã) -αI z (q 0 λ 0 /α, Ṽ -q 0 λ 0 /α) where 2 FW (1) (x) = min θ 2αθ + (α -1) log(1 -θ) + E λ W log[xλ W + (1 -θ)(1 -αθ)] .
(5.9)

The replica symmetric potential can furthermore be restated in terms of a scalar mutual information and conditional entropy

φ(A, V, Ã, Ṽ ) = 1 2 -ÃV -αA Ṽ + FW (AV ) + I(x; x + ξ 0 √ A ) + H(t 1 | ξ1 ; Ṽ ; ρ) , (5.10)
where ξ 0 and ξ1 are standard Gaussian variables, ρ = λ 0 q 0 /α, x follows p x (x) and t 1 follows

p(t 1 |ξ 1 ; V, q 0 ) = D ξ p out (t 1 | √ q 0 -V + √ V ξ).

Multi-layer

We wish now to estimate the entropy at the next layer H(t (2) |W (1) , W (2) ). The replica potential for the reconstruction of t (1) knowing t (2) is according to the single-layer GLM formula

φ 2 (A 2 , V 2 , Ã2 , Ṽ2 ) = 1 2 -Ã2 V 2 -α 2 A 2 Ṽ2 + FW (A 2 V 2 ) + I(t 1 ; t 1 + ξ 1 √ A 2 ) + H(t 2 | ξ2 ; V 2 ; ρ2 ) ,
with ξ 1 and ξ2 standard Gaussian variables, ρ1 = λ 1 q 1 /α 1 ,

α 1 = N 1 /N 0 , λ 1 = E λ W (1) [λ W (1)
] and

q 1 = dt 1 p (1)
out (t 1 ). Using the identity I(x; y) = H(x) -H(x|y) and the analytical expression of the entropy of a Gaussian distribution we have

I(t 1 ; t 1 + ξ 1 Ã2 ) = H t 1 + ξ 1 Ã2 -H ξ 1 Ã2 = H t 1 + ξ 1 Ã2 - 1 2 log(2πe Ã-1 2 ).
(5.11)

Yet the variable t 1 is itself the output of a GLM so that the entropy H t 1 + ξ 1 / Ã2 is again given by the single layer formula simply with the additional noise ξ 1 . Applying this recursive heuristic argument leads to a replica formula with an additive potential over the layers, lim

N0→∞ 1 N 0 H(t ( ) ) = min extr A,V , Ã, Ṽ φ (A, V , Ã, Ṽ ), (5.12)
with four -dimensional vectors A, V , Ã, Ṽ , and

φ (A, V , Ã, Ṽ ) = I t 0 ; t 0 + ξ 0 Ã1 - 1 2 k=1 αk-1 Ãk V k + α k A k Ṽk -F W (k) (A k V k ) + -1 k=1 αk H(t k | ξk ; Ãk+1 , Ṽk , ρk ) - 1 2 log(2πe Ã-1 k+1 ) + α H(t | ξ ; Ṽ , ρ ), (5.13)
where

α k = N k /N k-1 , αk = N k /N 0 , ρ k = dt p (k-1) out (t) t 2 , ρk = E λ W (k) [λ W (k) ]ρ k /α k ,
and ξk -s are standard Gaussian variables. In the computation of the conditional entropies in (5.13), the scalar t k -variables are generated from p(t 0 ) = p x (t 0 ) and

p(t k | ξk ; Ã, Ṽ , ρ) = E ξ,z p (k) out (t k + ξ/ Ã| ρ -Ṽ ξk + Ṽ z), ∀k = 1, . . . , -1, (5.14) p(t | ξ ; Ṽ , ρ) = E z p ( ) out (t | ρ -Ṽ ξ + Ṽ z), (5.15)
where ξ and z are independent standard Gaussian random variables. Finally, the function F W (k) (x) depends on the distribution of the eigenvalues of W ( ) W ( ) following the definition (5.9). The heuristic procedure of derivation presented here gives an intuition for the result of the careful replica computation of [START_REF] Kabashima | Inference from correlated patterns: A unified theory for perceptron learning and linear vector channels[END_REF] applied to the multi-layer GLM factor graph of p(x|{W (k) }, t ( ) ) represented on Figure 5.1b. The computation of the entropy of the output of a multi-layer network in the large dimensional limit, a computationally difficult task, has thus been reduced to an extremization of a function of 4 variables, that requires evaluating single or bidimensional integrals. This extremization can be done efficiently by means of a fixed-point iteration starting from different initial conditions, as detailed in the Supplementary Material of [START_REF] Gabrié | Entropy and mutual information in models of deep neural networks[END_REF] and supported in the dedicated package [START_REF] Manoel | Deep Neural Networks Entropy from Replicas[END_REF]. Validity of the formula The formula (5.12) is conjecture to give an exact limit of the entropy density given weight matrices are independent samples of the orthogonally invariant ensemble. Coauthors in [START_REF] Gabrié | Entropy and mutual information in models of deep neural networks[END_REF] rigorously proved the replica formula in 2-layer networks, for the restricted ensemble of weight matrices with Gaussian i.i.d entries. Although a general proof remains today out of reach, there are reasons to believe that the presented proof can be extended to an arbitrary number of layers, still in the case of Gaussian matrices. Further credentials to the result of the replica computation are coming from its agreement with different heuristics. The saddle point equations associated with the extremization of the replica potential (5.13) can be shown to recover fixed points of the ML-AMP [START_REF] Manoel | Multi-layer generalized linear estimation[END_REF] and ML-VAMP [START_REF] Fletcher | Inference in Deep Networks in High Dimensions[END_REF] algorithms for inference in multi-layer GLMs. In a related contribution, Reeves [START_REF] Reeves | Additivity of information in multilayer networks via additive Gaussian noise transforms[END_REF] also proposed a formula for the mutual information between adjacent layers (closely related to the entropy of the output) in deep networks, using different information-theoretic arguments. It similarly exhibits layer-wise additivity, and the two formulas are conjectured to be equivalent.

Comparison with non-parametric entropy estimators

The proposed strategy to approximate the entropy in multi-layer networks exploits the knowledge of the underlying model (input distribution, weight matrices, stochastic activations etc.) and produces an estimator of great accuracy for large neural networks.

The approach of non parametric estimation is entirely different. It relies only on a set of samples drawn from the distribution of interest to evaluate its entropy. The kernel-density approach of Kolchinsky et. al. [START_REF] Kolchinsky | Nonlinear Information Bottleneck[END_REF] consists in fitting a mixture of Gaussians (MoG) to the samples and subsequently compute an upper bound on the entropy of the MoG [START_REF] Kolchinsky | Estimating mixture entropy with pairwise distances[END_REF]. The method of Kraskov et al. [START_REF] Kraskov | Estimating mutual information[END_REF] uses nearest neighbor distances between samples to directly build an estimate of the entropy. Both methods require the computation of the matrix of distances between samples. It is unfortunately computationally hard to test how these estimators behave in high dimension as even for a known distribution the computation of the entropy is intractable in most cases. The replica method proposed here offers a valuable point of comparison for cases where it is rigorously exact.

We place ourselves in the setting of the rigorous Theorem of [START_REF] Gabrié | Entropy and mutual information in models of deep neural networks[END_REF] to numerically test the different estimators (see Figure 5.2). The weight matrices entries are i.i.d. Gaussian with mean 0. Their standard-deviation is rescaled by a factor 1/ √ N and then multiplied by a coefficient σ varying between 0.1 and 10, i.e. around the recommended value for initialization in the training of deep neural networks. An additive white Gaussian noise of small variance (∆ = 10 -5 ) is added right before the activation function t (1) = f (W 1 X + 1 ) with 1 ∼ N (0, ∆I N ), (5.16)

t (2) = f (W 2 f (W 1 X) + 2 ) with 2 ∼ N (0, ∆I N ).
(5.17)

The non-parametric estimators [START_REF] Kolchinsky | Nonlinear Information Bottleneck[END_REF][START_REF] Kraskov | Estimating mutual information[END_REF] were evaluated using 1000 samples, as the cost of computing pairwise distances is significant in high dimension. The entropy estimate is stable over independent draws of a sample of this size (error bars smaller than marker size). On Figure 5.2, we compare the different estimates of H(t (1) ) and H(t (2) ) for different activation functions: linear, hardtanh or relu. The hardtanh activation is a piecewise linear approximation of the tanh, hardtanh(x) = -1 for x < -1, x for -1 < x < 1, and 1 for x > 1, for which the integrals in the replica formula can be evaluated faster than for the tanh.

In the linear and hardtanh case, the non-parametric methods are following the tendency of the replica estimate when σ is varied, but appear to systematically over-estimate the entropy. For linear networks with Gaussian inputs and additive Gaussian noise, every layer is also a multivariate Gaussian and therefore entropies can be directly computed in closed form (exact in the plot legend). When using the Kolchinsky estimate in the linear case we also check the consistency of two strategies, either fitting the MoG to the noisy sample or fitting the MoG to the deterministic part of the t ( ) and augment the resulting variance with ∆, as done in [START_REF] Kolchinsky | Nonlinear Information Bottleneck[END_REF] (Kolchinsky et al. parametric in the plot legend). In the network with hardtanh non-linearities, we check that for small weight values, the entropies are the same as in a linear network with same weights (linear approx in the plot legend, computed using the exact analytical result for linear networks and therefore plotted in a similar color to exact). Lastly, in the case of the relu-relu network, we note that non-parametric methods are predicting an entropy increasing as the one of a linear network with identical weights, whereas the replica computation reflects its knowledge of the cut-off and accurately features a slope equal to half of the linear network entropy (1/2 linear approx in the plot legend). While non-parametric estimators are powerful tools able to approximate entropies from the mere knowledge of samples, they inevitably introduce estimation errors. The replica method is taking the opposite view. While being restricted to a class of models, it can leverage its knowledge of the neural network structure to provide a reliable estimate. To our knowledge, there is no other entropy estimator able to incorporate this information about the underlying multi-layer model.

Mean-field information trajectories over training of deep networks

The information bottleneck theory [START_REF] Tishby | The information bottleneck method[END_REF] applied to neural networks consists in computing the mutual information between the data and the hidden learned representations on the one hand, and between labels and again hidden learned representations on the other hand [START_REF] Tishby | Deep Learning and the Information Bottleneck Principle[END_REF][START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF]. A successful training will maximize the information with respect to the labels it learns to predict. Tishby and collaborators argue that it should simultaneously minimize the information with respect to the input data. The idea is that by getting rid of unecessary information from the input to predict the output, deep neural networks would prevent overfitting and lead to a good generalization. While this intuition suggests new learning algorithms and regularizers [START_REF] Chalk | Relevant sparse codes with variational information bottleneck[END_REF][START_REF] Achille | Information Dropout: Learning Optimal Representations Through Noisy Computation[END_REF][START_REF] Alexander A Alemi | Variational information bottleneck[END_REF][START_REF] Achille | Emergence of Invariance and Disentangling in Deep Representations[END_REF][START_REF] Kolchinsky | Nonlinear Information Bottleneck[END_REF][START_REF] Ishmael Belghazi | MINE: Mutual Information Neural Estimation[END_REF][START_REF] Zhao | InfoVAE: Information Maximizing Variational Autoencoders[END_REF], we can also hypothesize that this mechanism is already at play in a priori unrelated commonly used optimization methods, such as the simple stochastic gradient descent (SGD), which already yield very good capacities of generalization. This fact was first claimed by [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF]. In this work, numerical experiments were run on very small neural networks, to allow for entropy estimation by binning of the hidden neurons activities. Afterwards, the authors of [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF] reproduced the results of [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF] on small networks using the continuous entropy estimator of [START_REF] Kolchinsky | Nonlinear Information Bottleneck[END_REF], but found that the overall behavior of mutual information during learning is greatly affected when changing the nature of nonlinearities. Additionally, these authors investigate the training of larger linear networks on i.i.d. normally distributed inputs where entropies at each hidden layer can be computed analytically for an additive Gaussian noise. The replica estimator we derived allows us to evaluate entropies and mutual informations in non-linear networks larger than in [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF][START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF], provided that the training data has a distribution belonging to a certain class and that weight matrices verify the assumptions of the formula.

65

x W (1) t (1) W ( ) t ( )

W ( +1) ŷ y z W (1)
t( 1)

W (p) t(p) W (p+1)

Tractable deep learning models

Revisited teacher-student scenario The multi-layer model presented above can be leveraged to simulate a prototypical setting of deep supervised learning on synthetic data sets amenable to the replica computation of entropies and mutual informations. We consider a teacher model generating input-output pairs {x, y} from a vector latent variable z with separable prior distribution p z (z) = N0 i=1 p z (z i ). The rule generating the target outputs y can be chosen arbitrarily (in blue on the Figure 5.3). However we impose that the model transforming z into x is a multi-layer neural networks with orthogonally invariant weight matrices (in black). This teacher model can be sampled from to generate a training data set. Subsequently, a student deep neural network (in red) is trained on the synthetic data set, under the weight constraint described below, to predict at its output ŷ the target y from the input x.

A more traditional teacher-student scenario would consist in a matched setting; where the teacher would be itself a feed forward neural network from the input x, following a simple prior distribution p x (x), to the target output y. The advantage of the proposed setting is that it allows to consider synthetic inputs x featuring some structure. In fact the generative network from z to x is corresponds to the common architecture of VAEs [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF][START_REF] D J Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] and GANs [START_REF] Goodfellow | Generative Adversarial Networks[END_REF]. Via unsupervised learning adjusting the weights { W (1) , • • • , W (p) } these generative models can generate samples mimicking real images.

Weight constraint At the start of a neural network training, weight matrices initialized as i.i.d. Gaussian random matrices satisfy the necessary assumptions of the replica formula. In their singular value decomposition W ( ) = U ( ) S ( ) V ( ) the matrices U ( ) ∈ O(N ) and V ( ) ∈ O(N -1 ), are typical independent samples from the Haar measure across all layers. To make sure weight matrices remain close enough to this assumption during learning, we define a custom weight constraint which consists in keeping U ( ) and V ( ) fixed while only the matrix S ( ) , constrained to be diagonal, is updated. The number of parameters is thus reduced from N × N -1 to min(N , N -1 ). We refer to layers following this weight constraint as USV-layers.

The relatively small number of degrees of freedom implies that USV-layers are harder to train than usual. In practice, we notice that they tend to require more parameter updates. Also, interleaving linear USV-layers to increase the number of parameters between non-linearities can significantly improve the final result of training. We conduct an experiment on the MNIST data set. The best train and test, losses and accuracies, for the different architectures are given in Table 5.1 and some learning curves are displayed on Figure 5.4. As expected we observe that USV-layers are achieving better classification success than the random projections, yet worse than the unconstrained fully connected layer. Stacking USV-layers to increase the number of trainable parameters allows to reach very good training accuracies, nevertheless, the testing accuracies do not benefit to the same extent from these additional parameters. On Figure 5.4, we observe that the version of the experiment with many (6) USV-layers overfits the training set (green curves with testing losses growing towards the end of learning). Adding explicit regularizers might allow to improve the generalization performances of models with USV-layers. This experiment confirms that USV-layers can learn to represent complex functions despite their restriction.

We further note that such a product decomposition is reminiscent of a series of works on 60000 training images and 10 000 testing images), featuring one non-linear (relu) hidden layer of 500 neurones and a softmax output layer. The 500 × 10 parameters of the weight matrix of the output layer are all being learned in all the versions of the experiments. Conversely, before the relu layer, we either (1) do not learn at all the 784 × 500 parameters which then define random projections, (2) learn all of them as a traditional fully connected network, (3) use a combination of 2 (3a), 3 (3b) or 6 (3c) consecutive USV-layers (without any intermediate non-linearity). For each version of the experiment the outcomes of two independent runs are plotted with the same color, it is not always possible to distinguish the two runs as they overlap. adaptive structured efficient linear layers (SELLs and ACDC) [START_REF] Moczulski | ACDC: A Structured Efficient Linear Layer[END_REF][START_REF] Yang | Deep Fried Convnets[END_REF] motivated this time by speed gains. In these layers only diagonal matrices are learned and the matrices U ( ) and V ( ) are permutations of Fourier or Hadamard matrices, so that the matrix multiplication can be replaced by fast transforms.

First layer type #train params

Following information during training

Combining the teacher-student approach with the learning of USV-layer in the student, we are able to evaluate the entropy H(t (k) ) at any layer of the student model during training. Indeed the Markov Chain starting at the latent variable z, running through the teacher generative model and the student network (see Figure 5.3) fulfills the assumptions of the multi-layer model defined in Section 5.1.1. In particular, the initial distribution p z (z) is separable.

The mutual information between two continuous random variables x and y with joint density 67 p x,y (x, y) and marginals p x (x) and p y (y) is defined by I(x; y) = dxdyp x,y (x, y) log p x,y (x, y) p x (x)p y (y) .

(5.18)

It quantifies the level of dependence of its two arguments. It is always positive and equal to zero if and only x and y are independent. The above definition can be considered as a generalization of the formula for discrete random variables following the definition of the differential entropy. An equivalent expression is I(x; y) = H(y) -H(y|x). The extension to continuous random variable may however be ill-defined as the previous integral can diverge.

To access mutual informations in deep neural networks, from the replica formula for entropies, we need to compute the conditional entropies between adjacent layers. Considering neural networks of large width, we use the central limit theorem to reduce their evaluation to a 2-dimensional integral

H(t (k) |t (k-1) ) = R N k dt R N k-1 dt p out t|W t p (k-1) (t ) log p out (t|W t ) (5.19) = N k dz N (z; 0, ρ(k) ) dh p out (h|z), (5.20)
from which we deduce

I(t (k) ; t (k-1) ) = H(t (k) ) -H(t (k) |t (k-1)
).

While we have defined our student model as a priori stochastic, traditional feed forward neural networks are deterministic. In the numerical experiments presented in the next Section, we train and test networks without injecting noise, and only assume a noise model in the computation of information-theoretic quantities. We choose to add a white Gaussian noise of small variance (∆ = 10 -5 ) right before the last activation function of the Markov Chain. This addition is necessary for conditional entropies, and consequently mutual informations, to remain finite. Our choice attempts to stay as close as possible to deterministic neural networks, yet remains inevitably somewhat arbitrary. This issue will be discussed in Section 5.3. Note that for the computation of H(t (k) ) the mapping between x and t (k-1) is kept deterministic, which allows to follow the mutual information with the input layer given the identity

I(t (k) ; x) = H(t (k) ) -H(t (k) |x) = H(t (k) ) -H(t (k) |t (k-1) ) = I(t (k) ; t (k-1) ).
(5.21)

Training experiments

Using a dedicated python package [START_REF] Gabrié | Learning Synthetic Data[END_REF], interfacing the package [START_REF] Manoel | Deep Neural Networks Entropy from Replicas[END_REF] for replica computations, we run training experiments on different instances of the deep learning models defined above. We seek to study the simplest possible training strategies achieving good generalization. Hence for all experiments we use plain stochastic gradient descent (SGD) with constant learning rates, without momentum and without any explicit form of regularization. The sizes of the training and testing sets are taken equal and scale typically as a few hundreds times the size of the input layer.

Linear trainings

In our first experiment we check on linear networks a potential caveat of our strartegy. For the replica formula to be correct, additionally to the restriction to USV-layers, the different matrices S ( ) should remain sufficiently uncorrelated during the learning. By focusing on linear networks trained on Gaussian i.i.d input data, the replica prediction can be compared to a closed form expression. We consider the training of a 4-layer linear network, on a synthetic data set of Gaussian inputs and outputs generated by a linear teacher (see Figure 5.5 for details). We repeat the training experiment for increasing layer width N , and report the maximum and the mean value of the squared error on the estimation of the I(x; t ( ) ) over all epochs of 5 independent training runs. We find that even if errors tend to increase with the number of layers, they remain objectively very small and decrease drastically as the size of the layers increases. Results therefore demonstrate (i) that the replica formula remains correct throughout the learning of the USV-layers and (ii) that the replica method gets closer and closer to the exact result in the limit of large networks, as The noise follow sN ( ; 0, 0.01I N ). The student network has 3 USV-layers, plus one fully connected unconstrained layer x → t (1) → t (2) → t (3) → ŷ. The training on the regression task is done using plain SGD for the MSE loss (ŷ -y) 2 . Upper-left: MSE loss on the training and testing sets during training by plain SGD for layers of size N = 1500. Best training loss is 0.004735, best testing loss is 0.004789. Lower-left: Corresponding mutual information evolution between hidden layers and input.

Center-left, center-right, right: maximum and squared error of the replica estimation of the mutual information as a function of layers size N , over the course of 5 independent trainings for each value of N for the first, second and third hidden layer.

x S (1) t

S

S

S (4) t

W (5) ŷ of size N 0 = 100, data x of size N 1 = 500 generated with matrix W gen with i.i.d. normally distributed entries of variance as 1/N 0 and noise ∼ N ( ; 0, 0.01I N 1 ). We then train a student to solve the binary classification problem of recovering the label y = sign(z 1 ) (in blue). Note that the rest of the initial code (z 2 , ..z N 0 ) acts as noise/nuisance with respect to the learning task. The dimensions of the student layers (in red) are 500-1000-500-250-100-2, with 4-USV layers with only matrices S (l) as trainable parameters and one unconstrained fully-connected layer before the output.

y z W (1)
theoretically predicted (5.12). In the following, we assume that it is also the case for non-linear networks.

Concerning the application of the information-bottleneck ideas to deep learning, a similar experiment of linear network training was studied in [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF]. In agreement with their observations, we find that the mutual informations I(x; t ( ) ) keep on increasing throughout the learning (bottom left pannel of Figure 5.5), without compromising the generalization ability of the student.

Non linear trainings

Finally, we apply the replica formula to estimate mutual informations during the training of nonlinear networks on correlated input data. We design a synthetic problem of classification described on Figure 5.6. In the following we vary activation functions and weight initializations for the student of fixed architecture. For training, we use plain SGD but this time to minimize the cross-entropy loss 1 . We start with the comparison of two students with activations either linear-relu-linear- relu-softmax 2 (top row of Figure 5.7) or linear-hardtanh-linear-hardtanh-softmax (bottom row).

Because USV-layers only feature O(N ) parameters instead of O(N 2 ) we observe that they require more iterations to train in general. In the case of the relu network, adding interleaved linear layers was key to successful training with 2 non-linearities, which explains the somewhat unusual architecture proposed. For the student model using hardtanh, this was actually not an issue (see the following experiment), however, we consider a similar architecture for fair comparison. This experiment is reminiscent of the setting of [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF], yet now tractable for networks of larger sizes. For both types of non-linearities we observe that the mutual information between the input and all hidden layers decrease during the learning, except for the very beginning of training where we can sometimes observe a short phase of increase (see zoom in insets). For the hardtanh layers this phase is longer and the initial increase of noticeable amplitude. In this particular experiment, the claim of [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF] that compression can occur during training even with non double-saturated activation seems corroborated (a phenomenon that was not observed by [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF]). Yet we do not observe that the compression is more pronounced in deeper layers and its link to generalization remains elusive.

For instance, we do not see a delay in the generalization w.r.t. training accuracy/loss in the student model with hardtanh despite of an initial phase without compression in two layers. Futhermore, we find that changing the weight initialization can drastically change the behavior of mutual informations during training while resulting in identical training and testing final performances. In an additional experiment, we consider a similar setting to the classification on correlated data presented above. On Figure 5.8 we compare 3 identical students with activations hardtanh-hardtanh-hardtanh-hardtanh-softmax. Initial weight entries are sampled from a zeromean Gaussian with variance 4/N -1 (for the model presented at the top row), 1/N -1 (middle row), and 0.25/N -1 (bottom row). The first column shows that training is delayed for the weight initialized at smaller values, but eventually catches up and reaches accuracies superior to 0.97 both entry at the position of the target class. The trained neural network has an ouput ŷ ∈ [0, 1] M , with components summing to one, interpreted as a probability distribution over the classes. The cross-entropy loss is then defined as (y, ŷ) = -M µ=1 yµ log ŷµ. 2 The softmax activation precisely allows the interpretation of the output as a probability vector. For a pre- in training and testing. Meanwhile, mutual informations have different initial values for the different weight initializations and follow very different paths. They either decrease during the entire learning, or on the contrary are only increasing, or actually feature an hybrid path. We further note that it is to some extent surprising that the mutual information would increase at all in the first row if we expect the hardtanh saturation to instead induce compression.

activation vector z ∈ R M , ŷ = softmax(z) = e z / M µ=1 e zµ .
These observed differences and non-trivial observations raise numerous questions, and suggest that within the examined setting, a simple information theory of deep learning remains out-ofreach. In the next Section, we will discuss more thoroughly hurdles in the success of this strategy and in particular the necessity of a noise regularization of the mutual information. To conclude this Section, we check that qualitative behaviors discussed above are robust to a modification of the amplitude of the injected noise a posteriori. In all the previous experiments we had added a Gaussian noise of variance ∆ = 10 -5 to regularize the computation of the mutual information.

Here we verify on still the same problem described on Figure 5.6, with hardtanh activations, that going to smaller levels of noise does not change the qualitative picture. On Figure 5.9, we compare the mutual information evolution during learning while varying ∆. As ∆ decreases we observe that the average level of mutual information is increasing as expected (it should diverge at zero noise), yet the relative evolution is qualitatively unchanged.

( ) z ( ) ( ) z( ) f ( ) (•) t ( ) W ( +1)

Further investigation of the information theoretic analysis of deep learning

While a few experimental works claimed to have verified that SGD training of deep neural networks leads to compression of the information in hidden layers [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF][START_REF] Noshad | Scalable Mutual Information Estimation using Dependence Graphs[END_REF], others have nonetheless pointed out caveats in the application of these information theoretic ideas to the traditional supervised deep learning setting [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF][START_REF] Goldfeld | Estimating Information Flow in Neural Networks[END_REF][START_REF] Kolchinsky | Caveats for information bottleneck in deterministic scenarios[END_REF][START_REF] Gabrié | Entropy and mutual information in models of deep neural networks[END_REF]. As already mentioned, the key issue lies in the definition of the mutual information in deterministic neural networks.

In a feed forward deep neural network used for supervised learning the mapping between the entry x and the -th hidden layer t ( ) is typically deterministic. Consequently the conditional density p(t ( ) |x) is a Dirac with entropy H(t ( ) |x) diverging to -∞ and, as noted by [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF][START_REF] Goldfeld | Estimating Information Flow in Neural Networks[END_REF], I(x; t ( ) ) is effectively +∞. An intuitive way to attempt to regularize this mutual information is to quantize the continuous random variables and resort to the definition in the discrete case, which is always well defined. This is the strategy adopted by [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF]. Nevertheless [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF] showed that for the same training run a compression phase could be either present or absent when using different conventions for partitioning continuous intervals. Hence the regularization of the mutual information a posteriori of learning does not yield robust observations.

Conversely we can examine the training of neural networks for which mutual information are directly well defined. While discrete activations are problematic to study gradient based training algorithms, the addition of noise in a continuous neural network during the learning phase is a common form of regularization to promote generalization that also avoids the divergence of mutual information.

Mutual information in noisy trainings

We consider the addition of noise in the layers of neural networks during training aiming at regularization. The intuition behind this strategy is to force the learning towards a solution that is robust to variations around the training samples and therefore will also be effective on unseen test samples. Dropout [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF][START_REF] Srivastava | Dropout: A Simple Way to Prevent Neural Networks from Overfitting[END_REF] and batch-normalization [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF] are two specific protocols of noise injection during training that lead to significant improvements of performances of deep learning models and are widely adopted by practitioners. They involve both multiplicative and additive noises. We imagine a small experiment with different stochastic deep neural networks and a non-parametric estimator of mutual information. Resorting to the latter, rather than the replica formula considered in Section 5.1, allows to relax the restrictions to synthetic data sets and USV-layers, yet we will see at the cost of a loss in sensitivity. We describe how the estimator of [START_REF] Kolchinsky | Estimating mixture entropy with pairwise distances[END_REF] can be used to compute information observables for a variety of noises, before examining numerical results on the MNIST data set.

Non-parametric estimation

We assume that noise is added at each layer to the pre-activations denoted z ( ) , in the form of a standard Gaussian white noise ( ) scaled by a function of z ( ) . Formally we have the recursion z ( ) = W ( ) t ( -1) , z( ) = z ( ) + n(z ( ) ) ( ) , t ( ) = f (z ( ) ), (5.22) where the function n(•) fixing the amplitude of the noise and the activation f (•) are applied component-wise (see Figure 5.10). We can compute an upper and lower bound of I(x, z( ) ) following [START_REF] Kolchinsky | Estimating mixture entropy with pairwise distances[END_REF][START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF] from empirical samples. The injected noise need not be additive Gaussian and these bounds can be derived for an arbitrary noise scaling function. We consider a set of P training examples {x (k) } P k=1 , that are transformed by the neural network up to the layer into z ( ) (k) . The distribution of the following noised pre-activation z ( ) can be interpreted as a mixture of distributions with centroids {z

( ) (k) } P k=1 , p z( ) (z ( ) ) = 1 P P k=1 p z( ) |z ( ) (k) = 1 P P k=1 N i=1 N z( ) i ; z ( ) k,i , n(z ( ) k,i ) 2 .
(5.23)

Following [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF] we therefore use the mutual information estimator for mixtures of [START_REF] Kolchinsky | Estimating mixture entropy with pairwise distances[END_REF],

Î(z ( ) ; z( ) ) = -

P k=1 1 P log P k =1 1 P exp -D p(z ( ) |z ( ) (k) ) || p(z ( ) |z ( ) (k) ) , ( 5.24)
where D is a generalized distance between probability densities. We further use the results of [START_REF] Kolchinsky | Estimating mixture entropy with pairwise distances[END_REF] showing that the estimator is an upper bound when using as a distance the Kullback-Leibler divergence, and is a lower bound for the Chernoff α-divergence. Plugging-in our notations and the definitions of distances we obtain the following expressions, for the upper bound Îup (z ( ) ; z( ) ) = -

P k=1 1 P log P k =1 exp - N i=1 log n(z ( ) k,i ) n(z ( ) k ,i ) + 1 2 n(z ( ) k,i ) 2 n(z ( ) k ,i ) 2 -1 (5.25) + (z ( ) k,i -z ( ) k ,i ) 2 2n(z ( ) k ,i ) 2
, and for the lower bound Îlow (z ( ) ; z( ) ) = -

P k=1 1 K log P k =1 exp - N i=1 1 4 (z ( )
k,i -z ( ) k ,i ) 2 n(z ( ) k ,i ) 2 + n(z ( ) k ,i ) 2 (5.26) + 1 2 log 1 2 n(z ( ) k ,i ) 2 + n(z ( ) k ,i ) 2 |n(z ( ) k ,i )n(z ( ) k ,i )|
where we considered the Chernoff 1/2-divergence for the lower bound.

We will only consider noises with zero mean so that, along the feed forward pass, the successive linear mixing of the products with weight matrices approximately makes z ( ) a deterministic function of x (that is the transformation of x by the neural network without the noise injection). Hence we expect, following the same argument as (5.21), that I(z ( ) ; z ( ) ) I(z ( ) ; x). Beyond this heuristic argument, the data processing inequality [START_REF] Cover | Elements of Information Theory[END_REF] ensures that I(z ( ) ; z ( ) ) ≥ I(z ( ) ; x), so that a drop of the first mutual information is reflected in the second if they are of comparable order of magnitude.

Thus we have a strategy to estimate mutual informations between layers from a set of P samples. This allows to consider 'real' data sets, of natural images for instance, and arbitrary architectures and training algorithm. However, the non-parametric strategy will suffer from two limitations. First, its approximation of the density as a mixture over the P samples limits its sensitivity to large mutual informations Î(z ( ) ; z( ) ) ≤ H(z ( ) ) = - The bound is saturated when the noised representations z( ) of the different samples are well separated, as also noted by [START_REF] Goldfeld | Estimating Information Flow in Neural Networks[END_REF]. Otherwise the estimator (5.24) examines the overlaps of the mixture components from the pairwise distances to evaluate the mutual information. This points at the second limitation of the procedure: the memory (and computational) burden in P 2 . While a large number of samples P is desirable to increase the sensitivity of the estimator upper-bounded by log P , the cost scales quadratically. This issue worsens as the dimension of variables increase as overlaps are typically vanishing.

Numerical experiments on MNIST

We run experiments training deep neural networks to solve the benchmark task of classifying the handwritten digits of the MNIST data set. We adopt a unique architecture with 5 hidden layers and 20 fully connected neurons in all hidden layers, with either tanh or relu activations, and 10 output softmax neurons to predict the 10 digit classes. The trainings are done with mini-batch stochastic gradient descent (batch-size of 100 and learning rate of 0.01) without other forms of regularization besides the noise injection under study. We consider both additive and multiplicative noises via different noise scaling functions: either constant n(z i ) = σ 0 , linear n(z i ) = σ 0 |z i |, quadratic n(z i ) = σ 0 z 2 i , or exponential n(z i ) = σ 0 (e |zi| -1). We select the optimal value for the parameter σ 0 in terms of the best accuracy achieved on the MNIST test data set, and report the values of the upper bound and lower bounds (5.25)-(5.26) for the last hidden layer during the training on Figure 5.11 for tanh activations and Figure 5.12 for relu activations. For stochastic deep networks with tanh activation, we observe a clear compression during the learning for multiplicative noises, and a slight compression in a second phase of training for the constant noise. Unfortunately, for the unbounded relu activation the estimator hits the limitation of the non-parametric approach induced by the size of the sample P = 5000. Increasing the sample size comes with costs scaling as P 2 while only yielding a sensitivity growing as log(P ). Therefore, the experiment is non-conclusive for relus: there may or may not be compressions happening above the log(P ) bound. We are unable to further comment on the necessity for saturated activations to observe compressions, but we further investigate the case of the tanh activations.

Saxe and collaborators [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF] argued that the loss of information in stochastic tanh networks results form the interplay of the saturation of the tanh activation and the blurring induced by noise. During neural network training, the weight magnitudes, initialized at small values, typically grow and with them the pre-activations as well. In turns, the amplitude of multiplicative noises follows this evolution. Therefore the greater information drops observed for the latter types of noises are expected. Similarly the initial phase of increase of the mutual information for the constant noise can be understood as the improvement of an effective signal-to-noise ratio as the pre-activations grow at the beginning of training while the noise is fixed.

To assess whether this geometric interpretation of information trajectories is sufficient we also report mutual informations for a network of similar architecture but with Gaussian random weight matrices of same variance. Thereby we test whether the mere increase of the weights amplitude explains the compression. For all noises, the random network transmits more information than the trained one, and the upper bound of the estimator is almost always saturated for the random network. Hence, the compression seen during training may be due to saturations yet 'organized' by the learning.

Discussion

The concepts and preliminary results reported by [START_REF] Tishby | Deep Learning and the Information Bottleneck Principle[END_REF][START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF] have spurred a significant interest in the deep learning community to the information theoretic analysis of neural network training. Nevertheless, the observables proposed as relevant -mutual informations between hidden layers and inputs or outputs -are ill-defined in deterministic neural networks that are commonly used for prediction in supervised learning. The application of the information bottleneck concepts to learning of purely deterministic rules also suffer some caveats investigated by [START_REF] Kolchinsky | Caveats for information bottleneck in deterministic scenarios[END_REF]. Otherwise, it is possible to regularize the definition of mutual informations by discretization [START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF] or by assuming noisy activations [START_REF] Kolchinsky | Nonlinear Information Bottleneck[END_REF][START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF]. Yet the outcome of experiments sometimes depends on this arbitrary choice of regularization [START_REF] Andrew M Saxe | On The Information Bottleneck Theory of Deep Learning[END_REF]. It was argued that the observed compressions in this context can nonetheless be interpreted as a measure of the clustering of the representations learned by the network [START_REF] Goldfeld | Estimating Information Flow in Neural Networks[END_REF]. Indeed, using discretization or the non-parametric estimate with added noise, we understand that overlaps in the hidden activation patterns of different inputs induce a drop of mutual information. The choice of characteristic strength of the regularization (bins spacing, noise amplitude etc.) then controls the scale of the clustering being probed. The above cited works could not observe compressions in this setting for unbounded activations, but we note that their estimator suffers from a limited sensitivity to high mutual information and interesting phenomenology may be hidden above the limiting upper bound. In particular, in Section 5.2.2, we report compressions during the training of relu networks on synthetic data sets, using the mutual information estimator based on the replica method. The issue of definition can also be circumvented by considering neural networks in the learning phase regularized by the addition of noise. On the example of the MNIST classification, using the non-parametric estimator [START_REF] Kolchinsky | Estimating mixture entropy with pairwise distances[END_REF], we observed that tanh activated network display varying level of compression but we are not able to draw conclusions for the case of relu activations.

Therefore we are recalled to the complexity of the estimation of mutual informations that appears as a serious obstacle to the validation of a theory of deep learning based on information theoretic principles. Non parametric methods are generic strategy of evaluation yet they are found insufficient to investigate all the interesting cases and suffer in particular when the dimension of the problem (i.e. the size of the hidden layers) increases. Using advanced methods from statistical physics we were able to follow precisely information trajectories for certain models of classification experiments -for different non-linear activations and for hidden layer of high dimensions. Still, these models are constrained to have weight matrices with a reduced number of degrees of freedom, while the concept of over-parametrization emerges as a key component to the success of deep learning (see e.g. [START_REF] Spigler | A jamming transition from under-to over-parametrization affects loss landscape and generalization[END_REF][START_REF] Geiger | Scaling description of generalization with number of parameters in deep learning[END_REF] for a discussion of the state-of-understanding and recent results). Another a priori promising non-parametric estimator, MINE [START_REF] Ishmael Belghazi | MINE: Mutual Information Neural Estimation[END_REF], relies on the optimization of a deep neural network. Yet its accuracy in high-dimension remains unverified. Albeit the replica formula could offer an interesting point of comparison in cases where it is known to be exact, it seems difficult to obtain convincing guarantees for the MINE estimator while it relies on neural network optimization. Our aim would precisely be to use it to explain what remains to this day mysterious in deep learning.

The concept of progressive invariance learned by the hidden representation remains a key intuition of the community to justify the advantage of deep neural networks. Yet the information theoretic lense as formulated by [START_REF] Tishby | Deep Learning and the Information Bottleneck Principle[END_REF][START_REF] Shwartz | Opening the Black Box of Deep Neural Networks via Information[END_REF], which could have yielded a quantitative verification, incurs conceptual and practical problems hard to resolve. A related yet different proposition was made by [START_REF] Song | Resolution and relevance trade-offs in deep learning[END_REF], with interesting results for the unsupervised learning of a certain class of deep neural networks with binary units. It remains to understand if the theory can be robustly generalized and tested on real valued feed forward neural networks used in supervised learning.

Towards a model for deep Bayesian (online) learning

One of the great successes of the statistical physics approach to neural networks learning theory initiated by Gardner [START_REF] Gardner | Maximum Storage Capacity in Neural Networks[END_REF] is the prediction of learning curves in the teacher-student scenario. It corresponds to the computation of the optimal generalization errors under the Bayesian learning paradigm as a function of parameters of the model such as the size of the training set or the level of noise in the teacher mapping from input to outputs. Following early works on the single layer perceptron, variations of a simple two-layer architecture were also considered: the (tree) committee machine [START_REF] Timothy | The statistical mechanics of learning a rule[END_REF][START_REF] Monasson | Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Networks[END_REF][START_REF] Opper | Mean field approach to bayes learning in feed-forward neural networks[END_REF][START_REF] Engel | Statistical Mechanics of Learning[END_REF][START_REF] Monasson | Learning and Generalization Theories of Large Committee-Machines[END_REF][START_REF] Aubin | The committee machine: Computational to statistical gaps in learning a two-layers neural network[END_REF]. In this model, only the first layer connecting the inputs to the hidden units is trained while the second layer has fixed weights. A current challenge in learning theory is to conduct similar analyses for neural networks with learnable parameters distributed in multiple layers.

To this aim, the possibility of decomposing complicated inference tasks into simpler well-known blocks is a promising lever. Let us rapidly examine the supervised learning of a two-layer feed forward neural network. A training set of P input-output pairs {x (k) , y (k) } P k=1 is generated using two weight matrices and noisy activation functions:

∀k = 1 • • • P, x 0,(k) ∼ p x0 (x (k) ) (6.1)
y (k) = g(W 0 (2) g(W 0 (1) x (k) ; 1 ); 2 ), 1,2 ∼ p ( 1,2 ) (6.2) with dimensions x 0,(k) ∈ R N , y (k) ∈ R Q , W 0 (1) ∈ R M ×N , W 0 (2) ∈ R Q×M .
Introducing the intermediate layer variables t ∈ R M , and the matrix notations X ∈ R N ×P , T ∈ R M ×P and Y ∈ R Q×P gathering the P samples, we can decompose the student model as Y = g(W (2) T ; 2 ) factorization to recover T and W (2) from observed Y , (6.3) T = g(W (1) X; 1 ) vectorized GLM to recover W (1) from observations T . (6.4)

The GLM model is rather straightforward to solve and combine following the recent multi-layer developments of AMP [START_REF] Manoel | Multi-layer generalized linear estimation[END_REF] and VAMP [START_REF] Fletcher | Inference in Deep Networks in High Dimensions[END_REF]. However, the matrix factorization problem, arising from the ignorance of the hidden states as the first layer weights are unknown, is more complicated. At finite rank, it can be solved by message passing and rigorously characterized in the thermodynamic limit [START_REF] Thibault Lesieur | MMSE of probabilistic lowrank matrix estimation: Universality with respect to the output channel[END_REF][START_REF] Thibault Lesieur | Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications[END_REF][START_REF] Kabashima | Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization[END_REF], which corresponds in our problem to a number M of hidden units remaining finite as the other dimensions are growing to infinity. Assuming few hidden units should therefore allow us to analyse the learning in the teacher-student scenario of this two-layer neural network by composing the low-rank matrix factorization analysis with a GLM. However, to be able to consider a number of hidden units of same order as the number of inputs and outputs, which is a situation closer to the practical usage of deep neural networks, a surrogate to the analysis of high-rank matrix factorization is needed. We formulate here a proposition towards this goal.

In this Chapter, we present in a first Section new results for GLMs with calibration variables. In a second Section we illustrate the newly derived algorithms and analyses with the example of gain calibration. Finally, we propose in a last Section a direction of research that should allow us to leverage these advances to treat in a constrained setting a high rank matrix factorization and the learning of multi-layer neural networks.

Cal-AMP revisited

We recall the blind calibration problem for the GLM already discussed in Chapter 3 Section 3.5.2. We have access to P observations y (k) ∈ R M gathered in a matrix Y ∈ R M ×P . They were generated from the P unknown signals x 0,(k) R N , similarly noted X 0 ∈ R N ×P , linearly mixed by a known weight matrix W , and pushed through a noisy channel denoted p s out,0 (equivalent to a stochastic Factor graph for the GLM on vector variables with not perfectly known channel including calibration variables, corresponding to the joint distribution (6.9).

p out (y µ |w µ X) µ = 1 • • • M x 1 x 2 x 3 x 4 p x (x i ) i = 1 • • • N (a) x 1 x 2 x 3 x 4 p x (x i ) i = 1 • • • N s 1 s 2 p s (s µ ) µ = 1 • • • M p s out (y µ |w µ X, s µ ) µ = 1 • • • M (b) .
activation g 0 (•) further down this Section). The channel is not perfectly known and depends on the realization of a calibration s 0 ∈ R M . Under the teacher model including prior distributions,

s 0 ∼ p s0 (s 0 ) (6.5) ∀k = 1 • • • P, x 0,(k) ∼ p x0 (x 0,(k) ) = N i=1 p x0 (x 0,i,(k) ) (6.6
)

y (k) ∼ p s out,0 (y (k) |W x 0,(k) , s 0 ) = M µ=1 p s out,0 (y (k),µ |W x 0,(k) , s 0,µ ) (6.7)
we are interested in the estimation of the unknown signal and calibration variable under a student model for priors and channel,

p(X, s|Y , W ) = 1 Z(Y , W ) p x (X)p s (s)p s out (Y |W X, s) (6.8) = 1 Z(Y , W ) N i=1 p x (x i ) M µ=1 p s (s µ )p s out (y µ |w µ X, s µ ), (6.9)
with x i ∈ R P , and y µ ∈ R P . Note that the distribution could be further factorized over the index k of the P observations given the teacher model. The corresponding message passing was derived in [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF]. Here we instead restrict to the level of factorization above and consider the AMP algorithm on vector variables.

Derivation through AMP on vector variables AMP for reconstruction of multiple samples

Before specializing to the calibration problem we are interested in, we present the AMP algorithm on vector variables. Without the calibration variable, the posterior measure we are interested in is

p(X|Y , W ) = 1 Z(Y , W ) N i=1 p(x i ) M µ=1 p out (y µ |w µ X/ √ N ), x i ∈ R P , y µ ∈ R P . (6.10)
where the known entries of matrix W are drawn i.i.d from a standard normal distribution (note that the scaling in 1/ √ N is here made explicit). The corresponding factor graph is given on Figure 6.1a. This setting was actually recently treated for a model of the committee machine in [START_REF] Aubin | The committee machine: Computational to statistical gaps in learning a two-layers neural network[END_REF]. The major difference with the fully factorized version on scalar variables is that we consider covariance matrices between variables coming from the P observations instead of assuming complete independence. We shall recall here the main steps of the derivation, which will be useful both in the following derivation of the State Evolution, and in the presentation of the derivation of the AMP algorithm in combined models.

Belief propagation

The belief propagation algorithm is identical to the algorithm presented in Section 3.3.2, except that messages are functions of variables in R P :

m(t) µ→i (x i ) = 1 Z µ→i i =i dx i p out   y µ | j W µj √ N x j   i =i m (t) i →µ (x i ) (6.11) m (t+1) i→µ (x i ) = 1 Z i→µ p x (x i ) µ =µ m(t) µ →i (x i ). (6.12)
To improve readability we drop the time indices in the following derivation, and only specify them in the final algorithm.

Relaxed BP We will now develop messages keeping only terms up to order O(1/N ) as we take the thermodynamic limit N → +∞ (at fixed α = M/N ). At this order, we will find that it is consistent to consider the messages to be approximately Gaussian, i.e. characterized by their means and co-variances. Thus we define

     xi→µ = dx x m i→µ (x) C x i→µ = dx xx T m i→µ (x)            ω µ→i = i =i W µi √ N xi →µ V µ→i = i =i W 2 µi N C x i →µ , (6.13)
where ω µ→i and V µ→i are related to the intermediate variable z µ = w µ X.

Expansion of mµ→i

(x i ) ∝ dξ µ pout y µ |ξ µ e i W µi √ N ξ µ x i i =i dx i m i →µ (x i )e i W µi √ N x i ξ µ x i (6.16) ∝ dξ µ pout y µ |ξ µ e iξ W µi √ N x i +ω µ→i -1 2 ξ V -1 µ→i ξ (6.17)
where developing the exponentials of the product in (6.11) allows to express the integrals over the x i as a function of the definitions (6.13), before re-exponentiating to obtain the final result (6.17). Now reversing the Fourier transform and performing the integral over ξ we can further rewrite mµ→i (x i ) ∝ dz µ p out y µ |z µ e

-1 2 z µ - W µi √ N x i -ω µ→i V -1 µ→i z µ - W µi √ N x i -ω µ→i (6.18) ∝ dz µ P out (z µ ; ω µ→i , V µ→i )e z µ -ω µ→i V -1 µ→i W µi √ N x i - W 2 µi 2N x i V -1 µ→i x i , ( 6.19)
where we are led to introduce the output update functions

P out (z µ ; ω µ→i , V µ→i ) = p out y µ |z µ N (z µ ; ω µ→i , V µ→i ) , ( 6.20)
Z out (y µ , ω µ→i , V µ→i ) = dz µ p out y µ |z µ N (z µ ; ω µ→i , V µ→i ) , (6.21)

g out (y µ , ω µ→i , V µ→i ) = 1 Z out ∂Z out ∂ω and ∂ ω g out = ∂g out ∂ω , ( 6.22)
where we recall that N (z; ω, V ) is the multivariate Gaussian distribution of mean ω and covariance V . Further expanding the exponential in (6.19) up to order O(1/N ) leads to the Gaussian parametrization mµ→i (

x i ) ∝ 1 + W µi µi √ N g out 1 x i + W µi µi 2 2N x i T (g out g out T + ∂ ω g out 1 )x i (6.23) ∝ e B µ→i T x i -1 2 x i T A µ→i x i , ( 6.24)
with

   B µ→i = Wµi µi √ N g out (y µ , ω µ→i , V µ→i ) A µ→i = - Wµi µi 2 N ∂ ω g out (y µ , ω µ→i , V µ→i ). (6.25)
Consistency with m i→µ Inserting the Gaussian approximation of mµ→i in the definition of m i→µ , we get the parametrization

m i→µ (x i ) ∝ p x (x i ) µ =µ e B µ →i T x i -1 2 x i T A µ →i x i ∝ p x (x i )e -1 2 (x i -λ i→µ ) T σ -1 i→µ (x i -λ i→µ ) (6.26)
with

     λ i→µ = σ i→µ µ =µ B µ →i σ i→µ = µ =µ A µ →i -1 . ( 6

.27)

Closing the equations Ensuring the consistency with the definitions (6.13) of mean and covariance of m i→µ we finally close our set of equations by defining the input update functions

Z x = dx p x (x)e -1 2 (x-λ) σ -1 (x-λ) (6.28) f x 1 (λ, σ) = 1 Z x dx x p x (x)e -1 2 (x-λ) σ -1 (x-λ) (6.29) f x 2 (λ, σ) = 1 Z x dx xx p x (x)e -1 2 (x-λ) σ -1 (x-λ) -f x 1 (λ, σ)f x 1 (λ, σ) , (6.30) so that xi→µ = f x 1 (λ i→µ , σ i→µ ) C x i→µ = f x 2 (λ i→µ , σ i→µ ). ( 6

.31)

Approximate message passing Given the scaling of the weights it is possible to further simplify the algorithm by considering the approximated marginals

m i (x i ) = 1 Z i p x (x i )e -1 2 (x-λ i ) σ -1 i (x-λ i ) with              λ i = σ i M µ=1 B µ→i σ i = M µ A µ→i -1 . ( 6
V (t) µ = N i=1 W 2 µi N C x i (t) (6.33
)

ω (t) µ = N i=1 W µi √ N x(t) i - N i=1 W 2 µi N (σ (t) i ) -1 C x(t) i σ i g out (t-1) µ (6.34)
2) Estimate mean and variance of the gap between optimal z µ and ω µ given y µ

∂ ω g out (t) µ = ∂ ω g out (y µ , ω (t) µ , V (t) µ ) (6.35) g out (t) µ = g out (y µ , ω (t) µ , V (t) µ ) (6.36)
3) Estimate mean and variance of x i given current optimal z µ σ

(t) i =   - M µ=1 W 2 µi N ∂ ω g out (t) µ   -1 (6.37) λ (t) i = x(t) i + σ (t) i   M µ=1 W µi √ N g out (t) µ   (6.38)
4) Estimate of mean and variance of x i augmented of the information about the prior

C x i (t+1) = f x 2 (λ (t) i , σ (t) i ) (6.39) x(t+1) i = f x 1 (λ (t) i , σ (t) i ) (6.40)
until convergence

Treatment of calibration variables

Heuristic derivation To include calibration variables, we need to consider the factor graph on Figure 6.1b and augment the belief propagation with a set of messages related to the s µ . Without going back through the entire derivation the final algorithm can easily be guessed. The posterior distribution on X in the presence of the calibration variable s is a special case of the GLM on vector variables examined above with the effective channel

p out (y µ |w µ X) = ds µ p s out (y µ |w µ X, s µ )p s (s µ ). (6.41)
Thus to reconstruct X one can directly use Algorithm 7, with output functions (6.20)-(6.22) that will include a marginalization over s: , σ i , C x i are assumed to be diagonal. However, we recall that BP is only exact on a tree, where the incoming message at each node are truly independent. On the dense factor graph of the GLM on scalar variables, this is approximately exact in the thermodynamic limit due to the random mixing, and small scaling, of the weight matrix W . Here by considering the reconstruction of P samples at the same time, sharing a given realization s of the calibration variable, additional correlations may arise. Writing the message passing on the vector variables in R P allows not to neglect them.

Z out (y µ , ω µ , V µ ) = dz µ ds µ p s out y µ |z µ , s µ p s (s µ )N (z µ ; ω µ , V µ ). ( 6
= f s 1 (y µ , ω µ , V µ ) = 1 Z out ds µ s µ dz µ p s out y µ |z µ , s µ p s (s µ )N (z µ ; ω µ , V µ ) (6.44) C s µ = f s 2 (y µ , ω µ , V µ ) = 1 Z out ds µ s 2 µ dz µ p s out y µ |z µ , s µ p s (s µ )N (z µ ; ω µ , V µ ) -ŝ2 µ . ( 6

State Evolution for Cal-AMP

In the thermodynamic limit, recall that the performance of the AMP algorithm can be characterized by a set of simpler equations corresponding to an averaging over the disorder (here X 0 , Y and W ), referred to as State Evolution. In Chapter 3 we have seen that there are two ways of deriving the State Evolution, either from the direct averaging of the AMP steps or from the saddle point equations of the replica free energy associated with the problem under the Replica Symmetric (RS) ansatz. In [START_REF] Aubin | The committee machine: Computational to statistical gaps in learning a two-layers neural network[END_REF], the teacher-student matched setting of the GLM on vectors is examined through the replica approach and the Bayes optimal State Evolution equations are obtained through this second strategy. In the following we present the alternative derivation of the State Evolution equations from the message passing and without assuming a priori matched teacher and student. To this end, our starting point will be the r-BP equations. Ultimately, we also introduce new State Evolution equations following the reconstruction of calibration variables.

State Evolution derivation in mismatched setting

Defintion of the overlaps The overlaps are here P × P matrices

q = 1 N N i=1 xi xT i , m = 1 N N i=1 xi x 0,i T , q 0 = 1 N N i=1 x 0,i x 0,i T . ( 6

.46)

Output parameters Under Gaussian statistics of the entries of W , the variable ω µ→i defined in (6.13) is a sum of independent Gaussian variables and follows itself a Gaussian distribution. Its first and second moments are

E W ω µ→i = 1 √ N i =i E W W µi xi →µ = 0 , (6.47) E W ω µ→i ω T µ→i = 1 N i =i E W W 2 µi xi →µ xT i →µ = 1 N N i=1 xi→µ xT i→µ + O 1/N (6.48) = 1 N N i=1 xi xT i -∂ λ f x 1 σ i B µ→i xT i -∂ λ f x 1 σ i B µ→i xT i T + O 1/N (6.49) = 1 N N i=1 xi xT i + O 1/ √ N (6.50)
where we used the fact that B µ→i defined in (6.25) is of order O(1/ √ N ). Similarly, the variable

z µ→i = i =i W µi √ N x i is Gaussian with first and second moments E W z µ→i = 1 √ N i =i E W W µi x 0,i = 0 , ( 6.51)
E W z µ→i z T µ→i = 1 N N i=1 x 0,i x 0,i T + O 1/ √ N . (6.52)
Furthermore, their covariance is

E W z µ→i ω T µ→i = 1 N i =i E W W 2 µi x 0,i xT i →µ = 1 N N i=1 x 0,i xT i→µ + O 1/N (6.53) = 1 N N i=1 x 0,i xT i -x 0,i ∂ λ f x 1 σ i B T µ→i + O 1/N (6.54) = 1 N N i=1 x 0,i xT i + O 1/ √ N . (6.55)
Hence we find that for all µ-s and all i-s, ω µ→i and z µ→i are approximately jointly Gaussian in the thermodynamic limit following a unique distribution N z µ→i , ω µ→i ; 0, Q with the bloc covariance matrix

Q =    q 0 m m q    . ( 6.56)
For the variance message V µ→i , also defined in (6.13), we have

E W V µ→i = i =i E W W µi N 2 C x i →µ = N i=1 1 N C x i→µ + O 1/N (6.57) = N i=1 1 N C x i + O 1/ √ N , ( 6.58)
where using the developments of λ i→µ and σ i→µ (6.27), along with the scaling of

B µ→i in O(1/ √ N ) we replaced C x i→µ = f x 2 (λ i→µ , σ i→µ ) = f x 2 (λ i , σ i ) -∂ λ f x 2 σ i B T µ→i = f x 2 (λ i , σ i ) + O 1/ √ N . (6.59)
Futhermore, we can check that lim

N →+∞ E W V 2 µ→i -E W V µ→i 2 = 0, (6.60)
meaning that all V µ→i concentrate on their identical mean in the termodynamic limit, which we note

V = N i=1 1 N C x i . (6.61)
Input parameters Here we use the re-parametrization trick to express y µ as a function g 0 (•) taking a noise µ ∼ p ( µ ) as a second input: y µ = g 0 (w µ X 0 , s 0,µ , µ ). Following (6.25) and (6.32),

σ -1 i λ i = M µ=1 W µi √ N g out y µ , ω µ→i , V µ→i (6.62) = M µ=1 W µi √ N g out   g0   i =i W µi √ N x 0,i + W µi √ N x 0,i , µ   , ω µ→i , V µ→i    (6.63) = M µ=1 W µi √ N g out   g0   i =i W µi √ N x 0,i , s µ , µ   , ω µ→i , V µ→i    + M µ=1 W 2 µi N ∂ z g out g 0 z µ→i , s µ , µ , ω µ→i , V µ→i x 0,i . (6.64)
The first term is again a sum of independent random variables, given the W µi are i.i.d. with zero mean, of which the messages of type µ → i are assumed independent. The second term has non-zero mean and can be shown to concentrate. Finally recalling that all V µ→i also concentrate on V we obtain the distribution σ -1 i λ i ∼ N (σ -1 i λ i ; α mx 0,i , αqI P ) (6.65) with q = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q)g out (g 0 (z, s 0 , ) , ω, V )× (6.66)

g out (g 0 (z, s 0 , ) , ω, V ) T , m = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q)∂ z g out (g 0 (z, s 0 , ) , ω, V ). ( 6.67)
For the inverse variance σ -1 i one can check again that it concentrates on its mean

σ -1 i = M µ=1 W µi 2 N ∂ ω g out (y µ , ω µ→i , V µ→i ) α χ (6.68) χ = -d p ( )ds 0 p s0 (s 0 ) d dz N (z, ω; 0, Q)∂ ω g out (g 0 (z, s 0 , ) , ω, V ) . ( 6

.69)

Closing the equations These statistics of the input parameters must ensure that consistently

V = 1 N N i=1 C x i = E λ,σ f x 2 (λ, σ) , (6.70) q = 1 N N i=1 xi x i = E λ,σ f x 1 (λ, σ)f x 1 (λ, σ) , (6.71) m = 1 N N i=1 xi x 0,i = E λ,σ f x 1 (λ, σ)x 0,i , (6.72)
which gives upon expressing the computation of the expectations

V = dx 0 p x0 (x 0 ) Dξ f x 2 (α χ) -1 αqξ + α mx 0 ; (α χ) -1 , (6.73) m = dx 0 p x0 (x 0 ) Dξ f x 1 (α χ) -1 αqξ + α mx 0 ; (α χ) -1 x 0 , (6.74) q = dx 0 p x0 (x 0 ) Dξ f x 1 (α χ) -1 αqξ + α mx 0 ; (α χ) -1 × f x 1 (α χ) -1 αqξ + α mx 0 ; (α χ) -1 . (6.75)
The State Evolution analysis of the GLM on the vector variables finally consists in iterating the equations (6.66), (6.67), (6.69), (6.73), (6.74) and (6.75) until convergence.

Reconstruction of the calibration variable

In parallel, one can follow the reconstruction of s by introducing the scalar overlaps

r = 1 M M µ=1 ŝ2 µ , ν = 1 M M µ=1 ŝµ s 0,µ , r 0 = 1 M M µ=1 s 2 0,µ . (6.76)
Recalling the definition of the estimator ŝ (6.44), and after following the steps of the above derivation, one can see that the overlaps can be computed from the State Evolution variables

r = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q) ŝ g 0 (z, s 0 , ) , ω, V 2 , ( 6.77)
ν = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q) ŝ g 0 (z, s 0 , ) , ω, V s 0 . (6.78)

Performance analysis

The mean squared error (MSE) on the reconstruction of X by the AMP algorithm is then predicted by MSE(X) = q -2m + q 0 , (6.79)

where the scalar values used here correspond to the (unique) value of the diagonal elements of the corresponding overlap matrices. The fixed points of the iterated equations can subsequently be plugged into (6.77) and (6.78) to compute the MSE in the reconstruction of the calibration variable MSE(s) = r -2ν + r 0 . (6.80)

Bayes optimal State Evolution

Similarly to the scalar case, the equation can be greatly simplified in the Bayes optimal setting where the statistical model used by the student (priors p x and p s , and channel p out ) is known to match the teacher. In this case, one can prove that q = m, V = q 0 -m, q = m = χ and r = ν. (6.81)

And the State Evolution can be reduced to a set of three equations

r = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q) ŝ g 0 (z, s 0 , ) , ω, q 0 -m 2 , (6.82) m = dx 0 p x0 (x 0 ) Dξ f x 1 (α m) -1 α mξ + α mx 0 ; (α m) -1 x 0 (6.83) m = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q)g out g 0 (z, s 0 , ) , ω, q 0 -m) × (6.84) g out g 0 (z, s 0 , ) , ω, q 0 -m) ,
with the bloc covariance matrix

Q =    q 0 m m m    . ( 6
.85)

Online algorithm and analysis

Additionally, we consider the analysis of the online reconstruction of the calibration variable as the observations y (k) ∈ R M are treated successively. For the problem of the reconstruction of x 0 in the classic GLM problem, we have already discussed in Chapter 3, Section 3.5.1 the streaming AMP algorithm and corresponding State Evolution proposed in [START_REF] Manoel | Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing[END_REF]. The idea was to use as an effective prior at step k + 1 the approximate posterior marginal constructed by AMP at step k. Here we readily adapt this strategy by using at step k + 1 as an effective prior on s µ the approximate posterior at step k.

Approximate message passing

The AMP algorithm consists here in restarting Cal-AMP for a single sample at each new observation y (k) , while updating the prior used for the calibration variable. From the definition of the approximate posterior (6.43), we obtain the recursion on the effective prior p (k+1) sµ on s µ :

p (k+1) sµ (s µ ) = m s,(k) µ (s µ ) (6.86) = 1 Z (k) out dz (k),µ p s out y (k),µ |z (k),µ , s µ p (k) sµ (s µ )N (z (k),µ ; ω (k),µ , V (k),µ ), (6.87)
where the output variables ω (k),µ and V (k),µ correspond to the values at convergence (or at the last iteration t max ) of the Cal-AMP algorithm at the previous step (k). In the Section 6.2, we will examine the gain calibration problem and specify effective strategies to implement this recursion within the AMP algorithm.

State Evolution

The streaming State Evolution analysis of the calibration reconstruction is also adapted using the above recursion. Note that the effective prior at a given step P depends on the output variables of the algorithm for all the previously seen samples -expending the recursion above we have:

p (P ) sµ (s µ ) = 1 Z (P ) out P k=1 dz (k),µ p s out y (k),µ |z (k),µ , s µ N (z (k),µ ; ω (k),µ , V (k),µ ) p s (s µ ) (6.88) Z (P ) out = Z (P ) out ({y (k),µ , ω (k),µ , V (k),µ } P k=1 ) (6.89)
where again for each k the output variables ω (k),µ and V (k),µ are the converged values for the corresponding step k. The dependence of the normalization of Z (P ) out on the output variables (6.89) is reflected in the definitions at step P of g out (P ) and ŝ(P ) . Therefore, the State Evolution involving the output functions will feature an averaging on all the output variables relative to the already processed samples. In the Bayes optimal setting, it becomes r (P ) = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q (P ) ) (6.90)

P -1 k=1 dω (k) dz (k) N (z (k) , ω (k) ; 0, Q (k) ) ŝ(P ) g 0 (z, s 0 , ) , ω, q 0 -m 2 , m (P ) = dx 0 p x0 (x 0 ) Dξ f x 1 (α χ) -1 √ α mξ + α m(P ) x 0 ; (α m(P ) ) -1 x 0 , (6.91) m(P ) = d p ( )ds 0 p s0 (s 0 ) dω dz N (z, ω; 0, Q (P ) ) P -1 k=1 dω (k) dz (k) N (z (k) , ω (k) ; 0, Q (k)
) g out (P ) g 0 (z, s 0 , ) , ω, q 0 -m 2 , (6.92)

where the bloc covariance matrices similar to (6.85) are function of the fixed points of each step.

In the following Section we will discuss how to implement this State Evolution in practice, focusing on the specific problem of gain calibration.

Experimental validation on gain calibration

Setting and update functions

As a test case, we consider the following problem of gain calibration. The input signal is known to be ρ-sparse and distributed according to a Gauss-Bernoulli distribution. Each component of the output includes a division by a calibration variable that is uniformly distributed in a positive interval [a, b]:

∀i = 1 • • • N , p x0 (x i ) = P k=1 ρ N (x (k),i ; 0, 1) + (1 -ρ) δ(x (k),i ) , (6.93) ∀µ = 1 • • • M , p s0 (s µ ) = 1 [a,b] /(b -a) , 0 < a < b, ( 6.94
) δ -1 ν = y (V + ∆I P ) -1 ω (6.104)

y µ = 1 s 0,µ (w µ X 0 + ) , ∼ √ ∆N ( , 0, I P ) . ( 6
g out (y, ω, V ) = 1 Z out ∂ ω Z out = (V + ∆I P ) -1 (ŝ(y, ω, V )y -ω), (6.98) ∂ ω g out (y, ω, V ) = C s (y, ω, V )(V + ∆I P ) -1 yy (V + ∆I P ) -1 -(V + ∆I P ) -1 . ( 6
and I(P, ν, δ, a, b) can be computed using gamma functions as explained in [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF].

Input functions For a Gauss-Bernoulli prior on the entries of X, and assuming the AMP variances σ i are diagonal matrices, the input update functions can be written component-wise with scalar arguments:

f x 1 (λ, σ) = ρ λ (1 + σ) 3/2 e -λ 2 2(1+σ)   ρ e -λ 2 2(1+σ) (1 + σ) 1/2 + (1 -ρ) e -λ 2 2σ σ 1/2   , ( 6.105)
and

f x 2 (λ, σ) = ρ 1σ(1 + σ) + (1λ) 2 (1 + σ) 5/2 e -λ 2 2(1+σ)   ρ e -λ 2 2(1+σ) (1 + σ) 1/2 + (1 -ρ) e -λ 2 2σ σ 1/2   -f x 1,k 2 . (6.106) so that xi,k = f x 1 (λ i,k , σ i,kk ) and C x i,kk = f x 2 ((λ i,k , σ i,kk ).

Algorithm 8 Offline Gain Calibration State Evolution

Input: matrix Y ∈ R M ×P and matrix W ∈ R M ×N :

Initialize: t = 0, m (0) = 0, V (0) = q 0 = ρ, ∀µ = 1 • • • N MC s 0,µ ∼ p s0 (s 0,µ ) repeat 1
) Draw Monte Carlo samples for update of m (6.92) and r (6.90)

∀µ = 1 • • • N MC z µ , ω µ ∼ N (z µ , ω µ ; 0, Q (t) ) ∼ p ( ) y µ = g 0 (z µ , s 0,µ , ) with Q (t) =    q 0 I P m (t) I P m (t) I P m (t) I P    ∈ R 2P ×2P 2) Compute integrands 2.2) Compute ŝµ (y µ , z µ , ω µ , V (t) ) (6.100) 2.3) Compute g out µ (y µ , z µ , ω µ , V (t) ) (6.98)
3) Update r and m

r (t) = 1 N MC N MC µ=1 ŝ(t) µ 2 m(t) = 1 N MC N MC µ=1 g outµ 2
3) Update m (t+1) by numerical integration using (6.91), and

V (t+1) = q 0 -m (t+1) . t = t + 1 until convergence Output: time series {V (t) , m (t) , m(t) ; t = 1 • • • t max }

Offline results

The offline AMP algorithm is directly given by Algorithm 7 replacing the values of the output and input functions presented in the previous paragraph. In our numerical tests, we additionally impose that co-variance matrices V µ , ∂ ω g out µ and σ i are diagonal. This assumption lightens numerics so as to consider larger number of samples P . It also allows a comparison with the results of [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF]. Under the assumption of diagonal covariance matrices, the State Evolution equations (6.83), (6.84) involve P × P matrices proportional to the identity (given the P examples are statistically equivalent). Therefore it is sufficient to consider the update of one diagonal element noted respectively m and m. While the update of m only requires a two dimensional integral that can be performed numerically, we resort to a Monte Carlo for the update of m. The procedure is described in Algorithm 8.

On the top row of Figure 6.2 and respectively of Figure 6.3, we report the performance of reconstruction by the Cal-AMP algorithm of the signal X 0 and the calibration variable s 0 , in the plane ρ -α, for different values of P . These phase diagrams are similar to the ones reported in [START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF] and will be compared with the online case described in the next Section. On Figure 6.4a, we check numerically that the derived State Evolution predicts the behavior of the AMP algorithm for gain calibration. The MSEs of the two procedures are indeed consistent along the iterations of the algorithm. On Figure 6.4b, we also report an almost perfect agreement of the fixed points of SE and AMP in terms of the MSE on X as we vary the number of samples P .

Online results

The online algorithms include supplementary operations to update the effective prior on the calibration variable from one step to the next. In this setting the recursion is yielding a posterior on the calibration variable at step P with an identical form to the posterior of the offline algorithm (albeit with parameters computed differently). Therefore ŝµ and C s µ can still be computed analytically following (6.100) and (6.101). We provide a pseudo-code for the online Cal-AMP in Algorithm 10 and a pseudo-code for online SE in Algorithm 9.

p (k+1)
On the bottom rows of Figure 6.2 and 6.3 we plot phase diagrams obtained with online Cal-AMP. Compared to the offline diagrams we find that the reconstruction requires more samples to achieve comparable levels of accuracy. On Figure 6.4c and 6.4d we check the consistency of the Cal-AMP and SE fixed points in terms of MSE on X. Along the iterations of the algorithm and comparing fixed points we find a very good agreements of the two procedures. Nevertheless, the Monte Carlo integration in the SE does not allow for great numerical precision for low errors and we start seeing discrepancies between SE and AMP for errors below 10 -8 .

Algorithm 9 Online Gain Calibration State Evolution

Input: matrix Y ∈ R M ×P and matrix W ∈ R M ×N :

Initialize: t = 0, ∀µ = 1 • • • N MC s 0 µ ∼ p s 0 (s 0 µ ) Λ 0,µ = 0 , Σ 0,µ = 0 for k = 1 • • • P do Initialize: t = 0, m (0)
(k) = 0, V

(k) = q 0 = ρ repeat 1) Draw Monte Carlo samples for update of m (6.92) and r (6.

Matrix factorization model and multi-layer networks

We now return to our original problem of interest, which is to find a tractable setting to perform and analyze high-rank matrix factorization, a necessary milestone on the road to the characterization of learning in multi-layer neural networks. We identify a constrained formulation of matrix factorization that can be approached as a multi-layer GLM including calibration variables. We demonstrate a principle of derivation of the AMP algorithm for multi-layer GLM and deduce from it the AMP algorithm for the constrained matrix factorization.

Constrained matrix factorization

We consider the problem of recovering matrices W 0 ∈ R Q×N and X 0 ∈ R N ×P after observation of their product through a noisy (non-linear) channel while introducing a constraint: Y = g 0 ( W 0 X 0 ; ), s.t. W 0 = Φ 1 S 0 Φ 0 . (6.124)

As usual g 0 (•) stands for the channel and incorporates a noise ∈ R Q×P drawn from a simple distribution p ( ). The matrices Φ 0 ∈ R M ×N and Φ 1 ∈ R Q×M of the constrained form of W 0 are known, while the matrix S 0 ∈ R M ×M is diagonal and encloses the degrees of freedom of W 0 .

Considering the teacher-student matched setting, the posterior joint distribution we are going to be interested in is

p(X, S|Φ 0 , Φ 1 , Y ) = 1 Z dU p 2 out (Y |Φ 1 U )p 2 out (U |Φ 0 X, s)p x (X)p x (S) (6.125)
with the distributions of the right hand-side assumed to factorize over their inputs and the corresponding factor graph represented in Figure 6.5a. In the following we will assume that N , M and Q are large but maintain fixed ratios α 0 = M/N and α 1 = Q/N . Thus we are considering extensive rank matrix factorization, while easing the inference problem by reducing the number of free variables in one of the factors. The advantage of the chosen constrained model is that it can be interpreted as the combination of two GLM models: Y = g 0,2 (Φ 1 U ; ) usual GLM with prior on signal U coming from the first layer, U = g 0,1 (Φ 0 X; S) GLM with calibration variables.

The inference of X and S can therefore be tackled using the recent progresses around multilayer models. In the following we start by recalling and straightforwardly extending the ML-AMP algorithm [START_REF] Manoel | Multi-layer generalized linear estimation[END_REF] introduced at P = 1 to arbitrary P . In a second step we re-introduce the calibration variables to obtain the AMP algorithm for the constrained matrix factorization under scrutiny.

Multi-layer vectorized AMP

The derivation of the multi-layer AMP on vector variables follows identical steps to the derivation of the single layer presented in Section 6.1.1 starting with a larger collection of messages. Without conducting the lengthy procedure, one can form an intuition for the resulting algorithm starting from the single-layer AMP. We discuss here this intuition focusing on the 2-layer case.

We are now considering the factor graph of Figure 6.5b and the resulting Algorithm 11. Compared to the single-layer case, an interface with a set of hidden variables u µ is inserted between the signals x i and the observations y a . In the neighborhood of the inputs x i the factor graph is identical to the single-layer and the input functions can be defined from a normalization partition identical to (6.28), yielding updates (6.149)-(6.150). Similarly, the neighborhood of the observations y a is also unchanged and the updates (6.139) and (6.140)

Z x (λ i , σ i ) = dx i p x (x i )e p 2 out (y a |Φ 1 a U ) a = 1 • • • Q u 1 u 2 x 1 x 2 x 3 x 4 p x (x i ) i = 1 • • • N s 1 s 2 p 1,s out (u µ |Φ 0 µ X, s µ ) µ = 1 • • • M (a) p 2 out (y a |Φ 1 a U ) a = 1 • • • Q u 1 u 2 x 1 x 2 x 3 x 4 p x (x i ) i = 1 • • • N p 1 out (u µ |Φ 0 µ X) µ = 1 • • • M (b)
ω 1 µ (t) = i Φ 0 µi √ N x(t) i - i (Φ 0 µi ) 2 N σ 0 i (t-1) -1 C x i σ 0 (t-1) i g 1 out µ (t-1) (6.143) V 1 µ (t) = i (Φ 0 µi ) 2 N C x i (t)
λ 0 i = σ 0 i   µ Φ 0 µi √ N g 1 out µ (t) - (Φ 0 µi ) 2 N ∂ ω g 1 out µ (t) xi   (6.148)
3) Update means and variances of variables of both layers, x and u:

x(t+1

) i = f x 1 λ 0 i (t) , σ 0 i (t)
(6.149)

C x i (t+1) = f x

Towards the analysis of learning in multi-layer neural networks

The presented elementary matrix factorization can be used in turn as a building block to study learning in multi-layer neural networks under the teacher-student paradigm. To conclude this Chapter we discuss how we believe this study could be done.

The statistical model we previously investigated as an example of matrix factorization (6.124) can be interpreted as a very simple feed forward generative model used in modern machine learning. Namely, a latent variable with a separable prior x ∈ R N is transformed by a stochastic single layer neural network with weight matrix W (parametrized by s) to create a data point y ∈ R Q . Usually, such a model is trained to fit an arbitrary (real) data set by a gradient descent following either the Generative Adversarial Networks [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] interpretation and the adversarial objective or the Variational Autoencoder interpretation [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF][START_REF] D J Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] and the maximum likelihood objective. In the constrained setting we have described, with a reduced number of degrees of freedom of the weight matrix W , the AMP algorithm is also a learning algorithm returning the mean a posteriori Ŵ = Φ 1 ŜΦ 0 as appropriate network parameters.

This interpretation is interesting for several reasons. On Figure 6.6a, we represent the graphical model for the single-layer teacher generative model and the factor graph for the AMP inference of the student (only factorized at the level of the matrices). In the teacher-student scenario, the theoretical performance of AMP can be tracked on a single instance by comparing the reconstructed Ŵ to the teacher W 0 and cross-checked by the State Evolution. These tools will thereby allow to conduct a theoretical analysis of unsupervised learning in a simple model. Moreover, AMP can also be a practical learning algorithm on arbitrary data (not generated from a teacher) and should be compared with other learning techniques. While it is likely that a message passing implementation of learning would be cumbersome compared to SGD, it could be more efficient in various ways. For instance, AMP may need less samples than gradient descent to reach a similar quality of training.

Furthermore, the case study can be extended to deep generative neural networks with constrained weight matrices by stacking the elementary model as represented on Figure 6.6b. The juxtaposition of models can again be interpreted as a special case of the multi-layer vectorized GLM and the inference can be performed by extending the 2-layer vectorized AMP Algorithm 11 to an arbitrary number of layers. The graphical representations (see Figure 6.6b) of the teacher generative model and the posterior distribution now include variables for the hidden layers noted T , which are new interface variables in the AMP inference.

Finally, the considered model comprises deep supervised learning as a special case. In fact, by fixing the values of the inputs X to the value of a training set with the corresponding outputs Y , the inference problem is only simpler. It consists in learning solely the weights of the deep neural network, corresponding to the supervised setting. Such scenario should therefore allow to replicate the statistical physics analyses of learning in simple architectures (perceptrons and committee machines) in deep networks. In particular it could enable the computation of learning curves predicting optimal generalization errors in the multi-layer case, albeit in the constrained regime of learning, with only the singular values of the weights as adjustable parameters. As traditional generalization bounds relying on worst case guarantees are found irrelevant in deep learning, the proposed case study appears as an interesting preliminary test to determine whether the statistical physics approach could be more informative here. Potentially, it could provide a scale to measure the performance of the currently popular training algorithms that were introduced empirically, such as SGD.

Conclusion and outlook

In this dissertation, we used different mean-field free energies and algorithms to answer various questions around the training of neural networks. In Chapter 4, we showed that the TAP free energy and the approximate message passing algorithms are of practical interest to train Boltzmann machines. They can also be leveraged to exploit learned RBMs as priors in Bayesian problems.

A straightforward perspective of these results is to now consider, through the same lense, the feed forward models used in unsupervised learning. Here also, mean-field methods could yield alternative algorithms competitive with the state-of-the art. In Chapter 5, we used a replica computation to test the relevance of information theory to explain the generalization paradox in deep learning. With this strategy we were able to probe prototypes of learning experiments with large networks and non-linearities. Our findings both confirmed and confounded some of the previous claims in the literature, finally pointing out the caveats of the proposed approach. Chapter 6 is in itself a perspective. We identified a direction that could lead to an analysis of learning in multi-layer networks along the lines of the original statistical mechanics of the perceptron. We validated the first step of this proposed program by deriving the State Evolution corresponding to the Cal-AMP algorithm, which allows to infer parameters in the channel of a GLM.

On the edge of validity

Nonetheless, we have also touched upon the limitations of the mean-field approach. In Chapter 5, we had to consider a restricted setting of learning in order to guarantee the accuracy of the replica formula of the entropy. Only the spectrum of the weight matrices was learned, which effectively reduced the number of degrees of freedom. A similar restriction is also included in the proposed direction to study multi-layer Bayesian learning in Chapter 6. Additionally, for both of these analyses, we can only consider synthetic data sets with specific distributions. More generally, the temptation to apply abusively results from one field to the other can be a dangerous pitfall of the interdisciplinary approach. We could mention here the characterization of the dynamics of optimization, an open question that we did not tackle in this dissertation. While physicists have extensively studied Langevin dynamics with Gaussian white noise, the continuous time limit of SGD is unfortunately not an equivalent in the general case. While some works attempt to draw insights from this analogy using strong assumptions (e.g. [START_REF] Choromanska | The Loss Surfaces of Multilayer Networks[END_REF][START_REF] Stanis Law Jastrzȩbski | Three Factors Influencing Minima in SGD[END_REF]), others seek precisely to understand the differences between the two dynamics in neural networks optimization (e.g. [START_REF] Baity-Jesi | Comparing Dynamics: Deep Neural Networks versus Glassy Systems[END_REF][START_REF] Simsekli | A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks[END_REF]). Alternatively, [START_REF] Sarao Mannelli | Passed & Spurious: analysing descent algorithms and local minima in spiked matrix-tensor model[END_REF] gained insights on the dynamics of gradient descent (and Langevin) in a different high-dimensional non-convex optimization problem inspired by the physics of spin glasses. This last work illustrates the modelling tradition of theoretical physics. One can learn from models a priori far from the exact neural networks desired, but that retain some key properties, while being amenable to theoretical characterization. Another example of this approach is [START_REF] Wang | A Solvable High-Dimensional Model of GAN[END_REF].

Thus the mean-field approach will certainly not provide complete answers to the still numerous puzzles on the way towards a deep learning theory. Yet, combined with the modelling tradition of physics, it will still probably provide many more interesting intuitions. More specifically, the different contributions presented in this dissertation, point us towards a promising set up to study the impact of structure in data, which we propose as a final outlook.

One outlook

In the complicated story behind generalization, more and more works are trying to disentangle the role played by optimization algorithms (e.g. [START_REF] Shallue | Measuring the Effects of Data Parallelism on Neural Network Training[END_REF]) and regularization methods (e.g. [START_REF] Santurkar | How Does Batch Normalization Help Optimization[END_REF]), but the role played by data structure, while certainly important [START_REF] Zhang | Understanding Deep Learning Requires ReThinking Generalization[END_REF], is harder to probe. In the teacher-student scenario, one considers learning on synthetic datasets. In the classical literature, data points are typically distributed as random white noise. This choice is merely the simplest to z (1) z (2) x (1) x (2) p x (x (1) )

p out (y|z) ψ x (x (1) , x (2) ) = δ(x (1)x (2) ) ψ z (z (1) , z (2) ) = δ(z (1) z (2) ) lim ∆→0 N (z (2) ; W x (2) , ∆) We recall here a possible derivation of G-VAMP discussed in Chapter3 Algorithm 2 . We consider a projection of the BP equations for the factor graph Figure 8.

Gaussian assumptions

) 155
Self consistency of the parametrizations at Dirac factor nodes Around ψ x the message passing equations are simply mψx→x (2) (x (2) ) = m x (1) →ψx (x (2) ), mψx→x (1) (x (1) ) = m x (2) →ψx (x (1) ) [START_REF] Timothy | The statistical mechanics of learning a rule[END_REF] and similarly around ψ z . Moreover, considering that messages are marginals to which the contribution of the opposite message is retrieved we have m x (1) →ψx (x (1) ) ∝ m x,(1) (x (1) )/ mψx→x (1) (x (1) ), [START_REF] Welling | A new learning algorithm for mean field Boltzmann machines[END_REF] m x (2) →ψx (x (2) ) ∝ m x,(2) (x (2) )/ mψx→x (2) (x (2) ) . [START_REF] Yang | Deep Fried Convnets[END_REF] Combining this observation along with (156) leads to updates (3.89) and (3.85) The same reasoning can be followed for the messages around ψ z leading to updates (3.91) and (3.87).

Input and output update functions

The update functions of means and variances of the marginals are deduced from the parametrized message passing. For the variable x (1) taking into account the prior p x , the updates are very similar to GAMP input functions:

x(1) ∝ dx (1) x (1) p x (x (1) ) mψx→x (1) (x (1) )

= 1

Z

(1) x dx (1) x (1) p x (x (1) )e -1 2 x (1) A (1) x

x (1) +B (1) x

x (1) = f x 1 (B (1) x , A (1) x ) , (160)

C x(1) = 1 Z (1)
x dx (1) x (1) x (1) p x (x (1) )e -1 2 x (1) A (1) x

x (1) +B (1) x

x (1) (161)

-f x 1 (B (1) x , A (1) x )f x 1 (B (1) x , A (1) x ) = f x 2 (B (1) x , A (1) x ) , [START_REF] Zhao | InfoVAE: Information Maximizing Variational Autoencoders[END_REF] where Z

(1)

x is as usual the partition ensuring the normalization. Similarly for the variable z (1) , the update functions are very similar to the GAMP output functions including the information coming from the observations: ẑ(1) ∝ dz (1) p out (y|z (1) ) mψz→z (1) (z (1) ) (163) = 1

Z

(1) z dz (1) p out (y|z (1) )e -1 2 z (1) A (1) z z (1) +B (1) z z (1) = f z 1 (B (1) z , A (1) z ) , (164)

C z (1) = 1 Z (1) z
dz (1) z (1) z (1) p out (y|z (1) )e -1 2 z (1) A (1) z z (1) +B (1) z z (1) (165)

-f z 1 (B (1) z , A (1) z )f z 1 (B (1) z , A (1) z ) = f z 2 (B (1) z , A (1) z ) .

(166)

Linear transformation For the middle factor node we consider the vector variable concatenating x = [x (2) z (2) ] ∈ R N +M . The computation of the corresponding marginal with the message passing then yields m x(x) ∝ lim ∆→0 N (z (2) ; W x (2) , ∆I M )e -1 2 x A (2) x x+B (2) x x e -1 2 z A (2) z z+B (2) z z .

(167)

The means of x (2) and z (2) are then updated through x(2) , ẑ(2) = arg min

x,z W x -z 2 /∆ + x A (2) x x -2B (2) x x + z A (2) z z -2B (2) z z , (

at ∆ → 0. At this point it is advantageous in terms of speed to consider the singular value decomposition W = U SV and to simplify the form of the variance matrices by taking them proportional to the identify, i.e. A (2) z = A

(2)

z I M etc. Under this assumption the solution of the minimization problem is x(2) = g x 1 (B (2) x , A (2) x , B (2) z , A (2) z ) = V D A (2) z -2 SU B (2) z + A (2) x -2

V B (2) x , (169) ẑ(2) = g z 1 (B (2) x , A (2) x , B (2) z , A (2) z ) = W g x 1 (B (2) x , A (2) x , B (2) z , A (2) z ) , (

with D a diagonal matrix with entries D ii = (A

(2) z -1 S 2 ii + A (2) x -1
) -1 . The scalar variances are then updated using the traces of the Jacobians with respect to the B (2) -s C x (2) = A

(2) x N tr ∂g x 2 /∂B (2) x

I N = 1 N N i=1 (A (2) z -1 S 2 ii + A (2)
x -1

) -1 I N (171) = g x 2 (B (2) x , A (2) x , B (2) z , A (2) z ) (172) (2) z

C z (2) = A (2) z M tr ∂g z 2 /∂B
I M = 1 M N i=1 S ii (A (2) z -1 S 2 ii + A (2) z -1
) -1 I M (173) = g z 2 (B (2) x , A (2) x , B (2) z , A (2) z ).

(174)

Figure 1 . 1 :

 11 Figure 1.1:Let's assume we wish to classify data points x ∈ R 2 with labels y = ±1. We choose as an hypothesis class the perceptron sketched on the left with sign activation. For given weight vector w and bias b the plane is divided by a decision boundary assigning labels. If the training data are linearly separable, then it is possible to perfectly predict all the labels with the perceptron, otherwise it is impossible.

  i.d. draws from the model p θ to have generated the training data D = {x (k) θ (x (k) ).

P

  k=1 δ(x -x (k) )/P and the model p θ : min θ KL(p D ||p θ ) = min θ dx p D (x) log p D (x) p θ (x) ⇐⇒ max θ P k=1 log p θ (x (k) ) , (1.13)

  Feed forward neural network.

Figure 2 . 1 :

 21 Figure 2.1: (a) Undirected probabilistic graphical model (left) and factor graph representation (right). (b) Left: Directed graphical model for p(x, y, W ) without assumptions of factorizations for the channel and priors. Middle: Directed graphical model reflecting factorization assumptions for p(x, y|W ). Right: Corresponding factor graph representation.

Figure 2 .

 2 1a. The corresponding set of cliques is the set of all the pairs with one input unit (indexed by i = 1 • • • N ) and one hidden unit (indexed by α = 1 • • • M ), joined with the set of all single units. The potential function are immediately recovered from (2.2),

Figure 3 . 1 :

 31 Figure 3.1: Graphical representations of the Generalized Linear Model. Left: Probabilistic graphical model of the teacher. Middle left: Factor graph representation of the posterior distribution on the signal x under the student statistical model. Middle right and right: Belief propagation updates (3.31) -(3.32) for approximate inference.

Figure 3 . 2 :

 32 Figure 3.2: Factor graph representation of the GLM for the derivation of VAMP

  Substituting β = 0 in the definition of A we have A(0, a, c) = -λ(0, a, c) a -ξ(0, a, c) (a 2 + c) + log Z0 (λ(0, a, c), ξ(0, a, c)), (4.20) with Z0 (λ(0, a, c), ξ(0, a, c)) = dx e λ(0,a,c) x+ξ(0,a,c) x 2 N i=1 p x (x i ; θ i )

Figure 4 . 1 :

 41 Figure 4.1: Graphical models for the Restricted Boltzmann Machine and the Deep Boltzmann Machine

Figure 4 . 2 :

 42 Figure 4.2: Fully connected BM factor graph for adaTAP

  H z u c P S 9 G T 6 w = = < / l a t e x i t > Pseudo log-likelihood < l a t e x i t s h a 1 _ b a s e 6 4 = " d m m r p 7 N 8 n N M D 5 g 5

  p w X 5 9 3 5 m L e u O M X M E f y B 8 / k D L j S V l A = = < / l a t e x i t > (a) MNIST TAP log-likelihood < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 s a U j h a t x y t X N q X H lP c C R S G F d H I = " > A A A B + 3 i c b V D L S g M x F L 3 j s 9 b X W J d u g k V w Y 5 m p g i 4 r b l x W 6 A v a o W Q y a R u a S Y Y k I 5 a h v + L G h S J u / R F 3 / o 1 p O w t t P X D h c M 6 9 y b 0 n T D j T x v O + n b X 1 j c 2 t 7 c J O c X d v / + D Q P S q 1 t E w V o U 0 i u V S d E G v K m a B N w w y n n U R R H I e c t s P x 3 c x v P 1 K l m R Q N M 0 l o E O O h Y A N G s L F S 3 y 0 1 b u u I y + E F Z 2 P 7 y E j K q O + W v Y o3 B 1 o l f k 7 K k K P e d 7 9 6 k S R p T I U h H G v d 9 b 3 E B B l W h h F O p 8 V e q m m C y R g P a d d S g W O q g 2 y + + x S d W S V C A 6 l s C Y P m 6 u + J D M d a T + L Q d s b Y j P S y N x P / 8 7 q p

  A H z u c P S 9 G T 6 w = = < / l a t e x i t > Pseudo log-likelihood < l a t e x i t s h a 1 _ b a s e 6 4 = " d m m r p 7 N 8 n N M D 5 g 5+ t R Y n c R D v 9 x E = " > A A A B / n i c b V D L S g M x F L 3 j s 9 b X q L h y E y y C G 8 t M F X R Z c O O y g n 1 A O 5 R M J t O G Z p I h y Q h l K P gr b l w o 4 t b v c O f f m L a z 0 N Y D F w 7 n 3 J v c e 8 K U M 2 0 8 7 9 t Z W V 1 b 3 9 g s b Z W 3 d 3 b 3 9 t 2 D w 5 a W m S K 0 S S S X q h

Figure 4 . 3 :

 43 Figure 4.3: Evolution of the normalized pseudo-log-likelihood and mean-field-log-likelihood over test sets during training for different training procedures (averages over 10 independent trainings with standard deviations reported as error bars). Hyperparameters of training are given in Table 4.1. (a) The naive mean-field appears as the least efficient, while others are approximately equivalent. (b) Persistent mean-field algorithms have difficulties in the likelihood in the first epochs of training. The complexity of the training set, 101 classes unevenly represented over only 4 100 training points, might explain this unexpected behavior. The persistent chains all converge to similar non-informative blurs in the earliest training epochs. Nevertheless,these algorithms eventually reach identical likelihood to the rest of the algorithms tested.

Figure 4 . 4 :

 44 Figure 4.4: Additional elements of comparison between training algorithms. Hyperparameters of training are given in Table 4.1. (a) 24 randomly selected persistent chains used to compute parameter updates in Algorithm 5 . Top: PCD, Middle: PMF, Bottom: P-TAP2. (b) Classification accuracy achieved by a logistic regression on the hidden representation at different stage of learning.

Figure 4 . 5 :

 45 Figure 4.5: (a) Comparison for the mean-field training of a binary RBM on MNIST of likelihood estimates.The AIS results are averaged over 100 runs. For each run, 14 500 intermediate distributions are generated between the initial and target distribution using the same schedule as in[START_REF] Salakhutdinov | On the quantitative analysis of Deep Belief Networks[END_REF]. The pseudo-likelihood is computed using a stochastic approximation over 100 of the 784 pixels. Hyperparameters of training are given in Table4.1. (b) Left: Evolution of the pseudo-likelihood along the training of a binary RBM with 784 visible units and 500 hidden units, on the 5000 first images of binary-MNIST, with a learning rate of 0.001 and batches of size 100. Different number of iterations τ of the TAP inference (Algorithm 3) and adaTAP inference (Algorithm 4) are compared. In all cases, a damping of 0.5 is used. Right: Computation time for one iteration of the inference algorithm, as a function of the batch size. Time is reported in seconds for identical experimental settings.

Figure 4 . 6 :

 46 Figure 4.6: Top row: Normalized (per-unit) TAP log-likelihood estimate computed for 10, 000 training data samples for MNIST and 2, 400 training samples for CBCL . The TAP free energy is estimated using the converged TAP solutions from initial conditions drawn from the data samples. Solid lines indicate the average normalized TAP log-likelihood over the tested training samples and shaded regions indicate standard error. Bottom row: Free energy estimates of TAP solutions as a function of training epochs. In the case of the MNIST experiments, the number of hidden units considered is M = N H = 100. Among the same TAP solutions as above, we retrieve the unique ones and follow their number (green line) along the training. We report as transparent blue dots their TAP free energy. The estimate of the total free energy via uniform averaging is given as a red line. Hyperparameters of training are given in Table 4.1. This experiment was led by E. W. Tramel and A. Manoel.

Figure 4 . 7 ,Figure 4 . 7 :

 4747 Figure 4.7: Comparison of initial conditions (colored dots) and converged TAP solutions (black dots) for the tested data sets at different stages of training. For each data set, a two-dimensional Isomap embedding is calculated over the initialization data. Subsequently, the TAP solutions a (k) are embedded in the same space. In each case, all initial variances are set to 0. Also, a random selection of the TAP solution marginals are plotted as images. (a) real-MNIST: Here, the ∼3,100 digits corresponding to the classes '3', '6', and '7' are drawn from the first 10,000 training samples of the MNIST data set as initializations. A reduced set of labels is used for readability. (b) CBCL: All available training face images are used as initializations. This experiment was led by E. W. Tramel and A. Manoel.

54

 54

Figure 4 . 8 :

 48 Figure 4.8: Left: Factor graph of the exact posterior including observations and RBM prior.Middle: Approximate posterior after mean-field inference in the RBM leading to an effective prior as seen by the outer loop of the algorithm. Right: Effective RBM factor graph incorporating information about the observations using the approximate posterior coming from the message passing on the CS problem.

Figure 4 . 9 :

 49 Figure 4.9: Left: CS reconstruction performance over the first 1, 000 digit images of the MNIST test set in the plane sparsity ρ-measurement ratio α. Results for non-i.i.d. factorized prior, the binary RBM and the GRBM are on the left, center, and right, respectively. The M = K information theoretic transition is indicated by the black dotted line, and the AMP transition for i.i.d factorized prior [77] by the solid one. Top: Average reconstruction accuracy in MSE measured in dB. Bottom: Average reconstruction correlation (x 0 -x0 ) T (x-x)/σx 0 σ x with original digit image. Right: Visual comparison of reconstructions for a single digit image (ρ = 0.25) for small values of α.

Figure 5 . 1 :

 51 Figure 5.1: (a) Multi-layer stochastic neural network with orthogonally invariant weights: the matrices U (k) and V (k) are independent samples of the Haar measure. (b) Factor graph representation of the 3-layer GLM posterior distribution p(x|W(1) , W(2) , W(3) , t(3) ).

Figure 5 . 2 :

 52 Figure 5.2: Entropy of latent variables in stochastic networks with equally sized layers N 0 = N 1 = N 2 = 1000, standard Gaussian inputs px = N (x; 0, I N ), i.i.d weight entries drawn from N (W iµ ; 0, σ/N ), as a function of the weight scaling parameter σ. Left column: linear network. Center column: hardtanh-hardtanh network. Right column: relu-relu network.

Figure 5 . 3 :

 53 Figure 5.3: Graphical representation of a teacher model generating a synthetic data set of input-output pairs {x, y} and a student model with learnable parameters trained to recover the input-output mapping.

Figure 5 . 4 :

 54 Figure 5.4:Training and testing curves for the training of a two-layer neural net on the classification of MNIST (60 000 training images and 10 000 testing images), featuring one non-linear (relu) hidden layer of 500 neurones and a softmax output layer. The 500 × 10 parameters of the weight matrix of the output layer are all being learned in all the versions of the experiments. Conversely, before the relu layer, we either[START_REF] Achille | Emergence of Invariance and Disentangling in Deep Representations[END_REF] do not learn at all the 784 × 500 parameters which then define random projections, (2) learn all of them as a traditional fully connected network, (3) use a combination of 2 (3a), 3 (3b) or 6 (3c) consecutive USV-layers (without any intermediate non-linearity). For each version of the experiment the outcomes of two independent runs are plotted with the same color, it is not always possible to distinguish the two runs as they overlap.

Figure 5 . 5 :

 55 Figure 5.5: Training of a 4-layer linear students of varying size on a regression task generated by a linear teacher. The teacher of outputNy = 4 is y = W x + , with input x ∼ N (x; 0, I N ) of size N .The random teacher matrix W has i.i.d. normally distributed entries of mean 0 and variance 1/N . The noise follow sN ( ; 0, 0.01I N ). The student network has 3 USV-layers, plus one fully connected unconstrained layer x → t(1) → t(2) → t(3) → ŷ. The training on the regression task is done using plain SGD for the MSE loss (ŷ -y)2 . Upper-left: MSE loss on the training and testing sets during training by plain SGD for layers of size N = 1500. Best training loss is 0.004735, best testing loss is 0.004789. Lower-left: Corresponding mutual information evolution between hidden layers and input. Center-left, center-right, right: maximum and squared error of the replica estimation of the mutual information as a function of layers size N , over the course of 5 independent trainings for each value of N for the first, second and third hidden layer.

Figure 5 . 6 :

 56 Figure 5.6: Graphical model representation of the teacher and student model for the learning experiments of non-linear networks, corresponding to results presented on Figure 5.7 and Figure 5.8. The simple generative model is x = W z + with normally distributed latent representation z ∼ N (0, I N 0 )

Figure 5 . 7 :

 57 Figure 5.7: Training of two student models on a binary classification task with correlated input data and either relu (top) or hardtanh (bottom) non-linearities. Left: training and generalization crossentropy loss (left axis) and accuracies (right axis) during learning. Best training-testing accuracies are 0.995 -0.991 for relu version (top row) and 0.998 -0.996 for hardtanh version (bottom row). Remaining columns: mutual information between the input and successive hidden layers. Insets zoom on the first epochs.

Figure 5 . 8 :

 58 Figure 5.8: Learning and hidden-layers mutual information curves for a classification problem with a student using hardtanh activations. Top: Initial weights at layer of variance 4/N -1 , best training accuracy 0.999, best test accuracy 0.994. Middle: Initial weights at layer of variance 1/N -1 , best train accuracy 0.994, best test accuracy 0.9937. Bottom: Initial weights at layer of variance 0.25/N -1 , best train accuracy 0.975, best test accuracy 0.974. The overall direction of evolution of the mutual information can be flipped by a change in weight initialization without changing drastically final performance in the classification task.

Figure 5 . 9 :

 59 Figure5.9: Learning and hidden-layers mutual information curves for a classification problem with correlated input data, using a 4-USV hardtanh layers and 1 unconstrained softmax layer for different levels of regularizing noise: ∆ = 10 -5 (blue), ∆ = 10 -6 (orange), ∆ = 10 -10 (green).

W

Figure 5 . 10 :

 510 Figure 5.10: Representation as graphical model of the assumed noise injection at each layer of the stochastic neural networks studied in Section 5.3.1.

Figure 5 . 11 :Figure 5 . 12 :

 511512 Figure 5.11: Results for 5-hidden layer network with tanh activation on MNIST Left: Average and standard deviation of the best accuracy on the MNIST test set over 10 runs for the different types of noise as a function of the noise amplitude parameter σ 0 . Right: Evolution the mutual information bounds for P = 5000 at the last hidden layer over the course of training for the different noises at the selected value of σ 0 (solid lines with solid colored region in between). The dashed lines and regions corresponds to the bounds computed for an identical network with random Gaussian weight matrices of same variance as the trained weight matrix at the same epoch.

Figure 6 . 1 :

 61 Figure 6.1: (a) Factor graph of the GLM on vector variables corresponding to the joint distribution (6.10). (b)Factor graph for the GLM on vector variables with not perfectly known channel including calibration variables, corresponding to the joint distribution (6.9).

. 95 )

 95 Output functionsIn this setting the output functions have analytical expressions. The channel distribution and partition function arep out (y µ |z µ ) = b a ds µ p s (s µ )(s µ ) P N (z µ ; s µ y µ , ∆),(6.96)Z out (y, ω, V ) = dz p out (y|z) = b ads p s (s)(s) P N (z; sy,

. 99 )ds s P e - 1 2δ (s-ν) 2 ,

 9912 These expressions involve the estimate and variance of the calibration variables under the posterior which can be computed asŝ(y, ω, V ) = f s 1 (y, ω, V ) =b a ds (s) P +1 N (z; sy, ∆) b a ds (s) P N (z; sy, ∆) = I(P + 1, ν, δ, a, b) I(P, ν, δ, a, b) , (6.100) C s (y, ω, V ) = f s 2 (y, ω, V ) = I(P + 2, ν, δ, a, b) I(P, ν, δ, a, b) -ŝ(y, ω, V ) 2 , (6.101) where I(P, ν, δ, a, b) = b a (6.102) δ = (y (V + ∆I P ) -1 y) -1 , (6.103)

Figure 6 . 2 :

 62 Figure 6.2: Normalized cross correlation between Cal-AMP estimate X and teacher signal X 0 for the gain calibration problem with calibration variables uniformly distributed in [0.95, 1.05] and N = 10 3 . Diagrams are plotted as a function of the measurement rate α = M/N and the sparsity level ρ for the offline (top row) and online (bottow row) algorithms, for increasing number of available samples P . The blue line α CS is the phase transition threshold for a perfectly calibrated channel. The pink line α min marks the strict lower bound on the number of measurements necessary for reconstruction. The online Cal-AMP requires more samples than the offline version to achieve comparable errors, nevertheless above the transition relatively low errors are already reached at P = 10. We note that the algorithms are sometimes unstable at low ρ, leading to unexpectedly high MSEs (top left corner of top right diagram).

Figure 6 . 3 :

 63 Figure 6.3: Normalized cross correlation between Cal-AMP estimate ŝ and teacher signal s 0 for the gain calibration problem with calibration variables uniformly distributed in [0.95, 1.05] and N = 10 3 . Diagrams are plotted as a function of the measurement rate α = M/N and the sparsity level ρ for the offline (top row) and online (bottow row) algorithms, for increasing number of available samples P . The pink line α min marks the strict lower bound on the number of measurements necessary for reconstruction. Results are similar to the diagrams in terms of errors on the signal X of Figure 6.2.

sµ (s µ ) = 1 Z

 1 (k+1) out dz (k),µ N (y (k),µ ; z (k),µ /s µ , ∆)N (z (k),µ ; w (k),µ , V (k),µ )p (k)sµ (s µ ),

Figure 6 . 4 :

 64 Figure 6.4: Comparison of mean squared errors on the signal X obtained with Cal-AMP and predicted by SE for the gain calibration problem with calibration variables uniformly distributed in [0.95, 1.05] and N = 10 4 .Along the iterations of the algorithm and comparing fixed points we find a very good agreements of the two procedures. Nevertheless, the Monte Carlo integration in the SE does not allow for great numerical precision for low errors and we start seeing discrepancies between SE and AMP for errors below 10 -8 .

2 ) 2 . 1 ) 91 Algorithm 10

 2219110 90) ∀µ = 1 • • • N MC z µ,k , ω µ,k ∼ N (z µ,k , ω µ,k ; 0, Q (t) (k) ) ∼ p ( ) y µ,k = g 0 (z µ,k , s 0 µ , Following step (2) of Algorithm 10 with -samples of previous steps {y µ,l , z µ,l , ω µ,l } l≤k-1 at convergence -current y µ,k , z µ,k , ω µ,k -current V Update Λ k,µ , Σ k,µ 2.2) Compute ŝµ 2.3) Compute g outµ,k 3) Update r (k) and m(k) . t = t + 1 until convergence end for Output: time series {V (t) (k) , m (t) (k) , m(t) (k) ; t = 1 • • • t max } P k=1Online Gain Calibration Approximate Message Passing Input: matrix Y ∈ R M ×P and matrix W ∈ R M ×N : Initialize: Λ 0,µ = 0, Σ 0,µ = 0, for k = 1

Figure 6 . 5 :

 65 Figure 6.5: (a) Factor graph representation of the proposed constrained matrix factorization setting corresponding to the joint distribution (6.125) with factorization on N , M and Q and with the notation sµ = Sµµ. (b) Factor graph representation of a generic 2-layer GLM with vector variables.

t

  = t + 1 until convergence Output: signal estimate x1 ∈ R N , and estimated covariance C x 1 ∈ R N ×N

Figure 6 . 6 :

 66 Figure 6.6: Teacher-student models for unsupervised learning Top row: Directed graphical model representations of the single-layer and multi-layer teacher generative models for synthetic data set Y . Bottom row: Corresponding factor graphs for inference of the latent representations and network parameters.

Figure 8 :

 8 Figure 8: Factor graph representation of the GLM for the derivation of VAMP

  We start by parametrizing marginals as well as messages coming out of the Dirac factors. For a = 1, 2:m x,(a) (x (a) ) = N (x (a) , x(a) , C x(a) ) , m z,(a) (z (a) ) = N (z(a) , ẑ(a) , C z (a) ) ,

1 Machine learning with neural networks and mean-field approximations (Context and motivation

  ) 1.1 Neural networks for machine learning . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Supervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Unsupervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 A brief history of mean-field methods for neural networks . . . . . . . . . . . . . .

2 Statistical inference and statistical physics (Fundamental theoretical frameworks)

  2.1 Statistical inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Statistical physics of disordered systems . . . . . . . . . . . . . . . . . . . . . . . .

2.1.1 Statistical models representations . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Some inference questions in neural networks for machine learning . . . . . . 2.1.3 Challenges in inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2

3 Selected overview of mean-field treatments: free energies and algorithms

  Naive mean-field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thouless Anderson and Palmer equations . . . . . . . . . . . . . . . . . . . . . . . Belief propagation and approximate message passing . . . . . . . . . . . . . . . . . .3.3 (Generalized) approximate message passing . . . . . . . . . . . . . . . . . . 3.4 Replica method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Steps of a replica computation . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Assumptions and relation to other mean-field methods . . . . . . . . . . . . 3.5 Extensions of interest for this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Streaming AMP for online learning . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 A special special case of GAMP: Cal-AMP . . . . . . . . . . . . . . . . . . 3.5.3 Algorithms and free energies beyond i.i.d. matrices . . . . . . . . . . . . . . 3.5.4 Model composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	(Techniques)
	3.1

3.1.1 Variational derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 When does naive mean-field hold true? . . . . . . . . . . . . . . . . . . . . . 3.2 3.2.1 Outline of the derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Illustration on the Boltzmann machine and important remarks . . . . . . . 3.2.3 Generalizing the Georges-Yedidia expansion . . . . . . . . . . . . . . . . . . 3.3 3.3.1 Generalized linear model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Belief Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3II Contributions 4 Mean-field inference for (deep) unsupervised learning 4.1 Mean-field inference in Boltzmann machines . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Georges-Yedidia expansion for binary Boltzmann machines . . . . . . . . . iii

Machine learning with neural networks and mean-field approximations (Context and motivation)

	1
	X	Matrix
	x	Vector
	I N ∈ R N ×N	Identity matrix
	•	Boltzmann average
	O(N ) ⊂ R N ×N	Orthogonal ensemble
	AMP	Approximate message passing
	BP	Belief Propagation
	CD	Contrastive Divergence
	DAG	Directed Acyclic Graph
	DBM	Deep Boltzmann Machine
	GAMP GLM	Generalized Approximate message passing Generalized Linear Model Part I
	i.i.d.	independent identically distributed
	r-BP RS RBM	relaxed Beleif Propagation Replica Symmetric Restricted Boltzmann Machine Background
	SE	State Evolution
	SGD	Stochastic Gradient Descent
	SK	Sherrington-Kirkpatrick
	TAP	Thouless Anderson Palmer
	VAE	Variational Autoencoder
	VAMP	Vector Approximate message passing
		xi 1

Table 5 . 1 :

 51 Training results for MNIST classification of a fully connected 784-500-10 neural net with a ReLU non linearity. The different rows correspond to different specifications of trainable parameters in the first layer (1, 2, 3a, 3b, 3c) describe in the caption of Figure5.4. We use plain SGD to minimize the cross-entropy loss. All experiments use the same learning rate 0.01 and batchsize of 100 samples. Results are averaged over 5 independent runs, and standard deviations are reported in parentheses.

  We defined the Fourier transform pout of p out (y µ |z µ ) with respect to itsargument z µ = w µ X, pout (y µ |ξ µ ) = dz µ pout (y µ |z µ ) eUsing reciprocally the Fourier representation of p out (y µ |z µ ),

			-iξ	µ	z µ .			(6.14)
	p out (y µ |z µ ) =	1 (2π M )	dξ µ pout (y µ |ξ µ ) e	iξ	µ	z µ ,	(6.15)
	we decouple the integrals over the different x i in (6.11),
	mµ→i

  .32) Defining likewise parameters ω µ , V µ and xi , C x i , and considering their relations to the original λ i→µ , σ i→µ , ω µ→i , V µ→i , xi→µ and C x i→µ we get the vectorized AMP for the GLM presented in Algorithm 7. Note that this step was detailed for Boltzmann machines in Section 4.1.2. Approximate Message Passing for vectors Input: matrix Y ∈ R M ×P and matrix W ∈ R M ×N : Initialize: xi , C x i ∀i and g out µ , ∂ ω g out

	Algorithm 7 µ	∀µ
	repeat
	1) Estimate mean and variance of z µ given current xi

  out y µ |z µ , s µ p s (s µ )N (z µ ; ω µ , V µ ).

	m s µ (s µ ) =	1 Z out	dz µ p s	(6.43)
	So that at convergence of Algorithm 7, we can compute the estimate and uncertainty on the
	calibration variable
	ŝµ

.42)

The estimate of a variable constructed by (sum-product) AMP is always the mean of the approximate posterior marginal distribution. For the calibration variable, the AMP posterior is already displayed in the above Z out ,

  .[START_REF] Gardner | The space of interactions in neural network models[END_REF] Relation to original Cal-AMP derivation In[START_REF] Schülke | Blind Calibration in Compressed Sensing using Message Passing Algorithms[END_REF][START_REF] Schülke | Blind sensor calibration using approximate message passing[END_REF], the Cal-AMP algorithm was derived from the belief propagation equations on the scalar variables of the fully factorized distribution (over N , M and P ). It is equivalent to Algorithm 7 if covariance matrices V µ , ∂ ω g out µ

  • • • P : do Initialize: xi,k , C x i,k ∀i and g outµ,k , ∂ ω g outµ,k ∀µ repeat 1) Estimate mean and variance of z (k) given current x(k) Exploit current y (k) and inherited Λ k-1,µ , Σ k-1,µ 2.1) Update recursion parameters given y (k) Estimates and variances xi , C x

							N
	V	(t) µ,k =	W 2 µi C x i	(t)	(6.110)
							i=1
							N	N
	ω µ,k = (t)	W µi	x(t) i,k -	W 2 µi C x (t) i,k g out	(t-1) µ,k	(6.111)
							i=1	i=1
	2) δ	(t) µ,k	-1	= y 2 µ,k /(V	(t) µ,k + ∆)	(6.112)
				ν µ,k = δ µ,k y µ,k ω (t) µ,k /(V (t) µ,k + ∆) (t)	(6.113)
	Σ (t)				(t) µ,k	(6.114)
	Λ (t)				(t) µ,k ν µ,k (t)	(6.115)
	2.2) Update estimates ŝ and C s
				ŝ(t)
	∂ ω g out	(t) µ,k = C s(t) µ y 2 µ,k /(V µ,k + ∆) 2 -(V (t)	(t) µ,k + ∆) -1	(6.118)
				g out	(t) µ,k = (ŝ (t) µ y µ,k -ω	(t) µ,k )/(V µ,k + ∆), (t)	(6.119)
	3) Estimate mean and variance of x given current optimal z
								M	 -1
	σ	(t) i,k =	 -	W 2 µi ∂ ω g out	(t) µ,k		(6.120)
								µ=1
									M	
	λ (t) i,k =	x(t) i,k + σ i,k (t)		W µi g out	(t) µ,k		(6.121)
								µ=1
	4) Estimate of mean and variance of x augmented of the information about the prior
	C x i,k	(t+1) = f x 2 (λ	(t) i,k , σ i,k ) (t)	(6.122)
				x(t+1) i,k	= f x 1 (λ	(t) i,k , σ i,k ) (t)	(6.123)
	t = t + 1
	until convergence
	end for
	Output:

k,µ = Σ k-1,µ + δ k,µ = Σ k,µ Σ k-1,µ Λ k-1,µ + δ µ = I(k + 1, Λ k,µ , Σ k,µ , a, b) I(k, Λ k,µ , Σ k,µ , a, b) (6.116) C s(t) µ = I(k + 2, Λ k,µ , Σ k,µ , a, b) I(k, Λ k,µ , Σ k,µ ,

a, b) (6.117) 2.3) Update g out and ∂ ω g out i , ŝµ , C s µ

  are following from the definition ofZ y out (ω 2 a , V 2 a ) = dz a p 2 out (y a |z a ) eAlgorithm 11 Generalized Approximate Message Passing for the 2-layer GLMInput: matrix Y ∈ R M ×P and matrices Φ 0 ∈ R M ×N , Φ 1 ∈ R Q×M : Initialize: xi , C x i ∀i, ûµ , C u µ , g 1 out µ , ∂ ω g 1

																out µ	∀µ, g 2 out a , ∂ ω g 2 out a	∀µ and t = 0.
	repeat
	1) Update auxiliary variables of second layer:
	ω 2 a	(t) =		µ	Φ 1 aµ √ N	û(t) µ -			µ		(Φ 1 aµ ) 2 N	σ 1 µ	(t-1) -1	C u	µ	(t) σ 1 (t-1) µ	g 2 out a	(t-1)	(6.137)
	V 2 a	(t) =	µ	(Φ 1 aµ ) 2 N	C u	µ	(t)			(6.138)
	g 2 out a	(t) = g 2 out (y a , ω 2 a	(t) , V 2 a	(t) )	(6.139)
	∂ ω g 2 out a	(t) = ∂ ω g 2 out (y a , ω 2 a	(t) , V 2 a	(t) )	(6.140)
	λ 1 µ	(t) = σ 1 µ	a	Φ 1 aµ √ N	g 2 out a	(t) -	(Φ 1 aµ ) 2 N	∂ ω g 2 out a	(t) ûµ	(6.141)
	σ 1 µ	(t) = -	a	(Φ 1 aµ ) 2 N	∂ ω g 2 out a	(t)	1 2 (z a -ω 2 a ) V 2 a -1	-1 (z a -ω 2 a ) ,	(6.127) (6.142)
	1) Update auxiliary variables of first layer:

Machine learning with neural networks and mean-field approximations (Context and motivation)

(Fundamental theoretical frameworks)

(Techniques)

Selected overview of mean-field treatments: free energies and algorithms(Techniques)

The pseudo likelihood consists in assuming p(x) ≈ N i=1 p(x i |x \i ), where x \i stands for all the components of x expect the i-th, and the conditional probabilities have a closed form expression. Note however that its evaluation for one given x still requires as many operations as the size of the data space which is typically of a few hundreds.

http://yann.lecun.com/exdb/mnist/

http://www.vision.caltech.edu/Image_Datasets/Caltech101/

http://cbcl.mit.edu/software-datasets/heisele/facerecognition-database.html

The change of variable to recover the equivalence of the potentials is A = û/λ 0 , V = q 0 -q, Ã = q and Ṽ = u -λ 0 q 0 /α.

The new auxiliary function is related to the former as FW (AV ) = FW (û(q 0 -q)/λ 0 ) = Fρ λ ,α(q0 -q, û/λ 0 ).

The cross-entropy loss has been found to be more effective than the mean square error in classification tasks. For a problem with M categories, the training outputs are encoded in vectors y ∈ {0, 1} M with only one non-zero

(x i -λ i ) σ -1 i (x i -λ i ) ,(6.126)

Acknowledgements

Additional work not covered in this dissertation

This dissertation does not cover an additional publication, using again mean-field methods yet in the context of Constrained Satisfaction Problems (CSPs), unrelated to deep learning, [START_REF] Nguyen | Inverse statistical problems: from the inverse Ising problem to data science[END_REF] identical to the single layer (6.21). At the interface however, the variables u µ play the role of outputs for the first GLM and of inputs for the second GLM, which translates into a normalization partition function of mixed form

Updates (6.145) and (6.146) are obtained by considering that the second layer acts as an effective channel for the first layer, i.e. from the normalization interpreted as

Finally, update equations (6.151) and (6.152) are in turn derived considering the first layer defines an effective prior for the hidden variables and the normalization as

The rest of the algorithm updates follows as usual from the self-consistency between the different variables introduced as they correspond to different parametrization of the same marginals. The schedule of updates and the time indexing reported in Algorithm 11 results from the entire derivation starting from the BP messages. The generalization of the algorithm to an arbitrary number of layers is easily obtained repeating the heuristic arguments presented here. Lastly, note that the corresponding State Evolution equations can be derived following identical steps as presented in Section 6.1.2.

AMP for constrained matrix factorization

We have already argued that the constrained matrix factorization problem of interest here (6.124) can be interpreted as the combination of two GLM models. It is in fact covered by the two-layer AMP algorithm presented in the previous Section, for the interface channel p 1 out (u µ |z 1 µ ) = ds µ p 1,s out (u µ |z 1 µ )p s (s µ ) = ds µ δ(u µ -s µ z µ )p s (s µ ), (6.131) and to which we should add the equations of estimation of the calibration variables. The interface partition function is

= du ds µ p s (s µ )N (s µ u; λ 1 µ , σ 1 µ )N (u;

where we changed the convention by adding the normalizations of the Gaussian measures compared to (6.128) (an operation that does not affect the definition of the update functions). For the calibration variables, the approximate posterior provided by AMP is therefore

The update functions derived from Z u out can be written in terms of expectations over this posterior. They are given in Appendix B. < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 I r Y t e F b 6 S e j K k a T p t b X y D q F c g o = " > A A A B 8 n i c b V B N S 8 N A F H y p X 7 V + V T 1 6 W S y C p 5 K I o N 4 K X j x 4 q G B s o S l l s 9 2 0 S z e b s P s i l N C / 4 c W D i l d / j T f / j Z s 2 B 6 0 O L A w z 7 / F m J 0 y l M O i 6 X 0 5 l Z X V t f a O 6 W d v a 3 t n d q + 8 f P J g k 0 4 z 7 L J G J 7 o b U c C k U 9 1 G g 5 N 1 U c x q H k n f C y X X h d x 6 5 N i J R 9 z h N e T + m I y U i w S h a K Q h i i u M w y n E 2 u B 3 U G 2 7 T n Y P 8 J V 5 J G l C i P a h / B s O E Z T F X y C Q 1 p u e 5 K f Z z q l E w y W e 1 I D M 8 p W x C R 7 x n q a I x N / 1 8 n n l G T q w y J F G i 7 V N I 5 u r P j Z z G x k z j 0 E 4 W G c 2 y V

Towards a model for deep Bayesian (online) learning

e r H e r Y / F a M O q d 4 7 h D 6 z P H 7 m / k 9 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 u 0 L 0 9 J 3 R E q F w 4 F k 2 P h p Q j E 5 7 w 4 = " > A

e r H e r Y / F a M O q d 4 7 h D 6 z P H 7 m / k 9 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 u 0 L 0 9 J 3 R E q F w 4 F k 2 P h p Q j E 5 7 w 4 = " > A

e r H e r Y / F a M O q d 4 7 h D 6 z P H 7 m / k 9 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 u 0 L 0 9 J 3 R E q F w 4 F k 2 P h p Q j E 5 7 w 4 = " > A

x y 5 w j + w P n 8 A W P q k N 8 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z 8 8 n N P e M + e j M Q y U l s q 4 q 2 P O / v 6 8 = " >

x y 5 w j + w P n 8 A W P q k N 8 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z 8 8 n N P e M + e j M Q y U l s q 4 q 2 P O / v 6 8 = " >

x y 5 w j + w P n 8 A W P q k N 8 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z 8 8 n N P e M + e j M Q y U l s q 4 q 2 P O / v 6 8 = " >

x y 5 w j + w P n 8 A W P q k N 8 = < / l a t e x i t > W (1) ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > W (2) ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > W (L) ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > W (2) ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > W (L) ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " w F W a y g z E 2 c X N 9 w V r u i a S G n u o J

J Z P X i X t i 7 p n + d 1 l r e E W d Z T h B E 7 h H D y 4 g g b c Q h N a w E D B M 7 z C m 2 O c F + f d + V i M l p x i 5 x j + w P n 8 A f r b k Q w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w F W a y g z E 2 c X N 9 w V r u i a S G n u o J P o = " >

J Z P X i X t i 7 p n + d 1 l r e E W d Z T h B E 7 h H D y 4 g g b c Q h N a w E D B M 7 z C m 2 O c F + f d + V i M l p x i 5 x j + w P n 8 A f r b k Q w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w F W a y g z E 2 c X N 9 w V r u i a S G n u o J P o = " >

J Z P X i X t i 7 p n + d 1 l r e E W d Z T h B E 7 h H D y 4 g g b c Q h N a w E D B M 7 z C m 2 O c F + f d + V i M l p x i 5 x j + w P n 8 A f r b k Q w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w F W a y g z E 2 c X N 9 w V r u i a S G n u o J P o = " >

x y 5 w j + w P n 8 A W P q k N 8 = < / l a t e x i t > Teacher W < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t >

s c q p l r x L / 8 w a Z j q 6 D g o k 0 0 1 S Q x a E o 4 0 g n q K o B j Z i k R P P c E E w k M 1 k R m W C J i T Z l t U w J 7 v K X V 4 l 3 0 b n p u P e X 7 a 5 T t 9 G E E z i F c 3 D h C r p w B z 3 w g M A M n u E V 3 q z C e r H e r Y / F a M O q d 4 7 h D 6 z P H 7 m / k 9 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 u 0 L 0 9 J 3 R E q F w 4 F k 2 P h p Q j E [START_REF] Daniel | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF]

w 4 = " >

K e f I S n R l l h K J E m i c 0 m q u / N w o c K 5 X H o Z m s c q p l r x L / 8 w a Z j q 6 D g o k 0 0 1 S Q x a E o 4 0 g n q K o B j Z i k R P P c E E w k M 1 k R m W C J i T Z l t U w J 7 v K X V 4 l 3 0 b n p u P e X 7 a 5 T t 9 G E E z i F c 3 D h C r p w B z 3 w g M A M n u E V 3 q z C e r H e r Y / F a M O q d 4 7 h D 6 z P H 7 m / k 9 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 u 0 L 0 9 J 3 R E q F w 4 F k 2 P h p Q j E [START_REF] Daniel | Storing infinite numbers of patterns in a spin-glass model of neural networks[END_REF]

K e f I S n R l l h K J E m i c 0 m q u / N w o c K 5 X H o Z m s c q p l r x L / 8 w a Z j q 6 D g o k 0 0 1 S Q x a E o 4 0 g n q K o B j Z i k R P P c E E w k M 1 k R m W C J i T Z l t U w J 7 v K X V 4 l 3 0 b n p u P e X 7 a 5 T t 9 G E E z i F c 3 D h C r p w B z 3 w g M A M n u E V 3 q z C e r H e r Y / F a M O q d 4 7 h D 6 z P H 7 m / k 9 0 = < / l a t e x i t > W (1) < l a t e x i t s h < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > real dataset learning from synthetic data {x k ; y k } 1kK < l a t e x i t s h a _ b a s e = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a _ b a s e = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a _ b a s e = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > < l a t e x i t s h a _ b a s e = " ( n u l l ) " > ( n u l l ) < / l a t e x i t > allow for analytical computations of interesting observables such as learning curves. An exciting direction is instead to introduce some tunable structure in the synthetic data. This strategy was already adopted in [START_REF] Kadmon | Optimal architectures in a solvable model of deep networks[END_REF][START_REF] Chung | Classification and Geometry of General Perceptual Manifolds[END_REF][START_REF] Cohen | Separability and Geometry of Object Manifolds in Deep Neural Networks[END_REF]. These works used mixture of Gaussians or parametric manifolds, which can be more or less clustered, to derive theoretical learning properties as a function of the data properties using mean-field methods.

In Chapter 5, we further generalized the classical matched teacher-student scenario to include generative models. Although it was not the case in the experiments run in this Chapter, these generative models can be trained following state-of-the art unsupervised learning methods to produce synthetic datasets looking like real ones (images, sounds, texts etc.), as sketched on Figure 6.7. Following ideas developed in Chapters 4 and 5, mean-field inference should remain efficient in these models during learning with usual algorithms such as SGD. Additionally, the perspectives of Chapter 6 envisage also the analysis of a Bayesian learning of the weights using message passing algorithms and State Evolutions, both in the generative model and in the student network.

The combination of these different ideas pushes the idea of modeling learning tasks in an exciting direction. The progress in mean-field inference in compositional models will possibly allow a theoretical characterization of these teacher-student settings. Even if not, the empirical study of tunable sophisticated scenarios mimicking real learning tasks should improve our understanding, among other points, of the role of data structures in deep learning.

Appendix B Update functions for constrained matrix factorization

We consider the interface partition function

from which we deduce the approximate posterior provided by AMP

The update functions derived from Z u out are written in terms of expectations over this posterior.

Input update functions f u 1 and f u 2

These updates do not have a closed form, yet they only require a one-dimensional numerical integral over the calibration variables and therefore remain tractable. Note that here again the corresponding State Evolution can be derived by combining the results from the multi-layer version and the Cal-AMP problem. In the same manner, the online AMP and SE shall be easily written.

R ÉSUM É

Les algorithmes d'apprentissage automatique utilisant des r éseaux de neurones profonds ont r écemment r évolutionn é l'intelligence artificielle. Malgr é l'engouement suscit é par leurs diverses applications, les excellentes performances de ces algorithmes demeurent largement inexpliqu ées sur le plan th éorique. Ces probl èmes d'apprentissage sont d écrits math ématiquement par de tr ès grands ensembles de variables en interaction, difficiles à manipuler aussi bien analytiquement que num ériquement. Cette multitude est pr écis ément le champ d' étude de la physique statistique qui s'attelle à comprendre, originellement dans les syst èmes naturels, comment rendre compte des comportements macroscopiques à partir de cette complexit é microscopique. Dans cette th èse nous nous proposons de mettre à profit les progr ès r écents des m éthodes de champ moyen de la physique statistique des syst èmes d ésordonn és pour d ériver des approximations pertinentes dans ce contexte. Nous nous appuyons sur les équivalences et les compl émentarit és entre les algorithmes de passage de message, les d éveloppements haute temp érature et la m éthode des r épliques. Cette strat égie nous m ène d'une part à des contributions pratiques pour l'apprentissage non supervis é des machines de Boltzmann. Elle nous permet d'autre part de contribuer à des r éflexions th éoriques en consid érant le paradigme du professeur-étudiant pour mod éliser des situations d'apprentissage. Nous d éveloppons une m éthode pour caract ériser dans ces mod èles l' évolution de l'information au cours de l'entraînement, et nous proposons une direction de recherche afin de g én éraliser l' étude de l'apprentissage bay ésien des r éseaux de neurones à une couche aux r éseaux de neurones profonds.

MOTS CL ÉS

Syst èmes d ésordonn és, inf érence, apprentissage automatique, passage de message, m éthode des r épliques ABSTRACT Machine learning algorithms relying on deep neral networks recently allowed a great leap forward in artificial intelligence. Despite the popularity of their applications, the efficiency of these algorithms remains largely unexplained from a theoretical point of view. The mathematical descriptions of learning problems involves very large collections of interacting random variables, difficult to handle analytically as well as numerically. This complexity is precisely the object of study of statistical physics. Its mission, originally pointed towards natural systems, is to understand how macroscopic behaviors arise from microscopic laws. In this thesis we propose to take advantage of the recent progress in mean-field methods from statistical physics to derive relevant approximations in this context. We exploit the equivalences and complementarities of message passing algorithms, high-temperature expansions and the replica method. Following this strategy we make practical contributions for the unsupervised learning of Boltzmann machines. We also make theoretical contributions considering the teacher-student paradigm to model supervised learning problems. We develop a framework to characterize the evolution of information during training in these model. Additionally, we propose a research direction to generalize the analysis of Bayesian learning in shallow neural networks to their deep counterparts.

KEYWORDS

Disordered systems, inference, message passing, replica method