Je Voudrais Également Remercier

Nicolas Macris

Jean

Francesco Caltagirone

Thibault Lesieur

Aukosh Jagannath

Mes premiers remerciements sont bien évidemment pour Marc pour tout ce qu'il a fait pour moi au cours de cette thèse. Toujours de bonne humeur, disponible, encourageant, il m'a très bien guidé et a su rendre ces trois années très enrichissantes et agréables. Je pense que j'aurais difficilement pu espérer avoir un encadrant aussi sympa, on a bien rigolé.

Ce fut également un bonheur de travailler avec Lenka Zdeborová et Florent Krzakala que je remercie profondément pour m'avoir fait voyager, mais surtout pour toutes ces discussions qui m'ont appris à parler physicien. Ces trois années n'auraient pas été aussi intéressantes si je n'avais pas croisé leur route.

us to understand when computational gaps appears. The manuscript is organised as follows.

• Chapter 1 introduces the basics concepts of Bayes-optimal inference that we will used repeatedly throughout this manuscript. I am grateful to Andrea Montanari for interesting discussions regarding Section 1.2.

• Chapter 2 presents a useful "decoupling principle" that will simplify the study of the more complex models investigated in this thesis.

• Chapter 3 and Chapter 4 focus on low-rank matrix estimation and establish the information-theoretic limits for this problems. They are based on the paper [START_REF] Lelarge | Fundamental limits of symmetric low-rank matrix estimation[END_REF] in collaboration with Marc Lelarge and the work [START_REF] Miolane | Fundamental limits of low-rank matrix estimation: the non-symmetric case[END_REF].

• Chapter 5 studies the community detection problem in the Stochastic Block Model, when there is two communities of unequal sizes. It is excerpt from the joint work [START_REF] Caltagirone | Recovering asymmetric communities in the stochastic block model[END_REF] together with Francesco Caltagirone and Marc Lelarge.

• Chapter 6 pursue the directions of Chapters 3-4. The first part studies the statistical limits of low-rank tensor estimation is based on the paper [START_REF] Thibault Lesieur | Statistical and computational phase transitions in spiked tensor estimation[END_REF] with Thibault Lesieur, Marc Lelarge, Florent Krzakala and Lenka Zdeborová. The second part analyzes maximum likelihood estimations, following the paper [START_REF] Jagannath | Statistical thresholds for tensor pca[END_REF] with Aukosh Jagannath and Patrick Lopatto.

• Chapter 7 focuses on the statistical and computational limits of estimation in Generalized Linear Models. It is based on the work [START_REF] Barbier | Optimal errors and phase transitions in high-dimensional generalized linear models[END_REF] with Jean Barbier, Florent Krzakala, Nicolas Macris and Lenka Zdeborová.

• Chapters 8 and 9 concern the Lasso estimator. They establish uniform control of the distribution of the Lasso and study methods to tune the penalization parameter. This work is a collaboration with Andrea Montanari [START_REF] Miolane | The distribution of the lasso: Uniform control over sparse balls and adaptive parameter tuning[END_REF].

All the inference models that we consider in this manuscript have an equivalent within the statistical physics literature. The only difference is that the problems we study here contain a planted solution (representing some signal) whereas their equivalent in physics can be seen as "pure noise" models. More precisely, the Spiked Wigner model from Chapter 3 corresponds to the Sherrington-Kirkpatrick model [START_REF] Sherrington | Solvable model of a spin-glass[END_REF] while the Spiked Wishart model from Chapter 4 is the analog to the bipartite SK model [START_REF] Barra | Equilibrium statistical mechanics of bipartite spin systems[END_REF] or the Hopfield model [START_REF] John | Neural networks and physical systems with emergent collective computational abilities[END_REF]. The spiked tensor model of Chapter 6 is linked to the p-spin model and the Generalized Linear Models from Chapter 7 can be seen as perceptrons [START_REF] Gardner | The space of interactions in neural network models[END_REF][START_REF] Gardner | Optimal storage properties of neural network models[END_REF][START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF] with various threshold functions. Finally, the Lasso estimator studied in Chapter 8 and 9 is linked to the "Shcherbina and Tirozzi" model [START_REF] Shcherbina | Rigorous solution of the gardner problem[END_REF].

Some notations R ≥0

non-negative numbers: [0, +∞) R >0

positive numbers: (0, +∞) E expectation with respect to all random variables E X expectation with respect to the random variable X only W 2

Wasserstein distance of order 2 D TV total variation distance D KL Kullback-Leibler divergence

(d) -→ convergence in distribution S n-1
unit sphere of R n (x; y) or x, y dot product between x and y x • y normalized dot product:

x • y = 1 n n i=1 x i y i x 0 0 norm of x: x 0 = #{i|x i = 0} |x| 1 norm of x: |x| = |x i | x 2 norm of x: x = x 2 i φ(x

Bayes-optimal inference

We introduce in this chapter some general properties of Bayes-optimal inference, that will be used repeatedly in the sequel. Let us first define what we mean by Bayes-optimal inference.

We consider a statistical problem where we would like to recover a signal vector X ∈ R n from some observations Y ∈ R m . We assume that (X, Y ) is drawn from some probability distribution µ over R n × R m . Given a performance criterion, a Bayes-optimal estimator (or simply Bayes estimator) is an estimator of X given Y that achieves the best performance for this criterion. For instance if we measure the performance of an estimator x by its mean square error MSE( x) = E Xx(Y ) 2 , then the Bayes-optimal estimator is simply the posterior mean x Bayes

(Y ) = E[X|Y ].
The goal of this chapter is to present some general properties of Bayes-optimal estimators. In Section 1. [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF] we introduce what we will call (according to the statistical physics terminology) the "Nishimori identity" which is nothing more than a rewriting of Bayes rule. In Section 1.2 we will study the links between various natural performance metrics for estimators and show that they are in some sense equivalent. In Sections 1.3 we analyse the special case where Y = √ λX + Z, where λ ≥ 0 and Z is some Gaussian noise. This is the starting point of the study of the "spiked" matrix and tensor models. Finally we consider in Section 1.4 a simple example to illustrate the tools of this chapter.

The Nishimori identity

In order to analyze Bayes-optimal estimators, we will need to examine the posterior distribution of X given Y . To do so we will often consider i.i.d. samples x (1) , . . . , x (k) from the posterior distribution P (• | Y ), independently of everything else. Such samples are called replicas. The (obvious) identity below (which is simply Bayes rule) is named after the works of Nishimori [START_REF] Nishimori | Exact results and critical properties of the ising model with competing interactions[END_REF][START_REF] Nishimori | Internal energy, specific heat and correlation function of the bond-random ising model[END_REF] on "gauge-symmetric" spin glasses. It states that the planted solution X behaves like a replica.

Proposition 1.1.1 (Nishimori identity)

Let (X, Y ) be a couple of random variables on a polish space. Let k ≥ 1 and let x (1) , . . . , x (k) be k i.i.d. samples (given Y ) from the distribution P(X = • | Y ), independently of every other random variables. Let us denote • the expectation with respect to P(X = • | Y ) and E the expectation with respect to (X, Y ). Then, for all continuous bounded function f E f (Y , x (1) , . . . , x (k) ) = E f (Y , x (1) , . . . , x (k-1) , X) .

Proof. It is equivalent to sample the couple (X, Y ) according to its joint distribution or to sample first Y according to its marginal distribution and then to sample X conditionally to Y from its conditional distribution P(X = • | Y ). Thus the (k + 1)-tuple (Y , x (1) , . . . , x (k) ) is equal in law to (Y , x (1) , . . . , x (k-1) , X).

Performance measure and optimal estimators

We consider two random vectors X and Y that live respectively in R n and R m . We assume (for simplicity) that X = 1 almost surely. As explained above, given the observations Y , our goal is to estimate X with an estimator x(Y ). In order to evaluate the performance of such an estimator, what criterion should we take?

The probably most natural way to characterize the performance of x is by its meansquared error: MSE( x) = E Xx(Y ) 2 .

By Pythagorean theorem, we know that the optimal estimator which respect to this metric is the posterior mean x(Y ) = E[X|Y ] which achieves the minimal mean square error:

MMSE(X|Y ) def = E X -E[X|Y ] 2 . (1.2.1)
However, the MSE is not always an appropriate criterion. Indeed in many cases it is only possible to recover X up to its sign: think for instance of the Spiked Wigner Model Y = XX T + Noise with X ∼ Unif(S n-1 ). In such case, E[X|Y ] = 0: the best estimator in term of MSE does not even depend on the observations Y ! For this kind of problems one should rather consider the correlation (also known as cosine similarity) in absolute value between x and the signal X:

sup x =1 E |( x(Y ); X)| or sup x =1 E ( x(Y ); X) 2 , (1.2.2)
where (•; •) denotes the Euclidean inner product and where the suprema are taken over all estimators x : R m → S n-1 .

Let us introduce some notations. We will use the Gibbs Bracket • to write expectations with respect to the posterior distribution of X given Y :

f (x) = E[f (X)|Y ],
for all measurable function f such that f (X) is integrable. In particular, we will be interested by the n × n positive semi-definite (random) matrix:

M def = xx T = E[XX T |Y ].
(1.2.3) M is the Bayes-optimal estimator (in terms of mean square error) for estimating the matrix

XX T : MMSE(XX T |Y ) = E XX T -E[XX T |Y ] 2 .
(1.2.4)

An easy computation gives MMSE(XX T |Y ) = 1 -E[Tr(M 2 )]. The matrix M is also related to the second quantity of (1.2.2) through its largest eigenvalue λ max (M ):

Lemma 1.2.1 sup x =1 E ( x(Y ); X) 2 = E λ max (M )
and the optimal estimator for this metric is a unit eigenvector of M associated to its largest eigenvalue λ max (M ).

Proof. Let x be an estimator of X. By the Nishimori identity (Proposition 1.1.1), we have

E ( x(Y ); X) 2 = E x(Y ) T XX T x(Y ) = E x(Y ) T xx T x(Y ) = E x(Y ) T M x(Y ) ,
the lemma follows.

Lemma 1.2.1 tells us that the top unit eigenvector v of M maximizes E [( x(Y ); X) 2 ]. In the following, we will show that under a simple condition (that will hold for the models we consider in this manuscript), the estimator v is "asymptotically optimal" (in the limit of large dimension) for the two metrics (1.2.2) and λ max v v T is optimal for the estimation of XX T in terms of mean square error.

To introduce this condition and the asymptotic limit, we need to consider to a sequence of inference problems. We assume that for all n ≥ 1 we have two random vectors X [n] and Y [n] respectively in S n-1 and R mn , for some sequence (m n ) n≥1 . Our goal is again to estimate X [n] from the observation of Y [n] when n is very large: we would like for instance to compute the limits of (1.2.2) and (1.2.4) as n → ∞. Moreover, we would like to know which estimators are "asymptotically optimal", i.e. whose performance reach in the n → ∞ limit the optimal one. In the following, in order to simplify the notations, we will write X and Y instead of X [n] and Y [n] .

Proposition 1.2.1

Let us denote by G n the posterior distribution of X given Y . Notice that G n is a random probability distributions on S n-1 . Assume that there exists q ∈ [0, 1] such that for x (1) , x (2) i.i.d.

∼ G n we have x (1) ; x (2) (d)

---→ n→∞ q.

(1.2.5)

Then Tr(M 2 )

(d)
---→ n→∞ q 2 and λ max (M )

(d) ---→ n→∞ q.
Proof. Let us compute Tr(M 2 ) = Tr xx T xx T = (x (1) ; x (2) ) 2 ---→ n→∞ q 2 , by assumption.

If q = 0, then the result is obvious since λ max (M ) 2 ≤ Tr(M 2 ). Notice that Tr(M 3 ) ≤ λ max (M )Tr(M 2 ), so it suffices to show that Tr(M 3 ) = (x (1) ; x (2) )(x (2) ; x (3) )(x (3) ; x (1) ) ---→ n→∞ q 3 , for x (1) , x (2) , x (3) i.i.d. ∼ G n . This follows from Lemma 1.2.2 below.

Lemma 1.2.2

Under the assumptions of Proposition 1.2.1, we have for x (1) , x (2) , x (3) i.i.d. ∼ G n , (x (1) ; x (2) )(x (2) ; x (3) )(x (3) ; x (1) )

(d)
---→ n→∞ q 3 . Lemma 1.2.2 will be proved in Appendix A. From Proposition 1.2.1 we deduce the main result of this section: Proposition 1.2.2

Let v be a leading unit eigenvector of M (which is defined by (1.2.3)). Under the assumptions of Proposition 1.2.1, we have

|( v; X)| (d) ---→ n→∞ √ q.
(1.2.6) (1.2.8)

Further lim n→∞ MMSE(XX T |Y ) = 1 -q 2 ,
Hence if q = 0, the Proposition follows easily. Assume now that q > 0. Using Pythagorean Theorem and the Nishimori identity (Proposition 1.1.1) we get (1) ; x (2) ) 4 ---→ n→∞ 1 -q 4 , where the last limit follows from the assumption (1.2.5). Since 4 , using Proposition 1.2.1, we deduce (recall that we assumed q > 0) that lim sup n→∞ E ( v; X) 4 ≤ q 2 . Together with (1.2.8) this gives that |( v; X)| ---→ n→∞ √ q.

E X ⊗4 -λ 2 max v ⊗4 2 ≥ E X ⊗4 -x ⊗4 2 = 1 + E x ⊗4 ; x ⊗4 -2E X ⊗4 ; x ⊗4 = 1 -E (x
E X ⊗4 -λ 2 max v ⊗4 2 = 1 + E λ 4 max -2E λ 2 max (X; v)
The next point is a consequence of Proposition 1.2.1 because MMSE(XX T |Y ) = 1-E[Tr(M 2 )]. To prove (1.2.7) simply notice that

E |( v; X)| 2 ≤ sup x =1 E |( x(Y ); X)| 2 ≤ sup x =1 E ( x(Y ); X) 2 = E[λ max ],
which proves (1.2.7) using (1.2.6) and Proposition 1.2.1.

From Proposition 1.2.2, we deduce that the estimator A def = λ max (M ) v v T achieves asymptotically the minimal mean square error for the estimation of XX T :

lim n→∞ E XX T -A 2 = 1 -q 2 .
Remark 1.2.1. For simplicity we assumed in this section that X 2 = 1 almost surely. However we will need to work in the next chapters under a slightly weaker condition, namely X 2 ---→ n→∞ 1. It is not difficult to modify the proofs of this section to see that Lemma 1.2.1, Proposition 1.2.1, Lemma 1.2.2 and Proposition 1.2.2 still hold, provided that X 2 ---→ n→∞ 1 for the Wasserstein distance of order 4 (i.e. X 2 ---→ n→∞ 1 in distribution and E X 8 ---→ n→∞ 1).

Bayesian inference with Gaussian noise

We will now focus on the following model:

Y = √ λ X + Z , (1.3.1)
where the signal X is sampled according to some probability distribution P X over R n , and where the noise Z = (Z 1 , . . . , Z n ) i.i.d.

∼ N (0, 1) is independent from X. In Chapters 3 and 4, X will typically be a low-rank matrix. The parameter λ ≥ 0 plays the role of a signal-to-noise ratio. We assume that P X admits a finite second moment: E X 2 < ∞.

Given the observation channel (1.3.1), the goal of the statistician is to estimate X given the observations Y . Again, we assume to be in the "Bayes-optimal" setting, where the statistician knows all the parameters of the inference model, that is the prior distribution P X and the signal-to-noise ratio λ. We measure the performance of an estimator θ (i.e. a measurable function of the observations Y ) by its Mean Squared Error MSE( θ) = E Xθ(Y ) 2 . One of our main quantity of interest will be the Minimum Mean Squared Error MMSE(λ)

def = min θ MSE( θ) = E X -E[X|Y ] 2 ,
where the minimum is taken over all measurable function θ of the observations Y . Since the optimal estimator (in term of Mean Squared Error) is the posterior mean of X given Y , a natural object to study is the posterior distribution of X. By Bayes rule, the posterior distribution of X given Y is

dP (x | Y ) = 1 Z(λ, Y ) e H λ,Y (x) dP X (x) , (1.3.2)
where

H λ,Y (x) = √ λ x T Y - λ 2 x 2 = √ λ x T Z + λ x T X - λ 2 x 2 . Definition 1.3.1
H λ,Y is called the Hamiltonian1 and the normalizing constant

Z(λ, Y ) = dP X (x)e H λ,Y (x)
is called the partition function.

Expectations with respect the posterior distribution (1.3.2) will be denoted by the Gibbs brackets • λ :

f (x) λ = E f (X)|Y = 1 Z(λ, Y ) dP X (x)f (x)e H λ,Y (x) ,
for any measurable function f such that f (X) is integrable.

Definition 1.3.2

F (λ) = E log Z(λ, Y
) is called the free energy 2 . It is related to the mutual information between X and Y by

F (λ) = λ 2 E X 2 -I(X; Y ) . (1.3.3)
Proof. The mutual information I(X; Y ) is defined as the Kullback-Leibler divergence between P (X,Y ) , the joint distribution of (X, Y ) and P X ⊗P Y the product of the marginal distributions of X and Y . P (X,Y ) is absolutely continuous with respect to P X ⊗ P Y with Radon-Nikodym derivative:

dP (X,Y ) dP X ⊗P Y (X, Y ) = exp -1 2 Y - √ λX 2 exp -1 2 Y - √ λx 2 dP X (x) . Therefore I(X; Y ) = E log dP (X,Y ) dP X ⊗P Y (X, Y ) = -E log dP X (x) exp √ λx T Y - √ λX T Y - λ 2 x 2 + λ 2 X 2 = -F (λ) + λ 2 E X 2 .
We state now two basic properties of the MMSE. A more detailed analysis can be found in [START_REF] Guo | Estimation in gaussian noise: Properties of the minimum mean-square error[END_REF][START_REF] Wu | Functional properties of minimum mean-square error and mutual information[END_REF].

Proposition 1.3.1 λ → MMSE(λ) is non-increasing over R ≥0 . Moreover • MMSE(0) = E X -E[X] 2 , • MMSE(λ) ----→ λ→+∞ 0. Proposition 1.3.2 λ → MMSE(λ) is continuous over R ≥0 .
The proofs of Proposition 1.3.1 and 1.3.2 can respectively be found in Appendix B.1 and B.2.

We present now the very useful "I-MMSE" relation from [START_REF] Guo | Mutual information and minimum mean-square error in gaussian channels[END_REF]. This relation was previously known (under a different formulation) as "de Brujin identity" see [START_REF] Aart | Some inequalities satisfied by the quantities of information of fisher and shannon[END_REF]Equation 2.12]. Proposition 1.

3.3

For all λ ≥ 0, 2 -MMSE(λ) .

∂ ∂λ I(X; Y ) = 1 2 MMSE(λ) and
F (λ) = 1 2 E x T X λ = 1 2 E X
(1.3.4) F thus is a convex, differentiable, non-decreasing, and 1 2 E X 2 -Lipschitz function over R ≥0 . If P X is not a Dirac mass, then F is strictly convex.

Proposition 1.3.3 is proved in Appendix B.3. Proposition 1.3.3 reduces the computation of the MMSE to the computation of the free energy. This will be particularly useful because the free energy F is much easier to handle than the MMSE.

We end this section with the simplest model of the form (1.3.1), namely the additive Gaussian scalar channel:

Y = √ λX + Z , (1.3.5)
where Z ∼ N (0, 1) and X is sampled from a distribution P 0 over R, independently of Z.

The corresponding free energy and the MMSE are respectively

ψ P 0 (λ) = E log dP 0 (x)e √ λ Y x-λx 2 /2
and

MMSE P 0 (λ) = E X -E[X|Y ] 2 . (1.3.6)
The study of this simple inference channel will be very useful in the following, because we will see that the inference problems that we are going to study enjoy asymptotically a "decoupling principle" that reduces them to scalar channels like (1.3.5).

Let us compute the mutual information and the MMSE for particular choices of prior distributions:

Example 1.3.1 (Gaussian prior: P 0 = N (0, 1)). In that case E[X|Y ] is simply the orthogonal projection of X on Y :

E[X|Y ] = E[XY ] E[Y 2 ] Y = √ λ 1 + λ Y.
One deduces MMSE P 0 (λ) = 1 1+λ . Using (1.3.4), we get I(X; Y ) = 1 2 log(1+λ) and ψ P 0 (λ) =

ψ P 0 (λ) = 1 2 (λ -log(1 + λ)),
where the supremum and infimum are both over the probability distributions that have unit second moment. The standard normal distribution P 0 = N (0, 1) achieves both extrema.

Example 1.3.2 (Rademacher prior: P 0 = 1 2 δ +1 + 1 2 δ -1 ). We compute ψ P 0 (λ) = E log cosh( √ λZ+ λ) -λ 2 and I(X; Y ) = λ -E log cosh( √ λZ + λ). The I-MMSE relation gives

1 2 MMSE(λ) = ∂ ∂λ I(X; Y ) = 1 -E 1 2 √ λ Z + 1 tanh √ λZ + λ = 1 -E tanh( √ λZ + λ) - 1 2 E tanh ( √ λZ + λ) = 1 2 -E tanh( √ λZ + λ) + 1 2 E tanh 2 ( √ λZ + λ)
where we used Gaussian integration by parts. Since by the Nishimori property E xX λ = E x 2 λ , one has E tanh(

√ λZ + λ) = E tanh 2 ( √ λZ + λ) and therefore MMSE(λ) = 1 - E tanh( √ λZ + λ).

A warm-up: the "needle in a haystack" problem

In order to illustrate the results seen in the previous sections, we study now a very simple inference model. Let (e 1 , . . . , e 2 n ) be the canonical basis of R 2 n . Let σ 0 ∼ Unif({1, . . . , 2 n }) and define X = e σ 0 (i.e. X is chosen uniformly over the canonical basis of R 2 n ). Suppose here that we observe:

Y = √ λnX + Z ,
where Z = (Z 1 , . . . , Z 2 n ) i.i.d.

∼ N (0, 1), independently from σ 0 . The goal here is to estimate X or equivalently to find σ 0 . The posterior distribution reads:

P(σ 0 = σ|Y ) = P(X = e σ |Y ) = 1 Z n (λ) 2 -n exp √ λne T σ Y - λn 2 e σ 2 = 1 Z n (λ) 2 -n exp √ λnZ σ + λn1(σ = σ 0 ) - λn 2 ,
where Z n (λ) is the partition function

Z n (λ) = 1 2 n 2 n σ=1 exp √ λnZ σ + λn1(σ = σ 0 ) - λn 2 .
We will be interested in computing the free energy F n (λ) = 1 n E log Z n (λ) in order to deduce then the minimal mean squared error using the I-MMSE relation (1.3.4) presented in the previous section.

Although its simplicity, this model is interesting for many reasons. First, it is one of the simplest statistical model for which one observes a phase transition. Second it is the "planted" analog of the random energy model (REM) introduced in statistical physics by Derrida [START_REF] Derrida | Random-energy model: Limit of a family of disordered models[END_REF][START_REF] Derrida | Random-energy model: An exactly solvable model of disordered systems[END_REF], for which the free energy reads 1 n E log σ 1 2 n exp √ λnZ σ . Third, as we will see in Section 6.1.1, this model correspond to the "large order limit" of a rank-one tensor estimation model.

We start by computing the limiting free energy:

Theorem 1.4.1

lim n→∞ F n (λ) =    0 if λ ≤ 2 log 2 , λ 2 -log(2) if λ ≥ 2 log 2 .
Proof. Using Jensen's inequality

F n (λ) ≤ 1 n E log E [Z n (λ)|σ 0 , Z σ 0 ] = 1 n E log 1 - 1 2 n + e √ λnZσ 0 + λn 2 -log(2)n ≤ 1 n E log 1 + e λn 2 -log(2)n + λ n ---→ n→∞ 0 if λ ≤ 2 log(2) , λ 2 -log(2) if λ ≥ 2 log(2) .
F n is non-negative since F n (0) = 0 and F n is non-decreasing. We have therefore F n (λ) ---→ n→∞ 0 for all λ ∈ [0, 2 log [START_REF] Michael Aizenman | Extended variational principle for the sherrington-kirkpatrick spin-glass model[END_REF]]. We have also, by only considering the term σ = σ 0 :

F n (λ) ≥ 1 n E log e √ λnZσ 0 + λn 2 2 n = λ 2 -log(2) .
We obtain therefore that F n (λ) ---→ n→∞ λ 2 -log(2) for λ ≥ 2 log [START_REF] Michael Aizenman | Extended variational principle for the sherrington-kirkpatrick spin-glass model[END_REF].

Using the I-MMSE relation (1.3.4), we deduce the limit of the minimum mean squared error MMSE n (λ) = min θ E X -θ(Y ) 2 :

MMSE n (λ) = E X 2 -2F n (λ) = 1 -2F n (λ) .
F n is a convex function of λ, thus (see Proposition C.1) its derivative converges to the derivative of its limit at each λ at which the limit is differentiable, i.e. for all λ ∈ (0, +∞) \ {2 log(2)}. We obtain therefore that for all λ > 0,

• if λ < 2 log(2), then MMSE n (λ) ---→ n→∞ 1: one can not recover X better than a random guess.

• if λ > 2 log(2), then MMSE n (λ) ---→ n→∞ 0: one can recover X perfectly.

Of course, the result we obtain here is (almost) trivial since the maximum likelihood estimator σ(Y ) = arg max

1≤σ≤2 n Y σ
of σ 0 is easy to analyze. Indeed, max σ Z σ 2 log(2)n with high probability so that the maximum likelihood estimator recovers perfectly the signal for λ > 2 log(2) with high probability.

Chapter 2 A decoupling principle

We present in this section a general "decoupling principle" that will be particularly useful in the study of planted models. We consider here the setting where X = (X 1 , . . . , X n ) i.i.d.

∼ P 0 for some probability distribution P 0 over R with support S. Let Y ∈ R m be another random variable that accounts for noisy observation of X. The goal is again to recover the planted vector X from the observations Y . We suppose that the distribution of X given Y takes the following form

P(X ∈ A | Y ) = 1 Z n (Y ) x∈A
dP ⊗n 0 (x)e Hn(x,Y ) , for all Borel set A ⊂ R n , (

where H n is a measurable function on R n × R m that can be equal to -∞ (in which case, we use the convention exp(-∞) = 0) and Z n (Y ) = dP ⊗n 0 (x)e Hn(x,Y ) is the appropriate normalization. We assume that E| log Z n (Y )| < ∞ in order to define the free energy

F n = 1 n E log Z n (Y ) =
1 n E log dP ⊗n 0 (x)e Hn(x,Y ) .

In the following, we are going to drop the dependency in Y of H n (x, Y ) and simply write H n (x).

We introduce now an important notation: the overlap between to vectors u, v ∈ R n . This is simply the normalized scalar product:

u • v = 1 n n i=1 u i v i .
One should see x as a system of n spins (x 1 , . . . , x n ) interacting through the (random) Hamiltonian H n . Our inference problem should be understood as the study of this spin glass model. A central quantity of interest in spin glass theory is the overlaps x (1) • x (2) between two replicas, i.e. the normalized scalar product between two independent samples x (1) and x (2) from (2.0.1). Understanding this quantity is fundamental because it allows to deduce the distance between two typical configurations of the system and thus encodes the "geometry" of the "Gibbs measure" (2.0.1).

In our statistical inference setting we have x (1) • x (2) = x (1) • X in law, by the Nishimori identity (Proposition 1.1.1). Thus the overlap x (1) • x (2) corresponds to the correlation between a typical configuration and the planted configuration. Moreover it is linked to the Minimum Mean Squared Error by

MMSE = 1 n E X -x 2 = E P 0 [X 2 ] -E x • X ,
where • denotes the expectation with respect to x which is sampled from the posterior P(X = • | Y ) (defined by Equation 2.0.1), independently of everything else.

In this section we will see a general principle that states that under a small perturbation of the Gibbs distribution (2.0.1), the overlap x (1) • x (2) between two replicas concentrates around its mean. Such behavior is called "Replica-Symmetric" in statistical physics. It remains to define what "a small perturbation of the Gibbs distribution" is. In spin glass theory, such perturbations are usually obtained by adding small extra terms to the Hamiltonian. In our context of Bayesian inference a small perturbation will correspond to a small amount of side-information given to the statistician. This extra information will lead to a new posterior distribution. In the following, we will consider two different kind of side-information and we show that the overlaps under the induced posterior concentrate around their mean.

The pinning Lemma

We suppose here that the support S of P 0 is finite. We make this assumption in order to be able to work with the discrete entropy.

In this section, we give extra information to the statistician by revealing a (small) fraction of the coordinates of X. Let us fix ∈ [0, 1], and suppose that we have access to the additional observations

Y i =    X i if L i = 1 , * if L i = 0 , for 1 ≤ i ≤ n,
where L i i.i.d.

∼ Ber( ) and * is a value that does not belong to S. The posterior distribution of X is now Hn(x) , (2.1.1) where Z n, is the appropriate normalization constant. For x ∈ S n we will write

P(X = x | Y , Y ) = 1 Z n,   i|L i =1 1(x i = Y i )     i|L i =0 P 0 (x i )   e
x = (x 1 , . . . , xn ) = (L 1 X 1 + (1 -L 1 )x 1 , . . . , L n X n + (1 -L n )x n ) . (2.1.2)
x is thus obtained by replacing the coordinates of x that are revealed by Y by their revealed values. The notation x allows us to obtain a convenient expression for the free energy of the perturbed model: From now we suppose 0 ∈ (0, 1] to be fixed and consider ∈ [0, 0 ]. The following lemma comes from [START_REF] Montanari | Estimating random variables from random sparse observations[END_REF] and is sometimes known as the "pinning lemma". It shows that the extra information Y forces the correlations between the spins under the posterior (2.1.1) to vanish. Lemma 2.1.1 (Lemma 3.1 from [START_REF] Montanari | Estimating random variables from random sparse observations[END_REF] ) For all 0 ∈ [0, 1], we have

F n, = 1 n E log Z n, = 1 n E log
0 0 d   1 n 2 1≤i,j≤n I(X i ; X j |Y , Y )   ≤ 2 n H(P 0 ) .
Let • n, denote the expectation with respect to two independent samples x (1) , x (2) from the posterior (2.1.1). Lemma 2.1.1 implies that the overlap between these two replicas concentrates: Proposition 2.1.2

There exists a constant C > 0 that only depends on P 0 such that for all 0 ∈ [0, 1],

0 0 d E 1 n n i=1 x (1) i x (2)
i -

1 n n i=1 x (1) i x (2) i n, 2 n, ≤ C 0 n .
Proof.

(x (1) • x (2) x (1) • x (2) n, ) 2 n, = (x (1) • x (2) ) 2 n, -x (1) • x (2) 2

n, = 1 n 2 1≤i,j≤n x (1) i x (2) i x (1) j x (2) j n, -x (1) i x (2) i n, x (1)
j x (2) j n, = 1 n 2 1≤i,j≤n x i x j 2 n, -x i 2 n, x j 2 n, .
Let now i, j ∈ {1, . . . , n}. The support of P 0 is finite and thus included in [-K, K] for some K > 0. This gives:

x i x j 2 n, -x i 2 n, x j 2 n, ≤ 2K 2 | x i x j n, -x i n, x j n, | = 2K 2 x i ,x j x i x j P(X i = x i , X j = x j |Y , Y ) -x i x j P(X i = x i |Y , Y )P(X j = x j |Y , Y ) ≤ 4K 2 D TV P(X i = •, X j = •|Y , Y ); P(X i = •|Y , Y ) ⊗ P(X j = •|Y , Y ) ≤ 4K 2 D KL P(X i = •, X j = •|Y , Y ); P(X i = •|Y , Y ) ⊗ P(X j = •|Y , Y ) by Pinsker's inequality. Since I(X i ; X j |Y , Y ) = E D KL P(X i = •, X j = •|Y , Y ); P(X i = •|Y , Y ) ⊗ P(X j = •|Y , Y ) ,
we get using Lemma 2.1.1: (1) • x (2) x (1) • x (2) n,

0 0 d E x
2 n, ≤ 4K 2 0 0 0 d 1 n 2 1≤i,j≤n I(X i ; X j |Y , Y ) ≤ 4K 2 2 0 H(P 0 ) n .

Noisy side Gaussian channel

We consider in this section of a different kind of side-information: an observation of the signal X perturbed by some Gaussian noise. It was proved in [START_REF] Babu | Tight bounds on the capacity of binary input random cdma systems[END_REF] for CDMA systems that such perturbations forces the overlaps to concentrate around their means. The principle here is in fact more general and holds for any observation system, provided some concentration property of the free energy.

We suppose here that the prior P 0 has a bounded support S ⊂ [-K, K], for some K > 0. Let a > 0 and (s n ) n ∈ (0, 1] N . Let (Z i ) 1≤i≤n i.i.d.

∼ N (0, 1) independently of everything else. The extra side-information takes now the form

Y i = a √ s n X i + Z i , for 1 ≤ i ≤ n. ( 2.2.1)
The posterior distribution of X given Y , Y is now

P (x | Y , Y ) = 1 Z (pert)
n,a P ⊗n 0 (x) exp H (pert) n,a (x) , where H (pert) n,a (x) = H n (x) + h n,a (x) and

h n,a (x) = n i=1 a √ s n Z i x i + a 2 s n x i X i - 1 2 a 2 s n x 2 i . Z (pert) n,a
is the appropriate normalization. Let us define φ : a → 1 ns n log dP ⊗n 0 (x)e H (pert) n,a (x) .

We fix now A ≥ 2. Define also v n (s n ) = sup 1/2≤a≤A+1 E|φ(a) -Eφ(a)|. The following result shows that, in the perturbed system (under some conditions on v n and s n ) the overlap between two replicas concentrates asymptotically around its expected value.

Proposition 2.2.1 (Overlap concentration)

Assume that v n (s n ) ---→ n→∞ 0. Then there exists a constant C > 0 that only depends on K such that for all A ≥ 2, (1) • x (2) -E x (1) • x (2) n,a

1 A -1 A 1 E x
2 n,a da ≤ C 1 √ ns n + v n (s n ) ,
where • n,a denotes the distribution of X given (Y , Y ). x (1) and x (2) are two independent samples from • n,a , independently of everything else.

Proposition 2.2.1 is the analog of [START_REF] Panchenko | The Sherrington-Kirkpatrick model[END_REF]Theorem 3.2] (the Ghirlanda-Guerra identities, see [START_REF] Ghirlanda | General properties of overlap probability distributions in disordered spin systems. towards parisi ultrametricity[END_REF]) and is proved analogously is the remaining of the section. Denote for x ∈ S n

U (x) = 1 ns n ∂ ∂a h n,a (x) = 1 n n i=1 1 √ s n Z i x i + 2ax i X i -ax 2 i .
Lemma 2.2.1

Let x be a sample from • n,a , independently of everything else. Under the conditions of Proposition 2.2.1, we have for all

A ≥ 2 1 A -1 A 1 E U (x) -E U (x) n,a n,a da ≤ C 1 √ ns n + v n (s n ) ,
for some constant C > 0 that only depends on K.

Before proving Lemma 2.2.1, let us show how it implies Proposition 2.2.1.

Proof of Proposition 2.2.1. By the bounded support assumption on P 0 , the overlap between two replicas is bounded by K 2 , thus E U (x (1) ) x (1) • x (2) n,a -E x (1) • x (2) n,a E U (x (1) )

n,a ≤ K 2 E U (x) -E U (x) n,a n,a .
(2.2.2) Let us compute the left-hand side of (2.2.2). By Gaussian integration by parts and using the Nishimori identity (Proposition 1.1.1) we get E U (x (1) ) n,a = 2a E x (1) • x (2) n,a . Therefore E x (1) • x (2) n,a E U (x (1) )

n,a

= 2a E x (1) • x (2) n,a

2

.

Using the same tools, we compute for x (1) , x (2) , x (3) , x (4) i.i.d. ∼ • n,a , independently of everything else:

E U (x (1) )(x (1) • x (2) ) n,a = 2aE (x (1) • X)(x (1) • x (2) ) n,a + 1 n √ s n n i=1 EZ i x [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF] i (x (1) • x (2) ) n,a - (1) • x (2) ) n,a = 2aE (x (1) • X)(x (1) • x (2) ) n,a + aE (x (1) • x (2) ) 2 n,a -aE (x (1) • x (3) + x (1) • x (4) )(x (1) • x (2) ) n,a = 2aE (x (1) • x (2) ) 2 n,a .

a n n i=1 E (x (1) i ) 2 (x
Thus, by (2.2.2) we have for all a ∈ [1, A] E x (1) • x (2) -E x (1) • x (2) n,a

2 n,a ≤ K 2 2 E U (x) -E U (x) n,a n,a ,
and we conclude by integrating with respect to a over [1, A] and using Lemma 2.2.1.

Proof of Lemma 2.2.1. φ is twice differentiable on (0, +∞), and for a > 0 φ (a) = U (x) n,a , (2.2.3) for some constant C > 0 (that only depend on K), because Eφ (a) = 2aE x • X n,a . It remains to show that A 1 E U (x) n,a -E U (x) n,a da ≤ CA v n (s n ) for some constant C > 0 that only depends on K.

φ (a) = ns n U (x) 2 n,a -U (x) 2 n,a + 1 n n i=1 2x i X i -x 2 i n,a . ( 2
We will use the following lemma on convex functions (from [START_REF] Panchenko | The Sherrington-Kirkpatrick model[END_REF], Lemma 3.2). We apply this lemma to λ → φ(λ) + 3 2 K 2 λ 2 and λ → Eφ(λ) + 3 2 K 2 λ 2 that are convex because of (2.2.4) and the bounded support assumption on P 0 . Therefore, for all a ≥ 1 and b ∈ (0, 1/2) we have

E|φ (a) -Eφ (a)| ≤ Eφ (a + b) -Eφ (a -b) + 6K 2 b + 3v n (s n ) b .
(2.2.5)

Notice that for all a > 0, |Eφ (a)| = |2aE x • X n,a | ≤ 2aK 2 . Therefore, by the mean value theorem Combining this with equation (2.2.5), we obtain ∀b ∈ (0, 1/2),

A 1 E|φ (a) -Eφ (a)|da ≤ CA b + v n (s n ) b . (2.2.6)
for some constant C > 0 depending only on K. The minimum of the right-hand side is achieved for b = v n (s n ) < 1/2 for n large enough. Then, (2.2.6) gives

Chapter 3

Low-rank symmetric matrix estimation

Introduction to the spiked matrix models

Estimating a low-rank object (matrix or tensor) from a noisy observation is a fundamental problem in statistical inference with applications in machine learning, signal processing or information theory. We focus in this chapter (and in Chapter 4) on the so-called "spiked" models where we observe a signal spike perturbed with some additive noise. We should consider here two popular models. The first one is often denoted as the spiked Wigner model. One observes

Y = λ n XX T + Z (3.1.1)
where the "spike" X = (X 1 , . . . , X n ) i.i.d.

∼ P 0 is the signal vector and Z is symmetric matrix that account for noise with standard Gaussian entries: (Z i,j ) i≤j i.i.d.

∼ N (0, 1). λ ≥ 0 is a signal-to-noise ratio.

The second model that we will consider in Chapter 4 is the non-symmetric version of (3.1.1), sometimes called spiked Wishart1 or spiked covariance model:

Y = λ n U V T + Z (3.1.2)
where U = (U 1 , . . . , U n ) i.i.d.

∼ P U , V = (V 1 , . . . , V m ) i.i.d.
∼ P V are independent. Z is a noise matrix with standard normal entries: Z i,j i.i.d.

∼ N (0, 1). λ > 0 captures again the strength of the signal. We are here interested in the regime where n, m → +∞, while m/n → α > 0. In both models (3.1.1)-(3.1.2) the goal of the statistician is to estimate the low-rank signals (XX T or U V T ) from the observation of Y . This task is often called Principal Component Analysis (PCA) in the literature.

These spiked models have received a lot of attention since their introduction by [START_REF] Johnstone | On the distribution of the largest eigenvalue in principal components analysis[END_REF]. From a statistical point of view, there are two main problems linked to the spiked models (3.1.1)-(3.1.2).

• The recovery problem: how can we recover the planted signal X / U , V ? Is it possible? Can we do it efficiently?

• The detection problem: is it possible to distinguish between the pure noise case (λ = 0) and the case where a spike is present (λ > 0)? Is there any efficient test to do this?

We will focus here on the recovery problem. We let the reader refer to [START_REF] Berthet | Optimal detection of sparse principal components in high dimension[END_REF][START_REF] Onatski | Asymptotic power of sphericity tests for high-dimensional data[END_REF][START_REF] Dobriban | Sharp detection in pca under correlations: all eigenvalues matter[END_REF][START_REF] Banks | Information-theoretic bounds and phase transitions in clustering, sparse pca, and submatrix localization[END_REF][START_REF] Perry | Optimality and sub-optimality of pca i: Spiked random matrix models[END_REF][START_REF] Alaoui | Finite size corrections and likelihood ratio fluctuations in the spiked wigner model[END_REF][START_REF] Jordan | Detection limits in the high-dimensional spiked rectangular model[END_REF] and the references therein for a detailed analysis of the detection problem.

The spiked models (3.1.1)-(3.1.2) has been extensively studied in random matrix theory. The seminal work of [START_REF] Baik | Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices[END_REF] (for the complex spiked Wishart model, and [START_REF] Baik | Eigenvalues of large sample covariance matrices of spiked population models[END_REF] for the real spiked Wishart) established the existence of a phase transition: there exists a critical value of the signal-to-noise ratio λ above which the largest singular value of Y / √ n escapes from the Marchenko-Pastur bulk. The same phenomenon holds for the spiked Wigner model, as observed by Edwards and Jones [START_REF] Edwards | The eigenvalue spectrum of a large symmetric random matrix[END_REF] using the heuristic replica method and then rigorously proved [START_REF] Péché | The largest eigenvalue of small rank perturbations of hermitian random matrices[END_REF][START_REF] Féral | The largest eigenvalue of rank one deformation of large wigner matrices[END_REF][START_REF] Capitaine | The largest eigenvalues of finite rank deformation of large wigner matrices: convergence and nonuniversality of the fluctuations[END_REF]. It turns out that for both models the eigenvector (respectively singular vector) corresponding to the largest eigenvalue (respectively singular value) also undergo a phase transition at the same threshold, see [START_REF] David | Statistical mechanics of learning multiple orthogonal signals: asymptotic theory and fluctuation effects[END_REF][START_REF] Paul | Asymptotics of sample eigenstructure for a large dimensional spiked covariance model[END_REF][START_REF] Nadler | Finite sample approximation results for principal component analysis: A matrix perturbation approach[END_REF][START_REF] Benaych | The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices[END_REF][START_REF] Benaych | The singular values and vectors of low rank perturbations of large rectangular random matrices[END_REF].

For the spiked Wigner model (3.1.1), the main result of interest for us is the following. For any probability distribution P 0 such that E P 0 [X 2 ] = 1, we have

• if λ ≤ 1, the top eigenvalue of Y /
√ n converges a.s. to 2 as n → ∞, and the top eigenvector x (with norm x 2 = n) has asymptotically trivial correlation with X:

1 n x, X → 0 a.s.

• if λ > 1, the top eigenvalue of Y / √ n converges a.s. to √ λ + 1/ √ λ > 2 and the top eigenvector x (with norm x 2 = n) has asymptotically nontrivial correlation with X: 1 n x, X 2 → 1 -1/λ a.s.

An analog statement for the spiked Wishart model is proved in [START_REF] Benaych | The singular values and vectors of low rank perturbations of large rectangular random matrices[END_REF]. These results give us a precise understanding of the performance of the top eigenvectors (or top singular vectors) for recovering the low-rank signals.

However these naive spectral estimators do not take into account any prior information on the signal. Thus many algorithms have been proposed to exploit additional properties of the signal, such as sparsity [START_REF] Iain | Sparse principal components analysis[END_REF][START_REF] Alexandre D'aspremont | A direct formulation for sparse pca using semidefinite programming[END_REF][START_REF] Zou | Sparse principal component analysis[END_REF][START_REF] Arash | High-dimensional analysis of semidefinite relaxations for sparse principal components[END_REF][START_REF] Deshpande | Sparse pca via covariance thresholding[END_REF] or positivity [START_REF] Montanari | Non-negative principal component analysis: Message passing algorithms and sharp asymptotics[END_REF].

Another line of works study Approximate Message Passing (AMP) algorithms for the spiked models above, see [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF][START_REF] Deshpande | Information-theoretically optimal sparse pca[END_REF][START_REF] Thibault Lesieur | Phase transitions in sparse PCA[END_REF][START_REF] Montanari | Estimation of low-rank matrices via approximate message passing[END_REF]. Motivated by deep insights from statistical physics, these algorithms are believed (for the models (3.1.1)-(3.1.2), when λ and the priors P 0 , P U , P V are known by the statistician) to be optimal among all polynomial-time algorithms. A great property of these algorithms is that their performance can be precisely tracked in the high-dimensional limit by a simple recursion called "state evolution", see [START_REF] Bayati | The dynamics of message passing on dense graphs, with applications to compressed sensing[END_REF][START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference[END_REF][START_REF] Bayati | Universality in polytope phase transitions and message passing algorithms[END_REF]. For a detailed analysis of message-passing algorithms for the models (3.1.1)-(3.1.2), see [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications[END_REF].

It turns out that fixed points of these AMP algorithms are stationary points of the socalled "TAP 2 free energy" from statistical physics. For the model (3.1.1), the minimization of the TAP free energy was studied in [START_REF] Zhou Fan | Tap free energy, spin glasses, and variational inference[END_REF], who showed that the minimizer was equal to the posterior mean of X, provided that λ was large enough.

In the following we will not consider any particular estimator but rather try to compute the best performance achievable by any estimator. We will suppose to be in the so-called "Bayes-optimal" setting, where the statistician knows the prior P 0 (or P U , P V ) and the signal-to-noise ratio λ. In that situation, we will study the posterior distribution of the signal given the observations. As we should see in the sequel, both estimation problems (3.1.1)-(3.1.2) can be seen as mean-field spin glass models similar to the Sherrington-Kirkpatrick model, studied in the ground-breaking book of Mézard, Parisi and Virasoro [START_REF] Mézard | Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications[END_REF]. Therefore, the methods that we will use here come from the mathematical study of spin glasses, namely from the works of Talagrand [START_REF] Talagrand | Mean field models for spin glasses: Volume I: Basic examples[END_REF][START_REF] Talagrand | Mean field models for spin glasses: Volume II: Advanced Replica-Symmetry and Low Temperature[END_REF], Guerra [START_REF] Guerra | Broken replica symmetry bounds in the mean field spin glass model[END_REF] and Panchenko [START_REF] Panchenko | The Sherrington-Kirkpatrick model[END_REF].

In order to further motivate the study of the models (3.1.1)-(3.1.2) let us mention some interesting special cases, depending on the choice of the priors P 0 / P U , P V .

• Sparse PCA. Consider the spiked Wishart model with P U = Ber( ) and P V = N (0, 1). In that case, one sees that conditionally on U the columns of Y are i.i.d. sampled from N 0, Id n + λ/nU U T , which is a sparse spiked covariance model. The spiked Wigner model with P 0 = Ber( ) has also been used to study sparse PCA.

• Submatrix localization. Take P 0 = Ber(p) in the spiked Wigner model. The goal of submatrix localization is then to extract a submatrix of Y of size pn × pn with larger mean.

• Community Detection in the Stochastic Block Model (SBM). As we should see in Chapter 5 recovering two communities of size pn and (1 -p)n in a dense SBM of n vertices is (in some sense) "equivalent" to the spiked Wigner model with prior

P 0 = p δ 1-p p + (1 -p) δ - √ p 1-p .
• Z/2 synchronization. This corresponds to the spiked Wigner model with Rademacher prior P 0 = 1 2 δ -1 + 1 2 δ +1 . • High-dimensional Gaussian mixture clustering. Consider the multidimensional version of the spiked Wishart model where U ∈ R n×k and V ∈ R m×k . If one takes P V (the distribution of the rows of V ) to be supported by the canonical basis of R k , the model is equivalent to the clustering of m points (the columns of Y ) in n dimensions from a Gaussian mixture model. The centers of the clusters are here the columns of U .

Information-theoretic limits in the spiked Wigner model

We consider in this chapter the spiked Wigner model (3.1.1). Let P 0 be a probability distribution on R that admits a finite second moment and consider the following observations:

Y i,j = λ n X i X j + Z i,j , for 1 ≤ i < j ≤ n , (3.2.1)
where

X i i.i.d. ∼ P 0 and Z i,j i.i.d.
∼ N (0, 1) are independent random variables. Note that we suppose here to only observe the coefficients of λ/nXX T +Z that are above the diagonal. The case where all the coefficients are observed can be directly deduced from this case. In the following, E will denote the expectation with respect to the X and Z random variables.

Our main quantity of interest is the Minimum Mean Squared Error (MMSE) defined as:

MMSE n (λ) = min θ 2 n(n -1) 1≤i<j≤n E X i X j -θ i,j (Y ) 2 = 2 n(n -1) 1≤i<j≤n E (X i X j -E [X i X j |Y ]) 2 ,
where the minimum is taken over all estimators θ (i.e. measurable functions of the observations Y ). We have the trivial upper-bound

MMSE n (λ) ≤ DMSE def = E P 0 [X 2 ] 2 -E P 0 [X] 4 ,
obtained by considering the "dummy" estimator θ i,j = E P 0 [X] 2 . One can also compute the Mean Squared Error achieved by naive PCA. Let x be the leading eigenvector of Y with norm x 2 = n. If we take an estimator proportional to x i x j , i.e. θ i,j = δ x i x j for δ ≥ 0, we can compute explicitly (using the results presented in Section 3.1) the resulting MSE as a function of δ and minimize it. The optimal value for δ depends on λ, more precisely if λ < E P 0 [X 2 ] -2 , then δ = 0 while for λ ≥ E P 0 [X 2 ] -2 , the optimal of value for δ is

E P 0 [X 2 ] -λ -1 E P 0 [X 2 ] -1
, resulting in the following MSE for naive PCA:

MSE PCA n (λ) ---→ n→∞ E P 0 [X 2 ] 2 if λ ≤ E P 0 [X 2 ] -2 , λ -1 (2 -λ -1 E P 0 [X 2 ] -2 ) otherwise. (3.2.2)
We will see in Section 3.3 that in the particular case of P 0 = N (0, 1), PCA is optimal:

lim n→∞ MSE PCA n = lim n→∞ MMSE n .
Posterior distribution and free energy. In order to formulate our inference problem as a statistical physics problem we introduce the random Hamiltonian

H n (x) = i<j λ n x i x j Z i,j + λ n X i X j x i x j - λ 2n x 2 i x 2 j . (3.2.3)
The posterior distribution of X given Y takes then the form

dP (x | Y ) = 1 Z n (λ) dP ⊗n 0 (x) exp i<j x i x j λ n Y i,j - λ 2n x 2 i x 2 j = 1 Z n (λ)
dP ⊗n 0 (x)e Hn(x) , (3.2.4) where Z n (λ) is the appropriate normalization. The free energy is defined as

F n (λ) = 1 n E log dP ⊗n 0 (x) e Hn(x) = 1 n E log Z n (λ).
We will first compute the limit of the free energy F n and then deduce the limit of MMSE n by an I-MMSE (see Proposition 1.3.3) argument. We express the limit of F n using the following function

F : (λ, q) → ψ P 0 (λq) - λ 4 q 2 = E log dP 0 (x) exp λqZx + λqxX - λ 2 qx 2 - λ 4 q 2 ,
(3.2.5) where Z ∼ N (0, 1) and X ∼ P 0 are independent random variables. Recall that ψ P 0 denotes the free energy (1.3.6) of the scalar channel (1.3.5). The main result of this section is:

Theorem 3.2.

(Replica-Symmetric formula for the spiked Wigner model)

For all λ > 0, F n (λ) ---→ n→∞ sup q≥0 F(λ, q) . (3.2.6)
Theorem 3.2.1 is proved in Section 3.4. In the case of Rademacher prior (P 0 = 1 2 δ -1 + 1 2 δ +1 ), Theorem 3.2.1 was proved in [START_REF] Deshpande | Asymptotic mutual information for the balanced binary stochastic block model[END_REF]. The expression (3.2.6) for general priors was conjectured by [START_REF] Thibault Lesieur | MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel[END_REF]. For discrete priors P 0 for which the map F(λ, •) has not more than 3 stationary points, the statement of Theorem 3.2.1 was obtained in [START_REF] Barbier | Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula[END_REF]. The full version of Theorem 3.2.1 as well as its multidimensional generalization (where X ∈ R n×k , k fixed) was proved in [START_REF] Lelarge | Fundamental limits of symmetric low-rank matrix estimation[END_REF]. Theorem 3.2.1 allows us to compute the limit of the mutual information between the signal X and the observations Y . Indeed, by using (1.3.3):

Corollary 3.2.1 lim n→+∞ 1 n I(X; Y ) = λE P 0 (X 2 ) 2 4 -sup q≥0 F(λ, q) .
We will now use Theorem 3.2.1 to obtain the limit of the Minimum Mean Squared Error MMSE n by the I-MMSE relation of Proposition 1.3.3. Let us define

D = {λ > 0 | F(λ, •) has a unique maximizer q * (λ) } .
We start by computing the derivative of lim n→∞ F n (λ) with respect to λ.

Proposition 3.2.1

D is equal to R >0 minus some countable set and is precisely the set of λ > 0 at which the function

f : λ → sup q≥0 F(λ, q) is differentiable. Moreover, for all λ ∈ D f (λ) = q * (λ) 2 4 .
Proof. Let λ > 0 and compute

∂ ∂q F(λ, q) = λψ P 0 (λq) - λq 2 ≤ λ 2 E P 0 [X 2 ] -q , because ψ P 0 is 1 2 E P 0 [X 2 ]-Lipschitz by Proposition 1.3.3. Consequently, the maximum of F(λ, •) is achieved on [0, E P 0 [X 2 ]]. If q * maximizes F(λ, •), the optimality condition gives q * = 2ψ P 0 (λq * ). Consequently ∂ ∂λ F(λ, q * ) = q * ψ P 0 (λq * ) - (q * ) 2 4 = (q * ) 2 4 .
Now, Proposition D.2 in Appendix D gives that the λ > 0 at which f is differentiable is exactly the λ > 0 for which

∂ ∂λ F(λ, q * ) = 1 4 (q * ) 2 q * maximizer of F(λ, •)
is a singleton. These λ are precisely the elements of D. Moreover, Proposition D.2 gives also that for all λ ∈ D, f (λ) = q * (λ) 2 4 , which concludes the proof.

We deduce then the limit of MMSE n :

Corollary 3.2.2 For all λ ∈ D, MMSE n (λ) ---→ n→∞ (E P 0 X 2 ) 2 -q * (λ) 2 . (3.2.7) Proof. By Proposition 1.3.3, (F n ) n≥1 is a sequence of differentiable convex functions that con- verges pointwise on R >0 to f . By Proposition C.1, F n (λ) ---→ n→∞ f (λ)
for every λ > 0 at which f is differentiable, that is for all λ ∈ D. We conclude using the I-MMSE relation (1.3.4):

n -1 4n E P 0 [X 2 ] 2 -MMSE n (λ) = F n (λ) ---→ n→∞ f (λ) = q * (λ) 2 4 . ( 3.2.8)
Let us now define the information-theoretic threshold

λ c = inf λ ∈ D q * (λ) > (E P 0 X) 2 . (3.2.9)
If the above set is empty, we define λ c = 0. By Corollary 3.2.2 we obtain that

• if λ > λ c , then lim n→∞ MMSE n < DMSE: one can estimate the signal better than a random guess.

• if λ < λ c , then lim n→∞ MMSE n = DMSE: one can not estimate the signal better than a random guess. Thus, there is no hope for reconstructing the signal below λ c . Interestingly, one can not even detect if the measurements Y contains some signal below λ c . If one denotes by Q λ the distribution of Y given by (3.2.1), the work [START_REF] Alaoui | Finite size corrections and likelihood ratio fluctuations in the spiked wigner model[END_REF] shows that for λ < λ c one can not asymptotically distinguish between Q λ and Q 0 : both distributions are contiguous.

Information-theoretic and algorithmic phase transitions

Approximate Message Passing (AMP) algorithms

Approximate Message Passing (AMP) algorithms, introduced in [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF] for compressed sensing, have then be used for various other tasks. Rigorous properties of AMP algorithms have been established in [START_REF] Bayati | The dynamics of message passing on dense graphs, with applications to compressed sensing[END_REF][START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference[END_REF][START_REF] Bayati | Universality in polytope phase transitions and message passing algorithms[END_REF][START_REF] Berthier | State evolution for approximate message passing with non-separable functions. Information and Inference[END_REF], following the seminal work of Bolthausen [START_REF] Bolthausen | An iterative construction of solutions of the tap equations for the sherrington-kirkpatrick model[END_REF]. In the context of low-rank matrix estimation an AMP algorithm has been proposed by [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF] for the rank-one case and then by [START_REF] Matsushita | Low-rank matrix reconstruction and clustering via approximate message passing[END_REF] for finite-rank matrix estimation. For detailed review and developments about matrix factorization with message-passing algorithms, see [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications[END_REF]. We will only give a brief description of AMP here and we let the reader refer to [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF][START_REF] Deshpande | Information-theoretically optimal sparse pca[END_REF][START_REF] Thibault Lesieur | MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel[END_REF][START_REF] Montanari | Estimation of low-rank matrices via approximate message passing[END_REF]. In this section, we follow [START_REF] Montanari | Estimation of low-rank matrices via approximate message passing[END_REF] who provides the most advanced results for our problem (3.1.1). For simplicity, we assume here that P 0 has a unit second moment:

x 2 dP 0 (x) = 1.
Starting from an initialization x 0 , the AMP algorithm produces vectors x 1 , . . . , x t according to the following recursion:

x t+1 = (Y / √ n)f t (x t ) -b t f t-1 (x t-1 ), (3.3.1)
where

b t = 1 n n i=1 f t (x t i
) and where the functions f t act componentwise on vectors. After t iterations of (3.3.1), the AMP estimate of X is defined by

x t = f t (x t ).
A natural choice for the initialization is to take x 0 proportional to ϕ 1 , the leading unit eigenvector of Y :

x 0 = n(λ 2 -1)ϕ 1 . We need now to specify the "denoisers" (f t ) t≥1 . Let us consider the following one-dimensional recursion:

   q 0 = (1 -λ -1 ) + , q t+1 = 2ψ P 0 (λq t ) = 1 -MMSE P 0 (λq t ). (3.3.2)
Recall the additive Gaussian scalar channel from Section 1.3:

Y 0 = √ γX 0 + Z 0 . Let us define g P 0 (y, γ) = E[X 0 | √ γX 0 + Z 0 = y].
We define then

f t (x) = g P 0 x/ √ q t , λq t . (3.3.3)
The next theorem is a consequence of the more general results of [START_REF] Montanari | Estimation of low-rank matrices via approximate message passing[END_REF], specified to our setting.

Theorem 3.3.1

For all t ≥ 0,

lim n→∞ | x t , X | x t X = lim n→∞ x t = √ q t .
Consequently,

MSE AMP t def = lim n→∞ 1 n 2 E XX T -x t ( x t ) T 2 = 1 -q 2 t . (3.3.4)
By Proposition 1.3.3, the function ψ P 0 is increasing and bounded. The sequence (q t ) t≥0 converges therefore to a point q ∞ ≥ 0 that verifies q ∞ = 2ψ P 0 (λq ∞ ). q ∞ is therefore a critical point of F(λ, •). In the case where q ∞ is the global minimizer of F(λ, •), i.e. q ∞ = q * (λ), we see using Corollary 3.2.2 that lim t→∞ MSE AMP t = MMSE(λ): AMP achieves the Bayesoptimal accuracy.

In the case where q ∞ = q * (λ), AMP does not reach the information-theoretically optimal performance. However, AMP is conjectured (see for instance [START_REF] Zdeborová | Statistical physics of inference: Thresholds and algorithms[END_REF][START_REF] Antenucci | Glassy nature of the hard phase in inference problems[END_REF]) to be optimal among polynomial-time algorithms, i.e. lim t→∞ MSE AMP t is conjectured to be the best Mean Squared Error achievable by any polynomial-time algorithm.

Examples of phase transitions

We give here some illustrations and interpretations of the results presented in the previous sections. Let us first study the case where P 0 = N (0, 1) where the formulas (3.2.6) and (3.2.7) can be evaluated explicitly. Indeed, we saw in Example 1.3.1 in Section 1.3 that ψ N (0,1) (q) = 1 2 q -log(1 + q) . We can then compute q * (λ) = (1 -λ -1 ) + which gives

lim n→∞ MMSE n (λ) =    0 if λ ≤ 1 , 1 λ 2 -1 λ if λ ≥ 1 .
Comparing the limit above with the performance of (naive) PCA given by (3.2.2) we see that in the case P 0 = N (0, 1), PCA is information-theoretically optimal.

However, as we see on (3.2.2), the MSE of PCA only depends on the second moment of P 0 : naive PCA is not able to exploit additional properties of the signal. We compare on Figure 3.1 the asymptotic performance of the naive PCA (3.2.2) and the Approximate Message Passing (AMP) algorithm (3.3.4) to the asymptotic Minimum Mean Squared Error for the prior

P 0 = p δ 1-p p + (1 -p) δ - √ p 1-p , ( 3.3.5)
where p ∈ (0, 1). This is a two-points distribution with zero mean and unit variance. It is of particular interest because it is related with the community detection problem in the (dense) Stochastic Block Model as we will see in Section 5.9. We see on the MMSE is equal to 1 for λ below the information-theoretic threshold λ c 0.6. One can not asymptotically recover the signal better than a random guess in this region: we call this region the "impossible" phase. For λ > 1 we see that spectral methods and AMP perform better than random guessing. This region is therefore called the "easy" phase, because non-trivial estimation is here possible using efficient algorithms. Notice also that AMP achieves the Minimum Mean Squared Error for λ > 1, as proved in [START_REF] Montanari | Estimation of low-rank matrices via approximate message passing[END_REF]. The region λ c < λ < 1 is more intriguing. It is still possible to build a non-trivial estimator (for instance by computing the posterior mean), but our two polynomial-time algorithms fail. This region is thus denoted as the "hard" phase because it is conjectured that polynomialtime algorithms can only provide trivial estimates (based on the belief that AMP is here optimal among polynomial-time algorithms).

Quite surprisingly, one can guess in which phase (easy-hard-impossible) we are, simply by plotting the "potential" q → -F(λ, q). This is done in Figure 3 (c) "Impossible" phase (λ = 0.5)

Figure 3.2:

Plots of q → -F(λ, q) for different values of λ and P 0 given by (3.3.5) with p = 0.05.

we know that the limit of the MMSE is equal to 1 -q * (λ)2 where q * (λ) is the minimizer of -F(λ, •). Thus when -F(λ, •) is minimal at q = 0, we are in the impossible phase. When q * (λ) > 0, the shape of -F(λ, •) indicates whether we are in the easy or hard phase. If the q = 0 is a local maximum, then we are in the easy phase, whereas when it is a local minimum we are in a hard phase. The shape of -F(λ, •) could be interpreted as a simplified "free energy landscape": the hard phase appears when the "informative" minimum q * (λ) > 0 is separated from the non-informative critical point q = 0 by a "free energy barrier" as in Figure 3.2 (b). The phase diagram from Figure 3.3 displays the three phases on the (p, λ)-plane. One observes that the hard phase only appears when the prior is sufficiently asymmetric, i.e. for p < p * = 1 2 -1

Proof of the Replica-Symmetric formula (Theorem 3.2.1)

We prove Theorem 3.2.1 in this section, following [START_REF] Lelarge | Fundamental limits of symmetric low-rank matrix estimation[END_REF]. We have to mention that other proofs of Theorem 3.2.1 have appeared since then: see [START_REF] Barbier | The stochastic interpolation method: A simple scheme to prove replica formulas in bayesian inference[END_REF][START_REF] Krzakala | Estimation in the spiked wigner model: A short proof of the replica formula[END_REF][START_REF] Mourrat | Hamilton-jacobi equations for mean-field disordered systems[END_REF].

Because of an approximation argument presented in Section 3.4.7 it suffices to prove Theorem 3.2.1 for priors P 0 with finite (and thus bounded) support S ⊂ [-K, K], for some K > 0. From now, we assume to be in that case.

The lower bound: Guerra's interpolation method

The following result comes from [START_REF] Krzakala | Mutual information in rankone matrix estimation[END_REF]. It adapts arguments from the study of the gauge symmetric p-spin glass model of [START_REF] Babu | Exact solution of the gauge symmetric pspin glass model on a complete graph[END_REF] to the inference model (3.2.1). It is based on Guerra's interpolation technique for the Sherrington-Kirkpatrick model, see [START_REF] Guerra | Broken replica symmetry bounds in the mean field spin glass model[END_REF]. We reproduce the proof for completeness. Proposition 3.4.1

lim inf n→∞ F n (λ) ≥ sup q≥0 F(λ, q) .
(3.4.1)

Proof. Let q ≥ 0. For t ∈ [0, 1] we define H n,t (x) = i<j λt n Z i,j x i x j + λt n x i x j X i X j - λt 2n x 2 i x 2 j + n i=1 (1 -t)λqZ i x i +(1-t)λqx i X i - (1 -t)λq 2 x 2 i .
Let • n,t denote the Gibbs measure associated with the Hamiltonian H n,t (x):

f (x) n,t = x∈S n P ⊗n 0 (x)f (x)e
   Y i,j = λt n X i X j + Z i,j for 1 ≤ i < j ≤ n, Y i = (1 -t)λqX i + Z i for 1 ≤ i ≤ n,
where

X i i.i.d. ∼ P 0 and Z i,j , Z i i.i.d.
∼ N (0, 1) are independent random variables. We will therefore be able to apply the Nishimori property (Proposition 1.1.1) to the Gibbs measure Hn,t(x) . We have ψ(1) = F n (λ) and

• n,t . Let us define ψ : t ∈ [0, 1] → 1 n E log x∈S n P ⊗n 0 (x)e
ψ(0) = 1 n E log x∈S n P ⊗n 0 (x) exp n i=1 λqZ i x i + λqx i X i - λq 2 x 2 i = 1 n E log n i=1   x i ∈S P 0 (x i ) exp λqZ i x i + λqx i X i - λq 2 x 2 i   = F(λ, q) + λq 2 4 .
ψ is continuous on [0, 1], differentiable on (0, 1). For 0 < t < 1,

ψ (t) = 1 n E i<j √ λ 2 √ nt Z i,j x i x j + λ n x i x j X i X j - λ 2n x 2 i x 2 j - n i=1 √ λq 2 √ 1 -t Z i x i -λqx i X i + λq 2 x 2 i n,t , (3.4.2)
where x is a sample from the Gibbs measure • n,t , independently of everything else. For 1 ≤ i < j ≤ n we have, by Gaussian integration by parts and by the Nishimori property

E Z i,j √ λ 2 √ nt x i x j n,t = λ 2n E x 2 i x 2 j n,t -E x i x j 2 n,t = λ 2n E x 2 i x 2 j n,t -E x (1) i x (1) j x (2) i x (2) j n,t = λ 2n E x 2 i x 2 j n,t -E x i x j X i X j n,t ,
where x (1) and x (2) are two independent samples from the Gibbs measure • n,t , independently of everything else. Similarly, we have for 1

≤ i ≤ n E √ λq 2 √ 1 -t Z i x i n,t = λq 2 E x 2 i n,t -E x i X i n,t . Therefore (3.4.2) simplifies ψ (t) = 1 n E i<j λ 2n x i x j X i X j - n i=1 λq 2 x i X i n,t = λ 4 E (x • X) 2 -2q x • X n,t + o n (1) = λ 4 E (x • X -q) 2 n,t - λq 2 4 + o n (1) ≥ - λq 2 4 + o n (1) , ( 3.4.3)
where o n (1) denotes a quantity that goes to 0 uniformly in t ∈ (0, 1). Then

F n (λ) -F(λ, q) - λ 4 q 2 = ψ(1) -ψ(0) = 1 0 ψ (t)dt ≥ - λ 4 q 2 + o n (1) .
Thus lim inf n→∞ F n (λ) ≥ F(λ, q), for all q ≥ 0.

Adding a small perturbation

It remains to prove the converse bound of (3.4.1). For this purpose, we need to show that the overlap x • X (where x is a sample from the posterior distribution of X given Y , independently of everything else) concentrates around its mean. To obtain such a result, we follow the ideas of Section 2.1 that states that giving a small amount of side information to the statistician forces the overlap to concentrate, while keeping the free energy almost unchanged.

Let us fix ∈ [0, 1], and suppose that we have access, in addition of Y , to the additional information, for 1

≤ i ≤ n Y i =    X i if L i = 1, * if L i = 0, (3.4.4)
where

L i i.i.d.
∼ Ber( ) and * is a value that does not belong to S. Recall the free energy that corresponds to this perturbed inference channel is

F n, = 1 n E log x∈S n P ⊗n 0 (x) exp(H n ( x)) , where x = (x 1 , . . . , xn ) = (L 1 X 1 + (1 -L 1 )x 1 , . . . , L n X n + (1 -L n )x n ) . (3.4.5)
From now we suppose 0 ∈ (0, 1] to be fixed and consider ∈ [0, 0 ]. We will compute the limit of F n, as n → ∞ and then let → 0 to deduce the limit of F n , because by Proposition 2.1.1 |F n, -F n | ≤ H(P 0 ) .

Aizenman-Sims-Starr scheme

The Aizenman-Sims-Starr scheme was introduced in [START_REF] Michael Aizenman | Extended variational principle for the sherrington-kirkpatrick spin-glass model[END_REF] in the context of the SK model. This is what physicists call a "cavity computation": one compare the system with n + 1 variables to the system with n variables and see what happen to the (n + 1) th variable we add.

With the convention F 0, = 0, we have

F n, = 1 n n-1 k=0 A (0)
k, where

A (0) k, = (k + 1)F k+1, -kF k, = E[log(Z k+1, )] -E[log(Z k, )] .
We recall that Z n, = x∈S n P ⊗n 0 (x)e Hn(x) where the notation x is defined by equation (3.4.5). Consequently lim sup

n→∞ 0 0 d F n, ≤ lim sup n→∞ 0 0 d A (0) n, . (3.4.6)
We now compare H n+1 with H n . Let x ∈ S n and σ ∈ S. σ plays the role of the (n + 1) th variable. We decompose H n+1 (x, σ) = H n (x) + σz 0 (x) + σ 2 s 0 (x), where

H n (x) = 1≤i<j≤n λ n + 1 Z i,j x i x j + λ n + 1 X i X j x i x j - λ 2(n + 1) x 2 i x 2 j , z 0 (x) = n i=1 λ n + 1 Z i,n+1 x i + λ n + 1 X i X n+1 x i , s 0 (x) = - λ 2(n + 1) n i=1 x 2 i .
Let ( Z i,j ) 1≤i<j≤n be independent, standard Gaussian random variables, independent of all other random variables. We have then H n (x) = H n (x) + y 0 (x) in law, where

y 0 (x) = 1≤i<j≤n √ λ n(n + 1) Z i,j x i x j + λ n(n + 1) X i X j x i x j - λ 2(n + 1)n x 2 i x 2 j .
We define the Gibbs measure • n, by

f (x) n, = 1 Z n, x∈S n P 0 (x)f ( x) exp(H n ( x)) , (3.4.7)
for any function f on S n . The Gibbs measure • n, corresponds to the posterior distribution of X given ( λ/(n + 1)X i X j + Z i,j ) 1≤i<j≤n and Y from (3.4.4). We will therefore be able to apply the Nishimori identity (Proposition 1.1.1) and Proposition 2.1.2 to the Gibbs measure • n, . Let us define σ = (1 -L n+1 )σ + L n+1 X n+1 . We can rewrite Z n+1, = x∈S n P ⊗n 0 (x)e H n (x) σ∈S P 0 (σ) exp(σz 0 ( x) + σ2 s 0 ( x)) and Z n, =

x∈S n P ⊗n 0 (x)e H n (x) e y 0 (x) . Thus

A (0) n, = E log σ∈S P 0 (σ) exp σz 0 (x) + σ2 s 0 (x) n, -E log exp(y 0 (x)) n, .
In the sequel, it will be more convenient to use slightly simplified versions of z 0 , s 0 and y 0 in order to obtain nicer expressions in the sequel. We define

z(x) = n i=1 λ n Z i,n+1 x i + λ n X i X n+1 x i = λ n n i=1 x i Z i,n+1 + λ(x • X)X n+1 , s(x) = - λ 2n n i=1 x 2 i = - λ 2 x • x , y(x) = √ λ √ 2n n i=1 Z i x 2 i + λ 2n 2 n i=1 x 2 i X 2 i - x 4 i 2 + √ λ n 1≤i<j≤n x i x j Z i,j + √ λ n X i X j - λ 2n 2 x 2 i x 2 j = √ λ √ 2n n i=1 Z i x 2 i + √ λ n 1≤i<j≤n x i x j Z i,j + λ 2 (x • X) 2 - 1 2 (x • x) 2 ,
where

Z i i.i.d.
∼ N (0, 1) independently of any other random variables. Define now

A n, = E log σ∈S P 0 (σ) exp(σz(x) + σ2 s(x)) n, -E log exp(y(x)) n, .
Using Gaussian interpolation techniques, it is not difficult to show that 0 0 d (A n, -A (0) n, ) ---→ n→∞ 0 because the modifications made in z 0 , s 0 and y 0 are of negligible order. Using (3.4.6) we conclude lim sup

n→∞ 0 0 d F n, ≤ lim sup n→∞ 0 0 d A n, .
(3.4.8)

Overlap concentration

Proposition 2.1.2 implies that the overlap between two replicas, i.e. two independent samples x (1) and x (2) from the Gibbs distribution • n, , concentrates. Let us define the random variables

Q = 1 n n i=1 x (1) i x (2) i n,
and b i = x i n, . Notice that Q = 1 n i b 2 i ≥ 0. By Proposition 2.1.2 we know that 0 0 d E (x (1) • x (2) -Q) 2 n, ---→ n→∞ 0 . (3.4.9)
Thus, using the Nishimori property (Proposition 1.1.1) we deduce:

0 0 d E (x • X -Q) 2 n, ---→ n→∞ 0 and 0 0 d E (x • b -Q) 2 n, ---→ n→∞ 0 .
(3.4.10)

The main estimate

Let us denote, for ∈ [0, 1],

F : (λ, q) → - λ 4 q 2 + (E P 0 X 2 ) λq 2 + (1 -)E log x∈S P 0 (x) exp λqZx + λqxX - λ 2 qx 2
where the expectation E is taken with respect to the independent random variables X ∼ P 0 and Z ∼ N (0, 1). The following proposition is one of the key steps of the proof.

Proposition 3.4.2

For all 0 ∈ [0, 1],

0 0 d (A n, -E[F (λ, Q)]) ---→ n→∞ 0 .
The proof of Proposition 3.4.2 is deferred to Section 3.4.6. We deduce here Theorem 3.

F n ≤ sup q≥0 F(λ, q).
By Proposition 2.1.1 we have

0 F n ≤ 0 0 d F n, + 1 2 H(P 0 ) 2 0 .
Therefore by equation (3.4.8) and Proposition 3.4.2

0 lim sup n→∞ F n ≤ lim sup n→∞ 0 0 d A n, + 1 2 H(P 0 ) 2 0 ≤ lim sup n→∞ 0 0 d EF (λ, Q) + 1 2 H(P 0 ) 2 0 . (3.4.11) It remains then to show that lim sup n→∞ d EF (λ, Q) ≤ 0 sup q≥0 F(λ, q) + O( 2 0 ). We have for ∈ [0, 1], sup q∈[0,K 2 ] |F (λ, q) -F(λ, q)| ≤ sup q∈[0,K 2 ] λq 2 E P 0 [X 2 ] + E log x∈S P 0 (x) exp( λqZx + λqxX - λ 2 qx 2 ) ≤ C ,
for some constant C that only depends on λ and P 0 . Noticing that Q ∈ [0, K 2 ] a.s., we have then

|EF (λ, Q) -EF(λ, Q)| ≤ C 0 , for all ∈ [0, 0 ] and therefore 0 0 d EF (λ, Q) ≤ 0 sup q≥0 F(λ, q) + 1 2 C 2 0 .
Combined with (3.4.11), this implies lim sup n→∞

F n ≤ sup q≥0 F(λ, q) + 1 2 H(P 0 ) 0 + 1 2 C 0 , for all 0 ∈ (0, 1]. Theorem 3.2.1 is proved.

Proof of Proposition 3.4.2

In this section, we prove Proposition 3.4.2 which is a consequence of Lemmas 3.4.1 and 3.4.2 below. In order to lighten the formulas, we will use the following notations

X = X n+1 and Z i = Z i,n+1 . Recall A n, = E log σ∈S P 0 (σ) exp(σz(x) + σ2 s(x)) n, -E log exp(y(x)) n, , (3.4.12)
where for σ ∈ S, σ = (1 -L n+1 )σ + L n+1 X . We recall that • n, denotes the expectation with respect to x sampled from the Gibbs measure defined by (3.4.7). The computations here are closely related to the cavity computations in the SK model, see for instance [START_REF] Talagrand | Mean field models for spin glasses: Volume I: Basic examples[END_REF]. Lemma 3.4.1

0 0 d E log σ∈S P 0 (σ) exp(σz(x) + σ2 s(x)) n, -(E P 0 X 2 )E λQ 2 + (1 -)E log σ∈S P 0 (σ) exp λQσZ 0 + λQσX - λσ 2 2 Q ---→ n→∞ 0 ,
where Z 0 ∼ N (0, 1) is independent of all other random variables.

Lemma 3.4.2 0 0 d E log exp(y(x)) n, - λ 4 EQ 2 ---→ n→∞ 0 .
We will only prove Lemma 3.4.1 here since Lemma 3.4.2 follows from the same kind of arguments (the full proof can be found in [START_REF] Lelarge | Fundamental limits of symmetric low-rank matrix estimation[END_REF]). The remaining of the section is thus devoted to the proof of Lemma 3.4.1.

Let us write f (z, s) = σ∈S P 0 (σ)e σz+σ 2 s and we define:

U = f (z(x), s(x)) n, , V = σ∈S P 0 (σ) exp   σ λ n n i=1 b i Z i + λQX σ - λQ 2 σ2   . Lemma 3.4.3 0 0 d E (U -V ) 2 ---→ n→∞ 0 . Proof. It suffices to show that d |EU 2 -EV 2 | ---→ n→∞ 0 and d |EU V -EV 2 | ---→ n→∞ 0. Let E Z denote the expectation with respect to Z = (Z i,n+1 ) 1≤i≤n only. Compute E Z V 2 = E Z σ 1 ,σ 2 ∈S P 0 (σ 1 , σ 2 ) exp (σ 1 + σ2 ) λ n n i=1 b i Z i + λQX (σ 1 + σ2 ) - λQ 2 (σ 2 1 + σ2 2 ) = σ 1 ,σ 2 ∈S P 0 (σ 1 , σ 2 ) exp (σ 1 + σ2 ) 2 λ 2 Q + λQX (σ 1 + σ2 ) - λQ 2 (σ 2 1 + σ2 2 ) = σ 1 ,σ 2 ∈S P 0 (σ 1 , σ 2 ) exp σ1 σ2 λQ + λQX (σ 1 + σ2 ) (3.4.13)
where we write for i = 1, 2, σi = (1 -L n+1 )σ i + L n+1 X , as before.

Let us show that

d |EU 2 -EV 2 | ---→ n→∞ 0. E Z U 2 = E Z f (z(x), s(x)) 2 n, = E Z f (z(x (1)
), s(x (1) ))f (z(x (2) ), s(x (2) )) n, (x (1) and x (2) are indep. samples from ) ), s(x (1) ))f (z(x (2) ), s(x (2) )) (1) ) + σ2 1 s(x (1) ) + σ2 z(x (2) ) + σ2 2 s(x (2) ) n,

• n, ) = E Z f (z(x ( 1
n, = σ 1 ,σ 2 ∈S P 0 (σ 1 , σ 2 )E Z exp σ1 z(x
.

The next lemma follows from the simple fact that for N ∼ N (0, 1) and t ∈ R, Ee tN = exp( t 2 2 ). Lemma 3.4. [START_REF] David | Representations for partially exchangeable arrays of random variables[END_REF] Let x (1) , x (2) ∈ S n and σ 1 , σ 2 ∈ S be fixed. Then

E Z exp   σ 1 λ n n i=1 x (1) i Z i + σ 2 λ n n i=1 x (2) i Z i   = exp λσ 1 σ 2 x (1) • x (2) + λσ 2 1 2n x (1) 2 + λσ 2 2 2n x (2) 2 .
Thus, for all x (1) , x (2) ∈ S n and

σ 1 , σ 2 ∈ S E Z e σ1 z(x (1) )+σ 2 1 s(x (1) )+σ 2 z(x (2) )+σ 2 2 s(x (2) ) = e λσ 1 σ2 x (1) •x (2) +λX (σ 1 (x (1) •X)+σ 2 (x (2) •X)) ,
where we used the fact that s(x) = -λ 2n x 2 for all x ∈ S n . We have therefore

E Z U 2 = σ 1 ,σ 2 ∈S P 0 (σ 1 , σ 2 ) exp λσ 1 σ2
x (1) • x (2) + λX σ1 (x (1) • X)

+ σ2 (x (2) • X) n, . Define g : (s, r 1 , r 2 ) ∈ [-K 2 , K 2 ] 3 → σ 1 ,σ 2 ∈S P 0 (σ 1 , σ 2 ) exp λσ 1 σ2 s + λX (σ 1 r 1 + σ2 r 2 ) .
We have E Z U 2 = g(x (1) • x (2) , x (1) • X, x (2) • X) n, and by (3.4.13),

E Z V 2 = g(Q, Q, Q).

Lemma 3.4.5

There exists a constant M that only depends on λ and K, such that g is almost surely M -Lipschitz.

Proof. g is a random function that depends only on the random variables X and L n+1 (because of σ1 and σ2 ).

g is C 1 on the compact [-K 2 , K 2 ] 3 . An easy computation show that ∀(s, r 1 , r 2 ) ∈ [-K 2 , K 2 ] 3 , ∇g(s, r 1 , r 2 ) ≤ 3λK 4 exp(3λK 4 ).
g is thus M -Lipschitz with M = 3λK 4 exp(3λK 4 ).

Using Lemma 3.4.5 we obtain |g(x (1) • x (2) , x (1) • X, x (2) • X)

-g(Q, Q, Q)| n, ≤ M (x (1) • x (2) -Q) 2 + (x (1) • X -Q) 2 + (x (2) • X -Q) 2 n,
.

We recall equation (3.4.13) to notice that g(Q, Q, Q) = E Z V 2 . Thus, using (3.4.9) and (3.4.10)

0 0 d E|E Z U 2 -E Z V 2 | ≤ M 0 0 d E (x (1) • x (2) -Q) 2 + (x (1) • X -Q) 2 + (x (2) • X -Q) 2 n,
, and the right-hand side goes to 0 by (3.4.9-3.4.10).

Showing that d |EU V -EV 2 | ---→ n→∞ 0 goes exactly the same way. We thus omit this part here for the sake of brevity, but the reader can refer to [START_REF] Lelarge | Fundamental limits of symmetric low-rank matrix estimation[END_REF] where all details are presented.

Using the fact that | log U -log V | ≤ max(U -1 , V -1 )|U -V | and the Cauchy-Schwarz inequality, we have

E| log U -log V | ≤ √ EU -2 + EV -2 E(U -V ) 2 .

Lemma 3.4.6

There exists a constant C that depends only on λ and K such that

EU -2 + EV -2 ≤ C . Proof. Using Jensen inequality, we have U ≥ f z(x) n, , s(x) n, . Then U -2 ≤ f z(x) n, , s(x) n, -2 ≤ σ∈S P 0 (σ) exp -2σ z(x) n, -2σ 2 s(x) n, .
It remains to bound E exp(-2σ z(x) n, -2σ 2 s(x) n, ). P 0 has a bounded support, therefore

E exp -2σ z(x) n, -2σ 2 s(x) n, ≤ C 0 E exp   -2σ n i=1 λ n x i n, Z i   = C 0 E exp(2λQσ 2 ) ≤ C 1 ,
for some constant C 0 , C 1 depending only on λ and K. Similar arguments show that EV -2 is upper-bounded by a constant.

Using the previous lemma we obtain

d E| log U -log V | ---→ n→∞ 0. We now compute E log V explicitly. Lemma 3.4.7 E log V = (E P 0 X 2 )E λQ 2 +(1-)E log σ∈S P 0 (σ) exp   σ λ n n i=1 b i Z i + λQσX - λσ 2 2 Q   .
Proof. It suffices to distinguish the cases L n+1 = 0 and

L n+1 = 1. If L n+1 = 1 then for all σ ∈ S, σ = X and log V = log   exp X λ n n i=1 b i Z i + λQX 2 - λX 2 2 Q   = X λ n n i=1 b i Z i + λX 2 2 Q .
L n+1 is independent of all other random variables, thus

E 1(L n+1 = 1) log V = (E P 0 X 2 ) λ 2 EQ ,
because the Z i are centered, independent from X and because X is independent from Q. The case L n+1 = 0 is obvious.

The variables (b i ) 1≤i≤n and (Z i ) 1≤i≤n are independent. Recall that

Q = 1 n n i=1 b 2 i . Therefore, X , Q, 1 √ n n i=1 b i Z i = X , Q, QZ 0 in law,
where Z 0 ∼ N (0, 1) is independent of Q, X . The expression of E log V from Lemma 3.4.7 simplifies

E log V = (E P 0 X 2 )E λQ 2 + (1 -)E log σ∈S P 0 (σ) exp λQσZ 0 + λQσX - λσ 2 2 Q , thus 0 0 d E log U -(E P 0 X 2 )E λQ 2 +(1-)E log σ∈S P 0 (σ) exp λQσZ 0 +λQσX - λσ 2 2 Q ---→ n→∞ 0 ,
which is precisely the statement of Lemma 3.4.1.

Reduction to distribution with finite support

We will show in this section that it suffices to prove Theorems 3.2.1 for input distribution P 0 with finite support.

Suppose the Theorem 3.2.1 holds for all prior distributions over R with finite support. Let P 0 be a probability distribution that admits a finite second moment: E P 0 X 2 < ∞. We are going to approach P 0 with distributions with finite supports. Let 0

< ≤ 1. Let K > 0 such that E P 0 [X 2 1(|X| ≥ K)] ≤ 2 . Let m ∈ N * such that K m ≤ . For x ∈ R we will use the notation x =      K m xm K if x ∈ [-K, K] , 0 otherwise. Consequently if x ∈ [-K, K], x ≤ x < x + K m ≤ x + .
We define P0 the image distribution of P 0 through the application x → x. Let n ≥ 1. We will note Fn the free energy corresponding to the distribution P0 and F the function F from (3.2.5) corresponding to the distribution P0 . P0 has a finite support, we have then by assumptions

Fn (λ) ---→ n→∞ sup q≥0 F(λ, q) . (3.4.14)
By construction we have for all 1

≤ i ≤ n, E(X i -Xi ) 2 ≤ 2 . Hence E (X i X j ) i<j -( Xi Xj ) i<j 2 ≤ 2(n -1)nE P 0 [X 2 ] 2 .
Consequently, by "pseudo-Lipschitz" continuity of the free energy with respect to the Wasserstein metric (see Proposition B.1 in Appendix B.4) there exist a constant C > 0 depending only on P 0 , such that, for all n ≥ 1 and all λ ≥ 0,

|F n (λ) -Fn (λ)| ≤ λC . (3.4.15)

Lemma 3.4.8

There exists a constant C > 0 that depends only on P 0 , such that

sup q≥0 F(λ, q) -sup q≥0 F(λ, q) ≤ λC .
Proof. First notice that both suprema are achieved over a common compact set [0,

E P 0 [X 2 + X2 ]].
Indeed, for q ≥ 0, [START_REF] David | Representations for partially exchangeable arrays of random variables[END_REF], we obtain that there exists a constant C depending only on P 0 such that ∀q ∈ [0,

∂ ∂q F(λ, q) = λψ P 0 (λq) - λq 2 ≤ λ 2 E P 0 [X 2 ] -q because ψ P 0 is 1 2 E P 0 [X 2 ]-Lipschitz by Proposition 1.3.3. Consequently, the maximum of F(λ, •) is achieved on [0, E P 0 [X 2 ]] and similarly the supremum of F(λ, •) is achieved over [0, E P 0 [ X2 ]]. Using Proposition B.1 in Appendix B.
E P 0 [X 2 + X2 ]], |F(λ, q) -F(λ, q)| ≤ λC . The lemma follows.
Combining Equation 3.4.14 and 3.4.15 and Lemma 3.4.8, we obtain that there exists n 0 ≥ 1 such that for all n ≥ n 0 ,

|F n -sup q≥0 F(λ, q)| ≤ λ(C + C + 1) ,
where C and C are two constants that only depend on P 0 . This proves Theorem 3.2.1.

Chapter 4

Non-symmetric low-rank matrix estimation

We consider now the spiked Wishart model (3.1.2). Let P U and P V be two probability distributions on R with finite second moment. We assume that Var P U (U ),

Var P V (V ) > 0. Let n, m ≥ 1, λ > 0 and consider U = (U 1 , . . . , U n ) i.i.d. ∼ P U and V = (V 1 , . . . , V m ) i.i.d.
∼ P V , independent from each other. Suppose that we observe

Y i,j = λ n U i V j + Z i,j , for 1 ≤ i ≤ n and 1 ≤ j ≤ m , (4.0.1)
where (Z i,j ) i,j are i.i.d. standard normal random variables, independent from U and V . In the following, E will denote the expectation with respect to the variables (U , V ) and Z.

We define the Minimum Mean Squared Error (MMSE) for the estimation of the matrix U V T given the observation of the matrix Y :

MMSE n (λ) = min θ    1 nm n i=1 m j=1 E U i V j -θ i,j (Y ) 2    = 1 nm n i=1 m j=1 E (U i V j -E [U i V j |Y ]) 2 ,
where the minimum is taken over all estimators θ (i.e. measurable functions of the observations Y ). In order to get an upper bound on the MMSE, let us consider the "dummy estimator" θ i,j = E[U i V j ] for all i, j which achieves a "dummy" matrix Mean Squared Error of:

DMSE = 1 nm n i=1 m j=1 E (U i V j -E[U i V j ]) 2 = E[U 2 ]E[V 2 ] -(EU ) 2 (EV ) 2 .

Fundamental limits of estimation

As in Chapter 3, we investigate the posterior distribution of U , V given Y . We define the Hamiltonian

H n (u, v) = i,j λ n u i v j Z i,j + λ n u i U i v j V j - λ 2n u 2 i v 2 j , for (u, v) ∈ R n × R m . (4.1.1)
The posterior distribution of (U , V ) given Y is then

dP u, v Y = 1 Z n (λ)
e Hn(u,v)

dP ⊗n U (u)dP ⊗m V (v), (4.1.2)
where Z n (λ) = e Hn(u,v) dP ⊗n U (u)dP ⊗m V (v) is the appropriate normalization. The corresponding free energy is

F n (λ) = 1 n E log Z n (λ) = 1 n E log e Hn(u,v) dP ⊗n U (u)dP ⊗m V (v) .
We consider here the high-dimensional limit where n, m → ∞, while m/n → α > 0. We will be interested in the following fixed point equations, sometimes called "state evolution equations". Definition 4.1.1

We define the set Γ(λ, α) as

Γ(λ, α) = (q u , q v ) ∈ R 2 ≥0 q u = 2ψ P U (λαq v ) and q v = 2ψ P V (λq u ) . (4.1.3) First notice that Γ(λ, α) is not empty. The function f : (q u , q v ) → (2ψ P U (λαq v ), 2ψ P V (λq u )) is continuous from the convex compact set [0, EU 2 ] × [0, EV 2 ] into itself (see Proposi- tion 1.3.3
). Brouwer's Theorem gives the existence of a fixed point of f : Γ(λ, α) = ∅.

We will express the limit of F n using the following function

F : (λ, α, q u , q v ) → ψ P U (λαq v ) + αψ P V (λq u ) - λα 2 q u q v . ( 4.1.4)
Recall that ψ P U and ψ P V , defined by (1.3.6), are the free energies of additive Gaussian scalar channels (1.3.5) with priors P U and P V . The Replica-Symmetric formula states that the free energy F n converges to the supremum of F over Γ(λ, α).

F n (λ) ---→ n→∞ sup (qu,qv)∈Γ(λ,α) F(λ, α, q u , q v ) = sup qu≥0 inf qv≥0 F(λ, α, q u , q v ) . (4.1.5)
Moreover, these extrema are achieved over the same couples (q u , q v ) ∈ Γ(λ, α).

This result proved in [START_REF] Miolane | Fundamental limits of low-rank matrix estimation: the non-symmetric case[END_REF] was conjectured by [START_REF] Thibault Lesieur | MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel[END_REF], in particular F corresponds to the "Bethe free energy" [START_REF] Thibault Lesieur | MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel[END_REF]Equation 47]. Theorem 4.1.1 is proved in Section 4.3. For the rank-k case (where P U and P V are probability distributions over R k ), see [START_REF] Miolane | Fundamental limits of low-rank matrix estimation: the non-symmetric case[END_REF]. As in Chapter 3, the Replica-Symmetric formula (Theorem 3.2.1) allows to compute the limit of the MMSE.

Proposition 4.1.1 (Limit of the MMSE)

Let

D α = λ > 0 F(λ, α, •, •) has a unique maximizer (q * u (λ, α), q * v (λ, α)) over Γ(λ, α) .
Then D α is equal to (0, +∞) minus a countable set and for all λ ∈ D α (and thus almost every λ > 0)

MMSE n (λ) ---→ n→∞ E[U 2 ]E[V 2 ] -q * u (λ, α)q * v (λ, α) . (4.1.6)
Again, this was conjectured in [START_REF] Thibault Lesieur | MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel[END_REF]: the performance of the Bayes-optimal estimator (i.e. the MMSE) corresponds to the fixed point of the state-evolution equations (4.1.3) which has the greatest Bethe free energy F. Proposition 4.1.1 follows from the same kind of arguments than Corollary 3.2.2 so we omit its proof for the sake of brevity. Proposition 4.1.1 allows to locate the information-theoretic threshold for our matrix estimation problem. Let us define

λ c (α) = inf λ ∈ D α | q * u (λ, α)q * v (λ, α) > (EU ) 2 (EV ) 2 . (4.1.7)
If the set of the left-hand side is empty, one defines λ c (α) = 0. Proposition 4.1.1 gives that λ c (α) is the information-theoretic threshold for the estimation of U V T given Y :

• If λ < λ c (α), then MMSE n (λ) ---→ n→∞ DMSE.
It is not possible to reconstruct the signal U V T better than a "dummy" estimator.

• If λ > λ c (α), then lim n→∞ MMSE n (λ) < DMSE.
It is possible to reconstruct the signal U V T better than a "dummy" estimator. Proposition 4.1.1 gives us the limit of the MMSE for the estimation of the matrix U V T , but does not gives us the minimal error for the estimation of U or V separately. As we will see in the next section with the spiked covariance model, one can be interested in estimating U U T or V V T , only. Let us define:

MMSE (u) n (λ) = 1 n 2 E 1≤i,j≤n U i U j -E[U i U j |Y ] 2 , MMSE (v) n (λ) = 1 m 2 E 1≤i,j≤m V i V j -E[V i V j |Y ] 2 . Theorem 4.1.2
For all α > 0 and all λ ∈ D α

MMSE (u) n (λ) ---→ n→∞ E P U [U 2 ] 2 -q * u (λ, α) 2 and MMSE (v) n (λ) ---→ n→∞ E P V [V 2 ] 2 -q * v (λ, α) 2 .
Theorem 4.1.2 is proved in Section 4.4. ∼ N (0, Σ), conditionally on Σ. Given the matrix Y = (Y 1 | • • • |Y m ), one would like to estimate the "spike" U U T . We deduce from Theorem 4.1.2 above the minimal mean squared error for this task, in the asymptotic regime where n, m → +∞ and m/n → α > 0.

Application to the spiked covariance model

Corollary 4.2.1

For all α > 0, the function

q → ψ P U (λαq) + α 2 q + log(1 -q)
admits for almost all λ > 0 a unique maximizer q * (λ, α) on [0, 1) and

MMSE (u) n (λ) = 1 n 2 E U U T -E[U U T |Y ] 2 ---→ n→∞ E P U [U 2 ] 2 - q * (λ, α) λ(1 -q * (λ, α)) 2 .
Proof. There exists independent Gaussian random variables

V = (V 1 , . . . , V m ) i.i.d.
∼ N (0, 1) and

Z i,j i.i.d. ∼ N (0, 1), independent from U such that Y = (Y 1 | • • • |Y m ) = λ n U V T + Z .
Therefore, the limit of the MMSE for the estimation of U U T is given by Theorem 4.1.2 above. It remains only to evaluate the formulas of Theorems 4.1.1 and 4.1.2 in the case P V = N (0, 1). As computed in Example 1.3.1, ψ N (0,1) (q) = 1 2 (qlog(1 + q)). Thus, the limit of the free energy (4.1.5) becomes (after evaluation of the supremum in q u ):

sup qv∈[0,1) ψ P U (λαq v ) + α 2 q v + log(1 -q v ) .
By Theorem 4.1.2 for all α > 0 and almost all λ > 0 this supremum admits a unique maximizer q * v (λ, α) and MMSE (u) n (λ) 2 where q * u verifies (recall that (q * u , q * v ) ∈ Γ):

→ E P U [U 2 ] 2 -q * u (λ, α)
q * v (λ, α) = 2ψ N (0,1) (λq * u (λ, α)) = λq * u (λ, α) 1 + λq * u (λ, α)
.

We deduce from the equation above that q * u (λ, α) = q * v (λ,α) λ(1-q * v (λ,α)) , which concludes the proof.

We will now compare the MMSE given by Corollary 4.2.1 to the mean squared errors achieved by PCA and Approximate Message Passing (AMP). Let u be a singular vector of Y / √ n associated with σ 1 , the top singular value of Y / √ n, such that u = √ n. Then results from [START_REF] Benaych | The singular values and vectors of low rank perturbations of large rectangular random matrices[END_REF][START_REF] Dobriban | Pca from noisy, linearly reduced data: the diagonal case[END_REF] give that almost surely:

lim n→∞ u • U 2 =    λ 2 α-1 λ(λα+1) if λ 2 α ≥ 1 ,

otherwise, and lim

n→∞

σ 1 =    (1+λ)(α -1 +λ) λ if λ 2 α ≥ 1 , 1 + 1/ √ α otherwise.
We are then going to estimate U U T using θ PCA = δ u u T , where δ is chosen in order to minimize the mean squared error. The optimal choice of δ is δ * = λ 2 α-1 λ(λα+1) +

, which can be estimated using σ 1 . We obtain the mean squared error of the spectral estimator θ PCA :

lim n→∞ MSE PCA n =      1+λ λ(λα+1) 2 -1+λ λ(λα+1) if λ 2 α ≥ 1 , 1 otherwise.
As in the symmetric case (see Section 3.3.1) one can define an Approximate Message Passing (AMP) algorithm to estimate U U T . For a precise description of the algorithm, see [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF][START_REF] Deshpande | Sparse pca via covariance thresholding[END_REF][START_REF] Thibault Lesieur | Phase transitions in sparse PCA[END_REF]. The MSE achieved by AMP after t iterations is:

lim n→∞ MSE AMP n = 1 -q t u 2 ,
where q t u is given by the recursion:

   q t u = 2ψ P U (λαq t v ) q t+1 v = 2ψ P V (λq t u ) , (4.2.2)
with initialization (q 0 u , q 0 v ) = (0, 0). We know by Proposition 1.3.3 that the functions ψ P U and ψ P V are both non-decreasing and bounded. This ensures that (q t u , q t v ) t≥0 converges as t → ∞ to some fixed point (q AMP u , q AMP v ) ∈ Γ. If this fixed point turns out to be the one that maximizes F(λ, α, •, •), i.e. that (q AMP u , q AMP v ) = (q * u (λ, α), q * v (λ, α)), then AMP achieves the minimal mean squared error! For the plots of Figure 4.1, we consider a case where the signal is sparse:

P U = sN (0, 1/s) + (1 -s)δ 0 , ( 4.2.3)
for some s ∈ (0, 1], so that E P U [U 2 ] = 1. We plot the different MSE on Figure 4.1. We chose λ = 1 so the "spectral threshold" (the minimal value of α for which PCA performs better than a random guess) it at α = 1 (green dashed line). This threshold corresponds also to the threshold for AMP: MSE AMP = 1 for α < 1 while MSE AMP < 1 for α > 1.

The information-theoretic threshold α IT is however strictly less than 1. For α ∈ (α IT , 1) inference is "hard": it is information-theoretically possible to achieve a MSE strictly less than 1, but PCA and AMP fail (and it is conjectured that any polynomial-time algorithm will also fail). However, even for α > 1, AMP does not always succeed to reach the MMSE. For α ∈ (1, α Alg ), MSE AMP is strictly less than 1 but is still very bad. So, the region α ∈ (1, α Alg ) is also a "hard region" in the sense that achieving the MMSE seems impossible for polynomial-time algorithms (under the conjecture that AMP is optimal among polynomialtime algorithms). The scenario presented on Figure 4.1 is not the only one possible: various cases have been studied in great details in [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications[END_REF]. See in particular Figure 6 from [START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications[END_REF] and the phase diagrams of Figure 7 and

Proof ideas

The proof of the Replica formula for the non-symmetric case is a little bit more involved compared to the symmetric case, because one can not use the convexity argument of Proposition 3.4.1 to obtain the lower bound. Indeed, a key step in the proof of Proposition 3.4.1 was the inequality (3.4.3) that was obtained by saying that for every q ≥ 0

E (x • X -q) 2 ≥ 0 , (4.3.1)
where x is a sample from the posterior distribution of X given some observations (we omit the notation's details here in order to focus on the main ideas). However, if we apply the strategy of Proposition 3.4.1 to the non-symmetric case, one obtains

E (u • U -q u )(v • V -q v ) (4.3.2)
where (u, v) is a sample from the posterior distribution of (U , V ) given some observations, instead of (4.3.1). Now, it not obvious anymore that (4.3.2) is non-negative. In order to prove it, one has to investigate further the distributions of the overlaps u • U and v • V . By following the approach used by Talagrand in [START_REF] Talagrand | Mean field models for spin glasses: Volume I: Basic examples[END_REF] to prove the TAP equations (discovered by Thouless, Anderson and Palmer in [START_REF] David J Thouless | Solution of'solvable model of a spin glass[END_REF]) for the Sherrington-Kirkpatrick model, one can show that the overlaps approximately satisfy (when n and m are large)

   u • U 2ψ P U (λαv • V ) v • V 2ψ P V (λu • U ) .
These are precisely the fixed point equations verified by (q u , q v ) ∈ Γ(λ, α). Thus one has

E (u • U -q u )(v • V -q v ) E (2ψ P U (λαv • V ) -2ψ P U (λαq v ))(v • V -q v ) ≥ 0 , (4.3.3)
because by Proposition 1.3.3, ψ P U is non-decreasing. One obtain thus the analog of the lower-bound of Proposition 3.4.1 for the non-symmetric case. The converse upper-bound is proved following the Aizenman-Sims-Starr scheme, as in the symmetric case.

In the following sections we will not, however, follow the proof strategy that we just described. This was done in [START_REF] Miolane | Fundamental limits of low-rank matrix estimation: the non-symmetric case[END_REF]. We will instead provide a more straightforward proof from [START_REF] Barbier | The layered structure of tensor estimation and its mutual information[END_REF] that uses an evolution of Guerra's interpolation technique, see [START_REF] Barbier | The stochastic interpolation method: A simple scheme to prove replica formulas in bayesian inference[END_REF].

Interpolating inference model

We prove Theorem 4.1.1 in this section. First, notice that is suffices to prove Theorem 4.1.1 for λ = 1, because the dependency in λ can be "incorporated" in the prior P U . We will thus consider in this section that λ = 1 and consequently alleviate the notations by removing the dependencies in λ. Second, it suffices to prove that

F n ---→ n→∞ sup qu≥0 inf qv≥0 F(α, q u , q v ) (4.3.4)
because the equality with sup (qu,qv)∈Γ(λ,α) F(α, q u , q v ) follows then from simple convex analysis arguments (Proposition C.6) presented in Appendix C. Third, by a straightforward adaptation of the approximation argument of Section 3.4.7 to the non-symmetric case, it suffices to prove (4.3.4) in the case where the priors P U and P V have bounded supports included in [-K, K] for some K > 0. We suppose now that the above conditions are verified and we will show that (4.3.4) holds.

Let q 1 , q 2 : [0, 1] → R ≥0 be two differentiable functions. For 0 ≤ t ≤ 1 we consider the following observation channel

         Y t = (1 -t)/n U V T + Z Y (u) t = αq 1 (t) U + Z (u) Y (v) t = q 2 (t) V + Z (v) , (4.3.5)
where

Z (u) i , Z (v) j i.i.d.
∼ N (0, 1), are independent from everything else. The observation channel (4.3.5) interpolates between the initial matrix estimation problem (4.0.1) (t = 0, provided that q 1 (0) and q 2 (0) are small), and two decoupled inference channels on U and V (t = 1). For r 1 , r 2 ≥ 0, we define the Hamiltonian:

H n,t (u, v; r 1 , r 2 ) = i,j (1 -t) n u i v j Z i,j + (1 -t) n u i v j U i V j - (1 -t) 2n u 2 i v 2 j + n i=1 √ αr 1 u i Z (u) i + αr 1 u i U i - αr 1 2 u 2 i + m j=1 √ r 2 v j Z (v) j + r 2 v j V j - r 2 2 v 2 j .
The posterior distribution of (U , V ) given

(Y t , Y (u) t , Y (v) t ) is then dP u, v Y t , Y (u) t , Y (v) t = 1 Z n,t
e Hn,t(u,v;q 1 (t),q 2 (t)) dP ⊗n U (u)dP ⊗m V (v) , (4.3.6) where Z n,t is the appropriate normalization. We will often drop the dependencies in q 1 (t), q 2 (t) and write simply H n,t (u, v). The Gibbs bracket • n,t denotes the expectation with respect to samples (u, v) from the posterior (4.3.6):

f (u, v) n,t = 1 Z n,t f (u, v)e Hn,t(u,v;q 1 (t),q 2 (t)) dP ⊗n U (u)dP ⊗m V (v) , ( 4.3.7)
for all function f for which the right-hand side is well defined. The corresponding free energy is then

f n (t) = 1 n E log Z n,t = 1 n E log e Hn,t(u,v) dP ⊗n U (u)dP ⊗m V (v) . (4.3.8) Notice that f n (0) = F n + O(q 1 (0) + q 2 (0)) f n (1) = ψ P U (αq 1 (1)) + m n ψ P V (q 2 (1)
) .

(4.3.9)

f n (1) looks similar to the limiting expression F defined by (4.1.4). We would therefore like to compare f n [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF] and F n = f n (0) + O(q 1 (0) + q 2 (0)). We thus compute the derivative of f n :

Lemma 4.3.1
For all t ∈ (0, 1),

f n (t) = α 2 q 1 (t)q 2 (t) - 1 2 E u • U -q 2 (t) m n v • V -αq 1 (t) n,t . (4.3.10)
Proof. Let t ∈ (0, 1). Compute

f n (t) = 1 n E ∂ ∂t H n,t (u, v) n,t
.

Using Gaussian integration by parts and the Nishimori property (Proposition 1.1.1) as in the proof of Proposition 1.3.3, one obtains:

1 n E ∂ ∂t H n,t (u, v) n,t = 1 2 αq 1 (t)E u • U n,t + 1 2 q 2 (t)E m n v • V n,t - 1 2 E (u • U )( m n v • V ) n,t
, which leads to (4.3.10).

Our goal now is to show that the expectation of the Gibbs measure in (4.3.10) vanishes. If this is the case, the relation

F n f n (0) = f n (1) -1 0 f n (t)
dt would give us almost the formula that we want to prove. The arguments can be summarized as follows:

• First, we show that the overlap u • U concentrates around its mean E u • U n,t .

• Then, we chose q 2 to be solution of the differential equation q 2 (t) = E u • U n,t in order to cancel the Gibbs average in (4.3.10).

Overlap concentration

Following the ideas of Section 2.2, we show here that the overlap u • U concentrates around its mean, on average over small perturbations of our observation model. Proposition 4.

3.1

Let R 1 , R 2 : [0, 1]×(0, +∞) 2 → R ≥0 be two continuous, bounded functions that admits partial derivatives with respect to their second and third arguments, that are continuous and non-negative. Let s n = n -1/32 . For ∈ [1, 2] 2 , we let q 1 (•, ), q 2 (•, ) be the unique solution of

   q 1 (0) = s n 1 q 2 (0) = s n 2    q 1 (t) = R 1 (t, q 1 (t), q 2 (t)) q 2 (t) = R 2 (t, q 1 (t), q 2 (t)). (4.3.11)
Then there exists a constant C > 0 that only depends on K, α, R 1 ∞ and R 2 ∞ , such that for all t ∈ [0, 1],

2 1 2 1 E u • U -E u • U n,t 2 n,t d 1 d 2 ≤ C n 1/8 ,
where • n,t is the Gibbs measure (4.3.7) with (q 1 , q 2 ) = (q 1 (•, ), q 2 (•, )).

Proof. The existence and uniqueness of the solution of the Cauchy problem (4.3.11) comes from the usual Cauchy-Lipschitz theorem (see for instance Theorem 3.1 in Chapter V from [START_REF] Hartman | Ordinary Differential Equations[END_REF]). Let us fix t ∈ [0, 1] The flow Q t : → (q 1 (t, ), q 2 (t, )) of (4.3.11) is a C 1 -diffeomorphism. Its Jacobian is given by the Liouville formula (see for instance Corollary 3.1 in Chapter V from [START_REF] Hartman | Ordinary Differential Equations[END_REF]):

J( ) def = det ∂Q t ∂ ( ) = s 2 n exp t 0 ∂R 1 ∂q 1 (s, Q s ( ))ds + t 0 ∂R 2 ∂q 2 (s, Q s ( ))ds ≥ s 2 n , (4.3.12)
because the partial derivatives inside the exponential are both non-negative. The quantity

E u • U -E u • U n,t 2 n,t
is a function of the signal-to-noise ratios q 1 and q 2 , that we denote by

V . Let us write Ω = Q([1, 2] 2 )/s n and M = max( R 1 ∞ , R 2 ∞ ) + 2. Notice that Ω ⊂ [1, M/s n ] 2 because q 1 , q 2 are by (4.3.11) non-decreasing and max( R 1 ∞ , R 2 ∞ )-Lipschitz. By the change of variable (r 1 , r 2 ) = Q( 1 , 2 )/s n we have 2 1 2 1 E u • U -E u • U n,t 2 n,t d 1 d 2 = 2 1 2 1 V (q 1 (t, 1 ), q 2 (t, 2 ))d 1 d 2 = Ω V (s n r 1 , s n r 2 ) s 2 n dr 1 dr 2 J(Q -1 t (s n r)) ≤ M/sn 1 M/sn 1 V (s n r 1 , s n r 2 )dr 1 dr 2 ,
where we used (4.3.12) for the last inequality. By the change of variable r 1 = a 2 , we have for all r 2 ≥ 0:

M/sn 1 V (s n r 1 , s n r 2 )dr 1 = √ M/sn 1 V (s n a 2 , s n r 2 )2ada ≤ 2 M s n √ M/sn 1 V (s n a 2 , s n r 2 )da.
By definition of V , the quantity V (s n a 2 , s n r 2 ) is the variance of the overlap u • U where u is sampled from the posterior distribution of U given Y t , a √ αs n U + Z (u) and √ s n r 2 V + Z (v) . By Proposition 2.2.1 we have for all 1 ≤ r 2 ≤ M/s n 1 M/s n -1

√ M/sn 1 V (s n a 2 , s n r 2 )da ≤ C 1 √ ns n + √ v n
where C > 0 is a constant that depends only on K, α,

v n = sup t∈[0,1] sup 0≤r 1 ,r 2 ≤M/sn E φ t (r 1 , r 2 ) -Eφ t (r 1 , r 2 )
and

φ t : (r 1 , r 2 ) → 1 ns n log u,v
dP ⊗n U (u)dP ⊗m V (v)e Hn,t(u,v;snr 1 ,snr 2 ) .

Consequently

2 1 2 1 E u • U -E u • U n,t 2 n,t d 1 d 2 ≤ 2 M s n 2 C 1 √ ns n + √ v n ≤ C s 2 n 1 √ ns n + √ v n ,
for some constant C > 0. We now use the following lemma to control v n : Lemma 4

.3.2

There exists a constant C > 0 (that only depends on K, M and α) such that

v n ≤ Cn -1/2 s -1 n .
We delay the proof of Lemma 4.3.2 to Section 4.3.5. We deduce that

2 1 2 1 E u • U -E u • U n,t 2 n,t d 1 d 2 ≤ 2C n 1/8
if we choose s n = n -1/32 .

Lower and upper bounds

From now we write E u • U n,t , as a function of (t, q 1 (t), q 2 (t)):

E u • U n,t = Q(t, q 1 (t), q 2 (t)). (4.3.13)
Notice that Q is continuous, non-negative on [0, 1] × (0, +∞) 2 , bounded by K 2 and admits partial derivatives with respect to its second and third argument. These derivatives are both continuous. Moreover, notice that

E P U [U 2 ] -Q(t, r 1 , r 2 ) = E U -E U Y t , √ αr 1 U + Z (u) , √ r 2 V + Z (v) 2
is of course non-increasing with respect to r 1 and r 2 . The partial derivatives of Q with respect to its second and third argument are thus non-negative.

For simplicity we will now omit the dependencies on λ and α in F. The proof of (4.3.4) will follow from the two matching lower-and upper-bounds below. Proposition 4.3.2

In the setting of Proposition 4.3.1, for ∈ [1, 2] 2 we let q 1 (t, ), q 2 (t, ) be the solution of (4.3.11), with the choice R 2 = Q. For this choice of functions q 1 , q 2 , we have:

F n = [1,2] 2 1 0 ψ P U (αq 1 (1, )) + αψ P V (q 2 (1, )) - α 2 q 1 (t, )q 2 (t, ) dtd + o n (1). Proof. Let us fix ∈ [1, 2] 2 .
With the choice R 2 = Q, we have for all t ∈ [0, 1]:

q 2 (t, ) = Q(t, q 1 (t, ), q 2 (t, )) = E u • U n,t .
The derivative of (4.3.10) becomes then by Proposition 4.3.1:

f n (t) = α 2 q 1 (t)q 2 (t) - 1 2 E u • U -E u • U n,t m n v • V -αq 1 (t) n,t = α 2 q 1 (t)q 2 (t) + o n (1)
where o n (1) denotes a quantity that goes to 0 as n → ∞, uniformly in t, . By Proposition B.1 we have f n (0) = F n + O n (s n ). We have also:

f n (1) = ψ P U (αq 1 (1, )) + αψ P V (q 2 (1, )) + o n (1).
We conclude by

F n = [1,2] 2 f n (0)d + o n (1) = [1,2] 2 f n (1) - 1 0 f n (t)dt d + o n (1) = [1,2] 2 1 0 ψ P U (αq 1 (1, )) + αψ P V (q 2 (1, )) - α 2 q 1 (t, )q 2 (t, ) dtd + o n (1).

Lower bound

One deduces from Proposition (4.3.2) the following lower bound:

Proposition 4.3.3 lim inf n→∞ F n ≥ sup q 1 ≥0 inf q 2 ≥0
F(q 2 , q 1 ) .

Proof. We apply Proposition 4.3.2 with R 1 = r, for some r ≥ 0. We get q 1 (t, ) = 1 s n + rt, so that:

F n = [1,2] 2 1 0 ψ P U (α(s n 1 + r)) + αψ P V (q 2 (1, )) - α 2 rq 2 (t, ) dtd + o n (1). = [1,2] 2 ψ P U (αr) + αψ P V (q 2 (1, )) - α 2 rq 2 (1, ) d + o n (1). ≥ inf q 2 ≥0 F(q 2 , r) + o n (1),
where we used the fact that ψ P U is 1 2 K 2 -Lipschitz, and that s n → 0. This proves the proposition since the last inequality holds for all r ≥ 0.

Upper bound

We will now prove the converse upper bound. Proposition 4.3.4

lim sup n→∞ F n ≤ sup q 1 ≥0 inf q 2 ≥0
F(q 2 , q 1 ) .

Proof. We apply Proposition 4.3.2 with

R 1 = 2αψ P V • Q. R 1 verifies the conditions of Proposi- tion 4.3.1 because ψ P V is a C 2 convex Lipschitz function (Proposition 1.3.3).
For simplicity we omit briefly the dependencies in of q 1 and q 2 . ψ P U is K 2 /2-Lipschitz, and [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF], where o n (1) is a quantity that goes to 0 as n → ∞, uniformly in ∈ [1, 2] 2 . Notice that by convexity of the functions ψ P U and ψ P V , we get [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. and similarly: ψ P V (q 2 (1)) ≤ 1 0 ψ P V (q 2 (t))dt + o n [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. We get by Proposition 4.3.2

q 1 (0) = s n = o n (1) so ψ P U (αq 1 (1)) = ψ P U α(q 1 (1) -q 1 (0)) + o n
ψ P U (αq 1 (1)) = ψ P U α 1 0 q 1 (t) + o n (1) ≤ 1 0 ψ P U (αq 1 (t))dt + o n
F n ≤ [1,2] 2 1 0 ψ P U (αq 1 (t, )) + αψ P V (q 2 (t, )) - α 2 q 1 (t, )q 2 (t, ) dtd + o n (1). = [1,2] 2 1 0 F(q 2 (t, ), q 1 (t, ))dtd + o n (1). (4.3.14) Since we chose R 2 = Q and R 1 = 2αψ P V • Q = 2αψ P V • R 2 , Equation (4.3.11) gives: ∀ ∈ [1, 2] 2 , ∀t ∈ [0, 1], q 1 (t, ) = 2αψ P V (q 2 (t, )).
By convexity of ψ P V , this gives that for all ∈ [1, 2] 2 and all t ∈ [0, 1] we have

F(q 2 (t, ), q 1 (t, )) = inf q 2 ≥0
F(q 2 , q 1 (t, )) ≤ sup

q 1 ≥0 inf q 2 ≥0
F(q 2 , q 1 ).

Together with (4.3.14), this concludes the proof.

Concentration of the free energy: proof of Lemma 4.3.2

In this section, we prove Lemma 4.3.2: we show that the perturbed free energy concentrates around its mean, uniformly in the perturbation. Lemma 4.3.2 will follow from Lemma 4.3.3 and Lemma 4.3.4 below. Let E z denote the expectation with respect to the Gaussian random variables Z, Z (u) , Z (v) .

Lemma 4.3.3

There exists a constant C > 0, that only depends on K, α, such that for all t ∈ [0, 1],

B ≥ 0 and (r 1 , r 2 ) ∈ [0, B] 2 , E |φ t (r 1 , r 2 ) -E z φ t (r 1 , r 2 )| ≤ Cn -1/2 s -1 n 1 + Bs n .
Proof. Let (r 1 , r 2 ) ∈ [0, B] 2 and consider U and V to be fixed (i.e. we first work conditionally on U , V ). Consider the function f : (Z, Z (u) , Z (v) ) → φ t (r 1 , r 2 ) .

It is not difficult to verify that

∇f 2 ≤ C ns 2 n (1 + Bs n )
for some constant C > 0 that depends only on K and α. The Gaussian Poincaré inequality (see [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF] Chapter 3) gives then

E z (φ t (r 1 , r 2 ) -E z φ t (r 1 , r 2 )) 2 ≤ C ns 2 n (1 + Bs n ) .
We obtain the lemma by integration over U , V and Jensen's inequality.

Lemma 4.3.4

There exists a constant C > 0, that only depends on K, α, such that for all t ∈ [0, 1],

B ≥ 0 and (r 1 , r 2 ) ∈ [0, B] 2 , E |E z φ t (r 1 , r 2 ) -Eφ t (r 1 , r 2 )| ≤ Cn -1/2 s -1 n 1 + Bs n .
Proof. It is not difficult to verify that the function

g : (U , V ) → E z φ t (r 1 , r 2 )
verifies a "bounded difference property" (see [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF], Section 3.2) because the components of U and V are bounded by a constant K > 0. Then Corollary 3.2 from [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF] (which is a corollary from the Efron-Stein inequality) implies that for all t ∈ [0, 1] and

r 1 , r 2 ∈ [0, B] E (E z φ t (r 1 , r 2 ) -Eφ t (r 1 , r 2 )) 2 ≤ Cn -1 s -2 n (1 + s n B) .
for some constant C depending only on K and α. We conclude the proof using Jensen's inequality.

Proof of Theorem 4.1.2

In order to prove Theorem 4.1.2, we are going to consider the following model with side information to obtain a lower bound on the MMSE. Suppose that we observe for γ ≥ 0

   Y λ = λ n U V T + Z Y γ = γ n U U T + Z (4.4.1)
where (Z i,j = Z j,i ) i≤j i.i.d.

∼ N (0, 1) are independent from everything else. Define the corresponding free energy

F n (λ, γ) = 1 n E log dP ⊗n U (u)dP ⊗m V (v) exp 1≤i≤j≤n γ n Y i,j u i u j - γu 2 i u 2 j 2 + i,j λ n Y i,j u i v j - λu 2 i v 2 j 2n .
Proposition 4.4.1

Recall that ψ * P U (resp. ψ * P V ) denotes the monotone conjugate (Definition C.2 in Appendix C) of ψ P U (resp. ψ P V ). For all λ, γ ≥ 0, we have For almost all γ > 0 the supremum of Proposition 4.4.1 is achieved at a unique q * u (λ, γ, α) and

F n (λ, γ) ---→ n→∞ f (λ, γ) def = sup qu,qv≥0 γq 2 u 4 + αλq u q v 2 -ψ * P U (q u /2) -αψ * P V (q v /2
MMSE (u) n (λ, γ) def = 1 n 2 E 1≤i,j≤n U i U j -E[U i U j |Y λ , Y γ ] 2 ---→ n→∞ E[U 2 ] 2 -q * u (λ, γ, α) 2 .
The model (4.4.1) was considered in [START_REF] Deshpande | Contextual stochastic block models[END_REF], in the special case P U = 1 2 δ -1 + 1 2 δ +1 and P V = N (0, 1). Theorem 6 from [START_REF] Deshpande | Contextual stochastic block models[END_REF] shows that one can estimate U U T better than a random guess if and only if γ 2 + αλ 2 > 1. Corollary 4.4.1 above is more precise and general because it gives the precise expression of the minimum mean squared error for any prior P U , P V . In particular the boundary γ 2 + αλ 2 = 1 is not expected to be the informationtheoretic threshold for sufficiently sparse or unbalanced priors, see the phase diagram of

MMSE (u) n (λ) = MMSE (u) n (λ, γ = 0) = E P U [U 2 ] -4 ∂F n ∂γ (λ, 0 + ). (4.4.3)
The sequence of convex functions (F n (λ, •)) n≥1 converges pointwise to f (λ, •) on R ≥0 . Thus, by Proposition C.1:

lim sup n→∞ ∂F n ∂γ (λ, 0 + ) ≤ ∂f ∂γ (λ, 0 + ). (4.4.4)
We need therefore the following lemma: Lemma 4.4.1

For all α > 0 and all λ ∈ D α , ∂f ∂γ (λ, 0

+ ) = q * u (α, λ) 2 4 .
Proof. Let α > 0 and λ ∈ D α . Then the supremum of (4.4.2) is uniquely achieved at (q * u (λ, α), q * v (λ, α)) because the couples achieving this supremum are by Proposition C.6 precisely the couples achieving the supremum of F(λ, α, •, •) over Γ(λ, α). The lemma follows then from the "envelope theorem" of Proposition D.2.

From Lemma 4.4.1 and equations (4.4.3)-(4.4.4) above, we conclude:

lim inf n→∞ MMSE (u) n (λ) ≥ E P U [U 2 ] 2 -q * u (λ, α) 2 .
Let (u, v) sampled from the posterior distribution of (U , V ) given Y , independently of everything else. Then MMSE (u)

n (λ) = E P U [U 2 ] 2 -E (u • U ) 2 + o n (1)
. This gives (the corresponding result for V is obtained by symmetry):

lim sup n→∞ E (u • U ) 2 ≤ q * u (λ, α) 2 and lim sup n→∞ E (v • V ) 2 ≤ q * v (λ, α) 2 . (4.4.5)
Now, we know by Proposition 4.1.1 that

E P U [U 2 ]E P V [V 2 ] -E (u • U )(v • V ) = MMSE n (λ) ---→ n→∞ E P U [U 2 ]E P V [V 2 ] -q * u q * v , which gives E (u • U )(v • V ) → q * u q * v .
By Cauchy-Schwarz inequality we have

E (u • U )(v • V ) 2 ≤ E (u • U ) 2 E (v • V ) 2
which gives, by taking the liminf:

(q * u q * v ) 2 ≤ lim inf n→∞ E (u • U ) 2 lim inf n→∞ E (v • V ) 2 .
Combining this with (4.4.5), we get that lim E (u•U ) 2 = (q * u ) 2 and the relation [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF] gives the result.

MMSE (u) n (λ) = E P U [U 2 ] 2 -E (u • U ) 2 + o n

Proof of Proposition 4.4.1

It suffices to prove the result in the case where P U and P V have bounded support, because we can then proceed by approximation as in Section 3.4.7. From now, we suppose to be in that case. Since the dependency in γ can be incorporated in the prior P U and the one in λ in the prior P V , we only have to prove Proposition 4.4.1 in the case γ = λ = 1. In the sequel we will therefore remove the dependencies in λ, γ. Define for r ≥ 0

L n (r) = 1 n E log dP ⊗n U (u)dP ⊗m V (v) exp H n (u, v) + n i=1 √ rZ i u i + rU i u i - r 2 u 2 i ,
where

Z i i.i.d.
∼ N (0, 1), independently of everything else and where the Hamiltonian H n (u, v) is defined by (4.1.1) (with λ = 1). L n is the free energy (expected log-partition function) for observing jointly Y = 1

L n (r) ---→ n→∞ L(r) (4.4.6)
where

L(r) = sup qu≥0 inf qv≥0 ψ P U (αq v + r) + αψ P V (q u ) - αq u q v 2 . Lemma 4.4.2 F n (λ, γ) ---→ n→∞ sup r≥0 L(r) - r 2 4 . (4.4.7)
Proof. We will follow the same steps than in Section 4.3: we will therefore only present the main steps. Let r : [0, 1] → R ≥0 be a differentiable function. For 0 ≤ t ≤ 1 we consider the following observation channel

     Y = 1/n U V T + Z Y t = (1 -t)/n U U T + Z Y t = r(t) U + Z , ( 4
f n (t) = - 1 4 E u • U -r (t) 2 n,t + r (t) 2 4 + o n (1), (4.4.9)
where o n (1) → 0, uniformly in t ∈ [0, 1]. By taking r(t) = rt for all t ∈ [0, 1], we obtain

F n = f n (0) = f n (1) - 1 0 f n (t)dt ≥ L n (r) - r 2 4 . Therefore lim inf F n ≥ lim inf n→∞ L n (r) -r 2 4 which gives lim inf F n ≥ L(r) -r 2 4 for all r ≥ 0, hence lim inf F n ≥ sup r≥0 L(r) -r 2 4 .
To prove the converse upper-bound we proceed as in Section 4.3 and chose r to be solution r(•; ) of the Cauchy problem:

r(0) = n -1/32 r (t) = E u • U n,t
where ∈ [1, 2] is a parameter. The analog of Proposition 4.3.1 holds:

2 1 E u • U -E u • U n,t 2 n,t d ≤ C n 1/8
for some constant C > 0. Using (4.4.9) we get

F n = f n (0) + o n (1) = 2 1 f n (1) - 1 0 f n (t)dt d + o n (1) = 2 1 L n (r(1, )) - 1 0 r (t, ) 2 4 dt d + o n (1) ≤ 2 1 1 0 L n (r (t, )) - r (t, ) 2 4 dtd + o n (1) ≤ sup 0≤r≤ρu L n (r) - r 2 4 + o n (1),
where

ρ u = E P U [U 2 ].
γq 2 u 4 + αλq u q v 2 -ψ * P U (q u /2) -αψ * P V (q v /2) .
This is a consequence of the following Lemma: Lemma 4.4.3

Let f, g be two non-decreasing lower semi-continuous convex functions on R ≥0 , such that f (0) and g(0) are finite. Let f * and g * denote their monotone conjugate (see Definition C.2 in Appendix C). Then

sup r≥0 sup q 1 ≥0 inf q 2 ≥0 f (q 1 ) + g(r + q 2 ) -q 1 q 2 - r 2 2 = sup q 1 ,q 2 ≥0 q 2 1 2 + q 1 q 2 -f * (q 2 ) -g * (q 1 )
Proof. Let r ≥ 0. Let us write g r : q → g(q + r). By Proposition C.6, we have sup

q 1 ≥0 inf q 2 ≥0
f (q 1 ) + g(r + q 2 ) -q 1 q 2 = sup

q 1 ,q 2 ≥0 q 1 q 2 -f * (q 2 ) -g * r (q 1 ) = sup q 2 ≥0 g(r + q 2 ) -f * (q 2 ) = sup q 1 ,q 2 ≥0 q 1 (q 2 + r) -f * (q 2 ) -g * (q 1 ) ,
where we used Proposition C.4 for the two last equalities. Therefore

sup r≥0 sup q 1 ≥0 inf q 2 ≥0
f (q 1 ) + g(r + q 2 ) -q 1 q 2 -r 2 2 = sup

q 1 ,q 2 ≥0 sup r≥0 q 1 r - r 2 2 + q 1 q 2 -f * (q 2 ) -g * (q 1 ) = sup q 1 ,q 2 ≥0
q 2 1 2 + q 1 q 2 -f * (q 2 ) -g * (q 1 ) .

Chapter 5

Community detection in the asymmetric stochastic block model

Introduction

The community detection problem is fundamental problem in statistics. It consists in dividing the vertices of a given graph into groups which are more densely connected than the rest of the graph. The stochastic block model is a popular model of random graphs that exhibit communities: it is generated according to an underlying partition of the vertices. This model has been studied for a long time in statistics (see [START_REF] Paul W Holland | Stochastic blockmodels: First steps[END_REF]) computer science (see [START_REF] Dyer | The solution of some random np-hard problems in polynomial expected time[END_REF]) and more recently in statistical physics (see [START_REF] Decelle | Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications[END_REF]).

Definition 5.1.1 (Stochastic block model (SBM))

Let M be a 2×2 symmetric matrix whose entries are in [0, 1]. Let n ∈ N * and p ∈ [0, 1].

We define the stochastic block model with parameters (M, n, p) as the random graph G defined by:

1. The vertices of G are the integers in {1, . . . , n}.

2.

For each vertex i ∈ {1, . . . , n} one draws independently X i ∈ {1, 2} according to P(X i = 1) = p. X i will be called the label (or the class, or the community) of the vertex i.

3. For each pair of vertices {i, j} the unoriented edge G i,j is then drawn conditionally on X i and X j according to a Bernoulli distribution with mean M X i ,X j .

Our main focus will be on the community detection problem: given the graph G, is it possible to retrieve the labels X better than a random guess?

We investigate this question in the asymptotic of large sparse graphs, when n → +∞ while the average degree remains fixed. Our quantitative results will then be obtained when the average degree tends to infinity. We define the connectivity matrix M as follows:

M = d n a b b c , (5.1.1)
where a, b, c, d remain fixed as n → +∞. In this case, it is not possible to correctly classify more than a certain fraction of the vertices correctly and we will say that community de- tection is solvable if there is some algorithm that recovers the communities more accurately than a random guess would.

The symmetric case where p = 1/2 and a = c has been extensively studied starting with [START_REF] Decelle | Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications[END_REF] and gives rise to an interesting phenomenon: if d(1 -b) 2 < 1 then community detection is not solvable [START_REF] Mossel | Reconstruction and estimation in the planted partition model[END_REF], while if d(1-b) 2 > 1, it is solvable (in polynomial time) [START_REF] Massoulié | Community detection thresholds and the weak ramanujan property[END_REF][START_REF] Mossel | A proof of the block model threshold conjecture[END_REF]. Much less is known in the case where (5.1.2) holds and p < 1/2. The aim of this work is to investigate this asymmetric case. The main question is: does the asymmetry change the location of the above mentioned threshold?

A simple argument (see below) shows that if pa+(1-p)b = pb+(1-p)c then non-trivial information on the community of a vertex can be gained just by looking at its degree and the community detection is then solvable. In this paper, we concentrate on the case:

pa + (1 -p)b = pb + (1 -p)c = 1.
(5.1.2)

Under condition (5.1.2), the matrix R defined by

R = pa (1 -p)b pb (1 -p)c (5.1.3)
is a stochastic matrix with two eigenvalues: [START_REF] Neeman | Non-reconstructability in the stochastic block model[END_REF] that it is easy to distinguish G from an Erdős-Rényi model with the same average degree. In this regime, the spectral algorithm based on the nonbacktracking matrix solves the community detection problem [START_REF] Bordenave | Non-backtracking spectrum of random graphs: community detection and non-regular ramanujan graphs[END_REF][START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. Here, we prove that if a vanishing fraction of labels is given, then a local algorithm (belief propagation) allows to solve the community detection problem. The case λ < 1 (known as below the Kesten-Stigum bound) is more challenging. It is shown in [START_REF] Neeman | Non-reconstructability in the stochastic block model[END_REF], that the community detection problem is still solvable for some values of p but it is expected in [START_REF] Decelle | Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications[END_REF] that no computationally efficient reconstruction is possible. In [START_REF] Neeman | Non-reconstructability in the stochastic block model[END_REF], some bounds are given on the non-reconstructability region but they are not expected to be tight.

λ 1 = 1 > λ 2 = 1 -b. Define λ = dλ 2 2 = d(1 -b) 2 . (5.1.4) If λ > 1, it is shown in
In this work, we are mainly interested in a regime where d tends to infinity while λ = d(1-b) 2 remains constant (in particular a, b, c tend to one). Note that we first let n tend to infinity and then let d tend to infinity mainly in order to get explicit formulas. In this regime, we show that for all values of p ∈ (p * , 1/2) with p * = 1 2 -1 2 √

3 , the situation is similar to the balanced case: below the Kesten-Stigum bound, i.e. when λ < 1, the community detection problem is not solvable. For p < p * , we compute a function p → λ sp (p) < 1 such that for λ < λ sp (p), the community detection problem is not solvable. As shown by [START_REF] Neeman | Non-reconstructability in the stochastic block model[END_REF], there are points in the region λ sp (p) < λ < 1 where the community detection problem is solvable but we do not expect the bound λ sp (p) to be tight, i.e. the information theoretic threshold for community detection should be above λ sp (p) for p < p * .

There is an important probabilistic interpretation of the matrix R relating to the local structure of the SBM. As explained below, the SBM converges locally toward a labeled Poisson Galton-Watson branching process with mean offspring d: the label of the root is 1 with probability p and 2 with probability 1 -p and then conditioned on the parent's label being i, its children's labels are independently chosen to be j with probability R ij . A problem closely related to the detection problem in the SBM is the reconstruction problem on this random tree: given some information about the labels at depth n from the root, is it possible to infer some information about the label of the root when n → ∞? It is known [START_REF] Mossel | Survey-information flow on trees[END_REF] that the Kesten-Stigum bound corresponds to census-solvability (i.e. knowing only the number of labels 1 and 2 at depth n allows to get some information about the label of the root). When d → ∞, we show that λ sp (p) corresponds to the solvability threshold for the reconstruction problem on the tree (i.e. knowing the labels at depth n allows to get some information about the label of the root). We also consider the reconstruction problem where the label of each node at depth n is revealed with probability q. Then in the region λ sp (p) < λ < 1, we compute the minimal value of q such that some information about the label of the root can be recovered from the revealed labels. Above the Kesten-Stigum bound, i.e. when 1 < λ, this minimal value is 0. The easy phase follows from [START_REF] Bordenave | Non-backtracking spectrum of random graphs: community detection and non-regular ramanujan graphs[END_REF], the impossible phase below the spinodal curve (red curve) is proved in this paper and the hard phase is a conjecture. The dotted curve corresponding to λ c (p) is the conjectured curve for solvability of the community detection problem (see discussion in Section 5.4).

We summarize our main results on the phase diagram of Figure 5.2:

• Above the Kesten-Stigum bound (blue line), reconstruction is possible by a local algorithm given that an arbitrary small fraction of the labels is revealed. Moreover, the local algorithms (with this arbitrary small side information) achieve then the best possible performance without side information (see Proposition 5.3.1).

• Between the blue and the red line, we show that local algorithms are efficient for reconstruction when a certain fraction of labels is revealed (see Proposition 5.4.2).

• We show that reconstruction is impossible below the spinodal curve (red line), see Proposition 5.4.1.

In Section 5.2, we define the community detection problem and its variation when some labels are revealed. In Section 5.3, we give our main results about reconstruction above the Kesten-Stigum bound and in Section 5.4, we describe what happens below the Kesten-Stigum bound. Section 5.5 defines the various notions of solvability for the problem of reconstruction on trees and gives our main result for this problem. We first use the cavity method on trees in Section 5.6 and then relate these results to the original problem of community detection in Section 5.7. Finally, we show in Section 5.9 that the community detection problem is in some sense equivalent to the low-rank matrix estimation problem of Chapter 3. This allows to use the results of Chapter 3 to obtain precise reconstruction limits for the stochastic block model.

The community detection problem

We are interested in inferring the labels X from the graph G. To do so, we aim at constructing an estimator (i.e. a function of the observation G) T (G) = (T 1 (G), . . . , T n (G)) ∈ {1, 2} n , such that T i (G) is 'close' to X i . We will measure the performance of T using the 'rescaled average success probability' defined as follows

P suc (T ) = 1 n n i=1 P(T i (G) = 1|X i = 1) + P(T i (G) = 2|X i = 2) -1 . (5.2.1)
The '-1' is here to rescale the success probability and to ensure that P suc = 0 for 'dummy estimators' (i.e. estimators that do not depend on the observed graph G). The optimal test with respect to this metric is

T opt i (G) =    1 if log P(X i =1|G) P(X i =2|G) ≥ log( p 1-p ), 2 otherwise. (5.2.2)
Let s 0 be uniformly chosen among the vertices of G, independently of all other random variables. The maximal achievable rescaled success probability reduces then to

P suc (T opt ) = P(T opt s 0 (G) = 1|X s 0 = 1) + P(T opt s 0 (G) = 2|X s 0 = 2) -1.
We define a notion of solvability for the community detection problem. Definition 5.

2.1

We say that the community detection problem on a given stochastic block model is solvable if lim inf n→∞ P suc (T opt ) > 0.

We have another equivalent characterization for solvability given by the following proposition: Proposition 5.2.1

We have P suc (T opt ) = D TV (P 1 , P 2 ),

where P 1 and P 2 denote the conditional distribution of the graph G, conditionally respectively on X s 0 = 1 and X s 0 = 2, where s 0 is a uniformly chosen random vertex of G. D TV denotes the total variation distance.

Proof. The set of estimators of X s 0 is precisely {1 + 1 A |A measurable set}. Consequently P suc (T opt ) = sup

A measurable P(G ∈ A c |X s 0 = 1) + P(G ∈ A|X s 0 = 2) -1 = sup A measurable P 1 (A c ) + P 2 (A) -1 = sup A measurable P 2 (A) -P 1 (A) = D TV (P 1 , P 2 ).
In our setting (5.1.1), the asymptotic degree distribution of a given vertex is a Poisson random variable. Let i ∈ {1, . . . , n} be a vertex of G, we will denote by d i its degree. Then we have

d i (d) ----→ n→+∞ Poi(d(pa + (1 -p)b)) conditionally on {X i = 1}, d i (d) ----→ n→+∞ Poi(d(pb + (1 -p)c)) conditionally on {X i = 2}.
If the asymptotic average degrees differ from class 1 to class 2, we see easily that the problem is solvable. Proof. Using the definition of solvability in terms of the total variation distance, we have:

lim inf n→∞ D TV (P 1 , P 2 ) = lim inf n→∞ sup event A |P 1 (A) -P 2 (A)| ≥ lim inf n→∞ sup B⊂N |P 1 ({d s 0 ∈ B}) -P 2 ({d s 0 ∈ B})| = D TV (Poi(d(pa + (1 -p)b)), Poi(d(pb + (1 -p)c))) > 0.
In the rest of the paper, we will always assume that (5.1.2) is valid, so that the average degree in the graph is d. Most of our results below will be obtained in the limit where first n tends to infinity and then d tends to infinity but the parameter λ will always remain fixed, as well as the parameter p ∈ [0, 1/2] corresponding to the proportion of nodes in the first community.

We now introduce a variation of the standard community detection problem where a fraction q of the vertices have their labels revealed. This labels correspond to 'sideinformation' given with the graph. More formally, in this setting the label X v of each vertex v ∈ {1, . . . , n} is observed with probability q ∈ [0, 1] independently of everything else and the estimator T (G, q) = (T 1 (G, q), . . . , T n (G, q)) is then a function of the observed graph G and the observed labels. The probability of success is again defined by (5.2.1) where the T i (G)'s are replaced by the T i (G, q)'s and the optimal test T opt (G, q) is then

T opt i (G, q) =    1 if log P(X i =1|G,Y ) P(X i =2|G,Y ) ≥ log( p 1-p ) 2 otherwise,
where Y denotes the (random) set of observed vertices. Note that we have lim inf n→∞ P suc (T opt (G, q)) ≥ q ≥ 0, so that the notion of solvability of Definition 5.2.1 does not make sense in this case where we have some side-information.

We end this section by some technical definitions. In order to state our main results, we need to define the following function F from R ≥0 to R:

F (µ) = λ (1 -p) 2 E 1 p + (1 -p) exp( √ µZ -µ/2) -1 ,
where Z ∼ N (0, 1). Note that F is also a function of the parameters λ and p which are considered as fixed. Definition 5.

(Spinodal curve)

The spinodal curve is defined as the function

λ sp : p → sup λ ≥ 0 | 0 is the unique fixed point of F . (5.2.3) Let us define p * = 1 2 - 1 2 √ 3 . (5.2.4)
The spinodal curve λ sp and p * are represented on Figure 5.2. p * corresponds to the critical value of p below which λ sp goes strictly below the 'Kesten-Stigum' line λ = 1. As we can see on the phase diagram Figure 5.2, a "hard region" appears when p < p * . The following conjecture shows that λ sp is well defined and summarizes its main properties. Conjecture 5.2.1 (i) If λ > 1, then F has two fixed points: 0 and α > 0. Moreover, 0 is unstable and α is stable.

(ii) For p * ≤ p ≤ 1/2, we have λ sp (p) = 1.

(iii) For 0 ≤ p < p * , we have λ sp (p) < 1 and if λ sp (p) < λ < 1, then F has three fixed points: 0 < β < α. Moreover, 0 and α are stable and β is unstable.

The analysis of the function F seems challenging and we were only able to verify Conjecture 5.2.1 numerically. Proof. The exact value of p * follows from the following (non-rigorous) argument. a small µ expansion of the function F gives

F (µ) ≈ λµ + λ 2 (1 -6p(1 -p))µ 2 .
Thus, if 1 -6p(1 -p) > 0 (i.e. if p < p * ), then F is convex in a neighborhood of 0. Thus, when λ < 1, F is likely to have 3 fixed points.

Reconstructability above the Kesten-Stigum bound

We first consider the case λ > In words, we see that if a vanishing fraction of the labels is revealed, then the probability to recover the true label of a typical vertex by the optimal algorithm is 2P(N (α/2, α) > 0) -1. Indeed, we believe that (5.3.1) should be an equality. On Figure 5.3, we give a drawing of this curve as a function of λ for p = 0.25 > p * and on Figure 5.4 for p = 0.005 < p * . Note that at this stage, we only gave an interpretation of the curve for λ > 1. We deal with the case λ < 1 in the next section.

Before that, we give a result which shows that if a vanishing fraction of the labels is revealed then the optimal recovery is achieved by a local algorithm. Similar results in the case where (5.1.2) does not hold have been proved in [START_REF] Mossel | Density evolution in the degree-correlated stochastic block model[END_REF]. In the large degree regime, our result improves Proposition 3 in [START_REF] Varun Kanade | Global and local information in clustering labeled block models[END_REF] which deals only with the case p = 0.5 and λ larger than a large constant C. The fact that local algorithms are very efficient as soon as q > 0 (even optimal in the limit q → 0) leads to linear time algorithms for community detection (when some labels are revealed). Indeed from a practical perspective, we believe that our analysis carries over to the labeled stochastic block model [START_REF] Heimlicher | Community detection in the labelled stochastic block model[END_REF][START_REF] Lelarge | Reconstruction in the labelled stochastic block model[END_REF]. It is then possible to devise new clustering algorithms based on a similarity graph which are shown to be optimal for a wide range of models [START_REF] Saade | Clustering from sparse pairwise measurements[END_REF] and also local semi-supervised learning clustering algorithms, see [START_REF] Saade | Fast randomized semi-supervised clustering[END_REF] for more details in this direction.

We now define local algorithms. For an integer t, a test T (G, q) = (T 1 (G, q), . . . , T n (G, q)) is t-local if each T i (G, q) is a function of the graph B t (G, i) induced by the vertices of G whose distance from i is at most t. We denote by Loc t the set of t-local tests and by Loc = ∪ t≥0 Loc t the set of local tests.

Proposition 5.3.2

If λ > 1, then we have for all 0 < q ≤ 1, sup

T ∈Loc lim d→∞ lim n→∞ P suc (T (G, q)) ≥ 2P(N (α/2, α) > 0) -1,
where α > 0 is the stable fixed point in Conjecture 5.2.1 (i).

Proposition 5.3.2 will follow from Corollary 5.7.2. Note in particular that as a vanishing fraction of labels is revealed, i.e. q → 0, the best local algorithm performs at least as well as the optimal algorithm with no revealed labels. An explicit description of an optimal local test (obtained using 'belief-propagation') is given in the proof of Corollary 5.7.2.

Non reconstructability below the spinodal curve

We now state our second main result which states that reconstruction is impossible below the spinodal curve. . Moreover thanks to [START_REF] Bordenave | Non-backtracking spectrum of random graphs: community detection and non-regular ramanujan graphs[END_REF], the Kesten-Stigum threshold is an upper bound on the solvability threshold so that we have λ c (p) ≤ 1. For p ≥ p * defined in (5.2.4), the spinodal curve is equal to the Kesten-Stigum threshold by Conjecture 5.2.1 (ii), so that we have in this case λ c (p) = 1 and moreover as soon as the community detection problem is solvable, it is solvable in polynomial time thanks to the results in [START_REF] Bordenave | Non-backtracking spectrum of random graphs: community detection and non-regular ramanujan graphs[END_REF]. Figure 5.3 is valid for λ < 1. However for p < p * , there is a gap between the spinodal curve and Kesten-Stigum threshold and we conjecture that λ sp (p) < λ c (p) < 1, see Figure 5.2. In the case of dense graphs (where the average degree d is of order n), the value of λ c (p) has been computed in the recent works [START_REF] Krzakala | Mutual information in rankone matrix estimation[END_REF] and [START_REF] Barbier | Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula[END_REF]. We conjecture that their expression (used in Figure 5.2) is still valid for sparse graphs in the large degree regime. Our next result states that it is possible to reconstruct the communities below the spinodal curve, given a sufficient amount of side-information. Consider the case where p < p * and λ sp (p) < λ < 1. As soon as q > βp(1-p) λ , we have

lim d→∞ lim inf n→∞ P suc (T opt (G, q)) ≥ sup T ∈Loc lim d→∞ lim inf n→∞ P suc (T (G, q)) ≥ 2P(N (α/2, α) > 0) -1,
where 0 < β < α are the fixed points defined in Conjecture 5.2.1 (iii). Proposition 5.4.2 will follow from Corollary 5.7.2. In the regime of Proposition 5.4.2 (λ sp (p) < λ < 1 and q > βp(1-p) λ ), we believe that local algorithms are indeed optimal. Figure 5.4 illustrates the case p < p * with p = 0.05 for which we have λ sp (0.05) ≈ 0.58. Also, if the number of revealed entries is sufficiently high (i.e. above the red curve) then local algorithms provide a great improvement in the probability of successfully recovering the label of a typical vertex (the black curve). A description of a local algorithm (beliefpropagation) achieving the lower bound in Proposition 5.4.2 is provided in the proof.

Reconstruction on trees

We will first concentrate on the reconstruction of the labels on trees. The tree structure makes the analysis simpler and allows to deduce results for the stochastic block model, because the SBM is asymptotically locally tree-like. In this section we are going to state the analogous of the well known (see for instance [START_REF] Van | Random graphs and complex networks[END_REF]) local convergence of the Erdős-Rényi random graph towards the Galton-Watson branching process, in terms of labeled graphs. The labeled stochastic block model (G, X) will converge locally towards a random labeled tree. We have to introduce first the notion of pointed labeled graphs.

Definition 5.5.1 (Pointed labeled graphs)

• A pointed labeled graph is a triple G = (g, s 0 , x) where g is a countable, locally finite and connected graph, s 0 is a distinguished vertex of g called the root of the graph and x = (x s ) s∈Vg ∈ {1, 2} Vg are the labels of the vertices.

• Two pointed labeled graphs are equivalent if there exists a graph isomorphism between them, that preserves the root and the labels.

• We define, for r ∈ N, [G] r , the ball of radius r of G, as the pointed labeled graph induced by the root of G and all the vertices at distance at most r from the root.

The randomly rooted stochastic block model (G, s 0 , X) with parameters (M, n, p) is therefore a random pointed labeled graph, that we will denote SBM n from now. We will also be interested in a second family of random pointed labeled graphs, that will correspond to the local limits of stochastic block models. Definition 5.

(Labeled Poisson Galton-Watson branching process)

Let A = δ 1 -δ 1 -δ δ (where δ, δ ∈ [0, 1]
) be a transition matrix. The labeled Poisson Galton-Watson branching process with parameters (A, p, d) is a random pointed labeled graph (T, s 0 , X), where

• (T, s 0 ) is a Galton-Watson tree with offspring distribution Poi(d) rooted at s 0 .
• The labels X of the vertices of T are then chosen as follows:

1. The label of the root X s 0 ∈ {1, 2} is chosen accordingly to P(X s 0 = 1) = p.

2. Given the label X p of the parent p of a node s, the probability that X s = i ∈ {1, 2} is equal to A Xp,i independently from all other random variables.

In the following, we will denote GW = (T, s 0 , X), the labeled Galton-Watson branching process with parameters (R, d, p) with R defined by (5.1.3). The next well known result states that SBM n converges locally toward GW.

Theorem 5.5.1

Let f be a (positive or bounded) function of pointed labeled graphs, that depends only on the ball of radius r. Then

E f (SBM n ) ---→ n→∞ E f (GW) .
A main ingredient for our proof will therefore be the analysis of this well-studied problem of reconstruction on trees [START_REF] Evans | Broadcasting on trees and the ising model[END_REF][START_REF] Mossel | Information flow on trees[END_REF][START_REF] Mossel | Survey-information flow on trees[END_REF]. In the rest of this section, we define the reconstruction problem on trees and give the required results.

We consider here GW = (T, s 0 , X) the labeled Poisson Galton-Watson branching process with parameters (R, p, d). We denote

L n = {v ∈ V T | d(s 0 , v) = n}
, the set of vertices at distance n from the root. We define then

X (n) = (X s ) s∈Ln and c (n) = (c (n) 1 , c (n) 2 ) = (#{s ∈ L n | X s = i}) i=1,2 .
We also define a random subset E n of the nodes at depth n as follows: let q ∈ [0, 1] and for n ∈ N, let E n be the random subset of L n obtained by including in E n each vertex s ∈ L n independently with probability q.

We have three kinds of reconstruction problems.

Definition 5.5.3 (Solvability, q-solvability and census solvability)

We say that the reconstruction problem is solvable if

lim inf n→∞ D TV (P (n) 1 , P (n) 2 ) > 0,
where

P (n) i
denotes the conditional distribution of (T, s 0 , X (n) ) given X s 0 = i. One defines analogously q-solvability (respectively census solvability) by replacing

P (n) i by P (n,q) i (respectively P (n) i ), the conditional distribution of (T, s 0 , (X s ) s∈En ) (respectively (T, s 0 , c (n) )) given X s 0 = i.
Solvability corresponds thus to the special case q = 1. Obviously, census solvability and q-solvability imply solvability, but we will see that solvability does not always imply census solvability.

Similarly to the stochastic block model case, this characterization of solvability in terms of total variation can be rewritten in terms of the maximal achievable success probability for the estimation of X s 0 , given X (n) (or c (n) ). We define the rescaled success probability of an estimator T as

P suc (T ) = P(T (X (n) ) = 1|X s 0 = 1) + P(T (X (n) ) = 2|X s 0 = 2) -1.
The maximal rescaled success probability is then defined as ∆ n = sup T P suc (T ) where the supremum is taken over all measurable function of X (n) . Even though we defined these quantities for the solvability problem, these definitions and the following result can be straightforwardly extended to q-solvability and census-solvability. The following lemma is the analog of Proposition 5.2.1. Lemma 5.5.1

∆ n = D TV (P (n) 1 , P (n) 2 ).
We recall here the census-solvability criterion for our particular case (which is a straightforward extension of the results presented in [START_REF] Mossel | Survey-information flow on trees[END_REF]).

Theorem 5.5.2

We consider the Poisson Galton-Watson branching process with parameter (R, p, d). If λ > 1, then the problem is census-solvable and q-solvable for all 0 < q ≤ 1. If λ < 1, then the problem is not census-solvable.

In the large d limit, we are able to get more quantitative results. We define

P opt = lim d→∞ lim n→∞ D TV (P (n) 1 , P (n) 2 ), P (q) opt = lim d→∞ lim n→∞ D TV (P (n,q) 1 , P (n,q) 2
).

Proposition 5.5.1

We consider the Poisson Galton-Watson branching process with parameter (R, p, d).

P opt =    2P(N (α/2, α) > 0) -1 > 0 if λ > λ sp (p), 0 if λ < λ sp (p),
where α is the stable fixed point in Conjecture 5.2.1. If λ sp < λ < 1, then we have

P (q) opt =    2P(N (α/2, α) > 0) -1 > 0 if q > βp(1-p) λ , 0 if q < βp(1-p) λ ,
where α and β are the fixed points defined in Conjecture 5.2.1 (iii).

In particular, Proposition 5.5.1 shows that the spinodal curve is the solvability threshold for the reconstruction on trees. Proposition 5.5.1 is proved at the end of the next section.

Cavity method on trees

To compute the optimal success probability ∆ n = D TV (P

(n) 1 , P (n)
2 ) for the reconstruction problem on trees, we need to study the behaviour of the optimal estimator. This estimator is computed, similarly to (5.2.2), using the marginal distributions of the labels. We aim therefore at computing these marginals.

Our approach here is closely related to the one of [START_REF] Montanari | Finding one community in a sparse graph[END_REF] which studies the problem of finding one single community. We establish rigorously the 'cavity equations', a recursive method to compute marginals, that originates from statistical physics.

We consider here the labeled branching process GW = (T, s 0 , X) with parameter (R, p, d). In order to obtain quantitative results, we will be interested in the asymptotic of large degrees d → ∞ while λ remains fixed. We also define = 1 -b = λ d . We have then

a = 1 + 1 -p p , b = 1 - and c = 1 + p 1 -p .
(5.6.1)

The cavity recursions

Let r ∈ N. For a given vertex s of GW = (T, s 0 , X), we note T s the subtree induced by s and its progeny. With a slight abuse of notation, we write p

(x) = p 1 x=1 (1 -p) 1 x=2 . We define also ψ(x, y) = a 1 y=x=1 b 1 y =x c 1 y=x=2
. We now introduce the belief-propagation algorithm for label reconstruction. This algorithm computes 'messages', that approximate the marginal distributions of the labels. The message of the vertex s ∈ [T ] r is defined as the following function from {1, 2} → R:

ν s r : x s →          p(x s )1 xs=Xs if s ∈ E r , p(x s ) if s ∈ L r \ E r , p(x s ) s→v xv ψ(x s , x v )ν v r (x v ) if s / ∈ L r .
(5.6.2)

Lemma 5.6.1 For all s ∈ [T ] r \ L r , ν s r (x s ) ∝ p(x s ) (xv ) v∈Ts∩[T ]r \{s} (xv) v∈Er =(Xv) (i→j)∈[T ]r∩Ts R x i ,x j , (5.6.3)
where ∝ means equality up to a multiplicative constant that is independent of x s .

Proof. We show this lemma by induction on the depth of

d s def = T s ∩ [T ] r . Since s ∈ L r , we have d s ≥ 1.
We consider the following recursion hypothesis:

H(d) : for all s ∈ [T ] r such that d s ≤ d (5.6.3) holds.
H(1) follows from the definition of ν s r . Suppose now that H(d) holds for some

d ≥ 1. Let s ∈ [T ] r such that d s = d + 1. Then, by induction ν s r (x s ) = p(x s ) (xv)s→v s→v ψ(x s , x v )ν v r (x v ) = p(x s ) (xv)s→v s→v ψ(x s , x v )p(x v ) (xu) u∈Tv ∩[T ]r \{v} (xu) u∈Er =(Xu) (i→j)∈[T ]r∩Tv R x i ,x j = p(x s ) (xv)s→v s→v R xs,xv (xu) u∈Tv ∩[T ]r \{v} (xu) u∈Er =(Xu) (i→j)∈[T ]r∩Tv R x i ,x j = p(x s ) (xv ) v∈Ts∩[T ]r \{s} (xv) v∈Er =(Xv) (i→j)∈[T ]r∩Ts R x i ,x j ,
which proves that H(d + 1) holds.

The next lemma states the well known fact that belief-propagation computes the exact marginals on trees. Lemma 5.6.2

ν s 0 r (x s 0 ) = P(X s 0 = x s 0 |T, E r , (X v ) v∈Er ).
Proof. The structure of T outside of [T ] r does not provide any information about the labels of the vertices, thus, using Lemma 5.6.1

P X s 0 = x s 0 | T, E r , (X s ) s∈Er = P X s 0 = x s 0 | [T ] r , E r , (X s ) s∈Er = p(x s 0 ) (xs) s∈[T ]r \{s 0 } (xs) s∈Er =(Xs) (i→j)∈[T ]r R x i ,x j = ν s 0 r (x s 0 ).

If we write ξ

s r = log( ν s r (1) ν s r (2)
), the recursive definition (5.6.2) of the messages gives for r ≥ 1,

ξ s 0 r = h + s,s 0 →s f (ξ s r ), (5.6.4)
where h = log( p 1-p ) and f : x → log ae x +b be x +c .

Definition 5.6.1

We define P r as the law of ξ s 0 r = log P(Xs 0 =1|T,Er,(Xs) s∈Er ) P(Xs 0 =2|T,Er,(Xs) s∈Er ) . We denote P (1) r and P (2) r the laws of ξ s 0 r respectively conditionally on {X s 0 = 1} and {X s 0 = 2}.

For r ≥ 1. Let us work conditionally on {X s 0 = 1}. By the branching property of GW, conditionally on {s 1 , . . . , s L } being the children of s 0 , the random variables ξ s 1 r , . . . , ξ s L r are independent, identically distributed according to

paP (1) r-1 + (1 -p)bP (2) r-1 . Indeed, conditionally on {X s 0 = 1}, s 1 , . . . , s L i.i.d.
∼ paδ 1 + (1 -p)bδ 2 . Equation (5.6.4) leads therefore to the following distributional recursions (the second one is obtained by the same arguments):

Proposition 5.6.1 (Cavity equations)

For all r ≥ 1,

ξ (1) r (d) = h + L 1,1 i=1 f (ξ (1) r-1,i ) + L 1,2 i=1 f (ξ (2) r-1,i ),
(5.6.5)

where ξ r ∼ P (1) r , L 1,1 ∼ Poi(pad), L 1,2 ∼ Poi((1 -p)bd), ξ (1) r-1,i ∼ P (1) r-1 , ξ (2) r-1,i ∼ P (2)
r-1 , and all these variables are independent.

ξ (2) r (d) = h + L 2,1 i=1 f (ξ (1) r-1,i ) + L 2,2 i=1 f (ξ (2) r-1,i ), (5.6.6)
where ξ r ∼ P

(2) r , L 2,1 ∼ Poi(pbd), L 2,2 ∼ Poi((1 -p)cd), ξ (1)
r-1,i ∼ P

(1)

r-1 , ξ (2) r-1,i ∼ P (2)
r-1 , and all these variables are independent.

Gaussian limit

We are interested in the limit distributions of P (1) r and P (2) r , because they encode the marginal distributions of the labels and thus allow to derive the optimal reconstruction performances. We study the recursions (5.6.5) and (5.6.6). More precisely, we show that P (1) r and P (2) r converge in Wasserstein sense toward Gaussian distributions when d → ∞ and λ remains fixed. Proposition 5.6.2 For all r ≥ 1,

P (1) r W 2 ----→ d→+∞ N (h + µ r 2 , µ r ), P (2) r W 2 ----→ d→+∞ N (h - µ r 2 , µ r ),
where

W 2
--→ denote the convergence with respect to W 2 , the Wasserstein distance of order 2, and (µ r ) r≥1 is defined by

   µ 1 = qλ p(1-p) µ k+1 = F (µ k ).
(5.6.7) Proposition 5.6.2 is proved in Section 5.8. Proposition 5.5.1 is now a consequence of Conjecture 5.2.1 and the following corollary.

Corollary 5.6.1

For all 0 ≤ q ≤ 1,

lim r→∞ lim d→∞ D TV (P (r,q) 1 , P (r,q) 2 ) = 2P(N (µ ∞ /2, µ ∞ ) > 0) -1,
where µ ∞ is the limit of (µ r ) defined in (5.6.7).

Proof. The optimal test according the performance measure P suc is

T (GW) r (T, E r , (X s ) s∈Er ) = 1 if ξ r ≥ log p 1-p , 2 otherwise.
(5.6.8)

Analogously to Lemma 5.5.1, we have

D TV (P (r,q) 1 , P (r,q) 2 ) = P suc (T (GW) r ) = P ξ (1) r ≥ log p 1 -p + P ξ (2) r < log p 1 -p -1 ---→ d→∞ 2P(N (µ r /2, µ r ) > 0) -1 ---→ r→∞ 2P(N (µ ∞ /2, µ ∞ ) > 0) -1,
where we have used Proposition 5.6.2.

Proofs for the stochastic block model

In this section, we apply the results that we obtained for the reconstruction on the branching process to derive bounds for the community detection problem on the stochastic block model. Consider the case, where a fraction 0 ≤ q ≤ 1 is revealed: one observes the graph G and additionally each label X v with probability q, for 1 ≤ v ≤ n, independently of everything else. Let us denote

E G = {1 ≤ v ≤ n | X v is revealed}.
Let s 0 be uniformly chosen among the vertices of G. For r ≥ 0 we define (analogously to the case of the branching process)

E G,r = ∂[G] r ∩ E G , i.e
. the vertices at the boundary of the ball of center s 0 and radius r, whose label has been revealed. Define

ξ G,r = log P(X s 0 = 1|[G] r , (X s ) s∈E G,r ) P(X s 0 = 2|[G] r , (X s ) s∈E G,r )
.

(5.7.1)

Let us denote P

(1)

G,r and P

(2)

G,r the laws of ξ G,r conditionally respectively on X s 0 = 1 and X s 0 = 2.

Proposition 5.7.1

For i = 1, 2;

P (i) G,r ---→ n→∞ P (i) r ,
where P (i) r is defined in Definition 5.6.1.

Proposition 5.7.1 is a consequence of the local convergence of the stochastic block model towards a branching process (Theorem 5.5.1).

Proof. Let us define

ϕ n : ( G, s 0 , X, E r ) → log P(X s 0 = 1|[G] r = [ G] r , (X s ) s∈E G,r = ( X s ) s∈ Er , E G,r = E r ) P(X s 0 = 2|[G] r = [ G] r , (X s ) s∈E G,r = ( X s ) s∈ Er , E G,r = E r )
and

ϕ ∞ : ( G, s 0 , X, E r ) →      log P(Xs 0 =1|[T ]r=[ G]r,(Xs) s∈Er =( Xs)
s∈ Er

,Er= Er)

P(Xs 0 =2|[T ]r=[ G]r,(Xs) s∈Er =( Xs) s∈ Er ,Er= Er) if [ G] r is a tree, 0 otherwise.
Let ( G, s 0 , X) be a fixed pointed labeled graph such that [ G] r is a tree. Let E r be a subset of the vertices in ∂[ G] r . A straightforward extension of Theorem 5.5.1 gives us

ϕ n ( G, s 0 , X, E r ) ---→ n→∞ ϕ ∞ ( G, s 0 , X, E r ). (5.7.2)
Another consequence of the local convergence of (SBM n ) toward GW is that one can couple (SBM n , E G,r ) n and (GW, E r ) on a probability space such that there exists n 0 ∈ N such that

∀n ≥ n 0 , [SBM n , E G,r ] r = [GW, E r ] r . Let n ≥ n 0 ϕ n (SBM n , E G,r ) = ϕ n (GW, E r ) ---→ n→∞ ϕ ∞ (GW, E r ).
On this probability space ϕ n (SBM n , E G,r ) converges almost surely to ϕ ∞ (GW, E r ), hence the convergence of the conditional distributions.

Define the local test

T loc r (G, X) =    1 if ξ G,r ≥ log p 1-p , 2 otherwise.
(5.7.3)

T loc r
is the optimal r-local test, with side information (X s ) s∈E G,r . Note that ξ G,r (and thus T loc r ) is computed by the belief propagation algorithm. Using the results on the branching process, we are now able to fully characterize the performance of T loc . Proposition 5.7.2

For all 0 ≤ q ≤ 1, lim r→∞ lim d→∞ lim n→∞ P suc (T loc r ) = 2P(N (µ ∞ /2, µ ∞ ) > 0) -1,
where µ ∞ is the limit of the sequence defined by (5.6.7).

Proof. Using Propositions 5.7.1 and 5.6.2

     P (1) G,r (d) ---→ n→∞ P (1) r (d) ---→ d→∞ N (h + µr 2 , µ r ), P (2) G,r (d) ---→ n→∞ P (2) r (d) ---→ d→∞ N (h -µr 2 , µ r ),
where µ r is defined by the recursion (5.6.7). Therefore (recall that h = log p 1-p )

P suc (T loc r ) = P(ξ G,r ≥ h|X s 0 = 1) + P(ξ G,r < h|X s 0 = 2) -1 ---→ n→∞ P (1) G,r (ξ G,r ≥ h) + P (2) G,r (ξ G,r < h) -1 ---→ d→∞ 2P(N (µ r /2, µ r ) > 0) -1 ---→ r→∞ 2P(N (µ ∞ /2, µ ∞ ) > 0) -1.

Upper bound

We can now deduce an upper bound on the optimal performance for the community detection problem.

Corollary 5.7.1

We have:

lim sup d→∞ lim sup n→∞ P suc (T opt ) ≤ 2P(N (µ ∞ /2, µ ∞ ) > 0) -1,
where µ ∞ is the limit of the sequence defined by (5.6.7) in the case q = 1.

Proposition 5.4.1 and the first part of Proposition 5.3.1 are then consequences of this corollary:

• when λ < λ sp (p), µ ∞ = 0 and consequently lim d→∞ lim sup n→∞ P suc (T opt ) = 0.

• when λ > 1, then µ ∞ = α, hence the first bound of Proposition 5.3.1.

Proof.

Let r > 0. Let s 0 be uniformly chosen from the vertices of G. We aim at estimating X s 0 from the rooted graph (G, s 0 ). As seen in Section 5.2, the optimal test in terms of rescaled success probability P suc is

T opt (G) =    1 if log( P(Xs 0 =1|G) P(Xs 0 =2|G) ) ≥ log( p 1-p ), 2 otherwise.
We are going to analyze the oracle

T * r (G, X) =    1 if log P(Xs 0 =1|G,(Xs) s∈∂[G]r ) P(Xs 0 =2|G,(Xs) s∈∂[G]r ) ≥ log p 1-p , 2 otherwise. Obviously, P suc (T opt ) ≤ P suc (T * r ). The oracle T * r uses extra information (X s ) s∈∂[G]
r but is a local test, i.e. a test that only depends on the ball of radius r:

ξ * G,r def = log P(X s 0 = 1|G, (X s ) s∈∂[G]r ) P(X s 0 = 2|G, (X s ) s∈∂[G]r ) = log P(X s 0 = 1|[G] r , (X s ) s∈∂[G]r ) P(X s 0 = 2|[G] r , (X s ) s∈∂[G]r )
. ξ * G,r is thus equal to ξ G,r from equation (5.7.1), when q = 1. We conclude using Proposition 5.7.2 and the fact P suc (T opt ) ≤ P suc (T * r ).

Lower bound

We now establish a lower bound for estimation when a fraction q of the labels is revealed.

Corollary 5.7.2

For all 0 ≤ q ≤ 1, lim inf

d→∞ lim inf n→∞ P suc (T opt (G, q)) ≥ 2P(N (µ ∞ /2, µ ∞ ) > 0) -1, (5.7.4)
where µ ∞ is the limit of the sequence defined by (5.6.7).

Proof. Here, we are going to bound P suc (T opt (G, q)) from below by the performance of the local test T loc r (that corresponds to the estimator derived from belief-propagation) defined by (5.7.3). Obviously, P suc (T opt (G, q)) ≥ P suc (T loc r ) and Proposition 5.7.2 gives then the result.

The second part of Proposition 5.3.1 follows from this corollary. Indeed, when λ > 1 and q > 0, µ ∞ = α > 0.

Corollary 5.7.2 leads also to proposition 5.4.2: when q > βp(1-p) λ , then µ 1 > β and thus

(µ k ) converges to the fixed point α > β of G: µ ∞ = α > 0.
We deduce also Proposition 5.3.2 from the proof of Corollary 5.7.2. Indeed, we will see that the lower bound in (5.7.4) is achieved by a local test.

Proof of Proposition 5.6.2

We prove Proposition 5.6.2 by induction over r ≥ 1. Proposition 5.6.2 follows from Lemmas 5.8.1 and 5.8.3 below.

Initialization

First of all, we are going to show that ξ

ξ (1) 1 W 2 ----→ d→+∞ N (h + µ 1 2 , µ 1 ), ξ (2) 1 W 2 ----→ d→+∞ N (h - µ 1 2 , µ 1 ).
Proof. We will only prove the convergence for ξ (1) 1 , the convergence for ξ

(2) 1 can be obtained analogously. We have

ξ (1) 0 =

+∞ with probability q h

with probability 1 -q, and ξ

(2) 0 = -∞ with probability q h with probability 1 -q.

Therefore, the recursion (5.6.5) gives

ξ (1) 1 (d) = h + L log a b + L log b c ,
where L ∼ Poi(padq) and L ∼ Poi((1 -p)bdq) are independent. By isolating the means

ξ (1) 1 (d) = h + (L -padq) log a b + (L -(1 -q)bdp) log b c + dq pa log a b + (1 -p)b log b c . (5.8.1) In the d → ∞ limit, log( a b ) = p + 1 2 2 (1-( 1-p p ) 2 )+o( 2 ) and log( b c ) = -1-p -1 2 2 (1-( p 1-p ) 2
)+o( 2), so the last term in (5.8.1) becomes

dq pa log a b + (1 -p)b log b c = qλ 1 p + 2p -1 2p(1 -p) + o(1) = qλ 2p(1 -p) + o(1).
Now, to deal with the remaining terms in (5.8.1) we are going to use the following corollary the Central Limit Theorem. Lemma 5.8.2

Let (a n ) n ∈ (0, +∞) N , such that a n ---→ n→∞ +∞. Let X n be a sequence of random variable such that X n ∼ Poi(a n ). Then 1 √ a n (X n -a n ) W 2 ---→ n→∞ N (0, 1). Proof. We define v n def = an an → 1. Let (Y (n) i ) i.i.d. ∼ Poi(v n ) -v n . By the central limit theorem we have 1 √ a n (X n -a n ) (d) = 1 √ a n an i=1 Y (n) i W 2 ---→ n→∞ N (0, 1).
Applying this result to the terms in (5.8.1), we obtain (L -padq) log( a b )

W 2 ---→ d→∞ N (0, λq p ) and (L -(1 -p)bdq) log( b c ) W 2 ---→ d→∞ N (0, λq 1-p ) and finally ξ (1) 1 W 2 ---→ d→∞ N (h + µ 1 2 , µ 1 ).

Induction

The following lemma (combined with Lemma 5.8.1) concludes the proof of Proposition 5.6.2 by induction.

Lemma 5.8.3

Suppose that, for a fixed r ≥ 1 we have

ξ (1) r W 2 ----→ d→+∞ N (h + µ r 2 , µ r ), ξ (2) r W 2 ----→ d→+∞ N (h - µ r 2 , µ r ).
Then

ξ (1) r+1 W 2 ----→ d→+∞ N (h + µ r+1 2 , µ r+1 ), ξ (2) r+1 W 2 ----→ d→+∞ N (h - µ r+1 2 , µ r+1 ),
where

µ r+1 = F (µ r ).
Proof. We first compute the limits of the mean and the variance of ξ (1,2) r+1 . To do so, we will need the following well known Wald formulas.

Lemma 5.8.4 (Wald formulas)

Let X 1 , . . . , X n be i.i.d. integrable real random variables, and T a N valued integrable random variable, independent from the X i . Then T i=1 X i is integrable and

E T i=1 X i = E[T ]E[X].
Moreover, if the variables (X i ) are square integrable and centered Var T i=1

X i = E[T ]E X 2 .
Using Wald formulas above and Proposition 5.6.1, we obtain

Eξ (1) r+1 = h + pad Ef (ξ (1) r ) + (1 -p)bd Ef (ξ (2) r ), (5.8.2)

Eξ

(2)

r+1 = h + pbd Ef (ξ (1) r ) + (1 -p)cd Ef (ξ (2) r ).
(5.8.3)

We will now compute an approximation for the function f in the large d limit.

f (ξ) = log( a b e ξ +1 e ξ + c b )
and we have

a b = 1 + 1 p + O( 2 ) and c b = 1 + 1 1-p + O( 2 ). Thus f (ξ) = log 1 + e ξ p(1 + e ξ ) + O( 2 ) -log 1 + 1 (1 -p)(1 + e ξ ) + O( 2 ) .
The Taylor-Lagrange formula ensures that

∀a ∈ [0, 1], ∀x ≥ 0, log(1 + ax) -ax + a 2 x 2 2 ≤ 1 3 x 3 . Therefore f (x) = e x p(1 + e x ) - 2 2p 2 e x 1 + e x 2 - 1 (1 -p)(1 + e x ) + 2 2(1 -p) 2 1 1 + e x 2 + O( 3 ) (5.8.4)
To simplify the computation, we need the following so-called 'Nishimori condition'.

Lemma 5.8.5

For all continuous bounded function g:

E g(ξ (2) r ) = p 1 -p E g(ξ (1) r )e -ξ (1) r .
Proof. This is a consequence of Bayes rule.

P(ξ 2 r ∈ A) = P(ξ r ∈ A|X s 0 = 2) = P(ξ r ∈ A, X s 0 = 2) 1 -p = 1 1 -p E(1(ξ r ∈ A)P(X s 0 = 2|GW, E r , (X s ) s∈Er )) = 1 1 -p E(1(ξ r ∈ A)P(X s 0 = 1|GW, E r , (X s ) s∈Er )e -ξr ) = p 1 -p E(1(ξ r ∈ A)e -ξr |X s 0 = 1).
We use Lemma 5.8.5 to deduce the following identities. Corollary 5.8.1

pE e ξ (1) r 1 + e ξ (1) r + (1 -p)E e ξ (2) r 1 + e ξ (2) r = p, pE   e ξ (1) r 1 + e ξ (1) r   2 + (1 -p)E   e ξ (2) r 1 + e ξ (2) r   2 = pE e ξ (1) r 1 + e ξ (1) r .
Replacing f by its approximation (5.8.4) in equations (5.8.2) and (5.8.3) and applying Corollary 5.8.1, we obtain that Eξ

(1) r+1 ----→ d→+∞ h -1 2 F (µ r ) and Eξ (2) r+1 ----→ d→+∞ h + 1 2 F (µ r ). Similar calculations show that Var(ξ (1) r+1 ) ----→ d→+∞ F (µ r ) and Var(ξ (2) r+1 ) ----→ d→+∞ F (µ r ).
It remains to show that ξ [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF] r+1 is converging toward a Gaussian distribution in the Wasserstein sense. We will need the following lemma. Lemma 5.8.6

For i = 1, 2, √ d E[f (ξ (i) r )] and d E[f (ξ (i) r ) 2 ] are both converging to constants when d → +∞. Proof. √ df (x) = √ d p e x 1+e x - √ d 1-p 1 1+e x + o(1). So √ d E f (ξ (i) r ) = √ d p E   e ξ (i) r 1 + e ξ (i) r   - √ d 1 -p E 1 1 + e ξ (i) r + o(1),
and all the terms in this expression are converging by weak convergence of ξ

(i)
r and the fact that ∼ λ d . The other limit is proved analogously.

Compute now

ξ (1) r+1 -Eξ (1) r+1 = L 1,1 i=1 f (ξ (1) r,i ) -Ef (ξ (1)
r,i ) + L 1,2 i=1 f (ξ (2) r,i ) -Ef (ξ (2)
r,i ) + (L 1,1 -EL 1,1 )Ef (ξ (1) r ) + (L 1,2 -EL 1,2 )Ef (ξ (2) r ) (5.8.5)
We write

X i = f (ξ (1) r,i ) -Ef (ξ (1)
r,i ) and

Y i = f (ξ (2) r,i ) -Ef (ξ (2)
r,i ). Let us decompose the first sum:

L 1,1 i=1 X i = EL 1,1 i=1 X i + L 1,1 i=1 X i - EL 1,1 i=1 X i S .
We first show that S

W 2 ---→ d→∞ δ 0 . Wald identities give us Var(S) = Var   |L 1,1 -EL 1,1 | i=1 X i   = E |L 1,1 -EL 1,1 | E[X 2 1 ] = 1 d E |L 1,1 -EL 1,1 | ---→ d→∞ 0 bVar f (ξ (0) r ) O(1) ---→ d→∞ 0.
And therefore

S W 2 ---→ d→∞ δ 0 because ES = 0.
Next, we apply the Central Limit Theorem to the sum

EL 1,1 i=1 X i = 1 (1 -p)a (1-p)a i=1 (1 -p)aX i i.i.d., bounded random variables .
We obtain that the sum converges with respect to the Wasserstein metric to a normal distribution.

The two first sums in (5.8.5) are independent and converge to Gaussian distributions in the Wasserstein sense. It remains to show that the last two terms are converging toward Gaussian distributions. This is indeed the case because

(L 1,1 -EL 1,1 )Ef (ξ (0) r ) = 1 √ d (L 1,1 -EL 1,1 ) √ dEf (ξ (0) r ). 1 √ d (L 1,1 -EL 1,1
) converges toward a Gaussian distribution and

√ dEf (ξ (0)
r ) converges (to a constant) as d → ∞. The last term is treated the same way. ξ

(1)

r+1 is therefore converging toward a Gaussian distribution in the Wasserstein sense. The mean and the variance of this Gaussian distribution are necessarily equal the limits of the means and the variance of ξ [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF] r+1 that we computed.

Connection with rank-one matrix estimation

We adopt now a completely different point of view from the previous sections, in order to relate the community detection problem to the rank-one matrix estimation problem of Chapter 3. Rougthly, we will show now that the graph G contains, in some sense, "as much information" about the classes X than the matrix Y given by

Y i,j = λ n X i X j + Z i,j for 1 ≤ i < j ≤ n where Z i,j i.i.d.
∼ N (0, 1) independently of everything else and

X i = φ p (X i ) where φ p (1) = 1-p p and φ p (2) = -p 1-p .
In the case of two symmetric communities (i.e. p = 1/2) this was done in [START_REF] Deshpande | Asymptotic mutual information for the balanced binary stochastic block model[END_REF] who showed that the mutual information of the two models where approximately equal:

1 n I(X; G) 1 n I( X; Y ).
Following the proof strategy of [START_REF] Deshpande | Asymptotic mutual information for the balanced binary stochastic block model[END_REF] we generalize this to the asymmetric case: Theorem 5.9.1 There exists a constant C > 0 such that, for d large enough

lim sup n→∞ 1 n I(X; G) -I( X, Y ) ≤ Cd -1/2 .
Theorem 3.2.1 that we will see in Chapter 3 allows us to compute the limit of 1 n I( X, Y ). Define

F(λ, q) = - λq 2 4 + E log p exp( 1 -p p ( λqZ 0 + λq X 0 ) - λ(1 -p) 2p q) + (1 -p) exp(- p 1 -p ( λqZ 0 + λq X 0 ) - λp 2(1 -p) q) (5.9.1)
where the expectation is taken over Z 0 ∼ N (0, 1) and X 0 ∼ pδ φp( 1) + (1 -p)δ φp(2) independently from Z 0 . Define also

I : λ → λ 4 -sup q≥0 F g (λ, q).
(5.9.2) Corollary 3.2.1 gives then

Corollary 5.9.1

There exists a constant C > 0 such that, for d large enough

lim sup n→∞ 1 n I(X; G) -I(λ) ≤ Cd -1/2 .
From Corollary 5.9.1 one can deduce the precise threshold for reconstruction in the Stochastic Block Model (when d → ∞).

Definition 5.9.1 (Estimator)

An estimator of the labels X is a function x : G → {1, 2} n that could depend on auxiliary randomness (random variables independent of X).

For a labeling x ∈ {1, 2} n and i ∈ {1, 2} we define S i (x) = {k ∈ {1, . . . , n}|x k = i}, i.e. the indices of the nodes that have the label i according to x. We now recall a popular performance measure for estimators.

Definition 5.9.2 (Community Overlap)

For x, y ∈ {1, 2} n we define the community overlap of the configuration x and y as

Overlap(x, y) = 1 n max σ i=1,2 #S i (x) ∩ S σ(i) (y) - 1 n #S i (x)#S σ(i) (y)
where the maximum is taken over the permutations of {1, 2}.

Two configurations have thus a positive community overlap if they are correlated, up to a permutation of the classes. We will then say that the community detection problem is solvable, if there exists an estimator (i.e. an algorithm) that achieves a positive overlap with positive probability.

Definition 5.9.3 (Solvability)

We say that the community detection problem is solvable (in the limit of large degrees) if there exists an estimator x(G) such that

lim inf d→∞ lim inf n→∞ E(Overlap( x(G), X)) > 0.
The next Theorem states that

λ c (p) def = inf λ > 0 I(λ) < λ 4 .
will be the threshold for solvability in the stochastic block model.

Theorem 5.9.2

• If λ > λ c (p), then the community detection problem is solvable.

• If λ < λ c (p), then the community detection problem is not solvable.

5.9.1

The limit of the mutual information: proof of Theorem 5.9.1

We are going to compute I(X; G) and

I( X, Y ). For x ∈ {1, 2} n we denote x = (φ p (x 1 ), . . . , φ p (x n )) ∈ S n p , where φ p (1) = 1-p p , φ p (2) = -p 1-p and S p = -p 1-p , 1-p p .
Recall the notations from (5.6.1):

a = 1 + 1 -p p , b = 1 - and c = 1 + p 1 -p .
(5.9.3)

For x ∈ {1, 2} we define P 0 (x) = P 0 ( x) = p, if x = 1 and P 0 (x) = P 0 ( x) = 1 -p, if x = 2.
For x ∈ {1, 2} n we will write, with a slight abuse of notation

P 0 (x) = P 0 ( x) = n i=1 P 0 (x i ).
By definition of the mutual information, a simple computation gives: Lemma 5.9.1

I( X, Y ) = -E   log x∈S n p P 0 ( x) exp   i<j x i x j λ n Z i,j - λ 2n ( x i x j -X i X j ) 2      . Define V i,j = (G i,j -E(G i,j |X i , X j )) = (G i,j -d n (1 + X i X j )). Lemma 5.9.2 For d large enough, I(X; G) + O(d 2 + n ) = -E log x∈{1,2} n P 0 (x) exp i<j ( x i x j -X i X j )V i,j -2 ( x i x j ) 2 V i,j - λ 2n ( x i x j -X i X j ) 2 .
Proof. By definition, I(X; G) = E log P(X,G) P(X)P(G) = -E log P(G) P(G|X) . Thus

I(X; G) = -E log x∈{1,2} n P 0 (x)P(G|x) P(G|X)
.

Recall that

P(G|x) = i<j M G i,j x i ,x j (1 -M x i ,x j ) 1-G i,j = exp i<j G i,j log M x i ,x j + (1 -G i,j ) log(1 -M x i ,x j ) .
This leads to

I(X; G) = -E log x∈{1,2} n P 0 (x) exp i<j G i,j log M x i ,x j M X i ,X j + (1 -G i,j ) log 1 -M x i ,x j 1 -M X i ,X j .
(5.9.4)

Notice that M x i ,x j = d n (1 + x i x j ). Therefore log( Mx i ,x j M X i ,X j ) = log( 1+ x i x j 1+ X i X j
). By the Taylor-Lagrange inequality, there exist a constant C > 0 such that, for small enough (i.e. for d large enough):

log 1 + x i x j 1 + X i X j -( x i x j -X i X j ) + 1 2 2 (( x i x j ) 2 -( X i X j ) 2 ) ≤ C 3 , (5.9.5) log 1 -M x i ,x j 1 -M X i ,X j + d n ( x i x j -X i X j ) ≤ C d 2 n 2 .
(5.9.6)

By summation and triangle inequality:

A i,j def = G i,j log M x i ,x j M X i ,X j + (1 -G i,j ) log 1 -M x i ,x j 1 -M X i ,X j = ( x i x j -X i X j )G i,j - 1 2 2 (( x i x j ) 2 -( X i X j ) 2 )G i,j -(1 -G i,j )( x i x j -X i X j ) d n + O G i,j 3 + d 2 n 2 ,
because of equations (5.9.5) and (5.9.6). Since

G i,j -V i,j = d n (1 + X i X j ) = d n + λ n X i X j , we get A i,j =( x i x j -X i X j )V i,j - 1 2 (( x i x j ) 2 -( X i X j ) 2 )V i,j + λ n X i X j ( x i x j -X i X j ) -2 (( x i x j ) 2 -( X i X j ) 2 ) d n (1 + X i X j ) -G i,j ( x i x j -X i X j ) d n + O G i,j 3 + d 2 n 2 =( x i x j -X i X j )V i,j - 1 2 (( x i x j ) 2 -( X i X j ) 2 )V i,j + λ n X i X j ( x i x j -X i X j ) - λ 2n (( x i x j ) 2 -( X i X j ) 2 ) -G i,j ( x i x j -X i X j ) d n + O G i,j 3 + d 2 n 2 + n =( x i x j -X i X j )V i,j - 1 2 (( x i x j ) 2 -( X i X j ) 2 )V i,j - λ 2n ( x i x j -X i X j ) 2 -G i,j ( x i x j -X i X j ) d n + O G i,j 3 + d 2 n 2 + n .
Notice that G i,j ∈ {0, 1}, we have therefore, for some constant C > 0,

i<j G i,j ( x i x j -X i X j ) d n ≤ C d n i<j G i,j . Therefore i<j A i,j = ( x i x j -X i X j )V i,j - 1 2 (( x i x j ) 2 -( X i X j ) 2 )V i,j - λ 2n ( x i x j -X i X j ) 2 + ∆ n where |∆ n | ≤ C ( d n + 3 ) i<j G i,j + d 2 + n , for some constant C > 0. Since I(X; G) = -E log x∈{1,2} n P 0 (x)e A i,j , we get E log x∈{1,2} n P 0 (x) exp i<j ( x i x j -X i X j )V i,j -2 (( x i x j ) 2 -( X i X j ) 2 )V i,j - λ 2n ( x i x j -X i X j ) 2 = -I(X, G) + O d n + 3 E i<j G i,j + d 2 + n = -I(X, G) + O(d 2 + n ).

Lindeberg argument

We recall the following Lindeberg generalization theorem (Theorem 2 from [START_REF] Babu | Applications of the lindeberg principle in communications and statistical learning[END_REF]) which is a variant of the generalized "Lindeberg principle" from [START_REF] Chatterjee | A generalization of the lindeberg principle[END_REF].

Theorem 5.9.3 (Lindeberg generalization theorem)

Let (U i ) 1≤i≤n and (V i ) 1≤i≤n be two collection of random variables with independent components and f :

R n → R a C 3 function. Denote a i = |EU i -EV i | and b i = |EU 2 i -EV 2 i |. Then |Ef (U ) -Ef (V )| ≤ n i=1 a i E|∂ i f (U 1:i-1 , 0, V i+1:n )| + b i 2 E|∂ 2 i f (U 1:i-1 , 0, V i+1:n )| + 1 2 E U i 0 |∂ 3 i f (U 1:i-1 , 0, V i+1:n )|(U i -s) 2 ds + 1 2 E V i 0 |∂ 3 i f (U 1:i-1 , 0, V i+1:n )|(V i -s) 2 ds . Define J(X, Z) = -E log x∈S n p P 0 ( x) exp i<j λ n Z i,j ( x i x j -X i X j ) -2 ( x i x j ) 2 - λ 2n ( x i x j -X i X j ) 2 .
We show, using Theorem 5.9.3, that J(X, Z) is close to I(X; G).

Lemma 5.9.3

For d large enough we have

1 n |I(X; G) -J(X, Z)| = O( + d 2 /n).
Proof. We apply here Theorem 5.9.3 conditionally to X to the function

Φ(u) = -log x∈S n p P 0 ( x) exp i<j ( x i x j -X i X j )u i,j -2 ( x i x j ) 2 u i,j - λ 2n ( x i x j -X i X j ) 2 .
Φ is C 3 with bounded derivatives. Notice that I(X; G) = EΦ(V ) and J(X, Z) = EΦ( λ n Z). Let us compute V i,j moments, conditionally to X.

E(V i,j |X) = 0 E(V 2 i,j |X) = 2 Var(G i,j |X) = 2 d n (1 + X i X j )(1 + O( d n )) = λ n + O( n ) = λ n E(Z 2 i,j ) + O( n ) Analogously, E(V 3 i,j |X) = O( n ).
Using the Lindeberg generalization theorem we obtain

E Φ( λ/nZ) -E Φ(V ) ≤ i<j O( /n) = O(n ).

Gaussian interpolation

It remains to show Lemma 5.9.4

I(X; Y ) = J(X, Z) + O(n ).
Proof. We define:

H(x, X, Z, ) = i<j x i x j λ n Z i,j - λ 2n ( x i x j -X i X j ) 2 - 1 2 ( x i x j ) 2 λ n Z i,j , F ( ) = E log x∈{1,2} n P 0 (x) exp(H(x, X, Z, )).
Notice that F (0) = I(X; Y ) and F ( ) = J(X, Z). We are going to control the derivative of F . We note • the expectation with respect to the Gibbs measure: g( x) def = x P 0 ( x)g( x) exp(H(x,X,Z, ))

x P 0 ( x) exp(H(x,X,Z, )) . The derivative of F reads

F ( ) = - 1 2 λ n i<j E Z i,j ( x i x j ) 2 .
Here (

x i x j ) 2 is a continuously differentiable function of Z i,j and ∂ Z i,j ( x i x j ) 2 = O(n -1/2 ).
Using Gaussian integration by parts:

F ( ) = - √ λ 2 √ n i<j E ∂ Z i,j ( x i x j ) 2 = O(1). We conclude |F (0) -F ( )| = O(n ).

From mutual information to solvability: proof of Theorem 5.9.2

We are first going to introduce an estimation metric that will allow us to make the link between the minimum mean squared error for matrix estimation, and the overlap for community detection. Define

MMSE G n (λ) = min θ 2 n(n -1) i<j E X i X j -θi,j (G) 2 = 2 n(n -1) i<j E X i X j -E( X i X j |G) 2 (5.9.7)
where the minimum is taken over all function θ of G. By considering the trivial estimator θ = 0, we see that MMSE G n (λ) ∈ [0, 1]. This estimation metric correspond (up to a vanishing error term) to the derivative of the mutual information between the graph G and the labels X. Proposition 5.9.1 Let λ 0 > 0. There exists a constant C > 0 such that, for all λ ∈ (0, λ 0 ], d ≥ 1 and n ≥ 1

1 n ∂I(X; G) ∂λ - 1 4 MMSE G n (λ) ≤ C d -1/2 + d n + d 1/2 λ -1/2 n + d 3/2 n -2 λ -1/2
(5.9.8)

Proof. We are going to differentiate H(X|G) with respect to λ. To do so we will use a differentiation formula from [START_REF] Deshpande | Asymptotic mutual information for the balanced binary stochastic block model[END_REF] (Lemma 7.1), which was first proved in [START_REF] Méasson | The generalized area theorem and some of its consequences[END_REF]. Let us recall the setting (taken from [START_REF] Deshpande | Asymptotic mutual information for the balanced binary stochastic block model[END_REF]) of this Lemma.

For n an integer, denote by P n the set of unordered pairs in [n] (in particular #P n = n 2 )). We will use e, e 1 , e 2 , . . . to denote elements of P n . For for each e = (i, j) we are given a oneparameter family of discrete noisy channels indexed by θ ∈ J (with J = (a 1 , a 2 ) a non-empty interval), with finite input alphabet X 0 and finite output alphabet Y. Concretely, for any e, we have a transition probability {p e,θ (y|x)} x∈X 0 ,y∈Y , (5.9.9) which is differentiable in θ. We shall omit the subscript θ since it will be clear from the context. We then consider X = (X 1 , X 2 , . . . , X n ) a random vector in X n , and Y = (Y ij ) (i,j)∈Pn a set of observations in Y Pn that are conditionally independent given X. Further Y ij is the noisy observation of X i X j ∈ X 0 through the channel p ij ( • | • ). In formulae, the joint probability density function of X and Y is

p X,Y (x, y) = p X (x) (i,j)∈Pn p ij (y ij |x i x j ) .
(5.9.10)

This obviously include the two-groups stochastic block model as a special case. In that case Y = G is just the adjacency matrix of the graph. In the following we write Y -e = (Y e ) e ∈Pn\e for the set of observations excluded e, and X e = X i X j for e = (i, j).

Lemma 5.9.5

With the above notation, we have: (5.9.11)

∂H(X|Y ) ∂θ = e∈Pn xe,
We apply Lemma 5.9.5 to the stochastic block model. Let λ 0 > 0 and λ ∈ (0, λ 0 ]. Instead of having G i,j ∈ {0, 1}, it will be more convenient to consider G i,j ∈ {-1, 1}: G i,j = 1 if i ∼ j, G i,j = -1 else. Notice that neither the mutual information nor MMSE G n are affected by this change. G i,j is, conditionally to X i X j , independent of any other random variable, and distributed as follows

P(G i,j = 1| X i X j ) = d n (1 + λ d X i X j )
The transition probability from equation (5.9.9) is then p λ (g i,j

| x i x j ) = 1-g i,j 2 +g i,j d n (1+ x i x j λ d ). Thus ∂ ∂λ p λ (g i,j | x i x j ) = 1 2n g i,j x i x j d λ .
Lemma 5.9.5 gives

∂H(X|G) ∂λ = 1 2n d λ i<j x i , x j g i,j g i,j x i x j E p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) log x i , x j p λ (g i,j | x i x j )p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) A i,j - 1 2n d λ i<j x i , x j g i,j g i,j x i x j E p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) log p λ (g i,j | x i x j ) B i,j
(5.9.12)

Compute B i,j = x i , x j g i,j g i,j x i x j p( x i , x j ) log p λ (g i,j | x i x j ) = p 2 1 -p p log( ad n ) -log(1 - ad n ) -2p(1 -p) log( bd n ) -log(1 - bd n ) + (1 -p) 2 p 1 -p log( cd n ) -log(1 - cd n ) = p(1 -p) log ac b 2 + p(1 -p)d(2 b n - a n - c n ) + O( d 2 n 2 ) = p(1 -p) 1 -p p + p 1 -p + 2 + O( 2 ) + O( d 2 n 2 ) = + O( 2 ) + O( d 2 n 2 )
and

A i,j = g i,j g i,j E E( X i X j |G -(i,j) ) log x i , x j p λ (g i,j | x i x j )p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) = E E( X i X j |G -(i,j) ) log x i , x j p λ (1| x i x j )p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) x i , x j p λ (-1| x i x j )p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) . Define âi,j = E( X i X j |G -(i,j) ).
A i,j = E âi,j log

x i , x j d n (1 + x i x j λ d )p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) x i , x j (1 -d n (1 + x i x j λ d ))p ( X i , X j )|G -(i,j) ( x i , x j |G -(i,j) ) = E âi,j log d n (1 + âi,j λ d ) 1 -d n (1 + âi,j λ d ) = E âi,j log d n + âi,j + O( 2 ) + O( d n ) . Eâ i,j = E X i X j = (E X 1 ) 2 = 0 therefore A i,j = Eâ 2 i,j + O( 2 + d n )
. By replacing A i,j and B i,j in (5.9.12), we have

∂H(X|G) ∂λ = 1 2n -1 i<j Eâ 2 i,j + O( 2 ) + O( d n ) - 1 2n -1 i<j + O( 2 ) + O( d 2 n 2 ) = 1 2n i<j Eâ 2 i,j -1 + O(n ) + O(d 1/2 λ -1/2 + d 3/2 n -1 λ -1/2 ).
(5.9.13) Define a i,j = E( X i X j |G). Using Bayes rule, we have

p( x i , x j |G) = p(G i,j | x i x j )p( x i , x j |G -(i,j) ) x i , x j p(G i,j | x i x j )p( x i , x j |G -(i,j)
) .

If G i,j = 1, then p( x i , x j |G) = d n (1 + x i x j )p( x i , x j |G -(i,j) ) x i , x j d n (1 + x i x j )p( x i , x j |G -(i,j) ) = (1 + x i x j )p( x i , x j |G -(i,j) ) 1 + âi,j .
Thus

a i,j = âi,j + E(( X i X j ) 2 |G -(i,j) ) 1 + âi,j = âi,j + O( ) If G i,j = -1, p( x i , x j |G) = (1 -d n (1 + x i x j ))p( x i , x j |G -(i,j) ) 1 -x i , x j d n (1 + x i x j )p( x i , x j |G -(i,j) ) = (1- d n (1+ x i x j ))p( x i , x j |G -(i,j) )+O( d n ).
Therefore a i,j = âi,j + O( d n ). Equation (5.9.13) becomes then

∂H(X|G) ∂λ = 1 2n i<j Ea 2 i,j -1 + O(n + d) + O(d 1/2 λ -1/2 + d 3/2 n -1 λ -1/2 ) = - 1 2n i<j E ( X i X j -E( X i X j |G)) 2 + O(n + d + d 1/2 λ -1/2 + d 3/2 n -1 λ -1/2 ).
(5.9.14)

Decomposing I(X; G) = H(X) -H(X|G) we obtain the desired result.

Consequently, if one consider a sufficiently large d (in order to apply Corollary 5.9.1) and one integrate equation (5.9.8) from 0 to λ > 0, and let n tend to infinity, lim sup

n→∞ λ 0 MMSE G n (λ )dλ -I(λ) ≤ Kd -1/2 , (5.9.15)
for some constant K > 0 depending on λ but not on d. Proposition 5.9.2

For λ < λ c (p) lim d→∞ lim inf n→∞ MMSE G n (λ) = 1
(5.9.16)

For λ > λ c (p) lim sup d→∞ lim sup n→∞ MMSE G n (λ) < 1 (5.9.17)
Proof. This is a consequence of equation (5.9.15) and the definition of λ c (p). We will show (5.9.16) first. I is continuous, so by definition of λ c , I(λ c ) = λ c /4. Equation (5.9.15) gives then lim sup

n→∞ λc 0 |1 -MMSE G n (λ)|dλ ≤ 4Kd -1/2
because MMSE G n (λ) ≤ 1. Equation (5.9.16) follows.

Equation (5.9.17) is proved analogously. If lim sup d→∞ lim sup n→∞ MMSE G n (λ 0 ) = 1 for some λ 0 > λ c , then lim sup d→∞ lim sup n→∞ | λ 0 0 1 4 MMSE G n (λ)dλλ 0 4 | = 0 which implies (by equation (5.9.15)) that I(λ 0 ) = 0 which contradicts the definition of λ c .

We would now like to rewrite the result of Proposition 5.9.2 in terms of Overlap instead of MMSE. The following two lemmas will be useful to make the link between the MMSE and the overlap. Lemma 5.9.6

Let n ∈ N * . Let (A 1 , A 2 ) and (B 1 , B 2 ) be two partitions of {1, . . . , n}. Then #A 1 ∩ B 1 - 1 n #A 1 #B 1 = #A 2 ∩ B 2 - 1 n #A 2 #B 2 ,
(5.9.18)

#A 1 ∩ B 1 - 1 n #A 1 #B 1 = -#A 1 ∩ B 2 - 1 n #A 1 #B 2 .
(5.9.19)

Proof. We prove (5.9.18) first.

Remark that #A 2 ∩ B 2 = #B 2 -(#A 1 -#B 1 ∩ A 1 ) and #A 2 #B 2 = n 2 -n(#A 1 + #B 1 ) + #A 1 #B 1 . So that #A 1 ∩ B 1 - 1 n #A 1 #B 1 = #A 2 ∩ B 2 - 1 n #A 2 #B 2 + #B 2 -#A 1 -n + #A 1 + #B 1 = #A 2 ∩ B 2 - 1 n #A 2 #B 2 .
To prove (5.9.19), write #A

1 ∩ B 1 = #A 1 -#A 1 ∩ B 2 . Thus #A 1 ∩ B 1 - 1 n #A 1 #B 1 = #A 1 -#A 1 ∩ B 2 - 1 n #A 1 (n -#B 2 ) = -#A 1 ∩ B 2 - 1 n #A 1 #B 2 .
Lemma 5.9.7

Recall that for x ∈ {1, 2} n and i = 1, 2 we define

S i (x) = {j | x j = i}. Let x = (x 1 , . . . , x n ) ∈ {1, 2} n . Recall that x = (φ p (x 1 ), . . . , φ p (x n )) where φ p (1) = 1-p p and φ p (2) = -p 1-p . Then 1 n n i=1 X i x i = 1 2p(1 -p) Overlap(x, X) + O S 1 (X) n -p + S 2 (X) n -(1 -p) .
Proof.

1 n n i=1 X i x i = 1 n 1 -p p #S 1 (x) ∩ S 1 (X) + p 1 -p #S 2 (x) ∩ S 2 (X) -(n -#S 1 (x) ∩ S 1 (X) -#S 2 (x) ∩ S 2 (X)) = 1 n 1 p #S 1 (x) ∩ S 1 (X) + 1 1 -p #S 2 (x) ∩ S 2 (X) -n = 1 n 1 p (#S 1 (x) ∩ S 1 (X) -p#S 1 (x)) + 1 1 -p (#S 2 (x) ∩ S 2 (X) -(1 -p)#S 2 (x)) = 1 n 1 p (#S 1 (x) ∩ S 1 (X) - 1 n #S 1 (X)#S 1 (x)) + 1 1 -p (#S 2 (x) ∩ S 2 (X) - 1 n #S 2 (X)#S 2 (x)) + O S 1 (x) pn p - S 1 (X) n + S 2 (x) (1 -p)n (1 -p) - S 2 (X) n .
Using Lemma 5.9.6, one obtains then

1 n n i=1 X i x i = 1 2p(1 -p) Overlap(x, X) + O S 1 (X) n -p + S 2 (X) n -(1 -p) .
We are now going to prove Theorem 5.9.2. Remark that

MMSE G n (λ) = 2 n(n -1) i<j E X i X j -E( X i X j |G) 2 = 2 n(n -1) i<j E 1 -(E( X i X j |G)) 2
We will denote • G the expectation with respect to the posterior distribution P(X = •|G). We have then

E(E( X i X j |G) 2 ) = E X i X j x i x j G , where x is sampled, conditionally to G, from P(•|G). Thus MMSE G n (λ) = 1 - 1 n 2 i,j E X i X j x i x j G + o(1) = 1 -E 1 n n i=1 X i x i 2 G
+ o(1). (5.9.20)

Suppose λ > λ c . Then (5.9.20) and Proposition 5.9.2 imply lim inf

d→∞ lim inf n→∞ E 1 n n i=1 x i X i G > 0.
Using Lemma 5.9.7, this gives lim inf

d→∞ lim inf n→∞ E Overlap(x, X) G > 0,
where x is sampled according to the posterior distribution of X. Sampling from the posterior distribution P(X = •|G) provides thus an estimator that achieves a non zero overlap: the community detection problem is solvable.

Suppose now λ < λ c . Suppose that the community detection problem is solvable. There is therefore an estimator a that achieves a non zero overlap. Lemma 5.9.7 gives then

α def = lim inf d→∞ lim inf n→∞ 1 n E| n i=1 X i a i | > 0 Compute now for δ ∈ (0, 1] 2 n(n -1) i<j E( X i X j -δ a i a j ) 2 = 1 n 2 i,j E( X i X j -δ a i a j ) 2 + o(1) = 1 n 2 i =j E( X 2 i X 2 j ) + O(δ 2 ) - 2δ n 2 E( n i=1 X i a i ) 2 + o(1) ≤ 1 + O(δ 2 ) -2δ 1 n E| n i=1 X i a i | 2 + o(1). So that lim inf d→∞ lim inf n→∞ MMSE G n (λ) ≤ lim inf d→∞ lim inf n→∞ 2 n(n -1) i<j E( X i X j -δ a i a j ) 2 = 1-2δα 2 +O(δ 2 ).
The right-hand side will be strictly inferior to 1 for δ sufficiently small. This is contradictory with Proposition 5.9.2 (recall that λ < λ c ). The community detection problem is not solvable. Theorem 5.9.2 is proved.

Chapter 6

Statistical limits of rank-one tensor estimation

We consider in this chapter the tensor-analog of the spiked Wigner model from Chapter 3, namely the "spiked tensor model". Let k ≥ 2 and consider

Y = λ n k-1 X ⊗k + Z , (6.0.1)
where

X ∈ R n such that X √ n and Z i 1 ,...,i k i.i.d.
∼ N (0, 1). The spiked tensor model (6.0.1) was introduced by Montanari and Richard [START_REF] Richard | A statistical model for tensor pca[END_REF], as a natural extension to tensors of the spiked matrix models studied in the previous chapters. The paper [START_REF] Richard | A statistical model for tensor pca[END_REF] shows that there exists a finite value λ c such that weak recovery is possible for λ > λ c . Montanari, Reichman and Zeitouni [START_REF] Montanari | On the limitation of spectral methods: From the gaussian hidden clique problem to rank-one perturbations of gaussian tensors[END_REF] obtained bounds on λ c that were then improved by Perry, Wein and Bandeira [START_REF] Perry | Statistical limits of spiked tensor models[END_REF] who obtained tight bounds in the k → ∞ limit. The threshold for hypothesis testing is also of order 1 [START_REF] Montanari | On the limitation of spectral methods: From the gaussian hidden clique problem to rank-one perturbations of gaussian tensors[END_REF][START_REF] Perry | Statistical limits of spiked tensor models[END_REF] and precise expressions for the threshold has recently been obtained in [START_REF] Chen | Phase transition in the spiked random tensor with rademacher prior[END_REF][START_REF] Chen | Phase transition in random tensors with multiple spikes[END_REF].

In this chapter we derive in Section 6.1 precise information-theoretic limits for estimation. We then turn our attention to the maximum likelihood estimator in Section 6.3 and compute its asymptotic performance.

Information-theoretic limits

Large order limit: k → ∞

We start by a simple analysis of the symmetric tensor estimation model (6.1.5) with Rademacher prior P 0 = 1 2 δ +1 + 1 2 δ -1 in the limit k → ∞. This large k scenario has been studied in [START_REF] Perry | Statistical limits of spiked tensor models[END_REF], where the detection problem was also investigated. We suppose to observe here

Y i 1 ,...,i k = λ n k-1 X i 1 . . . X i k + Z i 1 ,...,i k , (6.1.1) for all (i 1 , . . . , i k ) ∈ {1, . . . , n} k . The Z i 1 ,...,i k are i.i.d. standard Gaussian, independent from X 1 , . . . , X n i.i.d. ∼ 1 2 δ +1 + 1 2 δ -1 .
We will denote by

F (k) n (λ) = 1 n E log x∈{-1,1} n 2 -n exp i 1 ,...,i k λ n k-1 Y i 1 ,...,i k x i 1 . . . x i k - λn 2 and MMSE (k) n (λ) = 1 n k E X ⊗k -E[X ⊗k |Y ] 2
the corresponding free energy and MMSE. We will see that simple arguments (that does not require the knowledge of the exact formulas of Theorem 6.1.1 and Corollary 6.1.2 that we will see in the next section) give that for large values of k we have We start with the study of the free energy. For λ 0 ∈ R we define

• if λ ≤ 2 log(2) -O(log(k) 2 /k 2 ) then lim n→∞ MMSE (k) n (λ) = 1, • if λ ≥ 2 log(2) then lim n→∞ MMSE (k) n (λ) = O(log(k) 2 /k 2 ).
f λ 0 (λ) =    0 if λ ≤ λ 0 , 1 2 (λ -λ 0 ) if λ ≥ λ 0 .

Proposition 6.1.1

There exists an increasing sequence

(λ ∼ k ) k≥2 such that λ ∼ 2 = 1/2 and λ ∼ k = 2 log(2) - O(log(k) 2 /k 2 ) and for all k, n, λ ≥ 0 f 2 log(2) (λ) ≤ F (k) n (λ) ≤ f λ ∼ k (λ) + λ n + log(2n + 1) n . (6.1.2)
Proof. We start with the lower bound. For all λ ≥ 0 we have F (k) n (λ) ≥ 0 and

F (k) n (λ) = 1 n E log x∈{-1,1} n 1 2 n exp i 1 ,...,i k λ n k-1 Y i 1 ,...,i k x i 1 . . . x i k - λn 2 ≥ 1 n E log   1 2 n exp i 1 ,...,i k λ n k-1 Y i 1 ,...,i k X i 1 . . . X i k - λn 2   = λ 2 -log(2) .
Let us now prove the upper-bound. For

x ∈ {-1, 1} n let us write Z(x) = n -k/2 i 1 ,...,i k Z i 1 ,...,i k x i 1 . . . x i k .
By Jensen's inequality, we have

F (k) n (λ) ≤ 1 n E log   x∈{-1,1} n 1 2 n E exp √ λnZ(x) + λn(x • X) k - λn 2 X, Z(X)   .
For x (1) , x (2) ∈ {-1, 1} n fixed, the covariance between Z(x (1) ) and Z(x (2) ) is E[Z(x (1) )Z(x (2) )] = (x (1) • x (2) 2k and therefore

) k . Consequently, for x ∈ {-1, 1} n the law of Z(x) conditionally on Z(X) is N (x • X) k Z(X), 1 -(x • X)
E exp √ λnZ(x) X, Z(X) = exp √ λn(x • X) k Z(X) + λn 2 (1 -(x • X) 2k ) .
We obtain thus:

F (k) n (λ) ≤ 1 n E log   x∈{-1,1} n 1 2 n exp √ λn|Z(X)| + λn 2 2(x • X) k -(x • X) 2k   ≤ 1 n E log   x∈{-1,1} n 1 2 n exp λn 2 2(x • X) k -(x • X) 2k   + λ n . Now, for k ∈ {-n, . . . , n}, we have # x ∈ {-1, 1} n n i=1 x i X i = k ≤ 2 n exp -nh(k/n) ,
where

h(t) = 1+t 2 log(1 + t) + 1-t 2 log(1 -t).
This gives

F (k) n (λ) ≤ 1 n E log   n k=-n exp n 2 2λ(k/n) k -λ(k/n) 2k -2h(k/n)   + λ n ≤ 1 n E log (2n + 1) exp n 2 max t∈[0,1] 2λt k -λt 2k -2h(t) + λ n = 1 2 max t∈[0,1] λ(2t k -t 2k ) -2h(t) + log(2n + 1) n + λ n . (6.1.3) The function λ → max t∈[0,1] λ(2t k -t 2k ) -2h(t) is continuous, 1-Lipschitz and equal to 0 for λ ≤ λ ∼ k where λ ∼ k def = inf t∈(0,1] 2h(t) 2t k -t 2k . Therefore, for λ ≥ λ ∼ k , max t∈[0,1] λ(2t k -t 2k ) -2h(t) ≤ λ -λ ∼ k , which combined to (6.1.3) proves (6.1.2). It remains to show λ ∼ 2 = 1/2 and 2 log(2) -O(log(k) 2 /k 2 ) ≤ λ ∼ k ≤ 2 log(2). Let us start with the case k = 2. The maximizer t of t → λ(2t 2 -t 4 ) -2h(t) verifies 4λ(t -t 3 ) = 2h (t) = log 1 + t 1 -t = 2 artanh(t) .
Therefore t = tanh(2λ(t -t 3 )). For λ ≤ 1/2, this equation admits a unique solution t = 0, whereas of λ > 1/2 it admits a second solution t > 0 and max t∈

[0,1] λ(2t 2 -t 4 ) -2h(t) > 0. Thus λ ∼ 2 = 1/2.
Let now k ≥ 3. Let t k be the largest minimizer of h k : t → 2h(t)/(2t k -t 2k ) on (0, 1]. One has t k ∈ (0, 1). For t ∈ (0, 1), h k (t) has the same sign as

t(log(1 + t) -log(1 -t)) 2h(t) - 2k(1 -t k ) 2 -t k , (6.1.4)
which is decreasing in k. This gives that (t k ) k≥3 is increasing and converges to 1, which is the only possible limit because t k cancels (6.1.4). One has also

t k k → 1. Define u k = 1 -t k . Since t k k → 1, one has ku k → 0. Then t k k = exp(k log(1 -u k )) = 1 -ku k + O(ku 2 k ). We get -log(u k ) + o(1) = k log(2) ku k + O(u 2 k k) 1 + o(1) = log(2)k 2 u k + o(1) . Therefore 2 log(k) = log(2)k 2 u k + log(k 2 u k ) + o(1) ∼ log(2)k 2 u k .
We deduce that 1 -

t k = u k ∼ 2 log(k) log(2)k 2 . We have then 2t k k -t 2k k = 1 -k 2 u 2 k + o(k 2 u 2 k ) and 2h(t k ) = 2 log(2) + O(u k ) + u k log(u k ). We conclude: λ ∼ k = h k (t k ) = 2 log(2) -O(log(k) 2 /k 2 ) .
Using the I-MMSE relations (1.3.4) as usual, we deduce: Corollary 6.1.1

If λ < λ ∼ k then MMSE (k) n (λ) ---→ n→∞ 1 , while for λ > 2 log(2): lim sup n→∞ MMSE (k) n (λ) ≤ 2 log(2) -λ ∼ k λ -2 log(2) = O(log(k) 2 /k 2 ) λ -2 log(2) .
Proof. By the I-MMSE relation (1.3.4), we have MMSE (k)

n (λ) = 1 -2F (k)
n (λ). If λ < λ ∼ k then by convexity and the fact that the free energy is non-decreasing:

0 ≤ F (k) n (λ) ≤ F (k) n (λ ∼ k ) -F (k) n (λ) λ ∼ k -λ ---→ n→∞ 0 ,
by Proposition 6.1.1. This proves the result for λ < λ ∼ k . For λ > 2 log(2) again, by convexity and Proposition 6.1.1 we have

lim inf n→∞ F (k) n (λ) ≥ lim inf n→∞ F (k) n (λ) -F (k) n (2 log(2)) λ -2 log(2) ≥ f 2 log(2) (λ) -f λ ∼ k (2 log(2)) λ -2 log(2) = 1 2 - 2 log(2) -λ ∼ k 2(λ -2 log(2)) ,
which concludes the proof.

The "abrupt" phase transition at λ = 2 log(2) that we see on Figure 6.1 reminds of the phase transition for the "needle in a haystack" problem seen in Section 1.4. This is not surprising, and this has been known for a long time in statistical physics: the Random Energy Model (which is the non-planted analog of the "needle in a haystack" problem) can be seen as the k → ∞ limit of the k-spin model (which corresponds to the spiked tensor model (6.1.1)), see [START_REF] Derrida | Random-energy model: Limit of a family of disordered models[END_REF].

Information-theoretic limits

We turn now our attention to the precise expression to the limit of the minimal mean square error in the model (6.0.1) for general priors P 0 . It will be useful (notably for the proof of Proposition 6.1.2 below) to consider a more general model, where we observe multiple tensors like (6.0.1), of different orders. This is the analog of the "mixed p-spin model" in statistical physics, see for instance [START_REF] Panchenko | The Sherrington-Kirkpatrick model[END_REF]. Let K ≥ 1 and λ 1 , . . . λ K ≥ 0. Assume that we observe, for all k ∈ {1, . . . , K}:

Y i 1 ,...,i k = λ k n k-1 X i 1 . . . X i k + Z i 1 ,...,i k for 1 ≤ i 1 , . . . , i k ≤ n, (6.1.5)
where

X = (X 1 , . . . , X n ) i.i.d. ∼ P 0 and (Z i 1 ,...,i k ) i 1 ,...,i k i.i.d.
∼ N (0, 1) are independent. We define the Hamiltonian

H n (x) = K k=1 i 1 ,...,i k λ k n k-1 Y i 1 ,...,i k x i 1 . . . x i k - λ k 2n k-1 (x i 1 . . . x i k ) 2 , (6.1.6) for x = (x 1 , . . . , x n ) ∈ R n .
The posterior distribution of X given Y reads then:

dP (X = x | Y ) = 1 Z n dP ⊗n 0 (x)e Hn(x) , ( 6.1.7)
where Z n is the appropriate normalizing factor. The free energy is thus

F n (λ) = 1 n E log Z n .
Again, we will express the limit of F n (λ) using the following "potential":

F : (λ, q) → ψ P 0 K k=1 λ k kq k-1 - 1 2 K k=1 (k -1)λ k q k , (6.1.8)
where ψ P 0 is the free energy of the scalar channel (1.3.5), defined by (1.3.6).

Theorem 6.1.1 (Replica-Symmetric formula for the spiked tensor model)

Let P 0 be a probability distribution over R, with finite moment of order 2K. Then, for all λ ∈ [0, +∞) K F n (λ) ---→ n→∞ sup q≥0 F(λ, q) . (6.1.9) Theorem 6.1.1 was proved in [START_REF] Thibault Lesieur | Statistical and computational phase transitions in spiked tensor estimation[END_REF] by the same arguments used for Theorem 3.2.1. The paper [START_REF] Barbier | The stochastic interpolation method: A simple scheme to prove replica formulas in bayesian inference[END_REF] gave then an alternative proof. In the case of a binary signal P 0 = 1 2 δ 1 + 1 2 δ -1 the work [START_REF] Babu | Exact solution of the gauge symmetric pspin glass model on a complete graph[END_REF] obtained bounds on the limit of F n . However, these bounds are not tight for all values of λ. By the I-MMSE relation (1.3.4) we deduce from Theorem 6.1.1 the limit of the Minimum Mean Squared Error:

MMSE (k) n (λ) = inf θ    1 n k E   i 1 ,...,i k X i 1 . . . X i k -θ(Y ) i 1 ...i k 2      ,
where the infimum is taken over all measurable functions θ of the observations Y . For all λ ∈ [0, +∞) K such that F(λ, •) admits a unique maximizer q * (λ) over R ≥0 , we have

MMSE (k) n (λ) ---→ n→∞ E P 0 X 2 k -q * (λ) k . (6.1.10)
The proof is exactly the same than for the matrix case, see the proof of Corollary 3.2.2. Proposition 6.1.2

Assume that P 0 admits a finite moment of order 2K. Let λ ∈ [0, +∞) K such that F(λ, •) admits a unique maximizers q * (λ). Then

1 n n i=1 x i X i (d) ---→ n→∞ q * (λ),
where x is a sample from the posterior distribution of X given Y , independently of everything else. Moreover, if λ k > 0 for some odd k, then

1 n n i=1 x i X i (d) ---→ n→∞ q * (λ).
From Proposition 6.1.2, we deduce using Proposition 1.2.2 and Remark 1.2.1:

Corollary 6.1.3
Assume that P 0 admits a finite moment of order max(2K, 8). Let λ ∈ [0, +∞) K such that F(λ, •) admits a unique maximizers q * (λ). Then

sup x E | x(Y ), X * | x(Y ) X * ---→ n→∞ q * (λ) ρ . (6.1.11)
Proof of Proposition 6.1.2. We define the random variable [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. Let us fix λ ∈ [0, +∞) K such that F(λ, •) admits a unique maximizer q * (λ). From Corollary 6.1.2 we get that for all k such that λ k > 0, we have

Q n def = 1 n n i=1 x i X i . Then for all k ∈ {1, . . . , K}, MMSE (k) n (λ) = E P 0 [X 2 ] k -E[Q k n ] + o n
E[Q k n ] ---→ n→∞ q * (λ) k . (6.1.12)
Let us fix such k. By the "I-MMSE" relation (Proposition 1.3.3) we have

∂F n ∂λ 2k (λ) λ + 2k = E[Q 2k n ] + o n (1).
Since F n is convex in λ 2k , we have by Proposition C.1:

lim sup n→∞ E[Q 2k n ] = lim sup n→∞ ∂F n ∂λ 2k (λ) λ + 2k ≤ ∂ ∂λ 2k sup q≥0 F(λ, q) λ + 2k = q * (λ) 2k . (6.1.13)
From (6.1.12) and (6.1.13) we get that

Q k n (d) ---→ n→∞ q * (λ) k , which leads to |Q n | (d) ---→ n→∞ q * (λ). If k is odd then Q n (d) ---→ n→∞ q * (λ).

Hardness of low-rank tensor estimation

The brutal jump of the minimal mean squared error on Figure 6.1 and the fact that tensor estimation is related (for large orders) to the "needle in a haystack" problem of Section 1.4 seems to indicate that the low-rank tensor estimation problem (6.0.1) for k ≥ 3 is computationally hard. Moreover it is known that computing an eigenvalue, an eigenvector or even the rank of k-tensor is NP-hard when k ≥ 3, [START_REF] Christopher | Most tensor problems are np-hard[END_REF]. The study of [START_REF] Richard | A statistical model for tensor pca[END_REF] supports this picture and shows that unless the signal-to-noise ratio λ goes to infinity with n, popular algorithms such as power iteration, tensor unfolding or message passing fail to recover the signal X, when X is uniformly distributed on the sphere of radius √ n.

The spiked tensor model is thus expected to be an extreme example of statistical problems that admit a statistical-to-algorithmic gap, when X ∼ Unif( √ n S n-1 ). The thresholds for estimation and detection are both order 1 in n; on the other hand, the thresholds for efficient testing and estimation are expected to diverge polynomially in n, λ alg = O(n α ). Sharp algorithmic thresholds have been shown for semi-definite and spectral relaxations of the maximum likelihood problem [START_REF] Samuel B Hopkins | Tensor principal component analysis via sum-of-square proofs[END_REF][START_REF] Samuel B Hopkins | Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectors[END_REF][START_REF] Kim | Community detection in hypergraphs, spiked tensor models, and sum-of-squares[END_REF] as well as optimization of the likelihood itself via Langevin dynamics [START_REF] Ben Arous | Algorithmic thresholds for tensor pca[END_REF]. Upper bounds have also been obtained for message passing and power iteration [START_REF] Richard | A statistical model for tensor pca[END_REF], as well as gradient descent [START_REF] Ben Arous | Algorithmic thresholds for tensor pca[END_REF]. All this approaches need at least α = (k -2)/4 in order to succeed. This suggests that we would be in a "hard regime" (where polynomial time algorithms can only achieve trivial performance) for all (finite) values of the signal-to-noise ratio λ. The work [START_REF] Thibault Lesieur | Statistical and computational phase transitions in spiked tensor estimation[END_REF] provides a more optimistic vision that can be summarized as:

• If the distribution of the signal has zero mean (i.e. E P 0 X = 0) we are indeed in a hard phase for all values of λ.

• However, if E P 0 X = 0 then polynomial-time algorithms (as AMP) can achieve a non-trivial performance and can even be optimal if E P 0 X is not too small.

Let us give some intuition about these points. As we have seen in Section 3.3 the presence of a "hard regime" is characterized (if we believe that AMP are optimal among polynomial-time algorithms) by the fact that q = 0 is a local minimum of the potential q → -F(λ, q). We thus expand around q = 0

-F(λ, q) = - λkq k-1 2 E P 0 [X] 2 + λ(k -1) 2 q k + O(q 2(k-1) ) , because ψ P 0 (0) = 0 and ψ P 0 = 1 2 E P 0 [X] 2 .
Consequently, if E P 0 X = 0 and k ≥ 3, then q = 0 is a local minimum of -F(λ, •) and we are in a "hard regime". But if the prior P 0 has a non-zero mean, then q = 0 is not a local minimum anymore and it is possible (with AMP for instance) to estimate the signal X with efficient algorithms.

The plots of Figure 6.2 confirm this picture. On the first plot (where P 0 = N (µ = 0, 1)) we observe that the local minimum q = 0 is separated from the global minimum by a barrier, which indicates hardness (see the discussion in Section 3.3.2). Since X = (X 1 , . . . , X n ) i.i.d. ∼ N (0, 1) is up to a normalization uniformly distributed on the sphere of radius √ n this is coherent with the results mentioned above. On the second plot, where the prior has a small mean µ = 0.15, the local minimum at q = 0 disappears and is replaced by another local minimum at q 0 , close to 0. It is possible in this situation to achieve non-trivial performance by efficient algorithms (as AMP), but it is again conjectured that their correlation with the planted solution X will be at most equal to q 0 , which is quite small compared to the optimal overlap q * (λ) 0.8. Polynomial-time algorithms can thus have non-trivial performance but are still far from optimal. On the third and fourth plot, we see that for larger means, the local minimum at q 0 disappears completely. It is now possible (using for instance AMP) to achieve the optimal performance in polynomial time.

Maximum likelihood estimation 6.3.1 Background and related work

We adopt here a slightly different normalization of the spiked tensor model in order to study maximum likelihood estimation. Suppose again that we are given an observation, Y , which is a k-tensor of rank 1 in dimension n subject to additive Gaussian noise. That is,

Y = λ √ nX ⊗k + Z, (6.3.1)
where X ∈ S n-1 , the unit sphere in R n and Z is an i.i.d. Gaussian k-tensor with Z i 1 ...i k ∼ N (0, 1) 1 . Throughout this section, we assume that X is drawn from an uninformative prior, namely the uniform distribution on S n-1 . Consequently, the typical order of magnitude of the entries of X is 1/ √ n: the scaling we take here is equivalent to the one we used in Section 6.1.

It is straightforward to show that maximizing the log-likelihood is equivalent to maximizing Y , x ⊗k over the sphere, x ∈ S n-1 . The maximum likelihood estimator (MLE) x ML λ is then defined as2

x ML λ = arg max

x∈S n-1 Y , x ⊗k . (6.3.2)
The goal of this section is to study the asymptotic behavior of the maximum likelihood estimator x ML λ . The function H λ : x → √ n x ⊗k , Y has been extensively studied in statistical physics and mathematics.

In the case λ = 0, the Hamiltonian H 0 was first studied by Crisanti and Sommers in [START_REF] Crisanti | The sphericalp-spin interaction spin glass model: the statics[END_REF] using the heuristic "replica method". They computed the value of its maximum (the "ground state"), a result that was then rigorously confirmed by [START_REF] Talagrand | Free energy of the spherical mean field model[END_REF][START_REF] Chen | The aizenman-sims-starr scheme and parisi formula for mixed p-spin spherical models[END_REF][START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF][START_REF] Chen | Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models[END_REF][START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF]. Crisanti and Sommers analyzed in [START_REF] Crisanti | Thouless-anderson-palmer approach to the spherical p-spin spin glass model[END_REF] the complexity of the function H 0 . In order to characterize how complex the function H 0 can be, one of the main quantity of interest is the number of critical points at a given energy (i.e. likelihood) and with a given correlation with the signal X:

Crt (M, E) def = x:∇H λ (x)=0 1 x, X ∈ M 1 1 n H λ (x) ∈ E (6.3.3) for M ⊂ [-1, 1
] and E ⊂ R. One can define similarly Crt 0 (M, E) the number of local maxima with correlation in M and energy in E. The paper [START_REF] Crisanti | Thouless-anderson-palmer approach to the spherical p-spin spin glass model[END_REF] shows (by non-rigorous methods) that in the case λ = 0, Crt 0 (E) exp(nS

0 (E)) provided E ⊂ [E ∞ (k), +∞) for some value E ∞ (k).
Another derivation of this prediction was done by Fyodorov [START_REF] Yan V Fyodorov | Complexity of random energy landscapes, glass transition, and absolute value of the spectral determinant of random matrices[END_REF] using tools from random matrix theory. The work of Auffinger, Ben Arous and Černỳ [START_REF] Auffinger | Complexity of random smooth functions on the high-dimensional sphere[END_REF][START_REF] Auffinger | Random matrices and complexity of spin glasses[END_REF] confirmed rigorously this picture by proving (using the Kac-Rice formula and results from random matrix theory) the conjectured limiting expression for

1 n log E[Crt ], 1 n log E[Crt 0 ]
, and for other even more precise quantities. Subag [START_REF] Subag | The complexity of spherical p-spin models-a second moment approach[END_REF] showed then, by a second moment computation, that the number of critical point (at which the Hamiltonian is larger than some value E ∞ (k)) is actually concentrated around its mean, giving a precise understanding of the typical number of critical points of the function H 0 .

In the case λ > 0, the model was first considered by Gillin and Sherrington [START_REF] Gillin | p > 2 spin glasses with first-order ferromagnetic transitions[END_REF] who analyzed the Gibbs measure associated to the Hamiltonian H λ . The study of the complexity of H λ was carried out by Ben Arous, Mei, Montanari and Nica [START_REF] Ben Arous | The landscape of the spiked tensor model[END_REF]. The authors compute the (normalized) logarithm of the expected number of local minima below a certain energy level via the Kac-Rice approach and show that there is a transition at a point λ s such that for λ < λ s this is negative for any strictly positive correlation, and for λ > λ s it has a zero with correlation bounded away from zero. In [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF], study the (normalized) logarithm of the (random) number of local minima via a novel (but non rigorous) replica theoretic approach. They argued that there are in fact two transitions for the log-likelihood, called λ s and λ c . First, for λ < λ s , all local maxima of the log-likelihood only achieve asymptotically vanishing correlation. For λ s < λ < λ c , there is a local maximum of the log-likelihood with non-trivial correlation but the maximum likelihood estimator still has vanishing correlation. Finally, for λ c < λ the maximum likelihood estimator has strictly positive correlation. In particular, if we let m(λ), denote the limiting value of the correlation of the maximum likelihood estimator and X, they predict that m(λ) has a jump discontinuity at λ c . We will verify this with Theorem 6.3.1 in the next section.

Main results

In order to state our main results, we need to introduce some notations. Define for t ∈ [0, 1),

f λ (t) = λ 2 t k + log(1 -t) + t (6.3.4)
and let

λ c = sup λ ≥ 0 sup t∈[0,1) f λ (t) ≤ 0 . (6.3.5)
As we will see with Proposition 6.4.1 below, it can be deduced from the results of Section 6.1 that λ c is the information-theoretic threshold for the model (6.3.1). Our main result is that the preceding transition is also the transition for which maximum likelihood estimation yields an estimator which achieves positive correlation with X. Let q * (λ) be defined by

q * (λ) =    0 if λ < λ c arg max t∈[0,1) f λ (t) if λ > λ c . (6.3.6)
As shown in Lemma 6.4.7, the function f λ admits a unique positive maximizer on [0, 1) when λ > λ c , so that this is well-defined. Let z k denote the unique zero on (0, +∞) of

ϕ k (z) = 1 + z z 2 log(1 + z) - 1 z - 1 k . (6.3.7)
Finally, let

GS k = √ k √ 1 + z k 1 + z k k . (6.3.8)
We then have the following.

Theorem 6.3.1

Let λ ≥ 0 and k ≥ 3. The following limit holds almost surely

lim n→∞ 1 √ n max x∈S n-1 x ⊗k , Y =        GS k if λ ≤ λ c √ k 1 + λ 2 q * (λ) k-1 1 + λ 2 kq * (λ) k-1 if λ > λ c . (6.3.9)
Furthermore, we have that for λ = λ c lim n→∞ x ML λ , X = q * (λ). (6.3.10)

As a consequence of Corollary 6.4.1, the maximum likelihood estimator achieves maximal correlation. Unlike the case k = 2, the transition in q * (λ) is not continuous. See Figure 6.3. Theorem 6.3.1 was proved in [START_REF] Jagannath | Statistical thresholds for tensor pca[END_REF] where it is also proved that λ c is the threshold for hypothesis testing, in the case of even k larger than 6. If we denote by Q λ the law of the observations Y given by (6.3.1), then for λ < λ c ,

D TV (Q λ , Q 0 ) ---→ n→∞ 0.
Below λ c it is therefore impossible to distinguish the tensor Y from pure noise. As a consequence of Theorem 6.3.1, the threshold λ c is also the threshold for multiple hypothesis testing: the maximum likelihood is able to distinguish between all of the hypotheses λ > λ c .

Regarding the spinodal transition

While the regime λ s < λ < λ c and the expected transition at λ s is not relevant for testing and estimation, there is still a natural interpretation from the perspective of the landscape of the maximum likelihood. In [START_REF] Ben Arous | The landscape of the spiked tensor model[END_REF][START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF], this is explained explained in terms of the complexity. There is also an explanation in terms of the optimization of the maximum likelihood. We end this section with a brief discussion of this phase. Let λ s be given by

λ s = (k -1) k-1 k(k -2) k-2 . (6.3.11)
Consider the constrained maximum likelihood,

E λ (m) = lim n→∞ 1 n max x∈S n-1 , X,x =m λn X, x k + √ n Z, x ⊗k . (6.3.12)
This limit exists and is given by an explicit variational problem (see (6.4.5) below). For λ > λ s , let q s (λ) be the (unique) positive, strict local maximum of f λ . By Lemma 6.4.7, this is well-defined and satisfies q s (λ) = q * (λ) for λ > λ c . In [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF], it is argued by the replica method that E λ (m) has a local maximum at q s (λ) for all λ > λ s . Establishing this rigorously is a key step in our proof of Theorem 6.3.1. In particular, we prove the following, which is a direct consequence of Lemma 6.4.4 below. Proposition 6.3.1

For λ > λ s , the function E λ has a strict local maximum at q s (λ).

It is easy to verify (by direct differentiation) that the map λ → E λ ( q s (λ)) is strictly increasing on (λ s , +∞). We have also that E λc ( q s (λ c )) = GS k by Lemma 6.4.4 and Lemma 6.4.8, so we get that for λ s < λ < λ c the strict local maximum at q s (λ) has E λ ( q s (λ)) strictly less than the maximum likelihood. In fact, (6.4.5) can be solved numerically, as it can be shown that one may reduce this variational problem, in the setting we consider here, to a two-parameter family of problems in three real variables. This is discussed in 6.4.2 below. In particular, see Figure 6.4 for an illustration of these two transitions in the case k = 4.

Proof of Theorem 6.3.1

In this section, we prove Theorem 6.3.1. We begin by providing a lower bound for the maximum likelihood for every λ ≥ 0 using results on the ground state of the mixed p- spin model recently proved in [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF][START_REF] Chen | Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models[END_REF]. We then use the information-theoretic bound on the maximal correlation achievable by any estimator from [START_REF] Thibault Lesieur | Statistical and computational phase transitions in spiked tensor estimation[END_REF] to obtain the matching upper bound. We end by proving the desired result for the correlation ( x ML λ , X). In the remainder of this section, for ease of notation, we let

H λ (x) = H(x) + λn x, X k , ( 6.4.1)
where

H(x) = √ n i 1 ,...,i k Z i 1 ,...,i k x i 1 . . . x i k .

Variational formula for the ground state of the mixed pspin model

We begin by recalling the following variational formula for the ground state of the mixed p-spin model. Consider the Gaussian process indexed by x ∈ S n-1 :

Y n (x) = √ n p≥1 a p 1≤i 1 ,...,ip≤n g i 1 ,...,ip x i 1 . . . x ip ,
where g i 1 ,...,ip are i.i.d. standard Gaussian random variables and p≥1 2 p a 2 p < ∞. This last condition ensures that the sum above is almost surely finite. The covariance of Y n is given by

E Y n (x)Y n (y) = nξ( x, y ),
where

ξ(t) = p≥1 a 2 p t p . Let C denote the subset of C([0, 1]
) of functions that are positive, non-increasing and concave. For any h ≥ 0, we let P h : C → R be

P h (φ) = ξ (x)φ(x) + 1 φ(x) dx + (h 2 + ξ (0))φ(0). Set G(ξ, h) = 1 2 min φ∈C P h (φ). (6.4.2)
Let us recall the following variational formula. For x ∈ S n-1 , we write x = (x 1 , . . . , x n ).

Theorem 6.4.1 ([51, 111])

For all h ≥ 0,

lim n→∞ 1 n max x∈S n-1 Y n (x) + h √ n n i=1 x i = G(ξ, h),
almost surely and in L 1 .

Remark 6.4.1. While the results of [START_REF] Chen | Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models[END_REF][START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF] are stated with ξ (0) = 0, they still hold when ξ (0) > 0 by replacing ξ → ξ(t) -ξ (0)t and h 2 → h 2 + ξ (0) . To see this, simply note that the Crisanti-Sommers formula still holds in this setting by the main result of [START_REF] Chen | The aizenman-sims-starr scheme and parisi formula for mixed p-spin spherical models[END_REF]. The reformulation from [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF]Eq. (1.0.1)] is then changed by this replacement by simply repeating the integration by parts argument from [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF]Lemma 6.1.1]. From here the arguments are unchanged under the above replacement.

The lower bound

By Borell's inequality, the constrained maximum likelihood (6.3.12) concentrates around its mean with sub-Gaussian tails. In particular, combining this with Borel-Cantelli we see that

E λ (m) = lim n→∞ 1 n E max x∈S n-1 , x,X =m λn x, X k + H(x) . (6.4.3) Clearly, lim inf 1 n E max S n-1 H λ ≥ E λ (m) for all m ∈ [-1, 1].
Recall the definition of λ s from (6.3.11) and q s (λ), see, e.g., Lemma 6.4.7. If we apply this for λ > λ s and m = q s (λ) > 1 -1 k-1 (by Lemma 6.4.7), Lemma 6.4.4 below will immediately yield the following lower bound. Lemma 6.4.1 For all λ > λ s ,

lim inf n→∞ E 1 n max x∈S n-1 H λ (x) ≥ √ k 1 + λ 2 q s (λ) k-1 1 + λ 2 kq s (λ) k-1 . (6.4.4)
We now turn to the proof of Lemma 6.4.4. We begin by observing the following explicit representation for E λ . Lemma 6.

4.2

For all m ∈ [-1, 1] the limit in (6.3.12) exists and

E λ (m) = λm k + G(ξ m , 0), (6.4.5)
where

ξ m (t) = (m 2 + (1 -m 2 )t) k -m 2k .
Proof. We begin by observing that by rotational invariance,

1 n E max x∈S n-1 , x,X =m λn x, X k + H(x) = λm k + 1 n E max x∈S n-1 , x 1 =m H(x) . Let x ∈ S n-1 such that x 1 = m. Then H(x) (d) = √ nm k g 1,...,1 + √ n k-1 j=0 k j 1/2 m j 2≤i 1 ,...,i k-j ≤n g i 1 ,...,i k-j x i 1 . . . x i k-j ,
where (g i 1 ,...,ip ) 1≤i 1 ,...,ip≤n p≤k are i.i.d. standard Gaussians.

So that

E max x∈S n-1 , x 1 =m H(x) = E max x∈S n-2 H m (x)
, where H m is given by:

H m (x) = √ n k-1 j=0 k j 1/2 m j (1 -m 2 ) (k-j)/2 1≤i 1 ,...,i k-j ≤n-1 g i 1 ,...,i k-j x i 1 . . . x i k-j .
The function H m is a Gaussian process with covariance

E H m (x)H m (y) = nξ m ( x, y ),
where ξ m is given by

ξ m (t) = k-1 j=0 k j m 2j (1 -m 2 ) k-j t k-j = (m 2 + (1 -m 2 )t) k -m 2k . (6.4.6)
We conclude using Theorem 6.4.1 to obtain the result.

We now observe that for m large enough, this formula has a particularly simple form. Lemma 6.

4.3

For all |m| ≥ 1 -1 k-1 we have:

E λ (m) = λm k + k(1 -m 2 ). (6.4.7)
Proof. In the setting of Theorem 6.4.1 it was also shown in [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF][START_REF] Chen | Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models[END_REF]

that if ξ (1) + h 2 ≥ ξ (1) then G(ξ, h) = ξ (1) + h 2 . Since ξ m (t) = k(1 -m 2 )(m 2 + (1 -m 2 )t) k-1 ξ m (t) = k(k -1)(1 -m 2 ) 2 (m 2 + (1 -m 2 )t) k-2 , the condition ξ m (1) ≥ ξ m (1) corresponds to (k -1)(1 -m 2 ) ≤ 1, i.e. |m| ≥ 1 -1 k-1 .
When this holds, we get that

E λ (m) = λm k + G(ξ m , 0) = ξ m (1) = λm k + k(1 -m 2 )
by (6.4.5).

We end with the desired explicit formula for E λ ( √ q s (λ)).

Lemma 6.4.4

For all λ > λ s , q s (λ) is a local maximizer of E λ and if we write

x(λ) = λ 2 kq k-1 s (λ), E λ q s (λ) = √ k 1 + x(λ) 1 + x(λ) k .
Proof. Differentiating the expression (6.4.7) for m ≥ 1 -

1 k-1 yields E λ (m) = λkm k-1 - √ km √ 1 -m 2 = k 1 1 -m 2 λkm k-1 + √ km √ 1 -m 2 -1 λ 2 km 2k-2 -λ 2 km 2k -m 2
so that the functions φ λ , f λ and m 2 → E λ (m) have precisely the same monotonicity on [1-1 k-1 , 1) (recall the expression of the derivatives f λ and φ λ given by (6.4.14)). Lemma 6.4.7 gives that q s (λ) is a local maximum of f λ and φ λ for λ > λ s , q s (λ) is therefore a local maximum of E λ .

Let us now compute E λ ( q s (λ)). By Lemma 6.4.7, q s (λ) = x(λ) 1+x(λ) . Consequently,

E λ (q s (λ) 1/2 ) = λq s (λ) k/2 + k(1 -q s (λ)) = √ k 1 + x(λ) 1 + x(λ) k .

The upper bound

We prove here the upper bound. Lemma 6.4.5

For all λ ≥ 0,

lim sup n→∞ E 1 n max x∈S n-1 H λ (x) ≤ GS k + λ 0 q * (t) k/2 dt. (6.4.8)
We defer the proof of this momentarily to observe the following information-theoretic bounds which will be useful in its proof. Proposition 6.4.1

Assume that X is uniformly distributed over S n-1 , independently of Z. Then for all λ ∈ (0, +∞)

\ {λ c } lim n→∞ E X ⊗k -E[X ⊗k |Y ] 2 = 1 -q * (λ) k .
This follows from the results of Section 6.1.2, by approximating the uniform measure on S n-1 by an i.i.d. Gaussian measure. For the completeness, we provide a proof in Section 6.4.6. As a consequence of this, we have the following. Corollary 6.4.1

Assume that X is uniformly distributed over S n-1 , independently from Z. Then for all measurable functions x : (R n ) ⊗k → S n-1 and for all λ = λ c we have

lim sup n→∞ E x(Y ), X k ≤ q * (λ) k/2 . Proof. Compute E X ⊗k - q * (λ) x(Y ) ⊗k 2 = E X ⊗k 2 + q * (λ) k E x(Y ) ⊗k 2 -2q * (λ) k/2 E x(Y ), X k = 1 + q * (λ) k -2q * (λ) k/2 E x(Y ), X k .
Recall that the posterior mean, E[X ⊗k |Y ], uniquely achieves the minimal mean-square error over all square-integrable tensor-valued estimators, T (Y ), for X ⊗k . The proposition follows then from Proposition 6.4.1 which gives

lim inf n→∞ E X ⊗k - q * (λ) x(Y ) ⊗k 2 ≥ lim inf n→∞ E X ⊗k -E X ⊗k Y 2 = 1 -q * (λ) k .
With this in hand we may now prove Lemma 6.4.5. Proof of Lemma 6.4.5. By Proposition 6.4.2 and an application of an envelope-type theorem (see, e.g., Proposition D.2), the map λ →

1 n E max S n-1 H λ (x) is differentiable for λ ≥ 0, with derivative ∂ ∂λ E 1 n max x∈S n-1 H λ (x) = E x ML λ , X k . (6.4.9)
By [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF][START_REF] Chen | Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models[END_REF] we know that

H λ (x) ≤ λ 0 lim sup n→∞ E x ML γ , X k dγ + GS k ≤ λ 0 q * (γ) k/2 dγ + GS k ,
where the second inequality follows from 6.4.1.

Proof of first part of Theorem 6.3.1

By an elementary but tedious calculation (see Lemma 6.4.9) the right sides of (6.4.4) and (6.4.8) are equal for λ ≥ λ c (recall that q * (λ) = q s (λ) for such λ). Thus for all λ > λ c ,

E 1 n max x∈S n-1 H λ (x) ---→ n→∞ √ k 1 + λ 2 q * (λ) k-1 1 + λ 2 kq * (λ) k-1
. (6.4.10)

We will now prove that for

λ ≤ λ c , Ď M n (λ) ---→ n→∞ GS k , where Ď M n (λ) is defined by Ď M n (λ) = E 1 n max x∈S n-1 H λ (x) .
Notice that Ď M n (λ) is convex as an expectation of a maximum of linear functions. By (6.4.9), it follows that Ď M n (0 + ) ≥ 0. (When k is odd, we use rotational invariance to see that it is in fact zero.) Consequently, Ď M n is non-decreasing on [0, +∞). By [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF][START_REF] Chen | Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models[END_REF] (see Theorem 6.4.1), lim n→∞ Ď M n (0) = GS k . By (6.4.10) and Lemma 6.4.9,

lim λ→λ + c lim n→∞ Ď M n (λ) = GS k .
Consequently, we obtain that for all λ ∈ [0, λ c ], lim n→∞ Ď M n (λ) = GS k . The almost sure convergence of (6.3.9) follows then from the convergence of the expectation Ď

M n (λ), combined with Borell's inequality for suprema of Gaussian processes (see for instance [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF]Theorem 5.8]) and the Borel-Cantelli Lemma. Remark 6.4.2. By (6.4.5) , E λ is given by a variational problem over the space C. We first observe that one can easily solve this variational problem numerical due to the following simple reductions. First note that if we let ξ m be as in (6.4.6), then (1/ ξ m ) is strictly positive, where the prime denotes differentiation in t. Thus by [START_REF] Jagannath | Low temperature asymptotics of spherical mean field spin glasses[END_REF]Theorem 1.2.4], the minimizer φ must be of the form φ(s) = 1 s dν s where ν s = θ 1 δ a +θ 2 δ b , where a, b ∈ [0, 1] and θ i ≥ 0. Thus the variational problem (6.4.5) is a variational problem over 4 parameters which can be solved numerically. These observations then rigorously justify the starting point of the discussion in [183, Section 4], namely the "RS" and "1RSB" calculation in [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF]Sect. 4.B] in the regime they analyze, called the "T = 0" regime there. We refer the reader there for a more in-depth discussion, see [START_REF] Ros | Complex energy landscapes in spiked-tensor and simple glassy models: Ruggedness, arrangements of local minima, and phase transitions[END_REF]Sect. 4.C].

Proof of second part of Theorem 6.3.1

We now turn to the second part of Theorem 6.3.1, namely (6.3.10). Let M n (λ) denote

M n (λ) = 1 n max x∈S n-1 H λ (x)
. Let λ ∈ (0, +∞) \ {λ c }. By (6.3.9) and Lemma 6.4.9

lim n→∞ M n (λ) = (λ) def =    GS k if λ ≤ λ c GS k + λ 0 q * (γ) k/2 dγ if λ > λ c
. By Proposition 6.4.2 and the Milgrom-Segal envelope theorem (see Proposition D.2), M n is differentiable in λ with derivative

M n (λ) = x ML λ , X k , almost surely. As M n is convex in λ (it is a maximum of linear functions), we see that for any 0 < h < λ, M n (λ -h) -M n (λ) h ≤ M n (λ) ≤ M n (λ + h) -M n (λ) h .
By taking the n → ∞ limit, we get that almost surely

(λ -h) -(λ) h ≤ lim inf n→∞ x ML λ , X k ≤ lim sup n→∞ x ML λ , X k ≤ (λ + h) -(λ) h . (6.4.11)
Since is differentiable for λ = λ c , we may take h → 0 to obtain lim n→∞ x ML λ , X k = q * (λ) k/2 almost surely, which proves (6.3.10).

Auxiliary results

Uniqueness of minimizers

This section gathers some basic lemmas that will be useful for the analysis. Proposition 6.4.2

Recall the definition (6.4.1) of H λ . We have the following • If k is odd, then H λ has almost surely one unique maximizer over S n-1 .

• If k is even, then H λ has almost surely two maximizers over S n-1 , x * and -x * .

Proof. We note the following basic fact from the theory of Gaussian processes, see, e.g. [START_REF] Kim | Cube root asymptotics[END_REF].

Lemma 6.4.6

Let (Z(t)) t∈T be a Gaussian process indexed by a compact metric space T such that t → Z(t) is continuous almost surely. If the intrinsic quasi-metric, d(s, t) 2 = Var Z(s) -Z(t) , is a metric, i.e., d(s, t) = 0 for s = t, then Z admits a unique maximizer on T almost surely.

Observe H λ is continuous on the compact S n-1 . For x 1 , x 2 ∈ S n-1 , we have

Var H λ (x 1 ) -H λ (x 2 ) = 2n 1 -x 1 , x 2 k .
If k is odd, then the proposition follows directly from the Lemma. If k is even, we apply the Lemma on the quotient space S n-1 / ∼ where ∼ denotes the equivalence relation defined by

x 1 ∼ x 2 ⇔ x 1 = x 2 or x 1 = -x 2 .

Study of the asymptotic equations

Define, for all q ∈ [0, 1]

φ λ (q) = λ 2 kq k-1 -log(1 + λ 2 kq k-1 ) -λ 2 (k -1)q k .
(6.4.12) Lemma 6.4.7

We have for all λ > 0, max

q∈[0,1) f λ (q) = max q∈[0,1] φ λ (q)
Furthermore, if we let

λ s = (k-1) k-1 k(k-2) k-2 :
• For λ < λ s , then the functions f λ and φ λ are decreasing on [0, 1).

• For λ > λ s , the functions f λ and φ λ have a strict local minimum at q u (λ) and a strict local maximum at q s (λ) where 0 < q u < k-2 k-1 < q s < 1, and both functions are strictly monotone on the intervals (0, q u ), (q u , q s ) and (q s , 1). Moreover, q s (λ) satisfies:

q s (λ) = λ 2 kq s (λ) k-1 1 + λ 2 kq s (λ) k-1 . (6.4.13)
Finally, for λ > λ c , q * (λ) = q s (λ) is the unique maximizer of f λ and φ λ over [0, 1).

Proof.

We have for q ∈ [0, 1)

φ λ (q) = k(k -1)λ 2 q k-1 1 + λ 2 kq k-1 h(q) and f λ (q) = h(q) 1 -q (6.4.14)
where h(q) = λ 2 kq k-1 -λ 2 kq k -q. It suffices therefore to study the variations of f λ . notice also that φ λ (q) = f λ (q) + h(q) -log(1 + h(q)).

Since f λ (q) = 0 implies h(q) = 0, this implies that max

q∈[0,1] φ λ (q) = max q∈[0,1) f λ (q)
and that these maxima are achieved at the same points. Let us now study the sign of the polynomial h(q):

h(q) = qkλ 2 q k-2 -q k-1 - 1 kλ 2 . (6.4.15)
One verify easily that the polynomial q k-2 -q k-1 achieves its maximum at k-2 k-1 and that the value of this maximum is (k-2) k-2 (k-1) k-1 . We get that for λ < λ s , h (q) < 0 for all q > 0. For λ > λ s we get that h admits exactly 3 zeros on R: 0 < q u (λ) < q s (λ) < 1. Since the maximum of h is achieved at k-2 k-1 we get that q u (λ) < k-2 k-1 < q s (λ). This proves the two points of the lemma; (6.4.13) simply follows from the fact that h(q s (λ)) = 0. The last statement of Lemma 6.4.7 is then an immediate consequence of the definition of λ c .

Recall that z k is defined as the unique zero of ϕ k (z) = 1+z z 2 log(1 +z) -1 z -1 k on (0, +∞).

Lemma 6.4.8

The mapping λ

→ q s (λ) is C ∞ on (λ s , +∞). Moreover λ 2 kq s (λ c ) k-1 = z k .
Proof. The first part follows from a straightforward application of the implicit function theorem.

We get in particular that the mapping λ → q s (λ) is continuous for λ > λ s . So by definition of λ c and Lemma 6.4.7, φ λc (q s (λ c )) = 0. Let us write

x = λ 2 kq s (λ c ) k-1 . 0 = φ λc (q s (λ c )) = x -log(1 + x) - k -1 k xq s (λ c ) = x -log(1 + x) - k -1 k x 2 1 + x ,
because q s (λ c ) = x 1+x (see (6.4.13)). This gives that ϕ k (x) = 0 and thus x = z k .

Lemma 6.4.9

Let λ > λ c and write x(λ) = λ 2 kq s (λ) k-1 . Then we have √ k

1 + x(λ) 1 + x(λ) k = GS k + λ λc q s (γ) k/2 dγ . Proof. Let us write g(λ) = √ k √ 1+x(λ)
1 + x(λ) k . By Lemma 6.4.4, q s (λ) is a local maximizer of E λ and thus a critical point of E λ . This gives

g (λ) = ∂ λ E λ ( q s (λ)) = ∂ λ E λ ( q s (λ)) = q s (λ) k/2 .
The lemma follows then from the fact that x(λ c ) = z k by Lemma 6.4.8 and the definition (6.3.8) of GS k .

Proof of 6.4.1

For P 0 a probability distribution over (R n ) ⊗k with finite second moment, we define the free energy

F P 0 (γ) = 1 n E log P 0 (dx) exp √ γn x, Z + γn x, X 0 - 1 2 γn x 2
where X 0 ∼ P 0 and Z i 1 ,...,i k ∼ N (0, 1) are independent. Proposition B.1 gives that for two probability distributions P 1 , P 2 on (R n ) ⊗k with finite second moment, we have

F P 1 (γ) -F P 2 (γ) ≤ γ 2 E P 1 X 1 2 + E P 2 X 2 2 W 2 (P 1 , P 2 ),
where W 2 (P 1 , P 2 ) denotes the Wasserstein distance of order 2 between P 1 and P 2 . Let µ n be the distribution of X ⊗k when X ∼ Unif(S n-1 ) and let ν n be the distribution of X ⊗k when X ∼ N (0, 1 n Id n ). Let us compute a bound on W 2 (µ n , ν n ). Let X be drawn uniformly over S n-1 , and G ∼ N (0, Id), independently from X. Then (X ⊗k , ( G X/ √ n) ⊗k ) is a coupling of µ n and ν n , so that, by definition of W 2 ,

W 2 (µ n , ν n ) 2 ≤ E X ⊗k -X G / √ n ⊗k 2 = E 1 n n i=1 G 2 i k/2 -1 2
where we use that E X k = 1. By the law of large numbers, it then follows that

lim n→∞ |F µn (γ) -F νn (γ)| → 0.
Recall the definition (6.4.12) of φ λ (q) and define

L(γ) = 1 2 max q∈[0,1] φ √ γ (q) = 1 2 max q∈[0,1) f √ γ (q)
, where the equality comes from Lemma 6.4.7. By Theorem 6.1.1 we have λ ≥ 0, F νn (γ) → L(γ) as n → ∞, which implies F µn (γ) → L(γ). The "I-MMSE relation" (Proposition 1.3.3) gives that γ → F µn (γ) is convex and differentiable over [0, +∞) and

F µn (λ 2 ) = 1 2 1 -E X ⊗k -E X ⊗k Y 2 .
By Proposition C.1, F µn converges to L for each λ > 0 at which L is differentiable. For γ < λ 2 c , L(γ) = 0, so L is differentiable on (0, λ 2 c ) with derivative equal to 0. For γ > λ 2 c , we know by Lemma 6.4.7 that f √ γ admits a unique maximizer q * ( √ γ) on [0, 1].

Proposition D.2 gives that L is differentiable at γ with derivative

L (γ) = 1 2 (∂ γ f √ γ )(q * ( √ γ)) = 1 2 q * ( √ γ) k .
We conclude that

lim n→∞ 1 2 1 -E X ⊗k -E[X ⊗k |Y ] 2 = lim n→∞ F µn (λ 2 ) =    0 if λ < λ c 1 2 q * (λ) k if λ > λ c .

Chapter 7

Phase transitions in Generalized Linear Models

Introduction: learning a linear classifier

Before presenting the Generalized Linear Models (GLM) in full generalities, we first focus on a simple supervised learning task: learning a linear classification rule. We assume here to have m points g 1 , . . . , g m ∈ R n , that are classified between two classes "+1" and "-1", according to the value of

Y µ = sign X * , g µ ,
where X * ∈ R n is a normal vector to the hyperplane that separates the two classes. The supervised learning task is, given the labeled dataset (g µ , Y µ ) 1 ≤ µ ≤ m , to learn the hyperplane Span(X * ) ⊥ that separates the two classes.

Rosenblatt's perceptron algorithm

Rosenblatt [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Rosenblatt | Principles of neurodynamics[END_REF] proposed an iterative algorithm to find a vector x that separates the data, i.e. such that sign( x, g µ ) = Y µ for all µ. The so-called "perceptron" algorithm goes as follows:

• Initialize x 0 = 0. • While I t def = {µ | sign( x t , g µ ) = Y µ } = ∅ choose µ t uniformly at random in I t and update x t :

x t+1 = x t + sign(Y µt )g µt . (7.1.1)

• If I t = ∅, then return x = √ n x t / x t .
Rosenblatt [START_REF] Rosenblatt | Principles of neurodynamics[END_REF] and then Novikoff [START_REF] Albert B Novikoff | On convergence proofs for perceptrons[END_REF] proved that the perceptron algorithm terminates after a finite number of iterations: Proposition 7.1.1

Let β = max µ g µ 2 and γ = min µ | X * , g µ |. Then the perceptron algorithm converges for t ≤ t max where

t max def = β X * 2 γ 2 .
Proof. Let t f be the time at which the algorithm converges, i.e. the first t for which Y µ = sign( x t , g µ ) for all µ. If such a t does not exist, we define t f = +∞. We have to prove that t f ≤ t max . From the update rule (7.1.1) we have for all t < t f :

x t+1 , X * = x t , X * + sign(Y µt ) g µt , X * ≥ x t , X * + γ,
so that x t , X * ≥ tγ for all t ≤ t f . On the other hand, one has

x t+1 2 = x t 2 + g µt 2 + 2sign(Y µt ) g µt , x t ≤ x t 2 + β,
by definition of β and because sign( g µt , x t ) = Y µt . We get x t 2 ≤ tβ, for all t ≤ t f . From what we have seen above and the Cauchy-Schwarz inequality, we have for all t ≤ t f :

t 2 γ 2 ≤ x t , X * 2 ≤ x t 2 X * 2 ≤ tβ X * 2 .
We conclude that t f ≤ t max .

What is now a typical order of magnitude of t max ? We will be interested in the asymptotic regime, where n, m → ∞ such that m/n → α > 0. Let us now suppose that the data points are randomly chosen as (g µ ) 1≤µ≤m i.i.d.

∼ N (0, Id n ) and assume that X * = √ n.

In that case, the random variables X * , g µ are i.i.d. N (0, n) random variables, so there exists constants c, C > 0 such that with high probability γ = min µ | X * , g µ | ≥ c √ n/m and max µ g µ 2 ≤ Cn. Consequently, with have with high probability

t max = O(n 3 ).
This means that it is possible in this model to find a separating hyperplane in polynomial time.

The generalization problem

We have seen in the previous section an efficient algorithm to find a separating hyperplane of our data. In other words, the perceptron algorithm is able to achieve a "training error" of 0 on the dataset (g µ , Y µ ) µ≤m . But does this algorithm generalize well? That is, given a new point g new ∼ N (0, Id n ) independent of g, who has label Y new = sign( X * , g new ), what is the probability that sign( x, g new ) = Y new ?

Let us suppose that X * = √ n. Since we assumed g new to be a standard Gaussian vector independent of g (and therefore also independent of x), a simple computation gives

P sign( x, g new ) = Y new = arccos 1 n x, X * π .
Not surprisingly, the generalization performance of the perceptron algorithm depends on the correlation between x and the planted solution X * . From now we take X * ∼ Unif({-1, 1} n ), independently from g. We plot on Figure 7.3 on page 114 the correlation The question is now: how does this compare with the best achievable correlation? Is it possible to generalize perfectly? To study this problem, we have to look at the posterior distribution of X * given g and Y which is simply the uniform distribution over

S n def = x ∈ {-1, 1} n ∀µ ∈ {1, . . . , m}, sign x, g µ = Y µ .
Obviously X * ∈ S n , and #S n decreases as m increases. In order to be able to estimate X * well, one would need that the points of S n are well correlated with X * . We introduce therefore for

I ⊂ [-1, 1], S n (I) def = S n x ∈ {-1, 1} n 1 n x, X * ∈ I .
The expected value of #S n (I) (or more precisely 1 n log E #S n (I) since #S n (I) is expected to be of the exponential order) can be computed easily: Proposition 7.1.2

Let h(q) = q+1 2 log(1 + q) + 1-q 2 log(1 -q) and ϕ(q) = arccos(q)/π. Define f α (q) = log(2) -h(q) + α log ϕ(q). Then, for any non-empty interval I ⊂ [-1, 1], we have (recall that m/n → α as n, m → ∞):

1 n log E #S n (I) ---→ n→∞ sup q∈I f α (q). Proof. For x ∈ {-1, 1} n , one writes q(x) = 1 n n i=1 x i X i . We have, conditionally on X * , E#S n (I) = x, q(x)∈I E m µ=1 1(sign( x, g µ ) = sign( X * , g µ )) = x, q(x)∈I P sign( x, g ) = sign( X * , g ) m , because (g µ ) µ≥1 i.i.d.
∼ N (0, Id n ). An easy computation gives P sign( x, g ) = sign( X * , g ) = ϕ(q(x)). For k ∈ Z we define

N n,k = # x ∈ {-1, 1} n q(x) = k/n . If n + k is odd, then N n,k = 0, otherwise N n,k = n (n+k)/2 . By Sterling approximation, there exists a constant c > 0 such that c √ n 2 n exp -nh(k/n) ≤ N n,k ≤ 2 n exp -nh(k/n) ,
while the upper-bound follows from the standard Chernoff bound for a sum of i.i.d. Rademacher random variables. Since

E#S n (I) = k, k/n∈I N n,k ϕ(k/n) m , we get 2 n max k,k/n∈I e -nh(k/n) ϕ(q) m ≤ E#S n (I) ≤ (2n + 1)2 n max q∈I e -nh(q) ϕ(q) m
and the result follows from taking the logarithm and dividing by n.

Let us define

α 1 = inf α > 0 sup q∈[-1,1] f α (q) = 0 1.45. ( 7
1 n log E[#S n (I )] ---→ n→∞ sup q∈I f α (q) < 0, because for α > α 1 , the maximum of f α is uniquely achieved at q = 1. Consequently P 1 n n i=1 X i x i ≤ 1 -= E #S n (I ) #S n ≤ E #S n (I ) ---→ n→∞ 0.
Perfect reconstruction is therefore possible for α > α 1 . But is α 1 the smallest value of α for which one can estimate X * perfectly, i.e. the smallest value of α for which (7.1.3) holds? In the physics literature, the current problem was introduced by Gardner and Derrida [START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF] who predicted (based on numerical simulations) that it was possible to learn X * perfectly for values of α larger than α IT = 1.35 ± 0.10. Then Györgyi [START_REF] Györgyi | First-order transition to perfect generalization in a neural network with binary synapses[END_REF] used the heuristic replica method to derive an exact formula for α IT 1.249.

Unfortunately, Proposition 7.1.2 does not allow us to obtain the expression of α IT . The issue is that the expected value of #S n does not capture the typical order of magnitude n log E#S n (given by Proposition 7.1.2), 1 n E log #S n and q * (α) (from Proposition 7.1.3), together with the correlation q AMP (α) achieved by GAMP and the correlation 1 n x, X * achieved by the perceptron algorithm (see Section 7.1.1) for n = 10000.

lim n→∞ 1 n E log #S n lim n→∞ 1 n log E#S n q * (α) q AMP (α) 1 n x, X *
of #S n : the mean of #S n is much larger than its median. This comes from exceptional events of extremely small probability on which #S n is exceptionally large, which make its first moment too large.

The right quantity to consider is 1 n E log #S n , which will not be parasitized by these exceptional events. This quantity is, however, harder to compute since the logarithm is now inside the expectation. Computing such quantities will be one of the main goals of this chapter: the result below will follow from an application of Theorems 7.3.1 and 7.3.2 presented in the next section. Proposition 7.1.3

Let us define for q ∈ [0, 1]

f α (q) = log(2)+inf r≥0 E log cosh( √ rZ+r) +2αE Φ √ q Z √ 1 -q log Φ √ q Z √ 1 -q - r(q + 1) 2 ,
where Z ∼ N (0, 1). Then we have

lim n→∞ 1 n E log #S n ---→ n→∞ max q∈[0,1] f α (q).
Further, for all values of α for which the maximum in q above is achieved at a unique point q * (α) (which is the case for all α outside a countable set), we have

1 n n i=1 X i x i P ---→ n→∞ q * (α). (7.1.4)
Notice that (7.1.4) implies that

sup x E x(Y , g), X * x(Y , g) X * ---→ n→∞ q * (α),
where the supremum is taken over all measurable functions x of Y and g. The critical value of α for which one can recover X * perfectly is then

α IT = inf α > 0 max q∈[0,1] f α (q) = 0 = inf α > 0 q * (α) = 1 1.249.
However, as we see on Figure 7.3 on page 114 the perceptron algorithm of Section 7.1.1 does not achieve the optimal correlation q * (α). It is a priori unclear if this can be done by an efficient algorithm. We will see in Section 7.4 that there exists a polynomialtime algorithm called GAMP that achieve asymptotically a correlation of q AMP (α), where q AMP (α) is the limit of the sequence (q t ) t≥0 given by the recursion (7.4.4). A bold conjecture from statistical physics states that it is impossible to do better that GAMP in polynomial time. One sees on Figure 7.3 on page 114 that there is a gap between √ q * and √ q AMP for α ∈ (α IT , α AMP ). In this regime it is theoretically possible to achieve a "perfect correlation" with the signal, but not with an efficient algorithm.

Generalized linear estimation: Problem statement 7.2.1 Definition

Let n, m ∈ N * . We define a Generalized Linear Model (GLM) as follows. Given a signal vector X * in R n and m "measurement vectors" Φ 1 , . . .

Φ m ∈ R n we sample Y 1 , . . . , Y m independently (conditionally on X * , Φ) according to Y µ ∼ P out • 1 √ n Φ µ , X * , 1 ≤ µ ≤ m. (7.2.1)
Here P out is a transition kernel: for all x ∈ R, P out (•|x) is a probability measure on R such that for all A ∈ B(R) the map x → P out (A|x) is measurable. The definition (7.2.1) of a Generalized Linear Model is somehow a bit more general than the classical definition of GLMs in statistics. We will discuss this in Section 7.2.2 below. Given the measurement vectors (Φ µ ) 1≤µ≤m and the observations (Y µ ) 1≤µ≤m , there are two main statistical tasks:

Y new ∼ P out (• | 1 √ n Φ new , X * ).
Let Φ ∈ R m×n be the matrix whose rows are Φ 1 , . . . , Φ m . The GLM (7.2.1) encompasses many statistical models of significant interest, depending on the choice of the kernel P out :

• The linear model: Y = 1 √ n ΦX * + σZ, where σ ≥ 0 and Z is some random noise.

• The Poisson regression: P out (•|x) = Pois(e x ), where Pois(θ) denotes the Poisson distribution of mean θ.

• The (real) phase-retrieval problem or "signless channel":

Y = 1 √ n |ΦX * | + σZ.
• The "1-bit compressed sensing", that we studied in Section 7.1: Y = sign(ΦX * ).

• The logistic model. This is the case when P out is given by

P out (y = 1 | x) = 1 -P out (y = -1 | x) = 1 1 + e -λx .
In the examples above, the functions absolute value | • | and sign act componentwise on the vector ΦX * .

Link with the classical definition of Generalized Linear Models in statistics

The definition of Generalized Linear Models in the statistical literature (see for instance [START_REF] Mccullagh | Generalized linear models[END_REF]) differs slightly from the one given in the previous section. GLM are usually specified by:

• A bijective function g : R → R, called "link function".

• A probability distribution P θ parametrized by θ ∈ R from the exponential family, i.e. whose admits a density f θ with respect to a reference measure ν given by

f θ (y) = h(y)e θy-b(θ) ,
for some functions h and b.

Then, given a "linear predictor"

η µ = 1 √ n Φ µ , X * , Y µ is sampled as Y µ ∼ P θµ
, where θ µ is given by the relation

g -1 (η µ ) = b (θ µ ). (7.2.2)
This choice of θ µ ensures that the linear predictor η µ is related to the mean of Y µ by

η µ = g(E[Y µ ]) = g(b (θ µ ))
. This definition of a Generalized Linear Model is obviously a particular case of the definition (7.2.1) since the kernel P out can reproduce the generation process that we just described.

Let us now mention some limitations of this "classical" definition, compared to (7.2.1):

• The support of P θ does not depend on θ (and therefore on Φ µ , X * ). Hence the classical definition does not encompass the clustering problem discussed in Section 7.1.

• The mean of Y µ is in one-to-one correspondence with η µ by (7.2.2), whereas with the definition (7.2.1) one could imagine that this mean depends differently on η µ (think for instance to the "signless channel" where

E[Y µ ] = |η µ |).

Bayesian framework

We will study the Generalized Linear Model (7.2.1) in a Bayesian framework, where the components X * 1 , . . . , X * n of X * are i.i.d. samples from a probability distribution P 0 over R. We will also assume that the measurement matrix Φ is independent of X * , with independent entries that have zero mean and unit variance.

We will only consider transition kernels P out that admits a transition density with respect to Lebesgue's measure or the counting measure on N. We will (with a slight abuse of notation) also use the notation P out (•|x) to denote this transition density. The posterior distribution of X * given Y , Φ takes the form:

dP (x|Y , Φ) = 1 Z(Y , Φ) dP ⊗n 0 (x) m µ=1 P out Y µ 1 √ n [Φx] µ (7.2.3) = 1 Z(Y , Φ) dP ⊗n 0 (x)e -H(x;Y ,Φ) (7.2.4)
where the Hamiltonian is defined as

H(x; Y , Φ) def = - m µ=1 log P out Y µ 1 √ n [Φx] µ (7.2.5)
and the partition function (the normalization factor) is defined as

Z(Y , Φ) def = dP ⊗n 0 (x)e -H(x;Y ,Φ) . (7.2.6)
The main quantity of interest here is the associated free energy:

F n def = 1 n E log Z(Y , Φ). (7.2.7)
We will compute in Section 7.3.2 the limit of

F n when n, m → ∞ while m/n → α > 0.
Example 7.2.1. In the case of the "planted perceptron" studied in Section 7.1, we had We will however sometimes need to take ∆ > 0 in order to "regularize" P out . In that case the transition kernel P out admits a transition density with respect to Lebesgue's measure, given by

P out (•|x) = δ sign(x) and P 0 = 1 2 δ 1 + 1 2 δ -1 . We see that Z(Y , Φ) = 1 2 n # x ∈ {-1, 1} n ∀µ
P out (y|x) = 1 √ 2π∆ dP A (a)e -1 2∆ (y-ϕ(x,a)) 2 . (7.2.9)
When ∆ = 0, we will only consider discrete channels where ϕ takes values in N 1 . In that case P out admits a transition density with respect the counting measure on N, that we will write also P out (•|x).

Let us end this section by looking at the link between the continuity properties of P out and ϕ. Clearly, if (7.2.8) holds and if for almost all a (with respect to P A ), the function ϕ(•, a) is continuous at some x 0 ∈ R, then x → P out (•|x) is continuous (for the weak convergence) at x 0 . The next proposition states that there exists functions ϕ that verify (7.2.8) and such that the converse is true.

Proposition 7.2.1

There exists a measurable function ϕ : R 2 → R such that for all x ∈ R, P out (•|x) is the law of ϕ(x, A) for A ∼ Unif([0, 1]). Moreover if x → P out (•|x) is continuous at x 0 for the weak convergence, i.e.

P out (•|x) ---→ x→x 0 P out (•|x 0 ).
then for almost all a ∈ [0, 1] the function ϕ(•, a) is also continuous at x 0 .

Proof. We define the cumulative distribution function of P out (•|x):

F (y|x) = y -∞ P out (dt|x),
and its generalized inverse:

ϕ(x, a) def = inf y ∈ R F (y|x) ≥ a .
For A ∼ Unif([0, 1]), we know that ϕ(x, A) (

= P out (•|x). Let now x 0 ∈ R be a point at which

x → P out (•|x) is continuous. The set S x 0 def = a ∈ R ∃y 1 = y 2 , a = F (y 1 |x 0 ) = F (y 2 |x 0 ) is countable. Indeed if a ∈ S x 0 then we can find q ∈ Q such that a = F (q|x 0 ). If we define ψ(a) = q then ψ : S x 0 → Q is injective hence S x 0 is countable. Let a ∈ R \ S x 0 . Since P out (•|x) ---→ x→x 0 P out (•|x 0 ), we have F (y|x) → F (y|x 0 ) for almost all y ∈ R. Let y 1 < ϕ(x 0 , a) < y 2 such that F (y i |x) → F (y i |x 0 ) for i = 1, 2.
Recall that a ∈ S x 0 so we have

F (y 1 |x 0 ) < a = F (ϕ(x 0 , a)|x 0 ) < F (y 2 |x 0 ).
Consequently for x close enough from x 0 we have F (y 1 |x) < a < F (y 2 |x) which implies that y 1 < ϕ(x, a) < y 2 . Since y 1 and y 2 can be chosen arbitrarily close to ϕ(x 0 , a) we conclude: ϕ(x, a) ---→ x→x 0 ϕ(x 0 , a).

Information-theoretic limits 7.3.1 Two scalar inference channels

As for the "spiked models" of the previous chapters, because of the decoupling principle seen in Chapter 2, the limit of the free energy (7.2.7) will be expressed in terms of free energies of simple scalar channels. The first one is the additive Gaussian channel (1.3.5) that we already studied in Section 1.3:

Y 0 = √ r X 0 + Z 0 , (7.3.1)
where X 0 ∼ P 0 and Z 0 ∼ N (0, 1) are independent and r ≥ 0. The inference problem consists of retrieving X 0 from the observations Y 0 . As we have seen in Section 1.3, the free energy ψ P 0 (r) (defined by (1.3.6)) associated to this problem is related to the mutual information between the signal and observations by:

I P 0 (r) def = I(X 0 ; √ r X 0 + Z 0 ) = rρ 2 -ψ P 0 (r). (7.3.2)
The second inference channel is a non-linear channel, associated to the transition kernel P out . Suppose that V, W * i.i.d.

∼ N (0, 1) where V is known and the inference problem is to recover the unknown W * from the observation of

Y 0 ∼ P out • | √ q V + √ ρ -q W * , ( 7.3.3)
where ρ > 0, q ∈ [0, ρ]. Notice that under the representation (7.2.8), the channel (7.

3.3) is equivalent to Y 0 = ϕ( √ q V + √ ρ -q W * , A) + √
∆Z with ∆ ≥ 0 and where (A, Z) ∼ P A ⊗ N (0, 1), independently of V, W * . The free energy for this model is again related to the normalization of the posterior

P (W * | Y 0 , V ) Ψ Pout (q) def = E log P out Y 0 | √ q V + √ ρ -q w e -w 2 /2 √ 2π dw, ( 7.3.4)
where P out denotes either the transition density with respect to Lebesgue's measure (given by (7.2.9)) in the case where ∆ > 0, or the density with respect to the counting measure over N, in the case of discrete observations (ϕ takes values in N and ∆ = 0). The free energy (7.3.4) is again linked to the mutual information between the observation Y 0 and the signal W * , given V :

I Pout (q) def = I(W * ; Y 0 |V ) = Ψ Pout (ρ) -Ψ Pout (q). (7.3.5)
We will prove in Section 7.8 that Ψ Pout is convex, differentiable and non-decreasing.

Replica-symmetric formula and mutual information

We present in this section a limiting formula for F n . The result holds under the following hypotheses.

(h1) The prior distribution P 0 admits a finite third moment and has at least two points in its support. (h2) There exists γ > 0 such that the sequence (E

[|ϕ( 1 √ n [ΦX * ] 1 , A 1 )| 2+γ ]) n≥1 is bounded. (h3)
The random variables (Φ µi ) are independent with zero mean, unit variance and finite third moment that is bounded with n. (h4) For almost-all values of a ∈ R (w.r.t. P A ), the function x → ϕ(x, a) is continuous almost everywhere.

We will also assume that one of the two following hypotheses hold: The hypotheses are here stated using the "random function" representation of (7.2.8). In many cases, it can be useful to state them using the transition kernel representation of (7.2.1). The hypotheses (h2) and (h4) are respectively equivalent 2 to: Under the above hypothesis (h5.a) (respectively (h5.b)), the transition kernel P out admits a density with respect to Lebesgue's measure on R (resp. the counting measure on N) that will be denoted by P out (•|x).

Definition 7.3.1

We will say that the kernel P out is

• non-informative if for almost all y ∈ R (under (h5.a)) -or all y ∈ N (under (h5.b))-the map x → P out (y | x) is almost everywhere equal to a constant. We say that P out is informative if it is not non-informative.

• symmetric if P out (•|x) = P out (•| -x) for almost all x ∈ R.
If P out is non-informative, it is not difficult to show that estimation is impossible.

Let us define the following potential

F. Call ρ def = E[X 2
] where X ∼ P 0 . Then for (q, r) ∈ [0, ρ] × R ≥0 we define

F(q, r) def = ψ P 0 (r) + αΨ Pout (q) - rq 2 . (7.3.6)
We will also write F(ρ, +∞) = lim r→∞ F(ρ, r), which is well-defined in R ∪ {-∞} since r → ψ P 0 (r) -r ρ 2 is non-increasing by Proposition 1.3.3. We need also to define the set of the critical points of F:

Γ def = (q, r) ∈ [0, ρ] × (R ≥0 ∪ {+∞}) q = 2ψ P 0 (r) r = 2αΨ Pout (q; ρ) , (7.3.7)
where, with a slight abuse of notation, we extend ψ P 0 and Ψ Pout by their limits: ψ P 0 (+∞) = lim r→∞ ψ P 0 (r) = ρ/2 and Ψ Pout (ρ) = lim q→ρ Ψ Pout (q). This last limit is well defined in R ∪ {+∞} by convexity of Ψ Pout . The elements of Γ are called "fixed points of the state evolution", because -as we shall see in Section 7.4 -they are related to the fixed points of some generalized approximate message passing algorithm.

lim n→∞ F n = sup q∈[0,ρ] inf r≥0 F(q, r) = sup (q,r)∈Γ F(q, r) . (7.3.8)
Moreover, if P out is informative, then the "sup inf" and the supremum over Γ in (7.3.8) are achieved over the same couples (q, r).

An immediate corollary of Theorem 7.3.1 is the limiting expression of the mutual information between the signal and the observations.

Corollary 7.3.1 i ∞ def = lim n→∞ 1 n I(X * ; Y |Φ) = inf q∈[0,ρ] sup r≥0 i RS (q, r) = inf (q,r)∈Γ
i RS (q, r) , (7.3.9)

where i RS (q, r)

def = I P 0 (r) + αI Pout (q) - r 2 (ρ -q) . (7.3.10)
Proof. This follows from a simple calculation:

1 n I(X * ; Y |Φ) = 1 n H(Y |Φ) - 1 n H(Y |X * , Φ) = -F n + 1 n E log P (Y |X * , Φ) = -F n + m n E log P out (Y 1 | Φ 1 , X * / √ n) . (7.3.11)
By the central limit theorem (that we can apply under hypotheses (h1)-(h3)) we have

S n def = 1 √ n Φ 1 , X * = 1 √ n n i=1 Φ 1,i X i (d) ---→ n→∞ N (0, ρ) .
Now, under the hypotheses (h2)-(h4) and either (h5.a) or (h5.b) it is not difficult to verify that

E log P out (Y 1 | Φ 1 , X * / √ n) = E P out (dy|S n ) log P out (y|S n ) ---→ n→∞ E P out (dy| √ ρ V ) log P out (y| √ ρ V ) = Ψ Pout (ρ)
where V ∼ N (0, 1). We conclude, using (7.3.11):

1 n I(X * ; Y |Φ) = -F n + αΨ Pout (ρ) + o n (1). ( 7
.3.12)

Optimal errors

We compute in this section the optimal errors for both estimation and prediction tasks.

Both will be determined by the value of q ≥ 0 that optimizes (7.3.8) (or equivalently in (7.3.9)). Proposition 7.3.1

Define

D * def = α > 0 (7.3.8) (or equivalently (7.3.9)) admits a unique optimizer q * (α) . (7.3.13)

Assume that the assumptions of Theorem 7.3.1 hold and that P out is informative. Then the set D * is equal to R >0 minus some countable set. Moreover the map α → q * (α) is continuous on D * .

Optimal reconstruction (or estimation) error

We first consider the problem of estimating X * given Y and Φ. The following theorem states that the optimizer q * (α) of the replica-symmetric formula (7.3.8) gives the asymptotic correlation between the planted solution X * and a sample from the posterior distribution

P (• | Y , Φ): Theorem 7.3.2
Assume that all the moments of P 0 are finite and that P out is informative. Assume that (h1)-(h2)-(h3)-(h4) hold and that either (h5.a) or (h5.b) holds. Then for all α ∈ D * ,

1 n n i=1 x i X * i (d)
---→ n→∞ q * (α), (7.3.14) where x = (x 1 , . . . , x n ) is sampled from the posterior distribution of the signal

P (• | Y , Φ)
given by (7.2.3), independently of everything else. Moreover, if P out is not symmetric (see Definition 7.3.1) then:

1 n n i=1 x i X * i (d) ---→ n→∞ q * (α). ( 7
E | x(Y , Φ), X * | x(Y , Φ) X * ---→ n→∞ q * (α) ρ , ( 7.3.16)
where the supremum is taken over all measurable functions x : R m × R m×n → R n \ {0}.

Consequently q * (α)/ρ is the best correlation with the signal X * that one can asymptotically achieve.

Optimal generalization (or prediction) error

We now consider the prediction problem: given the dataset Φ, Y and a new point Φ new , we would like to estimate

Y new ∼ P out (•| Φ new , X * / √ n).
We define the generalization error of an estimator Y (Φ new , Φ, Y ) by

E gen ( Y ) def = E Y new -Y (Φ new , Φ, Y ) 2 .
(7.3.17)

The optimal generalization error is then defined as the minimum of E gen over all estimators Y :

E opt gen def = min Y E gen ( Y ) = E Y new -E[Y new |Φ new , Φ, Y ] 2 . (7.3.18)
In order to express the optimal generalization error we introduce the following function (recall that Y 0 , V, W * come from the channel (7.3.3)):

E(q) def = MMSE( Y 0 |V ) = E Y 0 -E[ Y 0 |V ] 2 (7.3.19) = E y 2 P out (dy| √ ρ V ) -E E W * yP out (dy| √ q V + √ ρ -q W * ) 2 (7.3.20)
where E W * denotes the expectation with respect to W * only. for all continuous bounded function f . ν n is therefore a random measure on R. Let ν be the posterior distribution of Y 0 given V in the second scalar channel (7.3.3), i.e.

f (y)dν(y) = E W * f (y)P out dy q * (α)V + ρ -q * (α)W *
for all continuous bounded function f , where E W * denotes expectation with respect to W * ∼ N (0, 1) only. Then, under the hypotheses of Theorem 7.3.2:

ν n (wd) ---→ n→∞ ν, where (wd)
---→ n→∞ denotes convergence in distribution of the random variables (ν n ) n≥1 to the random variable ν (See [START_REF] Kallenberg | Random measures, theory and applications[END_REF]Chapter 4] for more details on this mode of convergence).

In other words, the data Y , Φ, Φ new contains asymptotically as much information on Y new than V on Y 0 ∼ P out • q * (α)V + ρ -q * (α)W * . Theorem 7.3.4 is proved in Section 7.7.3. Theorem 7.3.4 implies Theorem 7.3.3 because:

E opt gen = E Var(Y new |Φ new , Φ, Y ) = E Var(ν n ) ---→ n→∞ E Var(ν) = E(q * (α)). (7.3.22)
To justify the above limit, we need to proceed by truncation since µ → Var(µ) is not a continuous bounded function over the set of probability measures. For a probability distribution µ on R and X ∼ µ we define for R > 0 the probability measure µ R as the law of sign(X) min(|X|, R), then µ → Var(µ R ) is a continuous (with respect to the weak topology) bounded function. By Theorem 7.3.

4 we get E Var(ν R n ) ---→ n→∞ E Var(ν R
) and (7.3.22) follows by letting R → ∞, using hypothesis (h2). Theorem 7.3.4 allows to compute more sophisticated optimal errors. Take for instance a classification problem with K ∈ N * classes (such as the one we studied in Section 7.1) where ϕ takes values in {1, . . . , K}. For these kind of problems, a natural error metric to consider for an estimator Y is the probability of misclassification:

P Y new = Y (Y , Φ, Φ new ) .
The best estimator for this error metric is known (see for instance [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF], Section 2.1) to be the Bayes classifier:

Y Bayes = arg max k∈{1,...,K} P(Y new = k | Y , Φ, Φ new ).
Notice that the corresponding probability of misclassification is equal to

P Y new = Y Bayes = E K k=1 ν n ({k})1 k = arg max i∈{1,...,K} ν n ({i}) = E 1 -max 1≤k≤K ν n ({k}) .
By Theorem 7.3.4 we deduce:

P Y new = Y Bayes ---→ n→∞ = E 1 -max 1≤k≤K ν({k}) = P Y 0 = Y Bayes 0 , where Y Bayes 0 is the Bayes classifier for estimating Y 0 ∼ P out • q * (α)V + ρ -q * (α)W * given V : Y Bayes 0 = arg max k∈{1,...,K} P( Y 0 = k | V ).

The generalized approximate message-passing algorithm

While the results presented until now are information-theoretic, the next one concerns the performance of a popular algorithm to solve random instances of generalized linear problems, called generalized approximate message-passing (GAMP). This approach has a long history, especially in statistical physics [START_REF] David J Thouless | Solution of'solvable model of a spin glass[END_REF][START_REF] Mezard | The space of interactions in neural networks: Gardner's computation with the cavity method[END_REF][START_REF] Kabashima | Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Baldassi | Efficient supervised learning in networks with binary synapses[END_REF], error correcting codes [START_REF] Richardson | Modern coding theory[END_REF], and graphical models [START_REF] Martin | Graphical models, exponential families, and variational inference[END_REF]. For a modern derivation in the context of linear models, see [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF][START_REF] Krzakala | Statistical-physics-based reconstruction in compressed sensing[END_REF][START_REF] Jeremy | Expectation-maximization gaussian-mixture approximate message passing[END_REF]. The case of generalized linear models was discussed by Rangan in [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF], and has been used for classification purpose in [START_REF] Ziniel | Binary linear classification and feature selection via generalized approximate message passing[END_REF]. We first need to define two so-called threshold functions that are associated to the two scalar channels (7.3.1) and (7.3.3). The first one is the posterior mean of the signal in channel (7.3.1) with signal-to-noise ratio r:

g P 0 (y, r) def = E[X 0 |Y 0 = y] . (7.4.1)
The second one is the posterior mean of W * in channel (7.3.3):

g Pout ( y, v, q) def = E[W * | Y 0 = y, √ q V = v] . (7.4.2)
These functions act componentwise when applied to vectors. Given initial estimates ( x 0 , v 0 ) -that we take equal to 0 in absence of additional information -for the means and variances of the elements of the signal vector X * , GAMP takes as input the observation vector Y and then iterates the following equations with initialization g 0 µ = 0 for all µ = 1, . . . , m (we denote by u the average over all the components of the vector u and Φ T is the transpose of the matrix Φ): From t = 1 until convergence,

                         V t = v t-1 ω t = Φ x t-1 / √ n -V t g t-1 g t µ = g Pout (Y µ , ω t µ , ρ -V t ) ∀ µ = 1, . . . m λ t = α g 2 Pout (Y , ω t , ρ -V t ) R t = x t-1 + (λ t ) -1 Φ T g t / √ n x t i = g P 0 (R t i , λ t ) ∀ i = 1, . . . n v t i = (λ t ) -1 ∂ R g P 0 (R, λ t )| R=R t i ∀ i = 1, . . . n. (7.4.3)
One of the strongest asset of GAMP is that its performance can be tracked rigorously in the limit n, m → ∞ while m/n → α via a procedure known as state evolution (SE), see [START_REF] Bayati | The dynamics of message passing on dense graphs, with applications to compressed sensing[END_REF][START_REF] Bayati | Universality in polytope phase transitions and message passing algorithms[END_REF] for the linear case, and [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF][START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference[END_REF] for the generalized one. The state evolution tracks the asymptotic value of the overlap between the true hidden value X * and its estimate by GAMP x t defined as q t def = lim n→∞ 1 n X * , x t via q 0 = 0 and q t+1 = 2ψ P 0 (r t ) , r t = 2αΨ Pout (q t ) . (7.4.4) From Theorem 7.3.1 we realize that the fixed points of these equations correspond to the critical points of the asymptotic free energy in (7.3.8). By Proposition 1.3.3 and Proposition 7.8.2 the functions Ψ P 0 and Ψ Pout are both non-decreasing. This gives that (q t , r t ) converges as t → ∞ to some couple (q AMP , r AMP ) ∈ Γ.

Perhaps more surprisingly, one can use GAMP for predicting a new output

Y new ∼ P out (• | Φ new , X * / √ n) where Φ new ∼ N (0, Id n ) is independent of X * , Φ, Y . As x t
is the GAMP estimate of the posterior expectation of X * , the natural heuristic is to consider for the posterior probability distribution (given Y , Φ) of the random variable Φ new , X * / √ n a Gaussian with mean Φ new , x t / √ n and variance ρ -q t . This allows to estimate the posterior mean of the output, which leads to the GAMP prediction:

Y GAMP,t def = y P out dy 1 √ n Φ new , x t-1 + ρ -q t w e -w 2 /2 √ 2π dw.
(7.4.5)

Estimation and generalization error of GAMP

The following claim, from [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF], gives the precise estimation error of GAMP. It is stated there as a claim because some steps of the proof are missing. The paper [START_REF] Javanmard | State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference[END_REF] affirms in its abstract to prove the claim of [START_REF] Rangan | Iterative estimation of constrained rank-one matrices in noise[END_REF], but without further details. For these reasons, we believe that the result holds, however we prefer to state it here as a claim (instead of a theorem).

Claim 7.4.1 (From [177])

We have almost surely for all t ∈ N, ∼ N (0, 1), and that x → P out (•|x) is continuous almost everywhere for the Wasserstein distance of order 2. Let t ∈ N. Assume that the limit (7.4.6) holds in probability and that there exists η > 0 such that E[| Y GAMP,t | 2+η ] remains bounded (as n grows). Then we have for all t ∈ N,

lim n→∞ 1 n x t , X * = lim n→∞ 1 n x t 2 = q t . ( 7
lim n→∞ E GAMP,t gen def = lim n→∞ E Y new -Y GAMP,t 2
= E(q t ) . (7.4.8) Proposition 7.4.1 is proved in Section 7.7.4. We see that this formula matches the one for the Bayes-optimal generalization error, see Theorem 7.3.3, up to the fact that instead of q * (α) (the optimizer of the replica formula (7.3.8)) appearing in the optimal error formula, here it is q t which appears. Thus clearly, when q t converges to q * (α) (we shall see in Section 7.5 that this is the case in many situations): GAMP is Bayes-optimal in a plethora of models (a task often believed to be intractable) and this for large sets of parameters.

Examples of phase transitions

We illustrate now the results of the previous sections to several models of interest in fields ranging from machine learning to signal processing, and unveil several interesting new phenomena in learning of generalized linear models. For various specific cases of prior P 0 and output P out , we evaluate numerically the free energy potential (7.3.6), its stationary points Γ and identify which of them gives the information-theoretic results, i.e. is the optimizer in (7.3.8). We also identify which of the stationary points corresponds to the result obtained asymptotically by the GAMP algorithm, i.e. the fixed point of the state evolution (7.4.4). Finally we compute the corresponding generalization error (7.4.8). We stress that in this section the results are based on numerical investigation of the resulting formulas: We do not aim at rigor that would involve precise bounds and more detailed analytical control for the corresponding integrals.

Generic observations

Using the functions g Pout and g P 0 introduced in Section 7.4 we can rewrite the fixed point equations (7.3.7) as

q = 2ψ P 0 (r) = E[g P 0 (Y 0 , r) 2 ] , (7.5.1) r = 2αΨ Pout (q) = α ρ -q E[g Pout ( Y 0 , √ q V, q) 2 ] , (7.5.2)
where the expectation in (7.5.1) corresponds to the scalar channel (7.3.1) and the expectation in (7.5.2) corresponds to the second scalar channel (7.3.3).

The non-informative fixed point: It is interesting to analyze under what conditions

q * = 0 is the optimizer of (7.3.8). Notice that q * = 0 corresponds to the error on the recovery of the signal as large as it would be if we had no observations at our disposition. Theorem 7.3.1 gives that any optimal couple (q * , r * ) of (7.3.8) should be a fixed point of the state evolution equations (7.5.1)-(7.5.2). A sufficient condition for (q, r) = (0, 0) to be a fixed point of (7.5.1)-(7.5.2) is that:

(a) The transition kernel P out is symmetric (see Definition 7.3.1).

(b) The prior P 0 has zero mean.

In order to see this, notice that if P out (y|z) is even in z then from the definition (7.4.2) of the function g Pout we have g Pout (y, 0, 0) = 0 and consequently from (7.5.2) we have Ψ Pout (0) = 0. For the second point, notice that we have ψ P 0 (0) = 1 2 E P 0 [X 0 ] 2 = 0. We assume now that P out is symmetric and that the prior P 0 has zero mean. In order for q = 0 to be the global maximizer q * of (7.3.8) or to be a relevant fixed point of the state evolution (7.4.4) (relevant in the sense that GAMP might indeed converge to it in a practical setting) we need q = 0 to be a stable fixed point of the equations (7.5.1)-(7.5.2). We obtain that q = 0 is stable if

2αΨ Pout (0) × 2ψ P 0 (0) = α dy Dz(z 2 -1)P out (y| √ ρz) 2 DzP out (y| √ ρz) < 1 , (7.5.3)
where Dz is the standard Gaussian measure. We conjecture that the condition (7.5.3) delimits precisely the region where polynomial-time algorithms do not perform better than "random guessing" (see the discussion below, where we will make this stability condition explicit for several examples of symmetric output channels). Note that the condition (7.5.3) also appears in a recent work [START_REF] Mondelli | Fundamental limits of weak recovery with applications to phase retrieval[END_REF] as a barrier for performance of spectral algorithms.

The exact recovery fixed point: Another particular fixed point of (7.5.1)-(7.5.2) is the one corresponding to exact recovery q = ρ. A sufficient and necessary condition for this to be a fixed point is that lim q→ρ Ψ Pout (q) = +∞, i.e.

Ψ Pout (ρ) -Ψ Pout (q) ρ -q --→ q→ρ +∞. (7.5.4)
Consider the second scalar channel (7.3.3) and let P V (respectively

P V,W * ) denote the conditional law of Y 0 given V (respectively V, W * ). Then we have Ψ Pout (ρ) -Ψ Pout (q) = E[D KL (P V , P V,W * )]
. If the statistical model (P out (•|θ)) θ∈R is regular (see Definitions 1 and 2 in [START_REF] Peter | Efficient and adaptive estimation for semiparametric models[END_REF]), and if we denote by J(θ) its Fisher information at θ, then we have for θ, θ 0 ∈ R

D KL (P out (•|θ 0 ), P out (•|θ)) = 1 2 (θ -θ 0 ) 2 J(θ 0 ) + o((θ -θ 0 ) 2 ).
This leads (after some computations) to

Ψ Pout (ρ) -Ψ Pout (q) = E[D KL (P V , P V,W * )] = 1 2 E[J( √ ρV )](ρ -q) + o q→ρ (ρ -q).
Consequently (7.5.4) can not hold for such model: perfect recovery is impossible.

In the case of a discrete deterministic channel, where P out (•|x) = δ ϕ(x) where ϕ : R → N, perfect reconstruction will sometimes be possible. Let us consider the "planted perceptron" problem from Section 7.1 where ϕ(x) = sign(x) (other functions ϕ can be treated analogously). In that case Ψ Pout (ρ) = 0 and

Ψ Pout (q) = 2E Φ q ρ -q Z log Φ q ρ -q Z = 2 ρ -q q +∞ -∞ e -ρ-q 2q x 2 Φ(x) log Φ(x)dx = -γ √ ρ -q + o q→ρ √ ρ -q ,
where γ = -2 √ ρ Φ(x) log Φ(x)dx. Hence Ψ Pout (q) ∼ γ(ρ -q) -1/2 : (7.5.4) holds, (ρ, +∞) ∈ Γ. We have now to verify if q = ρ is a stable fixed point of (7.4.4). Using the change of variables u = ρ -q we have to check whether u = 0 is a stable fixed point of the function

f (u) = ρ -ψ P 0 (αΨ Pout (ρ -u)) = MMSE P 0 (αΨ Pout (ρ -u)).
The stability of u = 0 depends now on the properties of P 0 through the behavior of MMSE P 0 (r) as r → ∞. This "large signal-to-noise ratio" asymptotic has been studied in details in [START_REF] Wu | Mmse dimension[END_REF] whose conclusions can be roughly summarized as follows:

MMSE P 0 (r) =    Θ(r -1 ) if P 0 has a "continuous part" o(r -1 ) if P 0 is "discrete".
We refer to the paper [START_REF] Wu | Mmse dimension[END_REF] for a precise definition of "continuous" and "discrete" and consider two examples below:

• P 0 = ρN (0, 1) + (1 -ρ)δ 0 , for some ρ ∈ (0, 1]. In that case, MMSE P 0 (r) ∼ ρ r and consequently f (u) ∼ γ -1 ρ √ u as u → 0. The fixed point u = 0 is therefore not stable: perfect reconstruction is not possible in this case. Indeed, as we will see on Figure 7.6 on page 132 (middle plot), the generalization error decreases but never reaches 0.

• P 0 = 1 2 δ -1 + 1 2 δ +1 .
In that case, MMSE P 0 (r) = o(r -2 ) and consequently f (u) = o(u) as u → 0. This time, the fixed point u = 0 is stable: perfect reconstruction is possible (provided that (ρ, +∞) maximizes F over Γ) as we saw it in Section 7.1.

Below we give several examples where exact recovery either is or is not possible, or where there is a phase transition between the two regimes.

Phase diagram of perfect learning

In this section we consider deterministic (noiseless) output channels and ask: How many measurements are needed in order to perfectly recover the signal?

As the number of samples (measurements) varies we encounter five different regimes of parameters:

• The tractable recovery phase: This is the region in the parameter space where GAMP achieves perfect reconstruction.

• The non-informative phase: Region where perfect reconstruction is informationtheoretically impossible and moreover even the Bayes-optimal estimator is as bad as a random guess based on the prior information and on the knowledge of the output function.

• The no recovery phase: Region where perfect reconstruction is information-theoretically impossible, but an estimator positively correlated with the ground truth exists.

• The hard phase: Region where the perfect reconstruction is information-theoretically possible, but where GAMP is unable to achieve it. At the same time, in this region GAMP leads to a better generalization error than the one corresponding to the noninformative fixed point. It remains a challenging open question whether polynomialtime algorithms can achieve perfect reconstruction in this regime.

• The hard non-informative phase: This phase corresponds to the region where perfect reconstruction is information-theoretically possible but where GAMP only achieves an error as bad as randomly guessing, given by the non-informative fixed point. In this phase as well, the existence of polynomial-time exact recovery algorithms is an open question.

The linear channel

The case of exact recovery of a sparse signal after it passed trough a noiseless linear channel, i.e. ϕ(x) = x, is studied in the literature in great details, especially in the context of compressed sensing [START_REF] Emmanuel | Near-optimal signal recovery from random projections: Universal encoding strategies[END_REF]. For a signal with a fraction ρ of non-zero entries, as soon as α > ρ, perfect reconstruction is theoretically possible, since the linear system Y = Φx admits almost surely a unique solution that has ρn non-zero entries, which is the signal X * . However solving Y = Φx over the space of ρn-sparse vectors remain (a priori) computationally difficult since it requires to test all the n ρn possible positions of the nonzero entries of x.

The whole field of compressed sensing builds on the realization that, using the 1 norm minimization technique:

x = arg min x | Y =Φx x 1 ,
one can efficiently recover the signal for α larger than a critical value α 1 (ρ):

α 1 (ρ) = min γ≥0 ρ(1 + γ 2 ) + 2(1 -ρ) (1 + γ 2 )Φ(-α) -αφ(α) . (7.5.5)
This is the so-called Donoho-Tanner transition [START_REF] David | Neighborliness of randomly projected simplices in high dimensions[END_REF][START_REF] David | High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension[END_REF][START_REF] David | Sparse nonnegative solution of underdetermined linear equations by linear programming[END_REF]. In our Bayesian context, when the empirical distribution of the signal is known, one can fairly easily beat the 1 transition and reconstruct the signal up to lower values of α using GAMP algorithm [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF][START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF][START_REF] Krzakala | Statistical-physics-based reconstruction in compressed sensing[END_REF][START_REF] Krzakala | Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices[END_REF]. In this case, three different phases are present [START_REF] Krzakala | Statistical-physics-based reconstruction in compressed sensing[END_REF][START_REF] Krzakala | Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices[END_REF]:

(i) For α < ρ, perfect reconstruction is impossible;

(ii) for ρ < α < α AMP reconstruction is possible, but not with any known polynomialcomplexity algorithm; (iii) for α > α AMP , the so-called spinodal transition computed with state evolution, GAMP provides a polynomial-complexity algorithm able to reach perfect reconstruction.

The line α AMP (ρ) depends on the distribution of the signal. For a Gauss-Bernoulli signal with a fraction ρ of non-zero (Gaussian) values we plot α 1 , α AMP and α IT (ρ) = ρ on Figure 7.4.

The rectified linear unit (ReLU) channel

Let us start by discussing the case of a generalized linear model with the ReLU output channel, i.e. ϕ(x) = max(0, x), with a signal coming from a Gauss-Bernoulli distribution P 0 = ρN (0, 1) + (1 -ρ)δ 0 , i.e. X * has a fraction ρ of non-zero (Gaussian) values. We are motivated by the omnipresent use of the ReLU activation function in deep learning, and explore its properties for GLMs that can be seen as a simple single layer neural network.

Our analysis shows that a perfect generalization (and thus a perfect reconstruction of the signal as well) is possible whenever the number of samples per dimension (measurement rate) α > 2ρ, and impossible when α < 2ρ. This is very intuitive, since half of the measurements (those non-zero) are giving as much information as in the linear case, thus the factor 2.

How hard is it to actually solve the problem with an efficient algorithm? The answer is given by applying the state evolution analysis to GAMP, which tells us that only for larger values of α, beyond the spinodal transition α AMP , does GAMP reach a perfect recovery. Notice, however, that this spinodal transition occurs at a significantly lower measurement rate α than one would reach just keeping the non-zero measurements. This shows that, actually, these zero measurements contain a useful information for the algorithm. The situation is shown in the center panel of Figure 7.5: The zero measurements do not help information-theoretically but they, however, do help algorithmically.

The sign-less channel

We now discuss the sign-less channel where only the absolute value of the linear mixture is observed, i.e. ϕ(x) = |x|. This case can be seen as the real-valued analog of the famous phase retrieval problem. We again consider the signal to come from a Gauss-Bernoulli distribution P 0 = ρN (0, 1) + (1 -ρ)δ 0 .

Sparse phase retrieval has been well explored in the literature in the regime where the number s of non-zeros is sub-leading in the dimension, s = o(n). This case is known to present a large algorithmic gap. While analogously to compressed sensing exact recovery is information-theoretically possible for a number of measurement Ω(s log(n/s)), best known algorithms achieve it only with Ω(s 2 / log n) measurements [START_REF] Oymak | Simultaneously structured models with application to sparse and low-rank matrices[END_REF], see also [START_REF] Soltanolkotabi | Structured signal recovery from quadratic measurements: Breaking sample complexity barriers via nonconvex optimization[END_REF] and references therein for a good discussion of other related literature. This is sometimes referred to as the "s 2 barrier". We are not aware of a study where, as in our setting, the sparsity is s = ρn and the number of measurements is αn with α and ρ of order 1. Perfect reconstruction is information-theoretically possible as soon as α > ρ: In other words, the problem is -information-theoretically-as easy, or as hard as the compressed sensing one. When α > ρ one can indeed perfectly reconstruct the signal by the following procedure: Try all 2 m choices of the possible signs for the m outputs, and solve a compressed sensing problem for each of them. Clearly, this should yields a perfect solution only in the case of the actual combination of signs.

Algorithmically, however, the problem is much harder than for the linear output channel. As shown in the left side of Figure 7.5 on page 131, for small ρ one requires a much larger fraction α of measurements in order for GAMP to recover the signal. For the linear channel the algorithmic transition α AMP (ρ) → 0 as ρ → 0 (see Figure 7.4) while for the sign-less channel we get α AMP (ρ) → 1/2 as ρ → 0. In other words if one looses the signs one cannot perform recovery in compressed sensing with less than n/2 measurements.

What we observe in this example for α < 1/2 is in the statistical physics literature on neural networks known as retarded learning [START_REF] Hansel | Memorization without generalization in a multilayered neural network[END_REF]. This appears in problems where the ϕ(x) function is symmetric, as seen in Section 7.5.1: There is always a critical point of the mutual information with an overlap value q = 0. For this problem, this critical point is actually "stable" (meaning that it is actually a local minimum in q in the mutual information (7.3.9)) for all α < 1/2, by (7.5.3), independently of ρ.

This has the two following implications:

(i) In the non-informative phase, when α < 1/2 and α < ρ, the minimum at q = 0 is actually the global one. In this case there is no useful information that one can The red line is the Bayes-optimal generalization error, while the green one shows the (asymptotic) performances of GAMP as predicted by the state evolution (SE), when different. For comparison, we also show the result of GAMP (black dots) and, in blue, the performance of a standard out-of-the-box solver, both tested on a single randomly generated instance. Left: Perceptron, with ϕ(x) = sign(x) and a Rademacher (±1) signal. The results of a logistic regression with fine-tuned ridge penalty are shown for comparison. Middle: Perceptron with Gauss-Bernoulli coefficients for the signal. The results of a logistic regression with fine-tuned 1 sparsity-enhancing penalty are very close to optimal. Right: The symmetric door activation rule with parameter K = K * 0.67449 chosen in order to observe the same number of occurrence of the two classes. Using Keras [START_REF] Chollet | keras[END_REF], a neural network with two hidden layers was able to learn approximately the rule, but only for much larger training set sizes (shown in inset).

exploit and no estimator can be better than a random guess. (ii) In the hard non-informative phase when ρ < α < 1/2, GAMP initialized at random, i.e. close to the q = 0 fixed point, will remain there. This suggests that in this region, even if a perfect reconstruction is information-theoretically possible, it will still be very hard to be better then a random guess with a tractable algorithm.

The symmetric door channel

The third output channel we study in detail is the symmetric door channel, where ϕ(x) = sign(|x| -K). In case of channels with discrete set of outputs exact recovery is only possible when the prior is also discrete. In the present case we take X * ∼ Unif({-1, 1} n ), i.e. P 0 = 1 2 δ +1 + 1 2 δ -1 . This channel was studied previously using the replica method in the context of optimal data compression [START_REF] Hosaka | Statistical mechanics of lossy data compression using a nonmonotonic perceptron[END_REF].

This output channel is in the class of symmetric channels for which overlap q = 0 is a fixed point. This fixed point is stable for α < α c (K). Exact recovery is informationtheoretically possible above α IT (K) and tractable with the GAMP algorithm above the spinodal transition α AMP (K). The values of these three transition lines are depicted in the right panel of Figure 7.5.

We note that α IT ≥ 1 is a generic bound on exact recovery for every K, required by a simple counting argument. While a-priori it is not clear whether this bound is saturated for some K, we observe that it is for K = K * 0.67449 defined by P(|Z| ≤ K * ) = 1/2, for Z ∼ N (0, 1). In that case half of the observed measurements are negative and the rest positive. The saturation of the α IT ≥ 1 bound was remarked previously in [START_REF] Hosaka | Statistical mechanics of lossy data compression using a nonmonotonic perceptron[END_REF]. However, we conjecture that this information-theoretic limit will not be achievable with known efficient algorithms since GAMP recovers the signal for α larger that α AMP > 1.

Examples of generalization errors

In this section we evaluate the Bayes-optimal generalization error (given by Theorem 7.3.3) and the generalization error of GAMP (given by Proposition 7.4.1) for several cases of priors and output functions. We study both regression problems, where the output is real-valued, and classification problems, where the output is discrete.

While in realistic regression and classification problems the matrix Φ corresponds to the data, and is thus not i.i.d. random, we view the practical interest of our theory as a benchmark for state-of-the art algorithms. The results of Sections 7.3 and 7.4 provide an exact asymptotic analysis of optimal generalization error and sample complexity for a range of simple rules. The challenge for state-of-the-art multi-purpose algorithms is to try to match as closely as possible the performance that can be obtained with GAMP that is fine-tuned to the specific form of the output and prior.

Threshold output: The perceptron

The example of non-linear output that is the most widely explored in the literature is the threshold output, where the deterministic output function is ϕ(x) = sign(x). In the setting of the present chapter, this is known as the perceptron problem [START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF], or equivalently, the one-bit compressed sensing in signal processing [START_REF] Petros | 1-bit compressive sensing[END_REF]. Its solution has been discussed in details within the replica formalism (see for instance [START_REF] Györgyi | First-order transition to perfect generalization in a neural network with binary synapses[END_REF][START_REF] Eric | The transition to perfect generalization in perceptrons[END_REF][START_REF] Opper | Generalization performance of bayes optimal classification algorithm for learning a perceptron[END_REF][START_REF] Kabashima | Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels[END_REF][START_REF] Xu | Bayesian signal reconstruction for 1-bit compressed sensing[END_REF]) and we confirm all of these heuristic computations within our approach.

In Figure 7.6 (left) we plot the optimal generalization error of the perceptron with a Rademacher signal, the state evolution prediction of the generalization error of the GAMP algorithm, together with the error actually achieved by GAMP on one randomly generated instance of the problem. We also compare these to the performance of a standard logistic regression. As expected from existing literature [START_REF] Gardner | Three unfinished works on the optimal storage capacity of networks[END_REF][START_REF] Györgyi | First-order transition to perfect generalization in a neural network with binary synapses[END_REF][START_REF] Hs Seung | Statistical mechanics of learning from examples[END_REF] we confirm that in this case the information-theoretic transition appears at a number of samples per dimension α IT = 1.249(1), while the algorithmic transition is at α AMP = 1.493 [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. Logistic regression does not seem to be able to match the performance on GAMP in this case.

In Figure 7.6 (center) we plot the generalization error for a Gauss-Bernoulli signal with density ρ = 0.2. Cases as this one were studied in detail in the context of one-bit compressed sensing [START_REF] Xu | Bayesian signal reconstruction for 1-bit compressed sensing[END_REF] and GAMP was found to match the optimal generalization performance with no phase transitions observed, which is confirmed by our analysis. In this case the logistic regression is rather close to the performance of GAMP.

Symmetric Door

The next classification problem, i.e. discrete output rule, that we study is the symmetric door function ϕ(x) = sign(|x| -K). In Figure 7.6 (right) we plot the generalization error for K = K * 0, 67449 such that 1/2 of the outputs are 1 and 1/2 are -1. The symmetric door output is an example of function for which the optimal generalization error for α < α IT = 1 (for that specific value of K, see phase diagram in the right panel of Figure 7.5) is as bad as if we were guessing randomly. The GAMP algorithm still achieves such a bad generalization until α stab 1.36 (defined by (7.5.3)), and achieves perfect generalization only for α > α AMP = 1.566 [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF].

Interestingly, labels created from this very simple symmetric door rule seem to be very challenging to learn for general purpose algorithms. We tried to optimize parameters of a two-layers neural network and only managed to get the performances shown in the inset The generalization error for three regression problems is plotted as a function of the number of samples per dimension α. The red line is again the Bayes-optimal generalization error, while the green one shows the (asymptotic) performances of GAMP as predicted by the state evolution (SE), when different. Again, we also show the result of GAMP on a particular instance (black dots) and, in blue, the performance of an out-of-the-box solver. Left: White Gaussian noise output and a Gauss-Bernoulli signal. For this choice of noise, there is no sharp transition (as opposed to what happens at smaller noises). The results of a Lasso with fine-tuned 1 sparsity-enhancing penalty are very close to optimal. Middle:

Here we analyze a ReLU output function ϕ(x) = max(0, x), still with a Gauss-Bernoulli signal. We show for comparison the results of maximum likelihood estimation performed with CVXPY -a powerful python-embedded language for convex optimization [START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF]-using two methods that are both amenable to convex optimization: In CVX-1 we use only the nonzero values of Y , and perform a minimization of the 1 norm of x subject to Y µ = Φ µ , x for µ ∈ {1, . . . , m} such that Y µ = 0, while in CVX-2, we use all the dataset, with the constraint that Y µ = Φ µ , x for µ ∈ {1, . . . , m} such that Y µ = 0 (as before) and the additional restriction Φ µ , x ≤ 0 for µ ∈ {1, . . . , m} such that Y µ = 0. In both case, a perfect generalization is obtained only for α 1. Right: The sign-less output function ϕ(x) = |x|. In inset, we show the performance for the estimation problem using PhaseMax [START_REF] Goldstein | Phasemax: Convex phase retrieval via basis pursuit[END_REF]. of Figure 7.6 (right). It is an interesting theoretical challenge whether a deeper neural network can learn this simple rule from fewer samples.

Linear regression

The additive white Gaussian noise (AWGN) channel, or linear regression, is defined by ϕ(x, A) = x + σA with A ∼ N (0, 1). This models the (noisy) linear regression problem, as well as noisy random linear estimation and compressed sensing. In this case (7.3.20) leads to lim n→∞ E opt gen = ρ -q * + σ 2 . (7.5.6)

This result agrees with the generalization error analyzed heuristically in [START_REF] Hs Seung | Statistical mechanics of learning from examples[END_REF] in the limit σ → 0. Figure 7.7 (left) depicts the generalization error for this example. The performance of GAMP in this case is very close to the one of Lasso, that we will study in mode details in the next chapter.

Rectified linear unit (ReLU)

In Figure 7.7 (center) we analyze the generalization error for the ReLU output function, ϕ(x) = max(0, x). This channel models the behavior of a single neuron with the rectified linear unit activation widely used in multilayer neural networks. For sparse Gauss-Bernoulli signals in Figure 7.7 (center) we observe again the informationtheoretic transition (at α = 2ρ = 0.4) to perfect generalization to be distinct from the algorithmic one (at α AMP = 0.589(1)). At the same time our test with existing algorithms were not able to closely match the performance of GAMP. This hence also remains an interesting benchmark.

Sign-less channel

In Figure 7.7 (right) we analyze the generalization error for the sign-less output function where ϕ(x) = |x|, that we already discussed in Section 7.5.2. The information-theoretic perfect recovery starts at α = ρ = 0.5, but the problem is again harder algorithmically for GAMP that succeeds only above α AMP = 0.90 [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. Again, the problem appears to be hard for other solvers. The state-of-the-art algorithm PhaseMax [START_REF] Goldstein | Phasemax: Convex phase retrieval via basis pursuit[END_REF] is for instance able to learn the rule using about four times as many measurements than needed informationtheoretically.

Proof of Theorem 7.3.1

This section is devoted to the proof of Theorem 7.3.1. We will do it under the following hypotheses:

(H1) The support of the prior distribution P 0 is included in [-S, S], for some S > 0. (H2) ϕ is a bounded C 2 function with bounded first and second derivatives w.r.t. its first argument. (H3) (Φ µi ) i.i.d.

∼ N (0, 1). (H4) ∆ > 0.

These stronger assumptions can then be relaxed to the weaker assumptions (h1)-(h2)-( h3)-(h4) and (h5.a) or (h5.b). This is done by approximation arguments similar to the ones of Section 3.4.7 for (H1) and (H2). The hypothesis (H3) is relaxed using the "generalized Lindeberg swapping trick" from [START_REF] Chatterjee | A generalization of the lindeberg principle[END_REF]. Finally the condition ∆ > 0 can be replaced by ((h5.b)) by approximation, using Corollary B.2. We refer to [START_REF] Barbier | Optimal errors and phase transitions in high-dimensional generalized linear models[END_REF] for the details of these arguments.

Since the observations (7.2.1) with P out given by (7.2.8), are equivalent to the rescaled observations

∆ -1/2 Y µ = ∆ -1/2 ϕ 1 √ n [ΦX * ] µ , A µ + Z µ , 1 ≤ µ ≤ m , (7.6.1)
the variance ∆ of the Gaussian noise can be "incorporated" inside the function ϕ. Thus, it suffices to prove Theorem 7.3.1 for ∆ = 1 and we suppose, for the rest of the proof, that we are in this case.

Interpolating estimation problem

We aim at computing the limit of the free energy F n . To do so, we introduce an estimation problem parametrized by t ∈ [0, 1] that interpolates between the original problem (7.2.1) at t = 0 and the two scalar problems described in Section 7.3.1 at t = 1 whose free energy is easy to compute. For t ∈ (0, 1) the interpolating estimation problem is a mixture of the original and scalar problems. This interpolation scheme is inspired from the interpolation paths used by Talagrand to study the perceptron, see [START_REF] Talagrand | Mean field models for spin glasses: Volume I: Basic examples[END_REF]. There are two major differences between the perceptron studied by Talagrand, and the "planted perceptron" (where there is a "planted solution" X * ) that we are investigating:

• In the planted case, the presence of a planted solution forces (under small perturbations) the correlations to vanish for all values of the parameters, see Chapter 2.

In the non-planted case, proving such decorrelation is much more involved, and is proved only in a limited region of the parameter space (the high-temperature phase), see [START_REF] Talagrand | Mean field models for spin glasses: Volume I: Basic examples[END_REF].

• However, in the planted case, there can be arbitrarily many solutions to the state evolution equations (7.3.7) (see Remark 21 in [START_REF] Wu | Optimal phase transitions in compressed sensing[END_REF]), whereas in the region studied by [START_REF] Talagrand | Mean field models for spin glasses: Volume I: Basic examples[END_REF], there is only one solution.

Let q, r : [0, 1] → R ≥0 be two continuously differentiable functions such that q(0) = r(0) = 0. Define

S t,µ def = 1 -t n [ΦX * ] µ + q(t) V µ + ρt -q(t) W * µ (7.6.2)
where

V µ , W * µ i.i.d.
∼ N (0, 1). Consider the following observation channels, with two types of observations obtained through

Y t,µ ∼ P out ( • | S t,µ ) , 1 ≤ µ ≤ m, Y t,i = r(t) X * i + Z i , 1 ≤ i ≤ n, (7.6.3)
where

(Z i ) n i=1 i.i.d.
∼ N (0, 1). We assume that V = (V µ ) m µ=1 is known. Then the inference problem is to recover both unknowns W * = (W * µ ) m µ=1 and X * = (X * i ) n i=1 from the knowledge of V , Φ and the observations Y t = (Y t,µ ) m µ=1 and Y t = (Y t,i ) n i=1 . Define u y (x) def = log P out (y|x) and, with a slight abuse of notations, for (x,

w) ∈ R n ×R m s t,µ = s t,µ (x, w µ ) def = 1 -t n [Φx] µ + q(t) V µ + ρt -q(t) w µ . (7.6.4)
Notice that S t,µ = s t,µ (X * , W * µ ). We introduce the interpolating Hamiltonian

H t (x, w; Y t , Y t , Φ, V ) def = - m µ=1 log P out (Y t,µ |s t,µ ) + 1 2 n i=1 Y t,i -r(t) x i 2 . (7.6.5)
The dependence in Φ and V of the Hamiltonian is through the (s t,µ ) m µ=1 . We also introduce the corresponding Gibbs bracket • n,t which is the expectation w.r.t. the posterior distribution of (X * , W * ) given (Y t , Y t , Φ, V ). It is defined as

g(x, w) n,t def = 1 Z t (Y t , Y t , Φ, V )
dP ⊗n 0 (x)Dw g(x, w) e -Ht(x,w;Yt,Y t ,Φ,V ) , (7.6.6)

for every continuous bounded function g on R n ×R m . In (7.6.6)

Dw = (2π) -m/2 m µ=1 dw µ e -w 2 µ /2
is the m-dimensional standard Gaussian distribution and Z t (Y t , Y t , Φ, V ) is the appropriate normalization (or partition function):

Z t (Y t , Y t , Φ, V ) def
= dP ⊗n 0 (x)Dw e -Ht(x,w;Yt,Y t ,Φ,V ) . (7.6.7)

Finally the interpolating free energy is

f n (t) def = 1 n E log Z t (Y t , Y t , Φ, V ) . (7.6.8)
Notice that:

f n (0) = F n f n (1) = ψ P 0 (r(1)) -1 2 (1 + ρr(1)) + m n Ψ Pout (q( 1
)) (7.6.9)

As discussed above, part of the potential (7.3.6) appears in f n [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. We would like to relate F n = f n (0) to f n [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. We thus compute the derivative of the free energy along the interpolation path (see Appendix 7.6.4 for the proof): Proposition 7.6.1

Assume that q ∞ , r ∞ = O(1) as n → ∞. For all t ∈ (0, 1)

f n (t) = - 1 2 E 1 n m µ=1 u Yt,µ (S t,µ )u Yt,µ (s t,µ ) -r (t) Q -q (t) n,t + r (t) 2 (q (t) -ρ) + o n (1) ,
where o n (1) is a quantity that goes to 0 as n → ∞, uniformly in t ∈ (0, 1). Recall that s t,µ = s t,µ (x, w µ ) is given by (7.6.4) where (x, w) is a sample from the posterior distribution of (X * , W * ) given Y t , Y t , Φ, V , independently of everything else. The expectation with respect to (x, w) only is denoted by the Gibbs bracket • n,t , see (7.6.6). Finally the overlap Q is

Q def = 1 n n i=1 X * i x i . ( 7
.6.10)

Overlap concentration

Proposition 7.6.1 is the analog of Lemma 4.3.1 that we have seen for the non-symmetric matrix estimation model in Chapter 4. As in the proof of Theorem 4.1.1 we will need to show that the overlap Q concentrates around its mean, in order to be able to cancel the first term of f n (t) (in Proposition 7.6.1) by choosing q (t) = E Q n,t . As seen in Section 2.2 Q concentrates around its mean on average over small perturbations of our model and we need in principle to introduce these small perturbations as in the proof of Theorem 4.1.1. However, in order to make the proof easier to read, we will assume here that for all choice of the functions r, q we have:

1 0 E Q -E Q n,t 2 n,t dt ---→ n→∞ 0. (7.6.11)
The reader is invited to refer to Proposition 4.3.2 or to the paper [START_REF] Barbier | Optimal errors and phase transitions in high-dimensional generalized linear models[END_REF] for a precise execution of the perturbation arguments. Proposition 7.6.2

Assume that (7.6.11) holds. Assume that we have q (t) = E Q n,t for all t ∈ [0, 1], and that r ∞ = O(1) as n → ∞. Then

F n = ψ P 0 (r(1)) + αΨ Pout (q(1)) - 1 2 1 0 q (t)r (t)dt + o n (1) , ( 7
.6.12)

where o n (1) denotes a quantity that goes to 0 as n → ∞.

Proof. By the Cauchy-Schwarz inequality

1 0 E 1 n m µ=1 u Yt,µ (S t,µ )u Yt,µ (s t,µ ) -r (t) Q -q (t) n,t dt 2 (7.6.13) ≤ 1 0 E 1 n m µ=1 u Yt,µ (S t,µ )u Yt,µ (s t,µ ) -r (t) 2 n,t dt × 1 0 E Q -q (t) 2 n,t dt.
Under assumptions (H2)-(H4) the first term of this product is bounded by some constant C that only depend on ϕ, α and r ∞ . The second term goes to 0 as n → ∞ by (7.6.11), since we assumed that for all t ∈ [0, 1], q (t) = E Q n,t . Consequently (7.6.13) goes to 0 as n → ∞. Therefore from Proposition 7.6.1:

f n (1) -f n (0) = 1 0 f n (t)dt = 1 0 q (t)r (t) -r (t)ρ dt + o n (1) . (7.6.14)
When combining (7.6.14) with (7.6.9) we reach the claimed identity.

Lower and upper matching bounds

We now possess all the necessary tools to prove Theorem 7. F(q, r). (7.6.15)

Proof. Let us fix r ≥ 0 and let us choose r(t) = rt and q to be the solution of the following order-1 differential equation (E Q n,t is indeed a function of q(t)):

q(0) = 1 and ∀t ∈ [0, 1], q (t) = E Q n,t .
We can then apply Proposition 7.6.2 (since q ∞ ≤ ρ and r ∞ = r are both O n (1)) to get

F n = ψ P 0 (r) + αΨ Pout (q(1)) - r 2 q(1) + o n (1) ≥ inf q∈[0,ρ] F(q, r) + o n (1)
and thus lim inf n→∞ F n ≥ inf q∈[0,ρ] F(q, r). This is true for all r ≥ 0 so we get Proposition 7.6.3.

Proposition 7.6.4

lim sup n→∞ F n ≤ sup r≥0 inf q∈[0,ρ]
F(q, r) . (7.6.16) Proof. We now chose (q, r) to be the solution of the following order-1 system of differential equations:

q(0) = 0 r(0) = 0 and ∀ t ∈ [0, 1], q (t) = E Q n,t r (t) = 2αΨ Pout E Q n,t .
(7.6.17)

As before, q ∞ ≤ ρ. By Proposition 7.8.1 the function Ψ Pout is C 1 and bounded, so r ∞ = O(1). We apply Proposition 7.6.2 to get

F n = ψ P 0 (r(1)) + αΨ Pout (q(1)) - 1 2 1 0 q (t)r (t)dt + o n (1) = ψ P 0 1 0 r (t)dt + αΨ Pout 1 0 q (t)dt - 1 2 1 0 q (t)r (t)dt + o n (1) ≤ 1 0 ψ P 0 (r (t)) + αΨ Pout (q (t)) - 1 2 q (t)
αΨ Pout (q (t)) - 1 2 q (t)r (t) = inf q∈[0,ρ] αΨ Pout (q) - 1 2 q r (t) .
Plugging this back in (7.6.18), we get:

F n ≤ sup r≥0 inf q∈[0,ρ] ψ P 0 (r) + αΨ Pout (q) - 1 2 q r + o n (1),
which proves Proposition 7.6.4.

Derivative of the interpolating free energy: Proof of Proposition 7.6.1

Recall that u y (x) is the x-derivative of u y (x) = log P out (y|x). Moreover denote P out (y|x) and P out (y|x) the first and second x-derivatives of P out (y|x). We will first prove that for all t ∈ (0, 1) (7.6.20) where recall

f n (t) = - 1 2 E 1 n m µ=1 u Yt,µ (S t,µ )u Yt,µ (s t,µ ) -r (t) Q -q (t) n,t + r (t) 2 (q (t) -ρ) - A n 2 ,
Q = 1 n n i=1 X * i x i and A n def = E 1 √ n m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) 1 √ n n i=1 (X * i ) 2 -ρ 1 n log Z t . (7.6.21)
Once this is done, we will prove that A n goes to 0 as n → ∞ uniformly in t ∈ [0, 1], in order to obtain Proposition 7.6.1.

Proof of (7.6.20) Recall definition (7.6.8) which becomes, when written as a function of the interpolating Hamiltonian (7.6.5),

f n (t) = 1 n
E dydy e -Ht(X * ,W * ;y,y ,Φ,V ) log dP ⊗n 0 (x)Dw e -Ht(x,w;y,y ,Φ,V ) . (7.6.22)

We will need the Hamiltonian t-derivative H t given by

H t (x, w; y, y , Φ, V ) = - m µ=1 ds t,µ dt u yµ (s t,µ ) - r (t) 2 r(t) n i=1
x i (y i -r(t)x i ) . (7.6.23) where we recall (7.6.4):

s t,µ = s t,µ (x, w µ ) def = 1 -t n [Φx] µ + q(t) V µ + ρt -q(t) w µ . (7.6.24)
The derivative of the interpolating free energy thus reads, for 0 < t < 1,

f n (t) = - 1 n E H t (X * , W * ; Y t , Y t , Φ, V ) log Z t T 1 - 1 n E H t (x, w; Y t , Y t , Φ, V ) n,t T 2 (7.6.25)
where recall that

Z t = Z t (Y t , Y t , Φ, V
) is given by (7.6.7). Let us compute T 1 . Let 1 ≤ µ ≤ m. Let us start with the following term

E dS t,µ dt u Yt,µ (S t,µ ) log Z t = 1 2 E -[ΦX * ] µ n(1 -t) + q (t) q(t) V µ + ρ -q (t) ρt -q(t) W * µ u Yt,µ (S t,µ ) log Z t . (7.6.26)
Let us compute the first term of the right-hand side of the last identity. By Gaussian integration by parts w.r.t Φ µi we obtain

1 n(1 -t) E [ΦX * ] µ u Yt,µ (S t,µ ) log Z t = 1 n(1 -t) n i=1
E dydy e -Ht(X * ,W * ;y,y ,Φ,V ) Φ µi X * i u Yt,µ (S t,µ ) log Z t (y, y , Φ, V )

= 1 n n i=1 E (X * i ) 2 u Yt,µ (S t,µ ) + u Yt,µ (S t,µ ) 2 log Z t + E X * i x i u Yt,µ (S t,µ )u Yt,µ (s t,µ ) n,t = E 1 n n i=1 (X * i ) 2 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) log Z t + E 1 n n i=1 X * i x i u Yt,µ (S t,µ )u Yt,µ (s t,µ ) n,t
, (7.6.27) where we used the identity

u Yt,µ (x) + u Yt,µ (x) 2 = P out (Y t,µ |x) P out (Y t,µ |x) .
We now compute the second term of the right hand side of (7.6.26). Using again Gaussian integrations by parts but this time w.r.t

V µ , W * µ i.i.d.
∼ N (0, 1) as well as the previous formula, we obtain similarly E q (t)

q(t) V µ + ρ -q (t) ρt -q(t) W * µ u Yt,µ (S t,µ ) log Z t = E dydy e -Ht(X * ,W * ;y,y ,Φ,V ) q (t) q(t) V µ + ρ -q (t) ρt -q(t) W * µ u Yt,µ (S t,µ ) log Z t = E ρ P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) log Z t + E q(t)
(X * i ) 2 -ρ log Z t + 1 2 E 1 n n i=1 X * i x i -q (t) u Yt,µ (S t,µ )u Yt,µ (s t,µ ) n,t
.

As seen from (7.6.23), (7.6.25) it remains to compute E[X * j (Y t,j -r(t)X * j ) log Z t ]. Recalling that for 1 ≤ j ≤ n, Y t,j -r(t)X * j = Z j and then using again a Gaussian integration by parts w.r.t Z j ∼ N (0, 1) we obtain

E X * j (Y t,j -r(t)X * j ) log Z t = E X * j Z j log Z t = E X * j ∂ Z j log Z t = -E X * j r(t)(X * j -x j ) + Z j n,t = -r(t) ρ -E X * j x j n,t . Thus, by taking the sum, - r (t) 2 r(t) E 1 n n i=1 X * i (Y t,i -r(t)X * i ) log Z t = r (t)ρ 2 - r (t) 2 E 1 n n i=1 X * i x i n,t . (7.6.29)
Therefore, for all t ∈ (0, 1),

T 1 = 1 2 E 1 √ n m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) 1 √ n n i=1 ((X * i ) 2 -ρ) 1 n log Z t + r (t)ρ 2 - r (t)q (t) 2 (7.6.30) + 1 2 E 1 n m µ=1 u Yt,µ (S t,µ )u Yt,µ (s t,µ ) -r (t) 1 n n i=1 X * i x i -q (t) n,t . ( 7

.6.31)

To obtain (7.6.20), it remains to show that T 2 = 0. This is a direct consequence of the Nishimori identity (Proposition 1.1.1):

T 2 = 1 n E H t (x, w; Y t , Y t , Φ, V ) n,t = 1 n E H t (X * , W * ; Y t , Y t , Φ, V ) = 0 . (7.6.32)
For obtaining the Lemma, it remains to show that A n goes to 0 uniformly in t ∈ [0, 1].

Proof that A n vanishes as n → ∞

We now consider the final step, that is showing that A n given by (7.6.21) vanishes in the n → ∞ limit uniformly in t ∈ [0, 1] under conditions (H1)-(H2)-(H3). First we show that

E 1 √ n m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) 1 √ n n i=1 (X * i ) 2 -ρ = 0 . (7.6.33)
Once this is done, we use the fact that 1 n log Z t concentrates around f n (t) to prove that A n converges to 0 as n → ∞. We start by noticing the simple fact that for all s ∈ R, P out (y|s)dy = 0. Consequently, for µ ∈ {1, . . . , m},

E P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) X * , S t = dyP out (y|S t,µ ) = 0 . (7.6.34)
Thus, using the "tower property" of the conditional expectation:

E n i=1 ((X * i ) 2 -ρ) m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) = E n i=1 (X * i ) 2 -ρ E m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) X * , S t = 0
which gives (7.6.33). We now show that A n goes to 0 uniformly in t ∈ [0, 1] as n → ∞.

Using successively (7.6.33) and the Cauchy-Schwarz inequality, we have

|A n | = E 1 √ n m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) 1 √ n n i=1 (X * i ) 2 -ρ 1 n log Z t -f n (t) (7.6.35) ≤ E 1 √ n m µ=1 P out (Y t,µ |S t,µ ) P out (Y t,µ |S t,µ ) 2 1 √ n n i=1 (X * i ) 2 -ρ 2 1/2 E 1 n log Z t -f n (t)
2 n i=1 (X * i ) 2 -ρ 2 ≤ mC E n i=1 (X * i ) 2 -ρ 2 = mnC Var (X * 1 ) 2 .
Going back to (7.6.36), therefore there exists a constant C > 0 such that

|A n | ≤ C E 1 n log Z t -f n (t) 2 1/2 . (7.6.41)
Using standard concentration arguments (bounded difference property, Gaussian Poincaré inequality) it is not difficult to show that under assumptions (H1), (H2), (H3) and (H4) there exists a constant C(ϕ, S, α, ∆, R) ≥ 0 such that for all functions q, r bounded by R we have

Var 1 n log Z t = E 1 n log Z t - 1 n E log Z t 2 ≤ C(ϕ, S, α, ∆, R) n . (7.6.42) Consequently, E[( 1 n log Z t -f n (t)) 2 ] → 0 as n → ∞ uniformly in t ∈ [0, 1]. Thus A n goes to 0 as n → ∞ uniformly in t ∈ [0, 1]
. This ends the proof of Proposition 7.6.1. αI Pout (q) + sup r≥0

Proofs of the limits of optimal errors

I P 0 (r) - r 2 (ρ -q) (7.7.1)
is concave (as an infimum of linear functions). Proposition D.2 gives that h is differentiable at α if and only if I Pout (q) q minimizer of (7.7.1) is a singleton. We assumed that P out is informative, so Proposition 7.8.4 gives that I Pout is strictly decreasing. We obtain thus that the set of points at which h is differentiable is exactly D * . Since h is concave, D * is equal to R >0 minus a countable set. Proposition D.2 gives also that h (α) = I Pout (q * (α)), for all α ∈ D * . The function h is concave, so its derivative h is non-increasing. Since I Pout is strictly decreasing, we obtain that α ∈ D * → q * (α) is non-decreasing. Let now α 0 ∈ D * . By concavity of h, h (α) → h (α 0 ) when α ∈ D * → α 0 . Therefore:

I Pout (q * (α)) -----→ α∈D→α 0 I Pout (q * (α 0 ))
which implies q * (α) → q * (α 0 ) by strict monotonicity of I Pout . The map α → q * (α) is therefore continuous on D * .

Limit of the overlap: Proof of Theorem 7.3.2

Let Q n denote the overlap Q n def = 1 n X * , x between X * and x = (x 1 , . . . , x n ) a sample from the posterior distribution P (X * | Y , Φ), independently of everything else. In this section we will show that |Q n | converges in probability to q * (α), for all α ∈ D * .

Upper bound on the overlap Proposition 7.7.1

For all α ∈ D * and for all > 0,

P |Q n | ≥ q * (α) + ---→ n→∞ 0 .
Let us fix α ∈ D * and let p ≥ 1. In order to obtain an upper bound on the overlap, we consider an observation model with some (small) extra information (that takes the form of a tensor of order 2p) in addition of the original model (7.2.1), i.e. we observe

   Y ∼ P out (• | ΦX * / √ n) , Y = λ n 2p-1 (X * ) ⊗2p + Z , (7.7.2)
where λ ≥ 0, Z = (Z i 1 ...i 2p ) 1≤i 1 ,...,i 2p ≤n i.i.d.

∼ N (0, 1) and (X * ) ⊗2p = (X i 1 . . . X i 2p ) 1≤i 1 ,...,i 2p ≤n . The next Proposition gives the limit of the mutual information between the signal and the observations (7.7.2).

Proposition 7.7.2

For all λ ≥ 0, lim n→∞

1 n I X * ; Y , Y Φ = I(λ) , (7.7.3)
where the right-hand-side is

I(λ) def = inf q∈[0,ρ] sup r≥0 I P 0 (r+2pλq 2p-1 ) + αI Pout (q) - r 2 (ρ -q) + 2p -1 2 λq 2p -ρpλq 2p-1 + λ 2 ρ 2p .
(7.7.4) Proposition 7.7.2 is proved using the same strategy than for Proposition 4.4.1, so we omit its proof for brevity. We are now in position to prove Proposition 7.7.1.

Proof of Proposition 7.7.1. The chain rule for the mutual information gives

I(X * ; Y , Y |Φ) = I(X * ; Y |Φ) + I((X * ) ⊗2p ; Y |Y , Φ).
Thus, using the I-MMSE relation of Proposition 1.3.3,

1 n ∂ ∂λ I X * ; Y , Y Φ λ=0 + = 1 n ∂ ∂λ I (X * ) ⊗2p ; Y Y , Φ λ=0 + = 1 2n 2p MMSE (X * ) ⊗2p Y , Φ .
By concavity of λ → I(X * ; Y , Y |Φ), we have using Proposition 7.7.2 and Proposition C.1:

lim inf n→∞ 1 2n 2p MMSE (X * ) ⊗2p Y , Φ = lim inf n→∞ 1 n ∂ ∂λ I X * ; Y , Y Φ λ=0 + ≥ I (0 + ) ,
For α ∈ D * and λ = 0 the infimum of (7.7.4) is achieved uniquely at q = q * (α), by definition of D * . Consequently, by Proposition D.2 we obtain I(0 + ) = 1 2 (ρ 2p -q * (λ) 2p ) and thus lim inf

n→∞ 1 n 2p MMSE (X * ) ⊗2p Y , Φ ≥ ρ 2p -q * (α) 2p .
One verifies easily that

1 n 2p MMSE (X * ) ⊗2p Y , Φ = ρ 2p -E Q 2p n + o n (1) , (7.7.5)
so we deduce that lim sup n→∞ E Q 2p n ≤ q * (α) 2p . Let > 0. By Markov's inequality we have

P |Q n | ≥ q * (α) + ≤ E Q 2p n (q * (α) + ) 2p .
By taking the lim sup in n on both sides we obtain lim sup

n→∞ P |Q n | ≥ q * (α) + ≤ q * (α) 2p (q * (α) + ) 2p ,
and Proposition 7.7.1 follows by taking the p → ∞ limit in the inequality above.

Limit of the overlap

Let us fix α ∈ D * . The sequence of the overlaps (Q n ) n≥1 is tight (because bounded in L 1 ). By Prokhorov's Theorem we know that the sequence of the laws of (Q n ) n≥1 is relatively compact. We can thus consider a subsequence along which it converges in law, to some random variable Q. In order to simplify the notations (and because working with an extraction does not change the proof) we will assume in the sequel that

Q n (d) ---→ n→∞ Q ,
for some random variable Q. We aim now at showing that |Q| = q * (α) almost surely and moreover, if P out is not symmetric, Q = q * (α) almost surely.

Lemma 7.7.1 (Upper bound on the overlap)

|Q| ≤ q * (α) almost surely.

Proof. Let > 0. The set [0, q * (α) + ] is closed, so by Portemanteau's Theorem

P |Q| ≤ q * (α) + ≥ lim sup n→∞ P |Q n | ≤ q * (α) + = 1 ,
by Proposition 7.7.1. So P |Q| ≤ q * (α) + = 1 for all > 0 which gives P |Q| ≤ q * (α) = 1.

In order to prove the converse lower bound, we will need to consider the following error quantity, for f : R → R a continuous bounded function:

E f,n (α) def = MMSE(f (Y new )|Φ new , Y , Φ) = E f (Y new ) -E[f (Y new )|Φ new , Y , Φ] 2 , (7.7.6)
which is the minimum mean-square error on f (Y new ). We define also

E f (q) def = MMSE(f ( Y 0 )|V ) = E f ( Y 0 ) -E[f ( Y 0 )|V ]
E f,n (α) ---→ n→∞ E f (q * (α)) (7.7.8)
where q * (α) is the optimizer of the replica-symmetric formula (7.3.8), see Proposition 7.3.1.

We first deduce Theorem 7.3.4 from Proposition 7.7.3. Proposition 7.7.3 will then be proved at the end of this Section. Proposition 7.7.

3 gives E f,n (α) ---→ n→∞ E f (q * (α)).
The function E f can be written as

E f (q) = 1 2 E h f √ qZ 0 + √ ρ -qZ 1 , √ qZ 0 + √ ρ -qZ 1 where Z 0 , Z 1 , Z 1 i.i.d.
∼ N (0, 1) and h f : (a, b) ∈ R 2 → (f (y 1 ) -f (y 2 )) 2 P out (dy 1 |a)P out (dy 2 |b). The next lemma follows from a simple central-limit-type argument (and can for instance be proved using Lindeberg "swapping trick", see [START_REF] Chatterjee | A generalization of the lindeberg principle[END_REF]).

Lemma 7.7.2

Let k ≥ 1. Let (s n ) n≥1 be a sequence of independent random variables that have zero mean, unit variance and finite third moment that is bounded with n. We write s n = (s 1 , . . . , s n ). For n ≥ 1 let x (1) n , . . . , x (k) n ∈ R n be some deterministic vectors. Assume that for all i ∈ {1, . . . , k},

1 n j (x (i) n,j ) 3 n≥1
is bounded and that for all i, j ∈ {1, . . . , k}

1 n x (i) n , x (j) n ---→ n→∞ R i,j ,
for some matrix R ∈ R k×k . Then

1 √ n s n , x ( ) n 1≤ ≤k (d) ---→ n→∞ N (0, R).
We deduce from Lemma 7.7.2 (that we apply for k = 2 conditionally on x, X * ):

x, Φ new √ n , X * , Φ new √ n (d) ---→ n→∞ (Z 1 , Z 2 ) , (7.7.9) where (Z 1 , Z 2 ) is sampled, conditionally on Q, from N 0, ρ Q Q ρ . Proposition 7.7.4
We have

E f,n (α) ---→ n→∞ 1 2 E h f (Z 1 , Z 2 ) ,
where (Z 1 , Z 2 ) is defined in Lemma 7.7.2 above.

Proof.

We have

E f,n = E f (Y new ) -E f (Y new ) Φ new , Φ, Y 2 = 1 2 E (f (y new ) -f (y)) 2 P out dy new Φ new , X * / √ n P out dy Φ new , x / √ n = 1 2 E h f x, Φ new √ n , X * , Φ new √ n .
By (7.7.9) above, we have

x,Φnew √ n , X * ,Φnew √ n (d) ---→ n→∞ (Z 1 , Z 2 )
. Using (h4) (and Remark 7.3.1) we can find a Borel set S ⊂ R of full Lebesgue's measure such that h f is continuous on S × S. The set of discontinuity points of h f has thus zero measure for the law of (Z 1 , Z 2 ). Indeed if we condition on Q:

• if |Q| < ρ, then (Z 1 , Z 2 ) has a density over R 2 . • if Q = ρ, then Z 1 = Z 2 almost surely, but h f is continuous on (s, s) s ∈ S that has full
Lebesgue's measure on the diagonal (x, x) x ∈ R .

• if Q = -ρ, then Z 1 = -Z 2 almost surely and we use then similar arguments as for the previous point.

We have therefore:

h f x, Φ new √ n , X * , Φ new √ n (d) ---→ n→∞ h f (Z 1 , Z 2 ) ,
and Lemma 7.7.4 follows from the fact that h f is bounded.

Let us now define:

H f : [-ρ, ρ] → R q → 1 2 E h f (G (q) ) (7.7.10)
where G (q) ∼ N 0, ( ρ q q ρ ) . Notice that H f is equal to the function E f on [0, ρ]. By Proposition 7.7.4 above and Proposition 7.7.3, we have: For all q ∈ (0, ρ], H f (-q) ≥ H f (q) with equality if and only if for almost-all x ∈ R we have f (y)P out (dy|x) = f (y)P out (dy| -x).

H f (q * (α)) = lim n→∞ E f,n (α) = E H f (Q) . ( 7
Proof. For x ∈ R we let P out (•|x) denote the law of f (Y ) for Y ∼ P out (•|x). Let q ∈ (0, ρ] and

Z 0 , Z 1 , Z 1 i.i.d.
∼ N (0, 1). Let Y and Y be two random variables, that are independent conditionally on Z 0 , Z 1 , Z 1 and distributed as:

Y ∼ P out (•| √ qZ 0 + √ ρ -qZ 1 ) and Y ∼ P out (•| - √ qZ 0 + √ ρ -qZ 1 ) .
Y and Y are independent conditionally on Z 0 , so we get

H f (-q) = 1 2 E (Y -Y ) 2 = 1 2 E (Y -E[Y |Z 0 ]) 2 + 1 2 E (Y -E[Y |Z 0 ]) 2 + 1 2 E (E[Y |Z 0 ] -E[Y |Z 0 ]) 2 = E (Y -E[Y |Z 0 ]) 2 + 1 2 E (E[Y |Z 0 ] -E[Y |Z 0 ]) 2 = H f (q) + 1 2 E (E[Y |Z 0 ] -E[Y |Z 0 ]) 2 .
We get that H f (-q) ≥ H f (q), with equality if and only if

E[Y |Z 0 ] = E[Y |Z 0 ] almost surely. This is equivalent to E f (y)P out (dy|v + √ ρ -qZ) = E f (y)P out (dy| -v + √ ρ -qZ), for a.e. v ∈ R,
where Z ∼ N (0, 1). Thus, if q = ρ then we are done. If q ∈ (0, ρ), writing

F (x) = f (y)P out (dy| √ ρ -qx),
and

g(x) = F (x) -F (-x), we get that G(v) def = E[g(v + Z)] = 0 for almost every v ∈ R. We compute the derivative of G: G (v) = E[Zg(v + Z)] = 0
. By Lemma 7.8.1, we get that g is equal to zero almost-everywhere, which concludes the proof.

We have now all the tools needed to prove Theorem 7.3.2. Using Lemma 7.7.3 and (7.7.11) above, we get that

EH f (|Q|) ≤ EH f (Q) = H f (q * (α)). Since H f is equal to E f on [0, ρ] this gives E E f (|Q|) ≤ E f (q * (α)) .
(7.7.12)

If q * (α) = 0, then Theorem 7.3.2 follows simply from Proposition 7.7.1. We suppose now that q * (α) > 0 and consider ∈ (0, q * (α)). We define p( ) = P |Q| ≤ q * (α) -. We are going to show that p( ) = 0. We assumed that P out is informative, so by Proposition 7.8.6 and Proposition 7.8.7 in Section 7.8, there exists a continuous bounded function f : R → R such that E f is strictly decreasing on [0, ρ]. In the following, f is assumed to be such a function. We have

E E f (|Q|) = E 1 |Q| ≤ q * (α) -E f (|Q|) + 1 |Q| > q * (α) -E f (|Q|) ≥ p( )E f (q * (α) -) + (1 -p( ))E f (q * (α))
.

because E f is non-increasing and because |Q| ≤ q * (α) almost surely (Lemma 7.7.1). Combining this with (7.7.12) leads to

p( )E f (q * (α)) ≥ p( )E f (q * (α) -) .
Since E f is strictly decreasing: E f (q * (α)) < E f (q * (α) -), which implies p( ) = 0. This is true for all > 0, consequently |Q| ≥ q * (α) almost surely. We get (using Lemma 7.7.1) that |Q| = q * (α) , almost surely.

We conclude that the only possible limit in law of the tight sequence

|Q n | n≥1 is q * (α). Therefore |Q n | → q * (α).
Let us prove the second part of Theorem 7.3.2. If q * (α) = 0, then this is obvious. Let us suppose now that q * (α) > 0 and let p = P(Q = -q * (α)). Suppose that p > 0. We have seen above that for all continuous bounded function f , EH f (Q) = H f (q * (α)). Hence

H f (q * (α)) = EH f (Q) = (1 -p)H f (q * (α)) + pH f (-q * (α)),
and consequently H f (q * (α)) = H f (-q * (α)). Lemma 7.7.3 gives then that f (y)P out (dy|x) = f (y)P out (dy| -x) for almost every x ∈ R, which then leads to P out (•|x) = P out (•| -x) for almost-every x ∈ R: P out is symmetric. We conclude that if P out is not symmetric, then necessarily p = 0 and Q = q * (α) almost surely.

Proof of Proposition 7.7.3

Let f : R → R be a continuous bounded function. Let , λ > 0. In order to get lowerbounds on the generalization error, we will consider the following additional observations

U µ = √ λ Y µ + Z µ , for 1 ≤ µ ≤ m = n , (7.7.13)
where

Z µ i.i.d.
∼ N (0, 1), and Y µ is given by ∼ N (0, Id n ) independently of everything else. We will use the following lemma, proved at the end of this section.

Y µ = f ( Y µ ) , Y µ ∼ P out • Φ µ , X * √ n , ( 7

Lemma 7.7.4

For all , λ ≥ 0 we have

1 n I(Y ; √ λY + Z |Y , Φ, Φ ) ---→ n→∞ inf q∈[0,ρ] sup r≥0 ĩRS (q, r, λ) -i ∞ , (7.7.15)
where i ∞ is given by Corollary 7.3.1 and ĩRS (q, r, λ)

def = i RS (q, r) + I(f (Y (q) ); √ λ f (Y (q) ) + Z |V ) (7.7.16) = I P 0 (r) + αI Pout (q; ρ) + I(f (Y (q) ); √ λ f (Y (q) ) + Z |V ) - r 2 (ρ -q) .
(7.7.17)

Recall that Y (q) is sampled from the "second scalar channel" (7.3.3):

Y (q) ∼ P out (• | √ q V + √ ρ -q W * ), where V, W * i.i.d.
∼ N (0, 1).

Lemma 7.7.5

For all α, λ > 0 the set

D α,λ def = ≥ 0 the infimum in (7.7.15
) is achieved at a unique q * α, ,λ (7.7.18) is equal to [0, +∞) minus some countable set. Moreover, → q * α,λ, is continuous on D α, .

Proof. This follows from the same arguments than in the proof of Proposition 7.3.1.

From there one can use the I-MMSE relation of Proposition 1.3.3:

∂ ∂λ

1 n I(Y ; √ λY + Z |Y , Φ, Φ ) = 1 2n MMSE Y √ λY + Z , Y , Φ, Φ = 2 MMSE(Y 1 |Y , U , Φ, Φ ).
We will express the limit of MMSE(Y 1 |Y , U , Φ, Φ ) using the function:

M f : (λ, q) → MMSE f (Y (q) ) √ λ f (Y (q) ) + Z , V . (7.7.19) Lemma 7.7.6
For all α, λ > 0, we have for all ∈ D α,λ \ {0}

lim n→∞ MMSE(Y 1 |Y , U , Φ, Φ ) = M f (λ, q * α, ,λ ) ,
where q * α, ,λ is the unique minimizer of (7.7.15).

Proof.

Let us fix α, > 0. Consider the function

h α, : λ → inf q∈[0,ρ] sup r≥0
ĩRS (q, r, λ) . (7.7.20) Corollary 4 from [START_REF] Milgrom | Envelope theorems for arbitrary choice sets[END_REF] gives that h α, is differentiable at λ if and only if

∂ ∂λ I f (Y (q) ); √ λf (Y (q) )+Z V = 2
M f (λ, q) q minimizer of (7.7.15) (or equivalently of (7.7.20))

is a singleton (the equality comes from the I-MMSE relation from Proposition 1.3.3). In such case, Corollary 4 from [START_REF] Milgrom | Envelope theorems for arbitrary choice sets[END_REF] also gives that

h α, (λ) = 2 M f (λ, q) , (7.7.21)
for all q minimizer of (7.7.20). So if now ∈ D α,λ \ {0}, then the minimizer is unique and thus h α, is differentiable at λ, with derivative h α,λ (λ) = M f (λ, q * α, ,λ )/2. However, by (7.7.15) in Lemma 7.7.4, h α, is the pointwise limit on R ≥0 of the sequence of concave functions ,U ,Φ,Φ )/2 and we conclude using the fact that = 0.

(h n ) n≥1 = λ → 1 n I Y ; √ λY + Z Y , Φ, Φ + i ∞ n≥1 . Consequently, Proposition C.1 gives h n (λ) ---→ n→∞ h α, (λ). By the I-MMSE relation (Proposi- tion 1.3.3) we have h n (λ) = MMSE(Y 1 |Y

Lemma 7.7.7

For all α ∈ D * (recall that D * is defined by (7.3.13)),

lim λ→0 lim →0 M f (λ, q * α, ,λ ) = E f (q * (α)) .
Proof. Let α ∈ D * and λ > 0. We have by definition of D α,λ , of D * and using the link between ĩRS and i RS given by (7.7.16), that 0 ∈ D α,λ . By Lemma 7.7.5 above, we have

q * α, ,λ --------→ →0, ∈D α,λ q * α,0,λ = q * (α) .
Analogously to Proposition 7.8.5, M f (λ, •) is continuous on [0, ρ], thus lim →0 M f (λ, q * α, ,λ ) = M f (λ, q * (α)). We obtain the result by taking lim λ→0 M f (λ, q * (α)) = E f (q * (α)), using the continuity of the MMSE at 0, see Proposition 1.3.2.

In order to simplify the proof, we assume that m = αn. By definition of the generalization error (7.7.6) and of the labels Y given by (7.7.14), we have

E f,n (α) = MMSE(Y 1 |Y , Φ, Φ ) . Lemma 7.7.8 For all α ∈ D * , lim inf n→∞ E f,n (α) ≥ E f (q * (α)) . Proof. Let α ∈ D * , λ > 0 and ∈ D α,λ \ {0}. Obviously, E f,n (α) ≥ MMSE(Y 1 |Y , U , Φ, Φ ) ---→ n→∞ M f (λ, q * α, ,λ ) ,
where we used Lemma 7.7.6. Consequently lim inf n→∞ E f,n (α) ≥ M f (λ, q * α, ,λ ) and we obtain the lower bound by letting , λ → 0 and using Lemma 7.7.7.

Let us now prove the converse upper bound.

Lemma 7.7.9

There exists a constant C > 0 (that only depend on f ) such that for all α, λ > 0 and all

∈ D α,λ \ {0} lim sup n→∞ E f,n (α + ) ≤ M f (λ, q * α, ,λ ) + Cλ .
Proof. We will let the signal-to-noise ratio (snr) of the observation of Y 1 go to zero. Let us denote by λ 1 this snr:

U 1 = √ λ 1 Y 1 + Z 1 .
We will let λ 1 go from λ to 0 while the other snr for the observations of U µ for µ = 2, . . . , n will remain equal to λ. Using Proposition 9 from [START_REF] Guo | Estimation in gaussian noise: Properties of the minimum mean-square error[END_REF],

∂ ∂λ 1 MMSE(Y 1 |Y , U , Φ, Φ ) = E Var(Y 1 |Y , U , Φ, Φ ) 2 ≤ E (Y 1 ) 4 ≤ f 4 ∞ .
We define

C def = f 4 ∞ .
Consequently, by the mean value theorem,

MMSE(Y 1 |Y , U , Φ, Φ ) -MMSE(Y 1 |Y , (U µ ) n µ=2 , Φ, Φ ) ≤ Cλ . (7.7.22)
Since (U µ ) n µ=2 contains less information than ( Y µ ) n µ=2 because of the additional Gaussian noise and the application of the function f , we have

MMSE(Y 1 |Y , (U µ ) n µ=2 , Φ, Φ ) ≥ MMSE(Y 1 |Y , ( Y µ ) n µ=2 , Φ, Φ ) = E f,n (α + -1/n) ≥ E f,n (α + ) .
The last identity combined with (7.7.22) leads to

MMSE(Y 1 |Y , U , Φ, Φ ) + Cλ ≥ E f,n (α + ) . ( 7
MMSE(Y 1 |Y , U , Φ, Φ ) = M f (λ, q * α, ,λ
). We conclude by taking the limsup in the inequality above.

Corollary 7.7.1

For all α ∈ D * , lim sup

n→∞ E f,n (α) ≤ E f (q * (α)) .
Proof. Let α ∈ D * , λ > 0 and 1 > 0 such that α -1 ∈ D * . Since by Lemma 7.7.5 the set

D α-1 ,λ is dense in R ≥0 , we can find 2 ∈ D α-1 ,λ such that 0 < 2 ≤ 1 .
Using Lemma 7.7.9 above, we have lim sup

n→∞ E f,n (α -1 + 2 ) ≤ M f (λ, q * α-1 , 2 ,λ ) + Cλ .
Now, using the fact that 2 ≤ 1 we have

lim sup n→∞ E f,n (α) ≤ lim sup n→∞ E f,n (α -1 + 2 ) ≤ M f (λ, q * α-1 , 2 ,λ ) + Cλ .
Now, by Lemma 7.7.7 we have

lim λ→0 lim 2 →0 M f (λ, q * α-1 , 2 ,λ ) + Cλ = E f (q * (α -1 ))
which leads to lim sup n→∞ E f,n (α) ≤ E f (q * (α -1 )). We conclude by letting 1 → 0 (recall that by Proposition 7.3.1 D * is dense in R ≥0 so it is possible to find 1 > 0 arbitrary small such that α -1 ∈ D * ), using the continuity of E f (by Proposition 7.8.5) and the continuity of q * (by Proposition 7.3.1).

Proof of Lemma 7.7.4.

Extending the interpolation method presented in Section 7.6, one can generalize Theorem 7.3.1 to take into account this additional side information. This gives directly

1 n I(X * ; Y , √ λY + Z |Φ, Φ ) ---→ n→∞ Ĩ∞ (α, , λ) def = inf q∈[0,ρ] sup r≥0
ĨRS (q, r, λ) (7.7.24) where ĨRS (q, r, λ) is given by ĨRS (q, r, λ)

def = I P 0 (r) + αI Pout (q; ρ) + I(W * ; √ λf (Y (q) ) + Z |V ) - r 2 (ρ -q) . (7.7.25)
Conditionally on (V, f (Y (q) )), the random variables W * and √ λf (Y (q) ) + Z are independent, therefore I f (Y (q) );

√ λf (Y (q) ) + Z V = I W * , f (Y (q) ); √ λf (Y (q) ) + Z V .
Now, by the chain rule of the mutual information we have

I W * , f (Y (q) ); √ λf (Y (q) )+Z V = I W * ; √ λf (Y (q) )+Z V +I f (Y (q) ); √ λf (Y (q) )+Z V, W * .
We obtain that

I W * ; √ λf (Y (q) ) + Z V = I f (Y (q) ); √ λf (Y (q) ) + Z V -I f (Y (q) ); √ λf (Y (q) ) + Z V, W * .
(7.7.26) Notice that the last mutual information in the above equation does not depend on q nor r. Therefore we have:

inf q∈[0,ρ] sup r≥0 ĨRS (q, r, λ) = -I f (Y (q) ); √ λf (Y (q) ) + Z V, W * + inf q∈[0,ρ] sup r≥0
ĩRS (q, r, λ) . (7.7.27) Now, by the chain rule, we have

1 n I(X * ; Y , √ λY + Z |Φ, Φ ) = 1 n I(X * ; Y |Φ) + 1 n I(X * ; √ λY + Z |Y , Φ, Φ ) . (7.7.28)
The limit of the left-hand side is given by (7.7.24). By Corollary 7.3.1, we have lim n→∞ I(X * ; Y |Φ)/n = i ∞ . It remains to investigate the last term of the equation above. By the arguments used to prove (7.7.26), we have

I(X * ; √ λY + Z |Y , Φ, Φ ) = I(Y ; √ λY + Z |Y , Φ, Φ ) -I(Y ; √ λY + Z |Y , Φ, X * , Φ ) = I(Y ; √ λY + Z |Y , Φ, Φ ) -I(Y ; √ λY + Z |X * , Φ ) . (7.7.29)
We have

I(Y ; √ λY + Z |X * , Φ )/n = I(Y 1 ; √ λY 1 + Z 1 |X * , Φ 1 )
and it is not difficult to show, using similar computations as in the proof of Corollary 7.3.1, that

I(Y 1 ; √ λY 1 + Z 1 |X * , Φ 1 ) ---→ n→∞ I f (Y (q) ); √ λf (Y (q) ) + Z V, W * ,
(recall that the right-hand side does not depend on q). Combining this with (7.7.29), (7.7.28), (7.7.24), Corollary 7.3.1 and (7.7.27), we obtain the desired result.

Optimal generalization error: Proof of Theorem 7.3.4

Let α ∈ D * . We will simply write q * instead of q * (α) in order to lighten the notations. For x ∈ R n we write Z(x) We are going to show that ν n f (d)

---→ n→∞ νf by the moment method. The sequence of the laws of (ν n f ) n≥1 is tight because bounded. Let us consider some subsequence (n l ) l≥1 along which it converges in distribution to some random variable S f . Let k ∈ N * .

E (ν n f ) k = E F (Z(x (1) )) • • • F (Z(x (k) )) ,
where x (1) , . . . , x (k) are i.i.d. samples from the posterior distribution of X * given Y , Φ, independently of everything else. We define the matrix

Q n = x (i) , x (j) 1≤i,j≤k .
We extract from (n l ) l≥1 another subsequence (n l ) l≥1 along which Q n converges weakly to some random variable Q. By a central-limit-type argument (Lemma 7.7.2), (Z(x (k) ), . . . , Z(x (k) ))

(d) ---→ n l →∞ (Z 1 , . . . , Z k )
where Z is sampled, conditionally on Q from N (0, Q). By hypothesis (h4) and Remark 7.3.1, F is continuous almost everywhere. We get:

E (ν n f ) k - → E F (Z 1 ) • • • F (Z k ) ,
where the limit is taken along the subsequence (n l ) mentioned above. On the other hand

E (νf ) k = E F ( √ q * V + √ ρ -q * W 1 ) • • • F ( √ q * V + √ ρ -q * W k ) = E F (Z 1 ) • • • F (Z k ) ,
where V, W 1 , . . . , W k i.i.d.

∼ N (0, 1) and Z ∼ N (0, R) where the matrix R ∈ R k×k is defined by R i,i = ρ and R i,j = q * , for all i = j.

By the Nishimori identity (Proposition 1.1.1) and the law of large numbers we have Q i,i = ρ almost surely for all i ∈ {1, . . . , k}. If P out is not symmetric, then by Theorem 7.3.2 we have Q = R almost surely which gives

E (ν n f ) k → E F (Z 1 ) • • • F (Z k ) = E (νf ) k .
If now P out is symmetric. By Theorem 7.3.2 we have for all i = j in {1, . . . , k}, Q i,j ∈ {-q * , q * } almost surely. So if q * = 0, then Q = R almost surely and we are done. Suppose that q * > 0. By Lemma 1.2.2 we have also for all i, j, ∈ {1, . . . , k} distinct, = Z i 1≤i≤k .

Q i,j Q j, Q ,i = (q * ) 3 ,
We assumed that P out is symmetric, therefore F (z) = F (|z|) for almost all z ∈ R. Thus F (η i Z i ) = F (Z i ) almost surely. This gives:

E (ν n f ) k → E F (Z 1 ) • • • F (Z k ) = E F (η 1 Z 1 ) • • • F (η k Z k ) = E F (Z 1 ) • • • F (Z k ) = E (νf ) k .
Recall that ν n f converges in distribution to S f along the subsequence (n l ). We get ES k f = E (νf ) k . This last equality holds for all k ≥ 1 and since νf is a bounded random variable we conclude that S f

Generalization error of GAMP: Proof of Proposition 7.4.1

Let us decompose: ∼ N (0, 1) and let E W denotes the expectation with respect to W only. We have

E GAMP,t gen def = E Y new -Y GAMP,t 2 = E Y 2 new + E Y GAMP,t 2 -2E Y new Y GAMP,
E Y new Y GAMP,t ---→ n→∞ E E W yP out (dy| q t V + ρ -q t W ) 2 .
(7.7.31)

Proof. Start by writing

E Y new Y GAMP,t = E y y P out dy Φ new , X * √ n P out dy Φ new , x t √ n + ρ -q t W
where W ∼ N (0, 1) is independent of everything else. Φ new ∼ N (0, Id n ) is independent of X * and x t , so, conditionally on X * , x t we have

Φ new , X * √ n , Φ new , x t √ n ∼ N 0, 1 n X * 2 x t ,X * x t ,X * x t 2 .
We assumed that (7.4.6) holds, i.e. X * , x t /n → q t and x t 2 /n → q t , in probability. By the law of large numbers X * 2 /n → ρ in probability. Consequently,

Φ new , X * √ n , Φ new , x t √ n (d) ---→ n→∞ N 0, ρ q t q t q t .
Since x → P out (•|x) is continuous almost everywhere for the Wasserstein distance of order 2, the function h : (a, b) → E W yy P out (dy|a)P out (dy |b + ρ -q t W ) with W ∼ N (0, 1) is continuous almost everywhere. Therefore

H n def = h Φ new , X * √ n , Φ new , x t √ n (d)
---→ n→∞ h( q t Z 0 + ρ -q t Z 1 , q t Z 0 ) , (7.7.32) where Z 0 , Z 1 i.i.d.

∼ N (0, 1). Let η > 0. We have by Jensen's inequality

E H n 1+η ≤ E Y new Y GAMP,t 1+η ≤ E 1 2 Y 2 new + 1 2 ( Y GAMP,t ) 2 1+η ≤ 1 2 E|Y 1 | 2+2η + 1 2 E Y GAMP,t 2+2η .
By assumption, there exists η > 0 such that the two last terms above remain bounded with n: H n is therefore bounded in L 1+η and is therefore uniformly integrable. From (7.7.32) we thus get

E Y new Y GAMP,t = E[H n ] ---→ n→∞ E h( q t Z 0 + ρ -q t Z 1 , q t Z 0 ) = E E W yP out (dy| q t V + ρ -q t W ) 2 .
Following the arguments of Lemma 7.7.10 one can also show that

E Y GAMP,t 2 ---→ n→∞ E E W yP out (dy| q t V + ρ -q t W ) 2 , E Y 2 new ---→ n→∞ E y 2 P out (dy| √ ρ V ) .
This proves (together with (7.7.30) and Lemma 7.7.10) Proposition 7.4.1.

The non-linear scalar channel

We prove here some properties of the free energy of the second scalar channel (7.3.3), where V, W * i.i.d. ∼ N (0, 1) and

Y (q) ∼ P out • √ q V + √ ρ -q W * . (7.8.1)
In this channel, the statistician observes V and Y (q) and wants to recover W * .

Study of the scalar free energy

We recall the definition of Ψ Pout :

Ψ Pout (q) = E log dw e -w 2 2 √ 2π P out Y (q) | √ q V + √ ρ -q w ,
where P out (•|x) denotes the density of P out (•|x) with respect to the Lebesgue's measure on R or the counting measure on N (we will always be in one of these two cases). In this section, we establish the main properties of Ψ Pout . Recall that I Pout (q) def = I(W * ; Y (q) |V ) = Ψ Pout (ρ) -Ψ Pout (q) so the properties we will prove on Ψ Pout can be directly translated for I Pout , and vice-versa. We will denote the expectation with respect the posterior distribution of W * given Y (q) , V by the Gibbs bracket • q : Suppose that for all x ∈ R, P out (• | x) is the law of ϕ(x, A) + √ ∆Z where ∆ > 0, ϕ : R×R → R is a measurable function and (Z, A) ∼ N (0, 1)⊗P A , for some probability distribution P A over R. In that case P out admits a density given by

f (w) q = f (w)e -w 2 /2 P out (Y (q) | √ q V + √ ρ -q w)dw e -w 2 /2 P out (Y (q) | √ q V + √ ρ -q w)dw . ( 7
P out (y|x) = 1 √ 2π∆ dP A (a)e -1 2∆ (y-ϕ(x,a)) 2 .
Assume that ϕ is bounded and C 2 with respect to its first coordinate, with bounded first and second derivatives. Then q → Ψ Pout (q) is convex, C 2 and non-decreasing on [0, ρ].

Proof. Under the hypotheses we made on ϕ, we will be able to use continuity and differentiation under the expectation, because all the domination hypotheses are easily verified.

We compute now the first derivative. Recall that • q , defined in (7.8.2), denotes the posterior distribution of W * given Y (q) . We will use the notation u y (x) = log P out (y|x). For q ∈ (0, ρ) we have

Ψ Pout (q) = 1 2 E u Y (q) ( √ q V + √ ρ -q w)u Y (q) ( √ q V + √ ρ -q W * ) q = 1 2 E u Y (q) ( √ q V + √ ρ -q w) 2 q ≥ 0 ,
where w ∼ • q , independently of everything else. Ψ Pout is therefore non-decreasing. Using the boundedness assumption on ϕ and its derivatives, it is not difficult to check that Ψ Pout is indeed bounded.

We will now compute Ψ Pout . To lighten the notations, we write u (w) for u Y (q) ( √ q V + √ ρ -q w). We compute

∂ q E u (w)u (W * ) q = E 1 2 √ q V - 1 2 √ ρ -q W * u (W * ) u (w)u (W * ) q (A) +2E 1 2 √ q V - 1 2 √ ρ -q W * u (W * )u (w) q (B) +E 1 2 √ q V - 1 2 √ ρ -q W * u (W * ) 2 u (w) q (C) -E u (W * )u (w) q 1 2 √ q V - 1 2 √ ρ -q w u (w) q (D) (7.8.3)
Notice that (A) = (C). Let w, w (1) , w (2) i.i.d. ∼ • q , independently of everything else. We compute, using Gaussian integration by parts and the Nishimori identity (Proposition 1.1.1) 1) )u (w (2) ) q = -E u (W * )u (w (1) )u (w (2) ) q -E u (W * )u (w (1) ) 2 u (w (2) ) q + E u (W * )u (w (1) )u (w (2) ) q u (w) q (7.8.6)

(A) = 1 2 E u (W * ) u (W * )u (w) q + 1 2 E u (W * ) u (W * )u (w) 2 q - 1 2 E u (W * ) u (W * )u (w) q u (w) q (7.8.4) (B) = E u (W * )u (w) q + E u (W * )u (w) 2 q -E u (W * )u (w) q u (w) q (7.8.5) (D) = -E 1 2 √ q V - 1 2 √ ρ -q W * u (W * )u (w (
We now replace (7.8.4), (7.8.5) and (7.8.6) in (7.8.3):

2Ψ Pout (q) = E u (W * ) 2 u (w) q + E u (W * ) 2 u (w) 2 q -E u (W * ) 2 u (w (1) )u (w (2) ) q + E u (W * )u (w) q + E u (W * )u (w) 2
q -E u (W * )u (w (1) )u (w (2) )

q -E u (W * )u (w (1) )u (w (2) ) q -E u (W * )u (w (1) ) 2 u (w (2) ) q + E u (w) 4 q .

Using the identity u

Y (x) + u Y (x) 2 = P out (Y |x)
Pout(Y |x) , this factorizes and gives Suppose that for all x ∈ R, P out (• | x) is the law of ϕ(x, A)+ √ ∆Z where ϕ : R×R → R is a measurable function and (Z, A) ∼ N (0, 1) ⊗ P A , for some probability distribution P A over R. Assume also that

Ψ Pout (q) = 1 2 E P out (Y | √ q V + √ ρ -q w) P out (Y | √ q V + √ ρ -qw) q -u Y (q) ( √ q V + √ ρ -q w) 2 q 2 ≥ 0 . ( 7
E[ϕ( √ ρZ, A) 2 ] < ∞ , (7.8.8)
and that we are in one of the following cases:

(i) ∆ > 0.

(ii) ∆ = 0 and ϕ takes values in N.

Then q → Ψ Pout (q) is continuous, convex and non-decreasing over [0, ρ].

Notice that (7.8.8) is for instance verified under hypotheses (h1)-(h2)-(h3)-(h4). Indeed, by the central limit theorem (that we apply under (h1)-(h3)), 1

√ n X * , Φ

---→ n→∞ N (0, ρ). Then using (h4) we get that

ϕ 1 √ n X * , Φ 1 , A 1 (d) ---→ n→∞ ϕ( √ ρZ, A).
Finally, by (h2) the sequence above is bounded in L 2 , hence is limit has a finite second moment which proves (7.8.8).

Proof. We deduce Proposition 7.8.2 from Proposition 7.8.1 above by an approximation procedure. Since Ψ Pout = Ψ Pout (ρ) -I Pout , we will work with the mutual information

I Pout . Let us define U (q) = ϕ √ q V + √ ρ -q W * , A and Y (q) = U (q) + √ ∆Z.
We start by proving Proposition 7.8.2 under the assumption (i). Let > 0. By density of the C ∞ functions with compact support in L 2 (see for instance Corollary 4.2.2 from [START_REF] Vladimir | Measure theory[END_REF]), one can find a C ∞ function ϕ with compact support, such that

E ϕ( √ ρ Z, A) -ϕ( √ ρ Z, A) 2 ≤ 2 .
Let us write

U (q) = ϕ( √ q V + √ ρ -q W * , A) and Y (q) = U + √ ∆Z.
We have by the chain rule for the mutual information

I(U (q) ; Y (q) |V ) = I(W * , U (q) ; Y (q) |V ) = I(U (q) ; Y (q) |V, W * ) + I(W * ; Y (q) |V ) = I(U (q) ; Y (q) |V, W * ) + I Pout (q)
and similarly, I Pout (q) = I( U (q) ; Y (q) |V ) -I( U (q) ; Y (q) |V, W * ). By Corollary B.1, there exists a constant C > 0 such that

|I( U (q) ; Y (q) |V ) -I(U (q) ; Y (q) |V )| ≤ C and |I( U (q) ; Y (q) |V, W * ) -I(U (q) ; Y (q) |V, W * )| ≤ C .
We get that for all q ∈ [0, ρ], |I Pout (q) -I Pout (q)| ≤ C . The function I Pout can therefore be uniformly approximated by continuous, concave, non-increasing functions on [0, ρ]: I Pout is therefore continuous, concave and non-increasing.

Let us now prove Proposition 7.8.2 under the assumption (ii). Under this assumption we have I Pout (q) = I(W * ; U (q) |V ) and by the case (i) we know that the function i ∆ (q) = I(W * ; U (q) + √ ∆Z|V ) is concave and non-increasing for all ∆ > 0. By Corollary B.2 we obtain that for all q ∈ [0, ρ] and all ∆ ∈ (0, 1] we have

I Pout (q) -i ∆ (q) ≤ 100e -1/(16∆) ,
which proves (by uniform approximation) that I Pout is continuous, concave and non-increasing.

Proposition 7.8.3

Under the same hypotheses than Proposition 7.8.2 above, Ψ out is differentiable over [0, ρ) and for all q ∈ [0, ρ)

Ψ Pout (q) = 1 2(ρ -q) E w 2 q ,
where we recall that • q is defined by (7.8.2).

Proof. The fact that Ψ Pout is differentiable on (0, ρ) follows from differentiation under the expectation sign. In order to see it, we define X = √ q V + √ ρ -q W * . Then, for all q ∈ [0, ρ):

Ψ Pout (q) = E dX e - (X- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (dy|X) log dx e - (x- √ q V ) 2 2(ρ-q)
2π(ρ -q) P out (y|x) . (7.8.9)

We are now in a good setting to differentiate under the expectation sign. We have for all q ∈ (0, ρ),

∂ ∂q    e - (X- √ q V ) 2 2(ρ-q) √ ρ -q    = e - (X- √ q V ) 2 2(ρ-q) 2 √ ρ -q 1 ρ -q - (X - √ qV ) 2 (ρ -q) 2 + V (X - √ qV ) √ q(ρ -q) . ( 7.8.10)
Thus

Ψ Pout (q) = 1 2 E    1 ρ -q - (X - √ q V ) 2 (ρ -q) 2 + V (X - √ q V ) √ q (ρ -q) log dx e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (Y |x)    + 1 2 E 1 ρ -q - (x - √ q V ) 2 (ρ -q) 2 + V (x - √ q V ) √ q (ρ -q) q
where the Gibbs brackets • q denotes the expectation with respect to x ∼ P (X|Y (q) , V ). The second term of the sum above is equal to zero. Indeed by the Nishimori identity (Proposition 1.1.1):

E 1 ρ -q - (x - √ q V ) 2 (ρ -q) 2 + V (x - √ q V ) √ q (ρ -q) q = E 1 ρ -q - (X - √ q V ) 2 (ρ -q) 2 + V (X - √ q V ) √ q (ρ -q) = 1 ρ -q E 1 -(W * ) 2 = 0 .
We now compute, by Gaussian integration by parts with respect to V ∼ N (0, 1):

E V (X - √ q V ) √ q (ρ -q) log dx 1 2π(ρ -q) e - (x- √ q V ) 2 2(ρ-q) P out (Y (q) |x) = E    -1 ρ -q log dx e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (Y (q) |x)    + E    (X - √ q V ) 2 (ρ -q) 2 log dx e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (Y (q) |x)    + E (X - √ q V )(x - √ q V ) (ρ -q) 2 q .
Bringing all together, we conclude:

Ψ Pout (q) = 1 2 E (X - √ q V )(x - √ q V ) (ρ -q) 2 q = 1 2(ρ -q) E w 2 q .
It remains to show that Ψ Pout (0) = 1 2ρ E w 2 0 = 0. This follows from taking the q → 0 limit in the equation above.

Proposition 7.8.4

Assume that the hypotheses of Proposition 7.8.2 hold and suppose also that the kernel P out is informative. Then Ψ Pout is strictly increasing on [0, ρ].

Proof. By contradiction we suppose that Ψ Pout is not strictly increasing on [0, ρ]. There exists thus q ∈ (0, ρ) such that Ψ Pout (q) = 0. By Proposition 7.8.3 this means that w q = 0 almost surely and therefore that

R P out (Y (q) | √ q V + √ ρ -q w)we -w 2 /2 dw = 0 almost surely. Let us write σ = √ ρ -q. Consequently, R P out (y | v + σw)we -w 2 /2 dw = 0 (7.8.11)
for almost all y in R (if we are under assumption (i)) or all y ∈ N (under assumption (ii)) and almost all v ∈ R. We will now use the following lemma: Lemma 7.8.1

Let Z ∼ N (0, 1) and let f : R → R be a bounded function. Suppose that for almost all

v ∈ R, E[Zf (v + Z)] = 0 .
Then, there exists a constant C ∈ R such that f (v) = C for almost every v.

Proof. Let us define the function

h : t → E[f (Z -t)] = 1 √ 2π f (x)e -(x+t) 2 /2 dx .
We have

h (t) = -1 √ 2π f (x)(x + t)e -(x+t) 2 /2 dx = -E[Zf (Z -t)]
= 0 and therefore h is equal to some constant C ∈ R. We are going to show that f = C almost everywhere. Without loss of generality we can assume that C = 0, otherwise it suffices to consider the function f = f -C. Now we have for all n ≥ 0, t ∈ R

0 = h (n) (t) = 1 √ 2π f (x) ∂ ∂t e -(x+t) 2 /2 dx = 1 √ 2π f (x)(-1) n H n (x + t)e -(x+t) 2 /2 dx ,
where H n is n th Hermite polynomial, defined as H n (x) = (-1) n e x 2 /2 d n dx n e -x 2 /2 . Therefore, for all n ≥ 0, f (x)H n (x)e -x 2 /2 dx = 0 , which implies that f = 0 almost everywhere since the Hermite functions form an orthonormal basis of L 2 (R).

We apply now Lemma 7.8.1 to (7.8.11) where the function f is given by f (x) = P out (y | σx). We thus obtain that for almost every y, P out (y | •) is almost everywhere equal to a constant: this contradicts the assumption that P out is informative.

Study of the generalization function

We turn now our attention to the study of the following "generalization function":

E f : [0, ρ] → R ≥0 q → E (f (Y (q) ) -E[f (Y (q) )|V ]) 2 (7.8.12)
where f : R → R is a continuous bounded function. We will prove that E f is continuous (Proposition 7.8.5) and strictly decreasing (Proposition 7.8.6) under the following hypotheses.

(a) For all x ∈ R,

P out (• | x) is the law of ϕ(x, A) + √ ∆ Z where ϕ : R × R → R is a measurable function and (Z, A) ∼ N (0, 1) ⊗ P A , for some probability distribution P A over R. (b) For almost all a ∈ R (w.r.t. P A ), ϕ(•, a) is continuous almost everywhere.
We suppose also that we are in one of the following cases:

Proof. Notice that

E f (q) = E f ϕ( √ ρ V, A) + √ ∆Z 2 -E V E W * ,Z,A f ϕ( √ q V + √ ρ -q W * , A) + √ ∆Z 2 .
The first term does not depend on q and the second one is continuous by Lebesgue's convergence theorem.

Proposition 7.8.6

Assume that the hypotheses of Proposition 7.8.5 hold. Suppose that x → f (y)P out (dy | x) is not almost-everywhere equal to a constant. Then E f is strictly decreasing on [0, ρ].

Proof. E f (q) = E[f (Y (q) ) 2 ] -E E[f (Y (q) )|V ] 2 .
Since the first term does not depend on q, it suffices to show that H :

q → E E[f (Y (q)
)|V ] 2 is strictly increasing on [0, ρ]. We have for q ∈ (0, ρ):

E[f (Y (q) )|V ] = f (y) e -w 2 /2 √ 2π P out (dy| √ q V + √ ρ -q w)dw = f (y) e - (x- √ qV ) 2 2(ρ-q)
2π(ρ -q) P out (dy|x)dx .

So we have, using (7.8.10):

∂ ∂q E[f (Y (q) )|V ] = f (y) 2 1 ρ -q - (x - √ q V ) 2 (ρ -q) 2 + V (x - √ q V ) √ q(ρ -q) e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (dy|x)dw = 1 2(ρ -q) E f (Y (q) ) 1 -W * 2 + √ ρ -q V W * √ q V .
We obtain

H (q) = 1 ρ -q E E[f (Y (q) )|V ] E f (Y (q) ) 1 -W * 2 + √ ρ -q V W * √ q V . ( 7.8.13)
We compute by Gaussian integration by parts:

E E[f (Y (q) )|V ] E f (Y (q) )V W * V = E V E[f (Y (q) )|V ] E f (Y (q) )W * V = E ∂ ∂V E[f (Y (q) )|V ] E f (Y (q) )W * V + E E[f (Y (q) )|V ] ∂ ∂V E f (Y (q) )W * V . (7.8.14)
We compute successively

∂ ∂V E[f (Y (q) )|V ] = ∂ ∂V f (y) e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (dy|x)dx = f (y) √ q (x - √ q V ) ρ -q e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (dy|x)dx = √ q √ ρ -q E f (Y (q) )W * V . ( 7.8.15)
∂ ∂V E[f (Y (q) )W * |V ] = ∂ ∂V f (y) x - √ q V √ ρ -q e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (dy|x)dx = f (y) - √ q √ ρ -q + √ q(x - √ q V ) 2 (ρ -q) 3/2 e - (x- √ q V ) 2 2(ρ-q) 2π(ρ -q) P out (dy|x)dx = √ q √ ρ -q E f (Y (q) ) -1 + W * 2 V . ( 7
H (q) = 1 ρ -q E E f (Y (q) )W * V 2 ≥ 0 .
Let us suppose now that H is not strictly increasing on [0, ρ]. This means that we can find q ∈ (0, ρ) such that H (q) = 0 and therefore E[f (Y (q) )W * |V ] = 0 almost surely. This gives that for almost all v ∈ R,

E W f (y)P out (dy| √ q v + √ ρ -q W ) = 0 ,
where E is the expectation with respect to W ∼ N (0, 1). Lemma 7.8.1 gives then that the function x → f (y)P out (dy|x) is almost everywhere equal to a constant: we obtain a contradiction. We conclude that H is strictly increasing on [0, ρ] and thus E f is strictly decreasing on [0, ρ].

Proposition 7.8.7

Assume that the hypotheses of Proposition 7.8.5 hold. If the channel P out is informative, then there exists a continuous bounded function f : R → R such that x → f (y)P out (dy|x) is not almost everywhere equal to a constant.

Proof. By contradiction, let us suppose that for all continuous bounded function

f : R → R we have f (y)P out (dy|x) = C f for almost all x ∈ R, for some constant C f ∈ R. Let X ∼ N (0, 1) and Y ∼ P out (•|X). We have then E[f (Y )|X] = C f = E[f (Y )
] almost surely. Let g : R → R be another continuous bounded function and compute:

E[g(X)f (Y )] = E g(X)E[f (Y )|X] = E[g(X)]E[f (Y )] .
It follows that X and Y are independent: The measures P out (dy|x) e -x 2 /2 √ 2π dx and E[P out (dy|X)] e -x 2 /2 √ 2π dx are therefore equal. Consequently, for almost every x, y we have

P out (y|x) = E[P out (y|X)] .
This gives that for almost every y, P out (y|•) is almost everywhere equal to a constant. We conclude that P out is not informative, which is a contradiction. In both plots, σ = 0.2. The true coefficients vector θ is chosen to be sN -sparse with s = 0.1. The entries on the support of θ are drawn i.i.d. N (0, 1). Cross-validation is carried out using 4 folds. SURE is computed using the estimator σ for the plot on the left, and the true value of σ on the right. Left: A standard random design with (X ij ) ∼ iid N (0, 1/n).

Risk MMSE minλ R * (λ) Lasso, λ = λ EST Lasso, λ = λ SURE Lasso, λ = λ k-CV Lasso, λ = σ √ 2 log N (a)
Risk Lasso, λ = λ EST Lasso, λ = λ SURE Lasso, λ = σ √ 2 log N Lasso, λ = λ k-CV minλ 1 N θ -θλ 2 (b) Correlated columns
Right: The rows of the design matrix X are i.i.d. Gaussian, with correlation structure given by an autoregressive process, see Eq. (8.4.4). Here we used φ = 2.

Unfortunately, this analysis provides limited insight into the choice of the regularization parameter λ which -in practice-can impact significantly the estimation accuracy. As an example, Figure 8.1.3 reports the result of a small simulation in which we compare four different methods of selecting λ. The bound of (8.1.3) suggests to set λ = σ √ c 0 log N . For the standard random design used in the left frame, the optimal constant is expected to be c 0 = 2 [START_REF] David | Minimax risk over p -balls for q -error[END_REF][START_REF] David L Donoho | The noise-sensitivity phase transition in compressed sensing[END_REF]. We compare this method to three procedures that adapt the choice of λ to the data: cross validation (CV), Stein's Unbiased Risk Estimate (SURE), and a procedure that minimizes an estimate of the risk (EST). We refer to the next sections for further details on these methods. Note that all of these adaptive procedures significantly outperform the 'theory driven' λ: over a broad range of sample sizes n, the resulting estimation error is 2 to 3 times smaller. Further, the error achieved by these methods is quite close to the Bayes optimum.

These empirical observations are not captured by the bound (8.1.3), or by similar results.

An alternative style of analysis postulates an idealized model for the data and derives asymptotically exact results. Throughout this paper we will consider the simplest of such models, by assuming that design matrix to have i.i.d. entries X ij ∼ N (0, 1/n). While this assumption is likely to be violated in practice, it allows to derive useful insights that are mathematically consistent, and susceptible of being generalized to a broader context. This type of analysis was first carried out in the context of the Lasso in [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF] and then extended to a number of other problems, see e.g. [START_REF] El | Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators: rigorous results[END_REF][START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF][START_REF] Donoho | High dimensional robust m-estimation: Asymptotic variance via approximate message passing[END_REF][START_REF] Thrampoulidis | Precise error analysis of regularized m-estimators in high-dimensions[END_REF][START_REF] El | On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators[END_REF][START_REF] Sur | A modern maximum-likelihood theory for high-dimensional logistic regression[END_REF]. As an example, Figure 8.1 reports the predictions of this analysis for the risk of the three adaptive procedure for selecting λ. The agreement with the numerical simulations is excellent.

Unfortunately, the results in [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF] (and in follow-up work) do not allow to derive in a mathematically rigorous way curves such as the ones in Figure 8.1. In fact earlier results hold 'pointwise' over λ and hence do not apply to adaptive procedures to select λ. Further they provide asymptotic estimates 'pointwise' over θ, and hence do not allow to compute -for instance-minimax risk.

In order to clarify these points, it is useful to overview informally the picture emerging from [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF][START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF]. Fix θ ∈ R N , λ ∈ R >0 , and let η(x; b) = (|x| -b) + sign(x) be the soft thresholding function. By the KKT conditions the Lasso estimator θ λ satisfies

θ λ = η θ d λ ; ατ , θ d λ = θ λ + ατ λ X T (y -X θ λ ) , ( 8.1.4)
where the vector θ d λ is also referred to as the 'debiased Lasso' [START_REF] Zhang | Confidence intervals for low dimensional parameters in high dimensional linear models[END_REF][START_REF] Van De Geer | On asymptotically optimal confidence regions and tests for high-dimensional models[END_REF][START_REF] Javanmard | Confidence intervals and hypothesis testing for high-dimensional regression[END_REF]. The above identity holds for arbitrary α, τ > 0. However, [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF] predicts that the distribution of the debiased estimator θ d λ simplifies dramatically for specific choices of these parameters. Namely, let Θ be a random variable with distribution given by the empirical distribution of (θ i ) i≤N (i.e., Θ = θ i with probability 1/N , for i ∈ {1, . . . , N }) and let Z ∼ N (0, 1) be independent of Θ. Define α * , τ * to be the solution of the following system of equations (we refer to Section 8.3.1 for a discussion of existence and uniqueness):

     τ 2 = σ 2 + 1 δ E (η(Θ + τ Z, ατ ) -Θ) 2 , λ = ατ 1 -1 δ P Θ + τ Z > ατ . (8.1.5)
When α, τ are selected in this way, θ d λ is approximately normal with mean θ (the true parameters vector) and variance τ 2 * : θ d ≈ N (θ , τ 2 * Id). More precisely, for any test function

f : R × R → R, with |f(x) -f(y)| ≤ L(1 + x + y ) x -y , almost surely, lim N →∞ 1 N N i=1 f(θ i , θ d λ,i ) = E f(Θ, Θ + τ * Z) , (8.1.6)
lim

N →∞ 1 N N i=1 f(θ i , θ λ,i ) = E f(Θ, η(Θ + τ * Z; α * τ * )) . (8.1.7)
This is an asymptotic result, which holds along sequences of problems with: (i) Converging aspect ratio n/N → δ ∈ (0, ∞); (ii) Fixed regularization λ ∈ (0, ∞); (iii) Parameter vectors θ = θ (n) whose empirical distribution converges (weakly) to a limit law p Θ . As emphasized above, this does not allow deduce the behavior of the Lasso with adaptive choices of λ (there could be deviations from the above limits for exceptional values of λ), or to compute the minimax risk (there could be deviations for exceptional vectors θ ). The importance of establishing uniform convergence with respect to the regularization parameter λ was recently emphasized by [START_REF] Mousavi | Consistent parameter estimation for lasso and approximate message passing[END_REF]. Among other results, these authors derive a uniform convergence statement for the related approximate message passing (AMP) algorithm. However, in order to establish uniform convergence, they have to construct an ad-hoc smoothing of the quantity of interest, which is roughly equivalent to discretizing the corresponding tuning parameter.

Our goal here is to obtain uniform (in λ) convergence results for the Lasso, hence providing a sound mathematical basis to the comparison of various adaptive procedures, as well as to the study of minimax risk.

The next sections organized as follows. Section 8.2 reviews related work. We state our main theoretical results in Section 8.3. In Section 8.4 we apply these results to two types of statistical questions: estimating the risk and noise level, and selecting λ through adaptive procedures. Further, we illustrate our results in numerical simulations. Finally, Section 8.5 outlines the main proof ideas, with the most technical legwork deferred to Chapter 9.

Related work

There is -by now-a substantial literature on determining exact asymptotics in highdimensional statistical models, and a number of mathematical techniques have been developed for this task. We will only provide a few pointers focusing on high-dimensional regression problems.

The original proof of [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF] was based on an asymptotically exact analysis of an approximate message passing (AMP) algorithm [START_REF] Bayati | The dynamics of message passing on dense graphs, with applications to compressed sensing[END_REF] that was first proposed in [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF] to minimize the Lasso cost function. Variants of AMP have been developed in a number of contexts, opening the way to the analysis of various statistical estimation problems. A short list includes generalized linear models [START_REF] Rangan | Generalized approximate message passing for estimation with random linear mixing[END_REF], phase retrieval [START_REF] Schniter | Compressive phase retrieval via generalized approximate message passing[END_REF][START_REF] Ma | Optimization-based amp for phase retrieval: The impact of initialization and 2 -regularization[END_REF], robust regression [START_REF] Donoho | High dimensional robust m-estimation: Asymptotic variance via approximate message passing[END_REF], logistic regression [START_REF] Sur | A modern maximum-likelihood theory for high-dimensional logistic regression[END_REF], generalized compressed sensing [START_REF] Berthier | State evolution for approximate message passing with non-separable functions. Information and Inference[END_REF]. This approach is technically less direct than others, but has the advantage of providing an efficient algorithm, and is and not necessarily limited to convex problems (see [START_REF] Montanari | Non-negative principal component analysis: Message passing algorithms and sharp asymptotics[END_REF] for a non-convex example).

As mentioned above, our work was partially motivated by the recent results of [START_REF] Mousavi | Consistent parameter estimation for lasso and approximate message passing[END_REF] that establish a form of uniformity for the AMP estimates -but not for the Lasso solution. It would be interesting to understand whether the approach of [START_REF] Mousavi | Consistent parameter estimation for lasso and approximate message passing[END_REF] could also be used to obtain uniform results for the Lasso or other statistical estimators.

Here we follow a different route that exploits powerful Gaussian comparison inequalities first proved by Gordon [START_REF] Gordon | Some inequalities for gaussian processes and applications[END_REF][START_REF] Gordon | On milman's inequality and random subspaces which escape through a mesh in rn[END_REF]. Gordon inequality allows to bound the distribution of a minimax value, i.e. the value of a random variable G * = min i≤N max j≤M G ij , where (G ij ) i≤N,j≤M is a Gaussian process, in terms of a similar quantity for a 'simpler' Gaussian process. The use of Gordon's inequality in this context was pioneered by Stojnic [START_REF] Stojnic | A framework to characterize performance of lasso algorithms[END_REF] and then developed by a number of authors in the context of regularized regression [START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF], M-estimation [START_REF] Thrampoulidis | Precise error analysis of regularized m-estimators in high-dimensions[END_REF], generalized compressed sensing [START_REF] Amelunxen | Living on the edge: Phase transitions in convex programs with random data[END_REF], binary compressed sensing [START_REF] Stojnic | Recovery thresholds for 1 optimization in binary compressed sensing[END_REF] and so on. The key idea is to write the optimization problem of interest as a minimax problem, and then apply a suitable version of Gordon's inequality. A matching bound is obtained by convex duality and then a second application of Gordon's inequality. In particular, convexity of the cost function of interest is a crucial ingredient.

While the Gaussian comparison inequality provides direct access to the value of the optimization problem, understanding the properties of the estimator can be more challenging. We identify here a property (that we call local stability) that allows to transfer information on the minimum (the Lasso cost) into information about the minimizer (the Lasso estimator). We believe that this strategy can be applied to other examples beyond the Lasso.

Independently, a different approach based on leave-one-out techniques was developed by El Karoui in the context of ridge-regularized robust regression [START_REF] El | Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators: rigorous results[END_REF][START_REF] El | On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators[END_REF].

Finally, a parallel line of research determines exact asymptotics for Bayes optimal estimation, under a model in which the coordinates of θ are i.i.d. with common distribution p Θ . In particular, the asymptotic Bayes optimal error for linear regression with random designs was recently determined in [START_REF] Barbier | The mutual information in random linear estimation[END_REF][START_REF] Reeves | The replica-symmetric prediction for compressed sensing with gaussian matrices is exact[END_REF] and is also a Corollary of Theorem 7.3.2 from the previous chapter. Of course -in general-Bayes optimal estimation requires knowledge of the distribution p Θ , and is not computationally efficient. We will use this Bayes-optimal error as a benchmark of our adaptive procedures.

Main results

Definitions

As stated above, we consider the standard linear model (8.1.2) where y = Xθ + σz , with noise z ∼ N (0, Id n ), and X a Gaussian design: (X i,j ) i≤n,j≤N i.i.d.

∼ N (0, 1/n). The Lasso estimator is defined by

θ λ = arg min θ∈R N L λ (θ) . ( 8.3.1)
(The minimizer is almost surely unique since the columns of X are in generic positions.)

We set δ = n/N to be the number of samples per dimension. We are interested in uniform estimation over sparse vectors θ . Following [START_REF] David | Minimax risk over p -balls for q -error[END_REF][START_REF] Johnstone | Function estimation and gaussian sequence models[END_REF] we formalize this notion using p -balls (which are convex sets only for p ≥ 1).

Definition 8.3.1

Define for p, ξ > 0 the p -ball

F p (ξ) = x ∈ R N 1 N N i=1 |x i | p ≤ ξ p , and for s ∈ [0, 1] F 0 (s) = x ∈ R N x 0 ≤ sN .
By Jensen's inequality we have for p ≥ p > 0, F p (ξ) ⊂ F p (ξ).

Let φ(x) = e -x 2 /2 √ 2π be the standard Gaussian density and Φ(x) = x -∞ φ(t)dt be the associated cumulative function. In the case of 0 balls (sparse vectors), a crucial role is played by the following sparsity level.

Definition 8.3.2

Define the critical sparsity as

s max (δ) = δ max α≥0    1 -2 δ (1 + α 2 )Φ(-α) -αφ(α) 1 + α 2 -2 (1 + α 2 )Φ(-α) -αφ(α)    .
See Figure 7.4 for a plot of s max . The critical sparsity curve first appears in the seminal work by Donoho and Tanner on compressed sensing [START_REF] David | Neighborliness of randomly projected simplices in high dimensions[END_REF][START_REF] David | High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension[END_REF]. These authors consider the noiseless case (z = 0) of model (8.1.2) and reconstruction via 1 minimization (which corresponds to the λ → 0 limit of the Lasso). They prove that 1 minimization reconstructs exactly θ with high probability, if θ 0 ≤ N (s max (δ) -ε), and fails with high probability if θ 0 ≥ N (s max (δ) + ε) (for any ε > 0). A second interpretation of the critical sparsity s max (δ) was given in [START_REF] David L Donoho | The noise-sensitivity phase transition in compressed sensing[END_REF][START_REF] Tropp | Convex recovery of a structured signal from independent random linear measurements[END_REF][START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF]. For θ 0 ≤ N (s max (δ) -ε), the Lasso achieves stable reconstruction. Namely, there exists

M = M (s, δ) < ∞ for s < s max (δ), such that, if θ 0 ≤ N s, θ λ -θ 2 ≤ M (s, δ)σ 2 .
Our results provide a third interpretation: uniform limit laws for the Lasso will be obtained on 0 balls only for s < s max (δ).

A crucial role in our results is provided by the following max-min problem:

max β≥0 min τ ≥σ ψ λ (β, τ ) , (8.3.2) ψ λ (β, τ ) def = σ 2 τ + τ β 2 - 1 2 β 2 + 1 δ E min w∈R w 2 2τ β -βZw + λ|w + Θ| -λ|Θ| .
The expectation above is with respect to (Θ, Z) ∼ µ θ ⊗ N (0, 1), where µ θ denotes the empirical distribution of the entries of the vector θ :

µ θ = 1 N N i=1 δ θ i . Proposition 8.3.1
The max-min (8.3.2) is achieved at a unique couple (β * (λ), τ * (λ)). Moreover, (β * (λ), τ * (λ)) is also the unique couple (β, τ ) ∈ (0, +∞) 2 that verify

     τ 2 = σ 2 + 1 δ E (η(Θ + τ Z, τ λ β ) -Θ) 2 β = τ 1 -1 δ E η (Θ + τ Z, τ λ β ) . (8.3.3)
We will also use the notation α * (λ) = λ/β * (λ) and

s * (λ) = E η (Θ + τ * (λ)Z, τ * (λ)α * (λ)) = P |Θ + τ * (λ)Z| ≥ α * (λ)τ * (λ) . ( 8.3.4)
We will sometimes omit the dependency on λ and write simply α * , β * , τ * , s * . The distribution µ * λ defined below will correspond (see Theorem 8.3.1 in the next section) to the limit of the empirical distribution of the entries of ( θ λ , θ ). Definition 8.

3.3

Let (Θ, Z) ∼ µ θ ⊗ N (0, 1). We denote by µ * λ the law of the couple

η Θ + τ * (λ)Z, α * (λ)τ * (λ) , Θ .

Results

We fix from now on 0 < λ min ≤ λ max and D ⊂ R N that can be either F p (ξ) for some ξ, p > 0, or F 0 (s) for some s < s max (δ). Our uniformity domain is defined by Ω = δ, σ, D, λ min , λ max . Namely, we will control θ λ uniformly with respect to θ ∈ D and λ ∈ [λ min , λ max ], with n/N = δ. We will call constant any quantity that only depends on Ω. In absence of further specifications, C, c will be constants (that depend only on Ω) that are allowed to change from one line to another. Our first result shows that the empirical distribution of the entries {( θ λ,i , θ i )} i≤N is uniformly close to the model µ * λ . We quantify deviations using the Wasserstein distance. Recall that, given two probability measures µ, ν on R d with finite second moment, their Wasserstein distance of order 2 is

W 2 (µ, ν) = inf γ∈C(µ,ν)
x -y 2 2 γ(dx, dy)

1/2 , ( 8.3.5)
where the infimum is taken over all couplings of µ and ν. Note that W 2 metrizes the convergence in Eq. (8.1.7). Namely lim n→∞ W 2 (µ n , µ * ) = 0 if and only if, for any test function [START_REF] Villani | Optimal transport: old and new[END_REF]. It provides therefore a natural way to extend earlier results to a non-asymptotic regime.

f : R × R → R, with |f(x) -f(y)| ≤ L(1 + x + y ) x -y , we have lim n→∞ f(x)µ n (dx) = f(x)µ * (dx)

Theorem 8.3.1

Assume that D = F p (ξ) for some ξ > 0 and p > 0. Then there exists constants C, c > 0 that only depend on Ω, such that for all ∈ (0, 1 2 ] sup

θ ∈D P sup λ∈[λ min ,λmax] W 2 µ ( θ λ ,θ ) , µ * λ 2 ≥ ≤ C -max(1,a)-1 N (1/p-1) + exp -cN 2 a log( ) -2 ,
where a = • Uniformity with respect to λ. We bound (in probability) the maximum (over λ) deviation between the empirical distribution µ ( θ λ ,θ ) and the predicted distribution µ * λ . (The supremum over λ is 'inside' the probability.)

• Uniformity with respect to θ . We bound the maximum probability (over θ ) of a deviation between µ ( θ λ ,θ ) and µ * λ . (The supremum over θ is 'outside' the probability.)

The reader might wonder whether it is possible to strengthen this result and bound the maximum deviation over θ ('move the supremum over θ inside'). The answer is negative.

In particular, we can choose the support of θ to coincide with a submatrix of X with atypically small minimum singular value. This will result in larger estimation error θ λθ 2 , and hence in a large Wasserstein distance W 2 ( µ ( θ λ ,θ ) , µ * λ ).

Remark 8.3.2. Note that Theorem 8.3.1 does not hold for 0 balls. This is probably a fundamental problem, since controlling W 2 distance uniformly over 0 balls is impossible even in the simple sequence model (or, equivalently, for orthogonal designs X). Namely, consider the case in which we observe

y i = θ i + z i , i ≤ N , where (z i ) i≤N i.i.d.
∼ N (0, τ 2 * ), and we try to estimate θ by computing θ λ,i = η(y i ; λ). Then there are vectors θ ∈ F 0 (s) such that the empirical law µ ( θ λ ,θ ) does not concentrate in Wasserstein distance around its expectation µ * λ , i.e. the law of (Θ, η(Θ + Z; λ)) for G ∼ N (0, τ * ). In order to see this, it is sufficient to consider the vector θ = (N, 2N, . . . , kN, 0, . . . , 0) . In Section 9.6.1 of Chapter 9, we prove that (for this choice of θ ) there exists a constant c 0 such that W 2 ( µ ( θ λ ,θ ) , µ * λ ) ≥ k/N with probability at least 1 -e -c 0 k for all N large enough.

We can think of several possibilities to overcome this intrinsic non-uniformity over 0 balls. One option would be to consider a weaker notion of distance between probability measures. Here we follow a different route, and prove uniform estimates over 0 balls for several specific quantities of interest. In order to state these results, we introduce the following quantities, which correspond to the risk and the prediction error (and are expressed in terms of the solution (τ * , β * ) of ( 8

.3.3)) R * (λ) = δ τ * (λ) 2 -σ 2 , ( 8.3.6)
P * (λ) = β * (λ) 2 + 2σ 2 δ s * (λ) - σ 2 δ . (8.3.7) Theorem 8.3.2
Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0.

There exists constants C, c > 0 that only depend on Ω, such that for all ∈ (0, 1] sup

θ ∈D P sup λ∈[λ min ,λmax] 1 N θ λ -θ 2 -R * (λ) 2 ≥ ≤ C 2 N q e -cN 2 , ( 8.3.8)
sup

θ ∈D P sup λ∈[λ min ,λmax] 1 n y -X θ λ 2 -β * (λ) 2 2 ≥ ≤ C 2 N q e -cN 2 , ( 8.3.9)
sup

θ ∈D P sup λ∈[λ min ,λmax] 1 n X(θ -θ λ ) 2 -P * (λ) 2 ≥ ≤ C 2 N q e -cN 2 , ( 8.3.10)
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ).
The statement (8.3.8) is proved in Section 9.3.2, while (8.3.9)-(8.3.10) are proved in Section 9.4 of Chapter 9.

So far we focused on the Lasso estimator θ λ . The debiased Lasso estimator is defined as

θ d λ = θ λ + X T (y -X θ λ ) 1 -1 n θ λ 0 .
This estimator plays a crucial role in the construction of confidence intervals and pvalues [START_REF] Zhang | Confidence intervals for low dimensional parameters in high dimensional linear models[END_REF][START_REF] Van De Geer | On asymptotically optimal confidence regions and tests for high-dimensional models[END_REF][START_REF] Javanmard | Confidence intervals and hypothesis testing for high-dimensional regression[END_REF][START_REF] Takahashi | A statistical mechanics approach to de-biasing and uncertainty estimation in lasso for random measurements[END_REF], and provide an explicit construction of the 'direct observations' model in the sense that θ d λ is approximately distributed as N (θ , τ 2 * Id). We let µ

(d)
λ be the law of the couple Θ + τ * (λ)Z, Θ , where (Θ, Z) ∼ μθ ⊗ N (0, 1).

Theorem 8.3.3

Let µ ( θ d λ ,θ ) denote the empirical distribution (on R 2 ) of the entries of ( θ d λ , θ ). There exists constants c, C > 0 such that for all ∈ (0, 1],

sup θ ∈F 4 (ξ) P sup λ∈[λ min ,λmax] W 2 ( µ ( θ d λ ,θ ) , µ (d) λ ) ≥ ≤ C 11 e -cN 17 .
Theorem 8.3.3 is proved in Section 9.6.5 of Chapter 9.

Applications

Estimation of the risk and the noise level

In order to select the regularization parameter and to evaluate the quality of the Lasso solution θ λ , it is useful to estimate the risk and noise level. The paper [START_REF] Bayati | Estimating lasso risk and noise level[END_REF] developed a suite of estimators of these quantities based on the asymptotic theory of [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF]. The same paper also proposed generalizations of these estimators to correlated designs. Here we revisit these estimators and prove stronger guarantees. First, we obtain quantitative bound on the consistency rate of our estimators. Second, our results are uniform over λ, which justifies using these estimators to select λ.

Let us start with the estimation of τ * (λ) which plays a crucial role in the asymptotic theory. We define

τ (λ) = √ n y -X θ λ n -θ λ 0 .
We will see with Theorem 9.6.1 presented in Section 9.6.4 of Chapter 9 that lim

N,n→∞ 1 N θ λ 0 = P(|Θ + τ * Z| ≥ τ * λ/β * ) def = s * (λ) .
Further, by Theorem 8.3.2, we have 1

√ n y -X θ λ = β * (λ) + o n (1). Recall that by (8.3.3) we have β * (λ) = τ * (λ) 1 -1 δ s * (λ) . We deduce τ (λ) = τ * (λ) + o n (1)
. More precisely we have the following consistency result.

Corollary 8.4.1

Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0. There exists constants C, c > 0 that only depend on Ω such that for all ∈ (0, 1] sup

θ ∈D P sup λ∈[λ min ,λmax] | τ (λ) -τ * (λ)| ≥ ≤ C -6 N q exp -cN 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ).
We next consider estimating the 2 error of the Lasso. Following [START_REF] Bayati | The lasso risk for gaussian matrices[END_REF], we define

R(λ) = τ (λ) 2 2 N θ λ 0 -1 + X T (y -X θ λ ) 2 N 1 -1 n θ λ 0 2 .

Corollary 8.4.2

Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0.

There exists constants C, c > 0 such that for all ∈ (0, 1],

sup

θ ∈D P sup λ∈[λ min ,λmax] R(λ) - 1 N θ λ -θ 2 ≥ ≤ C 6 N q e -cN 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ
1 N θ λ -θ 2 δ(τ * (λ) 2 - σ 2 ) δ( τ (λ) 2 -σ 2 ), the estimator σ 2 (λ) = τ (λ) 2 - N n R(λ) = τ (λ) 2 1 + N n - 2 n θ λ 0 - X T (y -X θ λ ) 2 n 1 -1 n θ λ 0 2 (8.4.1)
is a consistent estimator of the noise level σ 2 .

Corollary 8.4.3

There exists constants C, c > 0 that only depend on Ω, such that for all ∈ (0, 1] sup

θ ∈D P sup λ∈[λ min ,λmax] σ 2 (λ) -σ 2 > ≤ C 6 N q e -cN 6 .
Finally, we consider the prediction error Xθ -X θ λ . Stein Unbiased Risk Estimator (SURE) provides a general method to estimate the prediction error, see e.g. [START_REF] Charles M Stein | Estimation of the mean of a multivariate normal distribution[END_REF][START_REF] Efron | The estimation of prediction error: covariance penalties and crossvalidation[END_REF][START_REF] Ryan J Tibshirani | Degrees of freedom in lasso problems[END_REF]. In the present case, it takes the form

P SURE (λ) = 1 n y -X θ λ 2 + 2σ 2 n θ λ 0 . ( 8.4.2)
Tibshirani and Taylor [START_REF] Ryan J Tibshirani | Degrees of freedom in lasso problems[END_REF] proved that P SURE (λ) is an unbiased estimator of the prediction error, namely

E P SURE (λ) = 1 n Xθ -X θ λ 2 + σ 2 . ( 8.4.3)
The next result establishes consistency, uniformly over λ and θ , with quantitative concentration estimates. Corollary 8.4.4

Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0.

There exists constants C, c > 0 that only depend on Ω such that for all ∈ (0, 1] sup

θ ∈D P sup λ∈[λ min ,λmax] 1 n Xθ -X θ λ 2 + σ 2 -P SURE (λ) ≥ ≤ C 6 N q e -cN 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ).
The same result holds if σ in (8.4.2) is replaced by an estimator of the noise level satisfying the same consistency condition as σ defined by (8.4.1) (cf. Corollary 8.4.3). This corollary follows simply from Theorem 9.6.1 from Chapter 9 and Theorem 8.3.2. Remark 8.4.1. Notice that exact unbiasedness of P SURE (λ) only holds if the noise z in the linear model (8.1.2) is Gaussian [START_REF] Ryan J Tibshirani | Degrees of freedom in lasso problems[END_REF]. In contrast, it is not hard to generalize the proofs in the present paper to include other noise distributions.

Adaptive selection of λ

As anticipated, we can use our uniform bounds to select λ through an adaptive procedure. We discuss here three such procedures, that have already been illustrated in Figure 8.1:

(i) Selecting λ by minimizing the estimate τ (λ), we denote this by λ EST ; (ii) Select λ as to minimize Stein's Unbiased Risk Estimate P SURE (λ), λ SURE ; (iii) Select λ by k-fold cross-validation, λ k-CV . We will next describe these procedures in greater detail, and state the corresponding guarantees.

Minimization of τ (λ). Since the 2 risk of the Lasso is by Theorem 8.3.2 approximately equal to R * (λ) = δ(τ * (λ) 2 -σ 2 ) and since by Corollary 8.4.1, τ is a consistent estimator (uniformly in λ) of τ * , a natural procedure for selecting λ is to minimize τ . We then define

λ EST = arg min λ∈[λ min ,λmax] τ (λ) .
The next result is an immediate consequence of Theorem 8. Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0. There exists constants C, c > 0 that only depend on Ω such that for all ∈ (0, 1] inf

θ ∈D P 1 N θ λ EST -θ 2 ≤ inf λ∈[λ min ,λmax] 1 N θ λ -θ 2 + ≥ 1 -C N q 6 e -cN 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ).

Minimization of SURE.

We define

λ SURE = arg min λ∈[λ min ,λmax] P SURE (λ) .
Here, it is understood that we can use either σ or σ(λ), cf. Eq. (8.4.1), in the definition of P SURE . We deduce from Corollary 8.4.4: Proposition 8.4.2

Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0.

There exists constants C, c > 0 that only depend on Ω such that for all ∈ (0, 1]

inf θ ∈D P 1 n X θ λ SURE -Xθ 2 ≤ inf λ∈[λ min ,λmax] 1 n X θ λ -Xθ 2 + ≥ 1 -C -6 N q exp -cN 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ).

Cross-validation.

We analyze now k-fold Cross Validation. Let k ≥ 2 and define n k = n(k -1)/k. We partition the rows of X in k groups: we obtain k-submatrices of size (n/k) × N that we denote X (1) , . . . , X (k) . Let us also write for i ∈ {1, . . . , k}, X (-i) for the submatrix of X obtained by removing the rows X (i) . We denote by y (i) , z (i) and y (\i) , z (\i) the corresponding subvectors of y and z.

The estimator R k-CV of the risk using k-fold cross validation if defined as follows. For i = 1, . . . , k solve the Lasso problem

θ i λ = arg min θ∈R N 1 2n k y (\i) -X (\i) θ 2 + λ n |θ| ,
and then compute

R k-CV (λ) = 1 N k i=1 y (i) -X (i) θ i λ 2 .
Finally, we set λ as follows

λ k-CV = arg min λ∈[λ min ,λmax] R k-CV (λ) .
The next Proposition shows that R k-CV (λ) is equal to the true risk (shifted by δσ 2 ) up to O(k -1/2 ).

Proposition 8.4.3

There exists constants c, C > 0 that depend only on Ω, such that for all k ≥ 2 such that s max (k -1)δ/k > s in the case where D = F 0 (s), we have

sup θ ∈D P sup λ∈[λ min ,λmax] R k-CV (λ) - 1 N θ λ -θ 2 -δσ 2 ≥ C √ k ≤ Ck 6 N q e -cN/k 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ).
Proposition 8.4.3 is proved in Section 9.6.7 from Chapter 9. It follows from Proposition 8.4.3 that with high probability,

1 N θ λ k-CV -θ 2 ≤ inf λ∈[λ min ,λmax] 1 N θ λ -θ 2 + O(k -1/2 ).

Numerical experiments

In this Section we compare numerically various different choices for the regularization parameter λ, namely λ EST , λ SURE and λ k-CV , presented in the previous section. For these experiments we take the components θ 1 , . . . , θ N to be i.i.d. from

P 0 = sN (0, 1) + (1 -s)δ 0 .
Within this probabilistic model, we can compare achieved by our various choice of λ to the Bayes optimal error (Minimal Mean Squared Error):

MMSE N = min θ E θ -θ(y, X) 2 = E θ -E[θ |y, X] 2 ,
where the minimum is taken over all estimators θ (i.e. measurable functions of X, y).

The limit of the MMSE has been recently computed by [START_REF] Barbier | The mutual information in random linear estimation[END_REF] and [START_REF] Reeves | The replica-symmetric prediction for compressed sensing with gaussian matrices is exact[END_REF] and is given by Theorem 8.4.1 below, which is a direct corollary from Theorem 7.3.2. Recall, that given two random variables U, V , their mutual information is the Kullback-Leibler divergence between their joint distribution and the product of the marginals:

I(U ; V ) def = D KL (p U,V p U ⊗ p V ).
Theorem 8.4.1

Define the function

Ψ δ,σ (m) = I P 0 σ -2 1 + m + δ 2 log(1 + m) - m 1 + m ,
where I P 0 (r) = I(Θ; √ rΘ + Z) for (Θ, Z) ∼ P 0 ⊗ N (0, 1). Then, for almost every δ > 0 the function Ψ δ,σ admits a unique maximizer m * (δ, σ) on R ≥0 and

MMSE N ---→ N →∞ δσ 2 m * (δ, σ) .
Figure 8.1 reports the risk achieved by the various choices of λ as a function of the number of samples per dimension δ. We also compare the data-driven procedures of the previous section to the theory-driven choice λ = σ √ 2 log N . In the left frame, we consider uncorrelated random designs: X i,j i.i.d.

∼ N (0, 1/n). On the right, we consider i.i.d. Gaussian rows with covariance structure determined by an auto-regressive model. Explicitly, the columns (X j ) 1≤j≤N of X are generated according to:

X 1 = u 0 , X j+1 = 1 √ 1 + φ 2 φX j + u j (8.4.4)
where

u j i.i.d.
∼ N (0, Id/n) and φ = 2. For both types of designs, λ EST , λ SURE and λ k-CV perform similarly, and substantially outperform the theoretical choice λ = σ √ 2 log N . For uncorrelated designs, the resulting risk is closely tracked by the asymptotic theory, and is surprisingly close to the asymptotic prediction for the Bayes risk MMSE N .

While our theory does not cover the case of correlated designs, the qualitative behavior is remarkably similar. We also observed that in this case, the risk estimator R(λ) is not consistent but its minimum is roughly located at the same value of λ as for uncorrelated designs.

Next we study adaptivity to sparsity. On Figure 8.2, we plot the risk as a function of the sparsity of the signal θ . We compare the three adaptive procedures (namely, λ EST , λ SURE and λ k-CV ), to the following choice

λ MM (s 0 ) = α 0 σ 1 - 1 δ M s 0 (α 0 ) , M s (α) = s(1 + α 2 ) + 2(1 -s) (1 + α 2 )Φ(-α) -αφ(α) , α 0 = arg min α≥0 M s 0 (α) ,
where s 0 < s max (δ) is a nominal value for the sparsity (for Figure 8.2, we use s 0 = 0.3). The value λ MM (s 0 ) is expected to be asymptotically minimax optimal over F 0 (s 0 ) [START_REF] David L Donoho | The noise-sensitivity phase transition in compressed sensing[END_REF]. Also in this example, adaptive procedures dramatically outperform the fixed choice λ = σ √ 2 log N , and also the minimax optimal λ at the nominal sparsity level.

Proof strategy

As mentioned above, our proofs are based on Gaussian comparison inequalities, and in particular on Gordon's min-max theorem [START_REF] Gordon | Some inequalities for gaussian processes and applications[END_REF][START_REF] Gordon | On milman's inequality and random subspaces which escape through a mesh in rn[END_REF]. In this section we review the application Risk/s Here θ is chosen to be sN -sparse, and we vary the sparsity level s. The entries on the support of θ are i.i.d. N (0, 1). Cross-validation is carried out using 4 folds. SURE is computed using the estimator σ. The minimax regularization λ MM (s 0 ) is used at the nominal level s 0 = 0.3.

MMSE min λ R * (λ) R * (λ), λ = λ MM s0 Lasso, λ = λ EST Lasso, λ = λ SURE Lasso, λ = λ k-CV Lasso, λ = λ MM s0 Lasso, λ = σ √ 2 log N
of this inequality to the Lasso as developed in [START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF]. We then discuss the limitations of earlier work, which does not characterize the empirical distribution of the Lasso estimator θ λ (or need extra sparsity assumptions [START_REF] Panahi | A universal analysis of large-scale regularized least squares solutions[END_REF]) nor uniform bounds as in Theorem 8.3.1. A key challenge is related to the fact that the Lasso cost function (8.1.1) is convex but not strongly convex. Hence, a small change in λ could cause a priori a large change in the minimizer θ λ .

In order to overcome these problems, we establish a property that we call 'local stability.' Namely, if the empirical distribution of ( θ λ , θ ) deviates from our prediction, then the value of the optimization problem increases significantly. This implies that the empirical distribution is stable with respect to perturbations of the cost (e.g. changes in λ). Gordon's comparison is again crucial to prove this stability property.

Finally, we describe how local stability is used to prove the results of the previous sections. A full description of the proofs is provided in Chapter 9.

Tight Gaussian min-max theorem

It is more convenient (but equivalent) to study w λ = θ λ -θ instead of θ λ . The vector w λ is the minimizer of the cost function

C λ (w) = 1 2n Xw -σz 2 + λ n |w + θ | -|θ | . (8.5.1)
Following [START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF], we rewrite the minimization of C λ as a saddle point problem:

min w∈R N C λ (w) = min w∈R N max u∈R n 1 n u T Xw -σz - 1 2n u 2 + λ n |w + θ | -|θ | . (8.5.2)
We apply the following Theorem from [START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF] which improves over Gordon's Theorem [START_REF] Gordon | On milman's inequality and random subspaces which escape through a mesh in rn[END_REF] by exploiting convex duality. ∼ N (0, 1), g ∼ N (0, Id N ) and h ∼ N (0, Id n ) be independent standard Gaussian vectors. Define

     C * (G) = min w∈Sw max u∈Su u T Gw + Q(w, u) , L * (g, h) = min w∈Sw max u∈Su u 2 g T w + w 2 h T u + Q(w, u) .
Then we have:

• For all t ∈ R, P C * (G) ≤ t ≤ 2P L * (g, h) ≤ t .
• If S w and S u are convex and if Q is convex concave, then for all t ∈ R

P C * (G) ≥ t ≤ 2P L * (g, h) ≥ t .
For the reader's convenience, we provide in Section 9.7.2 of Chapter 9 a proof of this theorem.

Because of Gordon's Theorem, it suffices now to study (see Corollary 8.5.1 below) for (g, g , h) ∼ N (0, Id N ) ⊗ N (0, 1) ⊗ N (0, Id n ). Proof. We will only prove the first point, since the second follows from the same arguments.

L λ (w) = 1 2   w 2 n + σ 2 h √ n - 1 n g T w + g σ √ n   2 + + λ n |w + θ | - λ n |θ | . ( 8
Define for (w,

u) ∈ R N × R n c λ (w, u) = 1 n u T Xw - σ n u T z - 1 2n u 2 + λ n |w + θ | -|θ | , l λ (w, u) = - 1 n 3/2 u g T w + 1 n u g σ + w 2 n + σ 2 h T u n - 1 2n u 2 + λ n |w + θ | -|θ | .
Notice that for all w ∈ R N , L λ (w) = max u∈R n l λ (w, u) and C λ (w) = max u∈R n c λ (w, u).

Let us suppose that X, z, g, h, g live on the same probability space and are independent. Let ∈ (0, 1]. Let σ max (X) denote the largest singular value of the matrix X. By tightness we can find K > 0 such that the event

σ max (X) ≤ K, z ≤ K, g ≤ K, h ≤ K, |g | ≤ K (8.5.4)
has probability at least 1 -. Let D ⊂ R N be a (non-empty, otherwise the result is trivial) closed set. Let us fix w 0 ∈ D. On the event (8.5.4) C λ (w 0 ) and L λ (w 0 ) are both upper bounded by some non-random quantity R. Let now w ∈ D such that C λ (w) ≤ R. We have then λ n |w+θ | ≤ R+ λ n |θ |, which implies that w is upper bounded by some non-random quantity R 1 . This implies that, on the event (8.5.4), the minimum of C λ over D is achieved on D ∩ B(0, R 1 ). Similarly on (8.5.4) the minimum of L λ over D is achieved on D ∩ B(0, R 2 ), for some non-random quantity R 2 . Without loss of generalities, one can assume R 1 = R 2 . On the event (8.5.4) we have

min w∈D C λ (w) = min w∈D∩B(0,R 1 ) C λ (w) = min w∈D∩B(0,R 1 ) max u∈B(0,R 3 ) c λ (w, u) ,
for some non-random R 3 > 0. This gives that for all t ∈ R, we have

P min w∈D C λ (w) ≤ t ≤ P min w∈D∩B(0,R 1 ) max u∈B(0,R 3 ) c λ (w, u) ≤ t + ,
and similarly

P min w∈D∩B(0,R 1 ) max u∈B(0,R 3 ) l λ (w, u) ≤ t ≤ P min w∈D L λ (w) ≤ t + .
Since the sets D ∩ B(0, R 1 ) and B(0, R 3 ) are compact, one can apply Theorem 8.5.1 to c λ and l λ and obtain:

P min w∈D C λ (w) ≤ t ≤ 2P min w∈D L λ (w) ≤ t + 2 .
The Corollary follows then from the fact that one can take arbitrarily small.

Local stability

In order to prove that (for instance) w λ verifies with high probability some property, let's say for instance that the empirical distribution of ( θ λ = θ + w λ , θ ) is close to µ λ , we define a set D ⊂ R N that contains all the vectors that do not verify this property, e.g.

D = w ∈ R N W 2 µ (θ +w,θ ) , µ * λ 2
≥ , for some ∈ (0, 1). The goal now is to prove that with high probability It remains now to study the cost function L λ , which is much simpler. This is done in Section 9.2 of Chapter 9. The key step will be to establish the following 'local stability' result (the next statement is an immediate consequence of Proposition 9.2.1 and Theorem 9.2.1 in Chapter 9. We prove in fact that the cost function L λ is strongly convex on a neighborhood of its minimizer.).

Theorem 8.5.2

The minimizer w * λ = arg min w L λ (w) exists and is almost surely unique. Further, there exists constants γ, c, C > 0 that only depend on Ω such that for all θ ∈ D, all λ ∈ [λ min , λ max ] and all ∈ (0, 1]

P ∃w ∈ R N , 1 N w -w * λ 2 > and L λ (w) ≤ min v∈R N L λ (v) + γ ≤ C e -cn 2 .
We do not obtain an equally strong result for the cost function C λ (w), but we prove the following statement, which is sufficient for obtaining uniform control (for the sake of argument, we focus here on the domain F p (ξ) and control of the empirical distribution).

Theorem 8.5.3

Assume that D = F p (ξ) for some ξ, p > 0. There exists constants C, c, γ > 0 that only depend on Ω such that for all ∈ (0, 1 2 ] sup

λ∈[λ min ,λmax] sup θ ∈D P ∃θ ∈ R N , W 2 µ (θ,θ ) , µ * λ 2 ≥ and L λ (θ) ≤ min L λ + γ ≤ C -max(1,a) exp -cN 2 a log( ) -2 ,
where a = 1 2 + 1 p .

Theorem 8.5.3 is proved in Section 9.3.1 of Chapter 9.

Sketch of proof of main results

For the sake of simplicity, we will illustrate the prove strategy by considering the empirical distribution of w λ = θ λ -θ , as the argument is similar for other quantities. According to Theorem 8.3.1, this should be well approximated by µ λ that is the law of Θ -Θ, when ( Θ, Θ) ∼ µ * λ , cf. Definition 8.3.3. As anticipated, Eq. (8.5.5) and Theorem 8.5.2, allow to control W 2 ( µ w λ , µ λ ) for a fixed λ ( µ w λ denotes the empirical distribution of the entries of w λ ). Namely, we can define D ε to be the set of vectors w such that W 2 ( µ w , µ λ ) ≥ ε > 0. We then prove that the minimizer w * λ of L λ has empirical distribution close to µ λ , and therefore by Theorem 8.5.2, L λ (w) > L λ (w * λ ) + γ for all w ∈ D , with high probability. This imply that the right-hand side of (8.5.5) is very small and we deduce that, with high probability, all minimizers or near minimizers of C λ (w) have empirical distribution close to µ λ , We now would like to prove Theorem 8.3.1 and show that with high probability µ w λ ≈ µ λ , uniformly in λ ∈ [λ min , λ max ]. To do so, we apply the above argument for λ = λ 1 , . . . , λ k , where λ 1 , . . . , λ k is an -net of [λ min , λ max ]. This implies that, with high probability for

λ ∈ {λ 1 , . . . , λ k }, W 2 ( µ w λ i , µ λ i ) ≤ ε. Next, for λ ∈ [λ i , λ i+1 ], we show that C λ i ( w λ ) = min w∈R N C λ i (w) + O(|λ i+1 -λ i |) .
Consequently if |λ i+1 -λ i | = O( ) (using again Eq. (8.5.5) and Theorem 8.5.2), we obtain that W 2 ( µ w λ , µ λ i ) = O( ) and therefore W 2 ( µ w λ , µ λ ) = O( ). We conclude that W 2 ( µ w λ , µ λ ) = O( ) for all λ ∈ [λ min , λ max ], with high probability, which is the desired claim.

If the strategy exposed above allows to obtain the risk of the Lasso and the empirical distribution of its coordinates, it is not enough to get its sparsity θ λ 0 or to obtain the empirical distribution of the debiased lasso

θ d λ = θ λ + X T (y -X θ λ ) 1 -1 n θ λ 0 .
Therefore, we will need to analyze the vector

v λ = 1 λ X T (y -X θ λ ) ,
which is a subgradient of the 1 -norm at θ λ . We are able to study v λ using Gordon's min-max Theorem because v λ is the unique maximizer of

v → min w∈R N 1 2n Xw -σz 2 + λ n v T (w + θ ) .
The detailed analysis is done in Section 9.5 from Chapter 9.

Chapter 9

Proofs of the results on the Lasso

This chapter is devoted to the proofs of the results of Chapter 8. It is organised as follows.

In Section 9.1, we study the max-min scalar problem (8.3.2). In Section 9.2 we investigate Gordon's optimization problem associated to the Lasso cost and deduce the results about the risk and the empirical law of the Lasso estimator in Section 9.3. We conduct the same kind of analysis for the Lasso residual u λ def = X θ λ -y in Section 9.4 and the subgradient v λ def = 1 λ X T (y -X θ λ ) of the 1 -norm at θ λ in Section 9.5. Section 9.6 gathers some auxiliary results and proofs such as the study of the sparsity of the Lasso estimator, the law of the debiased Lasso θ (d) λ , or the performance of k-fold Cross Validation. Finally Section 9.7 contains a recap of the notations, a proof of the "tight Gordon min-max" Theorem and some basic concentration results.

Study of the scalar optimization problem

In this section we study the scalar optimization problem (8.3.2):

max β≥0 min τ ≥σ σ 2 τ + τ β 2 - 1 2 β 2 + 1 δ E min w∈R w 2 2τ β -βZw + λ|w + Θ| -λ|Θ| , ( 9.1.1)
where (Θ, Z) ∼ P 0 ⊗ N (0, 1), for some probability distribution P 0 with finite first moment:

E P 0 |Θ| < ∞.
Of course, we will be mainly interested by the case where P 0 = µ θ , the empirical distribution of the entries of θ . Define For all δ ∈ (0, 1), the equation

ψ λ (β, τ ) = σ 2 τ + τ β 2 - 1 2 β 2 + 1 δ E min w∈R w 2 2τ β -βZw + λ|w + Θ| -λ|Θ| .
(1 + α 2 )Φ(-α) -αφ(α) = δ 2
admits a unique positive solution α min = α min (δ) > 0.

Proof. Let ϕ : α → (1 + α 2 )Φ(-α) -αφ(α). ϕ is continuous on R ≥0 , we have ϕ(0) = 1 2 and ϕ(+∞) = 0. It remains to show that ϕ is strictly decreasing on R ≥0 . Compute ϕ (α) = 2αΦ(-α) -2φ(α) and ϕ (α) = 2Φ(-α) > 0. Since ϕ (+∞) = 0, we have that for all α ≥ 0, ϕ (α) < 0. ϕ is thus strictly decreasing on R ≥0 .

Let us define

β max = β max (δ, λ) = λ α min (δ)
.

More generally, we will always write α = λ/β. We prove in this section the following theorem and some auxiliary results.

Theorem 9.1.1

The max-min (8.3.2) is achieved at a unique couple (β * , τ * ) and 0 < β * < β max . Moreover, (τ * , β * ) is also the unique couple in (0, +∞) 2 that verify

     τ 2 = σ 2 + 1 δ E (η(Θ + τ Z, τ λ β ) -Θ) 2 β = τ 1 -1 δ E η (Θ + τ Z, τ λ β ) . ( 9
σ 2 τ + τ β 2 - 1 2 β 2 = σ 2 2 .
Define for α ≥ 0 and y ∈ R, Lemma 9.1.4

α (y) = min x∈R 1 2 (y -x) 2 + α|x| and for Z ∼ N (0, 1), x ∈ R, ∆ α (x) = E α (x + Z) -α|x| . Lemma 9.1.3 E min w∈R w 2 2τ β -βZw + λ|w + Θ| -λ|Θ| = τ βE ∆ α Θ τ - βτ 2 . ( 9
• If β > β max , ψ λ (β, τ ) ----→ τ →+∞ -∞ . • If β = β max , ψ λ (β, τ ) ----→ τ →+∞ - β 2 2 - λ δ E Θ .
Proof. By (9.1.3) and the fact that ∆ α (0) = 1 2 + αφ(α) -(α 2 + 1)Φ(-α) by Lemma 9.6.13, we get that for all β, τ > 0

ψ λ (β, τ ) = σ 2 β 2τ - β 2 2 + τ β δ δ 2 + αφ(α) -(α 2 + 1)Φ(-α) + β δ ξ α (τ ) ,
where α = λ/β and

ξ α (τ ) = τ E ∆ α Θ τ -∆ α (0) .
Using the definition of β max : if β > β max then α < α min and therefore δ 2 +αφ(α)-(

α 2 +1)Φ(-α) < 0. If β = β max , δ 2 + αφ(α) -(α 2 + 1)Φ(-α) = 0. It remains to compute the limit of ξ α (τ ) as τ → ∞.
Using the expression (see Lemma 9.6.13) of the left-and right-derivatives of ∆ α at 0, we have almost surely:

τ ∆ α Θ τ -∆ α (0) ---→ τ →∞ -α|Θ| .
Suppose that E|Θ| < ∞. By Lemma 9.6.13, ∆ α is α-Lipschitz. Consequently, for all τ > 0:

τ ∆ α Θ τ -∆ α (0) ≤ α|Θ| .
Since we have assumed that E|Θ| < ∞ we can apply the dominated convergence theorem to obtain that ξ α (τ ) ----→

τ 2 = σ 2 + 1 δ E w * (α, τ ) 2 = σ 2 + 1 δ E (η(Θ + τ Z, τ λ β ) -Θ) 2 . (9.1.4)
does not admits any solution on (0, +∞). For all β ∈ (0, β max ), the function ψ λ (β, •) admits a unique minimizer τ * (β) on (0, +∞) that is also the unique solution of (9.1.4). Moreover, α → τ * (α) is C ∞ on (α min , +∞) and for all α > α min

∂τ * ∂α (α) ≤ (α + 1) τ * (α) 3 δσ 2 .
Proof. Most of this lemma was already proved in [START_REF] David L Donoho | Message-passing algorithms for compressed sensing[END_REF], we however provide a full proof for completeness. We have to study the fixed point equation

τ 2 = σ 2 + 1 δ E w * (α, τ ) 2 = F α (τ 2 ) ,
where α = λ/β. We can compute F α explicitly:

F α (τ 2 ) = σ 2 + τ 2 δ 1 + α 2 + E (x 2 -α 2 -1) Φ(α-x)-Φ(-α-x) -(x+α)φ(α-x)+(x-α)φ(x+α) ,
where we used the notation x = Θ τ . We can then compute the derivatives:

F α (τ 2 ) = 1 δ (1+α 2 )E[Φ(x-α)+Φ(-x-α)]- 1 δ E[(x+α)φ(x-α)-(x-α)φ(x+α)] , F α (τ 2 ) = -1 2δτ 2 E x 3 (φ(x -α) -φ(x + α)) ≤ 0 .
F α is therefore concave. By dominated convergence

F α (τ 2 ) ----→ τ →+∞ 2 δ (1 + α 2 )Φ(-α) -αφ(α) < 1 if β < β max ≥ 1 if β ≥ β max .
Since F α (0) = σ 2 > 0 and by concavity of F α , the fixed point equation admits a unique solution τ * (α) if and only is β ∈ (0, β max ). In that case we have also F α (τ * (α)) < 1.

Let us now assume that β ∈ (0, β max ). We have almost surely

∂ ∂τ min w∈R w 2 2τ β + βZw + λ|w + Θ| = - β 2τ 2 w * (α, τ ) 2 .
Since |w * (α, τ )| ≤ ατ + τ |Z|, we have by derivation under the expectation

∂ ∂τ ψ λ (β, τ ) = β 2 - βσ 2 2τ 2 - β 2δτ 2 E w * (α, τ ) 2 = β 2τ 2 τ 2 -σ 2 + 1 δ E w * (α, τ ) 2 .
Consequently, τ * (β) is the unique minimizer of ψ λ (β, •) over (0, +∞).

Let us now compute ∂τ 2 * ∂α . Since F α is a C ∞ function of τ 2 , one can apply the implicit function theorem to obtain that the mapping α → τ * (α) 2 is C ∞ and moreover:

∂τ 2 * ∂α (α) = ∂Fα ∂α (τ 2 * (α)) 1 -F α (τ 2 * (α)) . (9.1.5) Compute ∂F α ∂α (τ 2 ) = 2τ 2 δ E [α(Φ(-α + x) + Φ(-α -x)) -(φ(α -x) + φ(α + x))] .
One verify easily that

-1 ≤ -2φ(0) ≤ 2αΦ(-α) -2φ(α) ≤ δ 2τ 2 ∂F α ∂α (τ 2 ) ≤ α . (9.1.6)
By concavity on has that F α (τ 2 * (α)) is smaller than the slope of the line between the points of coordinates (0, σ 2 ) and (τ 2 * (α), τ 2 * (α)):

F α (τ 2 * (α)) ≤ 1 - σ 2 τ 2 * (α) . (9.1.7)
From equations (9.1.5-9.1.6-9.1.7) we get

∂τ 2 * ∂α (α) ≤ 2 τ * (α) 4 σ 2 δ (α + 1) .
The result follows then from the fact that ∂τ 2 * ∂α (α) = 2τ * (α) ∂τ * ∂α (α).

Define now Ψ

λ : β → min τ ≥σ ψ λ (β, τ ) . Lemma 9.1.6
The function Ψ λ is differentiable on (0, β max ) with derivative

Ψ λ (β) = τ * (α) -β - 1 δ E [Zw * (α, τ * (α))] (9.1.8) = τ * (α) 1 - 1 δ E Φ Θ τ * (α) -α + Φ - Θ τ * (α) -α -β . (9.1.9)
Proof. Ψ λ is differentiable on (0, β max ) (because of Lemma 9.1.5) with derivative

Ψ λ (β) = 1 2 σ 2 τ * (α) + τ * (α) -β + 1 δ 1 2τ * (α) E[w * (α, τ * (α)) 2 ] -E[Zw * (α, τ * (α))] = τ * (α) -β - 1 δ E [Zw * (α, τ * (α))] ,
because of (9.1.4). The second equality follows by Gaussian integration by parts.

Corollary 9.1.1

The function Ψ λ achieves its maximum over R ≥0 at a unique β * ∈ (0, β max ).

Proof. Ψ λ is the minimum of a collection of 1-strongly concave functions: it is therefore 1strongly concave and admits thus a unique maximizer β * over R ≥0 . By Lemma 9.1.4 we know that β * < β max . Indeed, notice that max

β Φ λ (β) ≥ Ψ λ (0) = -λ δ E|Θ|. Lemme 9.1.4 gives that β * ∈ [0, β max ), because Ψ λ (β max ) ≤ Ψ λ (0) -1 2 β 2 max < Ψ λ (0)
. By dominated convergence:

E Φ Θ τ * (α) -α + Φ - Θ τ * (α) -α ----→ β→0 + 0 . Indeed, when β → 0 + , α = λ/β → +∞ and | Θ τ * (α) | ≤ |Θ| σ .
Therefore by Lemma 9.1.6 we obtain lim inf

β→0 + Ψ λ (β) ≥ σ > 0 .
By concavity, we deduce that β * ∈ (0, β max ).

Proposition 9.1.1

The function λ → β * (λ) is C ∞ and is 2α -1 min -Lipschitz over (0, +∞). λ → α * (λ) is C ∞ over (0, +∞) and strictly increasing.

Proof. Let us define γ

* (λ) = β * (λ)/λ. γ * (λ) is the unique maximizer of min τ ≥σ σ 2 τ + τ γ 2 - λ 2 γ 2 + 1 δ E min w∈R w 2 2τ γ -γZw + |w + Θ| -|Θ| = h(γ) - λ 2 γ 2 ,
where h is a concave C ∞ function on R >0 . γ * (λ) is thus the unique solution of

h (γ) -λγ = 0 , on R >0 . γ → h (γ) -λγ is C ∞ with derivative γ → h (γ) -λ < 0. Consequently, the implicit function theorem gives that the mapping λ ∈ R >0 → γ * (λ) is C ∞ and that ∂γ * ∂λ (λ) = -γ * (λ) λ -h (γ * (λ)) < 0 . One deduces that λ → α * (λ) = γ * (λ) -1 is C ∞ and strictly increasing and that λ → β * (λ) = λγ * (λ) is C ∞ . Moreover ∂β * ∂λ (λ) = λ ∂γ * ∂λ (λ) + γ * (λ) ≤ 2γ * (λ) ≤ 2 α min .
Proof of Theorem 9.1.1. By Corollary 9.1.1, the maximum in β in (8.3.2) is achieved at a unique β * ∈ (0, β max ). To this β * corresponds a unique τ * (β * ) that achieves the minimum in (8.3.2), by Lemma 9.1.5. By (9.1.4) and (9.1.9) we obtain that (τ * (β * ), β * ) is solution of the system (9.1.2). Let now (τ, β) ∈ (0, +∞) 2 be another solution of (9.1.2). τ is therefore solution of (9.1.4) which gives that β ∈ (0, β max ) and τ = τ * (β) by Lemma 9.1.5. The second equality in (9.1.2) gives that Ψ λ (β) = 0 and thus that β = β * by strong concavity of Ψ λ . We conclude (τ, β) = (τ * (β * ), β * ).

Control on β * , τ *

The goal of this section is to show that β * and τ * remain bounded when θ varies in D.

Theorem 9.1.2

There exists constants β min , τ max > 0 that only depend on Ω such that for all θ ∈ D and all λ ∈ [λ min , λ max ],

β min ≤ β * (λ) < β max and σ ≤ τ * (λ) ≤ τ max .
To prove Theorem 9.1.2, we separate the case where D = F p (ξ) (where it follows from Lemma 9.1.9 and Corollary 9.1.2 below) from the case where D = F 0 (s) (where it follows from Lemmas 9.1.11 and 9.1.12).

Technical lemmas Lemma 9.1.7

We have

max β≥0 min τ ≥σ ψ λ (β, τ ) = ψ λ (β * , τ * (β * )) = 1 2 β 2 * + λ δ E |w * (α * , τ * (β)) + Θ| -|Θ| = 1 2 β 2 * + τ * (β * ) λ δ E H α * Θ τ * (β * )
,

where

H α (x) = (x -α)Φ(-α + x) + (-x -α)Φ(-x -α) + φ(-x + α) + φ(x + α) -|x| .
Proof. Using the optimality condition (9.1.4) of τ * (β), we have for all β ∈ (0, β max )

ψ λ (β, τ * (β)) = - 1 2 β 2 + βτ * (β) - β δ E [Zw * (α, τ * (β))] + λ δ E [|w * (α, τ * (β)) + Θ| -|Θ|] .
At β * the optimality condition (see (9.1.8)) reads

β * = τ * (β * ) -1 δ E Zw * (α * , τ * (β * )) , thus ψ λ (β * , τ * (β * )) = 1 2 β 2 * + λ δ E |w * (α * , τ * (β)) + Θ| -|Θ| . (9.1.10) Compute for α, τ > 0 E w * (α, τ ) + Θ = E η(Θ + τ Z, ατ ) = τ E η Θ τ + Z, α . (9.1.11) Now, for x ∈ R, E|η(x + Z, α)| = +∞ α-x (x + z -α)φ(z)dz + -α-x -∞ (-x -z -α)φ(z)dz = (x -α)Φ(x -α) + (-x -α)Φ(-x -α) + φ(α -x) + φ(α + x) = H α (x) + |x| .
and we obtain the Lemma by putting this together with (9.1.11) and (9.1.10).

The next Lemma summarizes the main properties of H α . Lemma 9.1.8 H α is a continuous, even function and for x > 0

H α (x) = Φ(x -α) -Φ(-x -α) -1 ∈ (-1, 0) .
H α is therefore 1-Lipschitz. H α admits a maximum at 0 and

H α (0) = 2φ(α) -2αΦ(-α) > 0 . Moreover H α (x) ----→ x→+∞ -α.
On p -balls Lemma 9.1.9

Assume that E |Θ| p ≤ ξ p for some ξ, p > 0. Then, there exists a constant β min = β min (δ, λ min , ξ, p, σ) such that for all λ ≥ λ min ,

0 < β min ≤ β * (λ) < β max .
Proof. Let β ∈ (0, β max ). By Lemma 9.1.6 we have

Ψ λ (β) = τ * (β) 1 - 1 δ E Φ Θ τ * (β) -α + Φ - Θ τ * (β) -α -β .
The function g α : x → Φ(x -α) + Φ(-x -α) is even, and increasing over R ≥0 . Let K > 0 such that ξ p K p σ p ≤ δ 4 . By Markov's inequality we have

P Θ τ * (β) ≥ K ≤ P Θ σ p ≥ K p ≤ 1 K p σ p E|Θ| p ≤ δ 4
.

Thus E g α Θ τ * (β) ≤ g α (K) + δ 4 .
As β → 0, α ≥ λ min /β → +∞. Since g α (K) -----→ α→+∞ 0 there exists β 0 = β 0 (K, λ min , δ) > 0 such that for all β ∈ (0, β 0 ), g α (K) ≤ δ 4 . Thus for all β ∈ (0, β 0 ),

Ψ λ (β) ≥ τ * (β) 1 - δ 4 + δ 4 δ -β ≥ σ 2 -β .
Let β min = min( σ 2 , β 0 ). We conclude that for all β ∈ (0, β min ), Ψ λ (β) > 0. By concavity we have then that β * ≥ β min . The other inequality β * < β max was already proved in Corollary 9.1.1.

Corollary 9.1.2

Assume that E |Θ| p ≤ ξ p for some ξ, p > 0. Then there exists a constant τ max = τ max (ξ, p, δ, s, λ min , λ max ) such that for all λ ∈ [λ min , λ max ],

σ ≤ τ * (β * (λ)) ≤ τ max . Proof. Let t ≥ ξ. By Markov's inequality we have P(|Θ| ≥ t) ≤ ξ t p ≤ 1, since θ ∈ F p (ξ). E H α * Θ τ * (α * ) = E 1(|Θ| < t)H α * Θ τ * (α * ) + E 1(|Θ| ≥ t)H α * Θ τ * (α * ) ≥ 1 - ξ t p H α * (0) - t τ * (α * ) -α * ξ t p ,
because by Lemma 9.1.8, H α * is 1-Lipschitz and for all x ∈ R, -α * ≤ H α * (x) ≤ H α * (0). Replacing H α * (0) by its expression given by Lemma 9.1.8 we get

E H α * Θ τ * (α * ) ≥ 2 φ(α * )-α * Φ(-α * ) - ξ t p α * +2(φ(α * )-α * Φ(-α * )) - t τ * (α * )
.

Since α * ≤ λ max /β min and φ(α * ) -α * Φ(-α * ) > 0 (because α * > α min ), we can find a constant t = t(δ, σ, λ min , λ max , p, ξ) ≥ ξ such that

(α * + 2(φ(α * ) -α * Φ(-α * ))) ξ t p ≤ φ(α * ) -α * Φ(-α * ) .
For this choice of t we have then

τ * (α * )λ δ E H α * Θ τ * ≥ λ δ τ * (α * )(φ(α * ) -α * Φ(-α * )) - λt δ .
Consequently by Lemma 9.1.2 and Lemma 9.1.7 we have

β 2 * 2 + λ δ τ * (φ(α * ) -α * Φ(-α * )) - λt δ ≤ ψ λ (β * , τ * (α * )) ≤ σ 2 2 ,
which finally gives

τ * (α * ) ≤ δσ 2 λ -1 + t φ(α max ) -α max Φ(-α max )
.

On sparse balls

Define the critical function:

M s : α → s(1 + α 2 ) + 2(1 -s) (1 + α 2 )Φ(-α) -αφ(α) .
M s corresponds to the worst mean squared error achievable by soft-thresholding with threshold α to estimate a vector θ ∈ F 0 (s) from the observations y = θ + w, where w ∼ N (0, I N ), see [START_REF] David | Ideal spatial adaptation by wavelet shrinkage[END_REF][START_REF] David | Minimax risk over p -balls for q -error[END_REF][START_REF] Johnstone | Function estimation and gaussian sequence models[END_REF].

Lemma 9.1.10 (From [71] )

Assume that

s < s max (δ) = δ max α≥0    1 -2 δ (1 + α 2 )Φ(-α) -αφ(α) 1 + α 2 -2 (1 + α 2 )Φ(-α) -αφ(α)    .
Then there exists α ≥ 0 such that M s (α) < δ. Proof. Let s < s max (δ). From the definition of s max (δ), we can find α ∈ R such that

δ 1 -2 δ (1 + α 2 )Φ(-α) -αφ(α) 1 + α 2 -2 (1 + α 2 )Φ(-α) -αφ(α) > s , which gives M s (α) < δ.
We assume in this section that s < s max (δ). Let us compute the derivatives

M s (α) = 2 (αs + 2(1 -s)(αΦ(-α) -φ(α))) , M s (α) = 2 (s + (1 -s)2Φ(-α)) > 0 .
Notice that M s (α) = 1 2 αM s (α)+M s (α) . Let α 0 be the unique α > 0 such that M s (α) = 0 and let α 1 < α 2 be such that M s (α 1 ) = M s (α 2 ) = δ. We can then easily plot the variations of M s :

α M s M s 1 2 M s 0 α 1 α 0 α 2 +∞ - 0 + 1 1 M s (α 0 ) M s (α 0 ) +∞ +∞ δ δ 1 1 s s M s (α 0 ) Lemma 9.1.11
Let s < s max (δ) and assume that P(Θ = 0) ≤ s. Then, there exists a constant β min = β min (δ, λ min , s, σ) such that for all λ ≥ λ min 0 < β min ≤ β * (λ) < β max (λ max , δ) := λ max /α min (δ) .

Proof. We already proved in Corollary 9.1.1 that β * (λ) < λ/α min . For all 0 < β < λ/α min , we have by Lemma 9.1.6

Ψ λ (β) = τ * (β) 1 - 1 δ E Φ Θ τ * (β) -α + Φ - Θ τ * (β) -α -β .
The function g α : x → Φ(x -α) + Φ(-x -α) is even, and increasing over R ≥0 . Therefore

Ψ λ (β) ≥ τ * (β) δ δ -s -(1 -s)2Φ(-α) -β .
Let β 0 = β 0 (λ min , δ, s) > 0 such that for all β ∈ (0, β 0 ), 2Φ(-α) ≤ 1 2 (δ -s). For all β ∈ (0, β 0 ) we have then

Ψ λ (β) ≥ σ(δ -s) 2δ -β .
Let β min = min( σ(δ-s) 2δ , β 0 ): for all β ∈ (0, β min ), Ψ λ (β) > 0. By concavity we conclude that β * ≥ β min . Lemma 9.1.12 Let s < s max (δ) and assume that P(Θ = 0) ≤ s. Then for all β, τ, λ > 0 we have

ψ λ (β, τ ) ≥ βσ 2 2τ - β 2 2 + τ β 2δ δ -M s (α) .
Proof. By (9.1.3) we have for all β, τ > 0

ψ λ (β, τ ) = β 2 σ 2 τ + τ - β 2 2 + τ β δ E ∆ α Θ τ - 1 2 .
Since by Lemma 9.6.13, ∆ α is even and non-increasing over R ≥0 , we have

E ∆ α Θ τ - 1 2 ≥ s∆ α (+∞) + (1 -s)∆ α (0) - 1 2 = -s α 2 2 + (1 -s) 1 2 + αφ(α) -(1 + α 2 )Φ(-α) - 1 2 = - 1 2 M s (α) .
Lemma 9.1.13

Let s < s max (δ) and assume that P(Θ = 0) ≤ s. Then the following inequalities hold

β * τ * (β * )(δ -M s (α * )) ≤ δ σ 2 + β 2 * , ( 9
ψ λ (β * , τ * (β * )) ≥ λτ * (β * ) δ E H α * Θ τ * (β * )
.

Since by Lemma 9.1.8, H α * is even, decreasing on R ≥0 , we have

E H α * Θ τ * (β * ) ≥ sH α * (+∞) + (1 -s)H α * (0) = -sα * + 2(1 -s) φ(α * ) -α * Φ(-α * ) = - 1 2 M s (α * ) ,
which proves (9.1.13). To prove (9.1.14) we use the optimality condition at β * :

0 = Ψ λ (β * ) = τ * (α * ) 1 - 1 δ E Φ Θ τ * (α * ) -α * + Φ - Θ τ * (α * ) -α * -β * . (9.1.15)
The function x → Φ(x -α * ) + Φ(-x -α * ) is even, increasing on R ≥0 . Therefore

E Φ Θ τ * (α * ) -α * + Φ - Θ τ * (α * ) -α * ≤ s + 2(1 -s)Φ(-α * ) = 1 2 M s (α * ) .
Combining this inequality with (9.1.15) leads to (9.1.14).

Proposition 9.1.2

Let s < s max (δ) and assume that P(Θ = 0) ≤ s. Then, there exists a constant τ max = τ max (δ, λ min , λ max , s, σ) such that for all λ ∈ [λ min , λ max ],

σ ≤ τ * (β * (λ)) ≤ τ max .
Proof. Let (β * , τ * ) be the unique optimal couple and recall α * = λ/β * . We distinguish 3 cases:

Case 1: α * ≥ α 0 . In that case 1 2 M s (α * ) ≤ M s (α 0 ) < δ. The inequality (9.1.14) gives

τ * (β * )(δ - 1 2 M s (α * )) ≤ β * ≤ β max , which gives τ * (β * ) ≤ βmax δ-Ms(α 0 ) . Case 2: α * ∈ [(α 1 + α 0 )/2, α 0 ].
In that case δ -M s (α * ) ≥ c > 0, for some constant c = c(δ, s) > 0. Now, by (9.1.12)

β * τ * (β * )(δ -M s (α * )) ≤ δ(σ 2 + β 2 * ) .
Therefore, Recall that α * (λ) = λ/β * (λ). Thus

τ * ≤ δ cβ min (σ 2 + β 2 max ) .
∂α * ∂λ (λ) = 1 β * (λ) - ∂β * ∂λ (λ) λ β * (λ) 2 ≤ β -1 min + 2α

Study of Gordon's optimization problem for w λ

In this section we study L λ defined by (8.5.3). Define, for w ∈ R N and β ≥ 0

λ (w, β) =   w 2 n + σ 2 h √ n - 1 n g T w + g σ √ n   β - 1 2 β 2 + λ n |w + θ | - λ n |θ | . (9.2.1)
So that L λ (w) = max β≥0 λ (w, β). Let us define the vector w λ ∈ R N by

w λ,i = η θ i + τ * (λ)g i , τ * (λ)λ β * (λ) -θ i . (9.2.2)
The goal of this section is to prove that, with high probability, the minimizer of L λ is close to w λ and that L λ is strongly convex around w λ . Proposition 9.2.1

L λ admits almost surely a unique minimizer w * λ on R N .

Proof. L λ is a convex function that goes to +∞ at infinity, so it admits minimizers over R N .

We distinguish two cases:

Case 1: there is a minimizer w such that

w 2 n + σ 2 h √ n -1 n g T w + g σ √ n > 0.
In that case, there exist a neighborhood O w of w such that for all w ∈ O w a(w

) := w 2 n + σ 2 h √ n - 1 n g T w + g σ √ n > 0 .
Thus for all w ∈ O w , L λ (w ) = 1 2 a(w

) 2 + λ n |w + θ | -λ n |θ |.
Recall that the composition of a strictly convex function and a strictly increasing function is strictly convex. L λ is therefore strictly convex on O w because a is strictly convex and remains strictly positive on O w and because x > 0 → x 2 is strictly increasing. w is thus the only minimizer of L λ .

Case 2: for all minimizer w we have

w 2 n + σ 2 h √ n -1 n g T w + g σ √ n ≤ 0.
Let w be a minimizer of L λ . The optimality condition gives

-1 λ   w 2 n + σ 2 h √ n - 1 n g T w + g σ √ n   +   w w 2 n + σ 2 h √ n -g   ∈ ∂|θ + w|.
We obtain then 0 ∈ ∂|θ + w| which implies w = -θ : L λ has a unique minimizer.

Local stability of Gordon's optimization Theorem 9.2.1

There exists constants γ, c, C > 0 that only depend on Ω such that for all θ ∈ D, all λ ∈ [λ min , λ max ] and all ∈ (0, 1]

P ∃w ∈ R N , 1 N w -w λ 2 > and L λ (w) ≤ min v∈R N L λ (v) + γ ≤ C e -cn 2 .
We deduce from Theorem 9.2.1 that for all ∈ (0, 1] with probability at least 1 -C -1 e -cn For all R > 0 there exists constants c, C > 0 that only depend on (Ω, R), such that for all ∈ (0, 1],

∀θ ∈ D, ∀λ ∈ [λ min , λ max ], P L λ (w λ ) ≤ min w ≤ √ nR L λ (w) + ≥ 1 - C e -cn 2 .
Proof. Notices that it suffices to proves the proposition for smaller than some constant. Let

θ ∈ D, λ ∈ [λ min , λ max ]. Let R > 0 and ∈ 0, min(1, σ 2 /2) . Define • λ (w, β) =   w 2 n + σ 2 - 1 n g T w   β - 1 2 β 2 + λ n |w + θ | - λ n |θ | . On the event 1 n h 2 -1 ≤ g σ √ n ≤ , ( 9.2.4)
which has probability at least 1 -Ce -cn 2 , we have, for all w ∈ B(0, R √ n) and β ∈ [0, β max ]:

| λ (w, β) -• λ (w, β)| = β w 2 n + σ 2 h √ n -1 + β g σ √ n ≤ β max σ 2 + R 2 1 n h 2 -1 + β max ≤ β max ( σ 2 + R 2 + 1) K .
For simplicity we write (β * , τ * ) = (β * (λ), τ * (λ)). We have on the event (9.2.4):

min w ≤R √ n L λ (w) = min w ≤R √ n max β≥0 λ (w, β) ≥ min w ≤R √ n λ (w, β * ) ≥ min w ≤R √ n • λ (w, β * ) -K .
Using the fact that for w ∈ B(0, R √ n)

w 2 n + σ 2 = min σ≤τ ≤ √ σ 2 +R 2    w 2 n + σ 2 2τ + τ 2    , we obtain that min w ≤R √ n • λ (w, β * ) = min σ≤τ ≤ √ σ 2 +R 2 β * 2 σ 2 τ +τ - β 2 * 2 + 1 n min w ≤R √ n β * 2τ w 2 -β * g T w + λ|w + θ | -λ|θ | . For all τ ∈ [σ, √ σ 2 + R 2 ] the function g → min w ≤R √ n β * 2τ w 2 -β * g T w + λ|w + θ | -λ|θ | is β max R √ n-Lipschitz. Therefore F(τ, g) = β * 2 σ 2 τ + τ - β 2 * 2 + 1 n min w ≤R √ n β * 2τ w 2 -β * g T w + λ|w + θ | -λ|θ | is β 2 max R 2 n -1 -sub-Gaussian.
Therefore there exists constants C, c > 0 such that for all τ ∈ [σ, √ σ 2 + R 2 ], we have

P (|F(τ, g) -EF(τ, g)| > ) ≤ Ce -cn 2 . F(•, g) is almost surely a β max (1 + R 2 σ 2 )-Lipschitz function on [σ, √ σ 2 + R 2 ]
. Therefore, by an -net argument one can find constants C, c > 0 that only depend on (Ω, R), such that for all > 0 the event

   sup τ ∈[σ, √ σ 2 +R 2 ] |F(τ, g) -EF(τ, g)| ≤    (9.2.5)
has probability at least 1 -C e -cn 2 . On the event (9.2.5) we have then min

τ ∈[σ, √ σ 2 +R 2 ] F(τ, g) ≥ min τ ∈[σ, √ σ 2 +R 2 ]
E [F(τ, g)] -. Notice that for all τ > 0 we have

1 n E min w ≤R √ n β * 2τ w 2 -β * g T w + λ|w + θ | -λ|θ | ≥ 1 n N i=1 E min w i ∈R β * 2τ w 2 i -β * g i w i + λ|w i + θ i | -λ|θ i | = 1 δ E min w∈R β * 2τ w 2 -β * Zw + λ|w + Θ| -λ|Θ| ,
where the last expectation is with respect (Θ, Z) ∼ µ θ ⊗N (0, 1). Consequently on the event (9.2.4) and (9.2.5), we have

(1 + K) + min w ≤R √ n L λ (w) ≥ min σ≤τ ≤ √ σ 2 +R 2 ψ λ (β * , τ ) ≥ min σ≤τ ψ λ (β * , τ ) = ψ λ (β * , τ * ) .
By Proposition 9.6.1 we have that

ψ λ (β * , τ * ) ≥ L λ (w λ ) -,
with probability at least 1 -Ce -cn 2 . Then, for all ∈ (0, 1) we have with probability at least 1 -C e -cn 2 min

w ≤R √ n L λ (w) + (K + 2) ≥ L λ (w λ ) .
Lemma 9.2.1

Let f be a convex function on R N . Let w ∈ R N and r > 0. Suppose that f is γ-strongly convex on the ball B(w, r), for some γ > 0. Assume that

f (w) ≤ min x∈B(w,r) f (x) + ,
for some ≤ r 2 γ 8 . Then f admits a unique minimizer x * over R N . We have x * ∈ B(w, r) and therefore

x * -w 2 ≤ 2 γ .

Moreover, for every x ∈ R N such that f (x) ≤ min f + we have x -w 2 ≤ 8 γ .

Proof. f is convex on B(w, r), it admits therefore a minimizer x * on B(w, r). By strong convexity we have

x * -w 2 ≤ 2 γ ≤ r 2 4 .
Consequently, x * is in the interior of B(w, r). By strong convexity, x * is then the unique minimizer of f over R N . By strong convexity, for any x outside of B(w, r) we have

f (x) > f (x * ) + 1 2 γ r 2 2 ≥ f (x * ) + . Consequently, if f (x) ≤ min f + then x ∈ B(w, r) and thus x -x * 2 ≤ 2 γ .
Proof of Theorem 9.2.1. Let t = min( 116 β min , σ). By Lemma 9.6.1 the event

w λ 2 n - E w λ 2 n ≤ t 2 , g T w λ n ≤ E g T w λ n + t, (9.2.6) g ≤ 2 √ N , σg √ n ≤ β min 4 , (1 - β min 8τ max ) ≤ h √ n ≤ 2
has probability at least 1 -Ce -cn , for some constants C, c > 0. On the event (9.2.6)

w λ 2 n + σ 2 ≥ E w λ 2 n + σ 2 -t 2 ≥ τ * -t .
Therefore

w λ 2 n + σ 2 h √ n ≥ h √ n τ * - h √ n t ≥ τ * - β min 8 -2t ≥ τ * - β min 4 .
Consequently, on the event (9.2.6) we have

w λ 2 n + σ 2 h √ n - 1 n g T w λ + σg √ n ≥ τ * - 1 n E g T w λ - 3 4 β min ≥ 1 4 β min , because τ * -1 n E[g T w λ ] = β * ≥ β min .
Moreover, on the event (9.2.6) the function

f : w → w 2 n + σ 2 h √ n - 1 n g T w + g σ √ n is 2 √ N n + 2
√ n -Lipschitz. We have seen above that on (9.2.6), f (w λ ) ≥ 1 4 β min . Thus we can find a constant r > 0 such that on the event (9.2.6) we have for all w ∈ B(w λ , r

√ n) f (w) > 1 8 β min .
By Lemma 9.6.14, the function f is a n -strongly convex on B(w λ , r √ n), for some constant a > 0. For all w ∈ B(w λ , r √ n) we have

L λ (w) = 1 2 f (w) 2 + λ n (|w + θ | -|θ |) .
Compute the Hessian for w ∈ B(w λ , r √ n):

∇ 2 1 2 f 2 (w) = f (w)∇ 2 f (w) + ∇f (w)∇f (w) T aβ min 8n Id N ,
which means that L is γ n -strongly convex on B(w λ , r √ n), for some constant γ > 0.

Notice that it suffices to prove Theorem 9.2.1 for ∈ (0, q] for some constant q > 0. Let ∈ (0, γr 2 8 ). Let now apply Proposition 9.2.2 with R = τ max + r: with probability at least 1

-C e -cn 2 L λ (w λ ) ≤ min w ≤R √ n L λ (w) + . (9.2.7)
Notice that on the event (9.2.6),

w λ 2 n ≤ E w λ 2 n + t 2 ≤ E w λ 2 n + σ 2 = τ 2 * ≤ τ 2 max .
Therefore, on (9.2.6), B(w λ , r

√ n) ⊂ B(0, R √ n).
Using then (9.2.7) we get

L λ (w λ ) ≤ min w∈B(w λ ,r √ n) L λ (w) + .
Consequently, on the events (9.2.6) and (9.2.7) that have probability at least 1 -C e -cn 2 , Lemma 9.2.1 gives that for all w ∈ R N such that L λ (w) ≤ min In order to prove this, we start by showing that the optimal Lasso cost concentrates around L * (λ).

Proposition 9.3.2

There exists constants c, C > 0 that only depend on Ω such that for all closed set D ⊂ R N and for all ∈ (0, 1],

P min w∈R N C λ (w) -L * (λ) ≥ ≤ C e -cn 2 .
Proof. By Corollary 8.5.1, we have We thus get

P min w∈R N C λ (w) -L * (λ) ≥ ≤ 2P min w∈R N L λ (w) -L * (λ) ≥ ≤ C e -cn
P min w∈D C λ (w) ≤ min w∈R N C λ (w) + ≤ 2P min w∈D L λ (w) ≤ L * (λ) + 2 + C e -cn 2 ≤ 2P min w∈D L λ (w) ≤ min w∈R N L λ (w) + 3 + 2P min w∈R N L λ (w) < L * (λ) - + C e -cn 2 ≤ 2P min w∈D L λ (w) ≤ min w∈R N L λ (w) + 3 + C e -cn 2 ,
for some constants c, C > 0, because of Corollary 9.2.1.

Local stability of the empirical distribution of the Lasso estimator: proof of Theorem 8.5.3

For w ∈ R N , let µ 0 (w) be the probability distribution over R 2 defined by

µ 0 (w) = 1 N N i=1 δ (w i +θ i ,θ i )
Theorem 8.5.3 follows from Proposition 9.3.1 and the following Lemma.

Lemma 9.3.1

Assume that D = F p (ξ) for some ξ, p > 0. There exists constants γ, c, C > 0 that depend only on Ω, such that for all ∈ (0, 1 2 ] we have

P min w∈D L λ (w) ≤ min w∈R N L λ (w) + 3γ ≤ C -max(1,a) exp -cN 2 a log( ) -2 , where D = w ∈ R N W 2 ( µ 0 (w), µ * λ ) 2 ≥ and a = 1 2 + 1 p .
Proof. By Theorem 9.2.1 and Proposition 9.6.2 there exists constants γ, c, C > 0 such that for all ∈ (0, 1 2 ] the event

∀w ∈ R N , L λ (w) ≤ min v∈R N L λ (v) + 3γ ⇒ 1 N w -w λ 2 ≤ 5 W 2 µ * λ , µ 0 (w λ ) 2 ≤ 4 (9.3.2)
has probability at least

1 -C -1 e -cn 2 -C -a exp -cN 2 a log( ) -2 ≥ 1 -C -max(1,a) exp -cN 2 a log( ) -2 ,
where a = 1 2 + 1 p . On the event (9.3.2), we have for all w ∈ D :

1 N w -w λ 2 ≥ W 2 µ 0 (w), µ 0 (w λ ) 2 ≥ W 2 ( µ 0 (w), µ * λ ) -W 2 (µ * λ , µ 0 (w λ )) 2 ≥ 4 .
This gives that on the event (9.3.2), for all w ∈ D , L λ (w) > min L λ (w) + 3γ is therefore empty: the lemma is proved.

Local stability of the risk of the Lasso estimator

We prove here the analog of Theorem 8.5.3 for the risk of the Lasso estimator.

Theorem 9.3.1

for the risk of the Lasso estimator. There exists constants C, c, γ > 0 that only depend on Ω such that for all ∈ (0, 1]

sup λ∈[λ min ,λmax] sup θ ∈D P ∃θ ∈ R N , 1 N θ -θ 2 -R * (λ) 2 ≥ and L λ (θ) ≤ min L λ +γ ≤ C e -cN 2 .
Theorem 9.3.1 follows from Proposition 9.3.1 and the following Lemma.

Lemma 9.3.2

There exists constants γ, c, C > 0 that only depend on Ω such that for all ∈ (0, 1] we have

P min w∈D L λ (w) ≤ min w∈R N L λ (w) + 3γ ≤ C e -cn 2 , where D = w ∈ R N w -N R * (λ) 2 ≥ N .
Proof. By Theorem 9.2.1 and Lemma 9.6.1 there exists constants γ, c, C > 0 such that for all

∈ (0, 1] the event ∀w ∈ R N , L λ (w) ≤ min v∈R N L λ (v) + 3γ ⇒ 1 N w -w λ 2 ≤ 5 w λ -N R * (λ) 2 ≤ N 4 (9.3.
3) has probability at least 1 -C e -cn 2 . On the event (9.3.3), we have for all w ∈ D :

1 N w -w λ 2 ≥ 1 N w -w λ 2 ≥ 1 N √ N - 1 2 √ N 2 ≥ 4 .
This gives that on the event (9.

ξ > 0, p > 0. Define K = 2ξ + 2δσ 2 λ min . Then ∀θ ∈ F p (ξ), P ∀λ ≥ λ min , 1 N w λ ≤ KN (1/p-1) + ≥ 1 -e -n/2 .
Proof. Since F p (ξ) ⊂ F p (ξ) for p ≥ p, it suffices to prove the Proposition for p ∈ (0, 1]: we suppose now to be in that case. With probability at least 1 -e -n/2 we have z ≤ 2 √ n and therefore min L λ ≤ L λ (θ ) ≤ 2σ 2 + λ n |θ | for all λ ≥ 0. One has thus with probability at least 1 -e -n/2 ,

∀θ ∈ F 1 (ξ), ∀λ > 0, λ n | θ λ | ≤ L λ ( θ λ ) ≤ 2σ 2 + λ n |θ | , which implies that 1 N | θ λ | ≤ 2δσ 2 λ + ξN 1/p-1 since 1 N |θ | ≤ 1 N N i=1 |θ i | p 1/p ≤ ξN 1/p-1 .

Proposition 9.3.4

Assume that 0 < δ < 1 and σ > 0. Let s < s max (δ). Then, there exists constants c, K > 0 such that

∀θ ∈ F 0 (s), P ∀λ ∈ [λ min , λ max ], 1 n | w λ + θ | -|θ | ≤ K ≥ 1 -2e -cn .
Proposition 9.3.4 follows from the arguments of [START_REF] Tropp | Convex recovery of a structured signal from independent random linear measurements[END_REF] that we reproduce below. Lemma 9.

3.3

Suppose that θ ∈ F 0 (s) for some s < s max (δ). There exists constants c, a > 0 that only depend on (δ, s) such that with probability at least 1 -e -cn , for all w

∈ R N such that |w + θ | -|θ | ≤ 0 we have Xw 2 ≥ a w 2 . Proof. Define K = D | • |, θ = r>0 u ∈ R N |θ + ru| ≤ |θ | , the descent cone of the 1 -norm at θ . Define ν min (X, K) = inf Xx x ∈ K, x = 1 ,
Let ω(K) be the Gaussian width of K:

ω(K) = E sup u∈K, u =1 g, u ,
where the expectation is taken with respect to g ∼ N (0, Id N ). The following result goes back to Gordon's work, [START_REF] Gordon | Some inequalities for gaussian processes and applications[END_REF], [START_REF] Gordon | On milman's inequality and random subspaces which escape through a mesh in rn[END_REF]. It can be found in for instance [START_REF] Tropp | Convex recovery of a structured signal from independent random linear measurements[END_REF] (Proposition 3.3).

Proposition 9.3.5

For all t ≥ 0, Let S 0 denote the support of θ . Let g ∼ N (0, Id N ), α ≥ 0 and define

P √ n ν min (X, K) ≥ √ n -1 -ω(K) -t ≥ 1 -e -t 2 /2 . Recall that M s (α) = s(1 + α 2 ) + 2(1 -s) (1 + α 2 )Φ(-α) -αφ(α) is the "critical function" studied in Section 9.1.2. Lemma 9.3.4 For all α ≥ 0 ω(K) 2 ≤ N M s (α) = N s(1 + α 2 ) + 2(1 -s) (1 + α 2 )Φ(-α) -αφ(α) .
v i = sign(θ i ) if i ∈ S 0 , α -1 g i 1(|g i | ≤ α) + sign(g i )1(|g i | > α) otherwise.
Notice that v ∈ ∂|θ |, therefore by (9.3.4):

ω(K) 2 ≤ E sup u∈K, u =1 g, u 2 ≤ E g -αv 2 ≤ E   i∈S 0 (g i -αsign(θ i )) 2 + i ∈S 0 η(g i , α) 2   ≤ N s(1 + α 2 ) + 2N (1 -s)((1 + α 2 )Φ(-α) -αφ(α)) .
Since s ≤ s max (δ), there exists (see Lemma 9.1.10) α ≥ 0 and t ∈ (0, 1) such that M s (α) ≤ δ(1 -t) 2 . Consequently ω(K) ≤ √ n(1 -t). Therefore, there exists some constants a, c > 0 that only depends on s and δ such that

P (ν min (X, K) ≥ a) ≥ 1 -e -cn .
On the above event, for all w ∈ K, Xw 2 ≥ a 2 w 2 , which proves the Lemma.

Proof of Proposition 9.3.4. Let us work on the event

∀w ∈ R N , |w+θ |-|θ | ≤ 0 ⇒ Xw 2 ≥ a w 2 z ≤ 2 √ n , (9.3.5)
which has probability at least 1 -3e -cn/2 . Let λ ∈ [λ min , λ max ]. Notice that on the event (9.3.5) we have min C λ ≤ C λ (0) ≤ 2σ 2 and therefore

λ n | w λ + θ | -|θ | ≤ 2σ 2 .
We distinguish two cases:

Case 1: | w λ + θ | -|θ | ≥ 0. In that case we obtain 1 n | w λ + θ | -|θ | ≤ 2σ 2 λ min . Case 2: | w λ + θ | -|θ | ≤ 0. In that case 2σ 2 ≥ C λ ( w λ ) ≥ 1 2n X w λ -σz 2 - λ n | w λ | ≥ 1 4n X w λ 2 - σ 2 2n z 2 - λ √ N n w λ ≥ a 2n w λ 2 -2σ 2 - λ √ δn w λ .
This implies that there exists a constant

C = C(s, δ, σ) > 0 such that 1 √ n w λ ≤ C(1 + λ). One conclude -Cδ -1/2 (1 + λ max ) ≤ - 1 n | w λ | ≤ 1 n | w λ + θ | -|θ | ≤ 0 .
Lipschitz continuity of the limiting risk and empirical distribution Proposition 9.3.6

The function λ → µ * λ is M -Lipschitz on [λ min , λ max ] with respect to the Wasserstein distance W 2 , for some constant M = M (Ω) > 0.

Proof. Let λ 1 , λ 2 ∈ [λ min , λ max ]. W 2 (µ * λ 1 , µ * λ 2 ) 2 ≤ E (η(Θ + τ * (λ 1 )Z, α * (λ 1 )τ * (λ 1 )) -η(Θ + τ * (λ 2 )Z, α * (λ 2 )τ * (λ 2 ))) 2 ≤ 2E (τ * (λ 1 )Z -τ * (λ 2 )Z) 2 + (α * (λ 1 )τ * (λ 1 ) -α * (λ 2 )τ * (λ 2 )) 2 ≤ 2 (τ * (λ 1 ) -τ * (λ 2 )) 2 + 2(α * (λ 1 )τ * (λ 1 ) -α * (λ 2 )τ * (λ 2 )) 2 ≤ 2(1 + α 2 max ) (τ * (λ 1 ) -τ * (λ 2 )) 2 + 2τ 2 max (α * (λ 1 ) -α * (λ 2 )) 2 .
Since by Proposition 9.1.3 the functions λ → α * (λ) and λ → τ * (λ) are both M -Lipschitz on [λ min , λ max ], for some constant M = M (Ω) > 0, we obtain:

W 2 (µ * λ 1 , µ * λ 2 ) 2 ≤ 2M 2 (1 + α 2 max + τ 2 max )(λ 1 -λ 2 ) 2 ,
which proves the Lemma.

Proposition 9.3.7

The function λ

→ R * (λ) = δ(τ * (λ) 2 -σ 2 ) is M -Lipschitz on [λ min , λ max ], for some constant M = M (Ω) > 0.
Proof. This is a consequence of Proposition 9.1.3. Assume that D is either F 0 (s) or F p (ξ) for some s < s max (δ) and ξ ≥ 0, p > 0. Define

Proofs of

q =    (1/p -1) + if D = F p (ξ) , 0 if D = F 0 (s) .
Then there exists constants K, C, c > 0 that depend only on Ω such that for all θ ∈ D

P ∀λ, λ ∈ [λ min , λ max ], L λ ( θ λ ) ≤ min x∈R N L λ (x) + KN q |λ -λ | ≥ 1 -Ce -cn . ( 9
L λ ( θ λ ) = L λ ( θ λ ) + (λ -λ) 1 n | θ λ | ≤ L λ ( θ λ ) + (λ -λ) 1 n | θ λ | = L λ ( θ λ ) + 1 n (λ -λ ) | θ λ | -| θ λ | ≤ min θ∈R N L λ (θ) + 1 n |λ -λ| | θ λ | -|θ | + | θ λ | -|θ | ≤ min θ∈R N L λ (θ) + 2KN q |λ -λ | .
∈ {1 . . . k}, ∀θ ∈ R N , L λ i (θ) ≤ min x∈R N L λ i (x)+γ ⇒ W 2 ( µ (θ,θ ) , µ * λ i ) 2 ≤ (9.3.8)
has probability at least 1 -CN q -max(1,a)-1 e -cN 2 a log( ) -2 . Therefore, on the intersection of the event in (9.3.6) and the event (9.3.8) we have for all λ ∈ [λ min , λ max ]

L λ i ( θ λ ) ≤ min x∈R N L λ i (x) + 2KN q |λ -λ i | ≤ min x∈R N L λ i (x) + γ , where 1 ≤ i ≤ k is such that λ ∈ [λ i-1 , λ i ].
This implies (since we are on the event (9.3.8)) that

W 2 µ ( θ λ ,θ ) , µ * λ i 2 ≤ . We conclude by W 2 µ ( θ λ ,θ ) , µ * λ 2 ≤ 2W 2 µ ( θ λ ,θ ) , µ * λ i 2 + 2W 2 (µ * λ , µ * λ i ) 2 ≤ 2 + 2M 2 (λ -λ i ) 2 ≤ 4 .
This proves the Theorem.

Proof of Theorem 8.

= M (Ω) > 0 such that λ → R * (λ) is M -Lipschitz on [λ min , λ max ],
as given by Proposition 9.3.7.

Let ∈ (0, 1] and define = min γ 2KN q , M +1 . Let k = (λ max -λ min )/ . Define, for i = 0, . . . , k:

λ i = λ min + i . By Theorem 9.3.1, the event ∀i ∈ {1 . . . k}, ∀θ ∈ R N , L λ i (θ) ≤ min x∈R N L λ i (x)+γ ⇒ 1 N θ-θ 2 -R * (λ i ) 2 ≤ (9.3.9)
has probability at least 1 -kC -1 e -cN 2 ≥ 1 -CN q -2 e -cN 2 . Therefore, on the intersection of the event in (9.3.6) and the event (9.3.9) we have for all λ ∈ [λ min , λ max ]

L λ i ( θ λ ) ≤ min θ∈R N L λ i (θ) + 2KN q |λ -λ i | ≤ min θ∈R N L λ i (θ) + γ , where 1 ≤ i ≤ k is such that λ ∈ [λ i-1 , λ i ].
This implies (since we are on the event (9.3.9)) that

1 N θ λ -θ 2 -R * (λ i ) 2 ≤ . We conclude by 1 N θ λ -θ 2 -R * (λ) 2 ≤ 2 1 N θ λ -θ 2 -R * (λ i ) 2 + 2 R * (λ i ) -R * (λ) 2 ≤ 2 + 2M 2 ( ) 2 ≤ 4 .
This proves (8.3.8).

9.4 Study of the Lasso residual: proof of (8.3.9)- (8.3.10) This Section is devoted to the proof of (8.3.9)-(8.3.10) from Theorem 8.3.2. Let us define

u λ = X w λ -σz = X θ λ -y . u λ is the unique maximizer of u → min w∈R N u T Xw -σu T z - 1 2 u 2 + λ(|θ + w| -|θ |) .
In Section 9.4.2 below, we prove the following Theorem:

Theorem 9.4.1

There exists constants c, C > 0 such that for all ∈ (0, 1], all θ ∈ D and all λ ∈ [λ min , λ max ] P We have almost surely

1 n u λ 2 -β * (λ) 2 2 ≥ ≤ C e -cn
min w∈R N max u∈R n c λ (w, u) = max u∈R n min w∈R N c λ (w, u) = c λ ( w λ , u λ ) .
Proof. By definition of w λ and u λ we have

c λ ( w λ , u λ ) = min w∈R N max u∈R n c λ (w, u) ≥ max u∈R n min w∈R N c λ (w, u) .
Let us prove the converse inequality. The optimality condition of w λ gives that there exists v ∈ ∂|θ + w λ | such that

X T u λ + λv = X T (X w λ -σz) + λv = 0 .
The function w → c λ (w, u λ ) is convex and

1 n X T u λ + λ n v = 0
is a subgradient at w λ . Therefore min w∈R N c λ (w, u λ ) = c λ ( w λ , u λ ), which proves the lemma.

We compute now Now by the same reasoning than Corollary 8.5.1 (we omit here the details for the sake of brevity) we have

P u λ ∈ D = P max u∈D min w∈R N c λ (w, u) ≥ max u∈R n min w∈R N c λ (w, u) ≤ P max u∈D min w∈R N c λ (w, u) ≥ L * (λ) -+ P max u∈R n min w∈R N c λ (w, u) ≤ L * (λ) -. By Lemma 9.
P max u∈D min w∈R N c λ (w, u) ≥ L * (λ) - ≤ 2P max u∈D min w∈R N m λ (w, u) ≥ L * (λ) - = 2P max u∈D U λ (u) ≥ L * (λ) -. Since U λ ≤ U λ we obtain P max u∈D min w∈R N c λ (w, u) ≥ L * (λ) - ≤ 2P max u∈D U λ (u) ≥ L * (λ) -.
Let E be the event of Lemma 9.4.1 above and let us work on the event

E 1 √ n u λ + σz -P * (λ) ≤ 1/2 , ( 9.4.3)
which has probability at least 1-C e -cn 2 (the fact that the second event in the intersection has this probability follows from standard concentration arguments as in Section 9.6.2). Let now u ∈ D , by the definition of D and the event above we have 1

√ n u -u λ ≥ 5 1/2 and thus 1 √ n u -u * λ ≥ 4 1/2 . By 1/n-strong concavity of U λ we get U λ (u) ≤ max u ∈R n U λ (u ) -8 ≤ L * (λ) -7 . Consequently P max u∈D U λ (u) ≥ L * (λ) - ≤ C e -cn 2
, which proves the result.

where (Θ, Z) ∼ µ θ ⊗ N (0, 1). For v ∈ R N we define

µ (v,θ ) = 1 N N i=1 δ (v i ,θ i ) .
Theorem 9.5.1

Assume that D = F p (ξ) for some ξ, p > 0. There exists constants C, c > 0 that only depend on Ω such that for all λ ∈ [λ min , λ max ] and all ∈ (0, 1 2 ],

sup

θ ∈D P W 2 ( µ ( v λ ,θ ) , ν * λ ) 2 ≥ ≤ C -max(1,a) exp -cN 2 a log( ) -2 ,
where a = 1 2 + 1 p .

Theorem 9.5.1 is proved in Section 9.5.3.

Theorem 9.5.2

Let D be F p (ξ) for some ξ > 0 and p > 0. For all ∈ (0,

W 2 ( µ ( v λ ,θ ) , ν * λ ) 2 ≥ ≤ C -max(1,a)-1 N (1/p-1) + e -cN 2 a log( ) -2 ,
where a = 1 2 + 1 p .

Theorem 9.5.2 is deduced from Theorem 9.5.1 in Section 9.5.3.

The norm of the subgradient

Let us define

κ * (λ) = β * (λ) 2 λ 2 1 + δ -2s * (λ) -δ σ 2 τ * (λ) 2 .
(9.5.2)

Theorem 9.5.3

There exists a constant C, c > 0 such that for all λ ∈ [λ min , λ max ] and all ∈ (0, 1],

sup

θ ∈D P 1 N v λ 2 -κ * (λ) 2 ≥ ≤ C e -cn 2 .
Theorem 9.5.3 is proved in Section 9.5.3. We deduce as before: Theorem 9.5.4

Let D be either F 0 (s) for s < s max (δ) or F p (ξ) for some ξ > 0 and p > 0. There exists constants C, c > 0 such that for all ∈ (0, 1],

sup

θ ∈D P sup λ∈[λ min ,λmax] 1 N v λ 2 -κ * (λ) 2 ≥ ≤ C 2 N q e -cn 2 ,
where q = 0 if D = F 0 (s) and q = (1/p -1) + if D = F p (ξ).

Theorem 9.5.4 is deduced from Theorem 9.5.3 in Section 9.5.3.

Upper bound on the sparsity of the Lasso estimator

Studying v λ allows to get an upper-bound on the 0 norm of θ λ . Indeed if θ λ,i = 0 then

| v λ,i | = 1: therefore θ λ 0 ≤ # i | v λ,i | = 1 .
For this reason, the following results will be used to prove Theorem 9.6.1 in Section 9.6.4.

Theorem 9.5.5

There exists constants C, c > 0 such that for all λ ∈ [λ min , λ max ] and all ∈ (0, 1],

sup

θ ∈D P 1 N # i | v λ,i | ≥ 1 -≥ s * (λ) + 2(1 + α max ) ≤ C 3 e -cn 6 .
Theorem 9.5.5 is proved in Section 9.5.3.

Theorem 9.5.6

Let D be either F 0 (s) for s < s max (δ) or F p (ξ) for some ξ > 0 and p > 0. We have for all ∈ (0, 1],

sup

θ ∈D P ∃λ ∈ [λ min , λ max ], 1 N # i | v λ,i | = 1 ≥ s * (λ) + ≤ C 6 N q e -cn 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1) + if D = F p (ξ).

Theorem 9.5.6 is deduced from Theorem 9.5.5 in Section 9.5.3.

Gordon's strategy for the subgradient Application of Gordon's Theorem

Let g ∼ N (0, Id N ) and h ∼ N (0, Id n ) be independent standard Gaussian vectors. We define:

V λ (v) = min w∈B      1 2   1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n   2 + + λ n v T (w + θ ) - λ n |θ |     
.

The following Proposition is the analog of Corollary 8.5.1.

Recall that w * λ is by Lemma 9.2.1 the unique minimizer of L λ over R N . Lemma 9.5.2

With probability at least 1 -2e -n/2 we have

min w∈R N L λ (w) = max v ∞≤1 V λ (v)
and the vector

v * λ = -λ -1   w * λ 2 n + σ 2 h √ n - 1 n g T w * λ + g σ √ n   +   h √ n w * λ w * λ 2 /n + σ 2 -g   (9.5.5) verifies v * λ ∞ ≤ 1 and is a maximizer of V λ .
Proof. By Proposition C.2, one can switch the min-max:

max v ∞≤1 V λ (v) = min w∈B max v ∞≤1      1 2   w 2 n +σ 2 h √ n - 1 n g T w+ g σ √ n   2 + + λ n v T (w+θ )- λ n |θ |      = min w∈B L λ (w) .
Let us work on the event h ≤ 2 √ n ∩{g ≤ √ n} which has probability at least 1-2e -n/2 . We have

λ n (|w * λ +θ |-|θ |) ≤ L λ (w * λ ) ≤ L λ (0) ≤ 5σ 2 . This gives w * λ ∈ B and thus max v ∞≤1 V λ (v) = min w∈B L λ (w) = min w∈R N L λ (w).
The optimality condition of w * λ gives that

v * λ = -1 λ   w * λ 2 n + σ 2 h √ n - 1 n g T w * + g σ √ n   +   h √ n w * λ w * λ 2 /n + σ 2 -g   ∈ ∂|θ + w * λ | . Therefore v * T λ (w * λ + θ ) = |w * λ + θ |.
Using the optimality condition again we obtain

V λ (v * λ ) = min w∈B    1 2 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n 2 + + λ n v * T λ (w + θ ) - λ n |θ |    = 1 2 1 n w * λ 2 + σ 2 h √ n - 1 n g T w * λ + g σ √ n 2 + + λ n |w * λ + θ | - λ n |θ | = min w∈R N L λ (w) = max v ∞≤1 V λ (v) .
Therefore v * λ achieves the optimal value.

Proposition 9.5.2

For all θ ∈ D and all λ ∈ [λ min , λ max ] we have for all ∈ (0, 1]

P 1 N v * λ -v λ 2 ≥ ≤ C e -cn .
Proof. By Theorem 9.2.1 we have for all ∈ (0, 1]

P 1 N w * λ -w λ 2 ≥ ≤ C e -cn ,
so we deduce the result from the expression (9.5.5) of v * λ and the concentration properties of w λ (see Section 9.6.2).

By the same arguments used for proving Lemma 9.5.2 it is not difficult to prove:

* λ =   w * λ 2 n + σ 2 h √ n - 1 n g T w * λ + g σ √ n   + .
Moreover, for all ∈ (0, 1] we have

P |b * λ -β * (λ)| > ≤ C e -cn .
Proof of Theorem 9.5.7

Let v ∈ R N such that v ∞ ≤ 1. We have by Proposition C.2 V λ (v) = min w∈B max β≥0 β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 + λ n v T (w + θ ) - λ n |θ | = max β≥0 min w∈B β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 + λ n v T (w + θ ) - λ n |θ | = max β≥0 min 0≤r≤R β r 2 + σ 2 h √ n - r √ n βg -λv + β g σ √ n - β 2 2 + λ n v T θ - λ n |θ | ,
because the minimization over the direction of w is easy to perform. Let us define for κ > 0

D κ = v ∈ B ∞ (0, 1), 1 N v -v * λ 2 ≤ κ 2 and V λ (v) ≥ max v ∞≤1 V λ (v ) - 1 8 κ 2 .
By concavity of V λ , D κ is convex.

Proposition 9.5.3

There exists a constant κ > 0 such that with probability at least 1 -Ce -cn we have ∀v

∈ D κ , V λ (v) = V λ (v) where V λ (v) = min w∈R N max β∈[β * -κ,β * +κ] β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 + λ n v T (w + θ ) - λ n |θ | .
In order to prove Proposition 9.5.3, we start with a Lemma: Lemma 9.5.4

For all v ∈ D κ , the function

f v : β → min w∈B β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 + λ n v T (w + θ ) - λ n |θ | admits a unique maximizer b λ (v) on [0, +∞) and one has |b λ (v) -b * λ | ≤ κ/2. Proof. Let v ∈ D κ . f v is 1-strongly concave so it admits a unique maximizer b λ (v) on R ≥0 . We have f v (b λ (v)) = max β≥0 f v (β) = V λ (v) ≥ max v ∞≤1 V λ (v ) - 1 8 κ 2 , because v ∈ D κ . Notice now that f v (b λ (v)) ≤ min w∈B λ (w, b λ (v)) because v T (w + θ ) ≤ |w + θ |.
Permuting the min-max (using Proposition C.2), we have max

v ∞≤1 V λ (v ) = max

Lemma 9.5.5

There exist constants K, κ > 0 such that with probability at least 1 -Ce -cn the following happens. For all

β ≥ 0, v ∈ R N such that |β -b * λ | ≤ 2κ and v -v * λ ≤ √ N κ the minimum over R N of w → β   1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n   - β 2 2 + λ n v T (w + θ ) - λ n |θ | is achieved on B(0, √ N K).
Proof. The minimization with respect to the direction of w is easy to perform: w has to be a non-negative multiple of βg -λv. It remains thus to minimizes with respect to the norm of w.

We have to show that under the conditions of the lemma, the minimum of

r ≥ 0 → β r 2 + σ 2 - r h βg -λv (9.5.6)
is achieved for r smaller than some constant. By Theorem 9.2.1 and Lemma 9.5.3 there exists a constant R > 0 (for instance R = τ max + 1) such that the event Let us now work on the event (9.5.

w * λ 2 ≤ nR 2 b * λ ≥ β min /2 h ≥ √ n/2 ( 9
7). Let v ∈ R N and β ≥ 0 such that v -v * λ ≤ √ N κ and |β -b * λ | ≤ 2κ. We have g - λ β v ≤ g - λ b * λ v * λ + λ β v - λ b * λ v * λ . Compute g - λ b * λ v * λ = h w * λ / √ n w * λ 2 n + σ 2 ≤ h a ,
with probability at least 1 -Ce -cn . Now

1 β v - 1 b * λ v * λ ≤ 1 β * v -v * λ + 1 β v - 1 b * λ v ≤ 2 β min v -v * λ + √ N min(β, b * λ ) 2 |β -b * λ | ≤ 2 β min v -v * λ + 16 √ N β 2 min |β -b * λ | ≤ 1 -a 4 √ n .
Putting all together:

1 h g - λ β v ≤ a + 1 -a 2 = 1 + a 2 < 1 .
This gives that the minimum of (9.5.6) is achieved for r

≤ σ √ 1-((1+a)/2) 2 . One can thus chose K = δσ √ 1-((1+a)/2) 2 .

Proof of

λ (v) of f v verify |b λ (v) -b * λ | ≤ κ/2 and therefore |b λ (v) -β * | ≤ κ. Consequently V λ (v) = max β∈[β * -κ,β * +κ] min w∈B β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 + λ n v T (w + θ ) - λ n |θ | . Now, for β ∈ [β * -κ, β * + κ], we have |β -b * λ | ≤ 2κ.
Since we are working on the event of Lemma 9.5.5, we obtain

V λ (v) = max β∈[β * -κ,β * +κ] min w∈R N β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 + λ n v T (w + θ ) - λ n |θ | ,
and Proposition 9.5.3 follows from the permutation of the minmax using Proposition C.2.

Lemma 9.5.6

There exists a constant C, c, γ > 0 such that V λ is γ/N -strongly concave, with probability at least 1 -Ce -cn .

Proof. The function f * : R N → R defined by

f * (v) = max w∈R N v T w - n λ max β∈[β * -κ,β * +κ] β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2
is the convex conjugate of the convex function

f : w ∈ R N → n λ max β∈[β * -κ,β * +κ] β 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n - β 2 2 = n λ ϕ 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n ,
where ϕ is the

C 1 function ϕ(x) =        1 2 x 2 if x ∈ [β * -κ, β * + κ] , (β * -κ)x -1 2 (β * -κ) 2 if x ≤ β * -κ , (β * + κ)x -1 2 (β * + κ) 2 if x ≥ β * + κ .
f is a proper closed convex function (because f is convex and its domain is R N ), therefore its convex conjugate f * is also a proper closed convex function. The Fenchel-Moreau Theorem gives then that f = f * * . Let us compute the gradient of

f for w ∈ R N ∇f (w) = n λ ϕ 1 n w 2 + σ 2 h √ n - 1 n g T w + g σ √ n   h √ n w/n w 2 n + σ 2 - g n   .
It is not difficult to verify that there exists a constant L such that ∇f is L-Lipschitz on R N , with probability at least 1 -Ce -cn . f = f * * is therefore 1/L-strongly smooth (see Definition C.1). By Proposition C.3 this gives that f * is 1/L strongly convex. One deduces then that V λ is γ/Nstrongly concave with γ = λ/(Lδ).

Let 0 < γ < 1/2 be a constant that verify the statement of Lemma 9.5.6 and let κ > 0 be a constant given by Proposition 9.5.3. Notice that it suffices to prove Theorem 9.5.7 for small enough and let ∈ (0, κ 2 ).

P ∃v ∈ B ∞ (0, 1), 1 N v -v λ 2 > and V λ (v) ≥ max v ∞≤1 V λ (v ) - 1 4 γ ≤ P ∃v ∈ B ∞ (0, 1), 1 N v -v * λ 2 > 2 and V λ (v) ≥ max v ∞≤1 V λ (v ) - 1 4 γ + C e -cn 2 ≤ P ∃v ∈ D κ , 1 N v -v * λ 2 > 2 and V λ (v) ≥ max v ∈Dκ V λ (v ) - 1 4 γ + C e -cn 2 , (9.5.8) because, if there exists v ∈ B ∞ (0, 1) such that 1 N v -v * λ 2 > 2 and V λ (v) ≥ max v ∞≤1 V λ (v )- 1
4 γ , we can construct v ∈ D κ that verifies the same conditions. Indeed:

• if 1 N v -v * λ 2 ≤ κ 2 , one simply take v = v. • otherwise, v = v * λ + κ(v -v * λ )/ v -v * λ is in D κ and by concavity V λ ( v) ≥ V λ (v).
Since with probability at least 1 -Ce -cn we have V λ (v) = V λ (v) for all v ∈ D κ and V λ is γ/N -strongly concave, the probability in (9.5.8) above is less that Ce -cn .

Proofs of the main results about the subgradient

Let us start with the analog of Proposition 9.3.1 for the costs functions V λ and V λ :

The empirical law of v λ : proof of Theorem 9.5.1

Theorem 9.5.1 follows now from Proposition 9.5.4 and the following Lemma. Lemma 9.5.8

There exists constants γ, c, C > 0 that depend only on Ω, such that for all ∈ (0, 1 2 ] we have

P max v∈D V λ (v) ≥ max v ∞≤1 V λ (v) -3γ ≤ C e -cn 2 log( ) -2 , where D = v ∈ B ∞ (0, 1) W 2 ( µ (v,θ ) , ν * λ ) 2 ≥ .
Proof. By Theorem 9.5.7 and Proposition 9.6.2 there exists constants γ, c, C > 0 such that for all ∈ (0, 1 2 ] the event

∀v ∈ B ∞ (0, 1), V λ (v) ≥ max B∞(0,1) V λ -3γ ⇒ 1 N v -v λ 2 ≤ 5 W 2 ν * λ , µ (v λ ,θ ) 2 ≤ 4 (9.5.10) has probability at least 1 - C e -cn 2 -C -a exp -cN 2 a log( ) -2 ≥ 1 -C -max(1,a) exp -cN 2 a log( ) -2 .
On the event (9.5.10), we have for all v ∈ D :

1 N v -v λ 2 ≥ W 2 µ (v,θ ) , µ (v λ ,θ ) 2 ≥ W 2 ( µ (v,θ ) , ν * λ ) -W 2 (ν * λ , µ (v λ ,θ ) ) 2 ≥ 4 .
This gives that on the event (9.5.10), for all

v ∈ D , V λ (v) < max v ∞≤1
V λ (v )-3γ . The intersection of (9.5.10) with the event max

v∈D V λ (v) ≥ max v ∞≥1
V λ (v) -3γ is therefore empty: the lemma is proved.

Proof of Theorem 9.5.1. Let γ > 0 be a constant that verify the statement of Lemma 9.5.8.

Let ∈ (0, 1 2 ] and define

D = v ∈ B ∞ (0, 1) W 2 ( µ v , ν * λ ) 2 ≥ D is a closed set. P ∃v ∈B ∞ (0, 1), W 2 ( µ v , ν * λ ) 2 ≥ and V λ (v) ≥ max V λ -γ = P max v∈D V λ (v) ≥ max v ∞≤1 V λ (v) -γ ≤ 2P max v∈D V λ (v) ≥ max v ∞≤1 V λ (v) -3γ + Ce -cn ≤ C e -cn 2 log( ) -2 ,
where we used successively Proposition 9.5.4 and Lemma 9.5.8.

Proof of Theorem 9.5.5 Lemma 9.5.9

There exists constants γ, c, C > 0 that depend only on Ω, such that for all ∈ (0, 1] we have

P max v∈D V λ (v) ≥ max v ∞≤1 V λ (v) -3γ 3 ≤ C 3 e -cn 6 , where D = v ∈ B ∞ (0, 1) 1 N # i |v i | ≥ 1 -> s * (λ) + 2(1 + α max ) .
Proof. Let ∈ (0, 1] and define

s = 1 N # i ∈ {1, . . . , N } |v λ,i | ≥ 1 -2 .
s is the mean of independent Bernoulli random variables. By Hoeffding's inequality we have

P s ≤ P |τ -1 * Θ + Z| ≥ α * -2α * + ≥ 1 -e -2N 2 . Compute P |τ -1 * Θ + Z| ≥ α * -2α * = E Φ Θ τ * (λ) -α * (λ) + 2α * (λ) + Φ - Θ τ * (λ) -α * (λ) + 2α * (λ) ≤ s * (λ) + 2α max .
We obtain P s ≤ s * (λ) + 2α max + ≥ 1 -e -2N 2 . By Theorem 9.5.7 there exists a constant γ > 0 such that the event

∀v ∈ B ∞ (0, 1), V λ (v) ≥ max B∞(0,1) V λ -3γ 3 ⇒ 1 N v -v λ 2 < 3 s ≤ s * (λ) + α max +
has probability at least 1 -C 3 e -cn 6 . We have on this event, for all

v ∈ D , 1 N v -v λ 2 ≥ 3 .
Therefore, on the above event we have max

v∈D V λ (v) < max v ∞≤1 V λ (v) -3γ 3 , which concludes the proof.
Proof of Theorem 9.5.5. Let γ > 0 be a constant that verify the statement of Lemma 9.5.9.

Let ∈ (0, 1] and define

D = v ∈ B ∞ (0, 1) 1 N # i |v i | ≥ 1 -≥ s * (λ) + 2(1 + α max ) . D is a closed set. P 1 N # i | v λ,i | ≥ 1 -≥ s * (λ) + 2(1 + α max ) ≤ P max v∈D V λ (v) ≥ max v ∞≤1 V λ (v) -γ ≤ 2P max v∈D V λ (v) ≥ max v ∞≤1 V λ (v) -3γ + C e -cn ≤ C e -cn 2 ,
where we used successively Proposition 9.5.4 and Lemma 9.5.9. Assume that D is F 0 (s) or F 1 (ξ) for some s < s max (δ) and ξ ≥ 0, p > 0. Let q = 0 if D = F 0 (s) and q = (1/p -1) + if D = F p (ξ). Then there exists constants K, C, c > 0 that depend only on Ω such that for all θ ∈ D

Uniform control over

P ∀λ, λ ∈ [λ min , λ max ], 1 N v λ -v λ 2 ≤ KN q |λ -λ | ≥ 1 -Ce -cn . ( 9
, λ ∈ [λ min , λ max ], 1 n u λ -u λ 2 ≤ KN q |λ -λ | .
Notice now that v λ = -1 λ X T u λ and that with probability at least 1-2e -n/4 , σ max (X) ≤ δ -1/2 +2 (by Proposition 9.7.4) which combined with the above inequality, prove the Proposition. Let k ≤ N and define the vector θ = (N, 2N, . . . , kN, 0, . . . , 0). With the definitions given in Remark 8.3.2, we claim that W 2 ( µ ( θ λ ,θ ) , µ * λ ) ≥ k/N with probability at least 1 -e -ck , for some constant c > 0. Indeed, consider the case λ = 0, τ * = 1, and let P, E denote probability and expectation with respect to the coupling that achieves the Wasserstein distance. This is a coupling for a triple of random variables (I, Θ, Z), with

I ∼ Unif {1, . . . , N } , (Θ, Z) ∼ µ θ ⊗ N (0, 1), with W 2 ( µ ( θ λ ,θ ) , µ * λ ) 2 = E (θ I -Θ) 2 + E (θ I + z I -Θ -Z) 2 ≡ A + B . ( 9.6.1)
We will proceed to bound separately the two terms above. Define δ i ≡ P(Θ = θ i |I = i), and δ max ≡ max i≤k δ i . Since Θ ∈ {0, N, . . . , kN } with probability one, we have

A ≥ 1 N k i=1 E (θ I -Θ) 2 I = i δ i ≥ N k i=1 δ i ≥ N δ max . (9.6.2)
For the second term, we have

B ≥ k i=1 E (θ I + z I -Θ -Z) 2 1 I=i 1 Θ=θ i = k i=1 E (z i -Z) 2 1 I=i 1 Θ=θ i (9.6.3) = k i=1 E (z i -Z) 2 1 Θ=θ i - k i=1 E (z i -Z) 2 1 I =i 1 Θ=θ i . (9.6.4)
Note that, by the coupling definition, P

I = i Θ = θ i = N P I = i ; Θ = θ i = 1 -δ i .
Using the fact that Θ and Z are independent random variables, together with Cauchy-Schwartz inequality,we get .6.8) where in the last step we used the fact that Z ∼ N (0, 1). Using (3 + 6x 2 + x 4 ) ≤ 4(1 + x 2 ) 2 , we thus conclude

B ≥ k i=1 E (z i -Z) 2 } P(Θ = θ i ) - k i=1 E (z i -Z) 4 1 Θ=θ i 1/2 P I = i; Θ = θ i 1/2 (9.6.5) ≥ 1 N k i=1 E (z i -Z) 2 } - 1 N k i=1 E (z i -Z) 4 1/2 P I = i Θ = θ i 1/2 (9.6.6) ≥ 1 N k i=1 E (z i -Z) 2 } - 1 N k i=1 δ 1/2 i E (z i -Z) 4 1/2 (9.6.7) ≥ 1 N k i=1 (1 + z 2 i ) - 1 N k i=1 δ 1/2 i 3 + 6z 2 i + z 4 i 1/2 . ( 9
B ≥ 1 -2δ 1/2 max N k i=1 (1 + z 2 i ) . ( 9
.6.9)

By concentration properties of chi-squared random variables, for any ε > 0, there exists c(ε) > 0 such that, with probability at least 1 -e -ck we have

1 k k i=1 z 2 i ≥ 1 -2ε.
Hence, with the same probability

W 2 ( µ ( θ λ ,θ ) , µ * λ ) 2 ≥ N δ max + 2k N (1 -2δ 1/2 max )(1 -ε) (9.6.10) ≥ k N . (9.6.11)
The last inequality follows by lower bounding the first term for δ max > 1/N , and the second for δ max ≤ 1/N , and fixing ε a sufficiently small constant.

Concentration properties of w λ

We prove in this section concentrations of the norms and some scalar product of w λ . Lemma 9.6.1

There exists constants c, C > 0 that only depend on Ω such that for all t ≥ 0 the event

1 n g T w λ -E 1 n g T w λ ≤ t , w λ 2 n -E w λ 2 n ≤ t and |w λ + θ | n -E |w λ + θ | n ≤ t
has probability at least 1 -Ce -ct 2 n -Ce -ctn .

Proof. The function

g → w λ = (η(θ i + τ * (λ)g i , α * (λ)τ * (λ)) -θ i ) 1≤i≤N
is τ max -Lipschitz. Consequently:

• g → |w λ +θ | n is δ -1/2 n -1/2 τ max -Lipschitz. Therefore |w λ +θ | n is τ 2 max δ -1 n -1 sub-Gaussian: for all t ≥ 0, P |w λ + θ | n -E |w λ + θ | n > t ≤ 2e -nt 2 δ/τ 2 max . • g → w λ √ n is n -1/2 τ max -Lipschitz. Therefore w λ √ n is τ 2 max n -1 sub-Gaussian. Its expectation is bounded by E w λ √ n ≤ (E w λ 2 n ) 1/2 = τ * ≤ τ max . By Proposition 9.7.3, we obtain that w λ 2 n
is (Cn -1 , Cn -1 )-sub-Gamma for some constant C and therefore for all t ≥ 0,

P w λ 2 n -E w λ 2 n > t ≤ 2e -cnt 2 + 2e -cnt . Now for i ∈ {1, . . . , N }, g i w λ,i = τ * g 2 i + g i w λ,i -τ * g i . |w λ,i -τ * g i | ≤ α * τ * and g i is 1-sub-Gaussian and E[|g i |] = 2 π ≤ 1.
Consequently, Lemma 9.7.1 gives that g i (w λ,i -τ * g i ) is 48τ 2 * α 2 * -sub-Gaussian. This gives that 1 n g T (w λ -τ * g) concentrates exponentially fast around its mean. So does 1 n g 2 .

Lemma 9.6.4

With probability at least 1 -e -1 128 N 2 , we have

W 2 ( µ, µ |r ) 2 ≤ + e -r 2 /2 . Proof. Obviously W 2 ( µ, µ |r ) 2 ≤ 1 N N i=1 (g i -g i|r ) 2
. The function x → x -x |r is 1-Lipschitz, so the variables (g i -g i|r ) 2 are i.i.d. [START_REF] Banks | Information-theoretic bounds and phase transitions in clustering, sparse pca, and submatrix localization[END_REF][START_REF] David | Representations for partially exchangeable arrays of random variables[END_REF]-sub-Gamma. Therefore for all ∈ [0, 1],

P 1 N N i=1 g i -g i|r 2 > E Z -Z |r 2 + ≤ e -1 128 N 2 .
And we conclude using E Z -Z |r 2 ≤ e -r 2 /2 , which we proved in the lemma above.

We need now some concentration results for empirical measures, in Wasserstein distance. The next proposition follows from a direct application of Theorem 2 from [START_REF] Fournier | On the rate of convergence in wasserstein distance of the empirical measure[END_REF] to distributions with bounded support. Notice that the results from [START_REF] Fournier | On the rate of convergence in wasserstein distance of the empirical measure[END_REF]

l = -R + (l -1)δ, -R + lδ ,
for l = 1, . . . , k. We define also B 0 = (-∞, R) ∪ [R, +∞). For l = 0, . . . k we write

I l = {i | θ i ∈ B l } and N l = #I l .
Let t > 0. Let l ∈ {1, . . . , k}. The random variables (g i|r ) i∈I l are i.i.d. and bounded by r. By the proposition above, one can couple i l ∼ Unif(I l ) with Z l ∼ N (0, 1) such that we have with probability at least 1 -Ce -ct 2 N .

E (Z l|r -g i l |r ) 2 ≤ tr 2 N N l ,
where E denotes the expectation with respect to i l and Z l . Let j l ∼ Unif(I l ) independently of everything else.

For l = 0, we define (i 0 , Z 0 ) ∼ Unif(I 0 ) ⊗ N (0, 1), independently of everything else. We have with probability at least 1 -Ce -ct 2 N :

E (Z 0|r -g i 0 |r ) 2 = E Z 2 0|r + E g 2 i 0 |r ≤ 2 + tr 2 N N 0 ,
where E denotes the expectation with respect to Z 0 and i 0 . Indeed, E g 2

i 0 |r = 1 N 0 i∈I 0 g 2 i|r ≤ 1 + tr 2 N
N 0 with probability at least 1 -Ce -ct 2 N . The equality comes from the fact that Z 0 and i 0 are independent. Finally, we define j 0 = i 0 .

Let us now define the random variable L whose law is given by P(L = l) = N l N , independently of everything else. Define

Y 1 = θ j L , Z L|r , Y 2 = θ i L , g i L |r . (Y 1 , Y 2
) is a coupling of (µ |r , µ |r ). Let E denote the expectation with respect to (i l , Z l ) 0≤l≤k and L. Then

E Y 1 -Y 2 2 = k l=0 N l N E θ i l -θ j l 2 + Z l|r -g i l |r 2 ≤ k l=1 N l N N N l tr 2 + δ 2 + N 0 N 2 + tr 2 N N 0 ≤ δ 2 + √ ktr 2 + 2 N 0 N ≤ δ 2 + √ ktr 2 + 2 ξ p R p ,
with probability at least 1 -C(k + 1)e -ct 2 N , where the last inequality comes from Markov's inequality, since θ ∈ F p (ξ).

Let now ∈ (0, 1 2 ]. Let us chose

r = -2 log( ), R = -1/p and k = -1/2-1/p ≤ 2 -1/2-1/p , so that δ = 2R/k ≤ 2 √ . Consequently E Y 1 -Y 2 2 ≤ (4 + 2ξ p ) + 2 √ 2 -1/4-1/(2p) | log( )|t . So if we chose t = | log( )| -1 5 4 + 1 2p we obtain P W 2 (µ |r , µ |r ) 2 ≤ (4 + 2ξ p + 2 √ 2) ≥ 1 -C -1/p-1/2 exp(-cN 2 1/2+1/p / log( ) 2 ).
Combining this with Lemmas 9.6.3 and 9.6.4 proves the proposition.

Sparsity of the Lasso estimator

The goal of this section is to prove: Theorem 9.6.1

Assume here that D is either F 0 (s) or F p (ξ) for some 0 ≤ s < s max (δ) and ξ > 0, p > 0.

There exists constants C, c > 0 that only depend on Ω, such that for all ∈ (0, 1)

sup θ ∈D P sup λ∈[λ min ,λmax] 1 N θ λ 0 -s * (λ) ≥ ≤ C 6 N q e -cN 6 ,
where q = 0 if D = F 0 (s) and q = (1/p -1)

+ if D = F p (ξ). Since #{i | | v λ,i | = 1} ≥ θ λ 0 , Theorem 9.5.6 gives that sup θ ∈D P ∃λ ∈ [λ min , λ max ], 1 N θ λ 0 ≥ s * (λ) + ≤ C 6
N q e -cN 6 . (9.6.12)

It remains to prove the converse lower bound in order to get Theorem 9.6.1. We start with the following 'local stability' property of the Lasso cost: Proposition 9.6.4

There exists constants C, c, γ > 0 that only depend on Ω such that for all ∈ (0, 1]

sup λ∈[λ min ,λmax] sup θ ∈D P ∃θ ∈ R N , 1 N θ 0 < s * (λ) - and L λ (θ) ≤ min L λ + γ 3 ≤ C 3 e -cN 6 .
Proposition 9.6.4 is a consequence of Proposition 9.3.1 and Lemma 9.6.5 below. Lemma 9.6.5

There exists constants γ, c, C > 0 that only depend on Ω such that for all ∈ (0, 1] we have

P min w∈D L λ (w) ≤ min w∈R N L λ (w) + 3γ 3 ≤ C 3 e -cn 6 , where D = w ∈ R N 1 N w + θ 0 < s * (λ) -. Proof. Define x λ = w λ + θ = η θ i + τ * g i , α * τ * 1≤i≤N
, and for r > 0

s r = 1 N # i ∈ {1, . . . , N } |x λ,i | ≥ r .
s r is a mean of independent Bernoulli random variables, by Hoeffding's inequality we have:

P s r ≥ P |Θ + τ * Z| ≥ α * τ * + r -4 ≥ 1 -e -N 2 /8 . Compute P |Θ + τ * Z| ≥ α * τ * + r = E Φ Θ τ * (λ) -α * (λ)- r τ * (λ) + Φ - Θ τ * (λ) -α * (λ)- r τ * (λ) ≥ s * (λ) - r σ .
Let us chose r = σ /4. We have then P s r ≥ s * (λ) -2 ≥ 1 -e -N 2 /8 . By Theorem 9.2.1 there exists a constant γ > 0 such that the event

∀w ∈ R N , L λ (w) ≤ min v∈R N L λ (v) + 3γ 3 ⇒ 1 N w -w λ 2 < σ 2 3 32 s r ≥ s * (λ) -2 (9.6.13)
has probability at least 1 -C 3 e -cn 6 . We have on this event, for all w ∈ D

1 N w -w λ 2 = 1 N w + θ -x λ 2 ≥ 2 r 2 = σ 2 3 32 .
Therefore, on the event (9.5.

P ∃λ ∈ [λ min , λ max ], 1 N θ λ 0 < s * (λ) - ≤ C 6 N q e -cn 6 .
This proves, together with (9.6.12), Theorem 9.6.1.

Proof of Theorem 8.3.3

Recall that the distributions µ * λ and ν * λ are respectively defined by Definition 8.3.3 and (9.5.1). Let ∈ (0, 1]. From now, we will work on the event

E = ∀λ ∈ [λ min , λ max ], W 2 ( µ ( θ λ ,θ ) , µ * λ ) 2 + W 2 ( µ ( v λ ,θ ) , ν * λ ) 2 ≤ 6 ∀λ ∈ [λ min , λ max ], 1 N θ λ 0 -s * (λ) + 1 N # i | v λ,i | = 1 -s * (λ) ≤ 2 ,
which has probability at least 1 -C -12 e -cN 17 from what we have just seen, and Theorems 8.3.1, 9.5.2, 9.6.1 and 9.5.6. From now, E and P will denote the probability with respect to the empirical distributions of the entries of the vectors we study, and the variables that we couple with them. Let λ ∈ [λ min , λ max ]. On the event E one can couple (Θ x , Z x ) ∼ μθ ⊗ N (0, 1) and (Θ v , Z v ) ∼ μθ ⊗ N (0, 1) with (Θ, Θ λ , V λ , Θ d λ ) which is sampled from the empirical distribution of the entries of (θ , θ λ , v λ , θ d λ ), such that

E Θ λ -η(Θ x + τ * Z x , α * τ * ) 2 + Θ -Θ x 2 ≤ 6 , E V λ + 1 α * τ * η(Θ v + τ * Z v , α * τ * ) -Θ v -τ * Z v 0 2 + Θ -Θ v 2 ≤ 6 .
Let

E 1 = Θ λ -η(Θ x + τ * Z x , α * τ * ) ≤ 2 , V λ + 1 α * τ * η(Θ v + τ * Z v , α * τ * ) -Θ v -τ * Z v ≤ 2 .
By Chebychev's inequality, P(E 1 ) ≥ 1 -C 2 , for some constant C > 0. Let us also define the event

E 2 = Θ x + τ * Z x = α * τ * and Θ v + τ * Z v = α * τ * .
Θ x + τ * Z x and Θ v + τ * Z v admit a density with respect to Lebesgue's measure. Therefore P(E 2 ) = 1. Lemma 9.6.6

The event

E 3 = | Θ λ | ∈ (0, 2 ] and | V λ | ∈ [1 -2 , 1)
has probability at least 1 -C 2 .

Proof. We denote here by O( 2) quantities that are bounded by C 2 , from some constant C.

Since Θ x + τ * Z x admits a density with respect to Lebesgue's measure we have

P η(Θ x + τ * Z x , α * τ * ) ∈ (0, 2 2 ] = 1 -O( 2 ) .
The second equivalence is proved exactly the same way.

Let us define

X d = η(Θ x + τ * Z x , α * τ * ) + Θ v + τ * Z v 0 -η(Θ v + τ * Z v , α * τ * ) .
We have

E( Θ d λ -X d ) 2 = 2E Θ λ -η(Θ x + τ * Z x , α * τ * ) 2 + 2E λ 1 -1 n θ λ 0 V λ -Θ v -τ * Z v + η(Θ v + τ * Z v , α * τ * ) 2 ≤ C 4 ,
for some constant C > 0, because on the event E, 1 N θ λ 0 -s * (λ) ≤ 2 , so

λ 1 -1 n θ λ 0 = λ 1 -1 δ s * (λ) + O( 2 ) = α * τ * + O( 2 ) .
By Lemma 9.6.8 above, we have on the event E,

X d =    Θ x + τ * Z x if Θ x + τ * Z x ≥ α * τ * or Θ x + τ * Z x ≤ -α * τ * , Θ v + τ * Z v otherwise.
Let us denote T x = (Θ x + τ * Z x , Θ x ) and One can therefore define two random variables T x = ( Θ x + τ * Z x , Θ x ) and

T v = (Θ v + τ * Z v , Θ v
T v = ( Θ v + τ * Z v , Θ v ) such that
• conditionally on E c , T x (respectively T v ) and T x (respectively T v ) have the same law.

• On the event

E c , Θ x + τ * Z x ≥ α * τ * ⇐⇒ Θ v + τ * Z v ≥ α * τ * and Θ x + τ * Z x ≤ -α * τ * ⇐⇒ Θ v + τ * Z v ≤ -α * τ * .
We define then

X d , Θ =             
T x on the event E provided that |Θ ( X d , Θ) ∼ µ d λ which is the law of (Θ + τ * Z, Θ) where (Θ, Z) ∼ µ θ ⊗ N (0, 1). Indeed, for every continuous bounded function f we have

E[f ( X d , Θ)] = E 1 E 1 |Θ x +τ * Z x |≥α * τ * f (T x ) + E 1 E 1 |Θ v +τ * Z v |<α * τ * f (T v ) + E 1 E c 1 | Θ x +τ * Z x |≥α * τ * f ( T x ) + E 1 E c 1 | Θ v +τ * Z v |<α * τ * f ( T v ) = E 1 E 1 |Θ x +τ * Z x |≥α * τ * f (T x ) + E 1 E 1 |Θ v +τ * Z v |<α * τ * f (T v ) + E 1 E c 1 |Θ x +τ * Z x |≥α * τ * f (T x ) + E 1 E c 1 |Θ v +τ * Z v |<α * τ * f (T v ) = E 1 |Θ x +τ * Z x |≥α * τ * f (T x ) + E 1 |Θ v +τ * Z v |<α * τ * f (T v ) = E 1 |Θ x +τ * Z x |≥α * τ * f (T x ) + E 1 |Θ x +τ * Z x |<α * τ * f (T x ) = E f (T x ) .
Let us now compute

E X d -X d 2 = E 1 E c X d -X d 2 ≤ C P(E c ) ≤ C ,
and Putting all together we obtain R(λ) = δτ * (λ) 2 -δσ 2 + O( ) = R * (λ) + O( ), and we conclude using Theorem 8.3.2.

E Θ -Θ 2 ≤ E 1 E Θ x -Θ 2 + E 1 E Θ v -Θ 2 + E 1 E c Θ x -Θ 2 + E 1 E c Θ v -Θ 2 ≤ 2 6 + 2C P(E c ) ≤ C . Therefore E ( Θ d λ , Θ) -( X d , Θ)
with probability at least 1 -Ce -cn for some constant C > 0. Similarly, we have with probability at least 1 -Ce -cn , 1 n w λ 2 -1 n w 2 ≤ C √ .

We conclude that with probability at least 1-Ce -cn we have for all w ∈ R N such that w-w λ

2 ≤ N , 1 n X w λ -σz 2 - 1 n w λ 2 - 1 n X w + σz 2 + 1 n w 2 ≤ C √ 1 + n n ≤ 2C √ n n .
Combining this with (9.6.18), we get that for all ∈ (0, 1],

P ∃w ∈ R N , w -w λ 2 ≤ N and 1 n X w + σz 2 - 1 n w 2 -σ 2 > 2C √ n n ≤ Ce -cn .
We conclude using Theorem 9.2.1 that

P ∃w ∈ R N , 1 n X w-σz 2 - 1 n w 2 -σ 2 ≥ √ n n and L λ (w) ≤ min v∈R N L λ (v)+γ ≤ C e -cn 2
for some constants c, C, γ > 0.

Using Proposition 9.3.1, we deduce Lemma 9.6.10

There exists constants γ, c, C > 0 that only depend on Ω such that for all θ in D and all λ ∈ [λ min , λ max ] such that for all ∈ (0, 1],

P ∃θ ∈ R N , 1 n X θ +σz -X θ 2 - 1 n θ-θ 2 -σ 2 ≥ √ n n and L λ (θ) ≤ min L λ +γ ≤ C e -cn 2 .
We have There exists constants C, c > 0 that only depend on Ω such that for all θ ∈ D, for all i ∈ {1, . . . , k} and for all ∈ (0, 1], Let ∈ (0, 1]. Let η = γ KN q and M = (λ max -λ min )/η . Define for j ∈ {0, . . . , M }, define λ j = min λ min + jη, λ max . We apply Lemma 9.6.10 with n = n/k, X = X (i) and z = z (i) to obtain that the event

θ i λ = arg min
P sup λ∈[λ min ,λmax] k n y (i) -X (i) θ i λ 2 - 1 n θ i λ -θ 2 -σ 2 ≥ k ≤ CN q -4 e -
E 1 = ∀j ∈ {1, . . . , M }, ∀θ ∈ R N , L (\i) λ j (θ) ≤ min L (\i) λ j + γ ⇒ k n y (i) -X (i) θ 2 - 1 n θ -θ 2 -σ 2 < √ k
has probability at least 1 -M C e -c 2 n . By Lemma 9.3.5 the event We obtain that on E 1 ∩ E 2 , which has probability at least 1 -CN q -2 e -cn 2 ∀λ ∈ [λ min , λ max ], k n y (i) -

E 2 = ∀λ, λ ∈ [λ min , λ max ], L (\i
X (i) θ i λ 2 - 1 n θ i λ -θ 2 -σ 2 < √ k.
Proposition 9.6.5

There exists constants c, C > 0 that only depend on Ω, such that for all θ ∈ D and for all i ∈ {1, . . . , k} has probability at least 1 -CN q e -cN . By Lemma 9.6.11, applied with = k -1 ,

P sup λ∈[λ min ,λmax] 1 N θ i λ -θ 2 -R * (λ) ≤ C √ k ≤ CN q k 4 e -cN/
E 3 = sup λ∈[λ min ,λmax] k n y (i) -X (i) θ i λ 2 - 1 n θ i λ -θ 2 -σ 2 ≤ 1
has probability at least 1 -Ck 4 N q e -cn/k 4 . On the event E 2 ∩ E 3 , we have, for all λ ∈ [λ min , λ max ],

L λ ( θ i λ ) = 1 2n y (i) -X (i) θ i λ 2 + 1 2n y (-i) -X (-i) θ i λ 2 + λ n | θ i λ | ≤ 1 k σ 2 + 1 n θ -θ i λ 2 + 1 + 1 2n k y (-i) -X (-i) θ i λ 2 + λ n | θ i λ | ≤ 1 k 1 + σ 2 + δ -1 (R (k) * (λ) + 1) + 1 2n k y (-i) -X (-i) θ λ 2 + λ n | θ λ | ≤ 1 k 1 + σ 2 + δ -1 (R (k) * (λ) + 1) + L λ ( θ λ ) + 1 2n k - 1 2n y -X θ λ 2 ≤ L λ ( θ λ ) + C k
for some constant C > 0. Let j ∈ {1, . . . , M }. We have L λ ( θ i λ ) = L λ j ( θ i λ ) -

λ j -λ n | θ i λ | and L λ ( θ λ ) ≤ L λ ( θ λ j ) = L λ j ( θ λ j ) + λ -λ j n | θ λ j | = min θ∈R N L λ j (θ) + λ -λ j n | θ λ j |.
So we get that on the event E 1 ∩ E 2 ∩ E 3 , for all j ∈ {1, . . . , M } and all λ ∈ [λ j-1 , λ j ],

L λ j ( θ i λ ) ≤ min θ∈R N L λ j (θ) + λ j -λ n | θ i λ j | -| θ λ j | + C k ≤ min θ∈R N L λ j (θ) + λ j -λ δ 2K 2 N q + C k ≤ min θ∈R N L λ j (θ) + η δ 2K 2 N q + C k ≤ min θ∈R N L λ j (θ) + C 0 k ,
for some constant C 0 > 0, because on E 1 we have ∀λ ∈ [λ min , λ max ], 1 N | θ i λ | -| θ λ | ≤ 2K 2 N q . By Theorem 9.3.1, there exists constants C, c, γ > 0 such that for all ∈ (0, 1] the event

E 4 = ∀j ∈ {1 . . . M }, ∀θ ∈ R N , L λ j (θ) ≤ min L λ j + γ 1 N θ -θ 2 -R * (λ j ) ≤ √
has probability at least 1 -CM -1 e -cN 2 . Consider the constant κ = C 0 γ . If k ≥ κ, then = C 0 γk ≤ 1 and the event E 4 has probability at least 1 -CM ke -cN/k 2 . So we obtain that on the event E 1 ∩ E 2 ∩ E 3 ∩ E 4 , which has probability 1 -CN q k 4 e -cn/k 4 , ∀j ∈ {1, . . . , M }, ∀λ ∈ [λ j-1 , λ j ],

1

N θ i λ -θ 2 -R * (λ j ) ≤ C √ k ,
≤ C ≤ C √ κ √ k ,
where C is a constant. We conclude that (in both cases) there exists a constant C > 0 such that ∀j ∈ {1, . . . , M }, ∀λ ∈ [λ j-1 , λ j ],

1

N θ i λ -θ 2 -R * (λ j ) ≤ C √ k ,
holds with probability at least 1 -CN q k 4 e -cN/k 4 Proposition 9.6.5 follows from the fact that for all λ ∈ [λ j-1 , λ j ], |R * (λ) -R * (λ j )| ≤ K 1 |λ -

λ j | ≤ 1 √ k .
Proof of Proposition 8.4.3. We apply Lemma 9.6.11 with = k -3/2 to obtain that with probability at least 1 -Ck 6 N q e -cn/k 6 we have ∀λ ∈ [λ min , λ max ], ∀i ∈ {1 . . . k}, k n y (i) -

X (i) θ i λ 2 - 1 n θ i λ -θ 2 -σ 2 ≤ 1 √ k .
By summing these inequalities for i = 1 . . . k and using the triangular inequality, we get

k n k i=1 y (i) -X (i) θ i λ 2 - 1 n k i=1 θ i λ -θ 2 -kσ 2 ≤ k √ k .
and then 1 N .6.22) By Proposition 9.6.5, we have with probability at least 1 -CN q k 4 e -cN/k 3 , ∀λ ∈ [λ min , λ max ], ∀i ∈ {1, . . . , k},

k i=1 y (i) -X (i) θ i λ 2 - 1 k k i=1 1 N θ i λ -θ 2 -δσ 2 ≤ δ √ k . ( 9
1 N θ i λ -θ 2 -R * (λ) ≤ C √ k .
This implies (again by summing and using the triangular inequality) that

1 k k i=1 1 N θ i λ -θ 2 -R * (λ) ≤ C √ k ,
which, combined with (9.6.22) proves Proposition 8.4.3.

The scalar lasso

In this section we study The minimum (9.6.23) is achieved at an unique point x * = η(y, α) and

α (y) =      1 2 y 2 if -α ≤ y ≤ α αy -1 2 α 2 if y ≥ α -αy -1 2 α 2 if y ≤ -α
Suppose now that y = x + Z , for some x ∈ R and Z ∼ N (0, 1). This gives (recall that ∆ α is even and continuous over R) that ∆ α is α-Lipschitz. From what we have seen above, we have also ∆ α (0 + ) = -∆ α (0 -) = -α. Compute now, using the fact that α is even: 9.6.9 A convexity lemma Lemma 9.6.14

∆ α (0) = E[ α (Z)] =
The function

f : x ∈ R N → x 2 n + σ 2 is σ 2 n(R 2 +σ 2 )
(R 2 + σ 2 ) 3/2 = f (tx + (1 -t)y) + 1 2 t(1 -t) 1 n x -y 2 σ 2 (R 2 + σ 2 ) 3/2 .

Toolbox

Notations recap

Recall that X is a n×N random matrix with entries X i,j i.i.d.

∼ N (0, 1/n). The random vectors z ∈ R n , g ∈ R N and h ∈ R n are standard Gaussian random vectors. The following table displays the main cost (or objective) functions used in this paper and their corresponding optimizers.

Definition

Optimizer In the definition of V λ above, B = w ∈ R N | |w| ≤ 2|θ | + 5σ 2 λ -1 min n + K , where K > 0 is the constant given by Lemma 9.5.5. The functions L λ , U λ and V λ are the "corresponding cost/objective functions" to C λ , U λ and V λ . A main part of the analysis is to show that w * λ , u * λ and v * λ are approximately equal to w λ , u λ and v λ given by:

L λ (θ) = 1 2n Xθ -y 2 + λ n |θ| θ λ C λ (w) = 1 2n Xw -σz 2 + λ n (|w + θ | -|θ |) w λ U λ (u) = min w∈R N u T Xw -σu T z -1 2 u 2 + λ(|θ + w| -|θ |) u λ V λ (v) = min
w λ = η θ + τ * (λ

Gaussian min-max Theorem

In this section, we reproduce the proof of the tight Gaussian min-max comparison theorem from [START_REF] Thrampoulidis | Regularized linear regression: A precise analysis of the estimation error[END_REF] for completeness, but also because we need a slightly more general version of this result.

We recall the classical Gordon's min-max Theorem from [START_REF] Gordon | Some inequalities for gaussian processes and applications[END_REF] (see also Corollary 3.13 from [START_REF] Ledoux | Probability in Banach Spaces: isoperimetry and processes[END_REF]): Theorem 9.7.1

Let X i,j and (Y i,j ), 1 ≤ i ≤ n, 1 ≤ j ≤ m be two (centered) Gaussian random vectors such that

       EX 2 i,j = EY 2 i,j
for all i, j , EX i,j X i,k ≥ EY i,j Y i,k for all i, j, k , EX i,j X l,k ≤ EY i,j Y l,k for all i = l and j, k .

Then, for all real numbers λ i,j : ∼ N (0, 1), g ∼ N (0, Id n 1 ) and h ∼ N (0, Id m 1 ) be independent standard Gaussian vectors. For u ∈ R n 1 +n 2 and v ∈ R m 1 +m 2 we define ũ = (u 1 , . . . , u n 1 ) and ṽ = (v 1 , . . . , v m 1 ). Define Then we have:

• For all t ∈ R, P C * (G) ≤ t ≤ 2P L * (g, h) ≤ t .

• If D u and D v are convex and if Q is convex concave, then for all t ∈ R P C * (G) ≥ t ≤ 2P L * (g, h) ≥ t .

Proof. Let us consider the Gaussian processes: X(u, v) = ṽ g T ũ + ũ h T ṽ , Y (u, v) = ṽT Gũ + ũ ṽ z , where z ∼ N (0, 1) is independent from G. Let (u, v), (u , v ) ∈ D u × D v and compute E Y (u, v)Y (u , v ) -E X(u, v)X(u , v ) = ũ ṽ ũ ṽ + (ũ T ũ )(ṽ T ṽ ) -ṽ ṽ (ũ T ũ ) -ũ ũ (ṽ T ṽ ) = ũ ũ -(ũ T ũ ) ṽ ṽ -(ṽ T ṽ ) ≥ 0 .

Therefore X and Y verify the covariance inequalities of Theorem 9.7.2: one can apply Theorem 9. Let us suppose now that D u and D v are convex and that G is convex-concave. We now apply the inequality we just proved, but with the role of u and v being switched (and -Q and -t instead of Q and t):

Basic concentration results

We recall in this section some elementary concentration results, see Chapter 2 from [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF] for a more detailed presentation of these facts.

Definition 9.7.1

A real random variable X is said to be

• σ 2 -sub-Gaussian if for every s ∈ R, log Ee s(X-E[X]) ≤ s 2 σ 2 2 ,

• (v, c)-sub-Gamma if for every s ∈ (-1/c, 1/c), log Ee s(X-E[X]) ≤ s 2 v 2(1 -c|s|) .

One deduces immediately from the above definition: Proposition 9.7.1

Let (X 1 , . . . , X n ) be independent real random variables. Define S = n i=1 X i .

• Suppose that for all i ∈ {1, . . . , n}, X i is σ 2 i -sub-Gaussian. Then S is n i=1 σ 2 isub-Gaussian.

• Suppose that for all i ∈ {1, . . . , n}, X i is (v i , c i )-sub-Gamma. Then S is n i=1 v i , max c isub-Gamma.

Proposition 9.7.2

Let X be a real random variable.

• if X is σ 2 -sub-Gaussian, then for all t > 0 P(X -E[X] ≥ t) ∨ P(X -E[X] ≤ -t) ≤ e -t 2 2σ 2 ,

• if X is (v, c)-sub-Gamma, then for all t > 0 If X is σ 2 -sub-Gaussian and has mean µ, then X 2 is a sub-Gamma random variable with parameters

P(X -E[X] ≥ √ 2ct + vt) ∨ P(X -E[X] ≤ -( √ 2ct + vt)) ≤ e -
   v = 16σ 2 + 4µ 2 σ 2 , c = 4σ 2 . Proof. Let µ = E[X] and Y = X -µ. X 2 = Y 2 + 2µY + µ 2 . E (Y 2 ) 2 = E[(X -µ) 4 ] ≤ 16σ 4 , E (Y 2 ) q = E (X -µ) 2q ≤ 2q!(2σ 2 ) q = 1 2
q!(16σ 2 )(2σ

E (XY -E[XY ]) 2q ≤ 2 2q-1 E[X 2q
] + 2 2q-1 m 2q ≤ q!(8σ 2 ) q + q!(4m 2 ) q ≤ q!(8σ 2 + 4m 2 ) q .

Largest singular value of a Gaussian matrix

The largest singular value of a n × N matrix A is defined as

σ max (A) = max x ≤1
Ax .

The next classical result is a simple consequence of Slepian's Lemma (see for instance [START_REF] Ledoux | Probability in Banach Spaces: isoperimetry and processes[END_REF], Section 3.3) and the classical Gaussian concentration inequality (see for instance [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF], Theorem 5.6).

B Proofs of some basic properties of the MMSE and the free energy B.1 Proof of Proposition 1.3.1

Let 0 < λ 2 < λ 1 . Define ∆ 1 = λ -1 1 , ∆ 2 = λ -1 2 and

   Y 1 = X + √ ∆ 1 Z 1 Y 2 = X + √ ∆ 1 Z 1 + √ ∆ 2 -∆ 1 Z 2 ,
where X ∼ P X is independent from Z 1 , Z 2 i.i.d.

∼ N (0, Id n ). Now, by independence between (X, Y 1 ) and Z 2 we have

MMSE(λ 1 ) = E X -E[X|Y 1 ] 2 = E X -E[X|Y 1 , Z 2 ] 2 = E X -E[X|Y 1 , Y 2 ] 2 ≤ E X -E[X|Y 2 ] 2 = MMSE(λ 2 ) . Next, notice that MMSE(λ 1 ) = E X -E[X|Y 1 ] 2 ≤ E X -E[X] 2 = MMSE(0) . (B.1)
This shows that the MMSE is non-increasing on R ≥0 . The first point is obvious while the second follows from:

0 ≤ MMSE(λ) = E X -E[X|Y ] 2 ≤ E X - 1 √ λ Y 2 = n λ ----→ λ→+∞ 0 .

B.2 Proof of Proposition 1.3.2

We start by proving that MMSE is continuous at λ = 0. Let λ ≥ 0 and consider Y , X, Z as given by (1.3.1). By dominated convergence one has almost surely that

E[X|Y ] = dP X (x)xe -1 2 √ λx-Y 2 dP X (x)e -1 2 √ λx-Y 2 --→ λ→0 E[X] .
Then by Fatou's Lemma we get

lim inf λ→0 MMSE(λ) ≥ E lim inf λ→0 X -E[X|Y ] 2 = E X -E[X] 2 .
Combining this with the bound MMSE(λ 2 . This proves that the MMSE is continuous at λ = 0.

) ≤ E X -E[X] 2 gives MMSE(λ) --→ λ→0 E X - E[X]
Let us now prove that the MMSE is continuous on R * ≥0 . We need here a technical lemma: Lemma B.1 For all λ > 0, p ≥ 1

E X -x λ 2p ≤ 2 p (2p!) λ p p! n p+1 .
Proof. We reproduce here the proof from [START_REF] Guo | Estimation in gaussian noise: Properties of the minimum mean-square error[END_REF], Proposition 5. We start with the equality

√ λ (X -x λ ) = √ λX -E[ √ λX|Y ] = Y -Z -E[Y -Z|Y ] = E[Z|Y ] -Z .
We have therefore

E X -x λ 2p = 1 λ p E E[Z|Y ] -Z 2p ≤ 2 2p-1 λ p E E[Z|Y ] 2p + Z 2p ≤ 2 2p λ p E Z 2p .
It remains to bound

E Z 2p ≤ n p E n i=1 Z 2p i = n p+1 (2p)! 2 p p! .
Let λ 0 > 0. The family of random variables Xx λ

2 λ≥λ 0 is bounded in L 2
by Lemma B.1 and is therefore uniformly integrable. The function λ → Xx λ 2 is continuous on [λ 0 , +∞), the uniform integrability ensures then that MMSE : λ → E Xx λ 2 is continuous over [λ 0 , +∞). This is valid for all λ 0 > 0: we conclude that MMSE is continuous over (0, +∞).

B.3 Proof of the I-MMSE relation: Proposition 1.3.3

MMSE(λ) = E X -x λ 2 = E X 2 + E x λ 2 -2E x X λ
Now, by the Nishimori property E x λ 2 = E (x (1) ) x (2) λ = E x X λ . Thus

MMSE(λ) = E X 2 -E x X λ . (B.2)
By (B.2) and (1.3.3), it suffices now to prove the second equality in (1.3.4). This will follow from the lemmas below.

Lemma B.2

The free energy F is continuous at λ = 0.

Proof. For all λ ≥ 0,

F (λ) = E log dP X (x)e -1 2 Y - √ λx 2 + 1 2 Y 2 = E log dP X (x)e -1 2 √ λX- √ λx+Z 2 + λE X 2 + n .
By dominated convergence dP X (x)e -

√ λX - √ λx + Z 2 ≤ 3 2 X 2 + E X 2 + Z 2 ,
for all λ ∈ [0, 1]. One can thus apply the dominated convergence theorem again to obtain that F is continuous at λ = 0.

Lemma B.3

For all λ ≥ 0,

F (λ) -F (0) = 1 2 λ 0 E x X γ dγ .
Proof. Compute for λ > 0

∂ ∂λ log Z(λ, Y ) = 1 2 √ λ x Z + x X - 1 2 x 2 λ .
Since E X 2 < ∞, the right-hand side is integrable and one can apply Fubini's theorem to obtain

F (λ 2 ) -F (λ 1 ) = λ 2 λ 1 E 1 2 √ λ x Z + x X - 1 2 x 2 λ dλ .
By Gaussian integration by parts, we have for all i ∈ {1, . . . , n} and λ > 0

EZ i x i λ = E ∂ ∂Z i x i λ = E √ λx 2 i λ - √ λ x i 2 λ = √ λE x 2 i λ -x i X i λ ,
where the last equality comes from the Nishimori property (Proposition 1.1.1). We have therefore

F (λ 2 ) -F (λ 1 ) = 1 2 λ 2 λ 1 E x X λ dλ .
By Lemma B.2, F is continuous at 0 so we can take the limit λ 1 → 0 to obtain the result.

By Proposition 1.3.2, the function λ → MMSE(λ) is continuous over R ≥0 . By (B.2) we deduce that λ → E x X λ is continuous over R ≥0 and therefore Lemma B.3 proves (1.3.4).

It remains only to show that F is strictly convex when P X differs from a Dirac mass. We proceed by truncation. For N ∈ N and x ∈ R we write x (N ) = x 1(-N ≤ x ≤ N ). We extend this notation to vectors x = (x 1 , . . . , x n ) ∈ R n by x (N ) = (x (N ) 1 , . . . , x (N ) n ). For X ∼ P X we define P (N ) X as the distribution of X (N ) . F (N ) , MMSE (N ) and • λ,N will denote respectively the corresponding free energy, MMSE and posterior distribution. One can compute the second derivative (since X (N ) is bounded, one can easily differentiate under the integral sign) and again, using Gaussian integration by parts and the Nishimori identity one obtains: If P 0 is not a Dirac measure, then the last term is strictly positive: this concludes the proof.

F (N ) (λ) = 1 2 E Tr xx T λ,N -x λ,N x T λ,N

B.4 Pseudo-Lipschitz continuity of the free energy with respect to the Wasserstein distance

Let P 1 and P 2 be two probability distributions on R n , that admits a finite second moment. We denote by W 2 (P 1 , P 2 ) the Wasserstein distance of order 2 between P 1 and P 2 . For i = 1, 2 the free energy is defined as

F P i (λ) = E log dP i (x) exp √ λx Z + λx X - λ 2 x 2 ,
where the expectation is with respect to (X, Z) ∼ P i ⊗ N (0, Id n ).

Proposition B.1

For all λ ≥ 0,

F P 1 (λ) -F P 2 (λ) ≤ λ 2 E P 1 X 2 + E P 2 X 2 W 2 (P 1 , P 2 ) .
A similar result was proved in [START_REF] Wu | Functional properties of minimum mean-square error and mutual information[END_REF] but with a weaker bound for the W 2 distance. By Proposition 1.3.3 we have for i = 1, 2, F P i (λ) = λ 2 E X i 2 -I(X i ; √ λX i + Z) where (X i , Z) ∼ P i ⊗ N (0, Id n ). We deduce immediately:

Corollary B.1

For all λ ≥ 0, ∼ N (0, Id n ) are independent from (X 1 , X 2 ) ∼ Q. Define

I
f (t) = E log dQ(x 1 , x 2 ) exp √ λtx 1 Y (t) 1 - λt 2 x 1 2 + λ(1 -t)x 2 Y (t) 2 - λ(1 -t) 2 x 2 2 .
We have f (0) = F P 2 (λ) and f (1) = F P 1 (λ). By an easy extension of the I-MMSE relation (1.3.4) we have for all t ∈ [0, 1]:

f (t) = λ 2 E X 1 x 1 -X 2 x 2 t
, where • t denotes the expectation with respect to (x 1 , x 2 ) sampled from the posterior distribution of (X 1 , X 2 ) given

Y (t) 1 , Y (t)
2 , independently of everything else. We have then

| 2 λ f (t)| = E X 1 (x 1 -x 2 ) -(X 2 -X 1 ) x 2 t ≤ E X 1 2 E x 1 -x 2 2 t 1/2 + E x 2 2 t E X 2 -X 1 2 1/2 = E X 1 2 E X 1 -X 2 2 1/2 + E X 2 2 E X 2 -X 1 2 1/2 ≤ E X 1 2 1/2 + E X 2 2 1/2 (W 2 (P 1 , P 2 ) + ) ,
where we used successively the Cauchy-Schwarz inequality and the Nishimori property (Proposition 1.1.1). We then let → 0 to obtain the result.

B.5 Zero-noise limit of the mutual information

The goal of this section is to prove Proposition B.

θ i = arg min u∈N |u -U i + √ ∆Z i |,
for all i ∈ {1, . . . , m}. Note that θ i is well-defined almost surely since there is a.s. a unique minimizer above. We have

P( θ i = U i ) ≤ P √ ∆|Z i | ≥ 1/2 = 2P N (0, 1) ≥ 1 2 √ ∆ ≤ 2 2 √ ∆ √ 2π e -1/(8∆) ≤ 2 √ ∆e -1/(8∆) ,
by usual bounds on the Gaussian cumulative distribution function. We have then

MMSE(U | U + √ ∆Z) ≤ E U -θ 2 = m i=1 E(U i -θ i ) 2 = m i=1 E 1( θ i = U i )(U i -θ i ) 2 ≤ m i=1 2E 1( θ i = U i )(U i -(U i + √ ∆Z i )) 2 + 2E 1( θ i = U i )(U i + √ ∆Z i -θ i ) 2 ≤ m i=1 2E 1( θ i = U i )∆Z 2 i + 1 2 E 1( θ i = U i ) ≤ m i=1 2∆P( θ i = U i ) 1/2 E[Z 4 i ] 1/2 + 1 2 P( θ i = U i ) ≤ me -1/(16∆) 2 √ 6∆ 5/4 + √ ∆ ≤ 6me -1/(16∆)
for ∆ ≤ 1. Plugging this inequality in (B.5), we obtain for all ∆ ∈ (0, 1],

|h

Corollary B.2

Let U be a random variable over N m with finite second moment, let X be a random variable over R n and let Z ∼ N (0, Id m ). We assume (U , X) to be independent from Z. Then, for all ∆ ∈ (0, 1], I(X; U + √ ∆Z) -I(X; U ) ≤ 100me -1/(16∆) .

Proof. We have by the chain rule of the mutual information: Let I ⊂ R be an interval, and let (f n ) n≥0 be a sequence of convex functions on I that converges pointwise to a function f . Then for all t ∈ I for which these inequalities have a sense f (t -) ≤ lim inf n→∞ f n (t -) ≤ lim sup n→∞ f n (t + ) ≤ f (t + ).

Proof. Let t ∈ I and h > 0. By convexity

f n (t + ) ≤ f n (t + h) -f n (t) h ---→ n→∞ f (t + h) -f (t) h ---→ h→0 f (t + ).
The first inequality follows from the same arguments.

Proposition C.3

Let f be a closed convex function over R n . Then f is γ-strongly convex if and only if f * is 1 γ -strongly smooth.

This result can be found in the book [START_REF] Zalinescu | Convex analysis in general vector spaces[END_REF], see Corollary 3.5.11 on page 217 and the Remark 3.5.3 below. A more accessible presentation of this result can be found in [START_REF] Sham M Kakade | Regularization techniques for learning with matrices[END_REF].

C.2 The monotone conjugate Definition C.2

We define the monotone conjugate (see [START_REF] Tyrell | Convex analysis[END_REF] Let f be a non-decreasing lower semi-continuous convex function on R ≥0 such that f (0) is finite. Then f * is another such function and (f * ) * = f .

Proposition C.5

Let f be a non-decreasing lower semi-continuous convex function on R ≥0 such that f (0) is finite. Then for all x, y ≥ 0:

x ∈ ∂f * (y) ⇐⇒ f (x) + f * (y) = xy ⇐⇒ y ∈ ∂f (x).

Let us now prove the second part of the Proposition: we now assume that f and g are differentiable, strictly convex. Let (q 1 , q 2 ) ∈ Γ be a couple that achieves the maximum of ϕ over Γ. It suffices to show that (q 1 , q 2 ) ∈ Γ. If q 1 > 0 and q 2 > 0, then this is trivial because f and g are differentiable.

Suppose now that q 1 = 0 (the case q 2 = 0 follows by symmetry). Since 0 = q 1 ∈ ∂g(q 2 ) and g is strictly increasing, we get that q 2 = 0, so that sup ψ = ϕ(q 1 , q 2 ) = f (0) + g(0). Notice that f * (f (0)) = -f (0) and g * (g (0)) = -g(0) so f (0) + g(0) = sup ψ ≥ ψ g (0), f (0) = f (0) + g(0) + f (0)g (0) ≥ f (0) + g(0).

(C.4)

We get that (g (0), f (0)) achieves the supremum of ψ, which implies that (g (0), f (0)) ∈ Γ. From (C.4) we get also that f (0) = 0 or g (0) = 0. Assume that f (0) = 0 (the case g (0) = 0 follows by symmetry), then 0 ∈ ∂f (g (0)) because (g (0), f (0)) ∈ Γ. Since f is strictly increasing, so we have g (0) = 0. We conclude that f (0) = g (0) = 0 which proves that (q 1 , q 2 ) = (0, 0) ∈ Γ.

We will need the following variants of Proposition C.6 in Chapter 7: Proposition C.7

Let f : [0, ρ] → R be a continuous convex non-decreasing function. Let g : R ≥0 → R be a convex, Lipschitz, non-decreasing function. Define ρ = sup x≥0 g (x + ) the Lipschitz constant of g. Then sup q 2 ≥0 inf q 1 ∈[0,ρ] f (q 1 ) + g(q 2 ) -q 1 q 2 = sup

q 1 ∈[0,ρ] inf q 2 ≥0
f (q 1 ) + g(q 2 ) -q 1 q 2 . Proof. We extend f on R ≥0 by setting f (x) = +∞ for all x > ρ. The extended function f is then a convex, lower semi-continuous function on R ≥0 and f (0) is finite. We have then sup q 2 ≥0 inf q 1 ∈[0,ρ] f (q 1 ) + g(q 2 ) -q 1 q 2 = sup q 2 ≥0 inf q 1 ≥0 f (q 1 ) + g(q 2 ) -q 1 q 2 = sup q 2 ≥0 g(q 2 ) -f * (q 2 ) , by Proposition C.4. g is ρ-Lipschitz so g * (x) = +∞ for all x > ρ. Consequently, by Proposition C.4, g(q 2 ) = sup q 1 ≥0 {q 1 q 2 -g * (q 1 )} = sup q 1 ∈[0,ρ] {q 1 q 2 -g * (q 1 )}. We get sup q 2 ≥0 inf q 1 ∈[0,ρ] f (q 1 ) + g(q 2 ) -q 1 q 2 = sup q 2 ≥0 sup q 1 ∈[0,ρ] q 1 q 2 -g * (q 1 ) -f * (q 2 ) = sup q 1 ∈[0,ρ] f (q 1 ) -g * (q 1 ) = sup

q 1 ∈[0,ρ] inf q 2 ≥0
f (q 1 ) + g(q 2 ) -q 1 q 2 . This concludes the proof of (C.5). It remains to see that a couple (q * 1 , q * 2 ) ∈ Γ that achieves the supremum in (C.5) also achieves the "sup-inf". This simply follows from the fact that ϕ(q * 1 , q * 2 ) = inf q 2 ϕ(q * 1 , q 2 ) and (C.5).

D Differentiation of a supremum of functions

We recall in this section two results about the differentiation of a supremum of functions from Milgrom and Segal [START_REF] Milgrom | Envelope theorems for arbitrary choice sets[END_REF]. Let X be a set of parameters and consider a function f : X × [0, 1] → R. Define, for t ∈ [0, 1]

V (t) = sup x∈X f (x, t) , X * (t) = x ∈ X f (x, t) = V (t) .

Proposition D.1 (Theorem 1 from [144] )

Let t ∈ [0, 1] such that X * (t) = ∅. Let x * ∈ X * (t) and suppose that f (x * , •) is differentiable at t, with derivative f t (x * , t).

• If t > 0 and if V is left-hand differentiable at t, then V (t -) ≤ f t (x * , t).

• If t < 0 and if V is right-hand differentiable at t, then V (t + ) ≥ f t (x * , t).

• If t ∈ (0, 1) and if V is differentiable at t, then V (t) = f t (x * , t).

Proposition D.2 (Corollary 4 from [144] )

Suppose that X is nonempty and compact. Suppose that for all t ∈ [0, 1], f (•, t) is continuous. Suppose also that f admits a partial derivative f t with respect to t that is continuous in (x, t) over X × [0, 1]. Then • V (t + ) = max

x * ∈X * (t) f t (x * , t) for all t ∈ [0, 1) and V (t -) = min

x * ∈X * (t) f t (x * , t) for all t ∈ (0, 1].

• V is differentiable at t ∈ (0, 1) is and only if f t (x * , t) x * ∈ X * (t) is a singleton.

In that case V (t) = f t (x * , t) for all x * ∈ X * (t).

  ) standard Gaussian density function φ(x) = e -x 2 /2 √ 2π Φ(x) standard Gaussian distribution function Φ(x) = x -∞ e -t 2 /2 √ 2π dt MMSE(X|Y ) Minimal Mean Square Error: MMSE(X|Y ) = E X -E[X|Y ] 2Chapter 1

E

  ( x(Y ); X)2 = q. (1.2.7)Proof. Let us abbreviate λ max def = λ max (M ). By Lemma 1.2.1 and Proposition 1.2.1 we haveE ( v; X) 2 = E[λ max ] ---→ n→∞ q.

x∈S n P 0 Proposition 2 . 1 . 1 For all n ≥ 1 1 Z n x∈S n 1 (

 0211111 (x) exp(H n ( x)) . and all ∈ [0, 1], we have|F n, -F n | ≤ H(P 0 ) .Proof. Let us computeP Y | Y , L = 1(x i = Y i for all i such that L i = 1)dP (x | Y ) = x i = Y i for all i such that L i = 1)e Hn(x) n i=1 P 0 (x i ) = Z n, Z n i|L i =1 P 0 (Y i ) = Z n, Z n P Y | L .Therefore, nF n, -nF n = H(Y | L) -H(Y | Y , L) and the proposition follows from the fact that 0 ≤ H(Y |Y , L) ≤ H(Y |L) = n H(P 0 ).

Lemma 2 . 2 . 2

 222 If f and g are two differentiable convex functions then, for any b > 0|f (a) -g (a)| ≤ g (a + b) -g (a -b) + d b ,whered = |f (a + b) -g(a + b)| + |f (a -b) -g(a -b)| + |f (a) -g(a)|.

A 1 Eφ

 1 (a + b) -Eφ (a -b) da = Eφ(b + A) -Eφ(b + 1) -Eφ(A -b) -Eφ(1 -b) = Eφ(b + A) -Eφ(A -b) + Eφ(1 -b) -Eφ(1 + b) ≤ 4K 2 b(A + 2) .

Figure 3 Figure 3 . 1 :

 331 Figure 3.1: Mean Squared Errors for the Spiked Wigner model with prior P 0 given by (3.3.5) with p = 0.05.

Figure 3 . 3 :

 33 Figure 3.3: Phase diagram for the spiked Wigner model with prior (3.3.5).

Theorem 4 . 1 . 1 (

 411 Replica-Symmetric formula for the spiked Wishart model)

Figure 4 . 1 :

 41 Figure 4.1: Mean Squared Errors for the spiked covariance model, where the spike is generated by (4.2.3) with s = 0.15, λ = 1. The right-hand side panel is a zoom of the left-hand side panel around α = 1.

  8.

Figure 3 .

 3 3 for a similar scenario. Let us now deduce Theorem 4.1.2 from Proposition 4.4.1. By the "I-MMSE" relation of Proposition 1.3.3:

√n

  U V T + Z and Y = √ rU + Z . By an straightforward extension of Theorem 4.1.1 we have for all r ≥ 0:

Figure 5 . 1 :

 51 Figure 5.1: The Stochastic Block Model with two communities of different sizes.

Figure 5 . 2 :

 52 Figure 5.2: Phase diagram for the asymmetric community detection problem. The easy phase follows from[START_REF] Bordenave | Non-backtracking spectrum of random graphs: community detection and non-regular ramanujan graphs[END_REF], the impossible phase below the spinodal curve (red curve) is proved in this paper and the hard phase is a conjecture. The dotted curve corresponding to λ c (p) is the conjectured curve for solvability of the community detection problem (see discussion in Section 5.4).

Lemma 5 . 2 . 1

 521 If pa + (1 -p)b = pb + (1 -p)c then the community detection problem is solvable.

Figure 5 . 3 :

 53 Figure 5.3: Lower bound for the probability to recover the true label of a typical vertex by an optimal local algorithm with side information for p > p * as a function of λ (i.e. function 2P(N (α/2, α) > 0) -1 for p = 0.25 > p * ).

Proposition 5 . 4 . 1

 541 If λ < λ sp (p) then lim d→∞ lim n→∞ P suc (T opt ) = 0. Proposition 5.4.1 will follow from Corollary 5.7.1. If λ c (p) def = inf λ ≥ 0 community detection is solvable denotes the solvability threshold, then Proposition 5.4.1 implies that λ sp (p) ≤ λ c (p)

Figure 5 . 4 :

 54 Figure 5.4: Necessary fraction of revealed labels (red) and corresponding lower bound of probability to recover the true label of a typical vertex by an optimal local algorithm (black) for p < p * , i.e. functions βp(1-p) λ (in red) and 2P(N (α/2, α) > 0) -1 (in black) appearing in Proposition 5.4.2 as a function of λ for p = 0.05 < p * .

Figure 6 . 1 :

 61 Figure 6.1: Minimal Mean Squared Errors MMSE (k) n for tensor estimation (6.1.1) with Rademacher prior, for k = 2, 3, 4, 6, as given by Corollary 6.1.2.

Corollary 6 . 1 . 2

 612

Figure 6 . 2 :

 62 Figure 6.2: Plots of q → -F(λ, q) for k = 3, λ = 10 and P 0 = N (µ, 1), for different values of µ.

100 Figure 6 . 3 :

 10063 Figure 6.3: Asymptotic correlation lim n→∞ | x ML λ , X | = q * (λ) as a function of the signalto-noise ratio λ, for different values of k.

Figure 6 . 4 :

 64 Figure 6.4: Asymptotic constrained maximum likelihood E λ (m) for k = 4 with λ = 1, 1.299, 1.35, 1.405, 1.5.Here λ s ≈ 1.299 and λ c ≈ 1.405. For λ < λ s , the function is (numerically) seen to be monotone. A secondary maximum occurs at the transition λ = λ s . This local maximum is bounded away from m = 0. Finally, at the information theoretic threshold λ c , the maximum likelihood is now maximized at this second point.

Figure 7 . 1 :

 71 Figure 7.1: Two classes separated by an hyperplane.

Figure 7 . 2 :

 72 Figure 7.2: Frank Rosenblatt with the "Mark 1 perceptron", a custom-built hardware that implements the perceptron algorithm (circa 1960).

1 n

 1 x, X * a simulation with n = 10000 and various values of α = m/n.

.1. 2 ) 1 3 )

 213 Proposition 7.1.2 gives that for α > α 1 , 1 n log E #S n ---→ n→∞ 0, which seems to indicate that #S n is of sub-exponential order in this regime: it should be possible to estimate X * almost perfectly: Corollary 7.1.Let x ∼ Unif(S n ), independently of everything else. Then we have for all α > α 1 , Proof. Let > 0 and I = [-1, 1 -]. By Proposition 7.1.2 we have

Figure 7 . 3 :

 73 Figure 7.3: Plots of the limits of 1n log E#S n (given by Proposition 7.1.2), 1 n E log #S n and q * (α) (from Proposition 7.1.3), together with the correlation q AMP (α) achieved by GAMP and the correlation 1 n x, X * achieved by the perceptron algorithm (see Section 7.1.1) for n = 10000.

  (a) The estimation task. The goal is here to recover the signal X * . (b) The prediction task. Given a new point Φ new and the dataset Φ, Y , we aim at predicting the corresponding output

  ∈ {1, . . . , m}, sign x, Φ µ = Y µ measures the number of possible values for X * given the observations of Y , Φ. Random function representation. We introduce now a convenient "random function" representation for the transition kernel P out . Let us consider a function ϕ : R × R → R and a probability distribution P A over R such that for all x ∈ R, P out (•|x) corresponds to the law of Y = ϕ(x, A) + √ ∆Z (7.2.8) where (A, Z) ∼ P A ⊗ N (0, 1) and ∆ ≥ 0. Notice that any transition kernel can be realized by (7.2.8) by simply taking ∆ = 0, A ∼ Unif([0, 1]) and ϕ(x, •) to be the generalized inverse of the cumulative distribution function of P out (•|x), see Proposition 7.2.1 below.

(

  h5.a) ∆ > 0. (h5.b) ∆ = 0 and ϕ takes values in N. Remark 7.3.1.

(

  h2') There exists γ > 0 such that E[|Y 1 | 2+γ ] remains bounded with n. (h4') x ∈ R → P out (•|x) is continuous almost everywhere for the weak convergence.

Theorem 7 . 3 . 1

 731 Suppose that hypotheses (h1)-(h2)-(h3)-(h4) hold. Suppose that either hypothesis (h5.a) or (h5.b) holds. Then

α 1 Figure 7 . 4 :

 174 Figure 7.4: Phase diagram of noiseless compressed sensing with Gauss-Bernoulli prior: P 0 = ρN (0, 1) + (1 -ρ)δ 0 .

Figure 7 . 5 :

 75 Figure 7.5: Phase diagrams showing boundaries of the region where exact recovery is possible (in absence of noise). Left: The case of ReLU activation function, ϕ(x) = max(0, x) with a Gauss-Bernoulli signal P 0 = ρN (0, 1) + (1 -ρ)δ 0 , as a function of the ratio between number of samples/measurements and the dimension α = m/n, and the fraction of nonzero components ρ. The dotted red line shows the algorithmic phase transition when using information only about the non-zero observations. Center: Analogous to the left panel, for the absolute value function: ϕ(x) = |x|. The dotted red line shows for comparison the algorithmic phase transition of the canonical compressed sensing. Right: Phase diagram for the symmetric door output function ϕ(z) = sign(|z| -K) for a Rademacher signal, as a function of α and K.

Figure 7 . 6 :

 76 Figure7.6: Generalization error in three classification problems as a function of the number of data-samples per dimension α. The red line is the Bayes-optimal generalization error, while the green one shows the (asymptotic) performances of GAMP as predicted by the state evolution (SE), when different. For comparison, we also show the result of GAMP (black dots) and, in blue, the performance of a standard out-of-the-box solver, both tested on a single randomly generated instance. Left: Perceptron, with ϕ(x) = sign(x) and a Rademacher (±1) signal. The results of a logistic regression with fine-tuned ridge penalty are shown for comparison. Middle: Perceptron with Gauss-Bernoulli coefficients for the signal. The results of a logistic regression with fine-tuned 1 sparsity-enhancing penalty are very close to optimal. Right: The symmetric door activation rule with parameter K = K * 0.67449 chosen in order to observe the same number of occurrence of the two classes. Using Keras[START_REF] Chollet | keras[END_REF], a neural network with two hidden layers was able to learn approximately the rule, but only for much larger training set sizes (shown in inset).

2 Bayesα 5 SE

 25 Figure 7.7:The generalization error for three regression problems is plotted as a function of the number of samples per dimension α. The red line is again the Bayes-optimal generalization error, while the green one shows the (asymptotic) performances of GAMP as predicted by the state evolution (SE), when different. Again, we also show the result of GAMP on a particular instance (black dots) and, in blue, the performance of an out-of-the-box solver. Left: White Gaussian noise output and a Gauss-Bernoulli signal. For this choice of noise, there is no sharp transition (as opposed to what happens at smaller noises). The results of a Lasso with fine-tuned 1 sparsity-enhancing penalty are very close to optimal. Middle:Here we analyze a ReLU output function ϕ(x) = max(0, x), still with a Gauss-Bernoulli signal. We show for comparison the results of maximum likelihood estimation performed with CVXPY -a powerful python-embedded language for convex optimization[START_REF] Diamond | CVXPY: A Python-embedded modeling language for convex optimization[END_REF]-using two methods that are both amenable to convex optimization: In CVX-1 we use only the nonzero values of Y , and perform a minimization of the 1 norm of x subject to Y µ = Φ µ , x for µ ∈ {1, . . . , m} such that Y µ = 0, while in CVX-2, we use all the dataset, with the constraint that Y µ = Φ µ , x for µ ∈ {1, . . . , m} such that Y µ = 0 (as before) and the additional restriction Φ µ , x ≤ 0 for µ ∈ {1, . . . , m} such that Y µ = 0. In both case, a perfect generalization is obtained only for α 1. Right: The sign-less output function ϕ(x) = |x|. In inset, we show the performance for the estimation problem using PhaseMax[START_REF] Goldstein | Phasemax: Convex phase retrieval via basis pursuit[END_REF].

2 ( 7 Proposition 7 . 7 . 3

 27773 .7.7) where Y 0 ∼ P out (• | √ qV + √ ρ -qW * ) is the output of the second scalar channel (7.3.3). Let f : R → R be a continuous bounded function. Assume that P out is informative and that (h1)-(h2)-(h3)-(h4) hold and that either (h5.a) or (h5.b) holds. Then for all α ∈ D * we have

def = 1 √

 1 n x, Φ new . By[START_REF] Kallenberg | Random measures, theory and applications[END_REF] Theorem 4.11] it suffices to show that for all continuous bounded function f , ν n f (d) ---→ n→∞ νf . Let f be a continuous bounded function and define F (z) = f (y)P out (dy|z).

=

  νf . νf is therefore the only point of accumulation (for the weak convergence) of the sequence (ν n f ) n≥1 . This sequence is tight because bounded we conclude that ν n f (d) ---→ n→∞ νf for all continuous bounded function f and therefore ν n

.8. 2 ) 7 . 8 . 1

 2781 Proposition

Proposition 7 . 8 . 5

 785 (i) ∆ > 0. (ii) ∆ = 0 and ϕ takes values in N.Under the hypotheses presented above, E f is continuous on [0, ρ].

Figure 8 . 1 :

 81 Figure 8.1:Estimation risk of the Lasso for different choices of λ, as a function of δ. N = 8000. In both plots, σ = 0.2. The true coefficients vector θ is chosen to be sN -sparse with s = 0.1. The entries on the support of θ are drawn i.i.d. N (0, 1). Cross-validation is carried out using 4 folds. SURE is computed using the estimator σ for the plot on the left, and the true value of σ on the right. Left: A standard random design with (X ij ) ∼ iid N (0, 1/n). Right: The rows of the design matrix X are i.i.d. Gaussian, with correlation structure given by an autoregressive process, see Eq.(8.4.4). Here we used φ = 2.

Figure 8 . 2 :

 82 Figure 8.2:Risk of the Lasso for different choices of λ. N = 10000, σ = 0.2, δ = 0.8. Here θ is chosen to be sN -sparse, and we vary the sparsity level s. The entries on the support of θ are i.i.d. N (0, 1). Cross-validation is carried out using 4 folds. SURE is computed using the estimator σ. The minimax regularization λ MM (s 0 ) is used at the nominal level s 0 = 0.3.

Theorem 8 . 5 . 1 (

 851 Theorem 3 from [205] )Let S w ⊂ R N and S u ⊂ R n be two compact sets and let Q : S w × S u → R be a continuous function. Let G = (G i,j ) i.i.d.

.5. 3 ) 8 . 5 . 1 (

 3851 Corollary a) Let D ⊂ R N be a closed set. We have for all t ∈ R P minw∈D C λ (w) ≤ t ≤ 2P min w∈D L λ (w) ≤ t . (b) Let D ⊂ R N be a convex closed set. We have for all t ∈ R P min w∈D C λ (w) ≥ t ≤ 2P min w∈D L λ (w) ≥ t .

  min w∈D C λ (w) ≥ minw∈R N C λ (w) + , for some > 0. Using Gordon's min-max Theorem (Corollary 8.5.1) we will be able to showP min w∈D C λ (w) ≤ min w∈R N C λ (w) + ≤ 2P min w∈D L λ (w) ≤ min w∈R N L λ (w) + + o N (1). (8.5.5) Informally, this is a consequence of the following two remarks. First, by applying parts (a) and (b) of Corollary 8.5.1 to the convex domain R N , we deduce that min w∈R N C λ (w) ≈ min w∈R N L λ (w). Second, by applying part (a) to the closed domain D, we obtain min w∈D C λ (w) min w∈D L λ (w)

9. 1 . 1 Basic properties of the scalar optimization problem Lemma 9 . 1 . 1 (

 11911 From [71] )

  We have max β min τ ψ λ (β, τ ) ≥ min τ ψ λ (0, τ ) = -λ δ E|Θ|. Then, by taking w = 0 one get max β≥0 min τ ≥σ ψ λ (β, τ ) ≤ max β≥0 min τ ≥σ

Lemma 9 . 1 . 5

 915 Definew * (α, τ ) = η Θ + τ Z, ατ -Θ .w * (α, τ ) is the minimizer of w → w 2 2τ β -βZw + λ|w + Θ| (recall that we always write α = λ/β).If β ≥ β max the equation

Case 3 : 3 •

 33 α * < (α 1 + α 0 )/2. In that case M s (α * ) ≤ -c, for some constant c = c(δ, s) > 0. Consequently by (9.1.13) we getτ * (β * ) ≤ σ The mapping λ → β * (λ) is C ∞ and 2α -1 min -Lipschitz on R >0 . • The mapping λ → τ * (λ) is C ∞ and M -Lipschitz on [λ min , λ max ], for some constant M (Ω) > 0.Proof. The first point has already been by Proposition 9.1.1. λ → τ * (λ) is the composition of the mappings λ → α * (λ) and α → τ * (α), that are both C ∞ by Lemma 9.1.5 and Proposition 9.1.1. Compute the derivative:

1

 1 v∈RN L λ (v) + we have w λ -w 2 ≤ There exists constants c, C > 0 that only depend on Ω such that for all closed set D ⊂ R N and for all ∈ (0, 1],P min w∈D C λ (w) ≤ min w∈R N C λ (w) + ≤ 2P min w∈D L λ (w) ≤ min w∈R N L λ (w) + 3 + C e -cn 2 .

  w∈R N C λ (w) > L * (λ) + ≤ P min w∈D C λ (w) ≤ L * (λ) + 2 + C e -cn 2 ,where we used Proposition 9.3.2 above. We can now apply the first point of Corollary 8.5.1 to obtain: P min w∈D C λ (w) ≤ L * (λ) + 2 ≤ 2P min w∈D L λ (w) ≤ L * (λ) + 2 . (9.3.1)

  v∈R N L λ (v) + 3γ . The intersection of (9.3.2) with the event min w∈D L λ (w) ≤ min w∈R N

Proof.

  Let v ∈ ∂|θ |. By convexity, we have for all w ∈ K we have w, v ≤ |w + θ | -|θ | ≤ 0. Now for all x ∈ R N and α ≥ 0 x -αv = sup w =1 x -αv, w ≥ sup w∈K, w =1 w, x -w, αv ≥ sup w∈K, w =1 w, x . (9.3.4)

Theorem 8 . 3 . 1 and

 831 Theorem 8.3.2-(8.3.8) are proved the same way. Proof of Theorem 8.3.1. Let γ > 0 as given by Theorem 8.5.3 and let K = K(Ω) > 0 as given by Lemma 9.3.5. Let M = M (Ω) > 0 such that λ → µ * λ is M -Lipschitz with respect to the Wasserstein distance W 2 on [λ min , λ max ], as given by Proposition 9.3.6. Let ∈ (0, 1] and define = min γ 2KN q , M +1 . Let k = (λ max -λ min )/ . Define, for i = 0, . . . , k: λ i = λ min + i . By Theorem 8.5.3, the event ∀i

  4.2 and Proposition 9.3.2 we can bound P max u∈R n min w∈R N c λ (w, u) ≤ L * (λ) -= P min w∈R N C λ (w) ≤ L * (λ) -≤ C e -cn 2 .

  λ is defined by Equation9.2.1) admits a unique maximizer b * λ over R ≥0 and b

8 κ 2 .

 82 β), where we recall that λ is defined by (9.2.1). We getmin w∈B λ (w, b λ (v)) ≥ maxβ≥0 min w∈B λ (w, β) -1 The function β → min w∈B λ (w, β) is 1-strongly concave and maximized (by Lemma 9.5.3 above) at b * λ , hence |b λ (v) -b * λ | ≤ κ/2.

.5. 7 ) has probability at least 1 -

 71 Ce -cn . Let us define the constants a =

Proposition 9 . 5 . 3 .

 953 Let us now fix a constant κ ∈ (0, β min /2) that verify the statement of Lemma 9.5.5. Let us work on the intersection of the event {|b * λ -β * (λ)| ≤ κ/2} with the event of Lemma 9.5.5. This intersection has by Lemma 9.5.3 and Lemma 9.5.5 probability at least 1 -Ce -cn . Let v ∈ D κ . By Lemma 9.5.4 the unique maximizer b

λ: proof of Theorems 9 . 5 . 2 , 9 . 5 .4 and 9 . 5 . 6

 95295956 Theorems 9.5.2, 9.5.4 and 9.5.6 are deduced from Theorems 9.5.1, 9.5.3 and 9.5.5 by annet argument, as we did to deduce Theorems 8.3.1 and 8.3.2 from Theorems 8.5.3 and 9.3.1. Since the ideas are the same, we only present here the key argument: Proposition 9.5.5

  ). Since Θ x +τ x * Z and Θ v +τ * Z v have the same law andP Θ x +τ * Z x ≥ α * τ * E = P Θ v + τ * Z v ≥ α * τ * E (by Lemma 9.6.8), we have P Θ x + τ * Z x ≥ α * τ * E c = P Θ v + τ * Z v ≥ α * τ * E c . Similarly we have P Θ x + τ * Z x ≤ -α * τ * E c = P Θ v + τ * Z v ≤ -α * τ * E c .

  x + τ * Z x | ≥ α * τ * , T v on the event E provided that |Θ x + τ * Z x | < α * τ * , T x on the event E c provided that | Θ x + τ * Z x | ≥ α * τ * , T v on the event E c provided that | Θ x + τ * Z x | < α * τ * .

2 ≤ 9 . 6 . 6 2 N ( 1 -1 n θ λ 0 ) 2 = 2 κ

 29662122 C and consequently W 2 ( µ ( θ d λ ,θ ) , µ d λ ) 2 ≤ C , on the event E which has probability at least 1 -C -12 e -cN 17 . Proof of Corollary 8.4.2 Let ∈ (0, 1]. Let us work on the intersection the events of Theorem 9.6.1,Corollary 8.4.1 and 9.5.3, which as probability at least 1 -C 6 e -cN 6 . Let λ ∈ [λ min , λ max ]. 1 N X T (y -X θ λ ) 2 = λ 2 N v λ 2 = λ 2 κ * (λ) + O( ) .We have also 1-1 n θ λ 0 = 1 -1 δ s * (λ) + O( ) = β * (λ)/τ * (λ) + O( ). Therefore X T (y -X θ λ ) λτ * (λ) β * (λ) * (λ) + O( ) = τ * (λ) 2 1 + δ -2s * (λ) -δσ 2 + O( ) .Now we have τ (λ) = τ * (λ) + O( ) and 1 N θ λ 0 = s * (λ) + O( ). Consequently τ (λ) 2 2 N θ λ 0 -1 = τ * (λ) 2 (2s * (λ) -1) + O( ) .

  ) θ + σz (\i) -X(\i) θ the minimizer of the Lasso cost (8.3.1) for δ(k) = k-1 k δ and σ (k) = k/(k -1)σ. Let τ (k) * (λ)be the τ * defined by Theorem 8.3.1, but with δ (k) instead of δ and σ (k) instead of σ. Define the corresponding 'risk':R (k) * (λ) = δ (k) τ (k) * (λ) 2 -(σ (k) ) 2 .It is not difficult to verify that the bounds on τ * , β * of Section 9.1.2 are uniform with respect to δ and σ. More precisely sup δ∈[δ min ,δmax] sup σ∈[σ min ,σmax] sup λ∈[λ min ,λmax]sup θ ∈D τ * (λ, δ, σ) + β * (λ, δ, σ) < +∞ ,where δ max , δ min , σ max , σ min > 0 such that s max (δ min ) > s if we are in the case D = F 0 (s). This gives that under the assumptions of Proposition 8.4.3, τ (k) * and R (k) * are bounded for all k ≥ 2 (that verify s max (δ(k -1)/k) > s in the case D = F 0 (s)) by some constant that depends only on Ω. Lemma 9.6.11

cn 4 .

 4 Proof. Let i ∈ {1, . . . , k}. Let us define for θ ∈ R N ,

λ

  (x)+KN q |λ-λ |(9.6.19) has probability at least 1 -Ce -cn . On the event E 2 , we have for all j ∈ {1, . . . , k} and all λ ∈ [λ j-1 , λ j ]λ j (x) + KN q η ≤ min x∈R N L (\i) λ j (x) + γ .

k 4 .

 4 Proof.Let us fix i ∈ {1, . . . , k}. By Proposition 9.3.7,λ → R * (λ) is K 1 -Lipschitz on [λ min , λ max ],for some constant K 1 > 0. By Propositions 9.3.3 and 9.3.4 there exists a constant K 2 > 0 such that the eventE 1 = ∀λ ∈ [λ min , λ max ], 1 N | θ i λ | -|θ | ≤ K 2 N q and 1 N | θ λ | -|θ | ≤ K 2 N q (9.6.20)has probability at least 1 -Ce -cn . Let us define η = minδ 2N q kK 2 , 1 K 1 √k and M = (λ maxλ min )/η . For all j ∈ {0, . . . , M }, we write λ j = min λ min + jη, λ max . By Theorem 8.3.2 the eventE 2 = ∀λ ∈ [λ min , λ max ], 1 N θ i λ -θ 2 -R (k) * (λ) ≤1 and 1 n y -X θ λ 2 -β * (λ) ≤ 1 (9.6.21)

  for some constant C > 0. If now k < κ. Then on the eventE 2 we have ∀j ∈ {1, . . . , M }, ∀λ ∈ [λ j-1 , λ j ], 1 N θ i λ -θ 2 -R * (λ j ) ≤ sup λ∈[λ min ,λmax] R (k) * (λ) + sup λ∈[λ min ,λmax]R * (λ) + 1

Lemma 9 . 6 . 13 Define

 9613 ∆ α (x) = E α (x + Z) -α|x| .The function ∆ α is continuous, even, decreasing on R ≥0 , α-Lipschitz. Moreover   ∆ α (0) = 1 2 + αφ(α) -(1 + α 2 )Φ(-α) lim x→±∞ ∆ α (x) = -α 2 2 and ∆ α (0 + ) = -∆ α (0 -) = -α.Proof. Since Z and -Z have the same law, one verify easily that ∆ α is an even function. We have for all x > 0∆ α (x) = E α (x + Z) -α = E [1(x + Z ∈ [-α, α])(x + Z -α)] ≤ 0 . α is convex, therefore x → E[ α (x + Z)] is non-decreasing. E[ α (Z)] = 0 because α is an odd function. Consequently, for all x > 0 -α ≤ E[ α (x + Z) -α] = ∆ α (x) .

α 0 z 2 α 2 2 .

 022 φ(z)dz + +∞ α (2αz -α 2 )φ(z)dz .By integration by partsα 0 z 2 φ(z)dz = -zφ(z) α 2 )φ(z)dz = -α 2 Φ(-α) + 2αφ(α) . Therefore ∆ α (0) = 1 2 + αφ(α) -(1 + α 2 )Φ(-α).We have almost surelyα (x + Z) -α|x| ----→ x→±∞ -Thus, by dominated convergence lim x→±∞ ∆ α (x) = -α 2 2 .

w∈B 1 2n 1 n 3 / 2 2 1n w 2 + 2 +λ 9 . 1 :

 113222291 Xw -σz 2 + λ n v T (θ + w) u g T w + 1 n 3/2 w h T u -σ n u T z -1 2n u 2 + λ n |w + θ | -|θ | u * λ V λ (v) = min w∈B 1 σ 2 h √ n -1 n g T w + g σ √ n + λ n v T (w + θ ) -λ n |θ | v *Table Main cost/objective functions

  )g, α * (λ)τ * (λ) -θ u λ = β * (λ) τ * (λ) τ * (λ) 2 -σ 2 h √ n -σ √ n z v λ = -α * (λ) -1 τ * (λ) -1 η θ + τ * (λ)g, α * (λ)τ * (λ) -θ -τ * (λ)g

1 Let

 1 D u ⊂ R n 1 +n 2 and D v ⊂ R m 1 +m 2 be compact sets and let Q : D u × D v → R be a continuous function. Let G = (G i,j ) i.i.d.

ṽT

  Gũ + Q(u, v) , L * (g, h) = min u∈Du max v∈Dv ṽ g T ũ + ũ h T ṽ + Q(u, v) .

YYṽT

  (u, v) + Q(u, v) ≤ t ≤ P min u∈Du max v∈Dv Y (u, v) + Q(u, v) ≤ t , (u, v) + Q(u, v) ≤ t z ≤ 0 Gũ + Q(u, v) ≤ t ≤ 2P min u∈Du max v∈Dv ṽ g T ũ + ũ h T ṽ + Q(u, v) ≤ t .

P

  min v∈Dv max u∈Du ṽT Gũ -Q(u, v) ≤ -t ≤ 2P min v∈Dv max u∈Du ṽ g T ũ + ũ h T ṽ -Q(u, v) ≤ -t ,which gives (using the fact that (G, g, h) and (-G, -g, -h) have the same law):P max v∈Dv min u∈Du ṽT Gũ + Q(u, v) ≥ t ≤ 2P max v∈Dv min u∈Du ṽ g T ũ + ũ h T ṽ + Q(u, v) ≥ t .By Proposition C.2, one can switch the min-max of the left-hand side, because Q is convexconcave and we are working on convex sets D u and D v . For the right-hand side, we simply use the fact that: max v∈Dv min u∈Du ṽ g T ũ + ũ h T ṽ + Q(u, v) ≤ min u∈Du max v∈Dv ṽ g T ũ + ũ h T ṽ + Q(u, v) , to conclude the proof.

2 . (B. 3 ) 2 ,

 232 By Cauchy-Schwarz inequality, we have for all positive, semi-definite matrix M ∈ R n×n , Tr(M ) 2 ≤ nTr(M 2 ). HenceF (N ) (λ) ≥ 1 2n E Tr xx T λ,N -x λ,N x Tby Jensen's inequality. Let now 0 < s < t. By integrating (B.3) we get

F

  (N ) (t) -F (N ) (s) ≥ 1 2n t s MMSE (N ) (λ) 2 dλ . (B.4)The sequence of convex functions (F (N ) ) N converges (by Proposition B.1) to F which is differentiable. Proposition C.1 gives that the derivatives (F (N ) ) N converge to F and therefore MMSE (N ) converges to MMSE. Therefore, equation (B.4) gives F (t) -F (s) s)MMSE(t) 2 .

  2 and Corollary B.2. Proposition B.2Let P U be a probability distribution over N m that admits a finite second moment. Let U ∼ P U and Z ∼ N (0, Id m ) be two independent random variables. Then H(U ) = -n∈N m P U (n) log P U (n) is finite and for all ∆ ∈ (0, 1],I(U ; U + √ ∆Z) -H(U ) ≤ 48me -1/(16∆) .Proof. Let us define for ∆ > 0, h(∆) = I(U ; U + √ ∆Z) = I P U (∆ -1 ). By Proposition 1.3.3 we have for all ∆ > 0,h (∆) = -1 2∆ 2 MMSE(U | U + √ ∆Z) . (B.5)We are now going to upper bound MMSE(U | U + √ ∆Z) by considering the following estimator:

U 2 .is bounded in L 1 .- 2 Z 2 E 2 ,

 21222 ∆)| ≤ 3m ∆ 2 e -1/(16∆) . (B.6)Since h(1) is finite and 1 0 e -1/(16∆) ∆ 2d∆ < +∞ we obtain that sup∆∈(0,1] |h(∆)| < +∞ . (B.7)By definition of h:h(∆) = I(U ; (B.8)By the previous equality and (B.7), the family of (non-negative) random variables Notice that (by dominated convergence) logP U (U )e -1 2 Z 2 = 1 -log P U (U )almost surely. This gives (by Fatou's Lemma) that this almost-sure limit is integrable and thus that H(U ) = -E log P U (U ) is finite. Let us now show that h(∆) ---→ ∆→0 H(U ). We have almostsurely log P U (U )e -1 2 Z 2 ≤ log U ∈N m P U (U ) exp -1 2∆ U + √ ∆Z -U 2 ≤ 0 .Since we now know that the left-hand side is integrable (because H(U ) is finite), we can apply the dominated convergence theorem to obtain that log P U (U )e -1 2 Z 2 = H(U ) -m which combined with (B.8) gives h(∆) ---→ ∆→0 H(U ). Now, using the bound on the derivative of h (B.6) we conclude that for all ∆ ∈ (0, 1], |h(∆) -H(U )| ≤ 3m ∆ 0 e -1/(16t) t 2 dt = 3m 16e -1/(16t) ∆ 0 = 48me -1/(16∆) .

  By applying Proposition B.2 twice, we get|I(U ; U + √ ∆Z) -H(U )|, |I(U ; U + √ ∆Z|X) -H(U |X)| ≤ 48me -1/(16∆) .Since I(X; U ) = H(U ) -H(U |X) we obtain the desired inequality.

Proposition C. 2 ( 1 A 2 x -y 2 .Remark C. 1 . 1 0 1 0

 2122111 Corollary 37.3.2 from [182])Let C and D be non-empty closed convex sets in R m and R n , respectively, and let f be a continuous finite concave-convex function onC × D. If either C or D is bounded, convex function f over R n is said to be • γ-strongly convex if x → f (x) -γ 2 x 2 is convex. • L-strongly smooth is f is differentiable everywhere and for all x, y ∈ R n we havef (y) ≤ f (x) + (y -x) T ∇f (x) + L If f is convex, differentiable over R n , and ∇f is L-Lipschitz, then f is L-strongly smooth. Indeed, if we take x, y ∈ R n and if we define h(t) = f ((1 -t)x + ty) we have f (y) -f (x) = h(1) -h(0) x) T ∇f ((1 -t)x + ty)dt = (y -x) T f (x) + (y -x) T ∇f ((1 -t)x + ty) -∇f (x) dt ≤ (y -x) T f (x) + tL x -y 2 dt ≤ (y -x) T f (x) + L 2 x -y 2 .

  p.110) of a non-decreasing convex function f : R ≥0 → R by: f * (x) = sup y≥0 {xy -f (y)}. (C.1) The most fundamental result on the monotone conjugate is the analog of the Fenchel-Moreau theorem: Proposition C.4 ([182] Theorem 12.4 )

  Let P 0 be a probability distribution on R with unit second moment E P 0 [X 2 ] = 1. By considering the estimator x =

	1 1+λ . We conclude:			√ 1+λ Y , one obtain MMSE P 0 (λ) ≤ λ
	sup P 0	MMSE P 0 (λ) =	1 1 + λ	and	inf P 0

1 2 λ -log(1 + λ) . Remark 1.3.1 (Worst-case prior).

  1 n E[max S n-1 H 0 → GS k . The reverse Fatou lemma gives then

	lim sup n→∞	E	1 n	max x∈S n-1

  .3.15) By Proposition 1.2.2 and Remark 1.2.1 in Section 1.2 we deduce from Theorem 7.3.2: Under the conditions of Theorem 7.3.2 we have for all α ∈ D

	Corollary 7.3.2

* sup x

  Under the assumptions of Theorem 7.3.2, we have for all α ∈ D * Let α ∈ D * . Let ν n be the law of Y new conditionally on Φ, Y , Φ new , i.e.

	Theorem 7.3.3
	E opt gen (α) ---→ n→∞ E(q * (α))	(7.3.21)
	where q * (α) is the optimizer of (7.3.8), see Proposition 7.3.1.
	Theorem 7.3.3 follows from a stronger result that we state below:
	Theorem 7.3.4

f (y)dν n (y) = f (y)P out dy x, Φ new / √ n dP (x|Y , Φ),

  ) with the optimal correlation given by Corollary 7.3.2, we see that if lim t→∞ q t = q * (α), then GAMP is information-theoretically optimal. Provided that Claim 7.4.1 holds we deduce the generalization error of GAMP:

	Comparing (7.4.7Proposition 7.4.1
	Suppose that hypotheses (h1)-(h2)-(h4) hold. Moreover suppose that either (h5.a)
	or (h5.b) holds. Assume that (Φ µi )	i.i.d.
		q t ρ	.	(7.4.7)

.4.6) Consequently lim n→∞ x t , X * x t X * =

  3.1 with the following matching lower-and upper-bounds and Propositions C.7 and C.8 in Appendix C.3. Let us start with the lower bound.

	Proposition 7.6.3
	lim inf

n→∞ F n ≥ sup r≥0 inf q∈[0,ρ]

7.7.1 Unicity of the optimizer q * of the replica formula: Proof of Proposition 7.3.1

  Since ψ P 0 and Ψ Pout are related to I P 0 and I Pout by the relations(7.

	3.2)-(7.3.5), the op-
	timizers of (7.3.8) and (7.3.9) are the same. We chose here to work with (7.3.9). The
	function
	h : α → inf q∈[0,ρ]

  .8.7)Ψ Pout is thus convex on [0, ρ]. It is not difficult to verify (by standard arguments of continuity under the integral) that Ψ Pout is continuous on [0, ρ], which gives that Ψ Pout is C 2 on its domain.

	Proposition 7.8.2

  ). Corollary 8.4.2 is proved in Section 9.6.6 of Chapter 9. Since by Corollary 8.4.2, Corollary 8.4.1, Theorem 8.3.2 we have with high probability R(λ)

  .1.12)-τ * (β * )λM s (α * ) ≤ δσ 2 ,

						(9.1.13)
	τ * (β * )(δ -	1 2	M s (α * )) ≤ β * .	(9.1.14)
	Proof. The inequality (9.1.12) simply follows from the previous lemma and from the fact that
	ψ λ (β, τ * (β)) ≤ max β≥0	min τ ≥σ	ψ λ (β, τ ) ≤	σ 2 2	,
	by Lemma 9.1.2. Let us prove (9.1.13). By Lemma 9.1.7, we have

  Since by Theorem 9.1.2, τ * (α * ) ≤ τ max (Ω) and α * ≤ λ max /β min (Ω), the derivative of τ * with respect to λ is bounded on [λ min , λ max ].

			-1 min λ max β -2 min .
	By Lemma 9.1.5, we have
	∂τ * ∂α	(α * (λ)) ≤ (α * (λ) + 1)	τ * (α * ) 3 δσ 2 .

  2 , 1 N w * λ -w λ 2 ≤ . From this we deduce easily that with the same probability |L λ (w * λ ) -L λ (w λ )| ≤ M , for some constant M > 0, which gives by Proposition 9.6.1:

	Corollary 9.2.1
	Define
	L * (λ) = ψ λ (β * (λ), τ * (λ)) .	(9.2.3)
	The exists constants c, C > 0 that only depend on Ω such that
	P min w∈R N L λ (w) -L * (λ) ≥	≤	C e -nc 2 .
	9.2.2 Proof of Theorem 9.2.1
	Proposition 9.2.2

  2 , where the last inequality comes from Corollary 9.2.1. The bound of the probability of the converse inequality is proved analogously.

	Proof of Proposition 9.3.1. Recall that L * (λ) is defined by (9.2.3). Let D ⊂ R N be a
	closed set.
	P min w∈D	C λ (w) ≤ min w∈R N	C λ (w) +
	≤ P min w∈D	C λ (w) ≤ min w∈R N	C λ (w) +	and min w∈R N	C λ (w) ≤ L * (λ) +
	+ P min

Control of the 1 -norm of the Lasso estimator Proposition 9.3.3 Let

  3.3), for all w ∈ D , L λ (w) > min

			v∈R N	L λ (v) + 3γ . The intersection
	of (9.3.3) with the event min w∈D	L λ (w) ≤ min w∈R N	L λ (w) + 3γ is therefore empty: the lemma is
	proved.
	9.3.2 Uniform control over λ: proofs of Theorems 8.3.1 and 8.3.2-
	(8.3.8)

  Ce -cn . Such K exists by Propositions 9.3.3 and 9.3.4. On the event (9.3.7) we have for all λ, λ ∈ [λ min , λ max ]:

				.3.6)
	Proof. K = K(Ω) > 0 be a constant such that for all θ ∈ D, the event
	∀λ ∈ [λ min , λ max ],	1 n	| θ λ | -|θ | ≤ KN q	(9.3.7)
	has probability at least 1 -

  Let γ > 0 as given by Theorem 9.3.1 and let K = K(Ω) > 0 as given by Lemma 9.3.5. Let M

3.2-(8.3.8).

  By Proposition 9.4.1, there exists a constant K such that with probability at least 1 -Ce -cn we have ∀λ

.5.11)

Proof.

  Using the same arguments that we use to deduce Theorems 8.3.1 and 8.3.2 (8.3.8) from Theorem 8.5.3 and Theorem 9.3.1 in Section 9.3.2, we deduce from Proposition 9.6.4 that for all ∈ (0, 1]

	sup
	θ ∈D

10) we have min w∈D L λ (w) > min w∈R N L λ (w) + 3γ 3 . We conclude P min w∈D L λ (w) ≤ min w∈R N L λ (w) + γ 3 ≤ C 3 e -cn 6 .

  Let x, y ∈ B(0, √ nR) and define for t ∈ [0, 1], g(t) = f (z t ), where z t = (tx + (1 -t)y). + σ 2 ) 3/2 .

	Compute
				g (t) =	1 n (x -y) T z t n + σ 2 zt 2	,
	and
	g (t) =	1 n x -y 2 zt 2 n + σ 2	-	1 n (x -y) T z t zt 2 n + σ 2 3/2 2
	=	1 n + σ 2 3/2 zt 2	1 n	x -y 2 z t n	2	+ σ 2 -	1 n	(x -y) T z t	2
	≥ (R 2 Consequently σ 2 zt 2 n + σ 2 3/2 1 n x -y 2 ≥ 1 n x -y 2	σ 2
	tf (x) + (1 -t)f (y) = tg(1) + (1 -t)g(0)
		≥ g(t) +	1 2	t(1 -t)	1 n	x -y 2	σ 2

3/2

-strongly convex on B(0, √ nR).

Proof.

Table 9 . 2 :

 92 "Asymptotic optimizers"

  t .

	Remark 9.7.1. The bound P(X >	√	2vt + ct) ≤ e -t implies that
	P(X > t) ≤	  	exp -t 2 8v exp -t 2c	for 0 < t ≤ 2v c 2 , for t ≥ 2v c 2 .
	Proposition 9.7.3

  2 ) q-2 . By Bernstein's inequality (see for instance Theorem 2.10 in[START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF])log Ee s(Y 2 -E[Y 2 ]) ≤ 16σ 2 s 2 2(1 -2σ 2 |s|) . 2µY is 4µ 2 σ 2 -sub-Gaussian, therefore log Ee 2µsY ≤ 2µ 2 σ 2 s 2 and log Ee s(X 2 -E[X 2 ]) = log Ee s(Y 2 -E[Y 2 ])+2µsY ≤ 1 2 log Ee 2s(Y 2 -E[Y 2 ]) + 1 2 log Ee 4µsY ≤ 16σ 2 s 2 1 -4σ 2 s + 4µ 2 σ 2 s 2 ≤ (16σ 2 + 4µ 2 σ 2 )s 2 1 -4σ 2 |s| .X 2 is therefore a Sub-Gamma random variable with variance factor v = 16σ 2 + 4µ 2 σ 2 and scale parameter c = 4σ 2 .Let X be a σ 2 -sub-Gaussian random variable. Define m = E[|X|]. Let Y be a random variable bounded by 1. Then XY is 16(m 2 + 2σ 2 )-sub-Gaussian.

	Lemma 9.7.1

Proof. We have |E[XY

]| ≤ m, therefore

  P 1 X 2 + E P 2 X 2 W 2 (P 1 , P 2 ) . Let > 0. Let us fix a coupling Q of X 1 ∼ P 1 and X 2 ∼ P 2 such that E X 1 -X 2 2 1/2 ≤ W 2 (P 1 , P 2 ) + .

	Proof of Proposition B.1. Let us consider for t ∈ [0, 1] the observation model
	 	Y 1 (t)	=	√	λtX 1 + Z 1 ,
		Y 2 (t)	=		λ(1 -t)X 2 + Z 2 ,

(X 1 ; √ λX 1 + Z) -I(X 2 ; √ λX 2 + Z) ≤ λ E where Z 1 , Z 2 i.i.d.

According to the physics convention, this should be minus the Hamiltonian, since a physical system tries to minimize its energy. However, we chose here to remove it for simplicity.

This is in fact minus the free energy, but we chose to remove the minus sign for simplicity.

This terminology usually refers to the case where V is a standard Gaussian vector. We consider here a slightly more general case by allowing the entries of V to be taken i.i.d. from any probability distribution.

named after Thouless, Anderson and Palmer[START_REF] David J Thouless | Solution of'solvable model of a spin glass[END_REF] who proposed a variational formula (which was recently rigorously proved by[START_REF] Chen | On the tap free energy in the mixed p-spin models[END_REF]) for the limiting free energy of a mean-field model of spin glasses, the Sherrington-Kirkpatrick model[START_REF] Sherrington | Solvable model of a spin-glass[END_REF].

√

, as computed in[18,

[START_REF] Caltagirone | Recovering asymmetric communities in the stochastic block model[END_REF]. For a more detailed analysis of the phase transitions in the spiked Wigner model, see[START_REF] Thibault Lesieur | Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications[END_REF] where many other priors are considered.

We note here that none of the results of this section are changed if one symmetrizes Z, i.e., if we work with the symmetric Gaussian k-tensor.

As shown in Proposition 6.4.2, x → x ⊗k , Y admits almost surely a unique maximizer over S n-1 if k is odd, and two maximizers x * and -x * if k is even. In the case of even k, x ML λ is simply picked uniformly at random among {-x * , x * }.

Notice that this allows to study any channel whose outputs belong to a countable set S by applying a injection u : S → N to the outputs.

Remerciements

Chapter 8

The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning

Introduction to the Lasso

Given data (x i , y i ), 1 ≤ i ≤ n, with x i ∈ R N , y i ∈ R, the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF][START_REF] Chen | Examples of basis pursuit[END_REF] Here X ∈ R n×N is the matrix with rows x 1 , . . . , x n , y = (y 1 , . . . , y n ), v denotes the 2 norm of vector v, and |v| its 1 norm. To fix normalizations, we will assume that the columns of X have 2 norm 1 + o [START_REF] Abbe | Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic bp, and the information-computation gap[END_REF]. (Note that this normalization is different from the one that is sometimes adopted in the literature, but the two are completely equivalent.) A large body of theoretical work supports the use of 1 regularization in the highdimensional regime n N , when only a small subset of the coefficients θ are expected to be large. Broadly speaking, we can distinguish two types of theoretical approaches. A first line of work makes deterministic assumptions about the design matrix X, such as the restricted isometry property and its generalizations [START_REF] Emmanuel | Decoding by linear programming[END_REF][START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF]. Under such conditions, minimax optimal estimation rates as well as oracle inequalities have been proved in a remarkable sequence of papers [START_REF] Candes | The Dantzig selector: Statistical estimation when p is much larger than n[END_REF][START_REF] Peter | Simultaneous analysis of Lasso and Dantzig selector[END_REF][START_REF] Van De | On the conditions used to prove oracle results for the lasso[END_REF][START_REF] Sahand N Negahban | A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers[END_REF][START_REF] Raskutti | Minimax rates of estimation for high-dimensional linear regression over q -balls[END_REF]. As an example, assume that that the linear model is correct. Namely, y = Xθ + σz , (8.1.2) for σ ≥ 0, z ∼ N (0, Id n ), and θ a vector with s 0 non-zero entries. Then, a theorem of Bickel, Ritov and Tsybakov [START_REF] Peter | Simultaneous analysis of Lasso and Dantzig selector[END_REF] implies that, with high probability, λ ≥ σ c 0 log N ⇒ θ λ -θ 2 ≤ Cs 0 λ 2 , (8. 1.3) for some constants c 0 , C that depend on the specific assumptions on the design. (The normalization of [START_REF] Peter | Simultaneous analysis of Lasso and Dantzig selector[END_REF] is recovered by setting σ 2 = σ 2 # /n, where σ 2 # is the noise variance of [START_REF] Peter | Simultaneous analysis of Lasso and Dantzig selector[END_REF].)

Study of Gordon's optimization problem

Recall that g ∼ N (0, I N ) and h ∼ N (0, I n ) are independent standard Gaussian vectors. Define for (w,

and U λ (u) = min w∈R N m λ (w, u), U λ (u) = m λ (w λ , u), where w λ is defined by (9.2.2). Obviously we have U λ (u) ≤ U λ (u). We write also

(9.4.1)

Lemma 9.4.1

There exists constants C, c > 0 such that for all ∈ (0, 1] and any λ ∈ [λ min , λ max ] we have with probability at least 1 -Ce -cn 2

• U λ is 1/n-strongly concave on R n and admits therefore a unique maximizer u * λ over R n .

• | max u∈R n U λ (u) -L * (λ)| ≤ , where L * (λ) is defined by Corollary 9.2.1.

Proof. By Lemma 9.6.1 and Lemma 9.6.2,

δ which is greater than some constant γ > 0. Indeed

remains greater than some strictly positive constant while θ vary in D and λ vary in [λ min , λ max ]. By Lemma 9.6.1 we have then that with probability at least 1 -Ce -cn , w T λ g ≥ 0 which implies that U λ is 1/n-strongly concave. Let us compute

By the concentration properties of w λ (see Section 9.6.2), have that with probability at least 1 -

with the same probability, which implies the third point by strong concavity.

Proof of Theorem 9.4.1

Let us only prove the second point since the first one follows from the same arguments. Let ∈ (0, 1] and define

Let us define for (w, u) ∈ R N × R n : Let D be either F 0 (s) for some s < s max (δ) or F p (ξ) for some ξ ≥ 0, p > 0. Let q = 0 if D = F 0 (s) and q = (1/p -1) + if D = F p (ξ). By Propositions 9.3.3 and 9.3.4 there exists a constant K = K(Ω) such that the event

has probability at least 1 -Ce -cn . Let us fix this constant K and let us write

We define also

The function U λ is 1/n-strongly concave. On the event (9.4.4), u λ is the (unique) maximizer of U λ .

Proof. Let us work on the event (9.4.4) and let λ ∈ [λ min , λ max ]. We have, by permutation of max and min: max Let q = 0 if D = F 0 (s) and q = (1/p -1) + if D = F p (ξ). There exists constants C, c, κ > 0 such that for all θ ∈ D the following event

Proof. Let us work on the event (9.4.4), which has probability at least 1 -

Study of the subgradient v λ

The goal of this section is to analyze the vector

which is a subgradient of the 1 -norm at θ λ . Let us define

Main results

Let

where K > 0 is some constant (depending only on Ω) that will be fixed later in the analysis (in fact K is the constant given by Lemma 9.5.5). Notice that w λ ∈ B, with probability at least 1 -e -n/2 . Define

With probability at least 1 -e -n/2 we have for all λ ≥ λ min

Proof. Let us work on the event { z ≤ 2 √ n} which has probability at least 1 -e -n/2 . On this event we have w λ ∈ B and therefore

where the permutation of the min-max is authorized by Proposition C.2. The optimality condition of w λ gives that

Using the optimality condition again we obtain

Therefore v λ achieves the optimal value.

The empirical law of the subgradient

Let ν * λ be the law of the couple

Proof. Let v ∈ R N . By Proposition C.2 one can permute the min-max and obtain:

We can the apply Gordon's Theorem (Corollary 9.7.1) in order to compare

which is equal to max v∈D V λ (v). Note that the maximums in (9.5.3) and (9.5.4) are not defined on compact sets (since D × R n is not bounded). One has therefore to follow the same procedure than for Corollary 8.5.1, and show that there exists a compact set K ⊂ R n such that with high probability, the maximum over u ∈ R n is achieved in K. For the sake of brevity we do not provide a complete execution of this argument and refer to the proof of Corollary 8.5.1.

Study of Gordon's optimization problem

In this section we study the optimization problem max v ∞≤1 V λ (v). Let us define

The goal of this section is to prove:

Theorem 9.5.7

There exists constants γ, c, C > 0 that only depend on Ω such that for all θ ∈ D, all λ ∈ [λ min , λ max ] and all ∈ (0, 1]

Proposition 9.5.4

There exists constants c, C > 0 that only depend on Ω such that for all closed set D ⊂ R N and for all ∈ (0, 1],

The proof of Proposition 9.5.4 is omitted for the sake of brevity, and because it follows from the exact same arguments than Proposition 9.3.1.

The norm of v λ : proof of Theorem 9.5.3

Lemma 9.5.7

There exists constants γ, c, C > 0 that depend only on Ω, such that for all ∈ (0, 1] we have

where

and κ * (λ) is defined by (9.5.2).

Proof. Similarly to Proposition 9.6.1 it is not difficult to prove that for all ∈ (0, 1],

for some constants c, C > 0. By Theorem 9.5.7 there exists constants γ, c, C > 0 such that for all ∈ (0, 1] the event

(9.5.9) has probability at least C e -cn 2 . On the event (9.5.9), we have for all v ∈ D :

This gives that on the event (9.5.9), for all

V λ (v ) -3γ . The intersection of (9.5.9) with the event max

V λ (v) -3γ is therefore empty: the lemma is proved.

Proof of Theorem 9.5.3. Let γ > 0 be a constant that verify the statement of Lemma 9.5.7.

Let ∈ (0, 1] and define

where we used successively Proposition 9.5.4 and Lemma 9.5.7.

Lemma 9.6.2

Proof. The next proposition simply follows from Lemma 9.6.4 and standard concentration arguments, so we omit its proof. Proposition 9.6.1

There exists constant C, c > 0 that only depend on Ω such that for all ∈ [0, 1],

Concentration of the empirical distribution

Proposition 9.6.2

Let θ ∈ F p (ξ), where p, ξ > 0. Let µ = µ θ ⊗ N (0, 1) and let µ be the empirical distribution of the entries of θ i , g i 1≤i≤N , where g 1 , . . . , g N i.i.d.

∼ N (0, 1). Then there exists constants C, c > 0 that only depends on ξ p , such that for all ∈ (0, 1 2 ],

where a = 1 2 + 1 p .

Before proving Proposition 9.6.2, we will need two simple lemmas. For r ≥ 0 and x ∈ R we use the notation

Let µ |r be the empirical distribution of the entries of θ i , g i|r 1≤i≤N .

Consequently, since the events E 1 has probability at least 1 -O( 2), we have 2) because we are working on E, we conclude that

. One can prove the same way that

, which gives the desired result.

Lemma 9.6.7

The event

has probability at least 1 -C 2 , for some constant C > 0.

On the event E we have

We deduce then from (9.6.14) that P | V λ | = 1 and Θ λ = 0 = O( 2 ) and finally

Lemma 9.6.8

The event E has probability at least 1 -C 2 and on E we have

and

Proof. Since E 1 , E 2 , E 3 and E 4 have all a probability greater than 1 -O( 2), the event E =

. On E we have

(because we are on the event E 1 )

⇐⇒ V λ = 1 (because we are on the event E 3 )

⇐⇒ Θ λ > 0 (because we are on the event E 4 )

⇐⇒ Θ λ > 2 (because we are on the event E 3 )

⇐⇒ η(Θ x + τ * Z x , α * τ * ) > 0 (because we are on the event E 1 )

(because we are on the event E 2 ) .

Proof of Proposition 8.4.3

Let n ∈ {1, . . . , n}. We consider a random n × N matrix X and a random vector z = (z 1 , . . . , z n ) such that X i,j i.i.d.

∼ N (0, 1/n) and z i i.i.d.

∼ N (0, 1) are independent and independent of everything else. Lemma 9.6.9

There exists constants γ, c, C > 0 that only depend on Ω such that for all θ in D and all λ ∈ [λ min , λ max ] such that for all ∈ (0, 1],

Proof. The vector w λ is independent from X , z . Hence

where χ is independent from w λ and follows a χ-squared distribution with n degrees of freedom.

We have therefore for all t ≥ 0

for some constants c, C > 0. We know by Lemma 9.6.1 and Lemma 9.6.2 that 1 n w λ 2 concentrates exponentially fast around τ * (λ) 2 -σ 2 , which is (Theorem 9.1.2) bounded by some constant. There exists therefore constants C, c > 0 such that

.6.17)

From (9.6.15)-(9.6.16) and (9.6.17) above, we deduce that for all t ≥ 0

for some constants c, C > 0. By Proposition 9.7.4, we know that

for some constant C > 0, with probability at least 1 -Ce -cn . Consequently

be two centered Gaussian processes. Suppose that the functions

Then for all t ∈ R P min

Proof. Define the random variable

X and Y are continuous on the compact set D u × D v and are therefore uniformly continuous on this set: d 0 > 0 almost surely. Let > 0. By tightness there exists a constant d > 0 such that

Q is continuous and thus uniformly continuous on D u × D v : there exists δ ∈ (0, d] such that for all z, z

which gives by taking the complementary:

By construction of δ we have with probability at least 1 -

and similarly for Y . We have therefore, for all t ∈ R P min

and thus Let G be a n × N random matrix, whose entries are i.i.d. N (0, 1). For all t ≥ 0 we have

Appendix A Proof of Lemma 1.2.2

In order to prove Lemma 1.2.2, we first need to introduce some definitions and results about exchangeable infinite arrays.

Definition A.1

An infinite symmetric random array R is a collection of random variables

We say that R is

• weakly exchangeable if for all n ≥ 1 and for all permutation σ of {1, . . . , n} we

• positive semi-definite if for all n ≥ 1 the matrix R |n def = (R i,j ) 1≤i,j≤n is positive semi-definite with probability one, i.e. ∀x ∈ R n , x R |n x ≥ 0.

On the one hand, the Aldous-Hoover Theorem [START_REF] David | Representations for partially exchangeable arrays of random variables[END_REF][START_REF] Douglas | Relations on probability spaces and arrays of random variables[END_REF] states that an infinite weakly exchangeable array is equal in distribution to (f (w, u k , v k , x k,k )) k,k ≥1 , for some function f and w, (u k ), (v k ), (x k,k ) i.i.d.

∼ Unif([0, 1]).

On the other hand, if (R k,k ) k,k ≥1 is a deterministic positive semi-definite array, then there exists a separable Hilbert space H, with scalar product (•;

The Dovbysh-Sudakov Theorem [START_REF] Ln Dovbysh | On the gram-de finetti matrices[END_REF] combines somehow these two results:

Let R be an infinite symmetric random array R which is weakly exchangeable and positive semi-definite. Then there exists a separable Hilbert space H (whose scalar product will be denoted by (• ; •)) and a random probability distribution η on H × R ≥0 such that R is equal in distribution to

where, conditionaly on η, (h k , a k ) k≥1 is a sequence of i.i.d. random variables with distribution η. δ k,k denotes here the Kronecker delta.

We refer to [START_REF] Panchenko | On the dovbysh-sudakov representation result[END_REF] and [START_REF] Austin | Exchangeable random measures[END_REF] for a proof. From the Dovbysh-Sudakov Theorem, one deduces easily:

be a random symmetric array, which is weakly exchangeable and positive semi-definite. Assume that we have almost surely

Then we have almost surely

Proof. We apply the Dovbysh-Sudakov Theorem to R. We obtain the existence of a random

where conditionally on η, (h k , a k )

∼ η. Let us work conditionally on η and let ν denotes the first marginal of η. From our hypothesis we get that |(h 1 ; h 2 )| = q, ν-almost surely which implies that h 1 2 = q, ν-almost surely. Indeed, if ν( h 1 2 = q) > 0 then there exists h ∈ H such that h 2 = q such that for all > 0, ν(B (h) > 0. We can now take > 0 small enough such that for all a, b ∈ B (h) we have (a; b) = q. We obtain that ν |(h 1 ; h 2 )| = q ≥ ν(B (h)) 2 > 0 which leads to a contradiction.

Since

s. the measure ν has at most two points in its support.

We have now all the tools needed to prove Lemma 1.2.2:

Proof of Lemma 1.2.2. The sequence (x (1) ; x (2) )(x (2) ; x (3) )(x (3) ; x (1) ) n≥1 is tight because bounded. Let us consider a subsequence (n ) ≥1 along which is converges in law, to some random variable Q.

Let us sample x (1) , x (2) , . . . , x k , . . . i.i.d. from the posterior distribution of X given Y and consider the following random symmetric weakly exchangeable positive semi-definite array

The scalar products (x (k) ; x (k ) ) are bounded, so the sequence of the laws of R n is tight. We can extract from (n ) ≥1 another subsequence (n ) ≥1 along which R converges in law to an array R. R is also a random, symmetric, weakly exchangeable, semi-definite positive array. Since R n 1,1 = 1 and R n 1,2 = q almost surely, we have R 1,1 = 1 and R 1,2 = q almost surely. Proposition A.1 above

R implies in particular (x (1) ; x (2) )(x (2) ; x (3) )(x (3) ; x (1)

Hence the only accumulation point of (x (1) ; x (2) )(x (2) ; x (3) )(x (3) ; x (1) ) n≥1 is q 3 : we conclude that (x (1) ; x (2) )(x (2) ; x (3) )(x (3) ; x (1) )

Proof. Let x ∈ ∂f * (y). We get that (f * ) * (x) = xy -f * (y) and therefore that f * (y) = xy -f (x), by Proposition C.4. This gives that x maximizes s → sy -f (s) over R ≥0 and thus y ∈ ∂f (x). It remains to show that y ∈ ∂f (x) =⇒ x ∈ ∂f * (y). This follows from Proposition C.4 and the implication x ∈ ∂f * (y) =⇒ y ∈ ∂f (x) that we just showed.

C.3 Some supremum formulas

This section gathers some tools to deal with "sup-inf" expressions that we encounter in this manuscript.

Proposition C.6

Let f, g be two convex Lipschitz functions on R ≥0 . For (q 1 , q 2 ) ∈ R 2 ≥0 we define ϕ(q 1 , q 2 ) = f (q 1 ) + g(q 2 ) -q 1 q 2 and ψ(q 1 , q 2 ) = q 1 q 2 -f * (q 2 ) -g * (q 1 ). Then the set Γ = (q 1 , q 2 ) ∈ R 2 ≥0 q 1 ∈ ∂g(q 2 ), q 2 ∈ ∂f (q 1 ) is non-empty and:

ψ(q 1 , q 2 ) = sup

and the two first suprema above are achieved and precisely at the same couples (q 1 , q 2 ).

If moreover f and g are both differentiable and strictly convex, then the same result holds for Γ replaced by

Proof. Let L f (resp. L g ) be the Lipschitz constant of f (resp. g). For x > L f , f * (x) = +∞ and (since f * is lower semi-continuous by Proposition C.5) f * (x) → +∞ as x → L f . Analogously, g(x) → +∞ as x → L g . The function ψ is therefore continuous on [0, L g ) × [0, L f ) and goes to -∞ on the border

The functions ψ achieves therefore its maximum at some (q 1 , q 2 ) ∈ [0, L g ) × [0, L f ). (q 1 , q 2 ) verifies then q 2 ∈ ∂g * (q 1 ) and q 1 ∈ ∂f * (q 2 ) which gives (q 1 , q 2 ) ∈ Γ by Proposition C.5. The set Γ is therefore non-empty and sup q 1 ,q 2 ≥0 ψ(q 1 , q 2 ) ≤ sup (q 1 ,q 2 )∈Γ ϕ(q 1 , q 2 ).

By definition of the conjugates f * and g * we have for all q 1 , q 2 ≥ 0 f (q 1 ) + f * (q 2 ) ≥ q 1 q 2 g(q 2 ) + g * (q 1 ) ≥ q 1 q 2 . We get that ϕ(q 1 , q 2 ) ≥ ψ(q 1 , q 2 ) with equality if and only if (q 1 , q 2 ) ∈ Γ, by Proposition C.5. This gives in particular that sup q 1 ,q 2 ≥0 ψ(q 1 , q 2 ) ≥ sup (q 1 ,q 2 )∈Γ ϕ(q 1 , q 2 ).

Hence, both supremum are equal and are achieved over the same couples because we have seen that all couple (q 1 , q 2 ) that achieves the supremum of ψ is in Γ.

We consider now the second equality. Using the definition of the monotone conjugate (C.1) and Proposition C.4:

q 1 q 2 -f * (q 2 ) -g * (q 1 ) .

Proposition C.8

Let g be a strictly convex, differentiable, Lipschitz non-decreasing function on R + . Define ρ = sup x≥0 g (x). Let f be a convex, continuous, strictly increasing function on [0, ρ], differentiable on [0, ρ). For (q 1 , q 2 ) ∈ [0, ρ] × R + we define ϕ(q 1 , q 2 ) = f (q 1 ) + g(q 2 ) -q 1 q 2 . Then sup

ϕ(q 1 , q 2 ) = sup (q 1 ,q 2 )∈Γ ϕ(q 1 , q 2 ) , (C.5)

where

where all the function are extended by there limits at the points at which they may not be defined (for instance g (+∞) = lim q→∞ g (q), f (ρ) = lim q→ρ f (q)). Moreover, the above extremas are achieved precisely on the same couples.

Proof. Let q * 1 be a maximizer of f -g * over [0, ρ]. q * 1 is well defined because f is continuous and g * is continuous over [0, ρ) and is either continuous or goes to +∞ at ρ (because g * is a lower semi-continuous convex function, see Proposition C.4). We distinguish 3 cases: Case 1: 0 < q * 1 < ρ. By strict convexity of g, ϕ(q 1 , •) admits a unique minimizer q * 2 and (g * ) (q * 1 ) = q * 2 by Proposition C.5. Thus, the optimality condition at q * 1 gives

The optimality of q * 2 gives then q * 1 ≤ g (q * 2 ). Suppose that q * 1 < g (q *

2 ). This is only possible when q * 2 = 0. Define q 1 = g (q * 2 ) = g (0). Remark that g * (q 1 ) = -g(0) = g * (q * 1 ). We supposed that q 1 > q * 1 thus, by strict monotonicity of f , f (q 1 ) -g * (q 1 ) > f (q * 1 ) -g * (q * 1 ) which contradict the optimality of q * 1 . We obtain therefore that q * 1 = g (q * 2 ).

Case 2: q * 1 = 0. The optimality condition gives now

where q * 2 is again the unique minimizer of ϕ(q * 1 = 0, •) = f (0) + g. g is strictly increasing, so q * 2 = 0. Therefore q * 2 = 0 = f (q * 1 = 0), by (C.6). As before we have necessarily, by optimality of q * 2 that q * 1 = g (q * 2 ).

Case 3: q * 1 = ρ. In that case arg min q 2 ≥0 {g(q 2 ) -q * 1 q 2 } = ∅ because g is strictly convex and ρ-Lipschitz. Proposition C.5 gives then that ∂g * (ρ) = ∅ which implies (see [START_REF] Tyrell | Convex analysis[END_REF]Theorem 23.3]) that (g * ) (ρ -) = +∞. Since q * 1 = ρ maximizes f -g * , we necessarily have then f (ρ -) = +∞. Using the slight abuse of notation explained in the Proposition, we have f (q * 1 ) = +∞ = q * 2 , where q * 2 = +∞ is the unique "minimizer" of ϕ(q * 1 , •), by strict convexity of g. By definition of ρ we have also g (q *

2 ) = g (+∞) = ρ = q * 1 .

We conclude from the tree cases above that the "sup-inf" in (C.5) is achieved, and that all the couples (q * 1 , q * 2 ) that achieve this "sup-inf" belong to Γ. Thus sup

ϕ(q 1 , q 2 ) ≤ sup (q 1 ,q 2 )∈Γ ϕ(q 1 , q 2 ) .

Let now be (q 1 , q 2 ) ∈ Γ. By convexity of g we see easily that ϕ(q 1 , q 2 ) = inf q 2 ϕ(q 1 , q 2 ). Thus, ϕ(q 1 , q 2 ) ≤ sup q 1 inf q 2 ϕ(q 1 , q 2 ). Therefore sup (q 1 ,q 2 )∈Γ ϕ(q 1 , q 2 ) ≤ sup

R ÉSUM É

Nous étudions des probl èmes statistiques classiques, tels que la d étection de communaut és dans un graphe, l'analyse en composantes principales, les mod èles de m élanges Gaussiens, les mod èles lin éaires (g én éralis és ou non), dans un cadre Bay ésien. Nous calculons pour ces probl èmes le "risque de Bayes" qui est la plus petite erreur atteignable par une m éthode statistique, dans la limite de grande dimension. Nous observons alors un ph énom ène surprenant: dans de nombreux cas il existe une valeur critique de l'intensit é du bruit au-del à de laquelle il n'est plus possible d'extraire de l'information des donn ées. En dessous de ce seuil, nous comparons la performance d'algorithmes polynomiaux à celle optimale. Dans de nombreuses situations nous observons un écart: bien qu'il soit possible -en th éorie -d'estimer le signal, aucune m éthode algorithmiquement efficace ne parvient à être optimale. Dans ce manuscrit, nous adoptons une approche issue de la physique statistique qui explique ces ph énom ènes en termes de transitions de phase. Les m éthodes et outils que nous utilisons proviennent donc de la physique, plus pr écis ément de l' étude math ématique des verres de spins.

MOTS CL ÉS

inf érence statistique, th éorie de l'information, physique statistique, verres de spin, d étection de communaut é dans des graphs, estimation de structure de faible rang, mod èle lin éaire g én éralis é, Lasso ABSTRACT We study classical statistical problems such as as community detection on graphs, Principal Component Analysis (PCA), sparse PCA, Gaussian mixture clustering, linear and generalized linear models, in a Bayesian framework. We compute the best estimation performance (often denoted as "Bayes Risk") achievable by any statistical method in the high dimensional regime. This allows to observe surprising phenomena: for many problems, there exists a critical noise level above which it is impossible to estimate better than random guessing. Below this threshold, we compare the performance of existing polynomial-time algorithms to the optimal one and observe a gap in many situations: even if non-trivial estimation is theoretically possible, computationally efficient methods do not manage to achieve optimality. From a statistical physics point of view that we adopt throughout this manuscript, these phenomena can be explained by phase transitions. The tools and methods of this thesis are therefore mainly issued from statistical physics, more precisely from the mathematical study of spin glasses. KEYWORDS statistical inference, information theory, statistical physics, spin glasses, community detection on graphs, lowrank estimation, generalized linear models, Lasso