Keywords:

applications à la M-estimation . . . . . . . . .

List of Figures

Gaussian mixture with the number of iterations in the HTGD (solid red), minibatch SGD (dashed green) and GD (dotted blue) algorithms . . . . . . . . . . 62 4.4 Evolution of the estimator of the location parameter θ = 0 of the balanced Gaussian mixture with the number of iterations in the HTGD (solid blue) and mini-batch SGD (dashed red) algorithms over 50 populations . . . . . . . . 62 5.1 Average over 50 runs of the risk estimate with the number of iterations (solid lines) +/-their standard deviation (dashed lines) . . . . . . . . . . . . . . . . 78 5.2 Average over 50 runs of the error test with the number of iterations (solid lines) +/-their standard deviation (dashed lines) . . . . . . . . . . . . . . . . . . . 79

6.1 Illustrative experiment with n = 50, q = 2, τ = 0.27 and p = 0. Figure 6.1(a) shows the training graph, where the position of each node is given by its 2D feature vector. Figure 6.1(b) depicts the same graph after applying a random transformation R to the features. On this graph, the Euclidean distance with optimal threshold achieves a reconstruction error of 0. In this manuscript, we present and study sampling strategies for statistical learning related problems. The goal is to deal with issues typically arising in a big data context when the number of observations and their dimensionality impose limitations on the learning process. We thus propose to tackle this issue by employing two sampling strategies:

• Speed-up the learning process by sampling the most useful observations.

• Scale-up the problem by discarding some observations to reduce the complexity and the size of the problem.

To introduce the problem we deal with, we first give a quick reminder on Empirical Risk Minimization (ERM) in the context of binary classification. The binary classification problem is considered a running example all along this manuscript. Because it can be easily formulated, it is undeniably the most documented statistical learning problem in the literature and many of its results extend to more general frameworks (e.g., multi-class classification, regression, ranking). Let (Ω, A, P) be a probability space and (X, Y ) a random pair defined on (Ω, A, P), taking its values in some measurable product space X × {-1, +1}, with common distribution P (dx, dy): the r.v. X represents some observation, hopefully useful for predicting the binary label Y . The distribution P can also be described by the pair (F, η) where F (dx) denotes the marginal distribution of the input variable X and η(x) = P{Y = +1 | X = x}, x ∈ X , is the posterior distribution. The objective is to build, based on the training dataset at disposal, a measurable mapping g : X → {-1, +1}, called a classifier, with minimum risk:

L(g) def = P{g(X) = Y }. (1.1)
It is well-known that the Bayes classifier g * (x) = 2I{η(x) ≥ 1/2} -1 is a solution of the risk minimization problem inf g L(g), where the infinimum is taken over the collection of all classifiers defined on the input space X . The minimum risk is denoted by L * = L(g * ).

Since the distribution P of the data is unknown, one substitutes the true risk with its empirical estimate

L n (g) = 1 n n i=1 I{g(X i ) = Y i }, (1.2)
based on a sample (X 1 , Y 1 ), . . . , (X n , Y n ) of independent copies of the generic random pair (X, Y ). The true risk minimization is then replaced by the empirical risk minimization where the minimum is taken over a class G of classifier candidates, supposed rich enough to include the naive Bayes classifier (or a reasonable approximation of the latter). Considering a solution g n of (1.3), a major problem in statistical learning theory is to establish upper confidence bounds on the excess of risk L( g n ) -L * in absence of any distributional assumptions and taking into account the complexity of the class G (e.g., described by geometric or combinatorial features such as the VC dimension) and some measure of accuracy of approximation of P by its empirical counterpart P n = (1/n) n i=1 δ (X i ,Y i ) over the class G. Indeed, the excess of risk of the empirical risk minimizers is typically bounded as follows

L( g n ) -L * ≤ 2 sup g∈G | L n (g) -L(g)| + inf g∈G L(g) -L * .
(1.4)

The second term on the right hand side is referred to as the bias and depends on the richness of the class G, while the first term, called the stochastic error, is controlled by means of results in empirical process theory, see Boucheron et al. (2005a). Unfortunately, one of the thing typically overlooked in this kind of analysis is how to efficiently solve the ERM problem, i.e how to find g n . It is usually approximated by some incremental optimization algorithm, iteratively computing estimator of the gradient of the empirical risk. We investigate efficient ways to scale-up the learning process and introduce sampling-based approachs to build approximations of g n . We do so in two different fashions:

• We replace the empirical risk L n (g) by an approximation based on fewer terms Ln (g).

It naturally makes the learning problem easier. Let gn be a minimizer of Ln (g), then (1.4) becomes:

L(g n ) -L * ≤ 2 sup g∈G | Ln (g) -L n (g)| + 2 sup g∈G | L n (g) -L(g)| + inf g∈G L(g) -L * .
We focus on appropriately controlling 2 sup g∈G | Ln (g) -L n (g)| which we typically do conditionally upon the observations. Such strategy is discussed and implemented in chapters 2 and 6 for two different problems.

• When computing estimator of the gradient, most incremental algorithms uniformly and independently sample observations within the dataset. We propose to use non uniform sampling methods to compute an estimator of the gradient of L n with smaller variance.

For the algorithms we propose, if we denote by g n (T ) the classifier obtained after T iterations of the optimization algorithm, then following the lines of [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF], inequality (1.4) becomes:

L(g n (T )) -L * ≤ L n (g n (T )) -L n ( g n ) (1) + 2 sup g∈G | L n (g) -L(g)| (2) + inf g∈G L(g) -L * .
where (1) corresponds to the optimization error and (2) corresponds to the stochastic error. This decomposition illustrates the well known fact (see [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF]) that when solving the Empirical Risk Minimization problem, we have to take into account the randomness induced by the observations so that the optimization accuracy is of the same order. We particularly pay attention to this fact and illustrate it both theoretically and empirically in chapter 3, 4 and 5.

The rest of this chapter is devoted to overview our different contributions. Here and throughout, the indicator function of any event E is denoted by I{E} and the variance of any square integrable r.v. Z by σ 2 (Z).

Learning from Survey Training Samples

This subsection is a summary of chapter 2. We place ourselves in the context of binary classification, when the observations used to train a classifier are drawn by means of a sampling/survey scheme and exhibit a complex dependence structure. We consider, (X 1 , Y 1 ), . . . , (X n , Y n ) a sample of independent copies of (X, Y ) observed on a finite population on I n := {1, . . . , n}. We call a survey sample of (possibly random) size N ≤ n of the population I n , any subset s := {i 1 , . . . , i n(s) } ∈ P(I n ) with cardinality N =: N (s) less than n. A sampling scheme is defined by a probability distribution R n on the set of all possible samples s ∈ P(I n ) conditionally on the observations D n = {(X i , Y i ) : i ∈ I n }. The probability that the unit i belongs to a random sample S drawn from the conditional distribution R n is called first order inclusion probability and is denoted by π i = P Rn {i ∈ S}.

We set π n = (π 1 , . . . , π n ). Given an observed sample S, it is fully determined by the r.v.

n = ( 1 , . . . , n ), where i = I{i ∈ S} for 1 ≤ i ≤ n.

Most available results (see Boucheron et al. (2005c) for instance) deal with the case where the dataset D n is at disposal. However this is not the case here as we only observe a subset of observations. Therefore, these results are not directly applicable to our problem, in particular because of the dependence structure induced by the sampling scheme. Nevertheless, we show that the theory of ERM can be extended to the case where statistical learning is based on observations obtained via survey samples.

Horvitz-Thompson risk. As defined in [START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF], for any classifier candidate g, the (not available) empirical risk L n (g) = n -1 1≤i≤n I{Y i = g(X i )} is replaced by its Horvitz-Thompson version :

L n (g) = 1 n n i=1 i π i I {g(X i ) = Y i } , (1.5)
where n = ( 1 , . . . , n ) denotes the vector in correspondence with the sample drawn.

While many sampling plans are of interest for our problem, we particularly pay attention to Rejective sampling, a sampling design R n of fixed size N ≤ n, that generalizes the simple random sampling without replacement (where all samples with cardinality N are equally likely to be chosen). This sampling plan is more difficult to analyse because the i are dependent r.v.. Therefore, when statistical learning is based on the observation of a sample drawn by means of a rejective scheme, classical results from learning theory do not immediately apply. Nevertheless, we show that similar results can be established for minimizer of (1.5) in the rejective case. it is due to the fact that this scheme form a collection of negatively associated (see [START_REF] Brändén | Negative dependence in sampling[END_REF], [START_REF] Kramer | Negative dependence and srinivasan's sampling process[END_REF] ) random variables, a rather tractable type of dependence structure. Using the negative association property, we show that for a given rejective sampling scheme n with first order inclusion probabilities π n and with κ n = N/(n × min i≤n π i ) we have for any solution ḡn of the minimization problem inf g∈G L n (g), an upper-bound on the stochastic error risk of order O P ((κ n (log n)/N ) 1/2 ).

Chapter 1. Introduction

The property of negative association being shared by many other sampling schemes, the same argument can be thus naturally applied to carry out a similar rate analysis for training data produced by such plans. However, this analysis cannot be extended to all sampling scheme. We circumvent this difficulty using the results established for rejective plan and relying on coupling argument. Consider a complex sampling scheme R * n with first order inclusion probabilities π * n = (π * 1 , . . . , π * n ) described by the vector * n = ( * 1 , . . . , * n ) (with not necessarily negatively associated r.v). Let ḡ * n be a minimizer of the HT empirical risk L * n (g) = (1/n) n i=1 ( * i /π * i )I{Y i = g(X i )} over a class G. Since we already established results in the rejective case, we introduce a rejective sampling scheme R n described by the r.v. n , with first order inclusion probabilities π n = (π 1 , . . . , π n ) as well as the following quantity:

Ľ n (g) = 1 n n i=1 i π * i I{Y i = g(X i )}, (1.6)
for any classifier g. Observe that (1.6) is different from the HT empirical risk L n (g) related to the rejective sampling scheme n because it involves the π * i 's instead of the π i 's. The excess of risk of the HT empirical risk minimizer can be expressed as follows:

L(ḡ * n ) -inf g∈G L(g) ≤ 2 sup g∈G L(g) -L n (g) + 2 sup g∈G L n (g) -L n (g) + 2 sup g∈G L n (g) -Ľ n (g) + 2 sup g∈G Ľ n (g) -L * n (g) . (1.7)
We controlled the first term on the right hand side of (1.7) by using Vapnik-Chervonenkis and McDiarmid inequalities (see e.g. [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF] and chapter A), assertion (i) of Proposition 2.4 established in the rejective case provides a control of the second term. The third term is bounded by means of the coupling argument while the last term is controlled by assumptions related to the closeness between the first order inclusion probabilities π * n and π n . More precisely, the assumptions required in the subsequent analysis involve the total variation distance between the sampling plans R n and R * n :

d T V (R n , R * n ) def = 1 2 s∈P(In) |R n (s) -R * n (s)|.
With κ * n = (N/n) min i≤n π * i and κ n = (N/n×min i≤n π i ), we show that L(ḡ * n )-inf g∈G L(g) is of the order of O P ((κ n (log n)/N ) 1/2 ) + 2(κ

* n + κ n )(n/N ) inf Rn d T V (R n , R * n )
, where the infinimum is taken over the set of rejective schemes R n with first order inclusion probabilities π n = (π 1 , . . . , π n ).

The rate bound obtained depends on the minimum error made when approximating the sampling plan by a rejective sampling plan in terms of total variation distance. It is of the same order as in case where observations are sampled uniformly up to a multiplicative term and show that learning with survey sample is possible when taking into account the first order inclusion probabilities.

Sampling Strategies for Stochastic Gradient Descent (SGD)

We present in this section a summary of the results establish in chapter 3, 4 and 5, in which we introduce the problem of non uniform sampling strategy for stochastic gradient descent (SGD in short). The Empirical Risk Minimization problem previously introduced is of the utmost importance and implementing efficient algorithms to solve this problem is a question that we tried to answer. Here we consider a more general framework than the binary classification one, and consider optimization problems of the form :

min θ∈Θ L n (θ) = min θ∈Θ 1 n n i=1 l(Z i , θ), (1.8)
where Θ is a Euclidean space, typically R d with d ≥ 1, and l(Z 1 , .), . . . , l(Z n , .) form a collection of real-valued convex continuously differentiable functions on Θ. Indeed, such an optimization problem typically arises in a broad variety of statistical learning problems, in particular supervised tasks, where the goal pursued is to learn a predictive model, fully determined by a parameter θ. The performance of the predictive function defined by θ, is typically measured by the expectation L(θ) = E[ ((X, Y ), θ)], referred to as the risk, where is a loss function assumed convex w.r.t. θ. It is usually assessed via its empirical counterpart

L n (θ) = 1 n n i=1 ((X i , Y i ), θ),
(1.9) based on n ≥ 1 supposedly available independent training examples (X 1 , Y 1 ), . . ., (X n , Y n ), copies of the random pair Z = (X, Y ). The minimization problem (1.8) can be solved incrementally, by means of variants of the stochastic approximation method originally introduced in the seminal contribution of [START_REF] Robbins | A stochastic approximation method[END_REF]. This one consists in computing successive estimates of a minimizer of (1.2) using the recursive equation

θ t+1 = θ t -γ t r t (θ t ) (1.10)
from a preliminarily picked initial value θ 0 ∈ Θ, where r t denotes an estimator of the gradient ∇ L n and γ t is the learning rate or step-size. The implementation of SGD is quite straightforward for the minimization of standard mean statistics, as it is usually performed by sampling uniformly at random (with or without replacement) a subsample of observations before computing an estimator of the gradient. In contrast to the batch approach, where all the data are used to estimate the gradient at each iteration (i.e. r t (θ) = ∇ L n (θ) for all t ≥ 0 and θ ∈ Θ), only subsets of the data sample are involved in the gradient estimation steps of sampled incremental algorithms, with the aim to reduce computational cost when n is large. In the most commonly used implementation of the stochastic gradient descent (SGD) algorithm, the gradient estimator is computed from a subset of reduced size S ≤ n uniformly drawn without replacement among all possible subsets of the dataset of size S at each step t ≥ 0. In practice, the main limitation of this incremental optimization technique is due to the stochastic noise induced by the choice at random of the data involved in the gradient estimator computation at each iteration. In particular, most of the theoretical justification of the SGD are established in a very general framework (see [START_REF] Robbins | A stochastic approximation method[END_REF] or Bach & Moulines (2011a) for instance) that encompasses the ERM case. We propose to introduce non uniform sampling strategy as well as a novel analysis highlighting the benefit of using non uniform sampling strategy for the ERM problem. We first introduce in chapter 3 a novel implementation of the SGD algorithm, where the data subset used at a given step is not picked uniformly at random among all possible subsets but drawn from a specific adaptive sampling scheme, depending on the past iterations. We then propose a general framework to extend these results using survey sampling theory in chapter 4 in which we also take into account the distribution of the observation in our final analysis. We conclude this section by considering the specific case where the empirical risk takes the form of a U -statistic and propose and efficient implementation of the SGD Algorithm in this case. Here and throughout, the gradient and Hessian operators with respect to θ are denoted by ∇ and ∇ 2 respectively. By convention, ∇ 0 corresponds to the identity operator and gradient values are represented as column vectors. For any vector V ∈ R d ,

Chapter 1. Introduction we denote by ||V || its euclidian norm and for any matrix A we denote by A T its transpose.

A Non Uniform Sampling Strategy for SGD

In order to speed up the learning process, we introduce a specific variant of the SGD algorithm with an adaptive sampling scheme, in the sense that it may vary with t, depending on the past iterations. We consider non uniform sampling with replacement. We first start by identifying a good sampling distribution by choosing the one minimizing the variance of the estimator. When drawing a sample S of size S with first order inclusion p i independently with replacement, the quantity

1 S i∈S ∇l(Z i , θ) p i (1.11)
is an unbiased estimator of ∇ L n (θ) with corresponding variance equal to :

1 S n i=1 ∇l(Z i θ) 2 p i - ||∇ L n (θ)|| 2 S .
(1.12)

To achieve the best estimation of the gradient (i.e minimizing the variance) at parameter θ conditionally upon the observations, it is therefore natural to sample observation Z i with probability: p * i (θ) = ∇l(Z i , θ) / n j=1 ∇l(Z j , θ) . Unfortunately, practical implementation of the above sampling scheme is not pertinent because it requires to evaluate all gradients to calculate the norms ∇l(Z 1 , θ t ) , . . . , ∇l(Z n , θ t ) at each iteration, which is precisely what we are try to avoid when using SGD. We therefore propose a sampling scheme approximating p * t := (p * i (θ t )) n i=1 without requiring any additional gradient evaluations. We use some old values of the gradient in our approximation. More precisely, the main idea is to replace each unknown gradient norm ∇l(Z i , θ t ) by a (possibly outdated) norm g t,i = ∇l(Z i , θ k ) at some former instant k = k(i, t) corresponding to the last time k ≤ t when the Z i was picked. More formally, we define the random sequence g t as g t+1,i = ∇l(Z i , θ t+1 ) if i ∈ {i

(1) t+1 , . . . , i

(S) t+1 } g t,i
otherwise.

(1.13)

Then, a natural way to approximate p * t is to construct pt = (p t,i ) n i=1 where we set for each i pt,i = g t,i n j=1 g t,j

.

(1.14)

It turns out that convergence cannot be guaranteed with this choice, because a certain component pt,i can get arbitrarily close to zero. A possible remedy is to enforce a greedy sampling scheme: ∀i ∈ {1, . . . , n}, p t,i = ρν i + (1 -ρ)p t,i , (1.15)

where ν = (ν 1 , . . . , ν n ) is a probability distribution with ν i > 0 for 1 ≤ i ≤ n, and 0 < ρ ≤ 1. This condition has the following interpretation : p t is a mixture between two laws of probability and one of this law (ν) is independent from the past. Now that we have defined our sampling strategy, the algorithm we propose is simply to compute at each iteration t an estimator of the gradient based on equation (1.11) by sampling observation according to p t := (p t,i ) n i=1 . This sampling strategy can also be efficiently implemented in practice and we show that sampling under this strategy only has an additional cost of O(log(n)). Theoretical results are then established by means of asymptotic argument where we show that with this sampling strategy, the asymptotic behaviour of θ t is optimal up to an error proportional to ρ. Note that all the result obtained in chapter 3 hold true conditionally upon the observations and therefore do no take into account the statistical nature of our problem (i.e we solve ERM because we do not know the true risk). We deal with this issue in the next section where we discuss of a similar problem (Non uniform sampling strategy for SGD) in the context of M-estimation. More specifically we use the framework of survey sampling introduced previously to extend our results . 1.3.2 Horvitz Thompson Gradient Descent (HTSGD) and Applications to M-Estimation

The previous sections strongly suggests to use sampling technique to scale up learning from datasets. We now show how to incorporate efficiently survey schemes into such iterative techniques for M-estimation. More precisely, we propose a specific estimator of the gradient, that is referred to as the Horvitz-Thompson gradient estimator (HTGD estimator in short). For the estimator thus produced, consistency and asymptotic normality results describing its statistical performance are established. The framework we consider is the same than in section 1.3.1. We define the Horvitz-Thompson estimator of the gradient n (θ) based on a survey sample S drawn from a design S n with (first order) inclusion probabilities {π i } 1≤i≤n and inclusion vector n = ( 1 , . . . , n ) as

Sn (θ) = 1 n i∈S 1 π i ∇l(Z i , θ) = 1 n n i=1 i π i ∇l(Z i , θ).
(1.16)

Equipped with this notation, we study the property of a SGD algorithm when the estimator of the gradient is computed by sampling observations within some dataset under some sampling plan S n (possibly depending upon t and the current value of the parameter). We denote by θ n (T ) the value of the parameter at time T . Conditioned upon the data D n = {Z 1 , . . . , Z n }, we study the asymptotic properties of the M -estimator produced by the HTGD algorithm.

The limit results stated below essentially rely on the fact that the HT estimator (1.16) of the gradient of the empirical risk is unbiased. Reduction of the asymptotic variance of θ n (T ) (of L n ( θ n (T )), respectively) is investigated later in the Poisson case (i.e when the i are independent r.v.). We show that under some appropriate assumptions, the sequence { θ n (t)} t≥0 converges to a minimizer of the empirical risk and

1/γ t θ n (t) -θ * n ⇒ N (0, Σ π,n ),
where the asymptotic covariance matrix Σ π,n , is the unique solution of a Lyapunov equation involving the π i 's and defined in chapter 4. By a direct application of the second order Delta method, we then characterize the behaviour of

L n ( θ n (t)) -L n (θ * n )
where θ * n is an empirical risk minimizer. We then discuss how one should choose π i in the Poisson case (case where the i are independent) and recall the result of chapter 3 by showing that sampling with π i proportional to ∇l(Z i , θ) yields optimal results when trying to minimize the variance of

L n ( θ n (t)) -L n (θ * n ).
Denoting by N the expected size of a sample, our analysis is then finally completed by studying the behaviour of θ n (t) as n, N tend to +∞ at appropriate rates. This is quite different from what we did in chapter 3 because we limited our analysis to the case where n is set in advance and all our results were obtained conditionally upon the observations. Doing so allow us to illustrate the well-known trade-off between (asymptotic) generalization and optimization errors, ruled by the limit behaviour of nγ t /N (see [START_REF] Bottou | The tradeoffs of large scale learning[END_REF] for instance). More precisely we show that under supplementary assumptions, if lim nγ t /N = Chapter 1. Introduction c > 0 , then we have the convergence in distribution:

lim n, N →∞ lim t→∞ √ n θ n (t) -θ * = N (0, Λ * + cΣ * ).
where lim n, N →∞ N Γ * n = Γ * and Λ * is the asymptotic covariance matrix involved in the TCL for M -estimator applied to θ * n -θ * . With γ t typically or order O(1/t), the condition lim nγ t /N = c > 0 gives an idea of how the number of iteration should be tuned according to the number of observation and the batch size to yield optimal results. Numerical experiments are then displayed in section 4.6.

Stochastic Gradient Descent Algorithms based on Incomplete U-Statistic

Here we discuss of the implementation of SGD for U-Statistics. We briefly introduce the problem and notations and explain the difference with the problems of section 1.3.1 and 1.3.2.

Generalized U -statistics are extensions of standard sample mean statistics. In machine learning, they are used as performance criteria in many problems, Metric Learning and AUC in particular are two examples that we consider in our experiments. It is defined as follows:

Definition 1.1. Let K ≥ 1 and (d 1 , . . . , d K ) ∈ N * K . Let X {1, ..., n k } = (X (k) 1 , . . . , X (k) n k ), 1 ≤ k ≤ K, be K independent samples of sizes n k ≥ d k and composed of i.i.d. random vari- ables taking their values in some measurable space X k with distribution F k (dx) respectively. Let H : X d 1 1 × • • • × X d K K → R be a measurable function, square integrable with respect to the probability distribution µ = F ⊗d 1 1 ⊗ • • • ⊗ F ⊗d K K . Assume in addition (without loss of generality) that H(x (1) , . . . , x (K) ) is symmetric within each block of arguments x (k) (valued in X d k k ), 1 ≤ k ≤ K.
The generalized (or K-sample) U -statistic of degrees (d 1 , . . . , d K ) with kernel H, is then defined as

U n (H) = 1 K k=1 n k d k I 1 . . . I K H X (1)
I 1 ; X

(2)

I 2 ; . . . ; X (K) I K , (1.17)
where n = (n 1 , . . . , n K ), the symbol

I 1 • • • I K refers to summation over all elements of Λ, the set of the K k=1 n k d k index vectors (I 1 , . . . , I K ), I k being a set of d k indexes 1 ≤ i 1 < . . . < i d k ≤ n k and X (k) I k = (X (k) i 1 , . . . , X (k) i d k ) for 1 ≤ k ≤ K.
Subsection 1.3.1 and 1.3.2 advocates the use of SGD to deal with the number of terms involved in (1.8). Note that when the empirical risk takes the form of a generalized U -statistics, the number of terms involved in the sum is of order O(n d 1 +...d K ) making the problem extremely difficult to solve. Nevertheless we show how to implement the SGD in this case. We place ourselves in the parametrized setting. Still denoting by Θ the parameter space, with H : K k=1 X d k k × Θ → R a convex loss function, we denote the empirical version of the risk by θ ∈ Θ → L n (θ) = U n (H(.; θ)). As we have mentioned before, the implementation of SGD is quite straightforward for the minimization of standard mean statistics, as it is usually performed by sampling uniformly at random (with or without replacement) a batch of observations before computing an estimator of the gradient. When the empirical risk takes the form of a U -statistic, the SGD algorithm could be implemented this way. It would lead to estimator of the gradient equal to:

gn (θ) = 1 K k=1 n k d k I 1 . . . I K ∇H(X (1)
I 1 ; X

(2) I 2 ; . . . ; X (K) 1.18) where I k refers to summation over all n k d k subsets X

I K ; θ), (
(k)

I k = (X (k) i 1 , . . . , X (k) i d k ) related to a set I k of d k indexes 1 ≤ i 1 < . . . < i d k ≤ n k and n = (n 1 , . . . , n K ).
In the case of U -Statistic, we prove that this strategy (which we later refer to as "computing a complete Ustatistic") is not efficient. We instead propose to proceed differently by drawing independently with replacement among the set of index vectors Λ, giving a gradient estimator in the form of a so-called incomplete U -statistic (see Lee (1990a)):

ḡB (θ) = 1 B (I 1 , ..., I K )∈D B ∇H(X (1)
I 1 , . . . , X (K)

I K ; θ), (1.19)
where D B is built by sampling B times with replacement in the set Λ. The parameter B is the number of terms averaged. For the same computational cost (i.e., taking

B = K k=1 n k d k
) and implementing SGD with (1.19) rather than (1.18) yields more accurate results, essentially because (1.19) has smaller variance w.r.t. to ∇L(θ) (except in the case where K = 1 = d 1 ). Intuitively, sampling an incomplete U -statistic is better because the number of observations involved is greater than the number of observation involved in the complete estimator. This is highlighted when we compare the performance of the SGD methods described above conditionally upon the observed data samples by studying both the asymptotic and non asymptotic behaviour of the SGD algorithm for both of the implementations. As we have done earlier in this chapter, we propose generalization bounds where (see [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF]), we decompose the stochastic error as follows:

E[L(θ t ) -L(θ * )] ≤ 2E sup θ∈Θ | L n (θ) -L(θ)| E 1 + E L n (θ t ) -L n (θ * n ) E 2
.

(1.20)

where θ * = argmin θ∈Θ L(θ). The generalization bound show the advantage of using an incomplete U -statistic (1.19) as the gradient estimator while highlighting the well-known fact that when using some optimization method to solve the Empirical Risk Minimization problem, we have to take into account the generalization bounds so that the optimization accuracy is of the same order (see subsection 1.3.2). Numerical experiments on Metric Learning problem and AUC optimization are then displayed.

Fast Learning Rates for Graph Reconstruction

This section is a summary of chapter 6 in which we present a brief overview of the graph reconstruction problem. We first set our context before describing the problem of interest. Let G = (V, E) be an undirected random graph with a set V = {1, . . . , n} of n ≥ 2 vertices and a set E = {e i,j : 1 ≤ i = j ≤ n} ∈ {0, 1} n(n-1) describing its edges: for all i = j, we have e i,j = e j,i = +1 if the vertices i and j are connected by an edge and e i,j = e j,i = 0 otherwise. We also assume that for all i ∈ V , a continuous r.v. X i is associated to vertex i. The X i 's are i.i.d. and for any i = j, the random pair (X i , X j ) gives some information useful to predict the occurrence of an edge connecting the vertices i and j. Conditioned upon the features (X 1 , . . . , X n ), any binary variables e i,j and e k,l are independent only if {i, j} ∩ {k, l} = ∅.

In particular, the conditional distribution of e i,j , i = j, is supposed to depend on (X i , X j ) solely and is described by: η (X i , X j ) = P {e i,j = +1 | (X i , X j )} .

(1.21)

Chapter 1. Introduction

The learning problem introduced by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF], referred to as graph reconstruction, consists in building a symmetric reconstruction rule g : X 2 → {0, 1}, from a training graph G, with nearly minimum reconstruction risk

R(g) = P {g(X 1 , X 2 ) = e 1,2 } , (1.22)
thus achieving a comparable performance to that of the Bayes rule g

* (x 1 , x 2 ) = I{η(x 1 , x 2 ) > 1/2}, whose risk is given by R * = E[min{η(X 1 , X 2 ), 1 -η(X 1 , X 2 )}] = inf g R(g).
The reconstruction risk (1.22) is replaced by its empirical version based on the labelled sample

D n = {(X i , X j , e i,j ) : 1 ≤ i < j ≤ n} related to G: R n (g) = 2 n(n -1) 1≤i<j≤n I {g(X i , X j ) = e i,j } .
(1.23)

Let g n be an empirical risk minimizer : min g∈G R n (g), where G is a class of reconstruction rules. As we have done before, the performance of g n is then measured by R( g n ) -inf g∈G R(g), which can be bounded if we can derive probability inequalities for the maximal deviation

sup g∈G | R n (g) -R(g)|.
(1.24)

Biau & Bleakley ( 2006) establish rate bounds of the order O P (1/ √ n) for the reconstruction risk of g n under appropriate complexity assumptions (namely, G is of finite VC-dimension). We prove that rates of order O P (log n/n) are always attained by the minimizers of the empirical reconstruction risk (1.23) with no additional assumptions. To establish this result, we rely on some arguments used in the fast rate analysis for empirical minimization of U -statistics (Clémençon et al., 2008a), although these results only hold true under restrictive distributional assumptions. Whereas the quantity (1.23) is not a U -statistic, we rewrite the difference between the excess of reconstruction risk of any candidate rule g ∈ G and its empirical counterpart as the sum of its conditional expectation given the X i 's, which is a U -statistic, plus a residual term. Denoting by Λ(g) = R(g) -R * the excess reconstruction risk with respect to the Bayes rule, its empirical estimator is given by

Λ n (g) = R n (g) -R n (g * ).
For all g ∈ G, one may write:

Λ n (g) -Λ(g) = U n (g) + W n (g), (1.25) where U n (g) = E [Λ n (g) -Λ(g) | X 1 , . . . , X n ]
is a one sample U -statistic of degree 2.

Under a certain "low-noise" condition, the analysis carried out by Clémençon et al. (2008a) shows that the small variance property of U -statistics lead to fast learning rates for empirical risk minimizers. We show that this condition is always fulfilled for the specific U -statistic U n (g) involved in the decomposition (1.25). This result is due to the fact that the empirical reconstruction risk is not an average over all pairs (i.e., a U-statistic of order 2) but an average over randomly selected pairs (random selection being ruled by the function η). We then conclude the proof of the results by establishing that the remaining term W n (g) is also of order O P (1/n).

We finally conclude our analysis by scaling up the learning process. Similarly to chapter 5, for large training graphs, the complexity of merely computing R n (g) is prohibitive as the number of terms involved in the summation is O(n 2 ). Just like we did in section B.3.3, we introduce a sampling-based approach to build approximations of the reconstruction risk with much fewer terms than O(n 2 ). Instead of the empirical reconstruction risk (1.23), we consider an incomplete approximation obtained by sampling pairs of vertices (and not vertices) with replacement. A parallel can easily be drawn with the results obtained in chapter 5 where we recommended to implement the SGD algorithm with incomplete U -Statistic, which corresponds in this case to sampling edges instead of nodes. Formally, we define the incomplete graph reconstruction risk based on B ≥ 1 pairs of vertices as

R B (g) = 1 B (i,j)∈P B I {g(X i , X j ) = e i,j } , (1.26)
where P B is a set of cardinality B built by sampling with replacement in the set

Θ n = {(i, j) : 1 ≤ i < j ≤ n} .
For any b ∈ {1, . . . , B} and all (i, j) ∈ Θ n , denote by b (i, j) the variable indicating whether the pair (i, j) has been picked at the b-th draw. The incomplete risk is then represented by:

R B (g) = 1 B B b=1 (i,j)∈Θn b (i, j) • I {g(X i , X j ) = e i,j } . (1.27) (B.38
) is an unbiased estimator of the true risk (1.22) and given the X i 's, its conditional expectation is equal to (4.2). When taking B = o(n 2 ), computational costs are significantly reduced, at the price of a slightly increased variance. We characterize the performance of solutions g B to the computationally simpler problem min g∈G R B (g) and show that with only B = O(n) pairs, the rate is of the same order (up to a log factor) as the one obtained by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] for the maximal deviation sup g∈G | R n (g) -R(g)| related to the complete reconstruction risk R n (g) with O(n 2 ) pairs. As expected, we show that the number B ≥ 1 of pairs of vertices plays the role of a tuning parameter, ruling a trade-off between statistical accuracy (taking B(n) = O(n 2 ) fully preserves the convergence rate) and computational complexity. Numerical experiments are displayed in section 6.5 to illustrate our different results.

PART I

Learning from Survey Training Samples

CHAPTER 2

Learning from Survey Training Samples

Abstract The generalization ability of minimizers of the empirical risk in the context of binary classification has been investigated under a wide variety of complexity assumptions for the collection of classifiers over which optimization is performed. In contrast, the vast majority of the works dedicated to this issue stipulate that the training dataset used to compute the empirical risk functional is composed of i.i.d. observations and involve sharp control of uniform deviation of i.i.d. averages from their expectation. Beyond the cases where training data are drawn uniformly without replacement among a large i.i.d. sample or modelled as a realization of a weakly dependent sequence of r.v.'s, statistical guarantees when the data used to train a classifier are drawn by means of a more general sampling/survey scheme and exhibit a complex dependence structure have not been documented in the literature yet. In this chapter, we show that the theory of empirical risk minimization can be extended to situations where statistical learning is based on survey samples and knowledge of the related inclusion probabilities. We prove that minimizing a weighted version of the empirical risk, referred to as the Horvitz-Thompson risk (HT risk), over a class of controlled complexity lead to a rate for the excess risk of the order O((κ n (log n)/N ) 1/2 ) with κ n = (N/n)/ min i≤n π i , when data are sampled by means of a rejective scheme of (deterministic) size N within a statistical population of cardinality N ≤ n with probability weights π i > 0. Extension to other sampling schemes are then established by a coupling argument. Beyond theoretical results, numerical experiments are displayed in order to show the relevance of HT risk minimization and that ignoring the sampling scheme used to generate the training dataset may completely jeopardize the learning procedure.

Introduction

Whereas statistical learning techniques crucially exploit data that can serve as examples to train a decision rule, they may also make use of weights individually assigned to the observations, resulting from survey sampling. Such weights could correspond either to true inclusion probabilities or else to calibrated or post-stratification weights, minimizing some discrepancy under certain margin constraints for the inclusion probabilities. In the context of statistical inference based on survey data, the asymptotic properties of specific statistics such as Horvitz-Thompson estimators (cf Horvitz & Thompson (1951)), whose computation involves not only the observations but also the weights, have been investigated, in particular, mean estimation and regression have been the subject of much attention, refer to [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF], [START_REF] Rosen | Asymptotic theory for successive sampling[END_REF], [START_REF] Robinson | On the convergence of the Horvitz-Thompson estimator[END_REF], [START_REF] Deville | Calibration estimators in survey sampling[END_REF], [START_REF] Berger | Rate of convergence to normal distribution for the Horvitz-Thompson estimator[END_REF] for instance, and a comprehensive functional limit theory for distribution function estimation is progressively documented in the statistical literature, see [START_REF] Gill | Large sample theory of empirical distributions in biased sampling models[END_REF], [START_REF] Breslow | Weighted likelihood for semiparametric models and two-phase stratified samples, with application to Cox regression[END_REF], [START_REF] Breslow | A Z-theorem with estimated nuisance parameters and correction note for "Weighted likelihood for semiparametric models and two-phase stratified samples, with application to Cox regression[END_REF], [START_REF] Breslow | Improved Horvitz-Thompson estimation of model parameters from two-phase stratified samples: applications in epidemiology[END_REF], [START_REF] Saegusa | Weighted likelihood estimation under two-phase sampling[END_REF]. At the same time, the last decades have witnessed a rapid development of the field of machine-learning. Revitalized by different breakout algorithms (e.g. SVM, boosting methods), its practice is now supported by a sound probabilistic theory based on recent non asymptotic results in the study of empirical processes, 16

Chapter 2. Learning from Survey Training Samples see Devroye et al. (1996a), [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization (with discussion)[END_REF], Boucheron et al. (2005a). However, most papers dedicated to theoretical results grounding the Empirical Risk Minimization approach, the main paradigm of statistical learning, assume that the training of a decision rule is based on a dataset formed of independent replications of a generic random vector Z, a collection of n ≥ 1 i.i.d. observations Z 1 , . . . , Z n namely. In contrast, few results are available in situations where the training dataset is generated by a more complex sampling scheme. One may refer to [START_REF] Bardenet | Concentration inequalities for sampling without replacement[END_REF] for concentration inequalities permitting to study the generalization ability of empirical risk minimizers when the training data are obtained by standard sampling without replacement (SWOR in abbreviated form) or to [START_REF] Steinwart | Learning from dependent observations[END_REF] in the case where the decision rule is learnt from a path of a weakly dependent stochastic process.

In this chapter, we extend the ERM theory to situations where the training dataset is generated by means of a more general sampling scheme, with possibly unequal probability weights. We first consider the case of rejective sampling (sometimes referred to as conditional Poisson sampling), an important generalization of basic SWOR. The rate bound results obtained by means of properties of so-termed negatively related random variables in this case are next shown to extend to a class of more general sampling schemes by a coupling argument. In addition, numerical experiments are carried out in order to provide empirical evidence of the approach developed. They show in particular that statistical accuracy of the ERM paradigm may fail if the sampling scheme underlying the training dataset is ignored.

The chapter is organized as follows. In section 2.2, the probabilistic framework of the present study is described and basic results of the probabilistic theory of classification are briefly recalled, together with some important notions of survey theory. The main theoretical results are stated in section 2.3, while illustrative numerical experiments are presented in section 2.4.

Background and Preliminaries

As a first go, we start with recalling key concepts pertaining to the theory of empirical risk minimization in binary classification, the flagship problem in statistical learning. A few notions related to survey theory are next described, which will be involved in the subsequent analysis. Throughout the chapter, the indicator function of any event E is denoted by I{E}, the Dirac mass at any point a by δ a , the power set of any set E byP(E), the cardinality of any finite set A by #A.

Binary Classification -Empirical Risk Minimization Theory

The binary classification problem is undeniably the most documented statistical learning problem in the literature and certain results extend to other general frameworks (e.g. multi-class classification, regression,ranking). With (Ω, A, P) a probability space and (X, Y ) a random pair defined on (Ω, A, P), taking its values in some measurable product space X × {-1, +1}, with common distribution P (dx, dy): the r.v. X models some observation, hopefully useful for predicting the binary label Y . The distribution P can also be described by the pair (F, η) where F (dx) denotes the marginal distribution of the input variable X and η(x) = P{Y = +1 | X = x}, x ∈ X , is the posterior distribution. The objective of binary classification is to build, based on the training dataset at disposal, a measurable mapping g : X → {-1, +1}, called a classifier, with minimum risk:

L(g) def = P{g(X) = Y }. (2.1)
IThe Bayes classifier g * (x) = 2I{η(x) ≥ 1/2} -1 is a solution of the risk minimization problem inf g L(g), where the infinimum is taken over the collection of all classifiers defined on the input space X . The minimum risk is denoted by L * = L(g * ). Since the distribution P of the data is unknown, one substitutes the true risk with its empirical estimate

L n (g) = 1 n n i=1 I{g(X i ) = Y i }, (2.2)
based on a sample (X 1 , Y 1 ), . . . , (X n , Y n ) of independent copies of the generic random pair (X, Y ). The true risk minimization is then replaced by the empirical risk minimization

min g∈G L n (g), (2.3)
where the minimum is taken over a class G of classifier candidates, supposed rich enough to include the naive Bayes classifier (or a reasonable approximation of the latter). Considering a solution g n of (2.3), a major problem in statistical learning theory is to establish upper confidence bounds on the excess of risk L( g n ) -L * in absence of any distributional assumptions and taking into account the complexity of the class G (e.g. described by geometric or combinatorial features such as the VCdimension) and some measure of accuracy of approximation of P by its empirical counterpart

P n = (1/n) n i=1 δ (X i ,Y i )
over the class G. Indeed, one typically bounds the excess of risk of the empirical risk minimizers as follows

L( g n ) -L * ≤ 2 sup g∈G | L n (g) -L(g)| + inf g∈G L(g) -L * .
The second term on the right hand side is referred to as the bias and depends on the richness of the class G, while the first term, called the stochastic error, is controlled by means of results in empirical process theory, see chapter A and Boucheron et al. (2005a).

Remark 2.1. (ON RISK SURROGATES) Although its study is of major interest from a theoretical perspective, the problem (2.3) is generally NP-hard. For this reason, the cost function I{-Y g(X) > 0} is replaced in practice by a non-negative convex cost function φ(Y g(X)), turning empirical risk minimization to a tractable convex optimization problem. Typical choices include the exponential cost φ(u) = exp(u) used in boosting algorithms, the hinge loss φ(u) = (1 + u) + in the case of support vector machines and the logit cost φ(u) = log(1 + exp(u)) for Neural networks, see [START_REF] Bartlett | Convexity, classification and risk bounds[END_REF] and the references therein.

In this chapter, we consider the situation where the training data used to compute of the empirical risk (2.2) is not an i.i.d. sample but the product of a more general sampling plan of fixed size N ≥ 1.

Sampling Schemes and Horvitz-Thompson Estimation

Let n ≥ 1. In the standard superpopulation framework we consider, (X 1 , Y 1 ), . . . , (X n , Y n ) is a sample of independent copies of (X, Y ) observed on a finite population I n := {1, . . . , n}. We call a survey sample of (possibly random) size N ≤ n of the population I n , any subset s := {i 1 , . . . , i n(s) } ∈ P(I n ) with cardinality N =: N (s) less than n. A sampling design is determined by a conditional probability distribution R n on the set of all possible samples s ∈ P(I n ) given the original data D n = {(X i , Y i ) : i ∈ I n }. For any i ∈ {1, . . . , n}, the first order inclusion probability, π i = P Rn {i ∈ S} is the probability that the unit i belongs to a random sample S drawn from the conditional distribution R n . We set π = (π 1 , . . . , π n ).

Chapter 2. Learning from Survey Training Samples

The second order inclusion probabilities are denoted by π i,j = P Rn {(i, j) ∈ S 2 }, for any i = j in {1, . . . , n} 2 . The information related to the observed sample S ⊂ {1, . . . , n} is fully enclosed in the r.v. n = ( 1 , . . . , n ), where i = I{i ∈ S} for 1 ≤ i ≤ n. The 1-d marginal conditional distributions of the sampling scheme n given D n are the Bernoulli distributions B(π i ) = π i δ 1 + (1 -π i )δ 0 , 1 ≤ i ≤ n, and the covariance matrix Γ n of the r.v. n has entries given by Γ n (i, j) = π i,j -π i π j , with π i,i = π i by convention, for 1 ≤ i, j ≤ n. Observe that, equipped with the notations above, 1≤i≤n i = n(S). One may refer to [START_REF] Cochran | Sampling techniques[END_REF], [START_REF] Deville | Réplications d'échantillons, demi-échantillons, Jackknife, bootstrap dans les sondages[END_REF] for accounts of survey sampling techniques. Notice also that, in many applications, the inclusion probabilities are built using some extra information, typically by means of auxiliary random variables W 1 , . . . , W n defined on (Ω, A, P) and taking their values in some measurable space W: ∀i ∈ {1, . . . , n}, π i = N h(W i )/ 1≤j≤n h(W j ), where N max 1≤i≤n h(W i ) ≤ 1≤i≤n h(W i ) almost-surely and h : W →]0, +∞[ is a measurable link function. The (X i , Y i , W i )'s are generally supposed to be i.i.d. copies of a generic r.v. (X, Y, W ). See [START_REF] Särndall | Model assisted survey sampling[END_REF] for more details. For simplicity, the π i 's are supposed to be deterministic in the subsequent analysis, which boils down to carrying out the study conditionally upon the W i 's in the example aforementioned.

Horvitz-Thompson risk. As defined in [START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF], the Horvitz-Thompson version of the (not available) empirical risk L n (g) = n -1 1≤i≤n I{Y i = g(X i )} of any classifier candidate g based on the sampled data {(X i , Y i ) : i ∈ S} with S ∼ R n is given by:

L n (g) = 1 n i∈S 1 π i I{g(X i ) = Y i } = 1 n n i=1 i π i I {g(X i ) = Y i } (2.4)
with the convention that 0/0 = 0 and where n = ( 1 , . . . , n ) denotes the vector in correspondence with the sample S. Observe that, conditionally upon the (X i , Y i )'s, the quantity (2.4), that shall be referred to as the empirical Horvitz-Thompson risk (empirical HT risk in short) throughout the chapter, is an unbiased estimate of the empirical risk L n (g). Its (pointwise) consistency and asymptotic normality are established in [START_REF] Robinson | On the convergence of the Horvitz-Thompson estimator[END_REF] and [START_REF] Berger | Rate of convergence to normal distribution for the Horvitz-Thompson estimator[END_REF] for a variety of sampling schemes. Limit results of functional nature are established in [START_REF] Gill | Large sample theory of empirical distributions in biased sampling models[END_REF] for specific biased sampling models, refer also to [START_REF] Breslow | Weighted likelihood for semiparametric models and two-phase stratified samples, with application to Cox regression[END_REF], [START_REF] Breslow | A Z-theorem with estimated nuisance parameters and correction note for "Weighted likelihood for semiparametric models and two-phase stratified samples, with application to Cox regression[END_REF], [START_REF] Saegusa | Weighted likelihood estimation under two-phase sampling[END_REF][START_REF] Bertail | Empirical processes in survey sampling[END_REF].

We investigate the statistical performance of minimizers ḡn of the HT risk (2.4) over the class G under adequate assumptions for the sampling scheme R n used to generate the training dataset. We point out that such an analysis is far from straightforward due to the possible dependence structure of the terms involved in the summation (2.4): except in the Poisson case (recalled below), concentration results for empirical processes cannot be directly applied to control maximal deviations of the type

sup g∈G |L n (g) -L(g)|.
Conditional Poisson sampling. One of the simplest sampling plan is undeniably the Poisson survey scheme (without replacement), a generalization of Bernoulli sampling originally proposed in [START_REF] Goodman | On the estimation of the number of classes in a population[END_REF] for the case of unequal weights: the i 's are independent and the sampling distribution is thus entirely determined by the first order inclusion probabilities

p n = (p 1 , . . . , p n ) ∈]0, 1[ n : ∀s ∈ P(I n ), P n (s) = i∈S p i i / ∈S (1 -p i ).
(2.5)

Observe in addition that the behaviour of the quantity (2.4) can be then investigated by means of results established for sums of independent random variables. However, the major drawback of this sampling plan lies in the random nature of the corresponding sample size, impacting significantly the variability of (2.4). The variance of the Poisson sample size is given by d n = n i=1 p i (1 -p i ), while the conditional variance of (2.4) is in this case:

n i=1 ((1 -p i )/p i )I{g(X i ) = Y i }.
For this reason, rejective sampling, a sampling design R n of fixed size N ≤ n, is often preferred in practice. It generalizes the simple random sampling without replacement (where all samples with cardinality N are equally likely to be chosen, with probability (n -N )!/n!, all the corresponding first and second order probabilities being thus equal to N/n and N (N -1)/(n(n -1)) respectively). Denoting by π n = (π 1 , . . . , π N ) its first order inclusion probabilities and by S n = {s ∈ P(I n ) : #s = N } the subset of all possible samples of size N , it is defined by:

∀s ∈ S n , R n (s) = C i∈s p i i / ∈s (1 -p i ),
(2.6)

where C = 1/ s∈Sn i∈s p i i / ∈s (1-p i ) and the vector p n = (p 1 , . . . , p n ) ∈]0, 1[ n yields first order inclusion probabilities equal to the π i 's and is such that i≤n p i = n. Under this latter additional condition, such a vector p n exists and is unique (see [START_REF] Dupacova | A note on rejective sampling[END_REF]) and the related representation (2.6) is then said to be canonical1 . Comparing (2.6) and (2.5) reveals that rejective R n sampling of fixed size N can be viewed as Poisson sampling given that the sample size is equal to N . It is for this reason that rejective sampling is usually referred to as conditional Poisson sampling. One must pay attention not to get the π i 's and the p i 's mixed up: the latter are the first order inclusion probabilities of P n , whereas the former are those of its conditional version R n . However they can be related by means of the results stated in [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF] (see Theorem 5.1 therein): ∀i ∈ {1, . . . , n},

π i (1 -p i ) = p i (1 -π i ) × (1 -(π -π i ) /d * n + o(1/d * n )) , (2.7) p i (1 -π i ) = π i (1 -p i ) × (1 -(p -p i ) /d n + o(1/d n )) ,
(2.8)

where d * n = n i=1 π i (1-π i ), π = (1/d * n ) n i=1 π 2 i (1-π i ) and p = (1/d n ) n i=1 (p i ) 2 (1-p i ).
More examples of sampling schemes with fixed size are given in section 2.8.

Main Results

We first consider the case where statistical learning is based on the observation of a sample drawn by means of a rejective scheme. As shall be seen below, the main argument underlying the results obtained relies on the fact that the related scheme form a collection of negatively associated (binary) random variables, a rather tractable type of dependence structure. This property being shared by many other sampling schemes of deterministic size, the same argument can be thus naturally applied to carry out a similar rate analysis for training data produced by such plans. Extensions of these results to more general sampling schemes are also considered by means of a coupling technique.

Chapter 2. Learning from Survey Training Samples

Horvitz-Thompson Empirical Risk Minimization in the Rejective Case

For clarity, we first recall the definition of negatively associated random variables, see [START_REF] Joag-Dev | Negative Association of Random Variables with Applications[END_REF].

Definition 2.2. Let Z 1 , . . . , Z n be random variables defined on the same probability space, valued in a measurable space (E, E). They are said to be negatively associated iff for any pair of disjoint subsets A 1 and A 2 of the index set {1, . . . , n} Cov (f ((Z i ) i∈A 1 ), g((Z j ) j∈A 2 )) ≤ 0, (2.9)

for any real valued measurable functions f : E #A 1 → R and g : E #A 2 → R that are both increasing in each variable.

The theorem stated below reveals that any rejective scheme n forms a collection of negatively associated r.v.'s. The proof is given in Appendix 2.6.

Theorem 2.3. Let n ≥ 1 and n = ( 1 , . . . , N ) be the vector of indicator variables related to a rejective plan on I n . Then, the binary random variables 1 , . . . , n are negatively associated.

The result above permits to handle the dependence of the terms involved in the summation (2.4). It is the key argument for proving the following proposition, which extends results for training datasets generated by basic sampling without replacement (i.e. in the case of all equal weights: π i = N/n for i = 1, . . . , n), refer to [START_REF] Bardenet | Concentration inequalities for sampling without replacement[END_REF] (see also [START_REF] Serfling | Probability inequalities for the sum in sampling without replacement[END_REF]).

Proposition 2.4. Suppose that the sampling scheme n is rejective with first order inclusion probabilities π n and that the class G is of finite VC dimension V < +∞. Set κ n = N/(n × min i≤n π i ). Then, the following assertions hold true.

(i) For any δ ∈ (0, 1), with probability larger than 1 -δ, we have: ∀N ≤ n,

sup g∈G L n (g) -L n (g) ≤ 2κ n log( 2 δ ) + V log(n + 1) N + 2κ n log( 2 δ ) + V log(n + 1) 3N .
(2.10)

(ii) For any solution ḡn of the minimization problem inf g∈G L n (g) is such that, for any δ ∈ (0, 1), with probability at least 1 -δ, we have: ∀n ≥ 1,

L(ḡ n ) -L * ≤ 2 2κ N log( 4 δ ) + V log(n + 1) N + 4κ n log( 4 δ ) + V log(n + 1) 3N + C V n + 2 2 log( 2 δ ) n + inf g∈G L(g) -L * .
The factor κ N involved in the bounds above reflects the influence of the sampling scheme (notice incidentally that κ n ≥ 1 since i≤n π i = N ). In the SWOR case, i.e. when π i = N/n for all i ∈ {1, . . . , n}, it is then minimum, equal to 1. As recalled in section 2.7, the canonical parameters p n are practically used to build a rejective sampling scheme n rather than its vector of first order inclusion probabilities (π 1 , . . . , π N ), whose explicit computation based on the p i 's is a difficult task, refer to [START_REF] Chen | Weighted finite population sampling to maximize entropy[END_REF] for dedicated algorithms. For this reason, one could be naturally tempted to minimize the alternative risk estimate L n (g) = (1/n) i≤n ( i /p i )I{Y i = g(X i )}. As proved in section 2.7, refinements of Eq. (2.7)-(2.8) show that

sup g∈G | L n (g) -L n (g)| ≤ 1 n n i=1 1 p i - 1 π i ≤ 6N κ N /(nd N ), (2.11)
one may directly derive a rate bound for solutions of inf g∈G L n (g) from bound (ii) in Proposition 2.4. In particular, the learning rate achieved by ḡn is preserved when 1/ √ N = O(min i≤n π i ) as n, N → +∞.

Extensions to More General Sampling Schemes

We now extend the rate bound analysis carried out in the previous subsection to more complex sampling schemes (described by a random vector * n possibly exhibiting a very complex dependence structure). In order to give an insight into the arguments which the extension is based on, additional notations are required. In this section, we consider a general sampling design R * n with first order inclusion probabilities π * n = (π

(g) = (1/n) n i=1 ( * i /π * i )I{Y i = g(X i )} over a class G.
We also consider a rejective sampling scheme R n described by the r.v. n , with first order inclusion probabilities π n = (π 1 , . . . , π n ) defined on the same probability space, as well as the following quantity:

Ľ n (g) = 1 n n i=1 i π * i I{Y i = g(X i )} (2.12)
for any classifier g. Observe that (2.12) differs from the HT empirical risk L n (g) related to the rejective sampling scheme n in the weights it involves, the π * i 's instead of the π i 's namely. Equipped with this notation, the excess of risk of the HT empirical risk minimizer can Chapter 2. Learning from Survey Training Samples be bounded as follows:

L(ḡ * n ) -inf g∈G L(g) ≤ 2 sup g∈G L(g) -L n (g) + 2 sup g∈G L n (g) -L n (g) + 2 sup g∈G L n (g) -Ľ n (g) + 2 sup g∈G Ľ n (g) -L * n (g) . (2.13)
Whereas the first term on the right hand side of (2.13) can be classically controlled using Vapnik-Chervonenkis and McDiarmid inequalities (see e.g. [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]), assertion (i) of Proposition 2.4 provides a control of the second term. Following in the footsteps of [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF], the third term shall be bounded by means of a coupling argument, i.e. a specific choice of the joint distribution of ( * n , n ) satisfying the distributional margin constraints, while the second term is controlled by assumptions related to the closeness between the first order inclusion probabilities π * n and π n . More precisely, the assumptions required in the subsequent analysis involve the total variation distance between the sampling plans R n and R * n :

d T V (R n , R * n ) def = 1 2 s∈P(In) |R n (s) -R * n (s)|.
Theorem 2.8. Suppose that Proposition 2.4's assumptions are fulfilled. Set κ * N = (N/n) min i≤n π * i and κ N = (N/n)/ min i≤n π i . Then, there exists a universal constant C < +∞ such that we have, ∀n ≥ 1,

E L(ḡ * n ) -inf g∈G L(g) ≤ 2 2κ N log( 4 δ ) + V log(n + 1) N + 4κ N log( 4 δ ) + V log(n + 1) 3N + C V n + 2 2 log( 2 δ ) n + 2(κ * N + κ N )(n/N )d T V (R n , R * n ), (2.14)
where the infinimum is taken over the set of rejective schemes R n with first order inclusion probabilities π n = (π 1 , . . . , π N ).

The proof is given in section 2.6. The rate bound obtained depends on the minimum error made when approximating the sampling plan by a rejective sampling plan in terms of total variation distance. In practice, following in the footsteps of [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF] or [START_REF] Berger | Rate of convergence to normal distribution for the Horvitz-Thompson estimator[END_REF], it can be controlled by exhibiting a specific coupling ( * n , n ). One may refer to Berger (1998) for many coupling results of this nature, in particular when the approximating scheme n is of rejective type.

Illustrative Numerical Experiments

In this section we display numerical experiments to illustrate the relevance of HT risk minimization. We first consider the case where g(X) = sign(k(X) T θ + b), where k is some mapping function, T denotes the transposition operator, θ, b are some parameters. As mentioned in 2.1, we consider the hinge loss as a convex surrogate of the 0 -1 loss and add some l 2 regularization term. This leads to the "Weighted SVM" formulation below:

min θ,b 1 n i∈S 1 π i max(0, 1 -Y i (k(X i ) T θ -b)) + λ θ 2 .
We use the gaussian r.b.f kernel and perform cross validation to appropriately choose the value of λ. We then consider the task of learning classification trees using the CART algorithm. These classifiers are trained using the scikit-learn library [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] Overall, taking into account the inclusion probability allows to consider a training set of reduced size and therefore reduce the computational complexity of the learning procedure without damaging the quality of the prediction.

The same conclusions can be drawn from the analysis of the following datasets which were obtained via a stratified sampling design. We point out that this sampling scheme involves negatively associated (binary) random variables so that the theoretical results obtained in this chapter apply to training data sampled by means of this scheme as well.

incaIndiv GJB privacy3 privacy4 The dataset incaIndiv 2 contain informations on the food consumption of the french population. The dataset GJB 3 contains questions about job seeking and the internet, workforce automation, online dating and smartphone use among Americans. The datasets privacy3 4 and Chapter 2. Learning from Survey Training Samples privacy45 contain questions about privacy and information sharing. On the datasets incaIndiv and incaCompl we try to predict whether or not someone is an adult, on the dataset GJB we will try to learn to predict the gender, and on the datasets privacy3 and privacy4 we will predict an answer to some questions among 5 possibilities.

We perform our experiments by randomly splitting the datasets incaIndiv, incaCompl,GJB into a training set (roughly 70 percent of the initial dataset) and a test set. The size of privacy3 and privacy4 being much smaller we perform 10-fold cross-validation.

incaIndiv GJB privacy3 privacy4

Weighted

Conclusion

Most theoretical studies providing a statistical explanation for the success of learning algorithms based on the ERM paradigm fully ignore the possible impact of the sampling scheme producing the training data and stipulate that observations are independent replications of a generic r.v. or are uniformly sampled without replacement in a larger dataset. Through the generalizable example of rejective sampling, this chapter shows that such studies can be extended to situations where training data are obtained by more general sampling schemes and possibly exhibit a complex dependence structure, provided that related probability weights are appropriately incorporated in the risk functional.

Technical

Bernstein's Inequality for Sums of Negatively Associated Random Variables

For simplicity, we first establish the following tail bound for negatively associated random variables, which extends the usual Bernstein inequality in the i.i.d. setting, see [START_REF] Bernstein | On a modification of chebyshev's inequality and on the error in laplace formula[END_REF] and section A.2.2. Proofs of Proposition 2.4 and Theorem 2.8 are then deduced from Theorem 2.3 and Theorem 2.9 (see section 2.6). Althought it is not done here, Hoeffding inequality for sums of negatively associated random variables could also be easily derived and we refer to the proofs of A.2.1.2 to establish this result.

Theorem 2.9. Let Z 1 , . . . , Z N be negatively associated real valued random variables such that

|Z i | ≤ c < +∞ a.s. E[Z i ] = 0 and E[Z 2 i ] = σ 2 i for 1 ≤ i ≤ n.
Then, for all t > 0, we have: ∀n ≥ 1,

P N i=1 Z i ≥ t ≤ exp - t 2 2 3 ct + 2 n i=1 σ 2 i .
Before detailing the proof, observe that a similar bound holds true for the tail probability P N i=1 Z i ≤ -t (and for P | N i=1 Z i | ≥ t as well, up to a multiplicative factor 2). Refer also to Theorem 4 in [START_REF] Janson | Large deviation inequalities for sums of indicator variables[END_REF] for a similar result in a more restrictive setting (i.e. for tail bounds related to sums of negatively associated r.v.'s).

Proof.

Similarly to what we did in section chapter A, the proof starts off with the usual Chernoff method: for all λ > 0,

P N i=1 Z i ≥ t ≤ exp -tλ + log E e t n i=1 Z i .
(2.15)

Next, observe that, for all t > 0, we have

E e t n i=1 Z i = E e tZn e t n-1 i=1 Z i ≤ E e tZn E e t n-1 i=1 Z i ≤ n i=1 E e tZ i , (2.16)
using the property (2.9) combined with a descending recurrence on i. The proof is finished by plugging (2.16) into (2.15), using an adequate control of the log-Laplace transform of the Z i 's throught Hoeffding's Lemma A.7 and optimizing finally the resulting bound w.r.t. λ > 0, just like in the proof of the classic Bernstein inequality, see chapter [START_REF] Bernstein | On a modification of chebyshev's inequality and on the error in laplace formula[END_REF].

On Biased HT Risk Minimization

Eq. (2.11) directly results from the following lemma.

Lemma 2.10. We have, for p i s such Suppose that d n ≥ 1. We have, for all i ∈ {1, . . . , n},

|1/π i -1/p i | ≤ 6 d n × (1 -π i )/π i .
Proof. The proof follows from the representation (5.14) on p1509 in [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF]. Denote by P N a Poisson sampling distribution on I n with inclusion probabilities p 1 , . . . , p n , the canonical parameters of R n . For all i ∈ {1, . . . , n}, we have:

π i p i 1 -p i 1 -π i =   s∈P(In): i∈In\{s} P (s)   -1 s∈P(In): i∈In\{s} P (s) h∈s 1 -p h j∈s (1 -p j ) + (p h -p i ) =   s: i∈In\{s} P N (s)   -1 s: i∈In\{s} P N (s) h∈s 1 -p h j∈s (1 -p j ) 1 + (p h -p i ) j∈s (1-p j )
Chapter 2. Learning from Survey Training Samples Now recall that for any x ∈] -1, 1[, we have:

1 -x ≤ 1 1 + x ≤ 1 -x + x 2 .
It follows that

π i p i 1 -p i 1 -π i ≤ 1 -   s: i∈In\{s} P (s)   -1 s: i∈In\{s} P (s) h∈s (1 -p h )(p h -p i ) j∈s (1 -p j ) 2 +   s: i∈In\{s} P (s)   -1 s: i∈In\{s} P (s) h∈s (1 -p h )(p h -p i ) 2 j∈s (1 -p j ) 3
Following now line by line the proof on p. 1510 in [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF] and noticing that j∈s (1 -

p j ) ≥ 1/2d n (see Lemma 2.2 in Hajek (1964)), we have h∈s (1 -p h )(p h -p i ) j∈s (1 -p j ) 2 ≤ 1 j∈s (1 -p j ) ≤ 2 d n and similarly h∈s (1 -p h )(p h -p i ) 2 j∈s (1 -p j ) 3 ≤ 1 j∈s (1 -p j ) 2 ≤ 4 d 2 n .
This yieds: ∀i ∈ {1, . . . , n},

1 - 2 d n ≤ π i p i 1 -p i 1 -π i ≤ 1 + 2 d n + 4 d 2 n and p i (1 -π i )(1 - 2 d n ) ≤ π i (1 -p i ) ≤ p i (1 -π i )(1 + 2 d n + 4 d 2 n ), leading then to - 2 d n (1 -π i )p i ≤ π i -p i ≤ p i (1 -π i )( 2 d n + 4 d 2 n )
and finally to

- (1 -π i ) π i 2 d n ≤ 1 p i - 1 π i ≤ (1 -π i ) π i ( 2 d n + 4 d 2 n ).
Since 1/d 2 n ≤ 1/d n as soon as d n ≥ 1, the lemma is proved.

Sampling Training Data -Technical Details

Further Details on the Rejective Scheme

Let N ≤ n and consider a vector π = (π 1 , . . . , π n ) of first order inclusion probabilities. Further define S n := {s ∈ P(I n ) : #s = N }, the set of all samples in population I n with cardinality N . The rejective sampling [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF]; [START_REF] Berger | Rate of convergence to normal distribution for the Horvitz-Thompson estimator[END_REF], sometimes called conditional Poisson sampling, exponential design without replacement or maximum entropy design, is the sampling design R n that selects samples of fixed size N (s) = N so as to maximize the entropy measure

H(R n ) = -s∈Sn R n (s) log R n (s)
, subject to the constraint that its vector of first order inclusion probabilities coincides with π. It is easily implemented in two steps:

1. Draw a sample S according to a Poisson plan P n , with properly chosen first order inclusion probabilities p n = (p 1 , . . . , p n ). The representation is called canonical if

n i=1 p i = n.
In that case, relationships between each p i and π i , 1 ≤ i ≤ n, are established in [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF].

If n(S)

= n, then reject sample S and go back to step one, otherwise stop.

Vector p must be chosen in a way that the resulting first order inclusion probabilities coincide with π, by means of a dedicated optimization algorithm [START_REF] Tillé | Sampling algorithms[END_REF]. The corresponding probability distribution is given for all s ∈ P(

I n ) by R n (s) = Pn(s) I{#s=N } s ∈Sn Pn(s ) ∝ i∈s p i i / ∈s (1 -p i ) × I{#s = N },
where ∝ denotes the proportionality.

Examples of Sampling Plan with Negatively Associated Random Variables

Below we list two examples of sampling scheme involving negatively associated r.v..

Stratified Sampling

A stratified sampling design permits to draw a sample S of fixed size N (S) = N ≤ n within a population I n that can be partitioned into K ≥ 1 distinct strata I n 1 , . . . , I n K (known a priori) of respective sizes N 1 , . . . , N K adding up to n. Let N 1 , . . . , N K be nonnegative integers such that

N 1 + • • • + N K = N ,
(s) = K k=1 N k n k -1
. Naturally, first and second order inclusion probabilities depend on the stratum to which each unit belong:

for all i = j in U n , π i (R str n ) = K k=1 n k N k I{i ∈ U N k } and π i,j (R str n ) = K k=1 n k (n k -1) N k (N k -1) I{(i, j) ∈ U 2 N k }.

Rao-Sampford Sampling

The Rao-Sampford sampling design generates samples s ∈ P(I n ) of fixed size N (s) = N with respect to some given first order inclusion probabilities

π RS := (π RS 1 , . . . , π RS n ), fulfill- ing the condition n i=1 π RS i = n, with probability R RS n (s) = η i∈s π RS i j / ∈s π RS j 1 -π RS j .
Here, η > 0 is chosen such that s∈P(In) R RS n (s) = 1. In practice, the following algorithm is often used to implement such a design Berger (1998):

Chapter 2. Learning from Survey Training Samples 1. select the first unit i with probability π RS i /n, 2. select the remaining n -1 units j with drawing probabilities proportional to π RS j /(1π RS j ), j = 1, . . . , n,

3. accept the sample if the units drawn are all distinct, otherwise reject it and go back to step one.

PART II

Sampling strategies for Stochastic Gradient Descent

CHAPTER 3

Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm

Abstract A wide collection of popular statistical learning methods, ranging from K-means to Support Vector Machines through Neural Networks, can be formulated as a stochastic gradient descent (SGD) algorithm in a specific setup. In practice, the main limitation of this incremental optimization technique is due to the stochastic noise induced by the choice at random of the data involved in the gradient estimator computed at each iteration. In this chapter, we introduce a novel implementation of the SGD algorithm, where the data subset used at a given step is not picked uniformly at random among all possible subsets but drawn from a specific adaptive sampling scheme, depending on the past iterations in a Markovian manner, in order to refine the current statistical estimation of the gradient. Beyond an algorithmic description of the approach we propose, rate bounds are established and illustrative numerical results are displayed in order to provide theoretical and empirical evidence of its statistical performance, compared to more "naive" SGD implementations. Computational issues are also discussed, revealing the practical advantages of the method promoted.

Introduction

In this chapter, we consider the generic minimization problem

min θ∈Θ L n (θ) = min θ∈Θ 1 n n i=1 l(Z i , θ), (3.1)
where Θ is a Euclidean space, typically R d with d ≥ 1, and l(Z 1 , .), . . . , l(Z n , .) form a collection of real-valued convex continuously differentiable functions on Θ. Such an optimization problem typically arises in a broad variety of statistical learning problems, in particular supervised tasks, where the goal pursued is to learn a predictive model, fully determined by a parameter θ, in order to predict a random variable Y (the response/output) from an input observation X taking its values in a feature space X . The performance of the predictive function defined by θ is measured by the expectation

L(θ) = E[l(θ; (X, Y ))],
referred to as the risk, where l is a loss function assumed convex w.r.t. θ. As mentionned before, the distribution (X, Y ) being unknown in practice, the risk functional is replaced by its statistical counterpart, the empirical risk namely, given by

L n (θ) = 1 n n i=1 l((X i , Y i ), θ), (3.2) based on n ≥ 1 supposedly available independent training examples (X 1 , Y 1 ), . . ., (X n , Y n ),
copies of the random pair (X, Y ). This supervised problem shall serve as a running example throughout the chapter. The minimization problem (3.2) can be solved incrementally, by 32

Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm means of variants of the stochastic approximation method originally introduced in the seminal contribution of [START_REF] Robbins | A stochastic approximation method[END_REF]. This consists in computing successive estimates of a minimizer of (3.2) using the recursive equation

θ t+1 = θ t -γ t r t (θ t ) (3.3)
from a preliminarily picked initial value θ 0 ∈ Θ, where r t denotes an estimate of the gradient ∇ L n and γ t is the learning rate or stepsize. In contrast to the batch approach, where all the data are used to estimate the gradient at each iteration (i.e. r t (θ) = ∇ L n (θ) for all t ≥ 0 and θ ∈ Θ), subsets of the data sample only are involved in the gradient estimation steps of sampled incremental algorithms, with the aim to reduce computational cost when n is large. In the most commonly used implementation of the stochastic gradient descent (SGD) algorithm, the gradient estimate is computed from a subset of reduced size S ≤ n uniformly drawn without replacement among all possible subsets of the dataset of size S at each step t ≥ 0.

In the present chapter, we introduce a specific variant of the SGD algorithm with an adaptive sampling scheme, in the sense that it may possibly be different from sampling without replacement (SWOR) and vary with t, depending on the past iterations. Rate bounds and limit theorems guaranteeing the theoretical validity of the methodology we propose are established.

In addition, the Markovian dynamics governing the evolution of the instrumental sampling distribution is shown to offer crucial advantages regarding computational efficiency. Finally, very encouraging experimental results are displayed, supporting the relevance of our method, in comparison to the usual mini-batch SGD implementation or alternative SGD techniques standing for natural competitors.

The chapter is structured as follows. A short review of the SGD methods documented in the literature, those based on non SWOR sampling schemes in particular, can be found in section 3.2. A description of the specific variant we propose in this chapter is given in section 3.3, together with a detailed discussion about the computational cost inherent to its implementation. The analysis assessing the validity of the estimate output by the algorithm proposed is carried out in section 3.4, whereas illustrative experiments are presented in section 3.5. Finally, some concluding remarks are collected in section 3.6.

Non Uniform Sampling (NUS) -State of the Art

We start off with a brief review of sampled incremental optimization algorithms, whose archetype is the celebrated SGD algorithm [START_REF] Robbins | A stochastic approximation method[END_REF]). Although relevant references in this area are much too numerous to be listed in an exhaustive manner, we point out that significant advances have been recently made in the design of efficient incremental methods, see for instance [START_REF]Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning[END_REF], [START_REF]Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization[END_REF], Johnson & Zhang (2013a), Shalev-Shwartz & Zhang (2012), [START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF] or Defazio et al. (2014) that achieve better performances than the traditional SGD method (for instance, by having the variance of the estimator going to 0 as in Johnson & Zhang (2013a), Defazio et al. (2014)). In order to study adaptative sampling scheme, we only considered the classical framework of SGD and did not compare to theses methods. In the original implementation of the SGD algorithms, a single observation (i.e. S = 1), indexed by i t+1 say, is chosen uniformly at random in {1, . . . , n} at each iteration t + 1 to form the gradient estimate ∇l(Z i t+1 , θ t ), its convergence following then from basic stochastic approximation theory. More recently, several papers have shown the possible gain from the use of non-uniform sampling, that is, choosing i t+1 according to a non-trivial distribution p well-suited to the specific optimization problem considered: this approach boils down to finding an optimal distribution, in the sense that it minimizes an upper bound on the convergence rate of the estimator [START_REF] Zhao | Stochastic Optimization with Importance Sampling[END_REF][START_REF] Needell | Stochastic gradient descent, weighted sampling and the randomized kaczmarz algorithm[END_REF], [START_REF] Clemencon | Scaling up m-estimation via sampling designs: The horvitz-thompson stochastic gradient descent[END_REF]. As far as NUS is concerned, it is natural to ask what relevant choice of the probability distribution must be chosen in order to achieve the smallest expected risk: for instance, the sampling scheme may depend on the Lipschitz constant of the gradient as proposed in D. Needell et al. (2014) or on upper bounds for the norm of the gradient, see [START_REF] Zhao | Stochastic Optimization with Importance Sampling[END_REF]. Despite these recent contributions, some questions remain open. 1) In general, usual analyses of NUS algorithms do not fully grasp the impact of the sampling scheme on the performance. For instance, Defazio et al. (2014) establish performance bounds which prove the convergence of NUS scheme for SAGA. Although the attractivity of NUS is demonstrated in the simulations, the bound itself does not fully reveal the performance gain w.r.t. uniform sampling. 2) As opposed to uniform sampling, the choice of an index i t+1 according to a non-trivial distribution on {1, . . . , n} is obviously more demanding in terms of computational time. The question of an efficient sampling implementation remains posed. In particular, it is important to quantify the increased complexity caused by NUS. 3) Proposed rules for choosing the sampling distribution p depend on global properties of the functions l(Z i , θ) (namely the Lipschitz constants L i ). They do not build upon the amount of information gathered on the optimization problem as the algorithm proceeds. An alternative is to use adaptive sampling, updating the sampling distribution p = p t at each iteration t in a Markovian fashion, as in the algorithm described below.

As shall be seen in the subsequent analysis, the NUS approach proposed in this chapter has theoretical and practical advantages regarding all these aspects. Whereas rate bound analysis by means of standard tools is poorly informative in general in the present setting, the asymptotic viewpoint developed in this chapter clearly highlights the benefit to using the specific NUS method we promote.

Adaptive Sampling SGD (AS-SGD)

We now turn to the description of the variant of SGD method considered in this chapter. The main novelty arises from the use of a specific instrumental sampling distribution evolving at each iteration. We also provide some insight into the gain one may expect from such a method and discuss the computational issues related to its implementation. Here and throughout, if I = (i 1 , . . . , i S ) is a S-uplet on {1, . . . , n} and h is a function on {1, . . . , n}, we use the (slightly abusive) notation i∈I h(i) to represent the sum S n=1 h(i n ). For the rest of this chapter, all expectations are taken conditionally upon the observations (i.e the randomness only lies in the sampling strategy) and for any θ ∈ Θ, ||θ|| denotes its euclidean norm.

The Algorithmic Principle

Let S ≤ n be fixed. At each iteration t ≥ 1, the generic AS-SGD is implemented in three steps as follows:

1. Compute the instrumental probability distribution p t on {1, . . . , n} from the information available at iteration t.

2. Form a random sequence of S indexes I t+1 = (i

(1) t+1 , . . . , i (S)
t+1 ) by sampling independently S times according to distribution p t .

Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm 3. Update the estimate using the equation

θ t+1 = Π K (θ t - γ t S i∈I t+1 ∇l(Z i θ t ) np t,i ) . (3.4)
where K is a compact convex set and Π K is the orthogonal projection on K. Let F t = σ(I 1 , . . . , I t ) be the σ-algebra generated by the past variables up to time t. Conditioned upon F t , we have: i

(1) t , . . . , i

(S) t i.i.d. ∼ p t . Set p t,j = P(i (1)
t+1 = j|F t ) for j = 1, . . . , n. Equipped with these notations, observe that the original SGD algorithm corresponds to the case where S = 1 and p t,j ≡ 1/n. When S > 1, notice that, in contrast to the mini-batch SGD (based on the SWOR scheme), one samples with replacement here and, due to the fact that the p t,j 's are not equal in general, it is necessary to normalize the individual gradients by N p t,i in order to guarantee the unbiasedness condition of the increment, which is classically required to ensure proper convergence of the algorithm, see [START_REF] Robbins | A stochastic approximation method[END_REF], D. Needell et al. (2014), [START_REF] Zhao | Stochastic Optimization with Importance Sampling[END_REF], [START_REF] Clemencon | Scaling up m-estimation via sampling designs: The horvitz-thompson stochastic gradient descent[END_REF]:

E   1 S i∈I t+1 ∇l(Z i θ t ) np t,i F t   = n j=1 p t,j ∇l(Z j , θ t ) np t,j = ∇ L n (θ t ) .
(3.5)

Ideal Sampling Distribution

In order to provide some insight into the specific dynamics we propose to build the successive sampling distributions p t , we first evaluate a bound on the amount of decrease of the functional. We assume that hypothesis below is fulfilled. Assumption 1. For all i ∈ {1, . . . , n}, the function θ → l(Z i , θ) is convex, continuously differentiable and its gradient ∇l

(Z i , θ) is L i -Lipschitz continuous with L i < +∞.
Let θ t be the sequence defined by (3.4) and K = Θ. By virtue of Assumption 1 and Theorem 2.1.5 in Nesterov & Nesterov (2004), we have

L n (θ t+1 ) ≤ L n (θ t ) + ∇ L n (θ t ), θ t+1 -θ t + L 2 θ t+1 -θ t 2 where L = 1 n i L i .
Taking the conditional expectation, we obtain that

E[ L n (θ t+1 )|F t ] ≤ L n (θ t ) -γ t ∇ L n (θ t ) 2 + Lγ 2 t ∆(θ t , p t ) 2n 2 ,
where

∆(θ t , p t ) = E   1 S i∈I t+1 ∇l(Z i , θ t ) p t,i 2 F t   = 1 S n i=1 ∇l(Z i , θ t ) 2 p t,i n 2 (S -1) ∇ L n (θ t ) 2 S .
At iteration t, the probability p * t = p * t (θ t ) ensuring the steepest descent on the above bound is clearly given by p * t = arg min p ∆(θ t , p) or equivalently by p * t,i = ∇l(Z i , θ t ) / n j=1 ∇l(Z j , θ t ) for i ∈ {1, . . . , n}, as mentioned in Zhao & Zhang (2014) and [START_REF] Clemencon | Scaling up m-estimation via sampling designs: The horvitz-thompson stochastic gradient descent[END_REF]. Unfortunately, practical implementation of the above sampling scheme is prohibitively complex, as it would require to evaluate all gradients to calculate the norms ∇l(Z 1 , θ t ) , . . . , ∇l(Z n , θ t ) at each iteration which is precisely what we try to avoid. The crucial point is therefore to propose a sampling scheme approximating p * t without requiring any additional gradient evaluations.

Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm 35

A Practical Sampling Distribution -Our Proposal

The main idea is to replace each unknown gradient norm ∇l(Z i , θ t ) by a (possibly outdated) norm g t,i = ∇l(Z i , θ k ) at some former instant k = k(i, t) corresponding to the last time k ≤ t when the i-th component was picked. More formally, we define the random sequence g t as

g t+1,i = ∇l(Z i , θ t+1 ) if i ∈ {i (1) t+1 , . . . , i (S) t+1 } g t,i
otherwise.

(3.6)

Then, a natural way to approximate p * t is to set for each i

pt,i = g t,i n j=1 g t,j
.

(3.7)

It turns out that convergence cannot be guaranteed with the choice (3.7), because a certain component pt,i can get arbitrarily close to zero, so that the i-th index is too rarely, or even never, picked1 . A possible remedy is to enforce a greedy sampling scheme or, as we will refer to it, a Doeblin-like condition on the transition kernel of the underlying Markov chain, see S. Meyn & Tweedie (2009):

∀i ∈ {1, . . . , n}, p t,i = ρν i + (1 -ρ)p t,i , (3.8)
where ν = (ν 1 , . . . , ν n ) is an arbitrary probability distribution satisfying ν i > 0 for 1 ≤ i ≤ n, and 0 < ρ ≤ 1. This condition has the following interpretation: p t is a mixture between two laws of probability and one of this law is independent from the past. The AS-SGD is summarized in Algorithm 1 below.

Algorithm 1 AS-SGD

Input: θ 0 , ρ, T , S, (γ t ) T -1 t=0 , ν Initialization: for i = 1 to n do Set g 0,t = ∇l(Z i , θ 0 ) end for A 0 = buildtree(g 0 ) for t = 0 to T -1 do
Define pt and p t according to (3.7) and (3.8)

I t+1 = sample(p t , S, ρ, ν, A t ) θ t+1 = Π K (θ t -γt nS i∈I t+1 ∇l(Z i ,θt) p t,i ) Update g t+1 according to (3.6) A t+1 = updatetree(A t , I t+1 , g t+1 ) end for Return θ T

Computationally Efficient Sampling

We point out that there is an additional computational price to pay when implementing NUS instead of uniform sampling. Given a non uniform distribution p = (p 1 , ..., p n ) on {1, . . . , n}, the simulation time needed to generate a r.v. with distribution p is larger than Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm in the case of uniform distribution. Indeed, one may resort to the inversion method (see [START_REF] Devroye | Non-uniform random variate generation[END_REF] for instance), which boils down to inserting an element in the sorted vector p = (p 1 , p 1 + p 2 , ..., p 1 + ...p n ) and requires log 2 (n) operations. Unfortunately, changing the i-th component of p (just like in the algorithm we propose) changes n -i components in the vector p. Our approach based on the notion of binary research tree is inspired from [START_REF] Devroye | Non-uniform random variate generation[END_REF] and overcome this difficulty.

Building / updating a tree. For simplicity, assume that n is even. Define a tree A t with n terminal leaves in correspondence with the indexes in {1, . . . , n} the weight g t,i is assigned to the leaf No. i. Each pair (2k + 1, 2(k + 1)) of adjacent terminal leaves, k ∈ {0, . . . , n/2}, admits a common ancestor, which the weight g t,2k+1 + g t,2(k+1) is assigned to. Continuing this way in a "bottom-up" fashion, the weight S t = i g t,i is assigned to the root node. The function buildtree used in Algorithm 1 generates such a tree from scratch and is used at the initialization t = 0. At step t + 1, g t+1 is essentially identical to g t except for a few S elements which have been updated. The tree A t+1 being close to A t , it does not have to be rebuilt from scratch. The routine updatetree given in section 3.7 provides a computationally efficient way to update the tree A t+1 from A t .

g 1 + ... + g 4 g 1 + g 2 g 1 g 2 g 3 + g 4 g 3 g 4 FIGURE 3.1: A binary tree for n = 4.
Simulating a random index. Suppose that we seek to generate a r.v., say i

(1)

t+1 , according to distribution pt . Using a r.v. U generated according to the uniform distribution on [0, 1], a path is generated from the root to one of the leaves by comparing U to successive thresholds. The generated variable i

(1) t+1 is defined as the index of the obtained leaf. The procedure is detailed in the subroutines sample and sample_tree (see section 3.7). Therefore updating an element i of the distribution is equivalent to update the path from i to the root of the tree and takes log 2 (n) operations, while sampling from our distribution is equivalent to follow a path from the root of the tree to one of its leaves. Table 3.1 summarizes the iteration complexity of the proposed method.

SGD AS -SGD Complexity Sc S(c + (2 -ρ) log 2 (n) )

Performance Analysis

We recall the transpose of a matrix A is denoted by A T . We start off by giving a rate bound analysis of the algorithm we proposed and then study the asymptotic behavior of the estimator θ t produced by our algorithms.

Preliminary results

In our analysis, we assume that Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm 37

Assumption 2. i) The function θ → L n (θ) is α-strongly convex, ii)
The minimizer θ * n of L n belongs to the interior of K.

The lemma stated below provides a bound on the Mean Square Error a t = E( θ t -θ * n 2 ), where θ t is generated by Algorithm 1. Its proof is strongly inspired by Bach & Moulines (2011a) andA .Nemirovski et al. (2009), where similar bounds are provided.

Lemma 3.1. Let Assumptions 1 and 2 hold true. Set γ t = γ 1 t -β where β ∈ (0, 1] and assume

γ 1 > β/(2α). For each t ∈ N * , a t ≤ Cγ t /ρ, (3.9)
where

C = max( 2B 2 ν γ 1 2αγ 1 -1 , a 1 γ 1 ) when β = 1 and C = max( B 2 ν γ 1 2α , a 1 γ 1 ) otherwise, with B ν = 1 Sn 2 n i=1 ν -1 i sup θ∈K ∇l(Z i , θ) 2 .
Remark 3.2. One mights easily check that choosing ν so as to minimize B ν would lead to take ν i ∝ sup θ∈K ∇l(Z i , θ) which is the sampling distribution proposed in [START_REF] Zhao | Stochastic Optimization with Importance Sampling[END_REF].

We emphasize the fact that sharper bounds on a t can be obtained. One could for instance easily generalize the approach of Bach & Moulines (2011a) which provides bounds on a t which are not only tighter but also valid under weaker assumptions on the step size. This would come at the price of a rather tedious bound in (3.9). As explained below, such an involved bound would actually be unnecessary for our purpose, and Lemma 3.1 is in fact sufficient to derive the main result of the next paragraph. It also admits a simpler proof provided in section 3.8.1. Before skipping to the main result, we first discuss the bound in (3.9). Lemma 3.1 establishes that the adaptive sampling scheme preserves the convergence rate in O(t -β ) obtained in the uniform sampling case. Nevertheless, Lemma 3.1 is merely a sanity check, because unfortunately, the bound does not suggest that adaptive non-uniform sampling generates a performance improvement: the minimum of the right hand side in (3.9) is attained for ρ = 1 which boils down to select a constant sampling probability. This is in contradiction with numerical results, which suggest on the opposite that strong benefits can be obtained by adaptively selecting the sampling probability. In order to obtain results that fully grasp the benefits of our adaptive sampling strategy, we investigate from now on the asymptotic regime t → ∞. The following Lemma provides an estimate of the value b t,i = E[ g i t -∇l(Z i , θ t-1 ) 2 ], which quantifies the mean square gap between the current (unobserved) gradients and the outdated gradients used to generate the next samples.

Lemma 3.3. Suppose that the assumptions of Lemma 3.1 hold true. For any

t ∈ N * , b t,i (2L i ) 2 2 β 1 -(1 -ρν i ) S C ρt β + o(t -β ) .
(3.10) Lemma 3.3 upper-bounds the gap between our approximation of the gradient and its true value.

The bound obtained depend on the batch-size and the step-size. We now state in the following section a Theorem Central Limit (TCL) on θ t -θ * n . The covariance matrix involved in the TCL depends on the sampling distribution and we show that its norm is minimal with the non uniform sampling strategy we propose.

Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm

Main results

We now turn to the analysis of the asymptotic behaviour and prove how our algorithm improve on SGD.

Assumption 3. The function L n is twice differentiable in a neighborhood of θ * n .

We introduce the sampling probability given by π * = ρν + (1 -ρ)π * , where,

π * i = ∇l(Z i , θ * n ) n j=1 ∇l(Z j , θ * n ) (3.11)
for any i = 1, . . . , n. We define

Q * = n i=1 ∇l(Z i , θ * n )∇l(Z i , θ * n ) T /(Sn 2 π * i ) and denote by H = ∇ 2 L n (θ * n ) the Hessian at point θ * .
Theorem 3.4. Suppose that Assumptions 1, 2 and 3 hold true and that the stepsize satisfies the condition of Lemma 3.1. Then the sequence (θ t -θ * n )/ √ γ t converges in distribution to a zero-mean Gaussian variable whose covariance matrix Σ = Σ(ρ, ν) is the solution to the following Lyapunov equation

ΣH + HΣ = Q * (if β < 1) Σ(I d + 2γ 1 H) + (I d + 2γ 1 H)Σ = 2γ 1 Q * (if β = 1) .
The proof is provided in section 3.8.3. The following Corollary is directly obtained by use of the second order delta-method [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF]. We denote by Tr(A) the trace of any square matrix A.

Corollary 3.5. Under the assumptions of Theorem 3.4, γ -1 t ( L n (θ t ) -L n (θ * )) converges in distribution to the r.v. V = (1/2)Z T Σ(ρ, ν) 1/2 HΣ(ρ, ν) 1/2 Z where Z is a Gaussian vector N (0, I d ). In addition, we have E(V ) = tr(HΣ(ρ, ν))/2. We now use Corollary 3.5 to compare our method with the best possible fixed choice of a sampling distribution. Note that the search for an optimal fixed distribution is also discussed in [START_REF] Clemencon | Scaling up m-estimation via sampling designs: The horvitz-thompson stochastic gradient descent[END_REF].

When the distribution is fixed, say to p, the asymptotic covariance of the normalized error is given by Σ(1, p) as defined in Theorem 3.4. Motivated by Corollary 3.5, we refer to the optimal fixed sampling distribution as the distribution p minimizing tr(HΣ(1, p)). It is straightforward to show that arg min

p tr(HΣ(1, p)) = π * ,
where π * is defined in (3.11). We also set σ 2 * = tr(HΣ(1, π * )). The following proposition follows from standard algebra and its proof is omitted due to the lack of space.

Proposition 1. Let Σ(ρ, ν) be the asymptotic covariance matrix defined in Theorem 3.4. Then,

σ 2 * ≤ tr(HΣ(ρ, ν)) ≤ σ 2 * (1 + Sρ/(1 -ρ)).
Proposition 1 implies that the asymptotic performance of the proposed AS-SGA can be made arbitrarily closed to the one associated with the best sampling distribution provided that ρ is chosen closed to zero. It is of course tempting to set ρ = 0 in (3.8) however in this case, the statement of Theorem 3.4 would be no longer valid.

Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm 39

Numerical experiments

We consider the l 2 -regularized logistic regression problem. Denoting by n the number of observations and by d the number of features, the optimization problem can be formulated as follows:

min θ∈R d 1 n n i=1 f (y i , x i , θ) + λ 2 θ 2 , (3.12)
where f (x, y, θ) = log(1 + exp(-yx T θ) the (y i ) n i=1 are in {-1, +1}, the (x i ) n i=1 are in R d and λ > 0 is a scalar. Note that for this problem one mights easily have access to the quantities L i and B i = sup θ∈R d ∇l(Z i , θ) .We used the benchmark dataset covtype with n = 581012, d = 54, λ = 1 √ n and γ t = γ 1 1+γ 1 λt as proposed in L. Bottou (2012) , where γ 1 is determined using a small sample of the training set. We considered the cases ν i = 1 n (ASGD), ν i ∼ L i (ASGD-Lip) and ν i ∼ B i (ASG-B), ran the algorithm for different values of the parameter ρ and compared it to the usual stochastic gradient descent with uniform sampling (SGD), lipschitz sampling (SGD-Lip) and upper-bound sampling (SGD-B) for the same parameters. In this scenario, C ∼ d , the computational times related to the SGD and the AS-SGD are comparable (see Table 1). Experiments suggest that choosing ν i ∼ L i leads to better performances and that using a small value of ρ leads to poor (respectively good) performance when θ 1 is far (respectively close) from θ * n . This suggests that a strategy could consist in running a classical SGD to get closer to θ * n and then run AS-SGD. One could also use the AS-SGD with a decreasing step-size policy. We dit not study these policies due to space limitations.

Conclusion

Motivated by recent work on SGD with non uniform probability, we introduced a novel implementation of the SGD algorithm with an adaptative sampling. We proposed a specific adaptive sampling scheme, depending on the past iterations in a Markovian manner that achieves low simulation cost. We also proposed a rigorous analysis to justify our approach and gave sufficient conditions to obtain theoretical guarantees.

Algorithms for Efficient NUS

In this section, we provide the main procedures called by Algorithm 1 to efficiently generate a collection of S i.i.d random indexes on {1, . . . , n}. If A is a tree with n leaves and e is a node, we denote by w(e) the weight of node e. The root of A is denoted by root(e). Father(e) is the father of a node e (and the empty set if e = root(A). Son(e) is the list of sons of e (and the emptyset if e is a leaf) and the elements of the list are referred to as son(e)[1], son(e) [2]. The functions isroot(e) and isleaf(e) return boolean values equal to one if e is the root or a leaf respectively. Finally, if e is a leaf, label(e) returns the index of the leaf e in {1, . . . , n}. The procedure buildtree is omitted but discussed in section 3.3.4. The algorithm Sample simply consists in writing the probability distribution as a mixture and is also omitted. ρ . We will now prove the lemma by induction. The property is checked for t = 1 by definition of C. Assume the result holds true for a t then, we have

Algorithm 2 Sample_tree

Input: A e =root(A) Draw U ∼ Uniform([0, w(e)]) repeat if U < w(son(e)[1]) then e ← son(e)[1]) else U ← U -w(son(e)[1]) e ← son(
a t+1 ((1 -2αγ t )γ t C + γ 2 t B 2 )/ρ and it is sufficient to show that (1 -2αγ t )γ t C + γ 2 t B 2 γ t+1 C which is equivalent to γ 2 t B 2 ν C(γ t+1 -γ t + 2αγ 2 t ). If β = 1, then, using 2αγ 1 > 1, we get: γ t+1 -γ t + 2αγ 2 t (2αγ 1 -1)γ 1 t(t+1))
> 0 and therefore

B 2 ν γ 2 t γ t+1 -γ t + 2αγ 2 t B 2 ν γ 1 t 2 t(t + 1) (2αγ 1 -1) 2B 2 ν γ 1 2αγ 1 -1 , (3.13) which gives the result. If 0 < β < 1, γ t+1 -γ t + 2αγ 2 t = γ 1 t+1 t -β u 1+β + 2α γ 2 1 t 2β 2α γ 2 1 t 2β - γ 1 β t β+1 γ 1 (2αγ 1 t 1-β -β) t β+1 > 0 since 2αγ 1 > β. We get B 2 γ 2 t γ t+1 -γ t + 2αγ 2 t B 2 γ 1 t 2β t β+1 (2αγ 1 t 1-β -β)) < B 2 2α ,
which concludes the proof.

Proof of Lemma 3.3

We consider a stepsize γ t such that

a t Cγ t+1 ρ , (3.14)
and we denote by I A the indicator function of any event A i.e., the r.v. equal to one on this event and to 0 elsewhere. Consider any index i and instant t and let A i k,t the event "the index i has not been picked since instant k". Using the Doeblin Condition we have:

P(A i k,t ) = E[I i t =0 ...I i k+1 =0 I i k =0 ] E[E[I i t =0 |F t-1 ]...I i k+1 =0 ] (1 -ρν i ) S E[I i t-1 =0 ...I i k+1 =0 ] (1 -ρν i ) S(t-k)
Conditionally upon A i k,t , we have g t,i = ∇l(Z i , θ k-1 ). Since (A i k,t ) k t is a partition of the state space, the law of total probability combined with Assumption 1 and the independence induced by the Doeblin Condition yields:

E[ g t,i -∇l(Z i , θ t-1 ) 2 ] L 2 i t k=1 E[ θ k-1 -
Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm

θ t-1 2 ](1 -ρν i ) S(t-k) . Using E[ θ k-1 -θ t-1 2 ] 2a t-1 + 2a k-1 4Cγ k /ρ leads to: E[ g t,i -∇l(Z i , θ t-1 ) 2 ] 4(L i ) 2 C ρ t k=1 γ k (1 -ρν i ) S(t-k) = 4(L i ) 2 C ρ t-1 k=0 γ t-k (1 -ρν i ) Sk .
For all 1 < t 0 < t, we have by splitting the sum in two terms:

t-1 k=0 γ t-k (1 -ρν i ) Sk γ t-t 0 1 -(1 -ρν i ) S + (1 -ρν i ) St 0 t-1 k=t 0 γ t-k .
Taking 2t 0 ∼ t for instance and using the classical integral test for convergence gives the result.

Proof of Theorem 3.4

The proof is prefaced by the following Lemma, whose proof is given at the end of this section.

Lemma 3.6. Under the Assumptions of Theorem 3.4, the sequence (θ t , p t ) converges to (θ * n , π * ) with probability one.

We now prove the main result. We use the decomposition

θ t+1 = θ t -γ t ∇ L n (θ t ) + γ t e t+1 + γ t η t+1
, where we set

D t+1 = (1/S) i∈I t+1 ∇l(Z i , θ t )/(np t,i ), e t+1 = ∇ L n (θ t ) -D t+1 , η t+1 = (Π K (φ t+1 ) -φ t+1 )/γ t , φ t+1 = θ t -γ t D t+1 .
We next check Conditions C1 to C4 in G. [START_REF] Yves F Atchade | On stochastic proximal gradient algorithms[END_REF]. Conditions C1 and C4 are immediate consequences of Assumptions 2 and 3. We check that e t satisfies Condition C2. First, by virtue of(3.5), (e t ) is a martingale increment sequence adapted to F t i.e., E(e t+1 |F t ) = 0.

Second, for all t ∈ N * and i = 1, . . . , n, p t,i ≥ ρν i > 0 with probability one. Therefore, it is straightforward to check that e t ≤ M a.s. for some constant M , which only depends on ρ, ν and B 1 , . . . , B N . Third, we analyze the asymptotic behaviour of the conditional covariance Q t = E(e t+1 e T t+1 |F t ). After some algebra, we obtain that

Q t = (1/n 2 ) n i=1 (1/(Sp t,i ))(∇l(Z i , θ t ) -∇ L n (θ t ))(∇l(Z i , θ t ) -∇ L n (θ t ))
T . Using Lemma 3.6 along with the continuity of ∇l(Z i , θ) for each i, we directly obtain that Q t tends to Q * with probability one. Condition C2 in G. [START_REF] Yves F Atchade | On stochastic proximal gradient algorithms[END_REF] is thus fulfilled. Turning to Condition C3, we shall prove that γ -1/2 t η t+1 converges to 0 in L 1 . Using Cauchy-Schwarz inequality:

E( γ -1/2 t η t+1 ) = E( γ -1/2 t η t+1 I η t+1 =0 ) ≤ E( η t+1 2 ) 1/2 γ -1 t P(η t+1 = 0) 1/2 . Defining u t = E( η t+1
2 ) and v t = P(η t+1 = 0), the above inequality reads

E( γ -1/2 t η t+1 ) ≤ √ u t v t γ t . (3.15)
We first analyze u t . Observe that

η t+1 2 = γ -2 t Π K (φ t+1 ) -φ t+1 2 ≤ 2γ -2 t ( Π K (φ t+1 ) -θ t 2 + φ t+1 -θ t 2 ) ≤ 4γ -2 t φ t+1 -θ t 2 = 4 D t+1 2 ,
Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic Gradient Descent Algorithm 43

where the last inequality is due to the non-expansiveness of the projection operator. Therefore Since p t,i admits a fixed deterministic lower-bound for each i and since the gradients ∇l(Z i , θ) are bounded on K, there exists a deterministic constant M such that D t+1 2 ≤ M . In particular, the sequence η t+1 2 is uniformly integrable. We now prove that η t+1 → 0 almost surely. Consider an > 0 such that the ball B(θ * , 2 ) of center θ * and radius is contained in K. By Lemma 3.6, for all ω on a set of probability one, there exists N (ω) such that θ t+1 (ω)θ * ≤ for all t ≥ N (ω). Using again that D t+1 (ω) is a bounded sequence and γ t → 0, it is clear that γ t D t+1 (ω) < for t large enough. Thus, for t large enough, φ t+1 (ω) ∈ K which implies that η t+1 (ω) = 0. Almost surely, the sequence η t+1 converges to zero. Putting all pieces together, η t+1 2 is a uniformly integrable sequence which tends a.s. to zero. As a consequence, u t = E( η t+1

2 ) tends to zero as t → ∞. We now analyze v t = P(η t+1 = 0). For > 0 be defined as above, note that the event {η t+1 = 0} is included in the event { φ t+1 -θ * ≥ }. By Markov inequality,

v t ≤ -2 E( φ t+1 -θ * 2 ) ≤ 2 -2 (E( φ t+1 -θ t 2 ) + E( θ t -θ * 2 )) ≤ 2 -2 γ 2 t E( D t+1 2 ) + Cγ t ρ ,
where we used Lemma 3.1 to obtain the last inequality. Recalling that D t+1 is bounded, it is clear that v t /γ t is a bounded sequence. Finally, by inequality (3.15), we conclude that [START_REF] Yves F Atchade | On stochastic proximal gradient algorithms[END_REF] is satisfied. This completes the proof of Theorem 3.4.

E( γ -1/2 t η t+1 ) tends to zero. Condition C3 in G.

Proof of Lemma 3.6

Almost sure convergence of θ t to θ * n directly follows from the Robbins-Siegmund Lemma and follows the same line of reasoning than L. Bottou (1998): it is therefore omitted. It remains to show that for each j = 1, . . . , n, pt,j → π * j a.s., where π * j = ∇l(Z j , θ * n ) / j ∇l(Z j , θ * n ) . Let A j denote the event that index j is picked infinitely often (i.e., there exists an infinite sequence (t k , n k ) k∈N on N * × {1, . . . , S} such that i

(n k ) t k = j).
As can be easily checked, the Doeblin condition p t,j ≥ ρν i ensures that A j has probability one. For a fixed ω ∈ A j and using the continuity of θ → ∇l(Z j , θ), we obtain that g t,j (ω) converges to ∇l(Z j , θ * n ) . As a consequence, pt,j (ω) → π * j (ω) and the result follows.

Introduction

In many situations, data are collected by means of a survey technique and the related weights (the true inclusion probabilities of the individual units forming the statistical population of interest) must be used by the statistician to compute unbiased statistics (see chapter 2). The availability of massive information in the Big Data era, which statistical procedures could theoretically now rely on, has motivated the recent development of parallelized/distributed variants of certain inference techniques or statistical learning algorithms, see [START_REF] Bekkerman | Scaling Up Machine Learning[END_REF], [START_REF] Mateos | Distributed sparse linear regression[END_REF], [START_REF] Navia-Vazquez | Distributed support vector machines[END_REF] or [START_REF] Bianchi | On-Line Learning Gossip Algorithm in Multi-Agent Systems with Local Decision Rules[END_REF] among others. It also strongly suggests to use sampling techniques as we did in the previous chapter, as a remedy to the apparent intractability of learning from datasets of explosive size, in order to break the current computational barriers, see [START_REF] Clémençon | Maximal Deviations of Incomplete U-statistics with Applications to Empirical Risk Sampling[END_REF] or [START_REF] Clémençon | Scaling-up Empirical Risk Minimization: Optimization of Incomplete U-statistics[END_REF].

It is the purpose of the present chapter to explore this approach further, by showing how to incorporate efficiently survey schemes into the SGD algorithm and highlight how it affects the statistical performance of the estimator produced. More precisely, the variant of the SGD method we propose involves a specific estimator of the gradient, that shall be referred to as the Horvitz-Thompson gradient estimator (HTGD estimator in short) throughout the chapter and accounts for the sampling design used to select the subsample for gradient evaluation at each iteration. For the estimator thus produced, consistency and asymptotic normality results describing its statistical performance are established under adequate assumptions on the first and second order inclusion probabilities. They reveal that accuracy may significantly increase, i.e. the asymptotic variance of the estimator produced by the HTGD procedure may be drastically reduced, when the inclusion probabilities of the survey design are picked adequately, depending on some supposedly available extra information, compared to a naive implementation with equal inclusion probabilities. This is thoroughly discussed in the particular case of the Poisson survey scheme. Although it is one of the simplest sampling designs, many more general survey schemes may be expressed as Poisson schemes conditioned upon specific events, see e.g. [START_REF] Berger | Asymptotic consistency under large entropy sampling designs with unequal probabilities[END_REF]. These theoretical results are also supported by strong empirical evidence. Many variants of the SGD technique, far too numerous to be listed here, have been 46Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation introduced these last few years in order to improve its scalability/speed,; attention should be paid to the fact that the analysis presented here only aims at sheding light on the impact of survey sampling on this technique, in its most generic form.

Theoretical Background and Preliminaries

As a first go, we start off with describing the mathematical setup and recalling key concepts in survey theory involved in the subsequent analysis. We recall that the indicator function of an event B is written I{B}. The square root of a symmetric semi-definite positive matrix B by B 1/2 .

Iterative M -Estimation and SGD Methods

Set two positive integers d and q. Let Z be an R d -valued random vector (r.v.) with unknown distribution P Z and Θ a compact subspace of R q equipped with the euclidean norm . . Consider a certain smooth loss function l : R d × Θ → R that is square P Z -integrable for any θ ∈ Θ. Given this theoretical framework, we are interested in solving the risk minimization problem min θ∈Θ L(θ), (4.1)

where

L : θ ∈ Θ → E[l(Z, θ)] ∈ R is the risk function with its empirical counterpart L n : θ ∈ Θ → 1 n n i=1 l(Z i , θ), (4.2)
based on the observation of n ≥ 1 independent copies Z 1 , . . . , Z N of Z (see Examples 4.1 and 4.2 below). As n → +∞, asymptotic properties of M -estimators, i.e. minimizers of L n (θ), have been extensively investigated, see van de Geer (2000) for instance.

Gradient descent. The popular approach to solve empirical risk minimization, introduced in chapter 3, consists in implementing variants of the standard gradient descent method, following the iterations

θ n (t + 1) = θ n (t) -γ t ∇ L n (θ t ), t ≥ 1, (4.3)
with an initial value θ n (0) arbitrarily chosen and a non-negative learning rate (step size or gain) γ t . The latter is taken such that +∞ t=1 γ t = +∞ and +∞ t=1 γ 2 (t) < +∞, see e.g. [START_REF] Bertsekas | Convex Analysis and Optimization[END_REF]. The true gradient is replaced by a counterpart computed from a subsample S ⊂ {1, . . . , n} of reduced size N n, so as to fulfill the computational constraints, and

Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation47 drawn at random (uniformly) among all possible subsets of same size at each iteration:

N : θ ∈ Θ → 1 N i∈S ∇l(Z i , θ). (4.4)
The variant of the SGD that we introduced in chapter 3 proposed to use more complex sampling schemes to speed-up the learning process. Unfortunately the results established were true conditionally upon the observations and only related to the empirical risk. We extend this analysis in the present chapter to the true risk by characterizing the asymptotic behaviour of L(θ n (t)) -L(θ * ). This analysis can be applied to the two following examples.

Example 4.1. (BINARY CLASSIFICATION) In the usual binary classification framework introduced in chapter 2, Y is a binary random output, taking its values in {-1, +1} say, and X is an input random vector valued in a high-dimensional space X , modeling some (hopefully) useful observation for predicting Y . Based on training data {(X 1 , Y 1 ), . . . , (X N , Y N )}, the goal is to build a prediction rule sign(h(X)), where h : X → R is some measurable function that minimizes the risk

L ϕ (h) = E [ϕ(-Y h(X))] .
Here, the expectation is taken over the unknown distribution of the random vector (X, Y ) and ϕ : R → [0, +∞) denotes a cost function, i.e. a measurable function such that ϕ(u) ≥ I{u ≥ 0} for any u ∈ R. For example, when ϕ is chosen as the convex function u ∈ R → (u + 1) 2 /2 ∈ R + , then the optimal decision function is given by h

* : x ∈ X → 2 P{Y = +1 | X = x} -1 ∈ [-1, 1] and the classification rule H * : x ∈ X → sign(h * (x)) ∈ {-1, +1}
coincides with the naive Bayes classifier. For simplicity, assume that ϕ is differentiable and that the decision function candidates h(x) belong to the parametric set {h(., θ) : θ ∈ Θ} of square integrable functions (with respect to the distribution of X) indexed by Θ ⊂ R q , q ≥ 1, such that θ → h(., θ) is differentiable. Finding the prediction rule with minimum risk amounts to solving (4.1) with Z = (X, Y ) and l(Z, θ) = ϕ(-Y h(X, θ)) for all θ ∈ Θ. In the ideal case where a standard gradient descent could be applied, a sequence θ t = (θ 1 (t), • • • , θ q (t)), t ≥ 1, would be iteratively generated using the update equation

θ(t + 1) = θ t + γ t E Y ∇h(X, θ t ) ϕ (-Y h(X, θ t ))
with learning rate γ t > 0. Naturally, as the distribution of (X, Y ) is unknown, the expectation involved in the t-th iteration cannot be computed and must be replaced by a statistical version:

1 n n i=1 Y i ∇h(X i , θ t ) ϕ (-Y i h(X i , θ t )),
in accordance with the Empirical Risk Minimization paradigm.

Example 4.2. (LOGISTIC REGRESSION) Consider the same probabilistic model as above, except that the goal pursued is to find θ ∈ Θ so as to minimize L n (θ) in (4.2) with Z i = (X i , Y i ) and l(Z i , θ) defined as

- Y i + 1 2 log exp(h(X i , θ)) 1 + exp(h(X i , θ)) + 1 -Y i 2 log 1 1 + exp(h(X i , θ))
for all i ∈ {1, . . . , n} and θ ∈ Θ. This is equivalent to maximizing the conditional loglikelihood given the X i 's related to the parametric logistic regression model:

P θ {Y = +1 | X} = exp(h(X, θ))/(1 + exp(h(X, θ))), θ ∈ Θ.

Survey Sampling and Horvitz-Thompson Estimation

We recall a few notations introduced in chapter 2 and 4. Let (Ω, A, P) be a probability space and n a positive integer. In the framework we consider throughout the chapter, it is assumed that Z 1 , . . . , Z n is a sample of i.i.d. random vectors defined on (Ω, A, P) and taking their values in R d . They are interpreted as independent copies of a generic r.v. Z observed on a finite population U n = {1, . . . , n}. A survey sample of the population is defined as a nonempty subset S ⊂ U n with cardinality N = N (S) less that n, selected at random according to a probability distribution R n on P(U n ), the power set of U n . The latter is called a sampling scheme/design/plan without replacement. We shall consider R n as a conditional distribution given the statistical population U n and the possible observations assigned to each of its units. In this setting, for any i ∈ U n , the probability that the unit i belongs to a random sample S drawn from such a R n is called the (first order) inclusion probability:

π i (R n ) = P Rn {i ∈ S}. We set π(R n ) = (π 1 (R n ), . . . , π N (R n )).
The second order inclusion probabilities are

π i,j (R n ) = P Rn {i ∈ S, j ∈ S}, for any (i, j) in U 2 n . In particular, π i,i (R n ) = π i (R n ).
When no confusion is possible, we shall omit to mention the dependence in R n when writing the first/second order probabilities of inclusion. The information related to the random sample S ⊂ U n is fully enclosed in the random vector n = ( 1 , . . . , n ) with components i = I{i ∈ S}, i ∈ U n . Given the statistical population, the conditional 1-d marginal distributions of the sampling scheme n are the Bernoulli distributions B(π i ) = π i δ 1 + (1 -π i )δ 0 , i ∈ U n , with δ x the Dirac mass at point x ∈ R. The conditional covariance matrix of the r.v. n is given by Γ n = {π i,j -π i π j } 1≤i,j≤n . Observe that n i=1 i = N (S), which can be fixed or random depending on R n .

Poisson schemes. One of the simplest survey designs is the Poisson scheme (without replacement), denoted by p n . For such a plan, conditioned upon the statistical population of interest, the i 's are independent Bernoulli random variables with parameters p 1 , . . . , p n in (0, 1]. Thus, the first order inclusion probabilities π i (p n ) = p i , i ∈ U n , fully characterize p n . The size N (S) of a sample S generated this way is random with mean n i=1 p i and goes to infinity as n → +∞ with probability one, provided that min 1≤i≤n p i remains bounded away from zero. In addition to its simplicity (regarding the procedure to select a sample thus distributed), the Poisson design plays a crucial role in sampling theory, insofar as it can be used to build a wide range of survey plans by conditioning arguments [START_REF] Hajek | Asymptotic theory of rejective sampling with varying probabilities from a finite population[END_REF]. For instance, a rejective sampling plan of fixed size N ≤ n corresponds to the distribution of a Poisson scheme n conditioned upon the event { n i=1 i = N }. One may refer to [START_REF] Cochran | Sampling techniques[END_REF], [START_REF] Deville | Réplications d'échantillons, demi-échantillons, Jackknife, bootstrap dans les sondages[END_REF] for accounts of survey sampling techniques and examples of designs to which the subsequent analysis applies.

Horvitz-Thompson estimators. We recall a few definitions and give the variance of Horvitz-Thompson estimators in the Poisson case. Suppose that independent random vectors Q 1 , . . . , Q n are observed on the population U n . They are viewed as copies of a generic r.v. Q taking its values in R q . A natural approach to estimate the total Q n = n i=1 Q i based on a sample S ⊂ U n generated from a survey design R n with positive (first order) inclusion probabilities {π i } 1≤i≤n consists in computing the Horvitz-Thompson estimator (HT estimator Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation49 in abbreviated form)

Q Rn = i∈S 1 π i Q i = n i=1 i π i Q i . (4.5)
Given the whole statistical population Q 1 , . . . , Q N , the HT estimator is an unbiased estimate of the total:

E (Q Rn | Q 1 , . . . , Q n ) = Q N almost-surely.
Its conditional variance is given by

V (Q Rn | Q 1 , . . . , Q n ) = n i,j=1 π i,j -π i π j π i π j Q i Q T j .
In particular, when the survey design is a Poisson plan P n with positive probabilities p 1 , . . . , p n , this turns into

V (Q Pn | Q 1 , . . . , Q n ) = n i=1 1 -p i p i Q i Q T i . (4.6) Remark 4.1. (AUXILIARY INFORMATION)
In practice, the first order inclusion probabilities are defined as a function of an auxiliary variable, say W taking its values in R d , d ≥ 1, which is observed on the entire population. Specifically, a link function π : R d → (0, 1] is chosen so that π i = π(W i ) for all i ∈ U n . When π(W ) and Q are dependent, proceeding this way may help us select more informative samples and consequently yield estimators with reduced variance. A more detailed discussion on the use of auxiliary information in the present context can be found in subsection 4.4.1.

Going back to the SGD problem, the Horvitz-Thompson estimator of the gradient ∇ L n (θ) based on a survey sample S drawn within the population U n = {1, . . . , n} from a design R n with vector of (first order) inclusion probabilities π n = (π 1 , . . . , π N ) and inclusion vector

n = ( 1 , . . . , n ) is Rn (θ) = 1 n i∈S 1 π i ∇l(Z i , θ) = 1 n n i=1 i π i ∇l(Z i , θ), θ ∈ Θ. (4.7)
As pointed out in Remark 4.1 and in section 3.3.2, this estimator would be most efficient if each π i was strongly correlated with the corresponding ∇l(Z i , θ), i ∈ U n . This suggests to devise a procedure where the survey design used to estimate the gradient may change at each step, as in the HTGD algorithm described in the next section. For instance, one could stipulate the availability of extra information W 1 , . . . , W N and assume the existence of a link function

π : W × Θ → (0, 1] such that π i = π(W i , θ) for all i ∈ U n .
Of course, such an approach would be beneficial only if the cost of the computation of the weight π(W i , θ) is smaller than that of the gradient ∇l(Z i , θ). As shall be seen in section 4.6, this happens to be the case in many situations encountered in practice.

The Horvitz-Thompson Gradient Descent

This section presents, in full generality, the variant of the SGD method we propose. It can be implemented in particular when some extra information about the target (the gradient vector field in the present case) is available, allowing hopefully for picking a sample yielding a more accurate estimation of the (true) gradient than that obtained by means of a sample chosen completely at random. Several tuning parameters must be picked by the user, including the 2. (ITERATIONS.) For t = 0, . . . , T (a) Draw a survey sample from U n = {1, . . . , n}, described by the inclusion vector

(t) n = (t) 1 , . . . , (t) n
, according to

R n = R n (θ n (t)) with inclusion probabili- ties π i (θ n (t)) for i ∈ U n .
(b) Compute the HT gradient estimate at θ n (t)

Rn (θ n (t)) := 1 n n i=1 (t) i π i (θ n (t)) ∇l(Z i , θ n (t)).
(c) Update the estimator

θ n (t + 1) = θ n (t) -γ t Rn (θ n (t)).
(OUTPUT.) The HTGD estimator θ n (T ).

Conditioned upon the data (Z 1 , W 1 ), . . . , (Z n , W n ), the asymptotic accuracy of the estimator or decision rule produced by the algorithm above as the number of iterations T tends to infinity is investigated in the next section under specific assumptions. Beyond consistency, special attention is paid to the issue of choosing properly the sampling plans R n (θ) so as to minimize the asymptotic variance of the estimator θ n (T ) or that of its empirical risk.

Conditional Asymptotic Analysis -Main Results

This section is dedicated to the analysis of the performance of the HTGD method, conditioned upon the observed population and under adequate constraints related to the (expected) size of the survey samples. We first discuss the case of Poisson survey schemes and next investigate how to establish limit results in a more general framework. They are similar to the results established in section 3.4.2 and serve as a preliminary for our main results. All expectations are taken conditionally upon the observations in this section.

Poisson Schemes with Unequal Inclusion Probabilities

Here we will recall a result establish in chapter 3. p n = (p 1 , . . . , p n ). Then, Eq. (4.6) implies

E Pn (θ) -∇ L n (θ) 2 | Z 1 , . . . , Z n = 1 n 2 n i=1 1 -p i p i ∇l(Z i , θ) 2 .
Searching for the parameter p n such that the L 2 distance between the empirical gradient evaluated at θ and the HT version given Z 1 , . . . , Z n is minimum under the constraint that the expected sample size is equal to N ≤ n yields the optimization problem

min pn∈(0,1] n n i=1 1 -p i p i ∇l(Z i , θ) 2 subject to n i=1 p i = N. (4.8)
Suppose that P{∇l(Z, θ) = 0} = 0 for all θ ∈ Θ; this is true in particular when the set {z ∈ R d : ∇l(z, θ) = 0} has finite cardinality and the distribution of Z is absolutely continuous with respect to the Lebesgue measure. Then we have ∇l(Z i , θ) > 0 with probability one for all i ∈ U n and θ ∈ Θ. As can be shown by means of the Lagrange multipliers method, in this setting the solution corresponds to weights being proportional to the values taken by the norm of the gradient just like in section 3.3.2.

p i (θ) := N ∇l(Z i , θ) n j=1 ∇l(Z j , θ)
, provided that the following condition is fulfilled:

p i (θ) ≤ 1 for all i ∈ U n . (4.9)
A straightforward application of Hoeffding's inequality shows that if

ε := E [ ∇l(Z, θ) ] sup z∈R d ∇l(z, θ) - N n ∈   0, E [ ∇l(Z, θ) ] sup z∈R d ∇l(z, θ)    ,
then condition (4.9) is satisfied with probability larger than 1 -exp -2nε 2 .

Remark 4.2. (ON THE SATURATION OF THE LINEAR CONSTRAINTS) When the latter condition is not satisfied, some of the conditions p i (θ) ≤ 1 are saturated and the solution of (4.8) is given by the Karush-Kuhn-Tucker method. Since the objective function is strictly convex and the constraints are affine, the following conditions, related to the Lagrangian

n i=1 1 -p i p i ∇l(Z i , θ) 2 + λ n i=1 p i -N + n i=1 µ i (p i -1),
are necessary and sufficient: (i)

n i=1 p i = N and for all i ∈ U n (ii) 0 < p i ≤ 1, (iii) ∇l(Z i , θ) 2 p 2 i = λ + µ i , (iv) µ i ≥ 0, (v) µ i (p i -1) = 0.
Denoting by m < N the number of components of the solution p n that are equal to 1 and by σ a permutation of U n such that ∇l(Z σ(1) , θ) ≤ . . . ≤ ∇l(Z σ(n) , θ) , the constraint (i) can be rewritten as

N = m + n-m i=1 ∇l(Z σ(i) , θ) / √ λ, so that p σ(i) = (N -m) ∇l(Z σ(i) , θ) / n-m j=1 ∇l(Z σ(j) , θ) for i ≤ n-m and p σ(i) = 1 for i ≥ n-m+1.
52Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation However, selecting a sample distributed this way requires to know the full statistical population ∇l(Z i , θ). In practice, one may consider situations where the weights are defined by means of a link function π(W, θ) and auxiliary variables W 1 , . . . , W n such that the inclusion probabilities are correlated with their corresponding gradient, as suggested previously. Observe in addition that the goal pursued here is not to estimate the gradient but to implement a stochastic gradient descent involving an expected number of terms fixed in advance, while yielding results close to those that would be obtained by means of a gradient descent algorithm with mean field (1/n) n i=1 ∇l(Z i , θ) based on the whole dataset. However, as shall be seen in the subsequent analysis (see Proposition 4.6), in general these two problems do not share the same solution from the angle embraced in this article.

In the next subsection, assumptions on the survey design under which the HTGD method yields accurate asymptotic results, surpassing those obtained with all equal inclusion probabilities (i.e. π i = N/n for all i ∈ U n ), are exhibited.

Limit Theorems -Conditional Consistency and Asymptotic Normality

We now consider a collection of general (i.e. not necessarily Poisson) sampling schemes {R n (θ)} θ∈Θ with positive first order inclusion probabilities {π n (θ)} θ∈Θ . Conditioned upon the data

D n = {Z 1 , . . . , Z n } (or D n = {(Z 1 , W 1 ), . . . , (Z n , W n )} in
the presence of extra variables, cf Remark 4.1) available in the population U n = {1, . . . , n}, we study the asymptotic properties of the M -estimator produced by the HTGD algorithm. The limit results stated below essentially rely on the fact that the HT estimator (4.7) of the gradient of the empirical risk is unbiased. Reduction of the asymptotic variance of θ n (T ) and L n (θ n (T )) will be investigated in the Poisson case in the next subsection. The asymptotic analysis also involves the regularity conditions listed below, which are classically required in stochastic approximation.

Assumption 4. The conditions below hold true.

For any

z ∈ R d , the mapping θ ∈ Θ → l(z, θ) is of class C 1 .
2. For any compact set K ⊂ Θ, we have with probability one: ∀i ∈ U n ,

sup θ∈K ∇l(Z i , θ) π i (θ) < +∞.
3. The set of stationary points

L n = {θ ∈ Θ : N (θ) = 0} is of finite cardinality.
Theorem 4.3. (CONDITIONAL CONSISTENCY) Suppose that Assumption 4 is fulfilled and that

• the learning rate decays to 0 so that t≥1 γ t = +∞ and t≥0 γ 2 (t) < +∞,

• the HTGD algorithm is stable, i.e. there exists a compact set K 0 ⊂ R q such that θ n (t) ∈ K 0 for all t ≥ 0.

Then, conditioned upon the data D n , the sequence {θ n (t)} t≥0 converges to an element of the set L n with probability one as t → +∞.

The stability condition is generally difficult to check. In practice, one may guarantee it by confining the sequence to a compact set fixed in advance and using a projected version of Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation53 the algorithm above. For simplicity, the present study is restricted to the simplest framework for stochastic gradient descent and we refer to [START_REF] Kushner | Stochastic Approximation and Recursive Algorithms and Applications[END_REF] or [START_REF] Borkar | Stochastic Approximation: a Dynamical Systems Viewpoint[END_REF] (see section 5.4 therein) for further details.

Consider a stationary point θ * n ∈ L n . The following local assumptions are also required to establish asymptotic normality results conditioned upon the event

E(θ * n ) = lim t→+∞ θ n (t) = θ * n .
Assumption 5. The conditions below hold true.

1. There exists a neighbourhood V of θ * n such that for all z ∈ R d , the mapping θ ∈ Θ → l(z, θ) is of class C 2 on V.

The Hessian matrix

H n = ∇ 2 L n (θ *
n ) is a stable q × q positive-definite matrix, i.e. its smallest eigenvalue l is positive.

3. For all (i, j) ∈ U 2 n , the mapping θ ∈ V → π i,j (θ) is continuous.

Under these assumptions, and similarly to the results establish in section 3.4.2 we have the following TCL.

Theorem 4.4. (CONDITIONAL CENTRAL LIMIT THEOREM) Suppose that Assumptions 4-5 are fulfilled and that γ t = γ 0 t -α for some constants α ∈ (1/2, 1] and γ 0 > 0. When α = 1, take γ 0 > 1/(2l) and set η := 1/(2γ 0 ); set η := 0 otherwise. Given the observations D n and conditioned upon the event E(θ * n ), we have the convergence in distribution as t → +∞

1/γ t (θ n (t) -θ * n ) ⇒ N (0, Σ πn ),
where the asymptotic covariance matrix Σ πn is the unique solution of the Lyapunov equation

H n Σ + ΣH n + 2ηΣ = Γ * n , (4.10) with Γ * n = Γ n (θ * n ) and, for all θ ∈ Θ, Γ n (θ) = 1 n 2 n i,j=1 π i,j (θ) π i (θ)π j (θ) -1 ∇l(Z i , θ) ∇l(Z j , θ) T . (4.11)
The result stated below provides the asymptotic conditional distribution of the error. Because it is a direct application of the second order delta method, the proof is omitted.

Corollary 4.5. (ERROR RATE) Under the hypotheses of Theorem 4.4, given the observations D n and conditioned upon the event E(θ * n ), as t → +∞ we have the convergence in distribution towards a non-central chi-square distribution:

1/γ t L n (θ n (t)) -L n (θ * n ) ⇒ 1 2 U T Σ 1/2 πn H n Σ 1/2 πn U,
where U is a q-dimensional standard Gaussian random vector.

Asymptotic Covariance Optimization in the Poisson Case

Now that the limit behavior of the solution produced by the HTGD algorithm has been described for general collections of survey designs R = {R n (θ)} θ∈Θ of fixed expected sample 54Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation size, we turn to the problem of finding survey plans yielding estimates with best accuracy. Formulating this objective in a quantitative manner, this boils down to finding R so as to minimize the asymptotic covariance matrix summary Σ 1/2 πn , for an appropriately chosen norm . on the space M q (R) of q × q matrices with real entries for instance, when it comes to estimate θ * n . In order to get a natural summary of the asymptotic variability, we consider here the Frobenius (Hilbert-Schmidt) norm, i.e. A = T r(A T A) = ( i,j a 2 i,j ) 1/2 for any A = (a i,j ) ∈ M q (R). For simplicity's sake, we focus on Poisson schemes and consider the case where η = 0 in Theorem 4.4. Notice that the cross terms (i = j) in Eq. (4.11), i.e. the U -statistic part of the conditional asymptotic variance, vanish in the Poisson case. The following result exhibits an optimal collection of Poisson schemes among those with N as expected sizes, in the sense that it yields an HTGD estimator with an asymptotic covariance of square root with minimum Frobenius norm. We point out that it is generally different from that considered in subsection 4.4.1, revealing the difference between the issue of estimating the empirical gradient accurately by means of a Poisson Scheme and that of optimizing the HTGD procedure.

Proposition 4.6. (OPTIMALITY) Consider the same assumptions as in Theorem 5 in the case where η = 0 and suppose that

N ≤ inf θ∈Θ n i=1 G n ∇l(Z i , θ) max 1≤i≤n G n ∇l(Z i , θ) , (4.12) with G n := H -1/2 n
. Then, the collection of Poisson schemes with positive inclusion probabilities {p * n (θ)} θ∈Θ defined for all θ ∈ Θ and i ∈ U n by

p * i (θ) = N G n ∇l(Z i , θ) n j=1 G n ∇l(Z j , θ)
is a solution of the minimization problem

min pn={pn(θ)} θ∈Θ Σ 1/2 pn subject to n i=1 p i (θ) = N.
In addition, we have

Σ 1/2 p * n 2 = 1 2    1 n 1 n n i=1 G n ∇l(Z i , θ * n ) 2 - 1 n 2 n i=1 G n ∇l(Z i , θ * n ) 2    .
Of course, the optimal solution exhibited in the result stated above is completely useless from a practical perspective, since the matrix H n is unknown in general and the computation of the values taken by the gradient at each point Z i is precisely what we are trying to avoid in order to reduce the computational cost of the GD procedure. However, we show in the next section that choosing inclusion probabilities positively correlated with the p * i (θ)'s is actually sufficient to reduce asymptotic variability (compared to the situation where equal inclusion probabilities are used). In addition, as illustrated by the two easily generalizable examples described in section 4.6, such a sampling strategy can be implemented in many situations.

Notice finally that, if we consider the asymptotic excess of empirical risk of the estimate L n (θ n (T )) -L n (θ * n ) rather than the asymptotic variance of the estimate itself, the survey Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation55 design R must be picked in order to minimize the quantity

E U T Σ 1/2 πn H n Σ 1/2 πn U | D n = E H 1/2 n Σ 1/2 πn U T H 1/2 n Σ 1/2 πn U | D n = T r (H n Σ πn ) ,
using the fact that U ∼ N (0, I q ) is chosen independent from D n here. Observing that

H n Σ πn + Σ πn H n = Γ * n in the case η = 0, we have T r (H n Σ πn ) = 1 2 T r(Γ * n ) = E[|| Pn (θ * n )|| 2 ]. (4.13)
Now, since we have in the Poisson case

T r(Γ * n ) = 1 n 2 n i=1 1 p i (θ * n ) -1 ∇l(Z i , θ * n ) 2 ,
the optimal Poisson scheme regarding this alternative criterion generally differs from that involved in Proposition 4.6 and boils down to that introduced in subsection 4.4.1 for optimal Horvitz-Thompson estimation of the gradient.

Extensions to More General Poisson Survey Designs

In this subsection, we still consider Poisson schemes and the case η = 0 for simplicity and now place ourselves in the situation where the information at disposal consists of a collection of i.i.d. random pairs

(Z 1 , W 1 ), . . . , (Z n , W n ) valued in R d × R d . Take a link function p : R d × Θ → (0, 1] such that θ ∈ Θ → p(w, θ
) is continuous for all w ∈ R d , then choose an expected sample size N ∈ {1, . . . , n} that satisfies

N ≤ inf θ∈Θ n i=1 p(W i , θ) max 1≤i≤n p(W i , θ) and define p i (θ) = N p(W i , θ) n j=1 p(W j , θ) , for all (i, θ) ∈ U n × Θ. (4.14)
Observe that for all θ ∈ Θ we have n i=1 p i (θ) = N and p i (θ) ∈ (0, 1] for all i ∈ U n . The computational cost of the inclusion probability p(W i , θ) is assumed to be much smaller than that of ∇l(Z i , θ) (see the examples in section 4.6) for all (i, θ) ∈ U n × Θ. The assumption introduced below involves the empirical covariance c n (θ) between G n ∇l(Z, θ) 2 /p(W, θ) and p(W, θ), for θ ∈ Θ:

c n (θ) = 1 n n i=1 G n ∇l(Z i , θ) 2 1 - 1 n n j=1 p(W j , θ) p(W i , θ) .
Assumption 6. The link function p(w, θ) fulfills the following condition:

c n (θ * n ) > 0.
The result stated below reveals to which extent sampling with inclusion probabilities defined by some appropriate link function may improve upon sampling with equal inclusion probabilities, p i = N/n for 1 ≤ i ≤ N , when implementing stochastic gradient descent. Namely, 56Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation the accuracy of the HTGD gets closer and closer to the optimum, as the empirical covariance c n (θ * ) increases to its maximum. Notice that in the case where inclusion probabilities are all equal, we have c n ≡ 0.

Proposition 4.7. Let N be fixed. Suppose that the collection of Poisson designs p with expected sizes N is defined by a link function p(w, θ) satisfying Assumption 6. Then, when Theorem 4.4 applies, we have Σ

1/2 pn ?F < Σ 1/2 pn ?F , as well as 0 ≤ Σ 1/2 pn 2 ?F -Σ 1/2 p * n 2 ?F = 1 2N σ 2 n (θ * n ) -c n (θ * n ) ,
where

σ 2 n (θ) = 1 n n i=1 G n ∇ψ(Z i , θ) 2 - 1 n n i=1 G n ∇ψ(Z i , θ) 2 denotes the empirical variance of the r.v. E ∇ 2 ψ(Z, θ) -1/2 ∇ψ(Z, θ) , θ ∈ Θ.
As illustrated by the easily generalizable examples provided in the next section, one may generally find link functions fulfilling Assumption 6 without great effort, permitting to gain in accuracy from the implementation of the HTGD algorithm.

Unconditional Asymptotic Analysis

Building upon the results of the previous section, we now investigate the behaviour of the HTGD estimator as n, N and t simultaneously tend to +∞ at appropriate rates. For the sake of simplicity we assume in this section that the minimizer θ * over the supposedly compact parameter space Θ is unique, as well as the empirical minimizer θ * n with probability one. All the results stated in this section can be directly extended to more general cases.

The assumption below, related to the asymptotic behavior of (4.11), is involved in the subsequent unconditional analysis.

Assumption 7. As both n and N tend to ∞, N Γ * n converges in probability toward a positive semi-definite matrix Γ * .

Although this condition may seem strong at first glance, it is actually fulfilled in several important situations. In particular, the following proposition shows it holds true in the Poisson case under weak conditions. Proposition 4.8. Suppose that the survey schemes are of Poisson type with link functions p(., θ) : R d → (0, 1), θ ∈ Θ, based on the auxiliary information W observed on the statistical population. Assume also that the following conditions are fulfilled.

(i) We have θ * n → θ * with probability one, as n → +∞.

(ii) The expected size N tend to infinity as n → ∞, so that

N n → c 0 ∈ [0, 1].
(iii) For all θ ∈ Θ:

E [p(W, θ)] < +∞ and E 1 p(W, θ) ∇l(Z, θ)∇θl(Z, θ) T < +∞.
Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation57

(iv) The essential supremum below is finite:

sup θ∈Θ sup x > 0 : P ∇l(Z, θ) p(W, θ) > x = 0 < +∞.
(v) We have: p = inf w,θ p(w, θ) > 0 and p = sup w,θ p(w, θ) < ∞.

Then, the quantities (4.14) define the inclusion probabilities of a Poisson scheme with probability one, as soon as N ≤ np/p. In addition, assumption 7 is fulfilled with

Γ * = E [p(W, θ * )] E 1 p(W, θ * ) ∇l(Z, θ * )∇l(Z, θ * ) T .
We are now ready to state the main result of this section, which illustrates the trade-off between (asymptotic) generalization and optimization errors, ruled by the limit behavior of nγ t /N .

Theorem 4.9. Suppose that Assumptions 4, 5, 7 are fulfilled and that the rate γ t satisfies the condition of Theorem 4.4 with α < 1 (and thus η = 0). Assume that the symmetric positive semi-definite matrix

H * = E[∇ 2 l(Z, θ * )] is invertible, set Λ * = (H * ) -1 E[∇l(Z, θ * )∇l(Z, θ * ) T ](H * ) -1
and denote by Σ * the unique solution of the Lyapunov equation: H * Σ + ΣH * = Γ * . The assertions below hold true. (ii) If lim n,N,t→+∞ n N γ t = 0, then we have the convergence in distribution:

lim n,N,t→+∞ √ n (θ n (t) -θ * ) = N (0, Λ * ).
(iii) If lim n,N,t→+∞ n N γ t = c > 0, then we have the convergence in distribution:

lim n,N →∞ lim t→∞ √ n (θ n (t) -θ * ) = N (0, Λ * + cΣ * ).
We first point out that, in contrast to case (ii) where they can be swapped, the limits involved in cases (i) and (iii) must be taken sequentially: assertion (i) (respectively, assertion (ii)) describes the asymptotic regime for large values of n and N , the number t of HTGD iterations is such that 1/γ t n/N (respectively, such that 1/γ t ∼ n/N ). In the asymptotic regime (i), corresponding to the 'Big Data' setup, the optimization error rules the limit behavior of the HTGD estimator, whereas the estimation error determines the asymptotic covariance structure in case (ii). Case (iii) corresponds to the situation where both terms impact the limit distribution. Just like in Corollary 4.5 for the conditional analysis, the asymptotic distribution of the error can be straightforwardly deduced from the Central Limit Theorem above by means of the delta method.

lim n,N →∞ lim t→∞ N γ t (L(θ n (t)) -L(θ * )) = 1 2 U T Σ * 1/2 H * Σ * 1/2 U.
(ii) If lim n,N,t→+∞ n N γ t = 0, then we have the convergence in distribution:

lim n,N,t→+∞ n (L(θ n (t)) -L(θ * )) = 1 2 U T Λ * 1/2 H * Λ * 1/2 U.
(iii) If lim n,t→+∞ n N γ t = c > 0, then we have the convergence in distribution:

lim n,N →∞ lim t→∞ n (L(θ n (t)) -L(θ * )) = 1 2 U T (Λ * + cΣ * ) 1/2 H * (Λ * + cΣ * ) 1/2 U.

Illustrative Numerical Experiments

For illustration purpose, this section shows how the results previously established apply to two problems by means of simulation experiments. For both examples, the performance of the HTGD algorithm is compared with that of a basic SGD strategy with the same (mean) sample size.

Linear logistic regression

Consider the linear logistic regression model corresponding to Example 4.2 with θ = (α, β) ∈ R × R d and h(x, θ) = α + β T x for all x ∈ R d . Let X be a low dimensional marginal vector of the input r.v. X, of dimension d d say, so that one may write X = (X , X ) as well as β = (β , β ) in a similar manner. The problem of fitting the parameter θ through conditional MLE corresponds to the case l((x, y), θ) = -log e α+β T x (y + 1)/2 + (1 -y)/2 1 + e α+β T x .

We propose to implement the HTGD with p((x , y), θ) ∝ ∇l ((X, Y ), θ) as link function, where l ((x, y), θ) = -log e α+β T x (y + 1)/2 + (1 -y)/2

1 + e α+β T x .

In order to illustrate the advantages of the HTGD technique for logistic regression, we considered the toy numerical model with parameters d = 11 and θ = (α, β 1 , . . . , β 10 ) = (-9, 0, 3, -9, 4, -9, 15, 0, -7, 1, 0), the 10 input variables being independent, uniformly distributed on (0, 1). The maximum likelihood estimators of θ were computed using the HTGD and SGD (mini-batch) . In order to compare them, the same number of iterations was chosen in each situation and a learning rate proportionnal to 1/ √ t was considered. As a first go, we So as to account for the randomness due to the data, we then simulated 50 samples according to the model for two population sizes, n = 500 and n = 1000. For both the HTGD and the mini-batch SGD algorithms, a sub-sample size of 20 was chosen. As shown in Table 4.1, the HTGD seems to be more robust to data randomness than SGD and GD. It is not surprising, since the sampling phase selects the most informative observations relative to the gradient descent, which makes HTGD less sensitive to the possible noise. It also provides more precise estimates, as illustrated by the results in

The Symmetric Model

Consider now an i.i.d. sample (X 1 , X 2 , . . . , X n ) drawn from an unknown probability distribution on R d , supposed to belong to the semi-parametric collection {P θ,f , θ ∈ Θ}, Θ ⊂ R d , dominated by some σ-finite measure λ. The related densities are denoted by f (x -θ), where θ ∈ Θ is a location parameter and a f (x) a (twice differentiable) density, symmetric about 0, i.e.f (x) = f (-x). The density f is unknown in practice and may be multimodal. For simplicity, we assume here that Θ ⊂ R but similar arguments can be developed when d > 1.

For such a general semi-parametric model, it is well-known that neither the sample mean nor the median (if, for instance, the distribution does not weight the singleton {0}) are good candidates for estimating the location parameter θ. In the semiparametric literature this model is referred to as the symmetric model, see [START_REF] Bickel | Efficient and Adaptive Estimation for Semiparametric Models[END_REF]. It is known that the tangent space (i.e. the set of scores) with respect to the parameter of interest θ and that with respect to the nuisance parameter are orthogonal. The global tangent space at P θ,f is given by

T L [P θ,f , P] = c f (x -θ) f (x -θ) + h(x -θ); c ∈ R, h ∈ Ṗ2 ,
where Ṗ2 is the tangent space with respect to the nuisance parameter:

Ṗ2 = h : E P θ,f [h(X)] = 0, h(x) = h(-x) and E P θ,f [h 2 (X)] < ∞ .
Orthogonality simply results from the fact that f (x) is an odd function and implies that the parameter θ can be adaptively estimated, as if the density f (x) was known, refer to [START_REF] Bickel | Efficient and Adaptive Estimation for Semiparametric Models[END_REF] for more details. In practice f (x) is estimated by means of some symmetrized kernel density estimator. Given a Parzen-Rosenblatt kernel K(x) (e.g. a Gaussian kernel) for instance, consider the estimate

f θ,n (x) = 1 N h n n i=1 K x -(X i -θ) h n ,
where h N > 0 is the smoothing bandwidth, and form its symetrized version (which is an even function)

f θ,n (x) = 1 2 f θ,n (x) + f θ,n (-x) .
Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation61

The related score is given by

s n (x, θ) = d dθ f θ,n (x)/ f θ,n (x).
In order to perform maximum likelihood estimation approximately, one can try to implement a gradient descent method to get an efficient estimator of θ. For instance, for a reasonable sample size n, it is possible to show that, starting for instance from the empirical median θ 0 with an adequate choice of the rate γ t , the sequence

θ n (t) = θ(t -1) + γ t 1 n n j=1
s n (X j -θ(t -1), θ(t -1))

converges to the true MLE. The complexity of this algorithm is typically of order 2T × n 2 if T ≥ 1 is the number of iterations, due the tedious computations to evaluate the kernel density estimator (and its derivatives) at all points X i -θ(t -1). It is thus relevant in this case to try to reduce it by means of (Poisson) survey sampling. The iterations of such an algorithm would be then of the form

θ n (t) = θ(t -1) + γ t 1 n n j=1 ε j p j s n (X j -θ(t -1), θ(t -1)), n j=1 p j = N.
As shown in section 4.4.3, the optimal choice would be to choose p j proportional to | s n (X jθ(t -1), θ(t -1))| at the t-th iteration:

p * j θ(t -1) = N | s n (X j -θ(t -1), θ(t -1))| n i=1 | s n (X j -θ(t -1), θ(t -1))| . (4.15)
Unfortunately this is not possible because s is unknown and replacing s(x-θ) by s n (x-θ(t-1), θ(t -1)) in (4.15) yields obvious computational difficulties. For this reason, we suggest to use the (much simpler) Poisson weights:

p j (θ) = N |X j -θ|/ n j=1 |X j -θ|.
Fig. 4.4 depicts the performance of the HTGD algorithm when θ = 0 and f (x) is a balanced mixture of two Gaussian densities with means 4 and -4 respectively and same variance σ 2 = 1, compared to that of the usual SGD method. Based on a population sample of size n = 1000, the HTGD and SGD methods have been implemented with N = 10 and T = 3000 iterations, whereas 30 iterations have been made for the basic GD procedure (with N = n = 1000) so that the number of gradient computations is of the same order for each method. For each instance of the algorithms we took θ 0 equal to the median of the population.

Conclusion

In this article, we have shown how survey sampling can be used in order to improve the accuracy of the stochastic gradient descent method in M -estimation, while preserving the complexity of the procedure. Beyond theoretical limit results, the approach we promote is illustrated by promising numerical experiments. Whereas massively parallelized/distributed approaches combined with random data splitting are now receiving much attention in the Big Data context, the present chapter explores a possible alternative way of scaling up statistical learning methods, based on gradient descent techniques.

Technical Proofs

Proof of Theorem 4.3

The conditional consistency of the HTGD algorithm described in Section 4.3 is obtained by applying Theorem 13 in [START_REF] Delyon | Stochastic Approximation with Decreasing Gain: Convergence and Asymptotic Theory[END_REF] (or Theorem 2.2 in Chapter 5 of [START_REF] Kushner | Stochastic Approximation and Recursive Algorithms and Applications[END_REF] among other references). Specifically, it states that if the following conditions are fulfilled, then θ n (t) converges as t → +∞ to some θ * n ∈ L n with probability 1:

• t≥1 γ t = +∞ and t≥0 γ 2 (t) = +∞, which was assumed,

• θ n (t) remains in a compact subset of Θ for all t ≥ 0, which was also assumed,

• θ ∈ Θ → -L n (θ) and θ ∈ Θ → ∇ L n (θ) are continuous, which is guaranteed by Assumption 4-(i),
• L n is finite, which corresponds to Assumption 4-(iii),

• for any compact subset K ⊂ Θ we have that sup θ∈K E Rn (θ) 2 | D n < +∞ with probability 1, wich we shall now check. 64Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation Let K be a compact subset of Θ, then

sup θ∈K E Rn (θ) 2 | D n = sup θ∈K 1 n 2 n i,j=1 π i,j (θ) π i (θ) π j (θ) ∇l(Z i , θ) T ∇l(Z j , θ) ≤ 1 n n i=1 sup θ∈K ∇l(Z i , θ) π i (θ) 2 ,
which is finite with probability 1 by virtue of Assumption 4-(ii).

Proof of Theorem 4.4

Our conditional Central Limit Theorem results from Theorem 1 in Pelletier (1998), the applicability of which needs to be checked.

First of all, rewrite the algorithm sequence as

θ n (t + 1) = θ n (t) -γ t ∇ L n (θ t ) + γ t ξ n (t + 1),
where ξ n (t + 1) := ∇ L n (θ t ) -Rn (θ n (t)). This way, -∇ L n (θ t ) appears as the mean field of the algorithm and ξ n (t + 1) as a noise term. Now consider the filtration F = {F t } t≥1 where for each t ≥ 1, F t is the σ-field generated by the indicator vectors

(1) n . . . , (t-1) n and by D n . Then Assumption 1-(ii) guarantees that {ξ n (t)} t≥1 is a sequence of increments of a q-dimensional square integrable martingale adapted to the filtration F: for all t ≥ 1 we have both E[ξ n (t + 1) | F t ] = 0 and

E ξ n (t + 1) 2 | F t = 1 n 2 n i,j=1 π i,j (θ n (t)) π i (θ n (t)) π j (θ n (t)) -1 ∇l(Z i , θ n (t)) T ∇l(Z j , θ n (t)) ≤ 1 n n i=1 sup θ∈K ∇l(Z i , θ) π i (θ) 2 < +∞.
Given this representation, our result is assured by Theorem 1 in [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF] provided that the following conditions hold true:

• ∇ L n (θ * n ) = 0, which was assumed, • on a neighborhood V of θ * n we have ∇ L n (θ) = H n (θ -θ * n ) + O( θ -θ * n
2 ), which results from a simple Taylor expansion made possible by Assumption 5-(i),

• -H n is a stable q × q matrix (the largest real part of its eigenvalues is negative), which corresponds to Assumption 5-(ii),

• γ is regularly varying with index -α ∈ (-1, 0] or γ t = γ 0 /t with γ 0 > 1/(2l) for all t ≥ 1, which was also assumed, • (B) E ξ n (t + 1) ξ n (t + 1) T | F t → Γ almost-surely on E(θ * n ) as t → +∞, with Γ a positive-definite deterministic matrix, which also needs to be checked.

• (A) sup

Let us start with condition (A). Since

0 ≤ ξ n (t+1) ≤ 1 n n i=1 ∇l(Z i , θ n (t)) π i (θ n (t))
almost-surely for all t ≥ 0, then for any b > 2

sup t≥0 E ξ n (t + 1) b | F t I {θ n (t) ∈ V} ≤ 1 n n i=1 sup θ∈V ∇l(Z i , θ) π i (θ) b < +∞
with probability 1, by Assumption 4-(ii).

Regarding condition (B), we have

E[ξ n (t + 1) ξ n (t + 1) T | F t ] = Γ n (θ n (t)) for all t ≥ 1,
where for any θ ∈ Θ,

Γ n (θ) = 1 n 2 n i,j=1 π i,j (θ) π i (θ) π j (θ) -1 ∇l(Z i , θ) ∇l(Z j , θ) T .
By virtue of the continuity assumptions of the inclusion probabilities (Assumption 5-(iii)) and of the gradient (Assumption 4-(i)), given the population data D n we have the almost-sure convergence

Γ n (θ n (t)) → Γ * n = Γ n (θ * n ) on the event E(θ * n )
as t → +∞. The limit matrix is, indeed, positive-definite and deterministic (given D n ).

Proof of Proposition 4.6

Let us start by calculating Σ 1/2 pn 2 for some collection of positive Poisson inclusion probabilities p n = {p n (θ)} θ∈Θ . In the case where η = 0, since H n is invertible by Assumption 5-(ii), the Lyapunov equation (4.10) can be rewritten as

Σ pn + H -1 n Σ pn H n = H -1 n Γ * n , which implies that Σ 1/2 pn 2 = 1 2 T r H -1 n Γ * n = 1 2 T r G n Γ * n G T n . Now recall that θ * n is
a stationary point, i.e. ∇ L n (θ * n ) = 0, and that we are considering a Poisson scheme (with independent inclusion variables). Thus,

Γ * n = 1 n 2 n i=1 1 p i (θ * n ) -1 ∇l(Z i , θ * n ) ∇l(Z i , θ * n ) T and then Σ 1/2 pn 2 = 1 2 1 n 2 n i=1 1 p i (θ * n ) -1 G n ∇l(Z i , θ * n ) 2 . (4.1)
Let us now turn to the definition of an optimal collection of Poisson plans. Using the Lagrange multiplier method like in subsection 4.4.1, we find that any p n minimizing (4.1) must satisfy the equalities

p i (θ * n ) = N G n ∇l(Z i , θ * n ) n j=1 G n ∇l(Z j , θ * n ) , i ∈ U n .
66Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation This is the case, in particular, of the collection p * n defined in Proposition 4.6. Condition (4.12) and the positive-definiteness of H n (Assumption 5-(ii)) ensure that p * i (θ) is almost-surely in (0, 1] for all θ ∈ Θ and i ∈ U n .

Proof of Proposition 4.7

Let us start by proving the first assertion. Using Eq. (4.1) with the corresponding inclusion probabilities, we immediately obtain

Σ 1/2 p N 2 -Σ 1/2 pn 2 = c n (θ * n ) 2 N ,
which is positive by Assumption 6.

Turning to the second assertion, observe that

Σ 1/2 pn 2 -Σ 1/2 p * n 2 = Σ 1/2 pn 2 -Σ 1/2 p N 2 + Σ 1/2 p N 2 -Σ 1/2 p * n 2 = 1 2 N σ 2 n (θ * n ) -c n (θ * n ) .
By definition of p * n (see Proposition 4.6), this quantity is always nonnegative.

Proof of Proposition 4.8

Consider a Poisson sampling plan with inclusion probabilities as in ((4.14)). We shall prove that Assumption 7 is fulfilled by establishing the asymptotic convergences (as n, N → +∞) of the three averages in brackets that appear in the following decomposition:

N Γ * n = 1 n n i=1 p(W i , θ * n ) × 1 n n i=1 1 p(W i , θ * n ) ∇l(Z i , θ * n )∇l(Z i , θ * n ) T - N n 1 n n i=1 ∇l(Z i , θ * n )∇l(Z i , θ * n ) T
Recall that (Z 1 , W 1 ), . . . , (Z n , W n ) were taken as independent copies of some generic r.v.

(Z, W ), which is thus independent from θ * n , for any n ∈ N * . The respective distributions of Z, W and (Z, W ) are denoted by P Z , P W and P Z,W .

First average

We shall verify that the first term in brackets tends to E [p(W, θ * )] almostsurely as n → +∞. For any n ∈ N * we have

1 n n i=1 p(W i , θ * n ) -E[p(W, θ * )] ≤ sup θ∈Θ 1 n n i=1 p(W i , θ) -E[p(W, θ)] + |E[p(W, θ * n )] -E[p(W, θ * )]| .
Thus, it suffices to check that both the supremum and the difference of expectations above (almost-surely) vanish as n grows.

The supremum can be controlled using Lemma 3.10 in van de Geer (2000). The parameter space Θ is assumed to be a compact metric space and the map θ ∈ Θ → p(w, θ) is supposed Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation67 to be continuous for all w ∈ R d . In addition, since the link function p was chosen to be bounded by some finite positive constant p, the envelope w ∈ R d → sup θ∈Θ |p(w, θ)| is P Wintegrable. By virtue of the aforementioned Lemma, these conditions are sufficient to obtain the uniform law of large numbers: as n → +∞, with probability one,

sup θ∈Θ 1 n n i=1 p(W i , θ) -E[p(W, θ)] → 0.
Let us now turn to the difference of expectations. Fix some w ∈ R d . The empirical risk minimizer θ * n is assumed to be strongly consistent (condition (i)) and the mapping θ ∈ Θ → p(w, θ) to be continuous. Thus, by the continuous mapping theorem, p(w, θ * n ) converges almost-surely to p(w, θ * ), as n → +∞. Then, because p is a bounded function, by the dominated convergence theorem we also have

E[p(w, θ * n )] → p(w, θ * ) as n → +∞. Next, for any n ∈ N * , the independence between W and θ * n implies E[p(W, θ * n )] = R d E[p(w, θ * n )] P W (dw).
Applying the dominated convergence theorem to this last integral, we finally obtain that

E[p(W, θ * n )] → E[p(W, θ * )] as n → +∞.
Second average The second term in brackets is a q × q matrix, the convergence of which shall be established element-wise. Let (k, h) ∈ {1, . . . , q} 2 and define the function θ). The element at the intersection of the k-th row and h-th column of this matrix is

Ψ k,h : (z, θ) ∈ R d × Θ → (∂l/∂θ k )(z, θ) × (∂l/∂θ h )(z,
1 n n i=1 1 p(W i , θ * n ) Ψ k,h (Z i , θ * n ).
Using the same reasoning as before, this quantity can be shown to converge almost-surely, as n, N → +∞, to E p(W, θ * ) -1 Ψ k,h (Z, θ * ) . We only need to check that

1. the map θ ∈ Θ → p(w, θ) -1 Ψ k,h (z, θ) is continuous for any (z, w) in R d × R d , 2. the envelope (z, w) ∈ R d × R d → sup θ∈Θ |p(w, θ) -1 Ψ k,h (z, θ)| is P Z,W -integrable, 3. the class of random variables {p(W, θ * n ) -1 Ψ k,h (Z, θ * n )} is uniformly P Z,W -integrable.
Condition (a) is guaranteed by the construction of p and by Assumption 4-(i).

An Intermediary Result

Before beginning to prove Theorem 4.9, we first establish the following lemma. It describes the limit behavior of the solution of the Lyapunov equation ((4.10)) as n, N → +∞.

Lemma 4.11. Suppose that the assumptions of Theorem 4.9 are fulfilled. Then as n, N tend to +∞, we have:

N Σ πn → Σ * in probability.
Proof. Observe first that it follows from Hence, we have:

H n Σ πn + Σ πn H n = Γ * n that Γ * n 2 = 2 H n Σ πn 2 + 2T r(H n Σ πn H n Σ πn ) 0 , 68Chapter
Γ * n √ 2 H n Σ πn .
We deduce from this inequality combined with assumptions 7 and the fact that H -1 n = O P (1) as n → ∞ (this can be deduced from the LLN H n → H * and the hypothesis that the Hessian matrices H n and H * are invertible) that

Σ πn = O P (1/N ) as n → ∞. (4.2)
Since H * Σ * + Σ * H * = Γ * and H n Σ πn + Σ πn H n = Γ * n , we have:

Γ * -N Γ * n = H * (Σ * -N Σ πn ) + (Σ * -N Σ πn )H * + + N (H n -H * )Σ πn + N Σ πn (H n -H * ). (4.3) Combining (4.3) with H * (Σ * -N Σ πn ) = (Σ * -N Σ πn )H * 1 √ 2 H * (Σ * -N Σ πn ) + (Σ * -N Σ πn )H * (4.4)
we easily get

H * (Σ * -N Σ πn ) 1 √ 2 Γ * -N Γ * n + √ 2 N Σ πn (H n -H * ) . (4.5)
By virtue of the LLN, we have H n -H * → 0 almost surely as n → ∞. Combining this with (4.2) and assumption 7 we see that the term on the right hand side of (4.5) converges toward 0 in probability as n → ∞. Combined with the invertibility of H * , this establishes the desired result.

4.1.7 Proof of Theorem 4.9

Consider the decomposition:

θ n (t) -θ * = θ n (t) -θ * n + θ * n -θ * = γ t N N/γ t (θ n (t) -θ * n ) + 1 √ n √ n (θ * n -θ * ) = γ t N N/γ t (θ n (t) -θ * n ) (1) + 1 √ n √ n (θ * n -θ * ) (2)
.

The term (2) above is asymptotically normal. By virtue of the classical Central Limit Theorem for M -estimators, see Theorem 5.23 in [START_REF] Van Der | Asymptotic statistics[END_REF] for instance, we have:

√ n (θ * n -θ * ) ⇒ N (0, Λ * ) as n → ∞. (4.6)
This suffices to establish assertion (ii) since the parameter space Θ is assumed to be compact here. Turning to term (1), holding n and N fixed, Theorem 4.4 claims that, in probability Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation69

along the sequence X (respectively, the sequence (X, W )):

1/γ t Σ -1/2 πn (θ n (t) -θ * n ) ⇒ Z as t → ∞, (4.7)
where Z denotes a q-dimensional centered Gaussian random vector with the identity as covariance, independent from the sequence X (from the sequence (X, W ) respectively). Now it follows from Lemma 4.11 combined with the continuity of the application that maps any symmetric positive semi-definite matrix to its square root that

(N Σ πn ) 1/2 → Σ * 1/2 in probability, as n, N → ∞. (4.8)
Given that one may write

N γ t (θ n (t) -θ * n ) = (N Σ πn ) 1/2 1/γ t Σ -1/2 πn (θ n (t) -θ * n ) ,
it results from (4.7) and (4.8) that the following convergence in distribution holds true:

lim n,N →∞ lim t→∞ N γ t (θ n (t) -θ * n ) = Σ * 1/2 Z. (4.9)
Assertions (i) and (iii) can be then deduced from (4.9) and (4.6) in a straightforward fashion (using the independence of the limits, regarding (iii)).

Rate Bound Analysis

Here we establish a rate bound for the HTGD algorithm under the additional assumption that the mapping θ → l(z, θ) is convex, referred to as Assumption 8, as we have done in chapter 4. Note that Assumptions 5 and 8 imply that θ * n is unique and L n is l strongly convex on V. For simplicity's sake, we suppose that the strong convexity property holds true on R d . The following result relies on standard arguments in stochastic approximation, see Nemirovski et al. (2009a), Bach & Moulines (2011a) or Nesterov (2004a). All expectations are taken conditionally upon the observations. Theorem 4.12. Under Assumptions 4, 5 and 8 and for a stepsize γ t = γ 0 t -α with some constants γ 0 > 0 and α ∈ (1/2, 1] (when α = 1, take γ 0 > 1/(2l)), there exists a constant

C α < +∞ such that: ∀t ≥ 1, E[ θ n (t) -θ * n 2 ] ≤ C α t α . (4.1)
Proof. We restrict ourselves to the case α = 1 and follow the proof of Bach & Moulines (2011a). By construction, we have we get

θ n (t + 1) -θ * n 2 = θ n (t) -θ * n 2 -2γ t Rn (θ n (t)) T (θ n (t) -θ * n ) + γ t Rn (θ n (t)) 2 . Since E[ Rn (θ n (t))|F t ] = ∇ L n (θ n (t)),
E[|θ n (t + 1) -θ * n | 2 | θ n (t)] = θ n (t) -θ * n 2 -2γ t ∇ L n (θ n (t)) T (θ n (t) -θ * n ) + γ 2 t E[ Rn (θ n (t)) 2 | θ n (t)].
The strong convexity property gives

L n (θ n (t)) -L n (θ * n ) ≤ ∇ L n (θ n (t)) T (θ n (t) -θ * n ) - l 2 θ n (t) -θ * n 2 and L n (θ * n ) -L n (θ n (t)) ≤ - l 2 θ n (t) -θ * n 2 , so that θ n (t) -θ * n 2 ∇ L n (θ n (t)) T (θ n (t) -θ * n ).
Combining this inequality with the previous one and taking the expectation, we obtain

E[ θ n (t + 1) -θ * n 2 ] ≤ E[ θ n (t) -θ * n 2 ](1 -2γ t l) + γ 2 t E[ Rn (θ n (t)) 2 ].
Under Assumption 4, we have E[ Rn (θ n (t)) 2 ] ≤ D for some constant D > 0. Using this bound and iterating the recursion, we finally obtain

E[ θ n (t + 1) -θ * n 2 ] E[ θ(1) -θ * n 2 ] t j=1 (1 -2lγ(j)) + D t j=1 γ 2 t t k=j+1 (1 -2lγ(k))
with the convention t k=t+1 (1 -2lγ(k)) = 1 We now substitute the expression of γ t and, using the following classical inequalities 1 + x e x and log(t + 1) -log(j + 1)

t k=j+1 1 k ,
we get

E θ n (t + 1) -θ * n 2 (E θ n (1) -θ * n 2 + D t j=1 1 j 2-2lγ 0 ) (t + 1) 2lγ 0 ,
where D is a positive constant. Since γ 0 > 1/(2l), we have

t j=1 1 j 2-2lγ 0 t 2lγ 0 -1 2lγ 0 -1
and we finally obtain the desired bound.

CHAPTER 5

Stochastic Gradient Descent based on incomplete U -Statistics

Abstract In many learning problems, ranging from clustering to ranking through metric learning, empirical estimates of the risk functional consist of an average over tuples (e.g., pairs or triplets) of observations, rather than over individual observations. In this chapter, we focus on how to best implement a stochastic approximation approach to solve such risk minimization problems. We argue that in the large-scale setting, gradient estimates should be obtained by sampling tuples of data points with replacement (incomplete U -statistics) instead of sampling data points without replacement (complete U -statistics based on subsamples). We develop a theoretical framework accounting for the substantial impact of this strategy on the generalization ability of the prediction model returned by the Stochastic Gradient Descent (SGD) algorithm. It reveals that the method we promote achieves a much better trade-off between statistical accuracy and computational cost. Beyond the rate bound analysis, experiments on AUC maximization and metric learning provide strong empirical evidence of the superiority of the proposed approach.

Background and Problem Setup

For clarity, we start with recalling the definition of generalized U -statistics and their crucial properties and providing examples of learning problems motivated by various applications where such data functionals are naturally involved. We recall that the variance of any square integrable r.v. Z is denoted by σ 2 (Z)

U -statistics: Definition and Examples

Generalized U -statistics are extensions of standard sample mean statistics, as defined below.

Definition 5.1.

Let K ≥ 1 and (d 1 , . . . , d K ) ∈ N * K . Let X {1, ..., n k } = (X (k) 1 , . . . , X (k) n k ), 1 ≤ k ≤ K, be K independent samples of sizes n k ≥ d k and composed of i.i.d. random vari- ables taking their values in some measurable space X k with distribution F k (dx) respectively. Let H : X d 1 1 × • • • × X d K K → R be a measurable function, square integrable with respect to the probability distribution µ = F ⊗d 1 1 ⊗ • • • ⊗ F ⊗d K K
. Assume in addition (without loss of generality) that H(x (1) , . . . , x (K) ) is symmetric within each block of arguments

x (k) (valued in X d k k ), 1 ≤ k ≤ K. The generalized (or K-sample) U -statistic of degrees (d 1 , . . . , d K ) with kernel H, is then defined as U n (H) = 1 K k=1 n k d k I 1 . . . I K H X (1)
I 1 ; X

(2)

I 2 ; . . . ; X (K) I K , ( 5
≤ i 1 < . . . < i d k ≤ n k and X (k) I k = (X (k) i 1 , . . . , X (k) i d k ) for 1 ≤ k ≤ K.
In the above definition, standard mean statistics correspond to the case where K = 1 = d 1 . More generally when K = 1, U n (H) is an average over all d 1 -tuples of observations. Finally, K ≥ 2 corresponds to the multi-sample situation where a d k -tuple is used for each sample k ∈ {1, . . . , K}.

The key property of the statistic (5.1) is that it has minimum variance among all unbiased estimates of

µ(H) = E H X (1) 1 , . . . , X (1) d 1 , . . . , X (K) 1 , . . . , X (K) d k = E [U n (H)] .
One may refer to Lee (1990a) for further results on the theory of U -statistics. In machine learning, generalized U -statistics are widely used for estimating properties of probability distributions (e.g. variance, Gini mean difference), for statistical hypothesis testing (i.e. Kendall tau, Mann-Whitney statistic) and used as performance criteria in various problems, such as those listed below.

Clustering. Given a distance D : X 1 × X 1 → R + , the quality of a partition P of X 1 with respect to the clustering of an i.i.d. sample X 1 , . . . , X n drawn from F 1 (dx) can be assessed through the within cluster point scatter:

W n (P) = 2 n(n -1) i<j D(X i , X j ) • C∈P I (X i , X j ) ∈ C 2 .
(5.2)

It is a one sample U -statistic of degree 2 with kernel

H P (x, x ) = D(x, x )• C∈P I{(x, x ) ∈ C 2 }.
Multi-partite ranking. Suppose that K independent i.i.d. samples

X (k) 1 , . . . , X (k)
n k with n k ≥ 1 and 1 ≤ k ≤ K on X 1 ⊂ R p have been observed. The accuracy of a scoring function s : X 1 → R with respect to the K-partite ranking is empirically estimated by the rate of concordant K-tuples (sometimes referred to as the Volume Under the ROC Surface):

VUS n (s) = 1 n 1 × • • • × n K K k=1 n k i k =1 I s(X (1) i 1 ) < • • • < s(X (K) i K ) .
The quantity above is a K-sample U -statistic with degrees d 1 = . . . = d K = 1 and kernel

Hs (x 1 , . . . , x K ) = I{s(x 1 ) < • • • < s(x K )}.
Metric learning. Based on an i.i.d. sample of labelled data (X 1 , Y 1 ), . . . , (X n , Y n ) on X 1 = R p × {1, . . . , J}, the empirical pairwise classification performance of a distance D : X 1 × X 1 → R + can be evaluated by:

R n (D) = 6 n(n -1)(n -2) i<j<k I {D(X i , X j ) < D(X i , X k ), Y i = Y j = Y k } (5.3)
which is a one sample U -statistic of degree three with kernel HD ((x, y), (x , y ), (x , y )) = I {D(x, x ) < D(x, x ), y = y = y }.

• Pairwise classification.

Based on an i.i.d. sample of labelled data (X 1 , Y 1 ), . . . , (X n , Y n ) on X 1 = R d × R, the empirical ranking performance of any antisymmetric ranking rule r : R d × R d → {-1, +1} (the quantity r(x, x ) being expected to be +1 when the label assigned to x is larger than that assigned to x ) is evaluated through the pairwise classification risk

P n (r) = 2 n(n -1) i =j I {r(X i , X j ) • (Y i -Y j ) < 0} .
It is a one sample U -statistic of degree 2 with kernel Hr ((x, y), (x , y )) = I{r(x, x ) • (y -y ) < 0}.

• Metric learning. Suppose that K independent i.i.d. samples X (k) 1 , . . . , X (k) n k with n k ≥ 1 and 1 ≤ k ≤ K on X 1 ⊂ R d are available.
The empirical performance of a similarity measure D : R d × R d → R + regarding its capacity to discriminate among the K populations when combined with a nearest-neighbor can be evaluated by:

R n (D) = 2 K k=1 1 n k (n k -1) 1≤i<j≤n k 1 + D(X (k) i , X (k) j ) -b + -2 1≤k<l≤K 1 n k n l n k i=1 n l j=1 1 + D(X (k) i , X (l) j ) -b + ,
where b > 0 is a tuning threshold parameter and z + denotes the positive part of any real number z.

It is a K sample U -statistic of degrees d 1 = . . . = d K = 2.

SGD Implementation based on Incomplete U -Statistics

Let Θ ⊂ R q with q ≥ 1 be some parameter space, we consider the risk minimization problem min θ∈Θ L(θ) with

L(θ) = E[H(X (1) 1 , . . . , X (1) d 1 , . . . , X (K) 1 , . . . , X (K) d K ; θ)] = µ(H(.; θ)),
where H :

K k=1 X d k k × Θ → R is a convex loss function, the (X (k) 1 , . . . , X (k) d k )'s, 1 ≤ k ≤ K, are K independent random variables with distribution F ⊗d k k (dx) on X d k k re- spectively so that H is square integrable for any θ ∈ Θ. Based on K independent i.i.d. samples X (k) 1 , . . . , X (k) n k with 1 ≤ k ≤ K, the empirical version of the risk function is θ ∈ Θ → L n (θ) = U n (H(.; θ)).
So far in this manuscript, we emphasized how important are stochastic method to solve the ERM problem, this is even truer in this case where the sample sizes n 1 , . . . , n K of the training datasets are such that computing the empirical gradient

g n (θ) def = ∇ L n (θ) = 1/ K k=1 n k d k I 1 . . . I K ∇H(X (1) I 1 ; X (2) I 2 ; . . . ; X (K) I K ; θ) (5.4)
just once is intractable due to the number #Λ = K k=1 n k d k of terms to be averaged. Note also that trying to adopt a similar approach to what we have done in chapter 3 and 4 is impossible as we would have to maintain a probability distribution over all the K k=1 n k d k couples of observations. A possible approach could consist in replacing (5.4) by a (complete) U -statistic computed from subsamples of reduced sizes n k << n k , {(X

(k) 1 , . . . , X (k) n k ) : k = 1, . . . , K}
say, drawn uniformly at random without replacement among the original samples, leading to the following gradient estimator:

gn (θ) = 1 K k=1 n k d k I 1 . . . I K ∇H(X (1)
I 1 ; X

(2) I 2 ; . . . ; X (K) (5.5) where I k refers to summation over all n k d k subsets X

I K ; θ),
(k)

I k = (X (k) i 1 , . . . , X (k) i d k ) related to a set I k of d k indexes 1 ≤ i 1 < . . . < i d k ≤ n k and n = (n 1 , . . . , n K ).
Provided that the n k 's are sufficiently small, it may be numerically tractable to average over the

K k=1 n k d k
terms involved in the definition of (5.5). Although this approach is very natural, one can obtain a better estimate for the same computational cost, as shall be seen below.

Monte-Carlo Estimation of the Empirical Gradient

From a practical perspective, the alternative strategy we propose is of disarming simplicity.

It is based on a Monte-Carlo sampling scheme that consists in drawing independently with replacement among the set of index vectors Λ, yielding a gradient estimator in the form of a so-called incomplete U -statistic (see Lee (1990a)):

ḡB (θ) = 1 B (I 1 , ..., I K )∈D B ∇H(X (1)
I 1 , . . . , X (K)

I K ; θ), (5.6)
where D B is built by sampling B times with replacement in the set Λ. We point out that the conditional expectation of (5.6) given the K observed data samples is equal to g n (θ). The parameter B, corresponding to the number of terms to be averaged, controls the computational complexity of the SGD implementation. Observe incidentally that an incomplete U -statistic is not a U -statistic in general. Hence, as an unbiased estimator of the gradient of the statistical risk L(θ), (5.6) is of course less accurate than the full empirical gradient (5.4) (i.e., it has larger variance), but this slight increase in variance leads to a large reduction in computational cost.

In our subsequent analysis, we will show that for the same computational cost (i.e., taking

B = K k=1 n k d k
), implementing SGD with (5.6) rather than (5.5) leads to much more accurate results. We will rely on the fact that (5.6) has smaller variance w.r.t. to ∇L(θ) (except in the case where K = 1 = d 1 ), as shown in the proposition below.

Proposition 5.2. Set B = K k=1 n k d k
. There exists a universal constant c > 0, such that we have:

σ 2 (g n (θ)) ≤ c • σ 2 θ / K k=1 n k and σ 2 (ḡ B (θ)) ≤ c • σ 2 θ / K k=1 n k d k , for all n ∈ N * K , with σ 2 θ = σ 2 (∇H(X (1) 1 , . . . , X (K) d K ; θ)).
Explicit but lengthy expressions of the variances are given in Lee (1990a).

Remark 5.3. The results of this chapter can be extended to other sampling schemes to approximate (5.4), such as Bernoulli sampling or sampling without replacement in Λ, following the proposal of [START_REF] Janson | The asymptotic distributions of Incomplete U -statistics[END_REF]. For clarity, we focus on sampling with replacement, which is computationally more efficient.

A Conditional Performance Analysis

As a first go, we investigate and compare the performance of the SGD methods described above conditionally upon the observed data samples. In this section, all expectations are taken conditionally upon the observations. Given a matrix M , we recall that we defined M T to be the transpose of M and M := T r(M M T ) to be its Hilbert-Schmidt norm. We assume that the loss function H is l-smooth in θ, i.e its gradient is l-Lipschitz, with l > 0. We also restrict ourselves to the case where L n is α-strongly convex for some deterministic constant α:

L

n (θ 1 ) -L n (θ 2 ) ∇ L n (θ 1 ) T (x -y) - α 2 θ 1 -θ 2 2
(5.7)

and we denote by θ * n its unique minimizer. We point out that the present analysis can be extended to the smooth but non-strongly convex case, see Bach & Moulines (2011b). A classical argument based on convex analysis and stochastic optimization (see Bach & Moulines (2011b); Nemirovski et al. (2009b) for instance) shows precisely how the conditional variance of the gradient estimator impacts the empirical performance of the solution produced by the corresponding SGD method and thus strongly advocates the use of the SGD variant proposed in Section 5.2.1.

Proposition 5.4. Consider the recursion θ t+1 = θ t -γ t g(θ t ) where E[g(θ t )|θ t ] = ∇ L n (θ t ), and denote by σ 2 n (g(θ)) the conditional variance of g(θ). With γ t = γ 1 /t β , the following holds.

1. If 1 2 < β < 1, then:

E[ L n (θ t+1 ) -L n (θ * n )] σ 2 n (g(θ * n )) t β γ 1 l2 β-1 ( 1 2α + lγ 2 1 2β -1 ) + o( 1 t β ) C 1 .
and with probability at least 1 -δ:

L n (θ t+1 ) -L n (θ * n ) σ 2 n (θ * n ) t β C 1 + D β log(L/δ) t β 2. If β = 1 and γ 1 > 1 2α , then: E[ L n (θ t+1 ) -L n (θ * n )] σ 2 n (g(θ * n )) t + 1 2 αγ 1 l exp(2αlγ 2 1 )γ 2 1 (2αγ 1 -1) + o( 1 t ) C 2 .
and with probability at least 1 -δ:

L n (θ t+1 ) -L n (θ * n ) σ 2 n (θ * n ) t C 2 + D log(L/δ) t
for some constants D and D β depending on the parameters L, α, γ 1 , a 1 .

Proposition 5.4 is another illustration of the fact that the convergence rate of SGD is dominated by the variance term and corroborate our approach of chapter 3 and 4 (see [START_REF] Zhao | Stochastic Optimization with Importance Sampling for Regularized Loss Minimization[END_REF]; Johnson & Zhang (2013b); Defazio et al. (2014) for instance).

As we have done in theorem 4.4, we can also give the asymptotic behaviour of the algorithm (when t → +∞), under the following assumptions:

A 1 The function L n (θ) is twice differentiable on a neighborhood of θ * n . A 2 The function ∇ L n (θ) is bounded.
Let us set Γ = ∇ 2 L n (θ * n ), then we have under the conditional probability:

Theorem 5.5. Let the covariance matrix Σ * n be the unique solution of the Lyapunov equation:

ΓΣ * n + Σ * n Γ -ηΣ * n = Σ n (θ * n ), (5.8) where Σ n (θ * n ) = E[g(θ * n )g(θ * n ) T ] and η = γ 1 > 1 2α if β = 1, 0 if not.
Then, under Assumptions A 1 -A 2 , we have:

1/γ t L n (θ t ) -L n (θ * ) ⇒ 1 2 U T (Σ * n ) 1/2 Γ(Σ * n ) 1/2 U,
where U ∼ N (0, I q ). In addition, in the case η = 0, we have:

(Σ * n Γ) 1/2 2 HS = E[U T (Σ * n ) 1/2 Γ(Σ * n ) 1/2 U ] = 1 2 σ 2 n (g(θ * n )).
(5.9) Theorem 5.5 reveals that the conditional variance term again plays a key role in the asymptotic performance of the algorithm. In particular, it is the dominating term in the precision of the solution. In the next section, we build on these results to derive a generalization bound in the spirit of [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF] which explicitly depend on the true variance of the gradient estimator. We also point out that the result established in section 4.5 (i.e an unconditional TCL) could also be established in a straightforward fashion, and comparison of norms of the asymptotic variance would also be straightforward thanks to equation (4.13).

Generalization Bounds

All expectations are now taken w.r.t the sampling procedure and the distribution of the observations. We introduce the following notations. Let θ * = argmin θ∈Θ L(θ) be the minimizer of the true risk. As proposed in [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF], the mean excess risk can be decomposed as follows:

∀n ∈ N * K , E[L(θ t ) -L(θ * )] ≤ 2E sup θ∈Θ | L n (θ) -L(θ)| E 1 + E L n (θ t ) -L n (θ * n ) E 2 .
(5.10)

Beyond the optimization error (the second term on the right hand side of (5.10)), the analysis of the generalization ability of the learning method previously described requires to control the estimation error (the first term). This can be achieved by means of the result stated below, which extends Corollary 3 in Clémençon et al. (2008b) to the K-sample situation.

Proposition 5.6. Let H be a collection of bounded symmetric kernels on

K k=1 X d k k such that M H = sup (H,x)∈H×X |H(x)| < +∞. Suppose also that H is a VC ma- jor class of functions with finite Vapnik-Chervonenkis dimension V < +∞. Let κ = min { n 1 /d 1 , . . . , n K /d K }. Then, for any n ∈ N * K E sup H∈H |U n (H) -µ(H)| ≤ M H 2 2V log(1 + κ) κ .
(5.11) for all δ ∈ (0, 1), we have with probability at least 1 -δ,

sup H∈H |U n (H) -µ(H)| ≤ M H 2 2V log(1 + κ) κ + log(1/δ) κ .
(5.12)

We are now ready to derive our main result.

Theorem 5.7. Let θ t be the sequence generated by SGD using the incomplete statistic gradient estimator (5.6)

with B = K k=1 n k d k
terms for some n 1 , . . . , n K . Assume that {L(.; θ) : θ ∈ Θ} is a VC major class class of finite VC dimension V s.t. (5.13) and N Θ = sup θ∈Θ σ 2 θ < +∞. If the step size satisfies the condition of Proposition 5.4, we have:

M Θ = sup θ∈Θ, (x (1) , ..., x (K) )∈ K k=1 X d k k |H(x (1) , . . . , x (K) ; θ)| < +∞,
∀n ∈ N * K , E[|L(θ t ) -L(θ * )|] CN Θ Bt β + 2M Θ 2 2V log(1 + κ) κ .
For any δ ∈ (0, 1), we also have with probability at least 1 -δ:

∀n ∈ N * K , |L(θ t )-L(θ * )| CN Θ Bt β + D β log(2/δ) t β +2M Θ 2 2V log(1 + κ) κ + log(4/δ) κ .
(5.14) for some constants C and D β depending on the parameters l, α, γ 1 , a 1 .

The generalization bound provided by Theorem B.15 shows the advantage of using an incomplete U -statistic (5.6) as the gradient estimator. In particular, we can obtain results of the same form as Theorem B.15 for the complete U -statistic estimator (5.5), but

B = K k=1 n k d k
is then replaced by K k=1 n k (following Proposition 5.2), leading to greatly damaged bounds. Using an incomplete U -statistic, we thus achieve better performance on the test set while reducing the number of iterations (and therefore the numbers of gradient computations) required to converge to a accurate solution. To the best of our knowledge, this is the first result of this type for empirical minimization of U -statistics. In the next section, we provide experiments showing that these gains are very significant in practice.

Numerical Experiments

In this section, we provide numerical experiments to compare the incomplete and complete U -statistic gradient estimators (5.5) and (5.6) when they rely on the same number of terms B. The datasets we use are available online. 1 In all experiments, we randomly split the data into 80% training set and 20% test set and sample 100K pairs from the test set to estimate the test performance. For SGD, we used a step size of the form γ t = γ 1 /t, where γ 1 is the initial value. The results below are with respect to the number of SGD iterations. Computational time comparisons can be found in the supplementary material. AUC Optimization We address the problem of learning a binary classifier by optimizing the Area Under the Curve, which corresponds to the VUS criterion (Eq. 5.2) when K = 2. Given a sequence of i.i.d observations

Z i = (X i , Y i ) where X i ∈ R p and Y i ∈ {-1, 1}, we denote by X + = {X i ; Y i = 1}, X -= {X i ; Y i = -1} and n = |X + ||X -|.
As done in [START_REF] Zhao | AUC Maximization[END_REF]; [START_REF] Herschtal | Optimising area under the ROC curve using gradient descent[END_REF], we take a linear scoring rule s θ (x) = θ T x where θ ∈ R p is the parameter to learn, and use the logistic loss as a smooth convex function upper bounding the Heaviside function, leading to the following ERM problem:

min θ∈R p 1 n X + i ∈X + X - j ∈X - log(1 + exp(s θ (X - i ) -s θ (X + i ))).
We use two datasets: IJCNN1 (∼200K examples, 22 features) and covtype (∼600K examples, 54 features). We try different values for the initial step size γ 1 and the batch size B. Some of the results, averaged over 50 runs of SGD, are displayed in Figure 5.1. As predicted by our theoretical findings, the incomplete U -statistic estimator consistently outperforms its complete variant on average. 2 We also observe a smaller variance between SGD runs when using the incomplete version.

Metric Learning

We now turn to a metric learning formulation, where we are given a sample of n i.i.d observations Z i = (X i , Y i ) where X i ∈ R p and Y i ∈ {1, . . . , c}. Following the existing literature [START_REF] Bellet | A Survey on Metric Learning for Feature Vectors and Structured Data[END_REF], we focus on (pseudo) distances of the form

D M (x, x ) = (x -x ) T M (x -x )
where M is a p × p symmetric positive semi-definite matrix. We again use the logistic loss to obtain a convex and smooth surrogate for (5.3). The ERM problem is as follows:

min M 6 n(n -1)(n -2) i<j<k I {Y i = Y j = Y k } log(1 + exp(D M (X i , X j ) -D M (X i , X k ))).
We use the binary classification dataset SUSY (5M examples, 18 features). Figure 5.2 shows that the performance gap between the two strategies is much larger on this problem than in Figure 5.1. This is consistent with the theory: one can see from Proposition 5.2 that the variance gap between the incomplete and the complete approximations is much wider for a one-sample U -statistic of degree 3 (metric learning) than for a two-sample U -statistic of degree 1 (AUC optimization).

Conclusion and Perspectives

In this chapter, we have studied a specific implementation of the SGD algorithm when the natural empirical estimates of the objective function are of the form of generalized U -statistics. This situation covers a wide variety of statistical learning problems such as multi-partite ranking, pairwise clustering and metric learning. The gradient estimator we propose in this context is based on an incomplete U -statistic obtained by sampling tuples with replacement. First, we have provided asymptotic and non-asymptotic rates for the convergence of this SGD-based learning method to the empirical risk minimizer. Our main result is a thorough analysis of the generalization ability of the predictive rules produced by this algorithm involving both the optimization and the estimation error in the spirit of [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF]. These results show that the SGD variant we propose far surpasses a more naive implementation (of same computational cost) based on subsampling the data points with replacement. Furthermore, we have shown that these performance gains are very significant in practice when dealing with large-scale datasets. Note all the results established can be extended to case where we sample tuples without replacement like we did in chapter 2 (in the general framework of rejective sampling) and would lead to upper bound of the same order.

Technical Proofs

Proof of Proposition 5.4

We follow the proof of Bach & Moulines (2011b) to derive bounds. We highlight the fact that since the loss function H is l-smooth, g n (θ t ) and gn (θ t ) are l-Lipschitz. We introduce the sequence γt = γ t (1 -lγ t ). In all generality we will denote by g t (θ) an unbiased estimator of the gradient at iteration t, l-Lipschitz in θ. We study the recursion θ t+1 = θ t -γ t g t (θ t ). We will make use of the two following classical inequalities from convex analysis (see Nesterov (2004b)):

L n (θ 1 ) -L n (θ 2 ) ∇ L n (θ 1 ) T (x -y) - α 2 θ 1 -θ 2 2
(5.15)

1 l g t (θ 1 ) -g t (θ 2 ) 2 (g t (θ 1 ) -g t (θ 1 )) T (θ 1 -θ2) (5.16)
As mentioned previously the analysis we proposed can easily be extended to a more general setting as in Bach & Moulines (2011b). We now begin the proof of the proposition:

θ t+1 -θ * n 2 = θ t -θ * n 2 -2γ t g t (θ t ) T (θ t -θ * n ) + γ 2 t g t (θ t ) 2
Using (5.16) we get

g t (θ t ) 2 2( g t (θ t ) -g t (θ * n ) 2 + g t (θ * n ) 2 ) 2l(g t (θ t ) -g t (θ * n )) T (θ t -θ * n ) + 2 g t (θ * n ) 2 ) (5.17) which together with E[g t (θ t )|θ t ] = g n (θ t ) gives E[ θ t+1 -θ * n 2 |θ t ] θ t -θ * n 2 -2γ t g n (θ t ) T (θ t -θ) + 2γ 2 t g t (θ * n ) 2
For the sake of simplicity we assume (1 -lγ t ) > 0 ∀t (which is eventually true since the sequence (γ t ) t 0 goes to 0 as t goes to infinity). Let

a t = E[ θ t -θ * n 2 ], σ 2 n (θ * n )
the variance (conditionally upon the data) of g t (θ * n ). Using (5.15) and taking the expectation we get the following recursion:

a t+1 a t (1 -2αγ t ) + 2γ 2 t σ 2 n (θ * n ) a 1 t j=1 (1 -2αγ j ) + 2σ 2 n (θ * n ) t j=1 γ 2 j t k=j+1 (1 -2αγ k ) (5.18)
with the convention t k=t+1 (1 -2αγ k ) = 1. Using 1 + x e x we get the following upper bound:

t j=1 (1 -2αγ j ) exp(-2α t j=1 γ j ) exp(2αl t j=1 γ 2 j )
We now need to distinguish two cases:

5.6.1.1 Case β = 1
If β = 1 we have:

1. log(t + 1) -log(j + 1)

t k=j+1 1 k 2. exp(2αl t k=1 1 k 2 ) exp(4αl) 3. exp(2αl t k=j+1 1 k 2 ) exp( 2αl j ) exp(2αl)
Under the assumption 2αγ 1 > 1:

a t+1 a 1 (t + 1) 2αγ 1 exp(4αlγ 2 1 ) + 2σ 2 n (θ * n ) exp(2αlγ 2 1 )γ 2 1 t j=1 (j + 1) 2αγ 1 j 2 1 (t + 1) 2αγ 1 a 1 (t + 1) 2αγ 1 exp(4αlγ 2 1 ) + 2 αγ 1 2σ 2 n (θ * n ) exp(2αlγ 2 1 )γ 2 1 (2αγ 1 -1)(t + 1)
which gives the result.

Case β < 1

If β < 1, let t 0 be a positive index, by splitting the sum in two parts we get:

t j=1 γ 2 j t k=j+1 (1 -2αγ k ) = t 0 j=1 γ 2 j t k=j+1 (1 -2αγ k ) + t j=t 0 +1 γ 2 j t k=j+1 (1 -2αγ k ) t k=t 0 +1 (1 -2αγ k ) t 0 j=1 γ 2 j + γ t 0 t j=t 0 +1 γ j t k=j+1 (1 -2αγ k )
where we used that the sequence (γ t ) t 1 is decreasing. Since

γ j = 1-(1-2αγ j ) 2α
+ lγ 2 j we have:

t j=t 0 +1 γ j t k=j+1 (1 -2αγ k ) = 1 2α t j=t 0 +1 t k=j+1 (1 -2αγ k ) - t k=j (1 -2αγ k ) + l t j=t 0 +1 γ 2 j t k=j+1 (1 -2αγ k ) 1 2α + l t j=t 0 +1 γ 2 j which leads to t j=1 γ 2 j t k=j+1 (1 -2αγ k ) exp(-2α t j=t 0 +1 γ j ) exp(2αl t j=t 0 +1 γ 2 j ) t 0 j=1 γ 2 j + γ t 0 2α + γ t 0 l t j=t 0 +1 γ 2 j
Taking t 0 ∼ t 2 and using the integral test for convergence:

Chapter 5. Stochastic Gradient Descent based on incomplete U -Statistics 1. t j=t 0 +1 γ j = γ 1 t j=t 0 +1 1 j β γ 1 (t+1) 1-β -(t 0 +1) 1-β 1-β γ 1 (t+1) 1-β 2(1-β) 2. t j=t 0 +1 γ 2 j = γ 2 1 t j=t 0 +1 1 j 2β γ 2 1 +∞ j=2 1 j 2β γ 2 1 2β-1 3. t 0 j=1 γ 2 j γ 2 1 (1 + 1 2β-1 ) = 2β 2β-1 γ 2 1
gives the following bound:

a t+1 a 1 exp(-2αγ 1 (t + 1) 1-β 2(1 -β) ) exp(2αl γ 2 1 2β -1 ) + 2σ 2 n (θ * n )(exp(-2α (t + 1) 1-β 2(1 -β) ) exp(2αl γ 2 1 2β -1 ) 2β 2β -1 γ 2 1 + 2 β γ 1 2αt β + γ 1 2 β t β 2lβ 2β -1 γ 2 1 ) = σ 2 n (θ * n ) γ 1 2 β t β ( 1 2α + lγ 2 1 2β -1 ) + o( 1 t β )
which concludes the proof.

5.6.2 Proof of Theorem 5.5

We recall that

Γ = ∇ 2 L n (θ * n ), Σ n (θ * n ) = E[g t (θ * n )g t (θ * n ) T ] and Σ * n is the solution of Lya- punov's equation: ΓΣ * n + Σ * n Γ -ηΣ * n = Σ n (θ * n ), (5.19)
Using classical results from stochastic approximation theory (see [START_REF] Delyon | Stochastic Approximation with Decreasing Gain: Convergence and Asymptotic Theory[END_REF]; G. [START_REF] Yves F Atchade | On stochastic proximal gradient algorithms[END_REF]; [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF] for instance) , we first show that under our assumptions:

1/γ t (θ t -θ * n ) ⇒ N (0, Σ * n ),
The asymptotic behavior of 1/γ t L n (θ t ) -L n (θ * ) is therefore a consequence of the second order delta method. We now turn to the second part of Theorem 5.5 and similarly to chapter 4:

E[U T (Σ * n ) 1/2 Γ(Σ * n ) 1/2 U ] = E[T r(Γ 1/2 (Σ * n ) 1/2 U U T (Σ * n ) 1/2 Γ 1/2 ] = T r(Γ 1/2 (Σ * n ) 1/2 E[U U T ](Σ * n ) 1/2 Γ 1/2 ) = T r(Γ 1/2 (Σ * n )Γ 1/2 ) = T r(ΓΣ * n ) = 1 2 T r(Σ n (θ * n )) = 1 2 σ 2 n (θ * n )
where we used the linearity of the trace, the linearity of the expectation, the dominated convergence theorem (to arrange the different terms) and Lyapunov's equation to conclude.

Proof of Theorem B.15

We prove a more general result and apply it to our specific setting. We consider the recursion defined in the proof of Proposition 2 and keep the same notations.

Theorem 5.8. Let θ t be the sequence generated by SGD and define

σ 2 = E[σ 2 n (g(θ * n ))]. Assume that {L(.; θ) : θ ∈ Θ} is a VC major class class of finite VC dimension V s.t M Θ = sup θ∈Θ, (x (1) , ..., x (K) )∈ K k=1 X d k k |H(x (1) , . . . , x (K) ; θ)| < +∞, (5.20)
If the step size satisfies the condition of Proposition 2, we have:

∀n ∈ N * K , E[|L(θ t ) -L(θ * )|] Cσ 2 t β + 2M Θ 2 2V log(1 + κ) κ .
For any δ ∈ (0, 1), we also have with probability at least 1 -δ:

∀n ∈ N * K , |L(θ t )-L(θ * )| Cσ 2 t β + D β log(2/δ) t β +2M Θ 2 2V log(1 + κ) κ + log(4/δ) κ .
(5.21) for some constant D β depending on the parameters l, α, γ 1 , a 1 and where

C = C 1 if β < 1 and C 2 otherwise.
Proof. For the sake of simplicity, we place ourselves in the special case where Θ is compact, but tedious calculations would lead to similar results under less restrictive assumptions. We therefore introduce the quantities M and M 1 that satisfy

g t (θ * n ) 2 M 2 and θ t -θ * n M 2 1 .
We now turn to the proof of the result:

L(θ t ) -L(θ * ) ≤ 2 sup θ∈Θ | L n (θ) -L(θ)| + L n (θ t ) -L n (θ * n ).
Taking a union bound we directly get:

P L(θ t ) -L(θ * ) l 2 σ 2 t β C + P | L n (θ t ) -L n (θ * n )| l 2 σ 2 t β C + 2 ) P 1 + P(sup θ∈Θ | L n (θ) -L(θ)| 4 ) P 2
The analysis of P 2 is classical and we refer to Clémençon et al. (2008b) to obtain that for all δ ∈ (0, 1), we have with probability at least 1 -δ,

sup θ∈Θ | L n (θ) -L(θ)| ≤ M H 2 2V log(1 + κ) κ + log(1/δ) κ .
(5.22)

We now focus on the second term. Nesterov & Nesterov (2004) for instance) we have:

Using L n (θ) -L n (θ * n ) l 2 θ -θ * n 2 (see
P | L n (θ t ) -L n (θ * n )| - l 2 σ 2 t β C 2 P θ t -θ * n 2 - σ 2 t β C l
Applying the recursion we get:

θ t+1 -θ * n 2 = θ t -θ * n 2 -2γ t g t (θ t ) T (θ t -θ * n ) + γ 2 t g t (θ t ) 2 = θ t -θ * n 2 -2γ t (g t (θ t ) -∇ L n (θ t )) T (θ t -θ * n ) -2γ t ∇ L n (θ t ) T (θ t -θ * n ) + γ 2 t g t (θ t ) 2
and using (5.17):

g t (θ t ) 2 2l(g t (θ t ) -g t (θ * n )) T (θ t -θ * n ) + 2 g t (θ * n ) 2 ) = 2l(g t (θ t ) -∇ L n (θ t ) -g t (θ * n )) T (θ t -θ * n ) + 2l(∇ L n (θ t )) T (θ t -θ * n ) + 2 g t (θ * n ) 2
Chapter 5. Stochastic Gradient Descent based on incomplete U -Statistics

which with ãt := θ t+1 -θ * n 2 gives ãt+1 ãt (1 -2αγ t ) + 2γ 2 t σ 2 n (θ * n ) -2γ t (g t (θ t ) -∇ L n (θ t )) T (θ t -θ * n ) -2γ 2 t l(g t (θ * n )) T (θ t -θ * n ) + 2γ 2 t ( g t (θ * n ) 2 -σ 2 n (θ * n ))
).

An immediate recursion leads to

ãt+1 ã1 t j=1 (1 -2αγ j ) + 2σ 2 n (θ * n ) t j=1 γ 2 j t k=j+1 (1 -2αγ k ) + 2 t j=1 γ 2 j t k=j+1 (1 -2αγ k ) ( g t (θ * n ) 2 -σ 2 n (θ * n )) -2 t j=1 γj t k=j+1 (1 -2αγ k ) (g t (θ t ) -∇ L n (θ t )) T (θ t -θ * n ) -2l t j=1 γ 2 j t k=j+1 (1 -2αγ k ) g t (θ * n ) T (θ t -θ * n )
The first two terms are analyzed in 5.6.1. We turn now to the tree remaining terms and we introduce the following quantities:

1. S 1,t = 2 t j=1 γ 2 j t k=j+1 (1 -2αγ k ) ( g t (θ * n ) 2 -σ 2 n (θ * n )) 2. S 2,t = 2 t j=1 γj t k=j+1 (1 -2αγ k ) (g t (θ t ) -∇ l n (θ t )) T (θ t -θ * n ) 3. S 3,t = 2l t j=1 γ 2 j t k=j+1 (1 -2αγ k ) g t (θ * n ) T (θ t -θ * n )
Placing ourself under the conditional probability and applying a union bound yields

P n   θ t+1 -θ * n 2 ã1 t j=1 (1 -2αγ j ) + 2σ 2 n (θ * n ) t j=1 γ 2 j t k=j+1 (1 -2αγ k ) +   P n (S 1,t 3 ) + P n (S 2,t 3 ) + P n (S 3,t 3 )
Under our assumptions, we have

g t (θ * n ) 2 M 2 , |g t (θ * n ) T (θ t -θ * n )| M M 1 and |(g t (θ t ) -∇ L n (θ t )) T (θ t -θ * n )| (2lM 1 + M )M 1
. Applying Azuma's A.9 inequality yields the following bounds (conditionally upon the observations):

P(S 1,t ) exp -2 4M 4 t j=1 γ 4 j t k=j+1 (1 -2αγ k ) 2 P(S 2,t ) exp -2 8M 2 1 (2lM 1 + M ) 2 t j=1 γ2 j t k=j+1 (1 -2αγ k ) 2 P(S 3,t ) exp -2 8M 2 M 2 1 t j=1 γ 4 j t k=j+1 (1 -2αγ k ) 2

Case β < 1

We have for β < 1:

T t=1 γ 4 j T k=j+1 (1 -2αγ k ) 2 T j=t 0 +1 (1 -2αγ j ) 2 t 0 j=1 γ 4 j + γ 2 t 0 T j=t 0 +1 γ 2 j t k=j+1 (1 -2αγ k ) 2 exp(-4α T j=t 0 +1 γ j ) exp(4αl T j=t 0 +1 γ 2 j ) t 0 j=1 γ 4 j + γ 2 t 0 ( 1 2α + l T j=t 0 +1 γ 2 j ) 2 exp(-4αγ 1 (T + 1) (1-β) 2(1 -β) ) exp( 4αlγ 2 1 2β -1 ) 4β 4β -1 γ 4 1 + γ 2 1 2 2β T 2β ( 1 2α + 2lβ 2β -1 ) 2 = γ 2 1 2 2β T 2β ( 1 2α + 2lβ 2β -1 ) 2 + o( 1 T 2β )
where we used

T j=1 x 2 j ( T j=1
x j ) 2 for x j = γ j t k=j+1 (1 -2αγ k ) 0 for the last inequality. We now analyze the second sum:

T j=1 γ2 j T k=j+1 (1 -2αγ k ) 2 T j=t 0 +1 (1 -2αγ j ) 2 t 0 j=1 γ2 j + γt 0 T j=t 0 +1 γj T k=j+1 (1 -2αγ k ) 2 exp(-4α T j=t 0 +1 γ j ) exp(4αl T j=t 0 +1 γ 2 j ) t 0 j=1 γ2 j + γt 0 T j=t 0 +1 1 -(1 -2αγ j ) 2α T k=j+1 (1 -2αγ k ) exp(-4α T j=t 0 +1 γ j ) exp(4αl T j=t 0 +1 γ 2 j ) t 0 j=1 γ2 j + γt 0 2α exp(-4αγ 1 (T + 1) (1-β) 2(1 -β) ) exp( 4αlγ 2 1 2β -1 ) 8β 2β -1 γ 2 1 + γ 1 2 β αT β = γ 1 2 β αT β + o( 1 T β )
which concludes this case.

Chapter 5. Stochastic Gradient Descent based on incomplete U -Statistics

Case β = 1

We have:

T t=1 γ 4 j T k=j+1 (1 -2αγ k ) 2 T t=1 γ 4 j exp(-4α T j=t+1 γ j ) exp(4αl T j=t+1 γ 2 j ) γ 4 1 exp(8αlγ 2 1 ) T t=1 (j + 1) 4αγ 1 j 4 (T + 1) 4αγ 1 γ 4 1 exp(8αlγ 2 1 )2 4αγ 1 (T + 1) 4αγ 1 T t=1 1 j 4-4αγ 1
We thus need to distinguish several cases:

1. If 2 < 4αγ 1 < 3 then T t=1 1 j 4-4αγ 1 1 + 1 3 -4αγ 1 = 4 -4αγ 1 3 -4αγ 1 so T t=1 γ 4 j T k=j+1 (1 -2αγ k ) 2 γ 4 1 exp(8αlγ 2 1 )2 4αγ 1 (T + 1) 4αγ 1 4 -4αγ 1 3 -4αγ 1 2. 4αγ 1 = 3 then T t=1 1 j 4-4αγ 1 1 + log(T ) so T t=1 γ 4 j T k=j+1 (1 -2αγ k ) 2 γ 4 1 exp(8αlγ 2 1 )2 3 1 + log(T ) (T + 1) 3 3. 4αγ 1 > 3 then T t=1 1 j 4-4αγ 1 (T + 1) 4αγ 1 -3 4αγ 1 -3 so T t=1 γ 4 j T k=j+1 (1 -2αγ k ) 2 γ 4 1 exp(8αlγ 2 1 )2 4αγ 1 (4αγ 1 -3)(T + 1) 3 CHAPTER 6

On Graph Reconstruction via Empirical Risk Minimization

Abstract The problem of predicting connections between a set of data points finds many applications, in systems biology and social network analysis among others. This chapter focuses on the graph reconstruction problem, where the prediction rule is obtained by minimizing the average error over all n(n -1)/2 possible pairs of the n nodes of a training graph. Our first contribution is to derive learning rates of order O(log n/n) for this problem, significantly improving upon the rates of order O(1/ √ n) established in the seminal work of [START_REF] Bibliography | Statistical Inference on Graphs[END_REF]. Strikingly, these fast rates are universal, in contrast to similar results known for other statistical learning problems (e.g., classification, density level set estimation, ranking, clustering) which require strong assumptions on the distribution of the data. Motivated by applications to large graphs, our second contribution deals with the computational complexity of graph reconstruction. Specifically, we investigate to which extent the learning rates can be preserved when replacing the empirical reconstruction risk by a computationally cheaper Monte-Carlo version, obtained by sampling with replacement B n 2 pairs of nodes. Finally, we illustrate our theoretical results by numerical experiments on synthetic and real graphs.

Introduction

Although statistical learning theory mainly focuses on establishing universal rate bounds (i.e., which hold for any distribution of the data) for the accuracy of a decision rule based on training observations, refined concentration inequalities have recently helped understanding conditions on the data distribution under which learning paradigms lead to faster rates. In binary classification, i.e., the problem of learning to predict a random binary label Y ∈ {-1, +1} from on an input random variable X based on independent copies (X 1 , Y 1 ), . . . , (X n , Y n ) of the pair (X, Y ), rates faster than 1/ √ n are achieved when little mass in the vicinity of 1/2 is assigned by the distribution of the random variable η(X) = P{Y = +1 | X}. This condition and its generalizations are referred to as the Mammen-Tsybakov noise conditions (see [START_REF] Mammen | Smooth discrimination analysis[END_REF][START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF][START_REF] Massart | Risk bounds for statistical learning[END_REF]. It has been shown that a similar phenomenon occurs for various other statistical learning problems. Indeed, specific conditions under which fast rate results hold have been exhibited for density level set estimation [START_REF] Rigollet | Fast rates for plug-in estimators of density level sets[END_REF], (bipartite) ranking (Clémençon et al., 2008a;[START_REF] Clémençon | Minimax learning rates for bipartite ranking and plug-in rules[END_REF][START_REF]Surrogate regret bounds for bipartite ranking via strongly proper losses[END_REF], clustering [START_REF] Antos | Individual convergence rates in empirical vector quantizer design[END_REF][START_REF] Clémençon | A statistical view of clustering performance through the theory of U-processes[END_REF] and composite hypothesis testing [START_REF] Clémençon | Overlaying classifiers: a practical approach to optimal scoring[END_REF].

In this chapter, we consider the supervised learning problem on graphs referred to as graph reconstruction, rigorously formulated by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF]. The objective of graph reconstruction is to predict the possible occurrence of connections between a set of objects/individuals known to form the nodes of an undirected graph. Precisely, each node is described by a random vector X which defines a form of conditional preferential attachment: 92 Chapter 6. On Graph Reconstruction via Empirical Risk Minimization one predicts whether two nodes are connected based on their features X and X . This statistical learning problem is motivated by a variety of applications such as systems biology (e.g., inferring protein-protein interactions or metabolic networks, see [START_REF] Jansen | A Bayesian networks approach for predicting protein-protein interactions from genomic data[END_REF][START_REF] Kanehisa | Prediction of higher order functional networks from genomic data[END_REF] and social network analysis (e.g., predicting future connections between users, see [START_REF] Liben-Nowell | The link prediction problem for social networks[END_REF]. It has recently been the subject of a good deal of attention in the machine learning literature (see [START_REF] Vert | Supervised graph inference[END_REF][START_REF] Bibliography | Statistical Inference on Graphs[END_REF][START_REF] Shaw | Learning a Distance Metric from a Network[END_REF], and is also known as supervised link prediction [START_REF] Lichtenwalter | New perspectives and methods in link prediction[END_REF][START_REF] Cukierski | Graph-based features for supervised link prediction[END_REF]. The learning task is formulated as the minimization of a reconstruction risk, whose natural empirical version is the average prediction error over the n(n -1)/2 pairs of nodes in a training graph of size n. Under standard complexity assumptions on the set of candidate prediction rules, excess risk bounds of the order O(1/ √ n) for the empirical risk minimizers have been established by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] based on a representation of the objective functional very similar to the first Hoeffding decomposition for second-order U -statistics (see [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF]. However, the computational complexity of finding an empirical risk minimizer, which scales at least as O(n 2 ) since the empirical graph reconstruction risk involves summing up over n(n -1)/2 terms was ignored. This makes the approach impractical when dealing with large graphs commonly found in many applications.

Building up on the above work, our contributions to statistical graph reconstruction are twofold:

Universal fast rates. We prove that a fast rate of order O(log n/n) is always achieved by empirical reconstruction risk minimizers, in absence of any restrictive condition imposed on the data distribution. This is much faster than the O(1/ √ n) rate established by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF]. Our analysis is based on a different decomposition of the excess of reconstruction risk of any decision rule candidate, involving the second Hoeffding representation of a U -statistic approximating it, as well as appropriate maximal/concentration inequalities.

Scaling-up ERM. We investigate the performance of minimizers of computationally cheaper Monte-Carlo estimates of the empirical reconstruction risk, built by averaging over B n 2 pairs of vertices drawn with replacement. The rate bounds we obtain highlight that B plays the role of a tuning parameter to achieve an effective trade-off between statistical accuracy and computational cost. Numerical results based on simulated graphs and real-world networks are presented in order to support these theoretical findings.

The chapter is organized as follows. In Section 6.2, we present the probabilistic setting for graph reconstruction and recall state-of-the-art results. Section 6.3 provides our fast rate bound analysis, while Section 6.4 deals with the problem of scaling-up reconstruction risk minimization to large graphs. Numerical experiments are displayed in Section 6.5, and a few concluding remarks are collected in Section 6.6.

Background and Preliminaries

We start by describing at length the probabilistic framework we consider for statistical inference on graphs, as introduced by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF]. We then briefly recall the related theoretical results documented in the literature.

A Probabilistic Setup for Preferential Attachment

In this chapter, G = (V, E) is an undirected random graph with a set V = {1, . . . , n} of n ≥ 2 vertices and a set E = {e i,j : 1 ≤ i = j ≤ n} ∈ {0, 1} n(n-1) describing its edges: for all i = j, we have e i,j = e j,i = +1 if the vertices i and j are connected by an edge and e i,j = e j,i = 0 otherwise. We assume that G is a Bernoulli graph, i.e. the random variables e i,j , 1 ≤ i < j ≤ n, are independent labels drawn from a Bernoulli distribution Ber(p) with parameter p = P{e i,j = +1}, the probability that two vertices of G are connected by an edge. The degree of each vertex is thus distributed as a binomial with parameters n and p, which can be classically approximated by a Poisson distribution of parameter λ > 0 in the limit of large n, when np → λ.

Whereas the marginal distribution of the graph G is that of a Bernoulli graph (also sometimes abusively referred to as a random graph), a form of conditional preferential attachment is also specified in the framework considered here. Precisely, we assume that, for all i ∈ V , a continuous r.v. X i , taking its values in a separable Banach space X , describes some features related to vertex i. The X i 's are i.i.d. with common distribution µ(dx) and, for any i = j, the random pair (X i , X j ) models some information useful for predicting the occurrence of an edge connecting the vertices i and j. Conditioned upon the features (X 1 , . . . , X n ), any binary variables e i,j and e k,l are independent only if {i, j} ∩ {k, l} = ∅. The conditional distribution of e i,j , i = j, is supposed to depend on (X i , X j ) solely, described by the posterior preferential attachment probability:

η (X i , X j ) = P {e i,j = +1 | (X i , X j )} . (6.1)
For instance, ∀(x 1 , x 2 ) ∈ X 2 , η (x 1 , x 2 ) can be a certain function of a specific distance or similarity measure between x 1 and x 2 , as in the synthetic graphs described in Section 6.5.

The conditional average degree of the vertex i ∈ V given X i (respectively, given (X 1 , . . . , X n )) is thus (n -1) x∈X η(X i , x)µ(dx) (respectively, j =i η(X i , X j )). Observe incidentally that, equipped with these notations, p = (x,x )∈X 2 η(x, x )µ(dx)µ(dx ).

Hence, the 3-tuples (X i , X j , e i,j ), 1 ≤ i < j ≤ n, are non-i.i.d. copies of a generic random vector (X 1 , X 2 , e 1,2 ) whose distribution L is given by the tensorial product µ(dx 1 )⊗µ(dx 2 )⊗ Ber(η(x 1 , x 2 )), which is fully described by the pair (µ, η). Observe also that the function η is symmetric by construction:

∀(x 1 , x 2 ) ∈ X 2 , η(x 1 , x 2 ) = η(x 2 , x 1 ).
In this framework, the learning problem introduced by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF], referred to as graph reconstruction, consists in building a symmetric reconstruction rule g : X 2 → {0, 1}, from a training graph G, with nearly minimum reconstruction risk

R(g) = P {g(X 1 , X 2 ) = e 1,2 } , (6.2)
thus achieving a comparable performance to that of the Bayes rule g * (x 1 , x 2 ) = I{η(x 1 , x 2 ) > 1/2}, whose risk is given by R

* = E[min{η(X 1 , X 2 ), 1 -η(X 1 , X 2 )}] = inf g R(g).

Related Results on Empirical Risk Minimization

Based on the labeled sample D n = {(X i , X j , e i,j ) : 1 ≤ i < j ≤ n} related to G, (6.2) is replaced by its empirical version1 :

R n (g) = 2 n(n -1) 1≤i<j≤n I {g(X i , X j ) = e i,j } .

(6.3)

We denote by g n , an empirical risk minimizer g n , i.e a solution of the optimization problem min g∈G R n (g), where G is a class of reconstruction rules of controlled complexity, hopefully rich enough to yield a small bias inf g∈G R(g) -R * . The performance of g n is measured by its excess risk R( g n ) -inf g∈G R(g), which can be bounded if we can derive probability inequalities for the maximal deviation

sup g∈G | R n (g) -R(g)|.
(6.4)

In the framework of classification, the flagship problem of statistical learning theory, the empirical risk is of the form of an average of i.i.d. r.v.'s, so that results pertaining to empirical process theory can be readily used to obtain bounds for the performance of empirical error minimization. Unfortunately, the empirical risk (6.3) is a sum of dependent variables. Following in the footsteps of Clémençon et al. (2008a), the work of [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] circumvents this difficulty by means of a representation of R n (g) as an average of sums of i.i.d. r.v.'s, namely

1 n! σ∈Sn 1 n/2 n 2 i=1 I{g(X σ(i) , X σ(i+ n 2 ) ) = e σ(i),σ(i+ n 2 ) },
where the sum is taken over all permutations of S n , the symmetric group of order n, and u denotes the integer part of any u ∈ R. Very similar to the first Hoeffding decomposition for U -statistics (see Lee, 1990b), this representation reduces the first order analysis of the concentration properties of (6.4) to the study of a basic empirical process (see Biau & Bleakley, 2006, Lemma 3.1). [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] thereby establish rate bounds of the order O(1/ √ n) for the excess of reconstruction risk of g n under appropriate complexity assumptions (namely, G is of finite VC-dimension). Note incidentally that (6.3) is a U -statistic only when the variable η(X 1 , X 2 ) is almost-surely constant (see [START_REF] Janson | The asymptotic distributions of generalized U-statistics with applications to random graphs[END_REF], for an asymptotic study of graph reconstruction in this restrictive context).

We finally point out that instead of estimating the reconstruction risk by R n (g), one could split the training dataset into two halves and consider the unbiased estimate of R(g) given by

1 n/2 n/2 i=1 I{g(X i , X i+ n/2 ) = e i,i+ n/2 }.
(6.5)

Since only independent r.v.'s are involved in the sum (6.5), the analysis of its generalization ability is much simpler. In particular, fast rates can be obtained under the following classical noise condition (see [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF]Boucheron et al., 2005c;[START_REF] Massart | Risk bounds for statistical learning[END_REF].

Assumption 8. There exists β > 0 and α ∈ [0, 1] such that for all t > 0: -α) .

P η(X 1 , X 2 ) - 1 2 t βt α/(1
One can then show that minimizers of (6.5) achieve a learning rate of order O(( log n n ) 1/(2-α) ). We make the following observations:

• Assumption 8 is always satisfied for α = 0 and corresponds to the classical learning rate of O( log(n)/n) obtained by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF].

• Fast learning rates of the same order as the one we obtained for the minimizer of R n (g) are achieved if and only if Assumption 8 is satisfied with α = 1. This corresponds to the case where the posterior preferential attachment probability η stays bounded away from 1/2 with probability one (cf [START_REF] Massart | Risk bounds for statistical learning[END_REF].

In fact, the assumption α = 1 is very restrictive. We illustrate this using the following toy example. Let N 0 ∈ N * . For each node 1 i n, we observe X i = (X 1 i , X 2 i ), where X 1 i and X 2 i are two distinct elements uniformly drawn from P({1, ..., N 0 }). Consider now the case where two nodes are likely to be connected if they share common preferences, say e i,j ∼ Ber(#(X i ∩X j )/#(X i ∪X j )). One can easily check that P(|η(X 1 , X 2 )-1 2 | = 0) > 0, so fast learning rates cannot be obtained for minimizers of (6.5). In contrast, the fast rates of Theorem 6.1 always hold for minimizers of R n (g), as shall be seen in next section.

Empirical Reconstruction is Always Fast!

In this section, we show that the rate bounds established by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] can be largely improved without any additional assumptions. Precisely, we prove that fast learning rates of order O(log n/n) are always attained by the minimizers of the empirical reconstruction risk (6.3), as revealed by the following theorem. For simplicity, our results are stated for the case of the classic 0-1 loss I{g(X i , X j ) = e i,j }, but we point out that the same arguments directly apply to other loss functions (see Section 5.4 of Boucheron et al., 2005b, for instance). This includes the following two practical examples:

• In order to solve the ERM problem with efficient optimization methods (e.g. gradientbased), one typically considers a convex surrogate of the 0-1 loss (e.g., hinge loss, logistic loss), as done in our experiments.

• Real-world networks can be very sparse (i.e., they have very few edges), leading to a highly imbalanced prediction problem. One may then consider a weighted loss in order to assign a higher cost to errors on edges than to errors on non-edges.

We now state our main theorem:

Theorem 6.1. (FAST RATES) Let g n be any minimizer of the empirical reconstruction risk (6.3) over a class G of finite VC-dimension V < +∞. For all δ ∈ (0, 1), we have w.p. at least

1 -δ: ∀n ≥ 2, R( g n ) -R * ≤ 2 inf g∈G R(g) -R * + C × V log(n/δ) n ,
where C < +∞ is a universal constant.2 Remark 6.2. (ON THE BIAS TERM) Apart from its remarkable universality, Theorem 6.1 takes the same form as in the case of empirical minimization of U-statistics (Clémençon et al., 2008a, Corollary 6), with the same constant 2 in front of the bias term inf g∈G R(g) -R * . As can be seen from the proof, this constant has no special meaning and can be replaced by any constant strictly larger than 1 at the cost of increasing the constant C. Note that the O(1/ √ n) rate obtained by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] has a factor 1 in front of the bias term. Therefore, Theorem 6.1 provides a significant improvement unless the bias overly dominates the second term of the bound (i.e., the complexity of G is too small).

Remark 6.3. (ON COMPLEXITY ASSUMPTIONS) We point out that a similar result can be established under weaker complexity assumptions involving Rademacher averages . As may be seen by carefully examining the proof of Theorem 6.1, this would require to use the moment inequality for degenerate U -processes stated in (Clémençon et al., 2008a, Theorem 11) instead of that proved by [START_REF] Arcones | U-processes indexed by Vapnik-Chervonenkis classes of functions with applications to asymptotics and bootstrap of U-statistics with estimated parameters[END_REF].

In the rest of this section, we outline the main ideas used to obtain this result. We rely on some arguments used in the fast rate analysis for empirical minimization of U -statistics (Clémençon et al., 2008a), although these results only hold true under restrictive distributional assumptions. Whereas the quantity (6.3) is not a U -statistic, one may decompose the difference between the excess of reconstruction risk of any candidate rule g ∈ G and its empirical counterpart as the sum of its conditional expectation given the X i 's, which is a U -statistic, plus a residual term. In order to explain the main argument underlying the present analysis, additional notation is required. Set

H g (x 1 , x 2 , e 1,2 ) = I{g(x 1 , x 2 ) = e 1,2 } q g (x 1 , x 2 , e 1,2 ) = H g (x 1 , x 2 , e 1,2 ) -H g * (x 1 , x 2 , e 1,2 )
for any (x 1 , x 2 , e 1,2 ) ∈ X × X × {0, 1}. Denoting by Λ(g) = R(g) -R * = E[q g (X 1 , X 2 , e 1,2 )] the excess reconstruction risk with respect to the Bayes rule, its empirical estimate is given by

Λ n (g) = R n (g) -R n (g * ) = 2 n(n -1) 1≤i<j≤n q g (X i , X j , e i,j ).
For all g ∈ G, one may write:

Λ n (g) -Λ(g) = U n (g) + W n (g), (6.6)
where

U n (g) = E [Λ n (g) -Λ(g) | X 1 , . . . , X n ] = 2 n(n -1) 1≤i<j≤n q g (X i , X j ) -Λ(g)
is a U -statistic of degree 2 with symmetric kernel q g (X 1 , X 2 ) -Λ(g), where we denote

q g (X 1 , X 2 ) = E[q g (X 1 , X 2 , e 1,2 ) | X 1 , X 2 ], and W n (g) = 2 n(n-1) i<j {q g (X i , X j , e i,j ) - q g (X i , X j )}.
Equipped with this notation, we can now sketch the main steps of the proof of the fast rate bound stated in Theorem 6.1. It is based on Eq. (6.6) combined with two intermediary results, each providing a control of one of the terms involved in it. The second order analysis carried out by Clémençon et al. (2008a) shows that the small variance property of U -statistics may yield fast learning rates for empirical risk minimizers when U -statistics are used to estimate the risk, under a certain "low-noise" condition. One of our main findings is that this condition is always fulfilled for the specific U -statistic U n (g) involved in the decomposition (6.6) of the excess of reconstruction risk of any rule candidate g, as shown by the following lemma. Lemma 6.4. (VARIANCE CONTROL) For any distribution L and any reconstruction rule g, we have

Var

(E [q g (X 1 , X 2 , e 1,2 ) | X 1 ]) ≤ Λ(g).
The fundamental reason for the universal character of this result lies in the fact that the empirical reconstruction risk is not an average over all pairs (i.e., a U-statistic of order 2) but an average over randomly selected pairs (random selection being ruled by the function η). The resulting smoothness is the key ingredient allowing us to establish the desired property.

Empirical reconstruction risk minimization over a class G being equivalent to minimization of Λ n (g) -Λ(g), the result below, combined with (6.6), proves that it also boils down to minimizing U n (g) under appropriate conditions on G, so that the fast rate analysis of Clémençon et al. (2008a) can be extended to graph reconstruction.

Lemma 6.5. (UNIFORM APPROXIMATION) Under the same assumptions as in Theorem 6.1, for any δ ∈ (0, 1), we have with probability larger than 1 -δ: ∀n ≥ 2,

sup g∈G W n (g) ≤ C × V log(n/δ) n ,
where C < +∞ is a universal constant.

The proof relies on classical symmetrization and randomization tricks combined with the decoupling method, in order to cope with the dependence structure of the variables and apply maximal/concentration inequalities for sums of independent random variables (see [START_REF] De La Pena | Decoupling : from dependence to independence[END_REF].

Based on the above results, Theorem 6.1 can then be derived by relying on the second Hoeffding decomposition (see [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF]. This allows us to write U n (g) as a leading term taking the form of a sum of i.i.d r.v.'s with variance 4V ar(E[q g (X 1 , X 2 , e 1,2 ) | X 1 ]), plus a degenerate U -statistic (i.e., a U -statistic of symmetric kernel h(x 1 , x 2 ) such that E[h(x 1 , X 2 )] = 0 for all x 1 ∈ X ). The latter can be shown to be of order O(1/n) uniformly over the class G by means of concentration results for degenerate U -processes.

Scaling-up Empirical Risk Minimization

The results of the previous section, as well as those of [START_REF] Bibliography | Statistical Inference on Graphs[END_REF], characterize the excess risk achieved by minimizers of the empirical reconstruction risk R n (g) but do not consider the computational complexity of finding such minimizers. For large training graphs, the complexity of merely computing R n (g) is prohibitive as the number of terms involved in the summation is O(n 2 ). In this section, we introduce a sampling-based approach to build approximations of the reconstruction risk with much fewer terms than O(n 2 ), so as to scale-up risk minimization to large graphs.

The strategy we propose, inspired from the notion of incomplete U -statistic (see Lee, 1990b), is of disarming simplicity: instead of the empirical reconstruction risk (6.3), we will consider

Extensions to Alternative Sampling Schemes

Although the Monte-Carlo scheme previously described is very appealing from a computational perspective, In this section, we show that the results of the previous section can be extended to other sampling techniques, such as Bernoulli sampling and sampling without replacement. Borrowing the terminology of survey theory, the (finite) population under study is the collection Θ n of all pairs of vertices of the graph G. Its cardinality is #Θ n = n(n -1)/2. In this context, a survey sample is any subset S of Θ n with (possibly random) cardinality m ≤ n(n -1)/2, referred to as its size. A survey scheme without replacement is thus any conditional probability distribution D on the power set of Θ n , P(Θ n ), given the data D n = {(X i , X j , e i,j ) : (i, j) ∈ Θ n }. The probability that the pair (i, j) ∈ Θ n belongs to the sample S drawn from D, conditioned upon D n , is denoted by π (i,j) = P D {(i, j) ∈ S} and termed a first order inclusion probability. The quantities π (i,j),(k,l) = P D {((i, j), (k, l)) ∈ S 2 }, for (i, j) = (k, l), are referred to as second order inclusion probabilities. Equipped with these notations, the Horvitz-Thompson version [START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF] of the empirical reconstruction risk of a rule g ∈ G based on a sample S ∼ D is then given by:

R (D) (g) = 2 n(n -1) (i,j)∈Λ i,j π i,j
I{g(X i , X j ) = e i,j }, (6.9)

where i,j = I{(i, j) ∈ S} for all (i, j) ∈ Θ n and 0/0 = 0 by convention. Provided that the π (i,j) 's are all strictly positive, (6.9) is an unbiased estimate of (6.3) and, when the size B ≤ n(n -1)/2 of the survey scheme is deterministic, its conditional variance given the training graph is V ar( (k,l) , where σ 2 (i,j),(k,l) is given by

R (D) (g) | D n ) = 4/(n(n -1)) 2 × (i,j) =(k,l) σ 2 (i,j),
I{g(X i , X j ) = e i,j } π (i,j) - I{g(X k , X l ) = e k,l } π (k,l) 2 × π (i,j),(k,l) -π (i,j) π (k,l)
for all (i, j) = (k, l). Two specific sampling techniques can naturally be considered.

Bernoulli sampling. Let B ≤ n(n -1)/2. This sampling plan corresponds to the situation where the (i,j) 's are i.i.d. Bernoulli r.v.'s with parameter 2B/(n(n -1)). In this case, the (random) size is a binomial variable of size n(n -1)/2 with B as expected value. Incidentally, we mention that Bernoulli sampling is a particular case of Poisson sampling (relaxing the assumption that the i,j 's are identically distributed), widely used for the purpose of graph sparsification (see e.g. Spielman, 2005, Section 6).

Sampling without replacement (SWOR). Fixing in advance B ≤ n(n -1)/2, one may uniformly draw a sample S among all samples of size B (there are n(n-1)/2 B such samples). In this case, we have π (i,j) = 2B/(n(n -1)) and π (i,j),(k,l) = 4B(B -1)/(n(n -1) 2 (n -2)) for all (i, j) = (k, l) in Θ n . This is a special case of rejective sampling, corresponding to the situations where the π (i,j) 's are all equal.

The following proposition reveals that, just like (6.7), the Horvitz-Thompson reconstruction risk (6.9), when based on SWOR or Bernoulli schemes, estimates the empirical reconstruction risk of rules in G uniformly well (provided that G is of finite VC dimension). Proposition 6.8. (UNIFORM DEVIATIONS) Suppose that G is of finite VC dimension V < +∞. For all δ ∈ (0, 1), we have with probability larger than 1 -δ: for all n ≥ 1, B ≤ shows the training graph, where the position of each node is given by its 2D feature vector. Figure 6.1(b) depicts the same graph after applying a random transformation R to the features. On this graph, the Euclidean distance with optimal threshold achieves a reconstruction error of 0.1311. In contrast, the reconstruction rule learned from B = 100 pairs of nodes (out of 1225 possible pairs) successfully inverts R and accurately recovers the original graph (Figure 6.1(c)). Its reconstruction error is 0.008 on the training graph and 0.009 on a held-out graph generated with the same parameters.

Synthetic Graph

We create a synthetic graph with n nodes as follows. Each node i has features X true i ∈ R q sampled uniformly over [0, 1]. We then add an edge between nodes that are at Euclidean distance smaller than some threshold τ , and introduce some noise by flipping the value of e i,j for each pair of nodes (i, j) independently with probability p. We then apply a random linear transformation R ∈ R q×q to each node to generate a "scrambled" version X i = RX true i of the nodes' features. The learning algorithm is only allowed to observe the scrambled features and must find a rule which accurately recovers the graph by solving the ERM problem above. Note that, denoting

D ij = R -1 X i -R -1 X j 2 , the posterior preferential attachment probability is given by η (X i , X j ) = (1 -p) • I{D ij ≤ τ } + p • I{D ij > τ }.
The process is illustrated in Figure 6.1. Using this procedure, we generate a training graph with n = 1, 000, 000 and q = 100. We set the threshold τ such that there is an edge between about 20% of the node pairs, and set p = 0.05. We also generate a test graph using the same parameters. We then sample uniformly with replacement B pairs of nodes from the training graph to construct our incomplete reconstruction risk. The reconstruction error of the resulting empirical risk minimizer is estimated on 1,000,000 pairs of nodes drawn from the test graph. We also compare the above scheme with an alternative strategy based on sampling nodes. Specifically, we randomly sample m nodes and use the reconstruction risk evaluated on the resulting sub-graph as an approximation to the risk on the full graph. To allow a fair comparison, we set m such that B m(m -1)/2 in order to get approximately the same computational complexity for both approaches. Table 6.2 compares the reconstruction error of both strategies for various values of B, averaged over 10 runs. Sampling nodes leads to significantly larger error than sampling pairs, as it only leverages information from a subset of training nodes. We have noticed experimentally that the error gap between these two strategies widens as the complexity of the class of reconstruction rules gets larger (for instance, if we increase the data dimension q). Figure 6.2 summarizes these results by displaying both the test error and the training time with respect to the number of terms in the risk estimate. For completeness, we also show the performance of the "dataset splitting" strategy (see Eq. 7 of the main text). It exhibits a good statistical/runtime trade-off but leads to suboptimal test error.

Real Network

Finally, we also validate our approach on Cit-HepTh, the high-energy physics theory citation network extracted from arXiv. 3 The graph has 27,770 nodes representing research chapters and 352,807 edges corresponding to a citation between two papers. We generate simple features based on the paper abstracts as follows. We first remove stop words and those with less than 4 characters, then apply a tokenizer and stemmer from the NLTK library4 and keep only the 300 most frequent words among all abstracts. Finally, we build a 300-dimensional bag-ofwords feature vector for each paper by counting the number of occurrences of these words in its abstract and applying an L 1 -norm normalization. We randomly split the nodes into a training set (80%) and a test set (20%). Note that the graph is very sparse: there is an edge between about 0.1% of the node pairs. Since classification error is not meaningful in such an imbalanced regime, we optimize a balanced error rate by sampling active edges with higher probability (this is equivalent to optimizing a weighted version of the reconstruction risk, see the remark at the beginning of Section 6.7). Note that the Euclidean distance (with threshold tuned on the training set) achieves a balanced test error of about 0.37.

Conclusion

In this chapter, we proved that the learning rates for ERM in the graph reconstruction problem are always of order O(log n/n). We also showed how sampling schemes applied to the population of edges (not nodes) can be used to scale-up such ERM-based predictive methods to very large graphs by means of a detailed rate bound analysis, further supported by empirical results. A first possible extension of this work would naturally consist in considering more general sampling designs, such as Poisson sampling (which generalizes Bernoulli sampling) used in graph sparsification (cf [START_REF] Spielman | Fast Randomized Algorithms for Partitioning, Sparsification, and Solving Linear Systems[END_REF], and investigating the properties of minimizers of Horvitz-Thompson versions of the reconstruction risk (see [START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF]. Another challenging line of future research is to extend the results of this chapter to more complex unconditional graph structures in order to account for properties shared by some realworld graphs (e.g., graphs with a power law degree distribution).

6.7 Technical Proofs 6.7.1 Proof of Lemma 6.4

For any reconstruction rule g, observe first that with probability one:

E[q g (X 1 , X 2 , e 1,2 ) | X 1 ] = E[E[q g (X 1 , X 2 , e 1,2 ) | X 1 , X 2 ] | X 1 ] = E X 2 [|1 -2η(X 1 , X 2 )|I{g(X 1 , X 2 ) = g * (X 1 , X 2 )}]]
Observing that we have

|1 -2η(X 1 , X 2 )| 2 ≤ |1 -2η(X 1 , X 2 )|
almost surely, and combining with Jensen inequality, we have

Var(E[q g (X 1 , X 2 , e 1,2 ) | X 1 ]) ≤ E X 1 [(E X 2 [|1 -2η(X 1 , X 2 )|I{g(X 1 , X 2 ) = g * (X 1 , X 2 )}]) 2 ] ≤ E[|1 -2η(X 1 , X 2 )|I{g(X 1 , X 2 ) = g * (X 1 , X 2 )}] = Λ(g).
6.7.2 Proof of Lemma 6.5

By definition, for all g, we have: ∀n ≥ 2,

W n (g) = 2 n(n -1) i<j {q g (X i , X j , e i,j ) -q g (X i , X j )} .
The proof relies on the key property: for all i = j, E[q g (X i , X j , e i,j ) -q g (X i , X j )|X i ] = E[q g (X i , X j , e i,j ) -q g (X i , X j )|X j ] = 0 almost surely. This basically implies that the process { W n (g)} g∈G "behaves" as a second order Rademacher Chaos. Mimicking the techniques introduced in De la [START_REF] De La Pena | Decoupling : from dependence to independence[END_REF], this can be deduced from the following two technical lemmas, which we prove separately in Section 6.7.3 and Section 6.7.4 for clarity. Lemma 6.10. (DECOUPLING) Let (X i ) n i=1 be an independent copy of the sequence (X i ) n i=1 . Consider r.v.'s valued in {0, 1}, {ẽ i,j , i < j}, conditionally independent given the X i 's and the X j 's and such that P(ẽ i,j = 1|X i , X j ) = η(X i , X j ). Then, for all q ≥ 1, we have:

E[sup g∈G | i<j q g (X i , X j , e i,j ) -q g (X i , X j )| q ] 4 q E[sup g∈G | i<j q g (X i , X j , ẽi,j ) -q g (X i , X j )| q ].
Thanks to the decoupling argument above, one can next introduce the following randomization.

Lemma 6.11. Let (σ i ) n i=1 and (σ i ) n i=1 be two independent sequences of i.i.d. Rademacher variables, independent from the (X i , X i , e i,j , ẽi,j )'s. Then, for all q ≥ 1, we have:

E[sup g∈G | i<j q g (X i , X j , ẽi,j )-q g (X i , X j )| q ] 4 q E[sup g∈G | i<j σ i σ j (q g (X i , X j , ẽi,j )-q g (X i , X j ))| q ].
Consider the conditional Rademacher average

E σ,σ [sup g∈G | i<j σ i σ j (q g (X i , X j , ẽi,j ) -q g (X i , X j ))| q ],
where E σ,σ denotes the expectation taken w.r.t. the (σ i , σ i )'s. Following Clémençon et al. (2008a), we can derive an exponential inequality using Markov's inequality and show that, w.p. at least 1 -δ,

sup g∈G | i<j σ i σ j ĥg (X i , X j , ẽi,j )| C × V log(n/δ) n .
6.7.3 Proof of Lemma 6.10

For any random variable ξ, we denote by L(ξ) its distribution. Let (X i ) n i=1 be an independent copy of (X i ) n i=1 , and F (respectively F ) be the sigma-field generated by {X 1 , ..., X n } (respectively {X 1 , ..., X n }). Let e i,j , 1 i < j n be Bernoulli random variables such that P(e i,j = 1|F, F ) = η(X i , X j ) (i.e, the conditional distribution of e i,j depends on (X i , X j ) only). As in De la [START_REF] De La Pena | Decoupling : from dependence to independence[END_REF], let (σ i ) n i=1 be independent Rademacher variables and define:

Z i = X i if σ i = 1 and X i otherwise, Z i = X i if σ i = 1 and X i otherwise.
Conditionally upon the X i and X i , the random vector (Z i , Z i ) takes the values (X i , X i ) or (X i , X i ), each with probability 1/2. In particular, we have (see De la Pena & Giné, 1999):

L(X 1 , ..., X n , X 1 , ..., X n ) = L(Z 1 , ..., Z n , Z 1 , ..., Z n ).
(6.10)

Let {ẽ i,j , 1 i < j n} be Bernoulli random variables such that P(ẽ i,j = 1|F, F ) = η(X i , X j ) and define for i < j:

êi,j =            e i,j if σ i = 1 and σ j = -1 e i,j if σ i = -1 and σ j = 1 ẽi,j if σ i = 1 and σ j = 1 ẽj,i if σ i = -1 and σ j = 1.
We also recall the following notations:

H g (x 1 , x 2 , e 1,2 ) = I{g(x 1 , x 2 ) = e 1,2 }, q g (x 1 , x 2 , e 1,2 ) = H g (x 1 , x 2 , e 1,2 ) -H g * (x 1 , x 2 , e 1,2 ) q g (X 1 , X 2 ) = E[q g (X 1 , X 2 , e 1,2 )|X 1 , X 2 ]
Let ĥg = q g -q g and notice that for all i < j:

E σ [ ĥg (Z i , Z j , êi,j )] = 1 4 ĥg (X i , X j , e i,j )+ ĥg (X i , X j , e i,j )+ ĥg (X i , X j , ẽi,j )+ ĥg (X i , X j , ẽj,i ) ,

where E σ denotes the expectation taken with respect to σ 1 , ..., σ n . Moreover, using

E[ ĥg (X i , X j , e i,j )|F] = E[ ĥg (X i , X j , e i,j )] = 0 and E[ ĥg (X i , X j , ẽi,j )|F] = E[q g (X i , X j , ẽi,j )|X i ] -E[E[q g (X i , X j , ẽi,j )|X i , X j ]|X i ] = 0, we easily get ĥg (X i , X j , e i,j ) = 4E[ ĥg (Z i , Z j , êi,j )]|F].
For all q > 1, we therefore have:

E[sup g∈G | i<j ĥg (X i , X j , e i,j )| q ] 4 q E[sup g∈G | i<j ĥg (Z i , Z j , êi,j )| q ].
We now use (6.10) combined with the fact that by construction, the law of êi,j only depends on the realizations Z i , Z j , i.e., P(ê i,j = 1|Z i , Z j ) = η(Z i , Z j ), to obtain

E[sup g∈G | i<j ĥg (Z i , Z j , êi,j )| q ] = E[sup g∈G | i<j ĥg (X i , X j , ẽi,j )| q ],
which concludes the proof.

Proof of Lemma 6.11

In this section, we find it more convenient to work with sums over {1 i = j n} than sums over {1 i < j n}, so that for i < j and any random variables a i,j , we set a j,i = a i,j .

Using the symmetry of our problem we have:

2 q E[sup g∈G | i<j ĥg (X i , X j , ẽi,j )| q ] = E[sup g∈G | i =j ĥg (X i , X j , ẽi,j )| q ],
where ĥg is defined as in Section 6.7.3. Re-using the notations used in Section 6.7.3, we further introduce (X i ) n i=1 , a copy of (X i ) n i=1 , independent from F, F , and denote by F its sigma-field. Let ẽ i,j , 1 i < j n Bernoulli random variables such that P(ẽ i,j = 1|F, F , F ) = η(X i , X j ). We now use classical randomization techniques and introduce our "ghost" sample:

E[sup g∈G | i =j ĥg (X i , X j , ẽi,j )| q ] = E[sup g∈G | i =j ĥg (X i , X j , ẽi,j ) -E F [ ĥg (X i , X j , ẽ i,j )]| q ] E    sup g∈G n j=1 i =j ĥg (X i , X j , ẽi,j ) -ĥg (X i , X j , ẽ i,j ) q    .
where E F denotes expectation with respect to the (X i ) n i=1 . Let (σ i ) n i=1 be independent Rademacher variables, independent of F, F and F , then we have:

E[sup g∈G | n j=1 i =j ĥg (X i , X j , ẽi,j ) -ĥg (X i , X j , ẽ i,j )| q F] = E[sup g∈G | n j=1 σ j i =j ĥg (X i , X j , ẽi,j ) -ĥg (X i , X j , ẽ i,j )| q |F] 2 q E[sup g∈G | n j=1 σ j i =j ĥg (X i , X j , ẽi,j )| q |F],
and get:

E[sup g∈G | i =j ĥg (X i , X j , ẽi,j )| q ] 2 q E[sup g∈G | n j=1 σ j i =j ĥg (X i , X j , ẽi,j )| q ].
We now repeat the same argument but for the (X i ) n i=1 . Let (X i ) n i=1 be a copy of (X i ) n i=1 , independent of F, F , and denote by F its sigma-field. Let ẽ i,j , 1 i < j n be Bernoulli random variables such that P(ẽ i,j = 1|F, F , F ) = η(X i , X j ). Then:

E[sup g∈G | n j=1 σ j i =j ĥg (X i , X j , ẽi,j )| q ] = E sup g∈G | n i=1 j =i σ j ĥg (X i , X j , ẽi,j ) -E F [σ j ĥg (X i , X j , ẽ i,j )]| q E sup g∈G | n i=1 j =i σ j ĥg (X i , X j , ẽi,j ) -σ j ĥg (X i , X j , ẽ i,j )| q . Let (σ i ) n i=1 be a copy of (σ i ) n i=1 , independent of (σ i ) n i=1 , F, F , F , we have E[sup g∈G | n j=1 i =j σ j ĥg (X i , X j , ẽi,j ) -σ j ĥg (X i , X j , ẽ i,j )| q |F , (σ i ) n i=1 ] = E[sup g∈G | n i=1 σ i j =i σ j ĥg (X i , X j , ẽi,j ) -σ j ĥg (X i , X j , ẽ i,j )| q |F , (σ i ) n i=1 ] 2 q E[sup g∈G | n i=1 σ i j =i σ j ĥg (X i , X j , ẽi,j )| q |F , (σ i ) n i=1 ].
Finally, we get

E[sup g∈G | n j=1 i =j ĥg (X i , X j , ẽi,j )| q ] 4 q E[sup g∈G | i =j σ i σ j ĥg (X i , X j , ẽi,j )| q ].
6.7.5 Proof of Theorem 6.1

We prove a more general version of Theorem 1.

Theorem 6.12. For any θ > 0, with probability at least 1 -δ, the empirical risk minimizer g n satisfies:

R( g n ) -R(g ) θ(R(g ) -R(g * )) + (1 + θ + 1 θ )DV log(n/δ) n
for some universal constant D.

The version of the main text is obtained by taking θ = 1 and adding R(g ) -R(g * ) on both sides of the inequality.

Proof. Following the analysis of Clémençon et al. (2008a), for all g ∈ G we rewrite:

Λ n (g) -Λ(g) = 2(T n (g) -Λ(g)) + W n (g) + W n (g).
where

T n (g) = 1 n n i=1 h g (X i ) is a sum of i.i.d random variables with h g (X i ) = E[q g (X i , X j , e i,j )|X i ], W n (g) = 1 n(n -1) i =j h g (X i , X j ) is a degenerate U-statistic with symmetric kernel h g (X i , X j ) = E[q g (X i , X j , e i,j )|X i , X j ] + Λ(g) -h g (X i ) -h g (X j ) and W n (g) = 1 n(n -1) i =j ĥg (X i , X j , e i,j )
with ĥg (X i , X j , e i,j ) = q g (X i , X j , e i,j ) -E[q g (X i , X j , e i,j )|X i , X j ].

We also recall that we proved in Lemma 2 that

Var (E [q g (X 1 , X 2 , e 1,2 ) | X 1 ]) ≤ Λ(g).
As mentioned before, the fact that we can upper-bound the variance of h g (X) by its expectation is the key property that will allow us to derive fast rates. We now follow the analysis of Boucheron et al. (2005b) and introduce the following quantities:

• H = {h g : g ∈ G}.

•

H * = {αh g : g ∈ G, α ∈ [0, 1]}. • ψ(r) = ER n f ∈ H * : E[f (X) 2 ] r
, where R n denotes the Rademacher chaos taken over the observation X i .

• For r > 0, we note G r = rhg max(r, √ Λ(g))

: g ∈ G .

For all h ∈ G r , E[h] -h 2 and Var(h) r so that applying Bousquet's Inequality for the Supremum of Empirical Processes [START_REF] Bousquet | A bennett concentration inequality and its application to suprema of empirical processes[END_REF]; Boucheron et al. (2005b) to the class G r gives that with probability at least 1 -δ/6, for any g ∈ G:

Λ(g) -T n (g) max(r, Λ(g)) r 2E[ sup hg∈Gr Λ(g) -T n (g)] + r 2 log(6/δ) n + 8 log(1/δ) 3n . Since E[sup hg∈Gr (Λ(g) -T n (g))] 2ER n [G r ] 2ψ(r) we get: Λ(g) -T n (g) max(r, Λ(g)) r 4ψ(r) + r 2 log(6/δ) n + 8 log(6/δ) 3n .
We now apply Bernstein's Inequality to h g and using the fact that Var(h g ) Λ(g ) Λ(g) for any g ∈ G, we get that with probability at least 1 -δ/6:

T n (g ) -Λ(g ) max(r, Λ(g)) 2 log(6/δ) n + 8 log(6/δ) 3n .

Summing the two inequality and taking a union bound we get that with probability at least 1 -δ/3, for all g ∈ G:

Λ(g)-T n (g) +T n (g )-Λ(g ) max(r, Λ(g)) r 4ψ(r) +2r 2 log(6/δ) n + 16 log(6/δ) 3n .
We now rewrite T n (g) as:

T

n (g) = 1 2 (Λ(g) + Λ n (g) -W n (g) -W n (g))
which we substitute in the previous inequality and obtain:

3 2 (Λ(g) -Λ(g )) 1 2 (Λ n (g) -Λ n (g )) + W n (g ) -W n (g) + W n (g ) -W n (g)
+ max(r, Λ(g)) r 4ψ(r) + 2r 2 log(6/δ) n + 16 log(6/δ) 3n .

We take g = g n so that Λ n ( g n ) -Λ n (g ) 0 and use Lemma 3 together with a result from Clémençon et al. (2008a) to obtain that with probability at least 1 -2δ/3:

W n (g ) -W n (g) + W n (g ) -W n (g) 2 sup g∈G |W n (g)| + 2 sup g∈G | W n (g)| 4CV log(3n/δ) n .
We finally get that with probability at least 1 -δ:

Λ( g n )-Λ(g ) 4CV log(3n/δ) n + max(r, Λ( g n )) r × 4ψ(r)+2r 2 log(6/δ) n + 16 log(6/δ) 3n .
Now, we either have Λ( g n ) r 2 , in which case we have in particular Λ( g n ) -Λ(g ) r 2 , or Λ( g n ) r 2 . Under the latter hypothesis: Substituting r * (δ) 2 for r and using its definition in the previous bound gives:

Λ( g n ) -Λ(g ) 4CV log(3n/δ) n + Λ( g n )) r 4ψ (
Λ( g n ) -Λ(g ) 4CV log(3n/δ) n + Λ ( g n )r * (δ).
Now, using for all θ > 0:

Λ( g n )r * (δ) 1 2 2θ 1 + θ Λ( g n ) + 1 + θ 2θ r * (δ)
gives that with probability at least 1 -δ:

R( g n ) -R(g ) θ(R(g ) -R(g * )) + (θ + 1) 2 4θ r * (δ) + (1 + θ)4CV log(3n/δ).
Putting all the pieces back together, we have shown

R( g n ) -R(g ) max(r * (δ) 2 , θ(R(g ) -R(g * )) + (θ + 1) 2 4θ r * (δ) + (1 + θ)4CV log(3n/δ)).
Convenient upper bound for r * (δ) can be found in Boucheron et al. (2005b):

r * (δ) CV log(n/δ) n ,
for some universal constant C. This concludes the proof.

6.7.6 Proof of Theorem 6.6

One may write for all g ∈ G, n ≥ 2 and B ≥ 1,

R B (g) -R n (g) = 1 B B b=1 Z b (g),
where

Z b (g) = i<j b (i, j) - 2 n(n -1) I{g(X i , X j ) = e i,j }
for all (g, b) ∈ G × {1, . . . , B}. Conditioned upon the (X i , X j , e i,j )'s, for all g ∈ G, the Z b (g)'s are i.i.d. centered random variables, bounded by 1. In addition, the collection G being of finite VC-dimension V , Sauer's lemma yields:

#{{I{g(X i , X j ) = e i,j } : g ∈ G} ≤ (1 + n(n -1)/2) V .
Applying Hoeffding's inequality to the Z b (g)'s conditioned upon the (X i , X j , e i,j )'s and the union bound leads to: ∀ζ > 0,

P sup g∈G 1 B B b=1 Z b (g) > ζ | {(X i , X j , e i,j )} (i,j)∈Λ ≤ 2(1 + n(n -1)/2) V exp -2Bζ 2 .
112 Chapter 6. On Graph Reconstruction via Empirical Risk Minimization

Taking the expectation w.r.t. the (X i , X j , e i,j )'s yields the desired bound.

6.7.7 Proof of Theorem 6.7

As done for Theorem 1, we prove the following generalization of Theorem 5.

Theorem 6.5. For any θ > 0, with probability at least 1-δ, the minimizer g B of the incomplete risk satisfies:

R( g B ) -R(g ) θ(R(g ) -R(g * )) + (1 + θ + 1 θ )D 1 V log(n/δ) n + log(n/δ) B .
for some universal constant D 1 .

Proof. We proceed in a similar fashion than for the proof of Theorem 6.12 and first start by recalling that we have with probability at least 1 -δ/4,

sup g∈G |W n (g)| CV log(4n/δ) n and sup g∈G | W n (g)| CV log(4n/δ) n .
We also recall that Theorem 4 gives that with probability at least 1 -δ/4:

sup g∈G | Λ B (g) -Λ n (g)| ≤ log 2 + V log 4(1+n(n-1)/2) δ 2B := C 1 V log(4n/δ) B .
We follow the proof of Theorem 6.12. For all r > 0 with probability at least 1 -δ/4:

Λ(g)-T n (g) +T n (g )-Λ(g ) max(r, Λ(g)) r 4ψ(r) + 16 log(8/δ) 3n + 2r 2 log(8/δ) n .
For any g ∈ G, let Λ B (g) = R B (g) -R B ( g B ) be the incomplete excess risk of g. We rewrite:

T n (g ) -T n (g) = 1 2 (Λ(g ) -Λ(g) + Λ n (g ) -Λ B (g ) + Λ B (g ) -Λ B (g) + Λ B (g) -Λ n (g) + W n (g ) -W n (g) + W n (g ) -W n (g)), (6.11)
which we substitute in the previous bound, take g = g B so that Λ B ( g B ) -Λ B (g ) 0 and get that with probability at least 1 -δ:

Λ( g B ) -Λ(g ) max(r, Λ(g)) r 4ψ(r) + 2r 2 log(8/δ) n + 16 log(8/δ) 3n + 4CV log(4n/δ) + C 1 V log(4n/δ) B .
Let r * 1 (δ) be solution of r = 4ψ(r) + 2r 2 log(8/δ) n + 16 log(8/δ) 3n .

Then we either have Λ( g B ) r * 1 (δ) 2 or:

R( g B ) -R(g ) = Λ( g B ) -Λ(g ) Λ( g B )r * 1 (δ) + 4CV log(4n/δ) + C 1 V log(4n/δ) B .
In the latter case we easily get that for all θ > 0,

R( g B ) -R(g ) θ(R(g ) -R(g * )) + (θ + 1) 2 4θ r * 1 (δ) + (1 + θ) 4CV log(4n/δ) + C 1 V log(4n/δ) B .
Upper-bounding r * 1 (δ) as in the proof of Theorem 6.12 gives the result.

6.7.8 Proof of Proposition 6.8

Proof. We first establish the following preliminary result.

Lemma 6.6. Suppose that the hypotheses of Proposition 6.8 are fulfilled. Then, we have:

∀g ∈ G, E R (D) (g) -R n (g) 2 | D n ≤ 2 B .
(6.12)

Proof. It suffices to notice that, for both sampling plans, we have for all (i, j) = (k, l) in Λ,

E i,j - 2B n(n -1) 2 ≤ 2B n(n -1) , (6.13)
as well as

E i,j - 2B n(n -1) k,l - 2B n(n -1) ≤ 4B n 2 (n -1) 2 ,
and to the fact that the collection { i,j : (i, j) ∈ Λ} is independent from the training data D n by assumption.

For the Bernoulli case, we apply Bernstein inequality to the sum Z(g) of the r.v.'s

i,j - 2B n(n -1)
I{g(X i , X j ) = e i,j } conditioned upon the graph D n , which are bounded by 1 and have conditional variance less than 2B/(n(n -1)). We obtain: ∀g ∈ G, ∀ζ > 0,

P {|Z(g)| > ζ | D n } ≤ 2 exp - ζ 2 2B + 2ζ/3 .
Using the union bound, one gets: ∀ζ > 0, ) is an open question. Extension of the results of this chapter to more complex dependence structures can also be investigated. For instance, the probability that two nodes are connected could be modelled as a function of their features, as in the present framework, but also on those of their neighbours in the graph,(i.e. relaxing the independence assumption for the e i,j 's) in order to account for properties shared by some real-world graphs (e.g., graphs with a power law degree distribution).

P sup g∈G |Z(g)| > Bζ | D n ≤ 2 exp - Bζ 2 2 + 2ζ/3 . Solving δ = 2(1 + n(n -1)/2)
We then focus on r.v. whose variance is bounded and establish Bernstein's inequality. We first set a few definitions :

Let (Ω, F, P) be a probability space. Let X be a random variable on this space and G a sub-σalgebra of F.

Notation 1. X is a real r.v. in L ∞ (Ω). The conditional essential supremum sup(X|G) is the unique real r.v. f X : Ω → R satisfying:

• X is G-measurable • X ≤ X a.s.
• If Ỹ : Ω → R verifies the two previous conditions then X ≤ Ỹ a.s.

It is straightforward to check that sup(X|G) ≥ E(X|G) and sup(X|G 1 ) ≥ sup(X|G 2 ) when G 1 ⊂ G 2 . Equipped with these notations we can state our main results in the following section.

A.2.1 Inequality for Bounded Random Variables(Azuma-Hoeffding and McDiarmid)

In this section we work with bounded r.v.. The theorem that we give now is strong enough to derive the fundamental inequality of Azuma-Hoeffding (A.9) and McDiarmid 's bounded difference inequality (A.10). We make use of this inequalities later in chapter 2,5 and 6. It involves the following notations:

Notation 2. Let X be a bounded r.v.. Let (F k ) 0≤k≤n be a filtration of F such that X is F nmeasurable. We denote X 1 , . . . , X n the martingale X k = E(X|F k ) and Y k = X k -X k-1 the associated martingale difference. The r.v. ran(X|G) := sup(X|G) + sup(-X|G) is the conditional range of X w.r.t G. We also denote:

• ran k = ran(Y k |F k-1 ) = ran(X k |F k-1 ) the conditional range,

• R 2 = n 1 ran k 2 the sum of squared conditional ranges, and r2 = ess sup(R 2 ) the maximum sum of squared conditional ranges.

A.2.1.1 A Preliminary Theorem

Here we establish the following theorem and give its proof. The methodology used to establish this result is classical and treated as an example. In particular its proof will be adapted to derive concentration inequalities in chapter 2.

Theorem A.5. (McDiarmid, 1998) Let X be a bounded r.v. with E(X) = µ, and (F k ) 0≤k≤n a filtration of F such that F 0 = {∅, Ω} and such that X is F n -measurable. Then for any t ≥ 0, P(X -µ ≥ t) ≤ e -2t 2 /r 2 , and more generally ∀r 2 ≥ 0, P((X -µ ≥ t) ∩ (R 2 ≤ r 2 )) ≤ e -2t 2 /r 2 .

APPENDIX B

Résumé des contributions en français

B.1 Motivation

Dans ce manuscrit, nous présentons et étudions des stratégies d'échantillonnage appliquées à problèmes liés à l'apprentissage statistique. L'objectif est de traiter les problèmes qui surviennent généralement dans un contexte de données volumineuses lorsque le nombre d'observations et leur dimensionnalité contraignent le processus d'apprentissage. Nous proposons donc d'aborder ce problème en utilisant deux stratégies d'échantillonnage:

• Accélérer le processus d'apprentissage en échantillonnant les observations les plus utiles.

• Simplifier le problème en écartant certaines observations pour réduire la complexité et la taille du problème.

Pour introduire le problème que nous traitons, nous introduisons très rapidement la minimisation du risque empirique (ERM) dans le contexte de la classification binaire. Le problème de classification binaire est considéré comme un exemple récurrent tout au long de ce manuscrit.

Parce qu'il peut être facilement formulé, il est indéniablement le problème d'apprentissage statistique le plus documenté dans la littérature et beaucoup de ses résultats s'étendent à des cadres plus généraux (e.g., Classification multi-classe, régression, ranking). Soit (Ω, A, P) un espace de probabilité et (X, Y ) une paire de variables aléatoires définie sur (Ω, A, P), prenant ses valeurs dans un espace mesurable X × {-1, +1}, avec distribution jointe P (dx, dy): la variable aléatoire X représente une observation, utile pour prédire le label Y . La distribution P peut aussi être décrite par la paire (F, η) où F (dx) désigne la distribution marginale de la variable d'entrée X et η(x) = P{Y = +1 | X = x}, x ∈ X , est la distribution postérieure. L'objectif est de construire, à partir de l'ensemble de données d'apprentissage à disposition, un mapping mesurable g : X → {-1, +1}, appelé classifier, avec riqsue minimum: (B.6) for some constants C and D β depending on the parameters l, α, γ 1 , a 1 . échantillon aléatoire S tiré de la distribution conditionnelle R n est appelée probabilité d'inclusion du premier ordre et est notée π i = P Rn {i ∈ S}. Nous définissons π n = (π 1 , . . . , π n ). Étant donné un échantillon observé S, il est entièrement déterminé par les variables aléatoires n = ( 1 , . . . , n ), où i = I{i ∈ S} pour 1 ≤ i ≤ n. La plupart des résultats disponibles dans la littérature (voir Boucheron et al. (2005c) par exemple) traitent du cas où l'ensemble de données D n est à disposition. Cependant, ce n'est pas le cas ici car nous n'observons qu'un sous-ensemble d'observations issue du schéma de sondage. Par conséquent, ces résultats ne sont pas directement applicables à notre problème, essentiellment à cause de la structure de dépendance induite par le schéma d'échantillonnage. Néanmoins, nous montrons que la théorie de l'ERM peut être étendue au cas où l'apprentissage statistique est basé sur des observations obtenues par sondages. Nous prouvons que, en minimisant une version pondérée du risque empirique, que nous nommons le risque d'Horvitz-Thompson , l'erreur stochastique peut être bornée par O P ((κ n (log n)/n) 1/2 ) (où κ n est défini plus tard) lorsque les données sont échantillonnés au moyen d'un schéma rejectif. Nous étendons ensuite ces résultats à d'autres schémas d'échantillonnage par un argument de couplage.

Risque de Horvitz-Thompson .

B.3 Stratégie d'échantillonnage pour l'algorithme du gradient stochastique

Nous présentons dans cette section un résumé des résultats établis dans les chapitres 3, 4 et 5, dans lequel nous présentons le problème de la stratégie d'échantillonnage non uniforme pour la descente de gradient stochastique (SGD en abrégé). Le problème de minimisation du risque empirique précédemment introduit est de la plus haute importance et la mise en oeuvre d'algorithmes efficaces pour résoudre ce problème est une question à laquelle nous avons tenté de répondre. Nous considérons ici un cadre plus général que celui de la classification binaire, et considérons les problèmes d'optimisation de la forme: ((X i , Y i ), θ), (B.13) basé sur n ≥ 1 exemples d'entraînement indépendants disponibles (X 1 , Y 1 ), . . ., (X n , Y n ), copies de la paire aléatoire Z = (X, Y ). Le problème de minimisation (B.12) peut être résolu de façon incrémentale, au moyen de variantes de la méthode approximation stochastique initialement introduite dans la contribution séminale de Robbins & Monro (1951)

I k = (X (k) i 1 , . . . , X (k) i d k ) pour 1 ≤ k ≤ K.
La sous-section B.3.1 et B.3.2 préconise l'utilisation de SGD pour gérer le nombre de termes polynomial dans (1.8). Notons que lorsque le risque empirique prend la forme d'une statistique U généralisée, le nombre de termes impliqués dans la somme est de l'ordre O(n d 1 +...d K ) rendant le problème extrêmement difficile à résoudre. Néanmoins, nous montrons comment implémenter le SGD dans ce cas. Nous nous plaçons dans le cadre paramétré. Notant encore Θ l'espace des paramètres, avec H : K k=1 X d k k × Θ → R une fonction de perte convexe , nous notons la version empirique du risque par θ ∈ Θ → L n (θ) = U n (H(.; θ)). Comme nous l'avons mentionné précédemment, la mise en oeuvre de SGD est assez simple pour la minimisation des statistiques standard, car elle est généralement réalisée en échantillonnant uniformément au hasard (avec ou sans remplacement) un sous-ensemble d'observations avant de calculer un estimateur du gradient. Lorsque le risque empirique prend la forme d'une U statistique, l'algorithme SGD pourrait être implémenté de cette façon. Cela conduirait à un estimateur du gradient égal à:

gn (θ) = 1 K k=1 n k d k I 1 . . . I K ∇H(X (1)
I 1 ; X

(2) I 2 ; . . . ; X Ceci est mis en évidence lorsque l'on compare conditionnellement aux observations les performances des méthodes SGD décrites ci-dessus en étudiant à la fois le comportement asymptotique et non asymptotique de l'algorithme SGD pour les deux implémentations. Comme nous l'avons déjà fait dans les chapitres précédents, nous proposons des bornes de généralisation similaire à (voir [START_REF] Bottou | The Tradeoffs of Large Scale Learning[END_REF]), nous décomposons l'erreur stochastique comme suit:

E[L(θ t ) -L(θ * )] ≤ 2E sup θ∈Θ | L n (θ) -L(θ)| E 1 + E L n (θ t ) -L n (θ * n ) E 2 .
(B.30)

B.4 Vitesse rapide pour la reconstruction de graphe

Cette section est un résumé du chapitre 6 dans lequel nous présentons un bref aperçu du problème de reconstruction de graphes. Nous introduisons dans un premier temps le contexte avant de décrire le problème d'intérêt. Soit G = (V, E) un graphe aléatoire non orienté avec un ensemble V = {1, . . . , n} de n ≥ 2 sommets et un ensemble E = {e i,j : 1 ≤ i = j ≤ n} ∈ {0, 1} n(n-1) décrivant ses arêtes: pour tout i = j, on a e i,j = e j,i = +1 si les sommets i et j sont reliés par une arête et e i,j = e j,i = 0 sinon. Nous supposons également que pour tout i ∈ V , une variable aléatoire continue X i est associé au sommet i. Les X i sont i.i.d. et pour tout i = j, la paire aléatoire (X i , X j ) représente un ensemble d'informations utiles pour prédire l'occurrence d'une arête reliant les sommets i et j. Conditionnellement à (X 1 , . . . , X n ), les variables aléatoirese i,j et e k,l ne sont indépendantes que si {i, j} ∩ {k, l} = ∅. En particulier, la distribution conditionnelle de e i,j , i = j, est supposée dépendre uniquement de (X i , X j ) et est décrite par:

η (X i , X j ) = P {e i,j = +1 | (X i , X j )} . obtenant ainsi dans le meilleur des cas une performance comparable à celle de la règle de Bayes g * (x 1 , x 2 ) = I{η(x 1 , x 2 ) > 1/2}, dont le risque est donné par R * = E[min{η(X 1 , X 2 ), 1η(X 1 , X 2 )}] = inf g R(g).

Le risque de reconstruction (B.34) n'étant pas disponible, il est remplacé par sa version empirique basée sur l'ensemble labélisé D n = {(X i , X j , e i,j ) : 1 ≤ i < j ≤ n} associé à G: R n (g) = 2 n(n -1) 1≤i<j≤n I {g(X i , X j ) = e i,j } . Biau & Bleakley (2006) établissent des bornes de l'ordre O P (1/ √ n) pour le risque de reconstruction de g n sous des hypothèses de complexité appropriées (à savoir que G a sa VCdimension finie). Nous prouvons que les taux d'ordre O P (log n/n) sont toujours atteints par les minimiseurs du risque de reconstruction empirique (B.35) sans autres hypothèses. Pour établir ce résultat, nous nous appuyons sur certains arguments utilisés dans l'analyse de taux rapide pour la minimisation empirique de U -statistiques (Clémençon et al., 2008a), bien que ces résultats soient établis sous des hyôthèses resctrictives. Alors que la quantité (1.23) n'est pas une U -statistique, nous réécrivons la différence entre l'excès de risque de reconstruction de toute règle candidate g ∈ G et sa contrepartie empirique comme somme de son espérance

  3.1 A binary tree for n = 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.2 Evolution of L n (θ t ) with S = 10 and ρ = 0.7 . . . . . . . . . . . . . . . . . 39 3.3 Evolution of L n (θ t ) with different values of ρ (ν i ∼ L i , S = 10) (left) and different sampling strategies (ρ = 0.7, S = 10) (right) . . . . . . . . . . . . . 40 4.1 Evolution of the estimator of β 5 with the number of iterations in the HTGD (solid), mini-batch SGD (dotted) and GD (dashed) algorithms with N = 10 (left) and N = 100 (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2 50 trajectories of the estimator of β 5 with the number of iterations in the HTGD (solid), mini-batch SGD (dotted) over 50 populations (left) and of θ 6 over 1 populations (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3 Evolution of the estimator of the location parameter θ = 0 of the balanced

  1311. In contrast, the reconstruction rule learned from B = 100 pairs of nodes (out of 1225 possible pairs) successfully inverts R and accurately recovers the original graph (Figure 6.1(c)). Its reconstruction error is 0.008 on the training graph and 0.009 on a held-out graph generated with the same parameters. . . . . . . . . . . . . . 101 6.2 Summary of results on the synthetic graph. . . . . . . . . . . . . . . . . . .

  FIGURE 3.2: Evolution of L n (θ t ) with S = 10 and ρ = 0.7

  FIGURE 3.3: Evolution of L n (θ t ) with different values of ρ (ν i ∼ L i , S = 10) (left) and different sampling strategies (ρ = 0.7, S = 10) (right)

48Chapter 4 .

 4 Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation

50Chapter 4 .

 4 Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation parameter N which controls the number of terms involved in the empirical gradient estimation at each iteration. HORVITZ-THOMPSON GRADIENT DESCENT ALGORITHM (HTGD) (INPUT.) Datasets {Z 1 , . . . , Z n } and {W 1 , . . . , W n }. Maximum (expected) sample size N ≤ n. Collection of sampling plans R n (θ) with positive first order inclusion probabilities π i (θ) for 1 ≤ i ≤ n, indexed by θ ∈ Θ with (expected) sample sizes less than N . Learning rate γ t > 0. Number of iterations T ≥ 1. 1. (INITIALIZATION.) Choose θ n (0) in Θ.

  Fix θ ∈ Θ and N ≤ n. Given Z 1 , . . . , Z n , consider a Poisson scheme P n on the population U n = {1, . . . , n} with positive parameter Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation51

  t (θ n (t) -θ * ) = N (0, Σ * ).

58Chapter 4 .

 4 Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation Corollary 4.10. Suppose that the assumptions of Theorem 4.9 are fulfilled. Let U be a ddimensional Gaussian centered r.v. with the identity as covariance matrix. The assertions below hold true. = +∞, then we have the convergence in distribution:

Chapter 4 .

 4 Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation59 drew a single sample of size n = 5000 on which the two algorithms were performed for 2000 iterations. Two sub-sample sizes were considered : N = 10 and N = 100. As can be seen on Fig.4.2, while virtually equivalent in terms of computation time, thus taking a larger sample improves the efficiency of the HTGD. It also appears to reach a better level of precision in less steps than both competitors, a phenomenon that is consistent on all 11 coordinates of θ.

FIGURE 4

 4 FIGURE 4.1: Evolution of the estimator of β 5 with the number of iterations in the HTGD (solid), mini-batch SGD (dotted) and GD (dashed) algorithms with N = 10 (left) and N = 100 (right).

60Chapter 4 .

 4 Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation

FIGURE 4

 4 FIGURE 4.2: 50 trajectories of the estimator of β 5 with the number of iterations in the HTGD (solid), mini-batch SGD (dotted) over 50 populations (left) and of θ 6 over 1 populations (right).

FIGURE 4 . 3 :Min

 43 FIGURE 4.3: Evolution of the estimator of the location parameter θ = 0 of the balanced Gaussian mixture with the number of iterations in the HTGD (solid red), mini-batch SGD (dashed green) and GD (dotted blue) algorithms

t≥0E

  ξ n (t + 1) b | F t I {θ n (t) ∈ V} < +∞ almost-surely for any b > 2, which we shall verify, Chapter 4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation65

  4. Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation

70Chapter 4 .

 4 Horvitz Thompson Stochastic Gradient Descent : Application to M-estimation

  Gradient Descent based on incomplete U -Statistics where n = (n 1 , . . . , n K ), the symbol I 1 • • • I K refers to summation over all elements of Λ, the set of the K k=1 n k d k index vectors (I 1 , . . . , I K ), I k being a set of d k indexes 1

78

  FIGURE 5.1: Average over 50 runs of the risk estimate with the number of iterations (solid lines) +/-their standard deviation (dashed lines)

  FIGURE 5.2: Average over 50 runs of the error test with the number of iterations (solid lines) +/-their standard deviation (dashed lines)

  2αγ k ) 2 .

  FIGURE 6.1: Illustrative experiment with n = 50, q = 2, τ = 0.27 and p = 0. Figure6.1(a) shows the training graph, where the position of each node is given by its 2D feature vector. Figure6.1(b) depicts the same graph after applying a random transformation R to the features. On this graph, the Euclidean distance with optimal threshold achieves a reconstruction error of 0.1311. In contrast, the reconstruction rule learned from B = 100 pairs of nodes (out of 1225 possible pairs) successfully inverts R and accurately recovers the original graph (Figure6.1(c)). Its reconstruction error is 0.008 on the training graph and 0.009 on a held-out graph generated with the same parameters.

  FIGURE 6.1: Illustrative experiment with n = 50, q = 2, τ = 0.27 and p = 0. Figure6.1(a) shows the training graph, where the position of each node is given by its 2D feature vector. Figure6.1(b) depicts the same graph after applying a random transformation R to the features. On this graph, the Euclidean distance with optimal threshold achieves a reconstruction error of 0.1311. In contrast, the reconstruction rule learned from B = 100 pairs of nodes (out of 1225 possible pairs) successfully inverts R and accurately recovers the original graph (Figure6.1(c)). Its reconstruction error is 0.008 on the training graph and 0.009 on a held-out graph generated with the same parameters.

  connu que le Bayes classifier g * (x) = 2I{η(x) ≥ 1/2} -1 est une solution du problème de minimisation du risque inf g L(g), où l'infinimum est pris sur la collection de tous les classifieurs définis sur l'espace X . Le risque minimum est noté L * = L(g * ). Puisque la distribution P des données est inconnue, le vrai risque est remplacé par son estimation empirique:L n (g) = 1 n n i=1 I{g(X i ) = Y i },(B.2) basé sur l'échantillon d'exemples (X 1 , Y 1 ), . . . , (X n , Y n ) de copies indépendantes de (X, Y ). La véritable minimisation du risque est alors remplacée par la minimisation empirique B.2 Apprendre de données de sondages Cette sous-section est un résumé du chapitre 2. Nous nous plaçons dans le contexte de la classification binaire, lorsque les observations utilisées pour former un classificateur sont issues d'un schéma d'échantillonnage/sondage et présentent une structure de dépendance complexe. Nous considérons, (X 1 , Y 1 ), . . . , (X n , Y n ) un échantillon de copies indépendantes de (X, Y ) observées sur une population finie I n := {1, . . . , n}. Nous appelons un survey sample de taille (éventuellement aléatoire) N ≤ n de la population I n , tout sous-ensemble s := {i 1 , . . . , i n(s) } ∈ P(I n ) avec cardinalité N =: N (s) inférieur à n. Un schéma d'échantillonnage est défini par une distribution de probabilité R n sur l'ensemble de tous les échantillons possibles s ∈ P(I n ) conditionnellement aux observations D n = {(X i , Y i ) : i ∈ I n }. La probabilité que l'unité i appartienne à Theorem B.1. Let θ t be the sequence generated by SGD using the incomplete statistic gradient estimator (5.6) with B = K k=1 n k d k terms for some n 1 , . . . , n K . Assume that {L(.; θ) : θ ∈ Θ} is a VC major class class of finite VC dimension V s.t.M Θ = sup θ∈Θ, (x (1) , ..., x (K) )∈ K k=1 X d k k |H(x (1) , . . . , x (K) ; θ)| < +∞, (B.5)and N Θ = sup θ∈Θ σ 2 θ < +∞. If the step size satisfies the condition of Proposition 5.4, we have:∀n ∈ N * K , E[|L(θ t ) -L(θ * )|] CN Θ Bt β + 2M Θ 2 2V log(1 + κ) κ .For any δ ∈ (0, 1), we also have with probability at least 1 -δ: ∀n ∈ N * K , |L(θ t )-L(θ * )| CN Θ

  un espace euclidien, typiquement R d avec d ≥ 1, et l(Z 1 , .), . . . , l(Z n , .) forme une collection de fonctions convexes continuellement dérivables valables sur Θ. En effet, un tel problème d'optimisation se pose typiquement dans une grande variété de problèmes d'apprentissage statistique, en particulier de tâches supervisées, où l'objectif poursuivi est d'apprendre un modèle prédictif, entièrement déterminé par un paramètre θ. La performance de la fonction prédictive définie par θ est typiquement mesurée par l'espérance L(θ) = E[ ((X, Y ), θ)], appelée le risque, où est un fonction de perte supposé convexe θ. Il est généralement évalué via son homologue empiriqueL n (θ) = 1 n n i=1

  , nous étudions la convergence d'un algorithme SGD lorsque l'estimateur du gradient est calculé en échantillonnant des observations dans un ensemble de données sous un plan d'échantillonnage S n (éventuellement en fonction de déoendant de t et la valeur actuelle du paramètre). On note θ n (T ) la valeur du paramètre à l'instant T . Conditionné sur les données D n = {Z 1 , . . . , Z n }, nous étudions les propriétés asymptotiques de l'estimateur M produit par l'algorithme HTGD. Les résultats limites indiqués ci-dessous reposent essentiellement sur le fait que l'estimateur HT (B.23) du gradient du risque empirique est non biaisé et sous certaines hypothèses listées dans la section précédente.Theorem B.10. (CONDITIONAL CENTRAL LIMIT THEOREM) Supposons que γ t = γ 0 t -α avec α ∈ (1/2, 1] et γ 0 > 0. Quand α = 1, γ 0 > 1/(2l) et η := 1/(2γ 0 ); η := 0 sinon. Conditionnellement à D n nous avons la convergence en distribution quand t → +∞ 1/γ t (θ n (t) -θ * n ) ⇒ N (0, Σ πn ),où la matrice de covariance asymptotique Σ πn est l'unique solution deH n Σ + ΣH n + 2ηΣ = Γ * n , (B.24) avec Γ * n = Γ n (θ * n ) and, for all θ ∈ Θ, j (θ) π i (θ)π j (θ) -1 ∇l(Z i , θ) ∇l(Z j , θ) T . (B.25)La réduction de la variance asymptotique de θ n (T ) (de L n ( θ n (T )), respectivement) est étudiée plus tard dans le cas de Poisson (c'est-à-dire quand les i sont indépendants). Par une application directe de la méthode Delta du second ordre, nous caractérisons alors le comportement de L n ( θ n (t)) -L n (θ * n ) où θ * n est un minimiseur empirique des risques. où n = (n 1 , . . . , n K ), le symbole I 1 • • • I K fait référence à la sommation sur tous les éléments de Λ, l'ensemble des vecteurs d'index K k=1 n k d k (I 1 , . . . , I K ), I k étant un ensemble de d k index 1 ≤ i 1 < . . . < i d k ≤ n k et X (k)

  I k de d k indexes 1 ≤ i 1 < . . . < i d k ≤ n k et n = (n 1 , . . . , n K ).Dans le cas de U -Statistiques" nous montrons que cette stratégie (que nous appellerons plus tard "complète U -statistique") n'est pas efficace. Nous proposons plutôt de procéder différemment en tirant indépendamment avec remise parmi l'ensemble des vecteurs d'index Λ, obtenant un estimateur du gradient sous la forme d'une incomplète U -statistique (voirLee (1990a)): où D B est construit en échantillonnant B fois avec remplacement dans l'ensemble Λ. Le paramètre B est le nombre de termes moyennés. Pour le même coût de calcul (ie, prendre B = K k=1 n k d k ) et implémenter SGD avec (B.29) plutôt que (B.28) donne de "meilleurs" solutions, essentiellement parce que (B.29) est un estimteur avec une plus petite variance (sauf dans le cas où K = 1 = d 1 ). Intuitivement, l'échantillonnage d'une incomplète U -statistique est préférable car le nombre d'observations présent dans cet estimateur est supérieur au nombre d'observations présent dans l'estimateur complet.

  (B.33) Le problème d'apprentissage introduit par[START_REF] Bibliography | Statistical Inference on Graphs[END_REF], appelé graph reconstruction, consiste à construire une règle symétrique de reconstruction g : X 2 → {0, 1}, à partir d'un graphe d'apprentissage G, avec un risque de reconstruction le plus petit possible: R(g) = P {g(X 1 , X 2 ) = e 1,2 } , (B.34)

  (B.35) Notez que (B.35) présente a prioir une structure de dépendance complexe car ce n'est pas une somme de variables aléatoires indépendantes. Soit g n un minimiseur du risque empirique: min g∈G R n (g), où G est une classe de reconstruction. Comme nous l'avons déjà fait précédemment, les performances de g n sont alors mesurées par R( g n )-inf g∈G R(g), qui peut être borné si nous pouvons obtenir des inégalités de concentration sur l'écart maximalsup g∈G | R n (g) -R(g)|.(B.36)

  More generally, when N = o(n) as n → +∞, as soon as the weights cannot vanish faster than N/n, the rate achieved by minimizers of the HT risk is of the order O( (log n)/N ).

	by random vectors n with negatively associated components, see Brändén & Jonasson (2012)
	or Kramer et al. (2011). Hence, Proposition 2.4's proof shows that the bounds stated above
	immediately extend to these cases. See section 2.8 for more details and references. Before
	showing how the rate bounds established can be extended to even more general sampling
	schemes, a few remarks are in order.
	Remark 2.5. (COMPLEXITY ASSUMPTIONS) We point out that the results stated can be estab-
	lished, essentially by means of the same argument as that developed in the section 2.6, under
	complexity assumptions of different nature, involving metric entropy conditions for instance
	(see e.g. van der Vaart & Wellner (1996)).
	Remark 2.6. (MODEL SELECTION) A slight modification of the argument involved in Proposi-
	tion 2.4 straightforwardly leads to bounds on the expected excess risk E[L(ḡ n )]-inf g∈G L(g).
	Following the Structural Risk Minimization principle (see Vapnik (2001)), such VC bounds
	can be next used as complexity regularization terms to penalize additively the HT risk (2.4)
	and, for a sequence of model classes G k with k ≥ 1 of finite VC dimension, select the classifier
	among the minimizers {arg min g∈G k	L	n (g), k ≥ 1}, which has approximately minimal risk.
	Remark 2.7. (BIASED HT RISK)
			Many sampling schemes (e.g. Rao-
	Sampford sampling, Pareto sampling, Srinivasan sampling) of fixed size are actually described

TABLE 2 .

 2 1: Average over 50 runs of the prediction error on D test and its standard deviation.

TABLE 3

 3

.1: Comparison of iteration complexities of AS-SGD and SGD with uniform sampling: c = complexity of pointwise gradient computation, S = sample size.

  Using the non expansiveness of the projection, the strong convexity and the definition of B ν , we get a t+1 (1-2αγ t )a t +γ 2

	Chapter 3. Adaptive Sampling Scheme for Incremental Optimization using Stochastic
	Gradient Descent Algorithm	41
	3.8 Technical Proofs
	3.8.1 Proof of Lemma 3.1
	t	B 2 ν
	Algorithm 3 Update_tree
		Input: A, I, g
		for i ∈ I do
		e =leaf(A, i)
		δ = g i -w(e)
		w(e) ← g i
		repeat
		e ←father(e)
		w(e) ← w(e) + δ
	e)[2]	until isroot(e)
	end if	end for
	until isleaf(e)	Return A
	Return label(e)

  The rest of the chapter is structured as follows. Basics in M -estimation and SGD techniques together with key notions in survey sampling theory are briefly recalled in section 4.2. Section 4.3 first describes the Horvitz-Thompson variant of the SGD in the context of a general M -estimation problem. In section 4.4, limit results are established in a general framework, revealing the possible significant gain in terms of asymptotic variance resulting from sampling with unequal probabilities in presence of extra information. They are next discussed in more depth in the specific case of Poisson surveys. Illustrative numerical experiments, consisting in fitting a logistic regression model (respectively, a semi-parametric shift model ) with extra information, are displayed in section 4.6.

Table 4

 4

		.3.
		n = 500 n = 1000
	HTGD	1.52		1.45
	SGD	2.21		2.09
	TABLE 4.1: Mean standard deviations of the final estimates of θ(= -9) across the 50 simu-
	lations
	Min. Median Max. Mean S.D.
		HTGD
	θ 5 -9.5	-8.7	-7.8	-8.6 1.45
	θ 6 13.3	14.6	15.9	14.5 1.52
		SGD
	θ 5 -9.9	-8.2	-7.4	-8.2 2.09
	θ 6 12.7	13.9	16.6	15.2 2.21

TABLE 4.2: Statistics on the global behavior of the final estimates of β 5 and β 6 across the 50 simulations

TABLE 4

 4

	.3: Statistics on the global behavior of the final estimates of the location parameter
	across the 50 simulations

TABLE 6

 6

	.1: Results (averaged over 10 runs) on synthetic graph with n = 1, 000, 000, q =
	100, p = 0.05.
		B = 0.01n B = 0.1n B = n B = 5n B = 10n
	Reconstruction error	0.2272	0.1543	0.1276 0.1185	0.1159
	Relative improvement	-	32%	17%	7%	2%
	Training time (seconds)	21	398	5,705	20,815	42,574

Table 6 .

 6 1 shows the test error (averaged over 10 runs) as well as the training time for several values of B. Consistently with our theoretical findings, B implements a trade-off between statistical accuracy and computational cost. For this dataset, sampling B = 5, 000, 000 pairs (out of 10 12 possible pairs!) is sufficient to find an accurate reconstruction rule. A larger B would result in increased training time for negligible gains in reconstruction error.

TABLE 6 .

 6 2: Reconstruction error on synthetic graph with parameters n = 1, 000, 000, q = 100, p = 0.05.

		B = 0.01n B = 0.1n B = n B = 5n B = 10n
	Sampling nodes	0.2552	0.1847	0.1411 0.1279	0.1233
	Sampling pairs of nodes	0.2272	0.1543	0.1276 0.1185	0.1159

TABLE 6 .

 6 3: Reconstruction error (averaged over 10 runs) on the Cit-HepTh graph.

		B = 10K B = 100K B = 1M B = 5M
	Balanced reconstruction error	0.3080	0.2629	0.2484	0.2464
	Relative improvement	-	15%	6%	<1%
	Training time (seconds)	418	1,675	4,481	18,895

Table 6 .

 6 3 shows the balanced test error (averaged over 10 runs) as well as the training time for several values of B. Despite the higher dimensional and sparse nature of the features, we are able to significantly improve over the Euclidean baseline using few training pairs. Furthermore, sampling B = 1M pairs is sufficient to get very close to the best performance: going from 1M to 5M pairs brings less than 1% relative improvement in test error at the expense of a 4 times increase in training time.

  Chapter 7. Conclusion, Limitations & Perspectives they can be extended in presence of some regularization term. When the regularization term is not differentiable, one typically use the stochastic proximal gradient descent (see[START_REF] Yves F Atchade | On stochastic proximal gradient algorithms[END_REF] for instance) which essentially boils down to perform one step of SGD on the data-fitting term before performing a proximal step on the regularization term. Incorporating non uniform sampling strategies is therefore a problem that would be very interesting to study for stochastic proximal gradient descent. Finally and as mentioned in chapter 3, significant advances have been recently made in the design of efficient incremental methods(see for instance[START_REF]Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning[END_REF],[START_REF]Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization[END_REF],Johnson & Zhang (2013a), Shalev-Shwartz & Zhang (2012),[START_REF] Schmidt | Minimizing finite sums with the stochastic average gradient[END_REF] orDefazio et al. (2014)) achieving better performances than the traditional SGD method, some of these methods do work when observation are sampled non uniformly but they do not shed light on how observations should be sampled or what would be the potential gain on the risk, which would be very interesting to study.For the problem we considered in chapter 6, we prove that the learning rates for ERM in the graph reconstruction problem are always of order O(log n/n), it would be interesting to see if we can establish a lower bound that matches this upper bound. Another challenging line of future research is to see what happens when the noise condition of assumption 8 is satisfied. It would straightforwardly lead to fast rate for minimizer of (6.5) but what it would change for minimizers of (6.3

d exp(-Bζ 2 /(2 + 2ζ/3)) leads to the first bound.

116

  . Celui-ci consiste à calculer les estimations successives d'un minimiseur de (B.2) en utilisant l'équation récursive θ t+1 = θ t -γ t r t (θ t ) (B.14) à partir d'une valeur initiale θ 0 ∈ Θ, où r t dénote un estimateur du gradient ∇ L n et γ t est le taux d'apprentissage où step-size. La mise en oeuvre de SGD est assez simple pour la minimisation d'un risque empirique prenant la forme d'une moyennes standard, car elle est généralement effectuée en échantillonnant uniformément au hasard (avec ou sans remplacement) un sous-échantillon d'observations avant de calculer un estimateur du gradient. Contrairement à l'approche batch, où toutes les données sont utilisées pour estimer le gradient à chaque itération (ie r t (θ) = ∇ L n (θ) pour tout t ≥ 0 et θ ∈ Θ), seuls les sous-ensembles de l'échantillon de données sont impliqués dans les étapes d'estimation de gradient de dans ces algorithmes incrémentaux, dans le but de réduire le coût de calcul lorsque n est grand. Dans l'implémentation la plus couramment utilisée de l'algorithme de la descente de gradient stochastique (SGD), l'estimateur de gradient est calculé à partir d'un sous-ensemble de taille réduite S ≤ n uniformément dessiné sans remplacement parmi tous les sous-ensembles possibles de taille S à chaque étape t ≥ 0. En pratique, la limitation principale de cette technique d'optimisation incrémentale est due au bruit stochastique induit par le choix aléatoire des données impliquées dans le calcul de l'estimateur de gradient à chaque itération. En particulier, la plupart des justifications théoriques du SGD sont établies dans un cadre très général (voir Robbins & Monro (1951) ou Bach & Moulines (2011a) par exemple) qui englobe le cas de la minimisation du risque empirique. Nous proposons d'introduire une stratégie d'échantillonnage non uniforme ainsi qu'une nouvelle analyse soulignant l'avantage d'utiliser une stratégie d'échantillonnage non uniforme pour le problème de la minimisation du risque empirique. Nous introduisons d'abord dans le chapitre 3 une nouvelle implémentation de l'algorithme SGD, où le sousensemble de données utilisé à un pas donné n'est pas choisi au hasard parmi tous les sousensembles possibles mais simulé à l'aide d'un schéma d'échantillonnage adaptatif spécifique, construit à partir des itérations passées. Nous proposons ensuite un cadre général pour étendre ces résultats en utilisant la théorie des sondages dans le chapitre 4 dans lequel nous prenons également en compte la distribution de l'observation dans notre analyse finale. Nous concluons cette section en considérant le cas spécifique où le risque empirique prend la forme d'une U -statistique et proposons une implémentation efficace de l'algorithme SGD dans ce cas. Ici dans le reste de cette sections, les opérateurs du gradient et de la hessienne par rapport à une variable θ sont notés ∇ et ∇ 2 respectivement. Par convention, ∇ 0 correspond à l'opérateur d'identité et les valeurs de gradient sont représentées comme des vecteurs de colonne. Pour tout vecteur V ∈ R d , on note ||V || sa norme euclidienne et pour toute matrice A on note A T sa transposée. B.3.1 Un stratégie d'échantillonnage non uniforme pour le SGD Afin d'accélérer le processus d'apprentissage, nous introduisons une variante spécifique de l'algorithme SGD avec un schéma d'échantillonnage adaptatif, en ce sens qu'il peut varier avec t, en fonction des itérations passées. Nous considérons un échantillonnage non uniforme avec remplacement. Nous commençons par identifier une bonne distribution d'échantillonnage en choisissant celle qui minimise la variance de l'estimateur. Lorsque l'on tire un échantillon S de taille S avec probébilités d'inclusion du premier ordre p i indépendamment avec remplacement, la quantité 1 S Pour obtenir la meilleure estimation du gradient (i.e en minimisant la variance) au paramètre θ conditionnellement aux observations, il est donc naturel d'échantillonner l'observation Z i avec probabilité:p * i (θ) = ∇l(Z i , θ) / n j=1 ∇l(Z j , θ) car ce choix minimise la valeur de la variance. Malheureusement, la mise en oeuvre pratique du schéma d'échantillonnage ci-dessus n'est pas pertinente car elle nécessite d'évaluer tous les gradients pour calculer les normes ∇l(Z 1 , θ t ) , . . . , ∇l(Z n , θ t ) à chaque itération, ce que nous essayons précisément d'éviter en utilisant l'algorithme du gradient stochastique. Nous proposons donc un schéma d'échantillonnage approximant p * t := (p * i (θ t )) n i=1 sans nécessiter d'évaluation de gradient supplémentaire. Nous utilisons quelques anciennes valeurs du gradient dans notre approximation. Plus précisément, l'idée principale est de remplacer chaque norme de gradient inconnue ∇l(Z i , θ t ) par une norme précédemment calculée g t,i = ∇l(Z i , θ k ) à un ancien instant k = k(i, t) correspondant au dernier instant k ≤ t où Z i a été tiré. Plus formellement, nous définissons la suite aléatoire g t comme étant égale à:g t+1,i = ∇l(Z i , θ t+1 ) if i ∈ {iNotez que tout les résultats obtenus dans le chapitre 3 restent vrais conditionnellement aux observations et ne tient donc pas compte de la nature statistique de notre problème (nous résolvons ERM car nous ne connaissons pas le vrai risque). Nous abordons ce problème dans la section suivante où nous discutons d'un problème similaire (Stratégie d'échantillonnage non uniforme pour SGD) dans le contexte de la M-estimation. Plus précisément, nous utilisons le cadre des sondages introduit précédemment pour étendre nos résultats. B.3.2 (HTSGD) et applications à la M-estimation Les sections précédentes suggèrent fortement d'utiliser les techniques de sondage pour améliorer l'algorithme d'apprentissage. Nous montrons maintenant comment incorporer des schémas de sondage efficaces dans de telles techniques itératives pour la M-estimation. Plus précisément, nous proposons un estimateur spécifique du gradient, que l'on appelle l'estimateur estimation du gradient de Horvitz-Thompson (HTGD en bref). Pour l'estimateur ainsi produit, les résultats de normalité asymptotique décrivant sa performance statistique sont établis. Le framework que nous considérons est le même que celui de la section 1.3.1. Nous définissons l' estimateur de Horvitz-Thompson du gradient n (θ) basé sur un échantillon de sondage S tiré par un design S n avec probabilités d'inclusion du premier ordre {π i } 1≤i≤n et le vecteur d'inclusion n = ( 1 , . . . , n ) as

			i∈S	∇l(Z i , θ) p i		(B.15)
	est un estimateur sans biais de ∇ L n (θ) avec une variance égale à:
	1 S	n i=1	∇l(Z i θ) 2 p i	-	||∇ L n (θ)|| 2 S	.	(B.16)
		g t,i			(1) t+1 , . . . , i otherwise.	(S) t+1 }	(B.17)

Notice that any vector p n ∈]0, 1[ n such that pi/(1 -pi) = cp i /(1 -p i ) for all i ∈ {1, . . . , n} for some constant c > 0 can be used to write a representation of Rn of the same type as (2.6)

https://https://www.data.gouv.fr/fr/datasets/

http://www.pewinternet.org/datasets/june-10-july-12-2015-gaming-jobs-and-broadband/

http://www.pewinternet.org/datasets/nov-26-2014-jan-3-2015-privacy-panel-3/

http://www.pewinternet.org/datasets/jan-27-feb-16-2015-privacy-panel-4/

Consider for instance the case n =

2, X = R, l(Z1, θ) = θ 2 , l(Z2, θ) = (θ -1) 2 and θ0 = 0.

http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/

Of course, the step size must be in an appropriate range. If it is unnecessarily small, both methods have comparable performance, while if it is too large, they both diverge.

A classical Lehmann-Scheffé argument shows that (6.3) is the estimator of (6.2) with smallest variance among all unbiased estimators.

Note that, throughout the chapter, the constant C is not necessarily the same at each appearance.

http://snap.stanford.edu/data/cit-HepTh.html

http://www.nltk.org

Consider γ 1 < 1 2L , then γj < 1 2 γ j , and

4γ 2 1 exp(8αlγ 2 1 )2 4αγ 1 (T + 1) 4αγ 1 T t=1 1 j 2-4αγ 1 4γ 2 1 exp(8αlγ 2 1 )2 4αγ 1 (4αγ 1 -1)(T + 1) bringing all the pieces back together and substituting the corresponding bounds give the result under the conditional probability . Taking the expectation over the distribution of the observations gives the result in terms of

and we get the final result.

PART III

Fast Learning Rates for Graph Reconstruction

Chapter 6. On Graph Reconstruction via Empirical Risk Minimization an incomplete approximation obtained by sampling pairs of vertices (and not vertices) with replacement. Formally, we define the incomplete graph reconstruction risk based on B ≥ 1 pairs of vertices as

I {g(X i , X j ) = e i,j } , (6.7)

where P B is a set of cardinality B built by sampling with replacement in the set Θ n = {(i, j) : 1 ≤ i < j ≤ n} of all pairs of vertices of the training graph G. For any b ∈ {1, . . . , B} and all (i, j) ∈ Θ n , denote by b (i, j) the variable indicating whether the pair (i, j) has been picked at the b-th draw ( b (i, j) = +1) or not ( b (i, j) = +0). The (multinomial) random vectors b = ( b (i, j)) (i,j)∈Θn are i.i.d. (notice that (i,j)∈Θn b (i, j) = +1 for 1 ≤ b ≤ B) and the incomplete risk can be then rewritten as

Observe that the statistic (6.7) is an unbiased estimate of the true risk (6.2) and that, given the X i 's, its conditional expectation is equal to (6.3). Considering (6.7) with B = o(n 2 ) as our empirical risk estimate significantly reduces the computational cost, at the price of a slightly increased variance:

for any reconstruction rule g. Note in particular that the above variance is smaller than that of the complete reconstruction risk based on a subsample of √ B vertices drawn at random (thus involving O(B) pairs as well). To characterize Var( R B (g)), we need to derive an explicit expression for V ar( R n (g)). This is done by relying on the second Hoeffding decomposition (see [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF] of R n (g). For all 1 ≤ i < j ≤ n, let us define

• K 2 (X i , X j ) = R(g) -K 1 (X i ) -K 1 (X j ) + E[I{g(X i , X j ) = e i,j }|X i , X j ],

• K 3 (X i , X j , e i,j ) = I{g(X i , X j ) = e i,j } -E[I{g(X i , X j ) = e i,j }|X i , X j ].

We have the following "orthogonal" decomposition:

Introducing the following variance terms:

• σ 2 2 = Var(K 2 (X 1 , X 2 )),

• σ 2 3 = Var(K 3 (X 1 , X 2 , e 1,2 )), one easily gets

Substituting the expression of Var( R n (g)) gives

Var R B (g

This shows that if B = O(n), Var( R B (g)) is of the same order as Var( R n (g)) while R B (g) is computationally much cheaper than R n (g) as it consists of only O(n) terms. In contrast to (6.7), the estimator obtained by sampling m nodes has a larger variance: it is equal to Var( R m (g)), which is of order 1/m = O(1/ √ B). We are thus interested in characterizing the performance of solutions g B to the computationally simpler problem min g∈G R B (g). The following theorem shows that, when the class G is of finite VC-dimension, the concentration properties of the incomplete reconstruction risk process { R B (g)} g∈G can be deduced from those of the complete version { R n (g)} g∈G . Theorem 6.6. (UNIFORM DEVIATIONS) Suppose that the class G is of finite VC-dimension V < +∞. For all δ > 0, n ≥ 1 and B ≥ 1, we have with probability at least 1 -δ:,

The finite VC-dimension hypothesis can be relaxed and a bound of the same order can be proved to hold true under weaker complexity assumptions involving Rademacher averages (see Remark 6.3). Remarkably, with only B = O(n) pairs, the rate in Theorem 6.6 is of the same order (up to a log factor) as that obtained by [START_REF] Bibliography | Statistical Inference on Graphs[END_REF] for the maximal deviation sup g∈G | R n (g) -R(g)| related to the complete reconstruction risk R n (g) with O(n 2 ) pairs. From Theorem 6.6, one can get a learning rate of order O(1/ √ n) for the minimizer of the incomplete risk involving only O(n) pairs. Unfortunately, such an analysis does not exploit the relationship between conditional variance and expectation formulated in Lemma 6.4, and is thus not sufficient to show that reconstruction rules minimizing the incomplete risk (6.7) can achieve learning rates comparable to those stated in Theorem 6.1. In contrast, the next theorem provides sharper statistical guarantees. Theorem 6.7. Let g B be any minimizer of the incomplete reconstruction risk (6.7) over a class G of finite VC-dimension V < +∞. Then, for all δ ∈ (0, 1), we have with probability at least

where C < +∞ is a universal constant.

This bound reveals that the number B ≥ 1 of pairs of vertices plays the role of a tuning parameter, ruling a trade-off between statistical accuracy (taking B(n) = O(n 2 ) fully preserves the convergence rate) and computational complexity. This will be confirmed numerically in Section 6.5.

The above results can be extended to other sampling techniques, such as Bernoulli sampling and sampling without replacement.

Performance of minimizer of (6.9) are given by the following proposition.

Proposition 6.9. Let g D B be any minimizer of (6.9) over a class G of finite VC-dimension V < +∞. When D is a Bernoulli plan or a SWOR plan of size B, then, for all δ ∈ (0, 1), we have with probability at least 1 -δ: ∀n ≥ 2,

where C < +∞ is a universal constant.

Numerical Experiments

In this section, we present some numerical experiments on large-scale graph reconstruction to illustrate the practical relevance of the idea of incomplete risk introduced in Section 6.4. Following a well-established line of work [START_REF] Vert | Supervised graph inference[END_REF][START_REF] Vert | A new pairwise kernel for biological network inference with support vector machines[END_REF][START_REF] Shaw | Learning a Distance Metric from a Network[END_REF], we formulate graph reconstruction as a distance metric learning problem [START_REF] Bellet | Metric Learning[END_REF]: we learn a distance function such that we predict an edge between two nodes if the distance between their features is smaller than some threshold. Assuming X ⊆ R q , let S q + be the cone of symmetric PSD q × q real-valued matrices. The reconstruction rules we consider are parametrized by M ∈ S q + and have the form

) is a (pseudo) distance equivalent to the Euclidean distance after a linear transformation L ∈ R q×q , with M = L T L. Note that g M (x 1 , x 2 ) can be seen as a linear separator operating on the pairwise representation vec((

hence the class of learning rules we consider has VC-dimension bounded by q 2 + 1. We define the reconstruction risk as:

where [•] + = max(0, •) is a convex surrogate for the 0-1 loss. In earlier work, ERM has only been applied to graphs with at most a few hundred or thousand nodes due to scalability issues. Thanks to our results, we are able to scale it up to much larger networks by sampling pairs of nodes and solve the resulting simpler optimization problem. In all experiments, we used the generic convex optimization method Adagrad (implemented in Python) and stopped the optimization when the training error had not been improving for the last 10 epochs.

Chapter 6. On Graph Reconstruction via Empirical Risk Minimization Turning next to the second bound, the exponential inequality tailored to the SWOR case (see [START_REF] Serfling | Probability inequalities for the sum in sampling without replacement[END_REF], Corollary 1.1) yields:

for all g ∈ G, ζ > 0. Using the union bound and then solving δ = 2(1 + n(n -1)/2) d exp(-Bζ 2 /2) completes the proof.

Note that following Section 6.7.7, one can easily derive a version of Theorem 5 for the minimizer of R D by replacing Λ B (g) with Λ D B (g) (the incomplete excess risk corresponding to the sampling plan D) in the decomposition (6.11) and making the appropriate modifications.

CHAPTER 7

Conclusion, Limitations & Perspectives

In this manuscript we study and implement sampling strategies for statistical learning related problems. By establishing theoretical results and displaying numerical experiments, we show that the different strategies that we propose are an efficient way to deal with scaling issues. In particular, we show how sampling strategies are useful to reduce the complexity induced by the size of the dataset, for problems typically arising in a Big Data context. We highlight the impact of such strategies by establishing upper bounds and asymptotic limit illustrating the trade-off between accuracy and statistical error. We also show how such strategies can be used to speed-up the learning process by providing an analysis in the spirit of [START_REF] Bottou | The tradeoffs of large scale learning[END_REF], bridging the gap between results from statistics and optimization.

For the first problem we consider in Chapter 2 (i.e learning from survey training samples), we show that learning is possible when taking into account the sampling procedure used to sample the observations. Our analysis focuses on the binary Classification problem but can be extended to more general frameworks. However, in Chapter 6, we introduce fast learning rates and the type of hypothesis on the data distribution required to prove them. It is therefore natural to investigate to which extent fast learning rates results can be established when observation are drawn from a general survey sampling scheme. The analysis we develop in chapter 2 would fail because of two main reasons:

• Fast learning rates are established because of small variance property of minimizer of the empirical risk. Unfortunately our analysis is based on upper bounding sup g∈G L n (g) -L n (g) and we can not expect to establish fast rates for this quantity. However, this difficulty is easily overcome by controlling the risk as we do in chapter 6 (or in Boucheron et al. (2005a)).

• The second issue is that we separate the randomness coming from the observations with the randomness coming from the sampling procedure when decomposing the risk. We then dealt with each term separately by using well known results and working conditionally upon the observations. The fast rates being established under assumptions on the data distribution, our analysis would therefore fail.

An other point of interest to study is what lower bounds can we expect to have on the quantity sup g∈G L n (g) -L n (g) . In the rejective case (and more generally for negatively associated r.v.) we expect it to be of order O(1/ √ N ) because negatively associated r.v. "behaves at least as well" as independent r.v. for which lower bound exists (see [START_REF] Lecué | Sharper lower bounds on the performance of the empirical risk minimization algorithm[END_REF] in the i.i.d case).

The second problem we consider in Chapter 3, 4 and 5 was the implementation of sampling strategies for SGD. A very simple extension to our results would be to see how

Concentration Inequalities and Applications to Empirical Risk Minimization

We introduce well-known results as well as the methodology typically applied to derive concentration inequalities. The ideas and results introduced in this chapter are of the utmost importance and are used all along this manuscript. We refer to [START_REF] Mcdiarmid | Concentration[END_REF] and [START_REF] Janson | On concentration of probability[END_REF] for good references on this subject, and [START_REF] Massart | Concentration Inequalities and Model Selection: Ecole d'Eté de Probabilités de Saint-Flour XXXIV[END_REF]; [START_REF] Boucheron | Concentration Inequalities: A Nonasymptotic Theory of Independence[END_REF] for a very complete review on concentration inequalities. References on classification and statistical learning theory include [START_REF] Vapnik | Theory of Pattern Recognition[END_REF]; Devroye et al. (1996b); [START_REF] Bousquet | Introduction to statistical learning theory[END_REF]; Boucheron et al. (2005a); [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]; [START_REF] Friedman | The elements of statistical learning[END_REF][START_REF] Vapnik | The nature of statistical learning theory[END_REF]. We keep the notations introduced for the binary classification problem of section 1.1. We recall that (Ω, A, P) is a probability space and Z = (X, Y ) a random pair defined on (Ω, A, P), taking its values in some measurable product space X × {-1, +1} and for some classifier g in G, L(g) = P{g(X) = Y } denotes its risk and L n (g) its empirical counterpart. We first present in section A.1 some classical result pertaining to the Vapnik-Chervonenkis theory as well as the methodology used to establish this result. In section A.2 we give traditional concentration inequalities that we use in this manuscript.

A.1 Vapnik-Chervonenkis's Inequality and The Method of Bounded Difference

This section is a step by step guide on how to establish Vapnik-Chervonenkis's inequality A.4 for the binary classification problem. Its purpose is to serve as a reference for the different problems we consider in this manuscript. Let g n be an empirical risk minimizer and g * ∈ arg min g∈G L(g). As mentioned in chapter 1, the ERM paradigm requires to establish upper bound for the quantity L( g n ) -L * . This is typically upper bounded by introducing the empirical counterpart of the risk and rewriting this quantity as:

Since g n is an empirical risk minimizer, the quantity L n ( g n ) -L n (g * ) is non positive. The quantities L( g n ) -L n ( g n ) and L n (g * ) -L(g * ) are upper bounded by sup g∈G |L(g) -L n (g)| which overall leads to the bound (1.4):

The class G is supposed rich enough to make the second term on the right hand side small, so that we only have to check that 2 sup g∈G | L n (g) -L(g)| is small to be sure that the empirical risk minimizer g n achieves a similar performance to the one achieved by g * . This result is

Then, for any t ≥ 0,

The same inequality holds true when replacing f (X) -µ by µ -f (X).

It is easy to check that Proposition A.1 can be applied to the r.v.

In particular, this gives that with probability at least 1 -δ

by distribution free constant depending on the number of observations n and the richness of the class G. A first step toward this direction is to introduce a ghost sample z 1 , . . . , z n independent of the z i with same distribution to symmetrize our quantity. We denote by L n the empirical risk based on the ghost sample. Then we have:

We then randomize this inequality by introducing rademacher r.v. σ 1 , . . . , σ n with P(σ i = 1) = P(σ i = -1) = 1 2 independent from our samples Z 1 , . . . , Z n and Z 1 , . . . , Z n . We have:

The quantity

is very interesting as it can be expressed as:

and is the first step toward a distribution-free bound. Indeed, if for any bounded set of vectors

It is very easy to establish upper bound for R n (V) when the set V is finite (see Boucheron et al. (2005a) for instance).

Proposition A.2. Assume V has cardinal N 0 and let V 0 ∈ R such that for any v ∈ V, ||v|| V 0 , then:

The proof of this proposition is given remark A.8. For our problem, G(Z n ) is a subset of {0, +1} n so its elements have norm smaller than √ n and its cardinality is upper bounded by 2 n . Unfortunately this is not satisfactory when plugged into proposition A.2 because th bound obtained would be a O(1). Note that if we could get an upper bound on the cardinal of G(Z n ) that would be polynomial in n instead of exponential, then proposition A.2 would give an upper bound of order O( log (n) n ). This is essentially what happens what when we assume that the class of indicator functions indexed by G has finite VC-dimension. It is defined for any set V as:

An other very important result is Sauer's Lemma (see [START_REF] Sauer | On the density of families of sets[END_REF] for instance) which states that a subset V with VC dimension V has its cardinality bounded by (n + 1) V .

Going back to our problem, the cardinal of G(Z n ) is called the VC shatter coefficient and let V n be its VC-dimension. The logarithm of the shatter coefficient is upper bounded by V n log(n + 1) thanks to Sauer's Lemma. If we assume that we have finite VC-dimension, then there exists V ≤ ∞ such that sup n∈N V n V . Applying proposition A.2 this leads to distribution-free upper bound known as Vapnik-Chervonenkis's inequality: Theorem A.4. (VAPNIK-CHERVONENKIS) For any distribution, we have with probability at least 1 -δ:

The VC-dimension is an important combinatorial parameter of the class, in particular, when G is a d-dimensional vector space of real-valued functions, the class of indicator functions {I{g(.) ≥ 0}} has finite VC-dimension V d. The rest of this chapter is devoted to present other useful results for statistical learning.

A.2 Concentration Inequalities for Empirical Risk Minimization

In this section we present popular inequalities often used in statistical learning. We first deal with bounded r.v. and establish McDiarmid's inequality and Azuma-Hoeffding's inequality.

The main idea used to derive this theorem is to express X -µ as a sum of martingale increment: X -µ = n i=1 Y k and control its Laplace transform using the following lemma:

Lemma A.6. Let (F k ) 0≤k≤n be a filtration of F with F 0 = {∅, Ω}, and (Y k ) 1≤k≤n be a martingale difference for this filtration such that each Y k is bounded. Let Z be any random variable. Then

Proof. This result can be easily proved by induction.

With this lemma, we decompose the expectation of a product into a product of expectations. It can be interpreted as doing 'almost as if' Y k was a sum of independent variables. The next step to prove Theorem A.5 is to control the Laplace transform of bounded r.v., it can be done using Hoeffding's Lemma:

Lemma A.7. (HOEFFDING'S LEMMA) Let X be a centered random variable and (a, b) ∈ R 2 such that a ≤ X ≤ b, then for any h > 0, we have E(e hX ) ≤ e

Equipped with these intermediate results, the proof of Theorem A.5 is quite straightforward:

Proof of Theorem A.5. We start with Chernoff method then apply Lemma A.6, before using Lemma A.7, and finally choose the value of parameter h.

122 Appendix A. Concentration Inequalities and Applications to Empirical Risk Minimization

k so that using Lemma A.6,

We take h = 4t/r 2 , we finally obtain

Remark A.8. Using Lemma A.7, we can prove proposition A.2. Indeed for any λ ≥ 0,

where we use the convexity of the exponential for the first inequality. Hoeffding's Lemma A.7

gives

Taking λ = log(N 0 )2n 2 /V 2 0 gives Theorem A.4.

A.2.1.2 Azuma-Hoeffding's Inequality and McDiarmid's Inequality

In this section, we apply Theorem A.5 and derive classical concentration inequalities as corollaries . The first result that we give is Azuma-Hoeffding's inequality that deals with sum of random variables.

Proposition A.9. (AZUMA-HOEFFDING INEQUALITY) Let (F k ) 0≤k≤n be a filtration of F such that F 0 = {∅, Ω}, Z a martingale and Y the associated martingale difference. If for every k, |Y k | ≤ c k , then we have

The same inequality holds true when replacing n k=1 Y k by -n k=1 Y k .

Proof. Apply Theorem A.5 with

k . Applying this inequality to -X, we obtain the desired result.

The second Theorem that can be derived from Theorem A.5 is McDiarmid inequality used previously in section A.1.

The same inequality holds true when replacing f (X) -µ by µ -f (X).

Proof. Lipschitz condition (A.6) implies that f is bounded, thus from Theorem A.5 we have

where r2 is defined with the filtration F k = σ(X 1 , . . . , X k ) and X = f (X 1 , . . . , X n ). We can easily establish an upper bound on r2 :

The following inequality is similar to Proposition A.9 and can be applied to sum of independent r.v.. It provides a tighter bound than the one in Proposition A.9.

The same inequality holds true when replacing S n -µ by µ -S n .

Proof. This is a immediate consequence of previous McDiarmid inequality (Proposition A.10) with

A.2.2 Bernstein-type Inequality (with Variance Term)

Now, we remove our boundedness hypothesis and instead make variance assumption. The theorem stated below will help us derive the popular Bernstein inequality, that is used to derive fast rate of convergence under appropriate variance control (see [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF]). We make use of a general version of this inequality in chapter 2 and 6. We first set a few notations:

Notation 3. The r.v. var(X|G) := E((X -E(X|G)) 2 |G) is called the conditional variance of X w.r.t. G and we set:

) the conditional variance,

• V = n 1 var k the sum of conditional variances and ν = ess sup(V ) the maximum sum of conditional variances,

) the conditional positive deviation,

• maxdev + = ess sup( max 0≤k≤n dev + k ) the maximum conditional positive deviation.

We now give the following theorem that will help us establish Bernstein-type inequalities.

Theorem A.12. [START_REF] Mcdiarmid | Concentration[END_REF] Let X be a r.v. with E(X) = µ and (F k ) 0≤k≤n a filtration of F such that F 0 = {∅, Ω} and such that X is F n -measurable. Let b = maxdev + the maximum conditional deviation assumed to be finite, and ν = ess sup V the maximum sum of conditional variances also assumed to be finite. Then, for any t ≥ 0,

2(ν+bt/3) , and more generally, for any v ≥ 0,

2(v+bt/3) .

In spirit, the proof of Theorem A.12 is similar to the proof of Theorem A.5 but require a different control of the Laplace transform to make use of variance assumptions. This is achieved by the following Lemma which combined with Hoeffding's lemma A.7 gives the result.

Lemma A.13. Let g defined for x = 0 by g(x) = e x -1-x

x 2

, and X a centered r.v. satisfting for some b ≥ 0, X ≤ b. Then E(e X ) ≤ e g(b)var(X) .

Proof. g is non decreasing so by taylor expansion of the exponential we have e x ≤ 1 + x + x 2 g(b) for x ≤ b. Taking expectation yields E(e X ) ≤ 1 + g(b)var(X) ≤ e g(b)var(X) .

Proof of Theorem A.12. The proof follows the same classical lines as the ones of Theorem A.5. Let Y 1 , . . . , Y n be the martingale differences associated to X and (F k ), and Z = 1 V ≤v . Exponential Markov inequality yields, for every h > 0,

Appendix A. Concentration Inequalities and Applications to Empirical Risk Minimization 125

From Lemma A.13, E(e hY k |F k-1 ) ≤ e h 2 g(hdev + k )var k ≤ e h 2 g(hb)var k so that from Lemma A.6 we obtain,

≤ e h 2 g(hb)v .

By setting h = 1 b ln(1 + bt v ) and using the fact that for every positive x, we have (1 + x) ln(1 + x) -x ≥ 3x 2 /(6 + 2x), we finally get

We finally state Bernstein's inequality as a corollary of Theorem A.12.

Proposition A.14. (BERNSTEIN INEQUALITY) Let X 1 , . . . , X n be n independent random variables with X k -E(X k ) ≤ b. We consider their sum S n = X k , the sum variance V = var(S n ) as well as the sum expectation E(S n ) = µ. Then, for any t ≥ 0,

2(V +bt/3) , and more generally,

2(v+bt/3) .

Remark A.15. (GAIN WITH RESPECT TO INEQUALITIES WITHOUT VARIANCE TERM) Assume that 0 ≤ X i ≤ 1 and consider renormalized quantities, namely Sn := S n /n, μ := µ/n, Ṽ = V /n 2 . Then, Hoeffding) and

with t typically of order between 1/n and 1/ √ n. Thus, if the variance Ṽ is small enough, Bernstein inequality 'almost' allows to have rates in e -nt instead of e -nt 2 . In other words, Bernstein-type inequality may give high probability bounds in 1 n log 1 δ instead of 1 n log 1 δ . This is of the utmost importance in the fast rate analysis we provide in chapter 6.

126 Appendix A. Concentration Inequalities and Applications to Empirical Risk Minimization Therefore ν = ess sup( var k ) = ess sup(V ) = V . Theorem A.12 applies and yields,

PART IV

Résumé des contributions en Français

Appendix B. Résumé des contributions en français du risque min g∈G L n (g), (B.3) où le minimum est pris sur une classe G de classifieurs, supposés assez riches pour inclure le classifieur naïf de Bayes (ou une approximation raisonnable de ce dernier). Considérant une solution g n de (B.3), un problème majeur dans la théorie de l'apprentissage statistique est d'établir des bornes de confiance sur l' excès de risque L( g n ) -L * en l'absence de toute hypothèse distributionnelle mais en prenant en compte de la complexité de la classe G (eg, décrite par des caractéristiques géométriques ou combinatoires telles que la dimension de Vapnick-Chervnonenkis VC), et un controle de l'approximation de P par sa contrepartie empirique

sur la classe G . En effet, l'excès de risque des minimiseurs du risque empirique est typiquement borné de ma façon suivante:

Le second terme du côté droit est appelé bias et dépend de la richesse de la classe G, tandis que le premier terme, appelé erreur stochastique, est contrôlé au moyen de résultats de la théorie des processus empiriques, voir Boucheron et al. (2005a). Malheureusement, l'une des choses généralement négligées dans ce type d'analyse est de savoir comment résoudre efficacement le problème de minimisation du ridque empirique, c'est-à-dire comment trouver g n . Ce problème est habituellement résolu par un algorithme d'optimisation incrémentale, calculant itérativement l'estimateur du gradient du risque empirique. Nous étudions des moyens efficaces d'améliorer le processus d'apprentissage et comment introduire des approches basées sur l'échantillonnage pour construire des approximations de g n . Nous le faisons de deux manières différentes:

• Nous remplaçons le risque empirique L n (g) par une approximation basée sur moins de termes Ln (g). Cela facilite naturellement le problème d'apprentissage. Soit gn un minimiseur de Ln (g), alors (B.4) devient:

Nous nous concentrons sur le contrôle approprié 2 sup g∈G | Ln (g) -L n (g)| que nous faisons généralement conditionnellement sur les observations. Une telle stratégie est discutée et implémentée dans les chapitres ?? et 6 pour deux problèmes différents.

• Lors du calcul de l'estimateur du gradient, la plupart des algorithmes incrémentaux échantillonnent uniformément et indépendamment les observations dans l'ensemble de données. Nous proposons d'utiliser des méthodes d'échantillonnage non uniformes pour calculer un estimateur du gradient de L n avec une variance plus faible. Pour les algorithmes que nous proposons, si nous notons g n (T ) le classificateur obtenu après T itérations de l'algorithme d'optimisation, alors suivant le raisonnement introduit dans de Bottou & Bousquet (2007), l'inégalité (B.4) peut être bornée par:

133

Comme défini dans [START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF], pour tout candidat classificateur g, le risque empirique (non disponible) L n (g) = n -1 1≤i≤n I{Y i = g(X i )} est remplacée par sa version de Horvitz-Thompson: (i) Pour tout δ ∈ (0, 1), avec probabilité au moins 1 -δ, nous havons: ∀N ≤ n,

(B.8)

(ii) Pour toute solution ḡn du problème de minimisation inf g∈G L n (g), nous avons que pour tout δ ∈ (0, 1), avec probabilité au moins 1 -δ, nous havons: ∀n ≥ 1,

Le facteur κ N présent dans les bornes reflète l'influence du schéma d'échantillonnage, (remarquons en particulier que κ n ≥ 1 puisque i≤n π i = N ).

)} sur une classe G. Puisque nous avons déjà établi des résultats dans le cas du schéma rejectif, nous introduisons un schéma d'échantillonnage rejectif R n décrit par les variables aléatoires n , avec probabilités d'inclusion du premier ordre π n = (π 1 , . . . , π n ) ainsi que la quantité suivante:

pour tout classifieur g. Observez que (B.9) est différent du risque empirique de Horvitz Thompson L n (g) lié au schéma d'échantillonnage réjectif n car elle fait intervenir les π * i au lieu des π i . L'excès de risque du minimiseur du risque empirique de Horvitz Thompson peut être borné comme suit:

, où le minimmum est pris sur l'ensemble des plans d'échantillonnage rejectif R n ave probabilité d'inclusion du premier ordre π n = (π 1 , . . . , π n ).

Theorem B.5. Supposons que les hypothèses de la proposition B.2 sont satisfaites. Soit

où le minimum est pris sur l'ensemble des schémas d'échantillonnage rejectif R n ayant pour probabilité d'inclusion du premier ordre les π n = (π 1 , . . . , π N ).

Le taux de convergence obtenu dépend de l'erreur minimale faite lors de l'approximation du plan d'échantillonnage par un plan d'échantillonnage réjectif en termes de distance de variation totale. Il est du même ordre dans le cas où les observations sont échantillonnées uniformément à un terme multiplicatif près et montrent que l'apprentissage avec un échantillon de sondage est possible en tenant compte des probabilités d'inclusion du premier ordre. . Cette stratégie d'échantillonnage peut également être mise en oeuvre efficacement dans la pratique et nous montrons que l'échantillonnage dans le cadre de cette stratégie n'a qu'un coût additionnel de O(log(n)). Les résultats théoriques sont ensuite établis au moyen d'un argument asymptotique où nous montrons qu'avec cette stratégie d'échantillonnage, le comportement asymptotique de θ t est optimal jusqu'à une erreur proportionnelle à ρ. Sous les hypothèses suivantes nous montrons d'abord la convergence de l'agorithme proposé:

Le lemme ci-dessous permet de contrôler l'erreur quadratique moyenne a t = E( θ t -θ * n 2 ), où θ t est généré par l'algorithme 1. La preuve de ce résultat est fortement inspirée de Bach & Moulines (2011a) et A. Nemirovski et al. (2009), où des bornes similaires sont établies.

Lemma B.6. Sous les hypothèses 9 et 10, soit

Ce résultat bien que classique quand l'on cherche à prouver la convergence d'un algorithme du gradient stochastique ne met pas en évidence l'avantage que l'on aurait d'utiliser l'algorithme que nous proposons.

Le corollaire suivant est directement obtenu par la méthode delta du second ordre [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF]. Nous notons Tr(A) la trace d'une matrice carrée A.

Corollary B.9. Sous les hypothèses que théorème B.8, γ -1 t ( L n (θ t ) -L n (θ * )) converge en distribution vers la variable aléatoire V = (1/2)Z T Σ(ρ, ν) 1/2 HΣ(ρ, ν) 1/2 Z où Z es un vecteur gaussien N (0, I d ). De plus, nous havons E(V ) = tr(HΣ(ρ, ν))/2. La Proposition 2 établit que la performance de l'algorithme que nous proposons peut être rendu aussi proche que possible de la performance associée à l'algorithme employant les probabilités optimales, pour peu que ρ soit proche de zero.

où U est une q-dimensionnelle variable aléatoire gaussienne centrée réduite.

Nous discutons ensuite comment choisir π i dans le cas où les variables aléatoires sont de type Poisson (cas où les i sont indépendants) et retrouvons le résultat du chapitre 3 en montrant que l'échantillonnage avec π i proportionnel à ∇l(Z i , θ) donne des résultats optimaux en essayant de minimiser la variance de L n ( θ n (t)) -L n (θ * n ). Proposition B.12. (OPTIMALITY) Sous les hypothèses du théorème 5 quand η = 0 avec

. Alors le sondage de Poisson avec probabilité d'inclusion {p * n (θ)} θ∈Θ defined for all θ ∈ Θ and i ∈ U n by

est solution du problème de minimisation

De plus, nous havons

Nous

. Supposons en plus (en toute généralité) que H(x (1) , . . . , x (K) ) soit symétrique en ses arguments x

Si le stepsize satisait les conditions de 5.4, on a:

Pour tout δ ∈ (0, 1), on a aussi avec probabilité au moins 1 -δ: ∀n

(B.32) pour des constantes C et D β dépendant des paramètres l, α, γ 1 , a 1 .

La limite de généralisation montre l'avantage d'utiliser la méthode incomplète pour obtenir un estimateur de gradient tout en mettant en lumière le fait bien connu que lorsque nous utilisons une méthode d'optimisation pour résoudre le problème de minimisation du risque empirique, nous devons prendre en compte les bornes de généralisation pour que l'erreur d'optimisation soit du même ordre (voir la sous-section B.3.2). conditionnelle étant donné les X i , qui sont des U -statistiques, plus un terme résiduel. Dénotant par Λ(g) = R(g) -R * l'excès de risque de reconstruction par rapport à la règle de Bayes, son estimateur empirique est donné par Λ n (g) = R n (g) -R n (g * ).

Pour tout g ∈ G, nous avons la décomposition suivante:

Λ n (g) -Λ(g) = U n (g) + W n (g), (B.37

est une U -statistique de degree 2.

Sous une certaine condition de «faible bruit», l'analyse effectuée par Clémençon et al. (2008a) 2006) a un facteur 1 en face du terme de biais. Donc, Théorème B.16 est un meilleur résultat à moins que le terme de biais ne domine complètement le second terme de la borne (i.e., la compléxité de G est trop faible).

Nous concluons enfin notre analyse en passant le processus d'apprentissage à l'échelle. Comme pour le chapitre 5, pour les graphes d'entraînement, la complexité de simplement calculer R n (g) est prohibitive car le nombre de termes impliqués dans la sommation est de l'ordre de O(n 2 ). Tout comme nous l'avons fait dans la section B.3.3, nous introduisons une approche basée sur l'échantillonnage pour construire des approximations du risque de reconstruction avec beaucoup moins de termes que O(n 2 ). Au lieu du risque de reconstruction empirique (B.35), nous considérons une approximation incomplète obtenue en échantillonnant des paires de sommets (et non pas sommets) avec remise. Un parallèle peut facilement être tiré avec les résultats obtenus dans le chapitre 5 où nous avons recommandé d'implémenter l'algorithme SGD avec des U -Statistiques incomplètes, ce qui correspond dans ce cas à tirer des arètes plutôt que de noeuds. Formellement, nous définissons le risque de reconstruction des expériences numériques illustrant et justifiant les différents résultats théoriques proposés dans cette sous-section sont aussi réalisés.