Anne Rejl

Corinne Jamroz

Sandrine Meilen

Guillemette Marot

Benjamin Guedj

Pascal Germain

Francis Bach

Gérard Biau

Pascal Massart

Adrien Saumard

Markus Reiß

Stéphane Boucheron

Nicolas Verzelen

Christophe Giraud

Martin Wahl

Jean-Jacques Daudin

Etienne Roquain

Fanny Villers

Adrien Ehrhardt

Cristian Preda

Vincent Rivoirard

Chi Tran Viet

Jonas Kahn

Thanh Mai

Pham Ngoc

Pierre Chainais

Patricia Reynaud-Bouret

Cécile Durot

Catherine Matias

Cristina Butucea

Antoine Chambaz

Emilie Lebarbier

Liliane Bel

Marie-Pierre Etienne

Christian Derquenne

Yannick Baraud

Lucien Birgé

Gwénaelle Castellan

Bertrand Michel

Cathy Maugis

Jean-Michel Poggi

Julien Jacques

Charles Bouveyron

Etienne Birmelé

Franck Picard

Philippe Besse

Béatrice Laurent

Servane Gey

Pierre Neuvial

Ambrym Odalric

Margot Maillard

Jean-François Correard

Sylvain Bouin

Vincent Karpf

Samuel Kubicki

Agathe Blanck

Julien Guilloux

Christophe Chiquet

Marie- Luce Ambroise

Mahendra Taupin

Julie Mariadassou

Sophie Aubert

Sophie Schbath

Emmanuel Dabo

Michel Creusé

Jean-Michel Koskas

Gilles Marin

Gilles Celeux

Stolz

Sylvie Barghav

Marie-Laure Huet

Benoît Martin- Magniette

Jean-Marie Fresse

Patricia Place

Élodie Dillies,. . . Everaere

First of all, I would like to thank the four referees of this manuscript who kindly accepted to take a large part of their time to write a report on my past and recent work. More precisely thank you to Bharath Sriperumbudur of whom I read the numerous papers on the kernel machinery, and to Sara van de Geer that I had the chance to meet briefly during my stay in Zürich in 2006. Je remercie également tout particulièrement Gilles Blanchard pour son abnégation et l'impressionnante clarté de ses articles, et enfin Arnak Dalalyan pour sa grande bienveillance et son soucis des détails.

Mes remerciements vont ensuite aux examinateurs de mon jury Gérard Biau, Catherine Matias et Éric Moulines qui ont chaleureusement accepté de prendre part à ma soutenance de HDR malgré leur emploi du temps plus que chargé ! Merci de me faire l'amitié de partager ces moments de science avec moi.

J'ai ici une pensée particulière pour Stéphane Robin qui a été mon directeur de thèse et, à ce titre, a fait naître en moi le goût de la recherche. C'est notamment en essayant de suivre son exemple que ma pratique de la recherche a mûri. Et même si nous n'avons pas collaboré ensemble depuis de nombreuses années, nul doute que le contenu de ce manuscrit découle de nos interactions passées. Merci à toi pour tes conseils et ton soutien ! Comme notre environnement est un facteur déterminant dans la qualité du travail produit, je profite de ces lignes pour remercier Christophe Biernacki qui a accepté de se porter garant de mon travail, a partagé avec moi plusieurs co-encadrements d'étudiants en thèse, et m'a enfin indirectement inititié à l'art de la création d'une équipe-projet Inria. L'équipe MODAL ainsi créée a été pour moi un environnement de travail précieux à de nombreux égards, et j'en salue ici tous les membres (et bien-sûr son staff technique

Foreword

This manuscript contains some part of the work I have carried out after my PhD as an assistant professor at the Paul Painlevé mathematics laboratory of the Lille University as well as a member of the MODAL project-team at INRIA.

My main motivation, these last years, was to use theory as a means to explain (and sometimes improve) the practice. To this end a large part of my work has been devoted to the analysis and/or improvement of various practical procedures such as the variational approximation in the stochastic block model (SBM) [START_REF] Celisse | Consistency of maximum-likelihood and variational estimators in the stochastic block model[END_REF], the use of the Gaussian distribution over a reproducing kernel Hilbert space (RKHS) [START_REF] Kellner | A one-sample test for normality with kernel methods[END_REF], or the development of statistical tools for multi-patient analysis of genomic markers [START_REF] Grimonprez | Mpagenomics: An r package for multi-patient analysis of genomic markers[END_REF].

But on top of that, most of my time has been devoted to studying cross-validation (CV), which is widely used in practice but still remains sometimes poorly understood. This explains why the present manuscript consists of a synthesis mainly focusing on my contributions to the CV understanding. The goal here is to provide the big picture on CV (in a limited time and space) by considering several complementary aspects of its use. This manuscript does not only report on some published papers. It also contains somewhat new ideas that have been partially explored to highlight potential future developments. Therefore several chapters contain new results (some of them at an early stage) which provide us (at least I hope so) with some new insight on the CV use. For this reason, Chapters 2 and 4 contain (sketches of) proofs helping to identify the main underlying ingredients, whereas some others only discuss published results such as Chapters 5 and 6.

The order of the chapters along this manuscript coincides with the successive questions usually raised by the use of CV and the need for its understanding. Chapter 1 introduces the main concepts and justifications for CV as a means to measure the performance of (statistical/machine) learning algorithms. It also describes a (new) general formulation of CV estimators that is then exploited to make some connections between CV estimators and the literature on U-statistics. Facing the ubiquitous problem of the computational resources saving, Chapter 2 describes several strategies leading either to closed-form expressions for specific CV estimators, or rather to efficient evaluation strategies.

The statistical performances of CV estimators are then discussed along Chapters 3-5. More precisely, Chapter 3 tackles risk estimation, while Chapter 4 details several strategies leading to concentration inequalities for the CV estimator. Some of them are at an early stage, but can still provide several clues towards tighter results. The CV performances for model selection are then discussed in Chapter 5 in the context of density estimation. By contrast, Chapter 6 addresses a different (and more applied) problem that is, the off-line detection of multiple abrupt changes (change-points) arising in a time-series. It illustrates how CV first, and then reproducing kernels, can help improving upon ongoing procedures. Finally the manuscript ends with Chapter 7 which enumerates several new lines of research that seem to be worth considering in the future.

Publications

Peer-reviewed journals Chapter 1

Cross-validation 1.1 Statistical framework

Let us start by introducing some notations that will be used all along the manuscript.

Notation

Let Z 1 , . . . , Z n ∈ Z denote n independent and identically distributed (i.i.d.) random variables with probability distribution P . The purpose of the statistical inference is to estimate a target feature f of the unknown distribution P , such as the density (with respect to a reference measure) or the regression function (see examples in what follows).

With F the set of all possible instances of f , the quality of any t ∈ F to approximate f is measured by a loss function L : F → R such that L(t) is minimal for t = f ,

L(f ) = inf t∈F L(t).
Throughout this manuscript we will consider loss functions defined by ∀t ∈ F, L(t) = L P (t) = E Z∼P [ γ(t; Z) ] ,

(1.1)

where γ : F × Z → R + is called a contrast function and E Z∼P [ • ] denotes the expectation with respect to Z ∼ P . From such a loss function L, two important quantities are the excess loss ∀t ∈ F, (f, t) = L(t) -L(f ), (1.2) and the (excess) risk of an estimator f = f (Z 1 , . . . , Z n )

R( f ) = E Z1,...,Zn∼P f, f (Z 1 , . . . , Z n ) . (1.3)
Note that the definition given by Eq. (1.1) covers most of classical statistical frameworks as illustrated by the following examples.

Density estimation aims at estimating the density f of P with respect to some given measure µ on Z . Then, F is the set of densities on Z with respect to µ . For instance, taking γ(t; x) = -ln(t(x)) in (1.1), the loss is minimal when t = f and the excess loss

(f, t ) = E Zi∼P ln f (Z i ) t(Z i ) = f ln f t dµ
is the Kullback-Leibler divergence between distributions tµ and f µ .

13

Prediction aims at predicting a quantity of interest Y ∈ Y given an explanatory variable X ∈ X and a sample (X 1 , Y 1 ), . . . , (X n , Y n ) . In other words, Z = X × Y , F is the set of measurable mappings X → Y and the contrast γ(t; (x, y)) measures the discrepancy between y and its predicted value t(x). Note that often in prediction, a cost function c : Y × Y → R + is also introduced that is related to the contrast function by ∀(x, y) ∈ X × Y, c(y, t(x)) = γ (t; (x, y)) .

Two classical prediction frameworks are regression and classification, which are detailed below.

Regression corresponds to a set Y ⊂ R (or R k for multivariate regression), the feature space X being typically a subset of R . Let f denote the regression function, that is f

(x) = E [ Y | X = x ] , so that ∀i ∈ {1, . . . , n} , Y i = f (X i ) + ε i with E [ ε i | X i ] = 0 . (1.4)
In regression the least-squares contrast is given by γ (t; (x, y)) = (t(x) -y) 2 , and the excess loss is

(f, t ) = E (X,Y )∼P (f (X) -t(X)) 2 .
Note that the excess loss of t is the square of the L 2 (P ) distance between t and f , so that prediction and estimation here are equivalent goals.

Remark 1.1. Let us mention that the above contrast and excess loss have been formulated in the out-ofsample error context, where prediction/estimation is not necessarily carried out at the same positions as the observations in the initial sample. By contrast in numerous settings, quantifying the performance of an estimator is easier in the in-sample context. The positions at which any new observation is made are then considered as deterministic. They are given by the n positions {x 1 , . . . , x n } in the initial sample. The model (1.4) becomes ∀i ∈ {1, . . . , n} ,

Y i = F i + ε i with E [ ε i ] = 0 , (1.5)
where F i = f (x i ) for every i, and only the i s are random variables. In this framework, the prediction performance of any vector t ∈ R n is measured by the contrast γ (t; Y ) = 1/n n i=1 (t i -Y i ) 2 , where t, Y ∈ R n . The excess loss is then

(F, t) = 1/n n i=1 (t i -F i ) 2 := t -F 2 n .
Classification corresponds a finite set Y (at least discrete). In particular, when Y = {0, 1} , the prediction problem is called binary (supervised) classification. With the 0-1 contrast function γ(t; (x, y)) = 1 t(x) =y , the minimizer of the loss is the so-called Bayes classifier f defined by ∀x ∈ X , f (x) = 1 η(x)≥1/2 , where η denotes the regression function η(x) = P (X,Y )∼P (Y = 1 | X = x) .

Note that using alternative convex losses such as the hinge, exponential, and logit ones has been also considered in countless works (see for instance the survey by [START_REF] Boucheron | Theory of classification: a survey of some recent advances[END_REF], for many references on the learning theory in the classification context).

Statistical/machine learning algorithms

In the present manuscript, any measurable mapping A : n∈N Z n → F is called a learning algorithm if, for every sample D n = (Z i ) 1≤i≤n ∈ Z n , it outputs an estimator of f denoted by A(D n ) = f A (D n ) ∈ F In situations where the learning algorithm is clearly identified from the context, this estimator is simply noted f (D n ) = f A (D n ) ∈ F. The quality of A for a given sample D n is then measured by L f (D n ) , which is a random variable that should be as small as possible.

Although it is out of the scope of the present manuscript to review all of them, several principles leading to classical learning algorithms are listed below to introduce some of the material that will be used in next chapters.

Minimum contrast estimators

They refer to a classical family of learning algorithms. Given some subset F of F called a model, a minimum contrast estimator over F refers to any estimator f (D n ) ∈ F minimizing (over F ) the empirical contrast The corresponding minimum contrast algorithm, associated with the model F , is the resulting mapping D n → f (D n ). The intuitive justification for using the empirical contrast L Pn (t) is that its expectation is equal to the loss L P (t), which is minimal for t = f . Minimizing L Pn (t) over a set F of candidate values for f hopefully leads to a good estimator of f . Note also that empirical contrast minimizers are particular instances of the broad family of M-estimators (van der [START_REF] Van Der Vaart | Weak convergence and empirical processes[END_REF]. Classical examples of minimum contrast estimators are:

• Maximum likelihood estimators in density estimation on [0, 1]: With γ(t; x) = -ln(t(x)), a possible choice for F is the vector space of piecewise-constant functions on the regular partition of X = [0, 1] into D equal-size intervals. This leads to the histogram estimator with D bins with length 1/D.

• Least-squares estimators in regression (Y ⊂ R): With γ(t; (x, y)) = (t(x) -y) 2 , let F denote the set of piecewise-constant functions on some fixed partition of X ⊂ R d . Then the resulting empirical contrast minimizer is called a regressogram.

Local averaging estimators

Density estimation:

• Nearest neighbor density estimators: If f denotes a density with respect to the Lebesgue measure λ(•) over R d , the Lebesgue differentiation theorem (Biau and Devroye, 2016, Theorem 20.18) suggests an estimator f k of f (for 1 ≤ k ≤ n) based on the k nearest neighbors of x among Z 1 , . . . , Z n . For every x ∈ R d ,

f k (D n ; x) = f k (x) = P Dn B x, R Dn k (x) λ B x, R Dn k (x)
,

where P Dn = 1/n n i=1 δ Zi is the empirical measure associated with D n , R Dn k (x) denotes the distance between x and its k-th nearest neighbor among Z 1 , . . . , Z n , and B (x, R) ⊂ R d refers to the ball centered at x with radius R. For 1 ≤ k ≤ n, the corresponding k-nearest neighbor algorithm is the mapping D n → f k (D n ; •).

• Kernel density estimators: From a kernel K : Z → R such that Z Kdz = 1 [START_REF] Tsybakov | Introduction à l'estimation non-paramétrique[END_REF] Section 1.2) and h > 0, the (Parzen-Rosenblatt) kernel density estimator of f is given, for every x ∈ Z, by

f h (D n ; x) = f h (x) = 1 n n i=1 K h (Z i -x) = K h P Dn (x),
where K h (•) = 1/hK(•/h) and P Dn = 1/n n i=1 δ Zi . For a given bandwidth h > 0, the resulting kernel density algorithm is then D n → f h (D n ; •).

Note that with the above notations, the kernel density estimator f h (D n ; x) is equal to the k nearest neighbor density estimator f k (D n ; x) with the particular choice of

K(•) = 1/21 |•|≤1 and h = h(D n ) = R Dn k (x)
.

CHAPTER 1. CROSS-VALIDATION

Regression A common strategy in nonparametric regression (Y ⊂ R) is to use local averaging estimators (Györfi et al., 2006, Section 2.1). Given some weights W n,i : X → R, a point-wise estimator of the regression function f is defined for every x ∈ X by

f (D n ; x) = argmin t∈R n i=1 W n,i (x) (Y i -t) 2 .
(1.6) Therefore W n,i modulates the influence of Y i in the evaluation of the estimator. This general formulation leads to widely used estimators by modifying the weights.

• Nadaraya-Watson kernel estimators: By choosing data-dependent weights such as W n,i (•) = K h (X i -•) for every 1 ≤ i ≤ n and h > 0, the minimizer of Eq. (1.6) leads to the Nadaraya-Watson kernel algorithm defined by

D n → f h (D n ; •) = n i=1 Y i K h (X i -•) n j=1 K h (X j -•) , if n j=1 K h (X j -•) = 0 = 0 , otherwise.
• Nearest neighbor estimators: From a given distance between points in X ⊂ R d (for instance based on the Euclidean norm), let us define the set V Dn k (x) of the k nearest neighbors of x (for 1 ≤ k ≤ n). Then, taking W n,i (x) = 1/k1 Xi∈V Dn k (x) for every x ∈ X , the minimizer of Eq. (1.6) is the kNN algorithm

D n → f k (D n ; •) = 1 k n i=1 Y i 1 Xi∈V Dn k (•) .
Numerous examples related to plug-in estimators in the classification context can be found for instance in the seminal textbook by [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF].

Variations around minimum contrast estimators

Iterative algorithms As illustrated by the previous examples, numerous estimators are defined as one minimizer of a functional Ψ(D n ; •) : F → R which depends on the sample D n and is defined over a given set F , that is

f (D n ) ∈ argmin t∈F Ψ(D n ; t).
(1.7)

Such an implicit definition often leads to estimators with no closed-form expression that can only be evaluated by numerical approximations. This phenomenon can result for instance from the difficult-tohandle expression of Ψ and/or from the constraints defining the set F over which the minimization must be performed. Some illustrations can be found for instance in [START_REF] Catoni | Challenging the empirical mean and empirical variance: a deviation study[END_REF] where robust estimators of the mean and variance are derived from a Huber-type loss, or in [START_REF] Celisse | Consistency of maximum-likelihood and variational estimators in the stochastic block model[END_REF] with maximum likelihood estimators in the stochastic block model where computing the log-likelihood is itself intractable. Solving (at least approximately) Eq. (1.7) then requires the use of iterative optimization algorithms to explore the set F and provide the approximate solution at the ith iteration, denoted by f i (D n ). A few instances of such iterative algorithms are the expectation-maximization (EM) algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF], the (stochastic) gradient descent algorithm [START_REF] Robbins | A stochastic approximation method[END_REF], the forward and backward stepwise variable selection strategies [START_REF] Hastie | The elements of statistical learning[END_REF], and the binary segmentation heuristic in change-point detection [START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF].

Then the resulting learning algorithm relies on the outcome of the used iterative optimization algorithm at the ith iteration, that is

D n → f i (D n ) ∈ F.
(1.8)

Remark 1.2. Worst-case bounds are usually derived in optimization to lower bound the minimal number of iterations required to (approximately) solve problems like that of Eq. (1.7) with a prescribed precision. However such lower bounds are unfortunately useless to describe the actual statistical performance of f i (D n ) with respect to the number i of iterations. In practice, this performance can be very different from that of the estimator at the limit (as i → +∞) as illustrated by Figure 1.1 (see also Section 7.1 about the theoretical analysis of such iterative learning algorithms). Learning algorithms based on calibration procedures Minimum contrast and local averaging estimators depend on unknown parameters (respectively the model and the bandwidth or number of neighbors). Actually f (D n ) = f θ (D n ), where θ denotes an unknown parameter. Its value has to be chosen from the data within a set of candidate values Θ.

A calibration (model selection) procedure must be applied to choose the value of θ from the data. Most of classical calibration procedures consists in minimizing a criterion C : Θ → R over the set Θ. Therefore the final estimator is f θ(Dn) (D n ), where

θ(D n ) ∈ argmin θ∈Θ C(θ).
If the calibration procedure C does not depend itself on any unknown parameter, then the resulting learning algorithm is

D n → f θ(Dn) (D n ).
However it is often the case that the calibration/model selection procedure depends itself on an unknown parameter that has to be user-specified. For instance, the final prediction error (FPE) criterion of [START_REF] Shibata | Approximate efficiency of a selection procedure for the number of regression variables[END_REF] and the generalized information criterion (GIC) introduced by [START_REF] Nishii | Asymptotic properties of criteria for selection of variables in multiple regression[END_REF] both depend on a regularization parameter λ > 0 which determines the weight of the penalty in the optimization of C λ (see also [START_REF] Shao | An asymptotic theory for linear model selection[END_REF], for an extensive comparison of model selection criteria). Let us notice that regularized least-squares strategies such as Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] and Ridge regression [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF] also enter this framework. It results the following learning algorithm, which itself requires a data-driven choice of λ,

D n → f θ λ (Dn) (D n ),
where θ λ (D n ) denotes the parameter value obtained by minimizing C λ (•) over Θ.

Cross-validation procedures

The present section reviews the main cross-validation (CV) procedures. The first goal is to introduce some important ideas and emphasize the respective merits of each CV procedure. Then several new connections between cross-validation estimators and U-statistics are exposed and discussed from Section 1.2.4. These connections are important since they will serve as a starting point in the theoretical analysis of CV procedures.

In all what follows, we focus on symmetric learning algorithms A that is, algorithms such that A(Z 1 , . . . , Z n ; •) = A(Z σ(1) , . . . , Z σ(n) ; •) a.s.,

where σ(•) denotes any permutation of {1, . . . , n}.

Estimate the risk by Hold-out

Validation error

A central question in statistical learning is the performance assessment of an estimator f = f (Z 1 , . . . , Z n ), which can be achieved by estimating the excess loss (1.2) or the risk (1.3) of this estimator. According to Eq. (1.1), doing so requires a new point Z ∼ P that is independent of the initial sample D n = {Z i } 1≤i≤n .

From an (almost) unlimited amount of data, one can use (for free) m additional independent copies Z n+1 , . . . , Z n+m ∼ P that have been left aside to estimate the loss of f . This estimator of L P f results from computing the empirical error of f on the so-called validation set Z n+1 , . . . , Z n+m , with m taken as large as possible, using that

1 m m i=1 γ f (D n ) ; Z n+i P -----→ m→+∞ E Z∼P γ f (D n ) ; Z = L P f a.s. .

Splitting the data: Hold-Out estimator

Unlike the previous situation, only a limited amount of observations is usually available. This motivates a completely different strategy of performance assessment. Its basic idea is that any Z i used to compute the estimator f can no longer serve for its performance assessment.

This gives rise to an important idea in statistics which consists in splitting the sample D n = {Z 1 , . . . , Z n } into two disjoint subsets as follows. For any integer 1 ≤ p ≤ n -1, let e ⊂ {1, . . . , n} denote a set of n -p indices randomly chosen, and ē = {1, . . . , n} \ e its complement of cardinality p. Let us now introduce (1.9)

D ē = {Z i | i ∈
Note that the quantity given by Eq. (1.9) is called the p-Hold-Out (HOp) estimator of the loss of the estimator f (D n ), namely L P f (D n ) .

Remark 1.3. Let us make a few important comments on the HOp estimator.

• As required in Eq. (1.1), the independence assumption between the training and test sets is fulfilled since Z 1 , . . . , Z n are assumed to be independent and training and test sets are disjoint sets.

• The splitting strategy depends on the cardinality p ∈ {1, . . . , n -1} of the test set that has to be chosen beforehand. This parameter allows to control an important trade-off arising from Eq. (1.9).

On the one hand, p determines the amount of data used to assess the performance of the estimator.

Intuitively this assessment will be all the more accurate as p is large. On the other hand, only n -p observations are devoted to compute the estimator A (D e ). Therefore the potential difference between A(D n ) and A (D e ) is likely to increase as p grows. The magnitude of the difference between A(D n ) and A (D e ) is related to the notion of stability for algorithm A. The connection with the notion of stability is further discussed at several places along the manuscript (see for instance Sections 3.1.3 and 4.2.2). Note that for a fixed value of p, the intuition suggests that this difference between A(D n ) and A (D e ) should vanish as n increases to +∞.

• For a given value of p, the estimator defined by Eq. (1.9) is computed from an arbitrary (random) split of the data into training and test sets. This random choice has at least two main drawbacks.

First, it induces an additional variability of the resulting HOp estimator. Indeed let us assume we are in the binary classification setting where the purpose is to classify any new observation into either the 0 or 1 class. The chosen random split of the data can put all 0-class observations in the training set and all remaining 1-class observations in the test set. This would certainly lead to a bad estimation of the true performance of A to classify new points.

Second, one could be tempted to say that some of the available information has been lost since only n -p observations (instead of n) have been used to compute the estimator. One could prefer a splitting strategy such that the n available data are involved in the computation of the estimator (not necessarily at the same time).

Exhaustive and non-exhaustive CV procedures

General formula of the CV estimator

As emphasized in Eq. (1.9), the HOp estimator crucially depends on a randomly chosen split of the data.

If the training set D e is not similar to the whole sample D n , then it results a bad assessment of the performance of the estimator, which should be avoided.

The cross-validation (CV) principle consists in repeating the splitting step leading to Eq. (1.9) several times to relax the dependence on a particular (arbitrary) split. For any 1 ≤ p ≤ n -1, let E n-p = {e ⊂ {1, . . . , n} | Card(e) = p} (the set of all possible subsets of {1, . . . , n} with cardinality p). Then any CV estimator of the risk of A(D n ) can be written as

R CV p (A, D n ) = R W p (A, D n ) = e∈En-p
W e e ∈En-p W e L P D ē (A (D e )) , (1.10)

where W : E n-p → R + is a mapping such that W (e) = W e is a random variable independent of D n . Importantly, W is fully characterized by the choice of one CV procedure (that is, of one particular splitting scheme). Note that the HOp procedure enters this definition by first choosing a training set E at random, uniformly over E n-p and then, defining W e = 1 if e = E and 0 otherwise.

Exhaustive CV procedure

Exhaustive CV procedures are procedures for which all possible splits of D n = (Z 1 , . . . , Z n ) into training and test sets (with respective cardinality n -p and p) are considered. From the above Eq. (1.10), it corresponds to the situation where W e = 1 for all e ∈ E n-p , which leads to e ∈En-p W e = n p .

For every 1 ≤ p ≤ n -1, there is a unique exhaustive CV procedure, which is called leave-p-out (LpO) [START_REF] Geisser | The predictive sample reuse method with applications[END_REF][START_REF] Shao | Model Selection by Cross-Validation[END_REF][START_REF] Zhang | Model selection via multifold cross validation[END_REF]. For any learning algorithm A, the Lpo estimator is defined by

R LpO p (A, D n ) = R ECV p (A, D n ) = n p -1 e∈En-p L P D ē (A (D e )) .
(1.11)

In the particular case where p = 1, the Lpo estimator reduces to the celebrated leave-one-out (L1O) estimator, that is

R ECV 1 (A, D n ) = 1 n n i=1 γ (A (τ i (D n )) ; Z i ) ,
where, for every sample D n = (Z 1 , . . . , Z n ), τ i (D n ) = (Z 1 , . . . , Z i-1 , Z i+1 , . . . , Z n ) ∈ Z n-1 [START_REF] Allen | The relationship between variable selection and data augmentation and a method for prediction[END_REF][START_REF] Geisser | The predictive sample reuse method with applications[END_REF][START_REF] Stone | Cross-validatory choice and assessment of statistical predictions[END_REF].

LpO and HOp estimators share several common features:

• Increasing p makes the resulting estimator more and more different from the one computed from the n available observations. Therefore the bias of the LpO estimator (as an estimator of the risk of A(D n )) intuitively increases as p grows. Obviously the extent to which increasing p influences the bias strongly depends on the underlying algorithm.

• In the same time, increasing p should enhance the performance evaluation since more and more observations are devoted to this task.

However unlike the HOp procedure, LpO exploits all available information since all observations contribute to the estimator computation and its performance evaluation (but not in the same time). Nevertheless the variance of the LpO estimator is more difficult to evaluate than that of the HOp estimator since it involves the interaction between all pairs of subsamples.

From a practical point of view, the main drawback of exhaustive CV procedures is their high computation cost. Computing the LpO estimator requires to successively consider all n p possible splits of the data, which is highly time consuming in general, and even prohibitive as n becomes large. For instance the computational complexity of computing the LpO estimator is of order O (n p × C A (n)) in time, where C A (n) denotes the time complexity of computing the output of algorithm A learned from n observations.

Unfortunately in many situations, C A (n) is itself high, which makes the L1O useless in practice. This is in particular true in the big data setup where n is so huge that even only one pass on the data is costly.

Non-exhaustive procedures

In order to overcome the computational limitation of the LpO procedure in general, several non-exhaustive strategies have been suggested [START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF]. In what follows, we only discuss some of them without trying to be exhaustive, which is out of the scope of the present manuscript.

V-fold cross-validation One of the most famous non-exhaustive CV procedures is the V -fold crossvalidation (V-FCV). Assuming for simplicity that the positive integer V divides the sample size n, V-FCV relies on a random partitioning of the whole sample into V disjoint subsets of cardinality p = n/V .

Let e 1 , . . . , e V denote the corresponding V subsets of indices such that ∪ V b=1 e b = {1, . . . , n} and Card(e b ) = p for every 1 ≤ b ≤ V . Then, the V-FCV estimator is defined for every learning algorithm A and sample D n , by

R F CV p (A, D n ) = 1 V V b=1 L P D ēb (A (D e b ))
.

The V-FCV procedure corresponds to Eq. (1.10) with random variables {W e } e∈En-p defined as follows:

1. ( Ē1 , . . . , ĒV ) denotes a random partition of {1, . . . , n} into V disjoints sets of cardinality p = n/V , 2. For every e ∈ E n-p , W e = 1, if e ∈ {E 1 , . . . , E V } 0, otherwise.

From a computational point of view, the complexity of V-FCV is O(V C A (n -n/V )) in time, which is by far less than that of the LpO procedure. Note also that V-FCV coincides with L1O for V = n.

One important drawback of the V-FCV procedure is that when V is small (for instance V = 2), the V-FCV estimator is computed from only one split of the data into two equal-size disjoint subsets. If the resulting training and test sets are quite different from the whole sample (for instance if most of the training set data belong to one class while the test set data belong to an other one), then the V-FCV estimator is highly mistaken due to high variability.

Remark 1.4 (Repeated V-FCV). A natural way to remedy the above drawback is to repeat the random partitioning step (Step 1 of the V-FCV procedure) B > 0 times. This procedure si called the B-repeated V-FCV [START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF].

Delete-p cross-validation

Except its high computational cost, the LpO procedure can be considered as an ideal CV procedure. Indeed unlike V-FCV for instance, it does not require any preliminary random partitioning of the data, which avoids any additional variability.

This remark suggests trying to approximate the LpO estimator in order to get an estimator with an additional variance remaining as small as possible, and a computational cost kept under control. This leads us to the so-called delete-p CV procedure [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF] which consists in randomly choosing B splits of the data among the n p possible ones and averaging the corresponding HOp estimators. For any 1 ≤ p ≤ n -1, any integer B > 0 and algorithm A, let {e b } 1≤b≤B denote the B randomly chosen training sets with cardinality n -p. Then the Delete-p estimator is given by

R Del p (A, D n ) = 1 B B b=1 L P D ēb (A (D e b ))
.

Two different strategies must be distinguished among Delete-p CV procedures: (i) the Monte-Carlo-p CV (MCCVp) which relies on a random uniform choice of the splits with replacement [START_REF] Picard | Cross-validation of regression models[END_REF], and (ii) the Repeated-Learning-Testing-p (RLTp) CV where the splits are drawn uniformly without replacement (Breiman et al., 1984a;[START_REF] Zhang | Model selection via multifold cross validation[END_REF].

• MCCVp: This estimator results from Eq. (1.10) by: 1. Choose randomly B sets E 1 , . . . , E B uniformly over E n-p with replacement, 2. For every e ∈ E n-p ,

W e = Card ({1 ≤ b ≤ B | e = E b }) .
• RLTp: Similarly, the definition of W e that leads to the RLTp estimator is given by: 1. Choose randomly B sets E 1 , . . . , E B uniformly over E n-p without replacement, 2. For every e ∈ E n-p ,

W e = 1, if e ∈ {E 1 , . . . , E B } 0, otherwise.
Note that the V-FCV procedure (with p = n/V ) can be interpreted as an instance of Delete-p CV by adding the partitioning constraint on the random sets Ē1 , . . . , ĒB .

Remark 1.5. It is noticeable that all the aforementioned CV procedures share the same expectation. All of them behave similarly at first order, that is in expectation. The main difference between them results from the variance (and the higher moments) resulting from the splitting scheme they are based on. Precisely quantifying this difference on the behavior of CV estimators is an important and challenging question that remains widely open up to now, even if some work has been done recently in that direction by [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] in density estimation with the quadratic loss.

LpO and minimum variance CV estimator

For all CV procedures described in previous Section 1.2.2, the general formula given by Eq. (1.10) allows to draw a comparison between the different resampling schemes in terms of bias and variance. This is the purpose of the next result which establishes first that all CV procedures share the same mean, but also that LpO has the smallest variance among CV procedures described in Section 1.2.2.

Proposition 1.1. For any 1 ≤ p ≤ n -1 and symmetric learning algorithm A, let R W p (A, D n ) denote the CV estimators defined by Eq. (1.10) from the weights {W e } e∈En-p . Then, The straightforward proof is not reproduced here. But it mainly relies on the following lemma.

E W R W p = R ECV p , Var R W p = Var Dn R ECV p + E Dn Var W R W p ≥ Var Dn R ECV
Lemma 1.1. For any 1 ≤ p ≤ n -1 and e ∈ E n-p , and any integer B ≥ 1, it comes Hold-p-Out:

P W W HOp e = 1 = n p -1 , V-Fold CV: P W W F CVp e = 1 = V × n p -1 , (with V = n/p ≥ 2)
RLT-p:

P W W RLT p e = 1 = B n p -1 , MCCV-p: P W W M CCV p e = b = B b n p -1 b 1 - n p -1 B-b , ∀ 0 ≤ b ≤ B.
Note that the last equality results from the fact that W e follows a multinomial distribution M B; n p , . . . , n p .

CV procedures all have the same bias, but differ from one another in terms of variance. In particular, LpO is the least variable CV procedure among those discussed in Section 1.2.2. However since explicitly computing the LpO estimator requires summing over n p terms, approximations such as MCCVp or RLTp remain computationally feasible alternatives.

Remark 1.6. An important (but still open) question when using such alternatives is to optimize the tradeoff between the computational cost and the statistical precision, which arises from the choice of B. A large value of B provides an estimator with a small variance, but increases the computational complexity that is of order O(B × C n-p ) in time, where C n denotes the time complexity induced by computing A(D n ; z) at any point z. In particular, B should be chosen large enough to marginally impact the variance of the resulting CV estimator compared to that of LpO (see also Section 3.2 and more precisely Proposition 3.3 for a quantification of the amount of variability induced by the non-exhaustive splitting scheme).

Connections with U-statistics

The present section aims at highlighting new existing connections between CV estimators and U-statistics. This enables exploiting existing results on U-statistics in the theoretical analysis of CV procedures (see Chapter 3 and Section 4.2.1 in particular).

Firstly, we start by describing explicitly the link between LpO and (complete) U-statistics. Secondly, we provide some preliminary attempts to relate non-exhaustive CV procedures to incomplete U-statistics.

Brief introduction to U-statistics

U-statistics arise in numerous statistical frameworks such as two-sample tests (Gretton et al., 2012a), density estimation [START_REF] Lerasle | Optimal kernel selection for density estimation[END_REF], Gini's mean difference [START_REF] Gerstenberger | On the efficiency of the Gini mean difference[END_REF], and ranking [START_REF] Clémençon | Ranking and empirical minimization of u-statistics[END_REF]. Their first theoretical analysis can be traced back at least to Hoeffding's paper [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF].

U-statistics ( (Denker, 1985, Chap. 1) and (Lehmann, 1999, Chap. 6)) have been introduced to estimate a parameter θ ∈ R that can be expressed as

θ = E [ h (Z 1 , . . . , Z m ) ] ,
(1.12)

where the integer m ≥ 1 is called the degree of θ and denotes the minimal integer such that a measurable map h : Z m → R exists and h (Z 1 , . . . , Z m ) is an unbiased estimator of θ.

Remark 1.7. Since h in Eq. (1.12) can be replaced without loss of generality by

1/m! σ h Z σ(1) , . . . , Z σ(m)
where the sum is taken over all permutations σ of {1, . . . , m}, it is usually assumed that h is symmetric with respect to its m arguments.

Let us now recall the definition of a U-statistics with kernel h and order m, that is Definition 1.1 (U-statistic of order m). For any integer m ≥ 1, let h : Z m → R be a measurable mapping assumed to be symmetric in its m arguments. Then for any integer n ≥ m,

U n (h) = n m -1 σ∈Πn h Z σ(1) , . . . , Z σ(m) (1.13)
is a U-statistic of kernel h and order m, where σ∈Πn denotes the sum over all permutations of {1, . . . , n}.

Let us mention that [START_REF] Hoeffding | A class of statistics with asymptotically normal distribution[END_REF][START_REF] Hoeffding | Probability inequalities for sums of bounded random variables[END_REF] have also investigated the case of non-symmetric kernels h. But in what follows, we mainly focus on symmetric ones.

U-statistics defined by Eq. (1.13) are also called complete U-statistics since their definition involves all possible permutations of {1, . . . , n}. For computational reasons at least, it can be worth considering incomplete U-statistics [START_REF] Blom | Some properties of incomplete u-statistics[END_REF] where only a subset of all possible permutations is considered (see Eq. (1.18)).

Most existing results on U-statistics are provided under the assumption that the order m is kept fixed, that is independent of the sample size n. This allows to use decoupling techniques to prove moments or concentration inequalities [START_REF] Adamczak | Moment inequalities for u-statistics[END_REF][START_REF] Arcones | A bernstein-type inequality for u-statistics and u-processes[END_REF][START_REF] De La Pena | Decoupling: From Dependence to independence[END_REF]. Resulting upper bounds only exhibit a dependence on m in the constants. However this conclusion becomes very different when the order m is allowed to depend on n, which would clearly worsen upper bounds derived by decoupling techniques (de la Pena and Giné, 1999). Such U-statistics with an order m allowed to increase with n are called infinite-order U-statistics. For instance, they are introduced and studied by [START_REF] Frees | Infinite order u-statistics[END_REF]; [START_REF] Heilig | Limit theorems for the infinite-degree u-process[END_REF]; [START_REF] Kohatsu-Hia | Weak convergence of infinite order u-processes[END_REF]; [START_REF] Rempala | Strong law of large numbers for u-statistics of varying order[END_REF].

The LpO estimator as a U-statistic

Here we focus on the LpO estimator given, for any learning algorithm A and any 1 ≤ p ≤ n -1 such that the following quantity exists, by

R ECV p (A, D n ) = n p -1 e∈En-p L P D ē (A (D e )) .
Despite some similarities the connection between the LpO estimator and U-statistics is not straightforward. One first difficulty is that L P D ē (A (D e )) is a function of the whole sample D n for every e ∈ E n-p . For bypassing this difficulty, we use the definition of

L P D ē (A (D e )) = p -1 i∈ē γ (A (D e ) ; Z i ) , so that R ECV p (A, D n ) = 1 p n p -1 e∈En-p i∈ē γ (A (D e ) ; Z i ) .
Then a candidate kernel arises that is the function h(D e , Z i ) = γ (A (D e ) ; Z i ), which depends on n -p + 1 arguments. However compared with usual assumptions on U-statistics given by Eq. (1.13), a second problem is that this candidate kernel is not a symmetric function. This last difficulty is overcome by the following result which defines a valid symmetric kernel of order m = n -p + 1.

Theorem 1.1. For any symmetric learning algorithm A and any 1 ≤ p ≤ n -1 such that the following quantities are well defined, the LpO estimator

R ECV p (A, D n ) is a U-statistic of order m = n -p + 1 with kernel h m : Z m -→ R defined by h m (Z 1 , . . . , Z m ) = 1 m m i=1 γ A(D (i) m ); Z i , (1.14)
where

D (i)
m denotes the sample (Z 1 , . . . , Z m ) where Z i has been withdrawn. Let us first notice that the kernel h m defined in Eq. (1.14) is symmetric. This will allow us to exploit some existing results in the U-statistics literature to describe the behavior of the LpO estimator.

However an important remark is that the order of the U-statistic is m = n -p + 1. Since the order depends on (and even increases with) n as p is kept fixed, one cannot apply the usual asymptotic normality result ( (Lehmann, 1999, Theorem 6.1.2, p. 369)) that holds true as long as m remains constant. At least the latter normality result still applies to LpO in the very restrictive case where n -p is a constant independent of n.

Finally let us also mention that the kernel h m itself is equal to the L1O estimator, that is

h m (Z 1 , . . . , Z m ) = 1 m m i=1 γ A(D (i) m ); Z i = R ECV 1 (A, D m ). (1.15)
This gives rise to a new strategy to analyze the behavior of the LpO estimator. It consists in:

• first, stating bounds controlling the performance of the L1O estimator,

• second, linking these bounds with the LpO estimator by exploiting the connection with U-statistics.

For instance our hope is typically to be able to improve upon inequalities such as (Arcones, 1995, Ineq. 1.4 and 2.10) since the latter cannot exploit the concentration properties of the kernel itself.

Proof of Theorem 1.1.

For every 1 ≤ t ≤ n -1, let E t = {e ⊂ {1, . . . , n} | Card(e) = t} denote the set of all possible subsets e of {1, . . . , n} with cardinality t. Then,

R ECV p (A, D n ) = 1 n p e∈En-p 1 p i∈ē γ (A (D e ) ; Z i ) = 1 n p e∈En-p 1 p i∈ē   v∈En-p+1 1 {v=e∪{i}}   γ (A (D e ) ; Z i ) ,
since there is a unique set of indices v with cardinality n -p + 1 such that v = e ∪ {i}. Then

R ECV p (A, D n ) = 1 n p v∈En-p+1 1 p n i=1   e∈En-p 1 {v=e∪{i}} 1 {i∈ē}   γ A D v\{i} ; Z i .
Furthermore for v and i fixed, e∈En-p 1 {v=e∪{i}} 1 {i∈ē} = 1 {i∈v} since there is a unique set e of indices such that e = v \ i. One gets

R ECV p (A, D n ) = 1 p 1 n p v∈En-p+1 n i=1 1 {i∈v} γ A D v\{i} ; Z i = 1 n n-p+1 v∈En-p+1 1 n -p + 1 i∈v γ A D v\{i} ; Z i , by noticing p n p = p n! p! (n-p)! = n! (p-1)!(n-p)! = (n -p + 1) n n-p+1 .

Non-exhaustive CV procedures and incomplete U-statistics

Coming back to non-exhaustive CV procedures such as n/p-FCV, RLTp or MCCVp (see Section 1.2.2), we now turn to the problem of making some possible connections with U-statistics.

The exhaustive CV procedure point of view Let us start by recalling the general formula of the CV estimator, where the resampling scheme (which determines the type of CV procedure) is encoded by the random weights W = {W e } e∈En-p .

R W p (A, D n ) = e∈En-p W e e ∈En-p W e L P D ē (A (D e )) .
(1.16)

After a careful look at the proof of Theorem 1.1, it clearly arises that the same strategy can be applied to the above general formula, leading to

R W p (A, D n ) = n n -p + 1 -1 v∈En-p+1 n n-p+1 p e ∈En-p W e i∈v W v\{i} γ A D v\{i} ; Z i .
This suggests the following candidate kernel with m = n -p + 1

h W m,v (D v ) = n m p e ∈En-p W e i∈v W v\{i} γ A D v\{i} ; Z i .
(1.17)

Trying now to reproduce the same reasoning as with the exhaustive CV procedure leads to make a few comments for comparing Eq. (1.17) to the kernel previously derived for the LpO estimator from Eq. (1.15). Firstly, the candidate kernel depends on the resampling weights W and the set of indices v as emphasized by the notation. This entails that this candidate kernel varies along the different possible sets of indices v unlike that of Eq. (1.15). This is in line with the framework explored for instance by [START_REF] Adamczak | Moment inequalities for u-statistics[END_REF]; De La Pena and Montgomery-Smith (1993); [START_REF] Giné | Exponential and moment inequalities for U -statistics[END_REF]; [START_REF] Houdré | Exponential inequalities, with constants, for U-statistics of order two[END_REF] where the kernel of the U-statistic is allowed to depend on the indices of the samples it is computed from. Secondly, another important consequence of the previous remark is that most of quantities h W m,v (D v ) are no longer the L1O estimators associated with the learning algorithm A computed from D v . Since we consider non-exhaustive CV procedures here, there exists v ∈ E m such that W v\{i} = 0 for some i ∈ v.

Incomplete U-statistics Another possible connection with U-statistics can be made from Eq. (1.16) by referring to the notion of incomplete U-statistics.

These have been introduced by [START_REF] Blom | Some properties of incomplete u-statistics[END_REF] to remedy the computational burden induced by considering all n m subsamples in the definition of U-statistics of order m. The theoretical properties of incomplete U-statistics have been studied for instance by [START_REF] Janson | The asymptotic distributions of incomplete u-statistics[END_REF]; [START_REF] Lee | On incomplete u-statistics having minimum variance[END_REF]; [START_REF] Weber | Incomplete degenerate u-statistics[END_REF].

With the same notation as in the definition of (complete) U-statistics (Definition 1.1), incomplete Ustatistics of order m ≤ n and kernel h are defined from a collection

B = {e b ∈ E m | 1 ≤ b ≤ B} of B n m
sets of distinct indices. Then, the associated incomplete U-statistic is defined by

U n (h) = 1 B σ∈Πn(B) h Z σ(1) , . . . , Z σ(m) , (1.18)
where Π n (B) denotes the subset of all permutations of {1, . . . , n} such that

B = {(σ(1), . . . , σ(m)) | σ ∈ Π n (B)}.
Using the weights {W e } e∈En-p as in Section 1.2.2 (see also [START_REF] Janson | The asymptotic distributions of incomplete u-statistics[END_REF]), this can be rephrased as

U n (h) = 1 B v∈B W v h (D v ) .
Note that this is somewhat similar to

R W p (A, D n ) = 1 p( e ∈En-p W e ) e∈En-p i∈ē W e γ A D v\{i} ; Z i , except (at least) that (Z 1 , . . . , Z m ) → γ (A (Z 1 , . . . , Z m-1 ) ; Z m )
is not symmetric in its arguments unlike h in the main part of the literature on incomplete U-statistics.

Chapter 2

Efficient computation of the cross-validation estimator

Resampling-based procedures such as cross-validation (CV) are a versatile tool to quantify the statistical precision of very different statistical algorithms (Section 1.1.2). However as illustrated by Section 1.2.2, applying such procedures turns out to be time consuming and can become even prohibited with large scale datasets. Reducing the computational burden induced by CV is therefore a great challenge.

In what follows we describe two different strategies aiming at saving the computational resource. The first one relies on deriving closed-form formulas for the CV estimator in different contexts. This reduces the computation cost without deteriorating the statistical precision. The second approach exploits the idea of replacing the CV estimator by an approximation for which a closed-form formula is available. Unlike the previous one, this is achieved at the price of a loss of statistical accuracy that has to be quantified.

Closed-form formulas

In all what follows, the main focus is given to LpO since the resulting estimator has the same bias as with other CV-procedures but enjoys the lowest variance. The purpose of the present section is to provide a general overview on strategies leading to closed-form formulas of the LpO estimator. We also detail a few examples to illustrate some slight variations leading to such closed-form formulas with more challenging loss functions or learning algorithms. These can serve as starting points to further derivations.

General principle

Let us start by introducing some new notations which turn out to be useful in deriving new closed-form expressions for the LpO estimator given by Eq. (1.11)

R ECV p (A, D n ) = n p -1 e∈En-p 1 p i∈ē γ (A (D e ) ; Z i ) . (2.1)
An important remark is that Eq. (2.1) can be interpreted as an expectation over the n p subsets in E n-p randomly drawn from a uniform distribution. Therefore let us introduce the random vector S = (S 1 , . . . , S n ) ∈ {0, 1}

n (respectively S ∈ {0, 1} n ) such that each S i ∈ {0, 1} is a Bernoulli random variable with parameter equal to the probability of randomly choosing a subset e containing i, that is

P S [ S i = 1 ] =
R ECV p (A, D n ) = 1 p n i=1 E S Si γ (A (D S ) ; Z i ) ,
where E S [ • ] is the expectation with respect to S, and

D S = {Z i | S i = 1, 1 ≤ i ≤ n}.
Conditioning on the event Si = 1 that is of probability p/n, it results

R ECV p (A, D n ) = 1 n n i=1 E S γ (A (D S ) ; Z i ) | Si = 1 . (2.2)
At this stage the latter expression leads to two slightly different approaches that are each further explored in the following sections:

• Use the specific properties of the "simple" function γ (•; •), say the squared-loss, to derive closed-form expressions by exploiting the linearity of the expectation with respect to S (Section 2.1.2),

• With contrast functions γ (•; •) that are more difficult to handle, exhibit (when possible) a finite number

Q i of values γ i q 1≤q≤Qi for each 1 ≤ i ≤ n, such that {γ (A (D e ; Z i )) | e ∈ E n-p } = γ i q | 1 ≤ q ≤ Q i .
Note that one necessary condition for the resulting closed-form formula to be tractable is that

i Q i n p . This leads to R ECV p (A, D n ) = 1 n n i=1 Qi q=1 γ i q P S γ (A (D S ) ; Z i ) = γ i q | Si = 1 , (2.3)
which is further exemplified in Section 2.1.3.

Probability distribution with respect to S: Simple examples

In what follows we distinguish several statistical problems that differ from one another either by the involved loss function or by the considered estimators for which a dedicated strategy has been developed.

Density estimation

Let us start by illustrating the derivation of a closed-form formula for the LpO estimator in the density estimation framework. Z 1 , . . . , Z n ∈ Z i.i.d.

∼ P and f = dP/dµ, where µ denotes the Lebesgue measure on Z and the unknown density f is assumed to belong to L 2 (Z).

• The case Z = [0, 1] is addressed with projection estimators [START_REF] Tsybakov | Introduction à l'estimation non-paramétrique[END_REF] given by

∀z ∈ [0, 1], A (D n ; z) = 1 n n j=1 λ∈Λ ϕ λ (z)ϕ λ (Z i ) , (2.4)
where {ϕ λ } λ∈Λ is an orthonormal family of L 2 ([0, 1]),

• The case Z = R is addressed with kernel density estimators [START_REF] Rosenblatt | Remarks on some non-parametric estimates of a density function[END_REF]) defined for every bandwidth h > 0 by

∀z ∈ R, A (D n ; z) = 1 n n j=1 K h (Z j -z) , (2.5)
where

K h (•) = 1/hK(•/h) and K(•) is a Parzen-Rosenblatt kernel.
We recall that the quadratic contrast associated with the squared loss is defined, for any candidate density t, by γ (t;

Z) = t 2 2 -2t(Z), where t 2 2 = Z t 2 (z) dµ(z).

Contrast expression

From the notation of Section 2.1.1, it first arises that these two estimators can be expressed in a unified way using a n × n matrix M = {M i,j } 1≤i,j≤n such that, for every 1 ≤ i ≤ n with Si = 1,

A (D S ; Z i ) = M i • S.
(2.6)

Each M i = (M i,1 , . . . , M i,n
) ∈ R n is the ith row vector of the matrix M , which depends on the underlying estimator, and S = (S 1 , . . . , S n ) T ∈ R n . Second, let us also emphasize that

A (D S ) 2 2 = Z (A (D S ; z)) 2 dµ(z) = j1,j2 S j1 S j2 Φ j1,j2 ,
where Φ j1,j2 depends on the underlying estimator. Then it results a (simple) quadratic expression with respect to S, that is

γ (A (D S ) ; Z i ) = A (D S ) 2 2 -2M i • S = j1,j2 S j1 S j2 Φ j1,j2 -2M i • S.
Expectation of the contrast with respect to S The expectation with respect to S in Eq. (2.2) becomes

E S γ (A (D S ) ; Z i ) | Si = 1 = E S   j1,j2 S j1 S j2 Φ j1,j2 -2M i • S | Si = 1   = j1 =i,j2 =i Φ j1,j2 E S S j1 S j2 | Si = 1 -2 j =i M i,j E S S j | Si = 1 .
The closed-form formulas of the LpO estimator are provided without any proof since they straightforwardly follow from the next technical result.

Lemma 2.1. With the notation introduced in Section 2.1.1, for every 1 ≤ i ≤ n, it comes

∀j = i, P S S j = 1 | Si = 1 = n-2 p-1 n-1 p-1 = n -p n -1 , ∀j 1 = j 2 ∈ {1, . . . , n} \ {i} , P S S j1 = 1, S j2 = 1 | Si = 1 = n-3 p-1 n-1 p-1 = (n -p)(n -p -1) (n -1)(n -2) •
Projection estimators For projection estimators on [0, 1], we immediately deduce Proposition 2.1 (Proposition 2.1 from Celisse (2014a)). Let {ϕ λ } λ∈Λ be an orthonormal family of L 2 ([0, 1]). For any sample D n = (Z 1 , . . . , Z n ) of independent random variables with density f , let A(D n , •) be the projection estimator of f defined by Eq. (2.4). Then for every

1 ≤ p ≤ n -1, the LpO estimator R ECV p (A, D n ) is given by R ECV p (A, D n ) = 1 n(n -p) λ∈Λ   n j=1 ϕ 2 λ (Z j ) - n -p + 1 n -1 1≤j = ≤n ϕ λ (Z j )ϕ λ (Z )   .
Examples of this formula applied to histograms, trigonometric polynomials, and Haar basis wavelets can be found in Celisse (2014a). Note also that the time complexity for computing this formula is of order O(Card(Λ) • n), which makes the LpO procedure fully achievable in the present situation.

Parzen-Rosenblatt kernel estimators For kernel density estimators on R, one gets a similar result.

CHAPTER 2. EFFICIENT COMPUTATION OF THE CROSS-VALIDATION ESTIMATOR

Proposition 2.2 (Celisse (2008) Proposition 3.3.1). Let K(•) ≥ 0 denote a symmetric kernel defined on R [START_REF] Parzen | On estimation of a probability density and mode[END_REF]. For any sample D n = (Z 1 , . . . , Z n ) of independent random variables with density f and any bandwidth h > 0, let A(D n , •) be the kernel density estimator of f defined by Eq. (2.5). Then for every

1 ≤ p ≤ n -1, the LpO estimator R ECV p (A, D n ) is given by R ECV p (A, D n ) = 1 n -p K h 2 2 + 1≤j =l≤n n -p -1 n(n -1)(n -p) (K h * K h ) (Z j -Z ) - 2 n(n -1) K h (Z j -Z ) ,
where K h (•) = 1/hK(•/h) and * denotes the convolution product between functions.

An example of this formula when applied with the Gaussian kernel can be found in (Celisse and Robin, 2008, Lemma 2.3). Note also that similar results for histograms and kernel density estimators has been derived by [START_REF] Rudemo | Empirical Choice of Histograms and Kernel Density Estimators[END_REF] in the particular case of p = 1, which coincides with the L1O procedure.

Regression

The present section is concerned with describing two derivation techniques applied in the regression context with different estimators. The first one (with projection estimators) is similar to what has been done in density estimation, whereas the second one (with nearest neighbors estimators) deserves a specific treatment.

Projection estimators with the squared loss Let us now consider the particular regression context where

Z i = (X i , Y i ) ∈ X × Y ⊂ [0, 1] × R and X i = i/n for i = 1, . . . , n. Furthermore for every 1 ≤ i ≤ n, Z i = f (X i ) + ε i , (2.7)
where ε 1 , . . . , ε n are assumed to be i.i.d., with E [ ε i ] = 0 and E ε 2 i < +∞. In this context we consider the projection estimator defined from a family of orthonormal vectors

{ϕ λ } λ∈Λ of L 2 ([0, 1]) by ∀x ∈ [0, 1], A(D n ; x) = λ∈Λ ϕ λ (x)   1 n n j=1 Y j ϕ λ (X j )   = n j=1 Y j 1 n λ∈Λ ϕ λ (x)ϕ λ (X j ) .
(2.8)

We refer interested readers to [START_REF] Tsybakov | Introduction à l'estimation non-paramétrique[END_REF], Section 1.7) for a more detailed description of projection estimators in the present context with examples, and to [START_REF] Genovese | Confidence sets for nonparametric wavelet regression[END_REF] for a specific application to wavelets. Let us also mention the clear connection between this expression and that of projection estimators in the density estimation framework given by Eq. (2.4). This is the reason why, with the contrast function γ (t; (x, y)) = (t(x) -y)

2 , we get similar results to those previously derived in the density estimation framework. More precisely, we first notice that there exists a n × n matrix M such that, for every

1 ≤ i ≤ n, A(D S ; X i ) = n j=1 Y j M i,j S j = M i • S,
where M i = (M i,1 , . . . , M i,n ) ∈ R n is a column vector corresponding to the ith row of the matrix M and S = (S 1 , . . . , S n ) T ∈ R n . Second, Eq. (2.2) leads to

R ECV p (A, D n ) = 1 n n i=1 Y 2 i -2Y i E S A (D S ; X i ) | Si = 1 + E S (A (D S ; X i )) 2 | Si = 1 = 1 n n i=1   Y 2 i -2Y i j =i Y j M i,j E S S j | Si = 1 + j1 =i,j2 =i Y j1 M i,j1 Y j2 M i,j2 E S S j1 S j2 | Si = 1   .
Then, Lemma 2.1 allows to derive the following closed-form formula.

Proposition 2.3 (Celisse (2008) Proposition 3.3.2). For any sample D n = (Z 1 , . . . , Z n ) from the model described in Eq. (2.7) where f denotes the regression function to be estimated, let A(D n , •) denote the projection estimator given by Eq. (2.8). Then for every

1 ≤ p ≤ n -1, the LpO estimator R ECV p (A, D n ) is given by R ECV p (A, D n ) = 1 n n i=1 Y 2 i + n -p n(n -1) i =j (Y j M i,j ) 2 -2Y i Y j M i,j + (n -p)(n -p -1) n(n -1)(n -2) i =j = Y j M i,j Y M i, ,
where M i,j = (n -p) -1 λ∈Λ ϕ λ (X i )ϕ λ (X j ), and the last sum is computed over all 3-tuples (i, j, ) such that i = j, i = , and j = .

The k-nearest neighbor predictor with the squared loss In the present statistical framework,

Z i = (X i , Y i ) ∈ X × Y ⊂ R d × R. Furthermore Z 1 , . . . , Z n are i.i.d.
with the same probability distribution P as Z = (X, Y ). The regression function to be estimated is given by f

(x) = E [ Y | X = x ],
for almost every x ∈ X . Here we also consider the quadratic contrast function given by γ (t; (x, y)) = (t(x) -y)

2 . We now tackle the problem of deriving a closed-form formula for the LpO estimator associated with the k-nearest neighbor (kNN) algorithm described in Section 1.1.2 among local averaging estimators. For every 1 ≤ i ≤ n, let σ i denote the (random) permutation of {1, . . . , n} such that

X σi(1) -X i ≤ X σi(2) -X i ≤ . . . ≤ X σi(n-1) -X i ,
(2.9)

for a given norm • in R d . Let us mention that the following reasoning will remain unchanged with any tie-breaking strategy.

Main idea For every 1

≤ k ≤ n -1, the kNN estimator of f satisfies A (D n ; X i ) = 1 k n j=1 Y j 1 {j∈V k (Xi)} , (2.10) where V k (X i ) = ∈ {1, . . . , n} | X -X i ≤ X σi(k) -X i .
Remark 2.1. Note that Eq. (2.10) has not the same structure as that of the previous projection estimator.

It still holds true that there exists a n × n matrix M such that, for any i such that S i = 0,

A (D S ; X i ) = j =i Y j M i,j S j .
But the main difference is that this matrix M = M S depends on S since the neighborhood of X i is computed from the training set points encoded by S.

We will overcome this main difference by using a conditioning argument that has been successfully applied in the classification framework by [START_REF] Mary-Huard | Exact cross-validation for knn: application to passive and active learning in classification[END_REF]. It relies on introducing an additional random variable R S k (X i ), which is equal to the rank in the whole sample of the kth nearest neighbor of X i in D S . For instance, let us assume p = 1 in Eq. (2.2), which mean that we remove only one point from the sample at each split. If one further assume that this point is removed from the k nearest neighbors of X i in the whole sample, then R S k (X i ) = k + 1.

Derivation We start with a similar expression to that of projection estimators

R ECV p (A, D n ) = 1 n n i=1   Y 2 i -2Y i j =i Y j E S M S i,j S j | Si = 1 + j1 =i,j2 =i Y j1 Y j2 E S M S i,j1 M S i,j2 S j1 S j2 | Si = 1   .
(2.11)

Let us illustrate the strategy by successively addressing each of the two terms depending on S.

• Left-most term:

Starting by introducing the random variable V S k (X i ), which denotes the set of indices of the k nearest neighbors of X i in D S (with X i ∈ D S ), it comes

j =i Y j E S M S i,j S j | Si = 1 = 1 k n-1 j=1 Y σi(j) E S 1 {σ i (j) ∈V S k (Xi)} S σi(j) | Si = 1 ,
where σ i is defined in Eq. (2.9) and does not depend on S. Further conditioning with respect to the event

R S k (X i ) = for k ≤ ≤ k + p -1 yields j =i Y j E S M S i,j S j | Si = 1 = k+p-1 =k 1 k n-1 j=1 Y σi(j) E S 1 {σi(j)∈V S k (Xi)} S σi(j) | Si = 1, R S k (X i ) = • P S R S k (X i ) = | Si = 1 = k+p-1 =k 1 k j=1 Y σi(j) E S S σi(j) | Si = 1, R S k (X i ) = • P S R S k (X i ) = | Si = 1 , since for any j ≤ , S σi(j) = 1 = σ i (j) ∈ V S k (X i ) given R S k (X i ) = .
Finally, one gets

j =i Y j E S M S i,j S j | Si = 1 = k+p-1 =k 1 k   k -1 -1 -1 j=1 Y σi(j) + Y σi( )   • n -p n -1 P [ H ( -1, n -2, p -1) = -k ] (2.12)
from applying the following technical lemma:

Lemma 2.2. With the same notation as above, for every

1 ≤ i ≤ n, it comes ∀k ≤ ≤ k + p -1, P S R S k (X i ) = | Si = 1 = n -p n -1 P [ H ( -1, n -2, p -1) = -k ] , P S S σi(j) = 1 | R S k (X i ) = , Si = 1 = 1, if j = , = k -1 -1 , if 1 ≤ j ≤ -1.
where H (d, N, p) denotes a hypergeometric variable such that

P [ H (d, N, p) = x ] = d x N -d p-x / N p .
• Right-most term:

This term is split into two parts as follows

j1 =i,j2 =i Y j1 Y j2 E S M S i,j1 M S i,j2 S j1 S j2 | Si = 1 = j =i Y 2 j E S M S i,j 2 S j | Si = 1 + 1 k 2 1≤j1 =j2≤n-1 Y σi(j1) Y σi(j2) E S 1 {σi(j1)∈V S k (Xi)} 1 {σi(j2)∈V S k (Xi)} S σi(j1) S σi(j2) | Si = 1 .
The first one is dealt with in the same way as the previous one, which leads to

j =i Y 2 j E S M S i,j 2 S j | Si = 1 = k+p-1 =k 1 k 2   k -1 -1 -1 j=1 Y 2 σi(j) + Y 2 σi( )   • n -p n -1 P [ H ( -1, n -2, p -1) = -k ] .
(2.13)

The second one is more tedious, but can be addressed with the same reasoning and using Lemma 2.3. With the same notation as above, for every 1 ≤ i ≤ n, it comes for every k ≤ ≤ k + p -1, that

P S S σi(j1) = 1, S σi(j2) = 1 | R S k (X i ) = , Si = 1 = k -1 -1 , if ∈ {j 1 , j 2 } , = (k -1)(k -2) ( -1)( -2) , otherwise.
All of this provides us with the closed-form expression of the LpO estimator for the kNN algorithm in regression.

Proposition 2.4. For any sample D n = (Z 1 , . . . , Z n ) of i.i.d.random variables from P and any integer 1 ≤ k ≤ n -1, let A(D n , •) denote the k nearest neighbors (kNN) estimator given by Eq. (2.10). Then for every

1 ≤ p ≤ n -k, the LpO estimator R ECV p (A, D n ) is given by R ECV p (A, D n ) = 1 n n i=1 Y 2 i - 2 n n i=1 Y i   k+p-1 =k 1 k   k -1 -1 -1 j=1 Y σi(j) + Y σi( )   • P S R S k (X i ) = | Si = 1   + 1 n n i=1   k+p-1 =k 1 k 2   k -1 -1 -1 j=1 Y 2 σi(j) + Y 2 σi( ) + 2 k -1 -1 1≤j1≤ -1 Y σi(j1) Y σi( ) + (k -1)(k -2) ( -1)( -2) 1≤j1 =j2≤ -1 Y σi(j1) Y σi(j2)   • P S R S k (X i ) = | Si = 1   ,
where for every 1 ≤ i ≤ n, σ i is defined by Eq. (2.9) and P S R S k (X i ) = | Si = 1 is explicitly computed from Lemma 2.2.

Probability distribution with respect to S: Difficult examples

The key properties exploited in the previous Section 2.1.2 are the linearity of the LpO estimator with respect to the random variables S i ∈ {0, 1} (for 1 ≤ i ≤ n) combined with the (simple) quadratic loss. However in many situations, the derivation is clearly more complicated. For instance, the choice of a highly non-linear loss function such as the {0, 1}-loss in binary classification or the log-loss in density estimation prevents us from applying the same approach.

Nevertheless it arises that we are still able to derive closed-form formulas for the LpO estimator in much more difficult contexts (at the price of some restrictions). The main ingredients to derive tractable closed-form formulas for the LpO estimator are:

• Calculate the (discrete) probability distribution of γ(A(D S ); Z i ) = ϕ(S) as a function of the random vector S (with S i = 0),

• The support of this discrete distribution is finite and its cardinality remains "computationally reasonable" (at least n p ).

Density estimation with the log-loss

Let us come back to the density estimation framework described at the beginning of Section 2.1.2. The main difference here lies in the use of the log -loss defined by the following contrast γ(t; Z) = -log [ t(Z) ], where t denotes any candidate density function. Before precisely describing the type of estimators we consider, let us start our derivation with the first steps, which will justify our requirements on the estimators leading to closed-form formulas.

Firstly with either projection or kernel density estimators (see Eq. (2.6)), Eq. (2.3) immediately provides

R ECV p (A, D n ) = 1 n n i=1 Qi q=1 γ i q P S -log [ A (D S ; Z i ) ] = γ i q | Si = 1 = 1 n n i=1 Qi q=1 γ i q P S A (D S ; Z i ) = exp -γ i q | Si = 1 = 1 n n i=1 Qi q=1 γ i q P S M i • S = exp -γ i q | Si = 1 , (2.14)
where, for every 1 ≤ i ≤ n, γ i q 1≤q≤Qi denotes the finite support of the distribution of the random variable γ (A (D S ) ; Z i ) = M i • S as a function of S (see Eq. (2.6)), and Q i its cardinality.

Secondly for positivity reasons of the M i s, we will restrict ourselves to histograms among projection estimators and to kernel density estimators.

Thirdly Eq. (2.14) clearly highlights that Q i has to be reasonably small for the formula to remain tractable. With a kernel density estimator, allowing the kernel to continuously depend on all the data (that is, having M i = (M i,1 , . . . , M i,n ) with all its coordinates distinct) unavoidably leads to Q i = n p almost surely. This suggests that keeping the computation cost under control requires to restrict ourselves to kernels that only depends on a finite number of points for each i, i.e. kernels that are compactly supported such as

K(z) = 1 [-1,1] (z)/2 for instance.
Remark 2.2. Note that other candidate density estimators could be considered with the log -loss such as the exponential families of piecewise polynomials described in [START_REF] Castellan | Density estimation via exponential model selection[END_REF] for instance.

Another interesting feature arising from Eq. (2.14) is the tight connection between the computation time (roughly driven by Q i ) and the number of distinct values of the coordinates of M i ∈ R n , which is related to the smoothness of the estimator.

Let us further mention that the requirements on the kernels can be relaxed if we are willing to allow any controlled approximation to the exact LpO estimator. For instance gathering close values among γ i q 1≤q≤Qi for every 1 ≤ i ≤ n would induce a small approximation error, but could considerably reduce Q i and the computation time (see also Section 2.2). This leads us to the following formula established with γ(t; Z) = -log [ t(Z) ].

Proposition 2.5. Let I = (I λ ) λ∈Λ be a partition of [0, 1] such that {ϕ λ } λ∈Λ denotes an orthonormal family of L 2 ([0, 1]) with ϕ λ = 1 I λ / |I λ |, |I λ | is the Lebesgue measure of I λ , and n λ = Card (I λ ∩ {Z 1 , . . . , Z n }). For any sample D n = (Z 1 , . . . , Z n ) of independent random variables with density f , let A(D n , •) be the corresponding histogram estimator of f . Then for every

1 ≤ p ≤ n -1, the LpO estimator R ECV p (A, D n ) is given by R ECV p (A, D n ) = 1 n n i=1 λ∈Λ 1 I λ (Z i ) n λ -1 =0 -log |I λ | P [ H(n λ -1, n -1, n -p) = ] , with R ECV p (A, D n ) = +∞, if there exists λ ∈ Λ such that P [ H(n λ -1, n -1, n -p) = 0 ] = 0.
A similar result is then available for the kernel

K(z) = 1 [-1,1] (z)/2.
Proposition 2.6. For any sample D n = (Z 1 , . . . , Z n ) of independent random variables with density f and any bandwidth h > 0, let A(D n , •) be the kernel density estimator of f defined by Eq. (2.5) with

K(z) = 1 [-1,1] (z)/2, for every z ∈ R. For every 1 ≤ i ≤ n and h > 0, set N i h = Card {1 ≤ j ≤ n | |Z j -Z i | ≤ h}. Then for every 1 ≤ p ≤ n -1, the LpO estimator R ECV p (A, D n ) satisfies R ECV p (A, D n ) = 1 n n i=1 N i h -1 =0 -log 2(n -p)h P H(N i h -1, n -1, n -p) = , with R ECV p (A, D n ) = +∞, if there exists 1 ≤ i ≤ n such that P H(N i h -1, n -1, n -p) = 0 = 0.
In the specific case where p = 1, formulas given by Propositions 2.5 and 2.6 have already been derived by [START_REF] Rudemo | Empirical Choice of Histograms and Kernel Density Estimators[END_REF].

Regression with the L 2 -loss

With the notation introduced in Section 2.1.1, let us consider:

• The regressogram estimator given, for every 1 ≤ i ≤ n such that Si = 1, by

A (D S ; Z i ) = j =i S j Y j M S i,j , with M S i,j = λ∈Λ 1 I λ (X j ) n =1 S 1 I λ (X ) 1 I λ (X i ),
(2.15)

• The kernel estimator given, for every 1 ≤ i ≤ n, by

A (D S ; Z i ) = j =i S j Y j M S i,j , with M S i,j = K h (X i -X j ) n =1 S K h (X i -X ) • (2.16)
The main difference with the previous situations is that these estimators both depend on S at their denominators.

In line with the previous comments following Eq. (2.14) and Remark 2.2, deriving tractable closed-form formulas of the LpO estimator depends on the support of the probability distribution of the denominator. For instance with K(x) = 1 [-1,1] (x)/2, this (finite) support is likely to have a small cardinality. More precisely with the same arguments as in the derivations of Propositions 2.5 and 2.6, the denominators in Eq. (2.15) and (2.16) have a hypergeometric distribution with known parameters, which allows us to prove the desired results, namely Theorem 1 in Arlot and Celisse (2011a) and Corollary 3.3.2 in [START_REF] Celisse | Model Selection Via Cross-Validation in Density Estimation, Regression and Change-Points Detection[END_REF].

Classification with the {0, 1}-loss and the kNN classifier

The present statistical framework is that of binary classification where

Z i = (X i , Y i ) ∈ X ×Y ⊂ R d ×{0, 1}.
Furthermore Z 1 , . . . , Z n are i.i.d. with the same probability distribution P as Z = (X, Y ). Here we consider the {0, 1}-loss given by γ (t; (x, y)) = 1 t(x) =y = (t(x) -y)

2 .

Our purpose is to derive a closed-form formula for the LpO estimator associated with the k-nearest neighbor (kNN) classifier, which is defined for every 1 ≤ k ≤ n -1 (with the same notation as in Section 2.1.2) by

∀x ∈ X , A (D n ; x) = 1, if n j=1 Y j 1 {j∈V k (x)} ≥ k/2,
(2.17) = 0, otherwise, where V k (x) = ∈ {1, . . . , n} | X -x ≤ X σx(k) -x , and σ x denotes the permutation of {1, . . . , n} such that

X σx(1) -x ≤ X σx(2) -x ≤ . . . ≤ X σx(n) -x .
Derivation Coming back to Eq. ( 2.3), it turns out that using the {0, 1}-loss simply leads to

Q i = 1 = γ i q , hence R ECV p (A, D n ) = 1 n n i=1 P S A (D S ; X i ) = Y i | Si = 1 .
Then further conditioning by the event

R S k (X i ) = for any k ≤ ≤ k + p -1, it results R ECV p (A, D n ) = 1 n n i=1 k+p-1 =k P S A (D S ; X i ) = Y i | R S k (X i ) = , Si = 1 P S R S k (X i ) = | Si = 1 .
• Since the last probability P S R S k (X i ) = | Si = 1 has been calculated in Lemma 2.2, it only remains to deal with

P S A (D S ; X i ) = Y i | R S k (X i ) = , Si = 1 .
• Let us then notice that

P S A (D S ; X i ) = Y i | R S k (X i ) = , Si = 1 = 1 {Yi=1} + P S A (D S ; X i ) = 1 | R S k (X i ) = , Si = 1 1 {Yi=0} -1 {Yi=1} = 1 {Yi=1} + P S   j=1 Y σi(j) S σi(j) ≥ k/2 | R S k (X i ) = , Si = 1   1 {Yi=0} -1 {Yi=1} = 1 {Yi=1} + P S   -1 j=1 Y σi(j) S σi(j) ≥ k/2 -Y σi( ) | R S k (X i ) = , Si = 1   1 {Yi=0} -1 {Yi=1} ,
where the permutation σ i (•) satisfies Ineq. (2.9), and the last equality results from the fact that given R S k (X i ) = , Y σi( ) belongs to the training sample almost surely.

• The conclusion comes from further noticing that the conditional probability distribution of

-1 j=1 Y σi(j) S σi(j) is the hypergeometric H -1 j=1 Y σi(j) , -1, k -1 .
Gathering all these calculations provides the following closed-form formula.

Proposition 2.7 [START_REF] Mary-Huard | Exact cross-validation for knn: application to passive and active learning in classification[END_REF], Proposition 1 and Section 2.3). For any sample D n = (Z 1 , . . . , Z n ) of i.i.d.random variables from P and any integer 1 ≤ k ≤ n -1, let A(D n , •) denote the k nearest neighbors (kNN) classifier given by Eq. (2.17). Then for every

1 ≤ p ≤ n -k, the LpO estimator R ECV p (A, D n ) is equal to R ECV p (A, D n ) = 1 n n i=1 1 {Yi=1} k+p-1 =k P   H   -1 j=1 Y σi(j) , -1, k -1   < k/2 -Y σi( )   P S R S k (X i ) = | Si = 1 + 1 n n i=1 1 {Yi=0} k+p-1 =k P   H   -1 j=1 Y σi(j) , -1, k -1   ≥ k/2 -Y σi( )   P S R S k (X i ) = | Si = 1 , with P S R S k (X i ) = | Si = 1 = n-p n-1 P [ H ( -1, n -2, p -1) = -k ] (see Lemma 2.
2). The closed-form expression given by Proposition 2.7 requires computing the k + p nearest neighbors of the n points in the sample. The induced time complexity is O (n • (nd + (k + p))) by using strategies like the so-called quickselect algorithm [START_REF] Martínez | Optimal sampling strategies in quicksort and quickselect[END_REF] where X i ∈ R d . This remains computationally reasonable since k + p ≤ n.

Let us mention that this expression holds with the classical kNN classifier where all the k nearest neighbors receive an equal weight 1/k. However alternative weights have been explored in the literature [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Cannings | Local nearest neighbour classification with applications to semi-supervised learning[END_REF]. For these more general estimators, similar closed-form expressions have been derived by [START_REF] Mary-Huard | Exact cross-validation for knn: application to passive and active learning in classification[END_REF], and independently by [START_REF] Steele | Exact bootstrap k-nearest neighbor learners[END_REF] for the bootstrap estimator.

Efficient computation of the CV estimator

In the previous Section 2.1, several exact closed-form formulas have been derived for the LpO estimator in various statistical settings. However Sections 2.1.1 and 2.1.3 have highlighted some limitations of the approaches aiming at deriving exact closed-form formulas. In particular Remark 2.2 explains that the computation time depends on the number of distinct coordinates of the vector M i ∈ R n , which is related to the smoothness of the estimator under consideration. In other words, focusing exclusively on exact formulas dramatically reduces the range of estimators for which such closed-form expressions do exist.

This leads us to two main remarks. On the one hand, since the available computational resources remain limited while the amount of available data keeps growing, an essential problem is to derive fast-tocompute CV estimators. Providing algorithmically efficient numerical algorithms for their evaluations is a crucial problem. On the other hand, one could allow for some approximation to the true CV estimator (at the price of a controlled additional statistical error) provided that the surrogate estimator could be efficiently computed.

Each of these remarks has been explored in the literature and is briefly exposed in the following sections. In particular the last one gives rise to a trade-off between the computation time (and more generally the available computational resources) and the statistical accuracy of the designed procedures.

Fast exact computations of the CV estimator

CV procedures are commonly used to calibrate the regularization parameter of numerous statistical algorithms such as Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]van de Geer and Lederer, 2013b), SVM (Schölkopf et al., 2004;Steinwart and Christmann, 2008a), Kernel Fisher discriminant analysis [START_REF] Scholkopft | Fisher discriminant analysis with kernels[END_REF], and Ridge regression [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF][START_REF] Solnon | Multi-task regression using minimal penalties[END_REF] to name but a few.

Coming back to Eq. (1.10)

R W p (A, D n ) = e∈En-p W e e ∈En-p W e L P D ē (A (D e )) ,
it appears that the high computational cost induced by CV is mainly due to recomputing Card (E n-p ) times the estimator A (D e ) for each training sample D e . Therefore this drawback would be (at least partially) overcome if one could express A (D e ) (that is costly to recompute too many times) in terms of A (D n ) (that is computed only once from all the available data) in such a way that the new resulting formula would be faster to compute.

Speed up the L1O procedure The above idea of linking A (D e ) with A (D n ) has been mainly exploited in the literature to derive new faster-to-compute formulas of the L1O estimator in several contexts. The main steps are briefly exposed in what follows.

• The first step consists in identifying typical situations where A (D n ) has a closed-form formula . One important remark is that numerous estimators A (D n ) can be computed by solving a linear system of equations [START_REF] Suykens | Least squares support vector machines[END_REF], which results in a closed-form expression. For instance, [START_REF] Cawley | Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers[END_REF] (with Kernel Fisher discriminant analysis) and [START_REF] Cawley | Fast exact leave-one-out cross-validation of sparse least-squares support vector machines[END_REF] (with least-squares SVM) show that

A(D n ; X) = R + H T H -1 H T t,
where R denotes a constant (n + 1) × (n + 1) matrix, H is a n × (n + 1) matrix computed from D n , and t ∈ R n is a known vector independent of D n .

• The second step relies on noticing that removing the ith row of H, leads to

H T (i) H (i) = H T H -h i h T i ,
where h i ∈ R n+1 denotes the column vector corresponding to the ith row of H. It then results that, for every 1 ≤ i ≤ n, (2.18) which remains problematic in terms of computation time. Indeed since the computation of such an inverse has a complexity O(n 3 ) in time, repeating this n times along the L1O procedure would be computationally too expensive.

A(τ i (D n ); X) = R + H T H -h i h T i -1 H T (i) t,
• The last step aims at removing the dependence of the inverse with respect to h i , which exploits the classical Sherman-Woodbury-Morrison formula Lemma 2.4 [START_REF] Henderson | On deriving the inverse of a sum of matrices[END_REF]). For any invertible d × d matrix A and two column vectors u, v ∈ R d , it comes

A + uv T -1 = A -1 - A -1 uv T A -1 1 + v T A -1 u • Therefore, it results for every 1 ≤ i ≤ n A(τ i (D n ); X i ) = R + H T H -1 H T (i) t + R + H T H -1 h i h T i R + H T H -1 1 -h T i (R + H T H) -1 h i H T (i) t.
(2.19) Unlike Eq. (2.18), the merit of Eq. (2.19) is that h i arises outside the inverse, which considerably reduces the computational burden.

Speed up other CV procedures More recently some efforts have been done to extend such improved numerical algorithms for more general CV procedures. For instance, [START_REF] An | Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression[END_REF] derived similar formulas based on the same reasoning and a slightly more general version of the the Sherman-Woodbury-Morrison formula. This improves on the naive computation of the V-FCV, RLT, and MCCV applied to the least-squares SVM and kernel ridge regression, which is a particular instance of the latter [START_REF] Suykens | Least squares support vector machines[END_REF]. Importantly, this strategy is applied to CV procedures where the cardinality p of the test set is allowed to vary (V-FCV with V = n/p for instance). Consequently these exact formulas result in procedures for which the computation time also depends on p. For instance, [START_REF] An | Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression[END_REF] notice the computation time of V-FCV decreases as V grows, whereas it would be the contrary with a naive implementation.

Remark 2.3. These approaches suffer two main limitations. They first heavily rely on closed-form formulas for the initial estimator A(D n ), which excludes numerous practical situations such as the density estimation with the log-loss and a parametric mixture of Gaussian density estimators.

Second, Eq. (2.19) does not allow to remove the summation over all the considered subsamples. Since the same problem occurs when considering other Cv procedures like V-FCV, this strategy cannot by applied with the LpO procedure.

Let us finally mention that several recent attempts have been made to speed-up CV procedures (and in particular V-FCV) [START_REF] Hubert | Fast cross-validation of high-breakdown resampling methods for pca[END_REF][START_REF] Joulani | Fast cross-validation for incremental learning[END_REF][START_REF] Krueger | Fast cross-validation via sequential testing[END_REF].

Fast approximations to the CV estimator

The present section goes one step further since it allows for approximating the CV estimator provided the resulting estimator is faster to compute than the true one. This has two main assets:

• It deals with estimators for which no closed-form formulas are available, which is a strong improvement upon the previous (exact) approach.

• Depending on the type of approximation we use, it can result in tractable formulas for the LpO estimator.

In what follows, we review the main ideas leading to connecting the estimator computed from the whole sample D n to that one computed from D e up to an approximation.

V-FCV and L1O

The idea of approximating the CV estimator to reduce its computation time is not new. For instance, [START_REF] Craven | Smoothing noisy data with spline functions[END_REF]; [START_REF] Wahba | Practical approximate solutions to linear operator equations when the data are noisy[END_REF] have introduced the Generalized CV (GCV) procedure to provide an approximation to the L1O estimator.

More recently [START_REF] Meijer | Efficient approximate k-fold and leave-one-out cross-validation for ridge regression[END_REF] designed a new approach (illustrated in the Generalized Linear Model (GLM) and the Cox' proportional hazards model) to provide an approximation to the V-FCV estimator that is faster to compute than the true one. It is based on a Taylor expansion of the score function combined with the use of the exact inverse matrix formula given by Lemma 2.4 [START_REF] Henderson | On deriving the inverse of a sum of matrices[END_REF].

• First, the approximation provided in [START_REF] Meijer | Efficient approximate k-fold and leave-one-out cross-validation for ridge regression[END_REF] results from an asymptotic Taylor expansion of the score function ˙

λ θ (D n ) = ∂ λ θ (Dn) ∂ θ |θ =θ
associated with the regularized log-likelihood,

λ θ (D n ) = θ (D n ) -λ 2 θ T Aθ, where λ > 0, θ ∈ R d , and A ∈ S + d (R), that is for any e ∈ E n-p , ˙ λ θ (D e ) = ˙ λ A λ (Dn) (D e ) + ∂ ˙ λ θ (D e ) ∂θ |θ =A λ (Dn) (θ -A λ (D n )) + o ((θ -A λ (D n ))) ,
where A λ (D n ) = argmin θ∈Θ λ θ (D n ) for every λ > 0. This leads to an approximation of the estimator computed from D e that is, (2.20) where the remainder terms have been neglected.

A λ (D e ) ≈ A λ (D n ) - ∂ ˙ λ θ (D e ) ∂θ |θ =A λ (Dn) -1 ˙ λ A λ (Dn) (D e ),
• Second, they perform an exact calculation relying on a variant of the Sherman-Woodbury-Morrison formula [START_REF] Henderson | On deriving the inverse of a sum of matrices[END_REF], which allows them to remove the dependence of the inverse with respect to e in Eq. (2.20).

Let us notice that the expression of the resulting estimator given by Eq. (2.20) still requires to recompute V times some quantities once plugged into the CV estimator. It cannot be applied with the LpO procedure (at least for p > 1), whereas it leads to closed-form expressions with p = 1. From a theoretical perspective [START_REF] Meijer | Efficient approximate k-fold and leave-one-out cross-validation for ridge regression[END_REF] do not provide any grounded quantification of the additional error incurred by the approximated CV estimator. They only carried out a comparison between the true and approximate CV estimators in an empirical study on true data.

LpO In the context of maximum likelihood estimation, [START_REF] Vidoni | Estimating the kullback-liebler risk based on multifold cross-validation[END_REF] derived asymptotic closed-form formulas for the LpO estimator in the Gaussian linear regression model (Vidoni, 2015, see Section 4). These formulas result from the same type of approach as the one of [START_REF] Meijer | Efficient approximate k-fold and leave-one-out cross-validation for ridge regression[END_REF]. Nevertheless [START_REF] Vidoni | Estimating the kullback-liebler risk based on multifold cross-validation[END_REF] addresses the case of LpO by exploiting an additional asymptotic approximation to the inverse in Eq. (2.20) to get a linearized estimator (Vidoni, 2015, Proposition 2.1) at the price of additional remainder terms

A (D e ) = A (D n ) - ∂ ˙ θ (D n ) ∂θ |θ =A(Dn) -1 ˙ A(Dn) (D ē) + o(n δ-1 ), (2.21)
where ˙ θ denotes the usual score function evaluated at θ ∈ Θ, A (D n ) is the maximum likelihood estimator computed from D n , and 0 ≤ δ < 1 is a constant such that p = O(n δ ) as n → +∞ by assumption. Unlike Eq. (2.20), the above equation depends on the test sample data D ē within the last score function ˙ A(Dn) (D ē) and no longer within the inverse. Therefore the same techniques as those earlier exposed in Section 2.1 provide closed-form formulas for the LpO estimator by exploiting the additivity of this score (under the independence assumption). Importantly, such formulas are available provided the considered contrast function enjoys some desirable properties allowing to apply approaches described in Sections 2.1.2 and 2.1.3.

Let us finally emphasize that Eq. (2.21) is derived without any precise (non asymptotic) quantification of the error induced by the approximation. Such a finite-sample performance quantification would be necessary in view of a model selection purpose.

Chapter 3

Risk estimation

Bias

Analyzing the bias of CV enables to minimize or to correct this bias; alternatively, when some bias is needed, such an analysis allows to tune the bias of CV as desired.

Theoretical assessment of the bias

From Proposition 1.1, it comes for every algorithm A, any sampling scheme encoded by W , and any

1 ≤ p ≤ n -1, that E R W p (A, D n ) = E Dn R ECV p (A, D n ) = E Dn-p [ L Z∼P (A (D n-p )) ] ,
where D n-p denotes a training sample of cardinality n -p, and Z ∼ P denotes a random variable independent of D n-p . Therefore, the expectation of the CV estimator of the risk only depends on n -p:

E R W p (A, D n ) = E Dn-p [ L Z∼P (A (D n-p )) ] .
This leads to the bias of the CV estimator of the risk of A(D n ), given by

Bias(A, n, n -p) = E Dn-p [ L Z∼P (A (D n-p )) ] -E Dn [ L Z∼P (A (D n )) ] , (3.1)
which is the difference between the risks of A respectively trained with n -p and with n observations. Since n -p < n , the bias of CV is usually nonnegative and tends to decrease when n -p increases. For instance this holds true when the risk of A(D n ) is a decreasing function of n , that is, when A is a smart rule. Note however that a classical algorithm such as 1-nearest-neighbour in classification is not smart (Devroye et al., 1996, Section 6.8). More precisely, (3.1) has led to several results on the bias of CV, which can be split into three main categories: asymptotic results (A is fixed and the sample size n tends to infinity), non-asymptotic results (where A is allowed to make use of a number of parameters growing with n), and empirical results. They are organized below by statistical framework.

Density estimation. The general behaviour of the bias of CV (positive, decreasing with n -p) is confirmed by several papers. Non-asymptotic expressions for the bias of LpO estimators for kernel and projection estimators with the quadratic risk were proved by [START_REF] Robin | Nonparametric density estimation by exact leave-p-out cross-validation[END_REF] and by Celisse (2014a). More precisely with projection estimators, the following proposition can be proved.

Proposition 3.1 (Corollary 2.4 in Celisse (2014a)). Let us consider the density estimation framework with the quadratic loss where the density f ∈ L 2 ([0, 1]), and let A be the learning algorithm leading to projection estimators built from an orthonormal family of L 2 ([0, 1]) denoted by {ϕ λ } λ∈Λ . Then for any 1 ≤ p ≤ n -1, the bias of the LpO estimator is given by

Bias(A, n, n -p) = p n(n -p) λ∈Λ Var [ ϕ λ (X 1 ) ] ≥ 0.
The (nonnegative) bias increases with p, which leads to the conclusion that L1O is the least biased risk estimator among CV procedures in the present context.

Asymptotic expansions of the bias of the L1O estimator for histograms and kernel estimators were previously derived by [START_REF] Rudemo | Empirical Choice of Histograms and Kernel Density Estimators[END_REF]; see [START_REF] Bowman | An alternative method of cross-validation for the smoothing of density estimates[END_REF] for simulations. [START_REF] Hall | On Kullback-Leibler loss and density estimation[END_REF] provided similar results with the log-likelihood contrast for kernel estimators by relating the performance of L1O to the interaction between the kernel and the tails of the target density.

Regression shows a similar picture. For LpO, non-asymptotic expressions of the bias were proved by [START_REF] Celisse | Model Selection Via Cross-Validation in Density Estimation, Regression and Change-Points Detection[END_REF] for projection and kernel estimators, and by Arlot and Celisse (2011a) for regressograms when the design is fixed. More recently [START_REF] Vidoni | Estimating the kullback-liebler risk based on multifold cross-validation[END_REF] has derived an asymptotic quantification of the bias of the LpO estimator with maximum likelihood estimators. For V-FCV and RLT, an asymptotic expansion of the bias was yielded by [START_REF] Burman | A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods[END_REF] for least squares in linear regression, and extended to spline smoothing [START_REF] Burman | Estimation of optimal transformations using v-fold cross validation and repeated learning-testing methods[END_REF]. Note that [START_REF] Efron | How biased is the apparent error rate of a prediction rule?[END_REF] proved non-asymptotic analytic expressions of the expectations of the L1O and GCV estimators in regression with binary data (see also [START_REF] Efron | Estimating the error rate of a prediction rule: improvement on cross-validation[END_REF].

Classification. For discriminating between two populations with shifted distributions, [START_REF] Davison | On the bias and variability of bootstrap and cross-validation estimates of error rate in discrimination problems[END_REF] compared the asymptotical bias of L1O and bootstrap. L1O is less biased when the shift size is n -1/2 : As n tends to infinity, the bias of L1O stays of order n -1 , whereas that of bootstrap worsens to the order n -1/2 . On synthetic and real data, [START_REF] Molinaro | Prediction error estimation: a comparison of resampling methods[END_REF] compared the bias of L1O, V-FCV and .632+ bootstrap: The bias decreases with n -p , and is generally minimal for L1O. Nevertheless, the 10-fold CV bias is nearly minimal uniformly over their experiments. Furthermore, .632+ bootstrap exhibits the smallest bias for moderate sample sizes and small signal-to-noise ratios, but a much larger bias otherwise. In binary classification, [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF] has derived an upper bound on the bias of the LpO estimator for the k-nearest neighbor classification rule (for 1 ≤ k ≤ n -1) with the {0, 1}-loss.

CV-calibrated algorithms. When a family of algorithms (A λ ) λ∈Λ is given, and λ is chosen by minimizing [START_REF] Stone | Cross-validatory choice and assessment of statistical predictions[END_REF] for the L1O, and [START_REF] Jonathan | On the use of cross-validation to assess performance in multivariate prediction[END_REF] for V-FCV). This bias is of different nature compared to the previous frameworks. Indeed, R W p (A λ , D n ) is biased for the same reason as the empirical contrast L P Dn (A(D n )) suffers some optimism as an estimator of the loss of A(D n ) . Estimating the risk of A λ (D n ) with CV can be done by considering the full algorithm A : D n → A λ(Dn) (D n ) , and then computing R W p (A ; D n ) . This procedure is illustrated in the seminal paper by [START_REF] Stone | Cross-validatory choice and assessment of statistical predictions[END_REF], Section 2, Examples III and V).

R W p (A λ ; D n ) over λ , R W p (A λ ; D n ) is biased for estimating the risk of A λ (D n ) (see

Bias correction

An alternative to choosing the CV estimator with the smallest bias is to correct this bias. [START_REF] Burman | A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods[END_REF][START_REF] Burman | Estimation of optimal transformations using v-fold cross validation and repeated learning-testing methods[END_REF] proposed a corrected V-FCV estimator

R corrF CV p (A; D n ) = R F CV p (A; D n ) + L P Dn (A(D n )) - 1 V V j=1 L P Dn A D ēj ,
where (ē 1 , . . . , ēV ) denotes a partition of {1, . . . , n} in V = n/p blocks of cardinality ≈ p, and D ēj is the sample data with indices in ēj . A similar correction holds for RLT. Both estimators have been proved to be asymptotically unbiased for least squares in linear regression. When the ēj s have the same cardinality p, the corrected V-FCV criterion is equal to the sum of the empirical contrast and the V -fold penalty [START_REF] Arlot | V -fold cross-validation improved: V -fold penalization[END_REF], defined by

penVF(A; D n ) = V -1 V V j=1 L P Dn A D ēj -L P De j A D ēj .
The V -fold penalized criterion was proved by [START_REF] Arlot | V -fold cross-validation improved: V -fold penalization[END_REF] to be (almost) unbiased in the non-asymptotic framework for regressograms. Further non-asymptotic oracle inequalities have been proved for such biascorrected resampling penalties by [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] in the density estimation context. Note also that other bias corrections have been proposed. For instance [START_REF] Davies | Cross validation model selection criteria for linear regression based on the Kullback-Leibler discrepancy[END_REF] derived a modified unbiased L1O estimator with the log-likelihood contrast.

Bias and stability

Stability of learning algorithms

The notion of stability has first been introduced by [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF] and further studied for instance by [START_REF] Kearns | Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation[END_REF] and [START_REF] Bousquet | Stability and generalization[END_REF]. This concept has emerged as an effective measure of the "smoothness" of a learning algorithm with respect to its input data. For an introduction to stability and connections with other topics such as reproducibility, see Yu (2013). Over the past decades, the use of stability to derive generalization bounds has received much attention in the statistical and machine learning communities. Existing results rely upon stability assumptions such as the hypothesis or uniform stability.

Main notions of stability in the literature. For the sake of completeness we now recall two basic notions of stability that will help us to justify the introduction of the new L q stability. Definition 3.1 (Hypothesis stability, [START_REF] Bousquet | Stability and generalization[END_REF], Definition 3). With the above notation, a learning algorithm has hypothesis stability β > 0 if

∀1 ≤ j ≤ n, E Dn,Z∼P [ |γ (A(D n ); Z) -γ (A(τ j (D n )); Z)| ] ≤ β,
where τ j (D n ) = (Z 1 , . . . , Z j-1 , Z j+1 , . . . , Z n ) denotes the sample D n where Z j has been removed.

For instance hypothesis stability holds true for the kNN binary classifier with the {0, 1}-loss (see Lemma 3.1 and also [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF], Ineq. (5.1)). This notion of stability is mainly used to derive polynomials upper bounds on the moments of the L1O estimator (see [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF], Eq. ( 7)) and (Bousquet and Elisseeff, 2002, Section 4.1)).

Since we are rather interested by exponential bounds (instead of polynomial ones), a stronger notion of stability has been introduced to this end. Definition 3.2 (Uniform stability, [START_REF] Bousquet | Stability and generalization[END_REF], Definition 6). With the above notation, a learning algorithm A has uniform stability β > 0 if

∀1 ≤ j ≤ n, sup Dn γ (A(D n ); •) -γ (A(τ j (D n )); •) ∞ ≤ β,
where τ j (D n ) = (Z 1 , . . . , Z j-1 , Z j+1 , . . . , Z n ) denotes the sample D n where Z j has been removed.

The notion of uniform stability is strong since it implies the (weaker) notion of hypothesis stability. In particular, (Bousquet and Elisseeff, 2002, Section 4.2) give several examples of uniform stable learning algorithms for which they derive PAC exponential generalization bounds on the L1O estimator. However these bounds are established under strong boundedness assumptions. From a more general point of view, uniform stability turns out to be somewhat restrictive as emphasized by (Kutin and Niyogi, 2002, Section 3.1).

Further insightful analyses of various notions of stability can be found in [START_REF] Kutin | Almost-everywhere algorithmic stability and generalization error[END_REF], [START_REF] Evgeniou | Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers[END_REF], [START_REF] Elisseeff | Stability of randomized learning algorithms[END_REF], [START_REF] Rakhlin | Stability results in learning theory[END_REF], [START_REF] Mukherjee | Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization[END_REF][START_REF] Shalev-Shwartz | Learnability, stability and uniform convergence[END_REF], [START_REF] Kale | Cross-validation and mean-square stability[END_REF], [START_REF] Kumar | Near-optimal bounds for cross-validation via loss stability[END_REF] and [START_REF] Villa | On learnability, complexity and stability[END_REF] to name but a few.

New notion of L q stability. Let us now introduce a new notion of L q stability [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF]. It is mainly motivated by the need to bridge the gap between the weak notion of hypothesis stability and its strong counterpart of uniform stability, respectively used to derive polynomial and exponential concentration inequalities. Definition 3.3 (L q stability, Definition 1 in [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF]). With the same notation as above, for any q ≥ 1, A is said β-L q stable if there exists β > 0 such that

∀ 1 ≤ j ≤ n, S q (A, n) = (E [ |γ (A(D n ); Z) -γ (A(τ j (D n )); Z)| q ]) 1/q ≤ β,
where the expectation is computed over D n and Z ∼ P , with Z independent of D n , and

τ j (D n ) = (Z 1 , . . . , Z j-1 , Z j+1 , . . . , Z n )
is the sample D n where Z j has been removed.

If q = 1, one recovers the hypothesis stability, also called L 1 stability in the literature. Note that uniform stability clearly implies L q stability for all q ≥ 1 simultaneously. For instance, the Ridge regression algorithm has been proved to be L q stable by [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF], Theorem 1) under weaker assumptions than the ones in (Bousquet and Elisseeff, 2002, Example 3). This new notion of stability turns out to be useful to derive exponential concentration inequalities (see Section 4.2.3).

Connection between stability and the bias of CV estimators

As illustrated by the above definitions ( Definitions 3.1, 3.2, and3.3), stability is usually related to the variation incurred by the learning algorithm A when removing one input point. Therefore the bias of the L1O estimator (given by (3.1)) can be straightforwardly related to the hypothesis stability with two successive uses of the Jensen inequality.

|Bias(A, n, n -1)| ≤ E [ |L P (A(D n-1 )) -L P (A(D n ))| ] ≤ E Dn,Z∼P [ |γ (A(D n ); Z) -γ (A(D n-1 ); Z)| ] ≤ β,
which implies that any β-hypothesis stable learning algorithm (Definition 3.1) has a bias which is upper bounded by β > 0.

However the bias of the LpO estimator R W p (A) is rather concerned with the variation incurred by A when removing 1 ≤ p ≤ n -1 input points from D n . A first naive idea would consist in upper bounding the bias by deriving successive upper bounds on

|A(D n-i ) -A(D n-i+1 )| for 1 ≤ i ≤ n -1.
Unfortunately, this only leads to a crude upper bound. Actually, quantifying this bias can sometimes be made more precisely from a direct calculation in the same way as [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF] for the kNN binary classifier with the {0, 1}-loss.

Lemma 3.1 [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF], Eq. ( 14)). For every 1 ≤ k ≤ n, let A k denote kNN classification algorithm, and let Z 1 , . . . , Z n denote n i.i.d.random variables such that for every

1 ≤ i ≤ n, Z i = (X i , Y i ) ∼ P , with Y i ∈ {0, 1}. Then for every 1 ≤ p ≤ n -k, E 1 A k (Dn;X) =Y -1 A k (Dn-p;X) =Y ≤ P [ A k (D n ; X) = A k (D n-p ; X) ] ≤ 4 √ 2π p √ k n ,
where (X, Y ) ∼ P is independent of D n .

This upper bound remains meaningful (smaller than 1) as long as p √ k ≤ n. In particular it does not handle the case of large values of p that is, p "close to" n.

Variance

With training sets of the same size n -p , CV estimators have the same bias, but still behave differently. Their variance Var R W p (A; D n ) captures most of the information to explain these differences.

Variability factors

Assume that Card(e j ) = n -p for every j . The variance of CV results from the combination of several factors, in particular the splitting ratio (n -p : p) and B .

Influence of the splitting ratio (n -p : p) . Let us consider the Hold-p-out estimator R HO p (A; D n ) of the risk. For a given split of {1, . . . , n} into training and test sets D e and D ē, [START_REF] Nadeau | Inference for the generalization error[END_REF] emphasize that Var R HO p

(A; D n ) = Var L P D ē (A(D e )) = E De Var Dē L P D ē (A(D e )) + Var Dn-p [ L P (A(D n-p )) ] = 1 p E Dn-p [ Var Z∼P (γ (A(D n-p ); Z)) ] + Var Dn-p [ L P (A(D n-p )) ] . (3.2)
Assuming n -p is fixed, the first term is proportional to 1/p . Therefore, more data for validation decreases the variance of R HO p , because it yields a better estimator of L P (A(D n-p )). Both terms show that the variance of R HO p also depends on the distribution of L P (A(D n-p )) around its expectation; which strongly depends on the stability of A .

Stability and variance.

General comments. When A is unstable, R ECV 1 (A) has often been pointed out as a variable estimator (Section 7.10, [START_REF] Breiman | Heuristics of instability and stabilization in model selection[END_REF][START_REF] Hastie | The elements of statistical learning[END_REF]. Conversely, [START_REF] Molinaro | Prediction error estimation: a comparison of resampling methods[END_REF] noticed, from a simulation experiment, that this trend disappears when A is stable. The relation between the stability of A and second order moments of R ECV 1 (A) was stressed by [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF] in classification. Note also that various techniques have been proposed for reducing the variance of R ECV 1 (A) (see Section 4.3.3 in Arlot and Celisse (2010)).

More recently, [START_REF] Kale | Cross-validation and mean-square stability[END_REF] have quantified the variance reduction allowed by using V-FCV instead of Hold-p-out and the link with the so-called notion of mean square stability. Consistently with the above observations, their result corroborates the intuition that this variance reduction is stronger when the learning algorithm is more stable.

Upper bounding the variance and stability From the link between the LpO estimator R ECV p and U-statistics earlier stated in Theorem 1.1, [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF]; [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF] have developed a new strategy allowing to upper bound the variance of the LpO estimator in terms of the stability of A, measured by means of the L q stability. This strategy follows several steps that we briefly recall in what follows.

• First step: It consists in upper bounding the variance of the LpO estimator in terms of that of L1O. Proposition 3.2 (Theorem 2.2 in Celisse and Mary-Huard ( 2015)). For any symmetric learning algorithm A and every 1 ≤ p ≤ n -1 such that the following quantities are well defined,

Var R ECV p (A, D n ) ≤ n n -p + 1 -1 Var R ECV 1 (A, D n-p+1 ) .
(3.3)

Note that the multiplicative factor in the right-hand side of Eq. (3.3) is equal to 1 as long as p ≤ n/2 + 1, which means that the above upper bound improves as p > n/2 grows. Let us also mention that one recovers the same order as in the usual upper bound on the variance of a U-statistic.

• Second step: It relies on a moment inequality derived from (Boucheron et al., 2013a) and repeated uses of the Minkowski inequality combined with Definition 3.3 for q = 2 to get a bound on the variance of the L1O estimator in terms of L 2 stability (see for instance Proposition 4.2).

• Third step: Combining the two previous steps leads to a bound on the variance of the LpO estimator in terms of the L 2 stability of A. Typical instances of resulting upper bounds on the variance are provided in [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF] for the Ridge regression algorithm with the quadratic loss, and in Celisse and Mary-Huard (2015) (via a similar but more refined approach) for the kNN binary classifier with the {0, 1}-loss.

Partial splitting and variance of the LpO estimator. When the splitting ratio (n -p : p) is fixed, the variance of CV is larger for partial data splitting methods as stated in Proposition 1.1. Choosing B < n p subsets (e j ) 1≤j≤B of {1, . . . , n} , usually randomly, induces an additional variability compared to exhaustive procedures such as LpO. The variability due to the choice of the data splits is maximal for hold-out, and minimal (null) for exhaustive splitting schemes like L1O (if p = 1) and LpO. With MCCV, this variability decreases like B -1 since the e j are chosen independently. The dependence on B is different for other CV estimators, such as RLT or V-FCV, because the randomly chosen e j s are not independent (see Section 1.2.3 for more details).

Note that the dependence of Var( R F CV p (A)) on V is more complex to evaluate, since B , n -p , and p simultaneously vary with V . Nevertheless, a non-asymptotic theoretical quantification of this additional variability of V-FCV has been obtained by [START_REF] Robin | Nonparametric density estimation by exact leave-p-out cross-validation[END_REF] in the density estimation framework (see also empirical considerations by [START_REF] Jonathan | On the use of cross-validation to assess performance in multivariate prediction[END_REF]. In the sequel we provide an extension of this non-asymptotic quantification to general splitting schemes encoded by the weight vector W .

From the simple expression of the variance of R W p given by Proposition 1.1, one can further quantify the additional variance incurred by any CV procedure that differs from LpO. More precisely, it comes Proposition 3.3 (see also [START_REF] Robin | Nonparametric density estimation by exact leave-p-out cross-validation[END_REF] for V-FCV). Let us first recall that the variance of the LpO estimator can be expressed as

Var R ECP p (A) = n p -1 E Dn x 2 e -[ E Dn [ x e ] ] 2 + e =e n p -2 E Dn [ x e x e ] ,
where x e = L P D ē (A(D e )). Furthermore since the distributions of W e and x e do not depend on e for any splitting scheme encoded by W , this leads to

Var R W p (A) = n p -1 + n p Var W [ W e ] B 2 E Dn x 2 e -[ E Dn [ x e ] ] 2 + e =e n p -2 + E W [ W e W e ] -E W [ W e ] E W [ W e ] B 2 E Dn [ x e x e ] .
The influence of the splitting scheme W arises from two additional terms with respect to what we would obtain for the LpO variance.

Focusing now on the Hold-p-out, V-FCV, and RLT-p procedures, it is possible to further specify the above expression since W e ∈ {0, 1} is then a Bernoulli random variable.

Corollary 3.1. With the Hold-p-out, V-FCV or RLT-p, it results Var R W p (A) = n p -1 + n p p W (1 -p W ) B 2 E Dn x 2 e -[ E Dn [ x e ] ] 2 + e =e n p -2 + E W [ W e W e ] -p 2 W B 2 E Dn [ x e x e ]
,

where

p W = E W [ W e ] = P W [ W e = 1 ]
does not depend on e.

Note that p W and E W [ W e W e ] depend on the splitting scheme and can be computed respectively from previous Lemma 1.1 and following Lemma 3.2. Lemma 3.2. For any 1 ≤ p ≤ n -1 and e = e ∈ E n-p , and any integer B ≥ 1, it comes Hold-p-Out:

P W W HOp e = 1, W HOp e = 1 = 0, V-Fold CV: P W W F CVp e = 1, W F CVp e = 1 = 0, if ē ∩ ē = ∅ (with V = n/p ≥ 2) = V (V -1)(p!) 2 n(n -1) . . . (n -2p + 1)
, otherwise, RLT-p:

P W W RLT p e = 1, W RLT p e = 1 = B(B -1) n p n p -1 -1
.

One can also emphasize that with the V-FCV procedure for instance, the formula in Corollary 3.1 can be further simplified. First, let us notice that only pairs (e, e ) such that e = e and ē ∩ ē = ∅ enter into the sum e =e . Second, for each such pair, the distribution of W e W e does no longer depend on (e, e ). The same conclusion holds true for E Dn [ x e x e ] which does not depend on (e, e ) provided e = e and ē ∩ ē = ∅.

A possible use of the above Corollary 3.1 is for choosing the sampling scheme. One could take W and the number B of splittings such that the additional variance (compared to that of LpO) remains acceptable, that is

∀e = e , max p W (1 -p W ) B 2 , E W [ W e W e ] -p 2 W B 2 ≤ η n p -2
, for some prescribed precision parameter 0 < η < 1. This gives rise to a trade-off between the available computational resources and a prescribed precision to achieve.

By exploiting the connection raised in Section 1.2.4 between LpO and U-statistics, another possible strategy to analyze the additional variance of R W p induced by non-exhaustive splitting schemes is to exploit previous works on incomplete U-statistics such as [START_REF] Lee | On incomplete u-statistics having minimum variance[END_REF]; [START_REF] Rempala | Incomplete u-statistics of permanent design[END_REF], which provide several guidelines to choose the best possible "design" in terms of variance. Let us also mention the recent work of [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] where a precise quantification of the variance of several CV estimators have been derived in the particular setting of density estimation with the L 2 -loss.

Asymptotic assessment of the variance

Precisely understanding how Var( R W p (A)) depends on the splitting scheme is complex in general (see Section 3.2.1). For instance the number B of splits is generally linked with p (at least for instance for LpO and V-FCV). Furthermore, the variance of CV strongly depends on the statistical framework and on the stability of A as discussed in Section 3.2.1. Therefore, radically different results have been obtained in different frameworks, in particular on the value of V for which the V-FCV estimator has a minimal variance [START_REF] Burman | A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods[END_REF]Hastie et al., 2009, Section 7.10). Despite these difficulties, the variance of several CV estimators has been assessed in various frameworks, as detailed below.

Regression. In a simple linear regression setting with homoscedastic data, [START_REF] Burman | A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods[END_REF] proved an asymptotic expansion of the variance of V-FCV

Var R F CV p (A, D n ) = 2σ 2 n + 4σ 4 n 2 4 + 4 V -1 + 2 (V -1) 2 + 1 (V -1) 3 + o n -2 .
(3.4)

Asymptotically, the variance decreases with V , implying that L1O asymptotically has the minimal variance among V-FCV estimators. Similar results have been derived for RLT as well.

Non-asymptotic closed-form formulas of the variance of the LpO estimator have been proved by [START_REF] Celisse | Model Selection Via Cross-Validation in Density Estimation, Regression and Change-Points Detection[END_REF] in regression, for projection and kernel estimators. On the variance of RLT in the regression setting, see [START_REF] Girard | Asymptotic comparison of (partial) cross-validation, GCV and randomized GCV in nonparametric regression[END_REF] for Nadaraya-Watson estimators, as well as [START_REF] Nadeau | Inference for the generalization error[END_REF] for several learning algorithms.

Density estimation. Closed-form formulas of the variance of the LpO risk estimator have been proved by [START_REF] Robin | Nonparametric density estimation by exact leave-p-out cross-validation[END_REF], and by Celisse (2014a) for projection estimators, that is Proposition 3.4 (Corollary 2.5 in Celisse (2014a)). With the same notation as above, let A(D n ) denote the projection estimator built from an orthonormal family

{ϕ λ } λ∈Λ of L 2 ([0, 1]). Then for every 1 ≤ p ≤ n-1, Var R ECV p (A, D n ) = n (n -1) 2 A + B n -p + C (n -p) 2 + O 1 n ,
where the O(•) does not depend on p, and A, B, C (A, C ≥ 0) are numerical constants only depending on {ϕ λ } λ∈Λ and the true unknown density.

This has the same flavor as the above Eq. (3.4) established for V-FCV in the regression setting. Let us emphasize that the monotonicity of the variance with respect to p depends on the sign of B, that is unknown in general (see (Celisse, 2014a, Proposition 2.3) for an analysis of the monotonicity of Var R ECV p with respect to p). In particular one important conclusion is that L1O leads (at least asymptotically) to the least variable CV estimator of the risk in that context. A similar quantification is also provided by (Celisse, 2008, Proposition 3.4.2) for kernel density estimators. Recently Arlot and Lerasle (Theorem 6 in 2015) provides, for several CV procedures, a precise quantification of the variance of the (difference between) CV estimators used with projection estimators in the density estimation framework.

Classification. For discriminating between two populations with shifted distributions, [START_REF] Davison | On the bias and variability of bootstrap and cross-validation estimates of error rate in discrimination problems[END_REF] showed that the gap between asymptotic variances of L1O and bootstrap becomes larger when data are noisier. [START_REF] Nadeau | Inference for the generalization error[END_REF] made non-asymptotic computations and simulation experiments with several learning algorithms. [START_REF] Hastie | The elements of statistical learning[END_REF] empirically showed that V-FCV has a minimal variance for some 2 < V < n , whereas L1O usually has a large variance. Simulation experiments by [START_REF] Molinaro | Prediction error estimation: a comparison of resampling methods[END_REF] suggest this fact mostly depends on the stability of the considered algorithm.

Variance estimation

There is no universal-valid under all distributions-unbiased estimator of the variance of RLT [START_REF] Nadeau | Inference for the generalization error[END_REF] and V-FCV [START_REF] Bengio | No unbiased estimator of the variance of K-fold cross-validation[END_REF]. In particular, [START_REF] Bengio | No unbiased estimator of the variance of K-fold cross-validation[END_REF] recommend the use of variance estimators taking into account the correlation structure between test errors.

Despite these negative results, (biased) estimators of the variance of R W p in regression and classification have been proposed and assessed by [START_REF] Bengio | No unbiased estimator of the variance of K-fold cross-validation[END_REF]; [START_REF] Markatou | Analysis of variance of cross-validation estimators of the generalization error[END_REF]; [START_REF] Nadeau | Inference for the generalization error[END_REF]. In the particular setting where p ≥ n/2, [START_REF] Fuchs | A u-statistic estimator for the variance of resampling-based error estimators[END_REF]; [START_REF] Fuchs | Minimization and estimation of the variance of prediction errors for cross-validation designs[END_REF]; [START_REF] Wang | Variance estimation of a general u-statistic with application to cross-validation[END_REF] derived unbiased estimators of the variance of the LpO estimator by exploiting the connection between LpO and U-statistics.

In the density estimation framework, [START_REF] Robin | Nonparametric density estimation by exact leave-p-out cross-validation[END_REF] proposed an estimator of the variance of the LpO risk estimator based on closed-form formulas.

Mean squared error

Optimality results for risk estimation

When the goal is to estimate the risk of an estimator associated with a given learning algorithm, the Mean squared error (MSE) criterion often enters into play as a means to compare different procedures.

Our purpose here is to collect previous results of Sections 3.1 (bias) and 3.2 (variance) about the performance of CV procedures used for risk estimation. As a main conclusion, it arises that L1O is (asymptotically) optimal for risk estimation in terms of MSE in numerous settings that are enumerated in what follows.

In density estimation using projection or kernel density estimators, [START_REF] Robin | Nonparametric density estimation by exact leave-p-out cross-validation[END_REF] have empirically observed that L1O has the smallest MSE among CV procedures, which has been theoretically established by (Celisse, 2014a, Theorem 2.1).

In least squares linear regression, [START_REF] Burman | A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods[END_REF] stated as well that L1O has the smallest MSE among V-FCV and RLT procedures, which has been supported by results of simulation experiments. This conclusion holds true in the more general framework empirically investigated by (Zhang and Yang, 2015, Section 7.1) where the problem consists in estimating the risk of more complex learning algorithms based on model selection criteria such as AIC or BIC. According to [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF], Section 8.2), L1O still performs the best in terms of MSE when used with unstable algorithms such as Lasso. For comparison, let us mention that in this context, 10-FCV is less variable but highly more biased.

Unbiased risk estimation and model selection

Let us now briefly discuss a few fundamental differences between two statistical purposes: risk estimation and model/learning algorithm selection (see also Chapter 5 for further details).

The previous Section 3.3.1 mainly focuses on the performance risk of CV procedures used for estimating the risk of a given estimator A(D n ). By contrast an other important question is to choose the "best" estimator among a collection {A λ (D n )} λ∈Λ of candidates, which corresponds to model/learning algorithm selection.

As pointed out by (Zhang and Yang, 2015, Section 7.2), it is important to keep in mind that risk estimation and model selection are often contradictory objectives. Coming back to CV, this means that the L1O procedure (asymptotically optimal for risk estimation in numerous settings) is not the universally best CV procedure in terms of model selection. Here are some of the reasons why one should remain cautious before stating such a general claim.

On the one hand, the bias incurred by a CV procedure is not the quantity that really matters when considering model selection since models are compared via differences of corresponding CV estimators (see (Arlot and Lerasle, 2015, Section 4)). Considering the difference between two CV estimators with the same large bias, this difference can be almost unbiased. Therefore a biased (but less variable) CV procedure such as 10-FCV could perform better in terms of model selection, which is supported by the conclusion of [START_REF] Breiman | Submodel selection and evaluation in regression. the x-random case[END_REF] for instance.

On the other hand, the performance of L1O for model selection depends itself on the type of model selection purpose we pursue, that is estimation/prediction or identification (see (Arlot and Celisse, 2010, Sections 6 and7) and(Celisse, 2014a, Sections 3.1 and3.2) for more details on this difference). For instance, [START_REF] Yang | Comparing Learning Methods for Classification[END_REF] addresses the identification purpose with CV procedures and describes settings where the "CV paradox" arises. More precisely, he exhibits settings where CV procedures are optimal for identification as long as the ratio p/n increases to 1 and n -p → +∞ as n grows. Note that it clearly excludes L1O from optimal CV procedures. With model selection for estimation, (Breiman and Spector, 1992, Section 5) illustrate the better performance of 10-FCV upon L1O in variable selection (see also (Zhang and Yang, 2015, Section 7.2)). However, (Celisse, 2014a, Theorem 3.1) proved that L1O (and more generally any LpO procedure such that p/n → 0 as n → +∞) is asymptotically optimal for estimation (see also [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] for similar results applying to V-FCV).

Chapter 4

Concentration of the cross-validation estimator

The CV estimator is used with two main purposes. The first one is to estimate the performance of a learning algorithm computed from a set of observations (Chapter 3). The second one is to calibrate the value of unknown parameters such as the partition for histograms, the bandwidth for kernel estimators, the regularization parameter in the Ridge regression (Chapter 5). Deriving concentration inequalities for the CV estimator is a convenient way to characterize the behavior of CV estimators with high probability.

The purpose of the present section is to describe two main strategies used to derive such concentration inequalities for the CV estimators. The first one (Section 4.1) heavily relies on the specificity of some settings where closed-form formulas can be derived, and simple but tedious calculations can be carried out to apply well-known concentration results such as the Bernstein or Talagrand concentration inequalities for instance. The second one (Section 4.2) is more general (and still at an early stage). It is designed to analyze the behaviour of the CV estimator without assuming such closed-form formulas are available for the CV estimator (or can be exploited in the derivation). This strategy mainly relies on moment inequalities, the notion of stability, and the connection between the CV estimator and U-statistics (see Section 1.2.4).

Exploit closed-form expressions

The present section details the first strategy, which heavily relies on closed-form formulas in the particular context of density estimation with the quadratic loss (Section 2.1.2) already discussed in Sections 3.1.1 for the bias and 3.2.2 for the variance. We focus on results established for the LpO procedure by Celisse (2014a) used with projection estimators. This approach has been extended by [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] to other resampling schemes.

Link between R ECV p and easy-to-handle quantities

For the first strategy relying on closed-form formulas, we first need to state the explicit link between the LpO estimator and important quantities on which concentration results will be applied.

To this end, let {ϕ λ } λ∈Λ(τ ) and {ϕ λ } λ∈Λ(τ ) denote two finite orthonormal families of L 2 ([0, 1]), and

F τ = Vect({ϕ λ } λ∈Λ(τ ) ) (respectively F τ = Vect({ϕ λ } λ∈Λ(τ )
)) be the corresponding finite dimensional vector spaces. For any density f ∈ L 2 ([0, 1]) and any index c ∈ {τ, τ }, let

• f c = argmin t∈Fc t -f , where t = [0,1] t 2 (x)dx, • f c = A c (D n ) = λ∈Λc P n ϕ λ ϕ λ be the minimum contrast estimator built from F c .
We are now in position to state the first main result of the present section, which is a key ingredient in our approach since it relates the LpO estimator to influential quantities such as the difference between bias terms (resp. variance terms). 51 Proposition 4.1 (Proposition A.2 in Celisse (2014a)). With the above notation and for every p ∈ {1, . . . , n -1}, the difference between the LpO estimators of A τ and A τ can be written as

R ECV p (A τ , D n ) -R ECV p (A τ , D n ) = n n -p E f τ -f τ 2 -E f τ -f τ 2 + f -f τ 2 -f -f τ 2 -ρ n,p f τ -f τ 2 -E f τ -f τ 2 + ρ n,p f τ -f τ 2 -E f τ -f τ 2 -2ρ n,p ν n (f τ -f τ ) + 1 n ρ n,p + n n -p ν n (φ τ -φ τ ) ,
where

ν n (t) = 1/n n i=1 (t(Z i ) -E [ t(Z i ) ]), with Z 1 , . . . , Z n ∼ f are independent random variables, φ c = λ∈Λc ϕ 2
λ for any c ∈ {τ, τ }, and

ρ n,p = 1 + 1 n -1 + n n -p 1 n -1 • Proposition 4.
1 deals with the difference between the LpO estimators evaluated at A τ and A τ . Considering this difference (instead of only one estimator) only simplifies the description of the strategy we followed (Section 4.1.2), but does not change the conclusions. The main contribution of Proposition 4.1 is to describe the link between these LpO estimators and simple quantities that can be handled by means of classical concentration inequalities as exposed in Section 4.1.2.

A noticeable feature of the present estimation setting is that the dependence with respect to p exclusively arises through multiplicative constants. This remark is for instance at the core of (Lemma 1 in [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] where an equivalence is proved between LpO and other resampling techniques.

Exponential concentration and empirical process theory

Deriving an exponential concentration inequality for the LpO estimators from Proposition 4.1 results from the successive applications of several classical concentration inequalities from the empirical process theory [START_REF] Massart | Concentration inequalities and model selection[END_REF][START_REF] Van De Geer | Empirical Processes in M-estimation[END_REF]. The main steps of this derivation are exposed in what follows using the notations of Proposition 4.1.

Bernstein's inequality

The Bernstein inequality given by Eq. ( 4.1) provides an upper bound for

ν n (f τ -f τ ) with t = (f τ -f τ ) / f τ -f τ .
More precisely for every x, η > 0, there exists a set of probability at least 1 -2e -x on which

ν n (f τ -f τ ) ≤ η/2 f τ -f τ 2 + 2η -1 Var (t(Z 1 )) x n + η -1 t ∞ x 3n 2 . The fact that f τ -f τ 2 ≤ 2 f τ -f 2 + f τ -f
2 allows relating the resulting deviations terms to both the approximation and estimation errors (using additional assumptions). Another use of the Bernstein inequality (4.1) provides the desired control on ν n (φ τ -φ τ ) in terms of deviation terms depending on the estimation errors

E f τ -f τ 2 and E f τ -f τ 2 .
Talagrand's inequality Considering the (random) estimation error f τ -f τ

2

, it is straightforward to express it as a supremum of the (centered) empirical process over the unit ball of the vector space F τ by

f τ -f τ 2 = λ∈Λ(τ ) [ (P Dn -P ) ϕ λ ] 2 = sup t∈Fτ , t ≤1 ν n (t) .
This simple reformulation allows to successively use Eq. ( 4.2) and ( 4.3) with Z = f τ -f τ to derive lower and upper bounds with high probability, where σ 2 and b can be linked to the estimation error

E f τ -f τ 2 .
Conclusions and remarks Combining all these concentration inequalities under specific assumptions provides the desired control of the deviations of the LpO estimator uniformly over a class candidate learning algorithms (see the proofs of Theorem 3.1 and 3.4 in Celisse (2014a)).

Note that [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] have recently extended the above approach to other resampling procedures such as V-FCV and random penalties. Their results also heavily rely on closed-form formulas available in the specific setting of density estimation with projection estimators and the quadratic loss (see for instance Eq. ( 33)). The technical tools involved in the proofs only slightly differ from the above ones (see for instance Lemma 15 in Section A.2 that deals with U-statistics of order two).

Technical results

For the sake of completeness, the three classical concentration inequalities that are the main tools of the above approach have been collected here.

Theorem 4.1 (Bernstein's inequality, Ineq. (2.10) in Boucheron et al. (2013a)). Let X 1 , . . . , X n be i.i.d.random variables defined on a mesurable space (X , T ), and let t denote a mesurable bounded real valued function. Then for every x > 0,

P ν n (t) > 2Var (t(X 1 )) x n + t ∞ x 3n ≤ e -x . (4.1)
Theorem 4.2 (Bousquet's version of Talagrand's inequality [START_REF] Bousquet | A Bennett concentration inequality and its application to suprema of empirical processes[END_REF]).

Let X 1 , . . . , X n be i.i.d.random variables defined on a mesurable space (X , T ). Let S denote a set of real valued functions such that sup t∈S t ∞ ≤ b and sup t∈S Var (t(X 1 )) = σ 2 . Denoting Z = sup t∈S ν n (t), then for every x > 0

P √ nZ ≤ √ nE(Z) + 2 (σ 2 + 2bE(Z)) x + bx 3 √ n ≤ e -x . (4.2)
Theorem 4.3 (Rio's version of Talagrand's inequality [START_REF] Klein | Concentration around the mean for maxima of empirical processes[END_REF]).

Let X 1 , . . . , X n be i.i.d.random variables defined on amesurable space (X , T ). Let S denote a set of real valued functions such that sup t∈S t ∞ ≤ b and sup t∈S Var (t(X 1 )) = σ 2 . Denoting Z = sup t∈S ν n (t), then for every x > 0

P √ nZ ≤ √ nE(Z) -2 (σ 2 + 2bE(Z)) x - 8bx 3 √ n ≤ e -x . (4.3)

Interests and limitations of this approach

On the one hand, the strategy described in Sections 4.1.1 and 4.1.2 provides accurate lower and upper bounds on the LpO estimator by exploiting closed-form formulas available in the specific context of density estimation with projection estimators and the quadratic loss. We should also keep in mind that such results (derived in this specific context) can serve as benchmarks to assess the accuracy of alternative strategies which would avoid exploiting closed-form expressions such as the ones discussed in Section 4.2.

On the other hand, the approach exposed along Sections 4.1.1 and 4.1.2 is limited to settings where the CV estimator can be expressed in terms of simple quantities for which accurate concentration results can be derived. For instance, the results provided by [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF]; Celisse (2014a) only hold true for several CV procedures in the specific (and somewhat narrow?) context of density estimation with the quadratic loss. However this strategy fails most of the time since we are usually not able to make such a tight connection between the CV estimator and simple quantities. Firstly, there is usually no such closedform formula for the CV estimator. Secondly even if a closed-form formula can be derived, its expression can be too difficult to handle as it happens in the change-point detection problem (Section 6.2.2), or in density estimation with the log-loss (Section 2.6).

Without exploiting closed-form expressions

In view of the limitations enumerated in Section 4.1.3, our main motivation here is to describe an alternative strategy leading to concentration inequalities for the CV estimator around its expectation. Such inequalities can be either moment or exponential inequalities. Our two main requirements about this strategy are that:

• it should ideally apply to any CV estimator: V-FCV, RLT, LpO, . . .

• it should not rely on any closed-form expression (which would limit its applicability).

Unlike the former approach discussed in Sections 4.1.1 and 4.1.2 (based on closed-form formulas) the present one is likely to be somewhat less accurate. This is the possible price to pay for a higher generality level.

Note that van der Laan et al. ( 2004) already derived exponential concentration results for general CV procedures with the log-loss. However the resulting upper bounds are not tight since they are only decreasing in the test set cardinality p. In particular, they are non informative for the L1O estimator and more generally for any CV procedure such that p remains constant with respect to the sample size n.

General strategy

In what follows we start by providing the first steps of the general strategy leading to concentration inequalities, which apply to any CV procedure. These steps exploit the connection between CV estimators and U-statistics previously raised in Section 1.2.4.

Secondly, we focus on the LpO estimator for which we illustrate the type of moment and exponential concentration inequalities we can derive with two learning algorithms: Ridge regression and kNN binary classification.

Upper bounding moments of the non-exhaustive CV estimators

The general approach we describe here relies on the next lemma, which relates moments of the V-FCV estimator to moments of a sum of "nearly independent and identically distributed" random variables. Lemma 4.1. With the notation used to derive Eq.(1.17), let the weights W encode the V-FCV (with p = n/V ) procedure defined in Section 1.2.2. Then the corresponding estimator satisfy

R W p (A, D n ) = 1 n! σ∈Πn H W m,v Z σ(1) , . . . , Z σ(n) ,
where Π n denotes the set of all permutations of {1, . . . , n}, v = (v 1 , . . . , v r ) and r = n/m , and

H W m,v (Z 1 , . . . , Z n ) = 1/r r i=1 h W m,vi (D vi ), with h W m,v (D v ) = n m p e ∈En-p W e i∈v W v\{i} γ A D v\{i} ; Z i .
Let us emphasize that a similar (but more involved) lemma could be established for other (nonexhaustive) CV procedures by taking into account the number of different (overlapping) subsamples for instance.

Sketch of proof of Lemma 4.1. The result comes from the main following ingredient which holds true with the splitting scheme W corresponding to V-FCV.

Let (E 1 , . . . , E V ) denote the V disjoint test samples of cardinality p randomly chosen according to the V-FCV sampling scheme. Then for any v ∈ {v 1 , . . . , v r },

σ∈Πn h W m,v (D v σ ) = (n -p + 1)!(p -1)! V b=1 i∈E b h W m,E b ∪{i} D E b ∪{i} ,
where (n -p + 1)!(p -1)! is the number of permutations mapping v onto each of the sets E b ∪ {i}.

Therefore from Lemma 4.1, one derives an upper bound on the centered moments of R F CV p (A, D n ) by using Jensen's inequality, that is for every q ≥ 1,

E R F CV p (A, D n ) -E R F CV p (A, D n ) q ≤ 1 n! σ∈Πn E H W m,v Z σ(1) , . . . , Z σ(n) q = E H W m,v (Z 1 , . . . , Z n ) q , (4.4)
where

H W m,v Z σ(1) , . . . , Z σ(n) = H W m,v Z σ(1) , . . . , Z σ(n) -E H W m,v Z σ(1) , . . . , Z σ(n)
, and the expectation E [ • ] applies to all random variables W and D n . Note that the last inequality holds true since the probability distribution of

H W m,v Z σ(1) , . . . , Z σ(n) does not depend on σ ∈ Π n .
The problem now reduces to upper bounding E H W m,v (Z 1 , . . . , Z n ) q in Eq. (4.4), which immediately provides that

E H W m,v (Z 1 , . . . , Z n ) q = E 1 r r i=1 h W m,vi (D vi ) q = r -q E r i=1 h W m,vi (D vi ) q , (4.5)
where

h W m,vi (D vi ) = h W m,vi (D vi ) -E h W m,vi (D vi ) .
Remark 4.1. The main features of the right-most term in Ineq. (4.5) are that: (i) it involves a sum of r identically distributed random variables, (ii) these random variables are not independent due to W , and

(iii) conditionally to W , the random variables h W m,vi (D vi ) i=1,...,r
are independent but no longer identically distributed. These properties remain to be further explored in the future for proving concentration inequalities applying to non-exhaustive CV estimators like V-FCV.

Validity of the upper bound: Application to LpO

Let us start this section by emphasizing that Lemma 4.1 also holds true with the exhaustive LpO estimator (where all weights W v\i = 1, a.s.) as proved in [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF]. With LpO, the random

variables h ECV m,vi (D vi ) i=1,...,r
in Eq. (4.5) are independent and identically distributed, which is easier to deal with in a first step. This justifies the main focus given to the LpO procedure from now on, although many steps in what follows could be transposed to other CV procedures.

Combining the upper bound from Eq. (4.4) with Eq. ( 4.5), it results for every q ≥ 1 that

E R ECV p (A, D n ) -E R ECV p (A, D n ) q ≤ r -q E r i=1 h ECV m,vi (D vi ) q , (4.6)
where

h ECV m,vi (D vi ) = 1/(n -p + 1) i∈v γ A D v\{i} ; Z i -L P (A(D n-p )) denotes the centered L1O estimator computed from D vi .
Therefore in the case where q = 2, using independence leads to (Celisse and Mary-Huard, 2015, Theorem 2.2)

Var R ECV p (A, D n ) ≤ n n -p + 1 -1 Var R ECV 1 (A, D n-p+1 ) . (4.7)
For comparison, let us remark that the upper bound in Eq. ( 4.7) has the same order of magnitude (except the integer part) as the classical upper bound on the variance of a U-statistic [START_REF] Serfling | Approximation theorems of mathematical statistics[END_REF], Lemma A, p. 181) that is,

Var R ECV p (A, D n ) ≤ n n -p + 1 -1 Var R ECV 1 (A, D n-p+1 ) .
But in contrast to the usual story with U-statistics where the kernel of the U-statistic does not depend on n, we will be able to further specify the upper bound with respect to n and p by evaluating Var R ECV 1 (A, D n-p+1 ) as well.

Besides, n n-p+1 = 1 as long as p ≤ n/2 + 1, in which case

1 2 ≤ n n -p + 1 -1 ≤ 1.
This means that the multiplicative factor in Eq. (4.7) only comes into play (and enhances the convergence rate with respect to n and p) with large values of p (p > n/2). For smaller values of p, our line of proof only provides that the variance of the LpO estimator is "the same as" the one of the L1O estimator computed from D n-p+1 . The same remark applies to Ineq. (4.6) as well.

Upper bounding moments of the LpO estimator: Two approaches

Our goal is now to derive upper bounds on the order-q moments (q ≥ 2) for the centered LpO estimator by means of Eq. (4.6).

Since the LpO estimator is a U-statistic, one could be tempted to apply existing moment inequalities already available for U-statistics such as those derived in [START_REF] Adamczak | Moment inequalities for u-statistics[END_REF]; [START_REF] Giné | Exponential and moment inequalities for U -statistics[END_REF] for instance. However these inequalities turn out to be too crude for our purpose since they have been derived through an extensive use of decoupling arguments (de la Pena and Giné, 1999). Whereas the latter arguments lead to increase the numeric constants with "usual" fixed-order U-statistics, it considerably deteriorates the convergence rates with U-statistics where the order is allowed to vary with n, which is the case of the LpO estimator (see Section 1.2.4) seen as an infinite order U-statistics [START_REF] Frees | Infinite order u-statistics[END_REF]).

Noticing that the right-hand side of Eq. ( 4.6) is the order-q moment of a sum of i.i.d.random variables, there are at least two approaches to derive an upper bound.

• The first one exploits the concentration properties of a sum of independent random variables. In particular it provides an upper bound for E r i=1 h ECV m,vi (D vi ) q using for instance that r i=1 h ECV m,vi (D vi ) is a sub-Gaussian random variable (Boucheron et al., 2013a, Section 2.4). Here this will be illustrated on the analysis of the Ridge regression estimator for which we provide an improved version upon the former [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF].

• The second approach turns out to be useful when no tight concentration result is available for r i=1 h ECV m,vi (D vi ). Then we rather use the Rosenthal inequality [START_REF] Ibragimov | On extremal problems and best constants in moment inequalities[END_REF]

, which relates E r i=1 h ECV m,vi (D vi ) q to r i=1 E h ECV m,vi (D vi ) q and r i=1 Var h ECV m,vi (D vi
) . This will be investigated with the kNN binary classifier on the basis of the work of [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF].

Sub-Gaussian behavior and Ridge regression

The purpose of what follows is to illustrate how to derive upper bounds on the polynomial moments of the LpO estimator by means of the new L q stability notion. Far from optimal in several respects, the present derivation should be only considered as a starting point towards a fully general and meaningful strategy.

To ease the presentation, we specify the notation to the regression problem where

Z = (X, Y ) ∈ X × Y ⊂ R d × R and the quadratic cost function c(t(x), y) = (t(x) -y)
2 is used. But the following is not limited to this case. Let us finally introduce, for any real-valued random variable U ∈ L q (P) (q ≥ 1), the notation

U q := (E [ |U | q ]) 1/q ,
which will be repeatedly used along the present section.

Moments and the L q stability. Let us start by introducing a new notion of stability-the L q stability (for q ≥ 1)-that is more general than the usual hypothesis stability [START_REF] Bousquet | Stability and generalization[END_REF], Definition 3) but still weaker than the uniform stability (Bousquet and Elisseeff, 2002, Definition 6) (see also Section 3.1.3). Our purpose is to show that the order-q moments of the centered L1O estimator can be upper bounded by introducing this new L q stability notion.

Definition 4.1 (L q stability, Celisse and Guedj (2016) Definition 1). Let A denote any symmetric learning algorithm, and c(•, •) be any cost function. Then for every q ≥ 1, A is said γ q -L q stable if there exists γ q > 0 such that, for all 1 ≤ j ≤ n,

S q (A, n) = (E [ |c (A(D n , X), Y ) -c (A(τ j (D n ), X), Y )| q ]) 1/q ≤ γ q ,
where the expectation is computed over D n and (X, Y ) ∼ P , with (X, Y ) independent of D n , and

τ j (D n ) = (Z 1 , . . . , Z j-1 , Z j+1 , . . . , Z n ) is the sample D n where Z j = (X j , Y j ) has been removed.
The above Definition 4.1 depends on the cost function c (•, •) and requires to bound the variation of A induced by removing one training point. This is in accordance with earlier definitions [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF][START_REF] Bousquet | Stability and generalization[END_REF][START_REF] Evgeniou | Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers[END_REF]. However, (simultaneously) controlling high order moments provides more information on the distribution of c (A(D n , X), Y ) than simply considering hypothesis stability, that is L q stability with q = 1. Let us also mention that other notions of stability have been introduced, which replace one training point by an independent copy [START_REF] Kutin | Almost-everywhere algorithmic stability and generalization error[END_REF][START_REF] Kale | Cross-validation and mean-square stability[END_REF][START_REF] Kumar | Near-optimal bounds for cross-validation via loss stability[END_REF]. Finally uniform stability obviously implies L q stability for every q ≥ 1.

We now provide an upper bound on the order-q moments of the centered L1O estimator (LpO with p = 1) which involves the L q stability introduced by Definition 4.1 and results from Boucheron et al. (2013a, Theorem 15.5). Proposition 4.2. With the above notation, for any real q ≥ 2,

R ECV 1 (A, D) -E R ECV 1 (A, D) q ≤ 2 √ κq 2 √ nS q (A, n -1) + 1 √ n c(A(τ j (D), X j ), Y j ) -c(A(τ j (D), X j ), Y j ) q , (4.8)
where κ < 1.271 denotes a universal constant.

A proof can be found in Celisse and Guedj (2016, Lemma 3). The above upper bound relates the variation of the L1O estimator around its expectation to two complementary phenomena. The first one is the variation incurred by the learning algorithm A when removing one observation from its input points (L q stability). The second one is the variation of the estimator A(τ j (D)) when evaluated at two different points. This has the same flavour as the expression of the variance of the Hold-p-out estimator derived in Eq. (3.2).

Let us emphasize that Ineq. (4.8) is established without taking into account the interaction between the different sub-samples whereas each couple of sub-samples share n -2 points in common. This suggests that the derived bound can be further improved, for instance following ideas developed in Section 7.2.2. In the particular case of the variance of the V-FCV estimator, [START_REF] Kale | Cross-validation and mean-square stability[END_REF] have derived an upper bound involving the interaction between stability and the variance of the estimator A(D n ). More precisely it quantifies the amount of variance reduction allowed for the V-FCV estimator by using a stable learning algorithm. Such an interaction is typically a key feature to look at in our derivation.

The Ridge regression algorithm. Let us now specify the upper bound in Proposition 4.2 to the Ridge regression. From the model Y i = f (X i ) + i for every 1 ≤ i ≤ n, let us assume that f (•) can be well approximated by the linear form < •, β * > R d for some β ∈ R d . Let us recall that for any λ > 0, the Ridge estimator of β is given by

A λ (D n ) = β λ = arg min β∈R d 1 n n i=1 (Y i -< X i , β > R d ) 2 + λ |β| 2 2 = 1 n Σ + λI d -1 X T Y, (4.9)
where X denotes the n × d design matrix, Y = (Y 1 , . . . , Y n ) , and Σ = 1/n n i=1 X i X T i = 1/n • X X denotes the empirical covariance matrix with E [ X i ] = 0 for all i, and Var(X i,j ) = 1 for all 1 ≤ j ≤ d by assumption. In the sequel, the estimated regression function f λ (•) will be denoted by A λ (D n , •) (with a slight abuse of notation) and is given by

∀x ∈ X ⊂ R d , A λ (D n , x) = x, β λ R d = x β λ .
Assumptions. To ease the reading of what follows, we provide simplified results under the following (somewhat restrictive) assumptions:

• Boundedness of the X i s:

Let us assume that there exists 0 < B X < +∞ such that

∀1 ≤ i ≤ n, |X i | 2 := d j=1 X 2 i,j ≤ B X , a.s. , (XBd)
• Boundedness of the Y i s:

Let us assume that there exists 0 < B Y < +∞ such that

∀1 ≤ i ≤ n, |Y i | ≤ B Y , a.s. . (YBd)
However let us emphasize that most of the forthcoming results can be extended, at the price of additional technicalities, to the unbounded case (at least for instance to the Gaussian setting for Y ). This more general situation will raise additional terms in the upper bounds related to the weight of the distribution tails.

Ridge regression and L q stability. Let us now provide the upper bounds for each of the two terms in the right-hand side of Eq.(4.8) for the Ridge regression.

Firstly, the following result provides a quantification of the L q stability of the Ridge algorithm for any q ≥ 1. This is an improvement upon Theorem 1 in [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF] in terms of the interplay between n and λ in the upper bound. Proposition 4.3. For any n > 1 and every λ > 0, let A λ be given by Eq. (4.9) and set c(y, y ) = (y -y )

2 . Then, assuming (XBd) and (YBd) hold true, A λ is γ q -L q stable for all q ≥ 1 with

γ q = (2B X B Y ) 2 λn 1 + B 2 X λn • (1 + 2B X d λ,2 ) 2 ,
where

d λ,2 = i=1 [ s i / √ n ] 2 / [ (s i / √ n) 2 + λ ] 2 , with {s i } 1≤i≤ denoting the singular values of X and = min (d, n).
Under the (XBd) and (YBd) assumptions, we recover the same bound as the one established in (Bousquet and Elisseeff, 2002, Example 3) for the uniform stability. In particular, note that d λ,2 can be upper bounded by a finite quantity independent of λ. However let us emphasize that the present derivation (relying on moments inequalities and simultaneous L q stability) can also be extended to the unbounded setting (at least for Y ) at the price of additional terms involving q that are related to the tail probability weights (see [START_REF] Guedj | Stability revisited: new generalisation bounds for the leave-one-out[END_REF] for some clues in that direction). Sketch of proof of Proposition 4.3. For any matrix X ∈ M n,d (R), let U ∈ U n (R) and V ∈ U d (R) denote unitary matrices of respective sizes n and d, and introduce a diagonal matrix S ∈ M n,d (R) such that X = U • S • V , which is the singular value decomposition (SVD) of X.

Claims.

1. It straightforwardly arises that

1 √ n Σ -1 λ X = 1 √ n Σ + λI d -1 X = V • T λ • U , where T λ ∈ M d,n (R) is a diagonal matrix with at most = min(d, n) singular values such that [ T λ ] i,i = (s i / √ n) • (s i / √ n) 2 + λ -1 and s i = [ S ] i,i , for all 1 ≤ i ≤ . 2. The Ridge estimator A λ (D n ) = β λ is given by A λ (D n ) = Σ -1 λ 1 n n i=1 X i Y i = 1 √ n V T λ U Y ,
where Y = (Y 1 , . . . , Y n ) . Therefore its Euclidean norm satisfies

|A λ (D n )| 2 = 1 √ n T λ U Y 2 = 1 √ n i=1 [ T λ ] 2 i,i [ U Y ] 2 i ≤ 1 √ n i=1 [ T λ ] 2 i,i nB Y ≤ d λ,2 B Y , (using (YBd)) with d λ,2 = i=1 [ s i / √ n ] 2 / [ (s i / √ n) 2 + λ ] 2 . 3. The above quantity d λ,2 = i=1 [ s i / √ n ] 2 / [ (s i / √ n) 2 + λ ] 2 < +∞ is upper bounded indepen-
dently of λ since only the non-zero singular values are involved in the sum.

4. The operator norm of Σ -1 λ can be also upper bounded by

Σ -1 λ op ≤ 1 λ ,
which is a tight bound for instance in the case where d > n.

Furthermore with Σ

(n) λ denoting the estimated covariance matrix computed from (X 1 , . . . , X n-1 ), the following equality

Σ -1 λ Σ (n) λ -I d = Σ -1 λ Σ (n) λ -Σ λ = Σ -1 λ Σ λ n -1 - X j X j n -1 , yields that, for any n ≥ 2, Σ -1 λ Σ (n) λ -I d op ≤ 1 λ • 2 B 2 X n -1 ≤ 4 B 2 X nλ • (with (XBd))
5. When removing the last observation, the Ridge estimator satisfies

A λ (D n ) -A λ (D n-1 ) = Σ -1 λ X Y n - (X n ) Y n n -1 + Σ -1 λ Σ (n) λ -I d Σ (n) λ -1 (X n ) Y n n -1 ,
where X n and Y n respectively denote the random matrix X and vector Y counterparts where the nth observation has been removed. Hence,

|A λ (D n ) -A λ (D n-1 )| 2 ≤ Σ -1 λ X j Y j n 2 + Σ -1 λ op • B X B Y n + Σ -1 λ Σ (n) λ -I d op • (d λ,2 B Y ) ≤ 2 B X B Y λn + Σ -1 λ Σ (n) λ -I d op • (d λ,2 B Y ) ≤ 2 B X B Y λn + 4 B 2 X nλ • (d λ,2 B Y ) = 2 B X B Y λn • (1 + 2B X d λ,2 ) .
6. With the quadratic cost function and every random variable (X,

Y ) ∈ R d × R, it results that |c (A λ (D n , X), Y ) -c (A λ (D n-1 , X), Y )| = X (A λ (D n ) -A λ (D n-1 )) 2 + 2 X (A λ (D n ) -A λ (D n-1 )) Y -X A λ (D n ) ≤ B 2 X • |A λ (D n ) -A λ (D n-1 )| 2 2 + 2B X • |A λ (D n ) -A λ (D n-1 )| 2 • [ B Y + B X |A λ (D n )| 2 ] ,
by using (XBd) and (YBd).

7. Combining all the above results provides us with

|c (A λ (D n , X), Y ) -c (A λ (D n-1 , X), Y )| ≤ B 2 X • |A λ (D n ) -A λ (D n-1 )| 2 2 + 2B X • |A λ (D n ) -A λ (D n-1 )| 2 • [ B Y + B X |A λ (D n )| 2 ] ≤ (2B X B Y ) 2 λn 1 + B 2 X λn • (1 + 2B X d λ,2 ) 2 .
Using now the same arguments as in the above proof, let us provide an upper bound on the variation incurred by the estimator A(τ j (D n )) when evaluated at different points.

Proposition 4.4. For any n > 1 and every λ > 0, let A λ be given by Eq. (4.9) and set c(y, y ) = (y -y )

2 . Then, assuming (XBd) and (YBd) hold true, A λ satisfies for any q ≥ 1 and 1 ≤ j ≤ n, that

c (A λ (τ j (D n ), X j ), Y j ) -c A λ (τ j (D n ), X j ), Y j q ≤ 2B 2 Y (1 + B X d λ,2 ) 2 .
The desired conclusion for the L1O estimator then results from plugging Propositions 4.3 and 4.4 in the right-hand side of Proposition 4.2.

Theorem 4.4. For any n > 1 and every λ > 0, let A λ be given by Eq. (4.9) and set c(y, y ) = (y -y )

2 . Then, assuming (XBd) and (YBd) hold true, it results for any real q ≥ 2

R ECV 1 (A λ , D) -E R ECV 1 (A λ , D) q ≤ 4 √ κq • B 2 Y λ √ n (2B X ) 2 1 + B 2 X λn + λ • (1 + 2B X d λ,2 ) 2 .
(4.10)

The interplay between n and λ in Theorem 4.4 is the same as in former results from [START_REF] Bousquet | Stability and generalization[END_REF]; [START_REF] Zhang | Generalization performance of some learning problems in hilbert functional spaces[END_REF] and Mohri et al. (2012, Corollary 11.2). We also recover the same 1/(nλ 2 ) dependence as in Corollary 4.2 of [START_REF] Zhang | Leave-one-out bounds for kernel methods[END_REF] when q = 2, where it is conjectured that this dependence cannot be weakened. However, [START_REF] Blanchard | Optimal rates for regularization of statistical inverse learning problems[END_REF] recently derived an upper bound on the magnitude of E R ECV 1 (A λ , D) (see for instance their Ineq. (5.4)) suggesting that (4.10) could be further refined. For instance, this could be achieved by means of self-bounding functions and stability as suggested in Section 7.2.2.

Proof of Theorem 4.4. From Propositions 4.3 and 4.4, we get

R ECV 1 (A λ , D) -E R ECV 1 (A λ , D) q ≤ 2 √ κq 2 √ n (2B X B Y ) 2 λn 1 + B 2 X λn • (1 + 2B X d λ,2 ) 2 + 2 √ n B 2 Y (1 + B X d λ,2 ) 2 ≤ 4 √ κq B 2 Y λ √ n (2B X ) 2 1 + B 2 X λn + λ • (1 + 2B X d λ,2 ) 2 .
As a conclusion, let us assume that the LpO estimator of the performance of the Ridge estimator can be computed (which is not the case in practice). An illustration of the interest of the present strategy then results from combining Eq. (4.6) (which connects the LpO and L1O moments) with the above theorem (namely Ineq. (4.10)).

Theorem 4.5. For any n > 1 and every λ > 0, let A λ be given by Eq. (4.9) and set c(y, y ) = (y -y )

2 . Let us further assume (XBd) and (YBd) hold true, and that nλ ≤ B 2 X . Then for any real q ≥ 2, it results that

R ECV p (A λ , D n ) -E R ECV p (A λ , D n ) q ≤ 12 eΓ 2 λ (n -p + 1) n n-p+1 √ q, where Γ λ = 4 √ κ B 2 Y λ • 2 (2B X ) 2 + λ • (1 + 2B X d λ,2 ) 2 .
Theorem 4.5 (respectively Theorem 4.4) proves that the centered LpO (resp. L1O) estimator is a sub-Gaussian variable. It illustrates how interesting is the strategy exploiting the link between the LpO estimator, U-statistics, and the (for instance) sub-Gaussian behavior of the L1O estimator. Note that the assumption λn ≥ B 2 X is only made to simplify the resulting bound, but can be easily removed without any strong modification of the conclusion.

As earlier noticed, there is no closed-form formula for the LpO estimator applied to the Ridge learning algorithm. However some ongoing work seem to indicate that such closed-form expressions can be derived for a (close) approximation to the LpO estimator (see Section 7.2.1). This remark puts results such as Theorem 4.5 in a new perspective since they could serve to quantify the performance of the approximated LpO estimator.

Let us comment on the concentration rate, which is O (n -p + 1)

n n-p+1 -1 . • In the case p ≤ n/2 + 1, n n-p+1 = 1 and n -p + 1 = n (1 -(p -1)/n) ≥ n/2. Then, 1 (n -p + 1) n n-p+1 ≤ 2 n ,
meaning that the concentration rate is of the order O(1/ √ n).

• In the case p > n/2 + 1, one gets Proof of Theorem 4.5. Firstly, let us remark that applying Theorem 4.4 under assumptions (XBd) and (YBd) provides that, for any 1

1 (n -p + 1) n n-p+1 = 1 n(1 -p-1 n ) • 1 1 + (p-1)/n 1-(p-
≤ i ≤ r = n m , R ECV 1 (A λ , D vi ) -E R ECV 1 (A λ , D vi ) q ≤ Γ λ √ m √ q, with Γ λ = Γ λ = 4 √ κ B 2 Y λ • 2 (2B X ) 2 + λ • (1 + 2B X d λ,2 ) 2 .
Secondly, the inequality a a ≤ a! e a , for any integer a ≥ 1 implies that, for every q ≥ 1,

E R 1 (A, D vi ) -E R 1 (A, D vi ) 2q ≤ Γ 2 λ m q (2q) q ≤ Γ 2 λ m q (2e) q q! = 2eΓ 2 λ m q q! .
Then, Theorem 2.1 in Boucheron et al. (2013a) implies that for any 1

≤ i ≤ r, R 1 (A, D vi ) ∈ G(ν) is a sub-Gaussian random variable with ν = 8eΓ 2 λ /m. Thirdly noticing that R 1 (A, D v1 ), . . . , R 1 (A, D vr ) are i.i.d.random variables, it results r i=1 R 1 (A, D vi ) ∈ G(rν)
is a sub-Gaussian random variable with parameter rν.

The conclusion follows from applying Lemma 4.3. Then for any real q ≥ 2,

R ECV p (A, D n ) -E R ECV p (A, D n ) q ≤ n m -1 r i=1 R 1 (A, D vi ) -E R 1 (A, D vi ) q ≤ n m -1 • 3 2 n m v √ q = 3 2v n m √ q ≤ 3 2 8eΓ 2 λ (n -p + 1) n n-p+1 √ q = 12 eΓ 2 λ (n -p + 1) n n-p+1 √ q.
The Rosenthal inequality and the k nearest neighbors classifier Unlike the previous section where sub-Gaussian properties of the L1O estimator have been exploited (see the proof of Theorem 4.5), we now turn to illustrate our general strategy by means of another tool which is the Rosenthal inequality. Its main interest is that, without assuming any known concentration property of the L1O estimator, it allows to upper bound E An optimized version of the Rosenthal inequality. We use a specific version of the Rosenthal inequality [START_REF] Ibragimov | On extremal problems and best constants in moment inequalities[END_REF] established with the optimal constant and involving a "balancing factor". In particular this balancing factor allows to take into account the difference of order of magnitude between

r i=1 E h ECV m,vi (D vi ) q and r i=1 Var h ECV m,vi (D vi ) .
Optimizing the upper bound with respect to this balancing factor leads to Proposition 4.5 (Proposition D.3 in Celisse and Mary-Huard (2015)). Let X 1 , . . . , X n denote independent real-valued random variables with symmetric distributions. Then for any real q > 2,

E n i=1 X i q ≤ 2 √ 2e q    q q n i=1 E [ |X i | q ] ∨ ( √ q) q   n i=1 E [ X 2 i ]   q    .
Note that the right-hand side in Proposition 4.5 exhibits a dependence with respect to q (resulting from an optimization step) that is non-improvable with this type of inequality. Another important remark is that Proposition 4.5 applied to the Ridge regression algorithm would lead to a worse upper bound than Theorem 4.5 under the same assumptions. More precisely it would lead to conclude to the sub-gamma (instead of sub-Gaussian) behavior for the LpO estimator due to the additional q q factor. This factor can be seen as the price to pay for the higher generality level of the present approach compared to the previous one since it does not exploit any sub-Gaussian behavior of n i=1 X i .

The k nearest neighbor {0, 1}-classifier. For 1 ≤ k ≤ n, the k nearest neighbors algorithm (kNN), denoted by A k [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF], consists in classifying any new observation x using a majority vote decision rule based on the label of the k closest points X (1) (x), . . . , X (k) (x) to x among the training sample X 1 , . . . , X n ∈ R d :

A k (D n ; x) := 1 if 1 k i∈V k (x) Y i = 1 k k i=1 Y (i) (x) > 0.5 0 otherwise , (4.11)
where

V k (x) = 1 ≤ i ≤ n, X i ∈ X (1) (x), . . . , X (k) (x)
denotes the set of indices of the k nearest neighbors of x among X 1 , . . . , X n according to a given metric (the Euclidean one for instance), and Y (i) (x) is the label of the i-th neighbor of x for 1 ≤ i ≤ k. Note that the definition of the neighbors requires a tie-breaking rule as well. Since repeated uses of the Stone lemma (Lemma 4.4) are made in the present work, we choose a tie-breaking rule making the Stone's lemma work (Biau and Devroye, 2016, Lemma 10.6, p.125). Among them, we use the smallest index among ties. We refer interested readers to [START_REF] Devroye | The strong uniform consistency of nearest neighbor density estimates[END_REF]; [START_REF] Fix | Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques, chapter Discriminatory analysisnonparametric discrimination: Consistency principles[END_REF]; [START_REF] Rogers | A finite sample distribution-free performance bound for local discrimination rules[END_REF] for seminal papers on the kNN algorithm and to [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF] for an extensive presentation.

As a preliminary result, let us also mention an existing upper bound on the L 1 stability (Definition 4.1) of the {0, 1}-kNN algorithm established in [START_REF] Devroye | Distribution-free inequalities for the deleted and holdout error estimates[END_REF].

Lemma 4.2. Let A k denote the kNN classifier, and set c(t(x), y) = 1 {t(x) =y} . Then for any independent copy (X, Y ) of (X i , Y i ) for 1 ≤ i ≤ n, and any integers 1 ≤ p, k with p + k ≤ n, it comes

P [ A k (D n ; X) = A k (D n-p ; X) ] ≤ 4 √ 2π p √ k n •
In other words, Lemma 4.2 with p = 1 entails that the kNN algorithm is

L 1 stable with S 1 (A k , n) ≤ 4 √ 2π √ k
n . Note that this bound only remains meaningful as long as p √ k/n → 0 as n → +∞. Having in mind that k → ∞ with k/n → 0 are sufficient requirements to achieve the universal consistency for the kNN classifier [START_REF] Stone | Consistent nonparametric regression[END_REF], this suggests to limit the use of Lemma 4.2 to the setting where p ≤ √ k. However the dependence of the bound with respect to √ k/n cannot be improved in a distribution-free context. This results from Proposition 5.1 in [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF] where an example has been built in which a lower-bound of order √ k/n has been derived.

≤ k ≤ n-1, let A k (D τ ; •) denote the kNN classifier and R ECV 1 (A k , D τ ) be the corresponding L1O estimator (m = n -p + 1). Then • for q = 2, E R ECV 1 (A k (D τ )) -E R ECV 1 (A k (D τ )) 2 ≤ C 1 k 3/2 m ; ( 4
.12)

• for any q > 2, The dependence of the right-hand side of Ineq. (4.12) with respect to k is tighter than that of Ineq. (4.13). This difference results from the difficulty to derive a tight bound for the expectation of

E R ECV 1 (A k (D τ )) -E R ECV 1 (A k (D τ )) q ≤ (C 2 √ q) q k √ m q , ( 4
n i=1 1 A k (D i τ ;Xi) =A k (D i,j τ ;Xi)
q with q > 2, where D i τ (resp. D i,j τ ) denotes the training sample D τ where Z i has (resp. Z i and Z j have) been removed. Using the bound of Ineq. (4.13) for the variance of the L1O estimator (with q = 2) would be possible as well. But this would lead to a worse sub-Gaussian deviation. Let us also mention that the bound in (4.12) is a strict improvement upon the k 2 /n which would result from Theorem 24.4 of [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF]. However this rate is likely to be sub-optimal although any precise answer about the optimality of this rate is still an open question to the best of our knowledge.

Plugging the bounds of Proposition 4.6 in the right-hand side of the inequality given by Proposition 4.5, it results

Theorem 4.6 (Theorem 3.2 in Celisse and Mary-Huard ( 2015)). For every p, k ≥ 1 such that p (4.11). Then there exist (known) constants C 1 , C 2 > 0 such that for every 1 ≤ p ≤ n -k,

+ k ≤ n, let R ECV p (A k , D n ) denote the LpO estimator of the kNN classifier A k (D n ; •) defined by
• for q = 2, E R ECV p (A k , D n ) -E R ECV p (A k , D n ) 2 ≤ C 1 k 3/2 (n -p + 1) ; (4.14) • for every q > 2, E R ECV p (A k , D n ) -E R ECV p (A k , D n ) q ≤ C 2 k √ n-p+1 q 1/2 q , (4.15)
with

C 1 = 128κγ d √ 2π
and

C 2 = 4γ d √ 2κ
, where γ d denotes the constant arising from Stone's lemma ( Lemma 4.4). Furthermore in the particular setting where n/2 + 1 < p ≤ n -k, then

• for q = 2, E R ECV p (A k , D n ) -E R ECV p (A k , D n ) 2 ≤ C 1 k 3/2 (n -p + 1) n n-p+1 , (4.16)
• for every q > 2,

E R ECV p (A k , D n ) -E R ECV p (A k , D n ) q ≤ n n -p + 1 Γ q max    k 3/2 (n -p + 1) n n-p+1 q 1/2 , k √ n -p + 1 n n-p+1 q 3/2    q , ( 4.17)
where

Γ = 2 √ 2e max √ 2C 1 , 2C 2 .
The right-hand side of Eq. (4.16) remains (at worse) of the same O(1/n) order as that of Eq. (4.14) as p ≤ n/2 + 1. By contrast the decay rate in Eq. (4.14) strongly worsens as p grows (with p > n/2 + 1), whereas the one in Eq. (4.16) (almost) remains unchanged. This difference is even stronger if one considers the right-most term in Eq. ( 4.17) where the rate is O(1/ √ n) with p = /n/2+1, but improves up to O(1/n) with p = n -1 for instance. Let us notice that the exponents √ q and q 3/2 are directly inherited from the use of the Rosenthal inequality. Therefore they cannot be enhanced following this line of proof.

Exponential concentration of the LpO estimator

The purpose of the present section is to derive exponential concentration inequalities for the CV estimator around its expectation. In what follows we focus on the LpO estimator.

Context

Let us first mention that since the LpO estimator is a U-statistic (see Section 1.2.4), we could try to apply available exponential concentration inequalities designed for U-statistics such as (Hoeffding, 1963, Ineq. 5.7), (de la Pena and Giné, 1999, Theorem 4.1.8), and[START_REF] Arcones | A bernstein-type inequality for u-statistics and u-processes[END_REF]. The main issues with such inequalities are that: (i) they heavily rely on the assumption that the kernel is bounded, which does not allow to take advantage of specific concentration properties of the kernel itself (the L1O estimator in the present setting), and (ii) they assume the order of the kernel is independent of n, which is precisely not true with CV procedures (at least with LpO).

Another important drawback of Hoeffding's inequality for nondegenerate U-statistics is its non-optimal dependence on the variance of the fixed-order U-statistic. This has been improved by (Arcones, 1995, Theorem 2). However this improvement is achieved by means of an intensive use of decoupling arguments (de la Pena and Giné, 1999) at the price of increasing the constants by factors depending on the order of the U-statistic. Since in our setting the order increases with the sample size n, this inequality turns out to be useless for us.

Our strategy is to derive exponential concentration inequalities by exploiting polynomial moment inequalities such as those derived in previous Section 4.2.2. Actually connections are already well known between moment inequalities and exponential concentration (see for instance Theorems 2.1 and 2.3 in Boucheron et al. (2013a)). Note that controlling polynomial moments is not the only way to derive exponential concentration inequalities. Alternative strategies consist for instance in upper bounding the Laplace-transform [START_REF] Baraud | A bernstein-type inequality for suprema of random processes with applications to model selection in non-gaussian regression[END_REF] or the Orlicz norm [START_REF] Lecué | Oracle inequalities for cross-validation type procedures[END_REF]van de Geer and Lederer, 2013a) of the random variable under consideration.

Main tool

In what follows we will exploit a general result, which relates the upper bounds on the order-q moments (expressed as a polynomial in q) to the deviation terms involved in the exponential concentration inequality. Proposition 4.7 [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF], Proposition D.1 and Lemma 8.10 in [START_REF] Arlot | Resampling and Model Selection[END_REF]). Let X denote a real valued random variable, and assume there exist C ≥ 1, λ 1 , . . . , λ N > 0, and α 1 , . . . , α N > 0 (N ∈ N * ) such that for any real q ≥ q 0 , E

[ |X| q ] ≤ C N i=1 λ i q αi q
. Then for every t > 0,

P [ |X| > t ] ≤ Ce q0 minj αj e -(mini αi)e -1 minj t N λ j 1 α j . ( 4

.18)

Furthermore for every x > 0,

P |X| > N i=1 λ i ex min j α j αi ≤ Ce q0 minj αj • e -x .
(4.19) Proposition 4.7 allows to derive exponential concentration results from upper bounds on the polynomial moments of a random variable.

On the one hand, Eq. ( 4.18) provides an upper bound on the probability that X is larger than every prescribed t > 0, which is useful for instance to derive moment inequalities.

On the other hand, Eq. ( 4.19) rather makes the connection between the polynomial arising from upper bounding the order-q moment of X and the deviation terms of X for every prescribed probability level. In particular this justifies why the dependence of the upper bound on the order-q moments with respect to q was crucial when deriving Theorem 4.5 and Eq. (4.17), respectively for the Ridge and the kNN algorithms.

Application to the Ridge and kNN algorithms

First applying Proposition 4.7 to the upper bound derived in Theorem 4.5 for the Ridge regression algorithm, it results Corollary 4.1 (Ridge algorithm and quadratic loss under boundedness). With the same notation and assumptions as in Theorem 4.5, for every x > 0,

P    R ECV p (A λ , D n ) -E R ECV p (A λ , D n ) > 12eΓ λ 2x (n -p + 1) n n-p+1    ≤ e • e -x ,
where Γ λ > 0 is a known numeric constant defined in Theorem 4.5.

With the definition of Γ λ from Theorem 4.5, the resulting deviation is of order O(1/(λ √ n)) whatever 1 ≤ p ≤ n -1. Similar concentration rates have been derived with L1O for instance in [START_REF] Bousquet | Stability and generalization[END_REF]; [START_REF] Zhang | Generalization performance of some learning problems in hilbert functional spaces[END_REF][START_REF] Zhang | Leave-one-out bounds for kernel methods[END_REF] for the kernel Ridge regression when deriving exponential inequalities under similar boundedness assumptions (see also the discussion following Theorem 4.5) With the same notation and assumptions as Theorem 4.6, for any p, k ≥ 1 such that p + k ≤ n, for every

x > 0 P R ECV p (A k , D n ) -E R ECV p (A k , D n ) > x ≤ 2 exp -(n -p + 1) x 2 ∆ 2 k 2 , (4.20)
where ∆ = 4 √ e max C 2 , √ C 1 with C 1 , C 2 > 0 defined in Theorem 4.6. Moreover if p > n/2 + 1, for every x > 0,

P    R ECV p (A k , D n ) -E R ECV p (A k , D n ) > √ 2eΓ √ n -p + 1    k 3/2 n n-p+1 x + 2e k n n-p+1 x 3/2       ≤ n n -p + 1 e • e -x , (4.21)
where Γ > 0 is a numeric constant defined in Eq. (4.17).

Eq. ( 4.20) establishes that the LpO estimator exhibits a sub-Gaussian behavior. In particular it concentrates around its expectation at rate O(1/ √ n -p + 1). Note that this rate as well as the dependence with respect to k is inherited from the use of the bounded difference inequality (Theorem 6.2 in Boucheron et al., 2013a), which suggests that at least the k 2 at the denominator of Eq. (4.20) can still be improved.

In comparison with the previous bound, Eq. (4.21) is somewhat tighter at least in the first deviation term where k 3/2 directly results from our upper bound on the variance of the L1O estimator. Note also that this deviation term is of order O(1/ √ n) whatever the choice of p, which is also an improvement upon the O(1/ √ n -p + 1) rate in Eq. (4.20). However another deviation term in x 3/2 also arises in Eq. (4.21). It is the price to pay for our strategy of proof relying on the Rosenthal inequality. Let us also emphasize that if n-p+1 = m remains constant, then the LpO estimator (as a U-statistic of order m) converges in distribution toward a Gaussian distribution if properly normalized. It suggests that, at least in this setting, the second deviation term could be removed. Nevertheless one can notice that this deviation term decreases very quickly as p grows. For instance it becomes of order O(1/n) as p becomes close enough to n. This last remark emphasizes that unlike the previous bound given in Eq. ( 4.20), the present one improves as n/2 + 1 < p < n grows.

Conclusions

Based on the remark that the LpO estimator is a U-statistic of order m = n -p + 1, we developed a general strategy to derive systematic upper bounds on the polynomial moments of the centered LpO estimator. The important tools in this derivation are moment inequalities such as Theorem 15.5 in Boucheron et al. (2013a) as well as upper bounds on the L q stability which hold simultaneously for all q ≥ q 0 for some q 0 ≥ 2.

As a first illustration of this strategy, we derived exponential concentration inequalities for the LpO estimator of the performance of the Ridge and kNN learning algorithms. The resulting inequalities still remain to be improved in several respects (concentration rate, meaning of the constants in the deviation terms). Considering the asymptotic distribution of the LpO estimator as a U-statistic of infinite order can be of some help to check the optimality of the deviation terms (see for instance [START_REF] Rempala | Weak limits of u-statistics of infinite order[END_REF]). However the present analysis already allows us to recover existing concentration results and will serve as a starting point for further analyzing the behavior of the LpO estimator.

Another important issue is to extend this general strategy to other CV procedures such as V-FCV, RLT, . . . The connection raised between U-statistics and the corresponding estimators (at least for V-FCV) is already a promising direction which requires some further exploration.

Technical results.

Lemma 4.3. Let N ∈ G(v) denote a sub-Gaussian random variable. Then, for every q ≥ 1, it results that A proof of this lemma can be found in [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF] (see Corollary 11.1).

N -E [ N ] q ≤ 3 √ 2v √ q. Lemma 4.4. Given n points (x 1 , ..., x n ) in R d ,
Chapter 5

Estimator selection

Cross-validation (CV) allows to perform estimator selection that is, to choose the best estimator among a list of candidates, no matter the paradigm these estimators are derived from (empirical contrast minimizers, nearest-neighbors, . . . ). In the literature, several instances of the estimator selection problem have been studied.

Model selection is one first instance [START_REF] Massart | Concentration inequalities and model selection[END_REF]. In this framework, each candidate estimator is chosen within a set of functions called the model. Instances of such estimators are histograms, regressograms, Lasso and Ridge estimators to name but a few. In that respect CV can serve to perform model selection, which is studied in what follows in the density estimation context. More precisely Section 5.2 focuses on the CV performance when used for estimation, whereas Section 5.3 addresses the analysis of the CV behavior with an identification purpose.

Parameter calibration is another instance of the estimator selection problem (which reduces to model selection in some cases). It arises when the candidate estimators in the family depend on parameters that have to be chosen (calibrated). For instance, the choice of the regularization parameter λ in the Ridge regression can be interpreted as a calibration problem, but also as a model selection one since it amounts to choose one model-that is, a ball with a given radius-among a nested family of models.

However the parameter calibration problem is not limited to such "classical" model selection question. Indeed most of model selection procedures themselves depend on unknown parameters. Penalized criteria such as those described in [START_REF] Barron | Risk bounds for model selection via penalization[END_REF]; [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] depend on unknown constants that remain to be calibrated for instance by means of the slope heuristic (Arlot and Massart, 2009a). In such a case-except computational considerations-CV can help calibrating the constant that is, choosing the best model selection procedure among several candidates. This strategy has been recently explored in [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF].

CV for estimator selection

Let us start by briefly reviewing some important distinctions between concepts used to describe the performance of estimator selection procedures. It is out of the scope of the present work to provide a thorough discussion on this topic. We refer interested readers to [START_REF] Yang | Can the strengths of aic and bic be shared? a conflict between model indentification and regression estimation[END_REF] for a insightful discussion about estimation and identification purposes, and to the first two sections of [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF] for a review of the different uses of CV procedures.

Estimation and identification purposes

Notation

In what follows, let us define F as the set of measurable functions on X and f ∈ F as the target. The estimation of f is made from a collection of candidate estimators {f τ } τ ∈T , where T denotes a countable set of indices. Furthermore depending on the context, the set T = T n is allowed to vary with n. From a given sample D n of cardinality n, A τ (D n ) = f τ denotes the estimator output by the learning algorithm A τ computed from D n , for all τ ∈ T . In the case where A τ (D n ) is defined relatively to a model, for instance as an empirical contrast minimizer over a given model, then this model is denoted by F τ .

CHAPTER 5. ESTIMATOR SELECTION

Estimation/prediction

Roughly speaking, the goal is to find an estimator f τ achieving a performance (measured in terms of excess loss) that is almost the same as the best possible one achieved by the oracle estimator f τ where

τ = argmin τ ∈T L P f τ = argmin τ ∈T L P f τ -L P (f ) .
The performance is then typically quantified in terms of an oracle inequality, which is a non-asymptotic result such as

L P f τ -L P (f ) ≤ C n L P f τ -L P (f ) + r n ,
with high probability, where C n ≥ 1 is a numerical constant and r n ≥ 0 satisfies r n = o P L P f τ -L P (f ) , as n → +∞. The performance of LpO as a model selection procedure used for estimation is explored in Section 5.2. When the leading constant C n in the above oracle inequality converges to 1 as n grows, the procedure is said to be efficient (asymptotically optimal). In particular, this implies that

L P f τ -L P (f ) L P f τ -L P (f ) P -----→ n→+∞ 1.
The prototypical examples of penalized criteria known to be optimal for estimation (efficient) with a small collection of models are Mallows' C p [START_REF] Mallows | Some comments on Cp[END_REF] (with the squared loss) and AIC [START_REF] Akaike | Information theory and an extension of the maximum likelihood principle[END_REF] (with the log -loss). Numerous alternative penalized criteria have been designed to remedy the deficiencies of the two above ones in non-asymptotic frameworks or with large collections of models [START_REF] Arlot | Model selection by resampling penalization[END_REF][START_REF] Baraud | Gaussian model selection with unknown variance[END_REF][START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Van De Geer | l1-regularization in high-dimensional statistical models[END_REF]. Oracle-type inequalities have been proved for the latter ones.

Identification

In contrast to the previous objective, the goal is here to recover the estimator f τ with the best possible performance over the family, that is

L P f τ -L P (f ) = inf τ ∈T L P f τ -L P (f ) .
The performance of the resulting estimator f τ is measured by deriving a lower bound on the probability of recovering this best estimator

P f τ = f τ .
(5.1)

From a non-asymptotic perspective, this probability should be as large as possible. When this probability converges to 1 as n increases, then the procedure is said estimator/model consistent (see for instance Eq. (2.1) in [START_REF] Shao | An asymptotic theory for linear model selection[END_REF]. When estimator selection reduces to model selection, then identification amounts to recover the model providing the closest approximation to the target (in terms of the loss function). If the target belongs to one candidate model for large enough values of n, then it leads to recover the smallest model in the collection that contains the target (see [START_REF] Schwarz | Estimating the dimension of a model[END_REF] with BIC and [START_REF] Shao | Model Selection by Cross-Validation[END_REF] with LpO for instance). This notion has been somewhat refined by introducing an additional parameter µ > 0 quantifying the difficulty level on the problem at hand that is, assuming

(1 + µ)L P f τ ≤ inf τ =τ L P f τ
holds true with high probability. The resulting lower bound on (5.1) can be expressed in terms of the parameter µ to reveal its influence on the final convergence rate.

For the HOp procedure, the case of only two candidate estimators (Card(T ) = 2) has been considered by [START_REF] Yang | Consistency of cross-validation for comparing regression procedures[END_REF] for regression, and [START_REF] Yang | Comparing Learning Methods for Classification[END_REF] for classification. Sufficient conditions on p and the convergence rates of the candidate classifiers are derived for achieving the desired consistency property. For the LpO procedure, this analysis has been extended to a finite family of estimators (2 < Card(T ) < +∞) by Celisse (2014a) in the density estimation context (see Section 5.3).

Estimation/prediction versus identification

In his seminal paper, [START_REF] Shao | An asymptotic theory for linear model selection[END_REF] proved that no deterministic penalty can share the optimality properties of both AIC and BIC. More precisely, in the nonparametric case, any efficient procedure which is minimaxrate optimal (like AIC) cannot be consistent in the same time. By contrast in the parametric case, any consistent procedure (like BIC) is efficient, but cannot be minimax-rate optimal. Numerous attempts have been made to design data-driven penalties combining the strengths of AIC and BIC criteria [START_REF] George | Calibration and empirical Bayes variable selection[END_REF][START_REF] Hansen | Bridging AIC and BIC: an MDL model selection criterion[END_REF][START_REF] Shen | Adaptive model selection[END_REF]. However [START_REF] Yang | Can the strengths of aic and bic be shared? a conflict between model indentification and regression estimation[END_REF] provided a definitive answer to this question in the linear regression model since he proved that no consistent model selection procedure (deterministic or not) can achieve minimax-rate optimality.

Let us also mention that [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF] have recently investigated the statistical performances of CV procedures when used to combine the assets of both AIC and BIC. More precisely, from a set of candidate estimators, AIC and BIC are applied in a first step to choose the best candidate. Then the second step consists in minimizing the CV estimator of the performance of the AIC-and BIC-candidates. It is proved, in the regression context, that CV procedures are consistent and automatically recover the best estimator from the AIC-and BIC-based ones. Note that there is no contradiction with respect to the previous remark about the strengths of AIC and BIC since Theorems 1 and 2 in [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF] do not establish the minimax-rate optimality.

Risk estimation and model selection

As suggested in Chapter 3, the selection performances of CV mostly depend on two factors. The first one is its bias as an estimator of the risk; in particular, when the collection of estimators T is not too large, minimizing an unbiased estimator of the risk leads to an efficient selection procedure. The second factor, usually less important-at least asymptotically-, is the variance of CV as an estimator of the risk. One could conclude that the best CV procedure for estimation is the one with the smallest bias and variance (at least asymptotically), for instance, L1O in the least-squares regression framework (see [START_REF] Burman | A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods[END_REF] and Section 7.1 in [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF]).

Howvever, the best CV estimator of the risk can strongly disagree the best model selection procedure. According to [START_REF] Breiman | Submodel selection and evaluation in regression. the x-random case[END_REF] for instance the best risk estimator is L1O, whereas 10-fold CV is more accurate for model selection. Such a difference mostly comes from three reasons. First, the asymptotic framework (A fixed, n → ∞) may not apply to models close to the oracle. Second, estimating the risk of each model with some bias can compensate the effect of a large variance, for instance when the signal-to-noise ratio of data is small. Third, what really matters in model selection is that, for any model selection procedure based on a criterion crit(•),

sign (crit(τ 1 ) -crit(τ 2 )) = sign L P fτ1 (D n ) -L P fτ2 (D n )
with the largest possible probability, for all τ 1 , τ 2 "close to" the oracle τ * (D n ) . This idea has been recently explored by [START_REF] Arlot | Contributions to statistical learning theory: estimator selection and change-point detection[END_REF]; [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] who developed a promising heuristic-based approach in the density estimation framework.

Therefore, specific studies are required to evaluate the performances of CV procedures in terms of model selection efficiency.

CV estimators and penalized criteria

For any symmetric learning algorithm A and any sample D n , the CV estimator of L P (A(D n )) can be written as

∀1 ≤ p ≤ n -1, R W p (A, D n ) = L P Dn (A(D n )) + R W p (A, D n ) -L P Dn (A(D n )) ,
where L P Dn (A(D n )) denotes the empirical contrast evaluated at A(D n ). The term between brackets can be interpreted as a (random) penalty, which allows to make a connexion between any CV estimator and existing penalized criteria. This remark has been already formulated in [START_REF] Celisse | Model Selection Via Cross-Validation in Density Estimation, Regression and Change-Points Detection[END_REF] and more recently exploited in Lemma 1 of [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] as a means to analyze CV procedures in the density estimation context. Some highlighting conclusions on the behavior of the CV procedures for model selection arise from calculating the expectation of this random penalty. This direction has been explored from an asymptotic point of view leading to meaningful connexions between CV procedures and well-known penalized criteria (see [START_REF] Shao | An asymptotic theory for linear model selection[END_REF] for many examples). For instance the efficiency of CV mostly depends on the asymptotics of (n -p)/n:

• When n -p ∼ n , CV is asymptotically equivalent to Mallows' C p , hence asymptotically optimal.

• When n -p ∼ λn with λ ∈ (0, 1) , CV is asymptotically equivalent to GIC κ with κ = 1 + λ -1 , which is defined as C p with a penalty multiplied by κ/2 .

The above results have been proved in linear regression by [START_REF] Shao | An asymptotic theory for linear model selection[END_REF] for LPO (see also [START_REF] Li | Asymptotic optimality for Cp, CL, cross-validation and generalized cross-validation: discrete index set[END_REF] for L1O, and [START_REF] Zhang | Model selection via multifold cross validation[END_REF] for RLT when B n 2 . In a general statistical framework, the model selection performance of several CV-based procedures applied to minimum contrast estimation algorithms was studied in a series of papers (van der [START_REF] Van Der Laan | Unified cross-validation methodology for selection among estimators and a general cross-validated adaptive epsilon-net estimator: Finite sample oracle inequalities and examples[END_REF][START_REF] Van Der Laan | The cross-validated adaptive epsilon-net estimator[END_REF]). An oracle-type inequality is proved, showing that up to a multiplying factor C n → 1, the risk of the algorithm selected by CV is smaller than the risk of the oracle with n-p observations m * (D n-p ) . In most frameworks, this implies the asymptotic optimality (efficiency) of CV as long as n/(n -p) = O (1). When p ∼ λn with λ ∈ (0, 1), this generalizes Shao's results. Note however that the above results are meaningless with the L1O estimator (p = 1) since "remainder terms" then become of order O(1/p) = O(1) and are no longer negligible.

Estimation and efficiency

In what follows, the main focus is given to the density estimation framework where model selection is carried out from a collection of finite dimensional vector spaces (leading to projection estimators). This framework has been exploited to provide an accurate theoretical analysis of the behavior of the LpO estimator used to find an estimator with almost the same performance as the best one within the considered collection. We refer to (Arlot and Celisse, 2010, Section 6.3) for existing results on the performance of CV procedures for estimation/prediction in various contexts. A large part of the material in the present section comes from Celisse (2014a,b)

Oracle inequality for the LpO estimator

Although CV is among the most widely used procedures for model selection, there are only a few nonasymptotic results characterizing its behavior as a model selection procedure used for estimation (see Section 5.1.1). In particular in the density estimation framework, there does not exist any satisfactory oracle inequality for CV procedures, and hence for LpO (with p > 1) (except the recent work [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] that will be discussed later).

As already mentioned van der Laan and Dudoit (2003); van der Laan et al. ( 2006) have provided oracle inequalities for general CV procedures. But their results only apply to the case where the test set cardinality p is approximately equal to n. In particular, this does not cover the case of the L1O estimator. Note also that [START_REF] Lecué | Oracle inequalities for cross-validation type procedures[END_REF] have produced oracle inequalities as well. But these ones (such as [START_REF] Lecué | Oracle inequalities for cross-validation type procedures[END_REF], Theorem 2.7)) involve an oracle estimator that is computed from only n -p observations. With a somewhat instable algorithm, removing p points could substantially change the resulting estimator and therefore the best possible performance.

Assumptions

With the same notation as in the previous section, let us review the assumptions under which our main result is derived.

• Square-integrable density:

f ∈ L 2 ([0, 1]).
(SqI)

Unlike [START_REF] Castellan | Density estimation via exponential model selection[END_REF] for instance, it is not assumed that the density f ≥ ρ, with a constant ρ > 0.

• Model regularity:

Let {ϕ λ } λ∈Λ denote a countable orthonormal family of L 2 ([0, 1]) such that {ϕ λ } λ∈Λ(τ ) is an orthonormal basis of the model F τ (τ ∈ T n ) with Card(Λ(τ )) = D τ . Let us further assume

∃Φ > 0, sup τ ∈Tn φ τ ∞ D τ ≤ Φ, with φ τ = λ∈Λ(τ ) ϕ 2 λ . (RegD)
The regularity (measured in terms of sup-norm) of the orthonormal basis of the model F τ has to remain controlled by the dimension of the vector space whatever the model in the collection. With a histogram estimator defined from a partition {I λ } λ∈τ of [0, 1], (RegD) requires |I λ | ≥ (ΦD τ ) -1 for every λ ∈ Λ(τ ). In other words histograms in the collection cannot be too irregular.

• Polynomial collection: There exists a T ≥ 0 such that

Card(T n ) ≤ n a T . (Pol)
This holds true if there exists α ≥ 0 such that Card ({τ

∈ T n , D τ = D}) ≤ D α , for all 1 ≤ D ≤ n.
In particular, a nested collection of models (such that F τ ⊂ F τ if τ < τ ) satisfies (Pol).

• Maximal dimension:

∃Γ > 0, sup τ ∈Tn D τ ≤ Γ n (log n) 2 • (Dmax)
In the sequel, Γ = 1 is considered all along the section to simplify the expressions. Note that proofs and conclusions remain unchanged with this particular choice.

• Estimation error and dimension: With • denoting the L 2 norm of measurable functions defined on [ 0, 1 ],

∃ξ > 0, inf τ ∈Tn √ nE f τ -f τ √ D τ ≥ ξ, (LoEx)
where f τ denotes the orthogonal projection of f onto the finite dimensional vector space F τ (model).

This assumption makes the estimation error and D τ /n comparable. It can be shown (see Lemma B.3 in the supplementary material Celisse, 2014b) that (LoEx) is fulfilled with any density f ∈ L 2 (0, 1) estimated by regular histograms such that one can find two constants η, ∈ (0, 1) such that the Lebesgue measure Leb ({x

∈ [0, 1] | f (x) ≥ η}) ≥ and > inf τ ∈Tn D τ -1
.

For instance, the latter inequality amounts to exclude too "small" models for which the support of the density f is included in only one interval I λ .

• Richness of the collection: There exist τ 0 ∈ T n and c rich ≥ 1 such that,

√ n ≤ D τ 0 ≤ c rich √ n. (Rich)
This requirement is rather mild since one can add such a model in our collection.

• Approximation property: There exist c , c u > 0 and > u > 0 such that, for every τ ∈ T n ,

c D - τ ≤ f -f τ 2 ≤ c u D -u τ . (Bias)
This assumption quantifies the approximation error incurred by model F τ in estimating f . It therefore relies on a smoothness assumption on f . Such an upper bound is classical for α-Hölderian functions with α ∈ (0, 1] and regular histograms for instance. Note that [START_REF] Stone | An asymptotically optimal histogram selection rule[END_REF] uses the same assumption (lower bound), which is the finite-sample counterpart of the classical assumption f -f τ > 0 for every τ ∈ T n usually made to prove asymptotic optimality for a model selection procedure (see [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF].

Main result

The performance of the LpO estimator with respect to p is described by the following oracle inequality, which establishes the CV non-asymptotic optimality.

Theorem 5.1 (Theorem 3.1 in Celisse (2014a)). Let f denote a density on [0, 1] such that (SqI) holds true, and set a collection of models {F τ } τ ∈Tn such that (RegD), (Pol), (Dmax), (Rich), (LoEx), and (Bias) are satisfied. For every p ∈ {1, . . . , n -1}, let τ = τ (p) denote the index of the model minimizing R ECV p (τ ) over T n . Then, there exist a positive integer n 0 = n 0 (f, Φ, ξ, Γ), a sequence (δ n ) N such that δ n = δ n (f, Φ, ξ, Γ) → 0 with nδ n → +∞ as n → +∞, and an event Ω with P( Ω) ≥ 1 -6/n 2 on which, n ≥ n 0 implies

f -f τ (p) 2 ≤ C n (p/n) inf τ ∈Tn f -f τ 2 with C n (p/n) = T + B ∨ T + V T - B ∧ T - V ≥ 1, (5.2)
where

T - B = 1 -δ n K(n, p), T - V = n n -p (1 -δ n ) [ 1 -4δ n ] -2δ n K(n, p) [ 3 -4δ n ] , T + B = 1 + δ n K(n, p), T + V = n n -p (1 + δ n ) [ 1 + 4δ n ] + 2δ n K(n, p) [ 3 + 4δ n ] ,
and

K(n, p) = 1 + 2 n-1 + p n-p 1 n-1 ≤ 2 + 2 n-1 •
In the right-hand side of Eq. ( 5.2), we could have written an additional remainder term of order δ n [ n/(n -p) + K(n, p) ] • o(1/n) which only depend on f and constants Φ, ξ, Γ. This additional remainder term has been put it in the leading factor C n (p/n) at the price of slightly increasing its value by an amount which turns out to be negligible. Therefore the leading factor C n (p/n) depends on the target density f . Let us notice that [START_REF] Birgé | From model selection to adaptive estimation[END_REF] proved oracle inequalities with a similar dependence of the leading constant on the target density (see for instance Theorems 3 and 4). The need for introducing the integer n 0 is essentially a sufficient condition for T - B and T - V to be positive. Recently, [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] derived a similar oracle inequality (Theorem 5) applying to more general resampling-based procedures. It reduces to ours for a particular choice of the constants, that is choosing V = n and C = (n/p -1/2)/(n/p -1), for 1/2 < C ≤ 2 with their notations (see also their Section 3.4).

From a general perspective Eq. ( 5.2) and Theorem 5 in [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] lead to the same conclusion that is, all values of p = p n such that p/n → 0 imply C n (p/n) → 1 as n → +∞, which leads to efficient (asymptotically optimal) model selection procedures. This means that, in the present context, any LpO procedure is efficient as long as p/n → 0, which remains true with V-FCV procedures (V = n/p → +∞). This holds in particular true with p = 1 that is, L1O is asymptotically optimal since

C or,n (p/n) := f -f τ (1) 2 inf τ ∈Tn f -f τ 2 a.s.
-----→ n→+∞ 1, (5.3) where τ (1) = argmin τ ∈Tn R ECV 1 (A τ , D n ). Note that this is also consistent with results by Shao (1997) established in the nonparametric regression case and saying that (asymptotically) unbiased risk estimation is a necessary condition for efficiency.

Non-asymptotic optimization with respect to p

The previous section allows to conclude that L1O is asymptotically optimal since it amounts to perform unbiased risk estimation. However this seems somewhat contradictory with empirical observations. Indeed many non-asymptotic settings illustrate that (biased) strategies such as 10-FCV can provide the best results [START_REF] Breiman | Submodel selection and evaluation in regression. the x-random case[END_REF]). Empirical assessment of the optimal constant Figure 5.1 displays the behavior of the optimal ratio C or,n (p/n) defined in Eq. ( 5.3) with respect to p/n. The density f is defined from a mixture of beta distributions and estimated by means of regular histograms (see Section 3.1.4 of Celisse (2014a) for details).

The curve flattens as n grows, while a "plateau"arises and moves to the left. On the one hand, this illustrates that there is much to lose by missing the optimal value of p/n with n small (n = 100). But as n increases, this curve flattens and all values of p/n belonging to the plateau perform almost the same. On the other hand, the ratio p/n has actually to decrease toward 0 for achieving efficiency as n becomes large, since the plateau slowly moves to the left and the curve is increasing with n = 10 000. Furthermore, the smallest value achieved by C or,n (p/n) slowly decreases as n grows. These two remarks are in accordance with the efficiency result (deduced from Theorem 5.1) saying that any ratio such that p/n → 0 implies C or,n (p/n) → 1.

Let us also emphasize that the minimum location of the curve p/n → C or,n (p/n) strongly differs from the choicep = 1 in the present experimental setting. Actually, it varies from p/n = 0.56 for n = 100 up to p/n = 0.24 for n = 10 000. Even for a such large amount of points, this empirically suggests that the L1O procedure can be suboptimal in non-asymptotic situations. This last remark points out the need for a deeper non-asymptotic understanding of the behavior of LpO as a model selection procedure used for estimation.

Optimizing the leading constant

Our suggestion consists in studying the function p/n → C n (p/n) to derive its minimum location. Considering this constant as a proxy to the optimal ratio C or,n (p/n), optimizing C n (p/n) with respect to p would lead to an approximation to the optimal value of p.

The following Corollary 5.1 provides the expression for the minimizer of p/n → C n (p/n).

Corollary 5.1 (Corollary 3.1 in Celisse (2014a)). With the notation and assumptions of Theorem 5.1, the constant

C n (p/n) is minimized over p ∈ {1, . . . , n -1} for 0 < p * n n = 1 - 1 -5δ n + 4δ 2 n -5δn n-1 + 8δ 2 n n-1 1 + (5δ n -8δ 2 n )(1 + 1 n-1 ) < 1•
Furthermore, the optimal ratio p * /n is slowly decreasing to 0 as n tends to +∞

p * n ∼ +∞ 10nδ n -----→ n→+∞ +∞.
(5.4)

In Eq. ( 5.4) the (approximately) optimal value of p is expressed in terms of the sequence δ n = δ n (f, Φ, ξ, Γ). In particular, C n (p/n) is minimized for p * /n slowly decreasing toward 0 as n grows. Let us notice that it excludes the case where p does not depend on n, which holds true with the L1O procedure.

Provided our optimization strategy is meaningful to infer the behavior of C or,n (p/n), this suggests that L1O can be suboptimal in the finite-sample setting. This claim is, at least empirically, supported by the simulation results where the minimum location of C or,n (p/n) is far from 1/n even for n = 10 000 (Figure 5.1). However even if the behavior described by Eq. ( 5.4) seems empirically supported by the simulation results (Figures 5.1 and fig.comparison.oracle.and.constant.behaviour), the validity of this approach remains to be theoretically grounded. Obviously optimizing the upper bound could lead to misleading conclusions, for instance if this upper bound is not tight enough. For instance deriving a lower bound with high probability for the optimal ratio C or,n (p/n) would help to clarify the validity of the present exploratory approach. An open question is also to clarify the potential link with the heuristic approach developed by (Arlot and Lerasle, 2015, Section 4) aiming at optimizing over the CV procedure.

Empirical assessment of the optimization strategy

Identification and estimator consistency

The present section mainly focuses on the theoretical analysis of the LpO procedure used for identification that is, for recovering the best estimator among the collection of candidates. Our analysis is carried out in the density estimation framework with the quadratic loss. We refer interested readers to (Arlot and Celisse, 2010, Section 7) for other results applying to CV procedures used for identification.

Assumptions

Let us now detail and comment our main assumptions used to study the asymptotic properties of the LpO procedure in terms of model/estimator consistency.

The best model. More precisely, our purpose is to recover the best model denoted by F τ and defined by τ := argmin

τ ∈Tn E f -f τ 2 , ( 5.5)
where τ is a deterministic quantity assumed to be unique and • denotes the L 2 norm on [ 0, 1 ].

Parametric and nonparametric models. Let us further assume that the countable collection {F τ } τ ∈Tn can be split into:

• Parametric models indexed by T P n for which there exist constants π, ρ > 0 (independent of n) such that sup

τ ∈T P n nE f τ -f τ 2 ≤ π , and inf τ ∈T P n ,f ∈Fτ f -f τ 2 ≥ ρ.
(5.6)

• Nonparametric models indexed by

T N P n such that n (log n) -2 inf τ ∈T N P n E f τ -f τ 2 -----→ n→+∞ +∞.
(5.7)

Then,

{F τ } τ ∈Tn = {F τ } τ ∈T P n ∪ {F τ } τ ∈T N P n . (P-NP)
Parametric models are models with convergence rate of order 1/n. Since Consistently with this remark, (5.6) requires that the largest dimension over parametric models is bounded by a constant independent of n, and that the bias of parametric models such that f ∈ F τ cannot decrease toward 0 with n. Otherwise, such a model would be nonparametric.

E f -f τ 2 ≈ f -f τ 2 + C • D τ /n,
Conversely (5.7) only requires the dimension of nonparametric models must be larger than (log n) 2 . In particular, this does not prevent nonparametric models from containing f or having their bias decreasing toward 0 as n grows.

Size of the gap. Since identifying τ cannot be achieved if other models can (asymptotically) perform as well as F τ , one also introduces a parameter µ > 0 and n 0 ∈ N * such that for all integers n > n 0 ,

(1 + µ)E f -f τ 2 ≤ inf τ ∈Tn\{τ } E f -f τ 2 .
(Gap)

This parameter µ quantifies the gap between the performance of the best estimator and that of other competitors. In the model selection framework for instance, a small value of µ means that the collection of models is allowed to include at least one other candidate with a very similar performance to that of τ . Recovering τ is then a difficult task. In other words, µ encodes the difficulty level of the identification problem at hand. Note that (Gap) excludes the case where two estimators in the collection have the same asymptotic positive risk. Therefore this assumption induces some restrictions on the collection of competing estimators. The same assumption (in probability rather than in expectation) has been made by [START_REF] Yang | Comparing Learning Methods for Classification[END_REF][START_REF] Yang | Consistency of cross-validation for comparing regression procedures[END_REF] with a simple collection of two candidate estimators. In what follows, the set of indices T n is required to be finite and can vary with n (see Theorems 5.2 and 5.3 where µ is assumed to be independent of n).

Remark 5.1. The requirement made by (Gap)-that µ is independent of n-can seem somewhat restrictive. On the contrary, one could be tempted to reverse the dependence between the collection of models and the parameter µ, saying that µ is defined by

µ = inf τ =τ E f -f τ 2 -E f -f τ 2 E f -f τ 2 •
A first consequence of this alternative definition is that µ depends on the target f , on the collection of models, and therefore on n.

For instance, from a nested collection of models, assuming that f belongs to one parametric model indexed by τ , and that (5.8) which implies that µ does not depend on n in this particular case. By contrast, a necessary and sufficient condition for µ to decrease with n is that the difference at the numerator has to be negligible with respect to the excess risk of the best model. Intuitively, this holds true if f does not belong to any model in the collection and f τ achieves the minimax rate n -2α 2α+1 over Hölder balls H(L, α) (with L > 0 and α ∈ (0, 1)). Then µ is approximately of order n -1 2α+1 . The increase of µ = µ(α) with α means that the problem becomes easier as the target f becomes more and more smooth.

E f -f τ 2 ≈ f -f τ 2 + C • D τ /n for all τ , it results that µ ≈ C D τ + 1 n - D τ n • C D τ n -1 = 1 D τ ,

Main results

Depending on whether the target f belongs or not to ∪ τ ∈Tn F τ , the two following results provide sufficient conditions on the LpO procedure for model selection consistency to hold. Their main contribution is to relate the cardinality p of the test set to the rate of convergence (that is, the excess risk) of f τ . Note that in addition, the model consistency property is established with a collection of models allowed to vary with n, which contrasts with earlier results (see for instance [START_REF] Yang | Consistency of cross-validation for comparing regression procedures[END_REF].

The target belongs to one candidate model

Let us start with the setting where f belongs to ∪ τ ∈Tn F τ . Since f does not depend on n, it means that f belongs to a parametric model, which entails the best estimator f τ achieves the parametric rate 1/n. Theorem 5.2 (Model consistency with f ∈ ∪ τ F τ ). Let ∪ τ ∈Tn F τ denote a collection of models satisfying (Pol) and (P-NP), τ ∈ T n given by (5.5) be such that (Gap) holds true, and assume (SqI), (RegD), (Dmax), and (LoEx). For every

1 ≤ p ≤ n -1, let us also define τ = τ (p) = argmin τ ∈Tn R ECV p (m). If the target f ∈ ∪ τ ∈Tn F τ , then all 1 ≤ p = p n ≤ n -1 such that log(n) 1 - p n -----→ n→+∞ 0, and n 1 - p n -----→ n→+∞ +∞,
(5.9)

leads to P [ τ = τ ] -----→ n→+∞ 1.
When f belongs to ∪ τ ∈Tn F τ , the best estimator f τ in a polynomial collection can be recovered by CV provided p/n converges to 1 as n tends to +∞. The proof establishes this rate (i) cannot exceed 1/n for distinguishing between parametric estimators (with convergence rate of order 1/n), and (ii) has to be faster than (log n) -1 to allow dealing with the polynomial complexity of the model collection. For instance a finite collection would lead to replace the (log n) -1 rate by a slower one determined by the control level of P [ τ = τ ]. Consistently with Remark 5.1, the (Gap) assumption is not restrictive since µ does not actually depend on n.

In the regression setting [START_REF] Yang | Consistency of cross-validation for comparing regression procedures[END_REF] proved that p/n → 1 (with n -p → +∞) enables to recover the best parametric estimator among two parametric candidates with HOp (see Corollary 1, (i)), while this requirement is no longer necessary when comparing parametric and nonparametric estimators. Our result is consistent with Yang's one, although our setting is somewhat more general since we compare the best parametric estimator with both parametric and nonparametric ones in the same time.

The target does not belong to any candidate model

Conversely if f does not belong to ∪ m F τ , all parametric models are biased according to (5.6) and f τ necessarily achieves a nonparametric rate, that is nR( f τ ) → +∞ as n tends to +∞. In this context we can prove the following Theorem 5.3. 2014a)). Let ∪ τ ∈Tn F τ denote a collection of models satisfying (Pol) and (P-NP), τ ∈ T n given by (5.5) be such that (Gap) holds true, and assume (SqI), (RegD), (Dmax), and (LoEx). For every 1 ≤ p ≤ n -1, let us also define τ = τ (p) = argmin τ ∈Tn R ECV p (m). Let us assume the target f ∈ ∪ τ ∈Tn F τ and R( f τ ) → 0 as n tends to +∞.

1. If D τ ≤ (log n) 4 for large enough values of n, then every

1 ≤ p = p n ≤ n -1 such that log(n) 1 - p n -----→ n→+∞ 0 and E f τ -f τ 2 n -p = o µR( f τ ) (5.10) leads to P [ τ = τ ] -----→ n→+∞ 1.
2. If D τ > (log n) 4 for large enough values of n, then every 1 ≤ p = p n ≤ n -1 such that

(log n) 5 n = o p n -p E f τ -f τ 2 and p n -p E f τ -f τ 2 = o µR( f τ ) (5.11) leads to P [ τ = τ ] -----→ n→+∞ 1.
Both Eq. ( 5.10) and ( 5.11) relate the optimal choice of p/n to the convergence rate of f τ and to µ, which quantifies the discrepancy between the best estimator and the second best one among the candidates. In the present context µ is not allowed to depend on n and could be removed that is, o µR( f τ ) = o R( f τ ) .

It has been nevertheless included in Eq. ( 5.10) and (5.11) to emphasize its possible influence on p.

For instance with the same technique of proof, Eq. ( 5.10) would lead to the desired conclusion if µ were allowed to depend on n. Actually the requirement D τ ≤ (log n) 4 suggests that the oracle estimator almost achieves a parametric rate (up to a "small" logarithmic factor). This implies that both R( f τ ) and

E f τ -f τ 2
are (approximately) of order 1/n. Then Eq. ( 5.10) becomes

log(n) 1 - p n -----→ n→+∞ 0 and 1 n -p = 1 n (1 -p/n) = o (µ) ,
which is exactly Eq. ( 5.9) if µ is constant with respect to n (which is almost true with small nonparametric models). By contrast, (5.11) cannot be easily extended to the case where µ is allowed to depend on n. Following the nonparametric example discussed in Remark 5.1, including large nonparametric models (D τ > (log n) 4 ) in the collection would lead to a fast decrease of µ with respect to n. However from the proof of (5.11) (Appendix C of Celisse (2014b)), it arises that µ has to be larger than δ n that is a decreasing sequence depending (at least) on the structure of the collection of models.

Note that Theorem 5.3 is the analogue of (Yang, 2007, Corollary 1, (ii)) derived in the linear regression framework.

Empirical assessment

Simulation experiments have been performed in the settings of Theorems 5.2 and 5.3, respectively when s belongs to (resp. does not belong to) the model collection. We refer to Celisse (2014a) Section 3.2.2 for details about the simulation experiments. The main results are illustrated respectively in Figures 5.3 and 5.4 where P [ τ = τ ] is displayed with respect to the ratio p/n. s belongs to the model collection Figure 5.3 displays the LpO performance when it is used for identification in two parametric settings corresponding to densities s 1 and s 2 .

As predicted by Theorem 5.2, CV reaches model selection consistency for recovering the best parametric estimator f τ on condition that p/n increases to 1 as n → +∞.

Comparing Panels (a) and (b), the convergence rate is slower in (b). This is consistent with the larger value of µ-which means that the problem is easier-in Panel (a) (where µ ≈ 1/2 from Eq.( 5 s does not belong to the model collection The converse situation arises in Figure 5.4 since CV reaches model selection consistency as long as p/n decreases to 0 as n → +∞. Let us emphasize that these experimental results have been obtained with µ allowed to depend on n (unlike (Gap)). This illustrates that the conclusions drawn from Theorem 5.3 are likely not limited to the case where µ does not depend on n. From the considered collection of models, it results that µ in Eq. ( 5.11) respectively satisfies µ α ≈ n -1 2α+1 and µ β ≈ n -1 2β+1 . Since α > β, it implies that µ α > µ β , which means that the experimental setting in Panel (a) is easier than the one in Panel (b). This explains why the model selection consistency (illustrated by both Panels (a) and (b)) is faster for the smoothest density s 3 than for s 4 .

Besides the highest probability is achieved for p/n ≈ 0.18 with n = 6 000 in Panel (b), whereas it is achieved for p/n ≈ 0.01 with only n = 3 000 in Panel (a). Up to the modification of (Gap) already discussed above, this observation can be related to the "lower bound" on p/n derived in Eq. (5.11) where

(log n) 5 nE f τ -f τ 2 = o p n -p ,
which means that p/n cannot decrease too fast to 0 as

n → +∞. Since E f τ -f τ 2 ≈ n -2α 2α+1 in Panel (a) is smaller than E f τ -f τ 2 ≈ n -2β 2β+1
in Panel (b), it results that the lower bound on p/n is smaller with s 3 than the one with s 4 . However deriving a fully justified explanation for this phenomenon remains a challenging task.

Conclusion

Dependence on closed-form formulas All of the material described in Sections 5.2 and 5.3 strongly relies on the closed-form formulas available in the density estimation framework by means of projection estimators. A natural question is then to know if similar (model selection) results could be derived from some of the concentration inequalities detailed in Section 4.2. Unfortunately the answer is negative. The aforementioned concentration results derived for the kNN estimator in binary classification or for the Ridge regression are not tight enough to deduce a finite-sample model selection result. For instance this limitation has been already discussed along the comments of Proposition 5.3 in [START_REF] Mary-Huard | New upper bounds on cross-validation for the k-nearest neighbor classification rule[END_REF] for the kNN classifier. Even if the provided bounds improve on ongoing ones, they are not tight enough with respect to their dependence on k. Similarly for the Ridge regression, the 1/( √ nλ) deviation rate in Corollary 4.1 should be replaced by 1/(nλ 2 ) as already discussed in the comments of this result and emphasized by [START_REF] Zhang | Leave-one-out bounds for kernel methods[END_REF]. This could be done with tighter constants in the deviation term, which cannot be achieved with the present derivation strategy. This issue is likely to be overcome using the alternative derivation strategy suggested in Section 7.2.2.

Optimal CV for model selection with an estimation purpose The oracle-type inequality exposed in Section 5.2 for the LpO estimator when used for estimation is the same as the one established more recently by [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF]. However their heuristic argument cannot be applied for optimizing with respect to p since the LpO estimator is a biased risk estimator. Understanding how the SN R ratio (see their Eq. ( 21)) varies as a function of p remains an open (but promising) question.

Optimal CV for model selection with an identification purpose Regarding the performance of LpO for identification when the oracle model belongs to large nonparametric models (Section 5.3), an important question is to relax the (Gap) assumption as pointed out in the comments following Theorem 5.3. In particular, linking the optimal value of p with the parameter µ would allow us to identify more convincing guidelines leading to an (almost) optimal value of p.

Chapter 6

Multiple change-point detection

The change-point detection problem has been tackled in numerous works (in the statistics and machine learning literature) which can be divided into two groups.

The first one is mainly concerned with the off-line setting where an entire times series is observed [START_REF] Brodsky | Nonparametric methods in change-point problems[END_REF]Carlstein et al., 1994;Tartakovsky et al., 2014). The goal is then to identify homogeneous segments along the time in which some features of the distribution of the observations-their mean or their variance, for instance-remain unchanged. When the number of change-points is known, this problem reduces to estimating the change-point locations as precisely as possible; in general, the number of change-points itself must be estimated. This problem arises in a wide range of applications, such as bioinformatics [START_REF] Curtis | Enabling dynamic network analysis through visualization in TVNViewer[END_REF][START_REF] Picard | A statistical approach for array CGH data analysis[END_REF], neuroscience [START_REF] Park | Anomaly detection on whole-brain functional imaging of neuronal activity using graph scan statistics[END_REF], audio signal processing [START_REF] Wu | Multiple change-point audio segmentation and classification using an MDL-based Gaussian model. Audio, Speech, and Language Processing[END_REF], temporal video segmentation [START_REF] Koprinska | Temporal video segmentation: A survey[END_REF], post-analysis of hacker attacks (Wang et al., 2014), social sciences [START_REF] Kossinets | Empirical analysis of an evolving social network[END_REF] and econometrics [START_REF] Mcculloh | Detecting Changes in a Dynamic Social Network[END_REF].

The second one deals with the on-line setting where new observations come at each time step and the question is to detect, as soon as possible, any structural change in (some features of) the distribution of the data along the time. This problem, which belongs to the more general anomaly detection, has become popular for detecting cyber-attacks (Lévy-Leduc and Roueff, 2009) and seismic events [START_REF] Ross | Automatic picking of direct p, s seismic phases and fault zone head waves[END_REF] for instance, or with social network analysis [START_REF] Frisén | Optimal sequential surveillance for finance, public health, and other areas[END_REF] and sensor networks monitoring [START_REF] Rice | Flexible smart sensor framework for autonomous structural health monitoring[END_REF].

Here we focus on the off-line change-point detection problem. In particular, the purpose of the following sections is illustrate how to overcome three main limitations of the ongoing literature on change-point detection. The first one (Section 6.2) addresses the problem of detecting multiple changes arising only in the mean of a signal in a heteroscedastic setting, that is while the variance is allowed to vary along the time. The second one (Section 6.3) describes a procedure that is sensitive to changes arising along the time in the full distribution (not only in the mean and/or variance) of complex objects not limited to real-valued vectors in R d . The third one (Section 6.4) addresses computational issues owing to the use of reproducing kernels.

The change-point detection problem

Let X be some measurable set and X 1 , . . . , X n ∈ X a sequence of independent X -valued random variables. For any i ∈ {1, . . . , n}, we denote by P Xi the distribution of X i . The change-point problem can then be summarized as follows: Given (X i ) 1≤i≤n , the goal is to find the locations of the abrupt changes along the sequence P X1 , . . . , P Xn . Note that the case of dependent time series is often considered in the change-point literature [START_REF] Bardet | Detecting abrupt changes of the long-range dependence or the self-similarity of a gaussian process[END_REF][START_REF] Bardet | Multiple breaks detection in general causal time series using penalized quasi-likelihood[END_REF][START_REF] Chang | Segmenting multiple time series by contemporaneous linear transformation[END_REF][START_REF] Lavielle | Least-squares estimation of an unknown number of shifts in a time series[END_REF]; as a first step, this work focuses on the independent case for simplicity.

An important example to have in mind is when X i corresponds to the observation at time t i = i/n of some random process on [0, 1], and we assume that this process is stationary over [t , t +1 ), = 0, . . . , D , for some fixed sequence 0 = t 0 < t 1 < • • • < t D +1 < t D +1 = 1. Then, the change-point problem is equivalent to localizing the change-points t 1 , . . . , t D ∈]0, 1[, which should be possible as the sample size 83 n tends to infinity. Note that we never make such an asymptotic assumption in the present work, where all theoretical results are non-asymptotic.

Example 6.1. The set X is R or R d , and the sequence (P Xi ) 1≤i≤n changes only through its mean. This is the most classical setting, for which numerous methods have been proposed and analyzed in the onedimensional setting [START_REF] Boysen | Consistencies and rates of convergence of jump-penalized least squares estimators[END_REF][START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF][START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF][START_REF] Korostelev | Mathematical statistics. Asymptotic minimax theory[END_REF]Zhang and Siegmund, 2007a) as well as the multi-dimensional case [START_REF] Bleakley | The group fused lasso for multiple change-point detection[END_REF][START_REF] Collilieux | A factor model approach for the joint segmentation with between-series correlation[END_REF][START_REF] Hocking | Learning sparse penalties for change-point detection using max margin interval regression[END_REF][START_REF] Picard | Joint segmentation of multivariate gaussian processes using mixed linear models[END_REF][START_REF] Soh | High-dimensional change-point estimation: Combining filtering with convex optimization[END_REF].

Example 6.2. The set X is R or R d , and the sequence (P Xi ) 1≤i≤n changes only through its mean and/or its variance (or covariance matrix). This setting is rather classical, at least in the one-dimensional case, and several methods have been proposed for it [START_REF] Andreou | Detecting multiple breaks in financial market volatility dynamics[END_REF][START_REF] Fryzlewicz | Multiple-change-point detection for auto-regressive conditional heteroscedastic processes[END_REF][START_REF] Picard | A statistical approach for array CGH data analysis[END_REF].

Adaptation to heteroscedasticity

The purpose here is to briefly describe how CV (and closed-form expressions) can help improving upon ongoing procedures by taking into account changes in the noise along the time (heteroscedasticity).

Context

As pointed out by [START_REF] Lavielle | Using penalized contrasts for the change-point problem[END_REF], multiple change-point detection procedures generally tackle one among the following three problems:

1. Detecting changes in the mean assuming the variance σ 2 is constant, 2. Detecting changes in the variance σ 2 assuming the mean is constant, 3. Detecting changes in the (mean, variance) with no distinction between changes in the mean, in the variance, and changes arising (at least) in both of them.

See for instance [START_REF] Bertin | Segmentation of multiple series using a lasso strategy[END_REF]; [START_REF] Gijbels | On the estimation of jump points in smooth curves[END_REF]; [START_REF] Picard | A statistical approach for array CGH data analysis[END_REF] to name but a few.

In applications such as Comparative Genomic Hybridization (CGH) data analysis (see (Arlot and Celisse, 2010, Section 6) for more details on CGH data) for instance, changes in the mean have an important biological meaning, since they correspond to the boundaries of amplified or deleted areas of chromosomes. However in the CGH setting, the variance σ 2 is not always constant, as assumed in Problem 1. In particular heteroscedasticity-that is, variations of σ 2 -can correspond to experimental artefacts or biological nuisance that should be removed. Therefore, CGH data analysis requires to solve a fourth problem, which is: 4. Detecting changes in the mean with no constraint on the variance σ

2 : [0, 1] → [0, ∞) .
Compared to Problem 1, the difference is the presence of an additional nuisance parameter σ 2 making Problem 4 harder. Up to the best of our knowledge, very few change-point detection procedures have been proposed for solving Problem 4 with no prior information on σ 2 (see for instance [START_REF] Muggeo | Efficient change point detection for genomic sequences of continuous measurements[END_REF]; [START_REF] Pein | Heterogeneous change point inference[END_REF]. Solving this problem is the purpose of the present section.

Notation. Let (t 1 , X 1 ), . . . , (t n , X n ) ∈ [0, 1] × R denote n independent random variables such that X i = f (t i ) + σ(t i ) • i , ∀1 ≤ i ≤ n,
where f denotes a real-valued piecewise-constant function, i is a zero-mean and reduced variable, and σ : [0, 1] → R + is any measurable function.

From the notation a, b = [a, b[∩N the set of integers between a and b (with b excluded), for any a < b, a given segmentation τ of (1, . . . , n) into D τ segments is defined from a set of D τ change-points τ 1 < τ 2 < . . . < τ Dτ < n + 1 (with τ 1 = 1 by convention) by

τ = { τ 1 , τ 2 , . . . , τ Dτ , n + 1 } .
Let us now introduce the vector space F τ of piecewise-constant functions defined from the segmentation τ . Then the performance of any g ∈ F τ is measured through the least squares empirical contrast given by

L Pn (g) = 1 n n i=1 (X i -g(t i )) 2 = Y -g n ,
where g = (g(t 1 ), . . . , g(t n )) ∈ R n . Then the empirical contrast minimizer f τ is given by

f τ = argmin g∈Fτ L Pn (g) = Dτ d=1 1 τ d+1 -τ d τ d+1 -1 i=τ d X i • 1 [τ d ,τ d+1 [ . (6.1)
In practical settings where the signal-to-noise ratio is low (which is a common situation) and only a finite (and small) number of observations are available, it is almost impossible to recover all true changepoints without including false change-points in the same time [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF]. This justifies the goal we pursue here that is estimating the underlying piecewise-constant regression function rather than focusing on the change-point locations. Fortunately in settings where the signal-to-noise ratio is large enough, any reasonable piecewise-constant estimator of the regression function would yield accurate estimates of the change-points. We also refer to [START_REF] Garreau | Consistent change-point detection with kernels[END_REF] who recently clarified the connection between estimating the piecewise-constant regression function and the estimated change-point locations.

Finding the change-points locations under heteroscedasticity

In this work we mainly focus on model selection-based approaches, which represent a large part of the literature (even if not all of it) on multiple change-points detection [START_REF] Baraud | Gaussian model selection with unknown variance[END_REF]Cleynen and Lebarbier, 2014b;[START_REF] Harchaoui | Multiple change-point estimation with a total variation penalty[END_REF][START_REF] Killick | Optimal detection of changepoints with a linear computational cost[END_REF][START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF]Zhang and Siegmund, 2007b) Let us also briefly mention that other approaches have been explored [START_REF] Frick | Multiscale change point inference[END_REF] where the FWER criterion is controlled, [START_REF] Matteson | A nonparametric approach for multiple change point analysis of multivariate data[END_REF] with a sequential testing procedure, [START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF] where an early stopping time is designed to avoid unnecessary computations).

In model selection-based procedures such as that one described in [START_REF] Birgé | Gaussian model selection[END_REF] and further explored in [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF], the problem of recovering multiple change-points is delineated into two successive steps:

1. The first step considers each possible number of change-points between two prescribed values. For each such number, the candidate change-points are identified by minimizing an empirical quality measure (the empirical contrast). This step outputs a collection of candidate segmentations that is, a collection of lists of candidate change-points (one such list for each number of change-points).

2. The second step consists in choosing the final number of change-points, then leading to the estimated segmentation. This is achieved by minimizing an appropriate penalized criterion where the large number of candidate segmentations has to be taken into account.

With these two steps in mind, mistakes can have two origins: (i) a poor collection of candidate segmentations at the first step, or/and (ii) an erroneous choice of the final number of change-points at the second step. In the present section, we essentially study the first step and illustrate the deficiency of the classical empirical risk minimization strategy in the heteroscedastic setting. We then argue that resampling techniques such as cross-validation can remedy this problem. The following results are exposed in Arlot and Celisse (2011a).

Deficiency of the empirical risk minimization in the heteroscedastic setting

From the above notations, very simple algebra shows that the expectation of the empirical contrast of f τ (see Eq.(6.1)) can be closely related to the risk as follows.

Lemma 6.1 ([Lemma 1 in Arlot and Celisse (2011a)).

E f -f τ 2 n = f -Π τ f 2 n + V (τ ), (6.2) E L Pn f τ = f -Π τ f 2 n -V (τ ) + 1 n n i=1 σ 2 (t i ), (6.3) where V (τ ) = 1 n Dτ d=1 1 τ d+1 -τ d τ d+1 -1 i=τ d σ 2 (t i ) ≥ 0,
and Π τ f denotes the orthogonal projection of f onto F τ .

In the usual homoscedastic setting (constant variance), the estimation error term V (τ ) = σ 2 D τ /n. On average all the segmentations τ with the same number of segments will have a similar estimation error. Then Eq. ( 6.2) suggests that the best segmentation with D segments is (on average) the one minimizing the approximation error term, that is f -Π τ f 2 n . When considering Eq. ( 6.3), the same remarks justify the empirical contrast minimization strategy to pick up the best segmentation τ (D) from all possible segmentations with D segments in the homoscedastic scenario.

By contrast the estimation error V (τ ) strongly comes into play when the variance is allowed to vary along the time. Due to the minus in front of V (τ ), Eq. ( 6.3) explains why minimizing the empirical contrast over all segmentations with D segments will lead to prefer segmentations maximizing V (τ ) that is, segmentations with change-points in noisy regions at the price of missing some true change-points (at least if D is small enough).

This phenomenon is illustrated by Figure 6.1. In the homoscedastic scenario (Fig. 6.1a), the minimum contrast estimator remains close to the oracle estimator (the estimator minimizing the true loss). But it clearly fails to detect important changes arising around t = 1/2 in the heteroscedastic scenario. Doing so it rather adds false change-points at t close to 0.2 where the noise level is the strongest (Fig. 6.1b).

Cross-validation to choose the best segmentation with a given number of segments

The previous result highlights one deficiency of the empirical contrast minimization strategy for choosing the best segmentation with a given number of segments when the variance is allowed to change (heteroscedastic setting). To remedy this limitation we need a criterion which provides a reasonable estimator of the risk (in particular of its estimation error term) and is possibly fast to compute.

Exploiting closed-form formulas for the LpO estimator in the change-points detection context (see Arlot and Celisse (2011a, Theorem 1) or Celisse (2008, Corollary 3.3.2)), Arlot and Celisse (2011a) consider an alternative minimization strategy where the optimized criterion is the LpO estimator rather than the empirical contrast. Figure 6.1 illustrates the behavior of the estimator chosen by L1O minimization for instance. The striking remarks are that it remains close to the empirical contrast minimizer when the variance remains constant (left picture), but leads to fewer change-points in noisy regions (unlike the latter) when the variance is allowed to vary (right picture). This observed behavior is justified by Proposition 1 in Arlot and Celisse (2011a) where the expectation of the LpO estimator is shown to be almost equal to the risk (Eq. (6.2)) as p becomes close to 1 under reasonable assumptions.

Further simulations experiments are carried out and confirm that the LpO minimization performs equally well as the empirical contrast minimization in the homoscedastic scenario, but clearly provides the better results in the heteroscedastic scenario, with a potential gain up to 20% in the reported simulation results (Arlot and Celisse, 2011a, Table 1 in Section 3). It is also noticeable that this gain in terms of statistical performance is achieved with (almost) the same computation cost as the one of the empirical contrast minimization. This results from the available closed-form formulas of the LpO estimator in the change-point framework (Arlot and Celisse, 2011a, Theorem 1).

New change-points detection procedures based on cross-validation

Choice of the number of change-points

The choice of the number of segments is usually made by optimizing a penalized criterion. For instance in the seminal work of [START_REF] Birgé | Gaussian model selection[END_REF] (further followed by [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF] in the context of change-point detection) 0 -type penalties are derived from concentration inequalities used to quantify the performance of the empirical contrast minimizer. Alternative penalized strategies also exist which are based on the convex relaxation of 0 penalties. For example a Lasso-type procedure is studied in [START_REF] Harchaoui | Multiple change-point estimation with a total variation penalty[END_REF] where the 1 constraint holds on the consecutive coordinates of the estimated mean vector.

Unfortunately neither of the two above approaches do apply to estimators defined as the minimizers of the LpO estimator, which have been observed to improve upon the empirical contrast minimizers in previous Section 6.2.2. Moreover deriving a new penalty applying to such the LpO minimizers would require first to prove concentration inequalities for the LpO estimator, which is a hard task in itself. Therefore these remarks lead Arlot and Celisse (2011a) to rather suggest the V-FCV procedure as a means to choose the number of change-points.

New general scheme for change-point detection procedures

Based on promising results summarized by Fig 4 and Table 2 in Arlot and Celisse (2011a), the authors introduce a general scheme to derive new change-point detection procedures, namely Procedure 1. At Procedure 1 General two-step scheme for change-point detection procedures Input:

observations: X 1 , . . . , X n ∈ X , constant: D max ∈ 1, n -1 .
Step 1: ∀D ∈ 1, D max , compute (by dynamic programming):

τ (D) ∈ argmin τ ∈T D n {crit 1 (τ )} and crit 1 ( τ (D)) (T D n : segmentations with D segments)
Step 2:

Find: D = argmin 1≤D≤Dmax {crit 2 ( τ (D))}.
Output: sequence of change-points: τ = τ ( D).

Step 1 of this general scheme, crit 1 (τ ) is used to find the best candidate segmentation τ (D) for each number D of segments. At Step 2, crit 2 is rather concerned by estimating L P f τ as tightly as possible to choose the best possible number of segments D.

With crit 1 (τ ) = L Pn f τ and crit 2 (τ ) = L P f τ , we would get an "oracle" procedure which recovers the best (oracle) segmentation from the collection of minimum contrast estimators.

With crit 1 (τ ) = L Pn f τ and crit 2 (τ ) = L Pn f τ + σ 2 Dτ n 5 + 2 log n Dτ , we (almost) recover the procedure used by [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF]. Let us notice that at Step 1 of the procedure, segmentations are gathered according to their numbers of segments D. In this latter setting, this turns out to be particularly relevant for solving the resulting optimization problem where the dynamic programming algorithm is used.

Finally the change-point procedure advocated by Arlot and Celisse (2011a, Section 5) consists in choosing crit 1 (τ ) to be the L1O estimator and crit 2 (τ ) to be the V-FCV estimator. From their extensive simulation experiments (Arlot and Celisse, 2011a, Section 5.2), one can draw two main conclusions. First, this new procedure achieves the overall best results even if it can be sometimes outperformed by an other competitor (which changes depending on the simulation settings). Second, regarding the possible choice of the LpO estimator as crit 1 (with p > 1), it seems that increasing p can sometimes improve the quality of the candidate segmentations. But the final results (after V-FCV) do not allow to identify any significant trend in that direction (Supplementary material of Arlot and Celisse, 2011a).

Conclusion

The empirical contrast minimization can fail (even when the number of segments is known) to precisely recover the change-point locations as long as the variance is allowed to change along the time (heteroscedastic setting). The new LpO minimization strategy appears as a reliable alternative to choose the change-points location since it remains robust to heteroscedasticity while being computationally efficient (by use of closed-form formulas).

In the most general setting where the number of segments is not known, it arises that V-FCV is an efficient and versatile tool which remains robust to heteroscedasticity unlike more classical penalties designed to exploit the homoscedastic setting such as the one of [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF].

This leads to conclude that without knowing in advance if the variance remains constant or not, successively applying LpO (crit 1 in Procedure 1) and then V-FCV (crit 2 in Procedure 1) should be used especially when false change-points are to be avoided. However this higher robustness is achieved at the price of increasing the amount of computation time from O(n 2 ) to O(n 2 • V ), which can become a limiting factor as n gets large. Let us still mention that parallel computing could be used for instance to overcome this difficulty, even if it still excludes considering large values of V (for instance V = n).

6.3 Changes in the full distribution and complex objects

Context

As explained at the beginning of Section 6.2, a large part of the literature on change-point detection deals with changes in the mean and/or variance of observations in R or R d . To this end, parametric models are often involved to derive change-point detection procedures. For instance, [START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF], [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF], [START_REF] Picard | Joint segmentation of multivariate gaussian processes using mixed linear models[END_REF] and [START_REF] Geneus | A changepoint detection method for profile variance[END_REF] make a Gaussian assumption, while [START_REF] Frick | Multiscale change point inference[END_REF] and Cleynen and Lebarbier (2014b) consider an exponential family.

The challenging problem of detecting abrupt changes in the full distribution of the data has been recently addressed in the nonparametric setting. However, the corresponding procedures suffer several limitations since they are limited to real-valued data or they assume that the number of true change-points is known. For instance, [START_REF] Zou | Nonparametric maximum likelihood approach to multiple change-point problems[END_REF] design a strategy based on empirical cumulative distribution functions that allows to recover an unknown number of change-points by use of BIC, but only applies to R-valued data. A similar conclusion applies to the strategy of [START_REF] Matteson | A nonparametric approach for multiple change point analysis of multivariate data[END_REF], which is moreover time-consuming due to an intensive permutation use, and only justified in an asymptotic setting. The kernel-based procedure proposed by [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF] enables to deal with complex data (not necessarily vectors). But it assumes the number of change-points to recover is known, which reduces its practical interest when no such information is available. Finally, many of these procedures are only theoretically grounded by asymptotic results, which makes their finite-sample performance questionable.

Other attempts have been made to design change-point detection procedures allowing to deal with complex data (that are not necessarily vectors). However, the resulting procedures do not allow to detect more than one or two changes arising in particular features of the distribution. For instance, Chen and Zhang (2015) describe a strategy based on a dissimilarity measure between individuals to compute a graph from which a statistical test allows to detect only one or two change-points. For a graph-valued time series, Wang et al. (2014) design specific scan statistics to test whether one change arises in the connectivity matrix.

Main contributions.

In this work we first describe a new efficient kernel-based multiple change-point detection procedure (KCP) allowing to deal with univariate, multivariate or complex data (DNA sequences or graphs, for instance) as soon as a positive semidefinite kernel can be defined for them. Among several assets, this procedure is nonparametric and does not require to know the true number of change-points in advance. Furthermore, it allows to detect abrupt changes arising in the full distribution of the data by using a characteristic kernel; it can also focus on changes in specific features of the distribution by choosing an appropriate kernel.

Secondly, our procedure (KCP) is theoretically grounded with a finite-sample optimality result, namely an oracle inequality in terms of quadratic risk, stating that its performance is almost the same as that of the best one within the class we consider (Theorem 6.1). A crucial point is that Theorem 6.1 holds true for any value of the sample size n; in particular it can be smaller than the dimensionality of the data. Unlike previous oracle inequalities in the change-point detection framework, our result requires neither the variance to be constant nor the data to be Gaussian. One main ingredient in the proof is derivation of a new concentration inequality for the quadratic norm of sums of independent Hilbert-valued vectors with exponential tails.

Motivating examples are provided in what follows to highlight the wide applicability of our procedure to various important settings.

Example 6.3. The set X is R or R d , and no assumption is made on the changes in the sequence (P Xi ) 1≤i≤n . For instance, when data are centered and normalized, as in the audio-track example [START_REF] Rabiner | Introduction to digital signal processing[END_REF], the mean and the variance of the X i can be constant, and only higher-order moments of (P Xi ) 1≤i≤n are changing. Only a few recent works deal with (an unknown number of ) multiple change-points in a fully nonparametric framework: [START_REF] Zou | Nonparametric maximum likelihood approach to multiple change-point problems[END_REF] for X = R, [START_REF] Matteson | A nonparametric approach for multiple change point analysis of multivariate data[END_REF] for X = R d . Note that assuming X = R and adding some further restrictions on the maximal order of the moments for which a change can arise in the sequence (P Xi ) 1≤i≤n , it is nevertheless possible to consider the multivariate sequence (p j (X i )) 0≤j≤d 1≤i≤n

, where p j is a polynomial of degree j for j ∈ {0, . . . , d}, and to use a method made for detecting changes in the mean (Example 6.1). For instance with R-valued data, one can take p j (X) = X j for every 1 ≤ j ≤ d, or p j equal to the j-th Hermite polynomial, as proposed by [START_REF] Lajugie | Large-margin metric learning for constrained partitioning problems[END_REF].

Example 6.4. The set X is the d-dimensional simplex (p 1 , . . . , p d ) ∈ [0, 1] d | p 1 + • • • + p d = 1 .
For instance, audio and video data are often represented by histogram features [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF][START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF][START_REF] Rabiner | Introduction to digital signal processing[END_REF]. In such cases, it is a bad idea to do as if X were R d -valued, since the Euclidean norm on R d is usually a bad distance measure between histogram data.

Example 6.5. The set X is a set of graphs. For instance, the X i can represent a social network [START_REF] Kossinets | Empirical analysis of an evolving social network[END_REF] or a biological network [START_REF] Curtis | Enabling dynamic network analysis through visualization in TVNViewer[END_REF] that is changing over time [START_REF] Chen | Graph-based change-point detection[END_REF]. Then, detecting meaningful changes in the structure of a time-varying network is a change-point problem. In the case of social networks, this can be used for detecting the rise of an economic crisis [START_REF] Mcculloh | Detecting Changes in a Dynamic Social Network[END_REF].

Other kinds of data could be considered, such as counting data [START_REF] Alaya | Learning the intensity of time events with change-points[END_REF]Cleynen and Lebarbier, 2014b), qualitative descriptors, as well as composite data, that is, data X i that are mixing several above examples.

The goal of this work is to propose a change-point algorithm that is (i) general enough to handle all these situations (up to the choice of an appropriate similarity measure on X ), (ii) in a non parametric framework, (iii) with an unknown number of change-points, and (iv) that we can analyze theoretically in all these examples simultaneously.

Detecting changes in the distribution with kernels

Our approach for solving the general change-point problem uses positive semidefinite kernels. It can be sketched as follows.

Kernel change-point detection procedure (KCP)

For any integer D ∈ 1, n , the set of sequences of D change-points is defined by

T D n = (τ 1 , . . . , τ D ) ∈ N D / 1 = τ 1 < τ 2 < • • • < τ D < τ D+1 = n + 1 (6.4)
where τ 1 , . . . , τ D are the change-points, and τ D+1 is added for notational convenience. Any τ ∈ T D n is a segmentation (of {1, . . . , n}) into D τ = D segments.

Let k : X × X → R be a positive semidefinite kernel, that is, a measurable function X × X → R such that for any x 1 , . . . , x n ∈ X , the n × n matrix (k(x i , x j )) 1≤i,j≤n is positive semidefinite. Examples of such kernels are given in Section 6.3.2. Then, we measure the quality of any candidate segmentation τ ∈ T D n with the kernel least-squares criterion introduced by [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF]:

L Pn (τ ) = 1 n n i=1 k(X i , X i ) - 1 n D =1   1 τ +1 -τ τ +1 -1 i=τ τ +1 -1 j=τ k(X i , X j )   . (6.5)
In particular when X = R and k(x, y) = xy, we recover the usual least-squares criterion

L Pn (τ ) = 1 n D =1 τ +1 -1 i=τ X i -X τ ,τ +1 -1 2 where X τ ,τ +1 -1 = 1 τ +1 -τ τ +1 -1 j=τ X j .
Note that Eq. (6.9) in Section 6.3.3 provides an equivalent formula for L Pn (τ ), which is helpful for understanding its meaning. Given the criterion (6.5), we cast the choice of τ as a model selection problem (as thoroughly detailed in Section 6.3.3), which leads to Algorithm 2 below, that we now briefly comment on.

• Step 1 of KCP consists in choosing the "best" segmentation with D segments, that is, the minimizer of the kernel least-squares criterion L Pn (•) over T D n , for every D ∈ 1, D max .

Procedure 2 Kernel Change-point Procedure (KCP)

Input: observations: X 1 , . . . , X n ∈ X , kernel: k : X × X → R, constants: c 1 , c 2 > 0 and D max ∈ 1, n -1 .
Step 1: ∀D ∈ 1, D max , compute (by dynamic programming):

τ (D) ∈ argmin τ ∈T D n {L Pn (τ )} and L Pn ( τ (D))
Step 2: find:

D ∈ argmin 1≤D≤Dmax L Pn ( τ (D)) + 1 n c 1 log n -1 D -1 + c 2 D .
Output: sequence of change-points: τ = τ D .

•

Step 2 of KCP chooses D by model selection, using a penalized empirical criterion. A major contribution of this work lies in the building and theoretical justification of the penalty n -1 c 1 log n-1 D-1 + c 2 D , see Section 6.3.3; a simplified penalty, of the form D n c 1 log n D + c 2 , would also be possible, see Section 6.3.3.

• Practical issues (computational complexity and choice of constants c 1 , c 2 , D max ) are discussed in Section 6.3.2. Let us only emphasize here that KCP is computationally tractable; its most expensive part is the minimization problem of Step 1, which can be done by dynamic programming (see [START_REF] Celisse | New efficient algorithms for kernel change-point detection[END_REF][START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF].

Examples of kernels

KCP can be used with various sets X (not necessarily vector spaces) as long as a positive semidefinite kernel on X is available. An important issue is to design relevant kernels, that are able to capture important features of the data for a given change-point problem, including non-vectorial data -for instance, simplicial data (histograms), texts or graphs (networks), see Section 6.1. The question of choosing a kernel is discussed in Section 6.3.5. Classical kernels can be found in the books by [START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF], Shawe-Taylor and Cristianini (2004) and Schölkopf et al. (2004) for instance. Let us mention a few of them:

• When X = R d , k lin (x, y) = x, y R d defines the linear kernel. When d = 1, KCP then coincides with the algorithm proposed by [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF].

• When X = R d , k G h (x, y) = exp[-x -y 2 /(2h 2 )
] defines the Gaussian kernel with bandwidth h > 0, which is used in the experiments of Section 6.3.4.

• When X = R d , k e h (x, y) = exp( x, y R d /h
) defines the exponential kernel with bandwidth h > 0. Note that, unlike the Gaussian kernel, the exponential kernel is not translation-invariant.

• When X = R, k H h (x, y) = 5 j=1 H j,h (x)H j,h (
y), corresponds to the Hermite kernel, where H j,h (x) = 2 j+1 √ πj!e -x 2 /(2h 2 ) (-1) j e -x 2 /2 (∂/∂x) j e -x 2 /2 denotes the j-th Hermite function with bandwidth h > 0. This kernel is used in Section 6.3.4.

• When X is the d-dimensional simplex as in Example 6.4, the χ 2 -kernel can be defined by

k χ 2 h (x, y) = exp -1 h•d d i=1 (xi-yi) 2 xi+yi
for some bandwidth h > 0. An illustration of its behavior is provided in the simulation experiments of Sections 6.3.4.

Note that more generally, [START_REF] Sejdinovic | Equivalence of distance-based and RKHS-based statistics in hypothesis testing[END_REF] proved that positive semidefinite kernels can be defined on any set X for which a semimetric of negative type is used to measure closeness between points. The so-called energy distance between probability measures is an example [START_REF] Matteson | A nonparametric approach for multiple change point analysis of multivariate data[END_REF]. In addition, specific kernels have been designed for various kinds of structured data, including all the examples of Section 6.1 [START_REF] Cuturi | Semigroup kernels on measures[END_REF][START_REF] Shervashidze | Scalable graph kernels[END_REF].

Practical issues

Computational complexity. The discrete optimization problem at Step 1 of KCP is apparently hard to solve since, for each D, there are n-1 D-1 segmentations of {1, . . . , n} into D segments. Fortunately, as suggested by [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF], this optimization problem can be solved efficiently by dynamic programming [START_REF] Auger | Algorithms for the optimal identification of segment neighborhoods[END_REF][START_REF] Kay | Fundamentals of statistical signal processing: detection theory[END_REF]: denoting by C k the cost of computing k(x, y) for some given x, y ∈ X , the computational cost of Step 1 then is O C k n 2 + D max n 4 in time and O(D max n + n 2 ) in space. Note that the O D max n 4 part of the time complexity results from the necessary computation of the so-called cost matrix. The coefficient (i, j) of this n × n cost matrix is equal to the statistical cost of the segment i, j -1 , which involves itself summing over a quadratic number of terms of the Gram matrix. By a careful optimization of the interplay between dynamic programming and the cost matrix computation, [START_REF] Celisse | New efficient algorithms for kernel change-point detection[END_REF] reduce the computational complexity to O((

C k + D max )n 2 ) in time and O(D max n) in space.
For given constants D max and c 1 , c 2 , Step 2 is straightforward since it consists in a minimization problem among D max terms already stored in memory. Therefore, the overall complexity of KCP is at most

O (C k + D max )n 2 in time and O(D max n) in space.
Setting the constants c 1 , c 2 . At Step 2 of KCP, two constants c 1 , c 2 > 0 appear in the penalty term. Theoretical guarantees (Theorem 6.1 in Section 6.3.3) suggest to take c 1 = c 2 = c large enough, but the lower bound on c in Theorem 6.1 is pessimistic, and the optimal value of c certainly depends on unknown features of the data such as their "variance", as discussed after Theorem 6.1. In practice the constants c 1 , c 2 must be chosen from data. To do so, we propose a fully data-driven method, based upon the "slope heuristic" [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF], that is explained in Section 6.3.4. Another way of choosing c 1 , c 2 is described in supplementary material.

Setting the constant D max . KCP requires to specify the maximal dimension D max of the considered segmentations, a choice that has three main consequences. First, the computational complexity of KCP is affine in D max , as discussed above. Second, if D max is too small -smaller than the number of true change-points that can be detected-, the segmentation τ provided by the algorithm will necessarily be too coarse. Third, when the slope heuristic is used for choosing c 1 , c 2 , taking D max larger than the true number of change-points might not be sufficient: better values for c 1 , c 2 can be obtained by taking D max larger, up to n. From our experiments, it seems that D max ≈ n/ √ log n is large enough to provide good results.

Theoretical analysis

We now provide theoretical guarantees for KCP. We start by reformulating it in an abstract way, which enlightens how it works.

Abstract formulation of KCP

Let H = H k denote the reproducing kernel Hilbert space (RKHS) associated with the positive semidefinite kernel k : X × X → R. The canonical feature map Φ : X → H is then defined by Φ(x) = k(x, •) ∈ H for every x ∈ X . A detailed presentation of positive semidefinite kernels and related notions can be found in several books [START_REF] Cucker | Learning Theory: An Approximation Theory Viewpoint[END_REF][START_REF] Scholkopf | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond[END_REF]Steinwart and Christmann, 2008b).

Let us define Y

i = Φ(X i ) ∈ H for every i ∈ {1, . . . , n}, Y = (Y i ) 1≤i≤n ∈ H n , T n = n D=1 T D
n the set of segmentations -see Eq. (6.4)-, and for every τ ∈ T n ,

F τ = f = (f 1 , . . . , f n ) ∈ H n s.t. f τ -1 +1 = • • • = f τ ∀1 ≤ ≤ D τ , (6.6)
which is a linear subspace of H n . We also define on H n the canonical scalar product by f, g = n i=1 f i , g i H for f, g ∈ H n , and we denote by • the corresponding norm. Then, for any g ∈ H n ,

Π τ g = argmin f ∈Fτ f -g 2 (6.7)
is the orthogonal projection of g ∈ H n onto F τ , and satisfies

∀g ∈ H n , ∀1 ≤ ≤ D τ , ∀i ∈ τ , τ +1 -1 , (Π τ g) i = 1 τ +1 -τ τ +1 -1 j=τ g j .
(6.8)

The proof of this statement has been deferred to Appendix Following [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF], the empirical risk L Pn (τ ) defined by Eq. (6.5) can be rewritten as

L Pn (τ ) = 1 n Y -µ τ 2 where µ τ = Π τ Y , (6.9)
as proved in Appendix.

For each D ∈ 1, D max , Step 1 of KCP consists in finding a segmentation τ (D) in D segments such that

τ (D) ∈ argmin τ ∈T D n Y -µ τ 2 = argmin τ ∈T D n inf f ∈Fτ n i=1 Φ(X i ) -f i 2 ,
which is the "kernelized" version of the classical least-squares change-point algorithm [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF].

Since the penalized criterion of Step 2 is similar to that of [START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF] and [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF], we can see KCP as a "kernelization" of these penalized least-squares change-point procedures.

Let us emphasize that building a theoretically-grounded penalty for such a kernel least-squares changepoint algorithm is not straightforward. For instance, we cannot apply the model selection results by [START_REF] Birgé | Gaussian model selection[END_REF] that were used by [START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF] and [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF]. Indeed, a Gaussian homoscedastic assumption is not realistic for general Hilbert-valued data, and we have to consider possibly heteroscedastic data for which we assume only that Y i = Φ(X i ) is bounded in Hsee Assumption (Db) in Section 6.3.3. Note that unbounded data X i can satisfy Assumption (Db), for instance by choosing a bounded kernel such as the Gaussian or Laplace ones. In addition, dealing with Hilbert-valued random variables instead of (multivariate) real variables requires a new concentration inequality.

Intuitive analysis

Section 6.3.3 shows that KCP can be seen as a kernelization of change-point algorithms focusing on changes of the mean of the signal [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF], for instance). Therefore, KCP is looking for changes in the "mean" of Y i = Φ(X i ) ∈ H, provided that such a notion can be defined.

If H is separable and [START_REF] Ledoux | Probability in Banach spaces[END_REF], also called the mean element of P Xi , by

E[k(X i , X i )] < +∞, we can define the (Bochner) mean µ i ∈ H of Φ(X i ) (Ledoux
∀g ∈ H, µ i , g H = E g(X i ) = E Y i , g H . (6.10)
Then, we can write

∀1 ≤ i ≤ n, Y i = µ i + ε i ∈ H where ε i = Y i -µ i .
The variables (ε i ) 1≤i≤n are independent and centered -that is, ∀g ∈ H, E[ ε i , g H ] = 0. So, we can understand µ τ as the least-squares estimator over

F τ of µ = (µ 1 , . . . , µ n ) ∈ H n .
An interesting case is when k is a characteristic kernel [START_REF] Fukumizu | Kernel measures of conditional dependence[END_REF], or equivalently, when H k is probability-determining (Fukumizu et al., 2004a,b). Then any change in the distribution P Xi induces a change in the mean element µ i . In such settings, we can expect KCP to be able to detect any change in the distribution P Xi , at least asymptotically. For instance the Gaussian kernel is characteristic (Fukumizu et al., 2004b, Theorem 4), and general sufficient conditions for k to be characteristic are known [START_REF] Sriperumbudur | Universality, characteristic kernels and RKHS embedding of measures[END_REF][START_REF] Sriperumbudur | Hilbert space embeddings and metrics on probability measures[END_REF].

CHAPTER 6. MULTIPLE CHANGE-POINT DETECTION

Notation and assumptions

We assume that H is separable, which is kind of a minimal assumption for two reasons: it allows to uniquely define the mean element -see Eq. (6.10)-, and most reasonable examples satisfy this requirement (Dieuleveut and Bach, 2014, p. 4). Let us further assume

∃M ∈ (0, +∞) , ∀i ∈ {1, . . . , n} , Y i 2 H = Φ(X i ) 2 H = k(X i , X i ) ≤ M 2 a.s. (Db)
For every 1 ≤ i ≤ n, we also define the "variance" of Y i by

v i = E Φ(X i ) -µ i 2 H = E k(X i , X i ) -µ i 2 H = E k(X i , X i ) -k(X i , X i ) , (6.11)
where X i is an independent copy of X i , and v max = max 1≤i≤n v i . Let us make a few remarks.

• If (Db) holds true, then the mean element

µ i exists since E[ k(X i , X i )] < ∞, the variances v i are finite and smaller than v max ≤ M 2 . Besides Y i admits a covariance operator Σ i that is trace-class with v i = tr(Σ i ) -µ i 2 H .
• If k is translation invariant, that is, X is a vector space and k(x, x ) = k(x-x ) for every x, x ∈ X , and some measurable function k : X → R, then (Db) holds true with M 2 = k(0) and v i = k(0) -µ i 2 H . For instance the Gaussian kernel is translation invariant (see Section 6.3.2).

• Let us consider the case of the linear kernel (x,

y) → x, y on X = R d . If E[ X i 2 R d ] < ∞, then, v i = tr(Σ i ) -µ i 2 R d
where Σ i is the covariance matrix of X i . In addition, (Db) holds true if and only if X i R d ≤ M a.s. for all i.

Oracle inequality for KCP

Similarly to the results of [START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF] and [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF] in the one-dimensional case, we state below a non-asymptotic oracle inequality for KCP. First, we define the quadratic risk of any µ ∈ H n as an estimator of µ by

R (µ) = 1 n µ -µ 2 = 1 n n i=1 µ i -µ i 2 H .
Theorem 6.1. We consider the framework and notation introduced in Sections 6.1-6.3.3. Let C ≥ 0 be some constant. Assume that (Db) holds true and that pen : T n → R is some penalty function satisfying

∀τ ∈ T n , pen(τ ) ≥ CM 2 n log n -1 D τ -1 + D τ . (6.12)
Then, some numerical constant L 1 > 0 exists such that the following holds: if C ≥ L 1 , for every y ≥ 0, an event of probability at least 1 -e -y exists on which, for every τ ∈ argmin τ ∈Tn {L Pn (τ ) + pen(τ )} , (6.13)

we have

R ( µ τ ) ≤ 2 inf τ ∈Tn {R ( µ τ ) + pen(τ )} + 83yM 2 n . (6.14)
In a few words, the idea is to take a penalty such that the empirical criterion L Pn (τ ) + pen(τ ) in Eq. (6.13) mimics (approximately) the oracle criterion R ( µ τ ). At least, the penalty must be large enough so that L Pn (τ ) + pen(τ ) ≥ R ( µ τ ) holds true simultaneously for all τ ∈ T n .

Oracle inequality rather than consistency result Theorem 6.1 applies to the segmentation τ provided by KCP when c 1 , c 2 ≥ L 1 M 2 . Theorem 6.1 shows that µ τ estimates well the "mean" µ ∈ H n of the transformed time series Y 1 = Φ(X 1 ), . . . , Y n = Φ(X n ). Such a non-asymptotic oracle inequality is the usual way to theoretically validate a model selection procedure [START_REF] Birgé | Gaussian model selection[END_REF]. This justifies the use of Eq. (6.14) to validate our new (model selection-based) change-point detection procedure called KCP. Moreover defining the performance of τ as the quadratic risk of µ τ used to estimate µ allows us to prove that KCP works well for finite sample size and for a set X that can have a large dimensionality (possibly much larger than the sample size n). The consistency of KCP for estimating the change-point locations, which is outside the scope of this work, is discussed in Section 6.3.5 and [START_REF] Garreau | Consistent change-point detection with kernels[END_REF].

Relaxations The constant 2 in front of the first term in Eq. ( 6.14) has no special meaning, and could be replaced by any quantity strictly larger than 1, at the price of enlarging L 1 and 83.

The value 2L 1 M 2 suggested by Theorem 6.1 for the constants c 1 , c 2 within KCP should not be used in practice because it is likely to lead to a conservative choice for two reasons. First, the minimal value L 1 for the constant C is derived from non optimized numerical constants arising from concentration inequalities. Second, the constant M 2 in the penalty is probably pessimistic in several frameworks. For instance with the linear kernel and Gaussian data belonging to X = R, similar oracle inequalities have been proved with M 2 replaced by the residual variance [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF]. In practice, as we do in the experiments of Section 6.3.4, we recommend to use a data-driven value for the leading constant C in the penalty, as explained in Section 6.3.2.

A nice feature of Theorem 6.1 is that it holds under mild assumptions: we only need the data X i to be independent and to have (Db) satisfied. Compared to previous results [START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF][START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF], we do not need the data to be Gaussian or homoscedastic. Furthermore, the independence assumption can certainly be relaxed by proving concentration inequalities similar to Propositions 1 and 3 in [START_REF] Arlot | Kernel change-point detection[END_REF] for some dependent X i .

Connexions with similar results

In the particular setting where X = R and k is the linear kernel (x, y) → xy, Theorem 6.1 provides an oracle inequality similar to the one proved by [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF] for Gaussian and homoscedastic real-valued data. The price to pay for extending this result to heteroscedastic Hilbert-valued data is rather mild: we only assume (Db) and replace the residual variance by M 2 .

A few oracle inequalities have been proved for change-point procedures, for real-valued data with a multiplicative penalty [START_REF] Baraud | Gaussian model selection with an unknown variance[END_REF], for discrete data [START_REF] Akakpo | Estimating a discrete distribution via histogram selection[END_REF], for counting data with a total-variation penalty [START_REF] Alaya | Learning the intensity of time events with change-points[END_REF], for counting data with a penalized maximum-likelihood procedure (Cleynen and Lebarbier, 2014b) and for data distributed according to an exponential family (Cleynen and Lebarbier, 2014a). Among these oracle inequalities, only the result by [START_REF] Akakpo | Estimating a discrete distribution via histogram selection[END_REF] is more precise than Theorem 6.1 (there is no log(n) factor compared to the oracle loss), at the price of using a smaller (dyadic) collection of possible segmentations, hence a worse oracle performance in general.

Experiments on synthetic data

This section summarizes some of the results of experiments on synthetic data that illustrate the performance of KCP. A thorough presentation of these experiments and results can be found in Arlot et al. (2012, Section 6).

Data generation process

Three scenarios have been considered with the sample size n = 1 000, the true number of segments D = 11, and N = 500 independent repetitions: (i) real-valued data with a changing (mean,variance), (ii) real-valued data with constant (mean,variance), and (iii) histogram-valued data (with 20 bins) as in Example 6.4. Figure 6.2 depicts one example for each scenario.

Parameters of KCP

For each sample, we apply our kernel change-point procedure (KCP, that is, Procedure 2) with the following choices for its parameters. We always take D max = 100. For the first two scenarios, we consider three kernels:

(i) The linear kernel k lin (x, y) = xy.

(ii) The Hermite kernel given by k H σ H (x, y) defined in Section 6.3.2. In Scenario 1, σ H = 1. In Scenario 2, σ H = 0.1.

(iii) The Gaussian kernel k G σ G defined in Section 6.3.2. In Scenario 1, σ G = 0.1. In Scenario 2, σ G = 0.16.

For Scenario 3, we consider the χ 2 kernel k χ 2 0.1 (x, y) defined in Section 6.3.2, and the Gaussian kernel

k G σ G with σ G = 1.
For choosing the constants c 1 , c 2 arising from Step 2 of KCP, we use the "slope heuristic" method, and more precisely a variant proposed by Lebarbier (2002, Section 4.3.2) for the calibration of two constants for change-point detection. We first perform a linear regression of L Pn ( τ (D)) against 1/n • log n-1 D-1 and D/n for D ∈ [0.6×D max , D max ]. Then, denoting by s 1 , s 2 the coefficients obtained, we define c i = -α s i for i = 1, 2, with α = 2. The slope heuristic has been justified theoretically in various settings (for instance by Arlot and Massart, 2009b, for regressograms) and is supported by numerous experiments [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF], including change-point detection [START_REF] Lebarbier | Quelques approches pour la détection de ruptures à horizon fini[END_REF][START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF]. The intuition behind the slope heuristic is that the optimal amount of penalization needed for avoiding to overfit with τ ∈ argmin τ {L Pn (τ )+pen(τ )} is (approximately) proportional to the minimal penalty: pen optimal (τ ) ≈ α pen minimal (τ ) for some constant α > 1, equal to 2 in several settings. The linear regression step described above corresponds to estimating the minimal penalty:

pen minimal (τ ) ≈ -s 1 • 1 n log n -1 D τ -1 -s 2 D τ n •
Then, multiplying it by α = 2 leads to an estimation of the optimal penalty, which has been done in our experiments.

Results

We now summarize the results of our experiments. Step 2 of KCP Step 2 of KCP is illustrated by Figures 6.3b Overall performance to recover the change-points The overall performance of KCP as a change-point detection procedure is illustrated by Figure 6.3d. Each true change-point has a probability larger than 0.5 to be recovered exactly by τ . This probability becomes even larger than 80% is one allows small mistakes (three positions before or after a true change-point). Importantly, such figures are obtained without overestimating much the number of change-points, according to Figure 6.3c.

Illustration of the KCP performance

Influence of the kernel

Comparison of three kernels in Scenario 2. Scenario 2 proposes a more challenging change-point problem with real-valued data: the distribution of the X i changes while the mean and the variance remain constant. The performance of KCP with three kernels -k lin , k H and k G -is shown on Figure 6 linear kernel k lin corresponds to the classical least-squares change-point algorithm [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF], which is designed to detect changes in the mean, hence it should fail in Scenario 2. KCP with the Hermite kernel k H is a natural "hand-made" extension of this classical approach, since it corresponds to applying the least-squares change-point algorithm to the feature vectors (H j,h (X i )) 1≤j≤5 . By construction, it should be able to detect changes in the first five moments on the X i . On the contrary, taking k = k G the Gaussian kernel fully relies on the versatility of KCP, which makes possible to consider (virtually) infinitedimensional feature vectors k G (X i , •). Since k G is characteristic, it should be able to detect any change in the distribution of the X i .

In order to compare these three kernels within KCP, let us first assume that the number of changepoints is known, hence we can estimate τ with τ (D ), where D * is the true number of segments. Then,Figures 6.4a,6.4b and 6.4c show that k lin , k H and k G behave as expected: k lin seems to put the changepoints of τ (D ) uniformly at random over {1, . . . , n}, while k H and k G are able to localize the true change-points with a rather large probability of success. The Gaussian kernel here shows a significantly better detection power, compared to k H : the frequency of exact detection of the true change-points is between 38 and 47% with k G , and between 17 and 29% with k H . The same holds when considering blocks of size 6: k G then detects the change-points with probability 70 to 79%, while k H exhibits probabilities between 58 and 62%.

Since k G is a characteristic kernel, these results suggest that KCP with a characteristic kernel k might be more versatile than classical least-squares change-point algorithms and their extensions. A more detailed simulation experiment would nevertheless be needed to confirm this hypothesis. We also refer to Section 6.3.5 for a discussion on the choice of k for a given change-point problem.

Structured data. Figure 6.5 illustrates the performance of KCP on some histogram-valued data (Scenario 3). Since a d-dimensional histogram is also an element of R d , we can analyze such data either with a kernel taking into account the histogram structure (such as k χ 2 ) or with a usual kernel on R d (such as k lin or k G ; here, we consider k G , which seems more reliable according to our experiments in Scenarios 1 and 2). Assuming that the number of change-points is known, taking k = k χ 2 yields quite good results according to Figure 6.5a, at least in comparison with k = k G (Figure 6.5b). Similar results hold with a fully data-driven number of change-points, as shown by Hence, choosing a kernel such as k χ 2 , which takes into account the histogram structure of the X i , can improve much the change-point detection performance, compared to taking a kernel such as k G , which ignores the structure of the X i .

Let us emphasize that Scenario 3 is quite challenging -changes are hard to distinguish on Figure 6.2c-, which has been chosen on purpose. Comparison to other procedures Complementary experimental results show that:

• linear penalties -that is, of the form CD/n, C > 0-similar to AIC (which would correspond to C = σ 2 ) and BIC (for which C = log(n)σ 2 /2) lead to overestimate the true number of segments, leading to include false positives with a large probability in Scenarios 1 and 2. Therefore, such a linear penalty seems less reliable than the refined shape proposed in the definition of KCP,

• the E-divisive procedure (ED) [START_REF] Matteson | A nonparametric approach for multiple change point analysis of multivariate data[END_REF] applied in Scenarios 1-2 (made for X = R d ) provides much more conservative results than KCP with k = k G , with a detection power much smaller than the one of KCP (detection frequencies 2 to 5 times lower for ED in Scenario 2 for instance).

Overall, KCP with k = k G clearly outperforms ED in Scenarios 1-2 for at least tow reasons: (i) ED uses a different similarity measures than ours; (ii) ED relies on a greedy strategy, in which τ (D +1) is obtained from τ (D) by adding one change-point, so that any mistake at the beginning of the process impacts the final segmentation.

Conclusion

The new kernel change-point procedure called KCP (Procedure 2) is based on a penalization procedure generalizing the one of [START_REF] Comte | A new algorithm for fixed design regression and denoising[END_REF] and [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF] to RKHS-valued data. Such an extension significantly broadens the range of possible applications of the algorithm, since it can deal with complex or structured data, and it can detect changes in the full distribution of the data -not only the mean or the variance. The new theoretical tools developed in the work -mostly, a concentration inequality for some function of RKHS-valued random variables-could be useful in other settings, such as clustering in reproducing kernel Hilbert spaces or functional data analysis.

Identification of the change-point locations

A natural question for a change-point procedure is its consistency for estimating the true change-point locations τ . The goal is to prove that d( τ , τ ) tends to zero almost surely as n tends to infinity, where d is some distance on T n (for instance Frobenius or Hausdorff). Many works prove such consistency results for other change-point procedures in various settings (for instance, [START_REF] Frick | Multiscale change point inference[END_REF][START_REF] Lavielle | Least-squares estimation of an unknown number of shifts in a time series[END_REF][START_REF] Matteson | A nonparametric approach for multiple change point analysis of multivariate data[END_REF][START_REF] Yao | Estimating the number of change-points via Schwarz criterion[END_REF]. Answering this question for KCP is the scope of [START_REF] Garreau | Consistent change-point detection with kernels[END_REF], which establishes that KCP is indeed consistent under mild assumptions.

Choosing the kernel k

A major practical and theoretical question about KCP is the choice of the kernel k. Fully answering this question is beyond the scope of the present work, but we can already provide a few guidelines from theoretical and experimental results.

First, simulation experiments show that the performance can strongly vary with k. They suggest that using a characteristic kernel yields a more versatile procedure when the goal is to detect changes in the full distribution of the data. Nevertheless any characteristic kernel used with a clearly bad choice of the bandwidth h can lead to a poor performance of KCP. Furthermore, for a given setting, a non characteristic kernel can be a good choice: for detecting changes in the mean of X i ∈ R d , k lin is known to work well [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF].

The problem of choosing a kernel has been considered for many different tasks in the machine learning literature, for instance choosing the best kernel for a two-sample or an homogeneity test. For choosing the bandwidth h of a Gaussian kernel, a classical heuristic is to take h equal to some median of ( X i -X j H ) i<j (see Gretton et al., 2012a, Section 8, and references therein). This idea can be used for change-point detection with KCP. A procedure for choosing the best convex combination of a finite number of kernels has been proposed by Gretton et al. (2012b), with the goal of building a powerful two-sample test. Another idea for combining several kernels, for instance the family k G h : h > 0 , has been studied by [START_REF] Sriperumbudur | Kernel choice and classifiability for RKHS embeddings of probability distributions[END_REF] for homogeneity and independence tests. Roughly, the idea is to replace the MMD test statistics -which depends on a kernel k-by its supremum over the considered family of kernels. Nevertheless, the extension of these two ideas to change-point detection with KCP does not seem straightforward.

Heteroscedasticity of data in H

A possible drawback of KCP is that it does not take into account the fact that the variance v i of Y i = Φ(X i ) can change with i: in general, the Y i s are heteroscedastic. In the case of real-valued data and the linear kernel k lin , Arlot and Celisse (2011b) have shown that heteroscedastic data can make KCP fail, and that this failure cannot be fixed by changing the penalty used at Step 2: all the segmentations τ (D) produced at Step 1 can be wrong.

When heteroscedasticity is a problem for KCP, which probably occurs for some kernels beyond k lin , we can think of combining KCP with the ideas of Arlot and Celisse (2011b), that is, replacing the empirical risk and the penalized criterion in Steps 1 and 2 of KCP by cross-validation estimators of the risk R ( µ τ ).

Efficient computations with reproducing kernels

Context As already discussed in Section 6.3.2, the computational complexity of KCP (Procedure 2) is strongly related to the way the dynamic programming algorithm [START_REF] Auger | Algorithms for the optimal identification of segment neighborhoods[END_REF][START_REF] Kay | Fundamentals of statistical signal processing: detection theory[END_REF] is used at Step 1.

In the context of detecting changes arising in some parameters of the distribution of real-valued observations, the classical use of dynamic programming (made for instance by [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF] with changes of the mean) has a O(n 2 ) time complexity. During the last years, several successful attempts have been made to reduce this time complexity by means of pruning strategies. Among others, [START_REF] Rigaill | A pruned dynamic programming algorithm to recover the best segmentations with 1 to k max changepoints[END_REF] solves the optimization problem at Step 1 with a reduced O(n log n) time complexity on average, while [START_REF] Killick | Optimal detection of changepoints with a linear computational cost[END_REF] provide a linear time version of the dynamic programming algorithm to solve a slightly different (but closely related) problem. Let us also mention the recent work by Maidstone et al. (2017a) which focuses on changes in the slope of a continuous piecewise-linear regression function. However these recent improvements are unfortunately not applicable in the present kernel-based framework.

Therefore in the present context of Procedure 2 (where reproducing kernels are involved), Step 1 suffers one strong practical limitation, which can be formulated in terms of a trade-off between computational and storage costs:

• The optimization problem at Step 1 can be solved by first computing and storing a n × n cost matrix, which induces a O(n 2 ) space complexity. However once this computation has been done, the "classical" dynamic programming algorithm applied to this cost matrix achieves its usual O(n 2 ) time complexity. With large sample sizes such as n ≈ 10 6 , storing such a huge matrix can be infeasible.

• Conversely, the n × n cost matrix can be computed on the fly within the dynamic programming algorithm. This avoids the storage of a n × n matrix, but increases the overall time complexity of the dynamic programming algorithm from O(n 2 ) to O(n 4 ) (with the classical implementation of Step 1 suggested for instance in [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF]). In particular such a slow runtime prevents us from considering huge sample sizes and therefore limits the practical applicability of KCP.

This inflated time complexity is not limited to the change-point detection problem. On the contrary it is rather ubiquitous in all learning procedures where reproducing kernels are involved (see for instance [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF] for an extended discussion on that problem).

Contribution

In what follows two computationally efficient strategies are described, which overcome the previous practical limitation and allow to deal with large sample sizes up to n ≈ 10 6 . First, in Section 6.4.1, a new implementation of the Step 1 in Procedure 2 is discussed. It stems from a step-by-step analysis of the former implementation. In particular, it outputs the exact solution to the optimization problem with a complexity of order O(n 2 ) in time and O(n) in space. This enhanced implementation allows us to deal with large sample sizes up to n ≈ 10 5 in reasonable time.

Second, Section 6.4.2 is mainly concerned with the change-point detection problem from a huge sample size (typically of order n ≈ 10 6 ). This is achieved by means of a new linear-time procedure returning an approximate solution to the optimization problem at Step 1 from a low-rank matrix approximation to the Gram matrix combined with the binary segmentation heuristic. Unlike Section 6.4.1 this "approximation" procedure is not yet theoretically grounded since it results from two approximation levels which each need to be analyzed. But the purpose here is only to make a first step towards an effective change-point detection procedure dealing with huge (but realistic) signals.

Reducing the computational cost of the dynamic programming step

In this section we explain how to avoid the preliminary calculation of the cost matrix required by [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF] to apply dynamic programming. The key idea is to compute the elements of the cost matrix on the fly when they are required by the dynamic programming algorithm. Roughly, this can be efficiently done by reordering the loops involved in Step 1 of Procedure 1 proposed in [START_REF] Arlot | Kernel change-point detection[END_REF]. This leads to the new exact Algorithm 3. It has a reduced space complexity of order O(n) compared to O(n 2 ) for the one used in [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF].

As exposed in Section 6.3.2, the main computational cost of the change-point detection procedure results from recovering the best segmentation with 1 ≤ D ≤ D max segments by solving

L D,n+1 = min τ ∈T D n Y -µ τ 2 H,n
(best fit to the data)

τ (D) = argmin τ ∈T D n Y -µ τ 2 H,n (best segmentation) (6.15)
for every 1 ≤ D ≤ D max , where Y = (Y 1 , . . . , Y n ) ∈ H n , and T D n denotes the collection of segmentations of {1, . . . , n} with D segments. This challenging step involves the use of dynamic programming [START_REF] Auger | Algorithms for the optimal identification of segment neighborhoods[END_REF][START_REF] Bellman | Applied Dynamic Programming[END_REF], which provides the exact solution to the optimization problem (6.15). Let us first provide some details on the usual way dynamic programming is implemented.

Limitations of the standard dynamic programming algorithm with kernels

Let τ denote a segmentation in D segments (with τ 1 = 1 and τ D+1 = n + 1 for notational convenience). For any 1 ≤ d ≤ D, the segment τ d , τ d+1 -1 = {τ d , . . . , τ d+1 -1} of the segmentation τ has a cost that is equal to (6.16) This cost quantifies the price that has to be paid for putting two consecutive change-points at τ d and τ d+1 .

C τ d ,τ d+1 = τ d+1 -1 i=τ d k(X i , X i ) - 1 τ d+1 -τ d τ d+1 -1 i=τ d τ d+1 -1 j=τ d k(X i , X j ).
Then the cost of the segmentation τ is given by

Y -μτ 2 H,n = D d=1 C τ d ,τ d+1 ,
which is clearly segment additive [START_REF] Arlot | Kernel change-point detection[END_REF][START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF]. Dynamic programming solves (6.15) for all 1 ≤ D ≤ D max by applying the following update rules (6.17) which exploits the property that the optimal segmentation in D segments over {1, . . . , n} can be computed from optimal ones with D -1 segments over {1, . . . , τ } (τ ≤ n). Making the key assumption that the cost matrix {C i,j } 1≤i,j≤n+1 has been stored, we can compute L D,n+1 with Algorithm 1.

∀ 2 ≤ D ≤ D max , L D,n+1 = min τ ≤n { L D-1,τ + C τ,n+1 },
Algorithm 1 Basic use of Dynamic Programming 1: for D = 2 to D max do 2:

for τ = D to n do 3:

L D,τ +1 = min τ ≤τ { L D-1,τ + C τ,τ +1 } 4:
end for 5: end for This algorithm is used by [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF] and suffers two main limitations. First it assumes that the C τ,τ have been already computed, and does not take into account the computational cost of its calculation. Second, it stores all C τ,τ in a O(n 2 ) matrix, which is memory expensive.

A quick inspection of the algorithm reveals that the main step at Line 3 requires O(τ ) operations (assuming the C i,j s have been already computed). Therefore, with the two for loops we get a complexity of O(D max n 2 ) in time. Note that without any particular assumption on the kernel k(•, •), computing Y -μτ 2 H,n for a given segmentation τ is already of order O(n 2 ) in time since it involves summing over a quadratic number of terms of the Gram matrix (see Eq. (6.16)). Therefore, there is no hope to solve (6.15) exactly in less than quadratic time without additional assumptions on the kernel.

From Eq. (6.16) let us also remark that computing each C i,j (1 ≤ i < j ≤ n) naively requires itself a quadratic number of operations. Computing the whole cost matrix would require a complexity O(n 4 ) in time. Taking this into account, the dynamic programming step (Line 3 of Algorithm 1) is not the limiting factor and the overall time complexity of Agorithm 1 is O(n 4 ).

Finally, let us emphasize that this high computational burden is not specific of detecting change-points with kernels. It is rather representative of most learning procedures based on reproducing kernels and the associated Gram matrix [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF].

Improved use of dynamic programming with reproducing kernels

Reducing space complexity From Algorithm 1, let us first remark that each C τ,τ is used several times along the algorithm. A simple idea to avoid that is to swap the two for loops in Algorithm 1. This leads to the following modified Algorithm 2, where each column C •,τ +1 of the cost matrix is only used once unlike in Algorithm 1.

Algorithm 2 Improved space complexity 1: for τ = 2 to n do 2:

for D = 2 to min(τ , D max ) do 3:

L D,τ +1 = min τ ≤τ { L D-1,τ + C τ,τ +1 } 4:
end for 5: end for Importantly swapping the two for loop does not change the output of the algorithm and does not induce any additional calculations. Furthermore, at step τ of the first for loop we do not need the whole n × n cost matrix to be stored, but only the column C •,τ +1 of the cost matrix. This column is of size at most O(n).

Algorithm 2 finally requires storing coefficients {L d,τ } 1≤d≤D, 2≤τ ≤n that are computed along the algorithm as well as successive column vectors {C •,τ } 2≤τ ≤n (of size at most n) of the cost matrix. This leads to an overall complexity of O(D max n) in space. The only remaining problem is to compute these successive column vectors efficiently. Let us recall that a naive implementation is prohibitive: each coefficient of the column vector can be computed in O(n 2 ), which would lead to O(n 3 ) to get the entire column.

Iterative computation of the columns of the cost matrix The last ingredient of our final exact algorithm is the efficient computation of each column vector {C •,τ } 2≤τ ≤n . Let us explain how to iteratively compute each vector in linear time.

First it can be easily observed that Eq. (6.16) can be rephrased as follows

C τ,τ = τ -1 i=τ k (X i , X i ) - A i,τ τ -τ = D τ,τ - 1 τ -τ τ -1 i=τ A i,τ ,
where D τ,τ = τ -1 i=τ k (X i , X i ), and A i,τ is given by

A i,τ = -k(X i , X i ) + 2 τ -1 j=i k(X i , X j ), if i < τ ,
and by further using A j,j = -k(X j , X j ) for any 1 ≤ j ≤ n. Second, both D τ,τ and {A i,τ } i≤τ can be iteratively computed from τ to τ + 1 by use of the two following equations:

D τ,τ +1 = D τ,τ + k(X τ , X τ ),
and

A i,τ +1 = A i,τ + 2k(X τ , X τ ), ∀i ≤ τ .
Therefore, as long as computing k(x i , x j ) requires O(1) operations, updating from τ to τ + 1 requires O (τ ) operations.

Remark 6.1. Note that for many classical kernels, computing k(x i , x j ) is indeed O(1) in time. If x i ∈ R q with q a positive integer being negligible with respect to other influential quantities such as D max and n, several kernels such as the Gaussian, Laplace, or χ 2 ones lead to a O(q) = O(1) time complexity for evaluating k(x i , x j ). By contrast in case where q is no longer negligible, the resulting time complexity is roughly multiplied by a factor q, which corroborates the intuition that the computational complexity increases with the "complexity" of the objects in X . This should be compared to the O(D max n 4 ) time complexity of the naive calculation of the cost matrix and to the O(n 2 ) space complexity of the standard Algorithm 1 from [START_REF] Harchaoui | Retrospective change-point estimation with kernels[END_REF].

Runtimes comparison to other implementations

The purpose of the present section is to summarize the comparison between Algorithm 3 and other competitors to illustrate their performances as the sample size increases with D max = 100. For all these simulation experiments we simulated data following a Gaussian distribution with mean 0 and variance 1. All simulations where run on a laptop with 7.7Gb of Ram and 4 Core CPU with 2.1GHz each. All simulation details can be found in Celisse et al. (2016, Section 3.1.3). The first comparison has been carried out between Algorithm 3 and the naive quartic computation of the cost matrix (Algorithm 1). Results for these algorithms are reported in Figure 6.6 (Left). Unsurprisingly, our quadratic algorithm (called Kernseg) is faster than a quartic computation of the cost matrix (called KCP) even for very small sample sizes (n < 320).

Second, we also compared the runtime of Kernseg (Algorithm 3) with that of a state-of-the-art procedure called ECP implemented in the R-package ecp of James and Matteson (2013) (see the middle panel of Figure 6.6). Since ECP is based on the binary segmentation heuristic applied to an energy-based distance, its worst-case complexity is at most O(D max n 2 ) in time, which is the same as that of Kernseg. Note also that the native implementation of ECP involves an additional procedure relying on permutation tests to choose the number of change-points. If B denotes the number of permutations, the induced complexity is then O(BD max n 2 ) in time. To be fair, we compared our approach and ECP with and without the permutation layer. Finally it is also necessary to emphasize that unlike Kernseg, ECP does not provide the exact but only an approximate solution to the optimization problem (6.15). Results are summarized in Figure 6.6 (Middle). It illustrates that our exact algorithm (Kernseg) has a quadratic complexity similar to that of ECP with and without permutations. Our algorithm is the overall fastest one even for a small sample size (n < 1 000). Although this probably results from implementation differences, it is still noteworthy since Kernseg is exact unlike ECP.

Finally, Figure 6.6 (Right) illustrates the worse memory use of ECP (with and without any permutations) as compared to that of the exact KS.Gau (Kernseg used with the Gaussian kernel). For n larger than 10 4 the quadratic space complexity of ECP is a clear limitation since several hundreds Gb of RAM are required.

Low-rank approximation to the Gram matrix and binary segmentation

Section 6.4.1 describes the improved algorithm Kernseg that carefully combines dynamic programming with the computation of the cost matrix elements. This new algorithm (Algorithm 3) provides the exact solution to the optimization problem given by Eq. (6.15). However without any further assumption on the underlying reproducing kernel, this algorithm only achieves the complexity O(n 2 ) in time, which is a clear limitation with large scale signals (n ≥ 10 5 ). Note also that this limitation results from using general reproducing kernels (and related Gram matrices) and cannot be overcome by existing algorithms to the best of our knowledge. For instance, the binary segmentation heuristic [START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF] -which is known to be computationally efficient for parametric models-suffers the same O(n 2 ) time complexity when used in the reproducing kernel framework (see also the forthcoming section about binary segmentation).

Let us remark however that for some particular kernels it is possible to reduce this time complexity. For example with the linear kernel given by k(x, y) = x, y R d , x, y ∈ R d , one can use the following trick (6.18) where • R d denotes the Euclidean norm in R d . The purpose of the present section is to describe a versatile strategy (i.e applicable to any kernel) relying on a low-rank approximation to the Gram matrix [START_REF] Fine | Efficient svm training using low-rank kernel representations[END_REF][START_REF] Smola | Sparse greedy matrix approximation for machine learning[END_REF][START_REF] Williams | Using the Nyström method to speed up kernel machines[END_REF]. This approximation allows to considerably reduce the computation time by exploiting (6.18). Note however that the resulting procedure achieves this lower time complexity at the price of only providing an approximation to the exact solution to (6.15) (unlike the algorithm described in Section 6.4.1).

1≤i =j≤n k(X i , X j ) = 1≤i≤n X i , n j=1 X j -X i R d = n i=1 X i 2 R d - n i=1 X i 2 R d ,

Low-rank approximation to the Gram matrix

The main idea is to follow the same strategy as the one described by [START_REF] Drineas | On the nyström method for approximating a gram matrix for improved kernel-based learning[END_REF] to derive a low-rank approximation to the Gram matrix K = {K i,j } 1≤i,j≤n , where K i,j = k(X i , X j ).

Assuming K has rank rk(K) n, we could be tempted to compute the best rank approximation to K by computing the rk(K) largest eigenvalues (and corresponding eigenvectors) of K. However such computations induce a O(n 3 ) time complexity which is prohibitive.

Instead, [START_REF] Drineas | On the nyström method for approximating a gram matrix for improved kernel-based learning[END_REF] suggest applying this idea on a (smaller) square sub-matrix of K with size p n. For any subsets I, J ⊂ {1, . . . , n}, let K I,J denote the sub-Gram matrix with respectively row and column indices in I and J. Let J p ⊂ {1, . . . , n} denote such a subset with cardinality p, and consider the sub-Gram matrix K Jp,Jp which is of rank r ≤ p. Further assuming r = p, the best rank-p approximation to K Jp,Jp is K Jp,Jp itself. This leads to the final approximation to the Gram Matrix K [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF][START_REF] Drineas | On the nyström method for approximating a gram matrix for improved kernel-based learning[END_REF] by

K = K In,Jp K + Jp,Jp K Jp,In , (6.19)
where I n = {1, . . . , n}, and K + Jp,Jp denotes the pseudo-inverse of K Jp,Jp . Further considering the SVD decomposition of K Jp,Jp = U ΛU, for an orthonormal matrix U, we can rewrite

K = Z Z, with Z = Λ -1/2 U K Jp,In ∈ M p,n (R).
Note that the resulting time complexity is O(p 2 n), which is smaller than the former O(n 3 ) as long as p = o( √ n). The column vectors {Z i } 1≤i≤n of Z act as new p-dimensional observations, and each K i,j can be seen as the classical inner-product between two vectors of R p , that is

K i,j = Z i Z j .
(6.20)

The main interest of this approximation is that, using Eq. (6.18), computing the cost of a segment of length t has a complexity O(t) in time unlike the usual O(t 2 ) that holds with general kernels.

Interestingly such an approximation to the Gram matrix can be also built from a set of deterministic points in X . This remark has been exploited to compute our low-rank approximation for instance in the simulation experiments as explained in Section 6.4.2.

Note that choosing the set J p of columns/rows leading to the approximation K is of great interest in itself for at least two reasons. First from a computational point of view, the p columns have to be selected following a process that does not require to compute the n possible columns beforehand (which would induce a O(n 2 ) time complexity otherwise). Second, the quality of K to approximate K crucially depends on the rank of K that has to be as close as possible to that of K (which remains unknown for computational reasons). However such questions are out of scope of the present work, and we refer interested readers to [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF]; [START_REF] Drineas | On the nyström method for approximating a gram matrix for improved kernel-based learning[END_REF]; [START_REF] Williams | Using the Nyström method to speed up kernel machines[END_REF] where this point has been extensively discussed.

Binary segmentation heuristic

Since the low-rank approximation to the Gram matrix detailed in the above section leads to finite dimensional vectors in R p (6.20), the change-point detection problem amounts to recover abrupt changes of the mean of a p-dimensional time-series. Therefore any existing algorithm usually used to solve this problem in the p-dimensional framework can be applied. An exhaustive review of such algorithms is out of the scope of the present work. However we will mention only a few of them to highlight their drawbacks and motivate our choice. Let us also recall that our purpose is to provide an efficient algorithm allowing: (i) to (approximately) solve Eq. (6.15) for each 1 ≤ D ≤ D max and (ii) to deal with large sample sizes (n ≥ 10 6 ).

Dynamic programming

The first algorithm is the usual version of constrained dynamic programming [START_REF] Auger | Algorithms for the optimal identification of segment neighborhoods[END_REF]. Although it has been recently revisited with p = 1 by Cleynen et al. (2014); Maidstone et al. (2017b); [START_REF] Rigaill | A pruned dynamic programming algorithm to recover the best segmentations with 1 to k max changepoints[END_REF], it has a O(n 2 ) time complexity with p > 1 , which excludes dealing with large sample sizes. Another version of regularized dynamic programming has been explored by [START_REF] Killick | Optimal detection of changepoints with a linear computational cost[END_REF] who designed the PELT procedure. It provides the best segmentation over all segmentations with a penalty of λ per change-point with an O(n) complexity in time if the number of change-points is linear in n. Importantly, the complexity of the pruning inside PELT depends on the true number of change-points. For only a few change-points, the PELT complexity remains quadratic in time. With PELT, it is not straightforward to efficiently solve Eq. (6.15) for each 1 ≤ D ≤ D max , which is precisely the goal we pursue. Note however that it would still be possible to recover some of those segmentations by exploring a range of λ values like in CROPS [START_REF] Haynes | Computationally efficient changepoint detection for a range of penalties[END_REF].

Binary segmentation A second possible algorithm is the so-called binary segmentation [START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF][START_REF] Olshen | Circular binary segmentation for the analysis of array-based DNA copy number data[END_REF][START_REF] Yang | Simple binary segmentation frameworks for identifying variation in DNA copy number[END_REF] that is a standard heuristic for approximately solving Eq. (6.15) for each 1 ≤ D ≤ D max . This iterative algorithm computes the new segmentation τ (D + 1) with D + 1 segments from τ (D) by splitting one segment of τ (D) into two new ones without modifying other segments. More precisely considering the set of change-points τ (D) = {τ 1 , . . . , τ D+1 }, binary segmentation provides

τ (D + 1) = argmin τ ∈T D+1 n |τ ∩ τ (D)= τ (D) Y -μτ 2 H,n .
Since only one segment of the previous segmentation is divided into two new segments at each step, the binary segmentation algorithm provides a simple (but only approximate) solution to Eq. (6.15) for each 1 ≤ D ≤ D max . We provide some pseudo-code for binary segmentation in Algorithm 5. It uses a sub-routine described by Algorithm 4 to compute the best split of any segment [τ, τ [ of the data. To be specific, this BestSplit routine outputs four things: (1) the reduction in cost of spliting the segment [τ, τ [, (2) the best change to split (3) the resulting left segment and (4) the resulting right segment.

In the binary segmentation algorithm candidate splits are stored and handled using a binary heap data structure [START_REF] Cormen | Introduction to algorithms[END_REF] using the reduction in cost as a key. This data structure allows to efficiently insert new splits and extract the best split in O(log(D max )) at every time step. Without such a structure inserting splits and extracting the best split would typically be in O(D max ) and for large D max the binary segmentation heuristic is at best O(n 2 ). Assuming the best split of any segment is linear in its length the overall time complexity of binary segmentation for recovering approximate solutions to (6.15) for all 1 ≤ D ≤ D max is around O (log(D max )n) in practice. The worst-case time complexity is O (D max n). A typical setting where it is achieved is with the linear kernel when i → X i = exp(i) for instance. At the i-th iteration of the binary segmentation algorithm, the best split of a segment of length n -i + 1 corresponds to one segment of length 1 and another one of length n -i.

An important remark is that binary segmentation only achieves this reduced O (log(D max )n) time complexity provided that recovering the best split of any segment is linear in its length. This is precisely what has been obtained by using the low-rank matrix approximation summarized by Eq. (6.20). Indeed with the low-rank approximation, computing the best split of any segment is linear in n and p. The resulting time complexity of binary segmentation applied to the p-dimensional vectors Z i s is thus O (p log(D max )n), which reduces to O (log(D max )n) as long as p is small compared to n. By contrast without this approximation, recovering the best split is typically quadratic in the length of the segment and binary segmentation would suffer an overall time complexity of order O(log(D max )n 2 ) or O(D max n 2 ).

Implementation and runtimes of the approximate solution

The approximation algorithm, referred to as ApKS for Approximation Kernseg, is the combination of the low-rank approximation step and of the binary segmentation discussed in the above sections. We provide the pseudo-code of this approximation algorithm, namely Algorithm 6. The resulting time complexity is then O(p 2 n + p log(D max )n), which allows dealing with large sample sizes (n ≥ 10 6 ). Our fast approximation algorithm (ApKS ) recovers a quadratic complexity if p = √ n. However its overhead is much smaller than that of the exact algorithm, which makes it more applicable than the latter with large signals in practice. Figure 6.7 illustrates that ApKS returns the solution in a matter of seconds with a sample size of n = 10 5 , which is much faster than Kernseg that requires a few minutes. The RBS implementation involves preliminary calculations which make it slower than ApKS with n ≤ 2 • 10 3 . However for larger values (n ≥ 10 4 ) RBS is as fast as ApKS with p = 10.

Statistical accuracy of the approximation

The purpose of the present section is to illustrate the behaviour of ApKS (in terms of statistical precision) as an alternative to Kernseg (which is more time consuming). Since we do not provide any theoretical warranty on the model selection performances of ApKS , we only show its results for several values of p ∈ {4, 10, 40, 80, 160} at the true number of segments D . For each value of p, we compute the approximation by: (i) evaluating the smallest and largest observed value (respectively denoted by m and M ), (ii) using an equally spaced grid of p deterministic values between m and M and (iii) use those p values to perform the approximation of the Gram matrix. All technical details about these experiments can be found in Celisse et al. (2016, Section 4).

From Figure 6.8 it clearly appears that the number of points used to build the low-rank approximation to the Gram matrix is an influential parameter that has to be carefully fixed. However as long as p is chosen large enough, the approximation seems to provide very similar results. This suggests that one q q q q q q q q q q q q q Ap4KS.Gau Ap10KS.Gau Ap40KS.Gau Ap80KS.Gau Ap160KS.Gau KS.Gau should find a trade-off between the statistical performances and the computation cost. Indeed from a statistical point of view increasing p is beneficial (or at least not detrimental). From a computational point of view however, increasing p is detrimental since it increases the time complexity (O(p 2 n)).

Let us finally emphasize that for large enough p the performances of ApKS are very close to those of KS.Gau. Given the low time complexity of ApKS compared to that of KS.Gau we argue that ApKS could be a promising alternative to KS.Gau for large signals (n 10 5 ). Nevertheless several questions related to the use of ApKS remain open such as the strategy to build the p-dimensional approximation, or to design a theoretically grounded model selection criterion similar to the one used in Procedure 2.

Conclusion

With large scale sample sizes (n ≈ 10 6 ) which are more and more common in the daily practice, changepoint detection procedures such as KCP cannot be applied with general kernels due to its native quartic time complexity (O(n 4 )). The main contribution of this work is to describe two efficient change-point detection procedures with reduced computational complexities.

The first one consists of an improved implementation of the interplay between the dynamic programming algorithm and the so-called cost matrix computation within KCP [START_REF] Arlot | Kernel change-point detection[END_REF]. The resulting procedure, called Kernseg, outputs the (exact) best segmentation for each number of segments with an overall complexity of order O(n) in space and O(n 2 ) in time.

By contrast, the second procedure (ApKS ) returns an approximation to the best segmentation by combining a low-rank matrix approximation to the Gram matrix with the binary segmentation heuristic. These approximations make it possible to reduce the computational complexity to O(n) in time and space, which allows us to deal with huge sample sizes (n ≈ 10 6 ) in a few seconds. If its statistical performance seems to remain close to that of the exact procedure in our simulation experiments, several questions remain widely open about the use of this approximation procedure.

Choice of the (number of ) rows/columns Finding the optimal low-rank matrix approximation remains a difficult task in practice.

For instance the true rank of the n × n Gram matrix K is unknown since its computation would induce a O(n 3 ) time complexity, which is prohibited to get an overall O(n) time complexity. This is a problem since any reliable approximation K to the Gram matrix should have its rank (lower but still) related to that of K. More generally, any time-efficient strategy leading to this low-rank approximation should avoid operations with a cost larger than O(n) in time. In particular this excludes the preliminary computation of the norm of each column vector of the Gram matrix as suggested in [START_REF] Drineas | On the nyström method for approximating a gram matrix for improved kernel-based learning[END_REF].

Several iterative strategies have been proposed to build such an approximation in linear time in practice. Among others Shawe-Taylor and Cristianini ( 2004) consider a Gram-Schmidt orthonormalization, [START_REF] Bach | Predictive low-rank decomposition for kernel methods[END_REF] describe an incomplete Cholesky decomposition, and [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF] introduce the CUR matrix decomposition. All these approaches suffer one main limitation: They only yield theoretical guarantees on the matrix approximation (for instance measured in terms of Frobenius norm) without considering the final purpose: classification, regression,. . . This results from the difficulty to analyze such iterative procedures in a meaningful way as emphasized by [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF].

In the present work, the low-rank approximation to the Gram matrix is built following a two-step empirical strategy. Firstly the support of the distribution of the X i ∈ R is estimated by using the minimum and maximum observed values. Secondly the approximation is computed from an equallyspaced grid of (deterministic) points covering this support. This rough strategy intuitively suffers several limitations. In particular the equally-spaced grid of points implicitly assumes that the corresponding (compactly supported) distribution is almost uniform, which can be far from being true. Moreover using such a grid of points when X i ∈ R d with d > 1 would become computationally demanding.

Nevertheless the observed performance of this low-rank approximation highlights interesting behaviors which would require further investigations. On the one hand, the number p of grid points used to build the approximation has to be large enough to reach a reasonable statistical precision. Having in mind that increasing p will also increase the computation time (O(p 2 n)), this suggests a trade-off arises between the statistical precision and the amount of time. On the other hand, the parameter p can be also interpreted as the rank of the approximation to the Gram matrix. In settings where the decay of the eigenvalues of the Gram matrix is very fast, using a low-rank approximation can be used as a means to regularize. For instance this avoids ill-conditioned problems which are responsible for numerical instability and slower convergence rates [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF].

Estimating the number of segments for the approximation procedure Unlike what has be done by [START_REF] Arlot | Kernel change-point detection[END_REF], there is no theoretical guarantee on the performance of ApKS in the present work. The purpose here was to illustrate the potential interest of such strategies in the kernel-based changepoint detection context. Deriving such a theoretical result would require to design a penalty from a tight evaluation of the approximation and estimation error terms for the resulting estimator. This strongly depends on the strategy used to choose the rows/columns of the low-rank approximation. To the best of our knowledge, such results are only available for strategies based on a random sampling of these rows/columns (see for instance Theorem 1 in [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF]. Note that several recent attempts have been made to incorporate some additional side information (summarized by leverage scores) into the sampling scheme [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF][START_REF] Musco | Recursive sampling for the nystrom method[END_REF].

f (D n ) ∈ argmin h∈F Ψ(D n ; h),
where Ψ(D n ; •) : F → R is a functional which depends on the sample D n and is defined over a given set F . In most cases no closed-form formulas are available for f (D n ) and an iterative optimization algorithm is used to get a sequence f t (D n ) 1≤t≤tmax of approximations, where t max denotes the maximum number of iterations we are allowed to compute with a given time budget B.

For example let us consider the same experiment setting as in [START_REF] Raskutti | Early stopping and non-parametric regression: an optimal data-dependent stopping rule[END_REF]. The regression function f (x) = |x -1/2| -1/2 for x ∈ [0, 1] is estimated by means of functions in the RKHS associated with the Gaussian kernel k γ (x, y) = exp -γ(x -y) 2 (γ > 0), and successive estimates result from applying the gradient descent algorithm. sample) risk (red curve) with respect to the number t of iterations. The curve exhibits:

• a fast decrease along the first iterations (before 100),

• a slower increase as the iteration number further grows (after 100),

• a global minimum achieved at t ≈ 70, which is by far smaller than t max = 1 000.

From such an iterative strategy, the idea is then to stop the process as early as possible to avoid the unnecessary computation of estimates beyond the global minimum location.

Early stopping rules Early stopping rules (ESR) are data-driven rules designed to avoid unnecessary calculations (therefore saving the computational resources). How to design and study ESR has been studied for a long time (see for instance [START_REF] Strand | Theory and methods related to the singular-function expansion and landweber's iteration for integral equations of the first kind[END_REF][START_REF] Wahba | Three topics in ill-posed problems[END_REF] in numerous contexts such as neural networks and stochastic gradient descent [START_REF] Morgan | Generalization and parameter estimation in feedforward nets: Some experiments[END_REF], greedy algorithms [START_REF] Barron | Approximation and learning by greedy algorithms[END_REF], boosting [START_REF] Bartlett | Adaboost is consistent[END_REF][START_REF] Bühlmann | Boosting with the l 2 loss: regression and classification[END_REF][START_REF] Zhang | Boosting with early stopping: Convergence and consistency[END_REF], conjugate gradient descent [START_REF] Blanchard | Optimal learning rates for kernel conjugate gradient regression[END_REF], . . . However most ongoing approaches suffer some limitations. Firstly, most of them have an asymptotic flavor. The stopping rule only depends on the sample through its cardinality and cannot be computed in practice [START_REF] Bühlmann | Boosting with the l 2 loss: regression and classification[END_REF][START_REF] Yao | On early stopping in gradient descent learning[END_REF][START_REF] Zhang | Boosting with early stopping: Convergence and consistency[END_REF]. Secondly, stopping rules are often derived from successive (more or less tight) upper bounds on the approximation and estimation error terms. This typically leads to criteria that only become (asymptotically) valid in terms of minimax rates (worst-case bounds) by contrast with (non-asymptotic) oracle-type inequalities [START_REF] Lin | Iterative regularization for learning with convex loss functions[END_REF][START_REF] Raskutti | Early stopping and non-parametric regression: an optimal data-dependent stopping rule[END_REF][START_REF] Wei | Early stopping for kernel boosting algorithms: A general analysis with localized complexities[END_REF][START_REF] Yao | On early stopping in gradient descent learning[END_REF].

High potential impact Countless optimization algorithms are concerned with designing efficient ESR, which could greatly improve their daily use. Among others, let us mention routinely used iterative algorithms such as (conjugate) gradient descent [START_REF] Blanchard | Optimal learning rates for kernel conjugate gradient regression[END_REF][START_REF] Yao | On early stopping in gradient descent learning[END_REF], stochastic gradient descent [START_REF] Dieuleveut | Nonparametric stochastic approximation with large step-sizes[END_REF], coordinate descent [START_REF] Wright | Coordinate descent algorithms[END_REF], and binary segmentation heuristic (in the change-point detection context) [START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF], . . . Several "classical" learning algorithms could also be improved by designing a dedicated ESR.

• Ridge regression: A first example is the Ridge learning algorithm, which natively depends on a regularization parameter λ > 0. The tuning of this influential parameter is usually made by optimizing the V-fold cross-validation (V-FCV) estimator over a grid of values of λ. This strategy is highly time consuming and introduces some additional variability related to using V-FCV.

By contrast, this grid search strategy can be reformulated as an iterative procedure (from the largest to the smallest values of λ) where at each step, the question is to decide whether we make one step further or not. Defining an ESR for such an iterative procedure would lead to a calibration strategy for λ as a by-product.

• Model selection with a nested collection of models: From a nested collection of models (S d ) 1≤d≤dmax , where 1 ≤ d ≤ d max denotes the dimension of the vector space S d , a possible solution for recovering the best model consists in first designing a penalized criterion d → pen(d) (derived from concentration inequalities for instance), and then optimizing

d = argmin 1≤d≤dmax L Pn f d + pen(d) ,
where f d denotes the empirical contrast minimizer over S d for each d. As expressed by the above optimization problem, this has to be carried out for all the values of d from 1 up to d max . In cases where the minimum location d is very small in comparison to d max (which has to be specified a priori ), this strategy leads to waste a large amount of time.

With a nested collection of models, that is where S d ⊂ S d+1 for all 1 ≤ d ≤ d max , the previous setting can be rephrased in terms of an iterative learning algorithm. Designing an ESR could help stopping the process close to the optimum while avoiding computing S d for d close to d max . Note also that this idea is not limited to the nested model setting, but could be extended to large collections of models as long as meta-models can be defined and ordered according to a meaningful criterion (as in the change-point detection framework).

In most of ongoing works about designing ESR, the analysis is carried out in one particular "model".

For instance Blanchard et al. (2016) derive an optimal ESR for one particular linear regression model with p covariates. The same remark holds true with [START_REF] Raskutti | Early stopping and non-parametric regression: an optimal data-dependent stopping rule[END_REF]; [START_REF] Wei | Early stopping for kernel boosting algorithms: A general analysis with localized complexities[END_REF] where the regression function is assumed to belong to the reproducing kernel Hilbert space associated with the underlying kernel.

When several such models are available (involving more or less covariates for instance), this analysis has to be embedded into the model selection framework where the potential bias of each model has to be taken into account. In particular, this can lead to modify the stopping rule. When using reproducing kernels, this question can be also related to the optimization of the kernel for a particular task. This is still a widely open problem in machine learning even if partial answers already exist or instance in the context of two-sample tests (Gretton et al., 2012b).

7.2 Efficient cross-validation 7.2.1 Approximating the CV estimator and closed-form formulas

Context In presence of a huge amount of data, statisticians/machine learners have to design learning strategies which have to be efficient from a practical point of view. This computational efficiency arises at two complementary levels:

1. at the learning algorithm level: each estimator (resulting from a given algorithm) has to be efficiently computed, 2. at the performance evaluation level: the performance assessment should be made as fast as possible.

The first level is the main concern of Section 7.1 about designing early stopping rules (ESR) in a given model to avoid wasting time with unnecessary calculations.

Efficient computations of the CV estimators

The present section mainly focuses on the second level, which has been previously discussed in Sections 2.1 and 2.2 of the present manuscript as far as crossvalidation is concerned. However these approaches aiming at deriving computationally efficient evaluations suffer some strong limitations. On the one hand, the main deficiency of Section 2.1 is that deriving closedform formulas for the LpO estimator is a difficult task, which cannot be always carried out (even at the price of great efforts). On the other hand, Section 2.2 enumerates recent approximations to V-FCV for which:

• some heavy computations are still required (depending on V ),

• some additional variability comes from the original random split of the data into V disjoint blocks.

Potential impact CV procedures are among the most used calibration strategies in the statistical/machine learning communities to tune unknown parameters. Therefore any improvement in the computation time of CV procedures would drastically enhance the daily practice of anyone who has to calibrate a learning procedure (Lasso, k-Nearest neighbors, kernel density estimator, . . . ) Let us emphasize that any improvement will be all the more strong as it will apply to a wide class of estimators.

Main ideas to explore

• Versatile strategy for approximating the LpO estimator:

ALpO: The existence of closed-form expressions for the LpO estimator depends on the contrast function γ and the estimator the LpO is applied to. The range of estimators and contrast functions for which such closed-form expressions are available can be widely enlarged at the price of allowing for some approximations. Assuming differentiability for γ and for the functional M (•) (defining an M-estimator), preliminary results show that a versatile strategy does exist which leads to closedform expressions for the approximated LpO (ALpO) estimator of M -estimators. For instance an m i (f ) with m i (f ) = m(f ; Z i ), ṁi (f ) denotes the gradient of m i at f , and mi (f ) the corresponding Hessian matrix at f . A typical example which can be investigated with this framework is the Ridge regression, where m i (θ) = Y i -X i θ 2 + λ θ 2 2 for some λ > 0. Assumptions and further refinements: The above approximation assumes that the Hessian matrix of M can be inverted over [f D , f De ]. This is typically true with the Ridge regression estimator, but certainly not true with the least-squares estimator computed in an over-parametrized model. This means that the strategy explored up to now with Tristan Mary-Huard and Julien Chiquet should be further improved, at least by relaxing some of the ongoing assumptions under which it has been derived.

Promising aspects: Nevertheless, the resulting formulas exhibit three important assets which can be emphasized. Firstly, the resulting approximated LpO (ALpO) estimator is expressed in terms of the derivatives of γ and M , which have to be computed (only) once from the whole sample. Therefore the computational cost is approximately of the same order as that of the empirical contrast in the same setting. Secondly, the formulas only depend on p through multiplicative coefficients. As a consequence, there is no additional cost induced by evaluating the ALpO estimator at different values of the splitting parameter p. Finally this general strategy can be applied to the density estimation framework as well as to non-parametric regression with reproducing kernels. Doing so only requires to specify the choices of γ and M in the different terms (as long as the assumptions under which the formula has been derived are fulfilled).

• Quantification of the approximation error suffered by ALpO:

An important motivation for deriving closed-form expressions of the ALpO estimator is that ALpO could provide better results than V-FCV (with p = n/V ). Intuitively, one sufficient condition for this to happen is that the gap (measured in terms of bias and variance) between the LpO and ALpO estimators is smaller than the one between the LpO and V-FCV estimators. For instance this can result from the larger variance of the V-FCV estimator in comparison to that of the LpO estimator (see Section 1.2.3).

Preliminary experimental results seem to support this idea, but a theoretical quantification of the induced approximation error is required to identify ranges of values of p or conditions under which this improvement is possible. At the early stage of the analysis, the ALpO estimator has been derived based on some asymptotic considerations. On this basis (and under some additional smoothness assumptions), an asymptotic quantification of the approximation error has been established. It suggests that this approximation is tight as long as p/n → 0 as n → +∞. This is somewhat expected since Eq. (7.1) quantifies how different f De can be from f D when p points are removed. These preliminary results remain to be made more precise, but it already suggests that considering larger values of p (with respect to n) would require some modifications in the ALpO derivation.

A future direction to explore is how to derive a first non-asymptotic quantification of the performance of the ALpO estimator in terms of both risk estimation and model selection/parameter calibration. Two possible strategies could be investigated. Firstly, one can try to quantify the gap between the LpO and ALpO estimators with the idea of exploiting existing results derived for LpO. Secondly, one can straightforwardly exploit the closed-form formulas derived for the ALpO estimator. By computing its expectation (and by means of concentration inequality results), one could establish a finite-sample quantification of its performance.

Concentration of the CV estimator and parameter calibration

Context The CV estimators serve at least two complementary objectives: (i) risk estimation (Chapter 3) and (ii) estimator selection/parameter calibration (Chapter 5). Let us also mention the recent work of [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF], where CV is seen as a means to choose between several model selection procedures, allowing for combining the assets of AIC and BIC for instance.

However only very few non-asymptotic results exist on the performance of the CV estimators, the most likely reason for that owing to the high technicalities induced by the CV definition. This is a strong limitation in the daily practice of countless CV users, and leads to numerous misconceptions as argued by [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF].

Concentration inequalities and parameter calibration One main tool to derive such finite-sample quantifications is the classical empirical process theory [START_REF] Van De Geer | Empirical Processes in M-estimation[END_REF] and more precisely concentration inequalities (Boucheron et al., 2013a). Concentration inequalities are at the core of model selection/parameter calibration procedures for which the classical finite-sample performance quantification comes in terms of an oracle-type inequality.

Very few concentration inequalities have been established for the CV estimators and Chapter 4 in the present work reviews some of them. However the existing inequalities suffer several drawbacks. Some of them are dedicated to a particular family of estimators and heavily rely on a closed-form formula (Section 4.1). The others are derived in a more general framework, but seem too loose for leading to meaningful results in the model selection context (Section 4.2).

Main ideas to explore

• LpO as a U-statistic:

Regarding the above remarks, deriving new meaningful concentration inequalities for the LpO estimator is of great interest since it can be seen as an early step towards useful model selection results. As a starting point, it is possible to further exploit the connection between LpO and U-statistics as already exposed in Section 4.2. For instance upper bounds on the polynomial moments of the LpO estimator can be derived from those of the L1O estimator. In particular this strategy has two assets compared to existing approaches:

1. it allows us to take advantage of the concentration properties of the kernel of the U-statistic, unlike most of existing concentration results which mainly rely on the boundedness of the kernel.

2. it deals with kernels of order m = m n that depends on n, unlike standard results where the order remains fixed with respect to n, allowing for decoupling arguments for instance [START_REF] Giné | Decouping: From Dependence to Independence[END_REF].

Deriving exponential concentration inequalities for the LpO estimator then reduces to upper bounding the polynomial moments of the L1O estimator. This can be achieved by exploiting existing concentration results for the L1O estimator, or by proving new ones if the latter are not tight enough. For instance one deficiency of the upper bound provided by Theorem 4.4 is that it does not lead to meaningful constants in the deviation term of Corollary 4.1. The resulting rate of convergence is not fast enough for exploiting this result in a model selection perspective. One main reason is that the term dealt with in Proposition 4.4 is too large due to the previous use of Jensen's inequality, which has destroyed the interactions between the different terms varying with j. To overcome this, new directions can be investigated. For instance under boundedness assumptions in the regression context, concentration inequalities for the L1O estimator can be proved by combining self-bounding functions as illustrated in Boucheron et al. (2013a, Section 6.11) and the stability of the considered learning algorithm.

• CV and model selection:

The model selection performance of CV has to be further explored to provide some guidelines to practitioners. In particular, an important distinction has to be made between model selection for estimation/prediction and model selection for identification. Such theoretical guarantees (and related guidelines) should remedy several ongoing misconceptions on the use of CV as the ones discussed by [START_REF] Zhang | Cross-validation for selecting a model selection procedure[END_REF].

A first idea is to interpret the LpO estimator as a penalized criterion by writing

R ECV p (A λ , D n ) = P n γ(A λ ) + pen p (A λ ) ,
where λ denotes some parameter to tune. Then an oracle inequality for the LpO estimator minimization could be derived from showing that pen p (A λ ) ≥ pen id (A λ ) with high probability, uniformly with respect to λ. (Let us recall that pen id (A λ ) is defined as the difference between the generalization error and the empirical contrast.)

Following this idea, the parameter p can be viewed as a means to set the balance between the fit to the data and the amount of regularization. In several settings (Celisse, 2014a), it can be shown that increasing p leads to increase the bias of the LpO estimator, that is increasing the expectation of the penalty pen p (A λ ). For instance p = 1 is almost equivalent to unbiased risk estimation (in a similar way to AIC or Mallow's C p ), whereas choosing p = n(1-1/ log n) leads to a penalty (asymptotically) close to BIC (see the discussion before Theorem 5 in [START_REF] Shao | An asymptotic theory for linear model selection[END_REF].

For these reasons the calibration of the parameter p cannot be done following the same idea as the so-called slope heuristic (Arlot and Massart, 2009a;[START_REF] Baudry | Slope heuristics: overview and implementation[END_REF]. Indeed unlike the latter which strongly exploits the overfitting of the (penalized) empirical contrast when the constant in front of the penalty is not large enough, the LpO estimator is an almost unbiased estimator of the risk at p = 1 and becomes an upwardly biased risk estimator as p grows. Therefore the optimal value of p is more related to the practical improvement allowed by a slight overpenalization as pointed out by [START_REF] Arlot | Contributions to statistical learning theory: estimator selection and change-point detection[END_REF] when model selection is performed for estimation/prediction. This question has been recently addressed in [START_REF] Arlot | Choice of v for v-fold cross-validation in least-squares density estimation[END_REF] where a new heuristic is described which aims at defining the best model selection procedure as the one maximizing a signal-to-noise ratio. However one restriction for applying this heuristic to the choice of the optimal value of p is that the LpO estimator is a biased risk estimator, a case which is not covered by the above heuristic. This optimal model selection strategy results from the combination of two ingredients.The first one is the intensive use of concentration inequalities allowing to determine the shape of the penalty as given by (6.12). The second one consists in applying the so-called "slope heuristic" leading to a data-driven choice of the multiplicative constant in front of the penalty [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF]. More precisely the slope heuristic used in the present work (see Section 6.3.4) relies first on estimating the minimal penalty

pen minimal (τ ) ≈ -s 1 • 1 n log n -1 D τ -1 -s 2 D τ n ,
and then on multiplying pen minimal (τ ) by a factor 2 to get the optimal penalty pen optimal (τ ), that is pen optimal (τ ) = 2 × pen minimal (τ ). This heuristic has been theoretically grounded in several settings (for instance by Arlot and Massart, 2009a, for regressograms).

New look at the slope heuristic in the change-point detection context By contrast, some situations also exist where multiplying by a factor 2 the minimal penalty is no longer optimal as proved for instance by [START_REF] Arlot | Data-driven calibration of linear estimators with minimal penalties[END_REF] with linear estimators. In the simulation experiments described in Section 6.3.4, the best results were not achieved with the multiplicative constant equal to 2, but rather with a somewhat smaller value around 1.7-1.8. This could suggest that the slope heuristic should be refined to take into account the specificity of the change-point detection problem and in particular the large collection of competing models. Despite the extensive simulation experiments of [START_REF] Lebarbier | Detecting multiple change-points in the mean of a Gaussian process by model selection[END_REF] carried out with the linear kernel and Gaussian variables, there is no theoretical justification for its use in the present change-point detection framework. Computing these expectations provides some insight on how to correct the empirical contrast evaluated at µ τ (D) to recover the best possible segmentation on average. In the present setting, the so-called minimal penalty to add to the empirical contrast is

pen min (τ ) = E Π τ (D) ε 2 + 2E Π τ (D) µ , ε ,
while the optimal amount of penalization is (by definition) given by pen opt (τ ) = 2 E Π τ (D) ε 2 + E Π τ (D) µ , ε = 2 pen min (τ ).

Unlike what has been explained above, the optimal penalty is no longer equal to 2 × pen min (τ ) as long as E Π τ (D) µ , ε ≈ 0, which certainly does not hold true when D < D and in settings where τ (D ) is not close to τ with high probability. This last remark also suggests that refining the slope heuristic would mainly enhance the procedure accuracy in difficult situations (with low signal-to-noise ratio) where τ (D ) can be far from τ . A first step towards any such improvement is to carry out an extensive simulation study to quantify the influence of E Π τ (D) µ , ε in practice. If the simulation results confirm the above reasoning, the next step is designing a new strategy to distinguish between the contribution of the two terms in pen min (τ ). From estimators of these two terms, one can: (i) add (an estimate of) pen min (τ ) to the empirical contrast, and (ii) add (an estimate of) E Π τ (D) ε 2 to it, which will provide us with an estimator of pen opt (τ ).

Any theoretical justification of the performance of such a data-driven procedure would require deriving tight lower and upper bounds with high probability for Π τ (D) ε 2 and Π τ (D) µ , ε which could be computed (or estimated) in practice.

On-line change-point detection

Context The change-point detection problem described in Section 6.1 is studied in the off-line context that is, after n observations have been collected. This off-line context turns out to be useful for discovering hidden structures underlying the data. By contrast, numerous practical examples require detecting such abrupt changes in a time-series observed on-line (in real-time). Among others, let us mention the detection of cyber-attacks (Lévy-Leduc and Roueff, 2009) and seismic events [START_REF] Ross | Automatic picking of direct p, s seismic phases and fault zone head waves[END_REF] for instance, or the social network analysis [START_REF] Frisén | Optimal sequential surveillance for finance, public health, and other areas[END_REF] and sensor networks monitoring [START_REF] Rice | Flexible smart sensor framework for autonomous structural health monitoring[END_REF]. This on-line context raises new specific constraints which require dedicated developments. For instance any reliable on-line procedure aims at minimizing the detection delay to avoid any deterioration/failure of the monitored system. This problem is also called anomaly detection. Another specific constraint is the need for providing any decision in a short amount of time imposed by the gap between two successive observations. However to the best of our knowledge, ongoing strategies for on-line change-point detection suffer two limitations: (i) they mainly focus on changes only arising in prescribed features of the distribution along the time (for instance the mean or the variance), (ii) most of them are designed to deal with specific objects from a particular application field (say networks) observed along the time, and cannot be applied to another type of object (such as DNA sequences) without deep modifications.

New look at the on-line change-point detection problem In the present on-line context the ongoing anomaly detection (AD) strategies mainly focus on detecting only one change called an anomaly. The observations come sequentially and, at each time step, the goal is to decide if one anomaly has arised or not. Ideally any true detection has to be made as soon as possible, while false detections should be avoided. This decision is made on the basis of a reference (null) distribution from which quantiles of the test statistic are computed. The reference distribution is either known by assumption (for instance with Gaussian data) or estimated from a reference dataset in a nonparametric perspective.

At this step, the notion of "anomaly" needs to be somewhat revisited to stick more tightly to practical situations. An anomaly is the by-product of a change in (some features of) the distribution that has generated the corresponding observation(s). Such a change is of interest as long as one is looking for detecting any modification in the distribution. But in numerous examples, all changes in the distribution are not of interest since some of them only reflect environmental modifications (but not atypical behaviors). For instance the number of connections to a store website is likely to be weaker during the night than in the evening. To be considered as an anomaly, the behavior of a user of this website has to be compared to the typical behavior in the same period of time.

Main ideas to explore

• Extending our off-line multiple change-point detection procedure to the on-line context: An important task is to provide a change-point detection tool that is able to automatically distinguish between the distributional changes related to environmental modifications (background signal), and changes induced by anomalous behaviors that are to be detected. Our proposal consists in extending our off-line multiple change-point detection procedure analyzed in Section 6.3.3 to the on-line context.

Unlike most of on-line procedures, this new kernel-based approach will benefit from the same assets as its off-line counterpart: (i) dealing with a wide range of data types by only choosing a relevant kernel (but without modifying the procedure or its analysis), and (ii) detecting changes arising in the distribution that are not limited to the mean or the variance.

At each time-step, say n, this new on-line procedure will recompute the best segmentation from 1 to n in D segments for 1 ≤ D ≤ D max . From a computational perspective, an important requirement is to keep under control the time complexity induced by recomputing the best segmentation at each time step. This can be done by exploiting the "on-line" formulation of the dynamic programming algorithm It is already possible to list some structural limitations of the new procedure. These limitations come as the price to pay for a higher flexibility to a varying environment.

-When an abrupt change arises in the distribution of the observations (which corresponds to a new segment), it is likely that several consecutive observations will be required for detecting this new segment. Therefore there is a given amount of time along which the observations at the beginning of this new segment will be seen as potential anomalies. This amount of time certainly depends on the size of the jump between the two consecutive segments, on the noise level around the corresponding change-point, and on the length of the segment on the left of the change-point.

-In the same way as the off-line KCP, the new on-line KCP (OKCP) relies on a calibration strategy of the penalty, which provides us with a data-driven choice of the constants c 1 , c 2 leading to the estimated number of segments at each time step. This calibration step has to be performed at each time step in principle, and is all the more costly as the maximum number of segments can be large. Therefore since the total number of segments increases with the number n of observations, it is necessary to build a strategy forgetting the past as long as it is far enough from the present. Intuitively, this should not be a great requirement since it is reliable to think that observations which are far enough from the present will no longer influence the existence of any new segment/anomaly. An important question will be to quantify theoretically this amount of time from which the past has no longer any influence on the present with high probability. This will help in considerably reducing the computational costs, which can be made even lighter by exploiting low-rank approximations to the Gram matrix to speed-up the update step from n to n + 1 in dynamic programming.

• Studying the influence of the reproducing kernel:

-The notion of characteristic kernel relates the difference between mean elements in the RKHS to the difference between probability distributions. This relationship can be exploited as long as such a characteristic kernel exists. But firstly, the conditions for the existence of such characteristic kernels are difficult to fulfill, which makes them difficult to identify in some examples. Secondly when the changes arise in specific features of the distribution, non-characteristic kernels can outperform characteristic ones by focusing on these specific features whereas characteristic kernels consider all of them (with possibly a lower detection power). These remarks give rise to interesting questions to solve:

1. Knowing a prescribed distribution feature, are we able to exhibit a class of kernels allowing us to detect changes arising in this feature? 2. Given a candidate kernel, do we know in which features it will be able to detect abrupt changes?

-All of this can be seen as a first step towards solving the more challenging question of tuning the kernel, which has been briefly discussed in Section 6.3.5. Our theoretical interpretation of KCP in Section 6.3.3 already suggests how the performance of KCP depends on k. Indeed, KCP focuses on changes in the mean µ 1 , . . . , µ n of the time series Y 1 , . . . , Y n ∈ H. A change between P Xi and P Xi+1 should be detected more easily when

µ i+1 -µ i 2 H = E k(X i+1 , X i+1 ) -2E k(X i+1 , X i ) + E k(X i , X i )
is larger, compared to the "noise level" max{v i , v i+1 }. When P Xi = P Xi+1 , we know that µ i+1 -µ i H is positive for any characteristic kernel k, while it might be equal to zero when k is not characteristic. But the fact that k is characteristic or not is not sufficient to guess whether k will work well or not, according to the above heuristic. For instance, all (characteristic )Gaussian kernels (with bandwidth ω > 0) do not perform the same depending on ω. In the change-point detection problem, a first idea is to extend the strategy exposed in Gretton et al. (2012b) to the multiple change-points detection problem as follows. Given a model selection procedure such as KCP, the work of [START_REF] Garreau | Consistent change-point detection with kernels[END_REF] can be used to derive a meaningful upper bound on the probability of missing the true segmentation. Then, this upper bound could be minimized as a function of the kernel to increase the detection power of the procedure. The two main difficulties of this approach are: (i) it strongly relies on the tight and meaningful upper bound on the probability of missing the true segmentation (or its closest proxy), and (ii) since this upper bound is likely to depend on unknown quantities, we will have to derive estimators for them and minimize the resulting (estimated) upper bound.

Figure 1 . 1 :

 11 Figure 1.1: Error of the iterative estimator (gradient descent) with respect to the number i of iterations. Black: Excess loss. Red: (Excess) Risk.

  ē} and D e = {Z i | i ∈ e} respectively called the test set and the training set. Then the splitting strategy consists in 1. using the training set D e to train the learning algorithm A that is, to compute the estimator A (D e ; •) = f (D e ; •) ∈ F, 2. assessing the performance of the estimator A (D e ) = f (D ē) by evaluating its empirical error on the test set D ē by writing L P D ē (A (D e )) = 1 p i∈ē γ (A (D e ) ; Z i ) .

  p , where E W [ • ] and Var W [ • ] denote respectively the expectation and the variance with respect to W , and E Dn [ • ] and Var Dn [ • ] expectation and variance with respect to D n ∼ P ⊗n

i=1

  Var h ECV m,vi (D vi ) .

  Moment inequalities. As emphasized by Proposition 4.5, the resulting upper bound on E r i=1 h ECV m,vi (D vi ) q results from deriving respective upper bounds for E h ECV m,vi (D vi ) q and Var h ECV m,vi (D vi ) for any 1 ≤ i ≤ r, since h ECV m,v1 (D v1 ), . . . , h ECV m,vr (D vr ) are i.i.d.centered random variables. These bounds are collected in the following result which is derived by combining the generalized Efron-Stein inequality (Theorem 15.5 in Boucheron et al. (2013b)), the above Lemma 4.2, and the Stone lemma (namely Lemma 4.4 mentioned for the sake of completeness).

Proposition 4 . 6 (

 46 Theorem 3.1 in Celisse and Mary-Huard (2015)). For every 1

  .13)with C 1 = 2 + 16γ d and C 2 = 4γ d √ 2κ, where γ d denotes the constant arising from Stone's lemma (Lemma 4.4) and κ < 1.271 is a universal constant (Proposition ??).

  any of these points belongs to the k nearest neighbors of at most kγ d of the other points, where γ d only depends on d.

  0

Figure 5 . 1 :

 51 Figure 5.1: Graph of the average of p/n → C or,n (p/n) over 500 repetitions.

Figure 5 .

 5 2 displays simulation results in the same experimental setting as Figure5.1. Let us also recall that p 0 = inf 1≤p≤n-1 C n (p/n).

  (a) n → Cn(p 0 /n) (blue plain) and n → Cn,or(p * /n) (black dashed). (b) n → p 0 /n (blue plain) and n → p * /n (black dashed).

Figure 5 .

 5 Figure 5.2: Comparison of the ratios and respective minimum locations with respect to n (averaged over 500 repetitions).

Theorem 5 . 3 (

 53 Model consistency with f ∈ ∪ m F τ , Theorem 3.4 of Celisse (

Figure 5 .

 5 Figure 5.3: p/n → P [ τ = τ ] averaged over N = 1 000 repetitions.

  .8)) than in Panel (b) (where µ ≈ 1/14). Unlike Panel (a) where τ remains almost unchanged as n increases, the best parametric estimator in Panel (b) changes with small values of n (as allowed by (5.5)), hence the slower convergence rate in (b).

Figure 5 . 4 :

 54 Figure 5.4: p/n → P [ τ = τ ] averaged over N = 1 000 repetitions.

Figure 6 . 1 :

 61 Figure 6.1: Graph of an instance of the observations Y i (red dots), f τ also called ERM (dashed blue), the oracle estimator (dotted black), the L1O optimal estimator (plain magenta) versus the postition t i ∈ [0, 1]. Fig. 6.1a: Homoscedastic setting with D τ = 4 for f τ and L1O, and D = 5 (true number of segments) for the oracle estimator. Fig. 6.1b: Heteroscedastic setting with D τ = 6 for f τ and L1O, and D = 10 (true number of segments) for the oracle estimator.

Figure 6 . 2 :

 62 Figure 6.2: Examples of generated signals (blue plain curve) in the three scenarios. Red vertical dashed lines visualize the true change-points locations.

  Figure 6.3 illustrates the typical behaviour of KCP when k is well-suited to the change-point problem we consider. It summarizes results obtained in Scenario 1 with k = k G the Gaussian kernel.

  distance (d F or d H ) between τ (D) and τ , as a function of D. Average risk R µ τ (D) , empirical risk L Pn µ τ (D)and penalized criterion as a function of D.

  Probability, for each instant i ∈ {1, . . . , n}, that τ = τ ( D) puts a change-point at i.

Figure 6 . 3 :

 63 Figure 6.3: Scenario 1: X = R, variable (mean, variance). Performance of KCP with kernel k G 0.1 . The value D and the localization of the true change-points in τ are materialized by vertical red lines.

  and 6.3c. The expectation of the penalized criterion is minimal at D = D (as well as for the risk of µ τ (D) ), and takes significantly larger values when D = D (Figure6.3b). Both curves exhibit a similar behavior and remain (uniformly) close to each other. As a result, KCP often selects a number of change-points close to its true value (Figure6.3c). Overall, this suggests that the model selection procedure used at Step 2 of KCP works fairly well.

Figure 6 . 4 :

 64 Figure 6.4: Scenario 2: X = R, constant mean and variance. Performance of KCP with three different kernels k. The value D and the localization of the true change-points in τ are materialized by vertical red lines. Probability, for each instant i ∈ {1, . . . , n}, that τ (D ) puts a change-point at i.

Figure 6 . 5 :

 65 Figure 6.5: Scenario 3: histogram-valued data. Performance of KCP with two different kernels k. Probability, for each instant i ∈ {1, . . . , n}, that τ (D ) puts a change-point at i. Vertical red lines show the true change-points locations.

  This update rule leads us to the following Algorithm 3, where each column C •,τ +1 in the first for loop is computed only once: Algorithm 3 Improved space and time complexity (Kernseg)1: for τ = 2 to n do 2: Compute the (τ + 1)-th column C •,τ +1 from C •,τ 3:for D = 2 to min(τ , D max ) do 4: L D,τ +1 = min τ ≤τ {L D-1,τ + C τ,τ +1 } point of view, each step of the first for loop in Algorithm 3 requires O(τ ) operations to compute C •,τ +1 and at most O(D max τ ) additional operations to perform the dynamic programming step at Line 4. Then the overall complexity is O(D max n 2 ) in time and O(D max n) in space.

Figure 6

 6 Figure 6.6: (Left) Runtime in seconds of Algorithm 3 as a function of the length of the signal (n) for D max = 100. (1-red) and a quartic computation of the cost matrix (2-violet). (Middle) Runtime in seconds of Algorithm 3 as a function of the length of the signal (1-red) and of ECP without permutation (2-blue) and ECP with the default number of permutations (3-green). (Right) Memory in mega-bytes of Algorithm 3 as a function of the length of the signal (1-red) and of ECP without permutation (2-blue) and ECP with the default number of permutations (3-green). The performances of ECP with or without permutation are exactly the same.

Figure 6 . 7 :

 67 Figure 6.7: Runtime as a function of n (length of the signal) for D max = 100. Runtime of our approximation algorithm with p = 100 (1-black) and p = √ n (2-orange), RBS (3-cyan), exact Algorithm 3 (4-red)

Figure 6

 6 Figure 6.8: Accuracy of ApKS with the Gaussian Kernel and for various p and of KS.Gau on (TCN,BAF) for a tumor percentage of 50% for D * .

  stopping rules and iterative learning algorithmsContext As explained in Section 1.1.1 (see Eq. (1.7)), numerous estimators are defined as the solution to an optimization problem such as

Figure 7 . 1 :

 71 Figure 7.1: Error of the iterative estimators with respect to the number t of iterations. Black: Excess loss. Red: Risk.

  elementary tool at the heart of our derivation is the following approximation used with fD , f De ∈ R d , f De -f D = De denotes some function in [f D , f De ], M (f ) = 1/n n i=1

  heuristic and reproducing kernels Context In the multiple change-point detection framework analyzed in Section 6.3.3, a penalized criterion has been designed to recover the best estimated segmentation from a huge collection of candidates.

=

  Main ideas to explore With τ (D) = argmin τ ∈T D n L Pn (τ ) and Π τ the orthogonal projector onto F τ , straightforward calculations lead toE µ -µ τ (D) 2 = E µ -Π τ (D) µ 2 + E Π τ (D) µ -µ τ (D) 2 E µ -Π τ (D) µ 2 + E Π τ (D) µ -Π τ (D) µ 2 -E Π τ (D) ε 2 -2E Π τ (D) µ , ε .

  ∀ 2 ≤ D ≤ D max , L D,n+1 = min τ ≤n { L D-1,τ + C τ,n+1 }, (7.2)which returns the best segmentation with D segments up to time n + 1 from those with D -1 segments up to time τ (with D ≤ τ ≤ n). Intuitively, this algorithm should be close to Algorithm 3 described earlier, except n increases at each time step.

  This suggests that the influence of p arises in the constants, but does not modify the global rate with respect to n. Nevertheless, increasing p when p ≤ n/2 + 1 deteriorates the upper bound, whereas the upper bounds improves as p grows when p > n/2 + 1. The worst concentration rate is obtained with p ≈ n/2. By comparison let us finally point out that existing results provided by van der Laan et al.

			,
			1)/n
	which remains of order O(1/n), and gives the same final O(1/	√	n) rate.
	(2004) lead to a O(1/ √ p) concentration rate, which is clearly worse than our O(1/ √	n).

  . Depending on our purpose, it could be useful to strengthen the O(1/( √ nλ)) rate to O(1/(nλ 2 )). This could be achieved if the constant Γ λ could be easily related to E R ECV p (A λ , D n ) , which is not the case with the present analysis (as with the one in Bousquet and Elisseeff (2002)). Since it is inherited from our upper bound on the L1O estimator, improving the latter would enhance the present one. Possible directions of improvements are discussed in Section 7.2.2. Second, let us now plug the upper bounds established in Theorem 4.6 for the kNN classification algorithm in Proposition 4.7. This leads to Corollary 4.2 (kNN binary classifier and {0, 1} loss, Proposition 4.2 in Celisse and Mary-Huard (2015)).

  Algorithm 4 BestSplit of segment [τ, τ [

	1: m = min τ <t<τ	τ <t<τ {C τ,t + C t,τ } and t = arg min	{C τ,t + C t,τ }
	2: Output four things (1) C τ,t -m, (2) t , (3) [τ, t[ and (4) [ t, τ [
	Algorithm 5 Binary Segmentation
	1: Segs = {[1, n + 1[}
	2: Changes = ∅
	3: CandidateSplit = ∅ [a binary heap]
	4: for D max iteration do
	5:	for aseg ∈ Segs do
	6:	Insert BestSplit(aseg) in CandidateSplit
	7:	end for
	8:	Extract the best split of CandidateSplit and recover: t, [τ, t[ and = [ t, τ [
	9:	Add t in Changes
	10:	Set Segs to { [τ, t[, [ t, τ [ }
	11: end for

Scientific locks

Remerciements

Algorithm 6 ApKS: Low rank approximation followed by binary segmentation 1: Compute the partial Gram-matrix K Jp,Jp 2: Use SVD to recover the p × n matrix Z 3: Run binary segmentation on Z

From this time complexity it arises that an influential parameter is the number p of columns of the matrix used to build the low-rank approximation. In particular this low-rank approximation remains computationally attractive as long as p = o( √ n). Figure 6.7 illustrates the actual time complexity of this fast algorithm (implemented in C) with respect to n for various values of p: (i) a constant value of p and (ii) p = √ n. To ease the comparison, we also plotted the runtime of the exact algorithm (Algorithm 3 detailed in Section 6. [START_REF]Change-point detection via cross-validation[END_REF].1) and that of a state-of-the-art procedure called RBS which relies on binary segmentation as well [START_REF] Pierre-Jean | Performance evaluation of DNA copy number segmentation methods[END_REF]. One main difficulty in the analysis of the iterative process is that we do not have access to the whole (estimated) risk curve up to the largest possible iteration number. Moreover the risk curve is not convex as a function of the iteration number. Going from one step to the next one does not actually optimize the risk itself but only an empirical proxy. Therefore one can be trapped into any local minimum that can arise. This makes the finite-sample performance of the oracle estimator (at the global minimum) definitely unachievable in the worst case (see the discussion introducing Eq. (1.17) in Blanchard et al., 2016, for instance).

KernSeg Exact and Heuristic

• Implicit definition of the optimum at each step:

In many realistic settings, the minimum location of the empirical criterion that is minimized at each step is only defined implicitly, that is without any closed-form expression for it. Classical instances of this arise with the coordinate descent algorithm [START_REF] Wright | Coordinate descent algorithms[END_REF], the binary segmentation heuristic [START_REF] Fryzlewicz | Wild binary segmentation for multiple change-point detection[END_REF], CART (Breiman et al., 1984b), and boosting algorithms for which the set of weak learners does not contain the (sub-)gradient (Biau and Cadre, 2017, Algorithm 1).

Main ideas to explore

• Designing ESR for iterative learning algorithms:

In contrast to what is done in [START_REF] Wei | Early stopping for kernel boosting algorithms: A general analysis with localized complexities[END_REF]; [START_REF] Yao | On early stopping in gradient descent learning[END_REF] for instance, improved ESR can arise from estimating (rather than upper bounding) the approximation and estimation errors. This can be done by means of a tight analysis of the empirical contrast minimization strategy and the use of concentration inequalities. A possible starting point for designing an ESR is to exploit ideas from Blanchard et al. (2016); [START_REF] Blanchard | Convergence rates of kernel conjugate gradient for random design regression[END_REF] where a maximum discrepancy-based stopping rule is derived. Preliminary results established in collaboration with Yaroslav Averyanov suggest that such stopping rules can be extended to the non-parametric regression setting where reproducing kernels are used to estimate the regression function. It is also important to emphasize that the filter representation of the estimators introduced in Blanchard et al. ( 2016) provides a general framework that turns out to be convenient to analyze a wide range of ongoing optimization/learning algorithms such as gradient boosting, stochastic gradient descent, . . . However the ESR of Blanchard et al. (2016), originally designed in the linear regression model, can lead to sub-optimal finitesample performances in settings where the true regression function almost belongs to the reproducing kernel Hilbert space compared to the stopping rule advocated by [START_REF] Raskutti | Early stopping and non-parametric regression: an optimal data-dependent stopping rule[END_REF], whereas it outperforms the latter in cases where the bias (approximation error) is larger. Modifying the stopping rule for taking into account the amount of bias suffered by the model is therefore an important step towards any improvement of the stopping rule.

• ESR and low-rank matrix approximation:

As earlier mentioned (Section 6.4.2), numerous machine learning procedures involve the storage and/or use of a n × n Gram matrix, which induces at least a O(n 2 ) time complexity. This high computational burden motivates the use of low-rank approximations to the Gram matrix such as the one proposed by [START_REF] Drineas | On the nyström method for approximating a gram matrix for improved kernel-based learning[END_REF], which relies on the choice of (the number of) rows/columns of the Gram matrix that will be serve for the approximation. Choosing (the number of) these rows/columns is a difficult task in itself and many iterative strategies have been designed, mainly inspired from the Gram-Schmidt orthonormalization [START_REF] Mahoney | Cur matrix decompositions for improved data analysis[END_REF][START_REF] Shawe-Taylor | Kernel methods for pattern analysis[END_REF]. However most of these iterative strategies only focus on approximating the Gram matrix instead of considering the final goal, that is classification for instance [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF]. To remedy this, [START_REF] Bach | Predictive low-rank decomposition for kernel methods[END_REF] have built a linear-time iterative strategy focusing on the prediction purpose. But it is so difficult to analyze that no theoretical guarantee does exist on its actual performance [START_REF] Bach | Sharp analysis of low-rank kernel matrix approximations[END_REF].

Designing such an iterative algorithm for which a theoretical analysis can be derived as well as an efficient ESR would greatly improve numerous machine/statistical learning procedures based on reproducing kernels.

• ESR and model selection:

Summary

The present manuscript mainly focus on cross-validation procedures (and in particular on leave-p-out (LpO)), describing its practical aspects as well as new strategies leading to non-asymptotic theoretical guarantees on its statistical performance (concentration inequalities, oracle inequalities). As a privileged application, cross-validation is also used to address the multiple change-points detection problem in the off-line context. This problem is then tackled in a more general framework by means of reproducing kernels and the model selection paradigm.

After introducing the cross-validation procedures in Chapter 1, ongoing strategies allowing us to efficiently compute cross-validation estimators are detailed in Chapter 2. In particular several of them yield closed-form expressions for the LpO estimator, which considerably reduces the computational cost. Such closed-form expressions have been already derived in density estimation with projection and kernel estimators, and with k-nearest neighbors estimators in the regression and binary classification contexts.

Chapter 3 discusses the statistical properties of the cross-validation estimators (used as risk estimators) in terms of bias, variance, and mean squared error. For instance among cross-validation estimators, it is established that the LpO one enjoys the lowest variance for a given test set cardinality. The leave-one-out (L1O) estimator is also proved to be asymptotically optimal in terms of mean squared error in density estimation with projection estimators.

Several approaches leading to concentration inequalities of the LpO estimator around its expectation are discussed in Chapter 4. A direct approach relying on the combination of closed-form expressions and the classical concentration inequalities of Bernstein and Talagrand is first exposed in the density estimation context. A more general approach is then described which exploits the link between the LpO estimator and U-statistics. Its main underlying idea is to deduce exponential concentration results for the LpO estimator from moment inequalities. The derivation of the preliminary results also involve the stability of the used learning algorithm.

The important question of model/statiscal algorithm selection is addressed in Chapter 5 in the particular case of density estimation. The optimality of the LpO-based model selection procedure is proved under some conditions both in the estimation purpose-by means of a non-asymptotic oracle inequality-and in the identification purpose-through a model consistency result.

Cross-validation is then used to tackle the multiple change-points detection problem in the off-line setting, where the variance is allowed to vary along the time (heteroscedastic setting). Chapter 6 summarizes the conclusions drawn from theoretical as well as empirical results about the behavior of cross-validation procedures. In particular, these conclusions lead us to suggest new model selection procedures relying on cross-validation. At the price of a higher computational cost, these procedures automatically take into account changes arising in the variance for instance, which improves the statistical performance. The more general question of detecting changes arising the full distribution of the observations (and not only in the mean) is also addressed by means of reproducing kernels. A new model selection procedure is designed that is based on a penalty derived in the reproducing kernel Hilbert space framework. Its non-asymptotic performance is quantified through an oracle inequality with high probability. Numerous aspects of the new procedure are also empirically assessed in the empirical study. For instance, the results illustrate that the chosen kernel clearly influences the final performance.

Finally the manuscript ends with Chapter 7 highlighting several challenging perspectives which could give rise to important improvements both on the practical and theoretical sides.