Keywords: kernel methods, rkhs, normality test, outlier detection Modèles Gaussiens et Méthodes méthodes à noyaux, rkhs, test de normalité, détection d'anomalie

When you get new rules that work, you're changing the physiology of your brain. And then your brain has to reconfigure itself in order to deal with it.

Chapter1

General Introduction

Issues adressed by this thesis

The world of statistics is traditionally divided into two halves: non-parametric statistics and parametric statistics. Parametric statistics -as the name suggests -are based on a parametric model, that is a collection {P θ } θ∈Θ of distributions that may have generated the observed dataset.

There are several advantages of using a parametric model: firstly the parameter θ stands as a summary of the underlying mechanism governing the data which enables interpretation, and secondly narrowing the possible representation of this mechanism to a simple model allows a better inference on θ -which is well-known by statisticians as the bias-variance trade-off. On the other hand, an inaccurate model may lead to bad performances of the learning task at play. Besides, a model must be defined with respect to the type of observed data, that is one must design specific models for vectorial data, qualitative data, structured objects, heterogeneous data, and so on.

By contrast, non-parametric statistics do not impose such distributional assumptions on data.

One family of such non-parametric procedures are kernel methods. Kernel methods rely on a transformation φ : X → H (the feature map) where X (the input space) is the set where our data take values and H is a typically high-dimensional space also called the feature space or kernel space. The idea is to map the initial dataset into H through φ and to work on the transformed data lying in the kernel space instead of the initial data. The interest of such mapping is to get properties on transformed data that initial data may not satisfy. As an illustration, let us take a look at Support Vector Machines (SVM) [BGV92; CV95] which is the archetype of kernel methods. SVM is an algorithm of supervised binary classification which separates two classes of observations by means of a linear boundary. Clearly using a linear classifier directly on the initial data may be problematic as the two classes may not be linearly separable. Hence SVM aims at linearly separating the two classes in the kernel space instead of the input space, which 1 is a feasible task because of the high dimensionality of the kernel space. Since the introduction of SVM, many other kernel methods have been conceived throughout the last two decades and cover a wide field of applications: regression [START_REF] Saunders | Ridge regression learning algorithm in dual variables[END_REF], denoising [START_REF] Schölkopf | Nonlinear Component Analysis as a Kernel Eigenvalue Problem[END_REF], hypothesis testing [START_REF] Gretton | A Kernel Method for the Two-Sample-Problem[END_REF][START_REF] Gretton | A Kernel Statistical Test of Independence[END_REF], outlier detection [Sch+01; Hof07], . . . A remarkable feature of kernel methods is that it is possible to do non-parametric statistics and parametric statistics at the same time, by applying a parametric statistical procedure in the kernel space. Such an approach is non-parametric in the sense that no distributional assumption is made in the input space. Instead, we consider a family of mappings and if this family is large enough, chances are that an adequately chosen mapping will "shape" the data to make it fit a given parametric model in the kernel space. So far, this approach has only been exploited by a few papers in the litterature, including [START_REF] Bouveyron | Kernel discriminant analysis and clustering with parsimonious Gaussian process models[END_REF] for classification and [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] for batch outlier detection.

Albeit promising, this parametric approach to kernel methods raises several questions:

• Is a given parametric model well-defined in the kernel space? The kernel space is often represented as a functional space (reproducing kernel Hilbert space or RKHS) which is typically infinite dimensional. Since the number of dimensions is infinite, special care must be taken to define properly a distribution in the kernel space, in particular such a distribution must be guaranteed to lie almost surely in the kernel space.

• Is a given parametric model realistic in the kernel space? Considering a family of feature maps, the question is whether this family is "rich" enough so that it contains a map that yields a distribution in the kernel space which is accurately described by a given parametric model. Another concern is whether such a map preserves the information carried by the initial data so that the parametric procedure in the kernel space still allows inference.

• How to test competing parametric models in the kernel space? Two or more parametric models may be in competition in the kernel space. To decide between those different models, a goodness-of-fit test may be used. The objective is to devise such a test that handles samples taking values in a possibly infinite-dimensional Hilbert space, such as a kernel space.

Therefore the aim of the present thesis is to bring answers to the questions above.

Outline of this thesis

This thesis is outlined as follows:

Chapter 2 consists in an introduction to kernel methods. The aim of the chapter is to explain the interest of kernel methods through motivating examples, to introduce essential notions such as kernels, positive definite functions and reproducing kernel Hilbert spaces, and also to provide several instances of kernels. Besides, we also mention elements that will be useful throughout

Outline of this thesis

Chapter 6 presents a goodness-of-fit test for normality dedicated to samples taking values in a general Hilbert space (which may be an RKHS). This test is based on the Maximum Mean Discrepancy (MMD) which is known to be used for homogeneity and independence testing. Our procedure allows to test a large variety of Gaussian models (zero-mean Gaussian, Gaussian with covariance of rank r, . . . ). Furthermore, a fast parametric bootstrap procedure adapted from [START_REF] Kojadinovic | Goodness-of-fit Testing Based On A Weighted Bootstrap: A Fast Large-Sample Alternative to the Parametric Bootstrap[END_REF] is successfully applied to our test and allows to reduce its computational cost. Our test displays good empirical performances and mild sensibility to high dimensions (unlike common multivariate normality tests), outperforming goodness-of-fit tests based on random projections [START_REF] Cuesta-Albertos | The Random Projection Method in Goodness of Fit for Functional Data[END_REF]. Finally, we propose an application of our test to covariance rank selection.

Finally, Chapter 7 concludes by providing some possible future works which could stem from the results of this thesis.

Chapter 2

Background: an Overview of Kernel Methods This chapter presents an introduction to kernel methods, a set of methods used in the machine learning community for the past two decades and relying on the same principle -the "kernel trick". The aim of the present chapter is not to be exhaustive, but to provide a broad overview of this topic.

Motivating examples

Broadly speaking, kernel methods are based on a transformation called feature map that maps available data taking values in a set X -called the input space -into a (typically) highdimensional space called feature space or kernel space. This introductory section aims at presenting three different cases that motivate the use of such a mapping: supervised binary classification, Fisher discriminant analysis and structured objects. Each of these examples highlights one useful aspect of kernelization, which are respectively:

• Benefit from the high-dimension nature of the feature space (Section 2.1.1),

• Applying a linear method in the kernel space to solve a non-linear problem in the input space (Section 2.1.2),

• Applying methods made for vectorial data in the kernel space to deal with non-vectorial data in the input space (Section 2.1.3).

Figure 2.1 -Left plot: Two groups of points not linearly separable in R 2 ; right plot: Two linearly separable classes after embedding into R 3 .

Supervised binary classification

Let us consider the problem of supervised binary classification. Assume that we are given n observations (x 1 , y 1 ), . . . , (x n , y n ) ∈ X × {-1, 1} where X is some set. Given a test point x ∈ X , the goal is to guess the corresponding y ∈ {-1, 1} based on the n observations. where sgn(.) denotes the sign of its argument.

The advantage of such a linear classifier is its simplicity. However, the drawback of such approach is that the existence of an hyperplane that separates the two classes is not always guaranteed. For instance, the left plot in Figure 2.1 1 displays two classes of points in R 2 , one class (blue, triangle) being enclosed by the other one (red, circle). It is clear that there exists no straight line in R 2 to separate these two groups of points, as a natural boundary would be instead the green circle displayed in the left plot of Figure 2.1. However, one can apply the following map to our observations:

R 2 → R 3 , (x 1 , x 2 ) → (x 1 , x 2 , x 2 1 + x 2 2 ) ,
which yields the transformed dataset represented in three dimensions in the right plot of 1 The plots in Figure 2.1 were taken from http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html Figure 2.1. In this new setting, it is possible to find a plane (e.g. the plane of equation z = 0.5) in R 3 which separates the two classes. This toy example illustrates the usefulness of embedding a dataset into a space of higher dimension. More generally, linear classifiers are known to be powerful enough in high-dimensional settings. This is related to the notion of the Vapnik-Chernovenkis dimension (or VC-dimension) of a set of classifiers. Considering a set of classifiers C ⊂ {(A, A c ) | A ⊂ X } as an ensemble of two-part partitions of a set X , the VC-dimension of C is defined as the maximum number h(C ) of points in X such that, for each of the 2 h(C ) ways of splitting these h(C ) points into two classes, there exists a classifier in C that classifies correctly all of the h(C ) points. In more formal terms,

h(C ) ∆ = max        h ∈ N * ∀(x 1 , . . . , x h ) ∈ X h , ∀I ⊆ {1, 2, . . . , h}, ∃(A, A c ) ∈ C , h i=1 1 1 x i ∈A 1 1 i∈I ∈ {0, h}        .
In the particular case of linear classifiers in R d , it is known that the corresponding VC-dimension is d + 1 (see [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF], Section 3.6), that is a set of n points -split into two groups -lying in a vector space of dimension d with d n -1 is always linearly separable.

Fisher discriminant analysis

Let x 1 , . . . , x n ∈ R D be a sample of n D-variate vectors split into M classes, and let y 1 , . . . , y n ∈ {1, 2, . . . , M} be the corresponding labels, that is y i = j means that x i belongs to the class j for every 1 i n and 1 j M. When d is large, it is desirable to search for a (linear) transformation T : R D → R d such that d is much smaller than D and T (x 1 ), . . . , T (x n ) allows to visualize data (when p 3) and identify the structure of the classes.

Fisher discriminant analysis (FDA) [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF] is a classical method of dimensionality reduction that consists in finding a direction that best separates those M clusters. Therefore it corresponds to a transformation T : R D → R d that is linear and such that d = 1. Introducing the pooled mean µ = (1/n) n i=1 x i and the inner class means µ j = n i=1 x i 1 1 y i =j / n i=1 1 1 y i =j for 1 j M, the goal is to find a direction u * ∈ R D which maximizes the following quantity Optimal projection (input space)

u * ∈ argmax u∈R D ,|u| 2 =1 u T S B u u T S W u , ( 2
x -5 0 5

Optimal projection (feature space)

x Figure 2.2 -Top-left: points X 1 , . . . , X n split into two classes in R 2 (red triangles and blue circles); bottom-left: overlapping projected points found by FDA; top-right: transformed points φ(X 1 ), . . . , φ(X n ) in the feature space; bottom-right: well separated projected points in the feature space found by FDA.

denote respectively the between class scatter matrix and within class scatter matrix.

The idea is that maximizing (2.1.1) is equivalent to maximizing the numerator u T S B u and minimizing the denominator u T S W u. A large u T S B u means that the within class means of the projected sample u T x 1 , . . . , u T x n are distant from each other, while a small u T S W u implies that the within class variances of the projected sample are all small. Therefore a larger value for the objective (2.1.1) indicates a better separability between the M clusters.

However, an optimal direction u * does not necessarily provide satisfactory separation for the projected sample, even though the clusters are not overlapping in R D before projection. For instance, the top left plot of Figure 2.2 presents two classes of points in R 2 (red triangles, blue circles) which lie on distinct subsets of R 2 . Despite this, the projected points onto the optimal direction found by FDA (Figure 2.2, bottom left plot) form two overlapping classes.

In the example of Figure 2.2, the use of an appropriate feature map may be helpful. For This example shows that through the use of a non-linear feature map φ, a problem in the input space for which linear methods such as FDA fail turns to a problem in the feature space that can be solved by such methods.

Text data and bag-of-words

In the era of Big Data, data are often available in a raw form, that is observations are not always represented as multivariate vectors with real entries, but as a batch of heterogeneous types of data: vectors but also graphs, trees, categorical data, text, images, audio streams,. . . The main problem is that most data analysis tools are designed for a specific type data -typically multivariate data. Instead of creating ad hoc methods corresponding to each conceivable type of data, one may instead transform the initial dataset to get vectorial data.

We illustrate this approach with the bag-of-words technique [START_REF] Joachims | Learning to Classify Text Using Support Vector Machines[END_REF] used to represent a text as a multivariate vector. The bag-of-words method is simple (Figure 2.3 2 ). Given a finite set of m relevant words (e.g. "dog", "cat", "economy", "kafkaesque" but not "the", "where", "but") denoted by w 1 , . . . , w m , an observed text x is mapped to a histogram φ(x) ∈ R m where [φ(x)] i is the number of occurrence of the word w i in the text x, for every i ∈ {1, . . . , m}. This way, any algorithm designed for multivariate data (linear classification, principal component analysis,. . . ) can be applied to φ(x) to treat x.

A straightforward way of choosing the relevant words is to take a dictionary and, based on a corpus of texts, to weed out the most observed words (stopword elimination) -typically words like "the" or "if" -and the least observed ones (document frequency thresholding).

Kernels: definition and basic properties

Section 2.1 showed through examples the usefulness of embedding data into a feature space H through a feature map φ. In practice, this embedding is represented by a function called kernel.

The objective of this section is to define formally the notion of kernel, how it is linked to positive definite functions and to illustrate the so-called "kernel trick", the latter allowing to perform all practical computations only in the input space. The notion of reproducing kernel Hilbert spaces (RKHS) [START_REF] Aronszajn | Theory of Reproducing Kernels[END_REF] is covered as well, in order to provide an explicit construction of the feature map and the feature space.

Definition

In the following, we introduce the formal definition of a kernel.

Definition 2.2.1 (Kernel). Let X be a set and k : X × X → R. k is called a (real valued) kernel if there exist a real Hilbert space H endowed with an inner product ., . H and a map φ : X → H such that:

∀x, y ∈ X , k(x, y) = φ(x), φ(y) H .

In particular, a kernel k can be defined by an expansion of the form k(x, y) = D r=1 φ r (x)φ r (y) , (2.2.2)

where (φ r ) 1 r D are real-valued functions defined on X and D ∈ N * ∪ {+∞}, when considering a feature map of the form φ : X → R D , x → (φ 1 (x) . . . φ D (x)) T .

A kernel k defined by an expansion as in (2.2.2) is called a Mercer kernel. Furthermore, Mercer's theorem [START_REF] Mercer | Functions of Positive and Negative Type, and their Connections with the Theory of Integral Equations[END_REF] states that conversely, a kernel k admit the representation (2.2.2) under the assumption that X is a compact topological space and k is continuous. However this result still holds true if the compactness assumption for X is removed [START_REF] Steinwart | Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHS[END_REF].

In Examples 2.2.1 to 2.2.3 thereafter, we provide examples of kernels defined through an implicit feature map as in Definition 2.2.1.

Example 2.2.1. (Linear kernel)

The most simple kernel on X = R d (d 1) is the canonical scalar product ∀x, y ∈ R d , k(x, y) = y T x .

The corresponding feature map/space are

H = X = R d , φ = Id R d .
Example 2.2.2. (Tree kernel)

In this example, we will define a tree kernel as proposed in [START_REF] Collins | Convolution Kernels for Natural Language[END_REF]. To compare two given trees T 1 and T 2 , we build a kernel following the idea that two trees are similar if they share many common subtrees. Let {S i } 1≤i≤M be the set of all the M possible subtrees.

Consider the feature map: h(T ) = (h 1 (T ), . . . , h M (T )) where h i (T ) counts the number of occurrences of S i in T .

The tree kernel is defined by the canonical dot product in R M :

k(T 1 , T 2 ) = M i=1 h i (T 1 )h i (T 2 ) .
Remark that M is exponentially much larger than the number of nodes in T 1 and T 2 . To be able to compute k, the tree kernel is written in another form. Let I i (n) the indicator equal to 1 if S i is rooted at the node n and 0 otherwise. Then h i (T ) = n∈T I i (n) and K can be formulated in a different way

k(T 1 , T 2 ) = M i=1         n 1 ∈T 1 I i (n 1 )                 n 2 ∈T 2 I i (n 2 )         = n 1 ∈T 1 n 2 ∈T 2 M i=1 I i (n 1 )I i (n 2 ) =C(n 1 ,n 2 )
.

On the first hand, k becomes a sum over the pairs of nodes (n 1 , n 2 ), and on the other hand, C(n 1 , n 2 ) can be computed quickly with a recursive procedure. Therefore the computation of the tree kernel does not depend on M.

Example 2.2.3. (Fisher kernel) An a priori about the behaviour of the dataset can be expressed by a generative model. Such a model involves a family of distributions P(.|θ) on X parameterized by θ ∈ Ω ⊆ R D where Ω is the parameter space. Haussler and Jaakkola (1999, [START_REF] Jaakkola | Using the Fisher kernel method to detect remote protein homologies[END_REF]) introduced a type of kernels called Fisher kernels which take into account such a model.

Fisher kernels are built upon the feature map

φ θ 0 : X → R D , φ θ 0 (x) = ∂ log P(x|θ) ∂θ i (θ 0 ) 1≤i≤D ,
where θ 0 is a given point in Ω. φ θ 0 (x) represents how to modify the current setting (with parameter set at θ 0 ) to maximize log(P(x|θ)) and make the model fit x better.

Two points x and y are similar if they "draw" the model towards similar directions, thus a Fisher kernel is defined as an inner product in R D between φ θ 0 (x) and φ θ 0 (y). Namely, let I be the Fisher information matrix

I = E X φ θ 0 (X) i φ θ 0 (X) j 1≤i,j≤D ,
where X follows the distribution P(.|θ 0 ). For any x, y ∈ X , the Fisher kernel k related to the model {P(.|θ)} θ∈Ω and a given parameter value θ 0 is defined by k(x, y) = φ θ 0 (x) T I -1 φ θ 0 (y) .

Note that given a kernel k, the corresponding pair of feature map/space (φ, H) is not unique.

For instance, consider the linear kernel of Example 2.2.1, k(x, y) = x T y for every x, y ∈ R d (d 1).

Example 2.2.1 defined φ = Id R d and H = R d as the corresponding feature map/space. However, one could have also considered the following feature space/map H and φ as

H = R 2d , φ : R d → R 2d , x → 1 √ 2 x, 1 √ 2 x ,
where for any u, v ∈ R d , (u, v) ∈ R 2d denotes the vector obtained by concatenating u and v. It is then trivial to check that ∀x, y ∈ X , k(x, y) = φ(x) T φ(y) = φ(x) T φ(y) .

Positive definite functions and reproducing kernel Hilbert spaces

The kernels shown in the examples of Section 2.2.1 were constructed by explicitly defining a feature map φ and a feature space H. However the function k : X × X → R is sometimes defined in the first place without any preliminary definition of φ and H. In fact, k(x, y) is often understood as a measure of similarity between x, y ∈ X and is directly constructed following this rationale. For instance if X is endowed with a distance function d : X × X → R + , the function k can be written as

∀x, y ∈ R d , k(x, y) = κ(d(x, y)) ,
for some function κ : R + → R with κ(0) = sup t∈R κ(t).

This raises the following question: how to tell whether a given k is a kernel or not? It turns out that answering this question does not require to explicitly build a feature space and a feature map and is related to the notion of symmetric, positive definite functions.

Definition 2.2.2 (Symmetric, positive definite function). A function k : X × X → R is positive definite if for every a 1 , . . . , a n ∈ R and x 1 , . . . , x n ∈ X :

n i,j=1
a i a j k(x i , x j ) ≥ 0 .

Moreover if k(x, y) = k(y, x) for every x, y ∈ X , k is symmetric.

Remark: Often in the literature, the expression positive definite function is replaced by positive semi-definite function. With the additional condition that i,j a i a j k(x i , x j ) = 0 if and only if a 1 = . . . = a n = 0, "positive definite" becomes "strictly positive definite" in the first naming, and "positive semi-definite" becomes "positive definite" in the second one, which may be confusing. In the following, we stick to the first terminology.

The interest of this notion is that k is a symmetric, positive definite function if and only if k is a kernel, as we show in the following.

Straightforwardly, a kernel k is necessarily symmetric and definite positive. Given a corresponding feature space H -equipped with the inner product ., . H and associated norm . H with h H = h, h H for every h ∈ H -and feature map φ, then for any a 1 , . . . , a n ∈ R and x 1 , . . . , x n ∈ X n i,j=1 a i a j k(x i , x j ) = n i,j=1 a i a j φ(x i ), φ(x j ) H = n i=1 a i φ(x i ) 2 H ≥ 0 , and for every x, y ∈ X k(x, y) = φ(x), φ(y) H = φ(y), φ(x) H = k(y, x) , by symmetry of the inner product of H. Therefore, a kernel k is always positive definite and symmetric.

Showing the converse requires to build a canonical feature map φ and feature space H. The construction of φ and H thereafter follows the proof of Theorem 4.16 from [START_REF] Steinwart | Support Vector Machines[END_REF]. Given a symmetric, positive definite function k, consider the following pre-hilbertian space of functions H 0 consisting of linear combinations of evaluations functions k(x, .), x ∈ X

H 0 =        n i=1 a i k(x i , .) n ∈ N * , a 1 , . . . , a n ∈ R, x 1 , . . . , x n ∈ X        .
which is equipped with the inner product ., . H 0 defined for any h = n i=1 a i k(x i , .) ∈ H 0 and h = m j=1 a j k(x j , .) ∈ H 0 as

h, h H 0 = n i=1 m j=1
a i a j k(x i , x j ) .

Let us check that ., . H 0 actually defines an inner product.

First of all, ., . H 0 is well defined since h, h H 0 can be written h, h = m j=1 a j h(x j ) = m i=1 a i h (x i ), which shows that h, h H 0 does not depend on the representation of h and h .

The positivity condition is fulfilled by the positive definiteness of k, since for every h = n i=1 a i k(x i , .)

∈ H 0 h, h H 0 = n i=1 n j=1
a i a j k(x i , x j ) 0 .

It is straightforward that ., . H 0 is symmetric and bilinear. Therefore it satisfies the Cauchy-Schwarz's inequality h, h H 0 h, h H 0 h , h H 0 which implies that •, • H 0 is definite as follows: for any h ∈ H 0 such that h, h

H 0 = 0, ∀x ∈ X , |h(x)| = | h, k(x, •) H 0 | | h, h H 0 ||k(x, x)| = 0 , hence h = 0.
Now we are in a position to define a feature space H as the completion of H 0 with respect to ., . H 0 . H is therefore a Hilbert space whose inner product ., . H satisfies

∀h, h ∈ H 0 , h, h H = h, h H 0 .
In particular, this implies the so-called reproducing property ∀x, y ∈ X , k(x, .), k(y, .) H = k(x, y) .

The reproducing property shows that choosing H as a feature space and

φ : X → H, x → k(x, .) , as a feature map yields ∀x, y ∈ X , k(x, y) = k(x, .), k(y, .) H = φ(x), φ(y) H , proving that k is a kernel.
H is called the reproducing kernel Hilbert space (RKHS) of k [START_REF] Aronszajn | Theory of Reproducing Kernels[END_REF] and is denoted H(k).

Furthermore, every kernel k is related to a unique RKHS ([SC08], Theorem 4.21).

Definition 2.2.3 (Reproducing kernel Hilbert space). For any symmetric, positive definite function k : X × X → R, there exists a unique Hilbert space H(k) equipped with an inner product ., . H(k) which satisfies:

• ∀x ∈ X , k(x, .) ∈ H(k) , • ∀f ∈ H(k), ∀x ∈ X , f , k(x, .) H(k) = f (x) .
Such an H(k) is called reproducing kernel Hilbert space with reproducing kernel k.

Apart from providing a canonical form for the feature space and the feature map, H(k) has the remarkable property of being the simplest" possible feature space corresponding to k, that is H(k) is the smallest possible feature space in the following sense: Theorem 4.21 from [START_REF] Steinwart | Support Vector Machines[END_REF] states that given any feature space/map pair ( H, φ) related to k, the following operator H → H(k), w → w, φ(.) H , is surjective. As Section 2.1.2 suggested, one of the motivations for the use of a kernel is to apply a linear algorithm in a feature space such as H. Such an algorithm deals with inner product such as w, φ(x) H where w ∈ H and x ∈ X , which shows that viewing the feature space as an RKHS suffices.

Finally, we illustrate this section by introducing a family of symmetric, positive definite functions (hence kernels): translation-invariant kernels.

Example 2.2.4. (Translation-invariant kernels) Let us set X = R d for some d 1 and define k as

∀x, y ∈ R d , k(x, y) ∆ = K(x -y) ,
where K : R d → R.

The function K is said to be positive definite if the corresponding kernel k is positive definite. Furthermore, assuming that K is continuous, Bochner's theorem states that K is positive definite if and only if K is the Fourier transform of a bounded positive measure on R d ([BT04], Theorem 20), that is

K(z) = e iw T z Λ(dw) ,
for some bounded positive measure Λ on R d . Therefore, the family of translation-invariant kernels encompasses several instances of commonly used kernels on R d such as:

• Gaussian kernel: k(x, y) = exp(-γ x -y 2 ), γ > 0 .
This kind of kernel corresponds to K(z) = exp(-γ z 2 ) which is positive definite since K is the characteristic function of a N (0, 2γI d ) Gaussian distribution.

• Laplace kernel: k(x, y) = exp (-γ xy ) , γ > 0 .

Here K(z) = exp(-γ z ) corresponds to the characteristic function of a random vector rU , where U is uniform on the unit sphere of R d and r is independent of U and follows a Cauchy distribution with density function f (t) = πγ 1 + (t/γ) 2 -1 .

• Cauchy kernel: The Cauchy kernel is a parametric kernel (with parameter σ > 0) with

formula k(x, y) = 1 1 + σ -2 x -y 2 ,
for every x, y ∈ R d Compared to the Gaussian and Laplace kernels, the Cauchy kernel emphasized more possible influences between distant points [START_REF] Basak | A least square kernel machine with box constraints[END_REF].

• B-spline kernels:

Define B 0 = 1 1 B 1,d the indicator function of the unit ball B 1,d of R d .
For every function f , g : R d → R, let f ⊗ g denote the convolution of f and g that is Some translation-invariant kernels such as the Gaussian kernel, the Laplace kernel and the Cauchy kernel admit actually in the more specific form k(x, y) = κ( xy ) where κ : R + → R.

(f ⊗ g)(x) = R d f (x )g(x -x)
These particular kernels are called radial kernels or radial basis function kernels (RBF kernels).

The "kernel trick"

As shown in Section 2.2.1, a kernel k is designed to match the inner product of some feature space. The feature space is typically high-dimensional -a property that is desired within some frameworks as the example of Section 2.1.1. However, manipulating high-dimensional vectors may be more of a curse than a blessing -especially in regard of computational costs.

In the following, we expose how using a kernel allows to eventually perform computations in the input space, hence avoiding the computational burden of high dimension. This practical usefulness of kernels is due to two aspects: the "kernel trick" and the representer theorem.

"Kernel trick"

Here is a simple illustration of the "kernel trick". Given x and y two elements in the input space and a kernel k, the distance between the embedded points φ(x) and φ(y) in H(k) can be calculated as follows

φ(x) -φ(y) 2 H(k) = φ(x) -φ(y), φ(x) -φ(y) H(k) = φ(x), φ(x) H(k) + φ(y), φ(y) H(k) -2 φ(x), φ(y) H(k) = k(x, x) + k(y, y) -2k(x, y) .
Therefore computing a distance in the RKHS only requires to evaluate the kernel k on elements of the input space. There is no need to manipulate directly any vectors in the feature space: this is the principle of the "kernel trick".

More generally, given a set of observations x 1 , . . . , x n ∈ X , if computations in the RKHS only consist of pairwise inner products φ(x i ), φ(x j ) H(k) for 1 i, j n, then calculating the Gram matrix K ∈ M n (R) defined by

K = k(x i , x j ) 1 i,j n ,
suffices. It turns out that most learning algorithms only require the Gram matrix, as implied by the representer theorem stated thereafter.

Representer theorem

Most learning algorithms can be cast as an optimization problem. Given observations (x 1 , y 1 ), . . . , (x n , y n ) ∈ X × R, the goal is to seek out a prediction function f * such that Y i = f * (x i ) + i for every 1 i n where 1 , . . . , n are error terms. Such a function is determined by minimizing (or maximizing) a functional of the general form

Ω(f ) = c ((x 1 , y 1 , f (x 1 )), . . . , (x n , y n , f (x n ))) + R(f ) , (2.2.3) where f ∈ F ⊆ R X with F a prescribed set of functions, c : (X × R 2 ) m → R + is a loss function
and R(f ) is a regularizer term which typically controls the "smoothness" of f in some sense to avoid overfitting. For instance in the case of ridge regression, an optimal regression function

f * : R d → R satisfies f * ∈ argmin f (x)=a T x, a∈R d n i=1 (a T x i -y i ) 2 + λ a 2 ,
where λ > 0. Here F = {x → a T x | a ∈ R d } is the set of linear regression functions, the loss function c is the squared loss n i=1 (a T x iy i ) 2 and R(f ) = λ a 2 . The representer theorem provides an explicit form for the optimizers of functionals as (2.2.3) when F is an RKHS. Originally introduced in [START_REF] Kimbeldorf | Some Results on Tchebycheffian Spline Functions[END_REF] in the special case of the squared loss, it has been generalized by [START_REF] Schölkopf | A generalized representer theorem[END_REF]. The latter version is stated in Proposition 2.2.4. Proposition 2.2.4 (Representer theorem, [START_REF] Schölkopf | A generalized representer theorem[END_REF]). Let k : X × X → R be a kernel where X is a non-empty set and let (x 1 , y 1 ), . . . , (x n , y n ) ∈ X × R.

Assume that the regularizer R(f ) in (2.2.3) is a strictly increasing (resp. decreasing) function of f H(k) . Then, any f * ∈ H(k) that minimizes (resp. maximizes) Ω(f * ) lies in the span of k(x 1 , .), . . . , k(x n , .),

that is ∃a 1 , . . . , a n ∈ R, f * = n i=1 a i k(x i , .) .
(2.2.4)

Proof of Proposition 2.2.4. We focus on the case where Ω(f ) is strictly increasing w.r.t. ||f ||, the other case being proved likewise.

Let V = span(k(x 1 , .), . . . , k(x n , .)), and f ∈ H(k). Let us split f into the sum of two orthogonal functions:

f = P V f ∈V + f ⊥ ∈V ⊥
, which is possible since V is finite dimensional and H(k) is a Hilbert space (hence complete).

Using the reproducing property and the fact that f ⊥ is orthogonal to V and k(x i , .) ∈ V for every 1 i n,

∀1 i n, f (x i ) = P V f (x i ) + f ⊥ (x i ) = P V f (x i ) , since f ⊥ (x i ) = f ⊥ , k(x i , .) H(k) = 0.
It follows

Ω(f ) = c((x 1 , y 1 , f (x 1 )), . . . , (x n , y n , f (x n ))) + R(f ) = c((x 1 , y 1 , P V f (x 1 )), . . . , (x n , y n , P V f (x n ))) + R(f ) .
Finally, since P V f and f T are orthogonal

||f || 2 = ||P V f || 2 + ||f ⊥ || 2 ≥ ||P V f || 2 .
Therefore since R is strictly increasing with respect to f H(k) ,

Ω(f ) = c((x 1 , y 1 , P V f (x 1 )), . . . , (x n , y n , P V f (x n ))) + R(f ) > c((x 1 , y 1 , P V f (x 1 )), . . . , (x n , y n , P V f (x n ))) + R(P V f ) = Ω(P V f ) .
It proves that any minimizer of Ω(f ) necessarily belongs to V .

Thus determining f * reduces to determining the scalars a 1 , . . . , a n ∈ R. We can therefore reformulate the problem in terms of matrices and vectors (called the dual problem) which is more convenient for resolution. Furthermore, the dual writing of the optimization problem does not involve the number of dimensions of the feature space but only the number of observations n.

Some examples of kernel methods

Now that we have properly introduced the definition of a kernel and some of their key properties, we are in a position to review some of the most encountered statistical and/or learning methods that make use of kernels. Many of them turn out to be generalized version of pre-existing procedures: Kernel Principal Component Analysis [START_REF] Schölkopf | Nonlinear Component Analysis as a Kernel Eigenvalue Problem[END_REF] which is a non-linear extension of PCA, Kernel Ridge Regression [START_REF] Saunders | Ridge regression learning algorithm in dual variables[END_REF], Kernel Fisher discriminant analysis [START_REF] Mika | Fisher Discriminant Analysis with Kernels[END_REF], Kernel Canonical

Correlation Analysis [START_REF] Lai | Kernel and nonlinear canonical correlation analysis[END_REF], among others.

Due to the vast variety of kernel methods, this section has not the ambition to present an exhaustive taxonomy of such methods. We restrict ourselves to three examples that will be mentioned or useful throughout this thesis: Support Vector Machine for classification, Kernel PCA for dimension reduction and Maximum Mean Discrepancy for two-sample and independence testing.

Support Vector Machine

Introduced by [START_REF] Boser | A Training Algorithm of Optimal Margin Classifiers[END_REF] and then [START_REF] Cortes | Support-vector networks[END_REF], Support Vector Machine (SVM) is a learning algorithm designed for binary classification. We recall the binary classification framework -that we have already encountered in Section 2.1.1: we are given n pairs of observations (x 1 , y 1 ), . . . , (x n , y n ) ∈ X ×{-1, 1} where X is an arbitrary set. The objective is to guess the class y ∈ {-1, 1} corresponding to a new point x ∈ X based on (x i , y i ) 1 i n .

The main idea of SVM is to construct a linear decision function that separates the two classes and to determine y according to the side of the decision hyperplane where the tested point x lies.

Furthermore, this decision boundary is not set in the input space X but in a high dimensional feature space H where x 1 , . . . , x n are embedded into, in order to make the two classes linearly separable. More precisely, assuming that X is endowed with a kernel k : X × X → R with corresponding feature space/map (H, φ), the SVM algorithm aims at finding a vector f ∈ H that solves the following constrained optimization scheme

minimize 1 2 f 2 H + C n i=1 ξ i over f ∈ H, b ∈ R, ξ 1 , . . . , ξ n ∈ R + such that y i ( f , φ(x i ) H + b) 1 -ξ i for all 1 i n , (2.3.5)
where C > 0.

Let us denote optimal separating hyperplane as

{h ∈ H | f * , h H + b * = 0} where (f * , b * ) is a
solution of (2.3.5). Thus a tested point x ∈ X will be assigned the class y ∈ {-1, 1} defined by

y = sgn( f * , φ(x) H + b * ) .
Let us discuss the inequality constraints y i ( f , φ(x i ) H + b) 1ξ i in (2.3.5). In a first time, we omit the variables ξ 1 , . . . , ξ n and focus on the simplified optimization scheme

minimize 1 2 w 2 H over f ∈ H, b ∈ R such that y i ( f , φ(x i ) H + b) 1 for all 1 i n . (2.3.6)
The effect of optimizing (2.3.6) is twofold. On the first hand, the constraints y i ( f , φ(x i ) H +b) 1 require that y i ( f , φ(x i ) H + b) is positive for every 1 i n, which means that every training point φ(x 1 ), . . . , φ(x n ) is correctly classified by the separating hyperplane. On the other hand, minimizing f 2 H makes the distance between the training points and the decision hyperplane as large as possible. To see this, rewrite (2.3.6) as follows minimize N

over

N ∈ R + , b ∈ R + , f H = 1 such that y i ( f , φ(x i ) H + b) 1 N for all 1 i n . where f = f / f 2 H(k) , b = b/ f 2 H(k) and N = f 2 H(k)
. The drawback of the optimal decision boundary that stems from (2.3.6) is that it tolerates no classification error among the training points. It may be a problem if one of the training points is Figure 2.5 -Illustration of Support Vector Machine an outlier. This lack of robustness may lead to a well-known problem in learning theory called overfitting, that is the learning algorithm sticks too much to the training set and causes a large generalization error -a tested point is more likely to be misclassified.

To tackle this overfitting issue, [START_REF] Cortes | Support-vector networks[END_REF] introduced the slack variables ξ 1 , . . . , ξ n in (2.3.5).

These slack variables allow a few training points to be exceptionally close to the separating hyperplane or even to be misclassified. This more robust approach leads to a smaller generalization error. Now that the objective of the SVM is clearly set, we show that solving (2.3.5) is feasible in practice, despite the fact that the optimization is done over a possibly infinite dimensional space H. In particular, the representer theorem can be applied in this framework so that the optimization problem does not depend on the dimension of H. Namely, (2.3.5) is equivalent to

minimize 1 2 f 2 H + C n i=1 max (0, 1 -y i ( f , φ(x i ) H + b)) over f ∈ H, b ∈ R . (2.3.7)
If the feature space H is chosen as the RKHS H(k), the reproducing property entails f , φ

(x i ) H(k) = f (x i ) for every 1 i n and (2.3.7) becomes minimize 1 2 f 2 H + C n i=1 max (0, 1 -y i (f (x i ) + b)) over f ∈ H, b ∈ R ,
which is of the same form as the objective (2.2.3) in Section 2.2.3 with c(x i , y i , f

(x i )) = C max(0, 1- y i (f (x i ) + b))
as the loss function -in this case, the hinge loss -and

R(f ) = (1/2) f 2 H(k)
. Hence the representer theorem states that the optimal f * admits the following expansion

f * = n i=1 a i k(x i , .) .
Introducing the vector a = (a 1 . . . a n ) T ∈ R n and the Gram matrix K = [k(x i , x j )] 1 l eqsi,j n , (2.3.5)

finally becomes minimize 1 2 a T Ka + C n i=1 ξ i over f ∈ H, b ∈ R, ξ 1 , . . . , ξ n ∈ R + such that y i (Ka + b) 1 -ξ i
for all 1 i n , which depends on n but not on the dimension of H(k).

Kernel PCA

Before introducing kernel principal component analysis (kernel PCA), let us recall the principle of linear PCA which is a classical tool in statistics for dimension reduction.

In the classical framework, we are given n i.i.d. observations X = [X 1 . . . X n ] in R m which are assumed zero-mean. The goal is to find the directions u 1 , . . . , u d that capture the major part of the variance of the observations. To do so, we first have to find the unit vector u 1 that maximizes the variance u T 1 Σu 1 , where Σ = E X XX T is the covariance matrix of X. The associated Laplacian to be maximized is

L(u 1 , λ) = u T 1 Σu 1 -λ(u T 1 u 1 -1) ,
which is equivalent to solving the eigenvalue equation

Σu 1 = λ 1 u 1 , (2.3.8)
where λ 1 the largest eigenvalues of Σ. The following vector u 2 is sought in the same way but such that it is orthogonal to u 1 , and so on. Therefore, the principal components u 1 , . . . , u d correspond to the first d eigenvectors of Σ.

In practice, we consider the sample covariance estimator Σ = (1/n) n i=1 X i X T i , so that the equation (2.3.8) becomes

1 n XX T u 1 = λ 1 u 1 .
In [START_REF] Schölkopf | Nonlinear Component Analysis as a Kernel Eigenvalue Problem[END_REF], the authors have proposed a kernelized version of the PCA, and applied it successfully to image denoising. Given a kernel k, kernel PCA applies usual PCA in the feature space H. For the sake of clarity, kernel PCA is presented in the case where H is a finite dimensional linear space but it can be extended as well to infinite dimensional settings by means of Hilbert-Schmidt operators algebra.

Assume H = R D where D is typically very large. Writing the feature map as φ(x) = (φ 1 (x), . . . , φ D (x)) T ∈ R D , the images of our observations in H are stored in the matrix

Φ ∈ M D,n (R) Φ = [φ(X 1 ), . . . , φ(X n )] .
As in classical PCA, we have to solve the following eigenvalue equation

1 n ΦΦ T u 1 = λ 1 u 1 .
(2.3.9)

We can not solve it directly, since the matrix ΦΦ T ∈ M D (R) is too large. Instead we transform

(2.3.9) and multiply it from the left by

Φ T 1 n Φ T Φ =K Φ T u 1 =v 1 ∈R n = λ Φ T u 1 =v 1 , where K = [ φ(X i ), φ(X j ) H ] 1 i,j n = [k(X i , X j )] 1 i,j n is the Gram matrix. Therefore, it becomes an eigenvalue problem in R n Kv 1 = nλ 1 v 1 .
Having found a wanted v 1 , we can express the eigenvector u 1 as follows

Φ T u 1 = v 1 ⇔ ΦΦ T u 1 =nλ 1 u 1 = Φv 1 ⇔ u 1 = 1 nλ 1 Φv 1 .
Finally we normalize u 1 by u 1 H to obtain ũ1 ∈ R D . Since

u 2 H = 1 n 2 λ 2 1 v T 1 Φ T Φv 1 = 1 n 2 λ 2 1 v T 1 Kv 1 = 1 nλ 1 , we get ũ1 = 1 √ nλ 1 Φv 1 .
Projecting a new embedded point φ(X) with X ∈ X onto the normalized principal component ũ1 yields with

φ(X) T ũ1 = 1 √ nλ 1 φ(X) T Φv 1 ,
φ(X) T Φ = (k(X 1 , X), . . . , k(X n , X)) .
Note that every actual computation has been made all along in the input space through k, which is an instance of the kernel trick. In the seminal paper of [START_REF] Schölkopf | Nonlinear Component Analysis as a Kernel Eigenvalue Problem[END_REF], an application of kernel PCA consists in "cleaning out" noisy data by projecting an embedded point φ(X) ∈ H onto principal components in the feature space to obtain P φ(X) ∈ H, then by going back into the input space.

The latter step is not trivial since most vector in H do not admit a pre-image through the feature map φ. In particular, there exists in general no element X * ∈ X satisfying φ(X * ) = P φ(X). Instead, X * is determined as a solution of the following optimization problem

X * = argmin x∈X φ(x) -P φ(X) 2 H ,
which is solved by gradient descent.

Another way to define X * is to consider the problem of going back to the input space as a regression problem [START_REF] Gokhan | Learning to find preimages[END_REF]. The regularization in the regression procedure allows to get a "smoother" approximate pre-image, which is shown to be efficient in practice for image denoising (see Figure 2.6 taken from [START_REF] Gokhan | Learning to find preimages[END_REF]).

Other instances of applications of kernel PCA include regularized binary classification [START_REF] Zwald | Performances d'Algorithmes Statistiques d'Apprentissage: "Kernel Projection Machine" et Analyse en Composantes Principales à Noyaux[END_REF] and outlier detection [START_REF] Hoffmann | Kernel PCA for novelty detection[END_REF].

Maximum Mean Discrepancy

Let X 1 , . . . , X n and Y 1 , . . . , Y m be two samples of i.i.d. X -valued random variables, where (X i ) 1 i n were drawn from a distribution P X and (Y i ) 1 i m from a distribution P Q , where P X and P Q are both defined on the measurable space (X , A). The purpose of a two-sample test is to test the null-hypothesis H 0 : P X = P Y against the alternative hypothesis H 1 : P X P Y . In order to set such a test, one needs a test statistic that measures the discrepancy between P X and P Y based on the two available samples. This test statistic is typically an estimator of some quantity ∆(P X , P Y ) that quantifies the gap between the unknown distributions P X and P Y .

A series of papers [Bor+06; Gre+07a; Gre+09] have proposed to use kernels for this task.

Their method rely on an Hilbert space embedding of distributions.

Definition 2.3.1 (Hilbert space embedding of distributions, Lemma 3 in [START_REF] Gretton | A Kernel Two-Sample Test[END_REF]). Let M + 1 (X ) be the set of probability distributions on X and k a (measurable) kernel on X that satisfies sup x∈X k(x, x) <

+∞.

Then the following mapping µ is well-defined

µ : M + 1 (X ) → H(k), P → µ[P ] ,
where for each

P ∈ M + 1 (X ), µ[P ] ∈ H(k) is the element of H(k) that satisfies ∀f ∈ H(k), µ[P ], f H(k) = E X∼P [ k(X, •), f H(k) ] .
For every

P ∈ M + 1 (X ), µ[P ] is called the Hilbert space embedding of P in H(k).
This can be seen as a generalization of the notion of moment-generating function -i.e.

a function of the type ϕ(w) = E X e w T X related to some random vector X. It is clear that a moment-generating function corresponds to a Hilbert space embedding with an exponential kernel k(x, y) = exp(x T y).

Through the embedding µ, each distribution P on X is represented by a mean element µ [P ] in H(k). This mapping is injective for a class of kernels called characteristic kernels.

Definition 2.3.2 (Characteristic kernel). Let k a kernel on X and µ the corresponding Hilbert space embedding of distributions.

k is said to be characteristic if and only if µ is injective, that is

∀P , Q ∈ M + 1 (X ), P = Q ⇐⇒ µ[P ] = µ[Q] .
Criteria for a kernel to be characteristic have been widely investigated in [Fuk+09; Sri+10; Through the Hilbert space embedding of distribution µ, the discrepancy between P X and P Y can be gauged by the following quantity

MMD(P X , P Y ) ∆ = µ[P X ] -µ[P Y ] 2 H(k) ,
that is equal to 0 if and only if P X = P Y whenever k is a characteristic kernel. MMD(P X , P Y ) is called the maximum mean discrepancy (MMD) between P X and P Y .

To set a two-sample test, MMD(P X , P Y ) is estimated based on (X i ) 1 i n and (Y i ) 1 i m . In-

troducing PX = (1/n) n i=1 δ X i (resp. PY = (1/n) m i=1 δ Y i ) the empirical measure of the sample (X i ) 1 i n (resp. (Y i ) 1 i m ) leads to the following test statistic MMD(P X , P Y ) ∆ = MMD( PX , PY ) = 1 n n i=1 k(X i , •) - 1 m m j=1 k(Y j , •) 2 H(k)
.

Expanding the squared norm and the kernel trick lead to a fully computable statistic

MMD(P X , P Y ) = 1 n 2 n i,j=1 k(X i , X j ) + 1 m 2 m i,j=1 k(Y i , Y j ) - 2 mn n,m i,j=1 k(X i , Y j ) .
The same rationale can be applied to design an independence test. Let (X 1 , Y 1 ), . . . , (X n , Y n ) ∈ X × Y be n independent copies of an X × Y -valued random variable (X, Y ) following the joint distribution P XY . Let P X and P Y denote respectively the marginal distributions of X and Y . An independence test aims at testing the null hypothesis H 0 that X and Y are independent against the alternative H 1 that they are dependent. By definition, X and Y are independent if and only if the joint distribution P XY and the product distribution P X ⊗ P Y coincide3 .

Therefore the independence testing framework is reformulated as follows

H 0 : P XY = P X ⊗ P Y versus H 1 : P XY P X ⊗ P Y ,
and an adequate test statistic is an the MMD between the empirical measures of P XY and P X ⊗ P Y [START_REF] Gretton | A Kernel Statistical Test of Independence[END_REF]. This statistic is called the Hilbert space independence criterion (HSIC). Namely, given a kernel K defined on X × Y , the HSIC writes

HSIC(X, Y ) = MMD(P XY , P X ⊗ P Y ) = 1 n n i,j=1 K ((x i , y i ), (., .)) - 1 n 2 n i,j=1
K (x i , y j ), (., .)

2

H(K) = 1 n 2 n i,j=1 K (x i , y i ), (x j , y j ) + 1 n 4 n i,j,k,l=1 K (x i , y j ), (x k , y l ) - 2 n 3 n i,j,k=1 K (x i , y i ), (x j , y k ) .
In [START_REF] Gretton | A Kernel Statistical Test of Independence[END_REF], kernels K of the form K((x, y), (x , y ) = k(x, x )l(y, y ) are considered where k is a kernel on X and l is a kernel on Y which leads to the following expression of the HSIC

HSIC(X, Y ) = 1 n 2 Tr (KHLH) ,
where Tr(.) denotes the trace of a matrix, K

= [k(X i , X j )] 1 i,j,n , L = [l(Y i , Y j )] 1 i,j,n and H = I n -(1/n)1 1 n 1 1 T n is a centering matrix.
In this case, [START_REF] Gretton | A simpler condition for consistency of a kernel independence test[END_REF] shows that if k and l are both characteristic, then the HSIC is consistent -that is HSIC(X, Y ) → 0 almost surely as n → +∞ under H 1 . This is a useful result since one does not need to check whether the product kernel K is characteristic.

Efficient computation of the kernel matrix

Through the kernel trick, we are able to run algorithm in some high dimensional feature space while performing actual calculations in the input space through evaluations of the kernel k on pairs of observations X 1 , . . . , X n ∈ X summarized by the Gram matrix (or kernel matrix)

K = k(X i , X j ) 1 i,j n
, so that computational costs are not affected by the number of dimensions of the feature space but only on the sample size n. However, kernel methods may still be computationally expensive when n is large. In particular, the obtainment of K requires n 2 entries to compute and store, while algorithms in linear time with respect to n are more desirable nowadays. This section is devoted to presenting methods used to tackle this problem by approximating K with low-rank representations, thus setting the computation/memory costs down to the order of O(n). More exactly, we are interested in factorizations of the Gram matrix as follows K = UV T , such that U, V ∈ M n,r (R) and r is much smaller than n. Such approaches include the Nyström method [START_REF] Williams | Using the Nyström method to speed up kernel machines[END_REF] and Random Kitchen Sinks with its variation Fastfood [RR07; LSS13].

Note that the goal is not to find the best low-rank factorization of K but to find a satisfactory low-rank representation of the Gram matrix at a reduced computational cost, typically in linear time. Actually the best r-rank approximation K r of K can be defined as the matrix K r of rank r < n that minimizes K -K r F where A 2 F = Tr(AA T ) defines the Froebenius norm for every A ∈ M n (R). The expression of K r depends on the eigenexpansion of the Gram matrix since it reads

K r = U r D r U r ,
where D r = diag(d 1 , . . . , d r ) contains the r largest eigenvalues of K on its diagonal and where the columns of U r are the eigenvectors of K (with unit norm) corresponding to d 1 , . . . , d r . However obtaining K r requires to compute -at least partially -the eigenexpansion of K which costs a too demanding execution time of order O(n 2 r).

Nyström method

The Nyström method consists in subsampling the columns of K to build a low-rank approximation of K. Namely, consider a subset I ⊆ {1, . . . , n} of cardinality r < n and define the approximate Gram matrix K as

K ∆ = K •,I K † I ,I K I ,• , (2.4.10)
where

K I ,I = [k(X i , X j )] i,j∈I , K I ,• = [k(X i , X j )] i∈I ,1 j n , K •,I = K T I ,
• and for any matrix A ∈ M r (R), A † denotes the pseudo-inverse4 of A.

The computation of (2.4.10) is in linear time with respect to n since K I ,• takes nr entries to calculate, K I ,I has r 2 entries to calculate and its inversion costs r 3 operations hence a total of

nr + r 2 + r 3 operations, which is of order O(n) if r is negligible compared to n. Note that KI,• = K I ,I K † I ,I K I ,• = K I ,•
, that is the rows (and likewise the columns) of K and K indexed by I coincide. This is due to the fact that the Nyström method amounts to project the embedded points k(X 1 , .), . . . , k(X n , .) onto the subspace V of H(k) spanned by the subsample {k(X i , .)} i∈I . To see this, let us introduce the orthogonal projector Π : H(k) → H(k) projecting onto V and L = Πk(X i , .), Πk(X j , .) H(k) the Gram matrix corresponding to projected points in the RKHS. It follows for every 1 i, j n with i ∈ I ,

L i,j = Πk(X i , .), Πk(X j , .) H(k) = Π * Πk(X i , .), k(X j , .) H(k) = Πk(X i , .), k(X j , .) H(k) = k(X i , .), k(X j , .) H(k) = K i,j ,
where Π * = Π is the adjoint operator of Π. Furthermore, there exists only one matrix K of rank r < n satisfying KI,• = K I ,• ([BJ05], Proposition 1), hence K = L.

There exist several strategies for the choice of the subset I . In [START_REF] Williams | Using the Nyström method to speed up kernel machines[END_REF], I is picked at random uniformly from the set of all subsets of {1, . . . , n} of cardinality p. [START_REF] Smola | Sparse greedy matrix approximation for machine learning[END_REF] uses sparse greedy approximation that consists in selecting the indexes in I one after the other -namely at each iteration, the selected index optimizes some criterion. However, to make this greedy selection efficient, the new index is chosen at each iteration from a randomly picked subset of the nonchosen indexes instead of trying all the remaining indexes. The proposal of [START_REF] Drineas | On the Nyström method for approximating a Gram matrix for improved kernel-based learning[END_REF] is close to that of [START_REF] Williams | Using the Nyström method to speed up kernel machines[END_REF] except that they assign different probabilities p i = K ii / n i=1 K ii to each i ∈ {1, . . . , n} and construct I by picking from {1, . . . , n} p subsequent times with replacement according to these probabilities.

It remains to determine the minimal value of r that makes the approximation of K satisfactory.

This problem is studied in [START_REF] Francis | Sharp analysis of low-rank kernel matrix approximations[END_REF] in the case of a uniform sampling of I . [START_REF] Francis | Sharp analysis of low-rank kernel matrix approximations[END_REF] points out that the quality of approximation of the Gram matrix should not be assessed by the matrix error K -K F but by the performance of the statistical task at stake when using the approximate Gram matrix. In particular, [START_REF] Francis | Sharp analysis of low-rank kernel matrix approximations[END_REF] focuses on the case of regression problems -such as SVMand shows that the generalization error obtained with K is of the same order as that with K if p is larger than a certain quantity that depends on the eigenspectrum decay of K and the regression function.

Random Fourier features

Another way to approximate k is to build a low-rank kernel k defined explicitly through a feature map φ : X → R p where the number of features p is small. Random Kitchen Sinks [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] proposes such an approach for translation-invariant kernels k : R d × R d → R. We recall the definition of translation-invariant kernels mentioned previously in Section ??. A kernel k is translation-invariant if there exists a function

K : R d → R such that ∀x, y ∈ R d , k(x, y) = K(x -y) .
Bochner's theorem states that K is the Fourier transform of a positive measure Λ on R d , that is

∀z ∈ R d , K(z) = R d
e iz T w Λ(dw) .

(2.4.11)

Note that since K is real-valued, (2.4.11) can be written in the following alternate way

∀z ∈ R d , K(z) = Re R d e iz T w Λ(dw) = R d
Re(e iz T w )Λ(dw) = R d cos(z T w)Λ(dw) , (2.4.12)

where Re(.) denotes the real part of a complex number.

Let us assume that K is properly normalized so that Λ is a probability measure -that is R d Λ(dw) = 1. The basic idea of Random Kitchen Sinks is to randomly draw a few points w 1 , . . . , w p ∈ R d from the distribution Λ and consider a Monte-Carlo estimation of the integral in (2.4.12). Namely, k is approximated by a low-rank kernel k as follows

∀x, y ∈ R, k(x, y) = K(x -y) = R d cos({x -y} T w)Λ(dw) 1 r r i=1
cos({x -y} T w i ) complexity of order O(dr). Repeating this computation for every x = X i with 1 i n leads to a running time of order O(ndr).

In the case of radial kernels where K(z) is a function of z , the fastfood method proposed by [START_REF] Le | Fastfood-approximating kernel expansions in loglinear time[END_REF] improves the running time of Random Kitchen Sinks with respect to d so that the complexity changes from linear to logarithmic dependence on d. Assuming in a first time that r = d, Fastfood bypasses the direct sampling of W and the computation of W x by replacing W with the matrix W

W ∆ = 2γ d SH d GΠH d B ∈ M d (R) , (2.4.13)
where S, G, B ∈ M d (R) are diagonal matrices, Π ∈ M d (R) is a permutation matrix and

H d ∈ M d (R)
is the so-called Walsh-Hadamard matrix defined recursively by

H 2 =       1 1 1 -1      
S d-1 of R d . Since k(•, •) = K(•-•
) is assumed to be a radial kernel, K is the Fourier transform of a spherical distribution Λ, that is the distribution of a random variable of the form rU where U is uniform on S d-1 and r is a positive random variable independent of U . Therefore setting the diagonal entries of S as i.i.d. copies of r/( G 2 F d) implies that the rows of W follow the distribution Λ. In the end, the actual difference between W and W is that the rows of W are not independent. Despite this dependency, [START_REF] Le | Fastfood-approximating kernel expansions in loglinear time[END_REF] proved that the approximate kernel k converges pointwise to k almost surely as r tends to infinity.

Chapter 3

Gaussian Models in Reproducing Kernel Hilbert Spaces

This chapter is devoted to defining probability distributions in an RKHS, especially Gaussian distributions. Conditions that guarantee that such distributions lie almost surely in a given RKHS will be of interest. In particular in the case of a Gaussian process parameterized by mean and covariance functions, we establish that such a process is well defined in the empirical setting where those parameters are estimated on the basis of a finite sample. In a second time, we present a few existing methods using Gaussian models in kernel spaces that can be found in the literature.

Gaussian distributions in RKHS

Let k : X ×X → R be a symmetric, positive definite function defined on a set X and let H(k) denote the corresponding RKHS. H(k) can be seen either as a function space -whose elements are seen as real functions -or as a mere Hilbert space -whose elements are seen as vectors. Each of these two points of view corresponds to a different way of defining a probability distribution on an RKHS: either through a stochastic process or through a random element. Definition 3.1.1 (Stochastic process). Let X be a non-empty set and (Ω, A, P) a probability space. A stochastic process Y is a collection Y = {Y (x)} x∈X of real-valued random variables defined on (Ω, A, P). Definition 3.1.2 (Random element). Let (Ω, A, P) be a probability space and (H, A ) a measurable space. A random element Z in (H, A ) is a (A, A )-measurable function Z : Ω → H, that is for every

B ∈ A , the pre-image Z -1 (B) = {ω ∈ Ω | Z(ω) ∈ B} belongs to A.
Let us focus on the case where H = H(k) is an RKHS. Since H(k) is endowed with an inner product •, • H(k) , the σ -algebra A can be defined as the cylindrical σ -algebra C H(k) , which is the coarsest σ -algebra that contains all the cylinder sets C A,h 1 ,...,h m ⊆ H(k) with

C A,h 1 ,...,h m ∆ = h ∈ H(k) h, h 1 H(k) . . . h, h m H(k) ∈ A ,
where m ∈ N * , h 1 , . . . , h m ∈ H(k) and A ⊆ R m is a Borel subset of R m . In other words, an RKHSvalued random element Z can be characterized by the distribution of its real-valued marginals

Z, h H(k) , h ∈ H(k).
A Gaussian distribution in H(k) corresponds to either a Gaussian stochastic process or a Gaussian random element. A multivariate Gaussian distribution is parametrized by its mean and its variance, which are respectively a vector and a square matrix. In the following, we show how the notions of mean and covariance are transposed to stochastic processes and RKHS-valued random elements.

The mean of a stochastic process

{Y (x)} x∈X is a map m : X → R defined by ∀x ∈ X , m(x) ∆ = E Y [Y (x)] , (3.1.1)
where we assume that the expectation in (3.1.1) exists for every x ∈ X .

The covariance function of a stochastic process

{Y (x)} x∈X is a map R : X × X → R defined by ∀x, x ∈ X , R(x, x ) ∆ = E Y [{Y (x) -m(x)}{Y (x ) -m(x )}] , (3.1.2)
assuming that the expectation in (3.1.2) exists for every x, x ∈ X .

In the case of a random element Z, the mean and the covariance are defined differently.

Assume that Z is a random element with weak first order, that is

∀h ∈ H(k), E Z [ Z, h H(k) ] < +∞ . (3.1.3)
Then the mean of the random element Z is the element µ ∈ H(k) that satisfies

∀h ∈ H(k), E Z [ Z, h H(k) ] = µ, h H(k) . (3.1.4)
The existence of µ is guaranteed by Riesz's Representation Theorem. Indeed, (3.1.3) implies that the linear map h → E Z [ Z, h H(k) ] is bounded hence continuous so that Riesz's Representation Theorem states the existence of µ ∈ H(k) so that (3.1.4) holds.

Assume that the random element Z is of weak second order, that is

∀h ∈ H(k), E Z [ Z, h 2 H(k) ] < +∞ . (3.1.5)
If (3.1.5) holds then the following bilinear form S :

H(k) × H(k) → R is well defined ∀h, h , S(h, h ) ∆ = E Z [ Z -µ, h H(k) Z -µ, h H(k) ] , (3.1.6)
The covariance operator of the random element Z is the operator Σ :

H(k) → H(k) that satisfies ∀h, h ∈ H(k), Σ • h, h H(k) = S(h, h ) . ( 3
= {Y (x)} x∈H(k) as ∀x ∈ H(k), Y (x) ∆ = Z, k(x, •) H(k) . (3.1.8)
Clearly Y defines a Gaussian stochastic process. Indeed for every a 1 , . . . , a m ∈ R and x 1 , . . . , x m ∈ X ,

m i=1 a i Y (x i ) = m i=1 a i Z, k(x i , •) H(k) = Z, m i=1 a i k(x i , •) H(k)
, is a Gaussian random variable because k(x i , •) ∈ H(k) for every 1 i m since H(k) is an RKHS.

The respective mean and covariances of Z and Y as defined in (3.1.8) can be linked as follows.

Because of the reproducing property of H(k), the mean function m :

X → R of Y is expressed with respect to the mean µ ∈ H(k) of Z as follows ∀x ∈ X , m(x) = µ, k(x, •) H(k) = µ(x) ,
so that m and µ actually coincide.

Straightforwardly, the covariance function R(x, x ) of Y satisfies for every

x, x ∈ X R(x, x ) = Σ • k(x, •), k(x , •) H(k) .
Defining conversely a random element Z from a stochastic process Y is less straightforward.

To do so, one has to assume that there exists a subset of trajectories -or sample paths -

T ⊆ {Y (x)(ω) | ω ∈ Ω} of Y that lie in H(k) with P(Y ∈ T ) = 1.
When this is the case, one can define the random element Z in H(k) as

Z(ω) ∆ = Y (•)(ω) ,
for every ω ∈ Ω such that Y ∈ T and and Z(ω) = 0 for instance for every ω ∈ Ω such that Y T .

However, such T does not necessarily exist. Here is a simple example that illustrates this. Let

k = •, •

Nuclear dominance

We saw that a H(k)-valued random element Z always defines a corresponding stochastic process

Y (x) = Z, k(x, •) H(k) but
R n = [R(x i , x j )] 1 i,j n and K n = [k(x i , x j )
] 1 i,j n for every n 1. If the quantity τ defined as

τ = lim n→+∞ Tr(R n K -1 n ) , (3.2.9) is finite, then P(Y ∈ H(k)) = 1.
Even though it provides some answers, the result of [START_REF] Driscoll | The Reproducing Kernel Hilbert Space Structure of the Sample Paths of a Gaussian Process[END_REF] relies on some restrictive assumptions, such as the almost sure continuity of Y or the continuity of the covariance function R(•, •).

[LB01] improved Driscoll's result by relaxing those assumptions. In particular, [START_REF] Lukic | Stochastic Processes wih Sample Paths in Reproducing Kenel Hilbert Spaces[END_REF] simplified the definition of τ in (3.2.9) by relating τ to the trace of some operator called dominance operator.

Definition 3.2.2 (Dominance operator). Let R and k be two symmetric, positive definite functions and H(R), H(k) denote their respective RKHS.

Assume H(R) ⊆ H(k). Then there exists a bounded operator L :

H(k) → H(R) such that ∀h ∈ H(k), ∀f ∈ H(R), L • h, f H(R) = h, f H(k) .
In particular,

L • k(x, •) = R(x, •) for every x ∈ X .
The operator L is called dominance operator of H(k) over H(R).

Moreover, if L : H(k) → H(k) is a positive, continuous and self-adjoint operator, then R(x, x ) = L • k(x, •), k(x , •) H(k) is a positive definite kernel such that H(R) ⊆ H(k).
The existence of L as asserted by definition 3.2.2 comes from Theorem 1.1 in [START_REF] Lukic | Stochastic Processes wih Sample Paths in Reproducing Kenel Hilbert Spaces[END_REF]. Remark that the operator L as defined in 3.2.2 coincide with the covariance operator Σ of the Gaussian random element Z defined from Y when Y ∈ H(k) almost surely. Indeed for every x, x ∈ X

Σ • k(x, •), k(x , •) H(k) = R(x, x ) = R(x, •), k(x , •) H(k) = L • k(x, •), k(x , •) H(k) ,
hence by bi-linearity and continuity of

•, • H(k) , Σ • f , g H(k) = L • f , g H(k) for every f , g ∈ H(k)
and Σ = L.

According to Proposition 4.5 in [START_REF] Lukic | Stochastic Processes wih Sample Paths in Reproducing Kenel Hilbert Spaces[END_REF], the quantity τ as defined in (3.2.9) corresponds to the trace of the dominance operator L. To define the trace of L, we need to assume that H(k) is separable so that an orthonormal Hilbertian basis {e i } i∈N * exists and

τ = Tr(L) ∆ = i 1 L • e i , e i H(k) .
Note that this definition of the trace does not depend on the choice of the orthonormal basis {e i } i 1 (see Theorem VI.18 in [START_REF] Reed | Methods of Modern Mathematical Physics I: Functional Analysis[END_REF]). Also remark that a sufficient condition for H(k) to be separable is that X is a separable topological space and k is continuous on X × X (see Lemma In particular, a consequence of Proposition 3.2.4 is that there cannot be any isotropic Gaussian process in an infinite-dimensional RKHS. In an RKHS H(k), an isotropic stochastic process is a stochastic process with covariance function R(x, x ) = k(x, x ) for every x, x ∈ X . In other words, the dominance operator of H(k) over H(R) is the identity operator L = Id H(k) . Given an orthonormal Hilbertian basis {e i } i∈N * of H(k),

Tr(L) = Tr(Id H(k) ) = i 1 Id H(k) • e i , e i H(k) = i 1 e i 2 H(k) = i 1 1 = +∞ ,
hence nuclear dominance does not hold and Proposition 3.2.4 entails the non-existence of an isotropic Gaussian process in H(k).

Validity of Gaussian processes in the empirical case

In this section, we consider the empirical case where a sample x 1 , . . . , x n ∈ X of X -valued observations is available and where the embedded data k(x 1 , •), . . . , k(x n , •) ∈ H(k) are modeled as a Gaussian process. The parameters of this Gaussian process need to be chosen with respect to the dataset. Hence two things are required: firstly, define empirical estimators for the mean and the covariance function of the Gaussian process, and secondly check whether the corresponding Gaussian process is well-defined in H(k) for this choice of parameters.

A natural estimator for the mean and the covariance function is to consider the mean/covariance of the empirical measure n i=1 δ k(x i ,•) , which gives rise to the empirical mean and the empirical covariance function. Definition 3.3.1 (Empirical mean). Given an embedded sample k(x 1 , •), . . . , k(x n , •), the corresponding empirical mean m : We show that R << k by exhibiting the corresponding dominance operator L:

X → R is defined by ∀x ∈ X , m(x) ∆ = 1 n n i=1 k(x i , x) . (3.3.10) Definition 3.3.2 (Empirical covariance function). Given an embedded sample k(x 1 , •), . . . , k(x n , •), the corresponding empirical covariance function R : X × X → R is defined by ∀x, x ∈ X , R(x, x ) ∆ = 1 n n i=1 [k(x i , x) -m(x)][k(x i , x ) -m(x )] . ( 3
L • f ∆ = 1 n n i=1 f , k(x i , •) H(k) k(x i , •) - m(•) n n i=1 f , k(x i , •) H(k) , (3.3.12) for every f ∈ H(k).
First of all, it is clear that L

• k(x, •) = R(x, •) for all x ∈ X . Let us check that L is a bounded operator. Since | f , k(x i , •) | H(k) f H(k) k 1 2 (x i , x i ) by Cauchy-Schwarz's inequality, one obtains that for f H(k) = 1, L • f H(k) 1 n n i=1 k(x i , x i ) + 1 n         1 n 2 n j,l=1 k(x j , x l )         1/2 n i=1 k(x i , x i ) 1/2 < +∞ ,
hence the boundedness of L.

Finally since the operator L is of finite rank (up to n), it has a finite trace. Therefore we can conclude that R << k.

Due to the high-dimensional nature of an RKHS, the empirical covariance function as defined in (3.1.2) may be unwieldy for some applications. For instance, in the methods described later in Sections 3.4.1 and 3.4.2, the covariance function needs to be "inverted" in the following sense.

Consider the eigen-expansion of the empirical covariance function R

∀x, x ∈ X , R(x, x ) = r j=1 λj φj (x) φj (x ) , (3.3.13)
where the pairs ( λj , φj ) 1 j r are the eigenvalues/eigenvectors of the corresponding dominance operator L -defined in (3.3.12) -that satisfy

∀1 j r, L • φj = λj φj ,
and φj , φl H(k) = δ jl for every 1 j, l r and δ jl denotes the Kronecker delta. Here r n denotes the rank of L. Therefore "inverting" R(•, •) means inverting the operator L which is done by inverting the eigenvalue in the eigen-expansion (3.3.13). This operation becomes problematic when r is very large as the smallest eigenvalues tend to 0 when r → +∞ -because of the finiteness of Tr( L) required by the nuclear dominance conditions. Hence inverting those small eigenvalues may result in computational instability in practice.

To overcome this issue, a class of parsimonious Gaussian models have been proposed [START_REF] Mcnicholas | Parsimonious Gaussian mixture models[END_REF][START_REF] Bouveyron | Kernel discriminant analysis and clustering with parsimonious Gaussian process models[END_REF] where the rd smallest (non-zero) eigenvalues are assumed to be equal to the same value (for some 1 d r). The parameter d of the model is called intrinsic dimension and has to be interpreted as the number of directions that carry most of the information of the dataset whereas the rd other directions consists of noise. Following this model, the empirical covariance function is "tweaked" in the following way: whereas the d largest eigenvalues of R remain unchanged , the other eigenvalues are replaced by their average value. The resulting parsimonious covariance estimator is formally defined thereafter.

Definition 3.3.4 (Parsimonious covariance estimator). Write the eigen-expansion of R as in (3.3.13).

The parsimonious sample covariance Rd (x, x ) is defined for every x, x by

Rd (x, x ) ∆ = d j=1 λj φj (x) φj (x ) + r l=d+1 γl r -d r j=d+1 φj (x) φj (x ) . (3.3.14)
It remains to show that a Gaussian process with Rd as covariance function is well-defined in an RKHS. This is done by first proving that nuclear dominance of R over Rd (Lemma 3.3.5) which combined with the already known relationship R << k entails Rd << k (Proposition 3.3.6).

Lemma 3.3.5. Nuclear dominance of the sample covariance estimator R over the parsimonious estimator Rd holds.

Proof of Lemma 3.3.5. Considering the eigenexpansion of R as written in (3.3.13), the dominance operator between R and Rd can be defined by

Ld • f = d j=1 f , φj H(k) φj +        1 r -d r l=d+1 λl        r j=d+1 λ-1 j f , φj H(k) φj .
Straightforwardly Ld ( R(x, .)) = Rd (x, .) for every x ∈ X . Let us check that Ld is bounded. For any

f H( R) ≤ 1, Ld • f H( R) d j=1 | f , φj | H(k) φj H( R) +        1 r -d r l=d+1 λl        r j=d+1 λ-1 j | f , φj | H(k) φj H( R) d j=1 f H(k) φj H( R) +        1 r -d r l=d+1 λl        r j=d+1 λ-1 j f H(k) φj H( R) d j=1 f H( R) φj H( R) +        1 r -d r l=d+1 λl        r j=d+1 λ-1 j f H( R) φj H( R) d j=1 φj H( R) +        1 r -d r l=d+1 λl        r j=d+1 λ-1 j φj H( R) < +∞ ,
where we used the inequality

f H(k) f H( R) due to H( R) ⊆ H(k) (see Theorem 1.1 in [LB01]).
Furthermore Ld has a finite trace because of the finiteness of its rank.

Tr( Ld L) = j≥1 Ld L φj , φj H(k) = j,l≥1 L φj , φl H(k) Ld φj , φl H(k) , since Ld is symmetric that is Ld f , g H(k) = f , Ld g H(k) for all f , g ∈ H(k).

Note that

Ld φj , φl

H(k) =        0 if j l or j > d or l > d 1 if j = l d which leads to Tr( Ld L) = j,l≥1 L φj , φl H(k) = d j=1 λk < +∞ .

Gaussian models and RKHS: examples of application

This section presents how a Gaussian model in an RKHS can be used in practice for some learning tasks. Three existing methods that make use of such models are reviewed: one for multigroup classification through a mixture of Gaussian processes in the kernel space [START_REF] Bouveyron | Kernel discriminant analysis and clustering with parsimonious Gaussian process models[END_REF],a method for outlier detection [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] and Gaussian process regression [START_REF] Rasmussen | Gaussian Processes for Machine Learning[END_REF].

Multigroup classification with a mixture of Gaussian processes

Consider an i.i.d. sample (X 1 , Z 1 ), . . . , (X n , Z n ) ∈ X × {1, . . . , C}. The n observations X 1 , . . . , X n are assumed to come from M different classes, and for each 1 i n the variable Z i represents the class membership of X i -that is Z i = j if X i comes from the j-th class. Given a new point X in X , the goal is to assign correctly X to one of the C classes.

One way of doing this is to compute the maximum a posteriori (MAP) probabilities P(Z = j | X = x) for each 1 j C, where Z denotes the class label of X and x is the observed value of X.

The new point is assigned to the class that maximizes the MAP.

In a first time, let us assume that X = R d . The MAP is calculated through the Bayes rule as

P(Z = j | X = x) = π j f j (x) C l=1 π l f l (x) , (3.4.15)
where π j = P(Z = j) and f j denotes the density function of X conditionally to Z = j for every 1 j C and x is the observed value of X.

A method to get a tractable expression for the MAP is to model the dataset X 1 , . . . , X n as a mixture of Gaussians, that is each f j is modeled as a Gaussian density ). Namely, D j (x) is written as a decreasing function of π j f j (x) that is

f j (x) = (2π) -d/2 |Σ j | -1/2 exp - 1 2 (x -µ j ) T Σ -1 j (x -µ j ) , where µ 1 , . . . , µ C ∈ R d are the respective mean vectors of each class, Σ 1 , . . . , Σ C ∈ M d (R)
D j (x) = -2 log(π j f j (x)) -d log(2π) = (x -µ j ) T Σ -1 j (x -µ j ) + log(|Σ j |) -2 log(π j ) , (3.4.16)
so that the new point X is assigned to the class that minimizes D j (x). Let us write the eigenexpansion of each matrix Σ j as Σ j = r l l=1 λ j,l u j,l u T j,l where r j ∈ {1, . . . , d} is the rank of Σ j , λ j,1 λ j,2 . . . λ j,r j > 0 are the non-zero eigenvalues of Σ j and u j,1 , . . . , u j,r j ∈ R d the corresponding eigenvectors. Then (3.4.16) can be written alternatively as

D j (x) = r j l=1 1 λ j,l [(x -µ j ) T u j,l ] 2 + r j l=1 log(λ j,l ) -2 log(π j ) .
(3.4.17)

However this approach may suffer some shortcomings. The actual distribution of X conditionally to Z = j may be far from a Gaussian distribution which may lead to bad classification performances. Moreover, this method does not cover the case of non-vectorial data, that is when X R d .

[BFG15] proposed a variation of this classification method by using a kernel. Let k be a kernel on X with corresponding feature map φ : X → H(k). The idea of [START_REF] Bouveyron | Kernel discriminant analysis and clustering with parsimonious Gaussian process models[END_REF] is to model the embedded data Y i = φ(X i ), i = 1, . . . , n -instead of the initial data X 1 , . . . , X n -as a mixture of Gaussian processes. Namely, Y | Z = j is modeled as a Gaussian process with mean m j : X → R and covariance function R j : X × X → R. Following Section 3.2, we assume that m j ∈ H(k) and R j << k so that these Gaussian processes are well-defined in H(k). The dominance operator of

H(k) over H(R j ) is denoted Σ j .
Remember that as remarked in Section 3.2, Σ j coincides with the covariance operator of the random element defined from Y conditionally to Z = j.

This kernelization allows to tackle the two issues mentioned above: there is no requirement about the nature of the set X where the initial dataset lies, and the kernel k can be chosen from a family of kernels that is rich enough so that the Gaussian model in the RKHS is reliable.

On the other hand, the transposition to the RKHS version is not straightforward. Indeed, the score functions D j (x) defined in the R d case were based on Gaussian densities, which was defined with respect to the Lebesgue measure on R d . In the case of an infinite-dimensional RKHS, there exists no canonical reference measure such as the Lebesgue measure on R d from which a Gaussian density could be defined. For this reason, we have to assume that H(k) is of finite dimension d even if it means that d is very large. The score function in (3.4.17) therefore becomes

D j (x) = d l=1 1 λ j,l φ(x) -m j , ϕ j,l 2 H(k) + d l=1 log(λ j,l ) -2 log(π j ) , (3.4.18)
where the pairs (λ j,1 , ϕ j,1 ), . . . , (λ j,d , ϕ j,d ) ∈ R + × H(k) are the eigenvalue/eigenvectors of the covariance operator Σ j , that is

Σ j • ϕ j,l = λ j,l ϕ j,l ,
for every 1 j C and 1 l d.

Since d is potentially very large, a parsimony assumption is made on the eigenvalues of Σ 1 , . . . , Σ C . Namely, we assume the existence of so-called intrinsic dimensions d 1 , . . . , d C ∈ {1, . . . , d} such that for every 1 j C and l > d j , λ j,l = λ with λ > 0. This way (3.4.18) becomes

D j (x) = d j l=1 1 λ j,l - 1 λ φ(x) -m j , ϕ j,l 2
H(k) + 1 λ φ(x) -m j 2 H(k) + d j l=1 log(λ j,l ) + (d -d j ) log(λ) -2 log(π j ) . (3.4.19)
Note that if d = +∞, this parsimony assumption entails that Tr(Σ j ) = +∞ for every 1 j C which violates one of the conditions for the relationships R j << k to hold.

In practice, (3.4.19) is expressed in a different form that does not involve φ or any direct calculations in H(k) (the "kernel trick"). Beforehand, we need to estimate some parameters on the basis of the sample (X 1 , Z 1 ), . . . , (X n , Z n ). The within class means m 1 , . . . , m C and covariance functions R 1 , . . . , R C are replaced by their empirical counterparts m1 , . . . , mC and R1 , . . . , RC as defined in Section 3.3. The empirical eigenvalues { λj,l } l and eigenfunctions { φj,l } l corresponding to Rj are estimated as in kernel PCA (see Section 2.3.2) from the eigendecomposition of the

matrix [k(X i , X i )] i,i ∈I j where I j = {i ∈ {1, . . . , n} | Z i = j}.
In particular the eigenfunctions admits an expansion of the form φj,l = i∈I j a

(i) j,l φ(X i ).
Besides, the class proportions π j are simply estimated by πj = n j /n where n j is the cardinal of I j . Plugging those empirical estimates into the score function (3.4.19) leads to the empirical score functions Dj (x)

Dj (x) = d j i=1       1 λj,l - 1 λ                i∈I j a (i) j,l k(x, x i ) - 1 n j i,i ∈I j k(x i , x i )          2 +          k(x, x) + 1 n 2 j i,i ∈I j k(x i , x i ) - 2 n j i∈I j k(x, x i )          2 + d j l=1 log( λj,l ) + (d -d j ) log( λ) -2 log( πj ) .

Batch outlier detection

The problem of outlier detection can be described as follows: given a sample of n observations X 1 , . . . , X n ∈ X , one assumes that there exists a subsample with indexes I out ⊂ {1, . . . , n} such that {X i } i∈I out -called the outliers -were generated by an underlying distribution Q different from the distribution P the {X i } i I out stem from. The goal is to identify those outliers. The difficulty lies in the fact that the number of outliers is typically very small compared to n. This issue is solved by assuming that outliers fall outside the support of P . Therefore, the problem amounts to define an acceptance region that contains non-outliers with high probability.

[Rot06] proposes to solve this problem by setting a Gaussian model in a feature space. Let us assume that the X 1 , . . . , X n take values in X = R d and consider a feature map φ :

R d → R p with related kernel k(x, x ) = φ(x) T φ(x ) for every x, x ∈ R d .
The proposed model assumes that φ(X) follows a N (µ, Σ) Gaussian distribution with mean vector µ ∈ R p and covariance matrix

Σ ∈ M p (R) when X ∼ P .
The useful quantity to detect outliers is the Mahalanobis distance D(φ(x), µ) between an embedded observation φ(x) ∈ R p and the mean vector µ

D(φ(x), µ) = (φ(x) -µ) T Σ -1 (φ(x) -µ) . (3.4.20)
When X is a non-outlier, the distance D(φ(X), µ) is expected to follow a χ 2 (p) chi-square distribution. To detect outliers, the empirical distribution of the D(φ(x 1 ), µ), . . . , D(φ(x n ), µ) is compared with the χ 2 (p) distribution by means of a quantile-quantile plot or QQ-plot. The QQ-plot consists in the pairs of points (F -1 (i/n), F-1 (i/n)) 1 i n where F -1 denotes the quantile function of the χ 2 (p) distribution and F-1 the empirical quantile function of D(φ(X), µ). In the case when no outlier is present, the QQ-plot should ideally be close to a straight line of equation y = x.

Thus a linear model is applied to the QQ-plot and a confidence interval around the fitted line is calculated. The observations declared as outliers are those that fall outside of the confidence interval.

Note that both µ and Σ are unknown and should be estimated. Moreover, the inversion of the covariance matrix Σ may be cumbersome when the number of dimensions p in the feature space becomes very large. However, [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] uses an alternative approach that bypasses this estimation phase by relating the Mahalanobis distance in (3.4.20) to a kernel ridge regression formulation of the outlier problem. Namely, they consider the regression problem

w * ∈ argmin w∈R p y -Φ T w 2 + δ w 2 with y = (1 . . . 1) T ∈ R n Φ = [φ(x 1 ) . . . φ(x n )] ∈ R p×n δ > 0 ,
where y represents the realizations of a latent variable Y that indicates whether X is an outlier or not. The solution of this regression problem is known to be w * = (K + δI) -1 y where K = Φ T Φ is the Gram matrix. Therefore the Mahalanobis distance (3.4.20) can be split into two parts by projecting φ(x) onto some vector w = αw * where α ∈ R is adequately chosen to get an expression of the form

D(φ(x), µ) = (φ(x) T w -m + ) 2 + D ⊥ where m + , D ⊥ ∈ R . (3.4.21)
The detailed expression of (3.4.21) writes for every 1 i n

D(φ(x i ), µ) = s -2 1 -s 2 y T (K + δI n ) -1 k i -s 2 2 -s -1 y T (K + δI n ) -1 k i 2 + nk T i (K + δI n ) -1 e i ,
where s 2 = n -1 y T K(K + δI) -1 y, k i is the i-th column of K and e i ∈ R n is the vector whose entries are all equal to 0 except for the i-th entry that is equal to 1. The advantage of this formulation is two-fold. Firstly, the parameters µ and Σ do not need to be estimated and in particular Σ does not need to be inverted. Secondly introducing the solution w * of the kernel ridge regression problem above leads to invert only the matrix (K+δI n ) instead of the Gram matrix K. Apart from avoiding computational instability, this also introduces a quantity that [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] calls effective degrees of freedom df = Tr(K(K + δI n ) -1 ). This quantity is useful when the number of dimensions p of the feature space is infinite so that D(φ(x), µ) is modeled as a χ 2 (df ) random variable instead of a χ 2 (p) variable.

As for theoretical guarantees, [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] focuses on the case of a Gaussian RBF kernel k(x, x ) = φ(x) T φ(x ) = exp(-γ xx 2 ) with parameter γ > 0 and justifies that in some sense this class of kernels is rich enough so that the Gaussian model in the feature space is relevant. Firstly in order to select an optimal γ, they consider a likelihood cross-validation procedure which necessitates a proper Gaussian density in the feature space. Since the feature space is infinite-dimensional for a Gaussian RBF kernel, a proper Gaussian density cannot be defined. However the problem is circumvented by setting a density p n (x) in the input space that "mimics" the contour lines of a Gaussian density in the feature space if the latter was finite dimensional, that is

p n (x) ∆ = C -1 exp - 1 2 D(φ(x), µ) with C = E X exp - 1 2 D(φ(X), µ) .
[Rot06] proves that when γ → +∞, p n (x) tends pointwise to a Parzen window density estimator

(1/n) n i=1 δ(x -x i )
where δ(•) denotes the Dirac function centered at 0. From this observation, [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] deduces that p n (x) is close to the actual distribution of X when γ, n → +∞ and therefore the Gaussian model in the feature space is accurate. However, this conclusion is questionable since the notion of density is the RKHS of k is not well-defined. Actually, it is proved later in this thesis (see Chapter 4) that for Gaussian RBF kernels, most of the marginals of an embedded variable φ(X) are close to a scale-mixture of isotropic Gaussians when γ → +∞ -that is a random variable of the form ηG where G is Gaussian and η is a positive variable independent of Gwhich contradicts the Gaussian model of [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF].

Gaussian process regression

A regression model can be cast in the following general form

y = f (x) + ,
where x is an X -valued variable called the input, y is a real valued variable called the output and is an error term. Given a sample (x 1 , y 1 ), . . . , (x n , y n ) ∈ X × R, the goal is to estimate the regression function f , or at least to predict the value f (x * ) for a test point x * ∈ X .

[RW06] have introduced Gaussian process regression as a method of regression by using Gaussian processes in a Bayesian manner.

To illustrate their method, we consider in a first time the standard linear model

X = R d , y = f (x) + with f (x) = x T w , (3.4.22)
for some w ∈ R d and where ∼ N (0, σ 2 ), σ 2 > 0. Instead of pursuing an estimator of w like other regression methods such as ridge regression or Lasso, the approach of [START_REF] Rasmussen | Gaussian Processes for Machine Learning[END_REF] consists in setting a prior distribution on w which is

w ∼ N (0, Σ) ,
for some covariance matrix Σ ∈ M d (R). This prior distribution induces a distribution on the stacked vector (y f (x * )) T where y = (y 1 . . . , y n ) T that is

                  y 1 . . . y n f (x * )                   ∼ N       0,       X T ΣX + σ 2 I n X T Σx * (x * ) T ΣX (x * ) T Σx *             , where X = [x 1 . . . x n ] ∈ M d,n (R).
Well-known formulas for the conditional distribution of covariates of Gaussian vectors yield the posterior distribution of f (x * )

f (x * ) | X, y, x * ∼ N (x * ) T ΣX(X T ΣX + σ I n ) -1 y, (x * ) T Σx * -(x * ) T ΣX(X T ΣX + σ I n ) -1 X T Σx * . (3.4.23)
An extension of (3.4.22) to a non-linear regression framework can be done by considering a (non-linear) feature map φ : X → R p and applying the linear regression method above to the embedded points φ(x 1 ), . . . , φ(x n ), which leads to the regression model

y = f (x) + with f (x) = φ(x) T w and w ∈ R p . (3.4.24)
Introducing the kernel k defined by

∀x, x ∈ X , k(x, x ) ∆ = φ(x) T Σφ(x ) ,
the kernel trick allows to express the posterior distribution of f (x * ) in this case as

f (x * ) | X, y, x * ∼ N k T x * (K + σ I n ) -1 y, k(x * , x * ) -k T x * (K + σ I n ) -1 k x * , (3.4.25)
where

k x * = (k(x 1 , x * ) . . . k(x n , x * )) T and K = [k(x i , x j )] 1 i,j n .
Note that in the procedure above works as well for more general feature maps φ : X → H with H R p , and in practice we can define directly the kernel k and bypass the explicit definition of Σ and φ.

Chapter 4

Asymptotic Distribution of Random

Projections in RKHS

This chapter presents theoretical results concerning embedded distributions in an RKHS induced by a Gaussian RBF kernel. The content of this chapter will be useful for the new outlier detection method introduced later in Chapter 5.

Introduction

Many statistical methods rely on distributional assumptions which often involve normality. This kind of Gaussian assumption can be found in classification [RW06; MM08; BFG15], dimension reduction [Bla+06; Die+10; DJS13] or novelty detection [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF]. However in most practical cases, this Gaussian assumption may be questionable. One possible way of dealing with this issue is to transform the dataset to make the normality assumption more reliable.

In the case of real-valued or multivariate data, a few approaches have been considered to get a nearly Gaussian distribution by using Box-Cox transformations [START_REF] Hernandez | The large-sample behavior of transformations to normality[END_REF] or more general power transformations [START_REF] Yeo | A new family of power transformations to improve normality or symmetry[END_REF]. To handle more general types of data, this transformation can be done through a positive definite kernel. Only a few papers have considered kernels to obtain Gaussian distributions. Among them, [START_REF] Bouveyron | Kernel discriminant analysis and clustering with parsimonious Gaussian process models[END_REF] proposes to perform supervised and unsupervised classification through a Gaussian mixture model in the kernel space. However, this work does not provide any theoretical guarantee about the normality of embedded data.

In [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF], novelty detection is conducted by setting a Gaussian density in the kernel space.

The validity of their method to detect outliers is justified by relating this method to density estimation of the distribution in the initial space. However, they do not assert straightforwardly that kernelized data are actually close to a Gaussian in general. All in all, there is a lack of understanding about the probabilistic behaviour of random variables embedded into an RKHS.

The purpose of the present chapter is to fill this gap and describe embedded distributions in a parametric class of RKHS. Namely, we study the asymptotic distribution of low-dimensional random projections in such RKHS when the "size" of the RKHS (which is controlled by the parameter of the kernel) increases.

Since RKHS -which are typically high-dimensional -are difficult to handle at first sight, let us resort to the case of R D when D is large. A series of several papers, ranging from [START_REF] Diaconis | Asymptotics of graphical projection pursuit[END_REF] to [START_REF] Dasgupta | A concentration theorem for projections[END_REF] have studied the distribution of low-dimensional projections of random vectors X in R D as D grows to infinity. In the seminal paper of [START_REF] Diaconis | Asymptotics of graphical projection pursuit[END_REF], unidimensional marginal probability distributions of X are proven to converge weakly to a N (0, σ 2 ) Gaussian distribution (for some σ 2 > 0) as D → +∞, with high probability over the direction of the projection (whose measure is given by the uniform distribution on the unit sphere S D-1 ). However, this result is based on the strong assumption that X behaves as an isotropic random vector, that is with a covariance matrix of the form ν 2 I where ν 2 > 0 and I denotes the identity matrix. They also provide

counter-examples where this isotropic assumption is not satisfied and where normality does not hold. A series of subsequent papers such as [START_REF] Weizsäcker | Sudakov's typical marginals, random linear functionals and a conditional central limit theorem[END_REF], [START_REF] Bobkov | On the central limit property of convex bodies[END_REF] or more recently [START_REF] Dasgupta | A concentration theorem for projections[END_REF] have generalized this result in broader frameworks. In particular, [START_REF] Dasgupta | A concentration theorem for projections[END_REF] considers the case where X is projected on a p-dimensional subspace with p 1 and for more general covariance structures of X. In this general setting, the distribution of the marginal is not necessarily Gaussian but an isotropic scale-mixture of Gaussians (SMG), that is a random variable sG where G ∼ N (0, I p ) and s is a real-valued random variable independent of G.

However, most of these findings hold in the finite-dimensional case and thus are not straightforwardly applicable to RKHS-valued variables since RKHS are often infinite-dimensional. The only exception is [START_REF] Weizsäcker | Sudakov's typical marginals, random linear functionals and a conditional central limit theorem[END_REF] which deals with random vectors in Hilbert spaces, but their result assumes that the direction of the projection is picked randomly from an isotropic Gaussian process, that is whose covariance operator is the identity operator. This makes their result unusable in the RKHS framework for the following reason. The covariance function of a Gaussian process must satisfy some conditions of so-called nuclear dominance [START_REF] Lukic | Stochastic Processes wih Sample Paths in Reproducing Kenel Hilbert Spaces[END_REF] for its trajectories to lie in an RKHS almost surely (see Section 3.2). These conditions require in particular that the covariance operator of the process must have a finite trace, which does not hold for an isotropic Gaussian process in an infinite-dimensional RKHS. Therefore [START_REF] Weizsäcker | Sudakov's typical marginals, random linear functionals and a conditional central limit theorem[END_REF] is not helpful in our case.

In the present work, we propose an extension of the results mentioned above from R D to RKHS. We restrict ourselves to the family of Gaussian RBF kernels k γ (x, y) = exp(-γd 2 (x, y))

with γ > 0 where d(•, •) is a metric on the input space X . The advantage of such kernels is that the covariance eigenstructure of an embedded distribution tends to be "flat" when γ is large, which is an ideal situation to get convergence to SMG according to results in R D . For reasons that will be explained later, the parameter γ controls the size of the RKHS in the same way D controls that of R D .

Our contribution is the following. First, we prove the weak convergence of random projections in the RKHS of k γ to a Gaussian scale-mixture as γ → +∞. Secondly, we examine the empirical framework where the distribution in the input space is represented by an i.i.d. sample X 1 , . . . , X n and show that convergence still holds as long as γ grows to infinity slowly enough compared to n. Finally, we show that when renormalizing properly k γ , most random projections in the RKHS converge weakly to a N (0, I p ) Gaussian distribution instead of a SMG. This may be an advantage in practice since it is no longer required to know the distribution of s in the SMG sG. The latter result holds when considering the empirical distribution and the true underlying distribution as well.

This chapter is outlined as follows. In Section 4.2, previous results on marginal distributions in the multivariate case are detailed and the rationale to extend these results to the RKHS case is given. In Section 4.3, two theorems describing the distribution of random projections in H(k γ )

-respectively in non-empirical and empirical cases -are stated. In Section 4.4, the results of Section 4.3 are extended to the case where k γ is adequately renormalized, which gives rise to a Gaussian limiting distribution instead of a SMG. In Section 4.5, we discuss how our results improve on existing results in R D and how the loss of information carried by random projections in H(k γ ) entails the existence of a "trade-off" value of γ in practice. Finally, the proofs of our main theorems are provided in Appendix 4.A.

From R D to reproducing kernel Hilbert spaces

In this section, we introduce existing results established in the multivariate case and present some properties of RBF kernels and their RKHS in order to link these two frameworks.

Concentration of random projections in R D

In the following, we state the main result of [START_REF] Dasgupta | A concentration theorem for projections[END_REF] since it is one of the most recent and general result of this kind to our knowledge.

Let X be some random variable with values in R D . X is projected onto a subspace generated by p 1 vectors of R D given by the rows of a random matrix Θ ∈ M p,D (R). To simplify, let us assume that the rows of Θ are independent Gaussian N (0, D -1 I D ) random variables, which are asymptotically orthonormal as D → +∞. In the following, the probability distribution of ΘX (conditionally on Θ) is denoted by f Θ .

Consider the probability distribution µ corresponding to the random variable X D / √ D (where . D denotes the Euclidean norm in R D ) and define the probability measure f which assigns to any Borelian subset

S ⊆ R D f (S) ∆ = ν σ (S)µ(dσ ) ,
where ν σ denotes the measure of an isotropic Gaussian N (0, σ 2 I p ). f corresponds to the distribution of a random variable σ G, where G ∼ N (0, I p ) and σ are independent with σ being a copy of X D / √ D. Such a distribution f is called a scale-mixture of (isotropic) Gaussians (SMG). Let d(Θ) = sup B |f Θ (B)f (B)| define the total variation distance between the probability measure f Θ and f where the supremum is taken over every ball B of R D . Theorem 4.2.1 proved by [START_REF] Dasgupta | A concentration theorem for projections[END_REF] gives an upper bound for d(Θ) which holds true with high probability over the projection matrix Θ.

Theorem 4.2.1 ([DHV06]). Assume X is centered and admits finite second moments. Let λ max (resp. λ avg ) denote the largest (resp. average) eigenvalue of the covariance matrix of X and let σ be the -quantile of the distribution of X D / √ D. Then for any ∈]0, 1/2[,

P Θ (d(Θ) > )       C 1 p 2 λ avg ln(1/ ) 3 σ 2       p exp - C 2 4 Dσ 2 pλ max ln(1/ ) , (4.2.1)
where P Θ is taken with respect to Θ and C 1 , C 2 are numerical constants.

The upper bound in (4.2.1) implies that the weak convergence of finite dimensional Gaussian random projections occurs when λ max /σ 2 = o(D) as D → +∞ (and for any fixed ∈]0, 1/2[). In [START_REF] Dasgupta | A concentration theorem for projections[END_REF], the term λ max /σ 2 is denoted ecc(X) and is presented as a measure of eccentricity of X. In a nutshell ecc(X) is small when the covariance eigenspectrum of X is flat and large otherwise. For instance when X is isotropic with independent entries (ie its covariance is of the form ν 2 I D with ν 2 > 0), λ max is equal to ν 2 and X 2 D /D converges almost surely to ν 2 as D → +∞ so that ecc(X) is close to 1 and the above convergence arises. On the contrary if the distribution of X is supported on a line (for example, X = Z.e 1 where e 1 = (1, 0, . . . , 0) ∈ R D and Z is some real-valued r.v.) , ecc(X) is of order D and the bound of Theorem 4.2.1 does not guarantee the weak convergence toward a SMG.

Note that the result of [START_REF] Diaconis | Asymptotics of graphical projection pursuit[END_REF] stated in the case where X is isotropic and p = 1 is a special case of Theorem 4.2.1. [START_REF] Diaconis | Asymptotics of graphical projection pursuit[END_REF] asserts that most unidimensional marginals of X converges weakly as D → +∞ to a Gaussian distribution instead of a SMG. This holds true in the setting of Theorem 4.2.1 since X 2 D /D converges to a deterministic quantity almost surely under the assumptions of [START_REF] Diaconis | Asymptotics of graphical projection pursuit[END_REF].

Connections with RBF kernel spaces

Our purpose is to extend the result of Theorem 4.2.1 from R D to kernel spaces, which will be done in Section 4.3. Since kernel spaces are typically infinite-dimensional, Theorem 4.2.1 is not straightforwardly applicable to RKHS. However, Section 4.2.1 provides two useful ingredients governing the weak convergence of projections in the multivariate case:

1 -the asymptotic behaviour of projections has to be studied with respect to D, that is the parameter driving the "size" of the ambient space, 2 -the covariance eigenspectrum of the projected vector has to remain "flat", which corresponds to a small eccentricity ecc(X).

Let us now consider the class of RBF kernels k γ with γ > 0 defined as follows ∀x, y ∈ X , k γ (x, y) = exp -γd2 (x, y) , where X is assumed to be a metric space endowed with a metric d(., .)1 . For each γ > 0, H(k γ ) denotes the RKHS corresponding to k γ and •, • γ denotes the associated inner product.

In the following, we recall two features of this family of kernels (in the case where d is the usual Euclidean metric) that satisfies the two criteria mentioned above and thus hint at an extension of Theorem 4.2.1 to RBF kernel spaces.

First, [START_REF] Tanaka | Theoretical Analyses on a Class of Nested RKHS's[END_REF] states that the RKHSs of RBF kernels are nested, that is

∀γ 1 , γ 2 ∈ R * + , γ 1 < γ 2 =⇒ H(k γ 1 ) ⊆ H(k γ 2 ) .
In this sense, the parameter γ controls the size of the RKHS H(k γ ). Thus increasing γ -that is enlarging H(k γ ) -may have the same effect as increasing D in Theorem 4.2.1.

Secondly, we show that the second criterion is met. Let λ 1 λ 2 . . . denote the eigenvalues of the covariance operator 2 Ek γ (X, .) ⊗ k γ (X, .). When X is a univariate N (0, ν 2 ) variable (ν 2 > 0), it is known (see [START_REF] Rasmussen | Gaussian Processes for Machine Learning[END_REF], p. 97) that for any k 1

λ k = γ k / √ 2ν 2 (4ν 2 ) -1 + γ + (16ν 4 ) -1 + γ(2ν 2 ) -1 k+1/2 ∼ γ→+∞ γ -1/2 √ 2ν 2 .
Note that the asymptotic expression of λ k as γ → +∞ does not depend on k. Therefore the covariance eigenspectrum of k γ (X, .) becomes flat as γ grows to infinity, hence a minimum eccentricity.

This fact can be extended for more general distributions of X as shows Proposition 4.2.2 that is proved in Appendix 4.B.1.

Proposition 4.2.2. Let (λ r ) r be the eigenvalues of the covariance operator

Σ γ = E X k γ (X, .) ⊗2 . Let supp(X) = {x ∈ X ∀ > 0, P(d 2 (x, X) < ) > 0}.
Assume that the distribution of X admits no point mass and that there exists a function A : supp(X) → R * + and s > 0 such that ∀x ∈ supp(X) , P d 2 (x, X) t ∼ A(x)t s , when t → 0 .

Then for any integer r,

λ r ∼ E X [A(X)]Γ (s + 1) γ s , when γ → +∞ ,
where Γ (s) = +∞ 0 x s-1 e -x dx denotes the Gamma function.

Therefore under the milder assumptions on X required by Proposition 4.2.2, (λ r ) r 1 admits a same asymptotic expression hence a "flat" eigenspectrum.

4.3

Σ γ = E X∼P k γ (X, .
) ⊗2 for some probability measure P . Then these p variables are asymptotically orthonormal when γ → +∞, that is

∀1 i p, h i 2 γ -→ γ→+∞ 1 a.s. ,
and

∀1 i, j p, i j, h i , h j γ -→ γ→+∞ 0 a.s. . The "projection" of k X onto V is denoted p V (X) =              k X , h 1 γ . . . k X , h p γ              ∈ R p ,
3 According to Section 3.2, those Gaussian processes are well defined in

H(k γ ) if H(Σ γ ) ⊆ H(k γ )
and Tr(Σ γ ) < +∞. The latter condition is easily checked as follows:

Tr(Σ γ ) = Tr(E X k X ⊗ k X ) = E X Tr(k X ⊗ k X ) = E X k X 2 = Ek γ (X, X) = 1 < +∞.
As for the former condition, it suffices to apply the last assertion of Definition 3.2.2 to the operator Σ γ .

We also consider the empirical case when P is not known and an i.i.d. sample X 1 , . . . , X n ∼ P is available. In this case, the covariance operator Σ γ is not known thus we resort to the empirical covariance Σ γ,n = (1/n) n i=1 k γ (X i , .) ⊗2 . Therefore we generate instead p independent (conditionally to the sample) Gaussian processes h 1,n , . . . , h p,n with mean zero and covariance Σ γ,n . In practice this is done by setting

h j,n = (1/n) n i=1 u (j) i k γ (X i , .
) for each 1 j p, where (u (j) i ) 1 i n,1 j p are i.i.d. N (0, 1). The linear subspace spanned by h 1,n , . . . , h p,n is denoted V n and the projection of k X onto V n (where X is independent of the sample X 1 , . . . , X n ) is denoted

p V n (X) =              k X , h 1,n γ . . . k X , h p,n γ              ∈ R p .
Let Tr(Σ 2 γ ) = E X Σ γ (X, X). Our main results in next sections focus on the discrepancy between the distributions of [

Tr(Σ 2 γ )] -1/2 p V (X) (resp. [Tr(Σ 2 γ )] -1/2 p V n (X)
) and that of some scale-mixture of Gaussians sG where G ∼ N (0, I p ) and s is some random vector independent of G (The normalization of p V (X) and p V n (X) is meant to prevent the variance from decreasing to 0).

Let ω be some random variable on R p and φ W (ω) = E W exp(i W , ω p ) denote the characteristic function of W for any r.v. W4 . If the distribution of ω is fully supported on R p , then we can consider the following L 2 -distance between distributions on R p [Sri+10]: for any pair of

p-variate random variables W , W , ∆(W , W ) ∆ = E ω φ W (ω) -φ W (ω) 2 1/2 .

Main assumptions

The results stated in Section 4.3.3 and Section 4.3.4 assume that the following assumptions hold: (A1) X is a separable metric space endowed with a metric d(., .), (A2) There exists a continuous, bounded function A : X → R + and s > 0 such that ∀x ∈ supp(P ) , P d 2 (x, X) t ∼ A(x)t s , when t → 0 , where supp(P ) = {x ∈ X ∀ > 0, P(d 2 (x, X) < ) > 0} defines the support of the distribution of X, and

A(x) = 0 otherwise, (A3) supp(P ) is a compact subset of X , (A4) (t, x) → P X (d 2 (x, X) t) is continuous.
The separability condition required by Assumption (A1) is made so that Σ γ admits a discrete eigenexpansion, that is there exists a sequence of eigenvalues (λ r ) r 1 ∈ (R + ) N * and eigenvectors (ψ r ) r 1 ∈ (H(k γ )) N * such that Σ γ = r 1 λ r ψ r ⊗ ψ r and (ψ r ) r 1 form an orthonormal family of vectors of H(k γ ). It is known (see e.g. Lemma 4.33 in [START_REF] Steinwart | Support Vector Machines[END_REF]) that the RKHS H(k γ ) is separable if k γ is continuous (straightforwardly this is true) and if the input space X is a separable topological space. Therefore under Assumption (A1) H(k γ ) is separable and since the operator

Σ γ : H(k γ ) → H(k γ ) is compact
P( x -X 2 t) µ B(x, √ t) = B(x, √ t) f (y)µ(dy) µ B(x, √ t) -→ t→0 f (x) , hence P( x -X 2 t) ∼ t→0 µ B(x, √ t) f (x) = π D/2 t D/2 f (x) Γ (D/2 + 1) ,
which correspond to Assumption (A2) with A(x) = π D/2 f (x)/Γ (D/2 + 1) and s = D/2. This holds for x ∈ S ⊆ supp(f ) for some S satisfying µ(supp(f ) \ S) = 0. Thus the continuity of f on its support implies that the result also holds for every x ∈ supp(f ).

Note that this extends to more general metrics in R D . For instance, consider Mahalanobis distances d(x, y) = (xy) S -1 (xy), x, y ∈ R D where S ∈ M D (R) is typically a covariance matrix. Applying Proposition 4.3.2 to the vector X = S -1/2 X straightforwardly entails that Assumption (A2) holds true for X. More generally one may consider metrics of the form

d(x, y) = ϕ(x) -ϕ(y) , (4.3.2)
for some ϕ : X → R q and q ∈ N * . Applying Proposition 4.3.2 to ϕ(X) implies that Assumption (A2) holds with A(x) = π q/2 f ϕ(X) (x)/Γ (q/2 + 1) and s = q/2, where f ϕ(X) denotes the density of

φ(X). Conversely, if d 2 (•, •) is of negative type 5 , d(•,
•) admits the form (4.3.2) where ϕ : X → H is an injective map and H is some Hilbert space (Proposition 3 in [START_REF] Sejdinovic | Equivalence of Distance-Based and RKHS-based Statistics in Hypothesis Testing[END_REF]), which makes Assumption (A2) true if H is of finite dimension.

On the other hand, Assumption (A2) may be false especially when X is infinite dimensional.

For example, consider X = C([0, 1], R) the space of continuous functions g : [0, 1] → R endowed with the supremum norm |g| ∞ = sup t∈[0,1] |g(t)|. Assume X = (X(t)) 0 t 1 is a fractional Brownian motion, that is X(0) = 0 a.s. and for any s, t ∈ [0, 1], E|X(s) -X(t)| 2 = |s -t| α for some α ∈ (0, 2).

According to Theorem 4.6 from [START_REF] Wenbo | Gaussian processes: inequalities, small ball probabilities and applications[END_REF], there exists 0 < K 1 K 2 < +∞ only depending on α such that for any ∈ (0, 1]

exp(-K 2 -2/α ) P (|X| ∞ ) exp(-K 1 -2/α )

Asymptotic distribution when P is known

We are now in a position to provide our main Theorem 4.3.3, which describes the asymptotic distribution of [Tr(Σ 2 γ )] -1/2 p V (X) as γ → +∞ when P is known. The proof of Theorem 4.3.3 is provided in Appendix 4.A.1.

Theorem 4.3.3. Let us assume (A1), (A2), (A3), (A4) and E ω ω < +∞ where • stands for the Euclidean norm of R p , and define a real-valued random variable s 2 by

s 2 law = Σ γ (X, X) Tr(Σ 2 γ )
• Then for every δ ∈ (0, 1), there exists an event with probability larger than 1δ on which

∆ 2           p V (X) Tr(Σ 2 γ ) , sG           8E ω [ ω ] pE X [A(X)]Γ (s + 1) log 2(p + 1) δ γ 2 -s/2 [1 + o γ (1)] , (4.3.4)
where G ∼ N (0, I p ) and where A : X → R + and s ∈ R * + are defined as in Assumption (A2) and Γ (.) denotes the Gamma function Γ (u) = +∞ 0 x u-1 e -x dx, ∀u 0.

In the special case X = R D with d(., .) as the Euclidean distance, Proposition 4.3.2 provides s = D/2 and A(x) = π D/2 f (x)/Γ (D/2 + 1), hence the upper bound in (4.3.4) becomes

∆ 2           p V (X) Tr(Σ 2 γ ) , sG           8E ω [ ω ] p log 2(p + 1) δ |f | 2 2π γ D/4 [1 + o γ (1)] , (4.3.5)
with high probability over h 1 , . . . , h p , where f denotes the density function of the distribution P of X and |f | 2 = ( f 2 ) 1/2 denotes the 2 -norm.

The upper bound in (4.3.5) hints at the influence of some key quantities on the distributional approximation of p V (X).

A large value of |f | 2 yields a loose bound which suggests that in this case weak convergence does not occur. This observation may be justified by the following rationale: a large |f | 2 corresponds intuitively to a distribution P concentrated around a few "peaks". Consider the limit case where |f | 2 tends to infinity and P gets close to a discrete distribution or at least admits a point mass. If P admits a point mass x 0 ∈ X such that P(X = x 0 ) = p > 0, then P(p V (X) = p V (x 0 )) p > 0 and the limit distribution of p V (X) would also admit a point mass which is not compatible with Therefore the covariance eigenvalues of k γ (X, .) of order larger than m vanish to 0 and weak convergence to a SMG may not happen.

Besides since the upper bound displays a term of order γ -D/4 , it is tempting to conclude that a high-dimensional input space fosters the weak convergence of p V (X). However, one has to be careful: when D varies, the input space -and consequently P and |f | 2 -do also change.

Thus a decreasing value of γ -D/4 might be canceled out by a high value of |f | 2 . To see this, let us consider a simple example. Assume X follows a multivariate Gaussian N (0, S) distribution in the input space, where the covariance matrix S = diag(l A special case of Theorem 4.3.3 is when P is a uniform measure on some subset S ⊂ R D .

(D) 1 , . . . , l (D) D ) is a diagonal matrix with l (D) 1 l (D) 2 . . . l (D) D > 0. The density function f of P is defined as f (x) = (2π) -D/2 det(S) -1/2 exp(-(1/2)x T S -1 x) for every x ∈ R D . Then straightforward algebra yields |f | 2 = (4π) -D/4 det -1/4 (S) (4πl
In this setting, s converges almost surely to 1 as γ grows to infinity hence the asymptotic distribution of [Tr(Σ 2 γ )] -1/2 p V (X) is Gaussian with mean 0 and covariance matrix I p . This is stated by Corollary 4.3.4 thereafter.

Corollary 4.3.4. Let us use the same notation and assumptions as in Theorem 4.3.3. Assume X = R D and P is the uniform measure on some measurable subset S ⊂ R D with finite Lebesgue measure µ(S) > 0. Then there exists an event with probability larger than 1δ for any δ ∈ (0, 1) on which [Tr(Σ 2 γ )] -1/2 p V (X) converges weakly to N (0, I p ) as γ → +∞.

Proof. It suffices to show that s 2 -→ γ→+∞ 1 almost surely. Lemma A.2.1 combined with Proposi- tion 4.3.2 yield Σ γ (X, X) = A(X)Γ (s + 1) (2γ) s [1 + o γ (1)] = f (X) π 2γ D/2 [1 + o γ (1)] = µ -1 (S) π 2γ D/2 [1 + o γ (1)] ,
and Lemma A.2.2 provides

Tr(Σ 2 γ ) = EA(X)Γ (s + 1) (2γ) s [1 + o γ (1)] = µ(S) f 2 (x)dx π 2γ D/2 [1 + o γ (1)] = µ -1 (S) π 2γ D/2 [1 + o γ (1)] ,
which proves that s 2 = 1 + o γ (1) as γ → +∞. Theorem 4.3.3 and Slutsky's lemma allow to conclude.

Asymptotic distribution when P is unknown

In this section, we consider the empirical case where P is unknown and only an i.i.d. sample X 1 , . . . , X n is available. The question is whether the convergence in distribution to the scalemixture of Gaussian as stated in Theorem 4.3.3 still holds when γ and n both grow to infinity.

Theorem 4.3.5 states that this is the case under the additional condition γ = o(n 1/s ). The proof of Theorem 4.3.5 is provided in Appendix 4.A.2.

Theorem 4.3.5. Let us assume (A1), (A2), (A3), (A4) and E ω ω < +∞, and let s 2 be defined as in Theorem 4.3.3. Then for every δ ∈ (0, 1), there exists an event with probability larger than 1 -4δ on which

∆ 2       p V n (X) Tr(Σ 2 ) , sG       C 1 γ -s/2 + C 2 γ s/2 n -1/2 ,
where G ∼ N (0, I p ) and

C 1 = 2 (s+6)/2 E ω p log 2(p + 1) δ EA(X)Γ (s + 1){1 + o γ (1)} C 2 = 1 EA(X)Γ (s + 1)       2E[ ω 2 ] log(1/δ)[1 + γ,n ] 1 -δ/2 +4E[ ω ] p log 2[p + 1] δ 2 s/2+1 + 2 s/2 • 3 • log(3/δ)         . with γ,n → 0 as γ → +∞ and γ s = o(n).
The fact that γ must be small compared to n for convergence to hold can be understood as follows. Since h 1 , . . . , h p are Gaussian processes with covariance Σ γ,n , they take values almost surely in the n-dimensional subspace S n of H(k γ ) generated by k γ (X 1 , .), . . . , k γ (X n , .) and V n is included in S n as a consequence. If n is fixed and γ grows to infinity, each embedded point k γ (x, .)

with x ∈ X \ {X 1 , . . . , X n } satisfies by the reproducing property k γ (x, .), k γ (X i , .) γ = k γ (X i , x) = exp(-γd 2 (X i , x)) -→ γ→+∞ 0 for any i = 1, . . . , n.
Therefore in this case k γ (x, .) tends to be orthogonal to S n and thus to V n , and the distribution of p V n (X) is highly concentrated around the origin and is far from a scale-mixture of Gaussians. This explains why n must be large enough compared to γ to counter this effect.

Straightforwardly, Corollary 4.3.4 still holds in the empirical case and a uniform distribution in the input space yields a N (0, I p ) Gaussian asymptotic distribution for p V n (X).

Asymptotic distribution when the kernel is renormalized

In Section 4.3, random projections in the RKHS H(k γ ) have been proved to converge weakly to an SMG sG where s 2 is equal to Σ γ (X, X) in distribution. However, if the latter equality holds P -almost surely, one gets a convergence in law to a N (0, I p ) Gaussian by normalizing the embedded variable k X by Σ γ (X, X) -which amounts to consider the renormalized kernel kγ kγ (x, y)

∆ = k γ (x, y) Σ γ (x, x)Σ γ (y, y) , (4.4.6)
where

Σ γ (•, •) is replaced by Σ γ,n (•, •) in the empirical framework.
In this case, we get a simpler limiting distribution that is parameter-free, whereas for k γ the asymptotic distribution involves the distribution of Σ γ (X, X) that must be estimated in practice.

The present section is devoted to show that this Gaussian convergence holds when renormalizing k γ as in (4.4.6) -both in the non-empirical and in the empirical case. In the non-empirical case, we study the asymptotic distribution of projections pV (X)

pV (X) =              kX , h 1 γ . . . kX , h p γ              with kX = Σ γ (X, X) -1/2 k X ,
and in the empirical case we study the projections pV n (X)

pV n (X) =              k(n) X , h 1,n γ . . . k(n) X , h p,n γ              with k(n) X = Σ γ,n (X, X) -1/2 k X .
Most of the notation and assumptions from Section 4.3 are continued in this section.

Asymptotic distribution when P is known

Theorem 4.4.1 thereafter provides an upper bound for the discrepancy between the distribution of pV (X) as defined above and the N (0, I p ) Gaussian distribution. This bound decreases to 0 when γ → +∞ -assessing the claimed asymptotic distribution of pV (X). The proof of this theorem is detailed in Appendix 4.A.3.

Theorem 4.4.1. Let us assume (A1), (A2), (A3), (A4) where A : X → R + as defined in (A2) satisfies 0 < a A(x) A < +∞ for every x ∈ supp(P ). Also assume that E ω ω < +∞.

Then for every δ ∈ (0, 1), there exists an event with probability larger than 1δ on which

∆ 2 ( pV (X), G) 8E ω pAΓ (s + 1) log 2(p + 1) δ γ 2 -s/2 [1 + o γ (1)] , (4.4.7)
where G ∼ N (0, I p ), s ∈ R * + is the same quantity introduced in Assumption (A2) and Γ (.) denotes the Gamma function Γ (u) = +∞ 0 x u-1 e -x dx, u 0.

Note the additional assumption that the function A(•) is bounded from below by a positive quantity a. This assumption is due to the renormalization of the projection p V (X) by Σ 1/2 γ (X, X) that needs to be lower bounded. Namely we use the equivalent term Σ γ (X, X) ∼ γ→+∞ A(X)Γ (s + 1)(2γ) -s (provided by Lemma A.2.1), hence a natural lower bound equivalent to aΓ (s + 1)γ -s . Theorem 4.4.2. Let us assume (A1), (A2), (A3), (A4) where A : X → R + as defined in (A2) satisfies 0 < a A(x) A < +∞ for every x ∈ supp(P ). Also assume that E ω ω < +∞.

Asymptotic distribution when P is unknown

Then for every δ ∈ (0, 1), there exists an event with probability larger than 1δ on which

∆ 2 pV n (X), G E ω 2        A a Γ (s + 1) (2γ) s + A 2 Γ (s + 1) log(1/δ) 2 1+2s a 3 γ s n + 2 (1-s)/2 κA log(1/δ) a 2 n        min 1 δ , 1 + ξ γ,n [1+o γ (1)] ,
(4.4.8)

where s ∈ R * + is the same quantity introduced in Assumption (A2), κ is a numerical constant and ξ γ,n is a quantity that tends to 0 when γ, n → +∞ and γ = o(n 3/2s ).

Unlike the case where k γ is not renormalized, the remarkable novelty is that the upper bound in (4.4.8) suggests that weak convergence holds when γ, n → +∞ even without any constraint on the growth of γ with respect to n. Empirical investigation in Section 4.5.3 below confirms that this is actually the case. However this raises a practical problem since γ can be chosen as large as wanted and there is no apparent "counter-weight" that forbids too large values of γ and leads to the existence of a "trade-off" optimal value of the hyperparameter. Section 4.5.3 below suggests that a possible counter-weight could be the loss of information carried by pV n (X) when γ → +∞, which is quantified by the dependence between pV n (X) and X.

Discussion

Advantages compared to previous results

We proved most of finite-dimensional projections of an embedded variable k X in a Gaussian RBF kernel space are close to a scale-mixture of Gaussians when the kernel hyperparameter γ is large, or to a N (0, I p ) Gaussian if the distribution in the input space is a uniform on a subset of R D (for some D 1) or if the kernel is renormalized adequately. This matches existing results about projections of a random vector X in R D for large D.

The main advantage of our result over previous results in R D is the lack of strong assumptions on X. As noted in Section 4.2.2, the closeness of projections in R D to a scale-mixture of Gaussians is due to two factors: the dimension D of the ambient space and a relatively flat covariance eigenspectrum of X. However, D is usually an immutable parameter whereas in the kernel case γ can be freely chosen and adapted to the dataset. As for the latter factor, it is tempting to flatten the eigenspectrum of X, for instance by making multiplying X at left by the square root of its inverse covariance matrix to get an isotropic vector. However, inverting a covariance matrix is a difficult task in the high-dimensional framework. It is usually done with methods involving shrinkage [START_REF] Ledoit | A well-conditioned estimator for large-dimensional covariance matrices[END_REF], matrix tapering [C+10] or sparsity [START_REF] Levina | Partial estimation of covariance matrices[END_REF], and most of these methods counterbalance the large number of covariates of X with parsimony assumptions. Because of these underlying assumptions, it is difficult to foretell whether a vector X renormalized by such a parsimonious estimator still admits marginals close to a SMG. On the other hand, Theorem 4.3.3 does not require any strong assumption on the distribution of X, thus avoiding any necessary preprocessing step on X. Moreover, our results include more general cases where the input space X is different from R D .

To renormalize or not to renormalize the kernel?

Whether one considers k γ or its renormalized version kγ , one obtains either SMG as marginals in the former case or N (0, I p ) Gaussians in the latter case. Since the latter distribution is simpler than the former, it would be natural to conclude that kγ should always be used in place of k γ . However there are some cases where k γ is preferable to kγ . This holds for instance in the application to outlier detection presented later in Chapter 5. It turns out that when transforming data through k γ , typical observations and outliers are naturally separated which allows to set a boundary between them and detect outliers (Section 5.2.1). However when using kγ , the renormalizing factor Σ 1/2 γ (X, X) tends faster to 0 when X is an outlier and γ → +∞, so that the renormalization cancels out that separation property.

Loss of information

In practice, the distribution of X is represented by a sample of n observations X 1 , . . . , X n which are independent copies of X. Consider the Gram matrix K γ = [k γ (X i , X j )] 1 i,j n . When γ → +∞, K γ converges entrywise to the identity matrix I n , which means that the information carried by the dataset (X i ) 1 i n has been entirely lost. This intuition is confirmed by the experiments whose results are summarized in Figure 4.1. The upper-left panel of Figure 4.1 displays a sample of simulated points in the input space R 2 generated from a uniform distribution on a "donut"-shaped subset of R 2 . The upper-right plot shows an instance of a random projection of k X in H(k γ ) (with γ = 400, p = 2 and n = 5000) which looks like a Gaussian (not a scale-mixture since X is uniformly distributed) as expected. We examine several two-dimensional projections of k X for increasing values of γ ranging from 50 to 5000 (100 trials for each value of γ). At each trial, we assess the normality of the projection with the Henze-Zirkler multivariate normality (MVN) test [START_REF] Henze | A Class of Invariant and Consistent Tests for Multivariate Normality[END_REF] and we measure the dependence between X and Tr -1/2 (Σ 2 γ ) • p V n (X) through a Hilbert Schmidt Independence Criterion (HSIC) test [START_REF] Gretton | A Kernel Statistical Test of Independence[END_REF]

-1/2 (Σ 2 γ ) • p V n (X).
of k X tend to become independent as γ → +∞ and corroborates the expected loss of information.

We carry the same experiments in the case where k is renormalized as in Section 4.4, and the corresponding results are represented in Figure 4.2. As foretold by Theorem 4.4.2, the convergence to a Gaussian distribution seems to hold when γ → +∞ even for a finite n (lower-left plot in Figure 4.2). However X and pV n (X) tend to become independent when γ grows to infinity which shows that a loss of information occurs as in the case where k is not renormalized.

All in all this negative effect on data information can be seen as a counterweight that defines a range of optimal values of γ for which a trade-off between distributional approximation and information is reached.

4.A Proofs of main results

This section presents in details the proofs of Theorem 4.3.3, 4.3.5, 4.4.1 and 4.4.2. All of these four proofs follow a similar outline consisting in three steps. Using the generic notation L to denote the quantity of interest (the distance between the distribution of a random projection and the limit law), the first step aims at linking L with its expectancy taken over the directions h 1 , . . . , h p (or h 1,n , . . . , h p,n in the empirical case) through a McDiarmid-type concentration inequality. In the second step, a tight upper bound for the expectancy of L is derived. Finally the results of two first steps are combined in the last step and yield the final upper bound of L.

4.A.1 Proof of Theorem 4.3.3

First step: apply a concentration inequality

In the following, the quantity of interest

∆ 2        p V (X) Tr(Σ 2 γ )
, sG

       is rewritten as a function L(h) of h = (h 1 . . . h p ) L(h) = ∆ 2           p V (X) Tr(Σ 2 γ )
, sG

          = E ω E X e i[Tr(Σ 2 γ )] -1/2 ω T p V (X) -E sG e isω T G 2 .
The goal is to apply the McDiarmid inequality (Lemma A.1.1) to bound the probability P(L(h 1 , . . . , h p ) > t) for some t > 0. This requires the existence of some deterministic quantity c > 0 such that for any 1 j p and h j , h j ∈ H(k γ ),

L(h -j ) -L(h) c ,
where h -j = (h 1 , . . . , h j-1 , h j , h j+1 , . . . , h p ). We show that this condition is fulfilled conditionally to the event E M = ∩ p j=1 { h j M} for any M > 0. Assume E M holds for some M > 0. Using Jensen's inequality (combined with the convexity of the modulus function) and the triangle inequality

yielding |z 1 | 2 -|z 2 | 2 = |z 2 1 | -|z 2 2 | z 2 1 -z 2 2 for any z 1 , z 2 ∈ C, L(h -j ) -L(h) E ω E X e i[Tr(Σ 2 γ )] -1/2 p k j ω k h k ,k X +ω j h j ,k X -E sG e isω T G 2 -E X e i[Tr(Σ 2 γ )] -1/2 p k j ω k h k ,k X +ω j h j ,k X -E sG e isω T G 2 E ω          E X e i p k j ω k h k ,k X [Tr(Σ 2 γ )] 1/2          e i ω j h j ,k X [Tr(Σ 2 γ )] 1/2 -e i ω j h j ,k X [Tr(Σ 2 γ )] 1/2          • E X e i p k j ω k h k ,k X [Tr(Σ 2 γ )] 1/2          e i ω j h j ,k X [Tr(Σ 2 γ )] 1/2 + e i ω j h j ,k X [Tr(Σ 2 γ )] 1/2          + 2E sG e isω T G          4E ω E X          e i ω j h j ,k X [Tr(Σ 2 γ )] 1/2 -e i ω j h j ,k X [Tr(Σ 2 γ )] 1/2          4E ω |ω j | E X | h j -h j , k X | [Tr(Σ 2 γ )] 1/2 4E ω ω Σ γ (h j -h j ), h j -h j Tr(Σ 2 γ ) 4E ω ω       λ 1 Tr(Σ 2 γ )       1/2 h j -h j 8E ω ω       λ 1 Tr(Σ 2 γ )       1/2 M := M(γ) ,
where we used the fact that t → exp(it) is 1-Lipschitz. Here λ 1 = sup u =1 Σ γ , u, u denotes the largest eigenvalue of Σ γ .

We

P (L(h) > t) P L(h) -E E M L(h) > t -E E M L(h) | E M + P(E c M ) exp - 2(t -E E M L(h)) 2 pM 2 (γ) + P(E c M ) .
Using the exponential Markov inequality and given some 0 < β < (2λ 1 ) -1 , P(E c M ) is bounded as follows

P(E c M ) pP ( h 1 > M) pe -βM 2 Ee β h 1 2 = pe -βM 2 j 1 (1 -2βλ j ) -1/2 = p exp         -βM 2 + (1/2) j 1 log 1 1 -2βλ j         p exp         -βM 2 + (1/2) j 1 2βλ j 1 -2βλ j         p exp -βM 2 + β(1 -2βλ 1 ) -1 ,
where we used the inequality log(x) x -1 and the equality Tr

(Σ γ ) = Tr(E X k ⊗2 X ) = E X Tr(k ⊗2 X ) = E X k γ (X, X) = 1. Assuming M < 1 -1/ √ 2, one sets β = (1 -2M)[ 1 + (2M/(1 -2M 2 )) 2 -1]/(2λ 1 ) entails P(E M ) p exp - M 2 2λ 1 , hence P(L(h) > t) exp - 2(t -E E M L(h)) 2 pM 2 (γ) + p exp - M 2 2λ 1 .
In other words, with probability larger than 1δ (for any M such that δ > pe -M 2 /(2λ 1 ) )

L(h) E h [L(h) | E M ] + M(γ) p 2 log 1 δ -pe -M 2 /(2λ 1 )
1pe

-M 2 2λ 1 -1 E h [L(h)] + 8E ω       λ 1 Tr(Σ 2 γ )       1/2 M p 2 log 1 δ -pe -M 2 /(2λ 1 )
. (4.A.9)

An upper bound for E h [L(h)] is given thereafter.

Second step: find an upper bound of E h L(h)

In the following, G and G denote two independent N (0, I p ) variables, and s, s are independent copies of Σ γ (X, X)/Tr(Σ 2 γ ). Then E h [L(h)] can be written

E h L(h) = E ω,h φ [Tr(Σ 2 γ )] -1/2 p V (X) (ω) -φ sG (ω) 2 = E ω,h E X,X e i[Tr(Σ 2 γ )] -1/2 ω T (p V (X)-p V (X )) +E G,G ,s,s e iω T (sG-s G ) -2E X,G,s e iω T ([Tr(Σ 2 γ )] -1/2 p V (X)-sG) . (4.A.10) Let ψ(t) = E ω e -t ω 2 /2 defined on R + and write s X = (Tr(Σ 2 γ )) -1 Σ γ (X, X) (likewise, s X = (Tr(Σ 2 γ )) -1 Σ γ (X , X )) and δ XX = Σ γ (X, X ). On the first hand, E ω,h,X,X e i[Tr(Σ 2 γ )] -1/2 ω T (p V (X)-p V (X )) = E ω,h,X,X e i[Tr(Σ 2 γ )] -1/2 p j=1 ω j k X -k X ,h j γ = E ω,X,X e -1 2 [Tr(Σ 2 γ )] -1 ω 2 Σ γ (k X -k X ),k X -k X γ = E X,X ψ [s X + s X -2[Tr(Σ 2 γ )] -1 δ XX ] .
On the other hand,

E ω,G,G ,s,s e iω T (sG-s G ) = E ω,s,s e -(s 2 +[s ] 2 ) ω 2 /2 = E s,s ψ(s 2 + [s ] 2 ) = E X,X ψ(s 2 X + s 2 X ) ,
and finally, using p V (X) ∼ N (0, Σ γ (X, X)I p ) with respect to h 1 , . . . , h p and conditionally to X,

E h,X,G,s,ω e iω T ([Tr(Σ 2 γ )] -1/2 p V (X)-sG) = E h,X,s,ω e i[Tr(Σ 2 γ )] -1/2 ω T p V (X)-s 2 ω 2 /2 = E X,s,ω e -(s 2 X +s 2 ) ω 2 /2 = E X,X ψ(s 2 X + s 2 X ) .
Plugging these three equalities into (4.A.10),

E h L(h) = E X,X ψ s 2 X + s 2 X -2[Tr(Σ 2 γ )] -1 δ XX -ψ(s 2 X + s 2 X ) 2 ψ ∞ E X,X       δ XX Tr(Σ 2 γ )       , (4.A.11)
where

ψ ∞ = sup{|ψ (t)| : t ∈ R + }. Note that ψ ∞ is finite since |ψ (t)| = E ω -(1/2) ω 2 e -t ω 2 /2
(1/2)E ω 2 < +∞. Here we are allowed to switch the mean operator and the derivation because of the Dominated Convergence Theorem and because t → exp(-t ω 2 /2) is 1-Lipschitz on R + .

Lemma A.2.3 states that

Eδ XX = E X A 2 (X)Γ 2 (s + 1) γ 2s [1 + o γ (1)] ,
and according to Lemma A.2.2,

Tr(Σ 2 γ ) = E X A(X)Γ (s + 1) 2 s γ s [1 + o γ (1)] ,
so that

E h L(h) 2 s E ω 2 EA 2 (X)Γ (s + 1) EA(X)γ s [1 + o γ (1)] . (4.A.12)

Conclusion of the proof

Continuing from (4.A.9), we proved that with probability larger than 1-δ for any δ > pe -M 2 /(2λ 1 ) ,

L(h) 1 -pe -M 2 2λ 1 -1 EL(h) + 8E ω       λ 1 Tr(Σ 2 γ )       1/2 M p 2 log 1 δ -pe -M 2 /(2λ 1 ) .
Besides, (4.A.12) provided the following upper bound for E h L(h)

E h L(h) 2 s E ω 2 EA 2 (X)Γ (s + 1) EA(X)γ s [1 + o γ (1)] . Lemma 4.B provides when γ → +∞ λ 1 = EA(X)Γ (s + 1) γ s [1 + o γ (1)] ,
and by Lemma A.2.2,

Tr(Σ 2 γ ) = EA(X)Γ (s + 1) (2γ) s [1 + o γ (1)] .
Hence if one sets pe -M 2 /(2λ 1 ) = δ/2 that is M = 2λ 1 log(2p/δ) , then with probability larger than 1δ for any 0 < δ < 1 and γ large enough so that M = 2EA(X)Γ (s + 1) log(2p/δ)γ

-s [1 + o γ (1)] < 1 -1/ √ 2, L(h) = ∆ 2           p V (X) Tr(Σ 2 γ ) , sG                    E ω 2 EA 2 (X)Γ (s + 1) EA(X) γ 2 -s + 8E ω pEA(X)Γ (s + 1) log 2(p + 1) δ γ 2 -s/2          [1 + o γ (1)] = 8E ω pEA(X)Γ (s + 1) log 2(p + 1) δ γ 2 -s/2 [1 + o γ (1)] .
4.A.2 Proof of Theorem 4.3.5

First step: apply a concentration inequality Let X = (X 1 , . . . , X n ) denote an i.i.d. sample generated by P . In the following, we consider the function L(•; X) :

H(k γ ) p → R defined by ∆ 2           p V n (X) Tr(Σ 2 γ )
, sG

          = L(h; X) = E ω E X e i[Tr(Σ 2 γ )] -1/2 ω T p V (X) -E sG e isω T G 2 , where p V n (X) = h 1,n , k X γ . . . h p,n , k X γ T with h = (h 1,n , . . . , h p,n ) following independently a
Gaussian process of mean zero and of covariance

Σ γ,n = (1/n) n i=1 k ⊗2 γ (X i , .
). The goal is to apply the McDiarmid inequality (Lemma A.1.1) to bound the probability P |X (L(h; X) > t) for some t > 0 and conditionally to X. This requires the existence of some deterministic quantity c > 0 such that for any 1 j p and h j,n , h j,n ∈ H(k γ ),

L(h -j ; X) -L(h; X) c ,
where h -j = (h 1,n , . . . , h j-1,n , h j,n , h j+1,n , . . . , h p,n ). We show that this condition is fulfilled conditionally to the event

E M = ∩ p j=1 { h j,n
M} for any M > 0. Assume E M holds for some M > 0. Using Jensen's inequality (combined with the convexity of the modulus function) and the triangle

inequality yielding |z 1 | 2 -|z 2 | 2 = |z 2 1 | -|z 2 2 | z 2 1 -z 2 2 for any z 1 , z 2 ∈ C, L(h -j ; X) -L(h; X) E ω E X e i[Tr(Σ 2 γ )] -1/2 p k j ω k h k,n ,k X +ω j h j,n ,k X -E sG e isω T G 2 -E ω E X e i[Tr(Σ 2 γ )] -1/2 p k j ω k h k,n ,k X +ω j h j,n ,k X -E sG e isω T G 2 E ω          E X e i p k j ω k h k,n ,k X [Tr(Σ 2 γ )] 1/2          e i ω j h j,n ,k X [Tr(Σ 2 γ )] 1/2 -e i ω j h j,n ,k X [Tr(Σ 2 γ )] 1/2          • E X e i p k j ω k h k,n ,k X [Tr(Σ 2 γ )] 1/2          e i ω j h j,n ,k X [Tr(Σ 2 γ )] 1/2 + e i ω j h j,n ,k X [Tr(Σ 2 γ )] 1/2          + 2E sG e isω T G          4E ω E X          e i ω j h j,n ,k X [Tr(Σ 2 γ )] 1/2 -e i ω j h j,n ,k X [Tr(Σ 2 γ )] 1/2          4E ω |ω j | E X | h j,n -h j,n , k X | [Tr(Σ 2 γ )] 1/2 4E ω ω Σ γ (h j,n -h j,n ), h j,n -h j,n Tr(Σ 2 γ ) 4E ω ω       λ 1 Tr(Σ 2 γ )       1/2 h j,n -h j,n 8E ω ω       λ 1 Tr(Σ 2 γ )       1/2 M := M(γ) ,
where we used the fact that t → exp(it) is 1-Lipschitz. Here λ 1 = sup u =1 Σ γ u, u denotes the largest eigenvalue of Σ γ .

Therefore, the McDiarmid inequality yields conditionally to E M and X for any

t > E h (L(h; X) | E M ) P h (L(h; X) > t) P h (L(h; X) -E h (L(h; X) | E M ) > t -E h (L(h; X) | E M ) | E M ) + P h (E c M ) exp - 2(t -E h [L(h; X) | E M ]) 2 pM 2 (γ) + P h (E c M ) .
Let λ1 λ2 . . . λn denote the eigenvalues of Σ γ,n . Using the exponential Markov inequality and given some 0 < β < (2 λ1 ) -1 , P h (E c M ) is bounded as follows

P h (E c M ) pP h h 1,n > M pe -βM 2 E h 1,n e β h 1,n 2 = pe -βM 2 j 1 (1 -2β λj ) -1/2 = p exp         -βM 2 + (1/2) j 1 log       1 1 -2β λj               p exp         -βM 2 + (1/2) j 1 2β λj 1 -2β λj         p exp -βM 2 + β(1 -2β λ1 ) -1 ,
where we used the inequality log(x) x -1 and the equality Tr(Σ γ,n ) = 1.

Assuming M < 1 -1/ √ 2, one sets β = (1 -2M)[ 1 + (2M/(1 -2M 2 )) 2 -1]/(2 λ1 ) entails P h (E M ) p exp - M 2 2 λ1 , hence P h (L(h; X) > t) exp - 2(t -E E M L) 2 pM 2 (γ) + p exp - M 2 2 λ1 .
that is with probability larger than 1δ (with δ > pe -M 2 /(2 λ1 ) ) over h and conditionally to X

L(h; X) E h|E M L(h; X) + M(γ) p 2 log       1 δ -pe -M 2 /(2 λ1 )       P -1 (E M )E h L(h; X) + M(γ) p 2 log       1 δ -pe -M 2 /(2 λ1 )       . (4.A.13)
The following section is devoted to finding an upper bound for E h L(h; X) that holds with large probability over the sample X.

Second step: find an upper bound for E h L(h; X)

E h L(h; X) can be expanded as follows

E h L(h; X) = E ω,h φ Tr(Σ 2 γ ) -1/2 p V (X) (ω) -φ sG (ω) 2 = E ω,h E X,X e i[Tr(Σ 2 γ )] -1/2 ω T (p V (X)-p V (X )) +E G,G ,s,s e iω T (sG-s G ) -2E X,G,s e iω T ([Tr(Σ 2 γ )] -1/2 p V (X)-sG) . (4.A.14) Let ψ(t) = E ω e -t ω 2 /2 defined on R + and write ŝX = Σ γ,n (X, X) Tr(Σ 2 γ ) s X = Σ γ (X, X) Tr(Σ 2 γ ) (likewise, ŝX = (Tr(Σ 2 γ )) -1 Σ γ,n (X , X ) and s X = (Tr(Σ 2 γ )) -1 Σ γ (X , X )) and δXX = Σ γ,n (X, X ) Tr(Σ 2 γ )
.

On the first hand,

E h,ω,X,X e i[Tr(Σ 2 γ )] -1/2 ω T (p V (X)-p V (X )) = E h,ω,X,X e i[Tr(Σ 2 γ )] -1/2 p j=1 ω j k X -k X ,h j,n γ = E ω,X,X e -1 2 [Tr(Σ 2 γ )] -1 ω 2 Σ γ,n (k X -k X ),k X -k X γ = E X,X ψ ŝX + ŝX -2 δXX .
On the other hand,

E G,G ,s,s ,ω e iω T (sG-s G ) = E s,s,ω e -(s+s ) ω 2 /2 = E s,s ψ(s + s ) = E X,X ψ(s X + s X ) ,
and finally, using p V n (X) ∼ N (0, Σ γ,n k X , k X γ I p ) with respect to h and conditionally to X, X 1 , . . . , X n ,

E h,X,G,s e iω T ([Tr(Σ 2 γ )] -1/2 p V (X)-sG) = E h,X,s e i[Tr(Σ 2 γ )] -1/2 ω T p V (X)-s ω 2 /2 = E X,s e -(ŝ X +s) ω 2 /2 = E X,X ψ(ŝ X + ŝX ) .
Plugging the three equalities into (4.A.14),

E h L(h; X) = E X,X [ψ(s X + s X -2 δXX ) -ψ(s X + s X )] + E X,X [ψ(ŝ X + ŝX -2 δXX ) -ψ(s X + s X -2 δXX )] + 2E XX [ψ(s X + s X ) -ψ(ŝ X + s X )] 2 ψ ∞ E X,X δXX + 4 ψ ∞ E X ŝX -s X 2 (4.A.15)
where

ψ ∞ = sup{|ψ (t)| : t ∈ R + }. Note that ψ ∞ is finite since |ψ (t)| = E ω -(1/2) ω 2 e -t ω 2 /2
(1/2)E ω 2 < +∞. Here we are allowed to switch the mean operator and the derivation because of the Dominated Convergence Theorem and because t → exp(-

t ω 2 /2) is 1-Lipschitz on R + . Since E X k γ (x, X) = E X k 2 γ/2 (x, X) = Σ γ/2 (x, x), it follows E X,X δXX = E X,X        1 n n i=1 k γ (X i , X)k γ (X i , X ) Tr(Σ 2 γ )        = 1 n n i=1 Σ 2 γ/2 (X i , X i ) Tr(Σ 2 γ )
. By Bennett's concentration inequality (Lemma A.1.2), with probability larger than 1δ over X

E X,X δXX E X        1 n n i=1 Σ 2 γ/2 (X i , X i ) Tr(Σ 2 γ )        + 2ν log(1/δ)n -1 + κc √ 2 log(1/δ)n -1 Tr(Σ 2 γ ) , (4.A.16)
where ν = Var(Σ 2 γ/2 (X, X)) and c = sup x∈X Σ 2 γ/2 (x, x).

First, Lemma A.2.2 provides

Tr(Σ 2 γ ) = EA(X)Γ (s + 1) 2 s γ s [1 + o γ (1)] ,
and combined with Lemma A.2.3 entails

E X        1 n n i=1 Σ 2 γ/2 (X i , X i ) Tr(Σ 2 γ )        = E X Σ 2 γ/2 (X, X) Tr(Σ 2 γ ) = E X {A(X)Γ (s + 1)γ -s } 2 EA(X)Γ (s + 1)(2γ) -s [1 + o γ (1)] = EA 2 (X) EA(X) Γ (s + 1) (γ/2) s [1 + o γ (1)] .
Lemma A.2.3 also leads to

ν = Var(Σ 2 γ/2 (X, X)) EΣ 4 γ/2 (X, X) = E X        A(X)Γ (s + 1) γ s 4 [1 + o γ (1)]        = EA 4 (X)Γ 4 (s + 1)γ -4s [1 + o γ (1)] ,
and by Lemma 4.B.1

c = sup x∈X Σ 2 γ/2 (x, x) λ 2 1 (Σ γ/2 ) sup x∈X k γ/2 (x, .) 4 γ/2 = λ 2 1 (Σ γ/2 ) = EA(X)Γ (s + 1) (γ/2) s 2 [1 + o γ (1)] ,
where λ 1 (Σ γ/2 ) denotes the largest eigenvalue of Σ γ/2 (the covariance operator).

Plugging all of the above into (4.A.16) provides with probability larger than 1δ over X

E X,X δXX                EA 2 (X)Γ (s + 1) EA(X)(γ/2) s + 2 log(1/δ)Γ 4 (s+1)EA 4 (X) nγ 4s + κ √ 2 log(1/δ)[EA(X)] 2 Γ 2 (s+1) n(γ/2) 2s EA(X)Γ (s + 1)(2γ) -s                [1 + o γ (1)] = EA 2 (X)Γ (s + 1) EA(X)(γ/2) s [1 + o γ,n (1)] .
Finally using Bennett's inequality again yields with probability larger than 1δ over X

E X (ŝ X -s X ) 2 = E X       Σ γ,n (X, X) -Σ γ (X, X) Tr(Σ 2 γ )       2 = E X        n i=1 k 2 γ (X i , X) -E X k 2 γ (X , X) nTr(Σ 2 γ )        2 E X       2 ν log(1/δ)/n + κ c√ 2 log(1/δ)/n Tr(Σ 2 γ )       2 ,
where κ is a numerical constant and

ν = Var X (k 2 γ (x, X )) E X k 4 γ (X , x) = Σ 2γ (X, X) = A(X)Γ (s + 1) (4γ) s [1 + o γ (1)] ,
and

c = sup x∈X k 2 γ (X, x) 1 , hence introducing a = inf{A(x) : x ∈ supp(P )} > 0 E X (ŝ X -s X ) 2 E X           2 log(1/δ)A(X)Γ (s+1) n(4γ) + κ √ 2 log(1/δ) n EA(X)Γ (s + 1)(2γ) -s           2 [1 + o γ (1)] E X 2 log(1/δ)A(X)Γ (s + 1) [EA(X)Γ (s + 1)] 2 γ s n       1 + κ log(1/δ)(2γ) s/2 naΓ (s + 1)       2 [1 + o γ (1)] = 2 log(1/δ)
EA(X)Γ (s + 1)

γ s n [1 + o γ,n (1)] ,
where the 'o γ,n (1)' tends to 0 when γ s = o(n).

Therefore gathering all of the above into (4.A.15) yields with probability larger than 1 -2δ over X

E h L(h; X) E ω 2          EA 2 (X)Γ (s + 1) EA(X)(γ/2) s + 2 2 log(1/δ)γ s EA(X)Γ (s + 1)n          [1 + γ,n ] , (4.A.17)
where γ,n → 0 as γ, n → +∞ and γ s = o(n).

Conclusion

Continuing from (4.A.13), we proved that with probability larger than 1δ for any δ > pe -M 2 /(2λ 1 ) ,

L(h; X) P -1 (E M )E h L + 8E ω       λ 1 Tr(Σ 2 γ )       1/2 M p 2 log       1 δ -pe -M 2 /(2 λ1 )       .
Besides, (4.A.17) provided the following upper bound that holds with probability larger than 1 -2δ over X

E h L(h; X) E ω 2          EA 2 (X)Γ (s + 1) EA(X)(γ/2) s + 2 2 log(1/δ)γ s EA(X)Γ (s + 1)n          [1 + γ,n ] ,
where γ,n → 0 as γ, n → +∞ and γ s = o(n).

Setting P(E c M ) = pe -M 2 /(2 λ1 ) = δ/2 that is M = 2 λ1 log(2p/δ) yields with probability larger than 1 -3δ over h and X

L(h; X) E ω 2 1 -δ/2          EA 2 (X)Γ (s + 1) EA(X)(γ/2) s + 2 2 log(1/δ)γ s EA(X)Γ (s + 1)n          [1 + γ,n ] + 8E ω       λ 1 Tr(Σ 2 γ )       1/2 p λ1 log 2[p + 1] δ .
Theorem 3.3.2 in [START_REF] Zwald | Performances d'Algorithmes Statistiques d'Apprentissage: "Kernel Projection Machine" et Analyse en Composantes Principales à Noyaux[END_REF] provides with probability larger than 1δ over X for any δ > 0 the inequality

| λ1 -λ 1 | 2 + 3 log(3/δ) √ n ,
hence with probability larger than 1 -4δ

L(h; X) E ω 2 1 -δ/2          EA 2 (X)Γ (s + 1) EA(X)(γ/2) s + 2 2 log(1/δ)γ s EA(X)Γ (s + 1)n          [1 + γ,n ] + 8E ω       λ 1 Tr(Σ 2 γ )       1/2 p log 2[p + 1] δ       λ 1 + 2 + 3 log(3/δ) √ n       . Lemma 4.B provides when γ → +∞ λ 1 = EA(X)Γ (s + 1) γ s [1 + o γ (1)] ,
and by Lemma A.2.2,

Tr(Σ 2 γ ) = EA(X)Γ (s + 1) (2γ) s [1 + o γ (1)] , therefore L(h; X) E ω 2 1 -δ/2          EA 2 (X)Γ (s + 1) EA(X)(γ/2) s + 2 2 log(1/δ)γ s EA(X)Γ (s + 1)n          [1 + γ,n ] + 8E ω p log 2[p + 1] δ       EA(X)Γ (s + 1) (γ/2) s {1 + o γ (1)} + 2 s+1 + 2 s .3 log(3/δ) √ n       ,
which reads in a more concise way

L(h; X) C 1 γ -s + C 2 γ s/2 n -1/2 + C 3 γ -s + C 4 n -1/2 .
To simplify this upper bound, we use the concavity of the square root function to obtain

C 3 γ -s + C 4 n -1/2 = C 1/2 3 γ -s/2 1 + (C 4 /C 3 )γ s n -1/2 C 1/2 3 γ -s/2 1 + (C 4 /2C 3 )γ s n -1/2 , hence L(h; X) C 1 γ -s + C 2 γ s/2 n -1/2 + C 1/2 3 γ -s/2 + C 4 2C 1/2 3 γ s/2 n -1/2 = C1 γ -s/2 + C2 γ s/2 n -1/2 .
where

C1 = 2 (s+6)/2 E ω p log 2(p + 1) δ EA(X)Γ (s + 1){1 + o γ (1)} C2 = 1 EA(X)Γ (s + 1)       2E[ ω 2 ] log(1/δ)[1 + γ,n ] 1 -δ/2 +4E[ ω ] p log( 2[p + 1] δ )(2 s/2+1 + 2 s/2 • 3 • log(3/δ))       .

4.A.3 Proof of Theorem 4.4.1 First step: apply a concentration inequality

In the following, we consider the function L :

H(k γ ) p → R defined by ∆ 2 ( pV (X), G) = L(h) = E ω E X e iω T pV (X) -E G e iω T G 2 ,
where h = (h 1 . . . h p ). The goal is to apply the McDiarmid inequality (Lemma A.1.1) to bound the probability P(L(h) > t) for some t > 0. This requires the existence of some deterministic quantity c > 0 such that for any 1 j p and h j , h j ∈ H(k γ ),

L(h -j ) -L(h) c ,
where h -j = (h 1 , . . . , h j-1 , h j , h j+1 , . . . , h p ).

We show that this condition is fulfilled conditionally to the event E M = ∩ p j=1 { h j M} for any M > 0. Assume E M holds for some M > 0. Using Jensen's inequality (combined with the convexity of the modulus function) and the triangle inequality yielding

|z 1 | 2 -|z 2 | 2 = |z 2 1 | -|z 2 2 | z 2 1 -z 2 2 for any z 1 , z 2 ∈ C, L(h -j ) -L(h) E ω E X e i p k j ω k h k , kX +ω j h j , kX -E G e iω T G 2 -E ω E X e i p k j ω k h k , kX +ω j h j , kX -E G e iω T G 2 E ω E X e i p k j ω k h k
, kX e iω j h j , kXe iω j h j , kX

• E X e i p k j ω k h k , kX e iω j h j , kX + e iω j h j , kX + 2E s,G e isω T G

4E ω E X e iω j h j , kXe iω j h j , kX

4E ω |ω j | E X h j -h j , k X 2 Σ γ (X, X) .
where we used the fact that t → exp(it) is 1-Lipschitz.

Lemma A.2.1 combined with assumption 0 < a A(X) A entails

Σ γ (X, X) = A(X)Γ (s + 1) (2γ) s [1 + A -1 (X)o γ (1)] AΓ (s + 1) (2γ) s [1 + a -1 o γ (1)] ,
where the 'o γ (1)' term is uniformly bounded with respect to X. Therefore,

L(h -j ) -L(h) 4E ω ω E X h j -h j , k X 2 AΓ (s + 1)(2γ) -s [1 + o γ (1)] 4E ω ω Σ γ (h j -h j ), h j -h j AΓ (s + 1)(2γ) -s [1 + o γ (1)] 4E ω ω (2γ) s λ 1 AΓ (s + 1) 1/2 h j -h j [1 + o γ (1)] 8E ω ω (2γ) s λ 1 AΓ (s + 1) 1/2 M [1 + o γ (1)] := M(γ) ,
Here λ 1 = sup u =1 Σ γ u, u denotes the largest eigenvalue of Σ γ .

Therefore, the McDiarmid inequality yields conditionally to E M and for any t > E h (L(h) | E M )

P (L(h) > t) P L(h) -E h [L(h) | E M ] > t -E h [L(h) | E M ] E M + P(E c M ) exp - 2(t -E h [L(h) | E M ]) 2 pM 2 (γ) + P(E c M ) .
Using the exponential Markov inequality and given some 0 < β < (2λ 1 ) -1 , P(E c M ) is bounded as follows

P(E c M ) pP ( h 1 > M) pe -βM 2 Ee β h 1 2 = pe -βM 2 j 1 (1 -2βλ j ) -1/2 = p exp         -βM 2 + (1/2) j 1 log 1 1 -2βλ j         p exp         -βM 2 + (1/2) j 1 2βλ j 1 -2βλ j         p exp -βM 2 + β(1 -2βλ 1 ) -1 ,
where we used the inequality log(x) x -1 and the equality Tr(Σ γ ) = 1.

Assuming M < 1 -1/ √ 2, setting β = (1 -2M)[ 1 + (2M/(1 -2M 2 )) 2 -1]/(2λ 1 ) entails P(E M ) p exp - M 2 2λ 1 , hence P(L(h) > t) exp - 2(t -E h [L(h)|E M ]) 2 pM 2 (γ) + p exp - M 2 2λ 1 .
In other words, with probability larger than 1δ for any δ > pe -M 2 /(2λ 1 )

L(h) E h [L(h) | E M ] + M(γ) p 2 log 1 δ -pe -M 2 /(2λ 1 )
1pe

-M 2 2λ 1 -1 E h L(h) + 8E ω (2γ) s λ 1 AΓ (s + 1) 1/2 M p 2 log 1 δ -pe -M 2 /(2λ 1 ) [1 + o γ (1)] . (4.A.18)
An upper bound for E h L(h) is derived in the second part thereafter.

Second step: find an upper bound of EL(h) E h [L(h)] can be expanded as

E h L(h) = E ω,h φ pV (X) (ω) -φ G (ω) = E ω,h E X,X e iω T ( pV (X)-pV (X )) +E G,G e iω T (G-G ) -2E X,G e iω T ( pV (X)-G) . (4.A.19)
Let ψ(t) = E ω e -t ω 2 /2 defined on R + and write δ2 X = Σ γ kX , kX γ = 1 (likewise, δ2 X = Σ γ kX , kX γ = 1) and δXX = Σ γ kX , kX γ . On the first hand,

E h,ω e iω T ( pV (X)-pV (X )) = E h,ω e i p j=1 ω j kX -kX ,h j γ = E ω e -1 2 ω 2 Σ γ ( kX -kX ), kX -kX γ = ψ [ δ2 X + δ2 X -2 δXX ] = ψ 2 -2 δXX .
On the other hand,

E G,G ,ω e iω T (G-G ) = E ω e -ω 2 = ψ(2) ,
and finally, using pV (X) ∼ N (0, I p ) with respect to h 1 , . . . , h p and conditionally to X,

E h,ω,G e iω T ( pV (X)-G) = E h,ω e iω T pV (X)-(1/2) ω 2 = E ω e -1 2 ω 2 δ2 X -1 2 ω 2 = ψ 1 + δ2 X = ψ (2) .
Plugging the three equalities into (4.A.19),

E h L(h) = E X,X ψ 2 -2 δXX -ψ(2) ,
and there exists ξ satisfying |ξ| δXX such that

(I) = 2E XX ψ (ξ) δXX 2 ψ ∞ E X,X | δXX | = 2 ψ ∞ E X,X δXX , (4.A.20)
where

ψ ∞ = sup{|ψ (t)| : t ∈ R + }. Note that ψ ∞ is finite since |ψ (t)| = E ω -(1/2) ω 2 e -t ω 2 /2
(1/2)E ω 2 < +∞. Here we are allowed to switch the mean operator and the derivation because of the Dominated Convergence Theorem and because t → exp(-t ω 2 /2) is 1-Lipschitz on R + .

It remains to find an equivalent term for E δXX when γ → +∞. Lemma A.2.1 and the assumption A(x) a > 0 for every x ∈ supp(P ) provides

E X,X δXX = E X,X           Σ γ (X, X ) Σ γ (X, X)Σ γ (X , X )           E X,X       γ s Σ γ (X, X ) A(X)A(X )Γ (s + 1) {1 + a -1 o γ (1)}       E X,X γ s Σ γ (X, X ) aΓ (s + 1) {1 + a -1 o γ (1)} ,
where the o γ (1) term is not random. It follows

E δXX γ s E X,X ,X e -γ[d 2 (X,X )+d 2 (X ,X )] aΓ (s + 1) {1 + a -1 o γ (1)} = γ s E X Σ 2 γ/2 (X , X ) aΓ (s + 1) {1 + o γ (1)} γ s E X A 2 (X )Γ 2 (s + 1)(2γ) -2s [1 + a -1 o γ (1)] aΓ (s + 1) {1 + o γ (1)} = E X [A 2 (X)]Γ 2 (s + 1) a(2γ) 2s [1 + o γ (1)] ,
which finally leads to

E h [L(h)] E ω 2 E X [A 2 (X)]Γ (s + 1) a(4γ) s [1 + o γ (1)] . (4.A.21)

Conclusion of the proof

Continuing from (4.A.18), we proved that with probability larger than 1δ for any δ > pe -M 2 /(2λ 1 ) ,

L(h) 1 -pe -M 2 2λ 1 -1 E h L(h) + 8E ω (2γ) s λ 1 AΓ (s + 1) 1/2 M p 2 log 1 δ -pe -M 2 /(2λ 1 ) [1+o γ (1)] .
Besides, (4.A.21) provided the following upper bound for E h L(h)

E h L(h) E ω 2 E X [A 2 (X)]Γ (s + 1) a(4γ) s [1 + o γ (1)] ,
and Lemma 4.B.1 provides when γ → +∞

λ 1 = EA(X)Γ (s + 1) γ s [1 + o γ (1)] .
Hence if one sets pe -M 2 /(2λ 1 ) = δ/2 that is M = 2λ 1 log(2p/δ) , then with probability larger than 1δ for any 0 < δ < 1

L(h)          E ω 2 E X [A 2 (X)]Γ (s + 1)a -1 (4γ) -s + 8E ω pAΓ (s + 1) log 2(p + 1) δ γ 2 -s/2          [1 + o γ (1)] = 8E ω pAΓ (s + 1) log 2(p + 1) δ γ 2 -s/2 [1 + o γ (1)] .

4.A.4 Proof of Theorem 4.4.2

First step: apply a concentration inequality Let X = (X 1 , . . . , X n ) denote an i.i.d. sample generated by P . In the following, we consider the function L(., X) :

H(k γ ) p → R defined by ∆ 2 pV n (X), G = L(h; X) = E ω E X e iω T pV n (X) -E G e iω T G 2 ,
with h = (h 1,n , . . . , h p,n ) following independently a Gaussian process of mean zero and of covari-

ance Σ γ,n = n -1 n i=1 k ⊗2 γ (X i , .
). The goal is to apply the McDiarmid inequality (Lemma A.1.1) to bound the probability P h (L(h; X) > t) for some t > 0 and conditionally to X. This requires the existence of some deterministic quantity c > 0 such that for any 1 j p and h j , h j ∈ H(k γ ), L(h -j ; X) -L(h; X) c , where h -j = (h 1,n , . . . , h j-1,n , h j,n , h j+1,n , . . . , h p,n ). We show that this condition is fulfilled conditionally to the event

E M = ∩ p j=1 { h j,n
M} for any M > 0. Note that h 1,n , . . . , h p,n are still independent under E M . Assume E M holds for some M > 0. Using Jensen's inequality (combined with the convexity of the modulus function) and the triangle inequality yielding

|z 1 | 2 -|z 2 | 2 = |z 2 1 | -|z 2 2 | z 2 1 -z 2 2 for any z 1 , z 2 ∈ C, L(h -j ; X) -L(h; X) E ω E X e i p k j ω k h k , k(n) X +ω j h j , k(n) X -E G e iω T G 2 -E ω E X e i p k j ω k h k , k(n) X +ω j h j , k(n) X -E G e iω T G 2 E ω E X e i p k j ω k h k , k(n) X e iω j h j , k(n) X -e iω j h j , k(n) X • E X e i p k j ω k h k , k(n) X e iω j h j , k(n) X + e iω j h j , k(n) X + 2E G e iω T G 4E ω E X e iω j h j , k(n) X -e iω j h j , k(n) X 4E ω |ω j | E X | h j -h j , k X | Σ 1/2 γ,n (X, X) 4E ω ω E X h j -h j , k X 2 Σ γ,n (X, X) ,
where we used the fact that t → exp(it) is 1-Lipschitz.

By Lemma A.2.4, for any δ ∈ (0, 1)

Σ γ,n (x, x) Σ γ (x, x) -1 2 log(1/δ )γ s aΓ (s + 1)n         1 + κ log(1/δ )(2γ) s aΓ (s + 1)n {1 + o γ (1)}        
, with probability larger than 1δ over X. This event is denoted B in the following.

It follows that under the events E M and B

L(h -j ; X) -L(h; X) 4E ω ω E X h j -h j , k X 2 Σ γ,n (X, X) 4E ω ω E X h j -h j , k X 2 Σ γ (X, X) 1 + ∆ γ,n 1/2
, where

∆ γ,n = 2 log(1/δ)γ s aΓ (s + 1)n + 2 (s+3)/2 κ log(1/δ)γ s aΓ (s + 1)n [1 + o γ (1)] . Lemma A.2.1 combined with assumption 0 < a A(X) A entails Σ γ (X, X) = A(X)Γ (s + 1) (2γ) s [1 + A -1 (X)o γ (1)] AΓ (s + 1) (2γ) s [1 + a -1 o γ (1)] ,
where the 'o γ (1)' term is uniformly bounded with respect to X. Therefore,

L(h -j ; X) -L(h; X) 4E ω ω E X h j -h j , k X 2 AΓ (s + 1)(2γ) -s [1 + o γ (1) + ∆ γ,n ] 4E ω ω Σ γ (h j -h j ), h j -h j AΓ (s + 1)(2γ) -s [1 + o γ (1) + ∆ γ,n ] 4E ω ω (2γ) s λ 1 AΓ (s + 1) 1/2 h j -h j [1 + o γ (1) + ∆ γ,n ] 8E ω ω (2γ) s λ 1 AΓ (s + 1) 1/2 M [1 + o γ (1) + ∆ γ,n ] ,
Here λ 1 = sup u =1 Σ γ , u, u denotes the largest eigenvalue of Σ γ . Lemma 4.B.1 provides

λ 1 = EA(X)Γ (s + 1) γ s [1 + o γ (1)] ,
which entails

L(h -j ; X) -L(h; X) 2 s/2+3 E ω ω M [1 + o γ (1) + ∆ γ,n ] ∆ = M(γ) .
Therefore, the McDiarmid inequality yields conditionally to E M ∩B and X for any t > E(L | E M ∩B)

P (L(h; X) > t) P L -E[L | E M ∩ B] > t -E[L | E M ∩ B] E M ∩ B + P(E c M ) + P(B c ) exp - 2(t -E[L | E M ∩ B]) 2 pM 2 (γ) + P(E c M ) + δ .
Let λ1 λ2 . . . λn denote the eigenvalues of Σ γ,n . Using the exponential Markov inequality and given some 0 < β < (2 λ1 ) -1 , P h (E c M ) is bounded as follows

P h (E c M ) pP h h 1,n > M pe -βM 2 Ee β h 1,n 2 = pe -βM 2 j 1 (1 -2β λj ) -1/2 = p exp         -βM 2 + (1/2) j 1 log       1 1 -2β λj               p exp         -βM 2 + (1/2) j 1 2β λj 1 -2β λj         p exp -βM 2 + β(1 -2β λ1 ) -1 ,
where we used the inequality log(x) x -1 and the equality Tr(Σ γ,n ) = 1.

Assuming M < 1 -1/ √ 2, one sets β = (1 -2M)[ 1 + (2M/(1 -2M 2 )) 2 -1]/(2 λ1 ) entails P h (E M ) p exp - M 2 2 λ1 , hence P h (L(h; X) > t) exp - 2(t -E[L | E M ∩ B]) 2 pM 2 (γ) + p exp - M 2 2 λ1 + δ . (4.A.22)
In other words, McDiarmid's inequality entails with probability larger than 1δ for any δ > pe -M 2 /(2 λ1 ) + δ

L E h [L | E M ∩ B] + M(γ) p 2 log       1 δ -δ -pe -M 2 /(2 λ1 )       E h [L | E M ∩ B] + 2 s/2+3 E ω ω M p 2 log       1 δ -δ -pe -M 2 /(2 λ1 )       {1 + o γ (1) + ∆ γ,n } P -1 (E M ∩ B)E h L + 2 s/2+3 E ω ω M p 2 log       1 δ -δ -pe -M 2 /(2 λ1 )       {1 + o γ (1) + ∆ γ,n } .
Second step: find an upper bound for E h L(h; X)

The mean term in the rhs above can be written

E h L(h; X) = E ω,h φ pV n (X) (ω) -φ G (ω) 2 = E ω,h E X,X e iω T ( pV n (X)-pV n (X )) +E G,G e iω T (G-G ) -2E X,G e iω T ( pV n (X)-G) . (4.A.23) Let ψ(t) = E ω e -t ω 2 /2 defined on R + .On the first hand, E h,ω,X,X e iω T ( pV (X)-pV (X )) = E h,ω,X,X e i p j=1 ω j k(n) X ,h j γ - k(n) X ,h j γ = E ω,X,X e -1 2 ω 2 {2-2 δXX } = E X,X ψ 2 -2 δXX , where δXX = Σ γ,n k X , k X γ Σ γ,n (X, X)Σ γ,n (X , X )
.

On the other hand,

E G,G ,ω e iω T (G-G ) = E ω e -ω 2 = ψ(2) ,
and finally, using pV n (X) ∼ N (0, I p ) with respect to h and conditionally to X, X 1 , . . . , X n ,

E h,X,G e iω T ( pV n (X)-G) = E h,X e iω T pV n (X)-ω 2 /2 = E ω e -ω 2 = ψ(2) .
Plugging the three equalities into (4.A.23),

E h L(h; X) = Eψ(2 -2 δXX ) -ψ(2) 2 ψ ∞ E δXX .
where

ψ ∞ = sup{|ψ (t)| : t ∈ R + }. Note that ψ ∞ is finite since |ψ (t)| = E ω -(1/2) ω 2 e -t ω 2 /2
(1/2)E ω 2 < +∞. Here we are allowed to switch the mean operator and the derivation because of the Dominated Convergence Theorem and because t → exp(-t ω 2 /2) is 1-Lipschitz on R + .

By Lemma A.2.4, the event

B =          Σ γ,n (x, x) Σ γ (x, x) -1 2 log(1/δ)γ s aΓ (s + 1)n         1 + κ log(1/δ)(2γ) s aΓ (s + 1)n {1 + o γ (1)}                 
occurs with probability larger than 1δ for any δ ∈ (0, 1] and any x ∈ X . Moreover under

B Σ γ (X, X) Σ γ,n (X, X) = 1 + Σ γ,n (X, X) Σ γ (X, X) -1 -1/2 1 - Σ γ,n (X, X) Σ γ (X, X) -1 -1/2 1 + 1 2 Σ γ,n (X, X) Σ γ (X, X) -1 1 + 2 log(1/δ)γ s aΓ (s + 1)n         1 + κ log(1/δ)(2γ) s aΓ (s + 1)n {1 + o γ (1)}        
.

Introducing A = sup{A(x) : x ∈ X } < +∞, it follows E δXX E X,X           1 n n i=1 k γ (X, X i )k γ (X i , X ) Σ γ,n (X, X)Σ γ,n (X , X )                   1 + log(1/δ)γ s 2aΓ (s + 1)n + 2 s+1 2 κ log(1/δ)γ s aΓ (s + 1)n         E X,X           1 n n i=1 k γ (X, X i )k γ (X i , X ) Σ γ (X, X)Σ γ (X , X )           {1 + o γ (1)}         1 + log(1/δ)γ s 2aΓ (s + 1)n + 2 s+1 2 κ log(1/δ)γ s aΓ (s + 1)n         E X,X        1 n n i=1 k γ (X, X i )k γ (X i , X ) aΓ (s + 1)(2γ) -s        {1 + o γ (1)} =         1 + log(1/δ)γ s 2aΓ (s + 1)n + 2 s+1 2 κ log(1/δ)γ s aΓ (s + 1)n         (1/n) n i=1 Σ 2 γ/2 (X i , X i ) aΓ (s + 1)(2γ) -s {1 + o γ (1)}         1 + log(1/δ)γ s 2aΓ (s + 1)n + 2 s+1 2 κ log(1/δ)γ s aΓ (s + 1)n         AΓ 2 (s + 1)(2γ) -2s aΓ (s + 1)(2γ) -s {1 + o γ (1)} =        A a Γ (s + 1) (2γ) s + A 2 Γ (s + 1) log(1/δ) 2 1+2s a 3 γ s n + 2 (1-s)/2 κA log(1/δ) a 2 n        {1 + o γ (1)} ,
that is

E h L(h; X) E ω 2        A a Γ (s + 1) (2γ) s + A 2 Γ (s + 1) log(1/δ) 2 1+2s a 3 γ s n + 2 (1-s)/2 κA log(1/δ) a 2 n        {1 + o γ (1)} , (4.A.24)
with probability larger than 1δ over X.

Conclusion of the proof

Continuing from (4.A.22), we proved that with probability larger than 1δ (over h) for any

δ > pe -M 2 /(2 λ1 ) + δ , L(h; X) E h L P(E M ∩ B) + 2 s/2+3 E ω ω M p 2 log       1 δ -δ -pe -M 2 /(2 λ1 )       {1 + o γ (1) + ∆ γ,n } 1/2 , where ∆ γ,n = 2 log(1/δ)γ s aΓ (s+1)n + 2 (s+3)/2 κ log(1/δ)γ s aΓ (s+1)n {1 + o γ (1)}.
Hence if one sets δ and M such that

δ = δ 4 and pe -M 2 /[2 λ1 ] = δ 4 P(E c M ) ,
then M = 2 λ1 log(4p/δ) and with probability larger than 1 -2δ over h and X L(h; X)

E h L(h; X) 1 -δ 2 + 2 s/2+3 E ω ω p λ1 log 2 + 4p δ {1 + o γ (1) + ∆ γ,n } 1/2 .
Theorem 3.3.2 in [START_REF] Zwald | Performances d'Algorithmes Statistiques d'Apprentissage: "Kernel Projection Machine" et Analyse en Composantes Principales à Noyaux[END_REF] provides with probability larger than 1δ for any δ > 0 the inequality

| λ1 -λ 1 | 2 + 3 log(3/δ) √ n , which entails combined with Lemma 4.B.1 λ1 EA(X)Γ (s + 1) γ s {1 + o γ (1)} + 2 + 3 log(3/δ) √ n .
Hence with probability larger than 1 -2δ

L(h; X) E h L(h; X) 1 -δ + 2 s/2+3 E ω ω p log 2 + 4p δ       EA(X)Γ (s + 1){1+o γ (1)} γ s + 2 + 3 log(3/δ) √ n       1 + o γ (1) + ∆ γ,n E h L(h; X) 1 -δ 2 + 2 s/2+3 E ω ω p log 2 + 4p δ        EA(X)Γ (s + 1) γ s + 3 log(3/δ) √ n + 3 • 2 s+3 2 κ log(3/δ) log(1/δ)γ s aΓ (s + 1)n 3/2        {1+o γ,n (1)} , (4.A.25)
where the 'o γ,n (1)' term converges to 0 when both γ and n tend to infinity.

This upper bound suggests that one must have γ s = o(n 3/2 ) so that L(h; X) converges to 0 with high probability. However it is easy to see that this condition is not necessary by deriving an alternative upper bound for L(h; X) that is simpler but less tight. A basic Markov inequality entails with probability larger than 1δ over h

L(h; X) 1 δ E h L(h; X) , (4.A.26)
so that with probability larger than 1 -3δ, L(h; X) is smaller than

E h L(h; X) 1 -δ/2 min                    1 δ , 1 + 2 s/2+3 E ω ω p log 2+4p δ EA(X)Γ (s+1) γ s + 3 √ log(3/δ) √ n + 3•2 s+3 2 κ √ log(3/δ) log(1/δ)γ s aΓ (s+1)n 3/2 {1+o γ,n (1)} E h L(h; X)                    .
Finally, (4.A.24) provided the following upper bound with probability larger than 1δ over the sample X

E h L(h; X) E ω 2        A a Γ (s + 1) (2γ) s + A 2 Γ (s + 1) log(1/δ) 2 1+2s a 3 γ s n + 2 (1-s)/2 κA log(1/δ) a 2 n        {1 + o γ (1)} ,
and with probability larger than 1 -4δ over h and X,

L(h; X) E ω 2        A a Γ (s + 1) (2γ) s + A 2 Γ (s + 1) log(1/δ) 2 1+2s a 3 γ s n + 2 (1-s)/2 κA log(1/δ) a 2 n        min 1 δ , 1 + ξ γ,n {1+o γ (1)} ,
where ξ γ,n is defined by

ξ γ,n = 2 s/2+3 E ω ω p log 2+4p δ EA(X)Γ (s+1) γ s + 3 √ log(3/δ) √ n + 3•2 s+3 2 κ √ log(3/δ) log(1/δ)γ s aΓ (s+1)n 3/2 E ω 2 A a Γ (s+1) (2γ) s + A 2 Γ (s+1) log(1/δ) 2 1+2s a 3 γ s n + 2 (1-s)/2 κA log(1/δ) a 2 n (1 -δ/2) {1 + o γ,n (1)},
and converges to 0 when γ, n → +∞ and γ = o(n 3/(2s) ).

4.B Additional lemmas

(P ) = {x ∈ X ∀ > 0, P(d 2 (x, X) < ) > 0}.
Assume that the distribution of X admits no point mass and that there exists a function A : supp(X) → R * + and s > 0 such that ∀x ∈ supp(X) , P d 2 (x, X) t ∼ A(x)t s , when t → 0 .

Then for any integer r,

λ r ∼ [E X A(X)]Γ (s + 1) γ s , when γ → +∞ ,
where Γ (s) = +∞ 0 x s-1 e -x dx denotes the Gamma function.

Proof. Let r > 0 be any integer. The r-th eigenvalue λ r of Σ γ satisfies the following equation

λ r = E X,X e -γd 2 (X,X ) ψ r (X)ψ r (X ) ,
where X is an independent copy of X and ψ r is the corresponding eigenvector of the covariance function C(x, x ) = Ek γ (X, x)k γ (X, x ) in L 2 (P ).

Let M > 0 be some arbitrary quantity. By conditioning,

λ r = pξ -+ ξ + , (4.B.27)
where

p = P (A -) , A -= d 2 (X, X ) M γ , A + = d 2 (X, X ) > M γ , ξ -= E e -γd 2 (X,X ) ψ r (X)ψ r (X ) A -, ξ + = E e -γd 2 (X,X ) ψ r (X)ψ r (X )1 1 A + .
It is straightforward that

p = E X P X d 2 (X, X ) M γ ∼ [E X A(X)]M s γ s . ( 4

.B.28)

Under the event A -, d 2 (X, X ) converges to 0 if M satisfies M = o γ (γ). Therefore, the continuity of the inner product ., . γ and the kernel

k γ implies ψ r (X ) = ψ r , k γ (X ) = ψ r , k γ (X ) + o γ (1) = ψ r (X) + o γ (1) so that ξ -= E e -γd 2 (X,X ) ψ r (X)ψ r (X ) A -= E e -γd 2 (X,X ) [ψ 2 r (X) + ] A - = E X ψ 2 r (X)E X e -γd 2 (X,X ) A -[1 + o γ (1)] , (4.B.29)
where the 'o(γ)' term put outside of the expectation comes from the boundedness of since

= ψ r (X)(ψ r (X ) -ψ r (X)) ψ r k X k γ (X , .) -k γ (X, .) 2 ψ r 2 < +∞ .
Let g(t; x) the function that satisfies g(t; x) → 0 when t → 0 for all x and P(d 2 (X,

X ) t) = A(x)(1 + g(t; x))t s . Fubini's Theorem entails E X e -γd 2 (x,X ) A -= E X [-e -y ] y=+∞ y=γ x-X 2 A - = E X +∞ γ x-X 2 e -y dy A - = +∞ 0 e -y E X 1 1 {γd 2 (x,
M s γ(s + 1, M) + e -M ∼ γ→+∞ Γ (s + 1) M s ,
and therefore It remains to show that ξ + is negligible in relation to γ -s .

ξ -∼ E X ψ 2 r (X)E X e -γd 2 (x,X ) ψ 2 r (x) A -∼ Eψ 2 r (X) Γ (s + 1) M s = Γ (s + 1) M s , ( 4
We use the convexity of exp(-x) that provides the inequalities e -x e -M -(e -Me -(M+ M) )(x -M)/ M when M x M + M for some M M, and e -x e -Me -M (x -M) for x M + 1.

ξ + = Ee -γd 2 (X,X ) ψ 2 r (X)1 1 A + - 1 2 Ee -γd 2 (X,X ) [ψ r (X) -ψ r (X )] 2 1 1 A + e -M E 1 - 1 -e - M M [γd 2 (X, X ) -M] ψ 2 (X)1 1 {M/γ d 2 (X,X ) (M+ M)/γ} + exp(-(M + M)) - 1 2 E [e -M -e -M (γd 2 (X, X ) -M)][ψ(X) -ψ(X )] 2 1 1 B e -M P( B)E ψ 2 r (X) B + e -M + 1 2 E [ψ r (X) -ψ r (X )] 2 1 1 B e -M P( B)E ψ 2 r (X) B + e -M + 1 2 E [ψ r (X) -ψ r (X )] 2 , (4.B.31) where B = {M/γ d 2 (X, X ) (M + M)/γ} and B = M/γ d 2 (X, X ) (M + 1)/γ .
It is shown that ξ + = o(e -M ) by checking that the rhs of (4.B.31) tends to 0. First,

E ψ 2 r (X) B = E X ψ 2 r (X)E X | B (1|X) = E X ψ 2 r (X) = 1 ,
and choosing M → +∞ such that M = o γ (γ) implies P( B) → 0 and exp(-M) → 0. Finally,

E [ψ r (X) -ψ r (X )] 2 = 2 -2E ψ r (X)ψ r (X ) B 2[Eψ 2 r (X) + Eψ 2 r (X )] = 4 .
Hence ξ + = O γ (e -M ) and setting M = q log(γ) for some q > s leads to λ r ∼ EA(X)Γ (s + 1)γ -s .

4.B.2 Asymptotic orthonormality of Gaussian processes

Lemma 4.B.2. Consider p independent zero-mean Gaussian processes h 1 , . . . , h p with covariance

Σ γ = E X∼P k γ (X, •) ⊗2
for some probability measure P . Then these p variables are asymptotically orthonormal when γ → +∞, that is

∀1 i p, h i 2 γ -→ γ→+∞ 1 a.s. , and
∀1 i, j p, i j, h i , h j γ -→ γ→+∞ 0 a.s. . Proof. Consider the eigendecomposition Σ γ = r 1 λ r Ψ ⊗2 r
where (Ψ r ) r form an orthonormal basis of H(k γ ). On the first hand,

E h h 2 γ = E h Tr(h ⊗2 ) = Tr(Σ γ ) = E X Tr(k ⊗2 X ) = Ek γ (X, X) = 1 .
On the other hand, since h, Ψ r γ are independent zero-mean Gaussians of respective variances

λ r , Var( h 2 γ ) = Var        r 1 h, Ψ r 2 γ        = r 1 Var h, Ψ r 2 γ = 2Tr(Σ 2 γ ) .
According to Lemma A.2.2, Tr(Σ 2 γ ) converges to 0 as γ → +∞. Therefore Var( h 2 γ ) converges to 0 and h 2 γ converges almost surely to its mean 1. If i j, then E h i , h j γ = 0 since h i and h j are zero-mean and independent. Also

Var( h i , h j γ ) = E h i , h j 2 γ = E h i Σ γ h i , h i γ = Tr(Σ 2 γ ) → 0 , hence h i , h j γ → 0 almost surely.
Chapter 5

A New Method for Online Outlier

Detection

In Chapter 4, we have described the probabilistic behaviour of random projections of an embedded variable in the RKHS of a Gaussian RBF kernel. As a reminder, we established that most of those projections converge weakly to a so-called scale-mixture of isotropic Gaussians when the hyperparameter of the kernel tends to infinity. The goal of the present chapter is to illustrate the usefulness of the aforementioned result in a practical problem, that is the problem of outlier detection. To this aim we describe a new method for outlier detection that is based on random projections in an RKHS.

The problem of outlier detection

Outlier detection (OD) is often mentioned in the literature under several alternative names such as anomaly detection or novelty detection, yet all refer to the same goal: to identify observations in a given dataset that depart from a "normal" statistical behaviour governing the majority of the dataset. Such "abnormal" observations are called outliers, while the non-outliers are called inliers in the following.

The latter definition of the problem may seem vague and actually there is no clear consensus on a proper definition of an outlier in the related literature. Indeed, the literature about OD is abundant and consists in many different kinds of approaches as pointed out in the extensive review of [START_REF] Pimentel | A review of novelty detection[END_REF]: probabilistic methods, distance-based methods or information-theoretic methods, among others. Each of these different classes of methods stems from a different characterization of outliers. For instance, a probabilistic approach assumes that the observations have been drawn from some probability distribution and outliers are points that lie in low-density regions, whereas a distance-based method labels as outliers observations that are abnormally 93 distant from most of the dataset and a domain-based approach assumes that the inliers lie in a subset of the space where data take values. All in all, the types of methods mentioned above can be deemed "domain-based" since they all eventually lead to define an acceptation region of the data space that contain most of the inliers and almost no outliers: in a probabilistic point of view, such an acceptation region corresponds to the region where the density function of the "normal" distribution is above a certain threshold; in a distance-based approach, this region consists in points whose distance to the rest of the dataset is smaller than some threshold.

Defining such an acceptation region is subject to two kinds of errors: labeling a inlier as an outlier (false alarm error) and calling an outlier a non-outlier (missed detection error). It is desired to control the probabilities of committing these types of errors. A convenient framework to achieve this goal is that of probabilistic methods for OD, that is the inliers are assumed to have been drawn from a known distribution so that the probability of false alarm error becomes available. However, the shortcoming of such methods is that the working model may not fit the actual data at all. Therefore our concern is to access the error probabilities while avoiding any restrictive assumption about the data distribution.

Formalized problem

From now on, the problem is formalized as follows. Consider a stream of X -valued data

X 1 , X 2 , . . . , X n , X,
for some set X . We assume throughout this chapter that X is endowed with a metric d(•, •).

The n previous points X 1 , . . . , X n are assumed to be independently drawn from a common distribution P and represent the inliers. The goal is to test whether the new point X was generated by P or by an alternative distribution P 1 . Note that this formulation is similar to homogeneity testing, where one tests the null-hypothesis that two samples X 1 , . . . , X n and X 1 , . . . , X m come from the same underlying distribution. The difference lies in the fact that one of the two samples consists of only one observation -that is reduced to X in our case. Since it is impossible to estimate the underlying distribution of X on the basis of one observation, some assumptions about the possible alternative distribution P 1 are necessary to make the problem tractable. Thus outlier detection consists in determining a region of X that contains most of the inliers and we are naturally led to define an outlier as an observation that lies outside of the support of the inlier distribution P -which is denoted supp(P ) and defined formally as supp(P ) = {x ∈ X | ∀ > 0, P X∼P (d(X, x) < ) > 0}. This assumption allows to get consistency results (i.e. almost surely detect an outlier as n grows to infinity) as we will see later in Section 5.3.2.

Therefore the outlier problem can be cast as the problem of testing the null-hypothesis H 0 against the alternative hypothesis H 1 which are both defined as

       H 0 : X ∼ P , H 1 : X ∼ P 1 where P 1 ∈ {Q : ν (supp(P ) ∩ supp(Q)) = 0} ,
where ν is a positive measure on X such that P and P 1 are absolutely continuous with respect to ν.

Note that we have presented the outlier problem in an online framework, that is single observations are assumed to arrive one after the other and are tested sequentially. This is opposed to batch OD, where a full sample is directly available in its entirety and a small sized subsample of outliers must be identified -see for instance the method proposed by [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF] mentioned in Section 3.4.2. Surprisingly, OD is often primarily formulated in the batch version and an online version is designed later -for instance OD with one-class SVM was first conceived in a batch fashion in [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF] then in an online manner as in [START_REF] Gretton | On-line one-class support vector machines. an application to signal segmentation[END_REF]. However, batch OD is typically more difficult than online OD since in the latter case a subsample of inliers is readily given and can serve as a reference, whereas in the former case the whole sample -outliers included -must be used to estimate useful quantities which may lead to erroneous estimations (especially if poorly robust estimators are used). Therefore we chose to formalize the OD problem in an online way as a more natural starting point.

Existing methods

As the extensive review by [START_REF] Pimentel | A review of novelty detection[END_REF] shows, numerous methods for OD have been proposed in the literature. We focus on two fairly recent ways of performing OD: one-class Support Vector Machine (oc-SVM) and kernel Principal Component Analysis (kernel PCA). SVM and kernel PCA have been already introduced in Section 2.3.1 and Section 2.3.2 respectively. These two techniques are kernel methods and as such, they are more general than OD methods performed solely in the input space X -in particular, the set X may take an arbitrary form.

One-class SVM

From a probabilistic point of view, the OD problem is linked to the distribution of the inlierswhich may be represented by a density function f if the input space X is R d . When it comes to define an acceptation region A , a good candidate is a subset of X on which the density f take large values, say

A = x ∈ X f (x) > τ , (5.1.1)
where τ > 0 is some prescribed threshold. Defining A this way requires to have access to the density function f which is usually unknown in practice. Therefore an estimator for f -based on a sample of inliers X 1 , . . . , X n -is required. However estimating a density function may be a difficult task, especially when X is a high-dimensional space where the curse of dimensionality arises and has to be circumvented by means of Gaussian assumptions [START_REF] Moghaddam | Principal Manifolds and Probabilistic Subspaces for Visual Recognition[END_REF] or sparsity assumptions [START_REF] Liu | Saprse Nonparametric Density estimation in High Dimensions Using the Rodeo[END_REF][START_REF] Vadivel | Generalized Subspace Based High Dimensional Density Estimation[END_REF] . That being said, estimating f is just an intermediate problem thus we should not be concerned about the accuracy of such estimation. More precisely, we are solely interested in the location of small density regions of X , hence even an estimate of f which poorly approximates f on high density regions may eventually allow to correctly detect outliers.

All in all, it may be more optimal to bypass the estimation of f and to consider an acceptation region A where the density f is replaced by some proxy function g that could achieve the goal of detecting outliers as well.

Following this rationale, let us rewrite (5.1.1) by replacing f by such a function g. Assuming that g belongs to some RKHS H(k) with reproducing kernel k : X × X → R, the reproducing property yields g(x) = g, k(x, •) H(k) for every x ∈ X . Therefore a point x ∈ X belongs or not to A according to the value of

sgn g, k(x, •) H(k) -τ .
In other words, inliers and outliers are separated in the RKHS by the hyperplane

H = h ∈ H(k) g, h H(k) -τ = 0 ,
which looks like the decision boundary of an SVM (Section 2.3.1). However the difference with standard SVM is that the training set essentially consists of inliers instead of two identified classes with comparable sizes. This leads to a special kind of SVM called one-class SVM (oc-SVM) that was introduced by [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF].

Instead of separating two given classes of points, oc-SVM separates the whole given sample from the origin in the RKHS, which is done through the following optimization scheme

minimize 1 2 g 2 H(k) + 1 νn n i=1 ξ i -τ over g ∈ H, τ ∈ R, ξ 1 , . . . , ξ n ∈ R + such that g, k(X i , •) H(k) τ -ξ i for all 1 i n , (5.1.2)
where ν ∈ (0, 1] is some parameter whose influence is discussed below.

By maximizing τ, the solution of (5.1.2) maximizes the distance between the embedded data and H , yet allows a few points to be misclassified via the slack variables ξ 1 , . . . , ξ n for a better generalization error as in the standard SVM. Despite the different format, the parameter ν plays a similar role as C in the standard SVM that is controlling how many observations are allowed to be misclassified. For instance Proposition 4 in [START_REF] Schölkopf | Estimating the support of a high-dimensional distribution[END_REF] states that ν is an upper bound on the proportion of declared outliers within the training set. However, this does not imply that the probability of committing a false alarm error when testing new points is also bounded by ν. More recently in [START_REF] Vert | Consistency and Convergence Rates of One-Class SVMs and Related Algorithms[END_REF], theoretical error bounds are provided for oc-SVM, however the bounded error is the classification error which encompasses false alarm errors and the error of missing an outlier altogether. Similarly [START_REF] Steinwart | Density Level Detection is Classification[END_REF] derives upper bounds for the classification error for density level detection (which is a problem related to outlier detection) through SVM. All in all this lack of control on error probabilities constitutes one weak point of oc-SVM.

Kernel PCA [START_REF] Hoffmann | Kernel PCA for novelty detection[END_REF] proposed to use kernel PCA to detect outliers in order to improve on the performance of one-class SVM.

In Section 2.3.2, we have introduced kernel PCA as a non-linear extension of principal component analysis. As a reminder, kernel PCA consists in applying standard PCA in the feature space instead of the input space. More precisely, kernel PCA seeks out principal components ϕ 1 , . . . , ϕ p ∈ H(k) which form an orthonormal family of vectors in H(k) and are such that the subspace V spanned by ϕ 1 , . . . , ϕ p captures most of the variance of the embedded data.

The main idea of [START_REF] Hoffmann | Kernel PCA for novelty detection[END_REF] is to use the reconstruction error to detect outliers. The reconstruction error RE(x) for a point x ∈ X is defined by

RE(x) = k(x, •) -Π V k(x, •) 2 H(k) ,
where Π V denotes the projection operator onto the subspace V and k(x,

x ) = k(x, •) -µ, k(x , •) - µ H(k) is the re-centered kernel with µ = (1/n) n i=1 k(X i , •).
A test point X is declared an outlier if the corresponding reconstruction error RE(X) is larger than some critical value τ > 0. In other words the acceptation region in the RKHS is the τ-neighborhood of the subspace V . [START_REF] Hoffmann | Kernel PCA for novelty detection[END_REF] argues that when k is a translation-invariant kernel that is k(x, x ) = κ(xx ) for some κ : X → R, this acceptation region encloses the embedded inliers more tightly than the acceptation region yielded by oc-SVM. This is due to the fact that embedded points all lie on the same sphere S 0 of H(k) when k is a translation-invariant kernel.

In this case, it is known that the intersection of S 0 with the separating hypersphere of oc-SVM coincides with the intersection of S 0 and the decision boundary S provided by Support Vector Domain Description (SVDD) [START_REF] Tax | Support vector domain description[END_REF] -which is also a sphere. Therefore oc-SVM and SVDD classify points in the same way. Since oc-SVM amounts to define the decision boundary as a sphere, it does not take into account possible heterogeneous variances of the embedded data along different directions in the RKHS, unlike kernel PCA. For this reason, kernel PCA yields a smallest acceptation region that encloses inliers in the RKHS hence kernel PCA is less prone to accepting actual outliers as inliers than oc-SVM.

Note that parametric OD methods that involve density estimation are contained in kernel PCA, since OD with Parzen window density estimation can be seen as a special case of OD with kernel PCA (as long as a positive definite kernel is involved). A Parzen window density estimator f is of the form f (x) = (1/nh) n i=1 K ((xx i )/h) where K(•) is some real positive function satisfying K(t)dt = 1 and h > 0 is the bandwidth parameter. It turns out that OD with f coincides with kernel PCA when choosing a number p of principal component equal to p = 0. In this case, the projector Π V is equal to 0 and the reconstruction error RE(x) writes

RE(x) = k(x, •) - 1 n n i=1 k(x i , •) 2 H(k) = k(x, x) + 1 n 2 n i,j=1 k(x i , x j ) - 2 n k(x i , x) .
Since k is chosen as a translation-invariant kernel, k(x, x) = κ(0) is constant, and (1/n 2 ) n i,j=1 k(x i , x j ) does not depend on x as well. Therefore, an outlier x corresponds to a large value of RE(x) which in turn corresponds to a small value of (1/n) n i=1 k(x i , x). The latter quantity matches the Parzen window density estimator with K(z) = hκ(hz). On the other hand, outlier detection with kernel PCA can be seen as a special case of [START_REF] Vito | Learning Sets with Separating Kernels[END_REF].

The main drawback of kernel PCA is its expensive computational cost. Indeed, finding the principal components ϕ 1 , . . . , ϕ p involves the eigendecomposition of the Gram matrix K = k(x i , x j ) 1 i,j n which is done in O(n 3 ) time. Only considering the p first principal components actually requires an execution time of order O(n 2 p). However the optimal value of p is not known, hence the OD algorithm must be reiterated for several possible values of p (up to the case p = n) in order to observe which choice of p yields the best performance -for instance this is done to compute the criterion proposed by [START_REF] Xiao | Model selection of Gaussian kernel PCA for novelty detection[END_REF] to select the optimal p.

New approach for outlier detection

Principle

We propose a new kernel-based method for outlier detection that circumvents the respective shortcomings of the two methods presented in Section 5.1.2. This new approach mostly relies on the results in Chapter 4 about the distribution of low-dimensional projections in Gaussian RBF kernel spaces, which will allow for a control of the probability of false alarm error.

Using the notation in Section 5.1.1, we assume a sample X 1 , . . . , X n , X is available where X 1 , . . . , X n ∼ P are i.i.d. X -valued variables representing the inliers and X is a test point. Besides, we consider the Gaussian RBF kernel k γ defined as

k γ (x, x ) ∆ = exp -γd 2 (x, x )
for every x, x ∈ X , and parameterized by γ > 0.

Our method is twofold. In the first step, a subspace V n of H(k γ ) is randomly selected and defined as

V n = Span(h 1 , . . . , h p ) ,
where h = (h 1 , . . . , h p ) are p i.i.d. zero-mean Gaussian processes with covariance function Σ γ,n (•, •):

Σ γ,n (x, x ) ∆ = 1 n n i=1 k γ (X i , x)k γ (X i , x ) for every x, x ∈ X , that is the empirical estimator of Σ γ (x, x ) = E X 1 ∼P k γ (X 1 , x)k γ (X 1 , x ) .
In the second step, the embedded test point k γ (X, •) ∈ H(k γ ) is projected onto V n . The projection of k γ (X, •) onto V n is represented via the p-variate vector p V n (X) whose entries are the coordinates of the projection with respect to the basis h. Since h forms asymptotically an orthonormal basis of V n when γ → +∞ (see Lemma 4.3.1), p V n (X) can be simply defined as

p V n (X) ∆ = k γ (X, •), h 1 H(k γ ) . . . k γ (X, •), h p H(k γ ) = h 1 (X) . . . h p (X) .
Finally we are in a position to introduce the test statistic

S n (X) ∆ = γ s p V n (X) 2 , (5.2.3) where • is the Euclidean norm of R p and s > 0. When X = R d , we set s = d/2.
Whether X is an outlier or an inlier, S n (X) exhibits two distinct behaviours.

If H 0 is true (X is an inlier), Proposition 4.3.5 states that when n → +∞ and γ grows slowly to infinity compared to n , the distribution γ s/2 p V n (X) is close to that of a random variable sG where G is a N (0, I p ) Gaussian vector and s 2 is a copy of γ s Σ γ (X 1 , X 1 ) independent of G and where X 1 ∼ P , which defines a scale-mixture of (isotropic) Gaussians (SMG). Therefore the Continuous Mapping Theorem and the continuity of • 2 entail that S n (X) is well-approximated by a variable

s 2 Q 2 where Q 2 ∼ χ 2 (p) is independent of s. Note that by Lemma A.2.1, s 2 = γ s Σ γ (X 1 , X 1 )
γ→+∞ 1 so that the distribution of S n (X) is not asymptotically reduced to a Dirac distribution at 0hence the γ s term in (5.2.3).

If H 1 is true (X is an outlier), S n (X) converges almost surely to 0. To see this, note that since h 1 , . . . , h p have Σ γ,n (•, •) as a covariance function, then

V n ⊆ V = Span k γ (x, •) x ∈ supp(P ) P -almost surely .
On the other hand, since an outlier X lies outside the support of P by definition, then for any

x ∈ supp(P )

k γ (x, •), k γ (X, •) H(k γ ) = exp -γd 2 (x, X) = O(e -γd 2 (X) ) -→ γ→+∞ 0 ,
where d(X) = inf d(X, x) x ∈ supp(P ) denotes the distance between X and the support of P . It follows that k γ (X, •) is asymptotically orthogonal to the subspace V -hence also to the subspace

V n almost surely -when γ → +∞ in such a way that

S n (X) = γ s p V n (X) 2 = O P 1 (γ s e -2γd 2 (X) ) -→ γ→+∞ 0 .
Now we are in a position to decide whether X is an outlier or not. Let us introduce the p-value

PV(X) = P s 2 ,Q 2 S n (X) > s 2 Q 2 , (5.2.4)
where the probability is evaluated solely over s 2 and Q 2 . Given a confidence level α ∈ (0, 1), the null-hypothesis H 0 is rejected if and only if PV(X) < α.

Note that we did not consider renormalized projections pV n (X) = Σ -1/2 γ,n (X, X)p V n (X) whereas we saw in Chapter 4 that they converge weakly to a N (0, I p ) instead of a SMG, which would have yielded a simpler null-distribution for our statistic. The problem is that when X is an outlier, pV n (X) is no more guaranteed to converge almost surely to 0 unlike p V n (X). This is due to the fact that the renormalizer Σ 1/2 γ,n (X, X) tends faster to 0 when X is an outlier than otherwise. Indeed when X is an outlier, Σ γ,n (X, X) = (1/n) n i=1 k 2 γ (X, X i ) exp -2γd 2 (X) and since S n (X) = O(e -γd 2 (X) ) as said above, so that there is no guarantee that pV n (X) 2 = S n (X)/Σ γ,n (X, X) tends to 0 a.s. under the alternative.

Practical implementation

This section presents in detail how our OD method is set up in practice.

For every j = 1, . . . , p, h j can be written as

h j = (1/ √ n) n i=1 u (j) i k γ (X i , •)
where the variables {u (j) i } 1 i n,1 j p are i.i.d. N (0, 1) Gaussians. This leads to the following matrix formulation of p V n (X)

p V n (X) = n -1/2 Uk x , where U = [u (j) i ] j,i ∈ M p,n (R) and k x = (k γ (X 1 , X) . . . k γ (X n , X)) ∈ R n .
This way the test statistic S n (X) writes

S n (X) = γ s p V n (X) 2 = n -1 γ s k x U Uk x .
The next step consists in determining the p-value PV(X) of the test as defined in (5.2.4). Intro-

ducing F χ 2 (p) (t) = P(Q 2 > t) the cumulative distribution function (cdf) of the χ 2 (p) distribution, PV(X) alternatively reads PV(X) = E s 2 F χ 2 (p) S n (X) s 2 , (5.2.5)
where the mean is taken only over s 2 . However s 2 is a copy of γ s Σ γ (X 1 , X 1 ) (X 1 ∼ P ) while Σ γ and the distribution P of X are unknown, therefore PV(X) needs to be estimated. To this aim, we can consider for each i = 1, . . . , n the quantities ŝi = γ s Σ γ,-i (X i , X i ) where

Σ γ,-i (X i , X i ) = 1/(n -1) j i k 2 γ (X j , X i ) is an estimate of Σ γ (X i , X i ) (conditionally to X i ). This leads to the estimated p-value PV(X) PV(X) = 1 n n i=1 F χ 2 (p)       S n (X) ŝ2 i       .
However, the estimation of the p-value may be improved upon in terms of execution time. Actually, computing the quantities ŝi requires to access the whole Gram matrix

K = [k γ (X 1 , X j )] 1 i,j n
which involves the computation of its n(n -1)/2 entries, then for each i = 1, . . . , n calculating Σ γ,-i (X i , X i ) takes in the order of n operations. All in all, the computational cost is of order O(n 2 ). This quadratic time can be reduced to linear time through a low-rank approximation of K by the methods presented in Section 2.4. Such a method yields an approximation of K as

K UU with U ∈ M n,r (R) , (5.2.6)
where r is much smaller than n. Using this matrix factorization, each Σ γ,-i (X i , X i ) can be approached as follows

Σ γ,-i (X i , X i ) = 1 n -1 e i K 2 e i - 1 n -1 1 n -1 e i UWU e i - 1 n -1 , ( 5.2.7)
where W = U U ∈ M r (R) and (e 1 , . . . , e n ) is the canonical base of R n . W can be calculated beforehand in O(r 2 n) time. Then the computation of Σ γ,-i (X i , X i ) through (5.2.7) takes O(r 2 ) operations, hence a total computation time of order O(r 2 n) linear with respect to n. The influence of the choice of r is discussed later in Section 5.4.1.

Low-rank approximation of the Gram matrix

Two different ways of approximating the Gram matrix as in (5.2.6) can be used. The first one involves the Nyström method (Section 2.4.1). As a reminder, Nyström method consists in randomly picking a subset I of cardinality r uniformly over {I ⊆ {1, . . . , n} | card(I ) = r} and approaching K by

K K •,I K -1 I ,I K I ,• ,
where

K •,I = [K ij ] 1 i n,j∈I , K I ,I = [K ij ] i,j∈I and K I ,• = K •,I . In this case U = K •,I K -1/2 I ,I . Let [ŝ nyst i ] 2 = γ s Σ nyst γ,-i (X i , X i ) where Σ nyst γ,-i (X i , X i ) denotes the approximation of Σ γ,-i (X i , X i ) based on
the Nyström method. We define the corresponding estimated p-value as

P V nyst (X) = 1 n n i=1 F χ 2 (p)        S n (X) [ŝ nyst i ] 2        .
The second method to approximate K uses Random Kitchen Sinks (Section 2.4.2). To be able to apply this method, we need to assume that X = R d endowed with the usual Euclidean norm • . Under this assumption, k γ (x, x ) = exp -γ xx 2 corresponds to the Fourier transform of the N (0, 2γI d ) Gaussian measure and can be approached as in (5.2.6) with

U = r -1              cos(w 1 X 1 ) sin(w 1 X 1 ) . . . cos(w r X 1 ) sin(w r X 1 ) . . . . . . . . . . . . . . . cos(w 1 X n ) sin(w 1 X n ) . . . cos(w r X n ) sin(w r X n )              , where w 1 , . . . , w r ∼ N (0, 2γI d ) i.i.d. Let [ŝ RKS i ] 2 = γ s Σ RKS γ,-i (X i , X i )
where each Σ RKS γ,-i (X i , X i ) denotes the approximation of Σ γ,-i (X i , X i ) based on RKS. We define the corresponding estimated p-value as

P V RKS (X) = 1 n n i=1 F χ 2 (p)       S n (X) [ŝ RKS i ] 2       .

Theoretical analysis

When testing a point X, there are two ways of committing an error: either by labeling X as an outlier while it is not (false alarm error) or by accepting X as an inlier while it is an outlier (missed detection error). In the following, we call false alarm rate the probability of a false alarm error, and missed detection rate the probability of a missed detection rate. In the hypothesis testing terminology, the false alarm rate corresponds to a Type-I error and missed detection rate to a Type-II error. The aim of this section is to study theoretically how those two kinds of error are controlled for our OD method. In order not to overload theoretical developments, our analysis focuses on the version of our OD procedure that relies on the exact p-value PV(X).

False alarm rate

When X ∼ P , Theorem 4.3.5 in Chapter 4 provides an upper bound of the general form

∆ 2 γ s/2 p V n (X), sG C 1 γ -s/2 + C 2 γ s/2 n -1/2 , (5.3.8)
with high probability over h 1 , . . . , h p and X 1 , . . . , X n and where γ denotes the parameter of the kernel k γ . Here ∆(•, •) is some distance between distributions on R p and s > 0 is some quantity related to the distribution P -for instance s = D/2 when X = R D and d(•, •) is the Euclidean distance.

The bound in (5.3.8) shows that the weak convergence of the null-distribution of S n (X) to s 2 Q 2 holds when γ and n both grow to infinity and γ = o(n 1/s ). More precisely, the right hand side of (5.3.8) is minimized by setting every additive terms of the bound at the same rate n -1/4 , that is by setting γ = Cn 1/(2s) for some constant C > 0. However this does not tell much about the rate of convergence of the actual false alarm rate P(P V (X) < α | H 0 ) to the prescribed level of confidence α. Theorem 5.3.1 thereafter fills this gap and its proof can be found in Section 5.A.1.

Theorem 5.3.1. Let X ∼ P and assume there exists a bounded, continuous function A : X → R + and s > 0 such that

∀x ∈ supp(P ) , P X ∼P d 2 (x, X ) t ∼ A(x)t s , when t → 0 ,
where supp(P ) = {x ∈ X ∀ > 0, P X ∼P (d 2 (x, X ) < ) > 0} defines the support of P and A(x) = 0 for every x supp(P ). Also assume that A(•) is lower bounded by some a > 0 on its support.

Then for any α, δ ∈ (0, 1), the actual false alarm rate is bounded as follows

P P V (X) < α H 0 α + C p τ -2 α E X [A(X)] 2 log(1/δ)Γ (s + 1)γ s an         1 + κ log(1/δ)(2γ) s aΓ (s + 1)n         {1 + o γ (1)} + 2C p τ -2 α γ s δ , (5.3.9)
where τ α is the α-quantile of the null distribution sG with G ∼ N (0, I p ), κ is a numerical constant and C p only depends on p.

One could use the upper bound (5.3.9) as a proxy to control the actual false alarm rate by choosing a (near-)optimal γ in a closed form. The problem is that such a closed form expression would involve some terms in the upper bound which depend on quantities related to the inlier distribution P -as an example when X = R D and d(•, •) is the Euclidean distance, A(•) is related to the density function f of P . But as we mentioned earlier, OD methods are made to avoid the complete estimation of the density function of P . In Section 5.5, we address the problem of selecting γ by presenting a procedure based on grid search and a one-sample test statistic.

Missed detection rate

We call missed detection rate the probability of committing a missed detection error, which is denoted by

β n β n = P X P V (X) > α H 1 = P X S n (X) > τ α H 1 ,
where P X is taken with respect to X and τ α is the α-quantile of s 2 Q 2 . In the hypothesis testing terminology, out OD procedure is said to be consistent if β n → 0 when n grows to infinity.

Theorem 5.3.2 thereafter provides an upper bound for β n that entails the consistency of our method. The proof of Theorem 5.3.2 is provided in Section 5.A.2.

Theorem 5.3.2 (Missed detection rate). Let X ∼ P 1 P and assume the same assumptions concerning the distribution P as in Theorem 5.3.1.

Let F denote the cdf of d(X) = inf{ Xx : x ∈ supp(P )} with X ∼ P 1 .

Then for any δ ∈ np (np) 2 +1 , np , there exists an event with probability (over h) larger than 1 -(2np/δ) -np/2 e -(np-δ)/2 under which

β n F           s log (2γ) + log(n) + log 1 aΓ (s+1) + 2 log(1/α) p + 2 2 log(1/α) p + log(δ) + o γ,n (1) 2γ           , (5.3.10)
where α ∈ (0, 1) is the prescribed confidence level and o γ,n (1) -→ γ,n→+∞ 0.

Note that Theorem 5.3.2 only assumes that the tested point X was generated by an alternative distribution P 1 different than P but not necessarily with a support distinct from the support of P . However the upper bound in (5.3.10) decreases to 0 only if the alternative H 1 holds, that is supp(P 1 ) ∩ supp(P ) is negligible. In this case the distance d(X) between X and supp(P ) is P 1 -almost surely non-null hence F(0) = 0. On the other hand, when the sample size n gets larger, the hyperparameter γ is "allowed" to increase as well while still controlling the false alarm rate at level α according to the analysis done in Section 5.3.1. For instance, we saw that setting γ ∝ n 1/(2s) leads to an asymptotic control of the false alarm rate. With this choice of γ and when n grows to infinity, the upper bound in (5.3.10) entails that β n decreases to 0 at the rate O(F(log(n)n -1/(2s) )) which tends to 0 since F(0) = 0. This way Theorem 5.3.2 guarantees that our OD procedure is consistent against H 1 .

Let us discuss the influence of some key quantities on the upper bound (5.3.10). First of all, since F is a cdf hence an increasing function, the upper bound increases when the lower bound a of A(•) gets smaller. To understand how a small a worsens the performance of our OD method, it may be helpful to resort to the multivariate case X = R d with d(•, •) as the Euclidean distance in which case A(x) ∝ f (x) and f (•) is the density of P (Proposition 4.3.2 in Chapter 4).

In this case, a must be interpreted as an indicator of the values of f on small density regions.

Therefore depending on the form of f , a small a may be influential in two different ways. The first case is when the distribution P is highly unbalanced, that is there is a large discrepancy between highest and smallest values of f . Thus a small a means that P is mostly concentrated on a smaller subset of its support. In this case our OD algorithm is likely to mistake this subset of supp(P ) as the entire support. In order to keep the false alarm rate at the prescribed level α, the OD procedure is compelled to compensate by erroneously "completing" this highly concentrated part of supp(P ) with a "dumb" subset of X outside of supp(P ), hence a larger missed detection The same experiments were conducted when using Nyström and Random Kitchen Sinks approximations as in Section 5.2.2. For Nyström method, false alarm and missed detection rates were measured for the same range of values of γ and p as above, in a first time for a small rank r = 50 and in the second time for a larger rank r = 400. The same was done for RKS with r = 50 and r = 300. The results are shown in Figures 5.4 to 5.7. When r is large enough (r = 400 for Nyström and r = 300 for RKS), performances comparable to those without Gram matrix approximation are observed. For Nyström with r = 400, choosing 25 γ 50 and p = 2 yields false alarm rates of about 0.6 and a missed detection rate below 0.01. Similarly for RKS with r = 300, setting γ = 50 and p = 2 yields a false alarm rate of 0.06 and a missed detection rate of 0.009. As long as r is large enough, the parameters γ and p show similar effects as previously, that is increasing γ and/or p decreases the missed detection rate and increases the false error rate. In particular the effect of a larger p on the false error rate seems to be more visible when Nyström/RKS approximation are used. This may be due to the fact that such an approximation implicitly consists in projecting embedded data onto a low-dimensional subspace in the RKHS (for instance, the subspace spanned by a small subsample of the embedded dataset for Nyström method). Therefore before projecting onto V n , the projected points does not lie in a high-dimensional RKHS but in a low-dimensional subspace of this RKHS, and this low dimensionality may prevent the weak convergence to a SMG from occurring hence the less controlled false alarm rates when p is not negligible compared to r.

All in all when it comes to calibrating γ and p, only false alarm rate control matters, since γ and p can be set as large as wanted when only considering missed detection error. In other words, optimizing the parameters is only constrained by false alarm rate control. This is advantageous since false alarm rate is related to the inlier distribution which is accessible as many inliers are available unlike outliers. A tuning procedure is proposed in details later in Section 5.5.

Comparison with other methods

We compare our OD method with the two existing OD methods presented in Section 5.1.2, that are oc-SVM and kernel PCA. These two latter methods use the same Gaussian RBF kernel

Breast cancer dataset

False positives

True positives

Our method oc-SVM Kernel PCA Figure 5.8 -ROC curves for our OD method (blue), oc-SVM (red) and kernel PCA (green), tested on synthetic "Spiral" dataset (left plot) and real-life "Cancer" dataset (right plot).

with parameter γ > 0 as ours. This comparison is done according to two criteria: detection performance and execution time.

Detection performance

To assess the detection performances of the three competing methods, we applied them to two different datasets:

• Synthetic dataset: it is the same "spiral" dataset as the one used in Section 5.4.1. The training set contains n = 500 inliers, and the tested set contains 200 inliers and 200 outliers.

• Real-life dataset: it is the "cancer" dataset from the UCI machine learning repository1 . It consists of 569 instances split into two groups, one with 357 instances labeled "benign" -these will be the inliers -and 112 instances labeled "malignant" -the outliers. Each instance is described by 10 covariates, and each covariate has been renormalized to have unit variance. The training set contains n = 200 instances, which yields a tested set with 157 inliers and 112 outliers.

For each method and dataset, a ROC curve is produced. For our method and kernel PCA, it consists in considering a varying threshold τ > 0 and rejects each tested observation X if and only if S n (X) < τ or RE(X) > τ. For the oc-SVM, the parameter ν is set at several values spanning the interval (0, 1]. Varying τ or ν yields couples of false positives/true positives pairs that constitutes the ROC curve. Our method Our method + Nystrom Our method + RKS One Class SVM Kernel PCA Figure 5.9 -Execution times (in seconds) with respect to n of our OD method (blue, solid), our method combined with Nyström approximation (cyan, dashed), our methods combined with Random Kitchen Sinks (dark blue, dotted), oc-SVM (red, dot-dash) and kernel PCA (green, long-dashed).

Parameter selection

This section presents a procedure to select the parameters γ and p of our OD method in practice.

In Section 5.4.1, it was observed that optimal parameters are mostly linked to false alarm rate control. Thus the parameters can be chosen solely on the basis of the inliers sample X 1 , . . . , X n ∼ P .

Let I = {1, . . . , n 0 } be indexes of a subsample of X 1 , . . . , X n with n 0 < n. For every i ∈ I , let s i = S n-n 0 (X i ) be the OD statistic applied to X i when using the subsample {X j } j I as the inliers sample. Our selection procedure relies on a Kolmogorov-Smirnov-like statistic KS(γ, p) that compares the empirical cumulative distribution function (cdf) of s (1) , . . . , s (n 0 ) to the cdf of ŝ2 Q 2 where ŝ2 follows the empirical distribution of {ŝ i } 1 i n as defined in Section 5.2.2 and is independent of Q 2 ∼ χ 2 (p). Namely, KS(γ, p) is defined as

KS(γ, p) = sup t 0 1 n 0 n 0 i=1 1 1 {s (i) t} - 1 n n j=1 F χ 2 (p) t ŝj = n 0 max i=0 i n 0 - 1 n n j=1 F χ 2 (p) s (i+1) ŝj ∨ i n 0 - 1 n n j=1 F χ 2 (p) s (i) ŝj ,
where F χ 2 (p) denotes the cdf of the χ 2 (p) distribution and s (0) s (1) . . . s (n 0 +1) are the ordered s 1 , . . . , s n with s (0) = 0 and s (n 0 +1) = +∞.

Given a range of parameter values γ 1 < γ 2 < . . . < γ m and p 1 < p 2 < . . . < p q , we compute every quantity KS(γ j , p l ) for every 1 j m and 1 l q. For each fixed p l , the plot {KS(γ j , p l )} 1 j m missed detection rate around 0.009.

5.A Technical details

5.A.1 Proof of Theorem 5.3.1

Let α ∈ (0, 1) be the prescribed level of confidence and τ α the α-quantile of the distribution of

s 2 Q 2 where Q 2 ∼ χ 2 (p).
The goal is to find an upper bound for the gap between α and the actual false alarm rate P(S n (X) < τ α ). Introducing the notation B α that denotes the ball of R p centered around 0 and of radius τ 1/2 and G ∼ N (0, I p ),

P(S n (X) < τ α ) = α + P(S n (X) < τ α ) -α = α + P(γ s/2 p V n (X) ∈ B α ) -P(sG ∈ B α ) = α + E X,h,s,G 1 1 B α (γ s/2 p V n (X)) -1 1 B α (sG) .
Standard calculations entail that the Fourier transform of 1

1 B α (•) is (2π) p/2 1 1 B α (x)e -iξ x = τ p/2 α ξ -p/2 J p/2 (τ α ξ ) ∆ = g(ξ) for every ξ ∈ R p , where J p/2 (•) denotes the Bessel function of the first kind J p/2 (t) = (t/2) p/2 √ piΓ ((p+1)/2) π 0 sin p (y)e -it cos(y) dy. Since g(ξ) is an integrable function of ξ, 1 1 B α (•)
is the inverse Fourier transform of g that is 1 1 B α (x) = (τ α /2π) p/2 ξ -p/2 J p/2 (τ α ξ )e ix ξ dξ.

It follows

P(S n (X) < τ α ) = α + E X,X 1 ,...,X n ,h,s,G (τ α /2π) p/2 ξ -p/2 J p/2 (τ α ξ ) e iγ s/2 ξ p V n (X) -e isξ G dξ = Fubini α + (τ α /2π) p/2 ξ -p/2 J p/2 (τ α ξ ) E X,X 1 ,...,X n ,h e iγ s/2 ξ p V n (X) -E s,G e isξ G dξ (5.A.11) = α + (τ α /2π) p/2 ξ -p/2 J p/2 (τ α ξ ) E X,X 1 ,...,X n e -γ s 2 Σ γ,n (X,X) ξ 2 -E X e -γ s 2 Σ γ (X,X) ξ 2 dξ α + 1 2 (τ α /2π) p/2 ξ -p/2+2 |J p/2 (τ α ξ )|dξ E X,X 1 ,...,X n γ s |Σ γ,n (X, X) -Σ γ (X, X)| , since x → e -x/2 is a 1/2-Lipschitz function, which yields more simply P(S n (X) < τ α ) α + C p τ -2 α E X,X 1 ,...,X n γ s |Σ γ,n (X, X) -Σ γ (X, X)| , (5.A.12)
where C p = (1/2)(2π) -p/2 ξ -p/2+2 |J p/2 ( ξ )|dξ after a change of variable was made.

Fixing X, Lemma A.2.4 provides the following inequality that holds on the event Ω X with probability larger than 1δ over the sample X 1 , . . . , X n

γ s |Σ γ,n (X, X) -Σ γ (X, X)| γ s Σ γ (X, X) 2 log(1/δ)γ s aΓ (s + 1)n         1 + κ log(1/δ)(2γ) s aΓ (s + 1)n [1 + o γ (1)]        
, which combined with Lemma A.2.1 yields

γ s |Σ γ,n (X, X) -Σ γ (X, X)| A(X) 2 log(1/δ)Γ (s + 1)γ s an         1 + κ log(1/δ)(2γ) s aΓ (s + 1)n         {1 + o γ (1)} .
(5.A.13) Plugging (5.A.13) into (5.A.12) and conditioning with respect to Ω X leads to

P(S n (X) < τ α ) α + C p τ -2 α E X E X 1 ,...,X n γ s |Σ γ,n (X, X) -Σ γ (X, X)| Ω X + 2γ s δ α + C p τ -2 α E X [A(X)] 2 log(1/δ)Γ (s + 1)γ s an         1 + κ log(1/δ)(2γ) s aΓ (s + 1)n         {1 + o γ (1)} + 2C p τ -2 α γ s δ , (5.A.14)
where we used the inequality |Σ γ,n (X, X) -Σ γ (X, X)| 2. This leads to the result of the lemma.

Note that we could have tried to plug directly the upper bound of Theorem 4.3.5 into (5.A.11) in the following way

P(S n (X) < τ α ) = α + (τ α /2π) p/2 ξ -p/2 J p/2 (τ α ξ ) E X,h e iγ s/2 ξ p V n (X) -E s,G e isξ G dξ α + (τ α /2π) p/2 ξ -p/2 |J p/2 (τ α ξ )| φ γ s p V n (X) (ξ) -φ sG (ξ) dξ α + (τ α /2π) p/2 ξ -p |J p/2 (τ α ξ )| 2 φ γ s p V n (X) (ξ) -φ sG (ξ) 2 dξ 1/2 = α + C∆(γ s p V n (X), sG) ,
where φ U denotes the characteristic function for any random variable U and C is some constant.

Theorem 4.3.5 asserts an inequality of the form ∆ 2 (γ s p V n (X), sG) C 1 γ -s/2 + C 2 γ s/2 n -1/2 for some constants C 1 and C 2 . However setting γ ∝ n 1/(2s) to minimize this upper bound would entail a convergence of the actual false alarm rate to α of order O(n -1/8 ) which is very slow. On the other hand, (5.A.14) yields for γ ∝ n 1/(2s) a convergence of order O( log(n)n -1/4 ).

5.A.2 Proof of Theorem 5.3.2

We must derive an upper bound for the probability P P V (X) > α H 1 which can be also expressed as P S n (X) > τ α H 1 where τ α satisfies P(s 2 Q 2 < τ α ) = α.

For every j = 1, . . . , p, let

h j = (1/ √ n) n i=1 u (j) k k γ (X i , •) where u (j) 1 , . . . , u (j)
n ∼ N (0, 1) independently so that h 1 , . . . , h p are independent zero-mean Gaussian processes with covariance function

Σ γ,n (•, •) = (1/n) n i=1 k γ (X i , •)k γ (X i , •). Let d(x) = inf{
xy : y ∈ supp(P )} denotes the distance between x and the support of P . Then,

S n (X) = γ s p V n (X) 2 = γ s p j=1 h j , k γ (X, •) 2 H(k γ ) = (γ s /n) p j=1 n k,l=1 u (j) k u (j) l e -γ(d 2 (X,X i )+d 2 (X,X j )) (γ s /n) p j=1 n k,l=1 |u (j) k ||u (j) l |e -2γd 2 (X) γ s p j=1 n l=1 [u (j) l ] 2 e -2γd 2 (X) γ s np[1 + n ]e -2γd 2 (X) ,
where n = o P (1) as n → +∞ because of the Law of Large Numbers.

Let us bound 1 + n with high probability. For any δ > 0 and λ ∈ (0, np/2), exponential Markov's inequality yields

P (1 + > δ) = P V 2 ∼χ 2 (np) V 2 > npδ E V 2 e λV 2 e -λnpδ = (1 -2λnp) -np/2 e -λnpδ ,
which is minimized at λ = 1/(2δ) -1/(2np). Therefore for any δ ∈ np (np) 2 +1 , np , the event B δ defined by

B δ = {1 + δ} ,
holds with probability larger than 1 -(2np/δ) -np/2 e -(np-δ)/2 .

From now on we assume that the event B δ holds true. Therefore

P (S n (X) > τ α ) P d 2 (X) < log(γ s npδ) + log(1/τ α ) 2γ .
By Lemma A.2.1 and writing a = inf{A(x) : x ∈ supp(P )} > 0 and X 1 ∼ P ,

s 2 = γ s Σ γ (X 1 , X 1 ) = A(X 1 )Γ (s + 1) 2 s [1 + A -1 (X 1 )o γ (1)] aΓ (s + 1) 2 s [1 + o γ (1)] ,
which leads to

τ α aΓ (s + 1)F -1 χ 2 (p) (α) 2 s [1 + o γ (1)] ,
where F -1 χ 2 (p) denotes the quantile function of the χ 2 (p) distribution. Therefore introducing F as the c.d.f. of d 2 (X)

P (S n (X) > τ α ) E n F                log(npδ) + s log (2γ) + log 1 aΓ (s+1) + log 1 F -1 χ 2 (p) (α) + o γ (1) 2γ                .
By the exponential Markov's inequality for any λ > 0

α = F χ 2 (p) (F -1 χ 2 (p) (α)) e λF -1 χ 2 (p) (α) E V 2 ∼χ 2 (p) e -λV 2 = e λF -1 χ 2 (p) (α)
(1 + 2λ) -p/2 , hence setting λ such that (1/2) log(1 + 2λ) = log(1/α)/p + for some > 0 log

        p F -1 χ 2 (p) (α)         log pλ (p/2) log(1 + 2λ) -log(1/α) = log e (2/p) log(1/α)+2 -1 2 (2/p) log(1/α) + 2 + log 1 + log(1/α) p (2 + 1/ ) log(1/α) p + 2 = 2 log(1/α) p + 2 2 log(1/α) p ,
where we used the inequalities e x -1 xe x and log(1 + x) x and set = log(1/α)/(2p) to minimize the bound.

This leads to

P (S n (X) > τ α ) F           log(n) +

Introduction

Non-vectorial data such as DNA sequences or pictures often require a positive definite kernel k so that further analysis is then carried out in the associated reproducing kernel Hilbert space (RKHS) H(k) where data are embedded into through the feature map x → k(x, .). For many applications, kernelized data -or even more general high-dimensional data -are assumed to fit a specific type of distribution, often a Gaussian distribution. For instance supervised and unsupervised classification are performed in [START_REF] Bouveyron | Kernel discriminant analysis and clustering with parsimonious Gaussian process models[END_REF] by modeling each class as a Gaussian process. In [START_REF] Roth | Kernel Fisher Discriminants for Outlier Detection[END_REF], outliers are detected by modelling embedded data as a Gaussian random variable and by removing points lying in the tails of that Gaussian distribution. This key assumption is also made in [START_REF] Srivastava | A Two-Sample Test in High Dimensional Data[END_REF] where a mean equality test is used in high-dimensional setting. Moreover, Principal Component Analysis (PCA) and its kernelized version Kernel PCA [START_REF] Schölkopf | Nonlinear Component Analysis as a Kernel Eigenvalue Problem[END_REF] are known to be optimal for Gaussian data as these methods rely on second-order statistics (covariance). Besides, a Gaussian assumption allows to use Expectation-Minimization (EM) techniques to speed up PCA [START_REF] Roweis | EM Algorithms for PCA and SPCA[END_REF].

Depending on the (finite or infinite dimensional) structure of the RKHS, Cramer-von-Misestype normality tests can be applied, such as Mardia's skewness test [START_REF] Mardia | Measures of Multivariate Skewness and Kurtosis with Applications[END_REF], the Henze-Zirkler test [START_REF] Henze | A Class of Invariant and Consistent Tests for Multivariate Normality[END_REF] and the Energy-distance test [START_REF] Szekely | A New Test For Multivariate Normality[END_REF]. However these tests become less powerful as dimension increases (see Table 3 in [START_REF] Szekely | A New Test For Multivariate Normality[END_REF]). An alternative approach consists in randomly projecting high-dimensional objects on one-dimensional directions and then applying univariate test on a few randomly chosen marginals [START_REF] Cuesta-Albertos | The Random Projection Method in Goodness of Fit for Functional Data[END_REF]. This projection pursuit method has the advantage of being suited to high-dimensional settings. On the other hand, such approaches also suffer a lack of power because of the limited number of considered directions (see Section 4.2 in [START_REF] Cuesta-Albertos | The Random Projection Method in Goodness of Fit for Functional Data[END_REF]).

[Gre+07a] introduced the Maximum Mean Discrepancy (MMD) which quantifies the gap between two distributions through distances between two corresponding elements in an RKHS.

The MMD approach has been used for two-sample testing [START_REF] Gretton | A Kernel Method for the Two-Sample-Problem[END_REF] and for independence testing (Hilbert Space Independence Criterion, [START_REF] Gretton | A Kernel Statistical Test of Independence[END_REF]). However to the best of our knowledge, MMD has not been applied in a one-sample goodness-of-fit testing framework.

Our main contribution presented in this chapter is to devise a one-sample statistical test of normality for data taking values in an RKHS, by means of the MMD principle. This implies that we will consider a second kernel k defined on H(k) and handle elements in the second RKHS H( k) to build our MMD-based test. However it must be remarked that our test also suits the case where H(k) is replaced by a more general Hilbert space. Therefore to avoid confusion due to this two levels of RKHS, we present our test as dealing with data in a general Hilbert space.

Our test features two possible applications: testing the normality of the data but also testing parameters (mean and covariance) if data are assumed Gaussian. The latter application encompasses many current methods that assume normality to make inferences on parameters, for instance to test the nullity of the mean [START_REF] Srivastava | A Two-Sample Test in High Dimensional Data[END_REF] or to assess the sparse structure of the covariance [SW03; BT11].

Once the test statistic is defined, a critical value is needed to decide whether to accept or reject the Gaussian hypothesis. In goodness-of-fit testing, this critical value is typically estimated by parametric bootstrap. Unfortunately, parametric bootstrap requires parameters to be computed several times, hence heavy computational costs (i.e. diagonalization of covariance matrices). Our test bypasses the recomputation of parameters by implementing a faster version of parametric bootstrap. Following the idea of [START_REF] Kojadinovic | Goodness-of-fit Testing Based On A Weighted Bootstrap: A Fast Large-Sample Alternative to the Parametric Bootstrap[END_REF], this fast bootstrap method "linearizes" the test statistic through a Fréchet derivative approximation and thus can estimate the critical value by a weighted bootstrap (in the sense of [START_REF] Burke | Multivariate tests-of-fit and uniform confidence bands using a weighted bootstrap[END_REF]) which is computationally more efficient. Furthermore our version of this bootstrap method allows parameters estimators that are not explicitly "linear" (i.e.

that consist of a sum of independent terms) and that take values in possible infinite-dimensional Hilbert spaces. where the dimension of the kept subspace is a crucial problem.

Here is the outline of the chapter. Section 6.2 sets our framework and Section 6.3 recalls the notion of MMD and how it is used for our one-sample test. The new normality test is described in Section 6.4, while both its theoretical and empirical performances are detailed in Section 6.5 in terms of control of Type-I and Type-II errors. A sequential procedure to select covariance rank is presented in Section 6.6.

Framework

Let (H, A) be a measurable space, and Y 1 , . . . , Y n ∈ H denote a sample of independent and identically distributed (i.i.d.) random variables drawn from an unknown distribution P ∈ P , where P is a set of distributions defined on A.

In our framework, H is a separable Hilbert space endowed with a dot product . , . More precisely, the tested hypothesis is that Y i follows a Gaussian distribution N (m 0 , Σ 0 ), where (m 0 , Σ 0 ) ∈ Θ 0 and Θ 0 is a subset of the parameter space Θ. 1 Following [START_REF] Lehmann | Testing Statistical hypotheses[END_REF], let us define the null hypothesis H 0 : P ∈ P 0 , and the alternative hypothesis H 1 : P P \ P 0 where the subset of null-hypotheses P 0 ⊆ P is

P 0 = {N (m 0 , Σ 0 ) | (m 0 , Σ 0 ) ∈ Θ 0 } .
The purpose of a statistical test T (Y 1 , . . . , Y n ) of H 0 against H 1 is to distinguish between the null (H 0 ) and the alternative (H 1 ) hypotheses. It requires two elements: a statistic n ∆2 (which 1 The parameter space Θ is endowed with the dot product (m, Σ), (m , Σ ) Θ = m, m H + Σ, Σ HS(H) , where HS(H) is the space of Hilbert-Schmidt (finite trace) operators H → H and Σ, Σ HS(H) = i≥1 Σe i , Σ e i H for any complete orthonormal basis (e i ) i≥1 of H. Therefore, for any θ ∈ Θ, the tensor product θ ⊗2 is defined as the operator Θ → Θ, θ → θ, θ Θ θ. For any θ ∈ Θ and h ∈ H( k), the tensor product h ⊗ θ is the operator

Θ → H( k), θ → θ, θ Θ h.
we define in Section 6.4.1) that measures the gap between the empirical distribution of the data and the considered family of normal distributions P 0 , and a rejection region R α (at a level of confidence 0 < α < 1). H 0 is accepted if and only if n ∆2 R α . The rejection region is determined by the distribution of n ∆2 under the null-hypothesis such that the probability of wrongly rejecting H 0 (Type-I error) is controlled by α.

Characteristic kernels on a Hilbert space

Within our framework the goal is to compare P the true distribution of the data with a Gaussian distribution P 0 = N (m 0 , Σ 0 ) for some (m 0 , Σ 0 ) ∈ Θ 0 . To achieve this goal, we rely on the notion of Maximum Mean Discrepancy (MMD) introduced previously in Section 2.3.3. Given a characteristic kernel k on H, we consider the following Hilbert space embedding of distributions of

H μ : M + 1, k (H) → H( k), P → μP = k(h, •)P (dh) , where M + 1, k (H) ⊆ M +
1 (H) where M + 1 (H) denotes the set of probability measures on H and M + 1, k (H) is the subset of probability measures P on H for which μP is well defined.

Hence the quantity of interest is

∆ 2 = μP -μP 0 2 H( k) . (6.3.1)
For the sake of simplicity, we use the notation

μN (m,Σ) = N [m, Σ]
to denote the Hilbert space embedding of a Gaussian distribution.

It remains to choose a kernel k defined on the Hilbert space H that is characteristic. Several criteria for a kernel to be characteristic have been investigated [Fuk+09; Sri+10; CS10; SFL11]. However most of these criteria are relevant for input spaces such as compact topological spaces, R d with a finite d ∈ N * or more generally locally compact Haussdorf spaces. The only exception is the case of integrally strictly positive definite kernels2 (see Theorem 7 in [START_REF] Sriperumbudur | Hilbert Space Embeddings and Metrics on Probability Measures[END_REF]) defined on a general topological space, but proving that a kernel is integrally strictly positive definite may be a difficult task.

In the following, we introduce two examples of kernels defined on the Hilbert space H and show that they are characteristic. Firstly, we consider the exponential kernel ∀x, y ∈ H, k(x, y) = exp( x, y H ) .

Secondly, we cover the case of the Gaussian RBF kernel ∀x, y ∈ H, k(x, y) = exp -γ xy 2 H , parameterized by γ.

Proposition 6.3.1 (Exponential kernel). The exponential kernel k(x, y) = exp( x, y H ) defined on a separable Hilbert space H is characteristic (for the class of probability measures P for which μP is well defined).

Proposition 6.3.1. Let C > 1. Cauchy-Schwarz's inequality entails

C μP -μQ 2 H( k) = sup g∈H( k), g H( k) 1 C g, μP -μQ H( k) = sup g∈H( k), g H( k) C E X∼P g(X) -E Y ∼Q g(Y ) sup w H =1, 0 t √ 2 log(C) E X∼P e t X,w H -E Y ∼Q e t Y ,w H ,
where we restricted ourselves to functions g of the form g = e t w,• H to get the last inequality.

Therefore if ∆(P , Q) = 0 then E X∼P e t X,w H -E Y ∼Q e t Y ,w H = 0 for every w H = 1 and 0 t 2 log(C). It follows that every marginals X, w H and Y , w H follow the same univariate distribution. Finally since H is a separable Hilbert space, the Cramer-Wald theorem ([Cue+06],

Proposition 2.1) entails that P = Q.

Proposition 6.3.2 (Gaussian kernel). Let k(x, y) = exp(-γ xy 2 H ) be a Gaussian kernel with γ > 0 and assume that H is a separable Hilbert space. Then k is a characteristic kernel.

Proof of Proposition 6.3.2. Since H is assumed separable, there exists an orthonormal Hilbertian basis (e i ) i 1 of H. Consider some operator T : H → H defined by T = i 1 ϕ i e ⊗2 i where i 1 ϕ i < +∞ and ϕ i > 0 for every i 1. Then there exists a zero-mean Gaussian process G with covariance operator 2γT that is well defined in H (since 2γT has a finite trace). The first step of the proof consists in proving that the "proxy" kernel kT (x, y) = exp(-γ T 1/2 (xy) 2 H ) defined on H is characteristic -here T 1/2 = i 1 ϕ 1/2 i e ⊗2 i . Following Theorem 7 from [START_REF] Sriperumbudur | Hilbert Space Embeddings and Metrics on Probability Measures[END_REF], it suffices to prove that kT is integrally strictly positive definite, that is kT (x, y)ν(dx)ν(dy) > 0 for every non-zero signed Borel measure ν on H. Let ν be such a measure, kT (x, y)ν(dx)ν(dy) = E G e i x-y,G H ν(dx)ν(dy)

= E G |ϕ ν (G)| 2 0 (6.3.2)
where ϕ ν (G) = e i x,G H ν(dx) denotes the Fourier transform of the measure ν.

Algorithm 1 Kernel Normality Test procedure Input: Y 1 , . . . , Y n ∈ H, k : H × H → R (kernel) and 0 < α < 1 (test level).

1. Compute K = Y i , Y j i,j (Gram matrix).

2. Compute n ∆2 (test statistic) from (6.4.3) that depends on K and k (Section 6.4.1)

3. (a) Draw B (approximate) independent copies of n ∆2 under H 0 by fast parametric bootstrap (Section 6.4.2). (b) Compute qα,n (1α quantile of n ∆2 under H 0 ) from these replications.

Output: Reject H 0 if n ∆2 > qα,n , and accept otherwise.

If (6.3.2) is null, then ϕ G is null almost everywhere (with respect to the distribution of G).

Since the eigenvalues i of T are non-null, the Gaussian distribution of G is supported on the entire H, therefore ν is a null measure which leads to a contradiction. Thus kT is integrally strictly positive definite, hence characteristic.

In the second step, we relate the embedding μT corresponding to kT to the embedding µ corresponding to the Gaussian kernel k(x, y) = exp(-γ xy 2 H ). Note that for every probability measure P on H,

μT [P ](•) = E X∼P kT (X, •) = E X∼P k(T 1/2 X, T 1/2 (•)) = μ[T 1/2 P ](T 1/2 (•)) , that is μT = A o μ o B where A : f ∈ H( k) → f oT 1/2
and B : P → T 1/2 P where T 1/2 P is the probability measure of a random variable T 1/2 X with X ∼ P . Since kT is characteristic, μT is an injective map, which entails that μ is also injective and that k is characteristic.

Kernel normality test

This section introduces our one-sample test for normality based on the quantity (6.3.1). As said in Section 6.2, we test the null-hypothesis H 0 : P ∈ {N (m 0 , Σ 0 ) | (m 0 , Σ 0 ) ∈ Θ 0 } where Θ 0 is a subset of the parameter space. Therefore our procedure may be used as test for normality or a test on parameter if data are assumed Gaussian. The test procedure is summed up in Algorithm 1.

Test statistic

As in [START_REF] Gretton | A Kernel Method for the Two-Sample-Problem[END_REF], ∆ 2 can be estimated by replacing μP with the sample mean μP = μ P = (1/n) which can be written by expanding the square of the norm and using the reproducing property of H( k) as follows Before stating Propositions 6.4.2 and 6.4.3, the following notation is introduced. For a symmetric operator L : H → H with eigenexpansion L = r≥1 λ r Ψ ⊗2 , its determinant is denoted |L| = r≥1 λ r . For any q ∈ R, the operator L q is defined as L q = r≥1 λ q r 1 1 {λ r >0} Ψ ⊗2 r .

∆2 = 1 n 2 n i,j=1 k(Y i , Y j ) - 2 n n i=1 N [ m, Σ](Y i ) + N [ m, Σ] 2 H( k) . ( 6
H( k) = E k(Y , Y ) -E k(Y , Y ) ≤ E k(Y , Y ) + E 2 k(Y , Y ) < +∞.
Proposition 6.4.2. (Gaussian kernel case) Let k(., .) = exp(-σ . -. 2 H ) where σ > 0. Then,

N [ m, Σ](.) = |I + 2σ Σ| -1/2 exp -σ ||(I + 2σ Σ) -1/2 (. -m)|| 2 H , N [ m, Σ] 2 H( k) = |I + 4σ Σ| -1/2 .
Proposition 6.4.3. (Exponential kernel case) Let k(., .) = exp( ., . H ). Assume that the largest eigenvalue of Σ is smaller than 1. Then,

N [ m, Σ](.) = exp m, . H + 1 2 Σ., . H , ||N [ m, Σ]|| 2 = |I -Σ2 | -1/2 exp (I -Σ2 ) -1/2 m 2 H .
The proofs of Propositions 6.4.2 and 6.4.3 are provided in Appendix 6.B.1.

For most estimators ( m, Σ), the quantities provided in Propositions 6.4.3 and 6.4.2 are computable via the Gram matrix K = Y i , Y j H 1≤i,j≤n . For instance, assume that ( m, Σ) are the classical estimators ( m, Σ) where m = (1/n) n i=1 Y i and Σ = (1/n) n i=1 (Y i -m) ⊗2 . Let I n and J n be respectively the n×n identity matrix and the n×n matrix whose all entries equal 1, H = I n -(1/n)J n and K c = HKH be the centered Gram matrix. Then for any ∈ R,

I + Σ = det I n + n K c ,
where det(.) denotes the determinant of a matrix and

(I + Σ) -1/2 Y i 2 H = (I n + n K c ) -1 i,i
, where [.] ii denotes the entry in the i-th row and the i-th column of a matrix.

Estimation of the critical value

Designing a test with confidence level 0 < α < 1 requires to compute the 1α quantile of the n ∆2 distribution under H 0 denoted by q α,n . Thus q α,n serves as a critical value to decide whether the test statistic ∆2 is significantly close to 0 or not, so that the probability of wrongly rejecting H 0 (Type-I error) is at most α.

Classical parametric bootstrap

In the case of a goodness-of-fit test, a usual way of estimating q α,n is to perform a parametric where U and U are i.i.d. random variables. In a nutshell, (6.4.4) is approximately an independent copy of the test statistic n ∆2 (under H 0 ). Therefore B replications [n ∆2 ] b can be used to estimate the 1α quantile q α,n of n ∆2 under the null-hypothesis.

However, this approach suffers heavy computational costs. In particular, each bootstrap replication involves the estimators ( mb , Σb ). In our case, this leads to compute the eigendecomposition of the B Gram matrices

K b = [ Y b i , Y b j ] i,j
of size n × n hence a complexity of order O(Bn 3 ).

Fast parametric bootstrap

This computational limitation is alleviated by means of another strategy described in [START_REF] Kojadinovic | Goodness-of-fit Testing Based On A Weighted Bootstrap: A Fast Large-Sample Alternative to the Parametric Bootstrap[END_REF].

N : Θ → H( k), (m, Σ) → N [m, Σ] ,
our bootstrap method relies on the following approximation

√ n μP -N [ m, Σ] √ n               μP -N [m 0 , Σ 0 ] = μP under H 0 -D (m 0 ,Σ 0 ) N [ m -m 0 , Σ -Σ 0 ]               1 √ n n i=1 [ k(Y i , .) -μP ] -D (m 0 ,Σ 0 ) N [Y i -m 0 , (Y i -m 0 ) ⊗2 -Σ 0 ] . (6.4.5)
Since (6.4.5) consists of a sum of centered independent terms (under H 0 ), it is possible to generate approximate independent copies of this sum via weighted bootstrap [START_REF] Burke | Multivariate tests-of-fit and uniform confidence bands using a weighted bootstrap[END_REF]. Given Z b 1 , . . . , Z b n i.i.d. real random variables of mean zero and unit variance and Zb their empirical mean, a bootstrap replication of (6.4.5) is given by

1 √ n n i=1 (Z b i -Zb ) k(Y i , .) -D (m 0 ,Σ 0 ) N [Y i , (Y i -m 0 ) ⊗2 ] . (6.4.6)
Taking the square of the norm of (6.4.6) in H( k) and replacing the unknown true parameters m 0 and Σ 0 by their estimators m and Σ yields the bootstrap replication

[n ∆2 ] b f ast of n ∆2 [n ∆2 ] b f ast ∆ = √ n μb P -D ( m, Σ) N [ mb , Σb ] 2 H( k)
, (6.4.7)

where

μb P = (1/n) n i=1 (Z b i -Zb ) k(Y i , .) , mb = (1/n) n i=1 (Z b i -Zb )Y i , Σb = (1/n) n i=1 (Z b i -Zb )(Y i -mb ) ⊗2 .
Therefore this approach avoids the re-computation of parameters for each bootstrap replication, hence a computational cost of order O(Bn 2 ) instead of O(Bn 3 ). This is illustrated empirically in the right half of Figure 6.1.

Fast parametric bootstrap for general parameter estimators

The bootstrap method proposed by [START_REF] Kojadinovic | Goodness-of-fit Testing Based On A Weighted Bootstrap: A Fast Large-Sample Alternative to the Parametric Bootstrap[END_REF] used in Section 6.4.2 requires that the estimators ( m, Σ) can be written as a sum of independent terms with an additive term which converges to 0 in probability. Formally, ( m, Σ) = (m 0 , Σ 0 ) + (1/n) n i=1 ψ(Y i ) + where Eψ(Y ) = 0, Var(ψ(Y )) < +∞ and P -→ n→+∞ 0. However there are some estimators which cannot be written in this form straightforwardly. This is the case for instance if we test whether data follow a Gaussian with covariance of fixed rank r (as in Section 6.6). In this example, the associated estimators are m = m = (1/n) n i=1 Y i (empirical mean) and Σ = Σr = r s=1 λs Ψ ⊗2 s where ( λs ) s and ( Ψs ) s are the eigenvalues and eigenvectors of the empirical covariance operator Σ = (1/n) n i=1 (Y i -μ) ⊗2 . We extend (6.4.7) to the general case when Θ 0 Θ and the estimators ( m, Σ) are not the classical ( m, Σ). We assume that the estimators ( m, Σ) are functions of the empirical estimators m and Σ, namely there exists a differentiable mapping T such that ( m, Σ) = T ( m, Σ), where T (Θ) ⊆ Θ 0 and T | Θ 0 = Id Θ 0 .

Under this definition, ( m, Σ) are consistent estimators of (m, Σ) when (m, Σ) ∈ Θ 0 . This kind of estimators are met for various choices of the null-hypothesis:

• Unknown mean and covariance: ( m, Σ) = ( m, Σ) and T is the identity map Id Θ ,

• Known mean and covariance: ( m, Σ) = (m 0 , Σ 0 ) and T is the constant map T (m, Σ) = (m 0 , Σ 0 ),

• Known mean and unknown covariance: ( m, Σ) = (m 0 , Σ) and T (m, Σ) = (m 0 , Σ),

• Unknown mean and covariance of known rank r: ( m, Σ) = ( m, Σr ) and

T (m, Σ) = (m, Σ r )
where Σ r is the rank r truncation of Σ.

By introducing T , we get a similar approximation as in (6.4.5) by replacing the mapping

N : Θ 0 → H( k) with N oT : Θ 0 → H( k). This leads to the bootstrap replication [n ∆2 ] b f ast := √ n μb P -D ( m, Σ) (N oT )[ mb , Σb ] 2 H( k) . (6.4.8)
The validity of this bootstrap method is justified in Section 6.4.2.

Finally we define an estimator qα,n of q α,n from the generated B bootstrap replications

[n ∆2 ] 1 f ast < . . . < [n ∆2 ] B f ast (assuming they are sorted) qα,n = [n ∆2 ] ( (1-α)B ) ,
where . stands for the integer part. The rejection region is defined by

R α = {n ∆2 > qα,
√ n μP -N [ m, Σ] L -→ n→+∞ G P -D (m 0 ,Σ 0 ) (N oT )[U P ] (ii) √ n μb P -D ( m, Σ) (N oT )[ mb , Σb ] L -→ n→+∞ G P -D (m 0 ,Σ 0 ) (N oT )[U P ] (i)
where (G P , U P ) and (G P , U P ) are i.i.d. random variables in H( k) × Θ.

If otherwise H 0 is false, (ii) still holds true (as long as m 0 and Σ 0 are well defined with respect to P ).

Furthermore, G P and U P are zero-mean Gaussian r.v. with covariances

Var (G P ) = E Y ∼P ( k(Y , .) -μP ) ⊗2 Var (U P ) = E Y ∼P Y -m 0 , (Y -m 0 ) ⊗2 -Σ ⊗2 cov (G P , U P ) = E Y ∼P ( k(Y , .) -μP ) ⊗ Y -m 0 , (Y -m 0 ) ⊗2 -Σ 0 .
By the Continuous Mapping Theorem and the continuity of . 2 H( k) , Proposition 6.4.4 guarantees that the estimated quantile converges almost surely to the true one as n, B → +∞, so that the type-I error equals α asymptotically.

Note that in [START_REF] Kojadinovic | Goodness-of-fit Testing Based On A Weighted Bootstrap: A Fast Large-Sample Alternative to the Parametric Bootstrap[END_REF] the parameter subspace Θ 0 must be a subset of R p for some integer p ≥ 1. Proposition 6.4.4 allows Θ 0 to be a subset of a possibly infinite-dimensional Hilbert space (m belongs to H and Σ belongs to the space of finite trace operators H → H).

Test performances

An upper bound for the Type-II error

Let us assume the null-hypothesis is false, that is P N (m 0 , Σ 0 ) or (m 0 , Σ 0 ) Θ 0 . Theorem 6.5.1 gives the magnitude of the Type-II error, that is the probability of wrongly accepting H 0 . The proof can be found in Appendix 6.B.3. Before stating Theorem 6.5.1, let us introduce or recall useful notation :

• ∆ = μP -(N oT )[m 0 , Σ 0 ]] H( k) , • q α,n = E qα,n , • m 2 P = E P ||D (m 0 ,Σ 0 ) (N oT )[Ψ (Y )] -k(Y , .) + μP || 2 H( k) ,
where

Ψ (Y ) = (Y -m 0 , [Y -m 0 ] ⊗2 -Σ 0 )
and D (m 0 ,Σ 0 ) (N oT ) denotes the Fréchet derivative of N oT at (m 0 , Σ 0 ). According to Proposition 6.4.4 and the continuous mapping theorem, qα,n corresponds to an order statistic of a random variable which converges weakly to G P -D (m 0 ,Σ 0 ) (N oT )[U P ] (as defined in Proposition 6.4.4). Therefore, its mean q α,n tends to a finite quantity as n → +∞. L and m 2 P do not depend on n as well. Then, for any n > q α,n ∆ -2 The first implication of Proposition (6.5.1) is that our test is consistent, that is

P n ∆2 ≤ qα,n ≤ exp             - n ∆ - q α,n n∆ 2 2m 2 P + Cm P M 1/2 (∆ 2 -q α,n /n)             f (α, B, M, ∆) , (6.5.9) where f (α, B, M, ∆) = (1 + o n (1)) 1 + C P b C ∆ 2 M 1/2 m P √ αB + o B (B -1/2 ) C ∆ 4
P(n ∆2 ≤ qα,n | H 0 false) -→ n→+∞ 0 .
Furthermore, the upper bound in (6.5.9) reflects the expected behaviour of the Type-II error with respect to meaningful quantities. When ∆ decreases, the bound increases (alternative more difficult to detect). When α (Type-I error) decreases, q α,n gets larger and n has to be larger to get the bound. The variance term m 2 P encompasses the difficulty of estimating μP and of estimating the parameters as well. In the special case when m and Σ are known, T = Id and the chain rule yields D (m 0 ,Σ 0 ) (N oT ) = (D T (m 0 ,Σ 0 ) N )o(D (m 0 ,Σ 0 ) T ) = 0 so that m 2 P = E|| φ(Y ) -μP || 2 reduces to the variance of μP . As expected, a large m 2 P makes the bound larger. Note that the estimation of the critical value which is related to the term f (α, B, M, ∆) in (6.5.9) does not alter the asymptotic rate of convergence of the bound.

Remark that assuming that qα,n is independent of n ∆2 is reasonable for a large n, since n ∆2 and qα,n are asymptotically independent according to Proposition 6.4.4.

Empirical study of type-I/II errors

Empirical performances of our test are inferred on the basis of synthetic data. For the sake of brevity, our test is referred to as KNT (Kernel Normality Test) in the following.

One main concern of goodness-of-fit tests is their drastic loss of power as dimensionality increases. Empirical evidences (see Table 3 Besides, note that in the following section, k is defined as a Gaussian kernel (see Proposition 6.4.2) with arbitrarily fixed parameter (σ = 1).

Finite-dimensional case (Synthetic data)

Reference tests. The power of our test is compared with that of two multivariate normality tests: the Henze-Zirkler test (HZ) [START_REF] Henze | A Class of Invariant and Consistent Tests for Multivariate Normality[END_REF] and the energy distance (ED) test [START_REF] Szekely | A New Test For Multivariate Normality[END_REF]. The main idea of these tests is briefly recalled in Appendix 6.A.1 and 6.A.2.

Null and alternative distributions. Results. In the small dimension case (Figure 6.2, left column), the actual Type-I error of all tests remain more or less around α (±0.02). Their Type-II errors are superimposed and quickly decrease down to 0 when n ≥ 200. On the other hand, experimental results reveal different behaviors as d increases (Figure 6.2, right column). Whereas ED test lose power, KNT and HZ still exhibits small Type-II error values. Besides, ED and KNT Type-I errors remain around the prescribed level α while that of HZ is close to 1, which shows that its small Type-II error is artificial. This seems to confirm that HZ and ED tests are not suited to high-dimensional settings unlike KNT.

Infinite-dimensional case (real data)

Dataset and chosen kernel. Let us consider the USPS dataset which consists of handwritten digits represented by a vectorized 8 × 8 greyscale matrix (X = R 64 ). A Gaussian kernel

k G (•, •) = exp(-τ 2 || • -• || 2
) is used with τ 2 = 10 -4 . Comparing sub-datasets "Usps236" (keeping the three classes "2", "3" and "6", 541 observations) and "Usps358" (classes "3", "5" and "8", 539 observations), the 3D-visualization (Figure 6.3, top panels) suggests three well-separated Gaussian components for "Usps236" (left panel), and more overlapping classes for "Usps358" (right panel enough number of random projections may allow RP to catch up KNT in terms of power. But RP has a computational advantage over KNT only when p = 1 where the RP test statistic is distribution-free. This is no longer the case when p ≥ 2 and the critical value for the RP test is only available through Monte-Carlo methods.

6.6 Application to covariance rank selection 6.6.1 Covariance rank selection through sequential testing

Under the Gaussian assumption, the null hypothesis becomes

H 0 : (m 0 , Σ 0 ) ∈ Θ 0 ,
and our test reduces to a test on parameters.

We focus on the estimation of the rank of the covariance operator Σ. Namely, we consider a collection of models (M r ) 1≤r≤r max such that, for each r = 1, . . . , r max ,

M r = {P = N (m, Σ r ) | m ∈ H(k) and rk(Σ r ) = r} .
Each of these models correspond respectively to the following null hypotheses H 0,r : rank(Σ) = r, r = 1, . . . , r max , and the corresponding tests can be used to select the most reliable model.

This can be seen as a case of multiple hypothesis testing, that could be solved through methods such as Benjamini-Hochberg procedure [START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF]. However such procedures usually impose to perform all of the tests and to sort all of the corresponding p-values. In order to eventually perform only a few tests, the null-hypotheses H 0,r are tested in a sequential procedure which is summarized in Algorithm 2. This sequential procedure yields an estimator r defined as r ∆ = min r H 0,r rejected for r = 1, . . . , r -1 and H 0,r accepted . or r ∆ = r max if all of the hypotheses are rejected.

Sequential testing to estimate the rank of a covariance matrix (or more generally a noisy matrix) is mentioned in [START_REF] Ratsimalahelo | Strongly consistent determination of the rank of matrix[END_REF] and [START_REF] Robin | Tests of Rank[END_REF]. Both of these papers focus on the probability to select a wrong rank, that is P( r r * ) where r * denotes the true rank. The goal is to choose a level of confidence α such that this probability of error converges almost surely to 0 when n → +∞.

There are two ways of guessing a wrong rank : either by overestimation or by underestimation.

Getting r greater than r * implies that the null-hypothesis H 0,r * was tested and wrongly rejected, hence a probability of overestimating r * at most equal to α. Underestimating means that at least

Robustness analysis

In practice, none of the models M r is true. An additive full-rank noise term is often considered in the literature [CTT14; JH12]. Namely, we set in our case Y = Z + (6.6.11)

where Z ∼ N (m, Σ r * ) with rk(Σ r * ) = r * and is the error term independent of Z. Note that the Gaussian assumption concerns the main signal Z and not the error term whereas usual models assume the converse [CTT14; JH12]. Figure 6.5 illustrates the performance of our sequential procedure under the noisy model (6.6.11). We set H = R 100 , n = 600, r * = 3 and Σ r * = Σ 3 = diag(λ 1 , . . . , λ 3 , 0, . . . , 0) where λ r = exp(-0.2r) for r ≤ 3. The noise term is = (λ 3 ρ -1 η i ) 1≤i≤100 where η 1 , . . . , η 100 are i.i.d. Student random variables with 10 degrees of freedom and ρ > 0 is the signal-to-noise ratio.

As expected, the probability of guessing the target rank r * decreases down to 0 as the signalto-noise ratio ρ diminishes. However, choosing a smaller level of confidence α allows to improve the probability of right guesses for a fixed ρ. without sacrificing much for smaller signal-to-noise ratios. This is due to the fact that each null-hypothesis H 0,r is false, hence the need for a smaller α (smaller Type-I error) which yields greater Type-II errors and avoids the rejection of all of the null-hypotheses.

6.A Goodness-of-fit tests 6.A.1 Henze-Zirkler test

The Henze-Zirkler test [START_REF] Henze | A Class of Invariant and Consistent Tests for Multivariate Normality[END_REF] relies on the following statistic

HZ = R d Ψ (t) -Ψ (t)
2 ω(t)dt , (6.A.12)

where Ψ (t) denotes the characteristic function of N (0, I), Ψ (t) = n -1 n j=1 e i t,Y j is the empirical characteristic function of the sample Y 1 , . . . , Y n , and d+4) . The H 0 -hypothesis is rejected for large values of HZ. Note that the sample Y 1 , . . . , Y n must be whitened (centered and renormalized) beforehand.

ω(t) = (2πβ) -d/2 exp(-||t|| 2 /(2β)) with β = 2 -1/2 [(2d + 1)n)/4] 1/(

6.A.2 Energy distance test

The energy distance test [START_REF] Szekely | A New Test For Multivariate Normality[END_REF] is based on

E(P , P 0 ) = 2E||Y -Z|| -E||Y -Y || -E||Z -Z || (6.A.13)
which is called the energy distance, where Y , Y ∼ P and Z, Z ∼ P 0 . Note that E(P , P 0 ) = 0 if and only if P = P 0 . The test statistic is given by

Ê = 2 n n i=1 E Z ||Y i -Z|| -E Z,Z ||Z -Z || - 1 n 2 n i,j=1 ||Y i -Y j || , (6.A.14)
where Z, Z i.i.d.

∼ N (0, I) (null-distribution). HZ and ED tests set the H 0 -distribution at P 0 = N ( μ, Σ) where μ and Σ are respectively the standard empirical mean and covariance. As for the Henze-Zirkler test, data must be centered and renormalized.

6.A.3 Projection-based statistical tests

In the high-dimensional setting, several approaches share a common idea consisting in projecting on one-dimensional spaces. This idea relies on the Cramer-Wold theorem extended to infinite dimensional Hilbert space. Let Z ∼ N ( m, Σ). The orthogonal projections Z, Ψ i are N (0, λ i ) and for i j, Z, Ψ i and Z, Ψ j are independent. Let Z be an independent copy of Z.

• Gaussian kernel case : k(., .) = exp(-σ • -• 2 H ) Let us first expand the following quantity

N [ m, Σ](y) = E Z exp -σ ||Z -y|| 2 = E Z exp        -σ r≥1 Z -y, Ψ r 2        = +∞ r=1 E Z exp        -σ r≥1 Z -y, Ψ r 2        (6.B.16) = +∞ r=1 E Z exp -σ Z -y, Ψ r 2 = +∞ r=1 (1 + 2σ λ r ) -1/2 exp -σ m -y, Ψ r 2 1 + 2σ λ r = I + 2σ Σ -1/2 exp(-σ (I + 2σ Σ)(y -m), y -m )
We can switch the mean and the limit in (6.B.16) by using the Dominated Convergence theorem since for every N ≥ 1

N r=1 exp -σ Z -y, Ψ r 2 ≤ 1 < +∞ . The second quantity ||N [ m, Σ]|| 2 is computed likewise ||N [ m, Σ]|| 2 = E Z,Z exp -σ ||Z -Z || 2 = E Z,Z exp        -σ r≥1 Z -Z , Ψ r 2        = +∞ r=1 E Z,Z exp        -σ r≥1 Z -Z , Ψ r 2        = +∞ r=1 E Z E Z (exp (-σ Z -Z , Ψ r ) | Z) = +∞ r=1 E Z (1 + 2σ λ r ) -1/2 exp -σ m -Z, Ψ r 2 1 + 2σ λ r = +∞ r=1 (1 + 2σ λ r ) -1/2 E U ∼N (0,λ r ) exp - σ U 2 1 + 2σ λ r = +∞ r=1 (1 + 2σ λ r ) -1/2 1 + 2σ λ r 1 + 2σ λ r -1/2 = +∞ r=1 (1 + 4σ λ r ) -1/2 = I + 4σ Σ -1/2 • Exponential kernel case : k(., .) = exp( •, • H ) Let us first expand the following quantity N [ m, Σ](y) = E Z exp ( Z, y ) = E U ∼N ( m,y , Σy,y ) exp (U ) = exp( m, y + (1/2) Σy, y ). Expanding ||N [ m, Σ]|| 2 , ||N [ m, Σ]|| 2 = E Z,Z exp ( Z, Z ) = E Z,Z exp        i≥1 Z, Ψ i Z , Ψ i        = ∞ i=1 E Z,Z exp ( Z, Ψ i Z , Ψ i ) .
(6.B.17)

We can switch the mean and the limit by using the Dominated Convergence theorem since

N i=1 exp ( Z, Ψ i Z , Ψ i ) a.s. -→ N →∞ exp ( Z, Z ) ≤ exp ||Z|| 2 + ||Z || 2 2 ,
and

E Z,Z exp ||Z|| 2 + ||Z || 2 2 = E Z exp ||Z|| 2 2 2 = E k1/2 (Z, Z) < +∞ .
The integrability of k1/2 (Z, Z) is necessary to ensure the existence of the embedding N [ m, Σ] related to the distribution of Z. As we will see thereafter, it is guaranteed by the condition λ1 < 1.

For each i,

E Z,Z exp ( Z, Ψ i Z , Ψ i ) = E Z E Z (exp ( Z, Ψ i Z , Ψ i ) | Z) = E Z exp λ i 2 Z, Ψ i 2 = (1 -λ 2 i ) -1/2 (6.B.18)
Plugging (6.B.18) into Equation (6.B.17),

||N [ m, Σ]|| 2 = ∞ i=1 (1 -γ 2 i ) -1/2 = I -Σ 2 -1/2 . ( 6
∞ i=1 [1 -λ 2 i ] = exp        -      - ∞ i=1 λ 2 i 1 -λ 2 i        ≥ exp -Tr(Σ 2 ) 1 -λ 2 1 . (6.B.20)
where we used the inequality: log(x) ≤ x -1. Since Σ is a finite trace operator, its eigenvalues converge towards 0 and λ 2 i ≤ λ i < 1 for i large enough. Thus, Tr(Σ 2 ) is finite and it follows from (6.B.20) that I -Σ 2 > 0.

6.B.2 Proof of Theorem 6.4.4

The proof of Proposition 6.4.4 follows the same idea as the original paper [START_REF] Kojadinovic | Goodness-of-fit Testing Based On A Weighted Bootstrap: A Fast Large-Sample Alternative to the Parametric Bootstrap[END_REF] and broadens its field of applicability. Namely, the parameter space does not need to be a subset of R d anymore.

The main ingredient for our version of the proof is to use the CLT in Banach spaces [START_REF] Hoffmann-Jorgensen | The Law of Large Numbers and the Central Limit Theorem in Banach Spaces[END_REF] instead of the multiplier CLT for empirical processes ([Kos07], Theorem 10.1).

We introduce the following notation :

• θ 0 = (m 0 , Σ 0 ) denotes the true parameters

• θ n = ( m, Σ) denotes the empirical parameters

• For any θ = (m, Σ) and y ∈ H, Ψ θ (y) = (y -m, (y -m) ⊗2 -Σ). • θ b 0,n = (1/n) n i=1 (Z b i -Zb )Ψ θ 0 (Y i ) and θ b n = (1/n) n i=1 (Z b i -Zb )Ψ θ n (Y i )
As a first step, we prove that

       √ n( μP -μP ), n -1/2 n i=1 Ψ θ 0 (Y i ), √ n μb P , √ nθ b 0,n        L -→ n→+∞ (G P , U P , G P , U p ) , ( 6
= n -1/2 n i=1 Z b i ( k(Y i , •) -μ) + √ n Zb ( μ -μ) = n -1/2 n i=1 Z b i ( k(Y i , •) -μ) + o P (1) √ nθ b 0,n = n -1/2 n i=1 Z b i Ψ θ 0 (Y i ) - √ n Zb (m -m, Σ -Σ) = √ n Zb + o P (1) , where √ n Zb L -→ n→+∞ N (
(Y -m 0 ) ⊗2 -Σ 0 2 = E Y -m 0 4 + Tr(Σ 0 -Σ 2 0 ) < +∞. Likewise μ → μ a.s. since E k(Y , •) -μ 2 = E k(Y , Y ) -μ 2 < +∞ by assumption.
Therefore the lhs in (6.B.21) can be written as

       √ n( μP -μP ), n -1/2 n i=1 Ψ θ 0 (Y i ), √ n μb P , √ nθ b 0,n        = n -
C 1 = E Y ( k(Y , .) -μP ) ⊗2 C 2 = E Y Ψ θ 0 (Y ) ⊗2 C b 1 = E Z b ,Y [Z b ] 2 ( k(Y , •) -μ) ⊗2 = C 1 C b 2 = E Z b ,Y [Z b ] 2 Ψ ⊗2 θ 0 (Y ) = C 2 .
W i is of finite variance if for every (g, u, g , u ) ∈ H( k) × Θ × H( k) × Θ , E (g, u, g , u ), W 2 < +∞.

Remark that E (g, u, g , u ), W 2 4 2Tr(C 1 )( g 2 + g 2 ) + 2Tr(C 2 )( u 2 + u 2 ) so it suffices to hence the conclusion of Proposition 6.4.4.

6.B.3 Proof of Theorem 6.5.1

The goal is to get an upper bound for the Type-II error P(n ∆2 ≤ q | H 1 ) . (6.B.23)

In the following, the feature map from H(k) to H( k) will be denoted as φ : H(k) → H( k), y → k(y, .) .

Besides, we use the shortened notation q := E qα,n,B for the sake of simplicity (see Section 6.5.1 for definitions). n ∆2 0 is a degenerate U-statistics so it converges weakly to a sum of weighted chi-squares ([Ser80], page 194).

√

nS n converges weakly to a zero-mean Gaussian by the classic CLT as long as E μP -N oT (θ 0 ), Ξ(Y i ) 2 is finite (which is true since k is bounded). It follows that ∆2

0 becomes negligible with respect to S n when n is large. Therefore, we consider a surrogate for the Type-II error (6.B.23) by removing ∆2 0 with a negligible loss of accuracy. Plugging (6.B.24) into (6.B.23) P(n ∆2 ≤ q) =P(n ∆2 0 + n∆ 2 + 2nS n ≤ q -) P(n ∆2 0 + n∆ 2 + 2nS n ≤ q) . (6.B.25)

Finally, using ∆2 0 ≥ 0 and conditioning on q yield the upper bound 3. "Replace" the estimator qα,n with the true quantile q α,n in the bound It remains to take the expectation with respect to q. In order to make it easy, q is pull out of the exponential term of the bound. This is done through a Taylor-Lagrange expansion where s = ∆ 2 -q n , s = ∆ 2 -q n , q ∈ (q ∧ q, q ∨ q) , and s ≥ 0 because of the assumption n > qL -2 .

P(n ∆2 ≤ q | q) ≤ P        n i=1 f (Y i ) ≥ n ŝ       , ( 6
(
The mean (with respect to q) of the right-side multiplicative term of (6.B.29) is bounded by

1 + 3n 2Mϑ E q exp 3| q -q| Mϑ 1 1 s≥0 1/2 E q( ŝ -s) 2 ,
because of the Cauchy-Schwarz inequality.

On one hand, E( qq) 2 → B→+∞ 0 (Lemma 6.B.1) implies E( qq) 2 → B,n→+∞ 0 and thus q → B,n→+∞ q ∞ weakly where q ∞ = lim n→+∞ q α,n (that is almost surely for q ∞ is a constant). Hence E q exp 3| q -q| Mϑ 1 1 s≥0 = E q 1 + o B (| q -q|) Mϑ 1 1 s≥0 ≤ 1 + E q(o B (| q -q|)1 1 s≥0 ) Mϑ .

Since the variable | q -q|1 1 s≥0 is bounded by the constant |nL 2 -q| ∨ |q| for every B, it follows from the Dominated Convergence Theorem 1 + E q(o B (| q -q|)1 1 s≥0 where s = ∆ 2 -q n , s = ∆ 2 -q n , q ∈ (q ∧ q, q ∨ q) .

Proof. A Taylor-Lagrange expansion of order 1 can be derived for h( ŝ) since the derivative of h h (x) = -(2/3)nMϑx 2 + 4nϑ 2 x (2ϑ 2 + (2/3)Mϑx) 2 exp -nx 2 2ϑ 2 + (2/3)Mϑx 1 1 x≥0 , is well defined for every x ∈ R (in particular, the left-side and right-ride derivatives at x = 0 coincide).

Therefore h( ŝ) equals h(s) + h (s)( ŝs)

= exp -ns 2 2ϑ 2 + (2/3)Mϑs 1 + exp (g(s)g(s)) g (s)1 1 s≥0 ( ŝs) , (6.B.37) where s = ∆ 2 -q n , s = ∆ 2 -q n , q ∈ (q ∧ q, q ∨ q) , and s ≥ 0 because of the assumption n > q∆ -2 . Chapter 7

General Conclusion and Perspectives

Nearly Gaussian marginals in an RKHS

In Chapter 4, we have evidenced that most p-dimensional projections of an embedded distribution in the RKHS of a Gaussian RBF kernel are close to some scale-mixture of Gaussians and derived a practical application to outlier detection from this observation in Chapter 5. Besides we also showed that with a proper renormalization of such a kernel, those projections become close to a N (0, I p ) Gaussian distribution instead. However a straightforward application of this

result is yet to be devised. In the following, we consider a few practical issues for which said result could be useful:

• Dimension reduction. In the multivariate setting, Non-Gaussian Component Analysis or NGCA [Bla+06; Die+10; DJS13] is a method of dimension reduction that relies on the assumption that a d-variate random vector X (with large d) can be decomposed as X = X 0 + X ⊥ where X 0 lies on a p-dimensional subspace V of R d and the noisy term X ⊥ is Gaussian. Hence the goal of NGCA is to reconstruct V . However as mentioned in Chapter 4, existing results in R d state that most marginals of such X when d is large are close to a scale-mixture of Gaussians instead of a Gaussian, which is not compatible with the assumptions of NGCA. That being said, embedding X into the RKHS of a renormalized Gaussian RBF kernel would yield nearly Gaussian marginals in most directions in the kernel space. Therefore it would be of interest to devise a kernel extension of NGCA to find the few remaining directions with non-Gaussian marginals. Such an extension is not straightforward since multivariate NGCA relies on a rewriting of the assumed decomposition of X in terms of its density, but in an infinte-dimensional RKHS there exists no canonical Haar measure (i.e. a shift-invariant measure) in the RKHS to define a density as in R d . NGCA could be adapted to the kernel case by considering for instance moment-generating functions instead of densities.

• Sampling from a known distribution. Given a density f of a distribution in R d , sampling a sequence x 1 , . . . , x n ∈ R d that emulates an i.i.d. sample from f is usually done through Monte Carlo Markov Chain methods [START_REF] Liang | Advanced Markov Chain Monte Carlo Methods: Learning from Past Samples[END_REF]. This class of methods generates a Markov chain x 1 , x 2 , . . . where the distribution of the x i converges weakly to an equilibrium distribution of density f . Albeit popular, these methods suffer some issues, typically a slow convergence of the Markov chain to the desired distribution, either because of a strong dependence between the successive samples or because of small acceptation rates. In such cases, a long Markov chain must be generated in practice, which leads to an intensive computational burden. Our result could be the basis of an alternative take on this sampling problem without using Markov chains. Indeed we could sample an i.i.d. sample from a N (0, I p ) Gaussian directly in a subspace of the RKHS and revert to the input space to find the corresponding sample that would look like an i.i.d. sample from f1 . This "revert" step would be the challenging part of such a sampling procedure since a given point in an RKHS does not admit an exact pre-image in the input space. However there exist several approaches to solve this problem [SSM97; BWS04].

Outlier detection

In Chapter 5, we introduced a new kernel-based method based on random projections in an RKHS (induced by a Gaussian RBF kernel) to perform outlier detection in an online setting. Our proposed method has the advantage of bypassing shortcomings of one-class SVM and kernel PCA -that are the lack of false alarm rate control for one-class SVM and a high computational cost for kernel PCA. Furthermore, we have provided theoretical guarantees about performances of our algorithm -in particular for missed detection rate -whilst such guarantees are lacking for kernel PCA to the best of our knowledge. The next step would be to transpose our OD method to the batch framework, where inliers and outliers are all provided at once in one sample. The problem to solve is that the random generation of the projection subspace V n in the RKHS depends on the empirical covariance function Σ γ,n (•, •) which estimates the covariance of the embedded inliers. However in a batch setting, this covariance would be estimated on the basis of inliers and outliers all together. This problem could be tackled by resorting to robust covariance estimators. In this case one should check that asymptotic results provided in Chapter 4 still holds with such estimators.

Normality test in Hilbert spaces

In Chapter 6, we introduced a new normality test suited to high-dimensional Hilbert spaces, that was based on an MMD statistic constructed from a characteristic kernel k. Investigating the influence of the kernel k on the performance of the test would be of interest. In the case where k is a Gaussian RBF kernel for instance, a method to optimize the Type-II error with respect to the hyperparameter of k would be welcomed. It turns out to be a challenging task. In the literature about hypothesis testing with the MMD, one of the few papers addressing this issue is [START_REF] Gretton | Optimal kernel choice for large-scale two-sample tests[END_REF] which optimizes a homogeneity test over convex combinations of a basis of kernels, but at the cost of modifying the MMD statistic to resort to a less powerful test.

We applied successfully a fast parametric bootstrap procedure that reduces the computational costs of our test from cubic to quadratic time with respect to the sample size n. However a quadratic execution time may still be too high when n becomes very large. For two-sample MMD tests, a linear time statistic can be devised to speed up the test (see Section 6 in [START_REF] Gretton | A Kernel Two-Sample Test[END_REF]).

However there are two issues: firstly the corresponding test is typically less powerful, and secondly it cannot be straightforwardly applied to the one-sample setting because of the estimation of the parameters (mean and covariance).

Finally, the choice of the level α for the sequential procedure (covariance rank selection) is another subject for future research. Indeed, an asymptotic regime for α has been exhibited to get consistency, but setting the value of α when n is fixed remains an open question.

A.2 Auxilary lemmas used in Chapter 4 and Chapter 5

Lemma A.2.1. Let X ∼ P and assume there exists a function A : X → R and s > 0 such that P(d 2 (X, x) t) = A(x)t s [1 + o(1)] as t → 0 for any x ∈ supp(P ) and A(x) = 0 for any x supp(P ). Also assume that supp(P ) is a compact subset of X and (t, x) → P(d 2 (x, X) t) is continuous. Then for any x ∈ supp(P ), Σ γ (x, x) = A(x)Γ (s + 1) (2γ) s 1 + A -1 (x) o γ (1) a.s. , when γ → +∞, where the 'o γ (1)' term is uniformly bounded with respect to x.

Proof. By definition

Σ γ (x, x) = E X k 2 γ (X, x) = Ee -2γd 2 (x,X) .
Fubini's Theorem entails where f (γ) is some arbitrary function such that f (γ) = o(γ) when γ → +∞. On the first hand, g : R + × supp(P ) → R is continuous since (t, x) → P X (d 2 (X, x) t) and x → A(x) are continuous. Therefore since supp(P ) is assumed compact, [0, f (γ)/(2γ)] × supp(P ) is a compact for the product topology of R + × X and by the continuity of g hence of |g|, the image |g| ([0, f (γ)/(2γ)] × supp(P )) is a compact of R + , namely |g|([0, f (γ)/(2γ)]×supp(P )) = [0, C γ ] with C γ 0. C γ is decreasing with respect to γ and decreases to 0 when γ → +∞ since g is continuous and g(0; x) = 0 for every x ∈ supp(P ). It follows where the 'o γ (1)' term is uniformly bounded with respect to x.

E X e -2γd 2 (x,X) = E X [-e -y ]
On the other hand, we show that the last additive term in (A.2.2) is negligible before γ -s for a proper choice of f (γ). where Γ (., .) denotes the upper incomplete Gamma function.

Equivalent terms for Γ (., t) when t → +∞ are well-known and yield

Γ (s + 1, f (γ)) ∼ f s (γ)e -f (γ) ,
hence an equivalent for (A.2.3) is A(x) -1 (2γ) s exp(-f (γ)) which converges to 0 if f (γ) = q log(γ) with q > s for instance. (Note that such a choice for f (γ) still fulfills the condition f (γ) = o(γ)) Lemma A.2.2. With the same assumption as Lemma A.2.1,

Tr(Σ 2 γ ) ∼
EA(X)Γ (s + 1) (2γ) s , as γ → +∞.

Proof. Tr(Σ 2 γ ) is expressed as follows

Tr(Σ 2 γ ) = E X Σ γ k X , k X γ ∆ = Eδ 2 X .
By Lemma A.2.1 and Slutsky's lemma, (2γ) 2s δ 2 X converges weakly to A(X)Γ (s + 1) when γ → +∞. Therefore, since A is assumed continuous and bounded, the weak convergence implies E(2γ) 2s δ 2 X → EA(X)Γ (s + 1) hence the result of the lemma.

Lemma A.2.3. Write δ XX = Σ γ (X, X ) where X, X are two i.i.d. random variables. When γ → +∞,

Eδ XX ∼ E X A 2 (X)Γ 2 (s + 1) γ 2s .
Proof. By definition of δ XX ,

Eδ XX = E Σ γ k X , k X γ = E X,X ,X k ⊗2 X k X , k X γ = E X,X ,X k γ (X , X)k γ (X , X)
= Ee -γ{d 2 (X,X )+d 2 (X,X )} = E X E X e -γd 2 (X,X ) 2 = E X Σ γ/2 (X, X) Proof. Since Σ γ,n (x, x) = (1/n) n i=1 k 2 γ (X i , x), Lemma A.1.2 can be applied with c = 1 and Var X (k 2 γ (X, x)) Ek 4 γ (X, x) = Σ 2γ (x, x) = ν which entails with probability at least 1δ

Σ γ,n (x, x) -Σ γ (x, x) 2ν log(1/δ) n       1 + κ log(1/δ) nν       ,
where K is an absolute constant. Therefore with high probability Gaussian Models and Kernel Methods

Abstract

Kernel methods have been extensively used to transform initial datasets by mapping them into a so-called kernel space or RKHS, before applying some statistical procedure onto transformed data. In particular, this kind of approach has been explored in the literature to try and make some prescribed probabilistic model more accurate in the RKHS, for instance Gaussian mixtures for classification or mere Gaussians for outlier detection. Therefore this thesis studies the relevancy of such models in kernel spaces. In a first time, we focus on a family of parameterized kernels -Gaussian RBF kernels -and study theoretically the distribution of an embedded random variable in a corresponding RKHS. We managed to prove that most marginals of such a distribution converge weakly to a so-called "scale-mixture" of Gaussians -basically a Gaussian with a random variance -when the parameter of the kernel tends to infinity. This result is used in practice to device a new method for outlier detection. In a second time, we present a one-sample test for normality in an RKHS based on the Maximum Mean Discrepancy. In particular, our test uses a fast parametric bootstrap procedure which circumvents the need for re-estimating Gaussian parameters for each bootstrap replication.

Keywords

  Consider for instance X = R d with d ∈ N * . A way of fulfilling this goal is to separate the two sets of points {x i | y i = -1} and {x i | y i = 1} by an hyperplane H of R d defined by a corresponding normal vector u ∈ R d and an offset b ∈ R, that is H = {x ∈ R d | x T u + b = 0}. Therefore a new point x ∈ R d is assigned to the class ŷ defined by ŷ ∆ = sgn x T u + b ,

  .1.1) where |u| 2 = √ u T u denotes the Euclidean norm in R D andS B = M j=1 (µ jµ)(µ jµ) T µ j )(x iµ j ) T 1 1 y i =j ,

Figure 2

 2 Figure 2.3 -Illustration of the bag-of-words approach: given a set of relevant words, transform a text into a histogram with occurrence of each of the relevant words, and compare texts by comparing corresponding histograms.

Figure 2

 2 Figure 2.4 -Trees (a) and (b) are compared by counting the number of their common subtrees. For instance, Tree (c) is a common subtree of (a) and (b).

  dx . Then define iteratively each function B i : R d → R by B i+1 = B i ⊗ B 0 for each i 0. The function B 2p+1 where p 0 are positive definite and therefore the kernel k p k p (x, y) = B 2p+1 (xy) for all x, y ∈ R d , defines a translation-invariant kernel [V+97].

Figure 2 . 6 -

 26 Figure 2.6 -Application of kernel PCA to image de-noising. From left to right: original noisy image; de-noised image with linear PCA; de-noised image with kernel PCA and gradient descent to learn the pre-image; de-noised image with kernel PCA and regularized regression to learn the pre-image.

CS10; SFL11 ]

 SFL11 . Examples of characteristic kernels include commonly used kernels such as the Gaussian kernel and the Laplacian kernel on R d .

  their respective covariance matrices and | • | denotes the determinant of a matrix. Now we are in a position to assign a score function to each class based on the MAP. Note that the denominator in (3.4.15) does not depend on the class label j, hence the score function D j (x) of the j-th class is defined solely via the numerator in (3.4.15

  the asymptotic distribution foretold by Theorem 4.3.3. Therefore the influence of |f | 2 displayed by the bound in (4.3.5) is consistent. Another way to explain the role of |f | 2 is to relate |f | 2 with the covariance eigenspectrum in H(k γ ). Remember that a small eccentricity of the covariance eigenspectrum fosters the convergence to a SMG, as Theorem 4.2.1 suggests in R D . When |f | 2 is large and P close to a discrete distribution (let us assume that the support is of cardinality m ∈ N * ), the support of the distribution of k γ (X, .) is contained in a subspace of H(k γ ) of finite dimension at most m.

  sup D 1 l (D)1 < 1/(2γ) leads to a loose upper bound in (4.3.5) as D grows to infinity.

  Theorem 4.4.2 extends Theorem 4.4.1 to the empirical case. The proof of Theorem 4.4.2 can be found in Appendix 4.A.4.

Figure 4 . 1 -

 41 Figure 4.1 -Upper-left: Observations in the input space (uniform on a subset of R 2 ); upper-right:Low-dimensional (p = 2) projection p V n (X) of embedded points in the RKHS H(k γ ) (γ = 400); lower-left: p-values given by Henze-Zirkler multivariate normality test on projections p V n (X) (renormalized by Tr 1/2 (Σ 2 γ )) with varying values for γ; lower-right: p-values given by the HSIC test for independence between X and Tr -1/2 (Σ 2 γ ) • p V n (X).

Figure 4 . 2 -

 42 Figure 4.2 -Upper-left: Observations in the input space (uniform on a subset of R 2 ); upperright: Low-dimensional (p = 2) projection pV n (X) of renormalized embedded points in the RKHS H(k γ ) (γ = 400); lower-left: p-values given by Henze-Zirkler multivariate normality test on projections pV n (X) with varying values for γ; lower-right: p-values given by the HSIC test for independence between X and pV n (X).

4.B. 1

 1 Covariance eigenvalues in an RBF RKHS Lemma 4.B.1. Let (λ r ) r be the eigenvalues of the covariance operator Σ γ . Let supp

0 ee

 0 X ) y} A -dy = +∞ -y P d 2 (x, X ) y/γ A -dy = M 0 e -y P d 2 (x, X ) y/γ P (d 2 (x, X ) M/γ) -y [1 + g(y/γ; x)]y s /γ s [1 + g(M/γ; x)]M s /γ s dy + e -M = 1 + o γ (1)

  .B.30) if M satisfies M → +∞ and M = o γ (γ). Here γ(., .) denotes the lower incomplete Gamma function γ(s, x) = x 0 e -y y s-1 dy.Combining (4.B.28) and (4.B.30) yields pξ -∼ EA(X)Γ (s + 1) γ s .

Figure 5

 5 Figure 5.1 -"Spiral" synthetic dataset γ 0.1 1 5 25 50 75 100 500 1000 3000 p = 2 0.054 0.060 0.052 0.052 0.058 0.060 0.062 0.124 0.219 0.452 10 0.061 0.046 0.058 0.061 0.062 0.075 0.086 0.260 0.400 0.667 25 0.043 0.050 0.067 0.068 0.079 0.080 0.096 0.296 0.450 0.730 50 0.045 0.042 0.059 0.071 0.083 0.092 0.106 0.311 0.469 0.759 100 0.027 0.043 0.067 0.071 0.084 0.096 0.106 0.317 0.477 0.779 500 0.054 0.034 0.068 0.075 0.090 0.101 0.114 0.320 0.488 0.805

Figure 5 . 2 -Figure 5 . 3 -

 5253 Figure 5.2 -False alarm rates with the "spiral" dataset γ 0.1 1 5 25 50 75 100 500 1000 3000 p = 2 0.947 0.950 0.807 0.074 0.018 0.012 0.009 0.001 0 0 10 0.950 0.947 0.446 0.012 0.005 0.001 0 0 0 0 25 0.972 0.920 0.245 0.007 0 0 0 0 0 0 50 0.966 0.906 0.142 0.004 0 0 0 0 0 0 100 0.989 0.866 0.09 0.003 0 0 0 0 0 0 500 0.988 0.798 0.054 0.001 0 0 0 0 0 0 Figure 5.3 -Missed detection rates with the "spiral" dataset

Figure 5 Figure 5 Figure 5 . 6 -Figure 5 . 7 -

 555657 Figure 5.4 -False alarm rates (left) and missed detection rates (right) with the "spiral" dataset and the Nyström approximation with r = 50

Finally, we illustrate

  our test and present a sequential procedure that assesses the rank of a covariance operator. The problem of covariance rank estimation is adressed in several domains: functional regression [CJ10; BMR13], classification[START_REF] Zwald | Performances d'Algorithmes Statistiques d'Apprentissage: "Kernel Projection Machine" et Analyse en Composantes Principales à Noyaux[END_REF] and dimension reduction methods such as PCA, Kernel PCA and Non-Gaussian Component Analysis[START_REF] Blanchard | Non-Gaussian Component analysis: a Semi-parametric Framework for Linear Dimension Reduction[END_REF][START_REF] Diederichs | Sparse non-Gaussian component analysis[END_REF][START_REF] Diederichs | Sparse non Gaussian component analysis by semidefinite programming[END_REF]

m

  and Σ are respectively the mean and the covariance operator of Y . The distribution of Y is denoted N (m, Σ).

  (1/n) n i=1 δ Y i is the empirical distribution. The null-distribution embedding N [m 0 , Σ 0 ] is estimated by N [ m, Σ] where m and Σ are appropriate and consistent (under H 0 ) estimators of m 0 and Σ 0 . This yields the estimator ∆2 = μP -N [ m, Σ] 2 H( k) ,

  The continuity of N [m, Σ] (with respect to (m, Σ)) and the consistency of ( m, Σ) yield N [ m, Σ]P -a.s. -→ n→∞ N [m 0 , Σ 0 ] P -a.s.. Finally, the continuity of • 2 H leads to ∆2 P -a.s. -→ n→∞ ∆ 2 .The expressions for N [ m, Σ](Y i ) and N [ m, Σ] 2 H( k) in (6.4.3) depend on the choice of k. Those are given by Propositions 6.4.2 and 6.4.3 when k is Gaussian and exponential. Note that in these cases, the continuity assumption of N [m, Σ] required by Proposition 6.4.1 is satisfied.

  bootstrap. Parametric bootstrap consists in generating B samples of n i.i.d. random variables Y ( m, Σ) (b = 1, . . . , B). Each of these B samples is used to compute a bootstrap replication [n ∆2 ] b = n μb P -N [ mb , Σb ] 2 H( k) , (6.4.4) where μb P , mb and Σb are the estimators of µ P , m and Σ based on Y b 1 , . . . , Y b n . It is known that parametric bootstrap is asymptotically valid [SMQ93]. Namely, under H 0 , ∀b = 1, . . . , B, n ∆2 , [n ∆2 ]

  Let us consider in a first time the case when the estimators of m and Σ are the classical empirical mean and covariance m = (1/n) n i=1 Y i and Σ = (1/n) n i=1 (Y i -m) ⊗2 . Introducing the Fréchet derivative [FSG08] D (m,Σ) N at (m, Σ) of the function

Figure 6 Figure 6 . 1 -

 661 Figure 6.1 (left plot) compares empirically the bootstrap distribution of [n ∆2 ] bf ast and the distribution of n ∆2 . When n = 1000, the two corresponding densities are superimposed and a two-sample Kolmogorov-Smirnov test returns a p-value of 0.978 which confirms the strong similarity between the two distributions. Therefore the fast bootstrap method seems to provide a very good approximation of the distribution of n ∆2 even for a moderate sample size n.

  Theorem 6.5.1. (Type II error) Assume sup x,y∈H 0 | k(x, y)| = M < +∞ where Y ∈ H 0 ⊆ H P -almost surely and qα,n is independent of n ∆2 .

  Mm 2 P , and C, C , C are absolute constants and C P b only depends on the distribution of [n ∆2 ] b f ast .

Figure 6

 6 Figure 6.2 -Type-I and type-II errors of KNT (+ blue), Energy Distance (• black), and Henze-Zirkler ( red). Two alternative distributions are considered: HA1 (rows 1 and 3) and HA2 (rows 2 and 4). Two settings are considered: d = 2 (left) and d = 100 (right).

Figure 6

 6 Figure6.3 -3D-Visualization (Kernel PCA) of the "Usps236" (top row, left) and "Usps358" (top row, right) datasets; comparison of Type-II error (bottom row, left: "Usps236", right: "Usps358") for: KNT (× blue) and Random Projection with p = 1 (• green), p = 5 ( purple) and p = 15 (+ red) random projections.

  Figure 6.5 -Illustration of the robustness of our sequential procedure under a noisy model.

  0, 1) (from the classical CLT) and ( m, Σ) → (m 0 , Σ 0 ) almost surely from the Law of Large Numbers (LLN) in Banach spaces (Theorem 2.1, [HP76]). More precisely, in our case, the LLN holds if E Ym 0 2 and E (Ym) ⊗2 -Σ 0 2 are finite, which is the case by assumption since E Ym 0 2 = Tr(Σ 0 ) < +∞ and E

  o P (1) , (6.B.22) whereW i = k(Y i , •) -μP , Ψ θ 0 (Y i ), Z b i ( k(Y i , •) -μ), Z b i Ψ θ 0 (Y i ) for i = 1, . . . ,n are i.i.d. terms. Now we are in a position to apply the CLT in Hilbert spaces ([HP76],Theorem 3.5) to n -1/2 n i=1 W i so that (6.B.22) entails (6.B.21). In particular the CLT holds if W i are of mean 0 and finite variance. Clearly, EW i = 0. As for the finite variance condition, let us first introduce the covariance operators C 1 , C b 1 : H( k) → H( k), C 2 , C b 2 : Θ → Θ as follows

+.

  show that Tr(C 1 ) and Tr(C 2 ) are finite. This is the case since Tr(C 1 ) = E P k(Y , Y ) -|| μP || 2 < +∞ and Tr(C 2 ) = E P ||Yµ 0 || 4 + Tr(Σ 0 -Σ 2 0 ) < +∞ by assumption. Therefore, the CLT in Hilbert spaces yields√ n( μP -μP ) GP (0, C 2 ) .Furthermore since (G P , U P ) and (G P , U P ) are jointly Gaussian andcov , n -1/2 n i=1 Ψ θ 0 (Y i ) = 0 cov √ n μb P , n -1/2 n i=1 Ψ θ 0 (Y i ) = 0, cov √ nθ b 0,n ,√ n( μP -μP ) = 0 , the limit Gaussian processes (G P , U P ) and (G P , U P ) are independent. Besides, (G P , U P ) and(G P , U P ) share the same cross-covariance operator C 1,2 : Θ → H( k) since Y ( k(Y , .) -μP ) ⊗ Ψ θ 0 (Y ) C b 1,2 = cov Z b ( k(Y , •) -μ), Z b Ψ θ 0 (Y ) = E[Z b ] 2 C 1,2 = C 1,2 ,therefore (G P , U P ) and (G P , U P ) are i.i.d. .Since D θ 0 (N oT ) is assumed continuous w.r.t. θ 0 , we get by the continuous mapping theoremthat -μP ) -D θ 0 (N oT )[n -1/2 n i=1 Ψ θ 0 (Y i )], √ n μb P -D θ 0 (N oT )[ G P -D θ 0 (N oT )[U P ], G P -D θ 0 (N oT )[U P ] .To get the final conclusion of Proposition 6.4.4, we have to prove two things.First, under the assumption that P = N (T (θ 0 )) (H 0 is true), considering the Fréchet derivative D θ 0 (N oT ) of N oT at θ 0 yields the following Taylor approximation√ n( μP -N oT [θ n ]) = √ n( μP -μP ) -√ n(N oT [θ n ] -N oT [θ 0 ]) = √ n( μP -μP ) -D θ 0 (N oT )[ √ n(θ nθ 0 )] + o P ( √ n||θ nθ 0 ||) = √ n( μP -μP ) -D θ 0 (N oT )[n -1/2 n i=1 Ψ θ 0 (Y i )] + o P ( √ n||θ nθ 0 ||) D θ 0 (N oT )[U P ] , because √ n(θ nθ 0 )converges weakly to a zero-mean Gaussian and by using the continuous mapping theorem with the continuity of θ 0 → D θ 0 N and of ||.|| Θ . Secondly, whether H 0 is true or not, √ n μb P -D θ n (N oT )[ D θ 0 (N oT )[ √ nθ b 0,n ] -D θ n (N oT )[ we must check that both (a) and (b) converge P -almost surely to 0. Since θ n (ω) → θ 0 P -almost surely and D θ (N oT ) = (D T (θ) N ) o (D θ T ) is continuous w.r.t. (N oT )[(m 0m n , Σ 0 -Σ n ] -D θ n (N oT )[ √ nθ b 0,n ] -→ 0 P -almost surely , so that it follows √ n μb P -D θ n (N oT )[ √ nθ b n ] -→ G P -D θ 0 (N oT )[U P ] ,

1.

  Reduce n ∆2 to a sum of independent termsThe first step consists in getting a tight upper bound for (6.B.23) which involve a sum of independent terms. This will allow the use of a Bennett concentration inequality in the next step.Introducing the Fréchet derivative D θ 0 (N oT ) of N oT at θ 0 , n ∆2 is expanded as followsn ∆2 =n μP -(N oT )(θ n ) 2 =n μP -μP + (N oT )(θ 0 ) -(N oT )(θ n ) + μP -(N oT )(θ 0 ) 2 =n μP -μP + D θ 0 (N oT )(θ 0θ n ) + o( θ 0θ n ) + μP -(N oT )(θ 0 ) i ) -μP -D θ 0 (N oT )(Ψ (Y i )) + √ n ( μP -(N oT )(θ 0 )) + o n √ n(θ 0θ n ) 2 :=n ∆2 0 + n∆ 2 + 2nS n + . oT (θ 0 ), Ξ(Y i ) , Ξ(Y i ) = φ(Y i ) -μP -D θ 0 (N oT )(Ψ (Y i )) ,and = o P (1) almost surely.

  .B.26) where f (Y i ) := μP -N oT (θ 0 ), Ξ(Y i ) , ŝ := ∆ 2 -Apply a concentration inequality We now want to find an upper bound for (6.B.26) through a concentration inequality, namely Lemma A.1.2 with ξ i = f (Y i ), t = n ŝ, ν 2 = Var(f (Y i )) and f (Y i ) ≤ c = M (P -almost surely). Lemma A.1.2 combined with Lemma 6.B.3 and 6.B.2 yields the upper boundP( n i=1 f (Y i ) ≥ n ŝ | q) ≤ expϑ 2 := ∆ 2 m 2 P ,and m 2 P = E Ξ(Y ) 2 .

≤.

  Lemma 6.B.2. Let f be defined as in (6.B.27). If Y ∈ H 0 ⊆ H P -almost surely and sup x,y∈H 0 | k(x, y)| = M, then for any y ∈ H 0 |f (y)| ≤M := ((y)| = μP -N oT (θ 0 ), D θ 0 (N oT )(Ψ (y)) -φ(y) + μP ≤∆ D θ 0 (N oT )(Ψ (y)) -φ(y) + μP ≤ ∆ lim t→0 E Z,Z ∼N oT (θ 0 +tΨ (y)) k(Z, Z) + k(Z , Z ) -2 k(Z, Z ) 1/2 + k(y, y) + E k(Y , Y ) = Var(f (Y )) = Ef 2 (Y i ) =E μP -N oT (θ 0 ), D θ 0 (N oT )(Ψ (Y )) -φ(Y ) + μP 2 ∆ 2 E D θ 0 (N oT )(Ψ (Y )) -φ(Y )Lemma 6.B.4. Let h be defined as in (6.B.28). Then, h( ŝ) ≤ exp -| ŝ -s| , (6.B.36)

For

  every x, y > 0, |g (x)| ≤ 3n/(2Mϑ) and then |g(x)g(y)| ≤ 3n|x -y|/(2Mϑ). It follows |g (s)| ≤ 3n 2Mϑ , and exp (g(s)g(s)) ≤ exp 3n 2Mϑ |s -s| = exp 3| q -q| 2Mϑ . Lemma 6.B.4 is proved.

e 0 ee

 0 -y E X 1 1 {2γd 2 (x,X) y} dy = +∞ -y P d 2 (x, X) y 2γ dyIntroduce the function g(t; x) satisfying P(d 2 (x, X) t) = A(x)(1 + g(t; x))t s and g(t; x) → 0 when t → 0.E X e -2γd 2 (x,X) = A-y y s g(y/2γ; x)dy+ A(x) (2γ) s +∞ f (γ)e -y y s g(y/2γ; x)dy , (A.2.2)

f (γ) 0 e

 0 -y y s g(y/2γ; x)dy C γ f (γ) 0 e -y y s dy C γ +∞ 0 e -y y s dy C γ Γ (s + 1) = o γ (1) ,

+∞f

  (γ) e -y y s g(y/2γ; x)dy+∞ f (γ) e -y y s (2γ) s A(x)f s (γ) -1 dy (2γ) s A(x)f s (γ) Γ (s + 1, f (γ)) , (A.2.3)

2 .

 2 Lemma A.2.1 entailsEδ XX = E X A 2 (X)Γ 2 (s + 1) γ 2s [1 + A -1 (X)o γ (1)] 2 = EA 2 (X)Γ 2 (s + 1) γ 2s [1 + o γ (1)] ,since EA(X) < +∞ because of the boundedness of A(.).Lemma A.2.4. Assume a = inf{A(x) : x ∈ supp(P )} > 0 and A = sup{A(x) : x ∈ supp(P )} < +∞. ThenΣ γ,n (x, x) Σ γ (x, x) least 1δ.Here κ is an absolute constant.

Σ

  γ,n (x, x) Σ γ (x, x) A.2.1 and introducing a = inf{f (x) :x ∈ supp(f )} > 0, Σ 2γ (x, x) Σ γ (x, x) = (4γ) -s/2 A(x)Γ (s + 1)(2γ) -s [1 + o γ (1)] γ s/2 aΓ (s + 1) [1 + o γ (1)] and ν = Σ 2γ (x, x) aΓ (s + 1)(2γ) -s [1 + o γ (1)] .

  and for every l∈ N * H 2l = H 2 ⊗ H l ,

	To understand how W emulates W , one must see that the rows of H d GΠH d B follow a N (0, I d )
	Gaussian distribution and have the same norm equal to G 2 F d. In other words the rows of
	(1/ G 2 F d)H d GΠH d B are uniformly distributed on the unit sphere
	where ⊗ stands for the Kronecker matrix product. The diagonal entries of B are i.i.d. Rademacher
	variables 5 , those of G are i.i.d. N (0, 1) Gaussians and those of S are i.i.d. variables whose
	distribution depends on the kernel k.
	The iterative definition of H d allows to compute any matrix product H d v where v ∈ R d
	recursively to get a O(d log(d)) computational cost instead of O(d 2 ). This way computing W x
	costs O(d log(d)) in time in the case r = d. If r > d is chosen as a multiple of d, W can be written
	as the result of stacking r/d square matrices W1 , . . . , Wr/d of size d sampled as in (2.4.13) which
	finally leads to a complexity of order O(r log(d)).

  Definition 3.1.3 (Gaussian stochastic process). Let Y = {Y (x)} x∈X be a stochastic process where X is a non-empty set. Y is a Gaussian stochastic process if for every a 1 , . . . , a m ∈ R and x 1 , . . . , x m ∈ X

	with m ∈ N * , m i=1 a i Y (x i ) is a (univariate) Gaussian random variable.
	Definition 3.1.4 (Gaussian random element). Let Z be a random element in (H(k), C H(k) ). Z is a
	Gaussian random element if for every h ∈ H(k), Z, h H(k) is a (univariate) Gaussian random variable.

  be the linear kernel on R m . In this case the RKHS H(k) consists in the set of every linear map u → u T v with v ∈ R m . Let Y = {Y (u)} u∈R m be a Gaussian stochastic process indexed by R m

	such that the Y (u) are i.i.d. N (0, 1) Gaussian variables. Clearly Y is not a linear map with high
	probability. This raises the question about conditions for a Gaussian process to lie in a given
	RKHS with probability one. This issue is treated in Section 3.2 thereafter which introduces the
	notion of nuclear dominance.

  that the converse is possible only if the paths of a given process Y belong to H(k) almost surely. In this section, we are interested in sufficient conditions that ensure this

	requirement.
	Let us assume that the process Y = {Y (x)} x∈X admits a mean function m : X → R and a
	covariance function R : X × X → R. In the case of a Gaussian process Y , [Dri73] studied the
	relationships between R and the kernel k that entail P(Y ∈ H(k)) = 1.
	Proposition 3.2.1 ([Dri73]). Assume X is a separable metric space and Y = {Y (x)} x∈X is a Gaussian
	process with mean m(•) ∈ H(k), covariance function R(•, •) continuous on X × X and whose paths are
	almost surely continuous on X . Let {x 1 , x 2 , . . .} be a countable dense subset of X and define the matrices

  Proposition 3.3.6. There is nuclear dominance of k over the parsimonious sample covariance estimator Rd , that is Rd << k.

Proof of Proposition 3.3.6. Having proved that k >> R and R >> Rd , one would like to deduce that k >> Rd . It is straightforward that H( Rd ) ⊆ H(k). A candidate for being the dominance operator of k over Rd is Ld L. Clearly Ld L is bounded since Ld and L are bounded. It remains to check that Tr( Ld L) < ∞:

  Let us further introduce additional notation needed for our main statements. Let X be a X -valued random variable with distribution P where X denotes some set. Using the notation introduced in Section 4.2.2, we consider the Radial Basis Function (RBF) kernel k γ parameterized by γ > 0, its RKHS H(k γ ) with corresponding inner product ., .

	Asymptotic distribution of random projections
	4.3.1 Notation

γ and norm . γ . As a shorthand, we denote the embedded variable k γ (X, .) taking values in H(k γ ) as k γ (X, .) = k X .

Following

[START_REF] Dasgupta | A concentration theorem for projections[END_REF]

, we examine the distribution of projections of k X onto p-linear subspaces of H(k γ ) denoted by V = Span(h 1 , . . . , h p ) where h 1 , . . . , h p ∈ H(k γ ) are random vectors. Ideally h 1 , . . . , h p should form an orthonormal family of H(k γ ) almost surely. For the sake of simplicity, we follow an approach similar as

[START_REF] Dasgupta | A concentration theorem for projections[END_REF]

and consider h 1 , . . . , h p p independent zero-mean Gaussian processes of covariance Σ γ , where Σ γ = E X k X ⊗ k X stands for the (non-centered) covariance operator of k X (to avoid additional notation, we also note Σ γ (x, x ) = Σ γ k γ (x, .), k γ (x , .) γ for any x, x ∈ X ) 3 . In this case, orthonormality holds true asymptotically as γ → +∞. This is guaranteed by Lemma 4.3.1 whose proof is provided in Appendix 4.B.2. Lemma 4.3.1. Consider p independent zero-mean Gaussian processes h 1 , . . . , h p with covariance

  and self-adjoint, it admits a discrete eigenspectrum. Assumption (A2) may seem strong, especially because of the non-dependence between the exponent s and x. But this assumption turns out to be mostly true when X = R D endowed with the Euclidean metric d(., .) = . -. , as stated by Proposition 4.3.2. Proposition 4.3.2. Assume X = R D and d(•, •) = • -• is the usual Euclidean distance. Let f denote the density function of X with respect to the Lebesgue measure µ of R D and assume f is continuous and

	bounded on its support. Then Assumption (A2) holds with A(x) = π D/2 f (x)/Γ (D/2 + 1) and s = D/2.

Proof of Proposition 4.3.2. Let x ∈ X and B(x, √ t) denote the ball of radius √ t and center x. Since f is a density function, f ∈ L 1 (µ) where µ is the Lebesgue measure of R D and Lebesgue's differentiation theorem entails for µ-almost every x ∈ supp(f )

  This hints at the idea that the polynomial rate of decay required by Assumption (A2) only corresponds to the case where X is of finite dimensional nature. Note that this does not imply that the new convergence results presented in Section 4.3.3 and Section 4.3.4

	,	(4.3.3)
	so that small ball probabilities in Assumption (A2) decrease to 0 at exponential rate instead of
	polynomial rate.

do not hold true when X is infinite-dimensional, even though the rates of decrease of the upper bounds might change. Besides the small ball probability in (4.3.3) only corresponds to a specific case (i.e. fractional Brownian motions). To the best of our knowledge, it seems difficult to provide a general expression that holds for most infinite-dimensional distributions, unlike that of Assumption (A2) which holds in all generality in the finite dimensional case. For this reason we restrict our analysis to finite dimensional input spaces, even if it means to preprocess infinite-dimensional data in practice through dimension reduction techniques (e.g. functional principal component analysis

[START_REF] Jones | Displaying the Important Features of Large Collections of Similar Curves[END_REF]

).

  apply the McDiarmid inequality to h 1 , . . . , h p conditionally to E M . Remark that h 1 , . . . , h p are still i.i.d. when conditioned to E M . This yields for any t > E(L | E M )

  H and the associated norm . H (defined by h H = h, h 1/2 H for any h ∈ H). Our goal is to test whether Y i is a Gaussian random element (r.e.) of H. A Gaussian r.e. in a general Hilbert space is defined similarly as a Gaussian r.e. in an RKHS (see Definition 3.1.4). 6.2.1. (Gaussian random element in a Hilbert space) Let (Ω, F , P) a measure space, (H, F ) a measurable space where H is a Hilbert space, and Y : Ω → H a measurable map. Y is a Gaussian r.e. of H if Y , h H is a univariate Gaussian random variable for any h ∈ H. Assuming that E Y ||Y || H < +∞, there exists m ∈ H such that: ∀h ∈ H, m, h H = E Y Y , h H , and a (finite trace) operator Σ : H → H satisfying: ∀h, h ∈ H, Σh, h H = cov( Y , h H , Y , h H ) .

	Definition

  Assume that P is Gaussian N (m 0 , Σ 0 ) where (m 0 , Σ 0 ) ∈ Θ 0 and ( m, Σ) are consistent estimators of (m 0 , Σ 0 ). Also assume thatE P k(Y , Y ) < + ∞ and N [m, Σ] is a continuous function of (m, Σ) on Θ 0 . Then ∆2 is a consistent estimator of ∆ 2 .Proof. First, note that μP exists since E k(Y , Y ) < +∞ implies E k1/2 (Y , Y ) < +∞. By the Law of Large Numbers in Hilbert Spaces[START_REF] Hoffmann-Jorgensen | The Law of Large Numbers and the Central Limit Theorem in Banach Spaces[END_REF], μP -→

	.4.3)
	Proposition 6.4.1 ensures the consistency of the statistic (6.4.3).
	Proposition 6.4.1.

n→∞ μP P -almost surely since E k(Y , •) -μP 2

  n } .

	Validity of the fast parametric bootstrap
	Proposition 6.4.4 hereafter shows the validity of the fast parametric bootstrap as presented in
	Section 6.4.2. The proof of Proposition 6.4.4 is provided in Section 6.B.2.
	Proposition 6.4.4. Assume E

P k1/2 (Y , Y ), Tr(Σ) and E P ||Ym 0 || 4 are finite. Also assume that T is continuously differentiable on Θ 0 . If H 0 is true, then for each b = 1, . . . , B,

  in[START_REF] Szekely | A New Test For Multivariate Normality[END_REF]) prove ongoing multivariate normality tests suffer such deficiencies. The purpose of the present section is to check if KNT displays a good behavior in high or infinite dimension.

  Two alternatives are considered: a mixture of two Simulation design. 200 simulations are performed for each test, each alternative and each n (ranging from 100 to 500). B is set at B = 250 for KNT. The test level is set at α = 0.05 for all tests.

	Gaussians with different means (µ 1 = 0 and µ 2 = 1.5 (1, 1/2, . . . , 1/d)) and same covariance
	Σ = 0.5 diag(1, 1/4, . . . , 1/d 2 ), whose mixture proportions equals either (0.5, 0.5) (alternative
	HA1) or (0.8, 0.2) (alternative HA2). Furthermore, two different cases for d are considered: d = 2
	(small dimension) and d = 100 (large dimension).

  ). References tests. KNT is compared with Random Projection (RP) test, specially designed for infinite-dimensional settings. RP is presented in Appendix 6.A.3. Several numbers of projections p are considered for the RP test : p = 1, 5 and 15.Simulation design. We set α = 0.05 and 200 repetitions have been done for each sample size.

	2nd Principal Component	-0.2 0.0 0.1 0.2 0.3
		-0.4
		-0.2	-0.1	0.0	0.1
			1st Principal Component
	Results. (Figure 6.3, bottom plots) RP is by far less powerful KNT in both cases, no matter
	how many random projections p are considered. Indeed, KNT exhibits a Type-II error near
	0 when n is barely equal to 100, whereas RP still has a relatively large Type-II error when
	n = 400. On the other hand, RP becomes more powerful as p gets larger as expected. A large

  , and Y , Z ∈ H denote two random variables with respective Borel probability measures P Y and P Z .If for every h ∈ H, Y , h = Z, h weakly then P Y = P Z .[Cue+06] suggest to randomly choose directions h from a Gaussian measure and perform a Kolmogorov-Smirnov test on Y 1 , h , . . . , Y n , h for each h, leading to the test statistic Consider the eigenexpansion of Σ = i≥1 λ i Ψ ⊗2

	Proposition 6.A.1. (Prop. 2.1 from [Cue+06]) Let H be a separable Hilbert space with inner product x∈R | Fn (x) -F 0 (x)| (6.A.15) where Fn (x) = (1/n) n i=1 1 1 { Y i ,h ≤x} is the empirical cumulative distribution function (cdf) of Y 1 , h , . . . , Y n , h and F 0 (x) = P( Y , h ≤ x) denotes the cdf of Z, h . Since [Cue+06] proved too few directions lead to a less powerful test, this can be repeated for several randomly chosen directions h, keeping then the largest value for D n (h). However the test statistic is no longer distribution-free (unlike the univariate Kolmogorov-Smirnov one) when the number of directions is larger than 2. •, • D n (h) = sup 6.B Proofs

Throughout this section, ., . (resp. ||.||) denotes either ., . H or . , . H( k) (resp. ||.|| H or ||.|| H( k) ) depending on the context. 6.B.1 Proof of Propositions 6.4.2 and 6.4.3 i where λ 1 , λ 2 , . . . is a decreasing sequence of positive reals and where {Ψ i } i≥1 form a complete orthonormal basis of H.

  Lemma 6.B.4).

	Lemma 6.B.4 rewrites the bound in (6.B.28) as
	exp -	ns 2 2ϑ 2 + (2/3)Mϑs	1 +	3n 2Mϑ	exp	3| q -q| 2Mϑ	1 1 s≥0 | ŝ -s| ,	(6.B.29)

  : kernel methods, rkhs, normality test, outlier detection Modèles Gaussiens et Méthodes à Noyaux Résumé Les méthodes à noyaux ont été beaucoup utilisées pour transformer un jeu de données initial en les envoyant dans un espace dit « à noyau » ou RKHS, pour ensuite appliquer une procédure statistique sur les données transformées. En particulier, cette approche a été envisagée dans la littérature pour tenter de rendre un modèle probabiliste donné plus juste dans l'espace à noyaux, qu'il s'agisse de mélanges de gaussiennes pour faire de la classification ou d'une simple gaussienne pour de la détection d'anomalie. Ainsi, cette thèse s'intéresse à la pertinence de tels modèles probabilistes dans ces espaces à noyaux. Dans un premier temps, nous nous concentrons sur une famille de noyaux paramétrée -la famille des noyaux radiaux gaussiens -et étudions d'un point de vue théorique la distribution d'une variable aléatoire projetée vers un RKHS correspondant. Nous établissons que la plupart des marginales d'une telle distribution est asymptotiquement proche d'un « scale-mixture » de gaussiennes -autrement dit une gaussienne avec une variance aléatoire -lorsque le paramètre du noyau tend vers l'infini. Une nouvelle méthode de détection d'anomalie utilisant ce résultat théorique est introduite. Dans un second temps, nous introduisons un test d'adéquation basé sur la Maximum Mean Discrepancy pour tester des modèles gaussiens dans un RKHS. En particulier, notre test utilise une procédure de bootstrap paramétrique rapide qui permet d'éviter de ré-estimer les paramètres de la distribution gaussienne à chaque réplication bootstrap.

Mots clés : méthodes à noyaux, rkhs, test de normalité, détection d'anomalie Laboratoire de Mathématiques Paul Painlevé UMR 8524 CNRS -Université Lille 1 -59655, Villeneuve d'Ascq Cedex -France

The picture in Figure2.3 was taken from http://www.gatsby.ucl.ac.uk/~gretton/coursefiles/Slides4A.pdf

P X ⊗ P Y is defined as the measure on X × Y such that for every measurable A ⊆ X and B ⊆ Y , P X ⊗ P Y (A × B) = P X (A)P Y (B).

In general, the pseudo-inverseM † of a matrix M ∈ M n,m (R) is defined by M † = lim δ→0 (M T M + δI m ) -1 M T = lim δ→0 M(MM T + δI n ) -1 .

A Rademacher random variable ξ is a {-1, 1}-valued r.v. such that P(ξ = 1) = P(ξ = -1) = 1/2.

Note that the metric d must be chosen such that the resulting kernel k γ is definite positive. This condition is fulfilled in the general case where d(x, x ) = ϕ(x)ϕ(x ) H for any x, x ∈ X , where ϕ : X → H and H is a separable Hilbert space.

Given two vectors u, v ∈ H where H denotes some prehilbertian space with dot product ., . , the tensor product u ⊗ v is defined as the operator u ⊗ v : H → H such that for any w ∈ H, (u ⊗ v) • w = v, w • u.

Here ., . p denotes the usual inner product of R p

A function : X × X → R is of negative type if for every x 1 , . . . , x n ∈ X and a 1 , . . . , a n ∈ R such that n i=1 a i = 0, n i,j=1 a i a j (x i , x j ) 0.

https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)

A kernel k : H × H → R is said integrally strictly definite positive iff k(x, y)µ(dx)µ(dy) > 0 for every finite non-zero signed Borel measure µ on H.

The density f would appear in the computations through the equivalent term given in Lemma A.2.1 combined with Proposition 4.3.2

Remerciements

rate. The second case is when P is concentrated homogeneously across its support -ideally when P is a uniform distribution. In this situation, a small a entails that supp(P ) is a large subset of R d hence the inliers sample X 1 , . . . , X n tends to be sparsely spread across supp(P ) so that it is more difficult for our OD algorithm to determine the boundary of supp(P ), which leads to poor performance.

The shape of the cdf F in the vicinity of 0 also influences the bound of β n in the following sense. If F(t) decreases to 0 when t → 0 at a large speed, then the upper bound decreases to 0 at a faster rate as well hence better performances. On the other hand, a fast decrease of F(t) near t = 0 means that outliers tend to lie far from the boundary of supp(P ). Therefore a small missed detection rate is expected, which is confirmed by the bound.

As for the confidence level α, reducing α makes the upper bound larger. In hypothesis testing terms, a smaller Type-I error α is supposed to make the Type-II error β n larger. Thus the bound reflects well the expected influence of α on β n .

Numerical results

False alarm and missed detection rates

In this section, we empirically observe the false alarm and missed detection rates on simulated data, in particular we discuss on the influence of the parameters γ and p on those two types of errors.

We apply our OD procedure on the synthetic dataset "spiral" displayed in Figure 5 outliers) are averaged out to get false alarm rates (resp. missed detection rate).

Figure 5.2 shows the false alarm rates obtained when applying our OD method on the 200 test inliers, for several values of γ ranging from 0.1 to 3000 and several values of p ranging from 2 to 500. The prescribed level of confidence was set at α = 0.05, so that false alarm rates close to 0.05 are expected. For γ up to 75 and p smaller than 10, the actual false alarm rates are satisfyingly close to 0.05. However when γ takes values beyond γ = 100, the false alarm rate is no longer controlled, being as high as 0.805 when γ = 3000 and p = 500. This is consistent with Theorem 4.3.5 which implies that γ must not be too large compared to n in the empirical setting. On the other hand, the bound of Theorem 4.3.5 also suggested that a larger p had a bad impact on the distributional approximation of S n (X) and in practice, the control of the type-I error seems to be significantly worsened by larger values of p, especially when γ 25.

Figure 5.3 displays the corresponding missed detection rates. As foretold by Theorem 5.3.2, the missed detection rate tends to 0 whenever γ → +∞ and decreases when p increases. Note Besides, each method is applied several times for several different values of its parameter, and the chosen parameters are those that maximize the AUC (area under curve). Namely, our method is parameterized by the pair (γ, p) that has optimal values (100, 50) for the "spiral" dataset and (0.05, 100) for the "cancer" dataset. Oc-SVM is only parameterized by γ that is optimal at γ = 3 for "spiral" and 0.05 for "cancer". Finally kernel PCA is parameterized by (γ, p) where p is the number of kept principal component, with optimal values (50, 50) for "spiral" and (0.8, 5) for "cancer".

Obtained ROC curves are displayed in Figure 5.8. One important observation to make is that the ROC curve of oc-SVM does not go below a certain level of false positive rate. For instance, it stays above approximately 0.07 for "spiral" and 0.10 for "cancer". Remember that ν controls the proportion of declared outliers in the training set. However when ν is set close to 0, the actual proportion of declared outliers among the tested actual outliers tends to increase back to large values instead of tending to 0. This shows empirically the lack of control on the false alarm error when using oc-SVM.

As for performance, it can be readily observed that oc-SVM yields a smaller AUC than kernel PCA -as expected -and our method on both datasets. Depending on the dataset, kernel PCA achieves a slightly larger AUC ("spiral" dataset) or our method performs better ("cancer" dataset).

In either case, the gap between the two methods is relatively small.

In order to decide between our method and kernel PCA, let us take a look at computation costs.

Execution time

In Figure 5.9, the execution time (in seconds) of the three competing methods are represented with respect to n. In addition, the version of our procedure using Nyström approximation and Random Kitchen Sinks are also represented. The parameter r -the rank of the low-rank approximation of the Gram matrix in Nyström method and RKS -is set at r = 50. As expected, kernel PCA is the most time-consuming method, with a running time up to 5 times larger than our procedure when n = 3000. Nyström method and RKS improves on the computational cost of our method, reducing the execution times up to about 2 times. Finally, oc-SVM is the fastest of all compared methods, achieving a 2 seconds running time when n = 3000.

As discussed in Section 5.1.2 when introducing kernel PCA, the high computational cost of kernel PCA is due to the eigen-decomposition of the Gram matrix which is of order O(n 3 ). On the other hand, we saw in Section 5.2.2 that the most time-consuming step in our procedure is the estimation of the p-value PV(X) because of the computation of the quantities Σ γ,-i (X i , X i ) for i = 1, . . . , n, which all in all costs O(n 2 ) in time but still outperforms kernel PCA computationally speaking. Nyström method or RKS reduces this computational cost down to O(nr 2 ) where r is the rank of the approximate Gram matrix. Note that Nyström method or RKS can also be applied to kernel PCA, however using a low-rank approximate Gram matrix restricts the number of possible kept principal components. We computed the KS(γ j , p l ) for the "spiral" dataset of section 5. For each fixed p, the curve { KS(γ j , p)} j describes a convex function, hence admitting a global minimum at some γ = γ * (p). Hence it would be natural to choose γ * (p) as the optimal γ for a fixed p. However the statistics s 1 , . . . , s n 0 were calculated on the basis of an inlier sample of smaller size nn 0 < n. To get a more accurate optimal γ for an inlier sample of full size n, we propose to multiply γ * (p) by a correction term (n/(nn 0 )) 1/(2s) where s is the same quantity occurring in (5.3.8). Indeed in Section 5.3.1 it was suggested that the optimal γ had the form γ * = Cn 1/(2s) for some constant C > 0. Finally the optimal pair (γ * , p * ) is chosen as

Note that when X = R D endowed with the Euclidean metric, s is readily given as s = D/2 according to Proposition 4.3.2 in Chapter 4.

In our simulations, we found p * = 2 and γ * = (500/250) 1/2 68 96.17 as optimal values.

According to the results in Figure 5.3, this corresponds to a false alarm rate around 0.062 and a Algorithm 2 Sequential selection of covariance rank

2. If H 0,r is rejected and r < r max , set r = r + 1 and return to 1. The bound in (6.6.10) decreases to 0 only if α converges to 0 but at a slow rate. Indeed, the Type-II errors β r (α) grow with decreasing α but converge to zero when n → +∞. For instance in the case of the sequential tests mentioned in [START_REF] Ratsimalahelo | Strongly consistent determination of the rank of matrix[END_REF] and [START_REF] Robin | Tests of Rank[END_REF], the correct rate of decrease for

Empirical performances

In this section, the sequential procedure to select covariance rank (as presented in Section 6.6.1) is tested empirically on synthetic data.

Dataset A sample of n zero-mean Gaussian with covariance Σ r * are generated, where n ranges from 100 to 5000. Σ r * is of rank r * = 10 and its eigenvalues decrease either polynomially (λ r = r -1 for all r ≤ r * ) or exponentially (λ r = exp(-0.2r) for all r ≤ r * ).

Benchmark To illustrate the level of difficulty, we compare our procedure with an oracle procedure which uses the knowledge of the true rank. Namely, the oracle procedure follows our sequential procedure at a level α oracle defined as follows

where n ∆2 r is the observed statistic for the r-th test and Z r follows the distribution of this statistic under H 0,r . Hence α oracle is chosen such that the true rank r * is selected whenever it is possible.

Simulation design To get a consistent estimation of r * , the confidence level α must decrease with n and is set at α = α n = exp(-0.125n 0.45 ). Each time, 200 simulations are performed.

Results

The top panels of Figure 6.4 display the proportion of cases when the target rank is found, either for our sequential procedure or the oracle one. When the eigenvalues decay polynomially, the oracle shows that the target rank cannot be almost surely guessed until Probabilities of finding the right rank with respect to n for our sequential test (• red) and the oracle procedure ( blue); bottom half: probabilities of overestimating the true rank with the sequential procedure compared with fixed alpha (+ green). In each case, two decreasing rate for covariance eigenvalues are considered : polynomial (left column) and exponential (right column). n = 1500. When n ≤ 1500, our procedure finds the true rank with probability at least 0.8 and quickly catches up to the oracle as n grows. In the exponential decay case, a similar observation is made. This case seems to be easier, as our procedure performs almost as well as the oracle when n ≥ 600. In all cases, the consistency of our procedure is confirmed by the simulations.

The bottom panels of Figure 6.4 compare α with the probability of overestimating r * (denoted by p + ). As noticed in Section 6.6.1, the former is an upper bound of the latter. But we must check empirically whether the gap between those two quantities is not too large, otherwise the sequential procedure would be too conservative and lead to excessive underestimation of r * . In the polynomial decay case, the difference between α and p + is small, even when n = 100. The gap is larger in the exponential case but gets narrower when n 1500.

Appendix A

Technical lemmas

A.1 Some useful concentration inequalities Lemma A.1.1 (McDiarmid's inequality). Let X 1 , . . . , X p be i.i.d. random variables taking values in some set X . Let L : X p → R be some real-valued function and assume there exists there exists c 1 , . . . , c p > 0 such that ∀j = 1, . . . , p, sup

x 1 ,...,x j ,x j ,...,x p ∈X |L(x 1 , . . . , x j , . . .) -L(x 1 , . . . , x j , . . .)| c j .

Then for any t > 0 P L(X 1 , . . . , X p ) -EL(X 1 , .