Je Remercie

Romuald Elie

Je Tiens À Remercier Également

Claire Lacour

Katia Meziani

Monsieur Mohamed Ben Alaya

Monsieur Julien Barral

Audrey Patout

Sylvie Cach

Mariam Sidibé

Keywords: Regression, heteroscedastic model, variance function, agregation, reject option, prediction interval, plug-in rule. vii

ma plus profonde gratitude à mes directeurs de thèse, Christophe

La première partie est consacrée à l'estimation de la fonction de variance à travers les procédures d'agrégation classiques. Plus précisément, nous nous intéressons à l'estimation de la fonction de variance par agrégation de type sélection modèle (MS) et convexe (C). Le but de la procédure MS est de sélectionner le meilleur estimateur parmi un ensemble de prédicteurs, celui de C consiste à sélectionner la meilleure combinaison convexe parmi les prédicteurs. Les prédicteurs sélectionnés sont alors appelés MS et C-estimateurs respectivement. La construction de MS et C-estimateurs repose sur une procédure en deux étapes. Dans une première étape, à partir d'un premier échantillon, nous construisons des estimateurs de la fonction de variance par la méthode basée sur les erreurs résiduelles. Dans une deuxième étape, nous les agrégeons à l'aide d'un deuxième échantillon. Nous établissons la consistance de MS-estimateur et de C-estimateur vis-à-vis du risque L 2 et illustrons ses performances numériques sur simulations.

La deuxième partie est dédiée à des applications où la fonction de variance joue un rôle crucial. Elle est divisée en deux chapitres. Le chapitre 3 est consacré à l'étude du problème de la régression où l'on est autorisé à s'abstenir de prédire. Nous nous concentrons sur le cas où le taux de rejet est fixe et dérivons la règle optimale qui repose sur le seuillage de la fonction de variance conditionnelle. Nous fournissons une procédure d'estimation semi-supervisée de la règle optimale impliquant deux ensembles de données : un premier ensemble de données étiquetées est utilisé pour estimer à la fois la fonction de régression et la fonction de variance conditionnelle ; un deuxième ensemble de données non étiquetées est exploité pour calibrer le taux de rejet à celui souhaité. Le prédicteur avec option rejet résultant s'avère presque aussi bon que le prédicteur optimal avec l'option rejet, à la fois en termes de risque et de taux de rejet. En outre, nous appliquons notre approche avec l'algorithme des K-plus proches voisins (K-PPV) et établissons des vitesses de convergence pour le prédicteur K-PPV avec option rejet résultant. Enfin, une étude numérique est réalisée pour illustrer l'intérêt de l'utilisation de la procédure proposée. Le chapitre 4 s'intéresse à la construction d'un intervalle de prédiction dans le modèle hétéroscédastique gaussien. L'intervalle optimal repose sur le seuillage de la densité conditionnelle de la loi gaussienne. La construction de cet intervalle repose sur une procédure d'estimation semi-supervisée en deux étapes (comme dans le Chapitre 3). Dans une première étape, à partir d'un premier échantillon de données étiquetées, nous estimons la densité conditionnelle en utilisant la règle plug-in. Plus précisément, l'estimateur de la densité conditionnelle est basé sur les estimateurs de la fonction de régression et de la fonction de variance. Dans une deuxième étape, nous calibrons le seuil à l'aide d'un deuxième échantillon de données non étiquetées. L'intervalle de prédiction construit offre des garanties de consistance et de bonnes performances numérique. De plus, nous établissons des vitesses de convergence pour l'intervalle de prédiction basé sur l'algorithme des K-PPV. Par ailleurs, une comparaison numérique à l'approche basée sur les conformal predictors suggère que notre méthode est plus stable et plus adaptée à des applications réelles où le nombre d'observations est petit.

Mots clefs:

Régression, modèle hétéroscédastique, fonction de variance, agrégation, option rejet, intervalle de prédiction, règle plug-in.

Quelques notations

• • : la distance euclidienne sur R d . • • 2 n := 1 n n i=1 (•) 2 .
• a n b n : a n ≤ cb n où c est une constante indépendante de n.

• a n ≍ b n : a n b n et b n a n .

• a n ∝ b n : a n = cb n où c est une constante indépendante de n.

• ⌈•⌉ : ⌈a⌉ est le plus grand entier ≤ a.

• [p] := {1, . . . , p} pour tout entier p ≥ 2.

• Λ p := {λ ∈ R p : λ j ≥ 0, p j=1 λ j = 1}.

• P X : la distribution marginale de X ∈ R d .

• ℓ p -norme sup : (sup x∈C |f (x)|) p où f : C ⊂ R d → R.

• log(n) signifie que l'inégalité est valable jusqu'à des constantes et des facteurs logarithmiques.

Chapitre 1

Introduction

La méthode de régression a été largement utilisée en statistiques. L'objectif général de la régression est d'expliquer une variable aléatoire Y , dite réponse, en fonction d'une variable explicative X, qui peut être aléatoire ou déterministe. Le but est de prédire Y à toute nouvelle entrée X. Dans l'analyse de régression, il existe deux principaux modèles : le modèle homoscédastique dans lequel la variance des erreurs de la régression est constante, et le modèle hétéroscédastique qui correspond au cas où la variance des erreurs dépend de X. Dans ce manuscrit, nous considérons le modèle de régression hétéroscédastique. Sur le plan statistique, cette hétéroscédasticité doit être détectée et estimée car sa non prise en compte dans l'estimation invalide les conclusions de nombreux problèmes d'inférence statistique tels que les tests statistiques qui supposent que les erreurs du modèle ont toutes la même variance. Un exemple classique en statistique qui illustre l'importance de la prise en compte de l'hétéroscédasticité est apparu dans la régression linéaire hétéroscédastique. L'estimateur des moindres carrés ordinaires (MCO) est non biaisé mais il n'est pas efficace. Il existe un estimateur des moindres carrés généralisés (c'est un estimateur des MCO sur un modèle transformé dont les aléas sont homoscédastiques et non autocorrélés) qui est généralement plus efficace que l'estimateur des MCO. En outre, les intervalles de prédiction obtenus à l'aide d'une approche qui estime la variance des erreurs en fonction des variables d'entrée sont susceptibles d'être plus réalistes que ceux obtenus en supposant que la variance des erreurs est constante, puisque l'estimation de l'incertitude prédictive dépend de l'estimation de la variance de la variable de réponse. Un autre aspect important de l'estimation de l'hétéroscédasticité du modèle est que l'estimation ponctuelle de la fonction de régression est directement liée à la variance en ce point. Malgré son importance à différents niveau, l'étude statistique du problème d'estimation de la fonction de variance est moins étudiée que celle de la fonction de régression dans la littérature statistique. Enfin, il faut noter que la fonction de variance a été étudiée, pas seulement en statistiques, mais dans de nombres domaines tels que la finance, l'économie et le traitement de signal. En particulier, la volatilité de la variable étudiée est un paramètre important à considérer.

Dans ce manuscrit, nous nous intéresserons au modèle de régression hétéroscédastique et élaborerons des méthodes statistiques capables de s'adapter aux différentes structures/hypothèses que l'on imposera à ce modèle. Le premier objectif de cette thèse est de proposer et d'étudier des estimateurs de la fonction de variance basés sur l'agrégation. Bien que le problème d'estimation de la fonction de variance est un objectif en soi sur le plan statistique que nous considérons dans cette thèse, un autre aspect de ce manuscrit est de pointer des problèmes où l'estimation de la variance est cruciale et enfin de proposer des solutions à ces problèmes basées sur l'estimation de celle-ci. En particulier, nous nous focaliserons sur les problèmes suivants :

• la régression avec option rejet : nous introduisons le problème de régression avec option rejet qui vise à construire des procédures d'estimation capables de ne pas prédire lorsque le doute dans la valeur prédite est trop grand. Il apporte entre autres une meilleure compréhension des situations où il est judicieux d'avoir recours à une méthode autorisant le rejet. Le prédicteur avec option rejet résultant repose sur un seuillage de la fonction de variance. Autrement dit, le fait de rejeter ou non dépend essentiellement de la fonction de variance : plus le bruit en une région de l'espace est fort, plus on a tendance à s'abstenir de prédire. Ainsi, en marge de l'étude du problème de rejet, nous proposons une estimation de ce prédicteur fondée sur une procédure semi-supervisée d'estimation de type plug-in et en étudiant ses propriétés statistiques.

• Intervalle de prédiction : l'objectif est de construire un intervalle de prédiction qui contient la variable à prédire avec grande probabilité dans le modèle gaussien hétéroscédastique. L'intervalle de prédiction optimal repose sur un seuillage de la densité conditionnelle qui dépend des fonctions de régression et de variance. L'estimation de cet intervalle repose sur une règle de type plug-in. En particulier, l'estimateur final dépend des estimateurs des fonctions de régression et de variance.

Modèle de régression

Dans ce chapitre, nous commençons par introduire le modèle de régression. Puis, nous présentons les estimateurs de la fonction de régression les plus utilisés dans la littérature, en particulier nous développons la procédure d'agrégation qui est un outil puissant utilisé dans le cadre de l'estimation de la fonction de régression. S'ensuit une présentation de l'estimation de la fonction de variance dans le cadre général. Une partie importante de l'introduction est consacrée à une présentation détaillée de l'apprentissage sous contrainte : l'option rejet et les intervalles de confiance. Enfin, nous décrivons les contributions de la thèse.

Modèle de régression

Le problème de régression est l'un des sujets les plus populaires et les plus classiques en statistique [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Scornet | Consistency of random forests[END_REF][START_REF] Stone | Consistent nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. Les modèles de régression étudient la relation entre la variable dite de réponse et une ou plusieurs variables explicatives. Par exemple, nous pouvons nous intéresser à l'influence de la vitesse sur le nombre d'accidents sur la route. Dans ce cas, le praticien s'intéresse à expliquer une variable à travers une autre. Afin de formaliser le problème, nous considérons un couple de variables aléatoires (X,

Y ) ∈ R d × R où X ∈ R d est la variable explicative et Y ∈ R est la variable à prédire associée à l'entrée X telle que Y = f * (X) + ξ (1.1) où • f * (x) = E[Y |X = x],
x ∈ R d , est appelée fonction de régression de Y sur X,

• ξ est le bruit dans le modèle qui satisfait les propriétés suivantes :

E[ξ|X] = 0 et E[ξ 2 ] < ∞.
Dans la suite, pour tout x ∈ R d , nous définissons

σ 2 (x) = E (Y -f * (X)) 2 |X = x
la fonction de variance conditionnelle.

Chapitre 1: Introduction Problème de régression : étant donné X ∈ R d , le but est de trouver la bonne prédiction de Y en supposant que (X, Y ) ∼ P où P est inconnue. Cela signifie que l'on souhaite trouver une fonction mesurable f : R d → R telle que f (X) approxime bien Y . Pour cela, nous introduisons le risque L 2 de f E |Yf (X)| 2 que nous voulons le plus petit possible avec l'hypothèse suivante E[Y 2 ] < ∞.

Pour toute fonction mesurable f , le risque L 2 de f peut être décomposé comme suit :

Lemme 1. Pour toutes les fonctions mesurables f :

R d → R avec E f 2 (X) < ∞, E |Y -f (X)| 2 -E |Y -f * (X)| 2 = E (f (X) -f * (X)) 2 .
(1.2)

Démonstration. Il suffit de remarquer que

E [(Y -f * (X))(f * (X) -f (X))] = E[E [(Y -f * (X))(f * (X) -f (X))|X]] = E[(f * (X) -f (X))E [(Y -f * (X))|X]] = E[(f * (X) -f (X))(E [Y |X] -f * (X))] = 0.
Le terme du membre de droite de l'Eq.(1.2) est appelé l'excès de risque ou l'erreur L 2 de f . Nous pouvons déduire du Lemme 1 que la fonction de régression f * est bien le meilleur prédicteur de la variable de réponse Y basé sur X au sens du risque L 2 :

R := E |Y -f * (X)| 2 = inf f E |Y -f (X)| 2
où l'infimum est pris sur toutes les fonctions mesurables f : R d → R. La quantité R est appelé la variance résiduelle et est l'erreur quadratique moyenne minimale de tout prédicteur de Y basé sur l'observation de la variable d'entrée X.

Et après ? Typiquement, la distribution de (X, Y ) et la fonction de régression f * sont inconnues. En particulier, il est impossible de prédire Y en utilisant f * (X). Au lieu de cela, nous observons un échantillon D n = {(X i , Y i )} n i=1 qui consiste en n copies indépendantes et identiquement distribuées (i.i.d.) de (X, Y ). L'objectif de la régression est d'utiliser cet échantillon pour construire un estimateur fn : R d → R de f * dont le risque L 2 est faible. Dans la section suivante, nous présentons les estimateurs de la fonction de régression.

Estimations de la fonction de régression

La littérature sur les méthodes d'estimation de la fonction de régression f * est vaste et impossible à recenser. Nous citons ici quelques références pertinentes pour la suite de l'exposé [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Scornet | Consistency of random forests[END_REF][START_REF] Stone | Consistent nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. Dans tout ce manuscrit, nous plaçons dans le cadre de la régression non-paramétrique [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. En somme, nous pouvons distinguer deux types d'estimateurs : les estimateurs linéaires et non-linéaires.

Estimateurs linéaires

Une méthode d'estimation de la fonction de régression f * à partir de l'échantillon d'apprentissage D n consiste à proposer un estimateur fn défini comme suit fn (x) = n i=1 W i (x)Y i , où les poids réels W i (x) vont être des fonctions boréliennes de x et X 1 , . . . , X n . Dans cette répresentation, fn (x) se présente comme une moyenne pondérée des Y i correspondants aux X i situés dans un voisinage de x. Ce voisinage est caractérisé par les poids W i (x). Cet estimateur est appelé estimateur par moyennage locale. Le théorème ci-dessous, initialement dû à Stone [START_REF] Biau | On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification[END_REF], fournit des conditions suffisantes sur les poids W i (x), pour assurer la consistance, dite universelle, de l'estimateur fn . Théorème 2 (Theorem 10.1 [START_REF] Biau | On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification[END_REF]). Supposons que les poids satisfassent les conditions suivantes quelle que soit la loi de X : (iii) Pour tout a > 0, n i=1 |W i (X)|1 { X i -X >a} converge vers 0 en probabilité.

(iv) n i=1 W i (X) converge vers 1 en probabilité.

(v) max 1≤i≤n W i (X) converge vers 1 en probabilité.

Alors fn est universellement L 2 -consistante , i.e.,

E | fn (X) -f * (X)| 2 → 0 quand n → ∞ (1.3) quelle que soit la loi du couple (X, Y ) avec E[|Y | 2 ] < ∞.
Les poids W i (x) peuvent être positifs et normalisés à 1, de telle sorte que (W 1 (x), . . . , W n (x)) peut être assimiler à un vecteur de probabilités. Dans la suite de ce paragraphe, nous présentons des exemples classiques d'estimateurs linéaires de la fonction de régression qui vérifient les conditions du théorème de Stone.

• L'algorithme des k plus proches voisins. Les k-plus proches voisins (k-Nearest neighbors (k-NN) en anglais) est une idée intuitive et est un exemple d'estimateurs linéaires de la fonction de régression f * . Pour estimer f * en un point x ∈ R d , nous regardons les k (k un entier strictement positif compris entre 1 et n) observations X i les plus proches de x et nous considérons comme valeur prédite en x la moyenne des Y i correspondants. L'estimateur résultant s'appelle estimateur des k-plus proches voisins qui est un estimateur de type moyenne locale. Nous désignons la suite (X (1) (x), Y (1) (x)), . . . , (X (n) (x), Y (n) (x)), la permutation de (X 1 , Y 1 ), . . . , (X n , Y n ) correspondante aux distances croissantes des X ix1 . En d'autres termes,

X (1) -x ≤ . . . ≤ X (n) -x .
Dans la suite nous supposerons que les égalités entre distances X ix = X jx se produisent avec probabilité zéro, ce qui est le cas lorsque la variable aléatoire Xx admet une densité par rapport à la mesure de Lebesgue. L'estimateur des k-plus proches voisins s'écrit donc

fknn (x) = 1 k n kn i=1 Y (i) (x), (1.4)
pour tout x ∈ R d . Notons que dans cette expression, k = k n est autorisé à dépendre de n. La consistance et la vitesse de convergence de l'erreur L 2 de fknn ont été étudiées dans de nombreux travaux (voir par exemple [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF]). En particulier, nous pouvons établir le résultat suivant de vitesse de convergence.

Chapitre 1: Introduction Théorème 3 (Theorem 6.2 [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF]). Soit d ≥ 3. Supposons que la variable X est bornée, que sup x∈R d σ 2 (x) ≤ L σ 2 et que f * est L-Lipschitz, c-à-d : il existe L > 0 telle que

|f * (x) -f * (z)| ≤ L x -z ∀x, z ∈ R d .
Alors pour k n ∝ n 2/(d+2) , nous avons E[( fknn (X)f * (X)) 2 ] n -2/(d+2) .

Pour d ≤ 2, la vitesse de convergence du Théorème 3 est également vérifiée sous la condition de forte densité [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF] : la distribution marginale P X admet une densité µ par rapport à la mesure de Lebesgue, telle que pour tout x ∈ R d , nous avons 0 < µ min ≤ µ(x) ≤ µ max < ∞. Le paramètre k joue un rôle important dans la performance de l'estimateur fknn . Le Théorème cidessus fournit une valeur de k qui n'est pas accessible en pratique (la constante dépend de paramètres inconnus). Diverses méthodes de choix de k sont donc disponibles. En pratique, la méthode utilisée le plus souvent est la validation croisée (pour plus de détails voir [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF]). Cette approche consiste à introduire un ensemble fini K n := {1, . . . , n} qui servira de grille pour le paramètre k. Puis, pour chaque k ∈ K n , nous calculons les deux estimateurs f (k) knn := fknn et f (k) knn,i de la fonction de régression sur D n et D n \{(X i , Y i )} respectivement. La sélection par validation croisée (nous considérons ici la version leave-one-out de la validation croisée) de k est alors donnée par

kcv = argmin k∈Kn 1 n n i=1 (Y i - f (k) knn,i (X i )) 2 .
Enfin, l'estimateur par validation croisée est donné par fknn (x) = f ( kcv ) knn (x). • Méthode à noyau. Les estimations à noyau ont été étudiées à l'origine dans le cadre de l'estimation de la fonction de densité par Rosenblatt [START_REF] Rosenblatt | Remarks on some nonparametric estimates of a density function[END_REF] et Parzen [70]. Ils ont été étendus au problème de régression par Nadaraya [START_REF] Nadaraya | On estimating regression[END_REF]66] et Watson [START_REF] Watson | Smooth regression analysis[END_REF]. La méthode à noyau donne un estimateur linéaire. Définition 4. On appelle noyau une fonction K : R d → [0, +∞[ telle que K = 1 et K 2 < ∞. Définition 5. Soit K un noyau et h > 0 un paramètre que l'on appellera fenêtre. L'estimateur de Nadaraya-Watson de f * est donné par

fNW (x) = n i=1 W N W i (x)Y i , où W N W i (x) = K X i -x h n j=1 K X j -x h 1 n j=1 K X j -x h =0 .
Le noyau K détermine la forme du voisinage autour du point x. Il est souvent pertinent de considérer des noyaux réguliers, ce qui permet d'obtenir des estimateurs réguliers à leur tour, par exemple

• le noyau gaussien : K(x) = 1 2π exp -x 2 2 ;

• le noyau d'Epanechnikov : K(x) = 3 4 1x 2 2 1 { x ≤1} . La fenêtre h := h n est le paramètre de régularisation du modèle qui dépend de n en général. Elle a un rôle crucial dans la consistance de l'estimateur fNW au sens du l'erreur L 2 au même titre que le paramètre k pour les k-ppv. Ainsi, les conditions suivantes

h n → 0 et nh n → ∞ lorsque n → ∞
sont minimales afin d'établir la consistance, par exemple au sens de l'erreur L 2 de l'estimateur fNW (Théorème 2). Dans ce cas également, de nombreuses méthodes de calibration de h n existent telles 1.2 Estimations de la fonction de régression que la validation croisée, plug-in ou Boostrap [START_REF] Pagan | Nonparametric Econometrics[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF]. Chaque méthode possède des avantages et des inconvénients. L'idée principale de la validation croisée est de minimiser le critère suivant sur h

CV ( fNW ) = 1 n n i=1 Y i - f (-i) N W (X i ) 2 , où f (-i)
N W est l'estimateur de la régression qui ne tient pas compte l'observation i. Le résultat suivant précise la vitesse de convergence de E ( fNW (x)f * (x)) 2 P X (dx) dans le cas du noyau naïf (K(x) = 1 { x ≤1} ). Théorème 6 (Theorem 5.2 [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF]). Supposons que la variable X est bornée, sup x∈R d σ 2 (x) ≤ L σ 2 et que f * est L-lipschitzienne. Alors pour h n ∝ n -1 d+1 , nous avons

E ( fNW (x) -f * (x)) 2 P X (dx) n -2 d+2 .
Remarque 7 ( Estimation par des polynômes locaux [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]). L'estimateur à noyau fNW peut s'écrire comme solution du problème de minimisation suivant :

fNW (x) = argmin θ n i=1 (Y i -θ) 2 K X i -x h n .
L'estimateur de NW apparaît ainsi comme un cas particulier de l'estimateur par polynômes locaux (LL) au point x fLL (x) = M l=0 θl (x)φ l (x) , où ( θ0 , . . . , θM ) = argmin

(θ 0 ,...,θ M ) n i=1 Y i - M l=0 θ l φ l (X i ) 2 K X i -x h n ,
et φ 1 , . . . , φ M sont des fonctions sur R d . La vitesse de convergence de fLL se dégrade très rapidement avec la dimension d. Cette propriété est connue sous le nom de fléau de la dimension. L'estimateur fLL est bien étudié dans le cas où d = 1 [START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] en prenant φ l = x l . Si M = 0, on revient au cadre de l'estimateur à noyau. Si M = 1, on parle de la régression linéaire locale.

Estimateurs non-linéaires

Nous pouvons citer les procédures non-linéaires populaires : les machines à vecteurs de support (support vector machines (SVM) en anglais) et les forêts aléatoires (random forest en anglais).

• Régression par Machines à Vecteurs de Support (SVM) [START_REF] Billings | Time series prediction using support vector machines, the orthogonal and the regularized orthogonal least-squares algorithms[END_REF][START_REF] Durbha | Support vector machines regression for retrieval of leaf area index from multiangle imaging spectroradiometer[END_REF][START_REF] Hastie | The elements of statistical learning: Data Mining, Inference, and Prediction[END_REF][START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]. Les SVM, initialement introduites dans le cadre de la classification, peuvent être étendues au problème de régression en conservant toutes les propriétés qui caractérisent l'algorithme. Dans le cas des SVM linéaires, la fonction de prédiction prend la forme f (x) := f (x, ω) =< x, ω > +b , avec x ∈ R d est la variable d'entrée, ω ∈ R d est le vecteur des poids et b est une constante à déterminer. Nous cherchons la relation affine f avec la plus petite valeur de ω. Nous résolvons donc le problème d'optimisation suivant :

min 1 2 ω 2 sous la contrainte |y i -f (x i )| ≤ ε, i = 1, . . . , n.
Chapitre 1: Introduction Ce problème n'ayant pas forcément de solution réalisable lorsque ε est trop petit. Les SVM utilisent une fonction de perte appelée ε-intensible (au fonction de seuillage) de la forme

|y -f (x)| ε = 0 si |y -f (x)| ≤ ε, |y -f (x)| -ε sinon.
où ε ≥ 0. Cela nous conduit à minimiser :

1 2 ω 2 + c n i=1 |y i -f (X i )| ε , sous des contraintes de type |y i -f (X i )| ε = |y i -f (X i )| -ε.
(ξ i + ξ * i ) , sous les contraintes    y i -f (X i ) ≤ ε + ξ * i f (X i ) -y i ≤ ε + ξ i ξ i , ξ * i ≥ 0, i = 1, .
. . , n. La constante c > 0 est un hyperparamètre qui permet de régler le compromis entre la quantité d'erreur autorisée et la quantité 1 2 ω 2 . La solution de ce problème est donnée en minimisant le Lagrangien suivant

L = ω 2 +c n i=1 (ξ i +ξ * i )- n i=1 (η i ξ i + η * i ξ * i )- n i=1 α i (ε + ξ i -y i + f (X i ))- n i=1 α * i (ε + ξ * i + y i -f (X i ))
où les η i , η * i , α i et α * i sont positifs et représentent les multiplicateurs de Lagrange. Nous obtenons alors la forme explicite suivante :

f (x) = n i=1 (α i -α * i ) < X i , x > +b
où la constante b peut se calculer grâce aux conditions de Karush-Kuhn-Tucker (KKT). La fonction de prédiction f dépend des X i et x à travers les produits scalaires < X i , x > et peut donc s'étendre à des formes plus générales. Ceci fait le lien avec les SVM non-linéaires. Dans le cas des SVM nonlinéaires, nous introduisons un espace S (appelé espace de redescription (feature space)) et un noyau

k(x i , x) =< Φ(x i ), Φ(x) > où Φ : R d → S une projection. Comme précédemment, nous obtenons la fonction de décision f f (x) = n i=1 (α i -α * i )k(X i , x) + b
Les noyaux les plus utilisés sont

• polynomial : k(x, z) = (γ < x, z > +c 0 ) d ; • sigmoïdal : k(x, z) = tanh(γ < x, z > +c 0 ) ; • linéaire : k(x, z) =< x, z > ; • base radiale : k(x, z) = exp -γ x -z 2 ,
1.2 Estimations de la fonction de régression où γ > 0 et c 0 ∈ R. Notons que la racine de l'arbre associé à R d contient toutes les observations de l'échantillon D n . L'algorithme commence par découper au mieux la racine en deux noeuds fils. Un découpage (split en anglais) est représentée par la forme suivante :

X (j) < v ∪ X (j) ≥ v .
où j ∈ {1, . . . , d} et v ∈ R. Nous traduisons ce split : les observations avec une valeur plus petite que v sur la j-ième variable seront dans le noeud fils de gauche, et les autres avec une valeur plus grande que v dans le noeud fils de droite. À ce stade, l'arbre CART choisit le meilleur split. Autrement dit, nous trouvons le meilleur couple (j, v) qui minimise la variance intra-noeuds suite au split d'un noeud en ses deux fils. De la même façon, nous répétons cette procédure sur chaque nouveau noeud fils jusqu'à atteindre un critère d'arrêt. Un exemple de critère d'arrêt peut être de ne pas découper des noeuds qui contiennent moins d'un certain nombre d'observations. Les noeuds terminaux sont appelés les feuilles de l'arbre. Nous associons à chaque feuille de l'arbre construit une valeur constante qui correspond à la moyenne des Y i des points qui se trouvent dans cette feuille. Nous appelons l'arbre développé jusqu'au bout un arbre maximal (le nombre de feuilles est égal au nombre d'observations). Cet arbre a un biais très faible, mais une variance très grande. A l'inverse, un arbre constitué uniquement de la racine a une très petite variance mais un biais élevé. La profondeur de l'arbre est donc un paramètre de régularisation à calibrer.

Pour se faire la seconde étape de l'arbre CART se nomme l'élagage qui permet d'éviter le surapprentissage. Elle se repose sur le choix d'un modèle parmi l'ensemble des sous-arbres élagués de l'arbre maximal. Ceci se fait par la minimisation d'un critère pénalisé où la pénalité est proportionnelle au nombre de feuilles de l'arbre.

Forêts aléatoires [START_REF] Breiman | Random forests[END_REF][START_REF] Scornet | Consistency of random forests[END_REF][START_REF] Scornet | On the asymptotics of random forests[END_REF][START_REF] Scornet | Random forests and kernel methods[END_REF][START_REF] Genuer | Arbres CART et Forêts aléatoires, Importance et sélection de variables[END_REF]. Les arbres de décisions ont une forte variabilité (changer une observation peut changer tout l'arbre). Pour réduire cet effet, on a recours au bagging. Les forêts aléatoires ont été introduites par Breiman [START_REF] Breiman | Random forests[END_REF] dont le principe consiste à agréger plusieurs arbres de décision. Définition 8. Soit { f (x, Θ 1 ), . . . , f (x, Θ q )} une collection de prédicteurs par arbre où (Θ 1 , . . . , Θ q ) est une suite de variables aléatoires i.i.d. . Le prédicteur des forêts aléatoires fRf est obtenu par agrégation de cette collection d'arbres

frf (x) = 1 q q i=1 f (x, Θ j ).
Notons qu'une famille de forêt aléatoire se distingue parmi les autres procédures par la qualité de ses performances sur de nombreux jeux de données. Dans ce contexte, nous parlons des Random Forests-RI (forêts aléatoires à variables d'entrée aléatoires). Les forêts aléatoires sont générées comme suit :

1. Nous donnons d'abord le nombre d'arbres q et le nombre de variables candidates pour découper un noeud (m ∈ N * ).

2. Nous tirons un échantillon bootstrap dans D n .

3. Nous construisons un arbre CART h(x, Θ q ) sur cet échantillon bootstrap : chaque coupure est sélectionnée en minimisant la fonction de coût de CART sur un ensemble de m variables choisies au hasard parmi les d variables.

Chapitre 1: Introduction 4. Enfin, l'estimateur des forêts aléatoires est la moyenne des prédictions individuelles des arbres

frf (x) = 1 q q j=1 f (x, Θ j ).
Un avantage majeur de l'algorithme de forêt aléatoire est qu'il peut donner de bons résultats même en grande dimension dû à l'étape de sélection aléatoire parmi l'ensemble des variables. De nombreux travaux sur l'analyse théorique des forêts aléatoires ont été menés comme ceux de [START_REF] Scornet | Consistency of random forests[END_REF][START_REF] Scornet | On the asymptotics of random forests[END_REF][START_REF] Scornet | Random forests and kernel methods[END_REF].

Conclusion.

Dans la littérature, il existe plusieurs estimateurs de la fonction de régression qui ont été bien étudiés d'un point de vue théorique et numérique : Bagging [START_REF] Breiman | Bagging predictors[END_REF], Boosting [START_REF] Schapire | The strength of weak learnability[END_REF], Réseaux de neurones [START_REF] Hastie | The elements of statistical learning: Data Mining, Inference, and Prediction[END_REF], Smoothing spline [START_REF] Wahba | Spline Models for Observational Data[END_REF], Wavelets [START_REF] Mallat | A wavelet tour of signal processing[END_REF], Lasso, Ridge, Elastic-Net, Fused-Lasso, Group-Lasso [START_REF] Hastie | The elements of statistical learning: Data Mining, Inference, and Prediction[END_REF] etc.

Simulations

Nous étudions les performances numériques des estimateurs construits à partir des algorithmes des forêts aléatoires (rf), des k-plus proches voisins (k-PPV), des machines à vecteurs de support (svm), des arbres de régression (tree) et du noyau (NW) ( frf , fsvm , fknn , fNW et ftree respectivement). Ces algorithmes seront utilisés dans la thèse. Pour les algorithmes rf, svm, knn, tree et NW, nous utilisons respectivement les packages R, randomForest, e1071, FNN, tree et NonpModelCheck avec les paramètres par défaut à l'exception de la fenêtre h et l'entier k qui sont choisis par validation croisée et maxnode = 23 est fixé pour rf. Nous étudions la performance de l'estimateur à noyau uniquement dans les modèles univariés car il est coûteux de les calculer en grande dimension. Les performances de l'estimateur f sont évaluées comme suit. On répète indépendamment 100 fois les étapes suivantes :

(i) Nous simulons deux ensembles de données D n et D T avec n = T = 1000.

(ii) À partir de D n , nous construisons les estimateurs f .

(iv) Enfin, sur D T , nous calculons leurs erreurs empiriques L 2 :

Err( f ) = 1 T T i=1 ( f (X i ) -f * (X i )) 2 .
En considérant les 100 répétitions, nous calculons la moyenne et l'écart-type de Err. Pour notre étude numérique, nous considérons 4 exemples du modèle suivant

Y = f * (X) + σ(X)ε , (1.5)
où la variable d'entrée X suit la loi uniforme sur [0, 1] d et ε suit la loi normale centrée réduite.

• Modèle

1 : d = 1, f * (X) = 0.75 sin(10πX), σ 2 (X) = 1 + 4X 2 . • Modèle 2 : d = 1, f * (X) = 0.75 sin(10πX), σ 2 (X) = (1 + 4X 2 )/5. • Modèle 3 : d = 50, f * (X) = X 2 1 + X 2 2 X 3 exp(-X 4 ) + X 6 -X 8 , σ 2 (X) = 0.5 + exp(-(X 1 -0.2) 2 ) + 2 sin(X 2 X 3 ) 2 + 2X 4 X 5 X 6 . • Modèle 4 : d = 50, f * (X) = X 2 1 + X 2 2 X 3 exp(-X 4 ) + X 6 -X 8 , σ 2 (X) = |0.1X 1 X 2 + X 2 3 -X 4 X 7 - X 2 6 + X 8 X 10 | 2 .
Pour chaque dimension d ∈ {1, 50}, les modèles ont la même fonction de régression, mais les fonctions de variance sont différentes. La fonction de variance dans les modèles 1 et 3 prend de grandes valeurs (des valeurs sont supérieures à 1). À l'opposé, elle prend des valeurs relativement modérées dans les modèles 2 et 4 (la majorité des valeurs sont inférieures à 1). Les résultats sont donnés dans la figure (1.2). Nous traçons le graphe de l'estimateur de fknn dans la figure (1.1). Nous faisons deux observations. Premièrement, les meilleurs estimateurs de f * dans chaque modèle sont fNW , fknn , frf et fsvm respectivement. De plus, l'estimateur ftree a de bonnes performances lorsque la fonction de variance prend des grandes valeurs. Deuxièmement, nous remarquons que les estimateurs dans les 4 modèles ont des performances similaires ce qui conduit à une concurrence entre eux. Finalement, nous concluons que plus la fonction de variance prend de grandes valeurs, plus l'estimation de f * devient difficile. Nous constatons que les estimateurs ont des performances proches dans chaque modèle. En général, l'existence de nombreuses méthodes d'estimation différentes conduit à des estimateurs éventuellement concurrents. Pour cela, nous allons présenter le cadre de l'agrégation d'estimateurs.

Agrégation d'estimateurs

La meilleure méthode d'estimation de la fonction de régression est l'agrégation. Nous présentons ci-après cette approche et ses propriétés.

Agrégation

n = {(X i , Y i )} n i=1 en deux sous-échantillons D n 1 et D n 2 avec n 1 + n 2 =
n tels que l'échantillon D n 1 soit utilisé pour construire les éléments du dictionnaire f1 , . . . , fM et que l'échantillon D n 2 serve à les agréger. En général, nous supposons que l'échantillon d'apprentissage a été fixé dans de nombreux travaux de sorte que les estimateurs f * , f1 , . . . , fM deviennent des fonctions fixées (déterministes).

Propriété de l'agrégat

La performance d'un estimateur f peut être mesurée par le risque quadratique, c'est-à-dire par

r( f ) = E[ f -f * 2 ]
avec f 2 = |f (x)| 2 P X (dx) où P X est la mesure de probabilité de X. Posons fθ = M j=1 θ j fj pour un certain ensemble de paramètres Θ ⊆ R M ou θ ∈ Θ. Trois grands types d'agrégation sont considérés dans la littérature :

1. Agrégation par sélection de modèle (MS) : construire un estimateur agrégat f qui est au moins aussi bon que le meilleur parmi les candidats f1 , . . . , fM à un terme résiduel près, c'est-à-dire

r( f ) ≤ min 1≤j≤M r( fj ) + △ MS n,M ,
où △ MS n,M tend vers 0 quand n tend vers l'infini. 2. Agrégation convexe (C) : construire un estimateur agrégat f qui est au moins aussi bon que la meilleure combinaison convexe de candidats f1 , . . . , fM à un terme résiduel près, c'est-à-dire

r( f ) ≤ min θ∈Θ r( fθ ) + △ C n,M , où △ C n,M tend vers 0 quand n tend vers l'infini, et Θ = {θ ∈ R M : θ j ≥ 0, M j=1 θ j = 1}. 3.
Agrégation linéaire (L) : construire un estimateur agrégat f qui est au moins aussi bon que la meilleure combinaison linéaire de candidats f1 , . . . , fM à un terme résiduel près, c'est-à-dire

r( f ) ≤ min θ∈R M r( fθ ) + △ L n,M , où △ L n,M tend vers 0 quand n tend vers l'infini, et Θ = R M .

Agrégation d'estimateurs

Les trois procédures d'agrégation ont été largement traité dans [START_REF] Bunea | Aggregation for gaussian regression[END_REF][START_REF] Juditsky | Functional aggregation for nonparametric regression[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF][START_REF] Tsybakov | Aggregation and minimax optimality in high-dimensional estimation[END_REF][START_REF] Yuhong | Aggregating regression procedures to improve performance[END_REF][START_REF] Lecué | Empirical risk minimization is optimal for the convex aggregation problem[END_REF][START_REF] Lecué | Aggregation via empirical risk minimization[END_REF]. Les quantités △ MS n,M , △ C n,M et △ L n,M caractérisent la précision de procédures d'agrégation MS, C et L respectivement. Dans le modèle gaussien, sous certaines hypothèses, les vitesses optimales d'agrégations de type sélection de modèle, convexe et linéaire (voir [START_REF] Tsybakov | Optimal rates of aggregation[END_REF]) sont données comme suit

       △ MS n,M ≍ log(M )/n; △ C n,M ≍ M/n si M ≤ √ n, log(1 + M/ √ n)/n si M > √ n; △ L n,M ≍ M/n.
En particulier, il est clair que

min θ∈R M r( fθ ) ≤ min θ∈Θ r( fθ ) ≤ min 1≤j≤M r( fj ) .
Les deux premiers types d'agrégation sont au coeur des travaux présentés dans ce manuscrit, aussi, nous présentons ci-après les performances numériques de MS-estimateur et C-estimateur sur quelques données simulées .

Simulations

Nous étudions les performances du MS-estimateur fMS et du C-estimateur fC dans les modèles présentés dans la section 1.2.3. Leurs constructions sont plus amplement détaillées dans la partie 1.6.1 en utilisant deux échantillons indépendants. Nous introduisons un ensemble F = { fs } 5 s=1 qui contient cinq estimateurs construits à partir des algorithmes des forêts aléatoires (rf), des k-plus proches voisins (K-PPV), des machines à vecteurs support (svm), des arbres de régression et des noyaux. Nous évitons la méthode à noyau dans les modèles de grande dimension. Nous répètons indépendamment 100 fois les étapes suivantes : (iv) Enfin, sur D T , nous calculons l'erreur empirique L 2 ( Err) de fMS , de fC et de tous les estimateurs de la fonction de régression f * obtenus à l'étape (iii).

À partir des En conclusion, la qualité des estimateurs de la fonction de régression dépend fortement de la fonction de variance conditionnelle σ 2 . Dans le paragraphe suivant, nous focalisons nos efforts à l'étude de la fonction σ 2 .

Estimation de la fonction de variance conditionnelle

La fonction de variance conditionnelle est une mesure de l'hétéroscédasticité et joue un rôle important dans de nombreux contextes de modélisation statistique. Elle est importante dans de multiples problèmes en finance telles que la mesure de la volatilité ou du risque [START_REF] Anderson | Estimating continuous-time stochastic volatility models of the short-term interest rate[END_REF] ou les rendements boursiers à long terme [START_REF] Mammen | Conditional variance forecasts for longterm stock returns[END_REF]. En statistique, elle est utile dans la sélection de la fenêtre de lissage optimale du noyau [START_REF] Fan | Design-adaptive nonparametric regression[END_REF], estimation du rapport signal sur bruit [START_REF] Verzelen | Adaptive estimation of high-dimensional signal-to-noiseratios[END_REF] et construction de bandes de confiance pour la fonction de régression [START_REF] Hart | Nonparametric Smoothing and Lack-of-Fit Tests[END_REF]. L'estimation de σ 2 a été sérieusement envisagée dans les années 1980 [START_REF] Müller | Estimation of heteroscedasticity in regression analysis[END_REF]. Elle a été motivée par la construction des intervalles de confiance I c pour la fonction f * en régression gaussienne. L'intervalle I c prend la forme habituelle suivante :

Étant donné α ∈ [0, 1], I c = f (x) -z α/2 σ2 (x) + s 2 , f (x) + z α/2 σ2 (x) + s 2
où f et σ2 sont les estimateurs de f * et σ 2 respectivement. La quantité s est l'estimation de l'erreur standard de f , et z α/2 peut être le quantile d'ordre α/2 de la loi normale ou un quantile determiné par boostrap. La construction de I c est bien détaillé par exemple dans les livres [START_REF] Fan | Local Polynomial Modelling and Its Applications[END_REF][START_REF] Loader | Local Regression and Likelihood[END_REF] et l'introduction de l'article [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF]. Cet intervalle dépend fortement de l'estimateur de la fonction de variance. En particulier, la longueur de I c est égale à Par conséquent, une mauvaise estimation de σ 2 conduit à des conclusions erronées.

2z α/2 σ2 (x) + s 2 .
Dans la suite, nous établissons un état de l'art sur le problème d'estimation de la fonction de variance. Dans la littérature, l'estimation de la fonction de variance est largement étudiée à la fois dans les cas de variables d'entrée fixées et aléatoires :

Design aléatoire

Dans ce manuscrit, nous nous focalisons sur le cas où les variables d'entrée sont aléatoires. Nous distinguons deux types d'approches : la méthode directe et la méthode basée sur les erreurs résiduelles. Dans la suite, nous présentons ces deux procédures en détail.

Méthode directe

La méthode directe repose sur une décomposition de la fonction de variance conditionnelle σ 2 en moments

σ 2 (x) = E Y 2 |X = x -(f * (x)) 2 .
Elle consiste à estimer séparément les deux termes du côté droit. Pour être plus précis, l'estimateur direct de σ 2 a la forme suivante

σ2 d (x) = ĝ(x) -f 2 (x) , (1.6)
où ĝ et f sont les estimateurs de E Y 2 |X = x et f * respectivement. Cette approche est par exemple considérée dans les articles [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF][START_REF] Härdle | Local polynomial estimators of the volatility function in nonparametric autoregression[END_REF]. Les auteurs dans [START_REF] Härdle | Local polynomial estimators of the volatility function in nonparametric autoregression[END_REF] ont étudié l'estimation de σ 2 dans le modèle autorégressif d'ordre 1 (en supposant que X i = Y i-1 ). Ils proposent une classe générale d'estimateurs de la fonction de volatilité basés sur les estimateurs par polynomiaux locaux (LP). Les définitions de ĝ et f par la méthode LP reposent sur les problèmes de minimisation suivants :

cn (x) = argmin c∈R l n i=1 (Y 2 i -c T U in ) 2 K Y i-1 -x h , c n (x) = argmin c∈R l n i=1 (Y i -c T U in ) 2 K Y i-1 -x h , où K : R → R est un noyau, h > 0 est une fenêtre et U in = F (u in ), F (x) =      1 x . . . x l-1 (l-1)!      , u in = Y i-1 -x h .
L'estimateur σ2 d de σ 2 est alors défini par La normalité asymptotique des estimateurs LP de la fonction de régression et la fonction de variance ont été établies. La méthode directe pour estimer la fonction de variance conditionnelle est simple à mettre en oeuvre. Toutefois, elle a un inconvénient majeur :

σ2 n (x) = cn (x) T F (0) -(c n (x) T F (0)) 2 .
(i) l'estimateur de σ2 d (1.6) n'est pas toujours positif en particulier lorsque différents paramètres de lissage sont utilisés pour établir les termes c n et cn .

Méthode basée sur les erreurs résiduelles

La méthode basée sur les erreurs résiduelles consiste en deux étapes. Dans une première étape, nous construisons un estimateur f de la fonction de régression f * . Dans une deuxième étape, un estimateur de σ 2 est obtenu en résolvant le problème de régression où la variable d'entrée est X et la variable à prédire est (Y -f (X)) 2 . Ainsi quasiment toutes les méthodes classiques d'estimation peuvent être considérées. Dans la suite nous axons notre développement sur les méthodes basées sur les moyennes locales, c'est-à-dire, peuvent s'écrire sous la forme σ2 (x) = n i=1 w i (x)(Y i -f (X i )) 2 où chaque w i (x) est une fonction borélienne réelle de x et de X 1 , . . . , X n . Bien souvent (mais pas toujours), les w i (x) sont positifs et normalisés à 1, de telle sorte que (w 1 (x), . . . , w n (x)) est en fait un vecteur de probabilités. Voici deux exemples classiques d'estimateurs de type moyenne locale.

Estimateur à noyau. L'estimateur à noyau pour σ 2 est obtenu en prenant w i (x) =

K X i -x h n i=1 K X i -x h où K est un noyau sur R d et h ≥ 0. En d'autres termes, pour x ∈ R d σ2 h (x) = n i=1 K X i -x h (Y i -f (X i )) 2 n i=1 K X i -x h . (1.7)
Les auteurs dans [START_REF] Kulik | Nonparametric conditional variance and error density estimation in regression models with dependent errors and predictors[END_REF] ont étudié l'estimateur (1.7) où ils ont également estimé lors de première étape la fonction de régression également par la méthode à noyau en supposant que la variable X a une variance unitaire et une densité. Ils ont établi les résultats asymptotiques de l'estimateur à noyau (1.7) dans ce cas. De plus, ils ont montré que l'estimation de la fonction de régression a une influence sur l'estimation de la fonction variance. L'inconvénient principal de cet estimateur est que l'estimation de f * a un effet négatif sur le comportement asymptotique de l'estimateur (1.7) dans le cas où les erreurs ε i ne sont pas indépendantes et identiquement distribuées.

Estimateur des k-plus proches voisins. En prenant les notations de la section 1.2.1, l'estimateur des k-plus proches voisins de σ 2 prend la forme suivante

σ2 (x) = 1 k n kn i=1 (Y (i) -f (X (i) )) 2 . (1.8)
Cet estimateur, dans le cas où f est également un estimateur des k-plus proches voisins, sera étudié en détail dans la suite de ce manuscrit. Nous remettons donc à plus tard l'analyse statistique de cette méthode.

En outre, il existe différentes variantes de la méthode basée sur les erreurs résiduelles pour la deuxième étape d'estimation de σ 2 pour d = 1 :

(i) Application d'un estimateur localement linéaire : Dans [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF], les auteurs ont utilisé la même méthode pour estimer la fonction de régression et la fonction de variance. Plus précisément, ils ont appliqué l'estimateur localement linéaire dans les deux étapes. Soit f (x) = â(x) l'estimateur localement linéaire qui résout le problème des moindres carrés pondérés suivant :

(â, b) = argmin a,b∈R n i=1 (Y i -a -b(X i -x)) 2 K 1 X i -x h 1 , ∀x ∈ R, où K 1 est un noyau tel que R K 1 (x)dx = 1 et h 1 > 0 est une fenêtre. Définissons Ẑi = (Y i - f (X i ))
2 pour tout i = 1, . . . , n. L'estimateur basé sur les erreurs résiduelles σ2 (x) = α(x) de la fonction de variance est donné comme suit

(α, β) = argmin α,β∈R n i=1 ( Ẑi -α -β(X i -x)) 2 K 2 X i -x h 2 ,
où K 2 est un noyau sur R et h 2 > 0 est une fenêtre. L'estimateur précédent de σ 2 est asymptotiquement aussi bon que dans le cas où la fonction de régression f * était donnée. Mais, il n'est pas nécessairement positif. Comme solution à cela, les auteurs dans [START_REF] Yu | Likelihood based-local linear estimation of the conditional variance function[END_REF] ont proposé des estimateurs basés sur une vraisemblance normale localisée de sorte que la fonction de variance estimée est toujours positive. Ils ont utilisé une forme localement linéaire pour estimer la fonction de régression, et une forme localement log-linéaire pour estimer la fonction de variance en séparant les fenêtres. Ils ont fourni une analyse asymptotique de leurs estimateurs.

(ii) Estimateur localement exponentiel : les auteurs dans [START_REF] Flavio | Nonparametric estimation of volatility functions: The local exponential estimator[END_REF] ont utilisé une approche similaire à celle dans [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF] (Application d'un estimateur localement linéaire). Ils ont considéré un estimateur localement linéaire pour estimer la fonction de régression f * (voir (i)). Cet estimateur est f (x) =

1.4 Estimation de la fonction de variance conditionnelle â(x). L'estimateur localement exponentiel de la variance conditionnelle est obtenu à partir de la solution du problème de minimisation suivant :

(α 1 , β1 ) = argmin α 1 ,β 1 ∈R n i=1 ((Y i -f (X i )) 2 -exp(α 1 + β 1 (X i -x))) 2 K X i -x h 2 ,
où h 2 > 0 est une autre fenêtre. L'estimateur exponentiel est alors donné par σ2 e (x) = exp( α1 (x)), et est donc toujours positif. Les auteurs ont prouvé qu'il a la même variance asymptotique (d'où la même efficacité) que l'estimateur de la fonction de variance obtenu dans (i).

(iii) Estimateur localement constant repondéré : étant donné l'estimateur de la fonction de régression de f dans (i), les auteurs dans [START_REF] Xu | Tilted nonparametric estimation of volatility functions with empirical applications[END_REF] ont proposé un estimateur à noyau repondéré pour estimer

σ 2 σ2 p (x) = n i=1 ŵi (x)K((X i -x)/h) Ẑi n i=1 ŵi (x)K((X i -x)/
w i (x) ≥ 0, n i=1 w i (x) = 1, et n i=1 w i (x)(X i -x)K h (X i -x) = 0 .
L'estimateur σ2 p est asymptotiquement équivalent à l'estimateur de σ 2 obtenu dans (i) et est positif dans des échantillons finis. L'existence de plusieurs estimateurs par la méthode basée sur les erreurs résiduelles permet de créer de la concurrence entre eux. En particulier, il est difficile de choisir le meilleur estimateur parmi eux. Dans ce contexte, il est naturel de poser la question suivante qui va guider nos recherches : Question 1. Quelle est la meilleure façon d'estimer la fonction de variance ? Nous apporterons des éléments de réponse à cette question en exploitant des arguments d'agrégation, voir section 1.6.1 pour plus de détail.

Design fixe

La méthode la plus populaire pour estimer la fonction de variance conditionnelle, lorsque la variable d'entrée X est déterministe, est la méthode des différences séquentielles (en anglais " the differencesequence method"). Le premier estimateur basé sur les différences a été développé dans [START_REF] Müller | Estimation of heteroscedasticity in regression analysis[END_REF]. Les auteurs ont considéré le modèle (1.5) où les variables d'entrée X i ∈ [0, 1] pour tout i = 1, . . . , n, les variables de bruit ε i ont un quatrième moment fini et la fonction de variance σ 2 est une fonction γ-lipschitzienne continue avec γ ∈ [0, 1]. L'estimateur initial a la forme

σ2 (X i ) =   j 2 j=j 1 w j Y j+i   2 ,
où j 1 = -⌈m/2⌉, j 2 = ⌈m/2 -1/4⌉ avec m est un entier fixé et la séquence de différence {w j } satisfait les conditions j 2 j=j 1 w j = 0, and

j 2 j=j 1 w 2 j = 1.

Chapitre 1: Introduction

Les auteurs ont montré que cet estimateur n'est pas consistant et ont proposé la modification suivante de l'estimateur : ils l'ont lissé à l'aide d'un noyau K, c'est-à-dire, ils définissent l'estimateur

σ2 (X) = 1 h n j=1 s j s j -1 K X -u h du σ2 (X j ) , avec s j = X j +X j+1 2 , 1 ≤ j ≤ n -1, s 0 = 0 et s n = 1.
Le paramètre h est la fenêtre de lissage qui satisfait la condition suivante : h → 0 et nh → 0 quand n → ∞ . L'estimateur σ2 a été montré consistant. Dans [START_REF] Wang | Effect of mean on variance function estimation in nonparametric regression[END_REF], les auteurs ont étudié les effets de la fonction de régression inconnue et lisse (en anglais smooth) sur l'estimation de la fonction de variance. Ils ont proposé un estimateur alternatif de σ 2 et ont évalué ses performances. Ils ont comparé les performances de leur estimateur avec l'estimateur basé sur les erreurs résiduelles proposé dans [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF]. Leurs résultats suggèrent que l'utilisation de la méthode basée sur les erreurs résiduelles n'est adaptée que lorsque la fonction de régression est lisse. Prolongeant les travaux de [START_REF] Brown | Variance estimation in nonparametric regression via the difference sequence method[END_REF][START_REF] Müller | Estimation of heteroscedasticity in regression analysis[END_REF][START_REF] Wang | Effect of mean on variance function estimation in nonparametric regression[END_REF] via des différences séquentielles, les auteurs de [START_REF] Cai | Variance function estimation in multivariate nonparametric regression[END_REF] ont étudié le cas général d'une fonction de variance multivariée hétéroscédastique.

Apprentissage sous contrainte

Dans la section précédente, nous avons énuméré certaines approches existantes pour estimer la fonction de variance. Ici, nous présentons deux problèmes d'apprentissage sous contrainte, l'option rejet et les intervalles de prédiction, dans lesquels la fonction de variance joue un rôle clé.

Option rejet en apprentissage

En général, dans un problème de prédiction, les algorithmes produisent toujours une prédiction, même lorsque celle-ci est susceptible d'être non pertinente. De nombreuses procédures d'estimation visent néanmoins à réduire les erreurs de prédiction, e.g., les réseaux de neurones, les méthodes à noyaux, l'approche des K plus proches voisins (K-PPV), les moindres carrés régularisés pour n'en nommer qu'un petit nombre. Cependant, même les méthodes les plus performantes commettent des erreurs qui peuvent dans certains cas avoir de lourdes conséquences. Le domaine médical est un champ d'application notable dans lequel il est souvent souhaitable de s'abstenir plutôt que de prendre une mauvaise décision. Cette capacité à s'abstenir de faire une prédiction a plusieurs avantages. Premièrement, en ne faisant des prédictions que lorsqu'elle est fiable, nous améliorons les performances. Deuxièmement, en évitant les erreurs de prédiction, nous pouvons augmenter la confiance d'un utilisateur dans l'algorithme proposé. Troisièmement, s'abstenir à produire une prédiction peut encore entraîner des gains de temps en ne nécessitant que des interventions humaines pour prendre des décisions dans un petit nombre de cas. Introduite, pour la classification, l'utilisation de l'apprentissage avec option rejet a récemment été développé en régression. Nous développons cidessous les deux cas d'application pour comprendre le problème.

Classification avec option rejet

L'idée de classification avec une option rejet remonte aux articles [START_REF] Chow | An optimum character recognition system using decision functions[END_REF][START_REF] Chow | On optimum error and reject trade-off[END_REF]. L'auteur a étudié à la fois la règle de décision de rejet optimale et le compromis entre le taux de rejet et l'erreur. En outre, cela se fait en particulier sous la fonction de perte (le coût) 0-1, en supposant que la distribution sousjacente est complètement connue. La classification avec une option rejet consiste à permettre à un prédicteur de refuser de répondre à une question en présence d'incertitude. Les instances doivent être rejetées chaque fois qu'aucune des probabilités a posteriori n'est pas suffisamment forte. La classification avec option a été développée dans le cadre de la théorie de l'apprentissage statistique [START_REF] Chow | On optimum error and reject trade-off[END_REF][START_REF] Herbei | Classification with reject option[END_REF][START_REF] Yuan | Classification methods with reject option based on convex risk minimization[END_REF][START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF][START_REF] Cortes | Boosting with abstention[END_REF]99,[START_REF] Hellman | The nearest neighbor classification rule with a reject option[END_REF][START_REF] Fumera | Support vector machines with embedded reject option[END_REF]. Elle est basée sur la classification binaire. Dans ce cas, une règle de classification avec l'option de rejet prend ses valeurs dans {0, 1, re} où re désigne le refus de répondre. Mathématiquement parlant, rappelons rapidement le problème de classification binaire. Soit (X, Y ) ∈ X × {0, 1} un couple admettant une loi de probabilité inconnue P sur X × {0, 1} où X est un 1.5 Apprentissage sous contrainte espace des entrées (features en anglais). Le vecteur X ∈ X est appelé vecteur des entrées et Y ∈ {0, 1} est l'étiquette associée à X. Une observation x peut être difficile à classer si la probabilité conditionnelle η(x) = P(Y = 1|X = x) est proche de 1/2. Soit g : X → {0, 1, re} un classifieur avec option rejet où re signifie rejeter. Ce classifieur a deux caractéristiques ; l'erreur P(g(X) = Y, g(X) = re) et le taux de rejet fixe P(g(X) = re). Pour réaliser un compromis entre ces deux mesures de qualité de l'estimateur de g, nous considérons le risque associée suivant P(g(X) = Y, g(X) = re) + dP(g(X) = re) . où d ∈]0, 1/2[ est un paramètre de régularisation qui permet de réaliser ce compromis. Il contrôle le poids que l'on souhaite donner à l'emploi de l'option rejet. Ce paramètre prend une valeur faible. Donc, on aura facilement recours au rejet. La règle optimale minimisant le risque précédent est donnée dans [START_REF] Chow | On optimum error and reject trade-off[END_REF] g

* (x) =    0 si 1 -η(x) > η(x) et 1 -η(x) > 1 -d, 1 si η(x) > 1 -η(x) et η(x) > 1 -d, re si max(1 -η(x); η(x)) ≤ 1 -d.
Les caractéristiques du classifieur de Bayes avec option rejet sont établies dans [START_REF] Chow | On optimum error and reject trade-off[END_REF][START_REF] Herbei | Classification with reject option[END_REF][START_REF] Yuan | Classification methods with reject option based on convex risk minimization[END_REF][START_REF] Lei | Classification with confidence[END_REF][START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF]. Par exemple, [START_REF] Herbei | Classification with reject option[END_REF] a étudié la vitesse de convergence des estimateurs plug-in et des minimiseurs du risque empirique. En outre, nous pouvons définir deux quantités : la probabilité de rejet et la probabilité de mauvaise classification. [START_REF] Lei | Classification with confidence[END_REF] se focalise à minimiser la probabilité de rejet à condition d'avoir une borne fixe sur la probabilité de mauvaise classification. À l'inverse, étant donné une borne fixe sur la probabilité de rejet, [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF] a étudié le problème minimisation de la probabilité de mauvaise classification. Dans cette thèse, nous nous intéressons au problème de régression avec option rejet.

Régression avec option rejet

Le rejet en régression a rarement été considéré dans la littérature. L'article [START_REF] Wiener | Pointwise tracking the optimal regression function[END_REF] a analysé la possibilité d'une option rejet dans le contexte de la régression des moindres carrés. Nous considérons un jeu de données étiquetées D n = {(X i , Y i )} n i=1 . Nous notons l'ensemble de toutes les fonctions de prédiction F. Étant donné une fonction de perte, ℓ : R × R → [0, ∞), nous quantifions la qualité de prédiction de tout f par le risque

R(f ) = E [ℓ(f (X), Y )] = R d ×R ℓ(f (x), y)dP(x, y).
La meilleure fonction de prédiction est la fonction de F minimisant le risque R :

f * ∈ argmin f ∈F R(f ).
Le risque empirique d'un prédicteur f sur D n est défini par

R(f ) = 1 n n i=1 ℓ(Y i , f (X i )).
Étant donné un échantillon D n , le minimiseur du risque empirique (ERM) sur

F est f = inf f ∈F R(f ).
Un prédicteur avec option rejet est une fonction mesurable de R d dans R ∪ ∅.

∀x ∈ {x ∈ R d , |Γ f (x)| = 1}, |f (x) -f * (x)| ≤ ε.
C n (X n+1 ) = C n (X 1 , Y 1 , . . . , X n , Y n ; X n+1 ) ⊆ R qui satisfait P (Y n+1 ∈ C n (X n+1 )) ≥ 1 -α, (1.9)
où P = P n+1 est la distribution jointe de (X 1 , Y 1 ), . . . , (X n+1 , Y n+1 ). Nous présentons les différentes notions de validité dans [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF] : Validité marginale. C'est la notion la plus classique pour les IP. L'équation (1.9) signifie que l'ensemble C n a la garantie de validité marginale. Soit ν la mesure de Lebesgue. L'ensemble de prédiction optimal de niveau 1α est

C α = argmin C:P((X,Y )∈C)≥1-α ν(C) := {(x, y) : p(x, y) ≥ t α } , où t α satisfait P (C α ) = 1 -α et p(X,
P (Y n+1 ∈ C n (x)|X n+1 = x) ≥ 1 -α pour tout P et presque x.
Elle implique la validité marginale mais pas l'inverse (voir [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF][START_REF] Lin | Locally valid and discriminative prediction intervals for deep learning models[END_REF] pour plus de détails). Un ensemble de prédiction est défini par

C * P (x) = inf x P (Y ∈ C(x)|X = x) ≥ 1 -α,
Si la loi jointe de (X, Y ) est connue, son expression est donnée comme suit C * P (x) = {y : p(y|x) ≥ t α (x)} , où t α (x) est choisi de cette manière : R p(y|x)1 {p(y|x)≥tα(x)} dy = 1α. L'ensemble C P = {C * P (x)} est appelé la bande oracle conditionnelle qui minimise ν(C(x)) pour tout x parmi toutes les bandes C * P .

Contributions

En général, la validité conditionnelle est obtenue que sur des hypothèses fortes sur la distribution de (X, Y ). La méthode proposée pour construire un ensemble de confiance est Validation locale [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF][START_REF] Lin | Locally valid and discriminative prediction intervals for deep learning models[END_REF]. Elle consiste à interpoler entre la validité marginale et la validité conditionnelle. Étant donné une partition A = {A k , 1 ≤ k ≤ m} de R d de sorte que ∪ m k=1 A k = R d , un ensemble de confiance vérifie la validation locale si

P (Y n+1 ∈ C n (x)|X n+1 ∈ A k ) ≥ 1 -α pour tout P et 1 ≤ k ≤ m.
La prédiction conforme est bien détaillée dans les articles [START_REF] Vovk | Algorithmic learning in a random world[END_REF][START_REF] Shafer | A tutorial on conformal prediction[END_REF][START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF]. Nous notons Z n+1 = (X n+1 , Y n+1 ) et z = (x, y). Nous formons un jeu de données augmenté D n+1,z correspondant à D n+1 avec Z n+1 = z. Nous définissons un score de conformité r i (z) = g(Z i , D n+1,z ) pour i = 1, . . . , n où g est une fonction mesurable basé sur D n+1,z . Puis, nous faisons l'hypothèse H 0 : Y n+1 = y en utilisant la fonction

π n (x, y) = 1 n + 1 n+1 i=1 1 {r i (z)≤r n+1 (z)} .
Enfin, le prédicteur de confiance est C n (x) = {y : (n + 1)π n (x, y) ≥ ⌊(n + 1)α⌋} .

Dans [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF], les auteurs ont étudié ce prédicteur en prenant g comme un estimateur à noyau de la jointe (X, Y ). L'IP résultant est marginalement valide. Mais, il est coûteux en temps de calcul : nous devons trouver le score de conformité r(z) pour chaque z. Les auteurs ont construit des bandes de prédiction avec une validation locale. Ils ont proposé un estimateur appelé COPS (en anglais conformal optimized prediction set) qui a toujours une garantie d'échantillon fini. Sous des conditions de régularité, cet estimateur converge vers une bande oracle à une vitesse optimale au sens minimax d'ordre (log(n)/n) β/(β(d+2)+1) , où β est la régularité de la classe de Holder. En grande dimension (d >> n), [START_REF] Lei | Distribution-free predictive inference for regression[END_REF] propose une stratégie pour construire un prédicteur de confiance pour la variable de réponse en utilisant n'importe quel estimateur de la fonction de régression.

Contributions

Les travaux présentés dans ce manuscrit sont regroupés en trois chapitres. Le premier chapitre donne une possible réponse à la Question 1. Il introduit une stratégie d'agrégation pour l'estimation de la fonction de variance. Le deuxième chapitre représente le problème de régression avec option rejet où l'on est autorisé à s'abstenir de prédire. Il décrit une réponse de la Question 2. Le troisième chapitre détaille la construction d'intervalle de prédiction en prédéterminant sa longueur. Tous ces travaux concernent le modèle de régression hétéroscédastique.

Estimation de la fonction de variance

Nous commençons cette section par une contribution à l'étude théorique de l'estimateur de K-PPV.

Estimation de la fonction de variance par K-PPV

Nous avons décrit dans les sections précédentes l'estimation des fonctions de régression et de variance par l'algorithme de K-PPV. Ici, nous présentons un nouveau résultat sur les vitesses de convergence de l'estimateur (1.4) et de l'estimateur (1.8) par rapport à la ℓ p -norme sup. Nous faisons tout d'abord les hypothèses suivantes : Hypothèse 1. Les fonctions f * et σ 2 sont lipschitziennes.

Hypothèse 2 (Hypothèse de forte densité). La distribution marginale P X satisfait l'hypothèse de forte densité si

• P X est supporté sur un ensemble régulier compact C ⊂ R d ,

• P X admet une densité µ w.r.t. à la mesure de Lebesgue telle que pour tout x ∈ C, nous avons 0 < µ min ≤ µ(x) ≤ µ max < ∞.

Hypothèse 3. Les variables Y -f * (X) et (Y -f * (X)) 2 -σ 2 (X)
satisfont une condition de bruit uniforme : il existe c 0 > 0 tel que

sup x∈C E [exp(λ (Y -f * (X))) | X = x] ≤ exp(c 2 0 λ 2 ), pour |λ| ≤ 1 c 0 .
Ces trois hypothèses sont assez classiques [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. L'hypothèse de forte densité a été introduite dans le contexte de la classification binaire par exemple dans [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]. L'hypothèse 3 exige que le conditionnel sur X soit sous-exponentielle uniformément sur C. Théorème 1. Supposons que les trois hypothèses précédentes sont satisfaites. Soit p ≥ 1 et k n ∝ n 2/(d+2) , alors

E sup x∈C fknn (x) -f * (x) p log(n) p n -p/(d+2) , E sup x∈C σ2 (x) -σ 2 (x) p log(n) p n -p/(d+2) .
Ce résultat présente les vitesses de convergence pour les estimateurs des fonctions de régression et de variance par K-PPV par rapport à la ℓ p -norme sup. Nous remarquons que les deux estimateurs ont la même vitesse de convergence qui est d'ordre log(n) p n -p/(d+2) et qui dépend de la dimension d et du paramètre p. En effet, la borne supérieure de la ℓ p -norme sup de l'estimateur σ2 dépend de la borne supérieure de la ℓ p -norme sup de l'estimateur fknn .

Estimation de la fonction de variance par les procédures d'aggrégation

Nous rappelons que l'agrégation est une approche populaire en apprentissage statistique pour estimer f * dans le modèle de régression (Section 1.3). Dans ce sens, nous développons l'agrégation pour estimer la fonction de variance σ 2 dans laquelle les estimateurs initiaux sont construits par la méthode basée sur les erreurs résiduelles (Section 1.4). Nous introduisons d'abord deux échantillons d'apprentissage indépendants :

D n = {(X i , Y i ), i = 1, . . . , n} et D N = {(X i , Y i ), i = n + 1, . . . , n + N } qui consistent
m ∈ argmin m∈[M 2 ] RN (σ 2 ŝ,m ) où RN (σ 2 ŝ,m ) = 1 N n+N i=n+1 | Y i -fMS (X i ) 2 -σ2 ŝ,m (X i )| 2 .
6. Sortie le MS-estimateur de la fonction de variance : σ2

MS := σ2 ŝ, m.
Agrégation convexe. Notre méthode se déroule toujours en deux étapes : une pour l'estimation de la fonction de régression f * et la seconde pour l'estimation de la fonction de variance σ 2 (c.f., Algorithme 2). Les différences avec l'agrégation C résident dans les étapes 2-3 et 5-6 définissant les poids pour les fonctions à agréger.

Résultats principaux. Dans la suite, nous décrivons la consistance de nos approches.

Soit R( fs ) = E |Y -fs (X)| 2 le risque quadratique pour fs pour tout s ∈ [M 1 ]. Nous définissons s * comme suit s * ∈ argmin s∈[M 1 ]
R( fs ) .

Nous introduisons également les hypothèses suivantes :

Hypothèse 4. Les fonctions f * et σ 2 sont bornées.

Hypothèse 5. Pour tout s ∈ [M 1 ] et tout m ∈ [M 2 ] , fs et σ2 s,m sont bornés pour presque tout D n .
Hypothèse 6 (Hypothèse de séparabilité). Il existe δ 0 > 0 telle que

δ * (D n ) = min s =s * {|R( fs ) -R( fs * )|} > δ 0 . Hypothèse 7. Y est borné ou Y provient du modèle gaussien Y = f * (X) + σ(X)ξ, où ξ ∼ N (0, 1) est indépendante de X. Hypothèse 8. Pour tout i ∈ [M 1 ], tout m ∈ [M 2 ] et tout λ ∈ Λ M 1 , fi et σ2 λ,m sont bornés pour presque tout D n . Hypothèse 9. Il existe une constante K ≥ 0 telle que pour chaque j ∈ [M 2 ] E |σ 2 λ 1 ,j (X) -σ2 λ 2 ,j (X)| ≤ K λ 1 -λ 2 1,M 1 , ∀λ 1 , λ 2 ∈ Λ M 1 presque sûrement D n .
Les hypothèses 1 jusqu'à 9 jouent un rôle crucial sur l'étude de la consistance de σ2

MS et de σ2 C respectivement. Les hypothèses 5 et 8 décrivent la bornitude des estimateurs initiaux qui sont construits sur l'échantillon D n . Les bornes de ces estimateurs ne sont pas nécessairement connues. Elles jouent un rôle clé dans la détermination de la vitesse de convergence de l'erreur d'estimation (la variance) de σ2

MS et de σ2 C . L'hypothèse 6 permet de contrôler la différence entre la variable aléatoire ŝ et le sélecteur déterministe s * avec grande probabilité. L'hypothèse 9 est une hypothèse forte. Elle est vérifiée par exemple pour les estimateurs de moyenne locale.

Pour tout

β ∈ Λ M 2 nous définissons σ2 λ,β : σ2 λ,β = M 2 j=1 β j σ2 λ,j . Sur D N , nous calculons β ∈ argmin β∈Λ M 2 RN (σ 2 λ,β ), où RN (σ 2 λ,β ) = 1 N n+N i=n+1 |(Y i -fC (X i )) 2 -σ2 λ,β (X i )| 2 .
6. Sortie le C-estimateur de la fonction de variance : σ2

C := σ2 λ, β .
Nous obtenons le premier résultat :

Théorème 2. Soit fMS et σ2 MS deux MS-estimateurs de f * et de σ 2 respectivement. Alors, il existe deux constantes absolues C 1 > 0 et C 2 > 0 tel que E |σ 2 MS (X) -σ 2 (X)| 2 ≤ E min m∈[M 2 ] E X |σ 2 s * ,m (X) -σ 2 (X)| 2 + C 1 min s∈[M 1 ] E fs -f * 2 N 1/2p + C 2 log(M 1 ) N 1/4p , 1.6 Contributions avec p = 1 si Y est borné et p = 2 si Y provient du modèle gaussien.
Ce Théorème donne la borne supérieure de l'erreur quadratique de σ2 MS . En résumé, le MS-estimateur est asymptotiquement aussi bon que le meilleur estimateur σ2 s * ,m qui est construit sur le premier échantillon D n au sens de l'erreur quadratique L 2 . La borne se compose de deux parties : la première partie est le biais du MS-estimateur σ2

MS et dépend du sélecteur déterministe s * ; la seconde partie est composée des deux termes restants et correspond à l'erreur d'estimation. Le premier terme est le terme de biais de fMS exprimé en fonction de la norme empirique, et le deuxième terme caractérise le prix à payer pour l'agrégation MS qui est d'ordre ; (log(M 1 )/N ) 1/4 dans le cas où Y est borné et (log(M 1 )/N ) 1/8 dans le cas où Y provient du modèle gaussien. Cette vitesse lente est du au fait que l'estimation de la fonction de variance repose sur f * que l'on doit également estimer.

Le deuxième résultat décrit la consistance de σ2

C .

Théorème 3. Soit fC et σ2 C deux C-estimateurs de f * et de σ 2 respectivement. Alors, il existe deux constantes absolues C 1 > 0 et C 2 > 0 tel que E |σ 2 C (X) -σ 2 (X)| 2 ≤ E inf β∈Λ M 2 E X |σ 2 λ,β (X) -σ 2 (X)| 2 + C 1 inf λ∈Λ M 1 E fλ -f * 2 N 1/2p + C 2 log(M 1 ) N 1/4p , avec p = 1 si Y est borné et p = 2 si Y provient du modèle gaussien.
Ce résultat décrit la borne supérieure de l'erreur quadratique de σ2 C . La borne se décompose en trois termes. Le premier terme représente le biais de σ2

C qui dépend de la variable aléatoire λ. Le deuxième et le troisième terme sont une borne du terme de variance de σ2

C . Le deuxième terme est le terme de biais de fC par rapport à la norme empirique. Le troisième terme est le prix à payer pour l'agrégation C.

Les deux procédures MS et C ont la même vitesse de convergence. En effet, les termes de la variance de σ2

MS et σ2 C sont basés sur la borne supérieure de l'erreur quadratique empirique de fMS et fC . De plus, fMS et fC ont la même vitesse de convergence (voir [START_REF] Tsybakov | Aggregation and minimax optimality in high-dimensional estimation[END_REF] pour plus de détails). En comparaison aux vitesses de convergence de fMS et fC au sens de l'erreur quadratique, nos vitesses sont lentes à cause de la double agrégation que nous devons effectuer pour l'estimation de la fonction de variance conditionnelle. L'optimalité de ces résultats reste un problème ouvert.

Régression avec option rejet et application au K-PPV

Dans le Chapitre 3, nous décrivons une première application de la fonction de variance conditionnelle. Nous introduisons le problème de régression avec option rejet qui vise à construire des procédures d'estimation capables de ne pas prédire lorsque le doute dans la valeur prédite est trop grand. En particulier, nous nous intéressons au cadre de travail dans lequel le taux de rejet est fixe, laissant ainsi la possibilité à l'humain d'agir sur une proportion des données jugées, par l'algorithme, trop délicate à traiter par l'algorithme lui-même. Cette décision de rejeter et donc de ne pas traiter certaines données par la machine est prise par l'algorithme et vise à rentabiliser l'effort de l'humain.

Nous avons présenté dans la Section 1.5.1 la définition d'un prédicteur avec option rejet Γ f . Nous rappelons que ce prédicteur est une fonction mesurable de R d dans R ∪ ∅. Il a deux caractéristiques :

• le taux de rejet : r (Γ f ) = P (|Γ f (X)| = 0) ; • l'erreur de prédiction : Err (Γ f ) = E (Y -f (X)) 2 | |Γ f (X)| = 1 .
Chapitre 1: Introduction Pour mesurer la performance de Γ f , nous considèrons un compromis entre ces deux quantités. Dans ce cas, le risque de Γ f est donné par

R λ (Γ f ) = E (Y -f (X)) 2 1 {|Γ f (X))|=1} + λ P (|Γ f (X)| = 0) , où 1 {•} est
la fonction indicatrice et λ > 0 est un paramètre de régularisation. En particulier, une forte valeur de λ impose une contrainte importante sur l'utilisation de l'option rejet. Inspiré de la classification avec option rejet [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF], nous présentons dans cette section le cadre d'étude où le prédicteur avec option rejet a un taux de rejet fixé à l'avance. Ceci traduit typiquement des situations où nous disposons de ressources humaines limitées pour traiter toutes les tâches. Pour ce faire, nous ajoutons une contrainte sur le taux de rejet du prédicteur. Plus précisément, soit ε ∈ (0, 1]. Le prédicteur optimal à taux de rejet ε, ou le ε-prédicteur optimal pour raccourcir est défini par :

Γ * ε ∈ arg min{Err (Γ f ) : r (Γ f ) ≤ ε} .
Notamment, sous l'hypothèse de continuité de la fonction de répartition F σ 2 de la variable aléatoire σ 2 (X), le problème d'optimisation précédent admet une solution explicite

Γ * ε (X) = {f * (X)} if F σ 2 (σ 2 (X)) ≤ 1 -ε , ∅ sinon .
Le ε-prédicteur optimal repose donc sur un seuillage de la fonction de variance. Nous pouvons montrer par ailleurs qu'il a exactement un taux de rejet égal à ε. Autrement dit, le fait de rejeter ou non dépend essentiellement de la fonction de variance. La nature de l'oracle suggère une approche semisupervisée de type plug-in qui repose sur l'estimation de f * et σ 2 , ainsi que F σ 2 . L'algorithme 3 détaille la construction du prédicteur plug-in de niveau de rejet ε.

Algorithme 3 Prédicteur plug-in de niveau ε 1. Nous introduisons deux échantillons indépendants : D n et D M = {X i } n+M i=n+1 qui contient M observations indépendantes, et non étiquetées, de même loi que X.

À partir de D n , nous construisons des estimateurs

f et σ2 de f * et σ 2 . 3. Sur D M , nous estimons F σ 2 : (a) Randomisation : σ2 (X i ) = σ2 (X i ) + ζ i , ζ i ∼ U([0, u]) avec u > 0, (b) Fonction de répartition empirique de σ2 (X i ) : Fσ2 (•) = 1 M n+M i=n+1 1 { σ2 (X i )≤•} .
4. Sortie du prédicteur plug-in de niveau ε

Γε (x) = f (x) if Fσ2 (σ 2 (x) + ζ) ≤ 1 -ε , ∅ sinon .
L'étape 3(a) de randomisation n'a pas d'effet sur la qualité de prédiction. Elle a pour objet d'assurer la continuité de la fonction de distribution de σ2 et donc d'obtenir un taux de rejet souhaité. De plus, la consistance de σ2 implique la consistance σ2 si u tend vers 0. L'excès de risque de tout prédicteur avec option rejet est défini comme la distance suivante :

E λε (Γ) = R λε (Γ) -R λε (Γ * ε )
. Le résultat suivant est le central de ce chapitre et décrit la consistance de notre approche.

Contributions

Théorème 4. Soit ε ∈ (0, 1) et λ ε = F -1 σ 2 (1 -ε).
Supposons que σ 2 est borné, f est un estimateur consistant de f * par rapport au risque L 2 et σ2 est un estimateur consistant de σ 2 par rapport au risque L 1 . Sous des hypothèses de regularité, nous avons

E E λε Γε -→ n,M →+∞ 0, et E r( Γε ) -ε M -1/2 .
Ce résultat prouve que le prédicteur Γε est asymptotiquement aussi bon que le prédicteur optimal et de niveau de rejet ε. La convergence de l'excès de risque du Γε nécessite que deux estimateurs consistants f et σ2 . En particulier, le théorème montre que le taux de rejet du Γε est de niveau de rejet ε jusqu'à un terme d'ordre M -1/2 . Dans la suite, nous considérons le cas où ces estimateurs sont obtenus en utilisant l'algorithme des K-plus proches voisins (K-PPV). Nous déterminons la vitesse de convergence de l'excès de risque de cet estimateur sous des hypothèses techniques (voir Chapitre 3) comme par exemple la condition α-marge qui garantit que la variable aléatoire σ 2 (X) ne puisse pas être trop proche du seuil λ ε .

Théorème 5. Soit ε ∈ (0, 1) et α > 0. Si k n ∝ n 2/(d+2) et u ≤ n -1/(d+2) , alors le ε-prédicteur basé sur K-PPV Γε satisfait E E λε Γε n -2/(d+2) + log(n) (α+1) n -(α+1)/(d+2) + M -1/2 .
Ce résultat porte plusieurs enseignements. Chaque terme de la vitesse de convergence ci-dessus décrit une caractéristique donnée du problème. La première repose sur l'erreur L 2 d'estimation de la fonction de régression. La seconde, qui dépend du paramètre α, est due à l'erreur d'estimation en norme sup de la fonction de variance. Le dernier terme est directement lié à l'estimation du seuil λ ε . Dans le cas α = 1, le deuxième terme est du même ordre que le terme correspondant à l'estimation de f * . Enfin, pour α > 1 et si la taille de l'échantillon des données non étiquetées M est suffisamment grande, alors la vitesse de l'excès de risque est la même que la vitesse de convergence de f par rapport à l'erreur L 2 . Cette dernière est alors la meilleure situation à laquelle nous pouvons nous attendre pour le paramètre de rejet.

Intervalle de prédiction

Dans le dernier chapitre, nous présentons une deuxième application de l'estimation de la fonction de variance. Nous abordons ici une nouvelle approche pour construire un intervalle de prédiction en donnant sa longueur a priori dans le modèle gaussien hétéroscédastique.

Nous supposons que le bruit ε suit une loi gaussienne centrée réduite. En particulier, la variable Y |X suit une loi gaussienne de moyenne f * (X) et de variance σ 2 (X). Sa densité est donnée comme suit :

p(y|X) = 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) .
Nous faisons les hypothèses suivantes :

Hypothèse 10. Il existe 0 < σ 0 < σ 1 < ∞ tel que pour tout x ∈ R d : σ 0 ≤ σ(x) ≤ σ 1 .
Cette condition est faible et garantit que la densité conditionnelle p(y|X) est bornée. Nous supposons de plus que p(y|X) est sans atome, c'est-à-dire Hypothèse 11. Pour tout y ∈ R, l'application t → P X (p(y|X) ≥ t) est continue sur R * + . Étant donné une observation X ∈ R d , notre objectif est de produire une plage de valeurs la plus précise possible où se trouve l'étiquette correspondante Y ∈ R. Soit Γ : R d → R un intervalle de prédiction. Il a deux caractéristiques :

• l'erreur : P (Y / ∈ Γ(X)) ;
• le taux de la longueur :

L(Γ) := E X [L(Γ(X)] = E X R 1 {y∈Γ(X)} dy où L(•) représente la longueur de l'intervalle.
Nous nous intéressons à minimiser l'erreur de l'intervalle de prédiction en ajoutant une contrainte sur le taux de la longueur du Γ(X). Nous fixons ℓ > 0. Le ℓ-intervalle de prédiction optimal est :

Γ * ℓ ∈ arg min{P (Y / ∈ Γ(X)) : Γ tel que L(Γ) ≤ ℓ} .
Concernant ce problème d'optimisation : Est-ce que Γ * ℓ admet une expression explicite ? La réponse est oui. D'abord, nous introduisons la quantité suivante : ∀t > 0

G(t) := R P X (p(y|X) ≥ t)dy.
La fonction G est décroissante sur R * + . Elle est continue grâce à l'hypothèse 11. Enfin, la formule explicite de Γ * ℓ est donnée comme suit

Γ * ℓ (X) = {y : p(y|X) ≥ λ ℓ }, avec λ * ℓ := G -1 (ℓ) où G -1 est la pseudo-inverse de G 2 .
Le ℓ-intervalle de prédiction optimal repose sur un seuillage de la densité conditionnelle. Son taux de la longueur est égal ℓ. Pour mesurer la performance d'intervalle Γ, nous considérons la mesure de risque suivante

R ℓ (Γ) = P (Y / ∈ Γ(X)) + λ * ℓ L(Γ) , où le paramétre λ * ℓ contrôle l'
p(y|x) = 1 2πσ 2 (x) exp - (y -f (x)) 2 2σ 2 (x) .
3. Sur D M , nous estimons λ * ℓ , en particulier, nous calculons la fonction empirique de G :

G(t) = R 1 M M i=1
1 {p(y|X i )≥t} dy .

4. Sortie d'intervalle de prédiction plug-in de longueur ℓ

Γℓ (x) = {y ∈ R : p(y|x) ≥ G-1 (ℓ)} .
Comme la variable Y n'est pas bornée, l'étude des propriétés théoriques de l'estimateur Γℓ pourrait être difficile. Dans ce contexte, nous faisons les modifications suivantes sans perte de généralité : Cette étape assure la continuité de l'application t → P X (p(y|X) ≥ t) sur R * + pour tout |y| ≤ s. • Discrétisation. Nous approximons G par la somme de Riemann basée sur la grille régulière

• Seuillage. Soit s > 0. Nous seuillons l'estimateur p p(y|x) = 1 2πσ 2 (x) exp - (y -f (x)) 2 2σ 2 (x) 1 {|y|≤s} .
G = {y 1 , . . . , y A } de [-s, s] pour un certain A ≥ 1. Soit (ζ 1 , . . . , ζ M ) M copies i.i.d. de ζ. Nous définissons Ĝ(t) = 2s M A A k=1 M i=1 1 {p(y k |X n+i ,ζ i )≥t} .
Nous sommes en train de déterminer un intervalle de prédiction empirique dont nous pouvons analyser à la fois d'un point de vue numérique et théorique. Cet estimateur est défini comme suit

Γ (X, ζ) = {y ∈ R : p (y|X, ζ) ≥ Ĝ-1 (ℓ)} .
Le résultat suivant décrit que l'estimateur Γ a une longueur moyenne égale à la valeur demandée ℓ avec une vitesse de convergence d'ordre s/ √ M sans aucune hypothèse.

Proposition 1. Supposons que A ≥ 4 √ M , alors E |L( Γ) -ℓ| s/ √ M .
Le théorème suivant donne la consistance de Γ par rapport à l'excès de risque : Théorème 6. Supposons les hypothèses 10-11. Considérons que s = log(min(n, N )

) et A = 4s √ M . Suppo- sons que E[|f * (X)|] ≤ ∞, u = u n → 0, √ sE ( f (X) -f * (X)) 2 -→ n→+∞ 0, et que s 5/2 E |σ 2 (X) -σ 2 (X)| -→ n→+∞ 0, alors E E λ ℓ Γℓ -→ n,N →+∞ 0 .
Nous appliquons notre méthodologie à l'algorithme des K-PPV en ajoutant la condition α-marge basée sur la variable p(Y |X). Théorème 7. Supposons que les hypothèses 1-2 et 10-11 sont satisfaites. Soit

k n ∝ n 2/(d+2) . Pour s = log (min(n, M )), A = 4s √ N et u n = 1/n, nous avons alors E E λ ℓ Γℓ log(n) n -(1+α)/(d+2) + min(n, N ) -(1+α) + N -1/2 .

Conclusion et perspectives de recherche

Conclusion. Cette thèse a proposé des procédures d'estimation de la fonction de variance dans le modèle de régression hétéroscédastique -l'algorithme des K-PPV -et les procédures d'agrégation classiques "agrégation de type sélection modèle et convexe" (MS et C respectivement). Nous avons utilisé la méthode basée sur les erreurs résiduelles. Sous des hypothèses techniques, nous avons déterminé la vitesse de convergence de ces méthodes. La vitesse de convergence de l'estimateur de la fonction de variance par K-PPV par rapport à la ℓ p -norme sup est d'ordre log(n) p n -p/d+2 . De plus, nous avons utilisé deux échantillons indépendants pour construire les estimateurs de la fonction de variance par MS et C. Nous avons montré que les deux approches d'agrégation ont la même vitesse de convergence qui est d'ordre (log(M 1 )/N ) -1/4 dans le cas où Y est borné et d'ordre (log(M 1 )/N ) -1/8 dans le cas gaussien. Ces vitesses lentes sont dues au fait que l'estimation de la fonction de variance repose sur la fonction de régression que l'on doit également estimer par les méthodes d'agrégation. Cette thèse a développé deux nouvelles applications de la fonction de variance : le problème de la régression avec option rejet et les intervalles de confiance en régression.

• Problème de la régression avec option rejet. Nous avons développé la prédiction avec option rejet dans le cadre de la régression. Nous nous sommes concentrés sur le cas où le taux de rejet est fixé. Nous avons fourni une règle optimale. Elle repose sur le seuillage de la fonction de variance. En utilisant deux échantillons indépendants (un de données étiquetées et un de données non étiquetées), nous avons construit un algorithme semi-supervisé basé sur le principe de type plug-in qui peut être appliqué à tout estimateur standard des fonctions de régression et de variance. Nous avons prouvé que notre prédicteur de type plug-in a un taux de rejet ε ∈ (0, 1) à un terme d'ordre M -1/2 près où M est la taille de l'échantillon des données non étiquetées. Nous avons obtenu un résultat de consistance de manière générale sur l'excès de risque qui dépend de la consistance des estimateurs des fonctions de régression et de variance par rapport à l'erreur L 2 . Nous avons également appliqué notre méthodologie à l'algorithme des K-PPV, c'est-à-dire nous avons estimé f * et σ 2 par K-PPV. Nous avons établi la vitesse de convergence pour le prédicteur de type plug-in basé sur K-PPV en utilisant la condition α-marge pour la variable σ 2 (X) autour du seuil λ ε . Nous avons remarqué que si α > 1 et si M est suffisamment grande, alors la vitesse de l'excès de risque est la même que la vitesse de convergence de f par rapport à l'erreur L 2 .

• Intervalle de prédiction en régression. Nous avons construit un intervalle de prédiction avec une longueur moyenne fixé dans le modèle gaussien hétéroscédastique. Nous avons dérivé une règle optimale qui repose sur le seuillage de la densité conditionnelle, qui elle-même dépend des fonctions de régression et de variance. Cette règle permet d'interpréter la variable de sortie. Comme dans le problème d'option rejet, nous avons proposé une approche de type plug-in pour l'intervalle de prédiction optimal et pour l'estimation de la densité conditionnelle. Cette approche peut être adaptée à n'importe quel estimateur des fonctions de régression et de variance. Nous avons étudié la consistance de notre prédicteur résultant en termes d'excès de risque et de taux de longueur. Nous avons prouvé que notre estimateur a la même longueur que l'intervalle de prédiction optimal à un terme d'ordre s/ √ M près, où le paramètre s ≥ 1. Nous avons déterminé la vitesse de convergence en se basant sur l'algorithme des K-PPV.

Enfin, les procédures proposées dans cette thèse sont facilement implémentables. Les résultats obtenus dans ce manuscrit ouvrent des pistes pour des futurs travaux Perspectives de recherche. Les travaux en cours portent sur plusieurs directions.

1. Estimation de la fonction de variance. Dans le deuxième chapitre, nous avons obtenu des vitesses de convergence de nos méthodes d'agrégation. Ce serait une motivation pour établir des bornes inférieures afin d'étudier l'optimalité des vitesses. De plus, une question intéressante serait d'étudier l'estimation de la fonction variance en grande dimension puisque travailler dans des espaces en grande dimension apporte souvent des difficultés théoriques qui viennent s'ajouter aux problèmes numériques de temps de calcul.

2. Intervalle de prédiction. Nous avons déterminé la borne supérieure de l'excès de risque de notre intervalle de prédiction empirique. Dans ce contexte, nous voudrions aborder la question des vitesses minimax en cas général. De plus, nous avons construit un intervalle de prédiction dans le cas où la variable Y |X admet une densité. Il pourrait être intéressant de proposer une construction dans un cadre général où cette hypothèse n'est pas nécessairement vérifiée.

Chapter 2

Variance function estimation in regression model via aggregation procedures

Abstract: In the regression problem, we consider the problem of estimating the variance function by the means of aggregation methods. We focus on two particular aggregation setting: Model Selection aggregation (MS) and Convex aggregation (C) where the goal is to select the best candidate and to build the best convex combination of candidates respectively among a collection of candidates. In both cases, the construction of the estimator relies on a two-step procedure and requires two independent samples. The first step exploits the first sample to build the candidate estimators for the variance function by the residual-based method and then the second dataset is used to perform the aggregation step. We show the consistency of the proposed method with respect to the L 2 -error both for MS and C aggregations. We evaluate the performance of these two methods in the heteroscedastic model and illustrate their interest in the quantile regression.

Introduction

Building efficient estimation of the level of noise is highly important for real applications and statistical analysis. In the heteroscedastic regression, which corresponds to the case where the variance of the errors depends on input variables, the heteroscedasticity must be detected and estimated. Indeed, not taking it into account in the estimation invalidates the conclusions of many statistical inference problems such as statistical tests which assume that the errors of the model all have the same variance. In addition, when using an approach that estimates the error variance as a function of input variables, the prediction intervals we obtain are likely to be more realistic than those obtained by assuming that the error variance is constant since the predictive uncertainty estimate depends on the estimate of the variance of the response variable. In general, testing and confidence intervals are two historical statistical problems where a bad calibration of the noise may lead to bad conclusions. Another important aspect of estimating the heteroskedasticity of the model is that the point estimate of the regression function is directly related to the variance function. The range of use of the variance structure in the data is even wider such as in selection the optimal kernel bandwidth [START_REF] Fan | Design-adaptive nonparametric regression[END_REF], estimation correlation structure of the heteroscedastic spatial data [START_REF] Opsomer | Kriging with nonparametric variance function estimation[END_REF], estimation of the quantile regression [START_REF] Koenker | Quantile Regression[END_REF][START_REF] Shan | Combining regression quantile estimators[END_REF], estimation of the signal-to-noise ratio [START_REF] Verzelen | Adaptive estimation of high-dimensional signal-to-noiseratios[END_REF], or choosing optimal design [START_REF] Müller | Estimation of heteroscedasticity in regression analysis[END_REF] and finding important applications, for instance, in finance with the problems of measuring volatility or risk [START_REF] Anderson | Estimating continuous-time stochastic volatility models of the short-term interest rate[END_REF] or long-term stock returns [START_REF] Mammen | Conditional variance forecasts for longterm stock returns[END_REF]. In our case, we highlight the interest in providing an efficient estimation of the variance function in the problem of regression with regret option where the good calibration of the rejection rule is highly dictated by the efficiency of the estimator of the noise level [START_REF] Denis | Regression with reject option and application to knn[END_REF]. We focus on the regression problem: we denote by (X, Y ) ∈ R d × R the couple of random variables where X is the feature vector and Y is the response variable such that

Y = f * (X) + W .
Here W is the noise and is such that

E[W |X] = 0 and E[W 2 ] < ∞. In particular for any x ∈ R d we write f * (x) = E [Y |X = x]
to denote the regression function and

σ 2 (x) = Var(Y |X = x) = E (Y -f * (X)) 2 |X = x to denote the conditional variance function.
Despite the popularity of the problem of estimating the noise level, there remains much to do. In particular, we study in the present chapter this problem from the aggregation perspective and build estimators of the conditional variance function based on Model Selection (MS) and Convex (C) aggregations. We study their consistency properties as well as their numerical performances.

Related work

Our literature review consists of three related fields:

Conditional variance estimation:

The problem of estimating the regression function is classical and widely studied, see for example [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Scornet | Consistency of random forests[END_REF][START_REF] Stone | Consistent nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF] and references therein.

Even though the problem of estimation of the conditional variance function is less studied, it has been considered in several works that can be cast into two groups according to the nature of the design (fixed or random).

When the design is fixed, the estimation of σ 2 has been studied mainly via residual-based methods [START_REF] Hall | Variance function estimation in regression: the effect of estimating the mean[END_REF][START_REF] Härdle | Local polynomial estimators of the volatility function in nonparametric autoregression[END_REF]74] and difference-based methods [START_REF] Brown | Variance estimation in nonparametric regression via the difference sequence method[END_REF][START_REF] Müller | Estimation of heteroscedasticity in regression analysis[END_REF][START_REF] Wang | Effect of mean on variance function estimation in nonparametric regression[END_REF]. Difference-based estimators do not require the estimation of the regression function f * . The first difference-based estimators have been developed by [START_REF] Müller | Estimation of heteroscedasticity in regression analysis[END_REF]. They considered an initial variance estimates which are squared weighted sums of m observations neighbouring the fixed point where the variance function is to be estimated. The authors showed that the proposed initial variance estimates are not consistent. To solve this problem, they smoothed them with a kernel estimate. [START_REF] Brown | Variance estimation in nonparametric regression via the difference sequence method[END_REF] presented a class of non-parametric variance estimators based on different sequences and local polynomial estimation and established asymptotic normality. [START_REF] Wang | Effect of mean on variance function estimation in nonparametric regression[END_REF] were interested in the effect of the unknown mean on the estimation of the variance function and proved numerically that the residual-based method performs better than the first-orderdifference-based estimator when the unknown regression function f * is very smooth.

In this work, we rather focus on random design. Less methods have been proposed to estimate the conditional variance function in this case. Most classical methods are the direct and the residualbased:

1. The direct method: a simple decomposition the conditional variance function σ 2 is rewritten as the difference of the first two conditional moments,

σ 2 (x) = E[Y 2 |X = x] -(E[Y |X = x]) 2 .
The direct method consists in estimating the two terms in the right side separately, see for example [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF][START_REF] Härdle | Local polynomial estimators of the volatility function in nonparametric autoregression[END_REF]. To be more specific, the direct estimator of σ 2 has the following form 2 , where ĝ and f are estimators of E[Y 2 |X = x] and f * , respectively. The main drawback of this approach is that it is not always nonnegative, for example, if different smoothing parameters are used in estimating those terms and adaptation to the unknown regression function f * is still not available. [START_REF] Härdle | Local polynomial estimators of the volatility function in nonparametric autoregression[END_REF] proposed a local polynomial regression estimates of those terms using the same bandwidth and the same kernel function. They established the asymptotic normality of local polynomial estimators of the regression function and the variance function.

σ2 d (x) = ĝ(x) -( f (x))
2. The residual-based method: this approach consists of two steps. First, one estimates the regression function and computes the squared residuals r = (Y -f (X)) 2 where f is the estimator of f * . Second, we estimate the variance function by solving the regression problem when the input is the feature X and the output variable is the residuals r . For more details, see [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF]67,74].

It exists many ways to study this method. For instance, [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF] applied a local linear regression in both steps and showed that their estimator is fully regression-adaptive to the unknown regression function. Using the local polynomial regression can be negative when the bandwidths are not selected appropriately. As a solution to this, [START_REF] Yu | Likelihood based-local linear estimation of the conditional variance function[END_REF] proposed estimators based on a localised normal likelihood, using a standard local linear form for estimating the mean and a local log-linear form for estimating the variance. [START_REF] Flavio | Nonparametric estimation of volatility functions: The local exponential estimator[END_REF] introduced an exponential estimator of the conditional variance in the second step to ensure the nonnegativity of the estimator of σ 2 .

[101] used a reweighted local constant estimator (kernel estimator) based on maximizing the empirical likelihood subject to a bias-reducing moment restriction. Moreover, such estimators have the form σ2 (X) = i ω i (X)(Y i -f (X i )) 2 where ω i (X) are weight functions ( [START_REF] Denis | Regression with reject option and application to knn[END_REF][START_REF] Kulik | Nonparametric conditional variance and error density estimation in regression models with dependent errors and predictors[END_REF]).

Recently, [START_REF] Denis | Regression with reject option and application to knn[END_REF] used the previous estimator and focused on estimating the regression function and the variance function respectively, by kNN. Under mild assumptions, they provided the rate of convergence of the kNN estimator of the conditional variance function in supremum norm. The residual-based method can be regarded as a generalized difference-based estimator. For more details, see [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF].

In this chapter, we focus on the residual-based method to estimate the variance function since they appear more tractable. In particular, we develop an aggregation procedures for this task.

Aggregation methods: Aggregation is a popular approach in statistics and machine learning. This technique is well known to estimate the regression function in the homoscedastic or heteroscedastic model. We refer the reader to the baseline articles [4,[START_REF] Bunea | Aggregation for gaussian regression[END_REF][START_REF] Juditsky | Functional aggregation for nonparametric regression[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF][START_REF] Tsybakov | Aggregation and minimax optimality in high-dimensional estimation[END_REF][START_REF] Yuhong | Aggregating regression procedures to improve performance[END_REF]. Given a set of estimators of regression function f * , the aggregation constructs a new estimator, called the aggregate, which mimics in a certain sense, the behavior of the best estimator in a class of estimates. There are several popular types of aggregation and we focus on two: the model selection aggregation (MS) which allows to select the best estimator from the set; the convex aggregation (C) where the goal is to select the best convex combination of functions in the set. In general, the aggregation procedures are based on sample splitting, that is, the original data set D N is split into two independent data sequences D k and D l with N = l + k ≥ 1. The first subsample D k is exploited to build M > 1 competing estimators of the regression function f * and D l is used to aggregate them. Most of the work has focused on fixing the first sample, resulting in fixed estimators (the estimators are then seen as fixed functions). Under mild assumptions, the authors in [START_REF] Tsybakov | Optimal rates of aggregation[END_REF] showed that the optimal rates for MS and C aggregation w.r.t. L 2 -error in gaussian regression model are of order log(M )

N , and

M N if M ≤ √ N , respectively, 1 N log M √ N + 1 if M ≥ √ N in both cases.
In this chapter, we consider aggregation methods for the conditional variance estimation. Up to our knowledge, such approaches have not been considered yet for this problem.

Main contribution

We develop the notions of model selection aggregation and convex aggregation to estimate the conditional variance function. To our knowledge, this work is the first to deal with this setting. We consider two independent datasets: the first will be used to build the initial estimators of the variance function and the second to aggregate them. We call these estimators the MS-estimator and Cestimator. We consider the residual-based method to build the initial estimators which is based on estimating the regression function in the first step. We focus on estimating the regression function by model selection aggregation and convex aggregation. In this chapter, the major part is then devoted to show the upper-bounds of L 2 -error of the MS-estimator and C-estimator when the initial estimators can be arbitrary or verify very weak conditions such that boundedness. We establish that the rate of convergence for MS and C procedures is of order O((log(M 1 )/N ) 1/8 ) when Y is unbounded and is of order O((log(M 1 )/N ) 1/4 ) when Y is bounded. Finally, we obtain numerical results which show the performance of our procedures.

Outline

The chapter is organized as follows. In the next section, the aggregation problems, the model selection and convex aggregations, are described in detail. Section 2.3 is focused on investigating the upper-bounds for the L 2 -error of our procedures. Finally, Section 2.4 presents a numerical comparison of the proposed method w.r.t. the heteroscedastic model as well as a direct application to the quantile regression.

Notations. We introduce some notation that is used throughout this chapter. Let p ≥ 2 be an integer, the set of integers {1, . . . , p} is denoted

Aggregation estimators

In this section, we describe an estimation algorithm of the variance function σ 2 by aggregation. In particular, we focus on two types of aggregations: the model selection aggregation (MS), and the convex aggregation (C). These aggregation problems, (MS) and (C), have been considered to estimate the unknow regression function in the regression model. The objective is to estimate f * by a combination of elements of a known set called dictionary made up of deterministic functions or preliminary estimators. The collection of estimators or algorithms is given and can be parametric, nonparametric or semi-parametric nature. Given a set of estimators, the MS-aggregation consists in constructing a new estimator which is approximately as good as the best estimator in the set, while the objective of C-aggregation is to construct a new estimator which is approximately as good as the best convex combination of the elements in the set, for more details see [4,[START_REF] Bunea | Aggregation for gaussian regression[END_REF][START_REF] Juditsky | Functional aggregation for nonparametric regression[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF][START_REF] Tsybakov | Aggregation and minimax optimality in high-dimensional estimation[END_REF][START_REF] Yuhong | Aggregating regression procedures to improve performance[END_REF]. Besides, to construct an aggregate of σ 2 , we first introduce two independent learning samples: D n = {(X i , Y i ), i = 1, . . . , n} and D N = {(X i , Y i ), i = n+1, . . . , n+N } which consist of respectively, n and N i.i.d. copies of (X, Y ).

Model selection aggregation

In this first paragraph, we detail how we perform MS-aggregation in order to estimate of the conditional variance function σ 2 by MS. It consists of two steps: one step for the estimation of the regression function f * and a second one devoted to the estimation of σ 2 . More precisely, in the first one we build 2.2 Aggregation estimators M 1 estimators of the regression function f1 , . . . , fM 1 based on D n with 2 ≤ M 1 < ∞. Then, we use the second dataset D N to estimate f * by MS: we select the optimal index, denoted ŝ as follows

ŝ ∈ argmin s∈[M 1 ] RN ( fs ), where RN ( fs ) = 1 N n+N i=n+1 |Y i -fs (X i )| 2 , (2.1)
and the aggregate of the regression function, denoted by fMS , is then given by fMS := fŝ .

(2.2)

In the second step, given the estimator fMS built on D N , we construct using back the sample D n M 2 estimators of the variance function σ 2 , denoted σ2 ŝ,1 , . . . , σ2 ŝ,M 2 , by residual-based method with 2 ≤ M 2 < ∞. Finally, based on D N again, we select the optimal single, denoted m, as follows

m ∈ argmin m∈[M 2 ] RN (σ 2 ŝ,m ) where RN (σ 2 ŝ,m ) = 1 N n+N i=n+1 | Ẑi -σ2 ŝ,m (X i )| 2 with Ẑi = Y i -fMS (X i ) 2 .
Therefore, the aggregate of the variance function, called MS-estimator and denoted σ2 MS , is defined as

σ2 MS := σ2 ŝ, m.
(2.3)

Convex aggregation

Convex aggregation procedures for nonparametric regression are discussed in [START_REF] Audibert | Aggregated estimators and empirical complexity for least square regression[END_REF][START_REF] Bunea | Aggregation for gaussian regression[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF]. We describe here an algorithm for aggregating estimates of the conditional variance function σ 2 by C-aggregation.

As for MS-aggregation, the construction of the aggregate of σ 2 needs two independent datasets D n and D N . The aggregation still proceeds in two steps: one for estimating f * and the second for the estimation of σ 2 . Each step is again decomposed in two parts. Firstly, we consider M 1 estimators of the regression function f * , { f1 , . . . , fM 1 }, based on D n , and for any λ ∈ Λ M 1 we define the linear combinations fλ fλ =

M 1 j=1 λ j fj .
Then, aggregates of the regression function based on the sample D N have the form

fC := fλ = M 1 j=1 λj fj , (2.4)
where the estimator λ is defined by λ ∈ argmin

λ∈Λ M 1 RN ( fλ ).
Once fC is obtained, we focus on the estimation of σ 2 . Based on the sample D n , we build M 2 estimators for the conditional variance function by the residual-based method, denoted σ2 λ,1 , . . . , σ2

λ,M 2
, and for any β ∈ Λ M 2 we define σ2 λ,β as follows

σ2 λ,β = M 2 j=1 β j σ2 λ,j .
Finally, based on D N , the aggregate estimate for σ 2 is given by

σ2 C := σ2 λ, β , (2.5)
where the estimator β is defined by

β ∈ argmin β∈Λ M 2 RN (σ 2 λ,β ), where RN (σ 2 λ,β ) = 1 N n+N i=n+1 | Ẑi -σ2 λ,β (X i )| 2 with Ẑi = (Y i -fC (X i )) 2 .
We called σ2 C the C-estimator.

Main results

This section is devoted to studying the L 2 -error of MS-estimator and C-estimator. Firstly, we introduce general conditions required on the model in Section 2.3.1. Secondly, we show the consistency of our methods in Sections 2.3.2 and 2.3.3.

Assumptions

The following assumptions are the bedrock of our theoretical analysis:

Assumption 1. The functions f * and σ 2 are bounded.

Assumption 2. Y is bounded or Y satisfies the gaussian model

Y = f * (X) + σ(X)ξ, (2.6)
where ξ is independent of X and distributed according to a standard normal distribution.

These assumptions are relatively weak and play a key role in our approach. They allow to use the Hoeffding's inequality in the case of boundness of Y or ξ. In particular, it is important to emphasize that Assumptions 1 and 2 guarantee that the variable Yf * (X) is sub-Gaussian (see Lemma 9 in the case where Y is bounded).

Upper bound for σ2

MS

We study the L 2 -error of the MS-estimator σ2

MS . Let R( fs ) = E |Y -fs (X)| 2 for all s ∈ [M 1 ]. We define s * as follows s * ∈ argmin s∈[M 1 ] R( fs ) . (2.7)
Besides, we need the following assumptions in the case of MS-aggregation: fs ∞ ≤ K 1 , and max

(s,m)∈[M 1 ]×[M 2 ] σ2 s,m ∞ ≤ K 2 .

Assumption 4 (Separability hypothesis).

There exists δ 0 > 0 such that

δ * (D n ) = min s =s * {|R( fs ) -R( fs * )|} > δ 0 .
Both assumptions are used to control the L 2 -error of the MS-estimator σ2 MS . Assumption 3 describes the boundedness of the estimators. It is in particular satisfied if the functions in the dictionaries are bounded. In our construction, the constants K 1 and K 2 do not need to be known. In practice, the response variable Y in the sample is finite and then it ensures that the candidates in the dictionaries are bounded. Assumption 4 is used for MS and helps us to ensure that the minimum of the risk R is well defined. It cannot be verified in practice since it depends on the distribution P. Let E be the expectation which is taken with respect to both X and the samples D n and D N . We establish the following result Theorem 9. Let fMS and σ2 MS be two MS-estimators of f * and σ 2 defined in Eq. (2.2) and (2.3), respectively. Then, under Assumptions 1-4, there exist two absolute constants

C 1 > 0 and C 2 > 0 such that E |σ 2 MS (X) -σ 2 (X)| 2 ≤ E min m∈[M 2 ] E X |σ 2 s * ,m (X) -σ 2 (X)| 2 + C 1 min s∈[M 1 ] E fs -f * 2 N 1/2p + C 2 log(M 1 ) N 1/4p , with p = 1 if Y is bounded and p = 2 otherwise.
The proof of this result is postponed to the Appendix. Let's give a sketch of the proof. The L 2 -error is the excess risk of σ2

MS where E

|σ 2 MS (X) -σ 2 (X)| 2 := E R(σ 2 MS ) -R(σ 2 ) with R(σ 2 ) = E |Z -σ 2 (X)| 2 . We introduce the minimizer σ2 MS := σ2 ŝ, m where m ∈ argmin m∈[M 2 ] R(σ 2 ŝ,m ). We consider the decomposition E |σ 2 MS (X) -σ 2 (X)| 2 = E R(σ 2 MS ) -R(σ 2 MS ) + E R(σ 2 MS ) -R(σ 2
) . We control the two terms in the right side separately. The first one is the estimation error (variance term). To control it, we need to introduce σ2

MS := σ2 ŝ, m where m ∈ argmin m∈[M 2 ] R N (σ 2 ŝ,m ), with R N (σ 2 ŝ,m ) = 1 N n+N i=n+1 |Z i -σ2 ŝ,m (X i )| 2 .
The upper bound of the variance depends on the L 2 -error of the aggregate fMS with respect to the empirical norm. The second one is the approximation error. Its upper-bound is linked to P(ŝ = s * ).

Theorem 9 gives the upper-bound for L 2 -error of σ2

MS . This bound consists of two parts: the first part is the bias of MS-estimator σ2

MS and depends on the deterministic selector s * ; the second part is composed of the two remaining terms and corresponds to the estimation error (variance). The first term is the bias term of fMS expressed in terms of the empirical norm • 2 N , and the second one characterises the price to pay for MS-aggregation and is of order (log(M 1 )/N ) 1/4p where p = 1 if Y is bounded and p = 2 otherwise. Note that this rate is slower than in the case of the estimation of the regression function f * . This slow rate is due to the double aggregation that we need to perform for the estimation of the conditional variance function.

Upper bound for σ2

C

In this part, we focus in studying the L 2 -error of σ2

C . The construction of σ2 C needs the following estimators { fi } M 1 i=1 and {σ λ,i } M 2 i=1 . We require the following conditions

Assumption 5. For all i ∈ [M 1 ], all λ ∈ Λ M 1 and all j ∈ [M 2 ]
, fi and σ2 λ,j are bounded a.s. D n .

Assumption 6. Suppose that there exists a constant K ≥ 0 such that for every j

∈ [M 2 ] E |σ 2 λ 1 ,j (X) -σ2 λ 2 ,j (X)| ≤ K λ 1 -λ 2 1,M 1 , ∀λ 1 , λ 2 ∈ Λ M 1 a.s D n .
Assumption 5 describes the boundedness of the candidates as Assumption 3. Assumption 6 is a strong condition. However, it holds, for instance, for estimators of the form σ2 λ,j (X) = i ω i (X)(Y ifλ (X i )) 2 where ω i (X) are weight functions, that are nonnegative and sum to one. The next theorem is the main result of this section, it displays the upper-bound of L 2 -error for σ2 C .

Theorem 10. Let fC and σ2 C be two C-estimators of f * and σ 2 defined in Eq. (2.4) and (2.5), respectively. Then, under Assumptions 1, 2, 5, and 6, there exist two absolute constants

C 1 > 0 and C 2 > 0 such that E |σ 2 C (X) -σ 2 (X)| 2 ≤ E inf β∈Λ M 2 E X |σ 2 λ,β (X) -σ 2 (X)| 2 + C 1 inf λ∈Λ M 1 E fλ -f * 2 N 1/2p + C 2 log(M 1 ) N 1/4p
, with p = 1 if Y is bounded and p = 2 otherwise.

As for Theorem 9, the upper-bound for the L 2 -error of C-estimator σ2 C is composed of three terms. The first one is the bias term of σ2

C which depends on the random selector λ, the second and third ones is a bound of the variance term that rely on the bias term of fC with respect to the empirical norm • 2 N and on the price to pay for convex aggregation which is of order (log(M 1 )/N ) 1/4p where p = 1 if Y is bounded and p = 2 otherwise.

We notice that both procedures MS and C have the same rate. Indeed, the variance term of σ2 MS and σ2

C is based on the upper bound for fMS and fC . Moreover, the aggregates fMS and fC have the same rate which is of order (log(M 1 )/N ) 1/2 with respect to the empirical norm • 2 N , see Proposition 2 and Proposition 3 in the Appendix. Let us now compare with the rates of fMS and fC with respect to L 2 -error and L 2 -risk. For the Gaussian and bounded regression model, the rate of the variance term of fMS and fC is of order log(M 1 )

N and M 1 N if M 1 ≤ √ N , respectively, 1 N log M 1 √ N + 1 if M 1 ≥ √ N in
both cases [START_REF] Bunea | Aggregation for gaussian regression[END_REF][START_REF] Lecué | Empirical risk minimization is optimal for the convex aggregation problem[END_REF][START_REF] Lecué | Aggregation via empirical risk minimization[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF]. We can deduce that our rates are very slow because our procedures need to estimate at the same time the unknown regression function f * and the variance function σ 2 by aggregation procedures.

Numerical results

This section is devoted to the numerical analysis of our procedures. In Section 2.4.

Data

Our numerical study relies on synthetic data: Heteroscedastic models: we propose four examples of heteroscedastic models (2.6):

• Model 1: let a ∈ {1/4, 1} and X = (X 1 , X 2 , X 3 ) have a uniform distribution on [0, 1] 3 . Let 1. f * (X) = 0.1 cos(X 1 ) + exp(-X 2 3 ); 2. σ 2 (X) = a 0.1 + exp(-7(X 1 -0.2) 2 ) + exp(-10(X 2 -0.5) 2 + exp(-50(X 3 -0.9) 2 . • Model 2: let X = (X 1 , . . . , X 10 ) have a uniform distribution on [0, 1] 10 . We define 1. f * (X) = 0.1 + exp(-X 2 1 ) + 0.2 sin(X 2 + X 3 + X 4 + 0.1X 2 5 ); 2. σ 2 (X) = 1 2 (0.5+ X 1 (1 -X 2 )+0.8X 3 X 4 +X 5 X 6 X 2 7 +0.9 exp(-500(X 8 +X 9 +X 10 -0.5) 2 )) 2 . • Model 3: sparse model 1. f * (X) = Xβ, β ∈ R p ; 2. σ 2 (X) = 1 2 0.3 + X 1 (1 -X 1 ) sin 2.1π X 2 +0.05 + 0.5X 3 + X 4 2 .
Bounded Y : we consider the following regression model when Y is bounded

Y = f * (X) + σ(X)̺
where ̺ have a uniform distribution on

[- √ 3, √ 3
]. We give the following examples of models :

• Model 4: let X have a uniform distribution on [0, 1] 2 and 1. f * (X) = X 1 + exp(-X 2 2 ); 2. σ 2 (X) = 0.01 + X 1 exp -(X 2 -0.9) 2 . • Model 5: let X = (X 1 , X 2 , X 3 ) have a uniform distribution on [0, 1] 3 and 1. f * (X) = X 1 + X 2 + 0.5 cos(X 3 ); 2. σ 2 (X) = 0.3 + X 1 (1 -X 1 ) sin (2.1)π X 2 +0.05 + X 3 2 .
We describe the previous models. We display in Figures 2.1 and 2.2 the histograms of the variance function for every model. Model 1 is a multivariate model in which the regression and variance functions are regular functions. In the case a = 1/4, the problem of estimation of σ 2 is hard since it takes a large proportion of values smaller than 1, while the case a = 1 is simpler because about 76.3% of the values of σ 2 are larger than 1 and 0.04% larger than 3. Moreover, Model 2 is also a multivariate model where we introduce higher order terms in the variance function. In this sense, the estimation of the variance function is hard since in addition, there are only 28% of values of σ 2 greater than 1. In Model 3 we consider a sparse model for the regression function where X is an N × p matrix (p is the number of predictors) with independent uniform entries, β ∈ R p is a vector of weights, and ξ ∈ R N is a standard Gaussian noise vector and is independent of the feature X. We fix p = 50. The vector β is chosen to be s-sparse where s < p, that means β has only first s coordinates different from 0;

β i = 1 {i≤s} .
Here, we choose s = 14. In addition, the variance function in this model is less difficult. Indeed, σ 2 takes only 24.8% values greater than 1. Finally, the last two examples are two models when Y is bounded. Model 4 is bivariate regression model where the estimation of σ 2 is difficult (about 99.8% of the values are less than 1). Lastly, considering Model 5, the values of σ 2 are between 0 and 3.12. There are 36.6% of values that are larger than 1. From this perspective the estimation of the variance function is less complicated. However, the presence of higher order terms makes the problem harder.

Benefit of aggregation

In this section, we improve the classical methods based on residual-based approach by considering aggregation. In the same time we compare MS and C aggregation.

Machines and simulation scheme

The construction of the aggregates σ2 MS and σ2 C is described in Sections 2.2.1 and 2.2.2. We recall that we focus on the residual-based method to compute the candidates of the variance function σ 2 . One of the advantages of using the aggregation approach is that the collection of candidates is chosen by the practitioner and can be arbitrary. We build three dictionaries

F = { fs } 12 s=1 , G 1 = σ2 ŝ,12 12
m=1
and G 2 = {σ2 λ,j } 12 j=1 that contain 12 machines each: the random forest with different number of trees (ntree=50, 150, 500), the kNN with different values of k (k = 7, 13, 22), the Lasso with different values of tuning parameter (λ = 0.5, 2), the Ridge with different values of tuning parameter (λ = 0.9, 3), regression tree and the Elastic Net regression with a penalty term λ = 1 and a parameter α = 0.6 that compromises between the ℓ 1 and the ℓ 2 terms in the penalty. The first dictionary is exploited to compute the aggregates fMS and fC while the last two are used to calculate respectively, σ2

MS and σ2 C with those 12 machines. For the 12 algorithms, we use the following R packages:

• Regression tree (R package tree, [START_REF] Ripley | tree: Classification and regression trees[END_REF]);

• k-nearest neighbours regression (R package FNN, [START_REF] Li | Fnn: Fast nearest neighbor search algorithms and applications[END_REF]);

• RandomForest regression (R package randomForest, [START_REF] Liaw | Classification and regression by randomforest[END_REF]);

• Lasso regression (R package glmnet, [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF]);

• Ridge regression (R package glmnet);

• Elastic Net regression (R package glmnet).

Other parameters are set by default. In addition to that, we use Optim function in R which is based on method BFGS to compute λ and β. Now, we evaluate the performances of σ2 MS and σ2 C on previous models. Besides, we provide estimation of the L 2 -error for σ2

MS and σ2 C and repeat independently L = 100 times the following steps: s,m } 1≤s,m≤12 and we choose the best among them in terms of empirical L 2 -error (always on the datatest D T ). That means, we take the smallest of empirical L 2 -error as follow: for all (s, m)

∈ [12] × [12] min 1 T T i=1 σ2 s,m (X i ) -σ 2 (X i ) 2 , (2.8)
and denote the best method. Finally, we compare it with our aggregation methods.

From these experiments, we compute the means and standard deviations of both empirical L 2errors Err for σ2 MS , σ2 C and the best method and we display the boxplot of the empirical L 2 -error.

Results

We present our results in Figures 2.3-2.8 and Tables 2.1 and 2.2. We make several observations. First, the convex aggregation method is better than the model selection aggregation method in all models.

The best can only serve as a benchmark to see the performance of a real estimator. For this reason, it is perfectly natural that "best" has better performances than our aggregation procedures on the test sample D T because the selection of the best couple (s, m) (see Eq. (2.8)) depends precisely on D T . Second, we notice that when n and N are enough, the MS-estimator σ2 MS and the C-estimator σ2 C have a similar performance, that is close to the performance of the best method. These results reflect our theory: the consistency of MS-estimator and the C-estimator. Third, we observe that the empirical L 2 -error of σ2

MS and σ2 C decreases faster in the simpler models (with respect to the estimation of the variance function) when n and N increase (see the evolution of the boxplots in Figures 2.4 and 2.8 as compared to Figures 2.3 and 2.6. In addition, our numerical results highlight an interesting fact: when we split data, it is advantageous to put more data in the second dataset D N used in the aggregation step. Indeed, it seems as illustrated in Table 2.2 that the methods have better performance for large samples D N is all cases. As an example, the mean error in Model 1 with a = 1 for C-aggregation is 0.33 when n = 1000 and N = 100 and 0.26 when n = 100 and N = 1000.

Real datasets

In this part, we consider two real datasets which are available on the UCI database. s,m } 1≤m≤4 of σ2 corresponding to the 4 procedures in F. In the last step, we compute the empirical L 2 -risk of

• MS-method: 1 T T i=1 (Y i -fMS (X i )) 2 -σ2 MS (X i ) 2 ;
• C-method:

1 T T i=1 (Y i -fC (X i )) 2 -σ2 C (X i ) 2 ;
• best-method (best empirical L 2 -risk): based on Step 4, we take the minimum of

1 T T i=1 (Y i -fs (X i )) 2 -σ2 s,m (X i ) 2 for all (s, m) ∈ [4] × [4].
From this estimates, we compute the mean and the standard deviation of the empirical L 2 -risk. The associated boxplots are given in second dataset. We observe that both of our aggregation methods achieve the same performance. Note that the aggregation methods are outperformed by the best method when n is large. This is mainly due to the fact that the method called " best" is computed without splitting the data (as explained before). Finally, we notice that it is advantageous to put a lot of data in the first dataset.

Application of variance function: Quantile regression

In this section, we illustrate the performance of our aggregation methods for quantile regression. Quantile regression allows a comprehensive analysis of the relationships between a response Y and input variables X. It is interesting in the entire conditional distribution of the dependent variable, and not only on its mean. For more details see for instance [START_REF] Koenker | Quantile Regression[END_REF][START_REF] Shan | Combining regression quantile estimators[END_REF][START_REF] Takeuchi | Nonparametric quantile estimation[END_REF]. We recall that in our work we observe (X,

Y ) ∈ R d × R such that Y = f * (X) + σ(X)ξ,
where ξ is the known noise with mean zero and unit variance. Let τ ∈ (0, 1). The conditional τquantile of Y given X = x, denoted by q τ , is given by

q τ (Y |X) = inf{y : F Y |X (y) ≥ τ },
where F Y |X is the cumulative distribution function of Y |X. In particular, the quantity q τ (Y |X) has the following form

q τ (Y |X) = f * (X) + σ(X)F -1 ξ (τ )
, where F ξ is the cumulative distribution function of ξ and is known. The plug-in approach is a possible procedure to estimate q τ . Given an estimator f of f * and an estimator σ2 of σ 2 , the plug-in estimator of q τ is qτ (Y |X) = f (X) + σ(X)F -1 ξ (τ ). It is clear that a good estimate of the conditional τ -quantile is related to a good estimate of the regression function f * and the variance function σ 2 . We evaluate the performance of qτ (Y |X) according to Firstly, we can see here again that our both aggregation based approaches have the same performances. Secondly, the svm method which is built on the union of two samples D n and D N is the best method for the median problem estimation (τ = 0.5) and knn method is the best procedure for the decile function. Finally, we can deduce that a good estimation of the regression and variance functions ensures a good estimation of the conditional τ -quantile.

Conclusion

In the regression setting, we estimated the variance function by the model selection and convex aggregation when the set of initial estimators are constructed by the residual based-method. We called the estimators of the two procedures the MS-estimator and C-estimator, respectively. We established the consistency of our estimators under mild assumptions and provided rate of convergence for these methods in L 2 -norm that are of order O((log(M 1 )/N ) 1/8 ) when Y is satisfied the gaussian model; and O((log(M 1 )/N ) 1/4 ) when Y is bounded. In future work, we are interested in studying on the one hand the optimality of our aggregation methods and on the other hand the performance of aggregation methods on high-dimensional features.

Appendix

This section gathers the proof of our results.

Proof of Theorem 9

Note that the quantity E |σ 2 MS (X)σ 2 (X)| 2 is the excess risk of the estimator σ2 MS and defines as follows: R(σ 2 ŝ,m ).

E |σ 2 MS (X) -σ 2 (X)| 2 := E R(σ 2 MS ) -R(σ 2 ) , where R(σ 2 ) = E |Z -σ 2 (X)| 2 is
(2.9)

We consider the following decomposition

R(σ 2 MS ) -R(σ 2 ) = R(σ 2 MS ) -R(σ 2 MS ) estimation error + R(σ 2 MS ) -R(σ 2 ) approximation error . (2.10)
Each of these errors is obviously positive. The random term R(σ 2 MS ) -R(σ 2 MS ) is called the estimation error (or the variance). It measures how close σ2 MS is to the best possible rule in

[M 2 ] in terms of the risk R. The deterministic term R(σ 2 MS ) -R(σ 2
) is called the approximation error (or the bias). We start with the following lemma

Lemma 1. Let σ2

MS be an aggregate defined in Equation (2.9). Then,

E R(σ 2 MS ) -R(σ 2 ) = E min m∈[M 2 ] E X |σ 2 ŝ,m (X) -σ 2 (X)| 2 .
This result explicitly determines the approximation error.

Proof of Lemma 1. For all m ∈ [M 2 ], the excess risk of σ2 ŝ,m is given as follows

E |Z -σ2 ŝ,m (X)| 2 -E |Z -σ 2 (X)| 2 = E X |σ 2 ŝ,m (X) -σ 2 (X)| 2 .
(2.11)

We apply min in Equation (2.11) and we get

E R(σ 2 MS ) -R(σ 2 ) = E min m∈[M 2 ] E X |σ 2 ŝ,m (X) -σ 2 (X)| 2 .

Appendix

Proof of Theorem 9. We thank the decomposition in Eq. (2.10), we have

E |σ 2 MS (X) -σ 2 (X)| 2 = E R(σ 2 MS ) -R(σ 2 MS ) + E R(σ 2 MS ) -R(σ 2 ) .
(2.12)

Step 1. Study of the term E R(σ 2 MS ) -R(σ 2 ) . We begin with the following Lemma Lemma 2. Let ŝ and s * be two estimators defined in (2.1) and (2.7), respectively. Then, under Assumptions 1, 2, 3 and 4, there exists an absolute constant C such that

P (ŝ = s * ) ≤ C log(M 1 ) N 1/2 .
Proof. Under Assumption 4, we have firstly

δ * (D n ) = min s =s * |R( fs * ) -R( fs )| > δ 0 > 0. Recall that R( fs * ) ≤ R( fŝ ).
On the event {ŝ = s * }, we have two cases

• RN ( fŝ ) < R( fs * ), and then

δ * (D n ) ≤ |R( fŝ ) -R( fs * )| ≤ | RN ( fs * ) -R( fs * )| ≤ max s∈[M 1 ]
| RN ( fs ) -R( fs )|.

• RN ( fŝ ) ≥ R( fs * ), and then

δ * (D n ) ≤ | RN ( fŝ ) -R( fŝ )| + | RN ( fs * ) -R( fs * )| ≤ 2 max s∈[M 1 ]
| RN ( fs ) -R( fs )|.

Therefore,

P (ŝ = s * ) ≤ P max s∈[M 1 ] | RN ( fs ) -R( fs )| ≥ δ 0 /2
We control this term using Bernstein's inequality. We check that the conditions for Bernstein's inequality are satisfied. For all s

∈ [M 1 ], set V i (s) = |Y i -fs (X i )| 2 = |f * (X i ) -fs (X i ) + σ(X i )ξ i | 2
for all i = n + 1, . . . , n + N . First, Assumptions 1 and 3 ensure that there exist a positive constants L 1 and L 2 such that |f * (X) -fs (X)| ≤ L 1 and |σ 2 (X)| ≤ L 2 . Second, note that since the variables V i (s) are i.i.d. and by the elementary inequality (x + y) 4 ≤ 2 3 (x 4 + y 4 ) for all x, y ∈ R, by Lemma 4, and by the elementary inequality x 4 + y 4 ≤ (x + y) 4 for all x, y ≥ 0 we have

n+N i=n+1 E V 2 i (s) ≤ 2 3 n+N i=n+1 E |f * (X) -fs (X)| 4 + σ 4 (X i )ξ 4 i ≤ 2 7 N (L 1 + L 2 ) 4 := v N ,
and for k ≥ 3 we follow the elementary inequality (x + y) 2k ≤ 2 2k-1 (x 2k + y 2k ) for all x, y ∈ R, Lemma 4, and the following elementary inequality x 2k + y 2k ≤ (x + y) 2k for all x, y ≥ 0

n+N i=n+1 E (V k i (s) ∨ 0) = n+N i=n+1 E |f * (X i ) -fs (X i ) + σ(X i )ξ i | 2k ≤ 2 2k-1 n+N i=n+1 E |f * (X i ) -fs (X i )| 2k + |σ 2 (X i )| k |ξ i | 2k ≤ 1 2 2 2k N L 2k 1 + 2 k+1 ( L 2 ) 2k (k)! ≤ 1 2 2 3k+1 N L 1 + L 2 2k k! ≤ 1 2 v N c k-2 k! .
where c

:= 8 L 1 + √ L 2 2 .
Using the Bernstein's inequality (Lemma 7), we get for all s ∈ [M 1 ]

P | RN ( fs ) -R( fs )| ≥ δ 0 2 ≤ 2 exp - N δ 2 0 2 10 (L 1 + √ L 2 ) 4 + 4cδ 0
By union bound on s ∈ [M 1 ], we obtain

P (ŝ = s * ) ≤ 2 exp log(M 1 ) - N δ 2 0 2 10 (L 1 + √ L 2 ) 4 + 4cδ 0 ≤ C log(M 1 ) N 1/2
, where C is a positive constant which depends on L 1 , L 2 and δ 0 .

By Lemmas 1 and 2, and under Assumptions 1 and 3 we get

E R(σ 2 MS ) -R(σ 2 ) = E min m∈[M 2 ] E X |σ 2 ŝ,m (X) -σ 2 (X)| 2 1 {ŝ=s * } + 1 {ŝ =s * } ≤ E min m∈[M 2 ] E X |σ 2 s * ,m (X) -σ 2 (X)| 2 + C log(M 1 ) N 1/2
where C is a constant which depends on K 2 , σ 2 and the constant in Lemma 2.

Step 2. Study of the term E R(σ 2 MS ) -R(σ 2 MS ) . To treat the estimation error, we introduce an aggregate σ2

MS which is based on minimization of the empirical risk of R

σ2 MS := σ2 ŝ, m , where m ∈ argmin m∈[M 2 ] R N (σ 2 ŝ,m ) , with R N (σ 2 ŝ,m ) = 1 N n+N i=n+1 |Z i -σ2 ŝ,m (X i )| 2 . Moreover, we consider the decomposition E R(σ 2 MS ) -R(σ 2 MS ) = E R(σ 2 MS ) -R(σ 2 MS ) + E R(σ 2 MS ) -R(σ 2 MS ) . Step 2.1. Study of the term E R(σ 2 MS ) -R(σ 2 MS ) . We decompose the term E R(σ 2 MS ) -R(σ 2 MS ) into two positive terms E R(σ 2 MS ) -R(σ 2 MS ) = E R(σ 2 MS ) -R N (σ 2 MS ) + E R N (σ 2 MS ) -R(σ 2 MS ) . (2.13)
We use the fact that R N (σ 2 MS ) ≤ R N (σ 2 MS ) in Eq. (2.13), and we get the uniform bound

E R(σ 2 MS ) -R(σ 2 MS ) ≤ 2E max (s,m)∈[M 1 ]×[M 2 ] |R N (σ 2 s,m ) -R(σ 2 s,m )| .
Then using Assumption 2, for some (s, m)

∈ [M 1 ]×[M 2 ], set T i (s, m) = |Z i -σ2 s,m (X i )| 2 = |σ 2 (X i )ξ 2 i - σ2 s,m (X i )| 2
for all i = n + 1, . . . , n + N . First, note that since the variables T i (s, m) are i.i.d. , conditionally on D n we have

|R N (σ 2 s,m ) -R(σ 2 s,m )| = 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)]) ≤ 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |≤L} + 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |>L} for any L > 0. Therefore, conditionally on D n E max (s,m)∈[M 1 ]×[M 2 ] |R N (σ 2 s,m ) -R(σ 2 s,m )| ≤ E max (s,m)∈[M 1 ]×[M 2 ] 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |≤L} + E max (s,m)∈[M 1 ]×[M 2 ] 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |>L} .
(2.14)

Appendix

Step 2.1.1. We control the first term on the r.h.s. of Eq. (2.14). On the event {|ξ| ≤ L} and under Assumptions 1 and 3, we get |T i (s, m)| ≤ c 1 L 4 + 2K 2 2 for all i = n + 1, . . . , n + N for some c 1 > 0 that depends on σ 2 . Conditionally on D n , we apply Hoeffding's inequality, for all (s, m)

∈ [M 1 ] × [M 2 ],
and all t ≥ 0

P 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |≤L} ≥ t ≤ 2 exp - N t 2 2(c 1 L 4 + 2K 2 2 ) 2 ,
Conditionally on D n , by a union bound on (s, m)

∈ [M 1 ] × [M 2 ],
we deduce that for all t ≥ 0

P max (s,m)∈[M 1 ]×[M 2 ] 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |≤L} ≥ t ≤ 2 exp log(M 1 M 2 ) - N t 2 2(c 1 L 4 + 2K 2 2 ) 2 .
We apply Lemma 6. Then, there exists a positive constant c such that

E max (s,m)∈[M 1 ]×[M 2 ] 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |≤L} ≤ c c 2 L 4 + c 3 log(M 1 M 2 ) N 1/2
, where c 2 is a positive constant that depends on c 1 and c 3 depends on K 2 .

Step 2.1.2. We control the second term on the r.h.s. of Eq. (2.14). By union bound on (s, m)

∈ [M 1 ] × [M 2 ]
, by Cauchy-Schwarz inequality, under Assumptions 1, 2 and 3, and Lemma 3 we obtain

E max (s,m)∈[M 1 ]×[M 2 ] 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |>L} ≤ M 1 s=1 M 2 m=1 1 N n+N i=n+1 E[|T i (s, m) -E[T i (s, m)]|1 {ξ i >L} ] ≤ M 1 s=1 M 2 m=1 1 N n+N i=n+1 E[|T i (s, m) -E[T i (s, m)]| 2 ]P(|ξ i | > L)] ≤ cM 1 M 2 P(|ξ 1 | > L) ≤ cM 1 M 2 exp(-L 2 /4) L 1/2 ,
where c is a positive constant which depends on ξ, σ

E max (s,m)∈[M 1 ]×[M 2 ] |R N (σ 2 s,m ) -R(σ 2 s,m )| ≤ c(c 2 L 4 + c 3 ) log(M 1 M 2 ) N 1/2 + cM 1 M 2 exp(-L 2 /4) L 1/2 .
Choosing L = 2 log(N ) and we get

E max (s,m)∈[M 1 ]×[M 2 ] |R N (σ 2 s,m ) -R(σ 2 s,m )| ≤ C log(N ) 4 log(M 1 M 2 ) N 1/2
, where C is a positive constant that depends on c 2 and c, and Step 2.1.2 is finished. We combine the results of the Step 2.1.1 and Step 2.1.2 and we get the following bound

E R(σ 2 MS ) -R(σ 2 MS ) ≤ 2C log(N ) 4 log(M 1 M 2 ) N 1/2 .
Remark 1. It is clear that when Y is bounded, there exists an absolute constant C > 0 such that

E R(σ 2 MS ) -R(σ 2 MS ) ≤ C log(M 1 M 2 ) N 1/2
.

Step 2.2. Study of the term E R(σ 2 MS ) -R(σ 2 MS ) . We start with the following decomposition

E R(σ 2 MS ) -R(σ 2 MS ) = E R(σ 2 MS ) -R N (σ 2 MS ) + E R N (σ 2 MS ) -R N (σ 2 MS ) + E R N (σ 2 MS ) -R(σ 2 MS ) .
(2.15) We use the same arguments in Step 2.1. to control the first term and the last term on the r.h.s. of Eq. (2.15), and we get the following bound

E R(σ 2 MS ) -R N (σ 2 MS ) + E R N (σ 2 MS ) -R(σ 2 MS ) ≤ 2E max (s,m)∈[M 1 ]×[M 2 ] |R N (σ 2 s,m ) -R(σ 2 s,m )| ≤ C log(N ) 4 log(M 1 M 2 ) N 1/2 . Remark 2. If Y is bounded, there exists an absolute constant C > 0 such that E R(σ 2 MS ) -R N (σ 2 MS ) + E R N (σ 2 MS ) -R(σ 2 MS ) ≤ C log(M 1 M 2 ) N 1/2
.

We now study the second term on the r.h.s. of Eq. (2.15). For that, we need the following decomposition

E R N (σ 2 MS ) -R N (σ 2 MS ) = E R N (σ 2 MS ) -RN (σ 2 MS ) + E RN (σ 2 MS ) -R N (σ 2 MS ) . (2.16)
Using RN (σ 2 MS ) ≤ RN (σ 2 MS ) in Eq. (2.16), we obtain the following inequality

E R N (σ 2 MS ) -R N (σ 2 MS ) ≤ 2E max m∈[M 2 ] | RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| .
We control the term E max

| RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| ≤ 1 N n+N i=n+1 | Ẑi -Z i | 2 + 2 N n+N i=n+1 | Ẑi -Z i |(|Z i | + K 2 ) , where K 2 is the bound of σ2 ŝ,m . The upper-bound of | RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| does not depend on m, therefore E max m∈[M 2 ] | RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| ≤ E 1 N n+N i=n+1 | Ẑi -Z i | 2 + 2E 1 N n+N i=n+1 | Ẑi -Z i |(|Z i | + K 2 ) .
(2.17) Note that, by Assumptions 1 and 3 we obtain for all i

= n + 1, . . . , n + N |f * (X i ) -fMS (X i )| ≤ f * ∞ + max s∈[M 1 ] fs ∞ ≤ f * ∞ + K 1 ≤ L 1 < ∞.
Since x 2y 2 = (xy)(x + y), (x + y) 2 ≤ 2(x 2 + y 2 ), we obtain the following inequality for all

i = n + 1, . . . , n + N | Ẑi -Z i | 2 = |(Y i -fMS (X i )) 2 -(Y i -f * (X i )) 2 | 2 = |(f * (X i ) -fMS (X i ))(2(Y i -f * (X i )) + (f * (X i ) -fMS (X i ))| 2 ≤ |f * (X i ) -fMS (X i )| 2 |(8|Y i -f * (X i )| 2 + 2L 2 1 | , (2.18
)

Control of E 1 N n+N i=n+1 | Ẑi -Z i | 2 .
First, since Assumptions 1-2 are satisfied, we have that for all 2.6 Appendix

i = n + 1, . . . , n + N , E |Y i -f * (X i )| 4 ≤ k 1 < ∞.
Second, by inequality (2.18), Cauchy-Schwarz inequality, Jensen's inequality, and under Assumptions 1, and 2, one gets

E 1 N n+N i=n+1 | Ẑi -Z i | 2 ≤ 2L 2 1 E fMS -f * 2 N + 8 N n+N i=n+1 E |Y i -f * (X i )| 2 |f * (X i ) -fMS (X i )| 2 ≤ 2L 2 1 E fMS -f * 2 N + 8 N n+N i=n+1 E [|Y i -f * (X i )| 4 ] E |f * (X i ) -fMS (X i )| 4 ≤ 2L 2 1 E fMS -f * 2 N + 8 √ k 1 L 1 N n+N i=n+1 E |f * (X i ) -fMS (X i )| 2 ≤ 2L 2 1 E fMS -f * 2 N + 8 k 1 L 1 E 1 N n+N i=n+1 |f * (X i ) -fMS (X i )| 2 ≤ C 1 E fMS -f * 2 N ,
where C 1 is a positive constant that depends on k 1 and L 1 .

Control of E 1 N n+N i=n+1 | Ẑi -Z i |(|Z i | + K 2 )
. First, since Assumptions 1-2 are satisfied, we have that for all i = n + 1, . . . , n

+ N , E (|Y i -f * (X i )| 2 + K 2 ) 2 ≤ k 2 < ∞.
Second, by Cauchy-Schwarz inequality and Jensen's inequality, one gets

1 N n+N i=n+1 E | Ẑi -Z i |(|Z i | + K 2 ) ≤ 1 N n+N i=n+1 E | Ẑi -Z i | 2 E [(|Y i -f * (X i )| 2 + K 1 ) 2 ] ≤ √ k 2 N n+N i=n+1 E | Ẑi -Z i | 2 ≤ k 2 E 1 N n+N i=n+1 | Ẑi -Z i | 2 ≤ C 2 E fMS -f * 2 N 1/4
, where C 2 is a positive constant that depends on C 1 and k 2 . Thus, there exists an absolute constant C such that

E max m∈[M 2 ] | RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| ≤ CE fMS -f * 2 N 1/4
.

We need the following proposition:

Proposition 2. Let fMS the aggregate defined in Eq. (2.2). Then, under Assumptions 2 and 3 there exists an absolute constant C such that

E fMS -f * 2 N ≤ min s∈[M 1 ] E fs -f * 2 N + C log(M 1 ) N 1/2 .
This result studies the upper-bound of empirical norm risk of the aggregate fMS and the proof of it exists in [START_REF] Tsybakov | Aggregation and minimax optimality in high-dimensional estimation[END_REF]. Besides, the Proposition 2 and the elementary inequality (x + y) 1/4 ≤ x 1/4 + y 1/4 for all x, y ≥ 0 give us the following inequality

E R(σ 2 MS ) -R(σ 2 MS ) ≤ C ′ min s∈[M 1 ] E fs -f * 2 N 1/4 + C " log(M 1 ) N 1/8
, where C ′ is a constant which depends on C 2 and C " is a constant which depends on C 2 and the constant in Proposition 2.

Merging the results of the Step 1 and Step 2 in Eq. (2.12) and we get the result.

Remark 3. In the case where Y is bounded and from Eq. (2.18), we observe that there exists a constant C 3 such that

| Ẑi -Z i | 2 ≤ C 3 |f * (X i ) -fMS (X i )| 2 . (2.

19)

By Jensen's inequality twice an inequality (2.17) and from Eq.(2.19), one gets there exists an absolute constant C 4 such that

E max m∈[M 2 ] | RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| ≤ C 4 E fMS -f * 2 N 1/2 . (2.20)
Finally, we apply Proposition 2 in Eq.(2.20) to get the result.

Proof of Proposition 2

From the definition of MS-estimator fMS , we get by a simple algebra that, for any s

∈ [M 1 ] fMS -f * 2 N ≤ fs -f * 2 N + 2 < fMS -fs , Y -f * >, where < fMS -fs , Y -f * >:= 1 N n+N i=n+1 ( fMS (X i ) -fs (X i ))(Y i -f * (X i )) . Therefore, one gets for any s ∈ [M 1 ] E fMS -f * 2 N ≤ E fs -f * 2 N + 2E < fMS -fs , Y -f * > . (2.21)
We control the second term in the r.h.s. of Eq (2.21). Firstly, we notice that

E < fMS -fs , Y -f * > ≤ E max 1≤j≤M 1 < fj -fs , Y -f * > .
Secondly, since Y -f * is ρ-subgaussian where ρ is a positive constant that depends on Y -f * , then the variables < fjfs , Y -f * > is ρ-subgaussian where ρ2 =

ρ 2 fj -fs 2 N N
. Moreover, under Assumption 3, it is clear that max 1≤j≤M 1 fjfs 2 N ≤ B where B is a constant which depends on K 1 . Therefore, we use Lemma 5 and we get

E max 1≤j≤M 1 < fj -fs , Y -f * > ≤ ρ √ B 2 log(M 1 ) N .
Thus,

E fMS -f * 2 N ≤ min s∈[M 1 ] E fs -f * 2 N + 2ρ √ B 2 log(M 1 ) N .

Proof of Theorem 10

We introduce the following aggregates Consider the following decomposition

E |σ 2 C (X) -σ 2 (X)| 2 = E R(σ 2 C ) -R(σ 2 C ) + E R(σ 2 C ) -R(σ 2 C ) + E R(σ 2 C ) -R(σ 2 ) . (2.22)

Appendix

Step 1. Study of the term E R(σ 2 C ) -R(σ 2 ) . We use the same proof of Lemma 1, and we get

E R(σ 2 C ) -R(σ 2 ) ≤ E inf β∈Λ M 2 E X |σ 2 λ,β (X) -σ 2 (X) . Step 2. Study of the term E R(σ 2 C ) -R(σ 2 C ) . We use the fact that R N (σ 2 C ) ≤ R N (σ 2 C
) , and we get the uniform bound

E R(σ 2 C ) -R(σ 2 C ) ≤ 2E sup (λ,β)∈Λ M 1 ×Λ M 2 |R N (σ 2 λ,β ) -R(σ 2 λ,β )| .
Since Λ M 2 (resp. Λ M 1 ) is compact, we have Λ M 2 ⊂ B(0, 1) (the closed unit ball) (resp. Λ M 1 ⊂ B(0, 1)), and there exists an

ǫ 2 -net Λ M 2 ǫ 2 of Λ M 2 (resp. an ǫ 1 -net Λ M 1 ǫ 1 of Λ M 1 ) w.r.t. • 1,M 2 (resp. • 1,M 1 ) such that |Λ M 2 ǫ 2 | ≤ (3/ǫ 2 ) M 2 (resp. |Λ M 1 ǫ 1 | ≤ (3/ǫ 1 ) M 1 ). In particular, for all β ∈ Λ M 2 (resp. λ ∈ Λ M 1 ) there exists β ǫ 2 ∈ Λ M 2 ǫ 2 (resp. λ ǫ 1 ∈ Λ M 1 ǫ 1 ) such that β -β ǫ 2 1,M 2 ≤ ǫ 2 (resp. λ -λ ǫ 1 1,M 1 ≤ ǫ 1 )
. From triangle inequality, one gets

|R N (σ 2 λ,β )-R(σ 2 λ,β )| ≤ |R N (σ 2 λ,β )-R N (σ 2 λ,β ǫ 2 )|+|R N (σ 2 λ,β ǫ 2 )-R N (σ 2 λ ǫ 1 ,β ǫ 2 )|+|R N (σ 2 λ ǫ 1 ,β ǫ 2 )-R(σ 2 λ ǫ 1 ,β ǫ 2 )| + |R(σ 2 λ ǫ 1 ,β ǫ 2 ) -R(σ 2 λ,β ǫ 2 )| + |R(σ 2 λ,β ǫ 2 ) -R(σ 2 λ,β )|. 1. Control of |R(σ 2 λ,β ǫ 2 )-R(σ 2 λ,β )|
. By Jensen's inequality, under assumptions 1-2-5 and E[ξ 2 ] = 1 we obtain

|R(σ 2 λ,β ǫ 2 ) -R(σ 2 λ,β )| ≤ E |Z -σ2 λ,β ǫ 2 (X)| 2 -|Z -σ2 λ,β (X)| 2 = E     M 2 j=1 β j -β ǫ 2 j σ2 λ,j (X)   2Z -σ2 λ,β (X) -σ2 λ,β ǫ 2 (X)   ≤ C 1 ǫ 2 ,
where C 1 is a constant that depends on the upper bounds of σ 2 and σ2 λ,j .

Control of |R

N (σ 2 λ,β ) -R N (σ 2 λ,β ǫ 2 )|.
Since Assumptions 1, 2, and 3 are satisfied, we obtain

|R N (σ 2 λ,β ) -R N (σ 2 λ,β ǫ 2 )| ≤ 1 N n+N i=n+1   M 2 j=1 |β j -β ǫ 2 j ||σ 2 λ,j (X i )|   |2σ 2 (X i )ξ 2 i -σ2 λ,β (X i ) -σ2 λ,β ǫ 2 (X i )| ≤ kǫ 2 C 2 N n+N i=n+1 ξ 2 i + C 3 ,
where k is the bound of σ2 λ,j , C 2 is the constant which depends on σ 2 and C 3 is the constant which depends on the upper bounds σ2 λ,β and σ2 λ,

β ǫ 2 . 3. Control of |R(σ 2 λ ǫ 1 ,β ǫ 2 ) -R(σ 2 λ,β ǫ 2 )|.
Under Assumptions 1, 2, 5, and 6, we get

|R(σ 2 λ ǫ 1 ,β ǫ 2 ) -R(σ 2 λ,β ǫ 2 )| ≤ E   M 2 j=1 β ǫ 2 j |σ 2 λ,j (X) -σ2 λ ǫ 1 ,j (X)||2σ 2 (X)ξ 2 -σ2 λ ǫ 1 ,β ǫ 2 (X) -σ2 λ,β ǫ 2 (X)|   ≤ M 2 j=1 β ǫ 2 j 2E E |σ 2 λ,j (X) -σ2 λ ǫ 1 ,j (X)|σ 2 (X)ξ 2 |D n , X + E |σ 2 λ,j (X) -σ2 λ ǫ 1 ,j (X)||σ 2 λ ǫ 1 ,β ǫ 2 (X) -σ2 λ,β ǫ 2 (X)| ≤ M 2 j=1 β ǫ 2 j 2E |σ 2 λ,j (X) -σ2 λ ǫ 1 ,j (X)|σ 2 (X)E ξ 2 + E |σ 2 λ,j (X) -σ2 λ ǫ 1 ,j (X)||σ 2 λ ǫ 1 ,β ǫ 2 (X) -σ2 λ,β ǫ 2 (X)| ≤ C 4 ǫ 1 ,
where C 4 is constant which depends on K and the upper bounds of σ 2 , σ2 λ ǫ 1 ,β ǫ 2 and σ2 λ,β ǫ 2 .

Control of |R

N (σ 2 λ ǫ 1 ,β ǫ 2 ) -R N (σ 2 λ,β ǫ 2 )|.
We use the same way as 3. and we obtain

|R N (σ 2 λ ǫ 1 ,β ǫ 2 ) -R N (σ 2 λ,β ǫ 2 )| ≤ ǫ 1 C 2 N n+N i=n+1 ξ 2 i + C 5 ,
where C 5 is constant which depends on the upper bounds of σ2 λ ǫ 1 ,β ǫ 2 and σ2 λ,β ǫ 2 .

Therefore, we deduce that

E sup (λ,β)∈Λ M 1 ×Λ M 2 |R N (σ 2 λ,β ) -R(σ 2 λ,β )| ≤ C k,C 2 ,C 3 ,C 4 ,C 5 (ǫ 1 +ǫ 2 )+E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 |R N (σ 2 λ,β ) -R(σ 2 λ,β )|   .
For some (λ,

β) ∈ Λ M 1 ǫ 1 × Λ M 2 ǫ 2 , set T i (λ, β) = |Z i -σ2 λ,β (X i )| 2 = |σ 2 (X i )ξ 2 i -σ2 λ,β (X i )| 2 for all i = n + 1, . . . , n + N . Let L > 0. Since the variables T i (λ, β) are i.i.d. , we have E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 |R N (σ 2 λ,β ) -R(σ 2 λ,β )|   ≤ E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 1 N n+N i=n+1 (T i (λ, β) -E[T i (λ, β)])1 {|ξ i |≤L}   + E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 1 N n+N i=n+1 (T i (λ, β) -E[T i (λ, β)])1 {|ξ i |>L}   .
(2.23)

Step 2.1. We control the first term on the r.h.s. of Eq. (2.23). On the event {|ξ| ≤ L} and under assumptions 1, 2 and 5, we get |T i (λ, β)| ≤ c 1 L 4 + c1 for all i = n + 1, . . . , n + N where c 1 is a positive constant which depends on the upper bound of σ 2 and c1 depends on the upper bound of σ2 λ,β . Conditionally on D n , we apply Hoeffding's inequality, for all (λ, β) ∈ Λ M 1 ǫ 1 × Λ M 2 ǫ 2 , and all t ≥ 0

P 1 N n+N i=n+1 (T i (λ, β) -E[T i (λ, β)])1 {|ξ i |≤L} ≥ t ≤ 2 exp - -N t 2 2(c 1 L 4 + c1 ) 2 ,
By a union bound on (λ, β) ∈ Λ M 1 ǫ 1 × Λ M 2 ǫ 2 and choosing ǫ 1 = ǫ 2 = 3 N , we deduce that for all t ≥ 0

P   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 1 N n+N i=n+1 (T i (λ, β) -E[T i (λ, β)])1 {|ξ i |≤L} ≥ t   ≤ 2 exp (M 1 + M 2 ) log(N ) - -N t 2 2(c 1 L 4 + c1 ) 2 .
We apply Lemma 6. Then, there exists a positive constant c such that

E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 1 N n+N i=n+1 (T i (λ, β) -E[T i (λ, β)])1 {|ξ i |≤L}   ≤ c(c 2 L 4 + c2 ) (M 1 + M 2 ) log(N ) N 1/2
, where c 2 is constant which depends on c 1 and c2 on c1 .

Step 2.2. We control the second term on the r.h.s. of Eq. (2.23). Thanks to the boundness of σ 2 and σ2 λ,β and E[ξ 4 ] = 3, we get E[T i (λ, β)] ≤ c 3 and T i (λ, β) ≤ c 4 ξ 4 i + c 5 for all i = n + 1, . . . , n + N where c 4 and c 5 are constants which depend on the upper bounds of σ 2 and σ2 λ,β , respectively. By

Appendix

Cauchy-Schwarz inequality and Lemma 3 we obtain

E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 1 N n+N i=n+1 (T i (s, m) -E[T i (s, m)])1 {|ξ i |>L}   ≤ c 4 N n+N i=n+1 E[ξ 4 i 1 {|ξ i |>L} ] + (c 3 + c 5 )P(|ξ 1 | > L) ≤ c4 P(|ξ 1 | > L) + (c 3 + c 5 )P(|ξ 1 | > L) ≤ c4 exp(-L 2 /4) √ L + (c 3 + c 5 ) exp(-L 2 /2) L ,
where c4 is a positive constant that depends on c 4 and ξ.

Merging the results of the Step 2.1 and Step 2.2 in Eq.(2.23), and we obtain

E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 |R N (σ 2 λ,β ) -R(σ 2 λ,β )|   ≤ c(c 2 L 4 + c2 ) (M 1 + M 2 ) log(N ) N 1/2 + c4 exp(-L 2 /4) √ L + (c 3 + c 5 ) exp(-L 2 /2) L .
Putting L = 2 log(N ), and we get

E   sup (λ,β)∈Λ M 1 ǫ 1 ×Λ M 2 ǫ 2 |R N (σ 2 λ,β ) -R(σ 2 λ,β )|   ≤ c 6 (M 1 + M 2 ) log 5 (N ) N 1/2
, where c 6 is constant which depends on c 2 . Thus,

E R(σ 2 C ) -R(σ 2 C ) ≤ C (M 1 + M 2 ) log 5 (N ) N 1/2
, where C is constant which depends on c 6 and c.

Remark 4. When Y is bounded, it is clear that there exists an absolute constant

C > 0 E R(σ 2 C ) -R(σ 2 C ) ≤ C (M 1 + M 2 ) log(N ) N 1/2 . Step 3. Study of the term E R(σ 2 C ) -R(σ 2 C
) . We use the same arguments of proof of Theorem 9 (Step 2.2), and we get that there exists two positive constants

C 1 and C 2 such that E R(σ 2 C ) -R(σ 2 C ) ≤ C 1 E fC -f * 2 N 1/p + C 2 α N , (2.24)
where p = 2 if Y is bounded, p = 4 otherwise, and

α N =      (M 1 +M 2 ) log(N ) N 1/2 if Y is bounded; (M 1 +M 2 ) log 5 (N ) N 1/2 otherwise.
In the sequel, we give the following proposition Proposition 3. Let fC be the aggregate defined in Eq. (2.4). Then, under Assumptions 2 and 5 there exists an absolute constant C > 0

E fC -f * 2 N ≤ min λ∈Λ M 1 E fλ -f * 2 N + C log(M 1 ) N .

Chapitre 2: Variance function estimation in regression model via aggregation procedures

The proof of this proposition is similar of the proof of Proposition 2. Thus, we apply Proposition 3 in inequality (2.24) and we get

E R(σ 2 C ) -R(σ 2 C ) ≤ C 1 min λ∈Λ M 2 E fλ -f * 2 N 1/p + C1 φ C N (M 1 ) ,
where C1 is a constant that depends on C 1 and the constant in Proposition 3, where p = 2 if Y is bounded, p = 4 otherwise, and

φ C N (M 1 ) =      log(M 1 ) N 1/4 if Y is bounded; log(M 1 ) N 1/8 otherwise.
Combining

Step 1, Step 2 and Step 3 in Eq (2.22) yields the result.

Technical lemmas

In this section, we gather several technical results which are used to derive the proof of results of this chapter.

Lemma 3. Let X be the standard gaussian distribution, then for any x > 0, it holds

P(X > x) ≤ exp(-x 2 /2) √ 2πx , and
P(|X| > x) ≤ 2 π exp(-x 2 /2) x .
Proof. Since X ∼ N (0, 1), one gets

P (X > x) = 1 √ 2π +∞ x exp(-u 2 /2)du ≤ 1 √ 2π +∞ x u x exp(-u 2 /2)du = exp(-x 2 /2) √ 2πx .
The second inequality follows from symmetry and the last one using the union bound

P(|X| > x) ≤ 2P (X > x) .
Lemma 4. Let X ∼ N (0, 1) and k ≥ 1, then

E |X| 2k ≤ 2 k+1 k!.
Proof.

E |X| 2k = +∞ 0 P |X| 2k > t dt = +∞ 0 P |X| > t 1 2k dt ≤ 2 +∞ 0 exp -t 1 k /2 dt u=t 1 k /2 = 2 k+1 k +∞ 0 u k-1 exp(-u)du = 2 k+1 k!.
Lemma 5. Let X 1 , . . . , X M be zero mean ν-subgaussian random variables, i.e., E [exp(rX i )] ≤ exp r 2 ν 2 2 for all r > 0. Then

E max 1≤i≤M X i ≤ ν 2 log(M ).

Appendix

Proof. By Jensen's inequality, for any r > 0

E max 1≤i≤N X i = 1 r E log exp r max 1≤i≤M X i ≤ 1 r log E exp r max 1≤i≤M X i = 1 r log E max 1≤i≤M exp (rX i ) ≤ 1 r log M i=1 E [exp (rX i )] ≤ 1 r log M i=1 E exp r 2 ν 2 2 = log(M ) r + ν 2 r 2 , taking r = 2 log(M ) ν 2
and we get the result.

Lemma 6. Let N ∈ N * , a ≥ 1, b and c be two non negative real numbers. Consider Z a positive random variable such that P (Z ≥ t) ≤ min(1, exp(a -bN t 2 ) .

(2.25)

Then, there exists a constant C > 0 not depending of N such that

E[Z] ≤ C a bN 1/2 .
Proof. By condition (2.25), we have

E[Z] ≤ +∞ 0 min(1, exp(a -bN t 2 )dt ≤ a bN 1/2 + +∞ ( a bN ) 1/2 exp(a -bN t 2 )dt. (2.26)
The following elementary inequality (xy) Lemma 7 (Bernstein's inequality). Let T 1 , . . . , T n be independent real valued random variables. Assume that there exists some positive numbers v and c such that

n i=1 E[T 2 i ] ≤ v ,
and for all integers k ≥ 3

n i=1 E[(T i ∨ 0) k ] ≤ k! 2 vc k-2 . Let S = n i=1 (T i -E[T i ]
), then for every any positive x we have

P (|S| ≥ x) ≤ 2 exp - x 2 2(v + cx)
.

Lemma 8 (Hoeffding's inequality). Let N ∈ N * and a > 0 be a real number. Let X 1 , . . . , X N be independent random variables having values in [-a, a], then for all t > 0

P 1 N N i=1 (X i -E[X i ]) > t ≤ 2 exp - N t 2 2a 2 .
Lemma 9 (Hoeffding's Lemma). Let X ∈ [a, b] be a bounded random variable with E[X] = 0. Then, for all

λ ∈ R E [exp(λX)] ≤ exp λ 2 (b -a) 2 8 .
Chapter 3

Regression with reject option and application to kNN

Abstract: We investigate the problem of regression where one is allowed to abstain from predicting. We refer to this framework as regression with reject option as an extension of classification with reject option. In this context, we focus on the case where the rejection rate is fixed and derive the optimal rule which relies on thresholding the conditional variance function. We provide a semi-supervised estimation procedure of the optimal rule involving two datasets: a first labeled dataset is used to estimate both regression function and conditional variance function while a second unlabeled dataset is exploited to calibrate the desired rejection rate. The resulting predictor with reject option is shown to be almost as good as the optimal predictor with reject option both in terms of risk and rejection rate. We additionally apply our methodology with kNN algorithm and establish rates of convergence for the resulting kNN predictor under mild conditions. Finally, a numerical study is performed to illustrate the benefit of using the proposed procedure.

Introduction

Confident prediction is a fundamental problem in statistical learning for which numerous efficient algorithms have been designed, e.g., neural-networks, kernel methods, or k-Nearest-Neighbors (kNN) to name a few. However, even state-of-art methods may fail in some situations, leading to bad decision-making. Obvious damageable incidences of an erroneous decision may occur in several fields such as medical diagnosis, where a wrong estimation can be fatal. In this work, we provide a novel statistical procedure designed to handle these cases. In the specific context of regression, we build a prediction algorithm that allows to abstain from predicting when the doubt is too important. As a generalization of the classification with reject option setting [START_REF] Chow | An optimum character recognition system using decision functions[END_REF][START_REF] Chow | On optimum error and reject trade-off[END_REF][START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF][START_REF] Herbei | Classification with reject option[END_REF][START_REF] Lei | Classification with confidence[END_REF][START_REF] Naadeem | Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option[END_REF][START_REF] Vovk | Algorithmic learning in a random world[END_REF], this framework is naturally referred to as regression with reject option. In the spirit of [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF], we opt here for a strategy where the predictor can abstain up to a fraction ε ∈ (0, 1) of the data. The merit of this approach is that it allows human action on the proportion of the data where the prediction is too difficult while standard machine learning algorithms can be exploited to perform the predictions on the other fraction of the data. The difficulty to address a prediction is then automatically evaluated by the procedure. From this perspective, this strategy may improve the efficiency of the human intervention.

In this chapter, we investigate the regression problem with reject option when the rejection (or abstention) rate is controlled. Specifically, we provide a statistically principled and computationally efficient algorithm tailored to this problem. We first formally define the regression with reject option framework, and explicitly exhibit the optimal predictor with bounded rejection rate in Section 3.2. This optimal rule relies on a thresholding of the conditional variance function. This result is the bedrock of our work and suggests the use of a plug-in approach. We propose in Section 3.3 a two-step procedure which first estimates both the regression function and the conditional variance function on a first labeled dataset and then calibrates the threshold responsible for abstention using a second unlabeled dataset. Under mild assumptions, we show that our procedure performs as well as the optimal predictor both in terms of risk and rejection rate. We emphasize that our procedure can be exploited with any off-the-shell estimator. As an example we apply in Section 3.4 our methodology with the kNN algorithm for which we derive rates of convergence. Finally, we perform numerical experiments in Section 3.5 which illustrate the benefits of our approach. In particular, it highlights the flexibility of the proposed procedure.

Rejection in regression is extremely rarely considered in the literature, an exception being [100] that views the reject option from a different perspective. There, the authors used the reject option from the side of ε-optimality, and therefore ensures that the prediction is inside a ball with radius ε around the regression function with high probability. Their methodology is intrinsically associated with empirical risk minimization procedures. In contrast, our method is applicable to any estimation procedure. Closer related works to ours appears in classification with reject option literature [START_REF] Bartlett | Classification with a Reject Option using a Hinge Loss[END_REF][START_REF] Chow | An optimum character recognition system using decision functions[END_REF][START_REF] Chow | On optimum error and reject trade-off[END_REF][START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF][START_REF] Herbei | Classification with reject option[END_REF][START_REF] Lei | Classification with confidence[END_REF][START_REF] Naadeem | Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option[END_REF][START_REF] Vovk | Algorithmic learning in a random world[END_REF]. In particular, the present work can be viewed as an extension of the classification with reject option setting. Indeed, from a general perspective, the present contribution brings a deeper understanding of the reject option. Importantly, the conditional variance function appears to capture the main feature behind the abstention decision. In [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF], the authors also provide rates of convergence for plug-in type approaches in the case of bounded rejection rate. However, their rates of convergence holds only under some margin type assumption [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]71] and a smoothness assumption on the considered estimator. On the contrary, we do not require these assumptions to get valid rates of convergence.

Regression with reject option

In this section we introduce the regression with reject option setup and derive a general form of the optimal rule in this context. We additionally highlight the case of fixed rejection rate as our main framework. First of all, before we proceed, let us introduce some preliminary notation. Let (X, Y ) be a random couple taking its values in R d × R: here X denotes a feature vector and Y is the corresponding output. We denote by P the joint distribution of (X, Y ) and by P X the marginal 3.2 Regression with reject option distribution of the feature X. Let x ∈ R d , we introduce the regression function

f * (x) = E [Y |X = x]
as well as the conditional variance function σ 2 (x) = E (Yf * (X)) 2 |X = x . We will give due attention to these two functions in our analysis. In addition, we denote by • the Euclidean on R d . Finally, | • | stands for the cardinality when dealing with a finite set.

Predictor with reject option

Let f be some measurable real-valued function which must be viewed as a prediction function. A predictor with reject option Γ f associated to f is defined as being any function that maps R d onto P (R) such for all x ∈ R d , the output Γ f (x) ∈ {∅, {f (x)}}. We denote by Υ f the set of all predictors with reject option that relies on f . Hence, in this framework, there are only two options for a particular x ∈ R d : whether the predictor with reject option outputs the empty set, meaning that no prediction is produced for x; or the output Γ f (x) is of size 1 and the prediction coincides with the value f (x). The framework of regression with reject option naturally brings into play two important characteristics of a given predictor Γ f . The first one is the rejection rate that we denote by r (Γ f ) = P (|Γ f (X)| = 0) and the second one is the L 2 error when prediction is performed

Err (Γ f ) = E (Y -f (X)) 2 | |Γ f (X)| = 1 .
The ultimate goal in regression with reject option is to build a predictor Γ f with a small rejection rate that achieves a small conditional L 2 error as well. A natural way to make this happen is to embed these quantities into a measure of performance. To this end, let consider the following risk

R λ (Γ f ) = E (Y -f (X)) 2 1 {|Γ f (X)|=1} + λ r (Γ f ) ,
where λ ≥ 0 is a tuning parameter which is responsible for compromising error and rejection rate: larger λ's result in predictors Γ f with smaller rejection rates, but with larger errors. Hence, λ can be interpreted as the price to pay for using the reject option. Note that the above risk R λ has already been considered by [START_REF] Herbei | Classification with reject option[END_REF] in the classification framework.

Minimizing the risk R λ , we derive an explicit expression of an optimal predictor with reject option.

Proposition 4. Let λ ≥ 0, and consider

Γ * λ ∈ arg min R λ (Γ f ) ,
where the minimum is taken over all predictors with rejection option Γ f ∈ Υ f and all measurable functions f . Then we have that 1. The optimal predictor with rejected option Γ * λ can be written as

Γ * λ (X) = {f * (X)} if σ 2 (X) ≤ λ ∅ otherwise . (3.1)
2. For any λ < λ ′ , the following holds

Err (Γ * λ ) ≤ Err (Γ * λ ′ ) and r (Γ * λ ) ≥ r (Γ * λ ′ ) .
Interestingly, this result shows that the oracle predictor relies on thresholding the conditional variance function σ 2 . We believe that this is an important remark that provides an essential characteristic of the reject option in regression but also in classification. Indeed, it has been shown that the optimal classifier with reject option for classification is obtained by thresholding the function f * (see for instance [START_REF] Herbei | Classification with reject option[END_REF]). However, in the binary case where Y ∈ {0, 1}, one has σ 2 (x) = f * (x)(1f * (x)), and then thresholding σ 2 and f * are equivalent. The second point of the proposition shows that the error and the rejection rate of the optimal predictor are working in two opposite directions w.r.t. λ and then a compromise is required. We illustrate this aspect with the airfoil dataset, and the kNN predictor (see Section 3.5) in the contiguous Figure 3.1. The two curves correspond to the evaluation of the error Err( Γλ ) (blue-solid line) and the rejection rate r( Γλ ) (red-dashed line) as a function of λ. In general any choice of the parameter λ is difficult to interpret. Indeed, one of the major drawbacks of this approach is that any fixed λ (or even an "optimal" value of this parameter) does not allow to control neither of the two parts of the risk function. Especially, the rejection rate can be arbitrary large.

For this reason, we investigate in Section 3.2.2 the setting where the rejection rate is fixed. We understand this rejection rate as a budget one has beforehand.

Optimal predictor with fixed rejection rate

In this section, we introduce the framework where the rejection rate is fixed or at least bounded. That is to say, for a given predictor with reject option Γ f and a given rejection rate ε ∈ (0, 1), we ask that Γ f satisfies following constraint r (Γ f ) ≤ ε. This kind of constraint has also been considered by [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF] in the classification setting. Our objective becomes to solve the constrained problem1 :

Γ * ε ∈ arg min{Err (Γ f ) : r (Γ f ) ≤ ε} . (3.2)
In the same vein as Proposition 4, we aim at writing an explicit expression of Γ * ε , referred in what follows to as ε-predictor. However, this expression is not well identified in the general case. Therefore, we make the following mild assumption on the distribution of σ 2 (X), which translates the fact that the function σ 2 is not constant on any set with non-zero measure w.r.t. P X .

Assumption 7. The cumulative distribution function

F σ 2 of σ 2 (X) is continuous.
Let us denote by F -1 σ 2 the generalized inverse of the cumulative distribution F σ 2 defined for all u ∈ (0, 1) as

F -1 σ 2 (u) = inf{t ∈ R : F σ 2 (t) ≥ u}.
Under Assumption 7 and from Proposition 4, we derive an explicit expression of the ε-predictor Γ * ε given by (3.2). Proposition 5. Let ε ∈ (0, 1), and let

λ ε = F -1 σ 2 (1 -ε). Under Assumption 7, we have Γ * ε = Γ * λε .
As an immediate consequence of the above proposition and properties on quantile functions is that r

(Γ * ε ) = P (|Γ * ε (X)| = 0) = P σ 2 (X) ≥ λ ε = P F σ 2 (σ 2 (X)) ≥ 1 -ε = ε
, and then the ε-predictor has rejection rate exactly ε. The continuity Assumption 7 is a sufficient condition to ensure that this property holds true. Besides, from this assumption, the ε-predictor can be expressed as follows

Γ * ε (x) = {f * (x)} if F σ 2 (σ 2 (x)) ≤ 1 -ε ∅ otherwise . (3.3)
Finally, as suggested by Proposition 4 and 5, the performance of a given predictor with reject option Γ f is measured through the risk R λ when λ = λ ε . Then, its excess risk is given by

E λε (Γ f ) = R λε (Γ f ) -R λε (Γ * ε )
, for which the following result provides a closed formula.

3.3 Plug-in ε-predictor with reject option Proposition 6. Let ε ∈ (0, 1). For any predictor Γ f , we have

E λε (Γ f ) = E X (f * (X) -f (X)) 2 1 {|Γf (X)|=1} + E X |σ 2 (X) -λ ε |1 {|Γf (X)| =|Γ * ε (X)|} .
The above excess risk consists of two terms that translates two different aspect of the regression with reject option problem. The first one is related to the L 2 risk of the prediction function f and is rather classical in the regression setting. On contrast, the second is related to the reject option problem. It is dictated by the behavior of the conditional variance σ 2 around the threshold λ ε .

Plug-in ε-predictor with reject option

We devote this section to the study of a data-driven predictor with reject option based on the plug-in principle that mimics the optimal rule derived in Proposition 5.

Estimation strategy

Equation (3.3) indicates that a possible way to estimate Γ * ε relies on the plug-in principle. To be more specific, Eq. ( 3.3) suggests that estimating f * and σ 2 , as well as the cumulative distribution F σ 2 would be enough to get an estimator of Γ * ε . To build such predictor, we first introduce a learning sample D n = {(X i , Y i ), i = 1, . . . , n} which consists of n independent copies of (X, Y ). This dataset helps us to construct estimators f and σ2 of the regression function f * and the conditional variance function σ 2 respectively. In this chapter, we focus on estimator σ2 which relies on the residual-based methods [START_REF] Hall | Variance function estimation in regression: the effect of estimating the mean[END_REF]. Based on D n , the estimator σ2 is obtained by solving the regression problem of the output variable (Y -f (X)) 2 on the input variable X. Estimating the last quantity F σ 2 is rather simple by replacing cumulative distribution function by its empirical version. Since this term only depends on the marginal distribution P X , we estimate it using a second unlabeled dataset D N = {X n+1 , . . . , X n+N } composed of N independent copies of X. This is an important feature of our methodology since unlabeled data are usually easy to get. The dataset D N is assumed to be independent of D n . We set

Fσ 2 (•) = 1 N N i=1 1 {σ 2 (X n+i )≤•} ,
as an estimator for F σ 2 . With this notation, the plug-in ε-predictor is the predictor with reject option defined for each x ∈ R d as

Γε (x) = f (x) if Fσ 2 (σ 2 (x)) ≤ 1 -ε ∅ otherwise . (3.4)
It is worth noting that the proposed methodology is flexible enough to rely upon any off-the-shelf estimators of the regression function f * and the conditional variance function σ 2 .

Consistency of plug-in ε-predictors

In this part, we investigate the statistical properties of the plug-in ε-predictors with reject option. This analysis requires an additional assumption on the following quantity

F σ2 (•) = P X σ2 (X) ≤ •|D n . Assumption 8. The cumulative distribution function F σ2 of σ2 (X) is continuous.
This condition is analogous to Assumption 7 but deals with the estimator σ2 (X) instead of the true conditional variance σ 2 (X). This difference makes Assumption 8 rather weak as the estimator σ2 (X) is chosen by the practitioner. Moreover, we can make any estimator satisfy this condition by providing a smoothed version of it. We illustrate this strategy with kNN algorithm in Section 3.4. Next theorem is the main result of this section, it establishes the consistency of the predictor Γε to the optimal one. Theorem 11. Let ε ∈ (0, 1). Assume that σ 2 is bounded, f is a consistent estimator of f * w.r.t. the L 2 risk, and σ2 is a consistent estimator of σ 2 w.r.t. the L 1 risk. Under Assumptions 7-8, the followings hold

E E λε Γε -→ n,N →+∞ 0, and E |r( Γε ) -ε| ≤ CN -1/2 ,
where C > 0 is an absolute constant.

This theorem establishes the fact that the plug-in ε-predictor behaves asymptotically as well as the optimal ε-predictor both in terms of risk and rejection rate. The convergence of the rejection rate requires only Assumption 8 which is rather weak and can even be removed following the process detailed in Section 3.4.2. In particular, the theorem shows that the rejection rate of the plug-in εpredictor is of level ε up to a term of order O(N -1/2 ). This rate is similar to the one obtained in the classification setting [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF]. It relies on the difference between the cumulative distribution F σ2 and its empirical counterpart Fσ 2 that is controlled using Dvoretzky-Kiefer-Wolfowitz Inequality [START_REF] Massart | The tight constant in the dvoretzky-kiefer-wolfowitz inequality[END_REF]. Interestingly, this result applies to any consistent estimators of f * and σ 2 .

The estimation of regression function f * is widely studied and suitable algorithm such as random forests, kernel procedures, or kNN estimators can be used, see [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Scornet | Consistency of random forests[END_REF][START_REF] Stone | Consistent nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. The estimation of the conditional variance function which relies on the residual-based methods has also been extensively studied based on kernel procedures, see for instance [START_REF] Fan | Efficient estimation of conditional variance functions in stochastic regression[END_REF][START_REF] Hall | Variance function estimation in regression: the effect of estimating the mean[END_REF][START_REF] Härdle | Local polynomial estimators of the volatility function in nonparametric autoregression[END_REF][START_REF] Kulik | Nonparametric conditional variance and error density estimation in regression models with dependent errors and predictors[END_REF][START_REF] Shen | Optimal estimation of variance in nonparametric regression with random design[END_REF]. In the next section, we derive rates of convergence in the case where both estimators f and σ2 rely on the kNN algorithm. In particular, we establish rates of convergence for σ2 in sup norm (see Proposition 9 in the supplementary material).

Application to kNN algorithm: rates of convergence

The plug-in ε-predictor Γε relies on estimators of the regression and the conditional variance functions. In this section, we consider the specific case of kNN based estimations. We refer to the resulting predictor as kNN predictor with reject option. Specifically, we establish rates of convergence for this procedure. In addition, since kNN estimator of σ 2 violates Assumption 8, applying our methodology to kNN has the benefit of illustrating the smoothing technique to make this condition be satisfied.

Assumptions

To study the performance of the kNN predictor with reject option in the finite sample regime, we assume that X belongs to a regular compact set C ⊂ R d , see [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]. Besides, we make the following assumptions.

Assumption 9. The functions f * and σ 2 are Lipschitz.

Assumption 10 (Strong density assumption). We assume that the marginal distribution P X admits a density µ w.r.t to the Lebesgue measure such that for all x ∈ C, we have 0 < µ min ≤ µ(x) ≤ µ max .

These two assumptions are rather classical when we deal with rate of convergence and we refer the reader to the baseline books [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF][START_REF] Tsybakov | Introduction to Nonparametric Estimation[END_REF]. In particular, we point out that the strong density assumption has been introduced in the context of binary classification for instance in [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]. The last assumption that we require highlights the behavior of σ 2 around the threshold λ ε .

Assumption 11 (α-exponent assumption). We say that σ 2 has exponent α ≥ 0 (at level λ ε ) with respect to P X if there exists c * > 0 such that for all t > 0

P X 0 < |σ 2 (X) -λ ε | ≤ t ≤ c * t α .
This assumption has been first introduced in [71] and is also referred as Margin assumption in the binary classification setting, see [START_REF] Mammen | Smooth discrimination analysis[END_REF]. For α > 0, Assumption 11 ensures that the random variable σ 2 (X) can not concentrate too much around the threshold λ ε . It allows to derive faster rates of convergence. Note that, if α = 0 there is no assumption.

kNN predictor with reject option

For any x ∈ R d , we denote by (X (i,n) (x), Y (i,n) (x)), i = 1, . . . n the reordered data according to the ℓ 2 distance in R d , meaning that X (i,n) (x)x < X (j,n) (x)x for all i < j in {1, . . . , n}. Note that Assumption 10 ensures that ties occur with probability 0 (see [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF] for more details). Let k = k n be an integer. The kNN estimator of f * and σ 2 are then defined, for all x, as follows

f (x) = 1 k n kn i=1 Y (i,n) (x) and σ2 (x) = 1 k n kn i=1 Y (i,n) (x) -f (X (i,n) (x)) 2 .
Conditional on D n , the cumulative distribution function F σ2 is not continuous and then Assumption 8 does not hold. To avoid this issue, we introduce a random perturbation ζ distributed according to the Uniform distribution on [0, u] that is independent from every other random variable where u > 0 is a (small) fixed real number that will be specified later. Then, we define the random variable σ2 (X, ζ) := σ2 (X) + ζ. It is not difficult to see that, conditional on D n the cumulative distribution F σ2 of σ2 (X, ζ) is continuous. Furthermore, by the triangle inequality, the consistency of σ2 implies the consistency of σ2 provided that u tends to 0. Therefore, we naturally define the kNN predictor with reject option as follows.

Let (ζ 1 , . . . , ζ N ) be independent copies of ζ and independent of every other random variable. We set

Fσ 2 (.) = 1 N N i=1 1 {σ 2 (X n+i )+ζ i ≤•} ,
and the kNN ε-predictor with reject option is then defined for all x and ζ as

Γε (x, ζ) = f (x) if Fσ 2 (σ 2 (x, ζ)) ≤ 1 -ε ∅ otherwise .

Rates of convergence

In this section, we derive the rates of convergence of the kNN ε-predictor in the following framework. We assume that Y is bounded or that Y satisfies

Y = f * (X) + σ(X)ξ , (3.5)
where ξ is independent of X and distributed according to a standard normal distribution. Note that these assumptions covers a broad class of applications. Under these assumptions, we can state the following result.

Theorem 12. Grant Assumptions 7, 9, 10, and 11. Let ε ∈ (0, 1), if k n ∝ n 2/(d+2) , and u ≤ n -1/(d+2) , then the kNN ε-predictor Γε satisfies

E E λε Γε ≤ C n -2/(d+2) + log(n) (α+1) n -(α+1)/(d+2) + N -1/2 ,
where C > 0 is a constant which depends on f * , σ 2 , c 0 , c * , α, C, and on the dimension d.

Each part of the above rate describes a given feature of the problem. The first one relies on the estimation error of the regression function. The second one, which depends in part on the parameter α from Assumption 11, is due to the estimation error in sup norm of the conditional variance E sup x∈C σ2 (x)σ 2 (x) ≤ C log(n)n -1/(d+2) stated in Proposition 9 in the supplementary material. Notice that for α = 1, the second term is of the same order (up to logarithmic factor) as the term corresponding to the estimation of the regression function. The last term is directly linked to the estimation of the threshold λ ε . Lastly, for α > 1, we observe, provided that the size of the unlabeled sample N is sufficiently large, that this rate is the same as the rate of f in L 2 norm which is then the best situation that we can expect for the rejection setting.

Numerical experiments

In this section, we present numerical experiments to illustrate the performance of the plug-in εpredictor. The construction process of this predictor is described in Section 3.3.1 and relies on estimators of the regression and the conditional variance functions. The code used for the implementation of the plug-in ε-predictor can be found at https://github.com/ZaouiAmed/Neurips2020_

RejectOption. For this experimental study, we consider the same algorithm for both estimation tasks and build three plug-in ε-predictors based respectively on support vector machines (svm), random forests (rf), and kNN (knn) algorithms. Besides, to avoid non continuity issues, we add the random perturbation ζ ∼ U [0, 10 -10 ] to all of the considered methods as described in Section 3.4.2. The performance is evaluated on two benchmark datasets: QSAR aquatic toxicity and Airfoil Self-Noise coming from the UCI database. We refer to these two datasets as aquatic and airfoil respectively. For all datasets, we split the data into three parts (50 % train labeled, 20 % train unlabeled, 30 % test). The first part is used to estimate both regression and variance functions, while the second part is used to compute the empirical cumulative distribution function. Finally, for each ε ∈ {i/10, i = 0, . . . , 9} and each plug-in ε-predictor, we compute the empirical rejection rate r and the empirical error Err on the test set. This procedure is repeated 100 times and we report the average performance on the test set alongside its standard deviation. We employ the 10-fold cross-validation to select the parameter k ∈ {5, 10, 15, 20, 30, 50, 70, 100, 150} of the kNN algorithm. For random forests and svm procedures, we used respectively the R packages randomForest and e1071 with default parameters.

Datasets

The datasets used for the experiments are briefly described below: QSAR aquatic toxicity has been used to develop quantitative regression QSAR models to predict acute aquatic toxicity towards the fish Pimephales promelas. This dataset is composed of n = 546 observations for which 8 numerical features are measured. The output takes its values in [0.12, 10.05]. Airfoil Self-Noise is composed of n = 1503 observations for which 5 features are measured. This dataset is obtained from a series of aerodynamic and acoustic tests. The output is the scaled sound pressure level, in decibels. It takes its values in [START_REF] Yuan | Classification methods with reject option based on convex risk minimization[END_REF]140].

Since the variance function plays a key role in the construction of the plug-in ε-predictor, we display in Figure 3.2 the histogram of an estimate of σ 2 (X) produced by the random forest algorithm. More specifically, for each i = 1, . . . , n, we evaluate σ2 (X i ) by 10-fold cross-validation and build the histogram of (σ 2 (X i )) i=1,...,n thereafter. Left and right panels of Figure 3.2 deal respectively with the aquatic and airfoil datasets and reflect two different situations where the use of reject option is relevant. The estimated variance in the airfoil dataset is typically large (about 40% of the values are larger than 10) and then we may have some doubts in the associated prediction. According to the aquatic dataset, main part of the estimated values σ2 is smaller than 1 and then the use of the reject option may seem less significant. However, in this case, the predictions produced by the plug-in ε-predictors would be very accurate.

Results

We present the obtained results in Figure 3.3 and Table 3.1. We make a focus on the values of ε ∈ {0, 0.2, 0.5, 0.8}. As a general picture, the results are reflecting our theory: the empirical errors of the plug-in ε-predictors are decreasing w.r.t. ε for both datasets and their empirical rejection rates are very close to their expected values. Indeed, Table 3.1 displays how precise is the estimation of the rejection rate whatever the method used. This is in accordance with our theoretical findings. Moreover, the empirical errors of the plug-in ε-predictors based on the random forests and kNN algorithms are decreasing w.r.t. ε for both datasets. As expected, the use of the reject option improves the prediction precision. As an illustration, for airfoil dataset and the predictor based on random forests, the error is divided by 2 if we reject 50% of the data. However, we discover that the decrease for the prediction error is not systematic. In the case of plug-in ε-predictor based on the svm algorithm and with the aquatic dataset, we observe a strange curve for the error rate (see Figure 3.3-left). We conjecture that this phenomenon is due to a poor estimation of the variance. Indeed, in Figure 3.4, we present the performance of some kind of hybrid plug-in ε-predictors: we still use the svm algorithm to estimate the regression function; the variance function estimation is done based on svm (dashed line), random forests (dotted line), and kNN (dash-dotted line). From Figure 3.4, we observe that the empirical error Err is now decreasing w.r.t. ε for the hybrid predictors based on svm and random forests, and that the performance is quite good.

Benefit of aggregation in regression with reject option

This section is a part of the article [START_REF] Zaoui | Variance function estimation in regression model via aggregation procedures[END_REF]. We illustrate our aggregation methods in the regression with reject option with two real datasets: Concrete Compressive Strength and Airfoil Self-Noise (see Section 2.4.3). We may have some doubts in the associated prediction on two real datasets since the estimated variance is large. We evaluate the performance of the procedure on two real datasets considering the same algorithm for both estimation tasks (same approach to estimate the regression and variance functions) and build four plug-in ε-predictors based respectively, on support vector machines (svm), random forests (rf), and Lasso (Lasso) and kNN (knn) algorithms. We take ζ ∼ U([0, 10 -7 ]). In particular, we run 100 times the procedure where we split the data each time in three: D N 1 with N 1 = 780 for Concrete Compressive Strength and N 1 = 1253 for Airfoil Self-Noise, D N with N = 150 and D T with T = 100. The dataset D T is exploited to calculate the empirical rejection rate r and the empirical error Err for each ε ∈ {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. From these estimations, we compute the average and standard deviation (between parentheses) of r and Err. The results are reported in Table 3.2 with ε ∈ {0, 0.2, 0.5, 0.8} and in Figure 3.5. First of all, we recall that for ε = 0, the measure of risk of ε-predictor match with the error of the approach we use to estimate the regression function f * . We remark that the best plug-in ε-predictor with ε = 0 is the rf method for concrete compressive strength and the svm method for airfoil self-noise. Their errors are 31.33 and 10.36 respectively. Notice that all the corresponding errors diminishes with ε and rejection rate is close to ε. We recall that our aim is not to build a regression rule that reduces the error rate. The motivation for introducing the plug-in ε-predictor is only to improve the confidence on prediction. Since the construction of the optimal rule depends on the estimators of f * and σ 2 , poor estimators would lead to bad plug-in ε-predictor. Now, we hope that our aggregation methods improve the accuracy of the procedure.

For a comparative study, we evaluate the performance of the plug-in ε-predictor considering the same algorithm for both estimation tasks of estimating the regression and the variance functions: we build four plug-in ε-predictors based respectively, on support vector machines (svm), random forests (rf), and Lasso (Lasso), kNN (knn) algorithms. We compare these methods to our aggregation procedures C and MS. The dictionaries F, G 1 and G 2 are described in Section 2.4.3. The labeled D M plays the role of D N in chapter 2. For each ε ∈ {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}, and each plug-in ε-predictor, we compute the empirical rejection rate r and the empirical error Err. We take ζ ∼ U([0, 10 -7 ]). So, we repeat independently 100 times the following steps: From these estimations, we compute the average and standard deviation (between brackets) of r and Err. The results are reported in Table 3.3 with ε ∈ {0, 0.2, 0.5, 0.8} and in Figure 3.6. Our main observation is that the C plug-in ε-predictor has the same performance as MS plug-in ε-predictor. According to the rejection rate, we recall that our theory related to this point is distribution free and this is also observed in Table 3.3 since all rejection rate have the approximately prescribed level ε. Importantly, note that both aggregation-based methods require splitting the data (part for estimation and a part for aggregation) while the other plug-in ε-predictors (such as rf) do not. However, our plug-in ε-predictors based on aggregation have a similar performance as the best. This result validates the relevance of our strategy.

Conclusion

We generalized the use of the reject option to the regression setting. We investigated the particular setting where the rejection rate is bounded. In this framework, an optimal rule is derived, it relies on thresholding of the variance function. Based on the plug-in principle, we derived a semi-supervised algorithm that can be applied on top of any off-the-shelf estimators of both regression and variance functions. One of the main features of the proposed procedure is that it precisely controls the probability of rejection. We derived general consistency results on rejection rate and on excess risk. We also established rates of convergence for the predictor with reject option when the regression and the variance functions are estimated by kNN algorithm. In future work, we plan to apply our methodology to the high-dimensional setting, taking advantage of sparsity structure of the data.

Approaches based on reject option may be helpful at least from two perspectives. First, when human action is limited by time or any other constraint, reject option is an efficient tool to prioritize the human action. On the other hand, in a world where automatic decisions need to be balanced and considered with caution, abstaining from prediction is one way to prevent from damageable decision-making. In particular, human is more likely able to detect anomalies such as bias in data.

In a manner of speaking, the use of the reject option compromises between human and machines! Our numerical and theoretical analyses support this idea, in particular because our estimation of the rejection rate is accurate.

While the rejection rate has to be fixed according to the considered problem, it appears that the main drawback of our approach is that border instances may be automatically treated while they would have deserved a human consideration. From a general perspective, this is a weakness of all methods based on reject option. This inconvenience is even stronger when the conditional variance function is poorly estimated.

Supplementary material

This supplementary material is organized as follows. Section 3.7.1 provides all proofs of results related to the optimal predictors (that is, Propositions 4, 5 11). In Sections 3.7.2 and 3.7.3 we prove Theorem 11 that establishes the consistency and Theorem 12 that states the rates of convergence of the plug-in ε-predictor Γε respectively. We further establish several finite sample guarantees on kNN estimator in Section 3.7.4. To help readability of the chapter, we provide in Section 3.7.5 some technical tools that are used for the proofs.

Proofs for optimal predictors

Proof of Proposition 4. By definition of R λ , we have for any predictor with reject option

Γ f R λ (Γ f ) = E (Y -f (X)) 2 1 {|Γ f (X)|=1} + λP(|Γ f (X)| = 0) = E (Y -f * (X) + f * (X) -f (X)) 2 1 {|Γ f (X)|=1} + λP(|Γ f (X)| = 0) = E (Y -f * (X)) 2 1 {|Γ f (X)|=1} + E (f * (X) -f (X)) 2 1 {|Γ f (X)|=1} +2E (Y -f * (X))(f * (X) -f (X))1 {|Γ f (X)|=1} + λP(|Γ f (X)| = 0) . Since E (Y -f * (X))(f * (X) -f (X))1 {|Γf (X)|=1} = 0 , and
E (Y -f * (X)) 2 1 {|Γf (X)|=1} = E σ 2 (X)1 {|Γf (X)|=1} , we deduce, R λ (Γ f ) = E (f * (X) -f (X)) 2 1 {|Γf (X)|=1} + E σ 2 (X)1 {|Γf (X)|=1} + λ(1 -1 {|Γf (X)|=1} ) = E (f * (X) -f (X)) 2 + (σ 2 (X) -λ) 1 {|Γf (X)|=1} + λ . (3.6)
Clearly, on the event {|Γ f (X)| = 1}, the mapping f → (f * (X)f (X)) 2 + σ 2 (X)λ achieves its minimum at f = f * . Then, it remains to consider the minimization of

Γ → E (σ 2 (X) -λ) 1 {|Γ(X)|=1} + λ , on the set Υ f * , which leads to {|Γ(X)| = 1} = {σ 2 (X) ≤ λ}.
Putting all together, we get

{|Γ * λ (X)| = 1} = {σ 2 (X) ≤ λ} and on this event Γ * λ (X) = {f * (X)} ,
and point 1. of Proposition 4 is proven. For the second point, we observe that for 0 < λ < λ ′ ,

{|Γ * λ (X)| = 1} = {σ 2 (X) ≤ λ} ⊂ {σ 2 (X) ≤ λ ′ } = {|Γ * λ ′ (X)| = 1} .
From this inclusion, we deduce r(Γ * λ ′ ) ≤ r(Γ * λ ). Furthermore, using the relation {|Γ * λ (X)| = 1} = {σ 2 (X) ≤ λ} and if we denote by a λ = P (|Γ * λ (X)| = 1) we have

Err (Γ * λ ) -Err (Γ * λ ′ ) = 1 a λ E (Y -f * (X)) 2 1 {σ 2 (X)≤λ} - 1 a λ ′ E (Y -f * (X)) 2 1 {σ 2 (X)≤λ ′ } = 1 a λ - 1 a λ ′ E (Y -f * (X)) 2 1 {σ 2 (X)≤λ} - 1 a λ ′ E (Y -f * (X)) 2 1 {λ<σ 2 (X)≤λ ′ } . (3.7)
By definition of σ 2 (X), we can write

E (Y -f * (X)) 2 1 {σ 2 (X)≤λ} = E 1 {σ 2 (X)≤λ} E (Y -f * (X)) 2 |X = E 1 {σ 2 (X)≤λ} σ 2 (X) ≤ λa λ ,
and then

1 a λ - 1 a λ ′ E (Y -f * (X)) 2 1 {σ 2 (X)≤λ} ≤ λ 1 - a λ a λ ′ .
In the same way, we obtain

1 a λ ′ E (Y -f * (X)) 2 1 {λ≤σ 2 (X)≤λ ′ } ≥ λ a λ ′ (a λ ′ -a λ ) = λ 1 - a λ a λ ′ .
From Equation (3.7), we then get

Err (Γ * λ ) ≤ Err Γ * λ ′ .
Proof of Proposition 5. First of all, observe that for any ε ∈ (0, 1), if we set

λ ε = F -1 σ 2 (1 -ε), then the optimal predictor Γ * λ given by (4.2) with λ = λ ε is such that, r Γ * λε = P σ 2 (X) ≥ λ ε = P F σ 2 (σ 2 (X)) ≥ 1 -ε = ε . We need to prove that any predictor Γ f such that r (Γ f ) = ε ′ with ε ′ ≤ ε, satisfies Err (Γ f ) ≥ Err Γ * λε . To this end, consider Γ * λ ε ′ with λ ε ′ = F -1 σ 2 (1 -ε ′ ).
On one hand, by optimality of Γ * λ ε ′ (cf. point 1. of Proposition 4), we have

Err (Γ f ) -Err Γ * λ ε ′ = 1 1 -ε ′ R λ ε ′ (Γ f ) -R λ ε ′ Γ * λ ε ′ ≥ 0 . On the other hand, since ε ′ ≤ ε implies λ ε ≤ λ ε ′ , point 2. of Proposition 4 reads as Err Γ * λε ≤ Err Γ * λ ε ′ .
Combining these two facts gives the desired result.

Proof of Proposition 11. First, from Equation (3.6), we have the following decomposition

R λε (Γ f ) = E (f * (X) -f (X)) 2 + σ 2 (X) -λ ε 1 {|Γf (X)|=1} + λ ε = E (f * (X) -f (X)) 2 1 {|Γf (X)|=1} + E (σ 2 (X) -λ ε )1 {|Γf (X)|=1} + λ ε .
Therefore, we deduce

E (Γ f ) = E (f * (X) -f (X)) 2 1 {|Γf (X)|=1} + E σ 2 (X) -λ ε 1 {|Γf (X)|=1} -1 {|Γ * ε (X)|=1} ,
and the result follows from the fact that all non zero values of

1 {|Γf (X)|=1} -1 {|Γ * ε (X)|=1} equal the sign of σ 2 (X) -λ ε due to the fact that {|Γ * ε (X)| = 1} = σ 2 (X) -λ ε ≤ 0 .

Proof of the consistency results: Theorem 11

The consistency of Γε consists in the introduction of a pseudo oracle ε-predictor Γε defined for all x ∈ R d by

Γε (x) = f (x) if σ2 (x) ≤ F -1 σ2 (1 -ε) ∅ otherwise . (3.8)
This predictor differs from Γε in that it knows the marginal distribution P X and then it has rejection rate exactly ε. Then, we consider the following decomposition

E E λε Γε = E R λε ( Γε ) -R λε ( Γε ) + E E λε Γε , (3.9)
and show that both terms in the r.h.s. go to zero.

• Step 1. E E λε Γε → 0. We use Proposition 11 and get the following result.

Proposition 7. Let ε ∈ (0, 1). Under Assumptions 7 and 8, the following holds

E E λε Γε ≤ E ( f (X) -f * (X)) 2 + E |σ 2 (X) -σ 2 (X)| + CE [|F σ2 (λ ε ) -F σ 2 (λ ε )|] ,
where C > 0 is constant which depends on the upper bounds of σ 2 and λ ε .

Proof of Proposition 7. Let ε ∈ (0, 1). First, we recall our notation

F σ2 (•) = P X σ2 (X) ≤ •|D n and λ ε = F -1 σ 2 (1 -ε).
We also introduce λε = F -1 σ2 (1ε) for the pseudo-oracle counterpart of λ ε . A direct application of Proposition 11 yields

E Γε ≤ E X f (X) -f * (X) 2 + E X |σ 2 (X) -λ ε |1 {| Γε(X)| =|Γ * ε (X)|} . (3.10)
We first observe that if σ 2 (X) ≤ λ ε and σ2 (X) ≥ λε , we have either of the two cases

• λε ≥ λ ε and then |σ 2 (X) -λ ε | ≤ |σ 2 (X) -σ 2 (X)|; • λε ≤ λ ε and then either |σ 2 (X) -λ ε | ≤ |σ 2 (X) -σ 2 (X)| or σ2 (X) ∈ ( λε , λ ε ).
Similar reasoning holds in the case where σ 2 (X) ≥ λ ε and σ2 (X) ≤ λε . Therefore

E |σ 2 (X) -λ ε |1 {| Γε(X)| =|Γ * ε (X)|} |D n ≤ E |σ 2 (X) -λ ε |1 {|σ 2 (X)-λε|≤|σ 2 (X)-σ 2 (X)|} |D n + 1 {λε≤ λε} E |σ 2 (X) -λ ε |1 {λε≤σ 2 (X)≤ λε} |D n + 1 { λε≤λε} E |σ 2 (X) -λ ε |1 { λε≤σ 2 (X)≤λε} |D n .
From the above inequality, since σ 2 is bounded, there exists a constant C > 0 such that

E |σ 2 (X) -λ ε |1 {| Γε(X)| =|Γ * ε (X)|} ≤ E |σ 2 (X) -σ 2 (X)| + CE |F σ2 ( λε ) -F σ2 (λ ε )| .
Now, from Assumptions 7 and 8, we have that F σ 2 (λ ε ) = 1ε = F σ2 ( λε ). Therefore, we deduce that

E |σ 2 (X) -λ ε |1 {| Γε(X)| =|Γ * ε (X)|} ≤ E |σ 2 (X) -σ 2 (X)| + CE [|F σ 2 (λ ε ) -F σ2 (λ ε )|] .
Putting this into Equation (3.10) gives the result in Proposition 7.

Since f and σ2 are consistent w.r.t. the L 2 and L 1 risks respectively, the first two terms in the bound of Proposition 7 converge to zero. It remains to study the convergence of the last term. To this end, we prove that

E [|F σ 2 (λ ε ) -F σ2 (λ ε )|] = E |1 {σ 2 (X)≤λε} -1 {σ 2 (X)≤λε} | ≤ P |σ 2 (X) -σ2 (X)| ≥ |σ 2 (X) -λ ε | .
Hence, for any β > 0, using Markov's Inequality we have

E [|F σ 2 (λ ε ) -F σ2 (λ ε )|] ≤ P |σ 2 (X) -λ ε | ≤ β + P |σ 2 (X) -σ2 (X)| ≥ β ≤ P |σ 2 (X) -λ ε | ≤ β + E |σ 2 (X) -σ 2 (X)| β .
Combining this last inequality with Proposition 7 and the consistency of f and σ2 w.r.t. the L 2 and L 1 risks respectively implies that for all β > 0 lim sup

n,N →+∞ E E λε Γε ≤ CP |σ 2 (X) -λ ε | ≤ β .
Since the above inequality holds for all β > 0, under Assumption 7, we deduce that

E E λε Γε → 0 ,
and then this step of the proof is complete.

• Step 2. E R λε ( Γε ) -R λε ( Γε ) → 0. Thanks to Equation (3.6), we have that R λε ( Γε ) -R λε ( Γε ) = E X (f * (X) -f (X)) 2 + (σ 2 (X) -λ ε ) 1 {| Γε(X)|=1} -1 {| Γε(X)|=1} .
Therefore, since σ 2 is bounded, there exists a constant C > 0 such that

E |R λε ( Γε ) -R λε ( Γε )| ≤ 2E (f * (X) -f (X)) 2 + CE |1 {| Γε(X)|=1} -1 {| Γε(X)|=1} | ≤ 2E (f * (X) -f (X)) 2 + CA ε , (3.11)
where

A ε = E |1 {| Γε(X)|=1} -1 {| Γε(X)|=1} | = E 1 { Fσ 2 (σ 2 (X))≥1-ε} -1 {F σ2 (σ 2 (X))≥1-ε} . (3.12)
Considering the fact that f is consistent w.r.t. the L 2 risk, it remains to treat the term A ε . We have

A ε ≤ P | Fσ 2 (σ 2 (X)) -F σ2 (σ 2 (X))| ≥ |F σ2 (σ 2 (X)) -(1 -ε)| ,
and then, for all β > 0, the following holds

A ε ≤ P |F σ2 (σ 2 (X)) -(1 -ε)| < β + P | Fσ 2 (σ 2 (X)) -F σ2 (σ 2 (X))| ≥ β . (3.13)
Under Assumption 8, the random variable F σ2 (σ 2 (X)) is uniformly distributed on [0, 1] conditionally on D n . Therefore, we deduce that

P |F σ2 (σ 2 (X)) -(1 -ε)| < β = E P X |F σ2 (σ 2 (X)) -(1 -ε)| < β |D n = E [2β|D n ] = 2β . (3.14)
According to the second term in the r.h.s. of Equation (3.13). we have that

P | Fσ 2 (σ 2 (X)) -F σ2 (σ 2 (X))| ≥ β ≤ P sup x∈R | Fσ 2 (x) -F σ2 (x)| ≥ β = E P D N sup x∈R | Fσ 2 (x) -F σ2 (x)| ≥ β|D n ,
where P D N is the probability measure w.r.t. the dataset D N . Since, conditionally on D n , Fσ 2 is the empirical counterpart of the continuous cumulative distribution function F σ2 , applying the Dvoretzky-Kiefer-Wolfowitz Inequality [START_REF] Massart | The tight constant in the dvoretzky-kiefer-wolfowitz inequality[END_REF], we deduce that

P | Fσ 2 (σ 2 (X)) -F σ2 (σ 2 (X))| ≥ β ≤ 2 exp(-2N β 2 ) . (3.15)
Putting (3.14) and (3.15) into Eq. (3.13), we have that for all β > 0

A ε ≤ 2 β + exp -2N β 2 . ( 3.16)
Since Equation (3.16) holds for all β > 0, we have that A ε → 0 as N, n → +∞. Hence, from the above inequality we get the desired result in Step 2:

E R λε ( Γε ) -R λε ( Γε ) → 0 .

Supplementary material

Combining Step 1 and Step 2 yields the convergence: E E λε Γε → 0 as N, n → +∞.

• Bound on E r( Γε ) . To finish the proof of Theorem 11, it remains to control the rejection rate E r( Γε ) and show that it satisfies E r( Γε )ε ≤ CN -1/2 for some constant C > 0. We observe that

E r( Γε ) -ε = E r( Γε ) -r( Γε ) ≤ A ε ,
where A ε is given by Eq. (3.12). Repeating the same reasoning as in Step 2 above, we bound A ε as in Eq. (3.13), and get from Dvoretsky-Kiefer-Wolfowitz Inequality (see Equation (3.15)), that for all β > 0,

P | Fσ 2 (σ 2 (X)) -F σ2 (σ 2 (X))| ≥ β ≤ 2 exp(-2N β 2 ) ,
and from Equation (3.14),

P |F σ2 (σ 2 (X)) -(1 -ε)| < β = 2β .
These two bounds combined the classical peeling argument of [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF] (see Lemma 12 below) imply the desired result:

A ε ≤ CN -1/2 .
(3.17)

Proof of rates of convergence: Theorem 12

In this section, we follow the same strategy as in Section 3.7.2 but here we care about rates of convergence. Moreover, we have to pay attention to the randomness we introduced in the predictor because of the use of kNN. As in Section 3.7.2, we introduce some pseudo-oracle predictor. However, this one needs to depend on the randomness we introduced in the definition of Γε (x, ζ). Define the pseudo-oracle ε-predictor Γε for all

x ∈ R d and ζ ∈ [0, u] as 2 Γε (x, ζ) = f (x) if σ2 (x, ζ) ≤ F -1 σ2 (1 -ε) ∅ otherwise .
To study the excess risk E E λε Γε of our predictor, we also consider a similar decomposition as in Eq. (3.9) and treat each of the two terms separately.

• Step 1. Study of E E λε Γε . We establish the following result.

Proposition 8. Assume that Assumptions 10 and 11 are fulfilled for some α ≥ 0, then the following inequality holds

E E λε Γε ≤ E f * (X) -f (X) 2 + C E sup x∈C σ2 (x) -σ 2 (x) 1+α + u 1+α ,
where C > 0 depends only on c * and α.

Proof. Let ε ∈ (0, 1). We recall that

λ ε = F -1 σ 2 (1 -ε) and λε = F -1 σ2 (1 -ε). Since ζ is distributed according to a Uniform distribution on [0, u], we observe that σ2 (X, ζ) -σ 2 (X) ≤ sup x∈C σ 2 (x) -σ2 (x) + u := ĥu .
Hence, according to Theorem 2.12 in [START_REF] Bobkov | One-dimensional empirical measures, order statistics and Kantorovich transport distances[END_REF] (recalled in Lemma 13), we have that conditionally on D n λελ ε ≤ ĥu .

Furthermore, since X and ζ are independent, we can use Proposition 11 and get

E E λε Γε ≤ E f (X) -f * (X) 2 + E |σ 2 (X) -λ ε |1 {| Γε(X,ζ)| =|Γ * ε (X)|} . On the event | Γε (X, ζ)| = |Γ * ε (X) , we note that |σ 2 (X) -λ ε | ≤ |σ 2 (X, ζ) -σ 2 (X)| + | λε -λ ε | .
Therefore, conditional on D n , we deduce the following

E (X,ζ) |σ 2 (X) -λ ε |1 {| Γε(X,ζ)| =|Γ * ε (X)|} ≤ E (X,ζ) |σ 2 (X) -λ ε |1 {|σ 2 (X)-λε|≤|σ 2 (X,ζ)-σ 2 (X)|+| λε-λε|} ≤ E X |σ 2 (X) -λ ε |1 {|σ 2 (X)-λε|≤2 ĥu} ≤ 2 ĥu P X |σ 2 (X) -λ ε | ≤ 2 ĥu .
Finally, applying Assumption 11, we deduce that there exists a constant C > 0 such that

E |σ 2 (X) -λ ε |1 {| Γε(X,ζ)| =|Γ * ε (X)|} ≤ C E sup x∈C σ 2 (x) -σ2 (x) 1+α + u 1+α ,
which ends the proof.

Based on Proposition 8, the control of E E λε Γε requires a bound on E f * (X) -f (X) 2 and on E sup x∈C σ 2 (x) -σ2 (x) 1+α . The first of these two terms relies on estimation of the regression function with kNN algorithm and is rather well studied. In particular, thanks to Proposition 14 we have with the choice

k n ∝ n 2/(d+2) E f (X) -f * (X) 2 ≤ Cn -2/(d+2) , (3.18)
where C > 0 is a constant which depends on f * , c 0 , C, and d. Then it remains to bound the second term which is the purpose of Proposition 9 that relies on the rate of convergence of the kNN estimator of the conditional variance σ2 in supremum norm. This result says that under our assumptions and for the choice k n ∝ n 2/(d+2) , we have that

E sup x∈C σ2 (x) -σ 2 (x) 1+α ≤ C log(n) (α+1) n -(α+1)/(d+2) ,
for a constant C > 0 that depends on f * , σ 2 , c 0 , C, and on the dimension d. Putting this last inequality and Eq. (3.18) into the upper bound on the excess risk of Γε from Proposition 8 we show that when we set u = u n ≤ n -1/(d+2) we can write

E E λε Γε ≤ C n -2/(d+2) + log(n) (α+1) n -(α+1)/(d+2) ,
where C > 0 is a constant which depends on f * , σ 2 , c 0 , c * , α, C, and on the dimension d. This ends the first step of the proof.

E R λε ( Γε ) -R λε ( Γε ) ≤ 2E f * (X) -f (X) 2 + CA ε ,
where A ε is defined similarly as in Equation (3.12) with a small modification due to the random perturbation we made on σ2 . Similarly we have

A ε ≤ P | Fσ 2 (σ 2 (X, ζ)) -{F σ2 (σ 2 (X, ζ))| ≥ |F σ2 (σ 2 (X, ζ)) -(1 -ε)| .
Therefore using the same arguments as in Step 2 of the proof of Theorem 11 to get (3.17), it is easy to see that there exists C > 0 such that A ε ≤ CN -1/2 . Then, we deduce

E R λε ( Γε ) -R λε ( Γε ) ≤ 2E f * (X) -f (X) 2 + CN -1/2 .
Finally, an application of Theorem 14 yields

E R λε ( Γε ) -R λε ( Γε ) ≤ C n -2/(d+2) + N -1/2 ,
where C > 0 is a constant which depends on f * , c 0 , C, and d. This ends Step 2 of the proof. Lastly, we combine the results in Step 1 and Step 2, together with the decomposition

E E λε Γε = E R λε ( Γε ) -R λε ( Γε ) + E E λε Γε ,
and get the desired bound on the excess risk.

Rate of convergence for kNN estimator

In this section , we focus on rates of convergence of kNN for the estimation of the regression function f * and the conditional variance function σ 2 . The proofs techniques are largely inspired by those in [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF][START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF], though we provide some additional steps to build for instance finite sample bounds for the sup norm in the problem of conditional variance estimation.

Regression function estimation

We provide the rate of convergence of the kNN estimator of f * in the regression model for which we make the following assumptions. We assume that f * is Lipschitz (Assumption 9) and that Assumption 10 are fulfilled. We recall that from Assumption 10, we have that P X is supported on a compact set C. Furthermore, we also assume that Yf * (X) satisfies a uniform noise condition: there exists c 0 > 0 such that

sup x∈C E [exp(λ (Y -f * (X))) | X = x] ≤ exp(c 2 0 λ 2 ), for |λ| ≤ 1 c 0 . (3.19)
This assumption is rather weak and requires that conditional on X is sub-exponential uniformly over C (see [START_REF] Vershynin | High-Dimensional Probability: An Introduction with Applications in Data Science[END_REF]). Using the same notation as in Section 3.3, we recall that the kNN estimator f of f is defined as follows

f (x) = 1 k n kn i=1 Y (i,n) (x) .
The purpose of the appendix is to provide rates of convergence for the kNN estimator f under the above assumption. To this end we require two auxiliary lemmata, which provide a control respectively with high probability and in expectation on the distance between a feature point and its neighbors uniformly over C.

Lemma 10. Assume Assumptions 9-10 hold. Then there exist C 1 > 0, which depends only on C, µ min , and on d and C 2 > 0, which depends on C and on d, such that for all t ≥ log(n)kn

C 1 n 1/d
, we have

P sup x∈C 1 k n kn i=1 X (i,n) (x) -x ≥ t ≤ C 2 exp log(n) -C 1 t d n/k n .
Proof. For any a ∈ R, let us denote by ⌊a⌋ the largest integer which is smaller or equal to a. Consider some x ∈ C. Following the same arguments as in proof of Theorem 6.2 in [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF], we split the data X 1 , . . . , X n into k n + 1 folds such that the first k n folds have the same size ⌊ n kn ⌋ and the last fold contains the remaining data if there are. We denote Xx j the nearest neighbor of x in the jth fold and then obviously kn i=1

X (i,n) (x) -x ≤ kn j=1 Xx j -x .
Let B2 (a, r) be the closed Euclidean ball in R d centered in a with radius r > 0. Since C is compact, we have C ⊂ B2 (0, R) for some R > 0, and therefore there exists an ε-net C ε of C w.r.t. . such that

|C ε | ≤ 3R ε d .
In particular, for all x ∈ C there exists x ε ∈ C ε such that xx ε ≤ ε. Then, for all x ∈ C and all j ∈ {1, . . . , k n }, there exists x ε ∈ C ε such that

Xx j -x ≤ Xx j -x ε + ε . (3.20)
Besides, we observe that

Xx j -x ε ≤ Xxε j -x ε + 2ε . (3.21) Indeed, if Xxε j -x ε + 2ε < Xx j -x ε we can write Xxε j -x + ε ≤ Xxε j -x ε + 2ε < Xx j -x ε ≤ Xx j -x + ε ,
which contradicts the fact that Xx j is the nearest neighbor of x in the jth fold. Hence, from Equations (3.20) and (3.21), we deduce that

sup x∈C 1 k n kn i=1 X (i,n) (x) -x ≤ 3ε + sup x∈Cε 1 k n kn j=1 Xx j -x .
From the above inequality, we obtain that for t > 6ε,

P sup x∈C 1 k n kn i=1 X (i,n) (x) -x ≥ t ≤ P   sup x∈Cε 1 k n kn j=1 Xx j -x ≥ t/2   . (3.22)
Our goal becomes to bound r.h.s. of the above inequality. Using union bound, we deduce that for all t > 6ε

P   sup x∈Cε 1 k n kn j=1 Xx j -x ≥ t/2   ≤ x∈Cε P   1 k n kn j=1 Xx j -x ≥ t/2   ≤ x∈Cε kn j=1 P Xx j -x ≥ t/2 . (3.23)
For each x ∈ C ε and j ∈ {1, . . . , k n }, by definition of Xx j and since (X i ) i=1,...,n are i.i.d., we have

P Xx j -x ≥ t/2 = P X (1,⌊ n kn ⌋) -x ≥ t/2 = (P ( X 1 -x ≥ t/2)) ⌊n/kn⌋ . (3.24)
On one hand, observe that for t ≥ 4R, (P ( X 1x ≥ t/2) = 0. On the other hand for t ≤ 4R, using the elementary inequality log(1a) ≤ -a for all a ∈ [0, 1), we have that

(P ( X 1 -x ≥ t/2)) ⌊n/kn⌋ ≤ exp - n k n P ( X 1 -x ≤ t/2) ,
which yields, thanks to Assumption 10, there exists C > 0 which depends on µ min and d such that

(P ( X 1 -x ≥ t/2)) ⌊n/kn⌋ ≤ exp -Ct d ⌊n/k n ⌋ .
We finally deduce from Equation (3.22), (3.23), and (3.24), that for all t ≥ 6ε

P sup x∈C 1 k n kn i=1 X (i,n) (x) -x ≥ t ≤ k n |C ε | exp -Ct d ⌊n/k n ⌋ .
Choosing ε = ( kn 6 d Cn )

X (i,n) (x) -x ≥ t ≤ C 2 exp log(n) -C 1 t d n/k n ,
which yields the expected result.

The second lemma establishes a control in expectation of the uniform distance.

Lemma 11. Under Assumption 10, there exist C > 0, which depends only on C, µ min , and on d such that

E sup x∈C 1 k n kn i=1 X (i,n) (x) -x p ≤ C k n log(n) n p/d .
Proof. Since Assumption 10 holds, we can use Lemma 10. Then there exist two non negative constants C 1 and C 2 such that for all t ≥ log(n)kn

C 1 n
1/d , we have

P sup x∈C 1 k n kn i=1 X (i,n) (x) -x ≥ t ≤ C 2 exp log(n) -C 1 t d n/k n .
Therefore an application of Lemma 14 implies directly the result.

Below, we state the main result of this section related to the rate of convergence in sup norm of the kNN estimator of the regression function.

Theorem 13. Assume Assumption 10 is satisfied. Moreover, let p ≥ 1 and k n ∝ n 2/(d+2) . Then

E sup x∈C f (x) -f * (x) p ≤ C log(n) p n -p/(d+2) ,
where C > 0 is a constant which depends on f * , c 0 , C, µ min and d.

Proof. First, we have that

f (x) -f * (x) = 1 k n kn i=1 Y (i,n) (x) -f * (X (i,n) (x)) + 1 k n kn i=1 f * (X (i,n) (x)) -f * (x) .
Therefore, since f * is L-Lipschitz, we then deduce that

sup x∈C f (x) -f * (x) ≤ sup x∈C 1 k n kn i=1 Y (i,n) (x) -f * (X (i,n) (x)) + L sup x∈C 1 k n kn i=1 X (i,n) (x) -x , which implies that E sup x∈C f (x) -f * (x) p ≤ 2 p-1 E sup x∈C 1 k n kn i=1 Y (i,n) (x) -f * (X (i,n) (x)) p + 2 p-1 L p E sup x∈C 1 k n kn i=1 X (i,n) (x) -x p . (3
Y (i,n) (x) -f * (X (i,n) (x)) p ≤ E max j∈{1,...,J} 1 k n kn i=1 Y (i,n) (x j ) -f * (X (i,n) (x j )) p .
Notice that conditional on X 1 , . . . , X n the random variables (Y (i,n) (x j )f * (X (i,n) (x j )) i=1,...,kn are independent with zero mean (see Proposition 8.1 in [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF]). Besides from Equation (3.19) they are uniformly sub-exponential over C, then we deduce from the Bernstein Inequality (see [START_REF] Vershynin | High-Dimensional Probability: An Introduction with Applications in Data Science[END_REF]) that for all t ≥ 0 and j = 1, . . . , J,

P 1 k n kn i=1 Y (i,n) (x j ) -f * (X (i,n) (x j )) ≥ t ≤ exp -ck n min t 2 K 2 , t K ,
where c > 0 is an absolute constant and K > 0 depends on c 0 in Eq. (3.19). Set v n = (d+2) log(n+1) ckn . Our choice of k n ensures that v n ≤ 1, and then we deduce from the union bound that for t ∈ (Kv n , K), P max j∈{1,...,J}

1 k n kn i=1 Y (i,n) (x j ) -f * (X (i,n) (x j )) ≥ t ≤ exp (d + 2) log(n + 1) -ck n t 2 /K 2 ,
and for t > K, P max j∈{1,...,J}

1 k n kn i=1 Y (i,n) (x j ) -f * (X (i,n) (x j )) ≥ t ≤ exp ((d + 2) log(n + 1) -ck n t/K) .
Considering these two cases, we can derive an exponential bound on the term P max j∈{1,...,J}

1 k n kn i=1 Y (i,n) (x j ) -f * (X (i,n) (x j )) ≥ t
for all t ≥ Kv n , therefore we can use similar arguments as in Lemma 14 and conclude that

E sup x∈C 1 k n kn i=1 Y (i,n) (x) -f * (X (i,n) (x)) p ≤ E max j∈{1,...,J} 1 k n kn i=1 Y (i,n) (x j ) -f * (X (i,n) (x j )) p ≤ C log(n) k n p/2 . (3.26)
Combining the above inequality, Equation (3.25), and Lemma 11, gives the desired result.

To conclude this section, we also provide the rate of convergence of the kNN estimator in L 2 -norm Theorem 14. Assume Assumption 10 is satisfied and let k n ∝ n 2/(d+2) , then

E f (X) -f * (X) 2 ≤ Cn -2/(d+2) ,
where C > 0 is a constant which depends on f * , c 0 , C, and d.

The proof of this result is provided in [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF] for d ≥ 3 (see Theorem 6.2). However, a small change implies that the same proof holds for all d under Assumption 10.

Conditional variance function estimation

We provide the rate of convergence of the kNN estimator of σ 2 . This proof is largely inspired by [START_REF] Biau | Lectures on the Nearest Neighbor Method[END_REF], though we are interested here in finite sample bounds. Proposition 9. Grant Assumptions 9 and 10. Let k n ∝ n 2/(d+2) , the following holds

E sup x∈C σ2 (x) -σ 2 (x) 1+α ≤ C log(n) (α+1) n -(α+1)/(d+2) ,
for all α ≥ 0, where C > 0 is a constant which depends on f * , σ 2 , c 0 , C, and on the dimension d.

Proof. First, we define the function σ2 by

σ2 (x) = 1 k n kn i=1 Y (i,n) (x) -f * (X (i,n) (x)) 2 , ∀x ∈ R d .
The function σ2 is the pseudo-estimator of σ 2 that would be used in the case where the function f * is known. By the triangle inequality, we have that for all x ∈ C, σ2 (x)σ 2 (x) ≤ σ2 (x) -σ2 (x) + σ2 (x)σ 2 (x) . Now, we observe that

σ2 (x) -σ2 (x) = 1 k n kn i=1 f * (X (i,n) (x)) -f (X (i,n) (x)) 2(Y (i,n) (x) -f * (X (i,n) (x))) + f * (X (i,n) (x)) -f (X (i,n) (x)) .
Therefore, we deduce

sup x∈C σ2 (x) -σ 2 (x) ≤ sup x∈C σ2 (x) -σ 2 (x) + sup x∈C f (x) -f * (x) 2 + 2 sup x∈C f (x) -f * (x) sup x∈C 1 k n kn i=1 (Y (i,n) (x) -f * (X (i,n) (x)) .
From the above inequality, using the fact that (a + b + c) p ≤ 3 p-1 (a p + b p + c p ) for p ≥ 1, a, b, c ∈ R and applying the Cauchy-Schwartz Inequality, we obtain

E sup x∈C σ2 (x) -σ 2 (x) 1+α ≤ C 1 E sup x∈C σ2 (x) -σ 2 (x) 1+α + C 2 E sup x∈C f (x) -f * (x) 2(1+α) +C 3 E sup x∈C f (x) -f * (x) 2(1+α) 1/2    E   sup x∈C 1 k n kn i=1 (Y (i,n) (x) -f * (X (i,n) (x)) 2(1+α)      1/2
, where C 1 , C 2 and C 3 are non negative reals. We finish the proof of the proposition by bounded the above l.h.s. This relies on controls of estimation error of kNN for the regression function f * and the conditional variance function σ 2 . Observe that when Y is either bounded or satisfies the model conditions in Eq. (3.5), we have that the random variables Y -f * (X) and (Y -f * (X)) 2 -σ 2 (X) satisfy the uniform noise condition (3.19). Indeed, while this fact is clear for Yf * (X), it also holds true for (Yf * (X)) 2σ 2 (X) since, conditionally on X, this random variable is either bounded (since σ 2 is bounded as well) or sub-exponential. Therefore, the result of Theorem 13 applies for the kNN estimators σ2 and f . Furthermore, using the result in Eq. (3.26), we deduce from the above inequality

E sup x∈C σ2 (x) -σ 2 (x) 1+α ≤ C log(n) (α+1) n -(α+1)/(d+2) ,
where C > 0 is a constant which depends on f * , σ 2 , c 0 , C, and the dimension d.

Technical tools

In this section, we state several results that may help for readability of the chapter. The first result is a direct application of the classical peeling argument of [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF].

Lemma 12 (Lemma 1 in [START_REF] Denis | Consistency of plug-in confidence sets for classification in semisupervised learning[END_REF]). Let X be a real random variable, (X n ) n≥1 be a sequence of real random variables and t 0 ∈ R. Assume that there exist C 1 > 0 and γ 0 > 0 such that

P X (|X -t 0 | ≤ δ) ≤ C 1 δ γ 0 , ∀δ > 0 ,
and a sequence of positive numbers a n tends towards infinity, C 2 , C 3 some positive constants such that

P Xn (|X n -X| ≥ δ|X) ≤ C 2 exp -C 3 a n δ 2 , ∀δ > 0, ∀n ∈ N.
Then, there exists C > 0 depending only on C 1 , C 2 and C 3 , such that

|E [1 Xn≥t 0 -1 X≥t 0 ] | ≤ Ca -γ 0 /2 n .
The next result describes the representation of ∞-Wasserstein distance (W ∞ ) on the real line. Let Z ∞ (R) be the collection of all compactly supported probability measures on R.

Lemma 13 (Theorem 2.12 in [START_REF] Bobkov | One-dimensional empirical measures, order statistics and Kantorovich transport distances[END_REF]). Let µ and ν be probability measures in Z ∞ (R) with respective distribution functions F and G. Then, W ∞ (µ, ν)

:= sup 0<t<1 |F -1 (t) -G -1 (t)| is the infimum over all h ≥ 0 such that G(x -h) ≤ F (x) ≤ G(x + h) for all x ∈ R.
The following result provides a bound on moments of a positive random variable provided a tail control.

Lemma 14. Let a ≥ 1, let b, c be two non negative real numbers, and let m ∈ N. Consider Z a positive random variable such that P (Z ≥ t) ≤ c exp (abt m ) , for all t ≥ (a/b) 1/m . Then for all p ≥ 1, there exists a constant C > 0 such that

E [Z p ] ≤ C(a/b) p/m .
Proof. Using the following equality which holds for any positive random variable Z, and any p ≥ 1

E [Z p ] = +∞ 0 P (Z ≥ t) pt p-1 dt , (3.27)
and the condition in Lemma 6, we deduce

E [Z p ] ≤ u 0 pt p-1 dt + c +∞ u exp (a -bt m )pt p-1 dt , (3.28)

Introduction

Prediction is one of the main goals in supervised learning, it consists in building, given historical data, a candidate output for a new observation. One common practice thereafter is to carry out inference on the output and then to ask for confidence in the predicted value, therefore, prediction interval (PI) appears as appropriate tools to handle this problem in the regression setting. A typical application is the prediction in the linear regression case when the data are assumed Gaussian with common variance. In this context, the notion of PI is well studied and well understood both from practice and theory. However, in the general case, inference as a post-processing step may produce irrelevant conclusions due to the stochastic nature of the data-driven prediction procedure (see for instance [START_REF] Berk | Valid post-selection inference[END_REF]). Therefore, in order to guarantee the theoretical validity of the prediction intervals, it is suitable to process at once both aspects of the problem, that is, one might design a data-driven procedure directly devoted to the prediction interval purpose.

In a classical setting of PI, one often asks for a pre-specified level of confidence for the predicted range of values (says 95% or 99% according to the problem). This is for instance the approach that is considered in the conformal prediction literature [START_REF] Vovk | On-line predictive linear regression[END_REF][START_REF] Vovk | Algorithmic learning in a random world[END_REF][START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF][START_REF] Lei | Distribution-free predictive inference for regression[END_REF]. However, this strategy may suffer from interpretability issues for problems where prediction task is difficult or when classical assumptions on the noise are not satisfied. Specifically, for relatively restrictive values of the confidence level, the resulting output might be so large that it becomes useless.

In contrast, our purpose is to produce for future observation a prediction interval with a predetermined expected length. This framework is completely different from the previous one since it does not ensure any coverage guarantee but rather ensures the interpretability of the predicted output. Indeed, since the length of the output interval is controlled, we do not expect for a given input instance x ∈ R d a too large set of candidate values.

Generally speaking, the range of values that we would output with PI has no reason to be an interval. However, in a Gaussian model, this range of values indeed forms an interval (or a union of it). In this chapter, we investigate the problem of PI under expected length constraint in the Gaussian regression setup. We aim at providing a general device that outputs a PI for a new feature. Our procedure relies on the plug-in principle and we propose in the present contribution a statistical analysis of it in this setting.

Main contributions. Denote by Γ : R d → P(R) a given prediction set, where P(R) is the set of subsets of R. One of the main inputs of the present work is the introduction of a novel framework for PI in the regression setting, taking sides of controlling the expected size E [L(Γ(X))] of the output predictor Γ while minimizing its error rate P (Y / ∈ Γ(X)), where L(Γ(X)) = R 1 y∈Γ(X) dy stands for the Lebesgue measure of Γ. We derive the optimal rule for this problem which is defined as

Γ * ℓ ∈ argmin Γ:E[L(Γ(X))]≤ℓ P (Y / ∈ Γ(X)) ,
where ℓ > 0 is a preset length chosen by the practitioner.

In the Gaussian framework, based on the plug-in principle, we then build a general procedure that estimate the optimum and prove that the resulting empirical predictor performs as well as Γ * ℓ both in term of expected length and error rate. Notably, the control on the expected length of the proposed estimator is distribution-free. Furthermore, our algorithm has two appealing properties. It can benefit from a semi-supervised setting and can be applied to any off-the-shelf machine learning algorithm.

On the other hand, we evaluate the performance of our estimator with respect to the symmetric difference distance and a risk measure which properly balances the expected length and the error rate. Specifically, we establish the consistency for our procedure under mild assumptions and provide rates of convergence under suitable assumptions on the distribution of the data.

We additionally conduct a numerical study that confirms our theoretical findings and shows how effective our method is in controlling the length, an important aspect to ensure the interpretability of the output. Finally, we provide a numerical comparison with the strategy which consists in building PI under expected coverage constraint. Our numerical experiment highlights that our proposed approach produced significantly more stable PI. In particular, our algorithm seems to be more adapted when the sample size of the training sample is moderate.

Related works.

A first line of work related to PI is confidence intervals. This is one of the most popular tools in statistical inference and differs from PI by the fact that the purpose there is to output a range of values for a given parameter of the model such that the mean, while our goal is here the prediction. The spectrum of applications of confidence interval is extremely wide and from some perspective PI can be seen as part of the confidence interval literature where we focus on building a confidence interval for the output of a new observation.

Probably the closest direction of works to ours is conformal prediction [START_REF] Vovk | On-line predictive linear regression[END_REF][START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF][START_REF] Lei | Distribution-free predictive inference for regression[END_REF]. The main difference relies on the way the expected length and the error rate of the prediction interval is considered. The goal there is to produce a PI with a pre-specified level of accuracy. The connection of PI with controlled expected size is important to figure out since, at the population level, each PI with controlled accuracy corresponds to a PI with controlled expected size. From practice however, the two approaches start to differ. We defer this discussion to Sections 4.4.2 and 4.5.2 where a complete comparison to PI with controlled accuracy is conducted.

Providing an output with a pre-defined length has rarely been considered. Probably the first reference that deals with such notion is [START_REF] Koopmans | Fixed length confidence intervals for parameters of the normal distribution based on two-stage sampling procedures[END_REF]. There, the authors build confidence intervals for the mean and variance in a Gaussian problem that reach given confidence level while being of size L. In contrast to that work, we deal with prediction intervals, our control on the length is in expectation which offers more flexibility on "hard" points, we do not focus on a pre-specified level of confidence but rather minimize the error under a size constraint, and we derive a statistical and a numerical analysis of our method.

Finally, let us notify that constraining the expected length is not novel. It has already been considered in the multi-class classification setting [START_REF] Denis | Confidence sets with expected sizes for multiclass classification[END_REF][START_REF] Chzhen | Minimax semi-supervised set-valued approach to multiclass classification[END_REF]. There, the control of the length is interpreted as the desired average number of output labels. Similar to the present work, the goal is to focus on a set of values for prediction while maintaining the interpretability of the output. The main difference with earlier work is that we deal here with real valued output which is more tricky. From this perspective the present chapter is a generalization of these previous works to the Gaussian regression setting.

Outline of the chapter.

Section 4.2 provides the main notation and describes the framework of prediction intervals under expected length constraint in the Gaussian regression. In particular, the explicit form of the optimal rule is provided. Section 4.3 introduces our data-driven procedure as well as its statistical analysis. This theoretical analysis is complemented with a numerical study presented in Section 4.5. Additional considerations beyond the Gaussian assumption and other frameworks of prediction intervals are considered in Section 4.4. A conclusion is provided in Section 4.6, while the proofs of our results are postponed to the appendix.

General framework

In the present contribution we focus on the Gaussian model, that is, we assume that (X,

Y ) ∈ R d × R are such that Y = f * (X) + σ(X) ε , (4.1)
where ε ∼ N (0, 1) is independent of X. In this expression, f * : R d → R is the regression function and σ : R d → R * + is the conditional variance function, both of them assumed to be unknown. The main assumptions that we consider throughout the chapter are presented in Section 4.2.1. The characterization of the optimal prediction interval under expected length constraint is provided in Section 4.2.2. Finally, we define the measure of performance dedicated to asses the quality of a prediction interval in Section 4.2.3.

Assumptions

Given an observation X ∈ R d , our goal is to produce the most accurate, in a certain sense to be specified later, a range of predicted values where the corresponding label Y ∈ R lies. Such predictions will be describe a set of P (R) and denoted by Γ(x) for each x ∈ R d . In other words, the predictor Γ is a mapping from R d onto P (R).

Throughout the chapter we denote by p(•|x) the conditional density of Y given x, that is, for all y ∈ R p

(y|x) = 1 √ 2πσ(x) exp - (y -f * (x)) 2 2σ 2 (x) ,
that is, we focus on the heteroscedastic Gaussian regression model. In order to avoid pathological situations, we impose the following mild assumptions on the regression and conditional variance functions.

Assumption 12. There exist 0 < σ

0 < σ 1 < ∞ such that for all x ∈ R d σ 0 ≤ σ(x) ≤ σ 1 .
Assumption 13. There exists

C 1 > 0 such that E [|f * (X)|] ≤ C 1 .
In addition, we consider an assumption which is PI context-specific. It ensures in particular the existence and uniqueness and the optimal PI. Note that similar assumption is considered in the setvalued classification framework [START_REF] Chzhen | Minimax semi-supervised set-valued approach to multiclass classification[END_REF]. Assumption 14 (Continuity). For all y ∈ R, the mapping t → P X (p(y|X) ≥ t) is continuous on R * + . In other word, we assume that p(y|X) is atomless.

Prediction interval with expected length

For a given predictor Γ two features are of interest, its error rate P(Y / ∈ Γ(X)) and its expected Lebesgue measure defined as

L(Γ) := E [L(Γ(X)] = E R 1 {y∈Γ(X)} dy .
Given ℓ > 0, we focus on the following problem

Γ * ℓ ∈ arg min{P (Y / ∈ Γ(X)) : Γ such that L(Γ) ≤ ℓ} . (4.2)
The next proposition provides the characterization of the optimal predictor under Assumption 14.

Proposition 10. Let ℓ > 0, under Assumption 14, the optimal predictor Γ * ℓ can be expressed as

Γ * ℓ (X) = {y ∈ R : p(y|X) ≥ λ * ℓ } ,
where λ * ℓ = G -1 (ℓ) with G(t) := R P(p(y|X) ≥ t)dy for all t > 0 1 . The parameter λ * ℓ , which corresponds to the value of the generalized inverse function G -1 at ℓ, plays a crucial role in our study since it fully determines the optimal predictor Γ * . This being said, let us comment on Proposition 10. First, an important consequence of the above proposition is that the predictor Γ * ℓ is an interval of length ℓ, that is L(Γ * ℓ ) = ℓ and we additionally can express Γ * ℓ as

Γ * ℓ (X) =   f * (X) -2σ 2 (X) log 1 √ 2πλ * ℓ σ(X) , f * (X) + 2σ 2 (X) log 1 √ 2πλ * ℓ σ(X)   .
Second, the function G defined in Proposition 10 is the extension to the regression case of the function G defined in [START_REF] Denis | Confidence sets with expected sizes for multiclass classification[END_REF] in the multi-class setting. Note that the function G is always well-defined and continuous for t > 0, since by Markov Inequality and Fubini Theorem,

G(t) = R P(p(y|X) ≥ t)dy ≤ 1 t R E [p(y|X)] dy ≤ 1 t E R p(y|X)dy ≤ 1 t .
Finally, we highlight that parameter λ * ℓ is simply the Lagrange multiplier of the minimization problem defined by Equation (4.2). Therefore, Γ * ℓ can be expressed as the minimizer of the unconstrained problem

Γ * ℓ ∈ arg min Γ P (Y / ∈ Γ(X)) + λ * ℓ E [L(Γ(X)] . (4.3)

Measures of performance

In this paragraph we introduce two ways to quantify the quality of a given prediction interval Γ . The first one, suggested by Equation (4.3), balances the error rate and the expected length of the predictor

R ℓ (Γ) = P (Y / ∈ Γ(X)) + λ * ℓ E [L(Γ(X)] ,
with λ * ℓ = G -1 (ℓ). This risk is particularly important from our perspective since minimizing it over all predictors lead to the optimal predictor Γ * ℓ , which reaches the requested expected length. A natural "distance" to the optimal predictor is then evaluated through the excess risk

E ℓ (Γ) = R ℓ (Γ) -R ℓ (Γ * ℓ ) .
The following proposition provides a closed formula for this term.

Proposition 11. Let ℓ ≥ 0. For any predictor Γ

E ℓ (Γ) = E Γ(X)△Γ * ℓ (X) |p(y|X) -λ * ℓ | dy .
Interestingly, the above result shows that the performance of a predictor Γ is directly linked to the behavior of the conditional density p(y|x) around the threshold λ * ℓ on the symmetric difference {Γ(X)△Γ * ℓ (X)}. A second measure of performance arises naturally when we deal with predictors that are intervals. It is the expectation of symmetric difference between the considered predictor Γ and optimal predictor Γ * ℓ defined for all predictor Γ as

H (Γ) = E [L (Γ(X)△Γ * ℓ (X))] = E Γ(X)△Γ * ℓ (X)
dy .

In some sense, we note that the measure H provides a stronger guarantee than the excess risk since E ℓ (Γ) ≤ C 2 H(Γ) where C 2 is a positive constant which depends on σ 0 . Besides, H(Γ) = 0 implies that Γ = Γ * ℓ while this property does not necessarily hold for the excess risk.

Data-driven procedure

In this section, we provide a general data-driven procedure to estimate the optimal predictor Γ * ℓ . Two key features are expected from the resulting empirical prediction interval. The expected length should be of order ℓ while keeping its error rate close to one obtained by the oracle predictor. The estimation procedure is presented in the Section 4.3.1, and its main properties are provided in Section 4.3.2. Finally, Section 4.3.3 is dedicated to the study of rates of convergence.

Empirical prediction interval

The result provided in Proposition 10 suggests that an empirical prediction interval can be obtained through the plug-in principle by considering estimators of the conditional density p and the parameter λ * ℓ = G -1 (ℓ). From a theoretical perspective, this learning task requires two independent samples. First, in order to build an estimator of the conditional density p, we estimate the functions f * and σ. Hence, we exploit a labeled sample D n = {(X i , Y i )} n i=1 and build based on it estimators f and σ of these two functions by the means of any machine learning algorithm. However, to establish theoretical guarantees, we require that the estimator σ satisfies similar assumption as Assumption 12.

To this end, we consider a thresholded version of the estimator σ denoted by σ and define for s > 0 as

σ2 (x) = σ2 (x)1 {s -1 ≤σ 2 (x)≤s} + s -1 1 {σ 2 (x)<s -1 } + s1 {σ 2 (x)>s} . A straightforward consequence of the definition of σ is that 1 s ≤ σ2 (x) ≤ s. Furthermore, if s satisfies 1 s ≤ σ 2 0 ≤ σ 2 1 ≤ s, we have for all x σ2 (x) -σ 2 (x) ≤ σ2 (x) -σ 2 (x) ,
Hence consistency of σ2 would imply the consistency of σ2 .

Based on f and σ, an estimator p of the conditional density p naturally derives and can be written for all (x, y) ∈ R d × R as

p(y|x) = 1 √ 2πσ(x) exp   - y -f (x) 2 2σ 2 (x)    .
The second step is devoted to the estimation of the parameter λ * ℓ and requires an unlabeled sample D N = {X n+1 , . . . , X n+N } which consists of i.i.d. observations of X and is independent of D n . Since λ * ℓ depends on the function G, it is suitable to consider the empirical counterpart of the function G, that we build based on p and define for all t ∈ [0, 1] as

G(t) = R 1 N N i=1 1 {p(y|X n+i )>t} dy .
As a result, the empirical prediction interval is defined2 point-wise as

Γ(x) = {y ∈ R : p(y|x) ≥ G-1 (ℓ)} .
The predictor Γ is very natural but has a few limitations: i) because Y is unbounded, the study of the theoretical properties of the estimator Γ might be difficult; ii) in addition, establishing a theoretical analysis on Γ involves similar assumption to Assumption 14 for G. More precisely, it requires that conditional on D n the cumulative distribution of p(y|X) is atomless; iii) furthermore, the above expression of Γ(x) is explicit but relies on computing an integral in order to evaluate the function G. This integral should be approximated. To circumvent all these issues, we consider the following modifications of the initial estimator Γ.

For i) -Thresholding. Let s > 0, we consider a thresholded version of p given by

p(y|x) = 1 √ 2πσ(x) exp   - y -f (x) 2 2σ 2 (x)    1 {|y|≤s} . ( 4
(t) = 2s M N M k=1 N i=1 1 {p(y k |X n+i ,ζ i )>t} .
Finally, the resulting empirical prediction interval writes as

Γ(X, ζ) = {y ∈ R : p(y|X, ζ) ≥ Ĝ-1 (ℓ)} . (4.6)

Theoretical guarantees

In this section, we provide the main properties of the empirical prediction interval Γ. We first illustrate that the prediction interval Γ has an expected length equal to the requested value ℓ. This is one of the main striking feature of our data-driven procedure.

Proposition 12. Assume that M > 4 √ N , then E L( Γ) -ℓ ≤ C s √ N ,
where C > 0 is an absolute constant.

The above result states that our methodology is able to produce a prediction interval with an expected length ℓ, irrespectively of the distribution of the data and of whether or not we have build accurate estimates for f * and σ. Importantly, Proposition 12 holds even if (X, Y ) does not satisfy Equation (4.1). From this perspective the control on the expected length of the produced prediction interval is distribution-free. Notice in particular that the stated bound depends only on the parameter s which should be specified by the practitioner (this choice is discussed later) and on the number N of unlabeled data. In some semi-supervised applications, the amount of these data can be very large so that we can get a good approximation of the marginal distribution P X and then we can expect a good control of the expected length almost for free. Let us also add that Proposition 12 is a fundamental step to show the following bound on the excess risk: Proposition 13. Let Assumption 14 be satisfied. For M > 4

√ N , we have

E E ℓ Γ ≤ C E R |p(y|X) -p(y|X)| dy + su + s √ N ,
where C > 0 is an absolute constant.

The above result shows that the excess-risk of Γ is mainly controlled by the L 1 -risk of the estimator of the conditional density. The residual terms are related to the randomization on the one hand and to the control of the expected length of Γ, given in Proposition 12, on the other hand. Proposition 13 is an intermediate step to establish consistency of the proposed prediction interval as well as to build explicit rates of convergence for the excess-risk of Γ. This is the purpose of the next paragraph and Section 4.3.3 respectively. Consistency result. Proposition 13 shows that the consistency of Γ with respect to the excess-risk relies to the consistency of the estimator p(y|x). In view of Equation (4.4), it is clear that the performance of p is directly linked to the statistical properties of f and σ. More precisely, we obtain the following result. Theorem 15. Let Assumptions 12, 13, and 14. Consider s = log(min(n, N )), M > 4 √ N , and u = u n → 0. Assume that √ sE ( f (X)f * (X)) 2 → 0, and s 5/2 E |σ 2 (X)σ(X)| → 0 , then the following holds

E E ℓ Γ ≤ C 2 E H Γ → 0 .
Let us make several comments on this theorem. First, under suitable assumptions, both excessrisk and expected symmetric difference of Γ converge to 0. Notably, since E E ℓ Γ ≤ C 2 E H Γ , consistency w.r.t. the expected symmetric difference implies consistency w.r.t. the excess-risk. From this perspective, symmetric difference control is a more difficult problem that excess-risk control.

In particular, E H Γ → 0 indicates that Γ = Γ * ℓ asymptotically. Another aspect that need to be discussed is the assumptions that are requested for the proof of Theorem 15. More specifically, consistency of f , and σ2 are naturally required to ensure that p is a consistent estimator of p. In particular, convergence of f and σ can be made possible by several learning algorithms such as kernel methods, local polynomials, regularized least-squares among many others.

Rates of convergence

Theorem 15 establishes the consistency of the prediction interval Γ under mild assumptions. In this section, we focus on rates of convergence. More structural assumptions are then required. We borrow conditions from [START_REF] Denis | Regression with reject option and application to knn[END_REF] introduced in the framework of regression with abstention. We assume that X belongs to a compact C, and we consider the following assumptions.

Assumption 15 (Regularity). The functions f * and σ 2 are Lipschitz.

Assumption 16 (Strong density assumption). The marginal distribution P X satisfies the strong density assumption

• P X is supported on a compact regular set C ⊂ R d ,
• P X admits a density µ w.r.t. to the Lebesgue measure such that 0 < µ min ≤ µ(x) ≤ µ max < ∞, for all

x ∈ C.

Assumption 17 (α-Margin assumption). We say that p(•|X) satisfies Margin assumption with parameter α ≥ 0 at level λ ℓ with respect to P X if there exist constants c 0 > 0 and t 0 > 0 such that for all 0 < t ≤ t 0 ,

R P X (|p(y|X) -λ ℓ | ≤ t) dy ≤ c 0 t α .
The above first two assumptions are rather classical when we deal with rates of convergence in nonparametric statistic. We refer the reader to the book [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF] for a more detail discussion. In addition, Assumption 17, also known as Tsybakov noise condition [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF], has been introduced in the binary classification setting to get fast rates of convergence [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]. In our setting, we notice that the Tsybakov noise condition is required around the threshold λ ℓ . Moreover, since we extend this assumption to the case of regression, we need to integrate it w.r.t. y ∈ R. Based on the above conditions, we can establish the following result. Proposition 14. Let Assumptions 12, 15, 16, and 17 be satisfied. For s = log(min(n, N )), and M > 4

√ N , we have that

E E ℓ Γ ≤ C   E   sup (x,y)∈C×[-s,s] |p(y|x) -p(y|x)| 1+α   + 1 min(n, N ) 1+α + u 1+α + log(N ) √ N   ,
where C > 0 is a constant which depends on f * , σ 2 , c 0 , α, and C.

As compared to the upper-bound that we get in Proposition 13, the bound here is better because of the exponent 1 + α against 1. However, it is obtained under stronger assumptions.

Estimators of regression and variance function.

The framework that we have described so far is quite general and allows to use any off-the-shelf machine learning algorithms to estimate the regression and the variance functions. In what follows, we propose a more concrete illustration of our approach by considering empirical prediction intervals Γ where both regression and variance functions are estimated with the kNN algorithm. Hereafter, we briefly recall the definition of the estimators that are based on the labeled sample D n . For any x ∈ R d , we denote by (X (i,n) (x), Y (i,n) (x)), i = 1, . . . n the reordered data according to the ℓ 2 distance in R d , meaning that

X (i,n) (x) -x < X (j,n) (x) -x ,
for all i < j in {1, . . . , n}. For simplicity, we assume that ties occur with probability 0. Let k = k n be an integer. The kNN estimator of f * and σ 2 are then defined, for all x ∈ R d , as follows

f (x) = 1 k n kn i=1 Y (i,n) (x) and σ2 (x) = 1 k n kn i=1 Y (i,n) (x) -f (X (i,n) (x)) 2 . (4.7)
The properties of these estimator are provided in [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF] for the regression function and in [START_REF] Denis | Regression with reject option and application to knn[END_REF] for the variance function. In particular, the authors in [START_REF] Denis | Regression with reject option and application to knn[END_REF] establish rates of convergence w.r.t. the sup-norm for the estimator σ.

Rates of convergence. The next result, which is an adaption of Proposition 3.1 in [START_REF] Chzhen | Minimax semi-supervised set-valued approach to multiclass classification[END_REF], is useful to derive upper-bound on the measure of risk H of Γ thanks to a control on the excess-risk.

Proposition 15. Let Assumptions 17 be satisfied. There exists an absolute constant

C 3 > 0 such that E H( Γ) ≤ C 3 E E ℓ Γ α/α+1
.

Importantly, this proposition, together with the inequality E ℓ (Γ) ≤ C 2 H(Γ) for all Γ, shows that under appropriate regularity condition consistency of Γ w.r.t. the distance H and the excess-risk are equivalent. The only difference is in the rates of convergence. The above result highlights the link between them under Assumption 17. In particular, we only have to establish rates of convergence w.r.t. E. Let us introduce the following notation. When a ∝ b, it means that the quantities a and b are equal up to a constant. Moreover log(n) says that the inequality holds up to some constants and logarithmic factors. Now, we state the main result of this section. n . The following holds

E E ℓ ( Γ) log(n) n -(1+α)/(d+2) + min(n, N ) -(1+α) + N -1/2 .
Several comments can be made from the above result. The first term is the classical nonparametric fast rate of convergence for the excess-risk under the Margin assumption and the Lipschitzness of the regression function. The last two terms that are related to the problem of PI estimation have different behavior according to the interplay between n and N . In particular, as soon as N ≤ n, More precisely, let β > 0. We consider the following problem

Γ * β = arg min P(Y / ∈Γ(X)≤β E [L(Γ(X)]] .
Under Assumptions 14 and 18 we can derive an expression of Γ * β based on thresholding of the conditional densities:

Γ * β = {y ∈ R, p(y|x) ≥ t β } , with t β defined as solution of E 1 {p(Y |X)≥t β } = R 1 {p(y|x)≥t β } p(y|x) dy = 1 -β .
Therefore, from the above equation, we deduce that

t β = H -1 (1 -β) ,
where H(t) = E 1 {p(Y |X)≥t} . Similarly to the procedure described in Section 4.3.1, we are able to provide a randomized prediction interval Γβ based on the estimator p. We point out that an important difference between the construction of estimators Γβ and Γ is the estimation of the function H. Indeed, this step require a labeled and not an unlabeled dataset, but do not request the discretization step. More formally, considering a labeled dataset D K = {(X i , Y i ), i = 1, . . . , K}, and (ζ 1 , . . . , ζ K ) the vector of perturbation, the estimator Ĥ of the function H is defined for each t > 0, as follows

Ĥ(t) = 1 K K i=1 1 {p(Y i |X i ,ζ i )≥t} .
Although a theoretical comparison with our proposed method is not our purpose, using similar arguments as in [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF], we can establish the consistency of Γβ under same assumptions as in Theorem 15.

E H Γβ → 0 .

In Section 4.5, we focus on a comparison between our method and the expected coverage approach from a numerical perspective.

Numerical experiments

This section is devoted to a numerical study of the performance of our procedure. More precisely, we analyze our approach on synthetic data in Section 4.5.1 and provide a comparison with the expected coverage approach described in Section 4.5.2.

Simulation study

We illustrate the performance of our procedure on the following model

Y = exp(-X 2 ) + dε 2 + 4 X 2 , X ∈ R d , (4.9)
where X = (X 1 , . . . , X d ) is such that for j = 1, . . . , d, the X j are i.i.d. simulated according to a Uniform distribution on [0, 1] and are independent from ε ∼ N (0, 1). Note that the considered model satisfies Equation 4.1, and that Assumptions 12, and 13 are fulfilled.

For our numerical experiments, we choose reasonable dimensions of the features space d ∈ {1, 5}. Before going further in our investigations, we display the boxplots of the output variable Y in Figure 4.1. We see that the range of values of Y is much larger for d = 5 and is included in [-5, 5] for both d = 1, 5. Besides, we choose to focus on ℓ ∈ {0.1, 0.5, 1, 2} which seems to be relevant values according to Figure 4.1 in order to still get interpretation of the output. For ℓ ∈ {0.1, 0.5, 1, 2}, we provide the evaluation of the expected length and the error rate for the oracle prediction set Γ * ℓ . To this end, we repeat 100 times the following scheme. From these repetitions, we compute the mean and standard deviation of the estimates. The obtained results are provided in Table 4.1.

Simulation scheme. To assess the performance of our procedure, we consider the following scheme. For d ∈ {1, 5} and ℓ ∈ {0.1, 0.5, 1, 2}, we repeat 100 the following steps. i) estimate f * and σ 2 from a training test of size n = 500. We consider the residual-based method [START_REF] Hall | Variance function estimation in regression: the effect of estimating the mean[END_REF]. The estimation of f * and σ 2 relies on the random forests algorithm from python library sklearn. We also choose u = 10 -5 for the parameter of the perturbation ζ (see Eq. From these experiments, we compute the empirical means and standard deviations expected length and the error rate. The results are provided in Table 4.2. A visual description of the behavior of our PI is also given in Figure 4.2.

Notice that the value of s that we consider here is different from the one suggested by the theory in 16. This is a minor point. The parameter s in the theory is set such that most of the labels lie in [-s, s] with high probability. This happens when n and M grow since s = log(min(n, N )). Our choice in practice ensures that this property holds regardless the values of n and N .

Expected length

Error rate Results. Two conclusions can be made from this first numerical study. First Tables 4.1 and 4.2 highlight how effective our method is in producing PI with (almost) exactly the right length. This is an important point and suggests that our strategy succeeds to enforce the constraint on the length prescribed by the optimization problem. Second, let us focus on a comparison between Γ * ℓ , the oracle PI, and its empirical counterpart Γ. Table 4.1 and Table 4.2 show how close are the performance of these two PI both in terms of expected length and of error rate. Interestingly, the performance of Γ is obtained with a moderate size N of the unlabeled sample that is used to estimate the threshold. These results also suggest that n = 500 is enough to have good estimations of the regression and variance functions. The closeness between Γ * ℓ and Γ is also illustrated in Figure 4.2.

ℓ d = 1 d = 5 d = 1 d = 5 0.1 0.

Numerical comparison with expected coverage approach

In this section, we numerically compare our procedure to the approach that constraint the expected coverage described in Section 4.4.2. We consider the model defined in Equation 4.9 with d = 5 and focus on the estimation of Γ * ℓ for ℓ = 2. With this expected length, the oracle predictor Γ * ℓ reaches an error rate of β = 0.17. Therefore, for this learning task, we are able to provide empirical PI for both approaches. That is to say, we compute Γ with ℓ = 2 as expected length and Γβ with β = 0.17 as expected error. In order to get a fair comparison of the methods, we repeat 20 times the following steps. For both approaches, we use a training set of size n = 500 to estimate the density p and we estimate the threshold of the considered procedure with a dataset of size N ∈ {10, 30, 50, 70, 100, 150, 200, 500, 1000}. Finally, we compute the expected length and error rate of both empirical PI over a test set of size T = 1000. From these repetitions, we compute empirical means and standard deviations. The results are displayed in Figure 4.3. As expected, in average, both methods behaves similarly. However there are important differences in favor of our approach. First, the convergence of our method is much faster to the mean value both for the expected length and the error rate. We notice that N = 10 is already enough for our method while more than 500 samples are needed for the method that focus on the coverage as constraint. Second, it seems that our construction is much more stable, in particular for length calibration. It illustrates the efficiency of our procedure to build prediction interval with the right The true value of the parameter is given by the dotted line.

expected length.

The two approaches are definitively not comparable in terms of objectives. Indeed, if we are really focused on constraining the error rate, then the length constraint appears (at first sight) sub-optimal and vice versa if we ask for interpretable outputs. However, our numerical analysis clearly suggests that our methodology is more stable: it induces a procedure with a lower variance.

Conclusion

In this chapter, we provide a general methodology to build prediction intervals with controlled expected length in the Gaussian regression. Our proposed algorithm is very effective in controlling the expected length of the output and then ensure the interpretability of the outcome. The theoretical analysis indicates that our method mimics the optimal rule w.r.t. the expected length and, under appropriate properties on the base estimators of the regression function, it is also efficient w.r.t. the symmetric difference distance and the excess-risk. Furthermore, a numerical study supports our theoretical results. Notably, it highlights good stability properties as compared to prediction intervals that focus on expected coverage constraints.

Our numerical comparison to PI under expected coverage constraint additionally opens a very significant door to the use of our method. Because of the stability of our method, one may think to the following two-stage procedure to produce a PI with error rate β.

• Step 1. Build the PI with error rate β and evaluate its length l;

• Step 2. Build our PI with average length l.

While we do not expect a significant improvement in average, the resulting prediction interval might be more stable. This will be the purpose of future investigation.

On the other hand, inference in the high-dimensional setting is a crucial challenge with modern data. Several successful studies consider the Gaussian homoscedastic linear regression [START_REF] Lu | Confidence intervals and regions for the lasso by using stochastic variational inequality techniques in optimization[END_REF][START_REF] Van De Geer | On asymptotically optimal confidence regions and tests for high-dimensional models[END_REF][START_REF] Minnier | A perturbation method for inference on regularized regression estimates[END_REF][START_REF] Belloni | Inference on treatment effects after selection among high-dimensional controls[END_REF]. An important direction for future research is to carry out PI i) for non Gaussian models; ii) and that can handle heteroscedastic model. Both of these questions have their applications in the high dimensional setting.

Appendix

This appendix is devoted to the proof of our main results. The proofs related to Section 4.2 are provided in Section 4.7.2, while Section 4.7.3 is devoted to the proofs of Section 4.3. Finally, Section 4.7.1 gathers useful results. In particular, we give rates of convergence for KNN estimates for both regression and variance function. Notice that in the whole appendix, C is a positive constant that may change from one line to another.

Technical results

In this section, we provide some useful properties that are used for the proof of our main results

Technical lemmas

The first tool we introduce is a generalization of the classical inverse transform theorem [START_REF] Van Der | Asymptotic statistics, volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics[END_REF]Lemma 21.1] to the continuous case. Let a > 0. We consider a random process (Z y ) y∈[-a,a] such that the function H defined by Proof. For every t ≥ 0, we have P(H(Z T ) ≤ t) = P(Z T ≥ H -1 (t)). Denote by dP T the marginal distribution of T . Since the variable T is independent of (Z y ) y∈[-a,a] and H is continuous, one gets

H(t) = 1 2a

Rates of convergence for K-NN estimators

In this section, we gather the results we use for K-NN estimators of both regression and variance function. The proof of this result is provided in [START_REF] Denis | Regression with reject option and application to knn[END_REF].

Proof of Section 4.2

In this section, we provide proofs related to the optimal confidence and to the excess-risk formula where (•) + stands for the positive part. First order optimality conditions for convex non-smooth minimization problems implies 0 ∈ ∂H(Γ * (λ * ℓ , X), λ * ℓ ) where ∂H is the sub-differential of H. Therefore, using the Fundamental Theorem of Calculus, we get E X R 1 {y∈Γ * (λ * ℓ ,X)} dy = ℓ. But, using the above definition of Γ * we can write by Fubini's theorem the left hand side term as E X R 1 {y∈Γ * (λ * ℓ ,X)} dy = R P ((p(y|X) ≥ λ * ℓ ) dy = G(λ * ℓ ). We then conclude that λ * ℓ = G -1 (ℓ). Notice that for this value, we have

L(Γ * ) = E X [L(Γ * (λ * ℓ , X)] = E X 1 {y∈Γ * (λ * ℓ ,X)} dy = G(λ * ℓ ) = ℓ .

Proof of Section 4.3

We now consider the theoretical properties of the prediction interval Γ. We first consider its expected length and then derive a finite sample bound on its excess-risk.

Length control

Proof of Proposition 12. To show this result, we need to introduce some pseudo-oracle predictor that has expected length ℓ. Since for y ∈ [-s, s], the events (A j (y)) j≥0 are mutually exclusive, we deduce

Excess-risk control

Proof of Proposition 13. Throughout the proof, we denote λℓ := Ḡ-1 (ℓ), where Ḡ is defined in Equation (4.12). We start with the following decomposition.

E ℓ Γ = E Γ + R ℓ ( Γ) -R ℓ ( Γ) . ( 4

Consistency Result

This section is devoted to the proof of Theorem 15. We first provide a result on the L 1 -integrated estimation error of p. Proof. To build this proof, we use the triangle inequality to split the term |p(y|X)p(y|X)| into 3. We then have to consider each of these terms consecutively. The first of these terms can be bounded as follows:

1 2πσ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) - 1 2πσ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) ≤ 1 2πσ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) 1 - σ(X) σ(X) = 1 2πσ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) σ(X) -σ(X) σ(X) . (4.22)
This upper-bound consists of two parts. One part which is the density of a Gaussian random variable (whose integral w.r.t. y is 1) and a second term which is independent of y. Observe that this second term |σ(X)σ(X)| is of the same order as |σ 2 (X) -σ2 (X)|. Indeed, notice that when σ(X) > σ(X) σ 2 (X) -σ2 (X) = (σ(X)σ(X))(σ(X) + σ(X)) ≥ σ 0 + 1 √ s (σ(X)σ(X)) ,

where in the last inequality, we use Assumption 12 and the fact that σ(X) ≥ 1/ √ s. Written differently, this means that |σ(X)σ(X)| ≤ √ s

1 + σ 0 √ s |σ 2 (X) -σ2 (X)| ≤ σ 0 √ s 1 + σ 0 √ s |σ 2 (X) -σ2 (X)| σ 0 ≤ C|σ 2 (X) -σ2 (X)| ,
(y -f (X)) 2 2σ 2 (X) - 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) = 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) - (y -f * (X)) 2 2σ 2 (X) -1 ≤ 1 2πσ 2 (X) × 2σ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) (y -f (X)) 2 -(y -f * (X)) 2 ≤ C 2πσ 2 (X)σ(X) exp - (y -f * (X)) 2 2σ 2 (X) (y -f (X)) 2 -(y -f * (X)) 2 .
Using the following decomposition

(y -f (X)) 2 -(y -f * (X)) 2 = ( f (X) -f * (X)) 2 + 2(y -f * (X)(f * (X) -f (X)) ,
we deduce

1 2πσ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) - 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) ≤ C 2πσ 2 (X)σ(X) exp - (y -f * (X)) 2 2σ 2 (X) f (X) -f * (X) 2 + |y -f * (X)|| f (X) -f * (X)| . (4.24)
In the case where (y -f (X)) 2 ≤ (yf * (X)) 2 , we obtain similar bound as in the above equation by

(y -f (X)) 2 2σ 2 (X) - 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) dy ≤ C σ(X) f (X) -f * (X) 2 + σ(X)| f (X) -f * (X)| .
Finally, using that σ(X) ≥ 1/ √ s, we deduce

R 1 2πσ 2 (X) exp - (y -f (X)) 2 2σ 2 (X) - 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) dy ≤ C √ s f (X) -f * (X) 2 + | f (X) -f * (X)| . (4.25)
The remaining term in the decomposition of |p(y|X)p(y|X)| is From Markov inequality and Assumption 13, the second term of the r.h.s. in the above inequality is bounded by .

1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) - 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) = 1 2πσ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) exp - (y -f * (X)) 2 2σ 2 (X) - (y -f * (X)) 2 2σ 2 (X) -1 .
E 1 {|f * (X)
Hence combining the above inequalities, we get for s ≥ 1

Rates of convergence

We start this section with a result on the estimation error of p w.r.t. the sup-norm. where C > 0 is a constant which depends on f * , σ 2 , and on the set C.

Appendix

Proof. We consider the same decomposition into 3 that we used in the proof of Proposition 16. Using the fact that σ(x) ≥ 1 √ s and Assumption 12, we get for all x ∈ C, and y ∈ [-s, s] (c.f., Eq. (4.22)), the first term is controlled as follows: (4.34) According to the second term, from Assumptions 15 and 16, and since f * is a Lipschitz function on the compact C, we have that |f * (x)| ≤ s for n, N large enough. Therefore, using the fact that x → exp(-x) is 1-Lipschitz on R + and that 1 s ≤ σ2 (x), we get (c.f., Eq. (4.24)) 1 2πσ where we recall that λℓ := Ḡ-1 (ℓ), with Ḡ defined in Eq. (4.12). Using similar arguments as those used in the proof of Theorem 4.4 in [START_REF] Denis | Regression with reject option and application to knn[END_REF] that is inspired by Theorem 2.12 in [START_REF] Bobkov | One-dimensional empirical measures, order statistics and Kantorovich transport distances[END_REF], it is not difficult to see that conditional on D n , λℓλ * ℓ ≤ sup Proof of Theorem 16. The proof is a straightforward application of Proposition 14, 17, and Theorem 17, where we also use the fact that for n, N large enough σ2 (x)σ 2 (x) ≤ σ2 (x)σ 2 (x) .

Sommaire 1 . 1 5 1. 2

 1152 Modèle de régression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Estimations de la fonction de régression . . . . . . . . . . . . . . . . . . . . . . . . .

1. 4 . 2 21 1. 5 22 1. 5 . 2 24 1. 6 25 1. 6 . 2 29 1. 6 . 3 31 1. 7

 42215225224625622963317 Design fixe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apprentissage sous contrainte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.5.1 Option rejet en apprentissage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intervalles de prédiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.6.1 Estimation de la fonction de variance . . . . . . . . . . . . . . . . . . . . . . . . Régression avec option rejet et application au K-PPV . . . . . . . . . . . . . . . Intervalle de prédiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion et perspectives de recherche . . . . . . . . . . . . . . . . . . . . . . . . . . 33

  (i) Pour toute fonction borélienne f : R d → R telle que E [|f (X)|] < ∞, il existe une constante C > 0 telle que E n i=1 |W i (X)||f (X i )| ≤ CE [|f (X)|] pour tout n ≥ 1. (ii) Il existe une constante D ≥ 1 telle que P ( n i=1 |W i (x)| ≤ D) = 1.

2 FIGURE 1 . 1 - 4 FIGURE 1 . 2 -

 211412 FIGURE 1.1 -Graphe de l'estimateur fknn (blue) et de la fonction de régression (rouge). Les points noirs sont les données aléatoires de l'échantillon de taille 1000 montrant l'hétéroscédasticité.

  (i) Nous simulons trois ensembles de données D n , D N et D T avec n = N = 500, et T = 1000. (ii) À partir de D n , nous construisons les estimateurs constituant F, puis à partir de D N , nous calculons fMS et fC . (iii) Pour comparaison, à partir de D n ∪ D N , nous calculons les estimateurs constituant F.

FIGURE 1 . 3 -FIGURE 1 . 4 -

 1314 FIGURE 1.3 -Boxplot de l'erreur empirique L 2 des estimateurs : fC , fMS , frf , fsvm , fknn , fNW et ftree dans le modèle 1.

1. 4 FIGURE 1 . 5 -FIGURE 1 . 6 -

 41516 FIGURE 1.5 -Boxplot de l'erreur empirique L 2 des estimateurs : fC , fMS , frf , fsvm , fknn et ftree dans le modèle 3.

FIGURE 1 . 7 -

 17 FIGURE 1.7 -Graphes des estimateurs fC , fMS , et de la fonction de régression dans le modèle 1. Les points noirs sont les données aléatoires de l'échantillon de taille 100 montrant l'hétéroscédasticité.

FIGURE 1 . 8 -

 18 FIGURE 1.8 -Graphes des estimateurs : fC , fMS , et de la fonction de régression dans le modèle 2. Les points noirs sont les données aléatoires de l'échantillon de taille 100 montrant l'hétéroscédasticité.

Algorithme 2 C

 2 -estimateur de la fonction de variance Étape 1 : Estimons f * par C 1. Soit M 1 ≥ 2 fixé. Nous considérons M 1 estimateurs de la fonction de régression f1 , . . . , fM 1 basée sur D n . 2. Pour tout λ ∈ Λ M 1 , nous définissons la combinaison linéaire fλ : fλ = M 1 j=1 λ j fj . Basée sur D N , nous déterminons λ ∈ argmin λ∈Λ M 1 RN ( fλ ).

3 . 1 j=1 λj fj , Étape 2 :

 312 Le C-estimateur de la fonction de régression est donné comme suit fC := fλ = M Estimons σ 2 par C 4. Étant donné l'estimateur fC construit sur D N , nous construisons M 2 estimateurs de la fonction de variance σ2 λ,1 , . . . , σ2 λ,M 2 construits à partir de D n par la méthode basée sur les erreurs résiduelles avec M 2 ≥ 2 fixé.

Algorithme 4

 4 équilibre entre les deux termes. Comme p(y|X) et λ * ℓ sont inconnues, nous estimons Γ * ℓ par algorithme semi-supervisé de type plugin : Algorithme 4. Intervalle de prédiction plug-in de longueur ℓ 1. Nous introduisons deux échantillons indépendants : D n et D M 2. À partir de D n , nous construisons un estimateur p(y|x) de p(y|x) par la règle plug-in, c'est-àdire, nous construisons des estimateurs f et σ2 de f * et de σ 2 respectivement

1. 7

 7 Conclusion et perspectives de recherche • Randomisation. Nous introduisons une perturbation aléatoire ζ distribuée selon une distribution uniforme sur [0, u], pour u > 0 et indépendant de (X, Y ) et nous considèrons p (y|X, ζ) = p(y|X) + ζ1 {|y|≤s} .

Contents 2 . 1

 21 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  [p]. Let n and N be integers. For any function f : R d → R, we define the empirical norm f 2 N = 1 N n+N i=n+1 |f (X i )| 2 and the supremum norm f ∞ = sup x∈R d |f (x)|. Moreover, we denote by Λ p := {λ ∈ R p : λ j ≥ 0, p j=1 λ j = 1} the simplex. Let • 1,p denote the ℓ 1 norm on R p , that is, λ 1,p := p j=1 |λ j |. For the sake of simplicity, let Z = (Yf * (X)) 2 .

Assumption 3 .

 3 For all s ∈ [M 1 ] and all m ∈ [M 2 ] , fs and σ2 s,m are bounded a.s D n . More precisely, there exist two positive constants K 1 and K 2 such that for all n ∈ N * max s∈[M 1 ]

Figure 2 . 1 - 4 values of σ 2 FrequencyFigure 2 . 2 -

 214222 Figure 2.1 -Histogram of values of σ 2 in Gaussian models.

1 .

 1 simulate three datasets D n , D N and D T with n ∈ {100, 1000}, N ∈ {100, 1000} and T = 1000; 2. based on D n , we compute the dictionary F, and then based on D N , we compute the aggregates fMS (that is ŝ) and fC (that is λ) of the regression function f * provided in Eqs (2.2) and (2.4); 3. based on D n and fMS (resp. fC ), we compute the collection G 1 (resp. G 2 ) and we calculate σ2 MS and σ2 C on D N ; 4. based on D n ∪ D N : firstly, we compute the collection F 1 ; secondly, for each estimate fs in F we calculate the estimators {σ 2 s,m } 1≤m≤12 of σ 2 corresponding to the 12 procedures in F; 5. finally, over D T , we compute the empirical L 2 -error of the aggregates σ2 MS and σ2 C . On the other hand, we compute the collection of estimators of the variance function {σ 2

1

 1 (a = 0.25) 0.028 (0.018) 0.031 (0.023) 0.013 (0.003) 0.011 (0.004) 0.014 (0.003) 0.011 (0.001) Model 1 (a = 1) 0.

  n = N = 1000

Figure 2 . 3 -

 23 Figure 2.3 -Boxplot of the empirical L 2 -error of the estimators in Model 1 (a = 0.25)

Figure 2 . 4 -

 24 Figure 2.4 -Boxplot of the empirical L 2 -error of the estimators in Model 1 (a = 1)

Figure 2 . 5 -

 25 Figure 2.5 -Boxplot of the empirical L 2 -error of the estimators in Model 2

Figure 2 .Figure 2 . 6 -

 226 Figure 2.6 -Boxplot of the L 2 -error of the estimators in sparse model when p = 50, and s = 14.

Figure 2 . 7 -

 27 Figure 2.7 -Boxplot of the L 2 -error of the estimators in Model 4.

Figure 2 . 8 -Figure 2 . 9 -

 2829 Figure 2.8 -Boxplot of the L 2 -error of the estimators in Model 5.

Figure 2 . 10 - 9 Figure 2 . 11 -

 2109211 Figure 2.10 -Boxplots of the L 2 -risk of our aggregation methods and the best method.

Figure 2 . 12 -

 212 Figure 2.12 -Curves of six estimators of the regression function and the variance function.

  the true risk of the variance function. Besides, we introduce a minimizer of the risk R, denoted by σ2 MS and given σ2 MS := σ2 ŝ, m , where m ∈ argmin m∈[M 2 ]

  m∈[M 2 ] | RN (σ 2 ŝ,m ) -R N (σ 2 ŝ,m )| . By definition of RN and R N , and under Assumption 3, we get for all m ∈ [M 2 ]

, where β ∈ argmin β∈Λ M 2 R 2 R

 22 N (σ 2 λ,β ) , and σ2 C := σ2 λ, β , where β ∈ argmin β∈Λ M (σ 2 λ,β ).

Contents 3 . 1 3 . 2

 3132 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Regression with reject option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 1 -

 31 Figure 3.1 -Err Γλ and r Γλ vs. λ.

Figure 3 . 2 -

 32 Figure 3.2 -Histogram of the estimates of σ 2 (X)

Figure 3 . 3 -

 33 Figure 3.3 -Visual description of the performance of three plug-in ε-predictors on the aquatic, and airfoil datasets.

Figure 3 . 4 -

 34 Figure 3.4 -Additional description of the performance of the plug-in procedure on aquatic dataset.

Figure 3 . 5 -

 35 Figure 3.5 -Visual description of the performance of four plug-in ε-predictors.

1 .

 1 simulate four datasets D n , D M , D N and D T with M = 150, N = 150 and T = 100; 2. based on D n , we compute the estimators in F, and then based on D M , we compute the aggregates fMS and fCM . Then, we compute the knn, Lasso, rf and svm estimators of the regression function on D n ∪ D M ; 3. based on D n and fMS (resp. fCM ), we compute the estimators in G 1 (resp. G 2 ). Then, based on D M we calculate σ2 MS and σ2 CM . From D n ∪ D M , we compute the knn, Lasso, rf and svm estimators of σ 2 ; 4. based on D N , we compute the empirical cumulative distribution function of the randomized estimators σ2 (X); 5. finally, over D T , we compute the empirical rejection rate r and the empirical error Err for the considered Γε .

Figure 3 . 6 -

 36 Figure 3.6 -Visual description of the performance of six plug-in ε-predictors.

. 4 ). 5 )

 45 For ii) -Randomization. To ensure the continuity of the conditional C.D.F. of p(y|X) for y ∈ [-s, s], we introduce a random perturbation ζ distributed according to a Uniform distribution on [0, u], for u > 0 and independent of (X, Y ). We then define the randomized version of p as p(y|X, ζ) = p(y|X) + ζ1 {|y|≤s} . (4For iii) -Discretization. To approximate G, we simply consider the Riemann sum based on the regular grid G = {y 1 , . . . , y M } of [-s, s] for some M ≥ 1. To this end, we introduce (ζ 1 , . . . , ζ N ) i.i.d. copies of ζ and then define Ĝ

Theorem 16 .

 16 Let Assumptions 12 and 15-17 be satisfied. Let k n ∝ n 2/d+2 , s = log(min(n, N )), M > 4 √ N , and u n = 1

Figure 4 . 1 -

 41 Figure 4.1 -Boxplot of the output Y for d = 1, 5Expected lengthError rateℓ d = 1 d = 5 d = 1 d = 5 0.1 0.1 (0.01) 0.1 (0.01) 0.81 (0.01) 0.94 (0.01) 0.5 0.49 (0.01) 0.49 (0.01) 0.34 (0.01) 0.71 (0.01) 1 0.99 (0.01) 0.99 (0.01) 0.07 (0.01) 0.48 (0.01) 2 1.99 (0.03) 1.99 (0.01) 0.00 (0.00) 0.17 (0.01)

i

  ) estimate λ * ℓ from an unlabeled dataset of size N = 1000 on a regular grid of size M = 1000 of the interval [-5, 5];ii) derive the resulting prediction interval on the same grid over a test set of size T = 1000;iii) based on the test set, compute the expected length and the error rate.

  (4.5)); ii) compute Ĝ-1 (ℓ) using an unlabeled dataset of size N = 100 on a regular grid of size M = 100 of the interval [-s, s], where s = max(-min(Y train ); max(Y train ), iii) derive the resulting prediction interval on a regular grid of size 1000 of [-s, s] over a test set of size T = 1000; iv) based on the test set, compute the expected length and the error rate.

Figure 4 . 2 -

 42 Figure 4.2 -Visual description of the empirical PI Γ and its oracle counterpart Γ * ℓ , with ℓ = 0.5 on the left and ℓ = 1 on the right for d = 1. The scatter plot of data is displayed and the graph of both regression function f * and estimator f is represented (solid line for f * , dashed line for f . The oracle PI Γ * ℓ (empirical PI Γ, respectively) is given in blue (orange, respectively).

  as a function of N for and

Figure 4 . 3 -

 43 Figure 4.3 -Comparison between Γ and Γβ . We plot the expected length (on the left) and the expected coverage (on the right) as a function of N over 20 repetitions for Γ (dashed) and Γβ (solid line in red).The true value of the parameter is given by the dotted line.

Lemma 15 .

 15 y ≥ t)dy , is continuous on R + . Let T uniformly distributed on [-a, a] and independent of (Z y ) y∈[-a,a] . We consider the random variable Z T and let U be distributed according to the uniform distribution on [0, 1]. ThenH(Z T ) L = U and H -1 (U ) L = Z T .

P

  (H(Z T ) ≤ t) = P(Z T ≥ H -1 (t)|T = y)) dP T (y) y ≥ H -1 (t)) dy = H(H -1 (t)) = t ,and we deduce that H(Z T ) L = U . For the second point of the Lemma, we observe that P(H -1 (U ) ≤ t) = P(U ≥ H(t)) y ≤ t|T = y))dy = P(Z T ≤ t) .

Theorem 17 .

 17 Grants Assumptions 15, 16, for k n ∝ n -2/d+2 , and all α > 0, the K-NN estimators defined in Equation (4.7) satisfyE sup x∈C | f (x)f * (x)| 1+α ≤ C log(n) 1+α n -(1+α)/(2+d) , E sup x∈C |σ 2 (x)σ(x)| 1+α ≤ C log(n) 1+α n -(1+α)/(2+d) .

Proof of Proposition 10 .

 10 First, let us consider the Lagrangian of the optimization problem 4.2. It can be written asH(Γ, λ) = P (Y / ∈ Γ(X)) + λ (E X [L(Γ(X)]ℓ) ,where λ ≥ 0 is a dual variable of the problem. Since,P (Y ∈ Γ(X)) = E X E 1 {Y ∈Γ(X)} |X = E X R p(y|X)1 {y∈Γ(X)} dy ,the Lagrangian reads asH(Γ, λ) = 1λℓ -E X R (p(y|X)λ)1 {y∈Γ(X)} dy .(4.10)Minimizing w.r.t. Γ leads to an optimal solution that can be written for all λ ≥ 0 and all x ∈ R d asΓ * (λ, x) = {y ∈ R : p(y|X) ≥ λ} .Injecting this value into 4.10 givesH(Γ * (λ, X), λ) = 1λℓ -E X R(p(y|X)λ) + 1 {y∈Γ * (λ,X)} dy ,

Proof of Proposition 11 .

 11 Let ℓ ≥ 0. Considering a similar decomposition as in the proof of Proposition 10, we can write the error rate of a predictor Γ asR ℓ (Γ) = 1 -E X R (p(y|X)λ * ℓ )1 {y∈Γ(X)} dy .(4.11)Therefore, we deduceE ℓ (Γ) = E X R (p(y|X)λ * ℓ ) 1 {y∈Γ * ℓ (X)} -1 {y∈Γ(X)} dy ,and the result follows from the fact that 1 {y∈Γ * ℓ (X)} -1 {y∈Γ(X)} = sgn(p(y|X)λ * ℓ ) since we have the equality between events {y ∈ Γ * ℓ (X)} = {p(y|X)λ * ℓ ≥ 0}, where sgn : R → {-1, 1} stands for the sign.

R 1 {R 1

 11 Let us then define the randomized predictor Γ(X, ζ) = {y ∈ R : p(y|X, ζ) ≥ Ḡ-1 (ℓ)} , (4.12)where Ḡ(t) := R P X,ζ (p(y|X, ζ) ≥ t)dy for all t > 0. Here again, the property L( Γ) := E X,ζ L Γ(X, ζ) = ℓ is due to the fact that the conditional on the data D n the r.v. p(y|X, ζ) has no atoms since it is randomized.Let us now consider the purpose of the proposition. We need to bound E L( Γ)ℓ . We can writeL( Γ)ℓ = L( Γ) -L( Γ) = E R 1 { Ĝ(p(y|X,ζ))≤ℓ} -1 { Ḡ(p(y|X,ζ))≤ℓ} dy (4.13) ≤ E Ĝ(p(y|X,ζ))≤ℓ} -1 { Ḡ(p(y|X,ζ))≤ℓ} dy ≤ E {| Ĝ(p(y|X,ζ))-Ḡ(p(y|X,ζ))|≥| Ḡ(p(y|X,ζ))-ℓ|} dy = R P | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ | Ḡ(p(y|X, ζ)) -ℓ| dy , where we use Fubini's theorem at last. Now notice that the above integral is limited to the compact [-s, s] since, this is the support of the function p(•|x, z) for all (x, z) ∈ R d × [0, u]. To bound this integral, we make use of the peeling technique of [2]. That is, we consider for δ > 0 and y ∈ [-s, s] A 0 (y) = 0 ≤ | Ḡ(p(y|X, ζ)) -ℓ| ≤ δ A j (y) = 2 j-1 δ ≤ | Ḡ(p(y|X, ζ)) -ℓ| ≤ 2 j δ , for j ≥ 1 .

P1 2 .

 2 | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ | Ḡ(p(y|X, ζ)) -ℓ| dy = s -s j≥0 P | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ | Ḡ(p(y|X, ζ)) -ℓ| , A j (y) dy . (4.14) Controlling this term relies on a bound on s -s P(A j (y))dy. It is clear that 0 ≤ Ḡ(t) = s -s P X (p(y|X, ζ) ≥ t|D n )dy ≤ 2s for all t ∈ [0, 1].We can apply Lemma 15 to say that Ḡ(Z T ) is uniformly distributed on [0, 2s] and then, for all j ≥ 0 and δ > 0, we deduce that s -sP(A j (y))dy = 2s 1 2s s -s P | Ḡ(p(y|X, ζ)) -ℓ| ≤ 2 j δ | D n dy = 2s × P | Ḡ(Z T ) -ℓ| ≤ 2 j δ|D n ≤ 2s 2 j+1 δ 2s = 2 j+1 δ . (4.15)Next, let us consider (4.14). We observe that for all j ≥ 1 s -sP | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ | Ḡ(p(y|X, ζ)) -ℓ| , A j (y) dy ≤ s -s P | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ 2 j-1 δ , A j (y) dy ≤ s -s E (Dn,X,ζ) P D N | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ 2 j-1 δ 1 A j (y) dy. (4.16) In Section 4.3.1, we have presented the predictor Γ that relies on the function Ĝ which is discretized. On the other hand, Ḡ is not discretized. Because of this difference, it is convenient, in order to control (4.16), to provide some additional notation. Let us define Ĝ(t) := 1 N {p(y|X n+i ,ζ i )≥t} dy . Let us now consider the second term. Conditional on (D n , X, ζ), the random variable Ĝ(p(y|X, ζ)) is an empirical mean of i.i.d. random variables of common mean Ḡ(p(y|X, ζ)) ∈ [0, 2s], we deduce from Hoeffding's inequality thatP D N | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ 2 j-2 δ|D n , X ≤ 2 exp -N δ 2 2 2j-116sTherefore, from Inequalities (4.14), (4.15), (4.16), and (4.18) one gets for δ = 4s √ N and M > 4 √ N s -s P | Ĝ(p(y|X, ζ)) -Ḡ(p(y|X, ζ))| ≥ | Ḡ(p(y|X, ζ)) -ℓ| dy ≤

R 1 { 21 )

 121 term of the r.h.s. in the above equation, thanks to Equation(4.11), we have thatR ℓ ( Γ) -R ℓ ( Γ) = E X,ζ R (p(y|X)λ ℓ ) 1 {y∈ Γ(X,ζ)} -1 {y∈ Γ(X,ζ)} dy .From Assumption 12, we have that |p(y|X)λ ℓ | is bounded by C 1 > 0 which depends on σ 0 . Hence, we deduce thatE R ℓ ( Γ) -R ℓ ( Γ) ≤ C 1 E R 1 {y∈ Γ(X,ζ)} -1 {y∈ Γ(X,ζ)} dy .This last inequality can be rewritten asE R ℓ ( Γ) -R ℓ ( Γ) ≤ C 1 E Ĝ(p(y|X,ζ))≤ℓ} -1 { Ḡ(p(y|X,ζ))≤ℓ} dy .Therefore, from Equation 4.13, and (4.19), we deduceE R l ( Γ) -R l ( Γ)Now we bound the first term in the r.h.s. in Equation(4.20). Thanks to Proposition 11, we have thatE ℓ ( Γ) = E X,ζ Γ(X,ζ)△Γ * ℓ (X) |p(y|X)λ * ℓ | dy .Now, we consider the following cases• If y ∈ Γ(X, ζ) \ Γ * ℓ (X), we have that p(y|X) < λ * ℓ and p(y|X, ζ) ≥ λℓ . Therefore, |p(y|X)λ * ℓ | = (λ * ℓ -λℓ ) + ( λℓp(y|X, ζ)) + (p(y|X, ζ)p(y|X)) . Using the fact that λℓp(y|X, ζ) ≤ 0, we get |p(y|X)λ * ℓ |1 {y∈ Γ(X,ζ)\Γ * ℓ (X)} dy ≤ (λ * ℓ -λℓ ) + p(y|X, ζ)p(y|X) 1 {y∈ Γ(X,ζ)\Γ * ℓ (X)} dy . • If y ∈ Γ * ℓ (X) \ Γ(X,ζ), we have that p(y|X) ≥ λ * ℓ and p(y|X, ζ) < λℓ . Therefore, |p(y|X)λ * ℓ | = (p(y|X)p(y|X, ζ)) + (p(y|X, ζ) -λℓ ) + ( λℓλ * ℓ ) . Using the fact that p(y|X, ζ) -λℓ < 0, we get|p(y|X)λ ℓ |1 {y∈Γ * ℓ (X)\ Γ(X,ζ)} dy ≤ ( λℓλ * ℓ ) + p(y|X, ζ)p(y|X) 1 {y∈Γ * ℓ (X)\ Γ(X,ζ)} dy .From the above considerations, we deduce the following inequalityE X,ζ Γ(X,ζ)△Γ * ℓ (X) |p(y|X)λ * ℓ | dy ≤ | λℓλ * ℓ | E R 1 {y∈ Γ(X,ζ)△Γ * ℓ (X)} dy + E R |p(y|X, ζ)p(y|X)| dy ≤ | λℓλ * ℓ | × (L( Γ) -L(Γ * )) + E R |p(y|X)p(y|X)| dy + 2su , where the last inequality is due to the fact that p(y|X, ζ) = p(y|X) + ζ1 y∈[-s,s] with |ζ| ≤ u. But L( Γ) = L(Γ * ) = ℓ by construction. Then, E E ℓ ( Γ) ≤ E R |p(y|X)p(y|X)| dy + 2su . Injecting this last inequality and (4.21) into (4.20) gives the announced result.

Proposition 16 .

 16 Under Assumption 12, we have thatE R |p(y|X)p(y|X)| dy ≤ C √ sE ( f (X)f * (X)) 2 + E f (X)f * (X) + Cs 5/2 E |σ 2 (X)σ 2 (X)| ,where C > 0 is a constant which depends on σ 0 and σ 1 in Assumption 12.

since 1 / 1 2πσ 2 1 2πσ 2 1 - 1 2πσ 2

 11212112 σ 0 ≤ C. The same reasoning holds in the case where σ(X) < σ(X) and then we conclude that|σ(X)σ(X)| ≤ C|σ 2 (X) -σ2 (X)| .Injecting this bound into (4.22) and using again Assumption 12, we deduce thatR (X) exp -(y -f (X)) 2 2σ 2 (X) -(X) exp -(y -f (X)) 2 2σ 2 (X) dy ≤ C |σ(X)σ(X)| ≤ C σ 2 (X) -σ2 (X) . (4.23)Let us now consider the second term in the decomposition of |p(y|X)p(y|X)|. Since x → exp(-x) Lipschitz on R + , from Assumption 12 we have that in the case where (y -f (X)) 2 ≥ (yf * (X)) 2 (X) exp -

1 2πσ 2

 12 switching the role of f by f * . Notice that R 12πσ 2 (X) exp -(yf * (X)) 2 2σ 2 (X) × |yf * (X)|dy is the expectation of the r.v. |Yf * (X)| where Y is Gaussian with expectation f * (X) and variance σ2 (X). Therefore, using the fact that E [|Z -E [Z]|] ≤ Var(Z) for any real valued random variable Z, we get R (X) exp -

4. 7 R 1 2 RPChapitre 4 :

 7124 Appendix• on the event { Ḡ-1 (ℓ) ≥ G -1 (ℓ)}, |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) ,• on the event { Ḡ-1 (ℓ) < G -1 (ℓ)},either p(y|X, ζ) ∈ ( Ḡ-1 (ℓ), G -1 (ℓ)) or |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) .Note that similar reasoning holds if y ∈ Γ * ℓ (X) \ Γ(X, ζ). Therefore, we deduce that conditional onD n , E Γ(X,ζ)△Γ * (X) dy ≤ E R 1 {|p(y|X,ζ)-p(y|X)|≥|p(y|X)-G -1 (ℓ)|} dy + 1 { Ḡ-1 (ℓ)<G -1 (ℓ)} E R 1 {p(y|X,ζ)∈( Ḡ-1 (ℓ),G -1 (ℓ))} dy + 1 { Ḡ-1 (ℓ)≥G -1 (ℓ)} E R 1 {p(y|X,ζ)∈(G -1 (ℓ), Ḡ-1 (ℓ))} dy .Using first the definition of Ḡ and then the fact that Ḡ( Ḡ-1 (ℓ)) = G(G -1 (ℓ)) = ℓ in this last inequality, we deduce the followingE Γ(X,ζ)△Γ * (X) dy ≤ E R 1 {|p(y|X,ζ)-p(y|X)|≥|p(y|X)-G -1 (ℓ)|} dy + E G(G -1 (ℓ)) -Ḡ(G -1 (ℓ)) . Now, we observe that E G(G -1 (ℓ)) -Ḡ(G -1 (ℓ)) ≤ E R 1 {p(y|X)≥G -1 (ℓ)} -1 {p(y|X,ζ)≥G -1 (ℓ)} dy ≤ E R 1 {|p(y|X,ζ)-p(y|X)|≥|p(y|X)-G -1 (ℓ)|} dy .Therefore, we have obtained E Γ(X,ζ)△Γ * (X) dy ≤ 2E {|p(y|X,ζ)-p(y|X)|≥|p(y|X)-G -1 (ℓ)|} dy (4.31) ≤ |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) dy .Let us consider the term in the r.h.s of Equation(4.31). Let δ > 0, we have thatR P |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) dy ≤ R P (|p(y|X, ζ)p(y|X)| ≥ δ) dy + R P p(y|X) -G -1 (ℓ) ≤ δ dy .From Markov's inequality, we deduceR P |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) dy ≤ 1 δ E R |p(y|X, ζ)p(y|X)| dy + G(G -1 (ℓ)δ) -G(G -1 (ℓ) + δ) . (4.32) Since p is supported on [-s, s], we observe that E R |p(y|X, ζ)p(y|X)| dy = E [-s,s] |p(y|X, ζ)p(y|X)| dy + E +∞ |y|≥s p(y|X)dy ≤ E [-s,s] |p(y|X)p(y|X)| dy + 2su + E +∞ |y|≥s p(y|X)dy . (4.33) Prediction interval with fixed expected length in the Gaussian heteroscedastic regression Now, we observe that E +∞ s p(y|X)dy = E 1 {|f * (X)|≤ s 2 } +∞ s p(y|X)dy + E 1 {|f * (X)|> s

  C and C ′ are two positive constants. Note that similar arguments yields E Therefore considering Equation (4.33) and Proposition 16 and defining s = log(min(n, N )), we get lim , ζ)p(y|X)| dy = 0 .Hence we obtain from Equation (4.32) that for all δ > 0 lim supn R P |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) dy ≤ G(G -1 (ℓ)δ) -G(G -1 (ℓ) + δ) .Since G is continuous, with δ → 0, we getlim n R P |p(y|X, ζ)p(y|X)| ≥ p(y|X) -G -1 (ℓ) dy = 0 .The above equation together with Equation (4.29), (4.30), (4.31) yields the desired result.

Proposition 17 . 2 + s 2

 1722 Let s = log(min(n, N )). Under Assumptions 12, 15, and 16, we have thatsup (x,y)∈C×[-s,s] |p(y|x)p(y|x)| ≤ C s sup x∈C f (x)f * (x) sup x∈C f (x)f * (x) + s 3 sup x∈C σ2 (x)σ 2 (x),

1 2πσ 2 1 2πσ 2

 1212 (x) exp -(y -f (x)) 2 2σ 2 (x) -(x) exp -(y -f (x)) 2 2σ 2 (x) ≤ Cs sup x∈C σ2 (x)σ 2 (x) .

(4. 36 )

 36 The combination of Equations (4.34),(4.35), and (4.36) gives the proposition.Proof of Proposition 14. We recall thatE ℓ Γ = E Γ(X,ζ)△Γ * ℓ (X) |p(y|X)λ * ℓ | dy .Now, we observe that for y ∈ Γ(X, ζ)△Γ * ℓ (X) |p(y|X)λ * ℓ | ≤ |p(y|X, ζ)p(y|X) | + λℓλ * ℓ ,

  (x,y)∈C×R |p(y|x)p(y|x)| + u := m(u) .Therefore, we deduce thatE Γ(X,ζ)△Γ * ℓ (X) |p(y|X)λ * ℓ | dy ≤ 2 m(u)E R 1 {|p(y|X)-λ * ℓ |≤2 m(u)} dy .Hence from the above inequality, and Assumption 17 we getE E ℓ Γ ≤ 2 1+α c 0 E m(u) 1+α .Therefore, from Equations (4.20) and (4.21), we obtain the following with s = log(min(n, N ))E E ℓ Γ ≤ C   E   sup (x,y)∈R×C |p(y|x)p(y|x)| 1+α   + u 1+α + log(N ) N   .Finally, since p is supported on [-s, s], we have sup (x,y)∈C×R |p(y|x)p(y|x)| ≤ sup (x,y)∈C×[-s,s] |p(y|x)p(y|x)| + sup (x,y)∈C×R\[-s,s]p(y|x) .

For2 8σ 2 1 ≤

 1 n, N large enough, we can assume, since f * is bounded, that |f * (X)| ≤ s/2. From Assumption 12, we have for n, N large enough sup (x,y)∈C×R\[-s,s] p(y|x) ≤ C exp -s exp(-s) ≤ C min(n, N ) , which yields the desired result.

  . 6 1.2.1 Estimateurs linéaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 Estimateurs non-linéaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.3 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Agrégation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.3 Agrégation d'estimateurs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.3.1 1.3.2 Propriété de l'agrégat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3.3 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4 Estimation de la fonction de variance conditionnelle . . . . . . . . . . . . . . . . . . . 16 1.4.1 Design aléatoire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

  Le problème d'optimisation précédent n'est pas quadratique. Pour palier ce problème, il est usuel de considérer le concept de marge souple en introduisant des variables d'écart ξ i et ξ * i pour formuler le problème d'optimisation

	min	1 2	ω 2 + c	n i=1

  Les arbres de régression (aussi appelés arbres de décision) sont des méthodes qui permettent d'obtenir des modèles prédictifs. Ils sont devenus très populaires au vu de leur faible temps de calcul, de leur capacité à gérer tous types de variables et à sélectionner les plus pertinentes, ainsi que la lisibilité et la facilité d'interprétation des résultats. L'algorithme le plus utilisé est l'arbre CART ( CART signifie Classification And Regression Trees). Il consiste à partitionner récursivement l'espace d'entrée R d de façon binaire et à déterminer une sous-partition optimale pour la prédiction.

• Les arbres de régression

[START_REF] Breiman | Classification and Regression Trees[END_REF][START_REF] Gey | Bornes de risque, détection de ruptures, boosting : trois thèmes statistiques autour de cart en régression[END_REF][START_REF] Genuer | Arbres CART et Forêts aléatoires, Importance et sélection de variables[END_REF]

.

  Nous commençons par présenter l'intérêt de l'agrégation d'estimateurs par l'exemple de l'adaptation à la régularité inconnue d'une fonction (la fonction peut par exemple être une densité ou une fonction de régression). Il est bien connu que le choix d'un paramètre dit de régularisation est crucial pour l'adaptation à la régularité d'une fonction. Ce paramètre peut être la fenêtre pour les méthodes à noyau, le paramètre de lissage pour l'estimateur Lasso ou la constante de régularisation pour les méthodes pénalisées. Les procédures du choix de ce paramètre par validation croisée, par boostrap parmi d'autres méthodes, permettent de sélectionner le paramètre d'une façon aléatoire (dépendante des données). L'objectif est de déterminer une seule valeur utile du paramètre. Ce qui se traduit par un problème de la sélection du modèle. Nous pouvons envisager d'utiliser plusieurs estimateurs au lieu d'un seul en les agrégeant d'une manière adéquate. Nous nous intéressons dans la suite de ce chapitre aux techniques d'agrégation dans le cadre de la régression.Dans le cadre plus développé de l'estimation de la fonction de régression, l'agrégation a pour but d'estimer la fonction de régression inconnue f * par une combinaison d'éléments d'un ensemble connu de fonctions appelé dictionnaire. Ces fonctions sont des estimateurs tels que : la régression li-

L'idée de combiner des prédicteurs (estimateurs) est ancienne. Cette approche, appelée agrégation d'estimateurs, a été développée tant d'un point de vue théorique qu'algorithmique. Elle a été étudiée dans les communautés du machine learning, statistique, et traitement d'images

[START_REF] Breiman | Bagging predictors[END_REF][START_REF] Freund | Boosting a weak learning algorithm by majority[END_REF][START_REF] Breiman | Random forests[END_REF][START_REF] Biau | On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification[END_REF][START_REF] Bunea | Aggregation for gaussian regression[END_REF][START_REF] Juditsky | Functional aggregation for nonparametric regression[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF]

. néaire, les k-PPV, les noyaux, les forêts aléatoires, les SVM, le lasso, le ridge, les réseaux de neurones, le bagging, le gradient boosting, etc. Étant donné M (M ≥ 2) estimateurs de f * notés f1 , . . . , fM , le problème de l'agrégation consiste à chercher un nouvel estimateur f en combinant les estimateurs précédents de manière appropriée. Ce nouveau estimateur f est appelé agrégat et sa construction est appelée agrégation. Les procédures d'agrégation sont généralement basées sur la division de l'échantillon principal D

1.5.2 Intervalles de prédiction

	Chapitre 1: Introduction
	Basé sur ERM, les auteurs ont analysé l'option rejet du point de vue de l'ε-optimalité, et ont garanti
	que la prédiction est à l'intérieur d'une boule de rayon ε autour du fonction de régression avec
	grande probabilité tout en rejetant seulement une partie limitée du domaine. L'inconvénient de cette
	approche est que leur méthodologie est associée seulement à des procédures de minimisation du
	risque empirique. De plus, la calibration de ε est très délicate. Il est naturel de poser la question
	suivante :
	Question 2. Est-il possible de généraliser le problème de régression avec option rejet en utilisant
	n'importe quelle procédure ?
	Nous apporterons les éléments de réponse dans la section 1.6.2.
	Les intervalles de prédiction (IP) jouent un rôle central en statistique inférentielle. La construction
	d'IP nécessite une estimation fiable des objets inconnus f * et σ 2 . En pratique, il est important de disposer de mesures précises de l'incertitude des prévisions d'un modèle pour éviter les conclusions
	erronées.
	L'intervalle de prédiction est une plage de valeurs susceptibles d'inclure le paramètre d'intérêt avec
	un certain degré de confiance. L'estimation en tout point n'est plus une valeur unique, mais une ré-
	gion de prédiction/tolérance.
	Bien que la construction d'IP est très ancienne, de nouvelles approches ont vu le jour. Ainsi, la
	prédiction conforme a fait récemment l'objet de nombreux efforts de recherche en statistique avec
	d'intéressants développements méthodologiques, informatiques et théoriques depuis le début des
	années 2000. Elle a été initialement formulée pour la tâche de classification [94, 79, 95] puis elle a
	été utilisée en régression [50, 51, 52, 84]. Étant donné l'échantillon D n , l'objectif est de prédire le la-bel Y n+1 associée à la nouvelle entrée X n+1 . Plus précisément, étant donné un niveau de confiance
	α ∈ [0, 1], la prédiction conforme permet de fournir un ensemble de confiance

  Y ) est la densité jointe de (X, Y ). La validité marginale n'est pas conditionnée par X n+1 . Dans ce cas, il est possible d'avoir des garanties plus solides par Validité conditionnelle. La validité conditionnelle est donnée par :

Agrégation de type sélection modèle. La

  respectivement en n et N copies i.i.d. de (X, Y ). Nous décrivons l'algorithme d'estimation de la fonction de variance σ 2 en utilisant deux types d'agrégation : agrégation de type sélection de modèle et l'agrégation convexe. Les estimateurs résultants sont appelés MS-estimateur et C-estimateur respectivement. méthode que nous proposons est en deux étapes et est décrite dans l'Agorithme 1. MS-estimateur de la fonction de variance Étape 1 : Estimons f * par MS 1. Soit M 1 ≥ 2 fixé. Nous considérons M 1 estimateurs de la fonction de régression f1 , . . . , fM 1 basée sur D n .

	Algorithme 1 : 2. Nous sélectionnons l'indice optimal	ŝ
	ŝ ∈ argmin s∈[M 1 ]	RN ( fs ), avec RN ( fs ) =	1 N	n+N i=n+1	|Y i -fs (X i )| 2 .
	3. Le MS-estimateur de la fonction de régression, noté fMS , est donné comme suit
			fMS := fŝ .
		Étape 2 : Estimons σ 2 par MS
	4. Étant donné l'estimateur fMS construit sur D N , nous construisons M 2 estimateurs de la fonc-tion de variance σ2 ŝ,1 , . . . , σ2 ŝ,M 2 , construits à partir de D n

par la méthode basée sur les erreurs résiduelles avec

  2 ≤ M 2 < ∞.5. Sur la base de D N , nous sélectionnons l'indice optimal m comme suit

  1, we describe four heteroscedastic models in the gaussian case and two models when Y is bounded. Second, we evaluate the performances of MS-estimator and C-estimator for different examples in Section 2.4.2. Once we have calibrated our estimate of the variance function σ 2 , we exploit it to consider the quantile regression in Section 2.4.4.

Table 2 .

 2 1 -Average and standard deviation of the empirical L 2 -error of the three estimators with n = N .

			n = 1000 N = 100			n = 100, N = 1000
		C	MS	Best	C	MS	Best
	Model	" Err	" Err	" Err	" Err	" Err	" Err
	Model 1 (a = 0.25) 0.023 (0.015) 0.028 (0.023) 0.012 (0.002) 0.018 (0.008) 0.019 (0.006) 0.012 (0.002)
	Model 1 (a = 1)	0.335 (0.265) 0.381 (0.343) 0.170 (0.014) 0.262 (0.090) 0.278 (0.081) 0.169 (0.018)
	Model 2	0.193 (0.132) 0.227 (0.189) 0.074 (0.013) 0.159 (0.055) 0.148 (0.054) 0.073 (0.010)
	Model 3	0.252 (0.082) 0.278 (0.149) 0.180 (0.015) 0.259 (0.029) 0.270 (0.035) 0.179 (0.015)
	Model 4	0.021 (0.014) 0.026 (0.027) 0.009 (0.002) 0.019 (0.012) 0.017 (0.015) 0.009 (0.002)
	Model 5	0.295 (0.144) 0.336 (0.209) 0.195 (0.019) 0.313 (0.079) 0.317 (0.095) 0.194 (0.015)

Table 2 .

 2 2 -Average and standard deviation of the empirical L 2 -error of the three estimators with n = N .

		0.15
	L^2 error	0.10
		0.05
		0.00
		C	MS	Best

Table 2 .

 2 3 -Performances of the six plug-in regression quantiles. We compute the means and standard deviations (between parentheses) of the L 2 -error of qτ (Y |X).

		C	MS	knn	rf	svm	Tree
	τ	Err	Err	Err	Err	Err	Err
	0.1 0.0037 (0.0028) 0.0043 (0.0050) 0.0027 (0.0019) 0.0091 (0.0033) 0.0217 (0.0056) 0.0071 (0.0024)
	0.5 0.0013 (0.0011) 0.0013 (0.0013) 0.0013 (0.0008) 0.0042 (0.0018) 0.0006 (0.0005) 0.0027 (0.0012)
	0.9 0.0039 (0.0028) 0.0043 (0.0047) 0.0026 (0.0016) 0.0089 (0.0029) 0.0216 (0.0051) 0.0123 (0.0033)

  Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Benefit of aggregation in regression with reject option . . . . . . . . . . . . . . .

. . . 3.2.1 Predictor with reject option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Optimal predictor with fixed rejection rate . . . . . . . . . . . . . . . . . . . . . 3.3 Plug-in ε-predictor with reject option . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Estimation strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Consistency of plug-in ε-predictors . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Application to kNN algorithm: rates of convergence . . . . . . . . . . . . . . . . . . . 3.4.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 kNN predictor with reject option . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Rates of convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Supplementary material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.1 Proofs for optimal predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.2 Proof of the consistency results: Theorem 11 . . . . . . . . . . . . . . . . . . . . 3.7.3 Proof of rates of convergence: Theorem 12 . . . . . . . . . . . . . . . . . . . . . 3.7.4 Rate of convergence for kNN estimator . . . . . . . . . . . . . . . . . . . . . . . 3.7.5 Technical tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 3 .

 3 1 -Performances of the three plug-in ε-predictors on the real datasets aquatic, and

	airfoil.
				aquatic					airfoil
		svm		rf		knn		svm		rf		knn
	1-ε	Err	1 -r	Err	1 -r	Err	1 -r	Err	1 -r	Err	1 -r	Err	1 -r
	1 0.8 1.08 (0.17) 0.81 (0.05) 1.04 (0.16) 0.80 (0.05) 1.98 (0.26) 0.80 (0.04) 1.38 (0.18) 1.00 (0.00) 1.34 (0.18) 1.00 (0.00) 2.29 (0.27) 1.00 (0.00) 11.81 (1.03) 1.00 (0.00) 14.40 (1.04) 1.00 (0.00) 35.40 (2.05) 1.00 (0.00) 8.27(0.86) 0.80 (0.03) 10.26 (0.95) 0.80 (0.03) 31.13 (1.96) 0.80 ( 0.03) 0.5 0.91 (0.18) 0.50 (0.06) 0.81 (0.18) 0.50 (0.06) 1.51 (0.30) 0.50 (0.06) 5.15 (0.92) 0.50 (0.04) 7.22 (0.92) 0.50 (0.3) 22.42 (2.13) 0.50 (0.03) 0.2 1.01 (0.32) 0.19 (0.05) 0.55 (0.21) 0.20 (0.05) 0.75 (0.37) 0.19 (0.05) 2.6 (0.64) 0.20 (0.03) 4.00 (0.74) 0.20 (0.03) 17.27 (3.00) 0.19 (0.03)

Table 3 .

 3

		Concrete compressive strength
		rf		svm		knn		Lasso
	ε	Err	r	Err	r	Err	r	Err	r
	0 31.33 (7.82) 0.00 (0.00) 46.83 (8.65) 0.00 (0.00) 87.32 (15.74) 0.00 (0.00) 111.72 (16.13) 0.00 (0.00)
	0.2 20.99 (5.72) 0.21 (0.06) 33.98 (7.54) 0.20 (0.06) 65.61 (16.12) 0.20 (0.06) 89.91 (14.63) 0.19 (0.05)
	0.5 13.47 (6.08) 0.48 (0.06) 21.73 (7.38) 0.50 (0.06) 46.71 (19.89) 0.50 (0.06) 66.86 (16.13) 0.50 (0.07)
	0.8 7.26 (8.29) 0.81 (0.05) 18.25 (11.70) 0.81 (0.05) 28.58 (28.66) 0.79 (0.11) 52.32 (16.99) 0.81 (0.05)
		Airfoil Self-Noise
		rf		svm		knn		Lasso
	ε	Err	r	Err	r	Err	r	Err	r
	0 12.57 (1.99) 0.00 (0.00) 10.36 (2.68) 0.00 (0.00) 33.97 (4.45) 0.00 (0.00) 22.51 (3.62) 0.00 (0.00)
	0.2 8.53 (1.48) 0.20 (0.05) 7.42 (2.09) 0.19 (0.05) 29.25 (4.03) 0.20 (0.06) 20.52 (3.17) 0.19 (0.06)
	0.5 6.25 (1.41) 0.52 (0.07) 3.89 (1.45) 0.50 (0.06) 22.42 (4.10) 0.49 (0.06) 15.51 (3.37) 0.50 (0.06)
	0.8 2.82 (1.00) 0.80 (0.05) 1.93 (0.93) 0.80 (0.05) 14.95 (7.12) 0.80 (0.07) 9.07 (3.19) 0.80 (0.05)

2 -Performances of the four plug-in ε-predictors on the real datasets Concrete compressive strength, and airfoil self-Noise.

Table 3 .

 3 3 -Performances of the six plug-in ε-predictors on the real datasets Concrete compressive strength, and Airfoil Self-Noise. 20.07 (5.65) 0.20 (0.05) 32.97 (7.03) 0.20 (0.05) 62.40 (13.93) 0.20 (0.06) 86.51 (11.72) 0.20 (0.05) 23.07 (6.70) 0.20 (0.05) 24.76 (6.91) 0.20 (0.05)

			Concrete compressive strength
		rf		svm		knn		Lasso		C		MS
	ε	Err	r	Err	r	Err	r	Err	r	Err	r	Err	r
	0	30.27 (7.16) 0.00 (0.00) 45.03 (9.03) 0.00 (0.00) 85.59 (16.11) 0.00 (0.00) 110.09 (16.01) 0.00 (0.00) 34.59 (7.74) 0.00 (0.00) 35.08 (7.44) 0.00 (0.00)
	0.2 0.5 13.25 (5.38) 0.48 (0.07) 22.45 (7.16) 0.49 (0.07) 44.98 (15.11) 0.49 (0.07) 64.95 (11.08) 0.49 (0.07) 13.24 (3.87) 0.49 (0.06) 15.14 (4.18) 0.49 (0.07) 0.8 7.91 (8.26) 0.80 (0.05) 17.92 (13.69) 0.80 (0.04) 30.02 (23.85) 0.78 (0.09) 55.91 (20.09) 0.80 (0.06) 8.50 (5.33) 0.80 (0.05) 10.19 (9.77) 0.80 (0.05)
				Airfoil Self-Noise
		rf		svm		knn		Lasso		C		MS
	ε	Err	r	Err	r	Err	r	Err	r	Err	r	Err	r
	0 0.2 8.67 (1.29) 0.20 (0.05) 7.02 (1.86) 0.21 (0.05) 29.15 (4.60) 0.20 (0.05) 21.16 (3.49) 0.20 (0.06) 6.19 (1.17) 0.21 (0.05) 5.82 (1.41) 0.21 (0.05) 12.80 (1.84) 0.00 (0.00) 10.49 (2.18) 0.00 (0.00) 33.78 (4.33) 0.00 (0.00) 23.23 (3.71) 0.00 (0.00) 10.30 (1.98) 0.00 (0.00) 10.77 (2.22) 0.00 (0.00) 0.5 6.08 (1.13) 0.49 (0.07) 3.84 (0.98) 0.48 (0.07) 21.95 (4.42) 0.49 (0.07) 14.83 (3.39) 0.49 (0.08) 4.09 (0.99) 0.49 (0.07) 3.73 (1.15) 0.49 (0.06) 0.8 2.92 (1.02) 0.80 (0.04) 1.97 (1.06) 0.80 (0.05) 14.25 (6.94) 0.81 (0.08) 8.76 (2.64) 0.80 (0.05) 1.68 (0.64) 0.80 (0.04) 1.67 (0.73) 0.80 (0.05)

•

  Step 2. Study of E R λε ( Γε ) -R λε ( Γε ) . Since X and ζ are independent, as in Step 2 of the proof of Theorem 11 (cf. Eq. (3.11)), we get

  1/d , we get that for t ≥ ( kn Cn ) 1/d ,

	P sup x∈C	1 k n	kn i=1

  .[START_REF] Denis | Prediction intervals with controlled length in the heteroscedastic gaussian regression[END_REF] Lemma 11 provides a bound on the second term in the r.h.s. of the above inequality. Then it remains to study the first term in the r.h.s. of Eq.(3.25). Let x ∈ C, and denote by N kn (x) = {X (1,n) (x), . . . X (kn,n) (x)} the set of the k n -nearest neighbors of x among {X 1 , . . . , X n }. We denote by B the set of all closed balls in R d . We observe that there exists ρ x > 0 such that N kn (x) ⊂ { B(x, ρ x ) ∩ {X 1 , . . . , X n }}, where B(x, ρ x ) is the closed ball centered on x with radius ρ x . Therefore {N kn (x), x ∈ C} ⊂ {{X 1 , . . . , X n } ∩ B, B ∈ B} .Besides, since the VC-dimension of the class of balls in R d is upper bounded by d + 2 (see for instance Corollary 13.2 in[START_REF] Devroye | A Probabilistic Theory of Pattern Recognition[END_REF]), Sauer Lemma implies that|{{X 1 , . . . , X n } ∩ B, B ∈ B}| ≤ S (B, n) ≤ (n + 1) d+2 ,where S (B, n) denotes the shatter coefficient of B by n points from C. We then deduce that |{N kn (x), x ∈ C}| ≤ (n + 1) d+2 , which implies in turn that there exists {x 1 , . . . , x J }, with J ≤ (n + 1) d+2 such that

	E	sup x∈C	1 k n	kn i=1

Table 4 .

 4 1 -Performance of the Oracle PI for ℓ ∈ {0.1, 0.5, 1, 2}.

		1 (0.01)	0.1 (0.01)	0.81 (0.01) 0.94 (0.01)
	0.5 0.49 (0.01) 0.49 (0.01) 0.34 (0.01) 0.71 (0.01)
	1	0.99 (0.01) 0.99 (0.01) 0.07 (0.01) 0.48 (0.01)
	2	1.99 (0.03) 1.99 (0.01) 0.00 (0.00) 0.17 (0.01)

  |> s Therefore, Assumption 12 and standard result on Gaussian tails yields for s ≥ 1 E 1 {|f * (X)|≤ s

				2 }	s	+∞	p(y|X)dy ≤ E 1 {|f * (X)|> s 2 } ≤	2C 1 s	.
	On the other hand, we observe that
	E 1 {|f * (X)|≤ s 2 }	s	+∞	p(y|X)dy ≤ E	s	+∞	1 2πσ 2 (X)	exp -	(y -s/2) 2 2σ 2 (X)	dy
											= E	+∞ s/2	1 2πσ 2 (X)	exp -	y 2 2σ 2 (X)	dy .
					2 }	s	+∞	p(y|X)dy ≤	1 √ 2π	exp -	s 2 8σ 2 1

En cas d'égalité Xix = Xjx avec i < j, Xi est déclaré plus proche de x que Xj

Si t = 0, nous avons G(0) = +∞ et nous utiliserons la convention G -1 (+∞) = 0.

Note that this set of estimators differ from the dictionary computed in step

since it is computed in the whole data Dn ∪ DN . We abuse in the notation to avoid extra notation that are irrelevant for the understanding.

We simplify it by svm and use the R package with default parameters: e1071.

By abuse of notation, we refer to Γ * λ as the solution of the penalized problem and to Γ * ε as the solution of the constrained problem.

The only difference between Γε(x, ζ) and Γε(x) given in(3.8) is the dependency in ζ that is hidden inside σ2 . To avoid useless additional notation, we write Γε for both pseudo-oracles.

When t = 0, we have G(t) = +∞ and then we will use the convention G -1 (+∞) = 0.

Here again, we use the convention G-1 (+∞) = 0.

Remerciements

Je suis très reconnaissant envers le Labex MME-DII de m'avoir accordé une

where u = (a/b) 1/m and where we used the trivial inequality P (Z ≥ t) ≤ 1 to bound the first term in the r.h.s. Since (a where we consider the changing of variable v = ((tu) m b) 1/m in the last equality. Finally, using that (a ′ + b ′ ) p ≤ 2 p-1 ((a ′ ) p + (b ′ ) p ) for all p ≥ 1, a ′ , b ′ ∈ R and given that u ≥ (1/b) 1/m , we show from the above inequality that

Inject this into Eq.(3.28) leads to the result.

Chapter 4

Prediction interval with fixed expected length in the Gaussian heteroscedastic regression

Abstract: We tackle the problem of building a prediction interval in heteroscedastic Gaussian regression. We focus on prediction intervals with constrained expected length in order to guarantee interpretability of the output. In this framework, we derive a closed form expression of the optimal prediction interval that allows for the development a data-driven prediction interval based on plug-in. The construction of the proposed algorithm is based on two samples, one labeled and another unlabeled. Under mild conditions, we show that our procedure is asymptotically as good as the optimal prediction interval both in terms of expected length and error rate. In particular, the control of the expected length is distribution-free. We also derive rates of convergence under smoothness and the Tsybakov noise conditions. We conduct a numerical analysis that exhibits the good performance of our method. It also indicates that even with a few amount of unlabeled data, our method is very effective in enforcing the length constraint.

the limiting term is N -1/2 and the rate becomes slow if n -(1+α)/(d+2) goes faster to 0. On the other hand, if the number of unlabeled data N is large with N ≫ n 1+α we recover the fast rate of convergence n -(1+α)/(d+2) . Between these two extremes, N -1/2 can still be the limiting term. However, we hope that in our semi-supervised setting, enough data are available to make this term negligible as compared to the others.

Extension and other approach

In this section, we discuss some points beyond the considered framework in this chapter. The extension of our results to other regression models is presented in Section 4.4.1. Another approach to build prediction interval based on the control of the expected error rate [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF] is described in Section 4.4.2. In particular, we exhibit the main differences with our considered procedure.

Beyond Gaussian setting

In the present work, we study prediction intervals under expected length constraint in the heteroscedastic Gaussian regression setup. The appealing aspect of this framework lies in the form of the optimal predictor

with λ * ℓ = G -1 (ℓ) and G(t) := R P(p(y|X) ≥ t)dy. Furthermore, the density p(y|X) has an explicit expression that exclusively depends on the regression and the conditional variance functions f and σ. Therefore, our proposed algorithm only involves estimators of f and σ to estimate the conditional density p. In particular, we do not consider any general procedure for density estimation.

In this paragraph, we discuss possible extensions outside the Gaussian framework but still considering the regression framework Y = f * (X) + σ(X) ε. In order to make sure that the optimal predictor is well defined, we require the following assumption. Assumption 18. We assume that the variable Y given X has density p(•|X).

If we do not assume that Y |X belongs to a given family of distribution, the characterization of the prediction interval (4.8) still holds but the expression of the conditional density can not be simplify. Therefore, a data-driven predictor, based on the plug-in principle, must rely on estimates p(•|x) of the conditional density p(•|x). The way to build the estimator Γ does not differ from the Gaussian case ones p is obtained (see Section 4.3). From the theoretical perspective, general properties such as Propositions 10 and 11 still hold and the question here is to investigate consistency results of the algorithm Γ. The control on the expected length of the prediction interval E L( Γ)ℓ ≤ C s √ N given in Proposition 12 is also still valid since this result is distribution-free. On the other hand, consistency for the excess-risk requires conditions. In the case where Y is bounded, if the estimator of the conditional probabilities is such that E R |p(y|X)p(y|X)|dy -→ n→+∞ 0, we can establish under Assumptions 12, 13, 14, and 18 that

Essentially, this result says that the estimation procedure that we study in this chapter extends beyond the Gaussian setting. In particular, we still manage to get consistency for bounded random variable. It is worth mentioning that consistency might also be obtained as soon as Y |X is sub-Gaussian. Then our method is statistically valid for general settings.

Prediction interval under expected coverage constraint

In this section, we present the approach which focuses on the construction of prediction interval under expected coverage. This method consists in minimizing the length of the prediction interval under a constraint on its expected error rate. This approach is for instance studied in [START_REF] Lei | Distribution-free prediction bands for non-parametric regression[END_REF].

Chapitre 4: Prediction interval with fixed expected length in the Gaussian heteroscedastic regression Then for all y ∈ [-s, s], conditional on (D n , X, ζ), the probability in Eq. (4.16) is bounded as follows

These two last terms are treated in different ways. For the first one, we observe that for all t ∈ [0, 1]

We

Injecting this inequality to (4.17) gives

Chapitre 4: Prediction interval with fixed expected length in the Gaussian heteroscedastic regression Hence, if σ 2 (X) ≥ σ2 (X), since x → exp(-x) is 1-Lipschitz on R + , we deduce from the above inequality that

Therefore, from Assumption 12, and since σ2 (X) ≥ 1/s, we get in the case where σ 2 (X) ≥ σ2 (X)

In the case where σ 2 (X) ≤ σ2 (X), using same arguments and additionally the fact that σ2 (X) ≤ s, we can obtain

where we used the fact that the integral w.r.t. y is the variance of Gaussian r.v. with variance σ2 (X)

and is then such that R 1 We bound the first term in the r.h.s. in the above inequality. Now, we study the second term in the r.h.s. of Equation (4.29). We observe that if y ∈ Γ(X, ζ) \ Γ * ℓ (X) the following holds