Keywords: Design consistency, Elastic net, Lasso, Random Forest, Ridge regression, XGBoost Model-assisted approach, Model-calibration, Nonparametric regression, Random forest, Survey data, Variance estimation Additive models, Bayesian additive regression trees (BART), CART, Cubist algorithm, Ensemble Methods, Nearest Neighbor, Item nonresponse, Random forest, Support vector regression (SVR), Survey data, Statistical learning, Tree boosting Missing data, Imputation, Survey sampling, Regression trees, Random forests

Je souhaiterais en premier lieu exprimer ma profonde gratitude à mes directeurs de thèse, Camelia et David. Je ne pouvais rêver meilleure ouverture au monde de la recherche. J'espère que les trois ans passés à leurs côtés m'ont permis d'acquérir une part de leur rigueur, de leur intuition et de leur passion pour la statistique. Ils font partie des personnes les plus bienveillantes qu'il m'ait été donné de rencontrer; depuis le premier jour, ils n'ont cessé de me conseiller, de partager leur expérience, en bref, de faire tout leur possible pour que je puisse atteindre mes objectifs. À ce titre, je leur suis in niment reconnaissant, et j'espère que nous continuerons à collaborer pendant encore de nombreuses années. Merci pour tout.

Je tiens aussi à remercier Jean-Michel Poggi, Anne Ruiz-Gazen et Yves Tillé de l'honneur qu'ils me font d'avoir accepté d'être les rapporteurs de cette thèse. Je remercie tout particulièrement Yves, pour m'avoir si bien accueilli à Neuchâtel et pour les nombreuses discussions que nous avons pu avoir; je remercie aussi Anne, pour ses retours et ses conseils prodigués à l'occasion de mon comité de suivi de thèse et lors des di érentes conférences où l'on s'est rencontré.

Je tiens à exprimer ma reconaissance à Aurélie Vanheuverzwyn, tout d'abord pour avoir accepté de faire partie de ce jury, mais aussi pour avoir grandement soutenu et participé aux projets de cette thèse. Un grand merci aussi pour avoir organisé des échanges qui, bien que trop peu nombreux dû au contexte sanitaire actuel, m'ont permis de découvrir quelques-unes des problématiques concrètes auxquelles les statisticiens peuvent être confrontés.

En n, je remercie également Clément Dombry d'avoir accepté de faire partie de mon jury de thèse, ainsi que pour ses conseils et son soutien constant depuis le master. J'adresse mes remerciements à l'intégralité des membres du Laboratoire de Mathématiques de Besançon, pour m'avoir si bien accueilli, intégré, et pour m'avoir permis de travailler dans une ambiance si chaleureuse. Je pense en particulier à Charlène et Claudia qui m'ont très fréquemment aidé, conseillé et soutenu dans les di érentes démarches que j'ai eu à entreprendre.

Un grand merci aussi à l'ensemble des membres de l'équipe Probabilités et Statistiques. Je souhaite remercier tout particulièrement Yacouba, pour son soutien et ses conseils, ainsi que pour sa passion pour la recherche qui est communicative. Je tiens aussi à remercier Louis, pour ses nombreux conseils, ainsi que Christophe, pour son soutien et ses encouragements.

Bien sûr, un grand merci à tous mes collègues doctorants avec qui j'ai pu avoir la chance de partager un match de ping-pong, une partie d'échecs, ou tout simplement un café. Je pense tout particulièrement à Valentin, qui m'a suivi dans bon nombre de mes idées farfelues, et sans qui l'ambiance au bureau aurait été bien moins drôle; à Cécile, que j'ai probablement fatiguée plus d'une fois, mais qui, comme à son habitude, n'osait que très rarement le dire; et bien sûr, à Loris, Youssef, Benjamin, Yoël, Mathilde, Romain, Chifaa, Marsault, Audrey, Charles, Mathieu, et tous les autres. Mes derniers remerciements s'adressent à ma famille et à mes amis, pour leur soutien constant, et pour avoir grandement contribué à rendre ces trois dernières années particulièrement agréables. Merci à tous.

I N T R O D U C T I O N

One of the main goals of survey sampling is the estimation of nite population parameters. Common examples of these parameters of interest include population totals, population means or population proportions, among others. Auxiliary information is often available and can be incorporated in the estimation procedures to increase the precision of the resulting point estimators. If the sampled units respond to all items, model-assisted estimation [START_REF] Cassel | Some results on generalized di erence estimation and generalized regression estimation for nite populations[END_REF] and calibration [START_REF] Deville | Calibration estimators in survey sampling[END_REF] provide exible solutions for making use of auxiliary variables and construct e cient estimators of nite population totals. In presence of missing data, auxiliary information can also be used to reduce the undesirable e ects of nonresponse. Popular methodologies include imputation or re-weighting the sampled elements.

This work was born from an idea of collaboration with Médiamétrie1 , the French audience company. Initially, we were interested in developing and analyzing matrix completion algorithms with survey data for estimating several nite population totals (i.e., multipurpose surveys). This topic is particularly promising, both from a theoretical and a practical point of view. Unfortunately, the COVID-19 pandemic made it di cult to meet and collaborate with Médiamétrie as much as we would have desired and this thesis also explored other research areas. Despite the sanitary complications, these three years have allowed us to have many enriching discussions with Médiamétrie. In particular, Médiamétrie continually supported us in the new research directions that the thesis had taken and, in particular, provided us with databases that were useful to assess the performances of the new methodologies. The project of matrix completion in surveys is still ongoing; more details about it are provided in Chapter 6. I hope that our future works in that area will be a useful contribution for Médiamétrie and that our collaboration will continue long after the time of this thesis.

Médiamétrie often has to deal with a large number of covariates. More generally, it is nowadays rather common for survey statisticians to face high-dimensional data. Therefore, in this thesis, the problem of estimating nite population totals in presence of a large number of auxiliary variables is considered. The scenarios of full response and missing data are both examined. In case of full response, I investigated the properties of existing model-assisted estimators in a high-dimensional asymptotic framework and I suggested a new class of model-assisted estimators based on random forests. In presence of missing data, I studied the use of statistical learning predictors based on a large number of covariates for imputation as well as the theoretical properties of imputed estimators based on regression trees and random forests.

Model-assisted estimation with high-dimensional data

Since the pioneering work of [START_REF] Särndal | On pi-inverse weighting versus best linear unbiased weighting in probability sampling[END_REF], [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] and [START_REF] Särndal | Cosmetic form of estimators in survey sampling[END_REF], model-assisted estimation procedures have attracted a lot of attention in the literature; see also [START_REF] Särndal | Model assisted survey sampling[END_REF] for a comprehensive discussion of the model-assisted approach. The main idea of model-assisted estimation is to estimate the functional relationship between the survey variable and the set of covariates by means of a predictive model, and to incorporate its predictions in the de nition of the estimator. When the predictions are close to the true values, the precision of the resulting estimator might increase. Many predictive models have been suggested in the literature, from parametric models [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] and penalized linear models such as the lasso [START_REF] Mcconville | Model-assisted survey regression estimation with the lasso[END_REF] and ridge [START_REF] Goga | Overview of ridge regression estimators in survey sampling[END_REF], to non-parametrics such as local polynomials [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], B-splines [START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF][START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF], penalized splines (Breidt et al., 2005, McConville and[START_REF] Mcconville | Survey design asymptotics for the model-assisted penalised spline regression estimator[END_REF], neural networks [START_REF] Montanari | Nonparametric model calibration in survey sampling[END_REF], generalized additive models [START_REF] Opsomer | Model-assisted estimation of forest resources with generalized additive models[END_REF] and regression trees [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF].

Most of the aforementioned literature is gravitating around predictors which are especially e ective when the number of covariates is relatively low; however, some of them tend to be relatively ine cient when used with a large number of covariates, a phenomenon known as the curse of dimensionality [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]; see also [START_REF] Giraud | Introduction to high-dimensional statistics[END_REF], [START_REF] Györ | A distribution-free theory of nonparametric regression[END_REF] for textbook discussions. Nowadays, it is no longer unusual for survey statisticians to face scenarios where a large number of auxiliary variables are available at the estimation stage. For example, Médiamétrie uses every day a panel of 7000 individuals and their television consumption is recorded every second, thus leading to 86400 covariates (Cardot et al., 2013a). Other applications on Médiamétrie data can also be found for instance in [START_REF] Goga | Principal component regression with survey data. application on the french media audience[END_REF]. Similarly, in [START_REF] Cardot | Comparison of di erent sample designs and construction of con dence bands to estimate the mean of functional data: An illustration on electricity consumption[END_REF], the authors considered samples of size 1 500 from a real data set collected by the Irish Commission for Energy Regulation Project concerning the electricity consumption of households and companies recorded every 30 min (for example, there are more than 300 variables over a week). In these examples, the assumption that the number of covariates is negligible with respect to the sample size may not be satis ed. Recently, to better account for these high-dimensional scenarios, [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF], [START_REF] Chauvet | Asymptotic e ciency of the calibration estimator in a high-dimensional data setting[END_REF], [START_REF] Ta | Generalized regression estimators with high-dimensional covariates[END_REF] investigated the asymptotic properties of linear model-assisted estimators when the number of covariates was allowed to increase to in nity.

In this thesis, I analyzed the performances of linear and penalized model-assisted estimators in a high-dimensional asymptotic framework (Dagdoug et al., 2022a) and suggested in [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF] the use of model-assisted estimator based on random forest algorithms [START_REF] Breiman | Random forests[END_REF]. Random forest predictors seem to remain relatively e cient in high-dimensional settings and seem to adapt well to sparsity [START_REF] Biau | Analysis of a random forests model[END_REF][START_REF] Scornet | Consistency of random forests[END_REF].

In Dagdoug et al. (2022a), we examined several model-assisted estimators from a designbased point of view and in a high-dimensional setting, including linear regression and penalized estimators. The consistency of model-assisted estimators based on linear regression, on the lasso and on ridge was established under relatively weak assumptions. We conducted an extensive simulation study using data from the Irish Commission for Energy Regulation Smart Metering Project, to assess the sensitivity of model-assisted estimators based on several statistical learning predictors to high-dimensional auxiliary information.

In [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF], we use random forests to estimate the functional relationship between the survey variable and the auxiliary variables. Several important results have been developed in this work. First, under regularity assumptions on the sampling design, on the survey variable and the random forest algorithms, we showed the L 1 convergence and the asymptotic normality of the estimator. The asymptotic variance of the estimator was derived and an L 1 consistent variance estimator was suggested. A nice and new non-asymptotic property has also been shown: this estimator can be written according to the observations which did not participate in the prediction, observations called "out-of-bag" in the "machine learning" literature. The use of these units, which makes it possible to avoid over tting, is new in the context of estimation for survey data. A model-calibration procedure for handling multiple survey variables was also discussed. The results of a simulation study suggested that the proposed point and estimation procedures perform well in terms of bias, e ciency and coverage of normal-based con dence intervals, in a wide variety of settings. We also suggested a new variance estimator based on K-fold cross-validation method. This new variance estimator constitutes an important advance in the theory of modern survey practice because it corrects the defect of over tting of traditionally used variance estimators and thus makes it possible to construct reliable con dence intervals. This new variance estimation method is all the more important as it is completely general and can be used with other nonparametric estimation methods such as splines, local polynomials, etc. Finally, adaptations of random forest algorithms to di erent sampling plans (strati ed, proportional to size) were proposed and the method was tested on audience data from Médiamétrie. More precisely, we had access to 3 882 auxiliary variables and used a sample size of 4 000 observations; the goal was to estimate the proportion of French individuals who listen to a radio of interest on a daily basis, both at the overall population level and for several domains of interest.

Treatment of item nonresponse with high-dimensional data

Missing data are present in most censuses and sample surveys. Nonresponse is particularly undesirable as unadjusted estimators might be biased and exhibit a substantial increase of variance [START_REF] Rubin | Inference and missing data[END_REF]; as such, the use of unadjusted estimators should be avoided and nonresponse treated. Two cases are usually distinguished: unit nonresponse and item nonresponse. Unit nonresponse is de ned by a complete lack of information of a given element, while item nonresponse characterizes elements for which some information is collected, but not all. The treatment of unit nonresponse is beyond the scope of this thesis. Usually, item nonresponse is handled using imputation, a procedure which consists in replacing missing values with arti cial values. Most often, these arti cial values are obtained by means of a predictive model. The theoretical properties of imputed estimators were investigated for several non-parametric methods such as the nearest neighbor [START_REF] Chen | Nearest neighbor imputation for survey data[END_REF], 2001[START_REF] Yang | Nearest neighbor imputation for general parameter estimation in survey sampling[END_REF], the score method [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF]Beaumont, 2007, Little, 1986), predictive mean matching [START_REF] Yang | Predictive mean matching imputation in survey sampling[END_REF], kernel regression [START_REF] Zhong | Jackknife empirical likelihood inference with regression imputation and survey data[END_REF], to cite just a few. For a comprehensive review about the missing data literature in surveys, see Chen and Haziza (2019) or [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF]. In the same way as for model-assisted estimators, most of the existing literature is centered around predictors which are sensitive to the number of auxiliary variables; that is, they are particularly e cient in low-dimensional settings, but might be particularly sensitive to the curse of dimensionality. For instance, using arguments of [START_REF] Abadie | Large sample properties of matching estimators for average treatment e ects[END_REF], it is shown in [START_REF] Yang | Nearest neighbor imputation for general parameter estimation in survey sampling[END_REF] that the nearest neighbor imputed estimator has a non-negligible bias whenever the number of covariates is strictly greater than one.

In this thesis, I also investigated the performances of imputed estimators based on predictors known for their high-dimensional e ciency. First, in Dagdoug et al. (2021a), we conducted a large-scale simulation study in which we compared imputed estimators based on numerous statistical learning predictors commonly used by machine learning practitioners. In particular, we included the traditional linear regression, the score method, as well as support vector machines [START_REF] Cortes | Support-vector networks[END_REF]Vapnik, 1995, Smola and[START_REF] Smola | A tutorial on support vector regression[END_REF], k-nearest neighbors, regression trees [START_REF] Breiman | Classi cation and regression trees[END_REF], random forest [START_REF] Breiman | Random forests[END_REF], gradient boosting (Chen andGuestrin, 2016, Friedman, 2001), Bayesian additive models (BART, [START_REF] Chipman | BART: Bayesian additive regression trees[END_REF]), additive models with B-splines, Cubist [START_REF] Quinlan | Combining instance-based and model-based learning[END_REF], [START_REF] Quinlan | Learning with continuous classes[END_REF]. Various relationships between the survey variable and the covariates were considered, as well as several sampling designs and nonresponse models; high-dimensional scenarios were considered as well. We discovered that imputed estimators based on complex algorithms (e.g. Cubist, Boosting, Bayesian additive regression trees, Random forests) can often outperform traditional parametric models: when the parametric model was well speci ed, then the imputed estimator based on it was more e cient than imputed estimators based on nonparametric complex algorithms; however, in most cases, the loss of e ciency of nonparametric models versus parametric was relatively low. On the other hand, when the parametric model was misspeci ed, imputed estimators based on nonparametric models were much more e cient. Overall, we found that imputed estimators based on Cubist, XGBoost and BART were very e cient in most scenarios and substantially improved over parametric estimators.

In [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF], we investigated both theoretically and empirically the performances of regression tree and random forest imputed estimators. We gave a result exhibiting a set of conditions on the predictor used for imputation under which the imputed estimator based on this predictor is L 2 consistent with respect to the joint distribution induced by the model, the nonresponse mechanism and the sampling design. The conditions that we found revealed that, if the predicted values are based on a predictor consistent for the regression function and the L 2 prediction error is bounded, even for nite samples, then the resulting imputed estimator converges in L 2 towards the parameter of interest. Using this result, the L 2 -convergence of the CART tree imputed is obtained. The L 2 -convergence of the random forest imputed estimator is obtained using in nite forests as a tool. We suggested variance estimators which seemed to perform well (from a bias and coverage rate point of view) on simulation studies. An application to data integration was also considered.

Organization of the dissertation and list of publications

The rest of this thesis is organized as follows. Chapter 1 is a presentation of the basic concepts of survey sampling. These concepts are presented in a uni ed framework as in [START_REF] Rubin-Bleuer | On the two-phase framework for joint model and design-based inference[END_REF], [START_REF] Boistard | Functional central limit theorems for single-stage sampling designs[END_REF], and [START_REF] Han | Complex sampling designs: Uniform limit theorems and applications[END_REF]. The mathematical formalism employed in this framework is rather di erent from the usual presentations of survey sampling theory given in [START_REF] Särndal | Model assisted survey sampling[END_REF], [START_REF] Tillé | Sampling and estimation from nite populations[END_REF] or [START_REF] Lohr | Sampling: design and analysis[END_REF], but it has the advantage to include both design and model inferences commonly used in the model-assisted literature as well as the model-design-nonresponse inferences used in the imputation literature. Chapter 1 also describes brie y basic sampling designs, the usual Horvitz-Thompson estimator [START_REF] Horvitz | A generalization of sampling without replacement from a nite universe[END_REF] and summarizes the main contributions of this dissertation. The chapters 2 and 3 are devoted to model-assisted estimation in high-dimensional settings; Chapter 2 presents the article Dagdoug et al. (2022a) while Chapter 3 presents the article [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF]. The chapters 4 and 5 are concerned with imputation in presence of a large number of covariates; Chapter 4 presents the article Dagdoug et al. (2021a) while Chapter 5 presents the work in progress [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF]

I N T R O D U C T I O N T O S U R V E Y S A M P L I N G A N D R E V I E W O F T H E M A I N C O N T R I B U T I O N S

1.1 A basic introduction to survey sampling

Superpopulation framework

Let Y be a survey variable, representing a characteristic of interest that the survey statistician wishes to study. Consider that the random variable Y is de ned on a probability space (Ω, M, P m ) taking values in a measurable space (E, E). The distribution of Y will be denoted by P Y := P m • Y -1 . Let N ∈ N * be a positive integer. De ne N independent and identically distributed (i.i.d.) random variables {Y k } k=1,...,N , with common law P Y . Then, the probability space (Ω, M, P m ) and the N-random vector Y := [Y 1 , Y 2 , ..., Y N ] de ne what we call in the sequel a superpopulation model as described in [START_REF] Rubin-Bleuer | On the two-phase framework for joint model and design-based inference[END_REF] and denoted by (Ω, M, P m , Y ).

As stated in [START_REF] Shao | The jackknife and bootstrap[END_REF], "the basic objective of statistical analysis is extracting all the information from the data to deduce "properties" of the "population" that generated the data". This statement holds true for both classical statistics (sometimes referred to as inferential statistics in the literature) and survey statistics; yet, the terms "properties" and "population" have di erent meanings in these two scenarios. In the next paragraph, we highlight these di erences and describe both approaches to inference.

In classical statistics, the term "population" is used to denote the probability distribution P Y from which the data has been generated. It can be viewed as an in nite population, and in survey statistics, as a superpopulation.

In survey statistics, however, the term "population" refers to a nite population, that is, a set of nite cardinality, as opposed to the conceptual "in nite population" of classical statistics. The cardinality of a nite population will be referred to as its size. We thus now consider a nite population U of N labeled elements

U := {u 1 , u 2 , ..., u N } = {1, 2, ..., N } ,
where u k denotes the k-th element of the population of interest U, also called the target population. To each element u k of U, we generate a realization y k of Y k (i.e., Y k (ω) := y k ∈ E for some ω ∈ Ω) and we say that U is generated by the superpopulation model (Ω, M, P m , Y ).

For simplicity, in the sequel, we use {y k } k∈U to denote the random variables {Y k } k∈U .

In classical statistics, the "properties" referred to above, can often be de ned as functionals of the distribution P Y . To be more precise, let (F, F ) denote a measurable space and M 1 + (E) be the set of probability measures on E. An "in nite population parameter" or "superpopulation parameter" θ M can often be represented as the image of an unknown element of M 1 + (E)

1.1 through a known statistical functional T : M 1 + (E) → F, that is, θ M := T(P Y ), where P Y is unknown. For instance, a typical example is given by the mean (whenever appropriate) of a real-valued distribution P Y , in which case

θ M = T(P Y ) = ∫ R ydP Y .
In survey statistics, the "properties" that we wish to estimate are functions of the random variables {y k } k∈U . These functions are called nite population parameters in the sequel. Common examples include the population total t y , the population mean ȳU and population proportion p c of elements having a particular characteristic denoted "C" ; these are de ned as follows:

t y := k∈U y k , ȳU := 1 N k∈U y k , p c := 1 N k∈U 1 {u k has characteristic "C"} . (1.1)
More generally, a nite population parameter θ U is de ned as the image of an unknown point (y 1 , ..., y N ) of E N through a known measurable function H : E N → F, that is, θ U := H (y 1 , y 2 , ..., y N ).

Therefore, in both classical and survey statistics, the aim is to estimate the image of a known function computed at an unknown point. However, the techniques to do so are sometimes quite di erent in both elds. Moreover, an additional important di erence is that, in the model-based framework of survey statistics, the parameter of interest θ U is a random variable taking values in F, whereas the parameter of interest θ M of classical statistics is deterministic. Indeed, see that θ U is measurable from Ω to F with realizations given by θ U (ω) = H (y 1 (ω), ..., y N (ω)) , ω ∈ Ω.

Therefore, di erent realizations of the superpopulation model might lead to di erent values of the nite population parameters of interest.

Most often, as in the examples given in (1.1), the nite population parameters considered for practical purposes are real-valued and the survey variable is real-valued as well. In some cases, however, the spaces E and F might not be the set of real numbers. For instance, in Cardot et al. (2013c), the authors considered the case of functional data in which E = F ([0; T], R), the set of functions from [0; T] to R, for some T ∈ R * + . Similarly, statisticians might be interested in estimating more complex parameters such as the population distribution function

F N (x) = 1 N k∈U 1 y k ∈]-∞;x] ,
in which case F = D (R, [0; 1]), the Skorokhod space of cadlag functions with values in [0; 1].

In this work, we solely focus on the estimation of the total of real-valued random variables, so that E = R and F = R. Indeed, since many population parameters can be expressed as functions of totals, the population total is an important population parameter. In the sequel, the aim is therefore to estimate the population total t y as de ned in (1.1).

Probability sampling

In order to estimate the unknown parameter t y , two approaches may be considered: a census or a sample survey. A census consists of collecting the values of the variable of interest for all population elements, in which case the true value of t y is known. However, due to practical constraints (e.g., cost, time, ...), it is usually impossible to have access to all population elements; a survey is used instead. A survey consists of selecting a subset s of the population, called a sample. An estimator based on the information gathered from the sampled elements is then used to infer on the parameter of interest. Throughout this work, several assumptions are made: 1) there is no coverage error 1 ; 2) the sample s is selected according to a probability sampling design; 3) no measurement error 2 is present in the data collected. Moreover, in Chapter 3 and Chapter 4, we consider the idealistic scenario of full response. A framework for handling nonresponse is described in Section 1.3. We also restrict ourselves to the case of sampling without replacement, that is, we assume that population elements cannot be selected more than once in the samples.

De nition 1.1.1. (Sampling design. [START_REF] Särndal | Model assisted survey sampling[END_REF]) Let S be the collection of subsets of U. A sampling design is a probability mass function p : S → [0; 1] satisfying s∈S p(s) = 1.

(1.2)

For each sampling design, we can de ne a probability measure on the design space, called a design probability.

De nition 1.1.2. (Design probability.) Let D be a sigma-algebra on S. The measurable space (S, D) is often called the design space. A design probability P p is a probability measure de ned on the design space (S, D).

The measure

P p := s∈S p(s)δ s is a design probability associated to the sampling design p, where δ s denotes the Dirac measure at s ∈ S. Each design probability can be characterized by a sampling design. Note that the sample s selected from U can be viewed as the realization of a random sample S, de ned on some probability space (Ω S , A S , P S ) taking values in the design space (S, D) such that P p = P S • S -1 . We summarize these ideas with a diagram illustrating the di erences of classical and survey statistics, as well as sample selection, see Figure 1. The size of the sample S is denoted by n S . A sampling design for which n s = n for all samples s ∈ S having a non-null probability of being selected is said to be of xed size.

It is often convenient to represent samples without replacement as vectors in {0; 1} N . We de ne the concept of sample membership indicators.

De nition 1.1.3 (Sample membership indicators).

For all k ∈ U, we de ne the sampling membership indicator I k (S) := 1 {k∈S} , 1 A coverage error happens when the speci cation of sampling units in the population from which a sample was selected does not match the target population, see [START_REF] Lohr | Sampling: design and analysis[END_REF] for details. 2 A measurement error is a situation in which a response in the survey di ers from the true value.

Generation of a population

Selection of a sample

Figure 1: Diagram illustrating the di erences between classical statistics and survey statistics. In the sample, y (k) denotes the k-th element selected in the sample, where the order is arbitrary.

which takes the value 1 if the element k is selected in a sample and 0 otherwise. For simplicity, we write I k for I k (S).

Since the map

φ : S -→ {0; 1} N , s -→ [I 1 , I 2 , ..., I N ] .
is a bijection from S to {0; 1} N , it follows that each sample in S can be uniquely represented as a vector of {0; 1} N . Some authors (e.g., [START_REF] Bonnéry | Propriétés asymptotiques de la distribution d'un échantillon dans le cas d'un plan de sondage informatif[END_REF], [START_REF] Conti | Resampling under complex sampling designs: Roots, development and the way forward[END_REF]) de ne sampling designs on {0; 1} N rather than on S; the two formulations are indeed equivalent.

The random variables {I k } k∈U are, by construction, de ned in the design space and follow a Bernoulli distribution {B(π k )} k∈U , each, respectively, where π k denotes the rst-order inclusion probability of element k, a notion de ned below. Note that {I k } k∈U are not necessarily independent nor identically distributed.

De nition 1.1.4 (First and second order inclusion probabilities). The rst-order inclusion probability π k of element k is the probability that element k is selected in a sample; it is de ned as:

π k := P S (k ∈ S) = P p ({k}) = s∈S: k∈s p(s), k ∈ U.
Similarly, the second-order inclusion probability π k of elements k and is the probability that elements k and are both selected in a sample; it is de ned by:

π k := P S ((k, ) ∈ S × S) = P p ({k, }) = s∈S: k, ∈s p(s), k, ∈ U.
Throughout this dissertation, we assume that, for all k ∈ U, π k > 0 and for all pairs (k, ) ∈ U × U, π k > 0. The design covariance Cov p (I k , I ) is denoted by

∆ k := Cov p (I k , I ) = π k, -π k π .
To infer on the parameter θ U based on a survey sample, the rst step is to use a sampling design to select a sample. The second step is to use an estimator θ of θ U and compute an estimate based on the data collected from the sample.

De nition 1.1.5 (Sampling strategy).

A sampling strategy is de ned as a pair p, θ which produces an estimate of the parameter of interest θ U according to a sampling design p.

A strategy (p, θ) is e cient if the estimator θ is precise under the sampling design p, namely if it has low variance computed with respect to p. The Horvitz-Thompson estimator [START_REF] Horvitz | A generalization of sampling without replacement from a nite universe[END_REF]Thompson, 1952, Narain, 1951) t π of t y is of particular interest.

De nition 1.1.6 [START_REF] Narain | On sampling without replacement with varying probabilities[END_REF], [START_REF] Horvitz | A generalization of sampling without replacement from a nite universe[END_REF]). The Horvitz-Thompson estimator of the population total t y is de ned as:

t π := k∈S y k π k = k∈S d k y k , (1.3)
where d k := π -1 k , denotes the sampling weight of element k ∈ U. It is also often referred to as the π-estimator or the expansion estimator.

The set {d k } k∈S is often referred to as the basic weighting system and can be interpreted as follows: if element k of U is selected in the sample, then it represents d k elements of the population. Next proposition displays the design properties of t π .

Proposition 1.1.1 [START_REF] Särndal | Model assisted survey sampling[END_REF]). . Let p be an arbitrary sampling design. The Horvitz-Thompson estimator t π has the following design properties. i) Provided that π k > 0 for all k ∈ U, the estimator t π is design-unbiased, that is, E p t π = t y , where E p [•] denotes the expectation with respect to the sampling design.

ii) The design variance of t π is given by

V p t π = k∈U l∈U ∆ kl y k π k y l π ,
where V p (•) denotes the expectation with respect to the sampling design.

iii) Provided that π k > 0 for all (k, ) ∈ U × U, the variance estimator

V π := k∈S l∈S ∆ k π k y k π k y π (1.4)
is design-unbiased for V p t π .

From a design-based point of view, the quantities {y k } k∈U in the expression of t π are not random, the only random quantities are the sampling membership indicators {I k } k∈U .

The unbiasedness property of the Horvitz-Thompson estimator is particularly attractive. Using arguments of [START_REF] Godambe | A uni ed theory of sampling from nite populations[END_REF], it can be shown that the Horvitz-Thomspon estimator is the only unbiased homogeneous linear 3 estimator of t y with weights independent of the sample.

Sampling without auxiliary information

As mentioned in the previous sections, two sources of randomness may be considered: the randomness due to the superpopulation model and the randomness due to the sampling design. Assuming that the sampling design and the survey variable are independent, functions of both sources of randomness might be considered using a product space representation with a product measure. This assumption is realistic if no auxiliary information 4 is available at the sampling stage. A more complex joint model-design probability will be de ned in Section 1.1.4. can be included in this representation, that is, sampling designs which do not use any additional information. The descriptions given below are only brief, we refer the reader to the references [START_REF] Särndal | Methods for estimating the precision of survey estimates when imputation has been used[END_REF], [START_REF] Thompson | Theory of sample surveys[END_REF], [START_REF] Fuller | Sampling statistics[END_REF] and [START_REF] Tillé | Sampling and estimation from nite populations[END_REF] for additional details.

Simple random sampling without replacement

Let n be the desired sample size. Simple random sampling without replacement (SRSWOR) of size n is the design which assigns the same probability to all without replacement samples of size equal to n and zero otherwise. That is, for s ∈ S,

p (s) =              N n -1 if s is of size n, 0 otherwise.
The rst-order inclusion probabilities are equal to π k = n/N for all k ∈ U and the second-order inclusion probabilities are equal to

π k = n (n -1) N (N -1) ,
for all k ∈ U. It also follows that the π-estimator of t y is given by

t π = N n k∈S y k .
The implementation of simple random sampling without replacement is fairly easy [START_REF] Fan | Development of sampling plans by using sequential (item by item) selection techniques and digital computers[END_REF]; however, the sampling strategy SRSWOR, t π might lead to a large variance for the Horvitz-Thompson estimator; this phenomenon might happen when the population variability of the survey variable is large. This strategy may be improved if additional auxiliary information is used at the sampling stage, for instance with strati ed or proportional to size sampling designs (described thereafter). As explained in [START_REF] Tillé | Sampling and estimation from nite populations[END_REF], "the use of simple random sampling is a way of selecting a sample without preconceptions on the studied population. However, if auxiliary information is known and the variable of interest Y is suspected of being linked to this auxiliary information, then it will be more interesting to use a design that integrates this auxiliary information".

Bernoulli sampling

Bernoulli sampling is another sampling design which gives equal inclusion probabilities to all units of the population. More precisely, let π ∈ [0; 1]. Bernoulli sampling design is the design for which the sample membership indicators {I k } k∈U are independent and identically distributed random variables, each I k follows a Bernoulli distribution of parameter π. The probability assigned to a sample s by a Bernoulli sampling is given by

p (s) = π n s (1 -π) 1-n s , s ∈ S,
where n s is the size of the sample s. Its implementation is simple as it su ces to draw N independent realizations of a random variable with uniform distribution U (]0; 1[) and select elements for which the realizations lie in the interval ]0; π[. The rst order inclusion probabilities are all equal to π k = π, for all k ∈ U and, we have π kl = π 2 , for all k l ∈ U. The π-estimator is then given by t π = π -1 k∈S y k .

For a Bernoulli design, the sample size n s = k∈U I k is random and follows a Binomial distribution B (N, π). The random size is an important drawback since it is impossible to know in advance the cost of the survey and the Horvitz-Thompson estimator tends to be ine cient due to the random sample size [START_REF] Särndal | Model assisted survey sampling[END_REF]. To remedy the issue of the random size, a natural idea would be to consider the conditional sampling design p (• |n s = n). This design is in fact a simple random sampling design of size n as described before; a proof can be found in [START_REF] Tillé | Sampling and estimation from nite populations[END_REF].

The parameter π needs to be chosen in order to implement the Bernoulli sampling. One can choose for example π such that the expected sample size is equal to n, the desired sample size, namely:

E p [n s ] = N π = n, (1.5)
which gives π = n/N.

Sampling with additional auxiliary information

In some cases, additional auxiliary information is known prior to sampling, either for all population elements or in a aggregated form. On the probability space (Ω, M, P m ), we de ne a generic random vector z : Ω → R q . The distribution of z is denoted by P z := P m • z -1 and we de ne N i.i.d. copies z 1 , z 2 , ..., z N on (Ω, M, P m ) with distribution P z . The superpopulation model then becomes (Ω, M, P m , Y , Z U ) , where Z U := [z 1 , z 2 , ..., z N ] ∈ R N×q . We assume that, conditionally on the auxiliary variables Z U , the sampling design is independent of the survey variable. In that case, the sampling design is said to be noninformative [START_REF] Pfe Ermann | The role of sampling weights when modeling survey data[END_REF].

On the other hand, if conditional independence is not satis ed, the sampling design is said to be informative, as considered in Pfe ermann and [START_REF] Pfe Ermann | Inference under informative sampling[END_REF] or [START_REF] Bonnéry | Propriétés asymptotiques de la distribution d'un échantillon dans le cas d'un plan de sondage informatif[END_REF], among others.

De nition 1.1.8. (Sampling design with auxiliary information.) Consider the design space (S, D). A sampling design p using Z U as auxiliary information is a function p : S × R N×q → [0, 1], such that i) for all s ∈ S, the map Z U → p (s, Z U ) is measurable.

ii) for all Z U ∈ R N×q , the map s → p (s, Z U ) is a sampling design.

Note that De nition 1.1.1 is a special case of De nition 1.1.8, by taking sampling designs such that p (s, Z U ) = p (s), for all s ∈ S and Z U ∈ R N×q . To each ω ∈ Ω, we can de ne a design probability P p as P p (•, ω) := s∈S p(s, Z U (ω))δ s (•).

1.1

Note that a sampling design with auxiliary information p(•, Z U ) is a random variable in (Ω, M, P m ), as for di erent realizations of Z U , we might obtain di erent probability measures on (S, D). The joint distribution of the sampling design and the superpopulation model cannot be described anymore as a product measure on the product space; we adopt the construction detailed for instance in [START_REF] Rubin | Inference and missing data[END_REF], [START_REF] Boistard | Approximation of rejective sampling inclusion probabilities and application to high order correlations[END_REF] and [START_REF] Han | Complex sampling designs: Uniform limit theorems and applications[END_REF].

De nition 1.1.9 (Joint model-design distribution with auxiliary information).

Consider a superpopulation probability space (Ω, M, P m ) and a design probability space S, D, P p . The joint model-design probability induced by a sampling design with auxiliary information is de ned as the probability P mz,p on the measurable space (Ω × S, M ⊗ D), uniquely de ned as

P mz,p (A × B) := ∫ A P p (B, ω) dP m (ω)
for all measurable rectangles A × B ∈ M × D.

Sample membership indicators and inclusion probabilities are de ned similarly as in the previous section, but are random variables in the product space. As such, we should write π k (Z U ) for the rst order inclusion probability of element k; for simplicity, however, we omit the dependence of Z U in the notations of the inclusion probabilities.

Poisson sampling

Bernoulli sampling has been de ned as a sampling design for which the sample membership indicators {I k } k∈U are i.i.d. . Poisson sampling generalizes Bernoulli sampling, by inducing sample membership indicators {I k } k∈U that are independent but no longer identically distributed; they are now Bernoulli variables with parameter π k ∈]0; 1[ (possibly dependent of Z U ), which are not necessarily the same for all k ∈ U. The sampling design p satis es

p (s, Z U ) = k∈s π k k∈U-s (1 -π k ) , s ∈ S.
As in Bernoulli sampling, the sample size n s is random. The inclusion probabilities are {π k } k∈U , speci ed by the statistician, and π k = π k π for all k ∈ U (by independence of the sample membership indicators).

The quantities {π k } k∈U need to be determined in order to implement the Poisson sampling. In Bernoulli sampling, the inclusion probability π can be chosen so that the expected sample size E p [n s ] matches the desired sample size. In Poisson sampling, the relation (1.5) is insu cient to determine {π k } k∈U . A natural criterion would be to minimize the sampling variance of the Horvitz-Thompson estimator under the constraint of a xed expected sample size:

         min [π 1 ,...,π N ] ∈ R N V p t π = min [π 1 ,...,π N ] ∈ R N k∈U 1 π k -1 y 2 k , s.t E p [n s ] = n.
(1.6)

1.1

This problem is easily solved either by using the Lagrangian or with the Cauchy-Schwartz inequality, a proof can be found in [START_REF] Särndal | Model assisted survey sampling[END_REF]. One nds

π k = ny k k∈U y k , k ∈ U.
(1.7)

It is impossible to de ne such inclusion probabilities since the values {y k } k∈U of the survey variable are unknown prior to sampling. Let Z be an auxiliary variable positively correlated to the survey variable Y and known for all k ∈ U, prior to sampling. We can de ne

π k = nz k k∈U z k , k ∈ U, (1.8)
that is, replacing the unknown values {y k } k∈U in (1.7) by {z k } k∈U . The inclusion probabilities (1.8) may be greater than one; let A be the set of n A elements such that nz k > k∈U z k . Then, we

set π k = 1 for all k ∈ A and π k = (n -n A ) z k k∈U-A z k for all π k ∈ U such that k A.
Designs that satisfy (1.8) are called probability proportional to size designs [START_REF] Särndal | Model assisted survey sampling[END_REF]. A Poisson sampling design with inclusion probabilities satisfying (1.8) will be almost as e cient as the strategy solving (1.6). However, the drawback of having a random sample size still remains. It was then suggested to consider xed size designs without replacement satisfying (1.8), referred to as πps-sampling designs. However, for such designs, it is di cult to compute the second-order inclusion probabilities which are required for estimating the variance of the Horvitz-Thompson estimator. For some probability proportional to size designs5 , [START_REF] Hájek | Asymptotic theory of rejective sampling with varying probabilities from a nite population[END_REF] suggested approximating the second order inclusion probabilities as a function of the rst-order inclusion probabilities. Moreover, recent advances in the eld provide exible algorithms to implement designs satisfying (1.8); in particular, we mention the cube method by [START_REF] Deville | E cient balanced sampling: the cube method[END_REF], widely used by national survey o ces. Its description is beyond the scope of this work, see [START_REF] Deville | E cient balanced sampling: the cube method[END_REF], [START_REF] Berger | Sampling with unequal probabilities[END_REF] or [START_REF] Tillé | Ten years of balanced sampling with the cube method: an appraisal[END_REF] for details.

Strati ed sampling

Let P = {U 1 , U 2 , ..., U H } be a partition of U, where the H elements of P are called strata. Let N h denote the number of elements of U h , so that N = N 1 + N 2 + ... + N H . From each stratum U h , we select independently a sample S h of n h elements according to a sampling design p h . The nal sample S of size n = H h=1 n h is:

S = S 1 ∪ S 2 ∪ ... ∪ S H .
The overall sampling design is de ned as

p (S, Z U ) = p 1 (S 1 , Z U ) × p 2 (S 2 , Z U ) × ... × p H (S h , Z U ) .
Since the strata form a partition of U, the population total t y may be written as :

t y = k∈U y k = H h=1 k∈U h y k = H h=1 t yh ,
where t yh := k∈U h y k . The rst and second-order inclusion probabilities with respect to the sampling design p h (•, Z U ) are de ned as follows:

π h k = P p,h ({k}, Z U ) , k ∈ U h , π h kl = P p,h ({k, }, Z U ) , k l ∈ U h , for h = 1, 2, .
.., H. The rst and second order inclusion probabilities with respect to p (•, Z) are given by:

π k = π h k , if k ∈ U h , π kl =          π h kl if k l ∈ U h , π h k π h l if k ∈ U h , l ∈ U h and h h ∈ {1, ..., H}.
Using these inclusion probabilities, we can de ne the π-estimator of t y as

t π = H h=1 t h ,
where t h := k∈S h y k /π h k is the Horvitz-Thompson estimator of t yh . Because the selections in di erent strata are independent, the design variance of t π is given by

V p t π = H h=1 V p t h .
The overall variance of the Horvitz-Thompson estimator is thus dependent of the variance of each of the H Horvitz-Thompson estimators, within each stratum. Therefore, the choice of the strati cation variable is important for the e ciency of the overall strategy. The choice of the sample size within each stratum is of great importance as well. The larger n h is, the better the inferences will be for U h due to a reduced variance, but increasing n h also increases the cost of the survey. It is therefore of practical interest to search for sample sizes {n h } h=1,...,H that best control the balance between precision and cost, a problem introduced by [START_REF] Neyman | On the two di erent aspects of the representative method: the method of strati ed sampling and the method of purposive selection[END_REF] and known as optimal sample allocation. The optimal sample size allocation, at a xed cost equal for all strata is given by

n h = n × N h S y Uh H h=1 N h S Y Uh , (1.9)
for h = 1, 2, ..., H, where S y Uh := (N h -1) -1 k∈U h (y k -ȳUh ) 2 1/2 is the standard deviation of the variable of interest Y within the population stratum. Naturally, we do not have this information; hence, if a variable Z, available to us, is correlated with the variable of interest Y , we de ne the Z-optimal allocation:

n h = n × N h S z Uh H h=1 N h S z Uh , for h = 1, 2, ..., H, where S zU h := (N h -1) -1 k∈U h z k -zU h 2 1/2
, the standard deviation of the variable Z within the stratum h.

If no auxiliary information other than the strati cation variable is known other than the strati cation variable, a popular strategy is to use proportional allocation, for which the sample sizes are given by

n h = n × N h N for h = 1, 2, .
.., H. It follows that the sampling fraction is the same for each stratum. If S Y U h = S 2 for all h = 1, ..., H, then proportional allocation is optimal.

Asymptotic theory in survey sampling

An important part of theoretical statistics is concerned with the study of the asymptotic theory, sometimes also called large sample theory. Indeed, it is often di cult to obtain information about the behavior of an estimator for a given xed sample size. Usually, more information can be deduced about statistics for large samples, i.e., when taking the limit of the sample size to in nity. The rationale behind the procedure is that the conclusions that we may establish in the limit might still hold (approximately) in practice for "large enough" sample sizes. A large part of this dissertation is devoted to investigate the asymptotic properties of survey estimators. However, de ning a formal asymptotic framework in survey sampling is not as straightforward as it is in most areas of statistics. The main di erence here is that the random sample S takes its values in the subsets of the nite population U. As such, it is not possible to let n, the sample size, increase to in nity if U is xed. We adopt the asymptotic framework developed in [START_REF] Isaki | Survey design under the regression superpopulation model[END_REF], which we describe below.

We start by considering a sequence {N v } v∈N , strictly increasing to in nity. Next, we consider a sequence of embedded nite populations {U v } v∈N of size {N v } v∈N , each generated by the superpopulation model. In each nite population U v , a sample S v of size n v is selected according to a sampling design p v (•, Z U v ). While the nite populations are assumed to be embedded, we do not require this property to hold for the samples {S v } v∈N . This asymptotic framework assumes that v goes to in nity, so that both the nite population sizes and the sample sizes go to in nity. To improve readability, we will use the subscript v only in the quantities U v , N v and n v ; quantities such as π k,v shall be denoted simply as π k .

This framework has been used to establish the asymptotic properties of various estimators in survey statistics. The asymptotic properties of the Horvitz-Thompson estimator are of particular 1.2 interest since many estimators can be described as functions of Horvitz-Thompson estimators. Conditions for the design-consistency 6 of the Horvitz-Thompson estimators were investigated by [START_REF] Isaki | Survey design under the regression superpopulation model[END_REF]. The L 2 design and joint consistency 7 of the Horvitz-Thompson estimator were established by various authors, e.g. [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF], [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF]. Central limit theorems 8 were obtained by various authors [START_REF] Madow | On the limiting distributions of estimates based on samples from nite universes[END_REF], [START_REF] Erdos | On the central limit theorem for samples from a nite population[END_REF], [START_REF] Hájek | Asymptotic theory of rejective sampling with varying probabilities from a nite population[END_REF], [START_REF] Krewski | Inference from strati ed samples: properties of the linearization, jackknife and balanced repeated replication methods[END_REF] and [START_REF] Bickel | Asymptotic normality and the bootstrap in strati ed sampling[END_REF], for particular sampling designs only. A review of some of these results can be found in [START_REF] Fuller | Sampling statistics[END_REF]. A joint model-design framework was introduced by [START_REF] Rubin-Bleuer | On the two-phase framework for joint model and design-based inference[END_REF] in which central limit theorems and convergence results were obtained for survey estimators. More recently, several authors [START_REF] Bertail | Empirical processes in survey sampling[END_REF], [START_REF] Boistard | Functional central limit theorems for single-stage sampling designs[END_REF], [START_REF] Han | Complex sampling designs: Uniform limit theorems and applications[END_REF] considered the problem of establishing limit theorems for survey empirical processes; a thorough review on the subject is given in [START_REF] Han | Complex sampling designs: Uniform limit theorems and applications[END_REF]. The variance estimator of the Horvitz-Thompson estimator given in (1.4) has been shown to be L 1 -consistent as well, see [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] or [START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF] for proofs.

Use of auxiliary information at the estimation stage 1.2.1 Model-assisted estimation in surveys

The previous section described several strategies for which auxiliary variables were incorporated in the sampling design, with the Horvitz-Thompson estimator. In this section, we describe how auxiliary information can also be used at the estimation stage to build more e cient estimators of t y .

We place ourselves in the superpopulation model as before, in which we de ne an additional generic random vector x ∈ R p on (Ω, M, P m ) and denote by P x its distribution. For simplicity, we assume that the support of P x belongs to the unit hypercube [0; 1] p . For simplicity of exposition, we further assume that the survey variable is compactly supported in an interval C 1,Y ; C 2,Y . We de ne a collection of covariates x 1 , x 2 , ..., x N i.i.d. with distribution P x and concatenated in a matrix X U := [x 1 , x 2 , ..., x N ] . Without loss of generality, we assume that the random vectors {x k } k∈U are to be used as auxiliary variables at the estimation stage, with techniques detailed in Section 1.3, while the random vectors {z k } k∈U are designed to be used at

A sequence of estimators { θ v } v ∈N of θ U,v is said to be design probability consistent if, for all > 0, lim v→∞ P p N -1 v | θ v -θ U,v | > = 0. Let 1 d < ∞. A sequence of estimators { θ v } v ∈N of θ U,v is said to be design L d consistent if lim v→∞ E p N -d v | θ v -θ U,v | d = 0, a.s.
The sequence is said to be L d consistent (for the joint distribution) if

lim v→∞ E N -d v | θ v -θ U,v | d = 0. A central limit theorem for { θ v } v ∈N estimators of θ U,v is a result formalizing the existence of a normalizing sequence {α v } v ∈N and a real σ such that such that α v → ∞ and α v N -1 v θ v -θ U,v L → N 0, σ 2 .
1.2 the sampling stage, as described in the previous section. Throughout the following sections, the aim is to build e cient strategies p, t for the estimation of t y . We assume that the sampling design p is already determined and the sample data already collected; hence, the e ciency of p, t is measured by the mean squared error of the estimator t computed with respect to the sampling design p. At our disposal, we have the information contained in the set

D ma := {(x k , y k ); k ∈ S} {x k ; k ∈ U\S}.
An important part of this work relies on the fact that the problem of estimating nite population totals is closely related to a prediction problem. Indeed, let f : R p → R be an arbitrary function and de ne the generalized di erence estimator of t y [START_REF] Cassel | Some results on generalized di erence estimation and generalized regression estimation for nite populations[END_REF] based on f as follows

t gd ( f ) := k∈U f (x k ) + k∈S y k -f (x k ) π k .
If f is a function independent of the sample S, the estimator t gd ( f ) is design-unbiased, namely, E p t gd ( f )t y = 0 and its design-variance is given by

V p t gd ( f ) = k∈U ∈U ∆ k y k -f (x k ) π k y -f (x ) π .
The design-mean square error (which is equal, in that case, to the design-variance) of the di erence estimator is therefore linked to the choice of the function f and to the quality of the covariates for predicting Y . If we assume that the survey variable can be represented as a function of the covariates as follows

y k = m(x k ), ∀k ∈ U, (1.10)
for a function m : R p → R, then it follows that

V p t gd (m) = 0.
In that case, the di erence estimator is an optimal estimator for t y in the sense that its design mean square error has the smallest value. We see therefore that the quality of the strategy (p, t gd ( f )) is closely related to the quality of the function f chosen in terms of predictor of Y , that is, how close are the quantities f (x k ) from y k , for all population elements k. In practice, Assumption (1.10) is often too simplistic; it would be more realistic to assume that

y k = m(x k ) + k , ∀k ∈ U, (1.11) with k denoting a sequence of i.i.d. random variables such that E [ k |x k ] = 0 and V ( k |x k ) = σ 2 .
The function m is often called the regression function. Under Model (1.11), the di erence estimator is expected to be an e cient estimator of t y ; for instance, [START_REF] Cassel | Some results on generalized di erence estimation and generalized regression estimation for nite populations[END_REF] proved that, t gd (m) is optimal in the sense that it minimizes the mean squared error with respect to the joint distribution among the class of design unbiased estimators of t y [START_REF] Cassel | Some results on generalized di erence estimation and generalized regression estimation for nite populations[END_REF].

1.2

In most cases, the di erence estimator based on m cannot be used as the regression function m is unknown. The idea of model-assisted estimation is to estimate the unknown function m by a regression function estimator (also called a predictor) m, tted on {(x k , y k ) ; k ∈ S}. The tted model is then used to construct the model-assisted estimator of t y [START_REF] Särndal | Model assisted survey sampling[END_REF]:

t ma = k∈U m(x k ) + k∈S y k -m(x k ) π k , (1.12)
where m(x k ) denotes the prediction of m at the point x k . The model-assisted estimator is therefore de ned as the population total of the estimated predictions { m(x k )} k∈U and the weighted sum of the sampled units of the estimated residuals {y km(x k )} k∈S . If the predictions { m(x k )} k∈S are close to the true values {y k } k∈S , then the rst term will dominate; if the predictions { m(x k )} k∈U are ine cient, however, the second term will counterbalance the ine ciency of the rst term and thus adds robustness to the estimator.

Whenever the predictor m is sample dependent, the estimator t ma ( m) is no longer design-unbiased and its properties do not follow from the properties of the di erence estimator. The properties of (1.12) has been investigated for many predictors, from parametric models [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF], to non-parametrics such as local polynomials [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], B-splines [START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF] and [START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF], penalized splines (Breidt et al., 2005, McConville and[START_REF] Mcconville | Survey design asymptotics for the model-assisted penalised spline regression estimator[END_REF], neural networks [START_REF] Montanari | Nonparametric model calibration in survey sampling[END_REF], generalized additive models [START_REF] Opsomer | Model-assisted estimation of forest resources with generalized additive models[END_REF] and regression trees [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF]. In the aforementioned articles, the design properties (asymptotic design unbiaseness and consistency) of these estimators were established. The conclusions made about model-assisted estimator are therefore independent of the quality of the model predictions. As explained in [START_REF] Särndal | Model assisted survey sampling[END_REF], the approach is model-based, but not model-dependent. Several authors, however, also investigated the joint design-model properties of these estimators, including [START_REF] Särndal | On pi-inverse weighting versus best linear unbiased weighting in probability sampling[END_REF], [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF], [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] and [START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF].

In most theoretical investigations of model-assisted estimators, the asymptotic properties were established in an asymptotic framework in which the number of covariates included in the model was kept xed, thus implying that the ratio p/n is negligible. However, nowadays, it is no longer unusual to face high-dimensional auxiliary information. These practical scenarios are therefore not included in the scenario usually considered because the assumption that p/n is negligible may not be satis ed. The behavior of these estimators in such scenarios was therefore, until recently, unknown. Recently, increasing attention has been devoted to establishing asymptotics properties of model-assisted estimators in a framework in which the number of covariates is increasing to in nity as well. This framework is called high-dimensional; the asymptotic results established in this framework include practical situations in which both p = p v9 and n v are large. Often, they can be seen as generalizations of the results obtained in a low-dimensional framework. In particular, [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF] studied dimension reduction through principal component analysis and established the design consistency of the resulting estimator in a setting in which the number of principal components is allowed to increase. More recently, [START_REF] Ta | Generalized regression estimators with high-dimensional covariates[END_REF] investigated the properties of model-assisted estimators based on linear regression and the Lasso, under the joint model-design distribution. [START_REF] Chauvet | Asymptotic e ciency of the calibration estimator in a high-dimensional data setting[END_REF] studied the design asymptotic properties of calibration estimators 10 , when the number p v of calibration variables is going to in nity.

In this dissertation, two articles focus on model-assisted estimation. First, we present the article Dagdoug et al. (2022a) entitled Model-assisted estimation in high-dimensional settings for survey data which investigates the asymptotic design behavior of linear and penalized linear model-assisted estimators in a high-dimensional framework. The complete article is presented in Chapter 3. Next, the article [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF] entitled Model-assisted estimation through random forests in nite population sampling deals with model-assisted estimators based on random forests. The nite sample properties of the resulting estimator were thoroughly investigated and its asymptotic properties were established (L 1 -consistency, determination of the asymptotic variance, suggestion of a L 1 -consistent variance estimator, asymptotic distribution). The complete article is provided in Chapter 4.

Model-assisted estimation in high-dimensional settings for survey data

For simplicity of exposure, we describe the predictors as if the available data was the population data D U := {(x k , y k ) ; k ∈ U}. Extension to their de nitions at the sample level will be detailed subsequently.

A description of linear and penalized linear models

In practice, it is often common and convenient for practitioners to assume that the regression function in (1.11) is a linear function of the covariates, that is, that there exists β in R p such that m(x) = x β. The unknown vector β can be estimated by β lr through the ordinary least square criterion at the population level:

β lr = arg min β∈R p k∈U (y k -x k β) 2 .
(1.13)

Provided that the matrix X U is of full rank, the solution of (1.13) is unique, a closed-form solution exists and is given by:

β lr = k∈U x k x k -1 k∈U x k y k .
The prediction at x k based on β lr is given by m(x k ) := x k β lr , for all k ∈ U.

10 Calibration is a procedure suggested by [START_REF] Deville | Calibration estimators in survey sampling[END_REF], widely used in practice. The main idea is to search for weights {w k } k ∈S , which are as close as possible to the original weighting system {d k } k ∈S , from a distance point of view, while satisfying the constraint that the w-weighted estimator of the covariates totals perfectly estimates their totals, whatever the sample S is. The reader is referred to [START_REF] Särndal | The calibration approach in survey theory and practice[END_REF] for a review on the subject.

In statistical learning, a popular method for improving the estimation of β in case of a large number of covariates is to have recourse to penalization to estimate the unknown vector β. More precisely, we de ne

β pen = arg min β∈R p k∈U y k -x k β 2 + t =1 λ ||β|| γ ν , (1.14)
with t ∈ N, λ ∈ R + , ν ∈ N and γ ∈ R + are hyper-parameters to be chosen before estimation.

Common choices include t = 1, γ 1 = 1 and η 1 = 1 for the lasso; t = 1, γ 1 = 2 and η 1 = 2 for ridge; t = 2, γ 1 = 1, η 1 = 1, γ 2 = 2 and η 2 = 2 for the elastic-net. The e ect of penalization is to decrease the norm of the vector of estimated coe cients. Some choices such as the Lasso or the Elastic-net are able to put some coe cients down to zero, and therefore can be seen as variable selection methods as well. The prediction at the point x k with a penalized linear model is given as

m pen (x k ) := x k β pen , k ∈ U.

The generalized linear regression estimator and its penalized counterparts

In practice, the vectors β lr and β pen cannot be computed as the y-values are recorded for the sample units only. An estimator of β lr , denoted by β lr , is obtained using the following weighted least square criterion at the sample level:

β lr = arg min β∈R p k∈S (y k -x k β) 2 π k . (1.15)
Again, the solution to (1.15) is unique provided that X S := (x k ) k∈S is of full rank and is given by:

β lr = k∈S x k x k π k -1 k∈S x k y k π k . (1.16)
Plugging m lr (x k ) := x k β lr in (1.12) leads to the model-assisted estimator based on linear regression, also called the Generalized regression estimator (GREG, [START_REF] Särndal | Model assisted survey sampling[END_REF]):

t greg = k∈U m lr (x k ) + k∈S y k -m lr (x k ) π k .
For the estimation of β pen , we de ne the following sample criterion:

β pen = arg min β∈R p k∈S 1 π k y k -x k β 2 + t =1 λ ||β|| γ ν .
A model-assisted estimator based on a penalized regression procedure is obtained from (1.12) by replacing m(x k ) with m pen (x k ) = x k β pen , leading to

t pen = k∈U m pen (x k ) + k∈S y k -m pen (x k ) π k .

High-dimensional asymptotics for linear and penalized linear models

To state the high-dimensional consistency of the GREG estimator, mild regularity conditions on the sampling design, the survey variable and the covariates are needed. These are the extension of those used in [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF], but adapted for the high-dimensional framework, see Chapter 3 for more details.

Result 1.2.1. Consider a sequence of GREG estimators { t greg } v∈N . Then,

1 N v ( t greg -t y ) = O p p 3 v n v .
Result 1.2.1 shows that, to guarantee the consistency of the GREG estimator t greg , the number of auxiliary variables must be relatively small with respect to the sample size, i.e.,

p 3 v /n v = o p (1).
Under stronger yet still realistic assumptions, the convergence rate can be improved to p 2 v /n v . Even then, the convergence rate is quite slow compared to the traditional asymptotic framework in which the GREG estimator is square-root consistent. With the same assumptions used in Result 1.2.1, we show that the penalized estimator t pen is consistent whenever the GREG estimator is and that, as a consequence, the penalized estimator cannot converge slower than the GREG estimator.

Result 1.2.2. Consider a sequence of penalized model-assisted estimators { t pen } v∈N of t y obtained by either ridge, lasso or elastic-net. Then,

1 N v ( t pen -t y ) = O p p 3 v n v .
Under relatively stronger regularity conditions than those needed for result 1.2.2 (see chapter 3 for more details), it is possible to improve on the convergence rate of some penalized estimators. For instance, consider the particular case of the ridge estimator, t ridge .

Result 1.2.3. Consider a sequence of penalized ridge estimators { t ridge } v∈N . Then,

1 N v E p t ridge -t y 2 = O p v n v .

Simulation study

In Dagdoug et al. (2022a), we conducted a large simulation study that included many scenarios. We chose to compare a wide range of model-assisted estimators, such as based on linear regression, penalized regression with lasso, ridge and elastic-net, regression trees, random forests, Cubist, gradient boosting, k-nearest neighbors and principal component regression, see Chapter 3 and 5 for details. We were interested in estimating the nite population total of four survey variables; some of them were linear in the auxiliary variables while others were not. To isolate the in uence of the dimension, we added an increasing number of noise variables that were not related to the survey variables and repeated our simulations. We used both equal and unequal sampling designs in the simulations.

Overall, Cubist and penalized regression estimators were the most e cient. Random forests, XGBoost, principal component regression, and, to a lesser extent, k-nearest neighbors, also improved on the Horvitz-Thompson estimator in most cases. The results of linear regression were very dependent of the survey variable considered and of the number of covariates considered. Our results also illustrated a few notable facts. First, whether or not the survey variable was linear in the auxiliary variables, the estimator based on linear regression was the most impacted by the addition of noise variables. For instance, in one scenario, its relative e ciency with respect to the Horvitz-Thompson estimator increased by almost 600%. For that same scenario, the other estimators had a relative e ciency which increased of only 14%, on average. The same observation was revealed in most scenarios tested, no matter the sampling design. Another interesting nding is that, when using unequal sampling designs with random forests, the estimator may exhibit a small sample bias if the hyper-parameters are not wellchosen (more precisely, if the covariates used in the sampling design are not su ciently taken into account).

Random forest for model-assisted estimation in nite population sampling

Regression trees and random forests are algorithms suggested for estimating the unknown regression function m in (1.11) and making predictions; these are non-parametric prediction methods or predictors. We begin by describing regression trees and random forests de ned at the population level.

A description of regression trees

A regression tree is a prediction method that can be viewed as an algorithm composed of two parts: a partitioning algorithm and a prediction rule. Let D N denote the set of N-tuples of vectors of

[0; 1] p × [C 1,Y ; C 2,Y ].
A partitioning algorithm is an algorithm which, given data points, de nes a partition of the space of covariates. That is, this is a deterministic function P : D N → P ([0; 1] p ) where P ([0; 1] p ) denotes the set of partitions of the unit hypercube of R p , see [START_REF] Nobel | Histogram regression estimation using data-dependent partitions[END_REF] for more details. Generally, partitions are created by successive splits with the objective of optimizing a certain criterion. The elements of the resulting partition P := {A 1 , A 2 , ..., A T } will be called the leaves or nodes of the tree.

A prediction rule is an algorithm which, takes as input a partition P := {A 1 , A 2 , ..., A T } and a dataset D U , and returns a prediction. In the case of regression trees, the prediction rule traditionally used returns the empirical average of the set {y k ; k ∈ U such that x k ∈ A(x)}, where A(x) denotes the leaf of the tree containing point x. More precisely, the prediction m tree (•, P, D U ) := m tree (•) is de ned by

m tree (x) = k∈U 1 x k ∈A(x) ∈U 1 x ∈A(x) y k ,
(1.17)

where,

1 x k ∈A(x) = 1 if x k ∈ A (x)
and 0 otherwise. In this article, unless otherwise stated, the term tree or regression tree will designate any regression tree built from any partitioning algorithm.

Figure 2 illustrates a regression tree based on two covariates, leading to the corresponding partition of R2 . Below we describe the CART partitioning algorithm, commonly used in practice.

Figure 2: Regression tree on R 2 (left) and its corresponding partition (right).

Example 1.2.1. CART partitioning algorithm [START_REF] Breiman | Classi cation and regression trees[END_REF].

In the CART partitioning algorithm, the partition is obtained by successive splits. More precisely, let A be a leaf of cardinality #(A) considered for the next split and C A be the set of all possible splits in the leaf A, which corresponds to all the pairs ( j, z) = (variable, position). De ne

mse(A) := 1 #(A) k∈U 1 x k ∈A (y k -ȳA ) 2 and ȳA := 1 #(A) k∈U 1 x k ∈A y k .
The splitting procedure is performed by nding the best split ( j * , z * ), that is, the one maximizing the following criterion

L( j, z) = mse(A) -mse (A L ) -mse (A R ) (1.18)
where

A L = k ∈ A; x k j < z , A R = k ∈ A; x k j z . Equivalently, note that maximizing (1.18) is similar to minimizing L( j, z) = mse (A L ) + mse (A R ) .
This criterion therefore searches for the split which would generate child nodes as homogeneous as possible, in terms of mean square error. Splits are always performed in the middle of two points. The procedure continues as long as a stopping criterion is not reached. The usual stopping criteria are obtained by specifying a minimum number of elements (n 0 ) in the terminal nodes, or a maximum depth (K) for the tree.

Example 1.2.2. Partitioning rule proposed by [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF].

The algorithm proposed by McConville and Toth can be described as follows:

1. Consider n 0 := n 11/20 , the minimum number of units in each node and choose α ∈]0; 0.5[, a con dence level.

3. Among the available p covariates, choose the one that has the test statistic with the lowest p-value in the hypothesis test H 0 : "∃C ∈ R such as E Y |X j ∈ A = C" for j = 1, ..., p. If none of these test statistics is signi cant (with respect to the α threshold set in step 1.), then A is a terminal leaf. In this case, return to step 1. for the next node.

4. Perform the split at a position z * ∈ arg max z L( j, z), with L de ned in the same way as for the CART criterion. This criterion is only optimized on the positions leading to child leaves containing at least n 0 elements in each child leaf.

For more details about trees and partitioning procedures, the reader is referred to [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF] or [START_REF] Györ | A distribution-free theory of nonparametric regression[END_REF], for comprehensive treatments of the topic.

Heuristic motivation and description of random forests

In practice, regression trees are particularly popular because their predictions can be easily understood and interpreted. However, their predictive e ciency may be low in some cases; see Figure 3 for an illustration motivating the use of random forests instead of regression trees. We have generated 100 observations with a covariate X 1 from a distribution U ([0; 1]) and a survey variable Y = 4 + 2X 2 1 + N (0; 0.2). The green curve is the real regression function m and we computed two estimations of m: the red curve is the tree-regression estimate of m based on the CART criterion with n 0 = 20 (see Exemple 1.2.1 from below) and the yellow curve is the random forest estimation of m. We can remark that the random forest predictor provides a much better estimation of m and we give below an heuristic explanation of this fact.

1.2

By construction, a regression tree belongs to the set of piecewise functions, a set of functions with nite complexity11 . Furthermore, the complexity of trees is necessarily lower or equal to N; thus, m tree belongs to the set of piecewise functions from R p to R with, at most, N di erent values. If the regression function m belongs to this function space, then m tree may be a good estimator of m. However, if m is a smooth function, say continuous, then m tree may be far from m as the complexity of m is +∞ if m is not constant. Yet, every continuous function can be seen as the uniform limit of a sequence of piecewise functions. The construction of such a sequence of piecewise functions converging to a continuous function is based on two facts: 1) the complexity must increase to in nity; 2) the diameter of each interval on which the piecewise function is constant must decrease to zero. It is thus possible to see N as an indicator of maximal complexity that a tree can reach. It follows that, asymptotically (for large N), it is possible for a tree to be an e cient estimator of a continuous regression function. For small samples, however, this may be another story.

While piecewise constant functions only have a small complexity, the following idea permits to increase it substantially. It is possible to show that, if { f b } B b=1 is a list of B piecewise functions, each with maximal complexity N, then the average function

f ave := 1 B B b=1 f b ,
belongs to the set of piecewise functions with maximal complexity B × N. Hence, the maximal complexity of f ave can be much greater than the complexity of the piecewise functions { f b } B b=1 . Naturally, for the gain of complexity to be important, the functions { f b } B b=1 must be di erent one from another (the intervals on which they are constant should be di erent).

A random forest is a predictor using this principle to estimate m: it is de ned as an average of B regression trees (thus, an average of piecewise functions). We see that the random forest represented by the orange curve in Figure 3 is still a piecewise constant function, but with a larger complexity than the red curve of the regression tree. Since the prediction rules described in Examples 1.2.1 and 1.2.2 are deterministic, it is clear that, for a xed set of elements, using the same partitioning rule to construct B trees would simply result in constructing the same tree B times. In this case, there would be no gain in complexity. Breiman thus suggested [START_REF] Breiman | Bagging predictors[END_REF][START_REF] Breiman | Random forests[END_REF] to introduce an additional randomness in the partitioning algorithm and/or prediction rule. The additional randomness introduced in the predictors can be de ned using the concept of stochastic predictors. Let Θ be a random variable de ned in a measurable space (J, J ). A stochastic predictor m is a measurable function such that m : R p × J → R. In other words, the predictor m may use a random variable Θ to make its predictions. It follows that the prediction method m is random with respect to Θ and, as such, an additional randomness is present.

Example 1.2.3. Let q ∈]0; 1[ and Θ be a random variable with Bernoulli distribution B(q); de ne m(x, Θ) := Θ||x|| 2 , where ||•|| 2 denotes the Euclidean norm. Then, m is a stochastic prediction model, meaning that, for two di erent realizations of Θ, the predictor m may generate di erent predicted values. An additional source of randomness is present, i.e., V Θ ( m(x, Θ)) > 0, where V Θ (•) denotes the variance with respect to the random variable Θ.

Two more concrete examples of how the randomization procedure can be incorporated and used in regression trees are given below.

Example 1.2.4. Breiman's random forests, [START_REF] Breiman | Random forests[END_REF]. The algorithm of Breiman can be described as follows:

1. Select B bootstrap samples12 in U, denoted by {U (Θ b )} B b=1 .

2. In each bootstrap sample, U(Θ b ), build a stochastic regression tree m ( • , Θ b ) by using the CART criterion, as described in Example 1.2.1, where the splitting criterion is optimized only on p 0 covariates among the p available. The p 0 covariates are chosen uniformly at random (without replacement) among the p covariates available, according to Θ b , at each split.

Example 1.2.5. Uniform random forests [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF], Scornet, 2016a).

All the B trees of the forest have the same behavior; as such, we describe only the behavior of a generic tree among the B belonging in the forest. We begin by considering [0; 1] p as the initial leaf. Then, recursively, the algorithm splits in the following fashion.

1. A node G is chosen uniformly at random among the existing nodes.

2. A covariate X j is chosen uniformly at random among the p covariates X 1 , X 2 , ..., X p .

3. A split is performed in the node G on X j at a position chosen uniformly at random.

The process is repeated K times, with K ∈ N, a parameter chosen by the user.

It is now possible to de ne the prediction of a random forest as an average of the predictions obtained from the B stochastic regression trees. More precisely, let {Θ (b) } B b=1 be a sequence of B i.i.d. random variables with distribution P Θ and { m tree (•, Θ (b) )} B b=1 be a sequence of stochastic regression trees; a random forest prediction is de ned as

m r f ( • , {Θ (b) } B b=1 ) := 1 B B b=1 m tree ( • , Θ (b) ). (1.19)
For simplicity of notations, we note m (B) r f the estimator in (1.19).

Remark 1.2.1. Initially, the term "random forest" was used to denote the initial algorithm of [START_REF] Breiman | Random forests[END_REF] and described in Example 1.2.4. However, the de nition given in this section describes a class of random forests algorithms, rather than a particular algorithm. Indeed, to each partitioning rule and randomization process, it is possible to de ne a "random forest" algorithm. Therefore, the de nition given above is quite general and includes many algorithms (including the original algorithm of [START_REF] Breiman | Random forests[END_REF]). It is also important to note that (non-stochastic) regression trees as de ned above are also part of this class; indeed, by taking B = 1 and a deterministic partitioning rule, we obtain a regression tree.

For more details about random forests and their implementation, the reader is referred to [START_REF] Biau | A random forest guided tour[END_REF] and [START_REF] Genuer | Les forêts aléatoires avec R[END_REF].

Random forests model-assisted estimator

The (random forest) di erence estimator is de ned as .20) where m (B) r f is given by (1.19). In practice, the estimator t (B) di f is not feasible. Indeed, it is built from the prediction method m r f , itself built from D U and therefore from unknown data. We propose to estimate the unknown prediction method m r f by m (B) r f 1 using the information in D ma :

t (B) di f := k∈U m (B) r f (x k ) + k∈S y k -m (B) r f (x k ) π k , ( 1
m (B) r f 1 (x) := 1 B B b=1 k∈S(Θ b ) π -1 k 1 x k ∈ A (b) (x) ∈S(Θ b ) π -1 1 x ∈ A (b) (x) y k , (1.21)
where S(Θ b ) denotes the sample which was used to build the b-th tree and A (b) (x) the leaf of the b-th tree containing the point x and obtained by applying a partitioning algorithm such as the CART algorithm given in Example 1.2.4 on the data D ma . The sums over the population are therefore replaced by sums across the sample and a weighting process is applied. More precisely, the sampling weights are incorporated into the numerator and the denominator of (1.21), thus making it possible to better take into account sampling designs with unequal probabilities. If the sampling design considered induces equal inclusion probabilities for all the elements of the population, then the weights in (1.21) cancel themselves and the estimator m (B) r f of m (B) r f is simply an estimator constructed by replacing population sums with sample sums and population partitions are replaced by sample partitions.

We now de ne a random forest model-assisted estimator as follows:

t (B) r f 1 = k∈U m (B) r f 1 (x k ) + k∈S y k -m (B) r f 1 (x k ) π k . (1.22)
Remark 1.2.2. As mentioned previously, the random forest de nition used before includes a wide class of algorithms. We can note that, given the de nitions of m (B) r f 1 in (1.21) and t (B) r f 1 in (1.22), the equation t (B) r f 1 de nes a class of estimators rather than a particular estimator. More precisely, let F r f (D ma , B) denote the set of weighted random forest functions with B trees, tted on {(x k , y k ); k ∈ S}. In this article, t (B) r f 1 actually represents any element of the set

T r f (D ma , B) := t = k∈U f (x k ) + k∈S y k -f (x k ) π k ; f ∈ F r f (D ma , B) .
Observe that T r f (D ma , 1) is the space of model-assisted estimators based on regression trees (stochastic or not); thus, the set T r f (D ma , 1) contains the estimator proposed by McConville and

1.2

Toth (2019). The results presented in this section being independent of B, the number of trees, the results presented below and in Chapter 4 are valid for any element t (B) r f 1 of

T r f (D ma ) := B∈N * T r f (D ma , B) .
Finite sample properties of the random forest estimator Proposition 1.2.1. Consider a random forest model-assisted estimator t (B) r f 1 .

1. The estimator t (B) r f 1 can be seen as an average of model-assisted estimators:

t (B) r f 1 = 1 B B b=1 t (b) tree1 ,
where t (b) tree1 denotes the model-assisted estimator based on the b-th tree in the forest, i.e.,

t (b) tree1 = k∈U m (b) tree1 (x k ) + k∈S y k -m (b) tree1 (x k ) π k ,
and m (b) tree1 is an estimator of (1.17).

2. The estimator t (B) r f 1 can be written

t (B) r f 1 = k∈U m (B) r f 1 (x k ) + 1 B B b=1 k∈O (S) b y k -m (b) tree1 (x k ) π k , (1.23)
where O (S) b := S -S(Θ b ) denotes the so-called "out-of-bag" elements of the b-th tree.

3. If m (B) r f 1 does not use a resampling mechanism13 , then t (B) r f 1 has the projection property14 :

t (B) r f 1 = k∈U m (B) r f 1 (x k ).
The point 1. above reveals that the estimator t (B) r f 1 is actually an average of B model-assisted estimators. More generally, it is possible to show that an average of model-assisted estimator remains a model-assisted estimator. Random forest estimators also have the following property: if t (B) r f 1 and t (B) r f 1 denote two forest estimators, each with B trees and built from the same algorithm, then their average ( t (B) r f 1 + t (B) r f 1 )/2 is a random forest estimator built on 2B trees. This property is no longer exact if we take the average of two forests with di erent numbers of trees. The point 2. shows that the random forest estimator computes its residuals only on the elements not selected to build the model. This is an unexpected positive point speci c to model-assisted estimators built on "bagging" [START_REF] Breiman | Bagging predictors[END_REF] type algorithms. Therefore, the second term to the right of (1.34) can be seen as a protection against ine cient predictions and against over tting. In particular, this implies that the e ciency of forests that do not include a resampling mechanism relies entirely on the prediction model (a consequence of point 3.). Note that it is also possible to write m (B) r f 1 as a weighted sum of the values {y k } k∈S :

m (B) r f 1 (x) = k∈S W (B) k1 (x)y k , (1.24)
where

W (B) k1 (x) = 1 B B b=1 π -1 k ψ (b) k 1 x k ∈ A(x) ∈S π -1 ψ (b) 1 x ∈ A(x) , k ∈ S, (1.25) with ψ (b) = 1 if ∈ S(Θ b
) and 0 otherwise.

Proposition 1.2.2. Consider a model-assisted random forest estimator t (B) r f 1 .

1. The estimator t (B) r f 1 can be seen as a weighted sum of the values {y k } k∈S :

t (B) r f 1 = k∈S w (B) k1 y k ,
where

w (B) k1 = 1 π k 1 + ∈U W (B) k1 (x ) 1 - I π , k ∈ S.
(1.26)

2. For all sampling designs, we have k∈S w (B) k1 = N, for all s ∈ S.

3. We have w (B) k1 = 1/π k for elements which are never selected in subsamples, i.e., k ∈ B b=1 O (S) b .

4. If bootstrap is used as resampling mechanism, the probability that an element is never selected converges to 0 when B increases to in nity.

5. The weights {w (B) k1 } k∈S are independent of the survey variable if and only if the partitioning rule used by the trees in the forest is independent of the survey variable.

The estimator t (B)

r f 1 is therefore an estimator which can be written as a weighted sum of the values of the survey variable. On the other hand, the weights {w (B) k1 } k∈S might be dependent of the survey variable if the partitioning rule is itself dependent of the survey variable (which is often the case, in practice). Therefore, the application of this weighting system to other survey variables must be done cautiously. On the other hand, when the splitting mechanism does not depend on the variable of interest, the estimator t (B) r f 1 therefore belongs to the class of linear estimators (i.e., it can be written as a weighted sum of the measurements of Y with weights independent of Y ). The previous proposition also reveals that the sum of the weights is always equal to the size of the population and that it is possible that some of these weights are in fact equal to the initial survey weights. However, for large forests, this phenomenon occurs only very rarely. Even when this scenario occurs, it should be noted that these elements are still used in the construction of the estimator: they contribute to the correction term in the form 2. Finally, we can note that, when no resampling mechanism is used in the algorithm, then the weights (1.26) are always positive, which is an attractive property.

Asymptotic properties the random forest estimator

The results that we describe below require certain regularity assumptions concerning the survey design, the survey variable and the forest algorithm on which the estimator is built, see [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF] for more details. Most of these assumptions are commonly used in the literature and veri ed in practice, see for example [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] and [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF] for more details. In particular, in Chapter 4, we restrict our work to the case of random forests where the resampling mechanism used is sampling without replacement, a slight modi cation from the original algorithm of Breiman.

Result 1.2.4. There exists constants C 1 > 0 and C 2 > 0 such that:

E p 1 N v t (B) r f 1 -t y C 1 √ N v + C 2 n 0v , a.s. (1.27)
where n 0v is the minimal number of elements allowed per terminal node in each tree of the forest.

It is therefore possible to bound the L 1 error of each estimator in the class T * r f (D ma ). Moreover, if n 0v tends to in nity, then this bound decreases to 0. Consequently, in this case, the estimators of T * r f (D ma ) are asymptotically unbiased and consistent for t y . In the rest of the section, we will therefore assume that n 0v tends to in nity as n tends to in nity. To get some of the following results, we will actually need to consider n 0v such that √ n v /n 0v converges to 0.

The following equivalence allows us to guide our suggestion regarding the asymptotic variance of the forest estimator and to determine its asymptotic distribution.

Result 1.2.5. The estimator t (B) r f 1 is equivalent to the generalized di erence estimator

t (B) di f : √ n v N v t (B) r f 1 -t y = √ n v N v t (B) di f -t y + o P (1), where t (B) di f is given in (1.20).
This result allows us to deduce the asymptotic variance of t (B) r f 1 , equal to

AV p 1 N v t (B) r f 1 = 1 N 2 v k∈U v ∈U v (π kl -π k π ) y k -m (B) r f (x k ) π k y -m (B) r f (x ) π .
(1.28)

In practice, this variance cannot be calculated and we therefore propose to estimate it by

V (B) r f 1 = 1 N 2 v k∈S v ∈S v π k -π k π π k y k -m (B) r f 1 (x k ) π k y -m (B) r f 1 (x ) π . (1.29)
This is a consistent and asymptotically unbiased estimator for the asymptotic variance of t (B) r f 1 , as guaranteed by the following result.

Result 1.2.6. The variance estimator

V (B) r f 1 is convergent for AV p N -1 v t (B) r f 1 , i.e. , lim v→∞ E p n v N 2 v V (B) r f 1 -AV p N -1 v t (B) r f 1 = 0.
In order to be able to determine asymptotic con dence intervals, it is necessary to determine the asymptotic distribution of the proposed estimator which is obtained under the additional assumption that the generalized di erence estimator t (B) di f follows a normal distribution. Result 1.2.7. Assume that

N -1 v t (B) di f -t y V p N -1 v t (B) di f L ----→ v→∞ N (0, 1) , then N -1 v t (B) r f 1 -t y V (B) r f 1 L ----→ v→∞ N (0, 1) .
As illustrated in Example 1.2.3, stochastic predictors have an additional source of variation, introduced by the randomization variables. The variance estimator (1.29) does not take into account these additional variations (induced by Θ, see 1.2.3). However, it is possible to show that there is a positive constant C such that

V Θ t (B) r f 1 N v C B .
Therefore, if B is chosen large enough, then the variations coming from the introduced randomization are negligible and do not need to be estimated.

Simulations and empirical investigations

In [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF], several simulation studies have been conducted. Simulations were performed in a large variety of scenarios in order to investigates the empirical performances of the random forest point estimator t (B) r f 1 . In most scenarios, the estimator was relatively e cient: when the relationship was linear in the covariates, t (B) r f 1 was slightly less e cient than the GREG estimator, yet remained e cient and improved on the Horvitz-Thompson estimator. When the relationship was not linear, however, important improvements were obtained with t (B) r f 1 over the GREG estimator. Our simulations also included a few high-dimensional scenarios, in which t (B) r f 1 was the only estimator included in the simulations to remain more e cient than the Horvitz-Thompson estimator.

We have also investigated the performances of the variance estimator V (B) r f 1 de ned in (1.29). We noted, in accordance with our theoretical results, that the estimator V (B) r f 1 is nearly unbiased and e cient when large sample and population sizes were used and when the minimal number of elements in each node n 0 is large enough. When this assumption was not satis ed, however, the estimator su ered from an important negative bias leading to an undercoverage for the con dence intervals. The problem detected here is in fact more general and may happen to all model-assisted estimators based on predictors capable of interpolating the data (i.e., exible enough). We suggested a cross-validated type variance estimator, which estimates e ciently the variance of the RF estimator, independently of the choice of n 0 , see Chapter 6 for more about this procedure.

Lastly, we conducted a simulation study investigating the in uence of the main hyperparameters of a random forest algorithm on the e ciency of the resulting model-assisted estimators. We found that the number of trees in the forest can be chosen arbitrarily large without risk of over tting; that the default number of variables considered for the splitting process p 0 = √ p led to satisfactory results in most cases; and that the most in uent parameter to be chosen was n 0 , which should be chosen neither too small nor too large. The cross-validated variance estimator discussed in the previous paragraph can also be used for guiding our choice of parameters. A more elaborate discussion is provided in the Conclusion chapter of this dissertation.

1.3 Item nonresponse and imputation in surveys 1.3.1 Basic framework and imputed estimators

In the previous sections, we made the assumption of full response, meaning that every information sought for the sampled elements could be collected. In practice, however, nonresponse happens in practically almost every survey. Chen and Haziza (2019) explains, "every time data are collected, it is virtually certain that one will face the problem of missing values. " Throughout this work, two assumptions are made: 1) each element of the population has a strictly positive probability of response; 2) the response of an element is independent from the response of other elements. It is clear that in some cases, these assumptions may not be satis ed. For instance, it is possible that some elements may never want to respond [START_REF] Kott | A note on handling nonresponse in sample surveys[END_REF]. However, these assumptions seem to be satis ed in most cases and are needed in order to have e cient nonresponse treatments. Interestingly, nonresponse is an undesirable phenomenon which can be modeled as a probability sampling design with unknown inclusion probabilities. Indeed, under the assumption that the response of each element is not a ected by the response of other elements, nonresponse is essentially a Poisson sampling design with unknown parameters {p k } k∈U where p k denotes the probability that unit k responds. To be more precise, let U r be a random variable taking value in the design space (S, D) with distribution P r called a nonresponse mechanism. In the literature, nonresponse mechanisms are classi ed in three di erent categories [START_REF] Rubin | Inference and missing data[END_REF]: missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR). In the rst category, it is assumed that the random variable U r is independent of the survey variable Y ; in the second scenario, it is assumed that, conditionally on the covariates X U , the random variable U r is independent of Y ; lastly, MNAR is used for distributions satisfying neither MCAR nor MAR. In this work, we restrict our investigations to the case of MAR.

Proposition 1.3.1. (Missing at random nonresponse mechanism.) Consider the design space (S, D). A nonresponse mechanism P r satisfying the missing at random assumption is a function P r : D × R N×p → [0, 1], such that i) for all A ∈ D, the map X U → P r (A, X U ) is measurable.

ii) for all X U ∈ R N×q , the map A → P r (A, X U ) is a probability measure on (S, D).

iii) The map

p r := S -→ [0; 1] , s -→ P r ({s}) .
is a Poisson sampling design.

Note that, given the covariates, S and U r are independent. In the imputation literature, it is common practice to work with the joint distribution induced by the sampling design, the superpopulation model and the nonresponse mechanism. The product space considered is the following.

De nition 1.3.1 (Joint model-design distribution with auxiliary information).

Consider the superpopulation probability space (Ω, M, P m ), the sampling design space S, D, P p and the response probability space (S, D, P r ). To each ω ∈ Ω, we de ne a product measure P p,r (A × B, ω) := P p (A, ω) P r (B, X U (ω)) for all measurable rectangles A × B ∈ D ⊗ D. The joint model-design-nonresponse probability induced by a nonresponse mechanism satisfying the MAR assumption is de ned as the probability P m,p,r on the measurable space (Ω × S × S, M ⊗ D ⊗ D), uniquely de ned as

P m,p,r (A × B × C) := ∫ A P p,r (B × C, ω) dP m (ω) for all measurable rectangles A × B × C ∈ M ⊗ D ⊗ D.
Each realization of U r de nes a subset of elements of U, which would be respondents if they were selected in the sample. Thus, the observed elements are given by the realizations of the random variable S r := S ∩ U r , of size n r . The set of sampled nonrespondents is denoted by S m := S\S r with size n m . As before, since each element of S can be represented as a vector in {0; 1} N , we de ne the random vector [r 1 , r 2 , ..., r N ] in the product space, where r k = 1 if k ∈ U r and 0 otherwise. By construction, {r k } k∈U is a set of independent Bernoulli random variables {B(p k )} k∈U , where the set of response probabilities {p k } k∈U is unknown, and may depend on the covariates under MAR.

In presence of nonresponse, the Horvitz-Thompson cannot be used as it is based on unknown values. Indeed, we have the decomposition

t π = k∈S r y k π k + k∈S m y k π k , (1.30)
where the second term of (1.30) is unknown. Instead, it is common to use an imputed estimator, de ned as

t imp = k∈S r y k π k + k∈S m y * k π k , (1.31)
where y * k is a proxy value used to replace missing y k , called an imputed value. If we assume that the set of covariates {x k } k∈S is fully observed for all sampled elements, then it is customary to use a predictor m, tted on D n r := {(x k , y k ) ; k ∈ S r } to de ne the imputed values y * k := m(x k ). In that case, the imputed estimator (1.31) is given by

t m = k∈S r y k π k + k∈S m m(x k ) π k .
(1.32)

In the article [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF], to establish our results, we laid out a set of conditions on m under which the imputed estimator t m converges in L 2 with respect to the joint distribution.

The conditions that we found on m reveal that if the predicted values are based on a predictor m consistent (in L 2 ) for the regression function m, and whose error is uniformly bounded, then t m converges in L 2 towards t y .

For this result, we assumed, in addition to the conditions on m, similar conditions such as those described in [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] for the consistency of the Horvitz-Thompson, see Chapter 5. ii) There exists a positive constant C, independent of v, such that

E m(x) -m(x) 2 r, X, I C. a.s.
Then, the sequence of imputed estimators

{ t m } is L 2 -consistent with rate E 1 N v t m -t y 2 = O γ v .
The conditions that we found to ensure the L 2 consistency of the imputed estimator have simple interpretations: i) requires that, for large samples, the predictor m estimates e ciently the true regression function m and ii) requires that, even for small samples, the error of estimation is bounded. Omitting condition ii), the result therefore states that, for an imputed estimator to be L 2 consistent, it is enough that it is based on a consistent prediction method. In other words, in that scenario, the problem of imputation is not harder to solve than the problem of regression.

The theoretical properties of imputed estimators were investigated for the nearest neighbor predictor [START_REF] Chen | Nearest neighbor imputation for survey data[END_REF], 2001[START_REF] Yang | Nearest neighbor imputation for general parameter estimation in survey sampling[END_REF], the score method [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF]Beaumont, 2007, Little, 1986), predictive mean matching [START_REF] Yang | Predictive mean matching imputation in survey sampling[END_REF], kernel regression [START_REF] Zhong | Jackknife empirical likelihood inference with regression imputation and survey data[END_REF], to cite just a few. For more information about the missing data literature in surveys, see Chen and Haziza (2019) or [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF].

In this dissertation, two articles were focused on imputed estimators. First, the article Dagdoug et al. (2021a) entitled Imputation procedures in surveys using nonparametric and machine learning methods: an empirical comparison, for which the main results are presented next. This article investigates the empirical performances of imputed estimators based on machine learning procedures, in a wide variety of scenarios. The complete article is presented in Chapter 4. Next, the article [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF] entitled Regression tree and random forest imputation in surveys with application to data integration is focused on the analysis of imputed estimators based on regression trees and random forests. The nite sample properties of the resulting estimators are thoroughly investigated and their L 2 -consistency towards t y is established.

Imputation procedures in surveys using nonparametric and machine learning methods: an empirical comparison

In the article Dagdoug et al. (2021a), we conducted an extensive simulation study to compare several nonparametric and machine learning imputation procedures in terms of bias and e ciency. The imputation procedures were evaluated in the case of nite population totals of continuous and binary variables and for population quantiles under both simple random sampling without replacement and proportional-to-size Poisson sampling. The Cubist algorithm, BART and XGBoost performed very well in a wide variety of settings. In general, these methods seem to be highly robust to model misspeci cation and seem to have the ability to capture nonlinear trends in the data. Additive models based on B-splines performed well in the case of population totals when the number of explanatory variables was small but broke down for large values of p. Finally, random forests performed relatively well in a high-dimensional setting.

In practice, the choice of an imputation procedure is not clear-cut and depends on the data at hand. If one is reasonably con dent about the correct speci cation of the rst moment of the imputation model (that includes the correct speci cation of the functional form and the correct speci cation of the vector of explanatory variables), parametric imputation procedures are expected to do well in terms of bias and e ciency. In addition, parametric imputation is simpler to understand and the results are easier to interpret, in general. In the case of complex/nonlinear relationships and/or in a high-dimensional setting, our empirical investigations suggest that machine learning procedures outperform traditional imputation procedures as they tend to be robust against model misspeci cation. However, these procedures require the speci cation of some regularization parameters. For instance, for XGBoost, one must specify the learning rate, the maximal depth and the coe cient of penalization. In support vector regression, the cost function and the kernel function must be selected, among others. In practice, the value for some of these parameters are determined through a cross-validation procedure. More details can be found in Chapter 5.

Regression tree and random forest imputation in surveys with application to data integration

Let m (B) r f 2 be an estimator of m obtained according to a random forest (with any partitioning rule), unweighted, built on {(x k , y k ); k ∈ S r }, that is,

m (B) r f 2 (x) = k∈S r W (B) k2 (x)y k ,
where

W (B) k2 (x) = 1 B B b=1 ψ (b) k 1 x k ∈ A(x) ∈S r ψ (b) 1 x ∈ A(x) , k ∈ S r .
The class of estimators imputed by random forests t (B) r f 2 is de ned by the set of elements of the form

t (B) r f 2 := k∈S r y k π k + k∈S m m (B) r f 2 (x k ) π k . (1.33)
Finite sample properties of the random forest imputed estimator Proposition 1.3.2. Consider a random forest imputed estimator t (B) r f 2 .

1. The estimator t (B) r f 2 can be seen as an average of imputed estimators:

t (B) r f 2 = 1 B B b=1 t (b) tree2 ,
where t (b) tree2 denotes the imputed estimator based on the b-th tree in the forest.

2. If the sampling design has equal inclusion probability, then the estimator t (B) r f 2 can be written

t (B) r f 2 = k∈S m (B) r f 2 (x k ) π k + 1 B B b=1 k∈O b (S r ) y k -m (b) tree2 (x k ) π k , (1.34)
where O b (S r ) := S r -S r (Θ b ) and m (b) tree2 denotes the b-th tree in forest m (B) r f 2 .

3. If the sampling design has equal inclusion probability and if m (B) r f 2 does not use a resampling mechanism, then t (B) r f 2 has the projection property:

t (B) r f 2 = k∈S m (B) r f 2 (x k ) π k .
Proposition 1.3.3. Consider a random forest imputed estimator t (B) r f 2 .

1. The t (B) r f 2 estimator can be written as a weighted sum of {y k } k∈S r :

t (B) r f 2 = k∈S r w (B) k2 y k ,
where

w (B) k2 = 1 π k + ∈S m W (B) k2 (x ) π = 1 π k + 1 B B b=1 ψ (b) k N b (x k , S m ) N b (x k , S r (Θ b )) , k ∈ S r , (1.35)
where N b (x k , S m ) denotes the weighted sum of elements of S m belonging to the node containing x k and N b (x k , S r (Θ b )) denotes the number of elements of S r (Θ b ) belonging to the node containing the point x k .

2. In the case of a deterministic tree, if the sampling design has equal inclusion probability, then we have

w (B) k2 = 1 π k × 1 + 1 B B b=1 ψ (b) k N (x k , S m ) N (x k , S r (Θ b ))
.

where N (x k , S m ) denotes the number of missing elements in the node containing x k .

3. If the initial weights are calibrated to the population size k∈S 1 π k = N, then k∈S w (B) k2 = N.

We have

w (B) k2 = 1 π k for elements k ∈ B b=1 O b (S r ).
5. If there are at least n 0 elements in the leaves of each tree, then the weights are bounded as follows

d k w (B) k2 d k 1 + n m n 0 , a.s. k ∈ S r .
(1.36)

These bounds can be reached.

6. The weights {w (B) k2 } k∈S r are independent of the survey variable if and only if the partitioning rule used by the trees in the forest is independent of the survey variable.

As with the model-assisted estimator, the forest imputed estimator can be written as a weighted sum of the values of the variable of interest. In the case of the imputed estimator, these weights reveal a lot of information about the behavior of the estimator. First of all, we observe that the imputation weights are always greater than or equal than the initial weights and are calibrated on the sum of the initial weights. If we consider the weights of an estimator based on a deterministic tree (e.g., CART, scoring method) with equal inclusion probabilities, we have

w (1) k2 = d k × 1 + N (x k , S m ) N (x k , S r ) = d k × 1 + R mr (x k ) , k ∈ S r .
We therefore observe that, if most of the elements with characteristics similar to an element k ∈ S r have not responded, then a signi cant weight will be attributed to element k because the ratio R mr (x k ) will be important. Otherwise, if almost all the elements with characteristics similar to the individual k ∈ S r have responded, the imputation weights will be very close to the initial weights. This is a desired behavior: when we have only a few elements similar to many other, the imputation weights of these elements need to be large; otherwise, if most elements of a given category are observed, it is enough to let the imputation weights of these remain close to its original weight. In particular, a responding element has an imputation weight equal to its initial weight if and only if all the elements in its leaf are respondents. The same interpretation is valid in the case of unequal inclusion probabilities.

On the other hand, if we consider stochastic trees and random forests, these properties are lost for forests with only a few trees (i.e., low B). Indeed, for the elements that have not been selected (there may be many of them for low B), the imputation weights are equal to the initial weights. Regardless of that, the sum of the imputation weights remains equal to the sum of the initial weights: a compensation phenomenon is therefore necessarily introduced and these weights can be relatively unstable. When B is large, this instability disappears and the behavior of the weights of the forest is very close to the behavior of the weights of a tree.

Asymptotic properties of the random forest imputed estimator

To study the asymptotic properties of estimators imputed by forests, it is useful to consider the in nite predictor m (∞) r f 2 de ned by

m (∞) r f 2 := lim B→∞ m (B) r f 2 = lim B→∞ 1 B B b=1 m (b) tree2
as well as the in nite estimator

t (∞) r f 2 := k∈S r y k π k + k∈S m m (∞) r f (x k ) π k = E Θ t (B) r f 2 . (1.37)
This estimator is of purely academic interest since, in practice, the number of trees that we can use is always nite. However, this in nite estimator is particularly interesting for three reasons: 1) it is simpler to study theoretically than the nite forest estimator; 2) it is more e cient than the nite forest estimator, as the proposition below reveals; 3) it is approachable to a given precision by the nite forest estimator.

Proposition 1.3.4. There exists C > 0 such that

0 E       t (B) r f 2 -t y N v 2       -E       t (∞) r f 2 -t y N v 2       C B .
(1.38)

Moreover, for all > 0,

P Θ | t (B) r f 2 -t (∞) r f 2 | > 2 exp -B 2 2n 2 m C 2,Y -C 1,Y min k∈U π k 2 .
This proposition therefore shows that it seems to be interesting to build large forests, in the sense that the in nite forest estimator is more e cient than the nite forest estimator. We now restrict our analysis to the case where, for the imputation in S v , we choose B v such that, if v 1 < v 2 then the number of trees used to impute, B v 1 , is strictly less than the number of trees used to impute, B v 2 . This allows the use of the result (1.38) implying, as soon as the estimator of in nite forests is consistent in L 2 , the consistency L 2 of the estimator of nite forests. Moreover, we restrict ourselves to the case where t (B) r f 2 is an imputed estimator based on the random forest algorithm in Breiman's original sense. More details about the assumptions for this result are given in Chapter 5.

Result 1.3.2. The estimator t (B) r f 2 converges in L 2 with respect to the joint distribution, that is,

lim v→∞ E 1 N v t (B) r f 2 -t y 2 = 0.
Finally, concerning the estimation of the variance, similarly as for model-assisted estimators, we show that the variations due to the randomization variables decrease when B increases:

V Θ t (B) r f 2 N v C B v .
It is therefore su cient to estimate the variance of t (B) r f 2 with respect to the joint distribution induced by the design, the model and the nonresponse mechanism. In most cases, the "naive" variance estimator

V naive := k∈S ∈S ∆ k π k r k y k + (1 -r k )y * k π k r y + (1 -r )y * π (1.39)
is a severely biased estimator; the use of speci c variance estimators is therefore necessary. Two approaches are traditionally used: the "two-phase" approach [START_REF] Särndal | Methods for estimating the precision of survey estimates when imputation has been used[END_REF], and the "reverse" approach [START_REF] Shao | Variance estimation for survey data with composite imputation and nonnegligible sampling fractions[END_REF]. The interested reader can refer to [START_REF] Haziza | Variance estimation procedures in the presence of singly imputed survey data: a critical review[END_REF] for a review of concepts and tools related to variance estimation of imputed estimators.

We suggested two corresponding estimators. For the two-phase approach,

V sar := V sam + V nr + 2 V mix , (1.40)
where

V sam := V naive + k∈S m d 2 k (1 -π k ) σ 2 V nr := k∈S γ 2 k σ 2 , V mix := k∈S γ k (d k -1) σ 2 ,
1.3 with σ is an estimator of the variance of the model residuals and γ k := r k w (B) k2d k for k ∈ S. Finally, for the reverse approach, assuming that the sampling fraction n v /N v is negligible, we suggested the following variance estimator:

V rev := k∈S ∈S ∆ k π k ξ (B) k π k ξ (B) π , (1.41)
where

ξ (B) k := m (B) r f 2 (x k ) + r k • 1 B B b=1 N b (x k , S) N b (x k , S r ) • y k -m (b) tree (x k ) , k ∈ S.
We also performed several empirical studies aiming at assessing the performances of point and variance estimators. The simulations suggest that point estimators behave well both in terms of bias and e ciency. Several variance estimators were used in the simulations, including the estimators de ned in (1.40) and (1.41). The results suggest that the estimators perform relatively well in terms of bias and coverage rate. An application to mass imputation is also included in this work; see Chapter 5 for details.

Introduction

Surveys conducted by national Statistical O ces (NSO) aim at estimating nite population parameters, which are those describing some aspects of the nite population under study. In this article, the interest lies in estimating the population total of a survey variable Y . Population totals can be estimated unbiasedly using the well-known Horvitz-Thompson estimator [START_REF] Horvitz | A generalization of sampling without replacement from a nite universe[END_REF]. In the absence of nonsampling errors, the Horvitz-Thompson estimator is unbiased with respect to the customary design-based inferential approach, whereby the properties of estimators are evaluated with respect to the sampling design; e.g., see [START_REF] Särndal | Model assisted survey sampling[END_REF]. However, Horvitz-Thompson type estimators may exhibit a large variance in some situations. The e ciency of the Horvitz-Thompson estimator can be improved by incorporating some auxiliary information, capitalizing on the relationship between the survey variable Y and a set of auxiliary variables x. The resulting estimation procedures, referred to as model-assisted estimation procedures, use a working model as a vehicle for constructing point estimators. Model-assisted estimators remain design-consistent even if the working model is misspeci ed, which is a desirable feature. When the working model provides an adequate description of the relationship between Y and x, model-assisted estimators are expected to be more e cient than the Horvitz-Thompson estimator.

The class of model-assisted estimators include a wide variety of procedures, some of which have been extensively studied in the literature both theoretically and empirically. When the working model is the customary linear regression model, the resulting estimator is the well-known generalized regression estimator (GREG); e.g., [START_REF] Särndal | On pi-inverse weighting versus best linear unbiased weighting in probability sampling[END_REF], [START_REF] Särndal | Cosmetic form of estimators in survey sampling[END_REF] and [START_REF] Särndal | Model assisted survey sampling[END_REF]. Other works include model-assisted procedures based on generalized linear models [START_REF] Firth | Robust models in probability sampling[END_REF]Bennett, 1998, Lehtonen and[START_REF] Lehtonen | Logistic generalized regression estimators[END_REF], local polynomial regression [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], splines [START_REF] Breidt | Model-assisted estimation for complex surveys using penalized splines[END_REF][START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF][START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF][START_REF] Mcconville | Survey design asymptotics for the model-assisted penalised spline regression estimator[END_REF], neural nets [START_REF] Montanari | Nonparametric model calibration in survey sampling[END_REF], generalized additive models [START_REF] Opsomer | Model-assisted estimation of forest resources with generalized additive models[END_REF], nonparametric additive models (Wang and Wang, 2011), regression trees (McConville and Toth, 2019[START_REF] Toth | Building consistent regression trees from complex sample data[END_REF] and random forests [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF].

Due to the recent advances of information technology, NSOs have now access to a variety of data sources, some of which may exhibit a large number of observations on a large number of variables. So far, the properties of model-assisted estimator have been established under the customary asymptotic framework in nite population sampling [START_REF] Isaki | Survey design under the regression superpopulation model[END_REF] for which both the population size N and the sample size n increase to in nity, while assuming that the number of auxiliary variables p is xed. In other words, existing results require n to be large relative to p. This framework is generally not adequate in the context of high-dimensional data sets as p may be of the same order as n, or even larger, i.e., p > n. A more appropriate asymptotic framework would let p increase to in nity in addition to N and n. [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF] studied dimension reduction through principal component analysis and established the design consistency of the resulting calibration estimator. More recently, [START_REF] Ta | Generalized regression estimators with high-dimensional covariates[END_REF] investigated the properties of the GREG estimator from a model point of view and when p is allowed to diverge and [START_REF] Chauvet | Asymptotic e ciency of the calibration estimator in a high-dimensional data setting[END_REF] studied the asymptotic variance of the calibration estimator when the number p of calibration variables is going to in nity.

The aim of this paper is to give a general consistency result for a class of model-assisted estimators when the number p of auxiliary variables is allowed to grow to in nity. This class of model-assisted estimators includes the GREG estimator as well as model-assisted estimators based on penalization methods such as ridge, lasso and elastic net. The latter methods were proposed to cope with multicolinearity between predictors in a high-dimension setting. Under mild regularity assumptions, we show that these model-assisted estimators are design-consistent provided that p 3 /n goes to zero. As we argue in Section 2.3, this rate can be improved if one is willing to make additional assumptions about the rate of convergence of the estimated regression coe cient. In particular, we lay out a set of additional conditions under which the model-assisted ridge estimator is consistent if p/n goes to zero and moreover, is √ n-consistent if p = O(n a ) with a ∈ [0, 1/2). Also, provided that the predictors are orthogonal, we show that both the model-assisted lasso and elastic net estimators are consistent provided that p/n goes to zero.

To the best of our knowledge, an empirical comparison of penalized or nonparametric model-assisted estimators in terms of bias and e ciency in a high-dimensional setting is currently lacking. We aim to ll this gap in the article. To assess the performance of several model-assisted estimators in a high-dimensional setting, we conduct a large simulation study using data from the Irish Commission for Energy Regulation Smart Metering Project. The data set consists of electricity consumption recorded every half an hour for a two-year period and for more than 6000 households and businesses, leading to highly correlated data. Due to the high-dimensional feature, model-assisted estimators based on a linear model tend to breakdown and penalized and reduction dimension based estimators may provide good alternatives.

The paper is organized as follows. In Section 2.2, we introduce the theoretical setup. In Section 2.3, we investigate the asymptotic properties of several model-assisted estimators: the GREG estimator as well as estimators based on ridge regression, lasso and elastic net. Section 2.4 contains an empirical comparison to assess the performance of several model-assisted estimators in terms of bias and e ciency. In our empirical experiments, we included model-assisted estimators based on ridge regression, lasso and elastic net, principal component regression as well as model-assisted estimators based on CART, random forests, XGBoost and CUBIST. We considered three sampling designs: simple random sampling without replacement, strati ed simple random sampling without replacement and strati ed xed-size without replacement proportional to size sampling. We make some nal remarks in Section 2.5. The technical details, including the proofs of some results, are relegated to the Supplementary Material.

The setup

Consider a nite population U = {1, 2, ..., N } of size N. We are interested in estimating t y = i∈U y i , the population total of the survey variable Y . We select a sample S from U according to a sampling design P(S) with rst-order and second-order inclusion probabilities {π i } i∈U and {π i } i, ∈U , respectively. In the absence of nonsampling errors, the Horvitz-Thompson estimator

t π = i∈S y i π i (2.1) is design-unbiased for t y provided that π i > 0 for all i ∈ U; that is, E p ( t π ) = t y , where E p (•)
denotes the expectation operator with respect to the sampling design P(S). In the sequel, unless stated otherwise, the properties of estimators are evaluated with respect to the design-based approach. Under mild conditions (Breidt andOpsomer, 2000, Robinson and[START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF], it can be shown that the Horvitz-Thompson estimator t π is design-consistent for t y . At the estimation stage, we assume that a collection of auxiliary variables, X 1 , X 2 , . . . , X p , is recorded for all i ∈ S. Moreover, we assume that the corresponding population totals are available from an external source (e.g., a census or an administrative le). Let x i = x i1 , x i2 , . . . , x ip be the x-vector associated with unit i. Also, we denote by X U = (x i ) i∈U the N × p design matrix and X S = (x i ) i∈S its sample counterpart.

Model-assisted estimation starts with postulating the following working model:

ξ : y i = f (x i ) + i , i ∈ U, (2.2)
where f (•) is an unknown function and the errors i are independent random variables such that

E ξ [ i |x i ] = 0 and V ξ ( i |x i ) = σ 2
, where σ 2 is an unknown parameter. Although we assume an homoscedastic variance structure, our results can be easily extended to the case of unequal variances of the form

V ξ ( i |x i ) = σ 2 ν(x i ) for some known function ν(•). The unknown function f (•) is estimated by f (•) from the sample data (x i , y i ) i∈S .
The tted model is then used to construct the model-assisted estimator

t ma = i∈U f (x i ) + i∈S y i -f (x i ) π i , (2.3)
where f (x) denotes the prediction at x under the working model (2.2). Whenever the predictor f (•) is sample dependent, the estimator t ma is design-biased, but can be shown to be asymptotically design-unbiased and design-consistent for a wide class of working models, as the population size N and the sample size n increase.

Least squares and penalized model-assisted estimators 2.3.1 The GREG estimator

Suppose that the regression function f (•) is approximated by a linear combination of X j , j = 1, . . . , p. The working model (2.2) reduces to ξ :

y i = x i β + i , i ∈ U, (2.4)
where β = β 1 , . . . , β p ∈ R p is a vector of unknown coe cients. Under a hypothetical census, where we observe y i and x i for all i ∈ U, the vector β would be estimated by β through the ordinary least square criterion at the population level:

β = arg min β∈R p || y U -X U β|| 2 2 = arg min β∈R p i∈U (y i -x i β) 2 , (2.5)
where y U = (y i ) i∈U . Provided that the matrix X U is of full rank, the solution to (2.5) is unique and given by

β = X U X U -1 X U y U = i∈U x i x i -1 i∈U x i y i . (2.6)
In practice, the vector β in (2.6) cannot be computed as the y-values are recorded for the sample units only. An estimator of β, denoted by β, is obtained from (2.6) by estimating each total separately using the corresponding Horvitz-Thompson estimator. Alternatively, the estimator β can be obtained using the following weighted least square criterion at the sample level:

β = arg min β∈R p y S -X S β Π -1 S y S -X S β = arg min β∈R p i∈S (y i -x i β) 2 π i , (2.7)
where Π S = diag (π i ) i∈S and y S = (y i ) i∈S . Again, the solution to (2.7) is unique provided that X S is of full rank and it is given by

β = X S Π -1 S X S -1 X S Π -1 S y S = i∈S x i x i π i -1 i∈S x i y i π i . (2.8) The prediction of f (•) at x under the working model (2.4) is f lr (x) = x β. Plugging f lr (•) in (2.
3) leads to the well-known GREG estimator [START_REF] Särndal | Model assisted survey sampling[END_REF]:

t greg = i∈U f lr (x i ) + i∈S y i -f lr (x i ) π i = i∈U x i β + i∈S y i -x i β π i .
(2.9)

If the intercept is included in the working model, the GREG estimator reduces to the population total of the tted values f lr (x i ) = x i β; that is, t greg = i∈U x i β. Also, the GREG estimator can be written as a weighted sum of the sample y-values:

t greg = i∈S w iS y i , (2.10)
where

w iS = 1 π i      1 -x i i∈S x i x i π i -1 i∈S x i π i - i∈U x i      , i ∈ S.
These weights can be also obtained as the solution of a calibration problem [START_REF] Deville | Calibration estimators in survey sampling[END_REF]. More speci cally, the weights w iS are such that the generalized chi-square distance

i∈S (w iS -π -1 i ) 2 /π -1
i is minimized subject to the calibration constraints i∈S w iS x i = i∈U x i . This attractive feature may not be shared by model-assisted estimators derived under more general working models.

Penalized least square estimators

While model-assisted estimators based on linear regression working models are easy to implement, they tend to breakdown when the number of auxiliary variables p is growing large. Also, when some of the predictors are highly related to each other, a problem known as multicolinearity, the ordinary least square estimator β given by (2.6) may be highly unstable. As noted by [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF], "the worse the conditioning of X U X U , the more β can be expected to be too long and the distance from β to β will tend to be large". In survey sampling, the e ect of multicolinearity on the stability of point estimators has rst been studied by [START_REF] Bardsley | Multipurpose estimation from unbalanced samples[END_REF] under the model-based approach. [START_REF] Chambers | Robust case-weighting for multipurpose establishment surveys[END_REF] and [START_REF] Rao | A ridge-shrinkage method for range-restricted weight calibration in survey sampling[END_REF] studied this problem in the context of calibration. These authors noted that the use of a large number of calibration constraints may lead to highly dispersed calibration weights, potentially resulting in unstable estimators.

In a classical iid linear regression setting, penalization procedures such as ridge, lasso or elastic-net can be used to help circumvent some of the di culties associated with the usual least squares estimator β. Let β pen be an estimator of β obtained through the penalized least square criterion at the population level:

β pen = arg min β∈R p i∈U y i -x i β 2 + t =1 λ ||β|| γ ν , (2.11)
where λ , ν and γ are positive real numbers, || • || ν is a given norm and t is a xed positive integer representing the number of di erent norm constraints. The values of ν , γ and t are typically predetermined. The tuning parameter λ controls the strength of the penalty that one wants to impose on the norm of β. Most often, the value of λ is selected through a crossvalidation procedure. The coe cients γ and ν are speci c to the penalization method. Hence, they a ect the properties of the resulting estimator β pen . Three special cases are considered below.

When t = 1, γ 1 = 2 and ν 1 = 2, λ 1 = λ, the estimator is known as the ridge regression estimator [START_REF] Hoerl | Ridge regression: biased estimation for nonorthogonal problems[END_REF]:

β ridge = arg min β∈R p i∈U y i -x i β 2 + λ||β|| 2 2 ,
where ||β|| 2 2 = p j=1 β 2 j is the usual Euclidean norm of β. The solution is given explicitly by (2.12) where I p denotes the p × p identity matrix. When t = 1, ν 1 = 1 and λ 1 = λ, the estimator β pen is known as the lasso estimator [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]: (2.13) where

β ridge = X U X U + λI p -1 X U y U = i∈U x i x i + λI p -1 i∈U x i y i ,
β lasso = arg min β∈R p i∈U y i -x i β 2 + λ||β|| 1 ,
||β|| 1 = p j=1 | β j | is the L 1 -norm of β.
As for the ridge, the lasso has the e ect of shrinking the coe cients but, unlike the ridge, it can set some coe cients β j to zero. Except when the auxiliary variables are orthogonal, there is no closed-form formula for the lasso estimator β lasso [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]. In survey sampling, McConville et al. ( 2017) investigated the design-based properties of the lasso model-assisted estimator for xed p.

The elastic-net estimator, that was suggested by [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF], combines two norms: the euclidean norm || • || 2 and the

L 1 norm, || • || 1 . If, in (2.11), we set t = 2, γ 1 = 1, ν 1 = 1, γ 2 = 2, ν 2 = 2, λ 1 = λα and λ 1 = λ(1 -α)
, the resulting estimator is the elastic-net estimator, which can be viewed as a trade-o between the ridge estimator and the lasso estimator, realizing variable selection and regularization simultaneously:

β en = arg min β∈R p i∈U y i -x i β 2 + λ α||β|| 1 + (1 -α)||β|| 2 2 ,
for λ > 0 and α ∈ [0, 1] a parameter that is usually chosen using a grid of multiple values of α.

The penalized regression estimator β pen in (2.11) is unknown as the y-values are not observed for the non-sample units. To overcome this issue, we use the following weighted penalized least square criterion at the sample level:

β pen = arg min β∈R p i∈S 1 π i y i -x i β 2 + t =1 λ ||β|| γ ν .
(2.14)

A model-assisted estimator based on a penalized regression procedure is obtained from (2.3) by replacing f (x) with f pen (x) = x β pen , leading to

t pen = i∈U f pen (x i ) + i∈S y i -f pen (x i ) π i = i∈U x i β pen + i∈S y i -x i β pen π i , (2.15)
where β pen is a generic notation used to denote the estimated regression coe cient obtained through either lasso, ridge or elastic net. Unlike the GREG estimator, t greg , the penalized modelassisted estimator is sensitive to unit change of the X-variables because β pen is sensitive to unit change. This is why, as in the classical regression setting, standardization of the X-variables is recommended before computing β pen . If the intercept is included in the model, then it is usually left un-penalized.

Remark 2.3.1. In the case of ridge regression, the estimator β ridge is given by

β ridge = X S Π -1 S X S + λI p -1 X S Π -1 S y S = i∈S x i x i π i + λI p -1 i∈S x i y i π i . (2.16)
Using (2.16) in (2.15) leads to the ridge model-assisted estimator tridge that can be expressed as a weighted sum of sampled y-values, tridge = i∈S w iS (λ)y i with weights given by

w iS (λ) = 1 π i      1 -x i i∈S x i x i π i + λI p -1 i∈S x i π i - i∈U x i      , i ∈ S.
These weights can also be obtained through a penalized calibration problem. It can be shown that they minimize the penalized generalized chi-square distance, i∈S (w iS -π

-1 i ) 2 /π -1 i + λ -1 || i∈S w iS x i -i∈U x i || 2
2 (Beaumont andBocci, 2008, Chambers, 1996). If some X-variables are left un-penalized in (2.11), the resulting weights ensure consistency between the survey estimates and their corresponding population totals associated with these variables.

We end this section by noting that the penalized model-assisted estimator t pen is sensitive to the choice of the penalty parameter λ . In the case of ridge regression, [START_REF] Bardsley | Multipurpose estimation from unbalanced samples[END_REF] suggested the ridge trace method for selecting the penalty parameter λ. This method consists of plotting the weights w iS (λ), i ∈ S for values of λ from a pre-determined grid values and to choose the value of λ for which the weights w iS (λ) are positive for all i ∈ S and i∈S w iS (λ)x i -i∈U x i is the smallest di erence among all the di erences considered for the grid values of λ. Using the fact that the modi ed penalty λ * = λ/(1 + λ) lies between 0 and 1 and is an increasing function of λ, [START_REF] Beaumont | Another look at ridge calibration[END_REF] proposed a method based on the bisection algorithm to rst determine λ * and then, λ. [START_REF] Guggemos | Penalized calibration in survey sampling: Design-based estimation assisted by mixed models[END_REF] implemented a Fisher scoring algorithm in order to nd the value of λ which maximizes a design-based estimated log-likelihood criterion. In case of the lasso model-assisted estimator, McConville et al. ( 2017) used a cross-validation procedure to choose the best value of λ. More research is needed to suggest a uni ed criterion in order to nd the best penalty in a sample-based framework. This is beyond the scope of the article. Most of the computer software use a cross-validation criterion to choose the best penalty parameter.

Consistency of the GREG and penalized GREG estimators in a high-dimensional setting

We adopt the asymptotic framework of [START_REF] Isaki | Survey design under the regression superpopulation model[END_REF] and consider an increasing sequence of embedded nite populations {U v } v∈N of size {N v } v∈N . In each nite population U v , a sample, of size n v , is selected according to a sampling design P v (S v ) with rst-order inclusion probabilities π i,v and second-order inclusion probabilities π i ,v . While the nite populations are considered to be embedded, we do not require this property to hold for the samples {S v } v∈N . This asymptotic framework assumes that v goes to in nity, so that both the nite population sizes {N v } v∈N , the samples sizes {n v } v∈N and the number of auxiliary variables {p v } v∈N go to in nity. To improve readability, we shall use the subscript v only in the quantities U v , N v , n v and p v ; for instance, quantities such as π i,v shall be simply denoted by π i .

The following assumptions are required to establish the consistency of the GREG and penalized GREG estimators in a high-dimensional setting.

(H1) We assume that there exists a positive constant

C 1 such that N -1 v i∈U v y 2 i < C 1 . (H2) We assume that lim v→∞ n v N v = π ∈ (0, 1).
(H3) There exist a positive constant c such that min

i∈U v π i c > 0; also, we assume that lim sup v→∞ n v max i ∈U v |π i -π i π | < ∞.
(H4) We assume that there exists a positive constant

C 2 such that, for all i ∈ U v , ||x i || 2 2 ≤ C 2 p v , where || • || 2 denotes the usual Euclidean norm. (H5) We assume that || β|| 1 = O p (p v )
, where β is the least square estimator given in (2.8) and || • || 1 denotes the L 1 norm.

The assumptions (H1), (H2) and (H3) were used by [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] in a nonparametric setting and similar assumptions were used by [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] to establish the consistency of the GREG estimator in a xed dimensional setting. These assumptions hold for many usual sampling designs such as simple random sampling without replacement, strati ed designs [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], or high-entropy sampling designs. Assumptions (H4) and (H5) can be viewed, respectively, as extensions of Assumption A.1 and Assumption A.3 in [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] to p v -dimensional vectors with p v growing to in nity. Assumption (H5) is not very restrictive in this high-dimensional setting as it requires that components of β are all bounded. When p v is xed, then our assumptions essentially reduce to those of [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF].

Result 2.3.1. Assume (H1)-(H5). Consider a sequence of GREG estimators { t greg } v∈N of t y . Then,

1 N v ( t greg -t y ) = O p p 3 v n v .
If the numbers of auxiliary variables {p v } v∈N and the sample sizes [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] is a special case of Result 3.1 with p v = O(1). Result 3.1 highlights the fact that the rate of convergence decreases as the number of auxiliary variables p v increases. Yet, this result guarantees the existence of a consistent GREG estimator, even when the number of auxiliary variables is allowed to diverge. An improved consistency rate for t greg may be obtained if, in (H5), the usual euclidean norm is used instead of L 1 -norm. Establishing the rate of convergence of the sampling error ββ may also be utilized to obtain a lower consistency rate for t greg ; e.g., see [START_REF] Chauvet | Asymptotic e ciency of the calibration estimator in a high-dimensional data setting[END_REF] .

{n v } v∈N satisfy p 3 v /n v = o(1), then N -1 v ( t greg -t y ) = o p (1). The √ n-consistency obtained by
The next result establishes the design-consistency of model-assisted penalized regression estimators. The proof is similar to that of Result 3.1 and is given in the Supplementary Material.

Result 2.3.2. Assume (H1)-(H5). Consider a sequence of penalized model-assisted estimators { t pen } v∈N of t y obtained by either ridge, lasso or elastic-net. Then,

1 N v ( t pen -t y ) = O p p 3 v n v .
If the numbers of auxiliary variables {p v } v∈N and the sample sizes

{n v } v∈N satisfy p 3 v /n v = o(1), then N -1 v ( t pen -t y ) = o p (1)
. The above result makes no use of the asymptotic convergence rate of β pen which depends on the penalization method. For example, if one can establish that ||

β pen || 1 = O p (γ v ), then N -1 v ( t pen -t y ) = O p (γ v p v /n v ).
Alternatively, improved consistency rates of t pen may be obtained if one can establish the magnitude of the sampling error β pen -β pen in a high-dimension setting. In other words, obtaining these improved rates requires additional assumptions, unlike Result 3.2 which is obtained under relatively mild assumptions.

Next, we show that, under additional assumptions on the auxiliary variables, the modelassisted ridge estimator is L 1 design-consistent for t y if p v /n goes to zero and that it has the usual

√ n-consistency rate if p v = O(n a v ) with 0 ≤ a < 1/2, which constitutes a signi cant improvement over Result 3.2.
Result 2.3.3. Assume (H1)-(H4). Also, assume that there exists a positive constant C such that

λ max (X U v X U v ) CN v , where λ max (X U v X U v ) is the largest eigenvalue of X U v X U v . Assume also that N v /λ v = O(1). 1. Then, there exists a positive constant C such that E p || β ridge || 2 2 C and 1 N v E p t ridge -t y = O p v n v .
If the numbers of auxiliary variables {p v } v∈N and the sample sizes

{n v } v∈N satisfy p v /n v = o(1), then N -1 v E p | t ridge -t y | = o(1). 2. E p (|| β ridge -βridge || 2 2 ) = O(p v /n v ). Thus, if p v /n v = o(1), then E p (|| β ridge -βridge || 2 2 ) = o(1).
3. We have the following asymptotic equivalence:

1 N v t ridge -t y = 1 N v t diff,˘-t y + O p p v n v ,
where

t diff,˘= i∈S v y i /π i - i∈S v x i /π i - i∈U v x i βridge and 1 N v E p t ridge -t y = O 1 √ n v + O p v n v . If p v = O(n a v ) with 0 ≤ a < 1/2, then 1 N v t ridge -t y = 1 N v t diff,˘-t y + o p (1) and 1 N v E p t ridge -t y = O 1 √ n v .
It follows from Result 3.3 that, for p v = O(n a v ) with 0 ≤ a < 1/2, the asymptotic variance of the model-assisted ridge estimator t ridge is equal to the variance of the generalized di erence estimator t diff,˘. For a = 1/2, we note that the model-assisted estimator is still √ n-design consistent but the remainder term is no longer negligible with respect to t diff,˘a nd the variability of this term should be consider to compute the asymptotic variance of t ridge . The case of modelassisted estimators based on lasso and elastic-net is more intricate. This is due to the fact that both estimators involve the L 1 -norm. As a result, a closed-form expression of these estimators cannot be obtained. However, if the predictors are orthogonal, a closed-form expression exists for the lasso and elastic-net estimators and improved consistency rates can be obtained; see Proposition 3.1 below. The case of non-orthogonal predictors is more challenging and is beyond the scope of this article.

Proposition 2.3.1. Suppose assumptions (H1)-(H3) and that the sampling design and the X-variables are such that the columns of Π -1/2 S v X S v are orthogonal. Suppose also that there exist positive quantities C 3 and C 4 such that max j=1,...,

p v N -1 v i∈U v x 4 i j ≤ C 3 < ∞ and min j=1,...,p v N -1 v i∈U v x 2 i j ≥ C 4 > 0. Then, N -1 v ( t greg -t y ) = O p ( p v /n v ) and N -1 v ( t pen -t y ) = O p ( p v /n v )
, where t pen denotes either the lasso or the elastic-net estimator.

Simulation study

In this section, we provide an empirical comparison of several model-assisted estimators. In addition to the estimators discussed in Section 2.3. In addition, we considered model-assisted estimators based on principal component regression [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF], regression trees [START_REF] Breiman | Classi cation and regression trees[END_REF], random forests [START_REF] Breiman | Random forests[END_REF], k-nearest neighbors, XGBoost [START_REF] Chen | XGBoost[END_REF] and Cubist [START_REF] Quinlan | Learning with continuous classes[END_REF]. For a description of these methods, see [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF] and Dagdoug et al. (2021a) and the references therein. We used data from the Irish Commission for Energy Regulation (CER) Smart Metering Project that was conducted in 2009-2010 (CER, 2011) 2 [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF]. This project focused on energy consumption and energy regulation. About 6000 smart meters were installed to collect the electricity consumption of Irish residential and business customers every half an hour over a period of about two years.

We considered a period of 14 consecutive days and a population of N = 6, 291 smart meters (households and companies). Each day consisted of 48 measurements, leading to 672 measurements for each household. We denote by X j = X(t j ), j = 1, . . . , 672, the electricity consumption (in kW) at instant t j and by x i j the value of X j recorded by the ith smart meter for i = 1, . . . , 6, 291. It should be noted that the matrix N -1 X X was ill-conditioned with a condition number equal to 254 753. This suggests that some of the X-variables were highly correlated with each other.

We generated four survey variables based on these auxiliary variables according to the following models: 0,1500);+ 3001 (X 5 > 200) + N (0, 1500);

Y 1 = 400 + 2X 1 + X 2 + 2X 3 + N (
Y 3 = 1 + cos(2X 1 + X 2 + 2X 3 ) 2 + 1 ; Y 4 = 4 + 3 • V {X 1 + X 2 } 2 -1/2 × {X 1 + X 2 } 2 + N (0, 0.01),
where V(•) denotes the empirical variance and the errors 1 in the model for Y 3 were generated from an E xp(10) and these errors were centered so as to obtain a mean equal to zero.

Our goal was to estimate the population totals t y j = i∈U y i j , j = 1, . . . , 4. From the population, we selected R = 2, 500 samples, of size n = 600, which corresponds to a sampling fraction n/N of about 10%. We considered three sampling schemes: simple random sampling without replacement, strati ed simple random sampling without replacement with optimal allocation and strati ed without replacement proportional to size sampling with proportional allocation.

In each sample, we computed twelve model-assisted estimators of the form

t ( j) ma = i∈U f ( j) (x i ) + i∈S y i -f ( j) (x i ) π i , j = 1, 2, . . . , 12,
where the predictors f ( j) (x i ), j = 1, 2, . . . , 12, were obtained using the following procedures:

Procedure 1: "LR" : Deterministic linear regression, leading to the GREG estimator.

Procedure 2: "CART": Classi cation and regression tree algorithm [START_REF] Breiman | Classi cation and regression trees[END_REF], leading to an estimator closely related to that of [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF] and implemented with the R-package rpart.

Procedure 3: "RF": Random forests with the algorithm of [START_REF] Breiman | Random forests[END_REF] with B = 1000 trees, a minimal number of elements in each terminal node n 0 = 5 and p 0 = √ p variables selected randomly at each split, where • denotes the customary oor function. The algorithm leads to the estimator described in [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF]. Simulations were implemented with the R-package ranger.

Procedure 4: "Ridge": Ridge regression with a regularization parameter determined by crossvalidation and implemented with the R-package glmnet. The estimator was studied by [START_REF] Goga | Overview of ridge regression estimators in survey sampling[END_REF].

Procedure 5: "Lasso": Lasso regression with a regularization parameter determined by crossvalidation and implemented with the R-package glmnet [START_REF] Mcconville | Model-assisted survey regression estimation with the lasso[END_REF].

Procedure 6: "EN": Elastic net regression with penalization coe cients determined by crossvalidation with the R-package glmnet.

Procedure 7: "XGB": XGBoost algorithm [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF] with 50 trees in the additive model, each tree being with a depth of at most 6 and a learning rate λ = 0.01. Simulations were implemented with the R-package XGBoost.

Procedure 8: "5NN": 5-nearest neighbors predictor with the euclidean distance and implemented with the R-package caret.

Procedure 9: "Cubist": A cubist algorithm [START_REF] Kuhn | Applied predictive modelling[END_REF] with 5 models in each predictor, implemented with the R-package cubist; the algorithm and its adaptation for survey data are described in Dagdoug et al. (2021a).

Procedure 10: "PCR1": Principal component regression based on the rst p 1/4 components kept and implemented with the R-package pls [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF].

Procedure 11: "PCR2": Principal component regression based on the rst p 2/4 components kept.

Procedure 12: "PCR3": Principal component regression based on the rst p 3/4 components kept.

As a measure of bias of the model-assisted estimators t ( j) ma , j = 1, 2, ..., 12, we computed the Monte Carlo percent relative bias de ned as

RB MC t ( j) ma = 100 × 1 R R r=1 ( t ( j,r) ma -t y ) t y , j = 1, 2, . . . , 12,
where t

( j,r) ma denotes the estimator t ( j) ma at the rth iteration, r = 1, . . . , R. As a measure of e ciency, we computed the relative of e ciency, using the Horvitz-Thompson estimator t π given by (2.1), as the reference. That is,

RE MC t ( j) ma = 100 × MSE MC ( t ( j) ma ) MSE MC ( t π ) , j = 1, 2, ..., 12,
where MSE MC ( t

( j) ma ) = R -1 R r=1 ( t ( j,r)
mat y ) 2 and MSE MC ( t π ) is de ned similarly. We were also interested in investigating to which extent the model-assisted estimators t ( j) ma , j = 1, . . . , 12 were a ected by the inclusion of a large number of predictors in the working models. To that end, in addition to the variables X 1 , . . . , X 5 , we included d noise predictors in the working models. These predictors were available in the Irish data set. We used the following values for d noise : 5, 10, 20, 50, 100, 200, 300 and 400.

Simple random sampling without replacement

In this section, we present the results obtained under simple random sampling without replacement (SRSWOR) of size n = 600. All the point estimators t ( j) ma , j = 1, . . . , 12, exhibited a negligible or small percent RB with a maximum value of about 3.1% (obtained in the case of the GREG estimator). For this reason, results pertaining to relative bias are not reported here.

Figures 4-7 display the relative e ciency of the model-assisted estimators t ( j) ma , j = 1, . . . , 12 as a function of the number of auxiliary variables incorporated in the working models. To improve readability, we have truncated some large values of RE, when applicable.

We begin by discussing the results on relative e ciency pertaining to the estimation of the total of the survey variable Y 1 . For low-dimensional settings, the GREG estimator was very e cient with values of RE below 10%. These results can be explained by the fact that Y 1 was linearly related to the x-variables. However, as the number of variables d noise increased, the e ciency of the GREG estimator rapidly deteriorated, suggesting that the performance of the GREG estimator is sensitive to the dimension of the x-vector. As expected, model-assisted estimators based on regularization methods such as ridge, lasso, elastic-net or dimension reduction methods such as principal components regression, performed generally very well. Unlike the GREG, these estimators were not much a ected by the number of auxiliary variables incorporated in the model. Turning to the model-assisted estimator based on a 5-nn, we note that it was less e cient than most competitors and that its e ciency got worse as d noise increased, a phenomenon referred to as the curse of dimensionality. The model-estimators based on XGBoost, Cubist and random forests performed quite well and did not seem to be a ected by the number of auxiliary variables incorporated in the model. Finally, the estimators based on CART were less e cient than those obtained through the other machine learning methods.

The results pertaining to the survey variable Y 2 and displayed in Figure 5 were fairly consistent with those obtained for the survey variable Y 1 with one exception: the Cubist algorithm was signi cantly more e cient than the other procedures in all the scenarios.

Turning to the survey variable Y 3 (see Figure 6), the model-assisted estimator based on random forests was signi cantly more e cient than the Horvitz-Thompson estimator, especially for large values of d noise . The other procedures led to estimators less e cient than the Horvitz-Thompson estimator with values of RE above 100. In particular, the GREG estimator broke down as the number of auxiliary variable increased. The performance of model-assisted estimators based on CART and XGBoost algorithms deteriorated as the dimension increased. In a high-dimension setting with highly correlated predictors, random forests improved over CART due to the random subsampling of p 0 variables among the p variables, generating then decorrelated trees [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF].

The results in Figure 7 about the survey variable Y 4 were similar to the ones in previous gures. Most estimators remained mostly una ected by the number of auxiliary variables d noise . Again, the model-assisted estimator based on the Cubist algorithm was the best in all the scenarios.

Strati ed simple random sampling with optimal allocation

In the second simulation study, we partitioned the Irish residential and business customer population into four strata U 1 , . . . , U 4 , using an equal quantile method with respect to the variable, X 1 , the electricity consumption at instant t 1 . From the population, we selected R = 2, 500 strati ed simple random samples, of size n = 600. The stratum sample sizes n h were determined using an X 2 -optimal allocation, where X 2 denotes the electricity consumption recorded at instant t 2 . This led to n 1 = 20, n 2 = 36, n 3 = 45 and n 4 = 499. The rst-order inclusion probabilities, π i = n h /N h , i ∈ U h and the sampling weights w i = π -1 i are shown in Table 1.

We con ned to the survey variables Y 1 and Y 3 only and we aimed at estimating t y 1 and t y 3 . It is worth pointing out that the resulting sampling design was informative as the variables used at the design stage (X 1 and X 2 ) were also related to the survey variables Y 1 and Y 3 . In fact, the Monte Carlo coe cient of correlation between the sampling weights and Y 1 was approximately equal to 0.402. We do not report the coe cient of correlation between the sampling weights and Y 3 as the relationship between Y 3 and the set of predictors X 1 , X 3 is not linear. Again, in each sample we computed twelve model-assisted estimators t ( j) ma , j = 1, . . . , 12 for each of t y 1 and t y 3 . Since most machine learning software packages do not take the sampling weights into account, we have included the design variables X 1 and X 2 in the set of predictors.

Stratum 1 2 3 4 π i 0.012 0.022 0.028 0.316 85 43.83 35.11 3.16 Table 1: First-order inclusion probabilities and sampling weights within strata.

w i = π -1 i 77.
We begin by discussing the results pertaining to the estimation of the total of the survey variable Y 1 . Figure 8 and Figure 9 display the Monte Carlo percent relative bias and the Monte Carlo relative e ciency as a function of the number of variables d noise . Except for the modelassisted estimators based on 5-nn and random forest, the other estimators exhibit a small value of RB for all values of d noise . Again, the 5-nn model-assisted estimator su ered from the curse of dimensionality. Turning to the estimator based on random forests, we note from Figure 8 that the bias increased as the number of predictors d noise increased. For instance, for d noise = 400, the value of RB was just above 10%. This signi cant bias may be explained by the fact that random forests is the only procedure among the ones considered in our simulation that randomly selects p 0 = √ p variables among the initial p predictors at each split. For instance, for d noise = 400, only 20 variables are randomly selected at each split. As a result, most predictions obtained through a random forests algorithm were based on misspeci ed working models, leading to potentially bad ts and large residuals. Also, each prediction corresponds to a weighted mean computed within each node with n 0 = 5 observations only. Therefore, each predictions corresponds to a ratio-type estimate based on 5 observations only. This, together with the fact that the sampling weights are highly variable, constitutes a conducive ground for the occurrence of small sample bias. In terms of e ciency, except for the GREG, the 5-nn and the random forest estimators, the other procedures performed well with values of RE ranging from 60% to 80%. The best procedures were Cubist and Lasso.

We now turn to the survey variable Y 3 . First, the Monte Carlo relative bias was negligible for all the estimation procedures and are not reported here. Results about relative e ciency are plotted in Figure 10. Random forests performed extremely well and their performance improved as d noise increased. This suggests that the method was able to extract the information contained in the predictors. This was also true for Cubist and XGBoost, although to a lesser extent.

To get a better understanding of the performance of random forests for the estimation of the total of the survey variable Y 1 , we conducted additional scenarios based on di erent values of the hyper parameters n 0 , the number of observations within each terminal nodes, and p 0 , the number of variables randomly selected at each split among the initial p model variables. We used the following values for n 0 and p 0 :

• n 0 = 5 observations and p 0 = √ p variables which are the default choices in the R-package ranger;

• n 0 = 5 observations and p 0 = p variables;

• n 0 = 5 observations and p 0 = √ p variables, with, in addition, the design variables X 1 , X 2 , as well as the vector of inclusion probabilities and the vector of strata that were selected with probability 1, at each split, besides the p 0 variables;

• n 0 = n 13/20 observations and p 0 = √ p variables. ma , j = 1, . . . , 12 for the estimation of the total of Y 1 with strati ed simple random sampling with X 2 -optimal allocation, n = 600 with increasing number of auxiliary variables Figure 9: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 1 with strati ed simple random sampling with X 2 -optimal allocation, n = 600 and increasing number of auxiliary variables

The Monte Carlo percent relative bias is displayed in Figure 11. We note that relative bias was much smaller (always less than 1%) when the design variables were considered besides p 0 variables at each split. To a lesser extent, the bias decreased when more observations ma , j = 1, . . . , 12 for the estimation of the total of Y 3 with strati ed simple random sampling with X 2 -optimal allocation, n=600 and increasing number of auxiliary variables were allowed in each terminal node. These results suggest, that, when the sampling design is informative, in order to avoid signi cant small sample bias, we recommend to force the design variables to be selected at each split. This option is available in the R package ranger.

Figure 11: Comparison of di erent con gurations of hyper-parameters for t r f for the estimation of the total of Y 1 with strati ed simple random sampling and X 2 -optimal allocation, n = 600.

Strati ed inclusion probability proportional-to-size sampling without replacement

We consider the strati ed population described in Section 2.4.2. In each stratum, we selected units according to a xed-size inclusion probability proportional-to-size sampling without replacement using X 2 , the electricity consumption at instant t = 2, as the size variable. In each stratum, we used the sample size n h were determined according to proportional allocation; i.e., n h = n • N h /N. The rst-order inclusion probabilities were then given by andh = 1, 2, 3, 4. As in Section 2.4.2, we focused on estimating t y 1 and t y 3 and we computed the same twelve model-assisted estimators t ( j) ma , j = 1, . . . , 12. The inclusion probabilities were highly correlated with the survey variable Y 1 , with a correlation coe cient of about 0.62; we do not report the coe cient of correlation in the case of Y 3 as the underlying relationship was nonlinear. Based on ndings from the Section 2.4.2, we adopted the following con guration for the random forest algorithm: we considered n 0 = 5 observations in each terminal node and, at each split, we randomly selected p 0 = √ p variables. Note that the design variables X 1 and X 2 as well as the vector of inclusion probabilities and the vector of stratum indicators were selected with probability 1 at each split in addition to the p 0 variables. All the estimators exhibited a negligible relative bias (less than 1%). Figure 12 and Figure 13 show the relative e ciency corresponding to t y1 t y3 , respectively. From Figure 12, we note that most estimators exhibited a behavior similar to that obtained in the case the strati ed simple random sampling based on an X 2 -optimal allocation (see Section 2.4.2). However, we note that the estimators PCR1 and PCR2 did poorly unlike in the case strati ed simple random sampling based on an X 2 -optimal allocation. This poor behaviour may be due to the fact that the sampling design was now much more informative and keeping a few principal components only may have led to a loss of information. The estimator PCR3 based on more principal components did better than PCR1 and PCR2. From Figure 13, we note that the use of model-assisted estimators led to signi cant improvement over the Horvitz-Thompson estimator, with value of relative e ciency ranging from 6% to 22%.

π i = n h x i2 j∈U h x j2 , i ∈ U h ,

Strati ed simple random sampling with proportional allocation

In this section, we consider a more realistic scenario based again on the Irish residential and business customer data. As a strati cation variable, we used the mean electricity consumption recorded during the rst week. Again, we constructed four strata using an equal-quantile method based, this time, on the mean electricity consumption; see also [START_REF] Cardot | Comparison of di erent sample designs and construction of con dence bands to estimate the mean of functional data: An illustration on electricity consumption[END_REF] who used a similar design. The mean trajectories during the rst week within each stratum are plotted in Figure 14. From Figure 14, we note that Stratum 1 corresponds to consumers with low global levels of electricity consumption, whereas Stratum 4 consists of consumers who have high levels of electricity consumption.

Our aim was to estimate the total electricity consumption recorded on the Monday of the second week and given by t y = 6291 i=1 384 j=336 y i j , where y i j is the electricity consumption recorded for the i-th unit at the j-th instant. Within each stratum, we selected a sample, of size n h , according to simple random sampling without replacement. The n h 's were determined according to proportional allocation; i.e, n h = n × (N h /N) with n = 600. In each of the 2,500 samples, we computed the same 12 model-assisted estimators as in the previous sections. Again, we computed the Monte Carlo percent relative bias and the relative e ciency for each the 12 estimators. The results are presented in Table 2. From Table 2, we note that the 5-nn model-assisted estimator was the only estimator to exhibit a non-negligible bias. Although it was less e cient than its competitors, it was more e cient than the Horvitz-Thompson estimator with a value of RE of about 65.6%. The ridge estimator was the most e cient with a value of RE equal to 4% and was closely followed by lasso, elastic-net, Cubist and principal components model-assisted estimators. The GREG estimator performed very well with a value of RE of about 9.3%. Random forests led to considerable improvement over the CART model-assisted estimator with values of RE of 17% and 41%, respectively. Still, random forests were less e cient than the GREG estimator, which is not surprising as the relationship between the survey variable and the auxiliary variables was linear.

Final remarks

In this paper, we have examined a number of model-assisted estimation procedures in a highdimensional setting both theoretically and empirically. If the relationship between the survey variable and the auxiliary information can be well described by a linear model, our results suggest that penalized estimators such as ridge, lasso and elastic net perform very well in terms of bias and e ciency, even in the case p = n. Model-assisted estimators based on random forests, Cubist and XGBoost methods were mostly una ected by the number of predictors incorporated in the working model, even in the case of complex relationships between the study and the auxiliary variables. As expected, the GREG estimator su ered from poor performances in the case of a large number of auxiliary variables.

The procedure Cubist stood out from the other machine learning procedure with very good performances in virtually all the scenarios. Further work is needed to establish the theoretical properties of model-assisted estimators based on Cubist in both a low-dimensional and high-dimensional settings.

Variance estimation is an important stage of the estimation process. Further research includes identifying the regularity conditions under which the variance estimators are designconsistent in a high-dimensional setting.

We end this article by mentioning that virtually all the machine learning software packages cannot handle design features such as unequal weights and strati cation. For instance, some random forests algorithms may involve a bootstrapping procedure and/or a cross-validation procedure. To fully account for the sampling design, both procedures must be modi ed so as to account for the design features. One notable exception is the R package RPMS [START_REF] Toth | rpms: Recursive Partitioning for Modeling Survey Data[END_REF] that has the ability to incorporate sampling weights for CART and random forests. Not fully accounting for the sampling design may be viewed as a form of model misspeci cation. However, model-assisted estimation procedures remain design-consistent even if the model is misspeci ed. In our experiments, several machine learning procedures (e.g., random forests, Cubist, XGboost) performed very well in most scenarios even though we did not modify the bootstrapping and cross-validation procedures to account for design features. In other words, it seems that, accounting for predictors that are highly predictive of the Y -variable, seems to be the preponderant factor with respect to the e ciency aspect of model-assisted estimators. We conjecture that fully accounting for the sampling design will likely lead to additional e ciency gains but that the predictive power of the model likely constitutes the "determining factor". Developing machine learning procedures that fully account for the sampling design is currently under investigation.

Supplementary material

Result 3.1. Assume (H1)-(H5). Consider a sequence of GREG estimators { t greg } v∈N of t y . Then,

1 N v ( t greg -t y ) = O p p 3 v n v .
If the numbers of auxiliary variables {p v } v∈N and the sample sizes

{n v } v∈N satisfy p 3 v /n v = o(1), then N -1 v ( t greg -t y ) = o p (1).
Proof. We adapt the proof of [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] to a high-dimensional setting. Let I i be the sample membership indicator for unit i such that

I i = 1 if i ∈ S and I i = 0, otherwise. Let α i := I i /π i -1 for all i ∈ U v .
We consider the following decomposition:

1 N v t greg -t y = 1 N v i∈U v α i y i - p v j=1 b j β j , (2.17)
where b j = 1 N v i∈U v α i x i j for j = 1, 2, ..., p v . Now, the rst term does not depend on the auxiliary information and we have (Breidt andOpsomer, 2000, Robinson and[START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF]):

E p 1 N v i∈U v α i y i 2 = 1 N 2 v i∈U y 2 i • E p (α 2 i ) + 1 N 2 v i∈U v ∈U v , i y i y • E p (α i α ).
(2.18)

We have

E p (α 2 i ) = (1 -π i )/π i 1/c and for i , E p (α i α ) = (π i -π i π )/π i π max i, ∈U v ,i |π i -π i π |/c 2
by Assumption (H3). It follows from (H1), (H2) and (H3) that

E p 1 N v i∈U v α i y i 2 1 cN 2 v i∈U v y 2 i + n v max i, ∈U v ,i |π i -π i π | c 2 n v N 2 v i∈U v ∈U v , i |y i y | 1 cN v + n v max i, ∈U v ,i |π i -π i π | c 2 n v 1 N v i∈U v y 2 i = O 1 n v (2.19)
and so,

1 N v i∈U v α i y i = O p 1 √ n v .
(2.20)

Now, consider the second term from the right-side of (2.17):

p v j=1 β j b j p v j=1 β 2 j p v j=1 b 2 j = || β|| 2 p v j=1 b 2 j || β|| 1 p v j=1 b 2 j .
(2.21)

By Assumption (H5), we have that || β|| 1 = O p (p v ). Furthermore,

p v j=1 b 2 j = 1 N v i∈U v α i x i 2 and 1 N 2 v E p i∈U v α i x i 2 2 = 1 N 2 v i∈U v ||x i || 2 2 E p (α 2 i ) + 1 N 2 v i∈U v i∈U v x i x E p (α i α ) 1 cN 2 v i∈U v ||x i || 2 2 + n v max i, ∈U v ,i |π i -π i π | c 2 n v N 2 v i∈U v i∈U v |x i x | 1 cN v + n v max i, ∈U v ,i |π i -π i π | c 2 n v 1 N v i∈U v ||x i || 2 2 = O p v n v , (2.22)
by Assumptions (H2)-(H4). It follows that

p v j=1 b 2 j = O p p v n v .
(2.23)

The result follows by using (2.17), (2.20), (2.21), (2.23) and Assumption (H5):

1 N v t greg -t y 1 N v i∈U v α i y i + p v j=1 β j b j = O p 1 √ n v + O p p 3 v n v = O p p 3 v n v .
Result 3.2. Assume (H1)-(H5). Consider a sequence of penalized model-assisted estimators { t pen } v∈N of t y obtained by either ridge, lasso or elastic-net. Then,

1 N v ( t pen -t y ) = O p p 3 v n v .
If the numbers of auxiliary variables {p v } v∈N and the sample sizes

{n v } v∈N satisfy p 3 v /n v = o(1), then N -1 v ( t pen -t y ) = o p (1).
Proof. From the proof of result (3.1), we only need to show that || βpen

|| 2 = O p (p v ) or || βpen || 1 = O p (p v )
, where βpen is one of the penalized regression coe cient: ridge, lasso and elastic-net. Consider rst the ridge regression coe cient, βridge . The ridge regression estimator has the advantage of having an explicit expression. We will show that

|| βridge || 2 < || β|| 2 for λ > 0. Let denote Tλ = X S v Π -1 S v X S v + λI p v = i∈S v x i x i π i + λI p v sample counterpart of T λ = X U v X U v + λI p v = i∈U v x i x i + λI p v . Moreover, let λ1 ≥ λ2 ≥ . . . ≥ λp v be the eigenvalues of i∈S v x i x i /π i
in decreasing order and vj the orthonormal corresponding eigenvectors, j = 1, . . . , p v . Then, the eigenvalues of the matrix Tλ are λ1 + λ ≥ λ2 + λ ≥ . . . ≥ λp v + λ ≥ λ > 0 with the same eigenvectors vj , j = 1, . . . , p v . Using the same arguments as those used in [START_REF] Hoerl | Ridge regression: biased estimation for nonorthogonal problems[END_REF], we obtain βridge =

p v j=1 ( λj + λ) -1 vj v j X S v Π -1 S v y S v and β = p v j=1 ( λj ) -1 vj v j X S v Π -1 S v y S v . Let denote by c j = v j X S v Π -1 S v y S v ∈ R, then || βridge || 2 2 = p v j=1 c 2 j ( λj + λ) 2 < || β|| 2 2 = p v j=1 c 2 j ( λj ) 2 for λ > 0. It follows that || βridge || 2 < || β|| 2 ≤ || β|| 1 = O p (p v ) and we get || βridge || 2 = O p (p v ).
We now consider the lasso regression estimator, βlasso , which minimizes the design-based version of the optimization problem given in (13) in the main article: βlasso = arg min

β∈R p i∈S v 1 π i (y i -x i β) 2 + λ||β|| 1 .
The lasso-estimator βlasso may be also obtained as the solution of a constrained optimization problem:

min β∈R p i∈S v 1 π i (y i -x i β) 2
under the constraint

||β|| 1 ≤ C,
for some small enough constant C > 0. If the ordinary least-square estimator β satis es the constraint, namely if || β|| 1 ≤ C, then the solution of the constrained optimization problem is β lasso = β; otherwise, if || β|| 1 > C, then the solution β lasso will be di erent from the least-square estimator β and ||

β lasso || 1 ≤ C < || β|| 1 . So, in both cases, we have || β lasso || 1 ≤ || β|| 1 = O p (p v ).
Finally, consider the elastic-net regression estimator, βen . Consider the following objective functions:

L ols (β) = i∈S v 1 π i (y i -x i β) 2 L en (β) = i∈S v 1 π i (y i -x i β) 2 + λ 1 ||β|| 1 + λ 2 ||β|| 2 2 = L ols (β) + λ 1 ||β|| 1 + λ 2 ||β|| 2 2 ,
where λ 1 = λα and λ 2 = λ(1 -α) with λ > 0 and α ∈ (0, 1). The cases α = 0 and α = 1 lead, respectively, to the ridge and lasso regression estimators which have been discussed above.

The ordinary least squares estimator β veri es β = arg min β∈R p L ols (β) and the elastic-net estimator veri es β en = arg min β∈R p L en (β). Since β minimizes L ols (β), we have L ols ( β)

L ols ( β en ). Similarly, we have L en ( β en ) L en ( β ols ). Therefore, the following inequalities hold:

L ols ( β) + λ 1 || β en || 1 + λ 2 || β en || 2 2 L ols ( β en ) + λ 1 || β en || 1 + λ 2 || β en || 2 2 = L en ( β en ) L ols ( β) + λ 1 || β|| 1 + λ 2 || β|| 2 2 = L en ( β ols ), which implies λ 1 || β en || 1 + λ 2 || β en || 2 2 λ 1 || β|| 1 + λ 2 || β|| 2 2 .
(2.24) Furthermore, since λ 1 > 0, we can write

λ 2 || β en || 2 2 λ 1 || β en || 1 + λ 2 || β en || 2 2 .
(2.25)

Using (2.24), (2.25) and the fact that || β|| 2 || β|| 1 , we obtain

λ 2 || β en || 2 2 λ 1 || β|| 1 + λ 2 || β|| 2 2 λ 1 || β|| 1 + λ 2 || β|| 2 1 which implies || β en || 2 2 α 1 -α || β|| 1 +|| β|| 2 1 = O p (p 2 v )
and so, ||

β en || 2 = O p (p v ).
Result 3.3. Assume (H1)-(H4). Also, assume that there exists a positive constant C such that

λ max (X U v X U v ) CN v , where λ max (X U v X U v ) is the largest eigenvalue of X U v X U v . Assume also that N v /λ v = O(1). 1. Then, there exists a positive constant C such that E p || β ridge || 2 2 C and 1 N v E p t ridge -t y = O p v n v .
If the numbers of auxiliary variables {p v } v∈N and the sample sizes

{n v } v∈N satisfy p v /n v = o(1), then N -1 v E p | t ridge -t y | = o(1). 2. E p (|| β ridge -βridge || 2 2 ) = O(p v /n v ). Thus, if p v /n v = o(1), then E p (|| β ridge -βridge || 2 2 ) = o(1).
3. We have the following asymptotic equivalence:

1 N v t ridge -t y = 1 N v t diff,˘-t y + O p p v n v ,
where

t diff,˘= i∈S v y i /π i - i∈S v x i /π i - i∈U v x i βridge and 1 N v E p t ridge -t y = O 1 √ n v + O p v n v . If p v = O(n a v ) with 0 ≤ a < 1/2, then 1 N v t ridge -t y = 1 N v t diff,˘-t y + o p (1) and 1 N v E p t ridge -t y = O 1 √ n v .
Proof.

1. As in the proof of result (3.2), we consider the eigenvalues of the matrix Tλ in decreasing order: λ1

+ λ ≥ λ2 + λ ≥ . . . ≥ λp v + λ ≥ λ > 0. The matrix Tλ is always invertible and the eigenvalues of T-1 λ are 0 < ( λ1 + λ) -1 ≤ ( λ2 + λ) -1 ≤ . . . ≤ ( λp v + λ) -1 ≤ λ -1 . We then obtain || T-1 λ || 2 λ -1 , ( 2
1 N 2 v i∈S v x i y i π i 2 2 = 1 N 2 v i∈U v ∈U v x i x y i I i π i y I π = 1 N 2 v Y X U v X U v Y 1 N v ||Y|| 2 2 1 N v ||X U v X U v || 2 ,
where Y = y i I i π i i∈U v . The symmetric and positive semi-de nite N v × N v matrix X U v X U v has the same non-null eigenvalues as those of the positive de nite

p v × p v matrix X U v X U v ,. Therefore, 1 N v ||X U v X U v || 2 = 1 N v λ max (X U v X U v ) C.
Using Assumptions (H1) and (H3), we have

1 N 2 v i∈S v x i y i π i 2 2 C N v ||Y|| 2 2 = C N v i∈U v y 2 i I i π 2 i C c 2 N v i∈U v y 2 i = O(1).
Finally, using also the fact that N v /λ = O(1), we have

|| β ridge || 2 2 || T-1 λ || 2 2 i∈S v x i y i π i 2 2 N 2 v λ -2 1 N 2 v i∈S v x i y i π i 2 2 = O(1).
It follows that

E p || β ridge || 2 2 = O(1).
(2.27)

To obtain the L 1 design-consistency of the ridge model-assisted estimator, we write as in the proof of Result 3.1:

1 N v t ridge -t y = 1 N v i∈U v α i y i - p v j=1 b j β j,ridge = 1 N v i∈U v α i y i - 1 N v i∈U v α i x i β ridge and E p 1 N v t ridge -t y E p 1 N v i∈U v α i y i + E p 1 N 2 v i∈U v α i x i 2 2 E p || β ridge || 2 2 = O 1 n v + O p v n v = O p v n v by (2.19), (2.22), (2.27).
2. We can write

β ridge -β ridge = T -1 λ i∈S v E iλ π i - i∈U v E iλ , (2.28)
where

E iλ = x i (y i -x i βridge ) with i∈U v E iλ = λI p v βridge .
Using the same arguments as those used in the proof of Result 3.1, we get

1 N 2 v E p i∈S v E iλ π i - i∈U v E iλ 2 2 1 cN v + n v max i, ∈U v ,i |π i -π i π | c 2 n v 1 N v i∈U v ||E iλ || 2 2 .
(2.29) Furthermore,

1 N v i∈U v ||E iλ || 2 2 2C 2 p v N v i∈U v y 2 i + i∈U v (x i βridge ) 2 = O(p v ) (2.30)
by Assumptions (H1) and (H4) and the fact that

1 N v i∈U v (x i βridge ) 2 = β ridge 1 N v i∈U v x i x i βridge || βridge || 2 2 1 N v ||X U v X U v || 2 = O(1).
(2.31)

2.6

To obtain the above inequality, we have also used the fact that || βridge || 2 = O(1) which can be proved by using the same arguments as the ones used for showing that || β ridge || 2 = O(1) in point (1). Expressions (2.29) and (2.30) lead to

1 N 2 v E p i∈S v E iλ π i - i∈U v E iλ 2 2 = O p v n v .
(2.32)

The result follows from (2.28), (2.32) and the fact that

||N v T-1 λ || 2 = O(1) : E p || β ridge -βridge || 2 2 = O p v n v .
(2.33)

3. We use the following decomposition:

1 N v t ridge -t y = 1 N v t diff,˘-t y - 1 N v i∈S v x i π i - i∈U v x i β ridge -βridge , and
1 N v t diff,˘-t y = 1 N v i∈S v y i π i - i∈U v y i - 1 N v i∈S v x i π i - i∈U v x i βridge = 1 N v i∈U v α i y i - 1 N v i∈U v α i x i βridge ,
where

α i = I i /π i -1, i ∈ U v . From (2.19), we have that N -2 v E p ( i∈U v α i y i ) 2 = O(n -1 v ) and we can get N -2 v E p i∈U v α i x i βridge 2 = O(n -1 v
) by using similar arguments as those used in the proof of Result 3.1 and (2.31). We obtain

1 N 2 v E p t diff,˘-t y 2 = O 1 n v .
The result follows since

1 N v E p t ridge -t y 1 N v E p t diff,˘-t y + 1 N 2 v E p i∈S v x i π i - i∈U v x i 2 2 E p β ridge -βridge 2 2 = O 1 √ n v + O p v n v
by using (2.22) and (2.33).

Proposition 3.1. Suppose assumptions (H1)-(H3) and that the sampling design and the X-variables are such that the columns of Π -1/2 S v X S v are orthogonal. Suppose also that there exist positive quantities C 3 and C 4 such that max j=1,...,p v N -1

v i∈U v x 4 i j ≤ C 3 < ∞ and min j=1,...,p v N -1 v i∈U v x 2 i j ≥ C 4 > 0. Then, N -1 v ( t greg -t y ) = O p ( p v /n v ) and N -1 v ( t pen -t y ) = O p ( p v /n v )
, where t pen denotes either the lasso or the elastic-net estimator.

Proof. From the proof of Result 3.1 (more speci cally, Equations 2.21 and 2.22), we need to show that

i∈U v ||x i || 2 2 /N v = O(p v ) and that || β|| 2 = O p (1)
. The same result holds for β lasso and β en . We have

i∈U v ||x i || 2 2 /N v = p v j=1 i∈U v x 2 i j /N v ≤ p v √ C 3 = O(p v )
under the assumption of uniformly bounded forth moment of X j , j = 1, . . . , p v .

We rst show that, under the assumed orthogonality condition, ||

β lasso || 2 ≤ || β|| 2 , || β en || 2 ≤ || β|| 2 and also || β|| 2 = O p (1).
Consider again the objective function L ols (β) as in the proof of Result 3.2. We can write

L ols (β) = i∈S v 1 π i (y i -x i β) 2 = i∈S v ( ỹi -x i β) 2 (2.34)
where ỹi = y i / √ π i and xi = ( xij )

p v j=1 = x i / √ π i for all i ∈ S v . Let XS v = Π -1/2 S v X S v = (x i ) i∈S v = ( X1 , . . . , Xp v ).
The columns of XS v , denoted by Xj , j = 1, . . . , p v are assumed to be orthogonal. This means that X j Xk = 0 for j k. The ordinary least-square estimator β is given by

β = ( X S v XS v ) -1 X S v ỹS v .
Under the orthogonality condition, X S v XS v is a diagonal matrix with diagonal elements given by

|| Xj || 2 2 = i∈S v x2 i j = i∈S v x 2 i j
π i , which corresponds to the Horvitz-Thompson estimator of i∈U v x 2 i j . Therefore, β = ( βj ) j∈S v and the j-th coordinate is given by βj = ( i∈S v x2 i j ) -1 i∈S v xij ỹi . The lasso estimator βlasso = ( βj,lasso ) p v j=1 as well as the elastic-net estimator βen = ( βj,en )

p v j=1
are obtained by using the cyclic soft-thresholding algorithm [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]:

βj,lasso = S λ ( i∈S v r i j xij ) i∈S v x2 i j and βj,en = S λα ( i∈S v r i j xij ) n v i=1 x2 i j + λ(1 -α)
,

where r i j = ỹi -k j xik βk and S λ (z) = sign(z)(|z| -λ) + is the soft-thresholding function with (|z| -λ) + = |z| -λ if |z| ≥ λ, and zero otherwise. If the columns of XS v are orthogonal, then i∈S v r i j xij = i∈S v xij ỹi
and βj,lasso is the soft-threshold estimator of the least-square estimator β j :

βj,lasso = S λ ( i∈S v xij ỹi ) i∈S v x2 i j
.

The elastic-net estimator is given by

βj,en = S λα ( i∈S v xij ỹi ) i∈S v x2 i j + λ(1 -α) . It follows that | βj,lasso | = |(| i∈S v xij ỹi | -λ) + | i∈S v x2 i j ≤ | i∈S v xij ỹi | i∈S v x2 i j = | βj |, j = 1, . . . , p v and || β lasso || 2 ≤ || β|| 2 . Similarly, || β en || 2 ≤ || β|| 2 .
We now show that || β|| 2 = O p (1). We have

|| β|| 2 ≤ ||N v ( X S v XS v ) -1 || 2 1 N v X S v ỹS v 2 . The matrix X S v XS v is diagonal with diagonal elements equal to i∈S v x 2 i j π i . Then, ||N v ( X S v XS v ) -1 || 2 = max j=1,...,p v 1 N -1 v i∈S v x 2 i j π i
and for all j = 1, . . . , p v :

1

N -1 v i∈S v x 2 i j π i = 1 N -1 v i∈U v x 2 i j + O p 1 √ n v = O p (1)
by using (H2), (H3) and the assumption of uniformly bounded fourth moment of X j , j = 1, . . . , p v . We have also used the fact that 1/(N -1

v i∈U v x 2 i j ) ≤ 1/(min j=1,...,p v N -1 v i∈U v x 2 i j ) ≤ 1/C 4 = O(1) for all j = 1, . . . , p v . Then, ||N v ( X S v XS v ) -1 || 2 = O p (1). (2.35) Now, 1 N v X S v ỹS v 2 2 ≤ 1 N v ỹS v 2 2 1 N v XS v X S v 2 .
We have

1 N v XS v X S v 2 = 1 N v X S v XS v 2 = max j=1,...,p v 1 N v i∈S v x 2 i j π i ≤ max j=1,...,p v 1 N v i∈U v x 2 i j ≤ C 3 and 1 N v || ỹS v || 2 2 = 1 N v i∈S v y 2 i π 2 i ≤ 1 c 2 N v i∈U v y 2 i ≤ C 1 c 2 by Assumption (H1). So, || 1 N v XS v ỹS v || 2 = O(1)
and combined with (2.35), we obtain || β|| 2 = O p (1).

Introduction

Since the pioneering work of [START_REF] Särndal | On pi-inverse weighting versus best linear unbiased weighting in probability sampling[END_REF], [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] and [START_REF] Särndal | Cosmetic form of estimators in survey sampling[END_REF], model-assisted estimation procedures have attracted a lot of attention in the literature; see also [START_REF] Särndal | Model assisted survey sampling[END_REF] for a comprehensive discussion of the modelassisted approach. At the estimation stage, auxiliary information is often available and can be incorporated in the estimation procedures to increase the precision of the resulting point estimators. The model-assisted approach starts with postulating a working model, describing the relationship between a survey variable Y and a set of p auxiliary variables X 1 , X 2 , . . . , X p . The model is tted to the sample observations to obtain predicted values, which then serve to build point estimators of population means/totals. Model-assisted estimators are asymptotically design-unbiased and design consistent, irrespective of whether or not the working model is correctly speci ed, which is an attractive feature; see [START_REF] Särndal | Model assisted survey sampling[END_REF] and [START_REF] Breidt | Model-assisted survey estimation with modern prediction techniques[END_REF], among others. When the working model holds, model-assisted estimators are expected to be highly e cient. However, when the sample size is small, the use of model-assisted estimators requires some caution as they may su er from small sample bias. In this article, we use random forests to estimate the functional relationship between Y and X 1 , X 2 , . . . , X p . In recent years, random forests have become attractive as National Statistical O ces have now access to a variety of data sources, potentially exhibiting a large number of observations on a large number of variables.

Consider a nite population U = {1, ..., k, ..., N } of size N. We are interested in estimating the population total of a survey variable Y , t y = k∈U y k . We select a sample S, of size n, according to a sampling design P(S | Z U ), where Z U denotes the matrix of design information, available prior to sampling for all the population units. Let I U = (I 1 , ..., I k , ..., I N ) be the N-vector of sample selection indicators such that I k = 1 if k ∈ S and I k = 0, otherwise. The rst-order and second-order inclusion probabilities are given by π

k = E [I k | Z U ] and π kl = E [I k I | Z U ] , respectively.
A basic estimator of t y is the well-known Horvitz-Thompson estimator given by

t π = k∈S y k π k . (3.1)
Provided that π k > 0 for all k ∈ U, the estimator (3.1) is design-unbiased for t y in the sense E t π | y U , Z U = t y , where y U = (y 1 , y 2 , ..., y N ) . The Horvitz-Thompson estimator makes no use of auxiliary information beyond what is already contained in the matrix Z U . We assume that a vector x k = (x k1 , x k2 , . . . , x k p ) of auxiliary variables is available for all k ∈ U. We also assume that y k , k ∈ U, are independent realizations from a working model ξ, often referred to as a superpopulation model:

E [y k | X k = x k ] = m(x k ), (3.2) V(y k | X k = x k ) = σ 2 ν(x k ),
where m(•) and ν(•) are two unknown functions and σ 2 is an unknown parameter. Suppose that Model (3.2) is tted at the population level and let m(x k ) be the populationlevel t associated with unit k obtained by tting a parametric or nonparametric procedure. This leads to the pseudo generalized di erence estimator

t pgd = k∈U m(x k ) + k∈S y k -m(x k ) π k . (3.3)
Because the values m(x k ) do not involve the sample selection indicators I 1 , . . . , I N , if follows that E t pgd | y U , Z U , X U = t y , where X U is the N × p matrix whose N rows are the vectors x 1 , . . . , x N . That is, the pseudo generalized di erence estimator (3.3) is design-unbiased for t y . In the sequel, we use the simpler notation

E p [•] instead of E [•|Z U , X U , y U ]
to denote the expectation operator with respect to the sampling design P(S|Z U ). Similarly, the notation V p [•] is used to denote the design variance of an estimator. Most often, the estimator (3.3) is unfeasible as the population-level ts m(x k ) are unknown. Using the sample observations, we t the working model and obtain the sample-level ts m(x k ). Replacing m(x k ) with m(x k ) in (3.3), we obtain the so-called model-assisted estimator of t y :

t ma = k∈U m(x k ) + k∈S y k -m(x k ) π k . (3.4) Unlike (3.
3), the estimator (3.4) is no longer design-unbiased, but can be shown to be designconsistent for t y for a relatively wide class of procedures m(•). The model-assisted estimator (3.4) is expressed as the sum of the population total of the predictions m(x k ) and an adjustment term that can be viewed as a protection against model-misspeci cation.

If m(x k ) = x k β with coe cients estimated by weighted least squares, the estimator (3.4) reduces to the well-known generalized regression (GREG) estimator; e.g., see Särndal et al. (1992, Chap. 6). Model-assisted estimators based on generalized linear models were considered by [START_REF] Lehtonen | Logistic generalized regression estimators[END_REF] and [START_REF] Firth | Robust models in probability sampling[END_REF], among others. There are some practical issues associated with the use of a parametric model such as linear and generalized linear models: they may lead to ine cient estimators if the function m(•) is misspeci ed or if the model fails to include interactions or predictors that account for curvature (e.g., quadratic and cubic terms). In contrast, nonparametric procedures are robust to model misspe ciation, which is a desirable property. A number of nonparametric model-assisted estimation procedures have been studied in the last two decades: local polynomial regression [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], B-splines [START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF] and penalized B-splines [START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF], penalized splines (Breidt et al., 2005, McConville and[START_REF] Mcconville | Survey design asymptotics for the model-assisted penalised spline regression estimator[END_REF], neural nets [START_REF] Montanari | Nonparametric model calibration in survey sampling[END_REF], generalized additive models [START_REF] Opsomer | Model-assisted estimation of forest resources with generalized additive models[END_REF], nonparametric additive models (Wang and Wang, 2011) and regression trees (McConville and[START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF][START_REF] Toth | Building consistent regression trees from complex sample data[END_REF].

In this paper, we propose a new class of model-assisted estimators of t y based on random forests (RF). Generally speaking, RF is an ensemble method that trains a (large) number of trees and combines them to produce more accurate predictions than a single regression tree would. Trees de ne a class of algorithms that recursively split the p-dimensional predictor space into distinct and non-overlapping regions. In other words, a tree algorithm generates a partition of regions or hyperrectangles of R p . For an observation belonging to a given region, the prediction is simply obtained by averaging the y-values associated with the units belonging to the same region. While regression trees are easy to interpret and allow the user to visualize the partition (Hastie et al., 2011, pp. 306), they may su er from a high model variance, hence their quali cation of "weak learners". A number of tree-based procedures have been proposed with the aim of improving the predictive performances of regression trees, including pruning [START_REF] Breiman | Classi cation and regression trees[END_REF], Bayesian regression trees [START_REF] Chipman | Bayesian CART model search[END_REF], gradient boosting [START_REF] Friedman | Greedy function approximation: A gradient boosting machine[END_REF] and RF [START_REF] Breiman | Random forests[END_REF].

Several empirical studies suggest that RF can outperform state-of-the-art prediction models; see e.g. [START_REF] Han | Comparison of random forest, arti cial neural networks and support vector machine for intelligent diagnosis of rotating machinery[END_REF], [START_REF] Hamza | An empirical comparison of ensemble methods based on classi cation trees[END_REF], Díaz-Uriarte and de Andrés (2006). RF are widely used due to their predictive performances and their ability to handle small sample sizes with a large number of predictors [START_REF] Scornet | Random forests and kernel methods[END_REF]. Also, RF algorithms can be parallelized, leading to a decrease in the training time. RF have been applied in a wide variety of elds, including medicine [START_REF] Fraiwan | Automated sleep stage identi cation system based on time-frequency analysis of a single EEG channel and random forest classi er[END_REF], time series analysis [START_REF] Kane | Comparison of arima and random forest time series models for prediction of avian in uenza h5n1 outbreaks[END_REF], agriculture [START_REF] Grimm | Soil organic carbon concentrations and stocks on barro colorado island -digital soil mapping using random forests analysis[END_REF], missing data [START_REF] Stekhoven | MissForest-non-parametric missing value imputation for mixed-type data[END_REF], genomics [START_REF] Qi | Random forests for bioinformatics[END_REF] and pattern recognition [START_REF] Rogez | Randomized trees for human pose detection[END_REF]. In recent years, neural networks and deep learning algorithms have attracted a lot of attention and have been shown to be e ective in a wide range of applications involving mostly unstructured data, such as speech recognition, image reconstruction and text translation; see [START_REF] Najafabadi | Deep learning applications and challenges in big data analytics[END_REF] and the references therein for a review on the topic. However, to exhibit high levels of performance, deep learning algorithms typically require huge amounts of data [START_REF] Arnould | Analyzing the tree-layer structure of deep forests[END_REF][START_REF] Najafabadi | Deep learning applications and challenges in big data analytics[END_REF]. This is seldom the case in surveys as most data sets consist of structured data consisting of (at most) a few hundred thousand observations and a few hundred survey variables. For an empirical comparison of RF and neural networks, see [START_REF] Han | Comparison of random forest, arti cial neural networks and support vector machine for intelligent diagnosis of rotating machinery[END_REF]. Finally, unlike RF algorithms that require the speci cation of a small number of hyper-parameters (see Section 3.6.3), gradient boosting, Bayesian regression trees or deep learning approaches depend upon the complex choice of a large number of hyper-parameters [START_REF] Bergstra | Algorithms for hyper-parameter optimization[END_REF].

To the best of our knowledge, only little is known about the theoretical properties of RF based on the original algorithm of [START_REF] Breiman | Random forests[END_REF]. Often, the theoretical investigations are made at the expense of simplifying assumptions; see for instance [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF] and [START_REF] Biau | Analysis of a random forests model[END_REF]. Two notable exceptions are [START_REF] Wager | Asymptotic theory for random forests[END_REF] and [START_REF] Scornet | Consistency of random forests[END_REF] who established the theoretical properties of an algorithm closely related to that of [START_REF] Breiman | Random forests[END_REF]. In a nite population setting, the theoretical properties of RF algorithms have yet to be established, even in the ideal situation of 100% response. This paper aims to ll this important gap. While we are mostly concerned with RF for regression, we can easily extend our methods to the case of RF for classi cation. Some recent empirical studies on the performance of RF for complex survey data can be found in [START_REF] Tipton | Properties of endogenous post-strati ed estimation using remote sensing data[END_REF], [START_REF] Buskirk | Finding respondents in the forest: A comparison of logistic regression and random forest models for response propensity weighting and strati cation[END_REF], De Moliner and Goga (2018) and [START_REF] Kern | Tree-based machine learning methods for survey research[END_REF].

The rest of the paper is organized as follows. Regression trees and RF are presented in Section 3.2. In Section 3.3, we suggest two classes of model-assisted estimators based on random forests: the rst is based on partitions built at the population level, while the second class is based on partitions built at the sample level. In Section 3.4, we establish the theoretical properties of model-assisted estimators based on RF and derive corresponding variance estimators. In Section 3.5, we describe a model-calibration procedure for handling multiple survey variables. In Sections 3.6.1-3.6.3, the nite sample properties of the proposed point and variance estimation procedures are evaluated through a simulation study, and in Section 3.6.4, we apply the proposed methods using data on radio audiences collected by Médiamétrie, a French audience company. The paper ends with some nal remarks in Section 3.7. Proofs of major results and further technical details are relegated to the Appendix and the Supplementary Material.

Regression trees and random forests

Regression trees

The original RF uses regression trees based on the classi cation and regression tree algorithm (CART) of [START_REF] Breiman | Classi cation and regression trees[END_REF], whereby the partition of the predictor space is generated by a greedy recursive algorithm. In this paper, we focus on the CART algorithm for regression, designed for handling quantitative survey variables Y , but our methods also applies to the case of binary survey variables. With regression trees, these estimated probabilities always lie between 0 and 1, which is a desirable feature. Alternative criteria may be used with binary variables, such as the Gini impurity or the entropy instead of the CART regression criterion (Hastie et al., 2011, Chapter 9). The CART algorithm for regression searches for the splitting variable and the splitting position (i.e., the coordinates on the predictor space where to split) for which the di erence in empirical variance in the node before and after splitting is maximized. As a starting point, we consider the hypothetical situation, where y k and x k are observed for all k ∈ U and assume that the regression tree is tted at the population level. We use the generic notation A to denote a node with cardinality #(A) considered for the next split, and C A to denote the set of possible splits in the node A, which corresponds to the set of all possible pairs ( j, z) = (variable, position). This splitting process is performed by searching for the best split ( j * , z * ) for which the following empirical CART population criterion is maximized:

L N ( j, z) = 1 #(A) k∈U 1 x k ∈A (y k -ȳA ) 2 -y k -ȳA L 1 x k j <z -ȳA R 1 x k j ≥z 2 , (3.5)
where

A L = k ∈ A; x k j < z , A R = k ∈ A;
x k j z and ȳA is the average of the y-values of units belonging to A. The best cut is always performed in the middle of two consecutive data points. In practice, it is common to impose a minimal number of observations N 0 (say) in each terminal node. In this case, the splitting process is performed until an additional split generates a terminal node with fewer observations than N 0 . The splitting process leads to the set

P U = A (U) 1 , . . . , A (U) j , . . . , A (U) J U (3.6) of J U hyperrectangles of R p such that A (U) j A (U) j = ∅
, for all j j ∈ {1, 2, . . . , J U } and

J U j=1 A (U) j = R p .
Thus, the set P U de nes a partition of R p , whose elements are called the terminal nodes. We use the generic notation A (U) (x k ) to denote a terminal node belonging to the partition P U given in (3.6) and that contains x k .

Figure 15 below illustrates how the recursive splitting procedure creates a partition in the simple case of two auxiliary variables X 1 and X 2 , based on 5 splits. Each grey rotated square represents a split (variable, position) performed at some position along one of the two auxiliary variables, X 1 or X 2 . The white ellipses represent the 6 terminal nodes, also represented by the scatter plot on the right; see also [START_REF] Biau | Cellular tree classi ers[END_REF] for a similar illustration. The prediction m tree (x k ) at the point x k is simply de ned as the average of the y-values of population individuals such that x belongs to A (U) (x k ):

m tree (x k ) = ∈U 1 x ∈A (U) (x k ) y N(x k ) , (3.7)
where N(x k ) = ∈U 1 x ∈A (U) (x k ) denotes,the number of units belonging to the terminal node A (U) (x k ). Given the partition P U , the population-level t m tree (x k ) may be viewed as the least squares type prediction obtained by tting a one-way ANOVA model with Y as the response variable and the node membership indicators {1 x k ∈A (U) j } J U j=1 as the set of explanatory variables; see (Hastie et al., 2011, Chapter 9) and the Supplementary Material for more details.

Random forests

To introduce random forests (RF) in a nite population setting, we again assume that y k and x k are observed for all k ∈ U. RF are based on a (large) number B (say) of regression trees. The prediction attached to unit k is de ned as the average of the predictions produced by each of the B regression trees. That is,

m r f (x k ) = 1 B B b=1 m (b) tree (x k ),
where m (b) tree (x k ) is the predicted value attached to unit k obtained from the bth regression tree, b = 1, . . . , B.

Obviously

, if m (1) tree (x k ) = . . . = m (B) tree (x k ), then m r f (x k ) = m (1)
tree (x k ). Such a situation would occur if each regression tree uses a deterministic splitting criterion in (3.5), which would lead to B identical partitions of R p . To cope with this issue, some amount of randomization is introduced in the tree building process, leading to B di erent predictions of m(•). The original algorithm of [START_REF] Breiman | Random forests[END_REF] is implemented as follows:

1. Select B bootstrap data sets with replacement from the population data set,

D U = {(x k , y k )} k∈U , each data set containing N pairs of the form (x k , y k );
2. Fit a regression tree on each bootstrap data set. Before each split is performed, m tr y predictors are selected randomly and without replacement from the full set of p predictors. The m tr y selected predictors are the split candidates to be considered for searching the best split in (3.5).

The algorithm stops when each terminal node contains less than a predetermined number of observations. This procedure leads to a set P U = P (1) U , P (2) U , . . . , P (B) U of B di erent partitions of R p , each of the form (3.6). The randomization used in the tree building process is denoted by the random variable θ (U) , assumed to belong to some measurable space (Θ, F ) and independent of the data [START_REF] Biau | A random forest guided tour[END_REF]. Let θ (U) b be the random variable associated with the bth tree. The random variables θ (U) b , b = 1, . . . , B, are assumed to be independent and their distribution is identical to that of the generic random variable θ (U) . In the RF algorithm of Breiman, the randomization is induced by the selection (with replacement) of observations in Step 1 of the above algorithm and the random selection of split variables in Step 2 of the above algorithm. A number of RF algorithms algorithms have been considered in the literature. For example, [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF], Scornet, 2016a) considered a simple RF algorithm called the uniform random forest (URF) algorithm. In the URF algorithm, a variable is selected with equal probability among the initial p predictors at each node and a split position is chosen uniformly in the node along the direction of the selected variable. The algorithm stops when each terminal node has a predetermined number of cuts. In this case, the randomization θ (U) b is characterized by the random selections of the node, the split variable and the location. For more details on RF algorithms, the reader is referred to [START_REF] Geurts | Extremely randomized trees[END_REF], [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF], [START_REF] Biau | Analysis of a random forests model[END_REF], [START_REF] Genuer | Variance reduction in purely random forests[END_REF], Scornet (2016a), among others. In the sequel, unless stated otherwise, we assume that the observations in Step 1 of the above algorithm are selected without replacement [START_REF] Scornet | Tuning parameters in random forests[END_REF], which we will refer to as subsampling. Also, for more generality, the splitting criterion is left unspeci ed.

3.3

: 90

Let m (1) tree (•, θ (U) 1 ), . . . , m (B) tree (•, θ (U) B ), denote the predictions obtained with the B stochastic or randomized regression trees. The RF prediction attached to unit k is de ned as a bagged estimator of B trees:

m r f (x k ) = 1 B B b=1 m (b) tree (x k , θ (U) b ). (3.8)
It is worth pointing out that considering a new set of predictors at each split leads to B trees which are less correlated with each other; that is, trees that are quite di erent from one another. As a result, the RF may lead to substantial gains in precision compared to a single tree (James et al., 2015, Chapter 8). The number of predictors selected at each split, denoted by m tr y , is thus an important tuning parameter in the RF algorithm. In practice, the choice m tr y = √ p seems to give good results, in general. In Section 3.6.3, we assess the impact of m tr y through a simulation study.

For any RF algorithm, the prediction at the point x k in (3.8) can also be expressed as

m r f (x k ) = ∈U W (x k )y , (3.9)
where

W (x k ) = 1 B B b=1 ψ (b,U) 1 x ∈A (U) x k ,θ (U) b N(x k , θ (U) b ) (3.10) is a prediction weight attached to unit k with N(x k , θ (U) b ) = ∈U ψ (b,U) 1 x ∈A (U) (x k ,θ (U) b
) denoting the number of observations belonging to the terminal node A (U) containing x k in the bth regression tree. The random variables ψ (b,U) in (3.10) depend on the resampling mechanism used in the RF algorithm and depend on θ (U) b , but are independent of the sampling design P(S | Z U ). In the case of subsampling, the random variables ψ (b,U) follow a Bernoulli distribution, ψ (b,U) ∼ B (N /N), where N denotes the number of units in each subsample. Note that the prediction m r f in (3.9) can be computed for either a continuous or a categorical y-variable. In the latter case, the prediction m r f in (3.9) corresponds to the population proportion of units who belong to a given category computed over the B trees.

Proposition 3.2.1. Consider the predictor weights W (x k ) given in (3.10).

i) The weights W (x k ) are uniformly bounded. That is,

0 < W (x k ) cN -1 0 for all ∈ U and all x k ∈ R p ,
where c is a positive constant that does not depend either on k, , or N 0 , the minimal number of observations in the terminal nodes.

ii) The weight functions sum up to one; that is, ∈U W (x k ) = 1 for all x k ∈ R p .

The proof of Proposition 3.2.1 is given in the Appendix.

Model-assisted estimation: Random forests

In Section 3.2, we assumed that y k and x k were observed for all k ∈ U, which led to the population-level ts m tree (x k ) and m r f (x k ) given by (3.7) and (3.9), respectively. However,

3.3

: 91 both (3.7) and (3.9) cannot be computed in practice as the y-values are observed only for k ∈ S. Moreover, the regression trees in Sections 3.2.1 and 3.2.2 were based on partitions built recursively at the population level so as to optimize the population criterion (3.5). As a result, these partitions depend on the vector of predictors {x k } k∈U but also on the unknown population values {y k } k∈U . While the former type of dependency is inherent to most parametric and nonparametric procedures, the latter is absent in many commonly used parametric and nonparametric procedures such as spline procedures [START_REF] Breidt | Model-assisted estimation for complex surveys using penalized splines[END_REF][START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF][START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF][START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF][START_REF] Mcconville | Survey design asymptotics for the model-assisted penalised spline regression estimator[END_REF]. Due to the dependency on the unknown population values {y k } k∈U , establishing the theoretical properties of model-assisted estimators based on RF is more challenging. For these reasons, in Section 3.3.1, we start by considering the simpler case of population partitions obtained using a variable Y * , assumed to be closely related to Y and available for all k ∈ U. While this assumption is somehow strong and not tenable in many practical situations, it provides some insights on how to tackle the problem in the presence of Y -dependency. Algorithms allowing to get rid of the Y -dependency have been suggested in the random-forest literature; see e.g. [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF], [START_REF] Biau | Analysis of a random forests model[END_REF] or Devroye et al. (2013, Chap. 20). Sample-based partitions are considered in Section 3.3.2.

Model-assisted estimation: Population-based partitions

In this section, we consider the case of a splitting criterion that does not depend on the data {y k } k∈s . We consider a variable Y * assumed to be closely related to Y and such that the values y * k are available for all k ∈ U. We seek population partitions P * U , independent of the survey variable Y , that maximize the following criterion:

L * N ( j, z) = 1 #(A) k∈U 1 x k ∈A y * k -y * A 2 -y * k -y * A L 1 x k j <z -y * A R 1 x k j ≥z 2 , (3.11)
where A R , A L are as in (3.5) and y * A is the average of the y * -values for the units belonging to a node A.

Based on (3.11), the population-level t at the point x k is given by

m * r f (x k ) = ∈U W * (x k )y , (3.12)
where the weights W * (x k ) in (3.12) are obtained from (3.10) by replacing A (U) with A * (U) , a generic member of the partition P * U . The weights { W * (•)} ∈U in (3.12) are known for all ∈ U and are independent of Y . Since m * r f (x k ) in (3.12) requires the y-values for all the population units, it cannot be computed. A simple solution consists of replacing the population total on the right hand-side of (3.12) by its corresponding Horvitz-Thompson estimator, which leads to

m * r f (x k ) = ∈S W * (x k )y π .
(3.13)

3.3

: 92

A model-assisted estimator of t y based on population RF is obtained by plugging m * r f (x k ) in (3.3):

t * r f = k∈U m * r f (x k ) + k∈S y k -m * r f (x k ) π k . (3.14)
Proposition 3.3.1. The RF estimator given in (3.14) can be expressed as

t * r f = k∈S w ks y k ,
where the weights w ks are given by

w ks = 1 π k 1 + ∈U W * k (x ) 1 - I π . k ∈ S (3.15)
Proof. By rearranging the sums, we get:

t * r f = k ∈S y k π k + ∈U 1 - I π m * r f (x ) = k ∈S y k π k + ∈U 1 - I π k ∈S W * k (x )
y k π k = k ∈S 1 + ∈U 1 - I π W * k (x ) y k π k .
Since the partitions P * U , are independent of both the survey variable Y and the sample S, the weights w ks given by (3.15) depend on the sample only through the sample selection indicators I , ∈ U, but are independent of Y . As a result, these weights may be used to estimate the population total of any survey variable, which is an attractive feature in multipurpose surveys. However, for RF algorithms based on the splitting criterion in (3.11), we expect the weights w ks to be e cient whenever the survey variable Y is highly correlated to the variable Y * . In multipurpose surveys where the survey variables are not necessarily correlated with one another, it may be preferable to use a splitting criterion that depends on the data {x k } k∈U as done in quantile random forests [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF], Scornet, 2016a).

Model-assisted estimation: Sample-based partitions

In this section, we seek sample partitions P S = P (1) S , . . . , P (b) S , . . . , P (B)

S using L n ( j, z) = 1 #(A) k∈S 1 x k ∈A (y k -ȳA ) 2 -y k -ȳA L 1 x k j <z -ȳA R 1 x k j ≥z 2 .
(3.16)

Based on the partition P S , we obtain the sample-level ts

m r f (x k ) = ∈S W (x k )y π , ( 3
.17)

3.3

: 93

where

W (x k ) = 1 B B b=1
ψ (b,S) 1

x ∈A (S) x k ,θ (3.18) and N(x k , θ (S) b ) = ∈U I π -1 ψ (b,S) 1

(S) b N(x k , θ (S) b ) , ∈ S,
x ∈A (S) x k ,θ

(S) b
denotes the estimated number of observations in the terminal node A (S) containing x k in the bth regression tree. The variable ψ (b,S) indicates whether or not unit has been selected in the bth sub-sample and is such that ψ (b,S) ∼ B (n /n) for RF based on subsampling, where n denotes the number of units in each sub-sample. Plugging m r f (•) in (3.4) leads to the RF model-assisted estimator

t r f = k∈U m r f (x k ) + k∈S y k -m r f (x k ) π k . (3.19)
Using similar arguments to those used in the proof of Proposition 3.3.1, we can show that t r f can be expressed as

t r f = k∈S w ks y k ,
where the weights w ks are given by

w ks = 1 π k 1 + ∈U W k (x ) 1 - I π , k ∈ S. (3.20)
Noting that k∈S W k (x ) π -1 k = 1 for all ∈ U, it follows from (3.20) that k∈S w ks = N for every sample S. That is, the sum of the weights w ks match the population size N perfectly, a desirable property shared by other nonparametric model-assisted estimators [START_REF] Breidt | Model-assisted estimation for complex surveys using penalized splines[END_REF][START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF][START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF]. Unlike the weights w ks in (3.15), the weights w ks depend on both the sample selection indicators I , ∈ U, and the partition P S that varies from one sample to another. This is due to the fact that the nodes A (S) are constructed so as to optimize the sample criterion (3.16). For this reason, the weights w ks , k ∈ S, are variable speci c in the sense that depend on the survey variable Y . To cope with this issue, we describe a model calibration procedure in Section 3.5 for handling multiple survey variables while producing a single set of weights.

Remark 3.3.1. In practice, the variables ψ (b,S) k in (3.18) are not generated for the units outside the sample. However, at least conceptually, nothing precludes de ning these variables for k

∈ U \ S. For k ∈ U \ S, we set ψ (b,S) k ∼ B ((N -n )/(N -n)) so that k∈U ψ (b,S) k = N . De ning the variables ψ (b,S)
k for units outside the sample will have no e ect on the predictions m r f (•) associated with the sample units since I k = 0 for k ∈ U \ S. This construction will prove useful in establishing the asymptotic properties of the proposed procedures; see Section 3.4.

As for the RF prediction built at the population level described in Section 3.2.2, the prediction m r f (x k ) in (3.17) can be expressed as a bagged predictor [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]. That is,

m r f (x k ) = 1 B B b=1 m (b) tree (x k , θ (S) b ), 3.3 : 94 where m (b) tree (x k , θ (S) b ) = ∈S π -1 ψ (b,S) 1 x ∈A (S) x k ,θ (S) b y / N(x k , θ (S) b )
is the prediction associated with unit k based on the bth stochastic regression tree. The model-assisted estimator t r f given by (3.19) can thus be viewed as a bagged estimator:

t r f = 1 B B b=1 t (b) tree (θ (S) b ),
where

t (b) tree (θ (S) b ) = k∈U m (b) tree (x k , θ (S) b ) + k∈S y k -m (b) tree (x k , θ (S) b ) π k
is the model-assisted estimator of t y based on the bth stochastic regression tree. As in the case of regression trees built at the population level (see Section 3.2.1), given the partition

P (b) S = {A (bS) j } J bS j=1 , the predictions m (b) tree (x k , θ (S)
b ) are least squares type predictions obtained by tting the one-way ANOVA model with Y as the response and the node membership indicators {1 x k ∈A (bS) j } J bS j=1 as the explanatory variables; see the proof of Proposition 3.3.3 and the Supplementary Material for more details. As a result, the estimator t (b) tree (θ (S) b ) is related to the customary post-strati ed estimator [START_REF] Särndal | Model assisted survey sampling[END_REF].

Under mild assumptions, Proposition 3.3.2 below shows that bagging improves the e ciency of model-assisted estimators. This is similar to what is encountered in the classical RF literature [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF].

Proposition 3.3.2. Let t (1) , . . . , t (b) , . . . , t (B) be a sequence of model-assisted estimators of t y and let t = B -1 B b=1 t (b) be a bagged estimator. Assuming that the t (b) 's have approximately the same design bias and design variance, then, for B large enough:

MSE p ( t) -MSE p ( t (1) ) V p ( t (1) ) max b b Cor p t (b) , t (b ) -1 0,
where MSE p (•) and Cor p (•) denote the mean squared error and correlation operators with respect to the sampling design.

The proof of Proposition 3.3.2 is given in the Appendix. We end this section by giving an alternative expression for t r f . Proposition 3.3.3. The RF estimator t r f given by (3.19) can be written as

t r f = k∈U m r f (x k ) + 1 B B b=1 k∈S 1 -ψ (b,S) k y k -m (b) tree (x k , θ (S) b ) π k , (3.21)
where m (b) tree (x k , θ (S) b ) is the predictor associated with unit k based on the bth stochastic regression tree.

The proof of Proposition 3.3.3 is given in the Appendix. It follows from Proposition 3.3.3, that the second term on the right hand-side of (3.21) vanishes if ψ (b,S) k = 1 for all k ∈ S. That is, the estimator t r f reduces to the so-called projection form [START_REF] Breidt | Model-assisted estimation for complex surveys using penalized splines[END_REF][START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF], Särndal et al., 1992) t r f = k∈U m r f (x k )

if the RF algorithm does not involve a resampling mechanism. In addition, the second term on the right hand-side of (3.21) vanishes if y k = c for all k, for some c ∈ R or if the trees in the forest are fully grown (i.e., each terminal node contains a single observation), which implies that the observations y k and the corresponding prediction m (b) tree (x k , θ (S) b ) coincide. When the estimator t r f can be expressed in the projection form, the weights w ks given by (3.20) are always positive and cannot exceed the number of terminal nodes from the largest tree of the forest.

In practice, a resampling mechanism is typically used with RF algorithms. In this case, the second term on the right hand-side of (3.21) does not vanish and is equal to the weighted sum of residuals computed for the non-resampled units, also called the out-of-bag individuals (James et al., 2015, Chapter 8), from each of the B trees. The second term on the right hand-side of (3.21) can then be viewed as a correction term which brings additional information from the units not used in computing the predictions m (b) tree (•, θ (S) b ), b = 1, . . . , B.

Asymptotic properties

To establish the asymptotic properties of the proposed estimators and to derive the associated variance estimators, we consider the asymptotic framework of [START_REF] Isaki | Survey design under the regression superpopulation model[END_REF]. We start with an increasing sequence of embedded nite populations

{U v } v∈N of size {N v } v∈N .
In each nite population U v , a sample of size n v is selected according to a sampling design

P v (S v = s v | Z U ).
While the nite populations are assumed to be embedded, we do not require this property to hold for the samples {S v } v∈N . This asymptotic framework assumes that v goes to in nity, so that both the nite population sizes and the samples sizes go to in nity. To improve readability, we shall use the subscript v only in the quantities U v , N v and n v ; quantities such as π k,v shall be denoted simply as π k .

Assumptions: RF model-assisted estimator t * r f

We make the following assumptions:

(H6) There exists a positive constant C such that sup k∈U v |y k | C < ∞.

(H7) We assume that lim

v→∞ n v N v = π ∈ (0, 1).
(H8) There exist positive constants λ and λ * such that min

k∈U v π k λ > 0 and min k, ∈U v π k λ * > 0.
Also, we assume that lim sup

v→∞ n v max k ∈U v |π k -π k π | < ∞.
Assumptions (H13)-(H15) have been extensively used in parametric, nonparametric and functional model-assisted estimation [START_REF] Breidt | Model-assisted estimation for complex surveys using penalized splines[END_REF][START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], Cardot et al., 2013c[START_REF] Goga | Réduction de la variance dans les sondages en présence d'information auxiliarie: Une approache non paramétrique par splines de régression[END_REF][START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF][START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF]. Assumption (H13) implies that the survey variable Y is uniformly bounded (Breidt andOpsomer, 2000, Cardot et al., 2010). Assumptions (H14) and (H15) deal with the rst and second order inclusion probabilities and they are satis ed for the classical xed-size sampling designs; see for example, [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] and [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF]. Furthermore, we assume that the minimum number of observations N 0v in a terminal nodes is growing to in nity and we make the following additional assumption

(C1) The number of subsampled elements N v is such that lim v→∞ N v /N v ∈ (0; 1].
This assumption requires that the number N v of elements in each subsample increases at the same speed as the population size N v , allowing each terminal node to have at least N 0v observations.

Assumptions: RF model-assisted estimator t r f

In addition to the above assumptions, we make the following assumptions to establish the asymptotic properties of tr f given by (3.19).

(H9) There exists a positive constant

C 1 such that n v max k ∈U v E p (I k -π k )(I -π )| P S C 1 .
(H10) The random forests based on population partitions and those based on sample partitions are such that, for all x ∈ R p :

E p m r f (x) -m r f (x) 2 = o(1), where m r f (x) = ∈U v 1 B B b=1
ψ (b,S) 1

x ∈A (S) x,θ

(S) b y N(x, θ (S) b ) with N(x, θ (S) b ) = ∈U v ψ (b,S) 1 x ∈A (S) x,θ (S) b
.

Assumption (H18) is similar to that used by [START_REF] Toth | Building consistent regression trees from complex sample data[END_REF] and [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF]; it requires that, as the sample and population size grow, the in uence of extreme observations on the sample partitions decreases. Assumption (H16) requires that the average number of elements at the population level in the sample partitions converges to the average number of population elements in the population partitions. It implicitly assumes that the sample partitions converge to the population partitions. A similar result was established in [START_REF] Toth | Building consistent regression trees from complex sample data[END_REF] in the case of regression trees. [START_REF] Toth | Building consistent regression trees from complex sample data[END_REF] evaluated the properties of point estimators with respect to the joint distribution induced by the superpopulation model and the sampling design. In a iid setting, [START_REF] Scornet | Consistency of random forests[END_REF] showed that the population partitions converge to the theoretical partitions. Assumption (H16) can thus be viewed as a design-based version of the result from [START_REF] Scornet | Consistency of random forests[END_REF]. In the Supplementary Material, we conduct a simulation study, whose results suggest that Assumption (H16) seems to be veri ed, at least in our experiments. More research is needed to provide a rigorous proof of Assumption (H16) in the design-based approach and is beyond the scope of this article.

As in the case of model-assisted estimators based on RF with population-based partitions, we assume that the minimum number of observations, n 0v , in the terminal nodes is also growing to in nity and we assume the following additional assumption about the RF resampling algorithm :

(C2) The number of subsampled elements n v is such that lim v→∞ n v /n v ∈ (0; 1].
This assumption requires that the number n v of elements in each subsample increases at the same speed as the sample size n v , allowing each terminal node to have at least n 0v observations.

Asymptotic results

In this section, we state some results pertaining to sequences of RF model-assisted estimators { t r f }. The corresponding results for the model-assisted estimators { t * r f } can be found in the Supplementary Material.

Result 3.4.1. Consider a sequence of RF model-assisted estimators { t r f }. Then, there exist positive constants C1 , C2 such that

E p 1 N v t r f -t y C1 √ n v + C2 n 0v
, with ξ-probability one.

If

n u v n 0v = O(1)
with 1/2 u 1, then there exists a positive constant C such that

E p 1 N v t r f -t y C √ n v
, with ξ-probability one.

Result 3.9.1 implies that the RF model-assisted estimator { t r f } is asymptotically designunbiased, i.e.,

lim v→∞ E p 1 N v t r f -t y = 0, with ξ-probability one,
and design-consistent in the sense that

lim v→∞ E p 1 {N -1 v | t r f -t y |>η} = 0, with ξ-probability one
for all η > 0. Moreover, if n 0v is large enough with respect to the sample size n v , the RF estimator t r f is √ n v -consistent. For a given partition, note that the number of terminal nodes is of order O(n v /n 0v ), and if n 0v satis es the condition from the Result 3.9.1, the number of terminal nodes is of order O(n 1-u ) for 1/2 u 1.

The next result shows that the RF model-assisted estimator t r f is asymptotically equivalent to the pseudo-generalized di erence estimator:

t pgd = k∈U m r f (x k ) + k∈S y k -m r f (x k ) π k , (3.22)
where m r f (x k ) is given by (3.9).

Result 3.4.2. Consider a sequence of RF estimators { t r f }. Assume also that

n u v n 0v = O(1) with 1/2 < u 1.
Then, { t r f } is asymptotically equivalent to the pseudo-generalized di erence estimator t pgd in the sense that

√ n v N v t r f -t y = √ n v N v t pgd -t y + o P (1).
From Proposition 3.9.2, it follows that the asymptotic variance of t r f can be approximated by the variance of (3.22). That is,

AV p 1 N v t r f = V p 1 N v t pgd = 1 N 2 v k∈U v ∈U v (π kl -π k π ) y k -m r f (x k ) π k y -m r f (x ) π .
(3.23)

While the RF model-assisted estimator t r f is design-consistent as long as n 0v and n v grow to in nity (Result 3.9.1), the asymptotic equivalence of t r f with the pseudo-generalized di erence estimator t pgd is obtained only for n 0v satisfying a certain rate. Stronger assumptions on higherorder inclusion probabilities (Breidt andOpsomer, 2000, McConville and[START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF] are required in order to show that the asymptotic mean squared error of t r f is equivalent to the variance of the pseudo-generalized di erence estimator. We do not pursue this further. Expression (3.23) suggests that t r f is e cient if the residuals y km r f (x k ) are small for all k ∈ U v . The asymptotic variance given in (3.23) cannot be computed in practice because the residuals,

y k -m r f (x k ), k ∈ U, are unknown. Assuming that π k > 0 for all pairs (k, ) ∈ U v × U v , a design-consistent estimator of AV p 1 N v t r f is given by V r f 1 N v tr f = 1 N 2 v k∈U v ∈U v I k I π k -π k π π k y k -m r f (x k ) π k y -m r f (x ) π , (3.24)
where m r f (x k ) is given by (3.17). To establish the design consistency of (3.24), we require the following additional assumption:

(H11) We assume that lim

v→∞ max i, j,k, ∈D 4,Nv |E p I i I j -π i π j (I k I -π k π ) | = 0, where D 4,N v
denotes the set of distinct 4-tuples from U v .

Assumption (H17) was suggested by [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] and, together with (H14)-(H15), is used to establish the design consistency of the unbiased estimator of the variance of the Horvitz-Thompson estimator k∈S v y k /π k , assuming that the survey variable Y has nite fourth moment. Assumption (H17) is satis ed for simple random sampling without replacement and strati ed simple random sampling without replacement. It is also satis ed for high entropy sampling designs [START_REF] Boistard | Approximation of rejective sampling inclusion probabilities and application to high order correlations[END_REF], Cardot et al., 2013c).

Result 3.4.3. Consider a sequence of RF model-assisted estimators { t r f }. Assume also that n u v n 0v = O(1) with 1/2 < u 1.
Then, the variance estimator V r f ( t r f ) is asymptotically design-consistent for the asymptotic variance AV p t r f . That is,

lim v→∞ E p n v N 2 v V r f ( t r f ) -AV p ( t r f ) = 0.
Finally, we establish the central limit theorem that can be used to obtain asymptotically normal con dence intervals of t y . To that end, we assume that t pgd is normally distributed, an assumption that is satis ed in many classical sampling designs; e.g., see Fuller (2009a).

(H12) The sequence of pseudo-generalized di erence estimators { t pgd } satis es

N -1 v t pgd -t y V p N -1 v t pgd L ----→ v→∞ N (0, 1),
where V p N -1 v t pgd is given by (3.23).

Result 3.4.4. Consider the sequence of RF estimators { t r f }. Then,

N -1 v t r f -t y V r f (N -1 v t r f ) L ----→ v→∞ N (0, 1) .
The proof of Result 3.4.4 is a direct application of Results 3.9.2 and 3.9.3, and is thus omitted.

A model calibration procedure for handling multiple survey variables

In practice, most surveys conducted by national statistical o ces (NSO) collect information on multiple survey variables. The collected data are stored in rectangular data les. A column of weights, referred to as a weighting system, is made available on the data le. This weighting system can then be applied to obtain an estimate for any survey variable. However, applying a RF algorithm yield the variable-speci c weights (3.20). In other words, the weights were derived to obtain an estimate of the total for a speci c survey variable Y . Hence, applying the weights (3.20) to other survey variables may produce ine cient estimators. A solution to this issue consists of developing multiple sets of weights, one for each survey variable. This is usually deemed undesirable by data users who are used to work with a single set of weights. In this section, we describe a model calibration procedure [START_REF] Wu | A model-calibration approach to using complete auxiliary information from survey data[END_REF], originally proposed by [START_REF] Montanari | Multiple and ridge model calibration for sample surveys[END_REF], that yields a single weighting system while accounting for multiple survey variables that are deemed important. Suppose that we can identify a subset of survey variables Y 1 , . . . , Y q , that are deemed important. We postulate the following working model for each variable:

E Y j k | X k = x k = m ( j) (x ( j) k ), j = 1, • • • , q, (3.25)
where m ( j) (•) is an unknown function and x

( j)
k is a vector of auxiliary variable associated with unit k for the variable Y j . We allow a di erent link functions m(•) and di erent sets of explanatory variables for each of the survey variables Y 1 , . . . , Y q . The interest lies in estimating the population totals t y 1 , . . . , t y q . We assume that each of these totals is estimated using a model-assisted estimator of the form (3.4) but with possibly di erent methods. For instance, some of the estimates may be based on a parametric working model, while others may be based on a nonparametric working model (e.g., RF). We can construct the set of q predicted values m (1) (x (1) k ), . . . , m (q) (x (q) k ), for k ∈ U. In addition, we assume that, at the estimation stage, a vector v k of size q of calibration variables is available for k ∈ S and that the corresponding vector of population totals t v = k∈U v k is known. In practice, survey managers often want to ensure consistency between 3.5 survey estimates and known population totals for important variables such as gender and age group.

Given these predictions m (1) (x (1) k ), . . . , m (q) (x (q) k ), and the vector calibration variables v, we seek calibrated weights w C k , k ∈ S, as close as possible to the initial weights π -1 k subject to the following q + q + 1 calibration constraints:

k∈S w C k = N, (3.26) k∈S w C k m ( j) (x ( j) k ) = k∈U m ( j) (x ( j) k ), j = 1, . . . , q, (3.27) k∈S w C k v k = k∈U v k .
(3.28)

More speci cally, we seek calibrated weights

w C k such that k∈S G(w C k /π -1 k )
is minimized subject to (3.26)-(3.28), where G(•) is a pseudo-distance function measuring the closeness between two sets of weights, such that [START_REF] Deville | Calibration estimators in survey sampling[END_REF].

G(w C k /π -1 k ) ≥ 0, di erentiable with respect to w C k , strictly convex, with continuous derivatives g(w C k /π -1 k ) = ∂G(w C k /π -1 k )/∂w C k such that g(1) = 0; see
The weights w C k are given by

w C k = π -1 k F( λ h k ),
where F(.) is the calibration function de ned as the inverse of g(.), λ is a q + q + 1-vector of estimated coe cients and

h k = 1, m (1) k -m (1)
, . . . , m

km

(q) , v 1k , . . . , v q k (3.29) with m ( j) k ≡ m ( j) (x ( j) k ) and m ( j) ≡ k∈S π -1 k m ( j) k / k∈S π -1 k , j = 1, • • • , q.
The calibrated weights w C k may be viewed as a compressed score summarizing the information contained in the q working models (3.25) and the vector of calibration variables v. The weighting system {w C k ; k ∈ S} may be then applied to any survey variable Y , which leads to the model calibration type estimator

t y,mc = k∈S w C k y k .
If the number of calibration constraints q + q + 1 is large, the resulting weights w C k may be highly dispersed leading to potentially unstable estimates t y,mc . A number of pseudo-distance functions such as the truncated linear and the logit methods may be used to limit the variability of the weights w C k ; see [START_REF] Deville | Calibration estimators in survey sampling[END_REF] for a description of these methods. A simple alternative is to use additional constraints on the weights as part of the calibration constraints. For instance, we may impose that w C k < w 0 , where w 0 is a threshold set by the survey statistician; see also [START_REF] Santacatterina | Optimal probability weights for inference with constrained precision[END_REF] for alternative constraints on the weights. Finally, we can relax the calibration constraints (3.26)-(3.28) by considering a L 2 -penalized criterion, leading to a ridge-type model calibration estimator; see [START_REF] Montanari | Multiple and ridge model calibration for sample surveys[END_REF]. [START_REF] Montanari | Multiple and ridge model calibration for sample surveys[END_REF] reports the results of a simulation study, assessing the performance of point estimators obtained through multiple and ridge model calibration methods.

Simulation study

Performance of point estimators

We conducted a simulation study to assess the performance of several model-assisted estimators, in terms of bias and e ciency. We generated a nite population of size N = 10, 000, consisting of a set of auxiliary variables and 8 survey variables. We rst generated 7 auxiliary variables X 0 , • • • , X 6 , according to the following distributions: X 0 ∼ U(0, 1); X 1 ∼ N (0, 1), X 2 ∼ Beta (3, 1), X 3 ∼ 2 × Gamma (3, 2), X 4 ∼ Bernoulli(0.7), X 5 ∼ Multinomial(0.4, 0.3, 0.3) and X 6 ∼ E(1). The variables X 1 , X 2 , X 3 , and X 6 have been standardized so as to have a mean and a variance equal to 0 and 1, respectively. To assess the performance of the proposed method in a high-dimensional setting, we also generated 100 additional auxiliary variables V 1 , V 2 , • • • , V 100 , from a uniform distribution U(-1, 1). Given the X-variables and the V-variables, we generated the survey variables according to the following models:

Model 1: Y 1 = 1 + 2 (X 0 -0.5) + N (0, 0.1) ; Model 2: Y 2 = 1 + 2 (X 0 -0.5) 2 + N (0, 0.1); Model 3: Y 3 = 2 + X 6 + X 2 + X 3 + X 4 + X 5 + N (0, 1); Model 4: Y 4 = 2 + (X 6 + X 2 + X 3 ) 2 + N (0, 1);
Model 5: Y 5 = 0.5X 5 + exp(-X 1 ) + 3X 4 + exp(-X 6 ) + E (1);

Model 6: Y 6 = V 2 1 + exp(-V 2 2 ) + N (0, 0.3); Model 7: Y 7 = V 2 1 + exp(-V 2 2 ) + N (0, 0.3); Model 8: Y 8 = 3 + V 1 V 2 + V 2 3 -V 4 V 7 + V 8 V 10 -V 2 6 + N (0, 0.5).
The errors in Model 5 have been scaled and centered so as to have a mean and a variance equal to 0 and 1, respectively. Models 1 and 2 were used in [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF], while Models 7 and 8 were introduced in Scornet (2017). Models 1-8 were generated so as to include a relatively wide range of relationships between the Y -variable and the set of explanatory variables: linear/non-linear relationships, presence/absence of quadratic terms and presence/absence of interactions. Our scenarios also included low, medium and highdimensional settings. From the population, we selected R = 5, 000 samples, of size n, according to simple random sampling without replacement. We used n = 250 and n = 1, 000. In each sample, we computed the following estimators: (i) The Horvitz-Thompson (HT) estimator given by (3.1); (ii) The generalized regression (GREG) estimator given by (3.4) with m(x k ) = x k β;

(iii) The model-assisted estimator (3.4) with m(x k ) obtained through regression trees (CART); and (iv) The model-assisted estimator (3.4) based on RF, where m(x k ) is given by (3.17). We considered three RF algorithms, each based on 1, 000 trees. The rst (RF1) was based on bootstrap. The second algorithm (RF2) was based on subsampling with a sampling fraction equal to 0.63 [START_REF] Scornet | Tuning parameters in random forests[END_REF]. For both RF1 and RF2, the minimum number of observations per terminal node was set to n 0 = 5. Finally, the third algorithm (RF3) was based on bootstrap with n 0 = √ n observations in each terminal node. In RF1-RF3, we used m tr y = √ p as it is the default number of variables considered for the splitting process in most software packages dealing with RF for regression.

For the estimators GREG, CART, RF1, RF2 and RF3, the predictions m(x k ) were obtained using the working models described in Table 3. For the survey variables Y 7 and Y 8 , the working models were based on a large number of super uous explanatory variables (50 and 100, respectively), which allowed us to assess the behavior of the resulting estimators in a medium/high dimensional setting.

Y 1 X 0 Y 2 X 0 Y 3 X 1 -X 6 Y 4 X 1 -X 6 Y 5 X 1 -X 6 Y 6 V 1 -V 10 Y 7 V 1 -V 50 Y 8 V 1 -V 100
We were interested in estimating the population totals t y j = k∈U y k j , j = 1, . . . , 8. As a measure of bias of an estimator ty j , we used the Monte Carlo percent relative bias de ned as

RB( t y j ) = 100 × 1 R R r=1 ( t (r) y j -t y j ) t y j ,
where t (r) y j denotes the estimator t y j in the rth iteration, r = 1, ..., R. As a measure of e ciency of an estimator ty j , we used the relative e ciency, using the Horvitz-Thompson estimator, t y j ,π , as the reference:

RE( t y j ) = 100 × MSE( t y j ) MSE( t y j ,π ) ,
where

MSE( t y j ) = 1 R R r=1 ( t (r) y j -t y j ) 2
and MSE( t y j ,π ) is de ned similarly. The results are displayed in Tables 4 and5. The simulations were performed using the R software with the package ranger [START_REF] Wright | ranger: A fast implementation of random forests for high dimensional data in c++ and r[END_REF]. We start by noting that all the estimators displayed a negligible bias in all the scenarios, as expected. Also, both RF1 and RF2 showed very similar performances in terms of bias and e ciency in all the scenarios. This is consistent with the empirical results of [START_REF] Scornet | Tuning parameters in random forests[END_REF]; i.e., the strategy based on bootstrap and the strategy based on subsampling with a sampling fraction of 0.63 led to similar performances. The results for RF3 were similar to those obtained for RF1 and RF2, which suggests that the number of observations in each terminal node did not seem to a ect the behavior of the point estimator, at least in our experiments. This may not be the case in other scenarios as we illustrate in Section 3.6.3.

In the case of a linear relationship (which corresponds to the survey variables Y 1 and Y 3 ), the GREG estimator was the most e cient, as expected. For instance, for the survey variables Y 3 , the value of RE for the GREG estimator was about 19.6%, whereas the RF1, RF2 and RF3 estimators showed a value of RE of about 34%. In the case of a nonlinear relationship (which corresponds to the survey variables Y 2 and Y 4 , . . . , Y 8 ), the GREG estimator was less e cient than RF1, RF2 and RF3. For instance, in the case of the variable Y 4 , the GREG showed a value of RE of about 81.1%, whereas the RE of RF estimators lied between 49.0% and 53.1%. For the variables Y 6 , Y 7 , Y 8 , the GREG estimator was even less e cient than the Horvitz-Thompson estimator with values of RE ranging from 105% to 127%.

In the case of a single explanatory variable (which corresponds to the survey variables Y 1 and Y 2 ), RF and regression trees displayed very similar performances. In contrast, the estimators RF1, RF2 and RF3 were more e cient than the CART estimator when the vector of explanatory variables was multi-dimensional (i.e., variables Y 3 , . . . , Y 9 ). In a high-dimensional setting (which corresponds to the survey variables Y 7 and Y 8 ), the RF estimators were more e cient than the Horvitz-Thompson estimator, even for n = 250.

Performance of the proposed variance estimator

We have also investigated the performance of the variance estimator V r f given by (3.24) in the case of RF with subsampling, in terms of relative bias and coverage of normal-based con dence intervals. We generated a population of size N = 100, 000 according to Model 5. The sample size was set to n = 500; 1, 000; 5, 000; 10, 000; 20, 000 and 50, 000. Here, we present the results for B = 1 but other values of B led to similar results and are not shown here. As we suspected that the number of observations in each terminal node, n 0 , may have an impact on the behavior of V r f , we used di erent values for n 0 : n 0 = n a/20 for a = 1; 3; 5; 7; 9; 11; 13; 15; 17. The choice n 0 = n 11/20 was advocated by [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF]. Figure 16 shows the Monte Carlo percent relative bias of V r f for di erent values of n and n 0 . Figure 17 shows the Monte Carlo coverage rate of the con dence interval, t r f ± 1.96 V r f , for di erent values of n and n 0 .

From Figure 16, we note that V r f is severely biased for small values of n 0 and as a consequence, the con dence intervals (see Figure 17) perform poorly for small values of n 0 because of the substantial underestimation of the true variance in these scenarios. For a given value of n 0 , we note that the bias decreases as n increases and for a given value of n, the bias decreases as n 0 increases. For n 0 = 13/20 , the con dence intervals perform relatively well with coverage rates close to the nominal rate. The signi cant bias for small values of n 0 is most likely due to over tting, which is characterized by the presence of arti cially small residuals y km(x k ) in each terminal node, which in turn, leads to underestimation. This issue was raised by [START_REF] Opsomer | Selecting the amount of smoothing in nonparametric regression estimation for complex surveys[END_REF] in the context of local polynomial regression. To cope with this issue, we suggest a variance estimator based on a K-fold criterion. More speci cally, we randomly split the sample S into K groups S κ , κ = 1, . . . , K, of approximately equal size. For k ∈ S κ , let m (-κ) (x k ) denote the prediction at the point x k built on S -S κ and

(-κ) k = y k -m (-κ) (x k ) the associated residual. The proposed K-fold variance estimator is given by V (K) = K κ 1 =1 K κ 2 =1 k∈S κ 1 ∈S κ 2 (∆ k /π k )( (-κ 1 ) k /π k )( (-κ 2 ) /π ).
In practice, the number of groups (or folds) is often set to K = 5 or K = 10. We tested the performance of V (5) in terms of bias and coverage probability, using the same scenarios as above. The bias was almost negligible for all sizes n and n 0 and the coverage rates lied between 93% and 96%, which constitutes a signi cant improvement over the results displayed in Figures 16 and17. More research is needed in order to establish the theoretical properties of the variance estimator based on a K-fold criterion evaluate. It will be treated elsewhere.

Choice of hyper-parameters

To get a better understanding of how the choice of hyper-parameters impacts the behavior of model-assisted estimators based on RF, we conducted additional scenarios. We rst identi ed the following important hyper-parameters involved in the RF algorithm of [START_REF] Breiman | Random forests[END_REF]:

i) The minimal number of observations, n 0 , in each terminal node;

ii) The number of trees in the forest B;

iii) The number of variables considered for the search of the best split in the optimization criterion (3.16);

iv) The resampling mechanism.

The additional scenarios were conducted using a nite population of size N = 10, 000 consisting of the survey variables Y 5 and Y 8 described in Section 3.6.1. Recall that the working model for the survey variables Y 5 included the predictors X 1 , . . . , X 6 , whereas it included the predictors V 1 , . . . , V 100 for the variable Y 8 (see Table 3).

From the population, we generated R = 10, 000 samples, of size n = 1, 000, according to simple random sampling without replacement. Figure 18 and Figure 19 show, respectively, the relative e ciency of the model-assisted estimators based on RF, t r f , corresponding to Y 5 and Y 8 , respectively, for several values of n 0 . Figure 18 suggests that t r f was much more e cient than the Horvitz-Thompson estimator for small values of n 0 and that the value of RE approached 100 as n 0 increased. This result can be explained by the fact that small values of n 0 led to homogeneous terminal nodes, which in turn led to small residuals y km r f (x k ). For the survey variable Y 8 , we note from Figure 19 that the value of n 0 did not seem to a ect the e ciency of the corresponding model-assisted estimator.

Figure 20 display the relative e ciency for several values of B, the number of trees in the forest for the survey variable Y 8 . As expected, a small value of B causes the estimator t r f to loose some e ciency. Figure 20 suggests that B = 50 led to good results and that the e ciency Figure 19: Relative e ciency of t r f for the survey variable Y 8 and for several values of n 0 of t r f was not much a ected by the number of trees B for B ≥ 50. Nevertheless, it is advisable to choose a large value of B if the computational capacity permits. The results for the survey variable Y 5 were very similar and so we omit them.

In most software packages, the default number of variables considered for the splitting process is m tr y = √ p in case of regression. In our simulations, this choice led to satisfactory results in most scenarios. Figure 21 shows the relative e ciency of t r f for the survey variable Y 8 and for several values of m tr y . Since the working model for Y 8 contained p = 100 explanatory variables, the default value √ p was equal to 10. Although the value √ p = 10 was not the best choice for optimal performances, it led e cient model-assisted estimators. Furthermore, the relative e ciency did not vary much for values B larger than 30.

Turning to the resampling mechanism, a common choice is to use bootstrap (with replacement), for which some of the results presented in the paper do not apply. However, as noted by several authors (see e.g. [START_REF] Scornet | Consistency of random forests[END_REF], [START_REF] Wager | Asymptotic theory for random forests[END_REF] and the references therein) and as shown in our simulations, selecting the points without replacement rather than with replacement does not seem to a ect the performance of the resulting model-assisted estimators in most cases.

Real data application

In this section, we apply the proposed methods using data collected by Médiamétrie, the company that measures the media audience in France. In this application, we focus on radio audiences. Each year, Médiamétrie conducts a survey aiming at gathering detailed information about French individuals 13 years of age and over, including socio-demographic variables and radio listening habits. We used the 2019 radio audience data that consisted of N = 26, 293 individuals. As a survey variable, we considered the binary variable Y , such that y k = 1 if an individual in the kth individual listens to the radio of interest on a daily basis, and y k = 0 otherwise. For con dentiality reasons, we omit the name of the radio broadcaster. We aimed at estimating the proportion of French individuals who listen to the radio of interest on a daily basis, both at the overall population level and for several domains of interest. For each individual, we had access to 43 socio-demographic variables (e.g., number of individuals in the household, age of each member of the household, gender, internet habits, occupation, etc.) and their listening habits of 21 other radios. For each individual, we also knew whether or not the individual listens to any of these 21 radios, for each interval of 7.5 minutes on a typical day. This led to a data set with p = 3, 882 variables, among which 3, 839 were binary.

From the data set, we selected a single sample of size n = 4, 000 according to a strati ed sampling design with 5 strata, each stratum corresponding to a French region: North-East, North-West, Île-de-France, South-east and South-West. The strata sample sizes were determined according to proportional allocation. We considered the following domains of interest: the sub-population of individuals who connects to the internet everyday, almost every day, once or twice per week, once to three times per month, very rarely, never, the sub-population of individuals with/without children, the sub-populations of individuals living in cities of size (less than 20, 000, between 20, 000 and 50, 000, between 50, 000 and 100, 000, between 100, 000 and 200, 000 and larger than 200, 000) and the sub-population of individuals living in households of size 1, 2, 3, 4, 5 and 5+

We computed the following estimates both at the overall level and at the domain level: (i) The Horvitz-Thompson estimator; (ii) the GREG estimator and (iii) the model-assisted estimator based on RF with hyper-parameters B = 1, 000, n 0 = n 11/20 and m tr y = √ p. The working models used for the GREG estimator and the model-assisted RF estimator included 3882 explanatory variables. In each scenario, we also computed a 95% con dence interval for the proportion in the population of individuals who listen to the radio of interest. Finally, we computed the ratio of the estimated variances, using the estimated variance of the Horvitz-Thompson estimator, as the reference. Note that the "true value" was known for each domain of interest. The results (in percentage) are given in Table ??.

From Table ??, we note that the Horvitz-Thompson estimator performed relatively well in most scenarios. Because of the large number of predictors, the GREG estimator su ered from signi cant small sample bias. For instance, the estimate based on the GREG estimate at the overall level was equal to 27.7%, far from the true value of about 13.5%. In terms of point estimation, the RF model-assisted estimator led to very similar results than those obtained with the Horvitz-Thompson estimator. However, RF led to substantial improvement in terms of estimated variance. Indeed, out of the 22 domains, the value of RV(RF) was smaller than 0.65 for 20 domains. The results suggest that, unlike the GREG estimator, the model-assisted estimator based on RF was not a ected by the large number of explanatory variables in the working model. The median length of the con dence intervals was equal to 5.4% for the Horvitz-Thompson estimator, 4.2% for the RF estimator and 3.8% for the GREG estimator.

Final remarks

In this paper, we have introduced a new class of model-assisted estimators based on random forests and derived corresponding variance estimators. We have established the theoretical properties of point and variance estimators obtained through a RF algorithm based on subsampling. The results of an empirical study suggest that the proposed estimators perform well in a wide variety of settings, unlike the GREG and CART estimators. In practice, this robustness property is especially attractive when the data and the underlying relationships are complex. The application on radio audience data recorded by the French company Médiamétrie showed that the RF proposed estimator performed well in this high-dimension setting. We have also described a model calibration procedure for handling multiple survey variables, yet producing a single set of weights, which is attractive from a data user's perspective.

In practice, virtually all survey face the problem of missing values. Survey statisticians distinguish unit nonresponse (when no information is collected on a sampled unit) from item nonresponse (when the absence of information is limited to some variables only). The treatment of unit nonresponse starts with postulating a nonresponse model describing the relationship between the response indicators (equal to 1 for respondents and 0 for nonrespondents) and a vector of explanatory variables. The treatment of item nonresponse starts with postulating an imputation model describing the relationship between the variable requiring imputation and a set of explanatory variables. In both unit and item nonresponse, determining a suitable model is crucial. Therefore, regression trees and RF may prove useful for obtaining accurate estimated response propensities and predicted values. To the best of our knowledge, a theoretical treatment of regression trees and RF in the context of either unit nonresponse or item nonresponse in a nite population setting is lacking. These topics are currently under investigation.

Traditionally, survey samples have been collected through probability sampling procedures and inferences were conducted with respect to the customary design-based framework. In recent years, there has been a shift of paradigm that can be explained by three main factors: (i) a dramatic decrease of response rates; (ii) a rapid increase in data collection costs; and (iii) the proliferation of nonprobabilistic data sources (e.g., administrative les, web survey panels, social media data, satellite information, etc.). To meet these new challenges, survey statisticians face increasing pressure to utilize these convenient but often uncontrolled data sources. While such sources provide timely data for a large number of variables and population elements, they often fail to represent the target population of interest because of inherent selection biases. The integration of data from a nonprobability source with data from a probability survey is a topic that is currently being scrutinized by National Statistical O ces. An approach to data integration is statistical matching or mass imputation; see [START_REF] Yang | Statistical data integration in survey sampling: A review[END_REF] for a very recent review on the topic. Again, regression trees and RF algorithms may prove useful in the context of integration of survey data. This topic is currently under investigation.

In a high-dimensional setting, RF may be used to select the most predictive predictors, which in turn may be used in the construction of model-assisted estimators of population totals/means. In this context, issues such as variable selection bias [START_REF] Strobl | Bias in random forest variable importance measures: Illustrations, sources and a solution[END_REF] in a nite population setting need to be investigated. This will be treated elsewhere.

Appendix

Proof of Proposition 3.2.1 Since

W (x k ) = 1 B B b=1 ψ (b,U) 1 x ∈ A (U ) x k ,θ (U ) b N(x k , θ (U) b ) , (3.30)
involves positive quantities only, the weights W (x k ) are nonnegative. Since ψ (b,U) ∈ {0, 1} for all ∈ U and for all b ∈ 1, 2, . . . , B, the weight can be bounded as follows:

W (x k ) = 1 B B b=1 ψ (b,U) 1 x ∈ A (U ) x k ,θ (U ) b N(x k , θ (U) b ) 1 B B b=1 N x k , θ (U) b -1 cN -1 0 .
where c does not depend on b nor on k or . To show ii), x b ∈ 1, 2, ..., B. The result follows by noting

that W (x k ) = N x k , θ (U) b -1 exactly N x k , θ (U) b times.
Proof of Proposition 3.3.2 Let { t (b) } be a sequence of estimators of t y . Then,

V p 1 B B b=1 t (b) = 1 B 2 B b=1 V p t (b) + B b=1 B b b =1 Cor p t (b) , t (b ) V 1/2 p t (b) V 1/2 p t (b ) . V p ( t (1) ) B + V p ( t (1) ) max b b Cor p t (b) , t (b )
and Bias ) ). So, for B large enough: ) ).

2 p (B -1 B b=1 t (b) ) = Bias 2 p ( t (1) ) = MSE p ( t (1) ) -V p ( t ( 1
MSE p 1 B B b=1 t (b) V p ( t (1) ) max b b Cor p t (b) , t (b ) -V p ( t (1) ) + MSE p ( t ( 1
Proof of Proposition 3.3.3 Consider the B partitions build at the sample level P S = { P (b) S } B b=1 . For a given b = 1, . . . , B, the partition P (b) S is composed by disjointed regions as follows P (b) S = { A (bS) j } J b S j=1 and for each b, consider the J bS dimensional vector ẑ(b)

k = 1 x k ∈ A (b S) 1 , . . . , 1 x k ∈ A (b S) J b S
where

1 x k ∈ A (b S) j = 1
if x k belongs to the region A (bS) j and zero otherwise for all j = 1, . . . , J bS . Since {A (bS) j } J b S j=1 is a partition, then x k will belong to only one region and so, the vector ẑ(b)

k will contain only one non zero component. We have S) ẑ(b) y (see also the supplementary materiel for more details). Now,

m r f (x k ) = B -1 B b=1 m (b) tr ee (x k , θ (S) b ) and m (b) tr ee (x k , θ (S) b ) can be written as m (b) tr ee (x k , θ (S) b ) = (ẑ (b) k ) β (b) where β (b) = ∈S π -1 ψ (b,S) ẑ(b) (ẑ (b) ) -1 ∈S π -1 ψ (b,
k ∈S y k -m r f (x k ) π k = 1 B B b=1 k ∈S y k -m (b) tr ee (x k , θ (S) b ) π k = 1 B B b=1 k ∈S (1 -ψ (b,S) k )(y k -m (b) tr ee (x k , θ (S) b )) π k + 1 B B b=1 k ∈S ψ (b,S) k (y k -m (b) tr ee (x k , θ (S) b )) π k .
For each b, consider the J bS dimensional vector 1 J b S whose elements are all equal to one, we have then

1 J b S ẑ(b) k = 1 for all k, so k ∈S ψ (b,S) k π k m (b) tr ee (x k , θ (S) b ) = k ∈S ψ (b,S) k π k (ẑ (b) k ) β (b) = 1 J b S k ∈S ψ (b,S) k π k ẑ(b) k (ẑ (b) k ) β (b) = ∈S ψ (b,S) π y .

Supplementary material Assumptions: population-based RF model-assisted estimator t * r f

To establish the properties of the proposed estimators, we will consider three categories of assumptions2 : assumptions on the sampling design, assumptions on the survey variable and, nally, assumptions on the random forests.

(H13) We assume that there exists a positive constant C such that

sup k ∈U v |y k | C < ∞.
(H14) We assume that lim

v→∞ n v N v = π ∈ (0; 1).
(H15) There exist positive constants λ and λ * such that min

k ∈U v π k λ > 0, min k, ∈U v π k λ * > 0 and lim sup v→∞ n v max k ∈U v |π k -π k π | < ∞. (C3) The number of subsampled elements N v is such that lim v→∞ N v /N v ∈ (0; 1].
Assumptions: sample-based RF model-assisted estimator tr f (H16) We assume that there exists a positive constant C 1 > 0 such that

n v max k ∈U v E p (I k -π k )(I -π )| P S ≤ C 1 .
(H17) The random forests based on population partitions and those based on sample partitions are such that, for all x ∈ R p :

E p m r f (x) -m r f (x) 2 = o(1).
where m r f (x) is given by

m r f (x) = ∈U v 1 B B b=1 ψ (b,S) 1 x ∈ A (S) x,θ (S) b N(x, θ (S) b ) y with N(x, θ (S) b ) = k ∈U v ψ (b,S) k 1 x k ∈ A (S) x,θ (S) b and m r f (x) = ∈U v 1 B B b=1 ψ (b,U) 1 x ∈ A (U ) x,θ (U ) b N(x, θ (U) b ) y with N(x, θ (U) b ) = k ∈U v ψ (b,U) k 1 x k ∈ A (U ) x,θ (U ) b
.

Below, we include a graph illustrating the convergence of the di erence m r fm r f towards 0 in L 2 where the regression function was de ned as m(X) = 2 + 2X 1 + X 2 + X 3 , with X 1 , X 2 and X 3 de ned as in Section 6 from the main paper. The population sizes were such that the sampling fraction was of 10%.

Asymptotic results of the population RF model-assisted estimator t * r f

The population RF model-assisted estimator is given by

t * r f = k ∈U v m * r f (x k ) + k ∈S v y k -m * r f (x k ) π k ,
where m * r f is the sample-based estimator of m by using RF built at the population level (for more details, see relation (13) from the main paper):

m * r f (x k ) = ∈S v 1 π W * (x k )y , (3.31)
where

W * (x k ) = 1 B B b=1 ψ (b,U) 1 x ∈ A * (U ) x k ,θ (U ) b N * (x k , θ (U) b ) and N * (x k , θ (U) b ) = ∈U v ψ (b,U) 1 x ∈ A * (U ) x k ,θ (U ) b
is the number of units falling in the terminal node

A * (U) x k , θ (U) b containing x k .
The estimator m * r f (x k ) can be written as a bagged estimator as follows:

m * r f (x k ) = 1 B B b=1 m * (b) tr ee (x k , θ (U) b ),
where m * (b) tr ee (x k , θ (U) b ) is the sample-based estimation of m based on the b-th stochastic tree:

m * (b) tr ee (x k , θ (U) b ) = ∈S v 1 π ψ (b,U) 1 x ∈ A * (U ) x k ,θ (U ) b N * (x k , θ (U) b )
y .

(3.32)

For more readability, we will use in the sequel m * (b) tr ee (x k ) instead of m * (b) tr ee (x k , θ (U) b ). Consider the pseudogeneralized di erence estimator:

t pgd = k ∈U v m * r f (x k ) + k ∈S v y k -m * r f (x k ) π k ,
where m * r f (x k ) is the population-based estimator of m by using RF built at the population level (for more details, see relation ( 12) from the main paper):

m * r f (x k ) = ∈U v W * (x k )y .
The estimator m * r f can be written as a bagged estimator as follows:

m * r f (x k ) = 1 B B b=1 m * (b) tr ee (x k )
and m * (b) tr ee (x k ) is the predictor associated with unit k and based on the b-th stochastic tree:

m * (b) tr ee (x k ) = ∈U v ψ (b,U) 1 x ∈ A * (U ) x k ,θ (U ) b N * (x k , θ (U) b ) y . ( 3
U = {A * (bU) j } J * bU j=1 . Consider z * (b) k = (1 x k ∈ A * (bU ) 1 , . . . , 1 x k ∈ A * (bU ) J * bU ) where 1 x k ∈ A * (bU ) j = 1 if x k belongs to the region A * (bU)
(x k ) = ∈U v ψ (b,U) 1 x ∈ A * (bU ) j N * (b) j y , for x k ∈ A * (bU) j ,
where N * (b) j is the number of units belonging to the region A * (bU) j :

N * (b) j = ∈U v ψ (b,U) 1 x ∈ A * (bU ) j , j = 1, . . . , J * bU . (3.34)
Then, m * (b) tr ee (x k ) can be written as follows:

m * (b) N ,r f (x k ) = (z * (b) k ) β * (b) , k ∈ U v (3.35)
where

β * (b) = ∈U v ψ (b,U) z * (b) (z * (b) ) -1 ∈U v ψ (b,U) z * (b) y .
Remark that β * (b) may be obtained as solution of the following weighted estimating equation:

∈U v ψ (b,U) z * (b) (y -(z * (b) ) β * (b) ) = 0.
Since the regions A * (bU) j , j = 1, . . . , J * bU , form a partition, then the matrix

∈U v ψ (b,U) z * (b) (z * (b)
) is diagonal with diagonal elements equal to N * (b) j , the number of units falling in the region A * (bU) j for all j = 1, . . . , J * bU . By the stopping criterion, we have that all N * (b) j ≥ N 0v > 0 for all j, so the matrix

∈U v ψ (b,U) z * (b) (z * (b)
) is always invertible and β * (b) is well-de ned.

Consider now m * (b) tr ee (x k ), the estimator of the unknown m * (b) tr ee (x k ). Then, m * (b) tr ee (x k ) is the weighted mean of y-values for sampled individuals belonging to the same region A * (bU) j as unit k :

m * (b) tr ee (x k ) = ∈S v 1 π ψ (b,U) 1 x ∈ A * (bU ) j N * (b) j,N y for x k ∈ A * (bU) j
and we can write:

m * (b) tr ee (x k ) = (z * (b) k ) β * (b) , k ∈ U v (3.36)
where

β * (b) = ∈U v ψ (b,U) z * (b) (z * (b) ) -1 ∈S v 1 π ψ (b,U) z * (b) y .
In the expression of β * (b)

, we do not estimate the matrix ∈U v ψ (b,U) z * (b) (z * (b) ) since it is known and besides, we guarantee in this way that we will always have non-empty terminal nodes at the population level. So, β * (b) will be always well-de ned whatever the sample S is.

Let denote by α

k = π -1 k I k -1 for all k ∈ U v
, where I k is the sample membership, I k = 1 if k ∈ S and zero otherwise. In order to prove the consistency of t * r f as well as its asymptotic equivalence to the pseudo-generalized di erence estimator t pgd , we use the following decomposition:

1 N v t * r f -t y = 1 N v t pgd -t y - 1 N v k ∈U v α k ( m * r f (x k ) -m * r f (x k )) = 1 N v t pgd -t y - 1 B B b=1 1 N v k ∈U v α k m * (b) tr ee (x k ) -m * (b)
tr ee (x k ) .

(3.37)

We will prove that each term form the decomposition (3.37) is convergent to zero. We give rst several useful lemmas.

Lemma 1. There exists a positive constant c1 such that:

n v N 2 v E p ( t pgd -t y ) 2 c1 .
Proof. First of all, from relation (3.31), m * r f (x k ) is a weighted sum at the population level of y-values with positive weights summing to one (see proposition 2.1. from the main paper). Then, we get that

sup k ∈U v | m * r f (x k )|
C by using also assumption (H13). We have:

1 N v ( t pgd -t y ) = 1 N v k ∈U v α k (y k -m * r f (x k ))
and

n v E p t pgd -t y N v 2 = n v N 2 v V p k ∈S v (y k -m * r f (x k )) π k n v N v • 1 λ + n v max k ∈U v |π k -π k π | λ 2 • 2 N v k ∈U v y 2 k + ( m * r f (x k )) 2 c1
by assumptions (H13)-(H15).

Lemma 2. There exists a positive constant c2 not depending on b = 1, . . . , B, such that

E p || β * (b) -β * (b) || 2 2 c2 N v N 2 0v for all b = 1, . . . , B.
Proof. We can write b) ) . As already mentioned before, the matrix T * (b) is diagonal with positive diagonal elements given by N * (b) j the number of units falling in the region A * (bU) j (see relation 3.34) for j = 1, . . . , J * bU and by the stopping criterion, we have that N * (b) j ≥ N 0v > 0. We obtain then

β * (b) -β * (b) = ∈U v ψ (b,U) z * (b) (z * (b) ) -1 ∈S v 1 π ψ (b,U) z * (b) y - ∈U v ψ (b,U) z * (b) y = ∈U v ψ (b,U) z * (b) (z * (b) ) -1 ∈U v α ψ (b,U) z * (b) y (3.38) Let denote by T * (b) = ∈U v ψ (b,U) z * (b) (z * (
||( T * (b) ) -1 || 2 = max j=1,...,J bU 1 N * (b) j ≤ N -1 0v , for all b = 1, . . . , B (
E p || β * (b) -β * (b) || 2 2 E p ||N v ( T * (b) ) -1 || 2 2 • 1 N v ∈U v α ψ (b,U) z * (b) y 2 2 N 2 v N 2 0v E p 1 N v ∈U v α ψ (b,U) z * (b) y 2 2 (3.40)
and

E p 1 N v k ∈U v α k ψ (b,U) k z * (b) k y k 2 2 = 1 N 2 v k ∈U v (ψ (b,U) k ) 2 y 2 k ||z * (b) k || 2 2 E p (α 2 k ) + k ∈U v ∈U v k ψ (b,U) k ψ (b,U) y k y (z * (b) k ) z * (b) E p (α k α ) 1 n v n v λN v + n v max k, ∈U v ,k |π k -π k π | λ 2 1 N v k ∈U v (ψ (b,U) k ) 2 y 2 k ||z * (b) k || 2 2 C 0 n v (3.
E p || β * (b) -β * (b) || 2 2 c2 N v N 2 0v .
Result 3.9.1. Consider a sequence of population RF estimators { t * r f }. Then, there exist positive constants C1 , C2 such that

E p 1 N v t * r f -t y C1 √ n v + C2 N 0v , with ξ-probability one. If N a v N 0v = O(1) with 1/2 a 1, then E p 1 N v t * r f -t y C √ n v
, with ξ-probability one.

Proof. We get from relation (3.37) :

1 N v E p t * r f -t y 1 N v E p t pgd -t y + 1 B B b=1 1 N v E p k ∈U v α k ( m * (b) tr ee (x k ) -m * (b) tr ee (x k )) .
Lemma 1 gives us that there exists positive constant C1 such that

1 N v E p t pgd -t y C1 √ n v . (3.42)
Now, by using relations (3.35) and (3.36), we can then write for any b = 1, . . . , B:

k ∈U v α k ( m * (b) tr ee (x k ) -m * (b) tr ee (x k )) = k ∈U v α k (z * (b) k ) ( β * (b) -β * (b) )
and

1 N v E p k ∈U v α k ( m * (b) tr ee (x k ) -m * (b) tr ee (x k )) E p 1 N v k ∈U v α k z * (b) k 2 2 1/2 E p || β * (b) -β * (b) || 2 2 1/2 (3.43)
and Then, from relations (3.43), (3.44) and lemma 2, we get that there exists a positive constant C2 such that, for any b = 1, . . . , B, we have:

E p 1 N v k ∈U v α k z * (b) k 2 2 = 1 N 2 v k ∈U v E p (α 2 k )||z * (b) k || 2 2 + k ∈U v ∈U v k E p (α k α )(z * (b) k ) z * (b) 1 n v n v λN v + n v max k ∈U v |π k -π k π | λ 2 • 1 N v k ∈U v ||z * (b) k || 2 2 C 2 n v (3.
1 N v E p k ∈U v α k ( m * (b) r f (x k ) -m * (b) N ,r f (x k )) C 2 n v c2 N v N 2 0v C2 N 0v (3.45)
by using also the assumption (H13). The result follows then from relations (3.42) and (3.45).

Result 3.9.2. Consider a sequence of RF estimators

{ t * r f }. If N a v N 0v = O(1) with 1/2 < a 1, then √ n v N v t * r f -t y = √ n v N v t pgd -t y + o P (1).
Proof. We get from relation (3.37) and lemmas (1) and the proof of result 3.9.1 (relation 3.45) that

√ n v N v t * r f -t y = √ n v N v t pgd -t y + 1 B B b=1 √ n v N v k ∈U v α k ( m * (b) tr ee (x k ) -m * (b) tr ee (x k )) . = √ n v N v t pgd -t y + O P √ n v N 0v = √ n v N v t pgd -t y + o P (1) provided that N a v N 0v = O(1) with 1/2 < a 1.
Result 3.9.3. Consider a sequence of population RF estimators { t * r f }. Assume that

N a v N 0v = O(1) with 1/2 < a 1, then the variance estimator V r f ( t * r f
) is design-consistent for the asymptotic variance AV p t * r f . That is,

lim v→∞ E p n v N 2 v V r f ( t * r f ) -AV p ( t * r f ) = 0.
Proof Consider the following decomposition

n v V p N -1 v t * r f -AV p N -1 v t * r f = n v V p N -1 v t * r f -V p N -1 v t pgd + n v V p N -1 v t pgd -AV p N -1 v t * r f
where

V p N -1 v t pgd is the pseudo-type variance estimator of V p N -1 v t pgd = AV p N -1 v t * r f given by V p N -1 v t pgd = 1 N 2 v k ∈U v ∈U v π k -π k π π k y k -m * r f (x k ) π k y -m * r f (x ) π I k I .
Now, to prove that the consistency of the rst term from right of (3.46), we use the same decomposition as in [START_REF] Goga | E cient estimation of non-linear nite population parameters by using non-parametrics[END_REF].

Denote ẽk = y k -m * r f (x k ), êk = y k -m * r f (x k ) and c k = π k -π k π π k π k π I k I .
Then,

n v ( V p N -1 v t * r f -V p N -1 v t pgd ) = n v N 2 v k ∈U v ∈U v c k ( êk ê -ẽk ẽ ) = n v N 2 v k ∈U v ∈U v c k [( êk -ẽk )( ê -ẽ ) + ẽk ( ê -ẽ ) + ẽ ( êk -ẽk )] = A 1 + A 2 + A 3 . For all k ∈ U v , êk -ẽk = m * r f (x k ) -m * r f (x k ) and thus, E p | A 1 | ≤ n v λ 2 N v + n v max k ∈U v |π k -π k π | λ * λ 2 1 N v k ∈U v E p ( êk -ẽk ) 2 ,
by assumptions (H14)-(H15). Therefore, it su ces to show that, for all k ∈ U v , one has E p ( êkẽk ) 2 = o(1) uniformly in k, which we show next. We have

E p ( m * r f (x k ) -m * r f (x k )) 2 1 B B b=1 E p ( m * (b) tr ee (x k ) -m * (b) tr ee (x k )) 2 .
We can write by using relations (3.35) and (3.36):

m * (b) tr ee (x k ) -m * (b) tr ee (x k ) = (z * (b) k ) ( β quantity going to zero provided that N a v N 0v = O(1) with 1/2 < a 1.
Using the same arguments, we obtain that

E p | A 2 | = o(1) and E p | A 3 | = o(1). We get then n v E p | V p N -1 v t * r f -V p N -1 v t pgd | = o(1).
The second term from right of (3.46) concerns the consistency of the estimator of the Horvitz-Thompson variance computed for the population residuals

y k -m * r f (x k ), k ∈ U v .
The proof of this consistency [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF] requires assumptions only on the higher order inclusion probabilities (H18) as well as nite forth moment of y km * r f (x k ) : 1) and the result follows.

1 N v k ∈U v (y k -m * r f (x k )) 4 ≤ 4 N v k ∈U v (y 4 k + ( m * r f (x k )) 4 ) < ∞. So, n v E p | V p N -1 v t pgd -AV p N -1 v t * r f | = o(

Asymptotic results: the sample RF model-assisted estimator t r f

The sample RF model-assisted estimator is given by

t r f = k ∈U v m r f (x k ) + k ∈S v y k -m r f (x k ) π k ,
where m r f is the estimator of m built at the sample level and by using RF based on partition built at the sample level (for more details, see relation ( 17) from the main paper):

m r f (x k ) = ∈S v 1 π W (x k )y ,
where

W (x k ) = 1 B B b=1 ψ (b,S) 1 x ∈ A (S) x k ,θ (S) b N(x k , θ (S) b ) and N(x k , θ (S) b ) = ∈S v π -1 ψ (b,S) 1 x ∈ A (S) x k ,θ (S) b
is the estimated number of units falling in the terminal node

A (S) x k , θ (S) b containing x k .
As in Section 3.9.1, the estimator m r f (x k ) can be written as a bagged estimator of m as follows:

m r f (x k ) = 1 B B b=1 m (b) tr ee (x k )
and m (b) tr ee (x k ) is the estimation of m based on the b-th stochastic tree:

m (b) tr ee (x k ) = ∈S v 1 π ψ (b,S) 1 x ∈ A (S) x k ,θ (S) b N(x k , θ (S) b ) y (3.46)
As in Section 3.9.1, for more readability, we note in the sequel m (b) tr ee (x k ) instead of m (b) tr ee (x k , θ (S) b ). Consider the pseudo-generalized di erence estimator:

tpgd = k ∈U v m r f (x k ) + k ∈S v y k -m r f (x k ) π k
where m r f is the estimation of m built at the population level by using RF based on partition built also at the population level (relation ( 9) from the main paper):

m r f (x k ) = ∈U v W (x k )y ,
where

W (x k ) = 1 B B b=1 ψ (b,U) 1 x ∈ A (U ) x k ,θ (U ) b N(x k , θ (U) b ) with N(x k , θ (U) b ) = ∈U v ψ (b,U) 1 x ∈ A (U ) x k ,θ (U ) b
is the number of units falling in the terminal node

A (U) x k , θ (U) b containing x k .
The estimator m r f can be also written as a bagged estimator as follows:

m r f (x k ) = 1 B B b=1 m (b) tr ee (x k )
and

m (b) tr ee (x k ) = ∈U v ψ (b,U) 1 x ∈ A (U ) x k ,θ (U ) b N(x k , θ (U) b ) y .
As in the previous section, we will write m r f and m r f in equivalent forms. Consider for that the B partitions build at the population level

P U = { P (b) U } B b=1 .
For a given b = 1, . . . , B, the partition P (b) U is composed by the disjointed regions

P (b) U = {A (bU) j } J bU j=1 . Consider z (b) k = (1 x k ∈ A (bU ) 1 , . . . , 1 x k ∈ A (bU ) J bU
)

where

1 x k ∈ A (bU ) j = 1 if x k belongs to the region A (bU) j
and zero otherwise for all j = 1, . . . , J bU . Since

P (b)
U is a partition, then x k belongs to only one region at the b-th step. Suppose for example that x k ∈ A (bU) j , then m (b) tr ee (x k ) is the mean of y-values for individuals for which x ∈ A (bU) j :

m (b) tr ee (x k ) = ∈U v ψ (b,U) 1 x ∈ A (bU ) j N (b) j y , for x k ∈ A (bU) j ,
where N (b) j is the number of units belonging to the region A (bU) j :

N (b) j = ∈U v ψ (b,U) 1 x ∈ A (bU ) j , j = 1, . . . , J bU . (3.47)
Then, m (b) tr ee (x k ) can be written as a regression-type estimator with z (b) k as explanatory variables:

m (b) tr ee (x k ) = (z (b) k ) β (b) , k ∈ U v (3.48)
where

β (b) = ∈U v ψ (b,U) z (b) (z (b) ) -1 ∈U v ψ (b,U) z (b) y .
Based on the same arguments as in Section 3.9.1, the matrix ∈U v ψ (b,U) z (b) (z (b) ) is diagonal with diagonal elements equal to N (b) j , j = 1, . . . , J bU . By the stopping criterion, we have that all N (b) j ≥ N 0 > 0, so the matrix ∈U v ψ (b,U) z (b) (z (b) ) is invertible and β (b) is well-de ned.

Consider now the B partitions build at the sample level P S = { P (b) S } B b=1 . For a given b = 1, . . . , B, the partition P (b) S is composed by the disjointed regions P (b)

S = { A (bS) j } J b S j=1 . Consider ẑ(b) k = (1 x k ∈ A (b) 1S , . . . , 1 {x k ∈ A (b) J b S } ) where 1 x k ∈ A (b S) j = 1 if x k belongs to the region A (bS) j
and zero otherwise for all j = 1, . . . , J bS . Here, the hat notation is to design the fact that the vector ẑ(b) k depends on random dummy variables

1 x k ∈ A (b S) j . Since { A (bS) j } J b S
j=1 form a partition, then x k belongs to only one terminal node. Suppose for example that x k ∈ A (bS) j , then m (b) tr ee (x k ) is a Hajek-type estimator:

m (b) tr ee (x k ) = ∈S v 1 π ψ (b,S) 1 x ∈ A (b S) j y N (b) j , for x k ∈ A (bS) j ,
where N (b) j is the estimated number of units falling in the terminal node A (bS) j :

N (b) j = ∈S v 1 π ψ (b,S) 1 x ∈ A (b S) j , j = 1, . . . , J bS .
Then, m (b) tr ee (x k ) can be written also as a regression-type estimator with ẑ(b) k as explanatory variables: (3.49) where

m (b) tr ee (x k ) = (ẑ (b) k ) β (b) , k ∈ U v ,
β (b) = ∈S v 1 π ψ (b,S) ẑ(b) (ẑ (b) ) -1 ∈S v 1 π ψ (b,S) ẑ(b) y .
As in Section 3.9.1, remark that β (b) may be obtained as solution of the following weighted estimating equation:

∈S v 1 π ψ (b,S) ẑ(b) (y -(ẑ (b) ) β (b) ) = 0. Since {A (bS) j } J b S
j=1 is a partition, then the matrix

∈S v 1 π ψ (b,S) ẑ(b) (ẑ (b)
) is diagonal with diagonal elements equal to N (b) j , j = 1, . . . , J bS . By the stopping criterion and assumption (H15), we have that

∈S v 1 π ψ (b,S) 1 x ∈ A (b) j S ≥ n 0v > 0, so ∈S v 1 π ψ (b,S) ẑ(b) (ẑ (b)
) is always invertible is and β (b) is wellde ned whatever the sample S is. We need to consider also a second pseudo-generalized di erence estimator:

t pgd = k ∈U v m r f (x k ) + k ∈S v y k -m r f (x k ) π k 3.9 123 where m r f (x k ) = ∈U v 1 B B b=1 ψ (b,S) 1 x ∈ A (S) (x k ,θ (S) b ) N(x k , θ (S) b ) y = 1 B B b=1 m (b) tr ee (x k ) with N(x k , θ (S) b ) = ∈U v ψ (b,S) 1 x ∈ A(x k ,θ (S) b ) and m (b) tr ee (x k ) = ∈U v ψ (b,S) 1 x ∈ A (S) (x k ,θ (S) b ) y N(x k , θ (S) b ) = (ẑ (b) k ) β (b) , k ∈ U v (3.50) for β (b) = ∈U v ψ (b,S) ẑ(b) (ẑ (b) ) -1 ∈U v ψ (b,S) ẑ(b) y . The matrix ∈U v ψ (b,S) ẑ(b) (ẑ (b) ) is also diagonal with diagonal elements equal to ∈U v ψ (b,S) 1 x ∈ A (b S) j ≥ n 0v > 0, j = 1, . . . , J bS so β (b)
is also well-de ned whatever the sample S is. In order to prove the consistency of the sample-based RF estimator t r f , we use the following decomposition:

1 N v (t r f -t y ) = 1 N v ( t pgd -t y ) - 1 N v k ∈U v α k ( m r f (x k ) -m r f (x k )).
(3.51)

We will give rst several useful lemmas. The constants used in the following results may not be the same as the ones from Section 3.9.1 even if they are denoted in the same way for simplicity.

Lemma 3. There exists a positive constant c1 such that:

n v N 2 v E p ( t pgd -t y ) 2 c1 .
Proof. The proof is similar to that of lemma 1. We also have that

sup k ∈U v | m r f (x k )|
C by using assumption (H13). Further,

n v E p t pgd -t y N v 2 n v N v • 1 λ + n v max k ∈U v |π k -π k π | λ 2 • 2 N v k ∈U v y 2 k + ( m r f (x k )) 2 c1
by assumptions (H13)-(H15).

Lemma 4. There exists a positive constant c2 such that:

n v N 2 v E p ( t pgd -t y ) 2 c2 .
3.9

Proof. Using (3.50), we get that m r f (x k ) can be written as a weighted sum of y-values with positive weights summing to unity, so sup k ∈U v | m r f (x k )| C by using also assumption (H13). Now,

t pgd -t y = k ∈U v α k (y k -m r f (x k ))
and

n v N 2 v E p ( t pgd -t y ) = n v N 2 v k ∈U v E p α 2 k (y k -m r f (x k )) 2 + n v N 2 v k ∈U v k, ∈U v E p (y k -m r f (x k ))(y -m r f (x ))E p (α k α | PS ) 2n v C 2 λN v + n v N 2 v k ∈U v k, ∈U v E p |y k -m r f (x k )||y -m r f (x )| max k ∈U v |E p (α k α | PS )| c2 ,
by assumptions (H14) and (H16).

Lemma 5. There exists a positive constant c3 not depending on b such that:

E p β (b) -β (b) 2 2 c3 n v n 2 0v , for all b = 1, . . . , B.
Proof. Let denote by b) ) . As already mentioned, the J bS × J bS dimensional matrix T (b) is diagonal with diagonal elements given by N (b)

T (b) = ∈S v 1 π ψ (b,S) ẑ(b) (ẑ (
j = ∈S v 1 π ψ (b,S) 1 x ∈ A (b) j S
the weighted somme of units falling in the region A (b) jS for j = 1, . . . , J bS and by the stopping criterion, we have that

N (b) j ≥ n 0v > 0. The matrix T (b) is then always invertible with ||( T (b) ) -1 || 2 ≤ n -1 0v for all b = 1, . . . B.
(3.52) Now, write

β (b) -β (b) = ( T (b) ) -1 ∈S v 1 π ψ (b,S) ẑ(b) y -T (b) β (b) = ( T (b) ) -1 ∈S v 1 π ψ (b,S) ẑ(b) y -m (b) tr ee (x ) = ( T (b) ) -1 ∈U v α E (b) (3.53)
where

E (b) = ψ (b,S) ẑ(b) (y -m (b) tr ee (x )) with ∈U v E (b) = 0. We have that ||ẑ (b) || 2 = 1 and sup ∈U v | m (b) tr ee (x ))| ≤ C for all ∈ U v and b = 1, . . . , B, then: || E (b) || 2 2 2C 2 .
Following the same lines as in lemma 4, we get that it exists a positive constant C0 not depending on b such that

1 N 2 v E p ∈U v α E (b) 2 2 C0 n v
, for all b = 1, . . . B.

(3.54)

We obtain then from relations (3.52) and (3.53) that:

E p β (b) -β (b) 2 2 E p N 2 v ||( T (b) ) -1 || 2 2 1 N 2 v ∈U v α E (b) 2 2 N 2 v n 2 0v 1 N 2 v E p ∈U v α E (b) 2 2 N 2 v n 2 0v C0 n v c3 n v n 2 0v (3.55)
by assumption (H14).

Result 3.9.4. Consider a sequence of sample RF estimators { t r f }. Then, there exist positive constants C1 , C2 such that

1 N v E p | tr f -t y | C1 √ n v + C2 n 0v . If n u v n 0v = O(1) with 1/2 u 1, then E p 1 N v t r f -t y C √ n v
, with ξ-probability one.

Proof. We use the decomposition given in relation (3.51):

1 N v (t r f -t y ) = 1 N v ( t pgd -t y ) - 1 N v k ∈U v α k ( m r f (x k ) -m r f (x k )). Now, E p 1 N v k ∈U v α k ( m r f (x k ) -m r f (x k )) 1 B B b=1 1 N v E p k ∈U v α k ( m (b) tr ee (x k ) -m (b) tr ee (x k ))
and using relations (3.49) and (3.50), we get:

1 N v E p k ∈U v α k ( m (b) tr ee (x k ) -m (b) tr ee (x k )) E p 1 N v k ∈U v α k ẑ(b) k 2 β (b) -β (b) 2 E p 1 N v k ∈U v α k ẑ(b) k 2 2 E p β (b) -β (b) 2 2 .
We have that ||ẑ (b) k || 2 = 1 for all k ∈ U v and b = 1, . . . , B. We can show then by using the same arguments as in the proof of lemma 4, that there exists a positive constant C 0 such that

E p 1 N v k ∈U v α k ẑ(b) k 2 2 C 0 n v
which together with lemma 5 gives us that there exists a positive constant C2 such that

1 N v E p k ∈U v α k ( m r f (x k ) -m r f (x k )) C2 n 0v . (3.56)
Now,

1 N v E p tr f -t y 1 N v E p t pgd -t y + 1 B B b=1 1 N v E p k ∈U v α k ( m (b) tr ee (x k ) -m (b) tr ee (x k )) C1 √ n v + C2 n 0v
by using lemma 4 and relation (3.56).

Result 3.9.5. Consider a sequence of RF estimators

{ t r f }. Assume that n u v n 0v = O(1) with 1/2 < u 1. Then, √ n v N v t r f -t y = √ n v N v t pgd -t y + o P (1).
Proof. We have

√ n v N v t r f -t y = √ n v N v t pgd -t y + √ n v N v k ∈U v α k ( m r f (x k ) -m N ,r f (x k )). (3.57) Now, √ n v N v k ∈U v α k ( m r f (x k ) -m r f (x k )) = √ n v N v k ∈U v α k ( m r f (x k ) -m r f (x k )) + √ n v N v k ∈U v α k ( m r f (x k ) -m r f (x k )). (3.58) Relation (3.56) gives us that √ n v N v k ∈U v α k ( m r f (x k ) -m r f (x k )) = O P √ n v n 0v = o P (1) (3.59) provided that n u v n 0v = O(1) with 1/2 < u 1.
Consider now the second term from the right-side of relation (3.58). We have:

n v N 2 v E p k ∈U v α k ( m r f (x k ) -m r f (x k )) 2 n v N 2 v (1 + λ) 2 λ 2 k ∈U v E p m r f (x k ) -m r f (x k ) + n v N 2 v k ∈U v k, ∈U v E p | m r f (x k ) -m r f (x k )|| m r f (x ) -m r f (x )| max k ∈U v |E p (α k α | PS )| n v N v (1 + λ) 2 λ 2 + C 1 λ 2 1 N v k ∈U v E p m r f (x k ) -m r f (x k ) 2 = o(1),
by assumptions (H14), (H15), ( H16) and (H17). It follows then that Result 3.9.6. Consider a sequence of population RF estimators { t r f }. Assume also that

√ n v N v k ∈U v α k ( m r f (x k ) -m r f (x k )) = o P (1). ( 3
n u v n 0v = O(1) with 1/2 < u 1.
Then, the variance estimator V r f ( t r f ) is asymptotically design-consistent for the asymptotic variance AV p t r f . That is,

lim v→∞ E p n v N 2 v V r f ( t r f ) -AV p ( t r f ) = 0. (3.61)
Proof. The proof follows the same steps as those of result (3.9.3). We need to show that

E p m r f (x k ) -m r f (x k ) 2 = o(1), (3.62) uniformly in k ∈ U v . We have m r f (x k ) -m r f (x k ) = m r f (x k ) -m r f (x k ) + m r f (x k ) -m r f (x k ) and E p ( m r f (x k ) -m r f (x k )) 2 1 B B b=1 E p ( m (b) tr ee (x k ) -m (b) tr ee (x k )) 2 1 B B b=1 E p ||ẑ (b) k || 2 2 β (b) -β (b) 2 2 1 B B b=1 E p β (b) -β (b) 2 2 c3 n v n 2 0v = o(1)
by lemma 5 and provided that

n u v n 0v = O(1) with 1/2 < u < 1.
The result (3.62) follows then by using also assumption (H17).

Introduction

In the last decade, the interest in machine learning methods has been growing in national statistical o ces (NSO). These data-driven methods provide exible tools for obtaining accurate predictions. The increasing availability of data sources (e.g., big data sources and satellite information) provides a rich pool of potential predictors that may be used to obtain predictions at di erent stages of a survey. These stages include the nonresponse treatment stage (e.g., propensity score weighting and imputation) and the estimation stage (e.g., model-assisted estimation and small area estimation). The imputation stage is the focus of the current paper.

Item nonresponse refers to the presence of missing values for some, but not all, survey variables. Frequent causes of item nonresponse include refusal to answer a sensitive question (e.g., income) and edit failures. The most common way of treating item nonresponse in NSOs is to replace a missing value with a single imputed value, constructed on the basis of a set of p explanatory variables, X = (X 1 , . . . , X p ), available for both respondents and nonrespondents. A variety of imputation procedures are available, ranging from simple (e.g., mean, historical and ratio imputation) to more complex (e.g., nonparametric procedures); e.g., see Chen and Haziza (2019) for an overview of imputation procedures in surveys. Every imputation procedure makes some (implicit of explicit) assumptions about the distribution of the variable Y requiring imputation. This set of assumptions is often referred to as an imputation model. At the imputation stage, it is therefore important to identify and include in the model all the appropriate explanatory variables that are predictive of the variable requiring imputation and determine a suitable model describing the relationship between Y and the set of explanatory variables X.

We distinguish parametric imputation procedures from nonparametric imputation procedures. In parametric imputation, the shape of the relationship between Y and X is predetermined; e.g., linear and generalized linear regression models. However, point estimators based on parametric imputation procedures may su er from bias if the functional form is misspeci ed or if the vector X fails to include interactions or predictors accounting for curvature. In contrast, with nonparametric methods, the shape of the relationship between Y and X is left unspeci ed. These methods have the ability to capture nonlinear trends in the data and tend to be robust to the non-inclusion of interactions or predictors accounting for curvature.

Commonly used nonparametric methods include kernel smoothing, local polynomial regression and spline-based regression models. While these methods provide some robustness against model misspeci cation, they tend to breakdown when the number predictors is large, a problem known as the curse of dimensionality. To mitigate this problem, one may employ additive models [START_REF] Hastie | Generalized additive models[END_REF]. However, when the dimension of X is very large, these models tend to fail and machine learning methods may provide an interesting alternative. The class of machine learning methods, that includes tree-based models such as random forests and boosting methods, provide more exible approaches able to adapt to complex non-linear and non-additive relationships between the survey variable requiring imputation and a set of predictors. These methods may also prove useful in the case of large data sets exhibiting a large number of observations on a large number of variables. Many machine learning procedures are relatively computationally e cient and can produce accurate predictions by o ering the user a kind of automatic variable selection that may prove useful in a high-dimensional setting.

However, both a theoretical treatment and an empirical comparison of machine learning imputation procedures in the context of missing survey data are currently lacking. In this paper, we aim to ll the latter gap by conducting an extensive simulation study that investigates the performance of several nonparametric and machine learning procedures in terms of bias and e ciency. To that end, we generated several nite populations with relationships between Y and X, ranging from simple to complex and generated the missing values according to several nonresponse mechanisms. We also considered both a low-dimensional and high dimensional settings. The simulation setup and the models are described in Section 4.4. We restricted our attention to population totals (Section 4.4) and population quantiles (Section 4.5) as the target parameters. The following procedures were included in our comparisons: the score method [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF]Beaumont, 2007, Little, 1986), K nearest-neighbour [START_REF] Chen | Nearest neighbor imputation for survey data[END_REF], additive models based on B-spline regression, regression trees [START_REF] Breiman | Classi cation and regression trees[END_REF], random forests [START_REF] Breiman | Random forests[END_REF], tree-based boosting methods [START_REF] Friedman | Greedy function approximation: A gradient boosting machine[END_REF] including XGBoost [START_REF] Chen | XGBoost[END_REF] and Bayesian additive regression trees [START_REF] Chipman | BART: Bayesian additive regression trees[END_REF], the cubist algorithm [START_REF] Quinlan | Combining instance-based and model-based learning[END_REF][START_REF] Quinlan | Learning with continuous classes[END_REF] and support vector regression [START_REF] Vapnik | Statistical Learning Theory[END_REF][START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]. In Section 4.3, we describe these models and the corresponding imputation procedures.

In recent years, machine learning procedures have received some attention in a survey sampling context. In the ideal situation of 100% response, the theoretical properties of model-assisted estimation procedures based on regression trees [START_REF] Mcconville | Automated selection of post-strata using a model-assisted regression tree estimator[END_REF] and random forests [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF] have been recently established. [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF] studied the theoretical properties of point and variance estimators based on random forests in a context of imputation for item nonresponse and data integration; see also De Moliner and Goga (2018), [START_REF] Tipton | Properties of endogenous post-strati ed estimation using remote sensing data[END_REF] for applications of random forests in surveys. A number of empirical investigations have been conducted to assess the performance of machine learning procedures in a context of propensity score estimation for unit nonresponse; e.g., [START_REF] Lohr | Using classi cation and regression trees to model survey nonresponse[END_REF], [START_REF] Gelein | Handling missing data with superpopulation model, design-based approach and machine learning[END_REF] and [START_REF] Kern | Tree-based machine learning methods for survey research[END_REF].

The machine learning procedures described in Section 4.3 slightly di er from their traditional implementation because of the inclusion of the sampling weights in the construction of imputed values. However, it should be noted that most of the machine learning software packages for obtaining predicted values assume simple random sampling and cannot handle unequal weights. Modifying machine learning algorithms to account for unequal weights may prove challenging.

When the design features (e.g., sampling weights, stratum indicators, etc.) are related to the survey variable requiring imputation, failing to incorporate them in the models may lead to biased estimators. To cope with this issue, we suggest to include all the appropriate design variables in the speci cation of the model. Standard machine learning software packages may then be safely used for creating a set of imputed values. In Section 4.4, we use Poisson sampling with inclusion probabilities proportional to a size variable X to select repeated samples from the nite population. The size variable X being related to the variable requiring imputation, including the X-variable in the speci ed models led to satisfactory results.

Preliminaries

Consider a nite population U = {1, 2, ..., N } of size N. Let Y denote a survey variable and y i be the y-values attached to unit i, i = 1, • • • , N. We are interested in estimating (i) the nite population total of the y-values, t y = i ∈U y i and (ii) the nite population quantile of order γ de ned as

Q γ := inf {t ∈ R; F N (t) γ} , where F N (t) = i ∈U 1 (y i t) /N
denotes the nite population distribution function.

From U, we select a sample S, of size n, according to a sampling design P (S = s) with rst-order inclusion probabilities π i = Pr(i ∈ S).

A complete data estimator of t y is the well-known Horvitz-Thompson estimator

t π = i ∈S y i π i , (4.1)
which is design-unbiased for t y provided that π i > 0 for all i ∈ U. A complete data estimator of the nite population quantile Q γ is given by

Q γ := inf t ∈ R; F(t) γ , (4.2)
where

F(t) = 1 N i ∈S 1 (y i t) π i (4.3)
with N = i ∈S 1/π i denoting the Horvitz-Thompson estimator of the population size N. Under mild regularity conditions [START_REF] Wang | On asymptotic normality and variance estimation for nondi erentiable survey estimators[END_REF], the complete data estimator Q γ is design-consistent for Q γ .

In practice, the Y -variable may be prone to missing values. Let r i be a response indicator such that r i = 1 if y i is observed and r i = 0, otherwise. Let S r = {i ∈ S; r i = 1} denote the set of respondents, of size n r , and S m = {i ∈ S; r i = 0} the set of nonrespondents, of size n m , such that S r ∪ S m = S and n r + n m = n. Available to the imputer is the data (y i , x i ) for i ∈ S r as well as the values of the vector x i for i ∈ S m .

Let y i be the imputed value used to replace the missing value y i and

y i = r i y i + (1 -r i ) y i
be the ith value of the Y -variable after imputation. Point estimators of t y and Q γ after imputation, often referred to as imputed estimators, are readily obtained from the complete data estimators (4.1) and (4.2) by replacing y i with y i . This leads to

t imp = i ∈S y i π i (4.4) and Q γ,imp = inf t ∈ R; F imp (t) γ , (4.5)
where

F imp (t) = 1 N i ∈S 1 ( y i t) π i (4.6)
denotes the imputed estimator of F N (t).

Remark 4.2.1. The population total t y , the distribution function F N (t) and the quantile of order γ, Q γ , may all be obtained as the solution of the following census estimating equation (Binder, 1983, Chen andHaziza, 2019):

U N (θ N ) = i ∈U u(y i ; θ N ) = 0, (4.7)
where θ N is a generic notation denoting a nite population parameter and u(y i ; θ) is a function of θ N . We assume that a solution to (4.7) exists and is unique. For instance, the population total t y can be obtained as a solution of (4.7) with u(y i ; θ N ) = y in -1 π i θ N ; the nite population distribution function F N (t) can be obtained as a solution of (4.7) with u(y i ; θ N ) = 1 (y i t) -θ N . Finally, the quantile Q γ of order γ can be obtained as a solution of (4.7) with u(y i ; θ N ) = 1 (y i θ N ) -γ. Other nite population parameters can be obtained as a solution of (4.7); e.g., see Chen and Haziza (2019). The imputed estimators t imp , Q γ,imp and F imp (t) given respectively by (4.4)-(4.6) can be obtained by solving the following sample estimating equation:

U imp ( θ imp ) = i ∈S 1 π i u( y i ; θ imp ) = 0,
where θ imp denotes an imputed estimator of θ N .

To construct the imputed values y i , we postulate the following imputation model ξ:

E ξ (y i |x i ) = f (x i ), (4.8) V ξ (y i |x i ) = σ 2 i , Cov ξ y i , y j |x i , x j = 0 for i j,
where f is an unknown function. Often, the variance structure σ 2 i is assumed to have the form σ 2 i = σ 2 a i , where a i > 0 is a known coe cient attached to unit i and σ 2 is an unknown parameter.

We assume that the data are Missing At Random [START_REF] Rubin | Inference and missing data[END_REF]:

f (y i |x i , r i = 1) = f (y i |x i , r i = 0). (4.9)
That is, we assume that the distribution of Y given x is the same for both respondents and nonrespondents.

If Condition (4.9) holds, the imputed values can be safely generated from f (y i |x i , r i = 1), which can be estimated from the observed data. In the context of imputation, the properties of point estimators are evaluated with respect to the joint distribution induced by the imputation, the sampling design and the unknown nonresponse mechanism. This framework is often referred to as the ξ pq-framework (Chen and Haziza, 2019). Note that our simulation setup in Section 4.4 is consistent with the ξ pq-framework as the simulation process involves (i) generating repeated nite populations; (ii) selecting a sample from each of population and (iii) generating a set of response indicators in each sample. Deterministic imputation consists of replacing the missing y i by y i = f (x i ), where f is an estimator of the unknown regression function f based on the responding units i ∈ S r . However, deterministic imputation methods tend to distort the distribution of the survey variable Y requiring imputation, potentially leading to biased estimators of quantiles (Chen andHaziza, 2019, Haziza, 2009). To cope with this issue, one can recourse to random imputation that consists of adding an appropriate amount of random noise to the deterministic value f (x i ). More speci cally, let e j := σ -1 j {y jf (x j )} for j ∈ S r , where σ j of an estimator of σ j (see Remark 4.2.2 below). We de ne the standardized residual e j = e j -∈S r w e ∈S r w , j ∈ S r .

In the case of random imputation, the missing y i is replaced by (4.10) where e i is selected at random from the set of standardized residuals { e j } j ∈S r with probability w j / ∈S r w .

y i = f (x i ) + σ i e i ,
Remark 4.2.2. To obtain an estimator σ i of σ i , one can postulate a model E(

2 i | x i ) = m(x i )
, where m is an unknown function. An estimator σ 2 i of σ 2 i is obtained by tting a parametric or a nonparametric procedure with the square residuals e 2 i as the response and x i as the set of predictors. In Section 4.3, except for the parametric imputation procedure discussed in Section 4.3.1, all the other procedures (Section 4.3.2-4.3.9) are nonparametric. In Section 4.4, these procedures are compared empirically in terms of bias and e ciency under a variety of settings.

A description of imputation methods

Parametric regression imputation

Parametric regression assumes that the rst moment (4.8) is given by

E ξ (y i |x i ) = f (x i , β), (4.11)
where β is a vector of coe cients to be estimated and f (•) is a predetermined function. An estimator β of β is obtained by solving the following estimating equations based on the responding units:

i ∈S r w i σ 2 i {y i -f (x i , β)} ∂ f (x i , β) ∂ β = 0, (4.12)
where w i > 0 is a weight attached to element i. Common choices for w i include w i = 1 and w i = π -1 i (Chen and Haziza, 2019). The imputed value y i under deterministic parametric regression imputation is given by

y i = f (x i , β), i ∈ S m . (4.13)
A special case of (4.13) is f (x i , β) = x i β, which corresponds to the customary linear regression model. In this case, the imputed value (4.13) reduces to (4.14) where

y i = x i β, i ∈ S m ,
β = j ∈S r w j σ -2 j x j x j -1 j ∈S r w j σ -2 j x j y j . (4.15)
The imputed value y i given by (4.14) can be written as a weighted sum of the respondent y-values: (4.16) where w i j = x i j ∈S r w j σ -2 j x j x j -1 w j σ -2 j x j . If the intercept is among the X-variables, then j ∈S r w i j = 1 for all i ∈ S m . A random counterpart of (4.13) is given by (4.10).

y i = j ∈S r w i j y j , i ∈ S m ,
Another important special case of (4.13) is the logistic regression model,

f (x i , β) = exp(x i β)/(1 + exp(x i β)),
which can be used for modeling binary variables. An estimator of β is obtained by solving (4.12), which requires a numerical algorithm such as the Newton-Raphson procedure. To eliminate the possibility of an impossible imputed value, a missing value to a 0 -1 variable is typically imputed by y i , where y i is a realization of a Bernoulli variable with parameter f (x i , β).

Under deterministic or random parametric regression imputation, the imputed estimator t imp is consistent for t y provided that the rst moment of the imputation model (4.8) is correctly speci ed. However, this type of imputation may lead to biased estimators of quantiles. In contrast, the use of a random parametric regression imputation procedure tend to preserve the distribution of the variable requiring imputation, leading to valid estimators; see Chen and Haziza (2019) for a discussion.

Imputation classes : the score method

The score method [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF]Beaumont, 2007, Little, 1986) consists of partitioning the sample S into H (say) imputation classes and imputing the missing values within each class independently from one class to another. It can be implemented as follows:

Step 1: For all i ∈ S, compute the preliminary values y LR i = x i β, where β is given by (4.15).

Step 2: Compute the empirical quantiles q 1 , q 2 , . . . , q H-1 of order 1/H, 2/H, . . . , (H -1)/H of the y LRvalues.

Step 3: Split the sample S into H classes, C 1 , . . . , C h , . . . , C H , such that

C h = i ∈ S : y LR i ∈ [q h-1 ; q h ) , h = 1, . . . , H,
with q 0 = -∞ and q H = +∞.

It is common practice to use either mean imputation or random hot-deck imputation within classes. For mean imputation, the imputed value for missing y i in the hth imputation class is given by

y i = j ∈S r ∩C h w j y j j ∈S r ∩C h w j = j ∈S r ∩C h w i j y j , i ∈ S m ∩ C h ,
where w i j = w j / j ∈S r ∩C h w j are the same for all i ∈ S m ∩ C h and j ∈S r ∩C h w i j = 1 for all i ∈ S m ∩ C h . For random hot-deck imputation, the imputed value is given by y i = y j , where the donor j ∈ S r ∩ C h is selected at random from the set of donors belonging to the hth imputation class with probability w j / j ∈S r ∩C h w j . Note that random hot-deck imputation within classes can be viewed as mean imputation within classes with added residuals.

K-nearest neighbours imputation

K-nearest neighbour (KNN) imputation is one of the simplest and widely used nonparametric imputation procedures. No explicit assumption is made about the regression function f relating Y and X. KNN imputation consists of replacing the missing value of a recipient by the weighted average of the y-values of its K closest respondents in terms of the X-variables.

with coe cients determined by a least squares criterion computed on the data (y i , x i ) i ∈S r [START_REF] Goga | B-spline based imputation procedures for the treatment of item nonresponse in surveys[END_REF]. The missing value y i is then imputed by y i = f (x i ), where

f (x i ) = q =1 β B (x i ) = b i β, x i ∈ [0; 1], (4.17) with b i = (B (x i )) q =1
denoting the vector of B-spline basis functions, and β = ( β )

q =1 minimizes β = arg min β ∈R q j ∈S r w j y j - q =1 β B (x j ) 2 = j ∈S r w j b j b j -1 j ∈S r w j b j y j ; (4.18)
see [START_REF] Goga | B-spline based imputation procedures for the treatment of item nonresponse in surveys[END_REF]. The expression of β is similar to that obtained with linear regression imputation given by (4.15) but unlike (4.15), the estimator (4.18) uses the B-spline functions B 1 , . . . , B q , whose number can vary as a function of the number of knots κ and the order v of the B-spline functions. The degree v of the piecewise polynomial does not seem to have a great impact on the model ts if a large enough number of interior knots is used [START_REF] Ruppert | Semiparametric regression, volume 12 of Cambridge Series in Statistical and Probabilistic Mathematics[END_REF]. This is why quadratic or cubic splines are mostly used in practice and an adequate number of interior knots will allow to obtain exible ts that capture local non-linear trends in the data. Knots are usually placed at the X-quantiles and their number may have a great e ect on the model ts: a large value of κ will lead to over tting, in which case a penalization criterion may be used in (4.18), while a small value of κ may lead to under tting. [START_REF] Ruppert | Semiparametric regression, volume 12 of Cambridge Series in Statistical and Probabilistic Mathematics[END_REF] give a practical rule for choosing the number κ of interior knots :

κ = min 1 4 × number of unique x i , 35 .
The imputed value (4.17) with B-spline regression can be also written as a weighted sum of the respondent y-values similar to (4.16), y i = j ∈S r w i j y j for all i ∈ S m with weights now given by

w i j = b i j ∈S r w j b j b j -1
w j b j . These weights do not depend on the y-values as in linear regression imputation and j ∈S r w i j = 1 since q j=1 B j (x) = 1 for all x ∈ [0; 1]. Unlike linear regression imputation, the weights w i j are now local due to the B-spline functions ensuring more exibility to model local nonlinear trends in the data.

We now turn to the multivariate case. For ease of presentation, we con ne to the case of two predictors, X 1 and X 2 . Additive models provide a simple way to model nonlinear trend in the data [START_REF] Hastie | Generalized additive models[END_REF] and extend the standard linear model by allowing non-linear functions between the response variable Y and each of the explanatory variables, while maintaining additivity. In the case of two predictors, the relationship between Y and X 1 , X 2 is expressed as a linear combination of unknown smooth functions f 1 and f 2 :

y i = α + f 1 (x i1 ) + f 2 (x i2 ) + i , (4.19)
where the i 's are independent errors with mean equal to zero. The model (4.19) is restricted to be additive and does not account for the potential interactions among the predictors. Accounting for interactions between X 1 and X 2 would require the additional predictor X 1 X 2 to be included in the model, leading to

y = f 1 (x 1 ) + f 2 (x 2 ) + f 3 (x 1 , x 2 ) + ξ,
where f 3 is a low-dimensional interaction function tted by using two-dimensional smoothers, such as local regression or two-dimensional splines. This is beyond the scope of this article. When the number of predictors is large, the number of potential interactions may be considerable, making the implementation of this procedure challenging. In such situations, random forests and boosting, discussed in sections 4.3.6 and 4.3.7, provide more exible approaches. But, as pointed out by [START_REF] James | An Introduction to Statistical Learning with Applications in R[END_REF], additive models provide a useful compromise between linear and fully nonparametric models. The unknown functions f 1 and f 2 in (4.19) can be estimated by using two B-spline basis B 1 = {B 11 , . . . , B 1q 1 } and B 2 = {B 21 , . . . , B 2q 2 }, which leads to

f 1 (x i1 ) = q 1 =1 β 1 B 1 (x i1 ) and f 2 (x i2 ) = q 2 =1 β 2 B 2 (x i2
), where β 1 and β 2 are determined, as before, by a least square criterion. To ensure the identi ability of α, additional constraints such as n r i=1 f 1 (x i1 ) = n r i=1 f 2 (x i2 ) = 0 are usually imposed. With these constraints, the estimators ( α, β 1 , β 2 ) are simply obtained as a regression coe cient estimator, for β 1 = ( β 1 )

q 1 =1 and β 2 = ( β 2 ) q 2
=1 . The imputed value for missing y i is given by

y i = α + f 1 (x i1 ) + f 2 (x i2 ), i ∈ S m .
(4.20)

In practice, a back tting algorithm is used to compute f 1 (•) and f 2 (•) iteratively [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]. However, when the number p of explanatory variables is large, the algorithm may not converge and additive models tend to breakdown. Finally, random versions of (4.17) and (4.20) are obtained by adding random residuals as in (4.10).

Regression trees

Regression trees through the CART algorithm have been initially suggested by [START_REF] Breiman | Classi cation and regression trees[END_REF]. Treebased methods are simple to use in practice for both continuous and categorical variables and useful for interpretation. They form a class of algorithms which recursively split the p-dimensional predictor space, the set of possible values for the X-variables, into distinct and non-overlapping regions of R p . The prediction f tr ee (x i ) at point x i corresponds to the average of the respondent y-values falling in the same region as unit i. When the number of X-variables is not too large, the splitting algorithm is quite fast, otherwise it may be time-consuming. Following [START_REF] Creel | Creating imputation classes using classi cation tree methodology[END_REF], we slightly adapt the original CART algorithm as well as the estimation procedure of f (•). The CART algorithm recursively searches for the splitting variable and the splitting position (i.e., the coordinates on the predictor space where to split) leading to the greatest possible reduction in the residual mean of squares before and after splitting.

More speci cally, let A be a region or node and let #(A) the number of units belonging to A. A split in A consists of nding a pair ( , z), where is the variable coordinates taking value between 1 and p, and z is the position of the split along the th coordinate, within the limits of A. Let C A be the set of all possible pairs ( , z) in A. The splitting process is performed by searching for the best split ( * , z * ) in the sense that

( * , z * ) = arg max ( ,z)∈ C A L( , z) (4.21) with L( , z) = 1 #(A) i ∈S r 1(x i ∈ A) (y i -ȳA ) 2 -y i -ȳA L 1(X i < z) -ȳA R 1(X i z) 2 , (4.22)
where X i j is the measure of jth variable X j for the ith individual, A L = {X ∈ A; X < z}, A R = {X ∈ A; X z} and X the th coordinate of X; ȳA is the average of y i for those units i such that x i ∈ A. In (4.21), 1(x i ∈ A) = 1 if x i ∈ A, and 1(x i ∈ A) = 0, otherwise. From (4.21), the best split ( * , z * ) is the one that produces a tree with the smallest residuals sum of squares (James et al., 2015, Chap. 8); that is, we seek ( * , z * ) that minimizes

( * , z * ) = arg min ( ,z)∈ C A i ∈S r :x i ∈ A y i -ȳA L 2 1(X i < z) + i ∈S r :x i ∈ A y i -ȳA R 2 1(X i z) .
The missing y i is replaced by y i = f tr ee (x i ), which corresponds to the weighted average of the respondent y-values falling into the same region as i ∈ S m :

y i = j ∈S r w j 1(x j ∈ A(x i )) j ∈S r w j 1(x j ∈ A(x i )) y j , i ∈ S m , (4.23)
where A(x i ) is the region from R p containing the point x i . With tree-based methods, the imputed value y i can also be expressed as

y i = j ∈S r w i j y j , i ∈ S m , (4.24)
where w i j = w j 1(x j ∈ A(x i ))/ j ∈S r w j 1(x j ∈ A(x i )) with j ∈S r w i j = 1. With regression trees and tree-based methods in general, the non-overlapping A-regions obtained by means of the CART algorithm depend on the respondent data {(y i , x i )} i ∈S r ; i.e., the same set of X-variables with a di erent set of respondents will lead to di erent non-overlapping A-regions. The resulting imputed estimator is similar to a post-strati ed estimator based on adaptative post-strata. Regression trees are simple to interpret and often exhibit a small model bias. However, they tend to over t the data if each A-region contains too few elements. To cope with this issue, regression trees may be pruned, meaning that super uous splits (with respect to a penalized version of (4.21)) are removed from the tree. Pruning a regression tree tends to reduce its model variance at the expense of increasing the model bias; see [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]. A random version of (4.24) is obtained by adding random residuals as in (4.10). Bagging and boosting methods may be used to improve the e ciency of tree-based procedures. This is discussed next.

Random forests

Random forest [START_REF] Breiman | Random forests[END_REF] is an ensemble method which achieves better accuracy than treeregression methods by creating a large number of di erent regression trees and combining them to produce more accurate predictions than a single model would. Random forests are especially e cient in complex settings such as small sample sizes, high-dimensional predictor space and complex relationships [START_REF] Hamza | An empirical comparison of ensemble methods based on classi cation trees[END_REF], Díaz-Uriarte and de Andrés ( 2006), among others). Since the article of [START_REF] Breiman | Random forests[END_REF], random forests have been extensively used in various elds such as medicine [START_REF] Fraiwan | Automated sleep stage identi cation system based on time-frequency analysis of a single EEG channel and random forest classi er[END_REF], time series analysis [START_REF] Kane | Comparison of arima and random forest time series models for prediction of avian in uenza h5n1 outbreaks[END_REF], agriculture [START_REF] Grimm | Soil organic carbon concentrations and stocks on barro colorado island -digital soil mapping using random forests analysis[END_REF], to cite just a few. Recently, their theoretical properties have been established by [START_REF] Scornet | Consistency of random forests[END_REF].

There exist a number of random forest algorithms (see [START_REF] Biau | A random forest guided tour[END_REF] for discussion). A widely used algorithm proceeds as follows [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF]:

Step 1: Consider B bootstrap data sets D 1 , D 2 , ..., D B , obtained by selecting with replacement n r pairs

(y i , x i ) from D = {(y i , x i )} i ∈S r .
Step 2: In each bootstrap data set D b for b = 1, . . . , B, t a regression tree and determine the prediction f (b) tr ee for the unknown f in (4.8) as described in section 4.3.5. For each regression tree, only p variables randomly chosen among the p variables are considered in the search for the best split in (4.21).

Step 3: The imputed value for missing y i is obtained by averaging the predictions at the point x i of the B regression tree predictions:

y i = 1 B B b=1 f (b) tr ee (x i ), i ∈ S m , (4.25)
where f (b) tr ee (x i ) is the prediction for the unknown f in (4.8) computed at x i and obtained with the bth regression tree as described in Section 4.3.5. More speci cally, from (4.23), the prediction f (b) tr ee (x i ) corresponds to the weighted average of y-values for j ∈ S r falling in the same region

A (b) (x i ) containing i ∈ S m .
A random version of (4.25) is obtained by adding random residuals as in (4.10). Although random forests are based on fully-grown trees, the accuracy of the predictions is improved by considering bootstrap of units and model aggregation, a procedure called bagging and used in statistical learning for reducing the variability. The number B of regression trees should be large enough to ensure a good performance without harming the processing time; see [START_REF] Scornet | Tuning parameters in random forests[END_REF]. The second improvement brought by random forest is the random selection at each split of p predictors, achieving decorrelated trees. The value of p is typically chosen as p √ p [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]. In random forest algorithms, a stopping criterion is usually speci ed so that the algorithm stops once a certain condition (e.g., on the minimum number of units in each nal nodes) is met.

Least square tree-boosting and other tree-boosting methods

As in bagging, boosting [START_REF] Friedman | Greedy function approximation: A gradient boosting machine[END_REF]) is a procedure that can be applied to any statistical learning methods for improving the accuracy of model predictions and is typically used with tree-based methods. While bagging involves the selection of bootstrap samples to create many di erent predictions, boosting is an iterative method that starts with a weak t (or learner) and improves it at each step of the algorithm by predicting the residuals of prior models and adding them together to make the nal prediction.

To understand how boosting works, consider a regression tree with non-overlapping regions A 1 , . . . , A J , expressed as

T(x, Θ) = J j=1 γ j 1(x i ∈ A j ).
(4.26)

The parameter Θ = {γ j , A j } J j=1 is obtained by minimizing

Θ = arg min Θ J j=1 i:x i ∈ A j L(y i , γ j ) = arg min Θ i ∈S r L(y i , T(x i , Θ)), (4.27)
where L denotes a loss function; e.g., the quadratic loss function. With the latter, given a region A j , estimating the constant γ j is usually straightforward as γ j = y j the average the y-values belonging to A j . However, nding the regions { A j } J j=1 and solving (4.27) in a traditional way may prove challenging and computationally intensive as it requires optimizing over all the parameters jointly. To overcome this di culty, one may use a greedy top-down recursive partitioning algorithm to nd { A j } J j=1 as described in Section 4.3.5. Alternatively, one may split the optimization problem (4.27) into many simple subproblems that can be solved rapidly. Boosting uses the latter and considers that the unknown f has the following additive form:

f (x) = M m=1 T(x, Θ m ), (4.28)
where T(x, Θ m ) for m = 1, . . . , M are trees determined iteratively by using a forward stagewise procedure [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]: at each step, a new tree is added to the expansion without modifying the coe cients and parameters of trees already added. Each added tree, usually referred to as a weak-learner, has a small size and slowly improves the estimation of f in areas where it does not perform well. For the quadratic loss function, after accounting for the survey weights, the algorithm becomes:

Step 1: Initialize the algorithm with a constant value: f 0 (x i ) = 0 and

γ 0 = arg min γ ∈R i ∈S r w i (y i -γ) 2 = 1 i ∈S r w i i ∈S r w i y i .
Step 2: For m = 1 to M:

(a) Given the current model f m-1 , t the regression tree that best predicts the residuals values y if m-1 (x i ), i ∈ S r and get the terminal regions (A jm ) J m j=1 .

(b) Given the terminal regions A jm , the optimal constants γ jm are found as follows:

γ jm = arg min γ j m i ∈S r :x i ∈ A j m w i L(y i , f m-1 (x i ) + γ jm ) = arg min γ j m i ∈S r :x i ∈ A j m w i (y i -f m-1 (x i ) -γ jm ) 2 for j = 1, . . . , J m . (c) Update f m (x i ) = f m-1 (x i ) + T(x i , Θ m ) where Θ m = { A jm , γ jm } J m j=1 and T(x i , Θ m ) = J m j=1 γ jm 1(x i ∈ A jm ).
Step 3: Output f M (x i ) and get the imputed value

y i = f M (x i ). (4.29)
A random version of (4.29) is obtained by adding random residuals as in (4.10). The number M of trees should not be too large and, for better performances, [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF] recommend to consider the same number of splits J m = J at each iteration. The value of J re ects the level of dominant interactions between the X-variables. The value J = 2 (one split) produces boosted models with only main e ects without interactions, whereas the value J = 3 allows for two-variable interactions. Empirical studies suggest that J = 6 generally leads to good results. As in ridge regression, shrinkage is used with tree boosting. In this case, Step 2. (c) of the above algorithm is replaced by a penalized version:

f m (x i ) = f m-1 (x i ) + νT(x i , Θ m ),
where the parameter ν ∈ (0, 1), called learning rate, is used to penalized large trees; usually ν = 0.1 or 0.01. Both M and ν control the performance of the model prediction.

XGBoost

Chen and Guestrin (2016) suggested a scalable end-to-end tree boosting system called XGBoost which is extremely fast. Here, we adapt the algorithm in order to account for the survey weights. Consider again a tree with formal expression given in (4.26). This tree learning algorithm consists of minimizing the following objective function at the m-th iteration:

Θ m = arg min Θ m { i ∈S r w i L(y i , f m-1 (x i ) + T(x i , Θ m ))} + Ω(T(x, Θ m )), (4.30)
where the penalty function Ω(T(x, Θ m )) = γJ + λ 2 J j=1 γ 2 j penalizes large trees in order to avoid over tting. The search problem is optimized by using a second-order Taylor approximation of L, and ignoring the constant term, the new optimization problem reduces to:

Θ m = arg min Θ m J j=1       γ j i ∈S r :x i ∈ A j w i g i + 1 2 γ 2 j ( i ∈S r :x i ∈ A j w i h i + λ)       + γJ, (4.31)
where g i and h i are the rst and second-order derivatives of the loss function computed at f m-1 (x i ).

With the quadratic loss function,

g i = 2( f m-1 (x i ) -y i ) and h i = 2.
The new objective function from (4.31) is a second-order polynomial with respect to γ j , so the optimal γ j is easily obtained as γ * j = -( i ∈S r :x i ∈ A j w i g i )/( i ∈S r :x i ∈ A j w i h i + λ), leading to the optimal value of the objective function as

-(1/2) J j=1 ( i ∈S r :x i ∈ A j w i g i ) 2 /( i ∈S r :x i ∈ A j w i h i + λ) + γJ.
This value is then used next as a decision criterion in a greedy top-down recursive algorithm to nd the optimal regions A j of the m-th tree to be added.

Bayesian additive regression trees (BART)

Bayesian additive regression trees (Chipman et al., 2010, BART) is similar to boosting in the sense that the unknown regression function f has an additive form as in (4.28). While boosting is completely nonparametric, BART makes a Gaussian assumption on the model errors:

y i = f (x i ) + i , i ∼ N 0, σ 2 , where f (x) = M m=1 T(x, Θ m ) = M m=1 T m (x, Γ m
) is assumed to be a sum of tree functions and Γ m = γ j , γ 2 , . . . , γ J m is the set of parameter values associated with the J m terminal nodes in each tree T(x, Θ m ).

As stated in [START_REF] Chipman | BART: Bayesian additive regression trees[END_REF], although similar in spirit to gradient boosting, BART di ers from boosting algorithms both by the way it weakens the individual trees by relying on a Bayesian framework, but also on how it performs the iterative tting. More speci cally, a prior is speci ed for the parameters of the model (T 1 , Γ 1 ), (T 2 , Γ 2 ), . . . , (T m , Γ m ) and σ 2 . The prior of T m can be decomposed into three components :

1. The probability that a node at depth J is a terminal node is given by α (1 + J) -β for α ∈ (0; 1) , β ≥ 0.

2. The distribution on the splitting variable assignments in each interior node is uniform.

3. The distribution of the splitting value conditional on the chosen splitting variable is also uniform.

Borrowing the illustrative example of [START_REF] Chipman | BART: Bayesian additive regression trees[END_REF], with the parameters α = 0.95 and β = 2, trees with 1, 2, 3, 4, 5 terminal nodes receive prior probabilities of 0.05, 0.55, 0.28, 0.09 and 0.03, respectively. Therefore, as in boosting, the BART model tends to favor trees with a small number of terminal nodes. However, the process of restricting the depth of regression trees (or equivalently the number of terminal nodes) in BART is di erent from the one used in boosting. For boosting, the depth of the trees is xed by the user and is similar for all trees used in the forest. For BART, the user speci es a probability for the trees to have a certain number of terminal nodes. As a result, the number of terminal nodes is random rather tan xed. Therefore, it is likely that trees have only a small number of terminal nodes with the BART model, but this number can vary depending on the data at hand. For γ j , a conjugate prior is chosen to make computations simpler; e.g., p(γ jm |T m ) is assumed to be N (γ γ , σ 2 γ ). Similarly, a conjugate prior is chosen for σ 2 , e.g., the inverse chi-square distribution. To generate the posterior distribution, the authors suggest the use of a Gibbs sampler. For general guidelines about the choices of these parameters, see [START_REF] Chipman | BART: Bayesian additive regression trees[END_REF]. The imputed value for missing y i is obtained as with the general boosting algorithm given in Section 4.3.7, where the prediction of each regression tree is the weighted average of the values in the terminal node containing x i .

Cubist algorithm

Cubist is an updated implementation of the M5 algorithm introduced by [START_REF] Quinlan | Learning with continuous classes[END_REF] and [START_REF] Quinlan | Combining instance-based and model-based learning[END_REF]. It is an algorithm based on regression trees and linear models, among other ingredients. Initially, Cubist was only available under a commercial license. In 2011, the code was released as open-source. The algorithm proceeds as follows (Kuhn and Johnson, 2013, Chap. 8):

Step 1: Create a partition P = { A 1 , A 2 , ..., A T } of R p . To do so, let C A be the set of all possible splits in a node A of cardinality , that is, the set of all possible pairs (position, variable). Then, the split is performed using the following criterion:

L (z, j) = arg max (z,j)∈ C A i ∈S r y i - 1 n r j ∈S r y j 2 - h=1 n h n r i:x i ∈D h y i - 1 n r j :x i ∈D h y j 2 ,
where D 1 , . . . , D denote the non-terminal nodes after each of the -1 previous splits and n h denotes the cardinal of elements in the node D h .

Step 2: In each node, a linear model is tted between the survey variable Y and the auxiliary variables that have been used to split the tree. More speci cally, consider the jth terminal node A j . Then, there exists a path from the rst node to the current node A j in the graph formed by the tree. This path uses p j variables among the set {X 1 , X 2 , ..., X p }. For instance, assume that a partition of 5 elements is created by the tree shown in Figure 22. Then, the linear model in the node A 1 is tted using the variables that created the path in red, that is, X 1 , X 4 and X 6 , and so p 1 = 3 for this node. The linear model tted in the node A 4 uses only one variable, X 1 , (the green path), so p 4 = 1. The coe cients β j ∈ R p j of the linear model in the node A j are estimated using the customary weighted least squares criterion:

β j = arg min β j ∈R p j i ∈S r w i y i -β j x (j) i 2 1 x i ∈ A j , where x (j)
i is the vector containing the measurements of the p j variables for unit i.

Step 3: In each node, a backward elimination procedure is performed using the adjusted error rate (AER) criterion. For instance, in the jth terminal node, we have where p * denotes the number of variables used in the current model which predicts y i for a prediction at the point x i . Each variable in the initial model is dropped and the AER is recomputed.

AE R(A j ) = #(A j ) + p * #(A j ) -p * i ∈S r :x i ∈ A j |y i -y i |,
Terms are dropped from the model as long as the AER decreases.

Step 4: Once the tree is fully grown, it is pruned by removing unnecessary splits. Starting at the terminal nodes, the AER is computed with and without the node. Whenever the node does not result in a decrease of the AER, it is pruned. This process is performed until no more node can be removed.

Step 5: To avoid over-tting, a smoothing procedure is performed. Let y i(j) be the predicted value obtained by tting the linear model in the jth child node and y i(p) be the predicted value obtained from the direct parent node. These predictions are combined as

y i = ay i(j) + (1 -a) y i(p) ,
where a = V(e (p) ) -Cov(e (j) , e (p) )

V(e (j)e (p) )

with e i(j) = y iy i(j) denoting the ith coordinate of the vector e (j) , e i(p) = y iy i(p) denoting the ith coordinate of the vector e (p) and V(•) and Cov(•, •) denoting the empirical model variance and covariance, respectively.

Step 6: Cubist can be used as an ensemble model. Once the Cubist algorithm is tted, the subsequent iterations of the algorithm use the previously trained algorithm to de ne an adjusted response y (m) i so that the next iteration of the algorithm uses

y (m) i = y i -(y (m-1) i -y i ),
where y (m) i is the value of the adjusted response y i for the mth iteration of the Cubist algorithm.

Step 7: The nal imputed value for missing y i is derived using a K nearest-neighbour rule:

y i = 1 K K k=1 1 0.5 + d k (t k + y (k) -t k ), (4.32)
where d k denotes the distance between x i and the kth neighbor, t k denotes the outcome of the kth neighbor and t k its predicted value.

A random version version of (4.32) is obtained by adding random residuals as in (4.10).

Support vector regression

Support vector machines [START_REF] Cortes | Support-vector networks[END_REF][START_REF] Smola | A tutorial on support vector regression[END_REF][START_REF] Vapnik | Statistical Learning Theory[END_REF], 2000) belong to the class of supervised learning algorithms and may be used for regression analysis. We start by considering the linear regression model

f (x i ) = β 0 + x T i β, β 0 ∈ R, β ∈ R p ,
before discussing the case of nonlinear relationships. In the customary regression framework, the goal is to minimize the residuals sum of squares. In Support Vector Regression (SVR), the goal is to minimize a function of the residuals plus a L 2 -penalization on the regression coe cient:

S = i ∈S r V (y i -f (x i )) + λ 2 ||β|| 2 , (4.33)
where V is the so-called -insensitive error measure de ned as V (x) = 0 if |x| < and |x| -otherwise [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF] for > 0; ε can be viewed as the allowed tolerance for tting; see Figure 1 in [START_REF] Smola | A tutorial on support vector regression[END_REF]. The optimization problem (4.33) may not have solution and supplementary tolerances ξ i , ξ * i (called also "the slack variables") on the individual tted errors are considered [START_REF] Smola | A tutorial on support vector regression[END_REF]. There exist several ways for incorporating weights in the optimization problem, leading to di erent weighted support vector regression solutions. We consider the method suggested by [START_REF] Lee | Weighted support vector machine for quality estimation in the polymerization process[END_REF] and [START_REF] Han | On weighted support vector regression[END_REF]:

minimize β 1 2 ||β|| 2 + C i ∈S r w i ξ i + ξ * i (4.34) and subject to y i -β 0 -x T i β + ξ i , β 0 + x T i β -y i + ξ * i . ξ i , ξ * i > 0, (4.35)
where C > 0 is the tuning parameter that provides a trade-o between the smoothness of the tted function and the deviation from the training data and w i = w i / j ∈S r w j ∈ (0, 1) denotes the normalized sampling weight associated with unit i. As a result, the w i 's are all smaller than one. As argued by [START_REF] Han | On weighted support vector regression[END_REF], incorporating weights in the objective function as in (4.34) has the e ect of shrinking the estimators β j to di erent extents. The solution of (4.33) and (4.35) is given by

β = i ∈S r α i -α * i x i , which leads to f (x) = i ∈S r α i -α * i < x i , x > +β 0 , (4.36) 4.4
: 144

where < •, • > is an inner product and α i > 0 and α * i > 0 denote the Lagrange multipliers verifying the quadratic programming problem:

min α i ,α * i i ∈S r (α i + α * i ) - i ∈S r y i (α i -α * i ) + 1 2 i,j ∈S r (α i -α * i )(α j -α * j ) < x i , x j > subject to 0 ≤ α i , α * i ≤ C i := C × w i , i ∈S r (α i -α * i ) = 0 and α i α * i = 0.
As a result, only a subset of the solution values ( α i -α * i ) are nonzero and the associated data values are called the support vectors. The solution β is written as a linear combination of these support vectors. Moreover, the prediction f (x) uses only the support vectors and the inner products between x and x i without requiring the computation of β. This property is useful for extending the method to handle nonlinear relationships.

We now consider the case of a nonlinear and unknown function f . We approximate f in a basis of functions {φ m } M m=1 as follows:

f (x) = M m=1 β m φ m (x) + β 0
and β 0 and β = (β m ) M m=1 minimize (4.34) and

subject to y i -β 0 - M m=1 β m φ m (x i ) + ξ i , β 0 + M m=1 β m φ m (x i ) -y i + ξ * i . ξ i , ξ * i > 0.
(4.37)

A similar derivation as before leads to

β = i ∈S r α i -α * i φ(x i ) for φ(x i ) = (φ m (x i )) M m=1 and f (x) = i ∈S r α i -α * i K(x i , x) + β 0 , where K(x i , x) =< φ(x i ), φ(x) >= M m=1 φ m (x i )φ m (x)
is a positive de nite kernel [START_REF] Smola | A tutorial on support vector regression[END_REF]. The computation of f (x) involves φ(x) only through inner products and using a kernel function makes the computation of f (x) possible without requiring φ(x). All is needed is the knowledge of K. Using K, it is possible to solve the optimization problem in a higher-dimensional space without having to compute any product in this space. Common choices of K(•, •) include the Gaussian kernel K(x i , x j ) = exp -||x ix j || 2 and the polynomial kernel K(x i , x j ) = 1 + x i x j q , q = 2, 3, . . . . The imputed value for the missing y i is given by

y i = j ∈S r α j -α * j K(x j , x i ) + β 0 . (4.38)
A random version version of (4.38) is obtained by adding random residuals as in (4.10). The reader is referred to [START_REF] Smola | A tutorial on support vector regression[END_REF] for a discussion on how to estimate β 0 .

Simulation study: the case of population totals

We conducted an extensive simulation study to investigate the performance of the imputation procedures described in Section 4.3 in terms of bias and e ciency.

The setup

For each scenario, we repeated R = 5, 000 iterations of the following process:

(i) A nite population of size N = 10, 000 was generated. The population consisted of a survey variable Y and a set of predictors X 1 , . . . , X p .

(ii) From the nite population generated in Step (i), a sample, of size n, was selected according to a given probability sampling design.

(iii) In each sample, nonresponse to item Y was generated according to a given nonresponse mechanism.

(iv) The missing values in each sample were imputed using several imputation procedures.

We now give a more in-depth discussion of each of the steps (i)-(iv). We rst generated ve predictors X 1 , . . . , X 5 , according to the following distributions: X 1 followed a normal distribution, X 1 ∼ N (0, 1) ; X 2 followed a Beta distribution, X 2 ∼ Beta (3, 1) ; X 3 followed a Gamma distribution, X 3 ∼ 2 × Gamma (3, 2) ; X 4 followed a Bernoulli distribution, X 4 ∼ B (0.7) ; and X 5 followed a multinomial distribution, X 5 ∼ Mult (0.4, 0.3, 0.3) . The predictors X 1 -X 3 were continuous, whereas the predictors X 4 and X 5 were discrete. The predictors X 1 -X 3 were standardized so as to have a zero mean and a variance equal to one. Given the predictors X 1 -X 5 , we generated the continuous survey variables Y 1 , . . . , Y 8 , according to the following models:

• Y 1 = 2 + 2X 1 + X 2 + 2X 3 + N (0, 1); • Y 2 = 2 + 2X 1 + X 2 + 2X 3 + Pareto(1, 4); • Y 3 = 2 + X 1 + X 2 2 + X 3 + N (0, 1); • Y 4 = 2 + 2X 1 + X 2 + 3X 3 X 4 + 1.51(X 5 = 1) -21(X 5 = 2) + N (0, 1); • Y 5 = 2 + 5X 3 1 + 4X 2 2 + X 3 X 4 + 1.51(X 5 = 1) -21(X 5 = 2) + N (0, 1); • Y 6 = 2 + (2X 1 + X 2 + 2X 3 ) 2 + N (0, 1) + Beta(3, 1); • Y 7 = 2 + (2X 1 + X 2 + 3X 3 X 4 + 1.51(X 5 = 1) -21(X 5 = 2)) 2 + N (0, 1); • Y 8 = 4 cos (X 1 ) + N (0, 1);
and the binary survey variables as follows:

• Y 9 = 1(S 1 > 1/2),
where

S 1 = 0.1 + 0.79 exp {1 + 0.5 (0.75 + 2X1 + 2X 2 + 2X 3 -X 4 -X 3 X 4 +1.51(X 5 = 1) -21(X 5 = 2))} -1 ; • Y 10 = 1(S 2 > 1/2), where S 2 = 0.55 × Q + 0.02 -0.01X 3 2 with Q = exp {1 + 0.4 × (6.5 + 2X 1 + 2X 2 + 2X 3 -X 4 -X 3 X 4 +1.51(X 5 = 1) -21(X 5 = 2))} -1 . (4.39)
4.4

: 146

For the survey variables Y 2 and Y 6 , note that we have generated errors for non-normal distribution to assess the robustness of the BART procedure that assumes a Gaussian distribution for the errors.

From each population, we selected samples, of (expected) size n = 1, 000, according to two sampling designs: (a) simple random sampling without replacement and (b) Poisson sampling with probability proportional to the values of the variable X 5 ; i.e., π i = 1, 000 × (x 5i / i ∈U x 5i ) for all i ∈ U. Simple random sampling without replacement was used for estimating the nite population total of the continuous survey variables Y 1 -Y 6 and Y 8 and the binary variables Y 9 and Y 10 , whereas Poisson sampling was used for estimating the totals of the survey variables Y 4 and Y 7 .

In each sample, nonresponse to the survey variable Y , = 1, . . . , 10, was generated according to four nonresponse mechanisms. That is, the response indicators r i were generated from a Bernoulli distribution with probability p gi , g = 1, . . . , 4, where (NR1): p 1i = 0.1 + 0.79 exp {1 + 0.5 (0.75

+ 2x i1 + 2x i2 +2x i3 -x i4 -x i3 x i4 + 1.51(x i5 = 1) -21(x i5 = 2))} -1 ; (NR2): p 2i = 0.5; (NR3): p 3i = 0.55 × q i + 0.02 -0.01x 3 i2 ; (NR4): p 4i = 0.5 × q i + 0.13 -0.1 (sin(x i1 ) + cos(x i2 )) ;
where q i is the ith value of Q given by (4.39). In (NR1)-(NR4), the model parameters were set so as to obtain a response rate of about 50% in each sample.

In each sample, the missing values were imputed according to eleven imputation procedures described in section 4.3. Some of the imputation procedures required the speci cation of some parameters (e.g., regularization parameter, depth of a regression tree, choice of a kernel, etc.). We have included several con gurations to assess the impact of these parameters on the performance of these procedures. Based on the di erent con gurations, we ended up with twenty-seven imputation procedures. More speci cally, we included the following procedures: Procedure 1: "LR" : Deterministic linear regression imputation; see Section 4.3.1. Procedure 2: "MWCα" : Mean imputation within classes, where the number of units in each class was set to α ∈ {50, 100, 250, 500}; see Section 4.3.2.

Procedure 3: "HDWCα" : Random hot-deck imputation within classes, where the number of units in each class was set to α ∈ {50, 100, 250}; see Section 4.3.2.

Procedure 4: "KNN" : K-Nearest-Neighbours imputation with K = 1 and K = 5 nearest neighbours and the euclidian distance and implemented with the R-package caret; see Section 4.3.3.

Procedure 5: "AMSα" : Additive models based on cubic B-splines with α equidistant interiors knots placed at the x-quantiles, where α ∈ {5, 10} and implemented with the R-package mgcv; see Section 4.3.4.

Procedure 6: "CART" : Imputation through regression trees with the CART algorithm and implemented with the R-package rpart; see Section 4.3.5.

Procedure 7: "RF1" : Imputation through random forest with B = 1000 trees, one observation per terminal node and 1 predictor considered for the search in each split. "RF2": Random forest with B = 1000 trees, 5 observations per terminal node and √ p predictors considered for each split, where p is the number of X-variables used in the imputation model, in our case p = 5. "RF3" : Random forest with B = 1000 trees, 10 observations per terminal node and √ p predictors considered for each split. Simulations were implemented with the R-package ranger; see Section 4.3.6.

: 147

Procedure 8: "XGB1": XGBoost algorithm with M = 50 trees each one with J = 3 nal splits and a learning rate of 0.1. "XGB2": XGBoost algorithm with M = 100 trees with J = 6 and a learning rate of 0.05. "XGB3": XGBoost algorithm with M = 250 trees with J = 10 and a learning rate of 0.01. Simulations were implemented with the R-package xgboost; see Section 4.3.7.

Procedure 9: "BART" : Imputation through Bayesian additive regression trees. Simulations were implemented with the R-package bartMachine; see Section 4.3.7.

Procedure 10: "CUBIST1": Cubist with one model. "CUBIST2" : Cubist with ve models. "CUBIST3" : Cubist with 5 models and unbiased estimation. Simulations were implemented with the R-package Cubist; see Section 4.3.8.

Procedure 11: "SVR1": Support vector regression imputation with a Gaussian kernel and the ν objective function. "SVR2": Support vector regression imputation with a polynomial kernel of degree 3 and the -insensitive objective function. "SVR3": Support vector regression imputation with a Gaussian kernel and the -insensitive objective function. "SVR4": Support vector regression imputation with a linear kernel and the -insensitive objective function. Simulations were implemented with the R-package e1071; see Section 4.3.9.

The imputation procedures used in our simulations were based on an imputation model that included the predictors X 1 , . . . , X 5 , without any interaction terms. Except for random hot-deck imputation (Procedure 3) and nearest-neighbour imputation (Procedure 4 with K = 1), for the binary variables Y 9 and Y 10 , note that we have generated zeroes and ones from independent Bernoulli distributions with parameter y i , where y i denotes the predicted value associated with unit i. Whenever y i < 0, we set it to y i = 0. Similarly, when y i > 1, we set it to y i = 1.

As a measure of bias of the imputed estimator t imp given by (4.4), we computed the Monte Carlo percent relative bias de ned as

RB MC ( t imp ) = 100 × 1 R R r=1 ( t (r) imp -t y ) t y , (4.40)
where t (r) imp denotes the imputed estimator t imp at the rth iteration, r = 1, . . . , 5, 000. As a measure of e ciency, we computed the relative of e ciency, using the complete data estimator t π given by (4.1), as the reference. That is,

RE MC ( t imp ) = 100 × MSE MC ( t imp ) MSE MC ( t π ) , (4.41)
where

MSE MC ( t imp ) = R -1 R r=1 ( t (r) imp -t y ) 2 and MSE MC ( t π ) is de ned similarly.

Simulation results

In Section 4.4.2, we discuss the simulation results pertaining to the continuous survey variables Y 1 , . . 9 and10 show the minimum, the median and the maximum Monte Carlo percent absolute RB and Monte Carlo percent RE only. The size variable X 5 used to obtain the rst-order inclusion probabilities was included as a predictor in the imputation models. The results in Tables 9 and10 were consistent with those obtained for simple random sampling without replacement. Again, the best methods were CUBIST3, BART and XGB1 in terms of either bias or e ciency.

Ranking

Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 1 LR CUBIST3 AMS5 BART XGB3 CUBIST3 CUBIST3 CUBIST3 2 CUBIST3 LR AMS10 CUBIST3 AMS5 BART AMS5 AMS5 3 MW50 AMS5 BART CUBIST1 AMS10 SVR3 AMS10 AMS10 4 AMS5 MWC50 CUBIST3 CUBIST2 XGB1 SVR1 MWC50 XGB1 5 AMS10 AMS10 CUBIST2 XGB1 XGB2 XGB1 BART BART

Binary survey variables

In this section, we present the results pertaining to the binary variables Y 9 and Y 10 . Again, for each imputation procedure, we obtained 2 × 4 = 8 sets of results. Tables 11 and12 show the minimum, the median and the maximum Monte Carlo percent absolute RB and Monte Carlo percent RE, respectively. The ranking for binary survey variables was slightly di erent from that obtained for the continuous survey variables. Nearest-neighbor (NN) imputation procedure was the best in terms of bias and e ciency. Recall that NN imputation did not rank among the best procedures for the continuous variables. NN imputation was followed by CUBIST, XGBOOST and BART.

High-dimensional setting

In this section, we investigate the performance of a subset of the imputation procedures considered in Section 4.4.1 in a high-dimensional setting. To that end, we used data from the Irish Commission for Energy Regulation (CER) Smart Metering Project conducted in 2009-2010(CER, 2011) ) that focused on energy consumption and energy regulation2 . About 6000 smart meters were installed in Irish residences and businesses. The customer's electrical consumption was collected every half an hour over a period of about two years.

We considered a subset of the original data set. We ended up with a population of N = 6291 smart meters (households and businesses) for a period of 14 consecutive days. For each population unit i (household or business), we had 2 × 7 × 48 = 672 measurements denoted by X j = X(t j ), j = 1, . . . 672. Each of these 672 measurements represents the electricity consumption (in kW) at instant t j . We denote by x i j the value of X j recorded by the smart meter i for i = 1, . . . , N at instant t j . It should be noted that these variables were highly correlated among themselves with a condition number of the matrix N -1 X T X computed using all the data, of about 60.000.

We created four survey variables based on a subset of the auxiliary variables X 1 , . . . , X 672 : Three high and very high dimensional settings were considered: in the rst setting, the imputation models used the rst 15 auxiliary variables X 1 , ..., X 15 , in the data set. In the second and third settings, the imputation models were based on the rst 100 and 300 auxiliary variables X 1 , ..., X 100 , and X 1 , ..., X 300 , respectively.

Y 1 = 400 + 2X 1 + X 2 + 2X 3 + N (0, 1500); Y 2 = 400 + X 1 X 2 + 2X 3 + N (0,
To impute the missing values, we con ned to a subset of the imputation procedures considered in Section 4.4.1: additive models, BART, CUBIST, XGBoost, random forests, nearest-neighbour imputation and support vector regression. Linear regression imputation and mean imputation within 20 classes were also considered. It is well known that the quality of predictions based on linear models tend to deteriorate substantially in the presence of a very large number of auxiliary variables. To cope with this issue, we also considered principal components analysis as a reduction-dimension method; see [START_REF] Cardot | Calibration and partial calibration on principal components when the number of auxiliary variables is large[END_REF] Table 13 shows the Monte Carlo percent relative bias (RB) and relative e ciency (RE) for p = 15 predictors. Table 14 shows the results for p = 100 and p = 300 predictors. For each scenario, the best imputation procedures are highlighted in bold. Note that the relative e ciency is now computed with respect to the mean square error of the imputed estimator based on the true imputation model. The additive models were considered in the rst setting only (p = 15 variables) because their performance deteriorated rapidly as the number p of variables increased. For p = 100 and p = 300 the back tting algorithm did not reach convergence in most scenarios.

From Tables 13 and14, we note that CUBIST and XGBoost were the best method in the vast majority of the scenarios. These methods were followed by BART and random forests. As expected, additive models performed poorly, which illustrates the curse of dimensionality. It is worth pointing out that random forests performed better in the high-dimensional setting than they did in the low-dimension setting considered in section 4.4.1. Finally, the strategy based on principal components analysis did relatively well in most scenarios.

Simulation study: the case of population quantiles

In this section, we turn our attention to population quantiles. Except for nearest-neighbour imputation, we con ned to the random versions of the imputation procedures described in Section 4.3. The target parameters were the quantiles of order γ 1 = 0.25, γ 2 = 0.5 and γ 3 = 0.75 that correspond to the rst quartile, the median and the third quartile, respectively. We considered a subset of the scenarios described in Section 4.4.1. First, we con ned to the case of the survey variables Y 3 and Y 6 and the nonresponse mechanisms (NR1) and (NR3) described in Section 4.4.1, leading to 2 × 2 = 4 scenarios. Also, samples -0,0 -0,0 0,5 0,2 -0,1 -1,3 0,0 -0,0 -0,11 -0,1 -0,0 0,0 were selected according to simple random sampling without replacement only. In each sample, we computed the imputed estimator Q γ,imp given by (4.5) for γ 1 = 0.25, γ 2 = 0.5 and γ 3 = 0.75. As in Section 4.4, we computed the Monte Carlo percent relative bias of Q γ,imp and the relative e ciency, From Figures 27-29, Cubist displayed a very good performance in terms of bias and e ciency for the three quantiles. The procedure XGBoost led to good results for Q 0.25 and Q 0.75 but performed poorly for Q 0.5 . Similarly, BART performed very well for both Q 0.5 and Q 0.75 but exhibited a poor performance for Q 0.25 . Support vector machine (SVR3) did relatively well for both Q 0.5 and Q 0.75 but was outperformed by Cubist and XGBoost for Q 0.25 . Again, the Cubist algorithm seemed to be insensitive to the target parameter, the model that has generated the Y -variable and the nonresponse mechanism, at least in our experiments.

Final remarks

In this paper, we have conducted an extensive simulation study to compare several nonparametric and machine learning imputation procedures in terms of bias and e ciency. The imputation procedures were evaluated in the case of nite population totals of continuous and binary variables and for population quantiles under both simple random sampling without replacement and proportional-to-size Poisson sampling. The Cubist algorithm, BART and XGBoost performed very well in a wide variety of settings. In general, these methods seem to be highly robust to model misspeci cation and seem to have the ability to capture nonlinear trends in the data. Additive models based on B-splines performed well in the case of population totals when the number of explanatory variables was small but broke down for large values of p. Finally, random forests performed relatively well in a high-dimensional setting. In practice, the choice of an imputation procedure is not clear-cut and depends on the data at hand. If one is reasonably con dent about the correct speci cation of the rst moment of the imputation model (that includes the correct speci cation of the functional form and the correct speci cation of the vector of explanatory variables), parametric imputation procedures are expected to do well in terms of bias and e ciency. In addition, parametric imputation is simpler to understand and the results are easier to interpret, in general. In the case of complex/nonlinear relationships and/or in a high-dimensional setting, our empirical investigations suggest that machine learning procedures outperform traditional imputation procedures as they tend to be robust against model misspeci cation. However, these procedures require the speci cation of some regularization parameters. For instance, for XGBoost, one must specify the learning rate, the maximal depth and the coe cient of penalization. In support vector regression, the cost function and the kernel function must be selected, among others. In practice, the value for some of these parameters are determined through a cross-validation procedure. To keep the processing time at a reasonable level, all the regularization parameters were predetermined in our experiments. Overall, it seems that Cubist is an excellent choice as it performed well in all the scenarios, unlike its main competitors (e.g., XGBoost, BART, random forest, etc.) whose performance varied from one scenario to another. From a computational point of view, most procedures were e cient. One notable exception is BART that proved to be highly computer intensive with an average processing time approximately twenty times larger than what was required for the other procedures.

Drawing inferences from survey data requires a variance estimate. It is well known that imputed values should not be treated as observed values. Otherwise, the resulting variance estimates tend to be much smaller, on average, than the true variance, especially if the nonresponse rates are appreciable. In the last three decades, a number of variance estimation procedures have been proposed for obtaining variance estimates that account for sampling, nonresponse and imputation. The reader is referred to [START_REF] Haziza | Variance estimation procedures in the presence of singly imputed survey data: a critical review[END_REF] for a comprehensive overview of variance estimation procedures in the presence of singly imputed data sets. Estimating the variance of imputed estimators obtained through machine learning procedures is challenging and requires further research. If the sampling fraction is negligible, one can recourse to the bootstrap procedure of [START_REF] Shao | Bootstrap for imputed survey data[END_REF] that consists of selecting bootstrap samples according to a complete data bootstrap procedure and reimputing the missing values within each bootstrap sample using the same imputation method that was used on the original data. If a machine learning procedure is used to impute the missing data, the Shao-Sitter procedure may be highly computer intensive. When the sampling fraction is not negligible, the problem of bootstrap variance estimation is more intricate (Chen et al., 2019). To make the variance estimation process simpler for survey practitioners, it would be desirable to derive a "universal" variance estimator based on Taylor expansion procedures that could be applicable to a wide class of machine learning imputation procedures, at least in the case of negligible sampling fractions. This is currently under investigation.

Investigating the performance of deep learning methods in the context of imputation for missing survey data would constitute a promising direction for future research. There exist a wide class of deep learning procedures based on relatively sophisticated algorithms that proved to be extremely e cient in the context of unstructured data such as signal processing or text analysis. However, for deep learning procedures to "shine" in terms of e ciency typically requires a huge volume of unstructured data, which is seldom the case in surveys. In practice, most data sets in surveys consist of structured data and contains, at most, a few millions observations and a few hundred survey variables. As noted by [START_REF] Choley | Deep learning with Python[END_REF]: "(...) gradient boosting (such as XGBoost) is used for problems where structure data is available, whereas deep learning is used for perceptual problems such as image classi cation". We believe that the class of imputation procedures considered in this article, that includes bagging and boosting among others, o ers a number of very good options that may be applicable to virtually all the surveys conducted by NSOs.

Introduction

Since the seminal paper of [START_REF] Breiman | Random forests[END_REF], random forests have been used in a variety of applications including medicine [START_REF] Fraiwan | Automated sleep stage identi cation system based on time-frequency analysis of a single EEG channel and random forest classi er[END_REF], time series analysis [START_REF] Kane | Comparison of arima and random forest time series models for prediction of avian in uenza h5n1 outbreaks[END_REF], agriculture [START_REF] Grimm | Soil organic carbon concentrations and stocks on barro colorado island -digital soil mapping using random forests analysis[END_REF], missing data [START_REF] Stekhoven | MissForest-non-parametric missing value imputation for mixed-type data[END_REF], genomics [START_REF] Qi | Random forests for bioinformatics[END_REF] and pattern recognition [START_REF] Rogez | Randomized trees for human pose detection[END_REF]. Random forests belong to the class of ensemble models, whereby a collection of B regression trees are constructed and a prediction is generated from each of the B trees. Unlike many nonparametric statistical procedures (e.g., kernel predictors, k-nearest neighbors, splines), random forests perform relatively well with high-dimensional data; see e.g., [START_REF] Hamza | An empirical comparison of ensemble methods based on classi cation trees[END_REF] and Díaz-Uriarte and de [START_REF] Díaz-Uriarte | Gene selection and classi cation of microarray data using random forest[END_REF]. Some recent theoretical investigations [START_REF] Biau | Analysis of a random forests model[END_REF][START_REF] Klusowski | Universal consistency of decision trees in high dimensions[END_REF][START_REF] Scornet | Consistency of random forests[END_REF] also suggest that random forests adapt well to sparse situations.

In surveys, the problem of missing data is ubiquitous. Estimators of population totals based on complete cases only, often referred to as unadjusted estimators, tend to exhibit large biases when the proportion of missing data is appreciable and the behavior of the responding units is di erent from that of the nonresponding units. In this article, we focus on the problem of item nonresponse, a term used to describe the absence of information on some, but not all, survey variables for a sample unit. The missing values are replaced by a plausible value constructed on the basis of auxiliary variables available for both respondents and nonrespondents, a process known as imputation. A large number of imputation procedures have been developed to compensate for missing values and to reduce the nonresponse bias to the best possible extent. The reader is referred to [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF] and Chen and Haziza (2019) for comprehensive discussions of imputation procedures in a survey sampling setting. Every imputation procedure relies on some implicit or explicit assumptions about the distribution of the survey variable requiring imputation. This set of assumptions is called an imputation model. In this context, tree-based methods such as random forests may prove useful for obtaining a set of imputed values. Because they are nonparametric in nature, random forests tend to be robust against model misspeci cation. Also, with the emergence of large data sets in National Statistical O ces (NSO), random forests have attracted a lot of attention in NSOs in recent years and are being scrutinized as an alternative to traditional imputation procedures. However, to the best of our knowledge, a theoretical investigation on the properties of random forests in the context of imputation for missing survey data is currently lacking. Assumption (H19) is common in the missing data literature [START_REF] Rubin | Inference and missing data[END_REF] and is required in order to estimate the regression function from the observed data. It states that, given the covariates x k , the survey variable y k is independent of the response indicators r k .

We assume that the relationship between the survey variable Y and the set of auxiliary variables x can be described by the following imputation model:

ξ : y k = m(x k ) + k , k ∈ S r , (5.2)
where m(x) := E [Y |X = x] denotes the regression function, and { k } k ∈S r is a sequence of independent and identically distributed (i.i.d.) white noise. We assume that: i) the regression function m is continuous; ii) the distribution of the covariates P x is supported on Supp(P x ), a compact subset the unit cube [0; 1] p ; iii) the residuals have a compact support; the survey variable Y has a distribution absolutely continuous with respect to the Lebesgue measure. These assumptions imply that the survey variable Y is almost surely bounded.

Let m be a estimator of m tted on D n r := {(x k , y k ); k ∈ S r }. The imputed estimator t m of t y based on the imputation procedure m is given by

t m := k ∈S r y k π k + k ∈S m m(x k ) π k , (5.3)
where m(x k ) denotes the imputed value associated with k ∈ S m .

To establish the asymptotic properties of (5.3), we consider the asymptotic framework of [START_REF] Isaki | Survey design under the regression superpopulation model[END_REF]. We consider an increasing sequence of embedded nite populations {U v } v ∈N of size {N v } v ∈N . In each nite population U v , a sample S v , of size n v , is selected according to a sampling design P v with inclusion probabilities π k,v and π k ,v . While the nite populations are assumed to be embedded, we do not require this property to hold for the samples {S v } v ∈N . This asymptotic framework assumes that v goes to in nity, so that both the nite population size N v and the sample size n v go to in nity. To improve readability, we shall use the subscript v only in the quantities U v , N v and n v ; quantities such as π k,v and π k ,v shall simply be denoted as π k and π k .

We now describe a set of conditions on m su cient for establishing the L 2 -consistency of t m . An imputed estimator t m is said to be mean square consistent or L 2 -consistent for t y if

E 1 N v t m -t y 2 ----→ v→∞ 0.
(5.4) Throughout this chapter, the expectation and variance operators are evaluated with respect to the joint distribution induced by the imputation model, the sampling design and the nonresponse mechanism.

We start with the regularity conditions needed for the L 2 -consistency of the complete data estimator (5.1).

(H20) We assume that the sampling design P v (•) is non-informative sampling and that a) The sampling fraction is such that lim

v→∞ n v N v = π * ∈ (0; 1).
b) There exists positive constants λ and λ * such that min

k ∈U v π k λ > 0, min k, ∈U v π k λ * > 0.
c) The sampling covariances are such that lim sup

v→∞ n v max k ∈U v |π k -π k π | < ∞.
Assumption (H20) is commonly used in the literature, see e.g., [START_REF] Robinson | Asymptotic properties of the generalized regression estimator in probability sampling[END_REF] and [START_REF] Breidt | Local polynomial regression estimators in survey sampling[END_REF]. It is known to hold for commonly used sampling designs. ii) There exists a positive constant C, independent of v, such that

E m(x) -m(x) 2 r, X, I C. a.s.
Then, the sequence of imputed estimators { t m } is L 2 -consistent with rate

E 1 N v t m -t y 2 = O γ v .
(5.5)

Proof. See Appendix 5.2.1.

Condition (i) in Result 5.2.1 requires the convergence of the L 2 -error of prediction towards zero whereas Condition (ii) requires that the conditional error is almost surely bounded, even for nite samples. Under some regularity assumptions, Condition (i) is satis ed for a large number of statistical procedures including linear regression and nonparametric regression procedures such as k-nearest neighbors and kernel regression under appropriate regularity conditions, see [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF]. Assuming a framework in which condition (ii) is satis ed, this result suggests that, in order to build a consistent imputed estimator, it is enough to use a consistent predictor to produce the imputed values. In that respect, imputation is not more di cult than regression. In a way, this is closely related to Theorem 2.2 of [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF], which states that it is enough to have a consistent regression estimate to obtain a consistent classi cation rule.

Remark 5.2.1. Result 5.2.1 can be used for establishing the L 2 -consistency of imputed estimators based on a wide class of parametric and nonparametric procedures, without having to impose any assumption on the superpopulation model. However, it may lead to suboptimal consistency rates without additional assumptions. Indeed, if no assumption about the joint distribution of (X, y) or about the smoothness of the regression function m(•) is made, an approach referred to as fully nonparametric, then there does not exist any guaranteed convergence rate in Result 5.2.1, no matter which imputation procedure is used (see Theorem 3.1 of [START_REF] Györ | A distribution-free theory of nonparametric regression[END_REF]). It follows that if one aims to obtain the convergence rate of t m based on Result 5.2.1, one has to consider a restricted class for the distributions of (X, Y ). By doing so, the rate of convergence of t m will be restricted by the minimax rate of convergence over the selected class, see e.g., [START_REF] Györ | A distribution-free theory of nonparametric regression[END_REF].

Tree imputation

In this section we begin by describing regression trees and partitioning predictors. We de ne an imputed estimator based on a regression tree. Its nite sample properties are discussed through the analysis of the underlying weighting system.

Trees and partitioning predictors

Partitioning predictors are algorithms that rst create a partition P = { A 1 , A 2 , ..., A T } of the predictor space based on D n r , and use it to make their predictions. The elements of P are called the (terminal) nodes. For a point x, de ne m tr ee (x, P) := (5.6) where A(x) denotes the node of P containing x and

k ∈S r 1 x k ∈ A(x) ∈S r 1 x ∈ A(x) • y k = k ∈S r W k (x, P)y k ,
W k (x, P) := 1 x k ∈ A(x) ∈S r 1 x ∈ A(x)
, k, ∈ U, (5.7)

denotes the prediction weights of m tr ee . In other words, the prediction m tr ee (x) of m tr ee is obtained by averaging the observations for the points that fall into the same node as the point x. We use the convention that m tr ee (x, P) = 0 if x is in an empty node. Since m tr ee can be written as a weighted sum of the observations, the properties of the tree imputed estimators are closely related to the properties of the prediction weights { W k (x , P)} k, ∈S r . For simplicity, in the sequel, we write W k (x) for the weights de ned in (5.7). Properties of the weights { W k (x , P)} k, ∈S r are given in Technical lemma 1.

Given the sample points, changing the partition may lead to di erent predictions. As such, a partitioning predictor is fully determined by both a set of points D n r and a partition P. Most often, the partition P is obtained as the output of a data partitioning algorithm; that is; an algorithm which takes the sample points as input, and outputs a partition of the regressor space. When the partitioning algorithm creates a partition by splitting recursively the regressor space, the algorithms are often called trees. Adopting the terminology of [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF], when the partitioning algorithm does not make use of the survey variable Y , we say that the partitioning rule, and, by extension, the partitioning predictor, has the X-property.

Example 5.3.1. CART algorithm [START_REF] Breiman | Classi cation and regression trees[END_REF]. In the CART algorithm, splits are created by a greedy algorithm that splits recursively the regressor space. More speci cally, let A denote a node containing #(A) respondents, considered for the next split, and C A the set of possible splits in the node A, which corresponds to the set of all possible pairs ( j, z) = (variable, position). Let mse

(A) = 1 #(A) k ∈S r 1 x k ∈ A (y k -ȳA ) 2
and ȳA be the average of the y-values of units belonging to A. This splitting process is performed by searching for the best split ( j * , z * ), i.e. the split for which the following criterion is maximized:

L( j, z) = mse(A) -mse (A L ) -mse (A R )
where A L = k ∈ A; x k j < z , A R = k ∈ A; x k j z . This criterion therefore searches for the split which would generate child nodes as homogeneous as possible, in terms of mean square error. Splits are always performed in the middle of two points. The procedure continues until a stopping criterion is reached. Usual stopping criteria consist of specifying a minimum number of elements (n 0 ) in the terminal nodes, or a maximum depth (K) for the tree.

For more details about trees and partitioning procedures, the reader is referred to [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF] or [START_REF] Györ | A distribution-free theory of nonparametric regression[END_REF].

Regression tree imputation

Let m tr ee (•, D n r ) := m tr ee (•) be a tree predictor. We de ne the regression tree imputed estimator t tr ee of t y as

t tr ee := k ∈S r y k π k + k ∈S m m tr ee (x k ) π k .
(5.8)

The properties of t tr ee are closely related to the behavior of its underlying weighting system.

Proposition 5.3.1. The tree estimator t tr ee de ned by (5.8) can be written as

t tr ee = k ∈S r w k y k ,
where the weights {w k } k ∈S r are given by

w k = 1 π k + ∈S m W k (x ) π = 1 π k + N (x k , S m ) N (x k , S r ) , k ∈ S r , (5.9) with N (x k , S m ) := ∈S m π -1 1 x ∈ A(x k ) denoting the estimated number of nonrespondents in A(x k ); similarly, N (x k , S r ) := ∈S r 1 x ∈ A(x k
) is used to denote the number of respondents that fall in A(x k ).

To get a better understanding of the weighting system {w k } k ∈S r , consider the special case of π k = π, for all k ∈ U. In that particular case, the estimation weights w k reduce to

w k = π -1 × 1 + N (x k , S m ) N (x k , S r ) = π -1 × 1 + R mr (x k ) = π -1 • g k ,
where g k := 1 + R mr (x k ) and R mr (x k ) denotes the ratio between the number of nonrespondents and the number of respondents in the node A (x k ). We see that a set "g-weights" is induced by the tree estimator in the case of equal inclusion probabilities (and only in that case). The weight w k will be large when the ratio of nonrespondents over respondents for elements similar to element k is large. If the neighbors of element k are mostly nonrespondents, there are only few such elements in the sample of respondents, so that this "category" would, in some sense, be under-represented. In that case, the ratio R mr (x k ) is large, so that the correction factor 1 + R mr (x k ) is signi cantly greater than 1.

Remark 5.3.1. Although Proposition 5.3.1 states that t tr ee can be written as a weighted sum of the sequence {y k } k ∈S r , it does not imply that t tr ee is a linear estimator of t y . Indeed, if the partition P induced by m tr ee does not have the X-property, then the estimation weights {w k } k ∈S r also have a Y -dependency, which implies that t tr ee is a nonlinear function of {y k } k ∈S r .

Proposition 5.3.2. The weights {w k } k ∈S r in (5.31) have the following properties.

i) The weights are calibrated to the population size N whenever the original weighting system {d k } k ∈U is:

k ∈S r w k = k ∈S d k := N.
ii) If there are at least n 0 elements in each node, the weights {w k } k ∈S r are bounded,

d k w k d k 1 + n m n 0 , a.s. k ∈ S r .
(5.10)

The bounds are sharp, i.e., each of the bounds can be attained.

The lower bound in (5.10) given by w k = d k is obtained where there is no missing elements in the node containing element k. The upper bound is obtained when all missing elements are in the node of element k, and if this node contains precisely the minimal number of elements possible. Two elements can also be observed from Proposition 5.3.2: 1) more nonrespondents leads to a more conservative inequality; 2) more elements in the nodes implies less volatile weights: the larger n 0 is chosen, the lower is the diameter of the support of {w k } k ∈S r , leading to less volatile weights.

Proposition 5.3.3. If the sampling design is such that π k = π for all k ∈ U, then t tr ee can be written in projection form, that is,

t tr ee = k ∈S m tr ee (x k ) π k .
Remark 5.3.2. When the sampling design induces unequal rst order inclusion probabilities, we can nd cases where Proposition 5.3.3 does not hold. This is due to the fact that the predictions of the tree predictor m tr ee are not weighted by the inclusion probabilities. If it was, Proposition 5.3.3 would hold without the equal inclusion probabilities assumption.

Properties of the tree imputed estimator

As for most imputed estimators, information about the distribution of t tr ee are di cult to obtain. In general, its bias and variance are unknown. In some cases, however, it is possible to obtain information about these quantities. First, in the particular case where the survey variable is constant, it is possible to fully characterize the distribution of t tr ee , as shown in Example 5.3.2.

Example 5.3.2. Assume that ξ :

y k = C, k ∈ U,
and that k ∈S d k = N. Then, we have t tr ee = t y , with probability one. In this particular case, t y is always perfectly estimated by t tr ee . Note that, in this case, we have t π = t y , with probability one as well.

Obviously, this scenario is not realistic as it is too simple to represent practical situations; consider the more practical example in which the survey variable is not constant, but the regression function is. In that case, the rst two moments of t tr ee can be obtained, see Example 5.3.3 below.

Example 5.3.3. Assume that ξ :

y k = C + k , k ∈ U, with { k } k ∈U is a sequence of i.i.d. random variables such that E [ k |x k ] = 0 and E 2 k |x k = σ 2 .
Assume also that k ∈S d k = N and that the tree predictor has the X-property. Then, it can be shown that t tr ee remains unbiased but its variance is now strictly positive.

In a general setup, Result 5.3.1 shows that the tree imputed estimator based on the CART criterion is L 2 consistent for t y , under some regularity conditions.

Result 5.3.1. Assume (H19) and (H20). Consider a sequence of tree imputed estimators { t tr ee } based on the CART criterion described in Example 5.3.1. Assume that:

1. No additional split is performed in a node if it contains one element or if the maximal depth K v is reached.

2. The regression function m is additive and bounded, i.e.

m v ∈ G v := {g(x) = p v j=1
g j (x j ), g j is bounded and Borel measurable, j = 1, 2, ...p v },

and ||m v || l 0 = #{ j = 1, 2, ..., p v ; m j non-constant} = o( √ K v ).
Then, if lim v→∞ K v = +∞ and lim v→∞ 2 K v log (n r p v ) /n r = 0, the tree estimator { t tr ee } is mean-square consistent for t y , i.e.

lim v→∞ E 1 N v t tr ee -t y 2 = 0.
The conditions given in Result 5.3.1 follow from the conditions of results from [START_REF] Klusowski | Universal consistency of decision trees in high dimensions[END_REF]. The conditions on the tree predictor states that the depth of the trees should increase as the sample and population sizes increase, but not too fast with respect to the number of respondents. The assumption that the regression function is additive in its covariates is technical only.

Remark 5.3.3. Corollary 4.3 of [START_REF] Klusowski | Universal consistency of decision trees in high dimensions[END_REF] holds in a high-dimensional framework as well, in which the number of covariates is allowed to increase to in nity, with "noise" variables. Interestingly, Result 5.2.1 carries over every property regarding the conditions and the convergence rate of the sequence of predictors, including high-dimensional convergence. As such, Result 5.3.1 also holds if p v diverges. As such, Result 5.3.1 also proves the existence of L 2 -consistent imputed estimators in a high-dimensional framework.

From trees to forest estimators

Randomized predictors and random forests

A random forest predictor is an ensemble method based on a large collection of regression trees. Its predictions are de ned as the average of the predictions of each of the trees in the forest. By noting that the prediction rules described in Examples 1.2.1 and 1.2.2 are deterministic, it is clear that, for a xed set of elements, using the same partitioning rule to construct B trees would simply result in constructing the same tree B times. Breiman suggested [START_REF] Breiman | Bagging predictors[END_REF][START_REF] Breiman | Random forests[END_REF] to introduce an additional randomness in the partitioning algorithm and/or prediction rule. The additional randomness introduced in the predictors can be de ned using the concept of stochastic predictors. Let Θ be de ned in a measurable space (J, J ). A stochastic predictor m is a measurable function such that m : R p × J → R. In other words, the predictor m might use a random variable to make its predictions. It follows that the prediction method m is random with respect to Θ and, as such, an additional source of randomness is present.

Example 5.4.1. Let q ∈]0; 1[ and Θ be a random variable with Bernoulli distribution B(q); de ne m(x, Θ) := Θ||x|| 2 , where ||•|| 2 denotes the Euclidean norm. Then, m is a stochastic prediction model, meaning that, for two di erent realizations of Θ, the prediction m may generate di erent values. An additional random source is present, i.e. one can show that V Θ ( m(x, Θ)) = q (1q)||x|| 2 2 > 0.

Two additional examples of how the randomization procedure can be incorporated and used in regression trees are given below.

Example 5.4.2. Uniform random forest [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF], Scornet, 2016a). All the B trees of the forest have the same behavior; as such, we describe only the behavior of a generic tree among the B belonging in the forest. We begin by considering [0; 1] p as the initial leaf. Then, recursively, the algorithm splits as follows:

1. A node G is selected uniformly at random. 2. A splitting variable X j is selected uniformly at random among the p auxiliary variables X 1 , X 2 , ..., X p . .13) where m (B) r f (x k ) denotes the prediction of m (B) r f de ned in (5.11) at the point x k . We begin our analysis of t (B) r f by establishing the link between forest estimators and tree estimators, described in Proposition 5.4.1.

t (B) r f := k ∈S r y k π k + k ∈S m m (B) r f (x k ) π k , ( 5
Proposition 5.4.1. The forest imputed estimator t (B) r f de ned in (5.13) is an average of (randomized) tree imputed estimators:

t (B) r f = 1 B B b=1 t (b) tr ee ,
where t (b) tr ee is the imputed estimator based on the b-th tree of the forest m (b) tr ee , that is,

t (b) tr ee = k ∈S r y k π k + k ∈S m m (b) tr ee (x k ) π k .
Thus a forest estimator is an average of randomized tree estimators. Many of the properties of tree estimators are also shared by randomized tree estimators, and thus with forest estimators as well.

In terms of nite sample properties, we need to distinguish deterministic tree estimators from randomized tree estimators when there is a resampling mechanism involved. If there is not resampling mechanism, then the nite sample properties of both types of estimators are similar. Indeed, if ψ k = 1 for all k ∈ S r , then every property stated in Section 5.3.1 hold. If not, however, some properties no longer hold. This is due to the fact that the weights of a randomized tree

W k (x ) = ψ k 1 x k ∈ A(x ,Θ) i ∈S r ψ i 1 x i ∈ A(x ,Θ) , k, ∈ S,
are not always symmetrical in k, . Speci cally, property iv) of Technical lemma 1 no longer holds anymore for such trees (in fact, symmetry holds only for elements k, ∈ ∩ B b=1 S r (Θ b )). To analyze the properties of forests, two main paths might be followed: rst, through Proposition 5.3.1, thus deducing properties of forests through the properties of randomized trees; second, through the fact that a forest predictor can be written in (almost) the same way as a tree predictor, with weights de ned in (5.12). As such, most proofs can be reproduced almost identically. For conciseness, we omit proofs that are obtained easily from similar arguments than those given in Section 5.3. Proposition 5.4.2. The forest t (B) r f estimator de ned in (5.13) can be written as

t (B) r f = k ∈S r w (B) k y k ,
where the weights {w (B) k } k ∈S r are given by

w (B) k = 1 π k + ∈S m W (B) k (x ) π = 1 π k + 1 B B b=1 ψ (b) k N b (x k , S m ) N b (x k , S r (Θ b ))
.

(5.14)

Similarly as for individual tree estimators, forest estimators belong to the class of linear estimators in Y if and only if each tree of the forest has the X-property; see Remark 5.3.1. The weights {w (B)) k } k ∈S r share the properties of the weights {w k } k ∈S r detailed in Proposition 5.3.2.

From nite to in nite forests

We now consider forests with a large number of trees. Such forests are more stable, hence easier to analyze. We begin our discussion on large forests by considering the notion of in nite forests predictors, de ned as m (∞) := E m (B) r f |X, I, r, y .

We emphasize that, in practice, m (∞) cannot be computed explicitly (but can be approached, see below).

It is called an in nite forest predictor because, by the strong law of large numbers, we have

lim B→∞ m (B) r f = lim B→∞ 1 B B b=1 m (b) tr ee = m (∞) . a.s.
Accordingly, de ne the in nite forest estimator as

t (∞) r f := k ∈S r y k π k + k ∈S m m (∞) r f (x k ) π k = k ∈S r w (∞) k y k .
(5.15)

Using the fact that an imputed forest estimator is a average of tree imputed estimators and the strong law of large numbers, it follows that

lim B→∞ t (B) r f a.s. = E t (B) r f |X, I , r, y = t (∞) r f .
(5.16)

We see therefore that, even though the in nite forest estimator cannot be computed, there is hope to approach it with a nite forest estimator based on large B. In fact, approaching the in nite forest is of particular interest, as reveals next lemma.

Lemma 6. Consider sequences of nite { t (B) r f } and in nite { t (∞) r f } forest estimators.

There exists C such that

0 E       t (B) r f -t y N v 2       -E       t (∞) r f -t y N v 2       C B .
We also obtain that

√ n v N v t (B) r f -t y = √ n v N v t (∞) r f -t y + O P n v B .
By Lemma 6, we see that the mean squared error of in nite forest is always lower of equal to the mean squared error of nite forest. As a consequence, it follows that in nite forests are more e cient than nite forests. Lemma 6 also reveals that the di erence between the two errors is bounded, and even decreases to 0 if B diverges. The following proposition postulates that, with high probability, the nite forest estimator can be made arbitrarily close to the (unknown) in nite forests. Stability is also recovered.

Proposition 5.4.3. Fix B ∈ N and > 0. The probability (with respect to P Θ ) that the nite forest estimator is not in an -neighborhood of the in nite forest estimator is bounded by (5.17) where Ω Y denotes the sample space of the random variable Y .

P Θ | t (B) r f -t (∞) r f | > 2 exp -B 2 2n 2 m sup ω ∈Ω Y Y (ω) -inf ω ∈Ω Y Y (ω) min k ∈U π k 2 ,
Obviously, when B → ∞, the bound given in Proposition 5.4.3 decreases to 0, which is convergence in probability of t (B) r f towards t (∞) r f . This, naturally, is not surprising as both convergence in L 2 and almost sure hold, as stated for instance in (5.16). However, the bound (5.17) provides the quantitative guarantee that, with large probability, the two estimators are close. In particular, the bound (5.17) can be used to choose the number of trees to be used in practical situations. The bound also illustrates the impact of the number of nonrespondents, with regards to the number of trees. A similar concentration inequality can be obtained for the estimation weights as well: for > 0, we have

P Θ |w (B) k -w (∞) k | > exp -2B 2 d 2 k n 2 m n 2 0 , k ∈ S r .

Convergence of random forest imputed estimators

We conclude this section by two examples of L 2 -consistent forest estimators. We begin by considering the case of uniform random forests, as described in Example 5.4.2, followed by Breiman's random forests. The proofs will rely on mainly the same ideas, but will require more restrictive assumptions for Breiman's random forests due to a high level of data dependency. An important part of our proof is based on the idea that the forests that we consider are, in some sense, large and stable: we will assume that, without rate requirement, the number of trees is strictly increasing: let v 1 < v 2 be positive integers, then the number of trees B v 1 in the forest predictor used to impute in S v 1 is strictly lower than the number of trees

B v 2 used in imputation of S v 2 , i.e. v 1 < v 2 =⇒ B v 1 < B v 2 .
The exact motivation of this requirement will be made clear in the proofs below.

Result 5.4.1. Assume (H19) and (H20). Consider a sequence of uniform forest imputed estimators { t (B) ur f } described in Example 5.4.2. Assume also that:

1. The number of steps L v increases as v increases such that

lim v→∞ L v = +∞ and lim v→∞ 2 L v n v = 0.
2. The number of trees in the forest increases, without rate requirement, i.e. lim v→∞ B v = +∞.

Then, the forest estimator { t (B) ur f } is mean-square consistent for t y , i.e.

lim v→∞ E 1 N v t (B) ur f -t y 2 = 0.
The conditions given in Result 5.4.1 follow from the conditions of results from Scornet (2016a).

Result 5.4.2. Assume (H19) and (H20). Consider a sequence of Breiman's random forest imputed estimators { t (B) br f } described in Example 5.4.3. Assume also that:

1. No additional split is performed in a node if it contains one element or if the maximal depth K v is reached.

2. The regression function m is additive with each component bounded, i.e.

m v ∈ G v := {g(x) = p v j=1
g j (x j ), g j is bounded and Borel measurable, j = 1, 2, ...p v },

and ||m v || l 0 = o( √ K v ).
Then, if lim v→∞ K v (p v /p 0v ) = +∞ and lim v→∞ 2 K v log (n r p v ) /n r = 0, the forest estimator { t (B) br f } is mean-square consistent for t y , i.e.

lim v→∞ E 1 N v t (B) br f -t y 2 = 0.

Variance estimation

It is well known that treating imputed values as observed values and using the naive variance estimator

V naive := k ∈S ∈S ∆ k π k r k y k + (1 -r k ) m (B) r f (x k ) π k r y + (1 -r k ) m (B) r f (x ) π
(5.18) may lead to a severe underestimation of the overall variance V( t (B) r f ). In this section, we derive two variance estimators that take into account all variation sources through the methods described in [START_REF] Särndal | Methods for estimating the precision of survey estimates when imputation has been used[END_REF] and [START_REF] Shao | Variance estimation for survey data with composite imputation and nonnegligible sampling fractions[END_REF]; we shall call these approaches the two-phase and the reverse frameworks, respectively. For more details about variance estimation of imputed estimators in surveys, the reader is referred to [START_REF] Haziza | Variance estimation procedures in the presence of singly imputed survey data: a critical review[END_REF]. Proposition 5.5.1 illustrates how variance estimation of large forest estimators is similar to variance estimation of tree estimators. For simplicity of notation, we let E Θ and V Θ be the expectation and variance operators with respect to the random variables {Θ b } B b=1 , conditionally on the every other random quantities.

Proposition 5.5.1. Consider sequences of nite { t (B) r f } and in nite { t (∞) r f } forest estimators. We have

V t (B) r f -t y N = V t (∞) r f -t y N + E       V Θ t (B) r f N       . (5.19) Furthermore, there exists C > 0 such that E V Θ N -1 v t (B) r f C × n 2 v N 2 v B v . Corollary 5.5.1. The contribution of E V Θ t (∞) r f to the overall variance V t (B) r f -t y given in (5.19) is at most of order O n 2 v N 2 v × n v B v .
Proposition 5.5.1 and Corollary 5.5.1 highlight that the contribution of the randomization variance can be made arbitrarily small by choosing a large value of B. More precisely, the contribution of the randomization variance is at most of order O f 2 v • n v /B v , which is small if either (or both): 1) the sampling fraction f v is small enough; 2) the number of trees is large enough. The other variance component is the variance of the in nite forest, which is, in some regards, a particular kind of regression tree. Hence, for conciseness, we describe the variance estimation procedures only for t tr ee .

Variance estimation through the two-phase framework

Following [START_REF] Särndal | Methods for estimating the precision of survey estimates when imputation has been used[END_REF], we consider the decomposition t tr eet y = t tr eet π + t πt y .

It follows that the overall mean-squared error of t tr ee can be written

E t tr ee -t y 2 = E t π -t y 2 + E t tr ee -t π 2 + 2E t tr ee -t y t π -t y , = V sam + V nr + 2V mix .
We propose, as described in [START_REF] Särndal | Methods for estimating the precision of survey estimates when imputation has been used[END_REF], to estimate these three terms separately. The rst term is the sampling variance, the second corresponds the nonresponse variance and the third is a mixed component. Following the method of [START_REF] Beaumont | Variance estimation when donor imputation is used to ll in missing values[END_REF], an estimator of the sampling variance is given by

V sam := V naive + k ∈S m d 2 k (1 -π k ) σ 2 ,
where σ 2 is an estimator of σ 2 . Usual sample variance estimators might be used on the data { e k } k ∈S r := {y km tr ee (x k )} k ∈S r . If heteroscedasticity is suspected in these residuals, one might regress { e k } k ∈S r on the covariates {x k } k ∈S r with a regression tree (or forest) to predict { σ k } k ∈S r ; see e.g. [START_REF] Haziza | Variance estimation procedures in the presence of singly imputed survey data: a critical review[END_REF] for more details about the procedure. An estimator of the nonresponse variance is given by

V nr := σ 2 k ∈S γ 2 k ,
where γ k := r k w kd k for k ∈ S. An estimator of the mixed component is given by

V mix := k ∈S γ k (d k -1) σ 2 .
An estimator of the total variance is therefore given by

V sar := V sam + V nr + 2 V mix .
(5.20)

Remark 5.5.1. As noted by [START_REF] Beaumont | Variance estimation under composite imputation: The methodology behind sevani[END_REF] and in [START_REF] Haziza | Variance estimation procedures in the presence of singly imputed survey data: a critical review[END_REF], the estimation of the nonresponse and mixed components is simpli ed when the imputation model is linear in the survey variable. However, as mentioned in Remark 5.3.1, a tree predictor (and estimator) is linear in the survey variable if and only if the partitioning algorithms has the X-property. For trees with the X-property, the overall variance of t tr ee is therefore taken into account by V sar . However, when the partitioning algorithm does not have the X-property, a rigorous justi cation of the variance estimator given in (5.20) is beyond the scope of this article. The rationale behind it is based on the assumptions that N -2 v V t tr eet y N -2 v V t tr eet y |P for large samples and that E y 2 k |P = σ 2 + o P (1). In words, the overall variance of t tr ee is taken into account by (5.20) if the in uence of the variations produced by the partitions to the overall variations is asymptotically negligible. Since most splitting criteria can be shown to converge to their theoretical counterparts (see e.g., [START_REF] Scornet | Consistency of random forests[END_REF]), we expect these assumptions to hold. Simulations provided in Section 5.7 also seem to corroborate these assumptions.

Variance estimation through the reverse framework

In the reverse framework (Fay, 1991, Shao and[START_REF] Shao | Variance estimation for survey data with composite imputation and nonnegligible sampling fractions[END_REF], conditionally on the nonresponse mechanism, the variance of a regression tree imputed estimator can be decomposed as V t tr eet y |r = E V t tr ee |r, y, X |r + V E t tr eet y |r, y, X |r :

= V 1 + V 2 .
(5.21)

Note that, if t tr ee is asymptotically unbiased, it follows that V t tr eet y |r ≈ V t tr eet y for large samples. It is known that, for single-stage sampling designs, the contribution of V 2 to the overall variance is at most of order O(n v /N v ), see e.g., [START_REF] Shao | Variance estimation for survey data with composite imputation and nonnegligible sampling fractions[END_REF], [START_REF] Haziza | Variance estimation procedures in the presence of singly imputed survey data: a critical review[END_REF]. Hence, if the sampling fraction is negligible, its computation may be omitted. In what follows, we make this assumption.

Using a linearization of t tr ee , it follows that

V 1 ≈ E V k ∈S d k ξ k |r, X, y r , where ξ k := m tr ee (x k ) + r k • N (x k , U) N (x k , U r ) • (y k -m tr ee (x k )) , k ∈ S, and
N(x k , U) := k ∈U 1 x k ∈ A(x) , N(x k , U r ) := k ∈U r 1 x k ∈ A(x) , m tr ee (x k ) := k ∈U 1 x k ∈ A(x) ∈U 1 x ∈ A(x)
• y k , with A(x) denoting the node of the population partition containing x and U r is the population of respondents. The quantities {ξ k } k ∈S are therefore unknown; they can be estimated by

ξ k := m tr ee (x k ) + r k • N (x k , S) N (x k , S r ) • (y k -m tr ee (x k )) , k ∈ S. An estimator of V 1 is given by V r ev := k ∈S ∈S ∆ k π k ξ k π k ξ π . (5.22)
If n/N is negligible, V r ev is an estimator of the total variance.

Mass imputation for data integration

In recent years, there has been a shift of paradigm in NSOs that can be explained by three main factors: (i) a dramatic decrease of response rates; (ii) increasing data collection costs; and (iii) the proliferation of nonprobabilistic data sources such as web survey panels, social media and satellite information. To meet these new challenges, NSOs face increasing pressure to utilize these convenient but often uncontrolled data sources. While such data sources provide timely data for a large number of variables and population units, they often fail to represent the target population of interest because of inherent selection biases. The integration of data from a nonprobability source to data from a probability survey is a topic that is currently being scrutinized by NSOs. The reader is referred to [START_REF] Beaumont | Pitfalls of making inferences from non-probability samples: Can data integration through probability samples provide remedies? surv[END_REF] and [START_REF] Yang | Statistical data integration in survey sampling: A review[END_REF] for recent overviews on data integration methods in a survey sampling setting.

In this section, we show how our methodology and results can be applied to the problem of data integration. Consider a nite population U of size N. Two independent samples S A ⊂ U and S B ⊂ U are observed. On the one hand, the sample S A of size n A is selected from the sampling frame according to a probability sampling design P A (•) with rst-order inclusion probabilities {π (A) k } k ∈U known for all population units. On the other hand, the sample S B of size n B is a sample where the inclusion probabilities {π (B) k } k ∈U are unknown. The survey variable is assumed to be observed only for the elements of S B , whereas the vectors of covariates {x k } k ∈S A and {x k } k ∈S B are observed both samples. The framework is summarized in Table 15 Because the inclusion probabilities of the sample S B are unknown, S B cannot be used directly to produce reliable estimates of t y . Moreover, a similar conclusion holds for S A since the measurements of the survey variable are unobserved for those elements. In that framework, it is common to consider the methodology of mass imputation; that is, a model m (S B ) r f is tted on {(x k , y k )} k ∈S B to de ne the following estimator of t y :

t mi := k ∈S A m (S B ) r f (x k ) π (A) k .
(5.23)

A similar mass imputation estimator can also be constructed with any other imputation model, including regression trees. In fact, the mass imputation estimator t mi can be viewed as an imputed estimator with n m = n and an imputation model coming from an auxiliary source. Thus, the mass imputation estimator inherits most of the properties proved for trees and forests imputed estimators. More precisely, our previous regularity conditions turn into the following in this framework.

The regularity conditions on the sampling design P A (•) are similar to those made for the sampling design P(•) in the previous sections. Regarding the (unknown) sampling design P B (•), it is enough to assume that it is non-informative and that each element has a strictly positive probability of being in the sample S B .

Under the assumptions mentionned in the above paragraph, the previous results of consistency hold for both tree and forest mass imputed estimators. Assuming that the sample S B is of size much greater than S A , i.e. n A /n B ≈ 0, we suggest, using the reverse framework, the following variance estimator: A) .

V mi = k ∈S A ∈S A π (A) k -π (A) k π (A) π (A) k m (B) r f (x k ) π (A) k m (B) r f (x ) π (
(5.24)

Simulations

In this section, we present the results of several empirical studies to assess the behaviors of the methodologies introduced in this article. First, in Section 5.7.1, we study the empirical performances of trees and forest imputed estimators; we compare them with other state-of-the-art imputed estimators. In Section 5.7.2, we investigated the performances of the variance estimators suggested in Section 5.5. Section 5.7.3 provides the results of simulations investigating the performances of mass imputed estimators. Section 5.7.4 focuses on the performances of the variance estimator suggested for mass imputed estimators.

Performances of point estimators

We generated a population U of size N = 10 000 consisting of a set of covariates X 1 , X 2 , ..., X 5 and 5 survey variables. We begin by de ning a matrix Z ∈ R N ×p with entries giving by Z i j ∼ N (5, 1). Next, We generated a design matrix X = A + E ∈ R N ×p , with A = SVD 2 (Z) where SVD 2 denotes the rank-2 singular value decomposition operator and E with components such that:

• E k ∼ N (0, 0.01) if A k Q 0.25 , • E k ∼ N (0, 0.8) if Q 0.25 < A k Q 0.5 , • E k ∼ N (0, 1.6) if Q 0.5 < A k Q 0.75 , • E k ∼ N (0, 2.4) if Q 0.75 < A k ,
where Q α denotes the empirical quantile of order α of A 2 , the second column of A. The rationale behind this construction was to use a design matrix X whose colums represent the covariates, which was full rank, yet with correlations between the covariates and with an underlying structure of strata (given by the quantiles of the column A 2 ) with di erent variances in each stratum.

Using X 1 -X 5 , we generated 5 survey variables according to:

• Y 1 = 2 + X 1 + X 2 + X 3 + X 4 + N (0, 1); • Y 2 = 2 + X 2 1 + X 2 2 + X 3 3 + X 4 + N (0, 1); • Y 3 = 2 + cos (X 1 + X 2 ) + N (0, 1); • Y 4 = 2 + 2X 1 + 5X 2 + X 2 1 X 2 3 X 2 4 + N (0, 1
);

• Y 5 = 2 + X 1 + 10 exp 21 X 5 >5 -31 X 5 <6 + N (0, 1).

The goal was to estimate the totals t y j := k ∈U y k j for j = 1, ..., 5, where y k j denotes the measure of the survey variable Y j for element k ∈ U. To this aim, we considered two scenarios: 1) we assumed that the underlying strata structure of the design matrix was unknown to the survey statistician; 2) the strata structure was known to the survey statistician, and thus this information could be incorporated into the sampling design. In the rst scenario, we used simple random sampling without replacement of size n = 1 000. In the second, we used the known strata structure to de ne a strati ed sampling with X 2 -optimal allocation. We note that the strati ed sampling design was informative as correlations between the survey variables and the inclusion probabilities were between 0.3 and 0.5.

Nonresponse to Y 1 , Y 2 , ..., Y 5 was generated according to a MAR nonresponse mechanism, attributing response probabilities de ned as follows

p k = 0.1 + 0.8 × logit(5 -0.25(x k1 + x k2 + x k3 + x k4 )), k ∈ U,
and r k ∼ B(p k ). Missing values were imputed by using the following 5 di erent imputation procedures:

1) The imputed estimator (LR) based on linear regression.

2) The imputed estimator (CART) based on a regression tree.

3) The imputed estimator (RF) based on random forest with:

• B = 1000 trees in the forests,

• Bootstrap as resampling mechanism,

• At least n 0 = n 11/20 elements in each terminal node,

• The CART splitting criterion was optimized by selecting m tr y = √ p + design variables with probability one at each split.

4) The imputed estimator (NN1) based on nearest neighbor imputation.

5) The imputed estimator (NN5) based on 5-nearest neighbors imputation.

A Monte-Carlo procedure of R = 5 000 iterations was used to evaluate the performances of these estimators. As a measure of bias, we used the Monte-Carlo relative bias (RB) de ned as

RB( t imp ) = 100 × 1 R R r=1 ( t (r) -t y ) t y , (5.25)
for an estimator timp . The Monte-Carlo relative e ciency (RE) with respect to the Horvitz-Thompson estimator was also computed:

RE(t imp ) = 100 × R r=1 (t (r) -t y ) 2 R r=1 (t (r) yπ -t y ) 2 .
(5.26)

The results in terms of relative e ciency and relative bias are reported in

Empirical performances of tree and forest mass imputed estimators

We generated a nite population of size N = 10 000 with two sets of auxiliary variables and four survey variables. The rst set of auxiliary variables consisted of X 1 ∼ N (2, 1) and X 2 ∼ N (2, 1). The second set of auxiliary variables consisted of Z 1 ∼ N (0, 1), Z 2 ∼ Beta (3, 1), Z 3 ∼ 2 × Gamma (3, 2), Z 4 ∼ Bernoulli(0.7), and Z 5 ∼ Multinomial(0.4, 0.3, 0.3). Given these variables, we generated the survey variables Y 1 -Y 4 according to the following models:

• Y 1 = 2 + X 1 + X 2 + N (0, 1) ; • Y 2 = 1 + 2X 3 1 + N (0, 0.5) ; • Y 3 = 2 + Z 2 1 + Z 2 + Z 2 3 + 1, 51 {Z 5 =1} + N (0, 1) ; • Y 4 = 2 + (Z 1 + Z 2 + Z 3 ) 2 + N (0, 1) + Beta(3, 1).
To estimate the population totals for the survey variables Y 1 and Y 2 , we performed 5, 000 iterations of the following process: we selected a probability sample, S A , of size n A = 500, according to simple random sampling without replacement. Independently, we selected a nonprobability sample S B , of size n B = 500, as follows: we partitioned the population into two strata : Stratum 1 consisted of the units k with x k1 < 2 and Stratum 2 contained the remaining units. In Stratum 1, we selected n 1 = 0, 7 × n B units according to simple random sampling without replacement. In Stratum 2, we selected n 2 = 0, 3 × n B units according, again, to simple random sampling without replacement. For the survey variables Y 3 and Y 4 , we used a similar procedure with a slight di erence: the strati cation was performed using the variable Z 1 instead of X 1 .

We mass imputed the missing elements {y k } k ∈S A according to the same imputation models as those described in Section 5.7.1, with the exception of the random forest algorithm for which we used B = 500 and n 0 = 10. For each imputation procedures, the imputations were obtained using the set of predictors described in Table 19.

Survey variable Vector of explanatory variable X

used in the working model

Y 1 X 1 -X 2 Y 2 X 1 -X 2 Y 3 Z 1 -Z 5 Y 4 Z 1 -Z 5
Table 19: Working models used.

We were interested in estimating the population total t y j = k ∈U y k j , j = 1, • • • , 4. For each imputation procedure (i)-(v), we computed the corresponding mass imputed estimator given by (5.23). In addition, we computed the naive estimator,

t naive = n -1 B k ∈S B y k (NAIVE).
As a measure of e ciency, we computed the relative bias given by (5.25) and the relative e ciency given by (5.26), using the unfeasible Horvitz-Thompson estimator, t π = k ∈S A y k /π k ( A) , as the reference.

The results are displayed in Table 20. As expected, the naive estimator is considerably biased in all the scenarios with value of absolute RB ranging from 15.3% to 60.9%. This can be explained that the participation mechanism depended on the variable X 1 (for Y 1 and Y 2 ) and the variable Z 1 (for Y 3 and Y 4 ) but neither of X 1 nor Z 1 was used in the estimation procedure. In the case of Y 1 , all the procedures led to negligible biases and LR was the most e cient. We note that CART was signi cantly less e cient than the other procedures with a value of RE equal to 269%. For the estimation of the total of Y 2 , all the procedures exhibited some moderate bias with values of absolute RB ranging from 3.4% (for NN) to 7.0%.

In terms of e ciency the best procedure as NN followed by RF. The other procedures were signi cantly less e cient. In the case of the survey variables Y 3 and Y 4 , the best procedure in terms of both RB and RE was RF. The imputation procedure was considerably ine cient. In the case of Y 4 , both NN and 5NN showed a signi cant bias.

Empirical performances of variance estimators for mass imputed estimators

In this section, we investigate the performance of V rf given by (5.24) in terms of bias and coverage of normal con dence intervals. We generated a population of size N = 50 000 consisting of a survey variable Y = 0, 3 + 2X + N (0, 0.4), where X ∼ N (0, 1). From the population, 5, 000 probability samples and nonprobability samples were selected using the setup in Section 7.2 for the survey variables Y 1 and Y 2 . We used n A = 500; 2000 and n B = 500; 2000; 10000.

In each sample, we computed (i) the imputed estimator (5.23) based on the random forest algorithm of Breiman; (ii) the variance estimator given by (5.24); and (iii) a 95% con dence interval of the form t mi ± 1.96 V r f . Table 21 reports the Monte Carlo percent relative bias of V r f and the Monte Carlo coverage probability of the con dence intervals. For n A = 500, the variance estimator showed a small bias. The bias decreased as the ratio n A /n B decreased. For n A /n b = 500/10000 = 0.05, the variance estimator was virtually unbiased. The coverage rates ranged from 94.1% to 94.7%. For n A = 2000, the variance estimator

Appendix

Result 5.2.1. Assume (H19) and (H20). Consider a sequence of predictors { m} tted on D n r and its population counterparts { m} tted on D

N := {(x k , y k ) ; k ∈ U} . If i)
The sequence of population predictors { m} satis es

lim v→∞ E m(x) -m(x) 2 = 0,
with a convergence rate denoted γ v .

ii) There exists a positive constant C, independent of v, such that

E m(x) -m(x) 2 r, X, I C. a.s.
Then, the sequence of imputed estimators { t m } is L 2 -consistent with rate (5.27) where t π denotes the HT estimator on complete data de ned in (5.1). We turn into the second term of the right-hand side of (5.27). Write

E 1 N v t m -t y 2 = O γ v . Proof. Write E 1 N v t m -t y 2 2E 1 N v t m -t π 2 + 2E 1 N 2 v t π -t y 2 ,
E 1 N 2 v t π -t y 2 = E 1 N 2 v k ∈U α 2 k y 2 k + E 1 N 2 v k, ∈U k α k α y k y 5.8
where α k := I k π -1 k -1. Under (H20) each of these term converge to zero with the rate O(n -1 v ). It remains to show that the rst term of (5.27) is converging to 0 with the rate O(γ v ).

Recall that

1 N v t m -t π = 1 N v k ∈S (1 -r k ) π k ( m(x k ) -y k ) , hence E 1 N v t m -t π 2 2E 1 N v k ∈S (1 -r k ) π k ( m(x k ) -m(x k )) 2
(5.28)

+ 2E 1 N v k ∈S (1 -r k ) π k (m(x k ) -y k ) 2
We now establish the consistency of the second term of (5.28) with the rate

O(n -1 v ). Write E 1 N v k ∈S (1 -r k ) (m(x k ) -y k ) 2 = E        1 N 2 v k, ∈S k (1 -r k ) π k (1 -r ) π l × k       
(5.29)

+ E 1 N 2 v k ∈S (1 -r k ) π k 2 2 k
For the rst term of (5.29), we use the law of total expectation as follows:

E        1 N 2 v k, ∈S l k (1 -r k ) π k (1 -r l ) π l × k        = E        1 N 2 v k, ∈S k (1 -r k ) π k (1 -r ) π E k X, I, r      
 Notice now that the random variables k and are independent for all k . Furthermore, recall that, for all k ∈ U, E [ k |X] = 0. Thus, it follows that E k X, I, r = E k X, I, r E X, I, r = 0.

(5.30)

Therefore, the rst term in (5.29) is 0. For the second term, a similar derivation does not work. However, we have that

E 1 N 2 v k ∈S (1 -r k ) π k 2 2 k N v λ 2 N 2 v max k ∈U E (m(x k ) -y k ) 2 = O(N -1 v ) since, for all k ∈ U, E 2 k X, I , r = σ 2 < ∞.
It remains to show that the rst term of (5.28) is O(γ v ). Bounding arguments ensures that

E 1 N v k ∈S (1 -r k ) π k ( m(x k ) -m(x k )) 2 n v λ 2 N v 1 N v k ∈U E m(x k ) -m(x k ) 2 .
Now, Condition ii) implies that there exists a positive constant C > 0, independent of v, such that

E m(x k ) -m(x k ) 2 r, X, I C, a.s.
and it follows from Condition i) and Lemma 7 that, for all k ∈ U,

E m(x k ) -m(x k ) 2 r, X, I P -→ 0.
Hence, by the Lebesgues dominated convergence theorem,

lim v→∞ E E m(x k ) -m(x k ) 2 r, X, I = 0, with the rate O(γ v ). Moreover, max(γ v , 1/n v ) = γ v .
The result follows.

Proposition 5.3.1. The tree estimator t tr ee de ned in (5.8) can be written as

t tr ee = k ∈S r w k y k ,
where the estimation weights {w k } k ∈S r are given by

w k = 1 π k + ∈S m W k (x ) π = 1 π k + N (x k , S m ) N (x k , S r ) , k ∈ S r , (5.31) with N (x k , S m ) := ∈S m π -1 1 x ∈ A(x k )
denoting the Horvitz-Thompson estimator of the number of elements of A(x k ) with elements of S m ; accordingly, N (x k , S r ) :=

∈S r 1 x ∈ A(x k ) is used to denote the cardinal of elements in S r that fall in A(x k ). Proof. Write t tr ee = k ∈S r y k π k + k ∈S m 1 π k ∈S r W (x k )y = k ∈S r y k π k + ∈S r k ∈S m 1 π k W (x k ) y = k ∈S r 1 π k + ∈S m W k (x ) π y k .
To prove the last equality, see that

∈S m W k (x ) π (1) = ∈S m π -1 1 x ∈ A(x k ) i ∈S r 1 x ∈ A(x k ) := N (x k , S m ) N (x k , S r ) ,
where equality (1) follows from the symmetry property given in Technical lemma 1 item iv).

Proposition 5.3.2. The estimation weights {w k } k ∈S r in (5.31) have the following properties.

i) The weights are calibrated to the population size N whenever the original weighting system {d k } k ∈U is:

k ∈S r w k = k ∈S d k := N.
ii) If there are at least n 0 elements in each node, the weights {w k } k ∈S r are bounded,

d k w k d k 1 + n m n 0 , a.s. k ∈ S r .
The bounds are sharp, i.e. each of the bounds can be attained.

Proof. For i), recall that, for all x ∈ R p , we have k ∈S r W k (x) = 1, as stated in Technical lemma 1 item i). Thus,

k ∈S r w k = k ∈S r 1 π k + ∈S m k ∈S r W k (x ) π = k ∈S r 1 π k + ∈S m 1 π = k ∈S 1 π k .
The bounds follow as a consequence of Technical lemma 1 item iii).

Proposition 5.3.3. If the sampling design is such that π k = π for all k ∈ U, then t tr ee can be written in projection form, that is,

t tr ee = k ∈S m tr ee (x k ) π k .
Proof. Note that

t tr ee = k ∈S m tr ee (x k ) π k + k ∈S r y k -m tr ee (x k ) π k .
It is therefore enough to show that

k ∈S r y k -m tr ee (x k ) π k = 0.
Hence, write

k ∈S r y k -m tr ee (x k ) π k = k ∈S r π -1 k y k - ∈S r W (x k )y = k ∈S r π -1 k y k - k ∈S r π -1 k ∈S r W (x k )y (2) = k ∈S r π -1 k y k - ∈S r π -1 k ∈S r W k (x ) y 5.8 190 = k ∈S r π -1 k y k - ∈S r π -1 y = 0.
Equality (2) follows from the fact that π k = π , for all k, ∈ S r and the symmetry property given in Technical lemma 1 item iv).

Result 5.3.1. Assume (H20) and (H19). Consider a sequence of tree imputed estimators { t tr ee } based on the CART criterion described in Example 5.3.1. Assume also that:

1. There is no more split in a node if there is either only one element in it or if the maximal depth K v is reached.

2. The regression function m is additive and bounded, i.e. Then, if lim v→∞ K v = +∞ and lim v→∞ 2 K v log (n r p v ) /n r = 0, the tree estimator { t tr ee } is mean-square consistent for t y , i.e. Therefore, Condition ii) of Result 5.2.1 holds as well. Hence, Result 5.2.1 guarantees the mean-square consistency of { t tr ee } is proved.

Lemma 6. Consider sequences of nite { t (B) r f } and in nite { t (∞) r f } forest estimators.

There exists C such that

0 E       t (B) r f -t y N v 2       -E       t (∞) r f -t y N v 2       C B .
We also obtain that

√ n v N v t (B) r f -t y = √ n v N v t (∞) r f -t y + O P n v B .
Proof. The proof essentially follows ideas described in Scornet (2016a). Write

t (B) r f -t y 2 = t (B) r f -t (∞) r f + t (∞) r f -t y 2 = t (B) r f -t (∞) r f 2 + t (∞) r f -t y 2 + 2 t (B) r f -t (∞) r f t (∞)
r ft y .

(5.32)

Next, notice that

E t (B) r f -t (∞) r f t (∞) r f -t y = E E t (B) r f -t (∞) r f
r, X, I, y t (∞) r ft y = 0.

Therefore, taking expectations on both sides of (5.32) gives

E t (B) r f -t y 2 = E t (B) r f -t (∞) r f 2 + E t (∞) r f -t y 2 ,
(5.33) so that

E t (B) r f -t y 2 -E t (∞) r f -t y 2 = E t (B) r f -t (∞)
r f 2 0.

(5.34)

Next, write

t (B) r f N v - t (∞) r f N v = 1 N v k ∈S m m (B) r f (x k ) -m (∞) r f (x k ) π k , so that E       t (B) r f N v - t (∞) r f N v 2       = 1 N 2 v • E       k ∈S m m (B) r f (x k ) -m (∞) r f (x k ) π k 2       n v N 2 v • E        k ∈S m m (B) r f (x k ) -m (∞) r f (x k ) 2 π 2 k        n v N v N 2 v λ 2 • max k ∈U E m (B) r f (x k ) -m (∞) r f (x k ) 2
Now, using Theorem 3.3 of Scornet (2016a), there exists a positive constant C such that, for all k ∈ U, We can note that this lemma is somewhat similar to that proven in [START_REF] Doob | Stochastic processes[END_REF] for the case of almost sure convergence.

E m (B) r f (x k ) -m (∞) r f (x k ) 2 C B , leading to E       t (B) r f N v - t (∞) r f N v 2       Cn v N v N 2 v λ 2 B v = O 1 B v .
Proposition 5.4.3. Fix B ∈ N and > 0. The probability (with respect to P Θ ) that the nite forest estimator is not in an -neighborhood of the in nite forest estimator is bounded by

P Θ | t (B) r f -t (∞) r f | > 2 exp -B 2 2n 2 m sup ω ∈Ω Y Y (ω) -inf ω ∈Ω Y Y (ω) min k ∈U π k 2 ,
where Ω Y denotes the sample space of the random variable Y .

Proof. Observe that

t (B) r f -t (∞) r f = k ∈S m m (B) r f (x k ) -m (∞) r f (x k ) π k , so that P Θ | t (B) r f -t (∞) r f | > = P Θ 1 B B b=1 k ∈S m m (b) tr ee (x k ) -m (∞) r f (x k ) π k > . De ne d (b) := k ∈S m π -1 k m (b)
tr ee (x k )m (∞) r f (x k ) . Note that, given the covariates, the sample membership indicators, the survey variable and the nonresponse indicators, the sequence { m (b) tr ee } B b=1 is a sequence of independently and identically distributed (according to P Θ ) random variables. The same holds therefore for the sequence { d Thus, for > 0,

P Θ | t (B) r f -t (∞) r f | > = P Θ B b=1 d (b) > B (3) 2 exp -2B 2 4n 2 m sup ω ∈Ω Y Y (ω) -inf ω ∈Ω Y Y (ω) min k ∈U π k 2 ,
where (3) follows from Hoe ding inequality for bounded random variables.

Result 5.4.1. Assume (H20) and (H19). Consider a sequence of uniform forest imputed estimators { t (B) ur f } described in Example 5.4.2. Assume also that:

1. The number of steps L v increases as v increases such that lim v→∞ L v = +∞ and lim v→∞ 2

L v n v = 0.
2. The number of trees in the forest increases, without rate requirement, i.e. lim v→∞ B v = +∞.

Then, the forest estimator { t (B) ur f } is mean-square consistent for t y , i.e.

lim v→∞ E 1 N v t (B) ur f -t y 2 = 0.
Proof. Similar arguments than those used in the proof of Result 5.3.1 in coordination with Corollary 1 of Scornet (2016a) leads to the consistency of the in nite forest estimator t (∞) ur f , that is,

lim v→∞ E       t (∞) ur f -t y N v 2       = 0.
Moreover, from Lemma 6, we have

0 E       t (B) ur f -t y N v 2       -E       t (∞) ur f -t y N v 2       C B .
Thus, if we consider large forests (i.e. with an increasing number of trees), the sequences

E N -2 v t (B) ur f -t y 2 and E N -2 v t (∞)
ur ft y 2 must have the same limit. Hence,

lim v→∞ E       t (B) ur f -t y N v 2       = 0,
which concludes the proof.

Proposition 5.5.1. Consider sequences of nite { t (B) r f } and in nite { t (∞) r f } forest estimators. We have

V t (B) r f -t y N = V t (∞) r f -t y N + E       V Θ t (B) r f N       .
Furthermore, there exists C > 0 such that

E V Θ t (B) r f C × n 2 v N 2 v B v .
Proof. By the law of iterated variance,

V t (B) r f -t y = V E Θ t (B) r f -t y + E V Θ t (B) r f -t y .
From (5.16), it follows that

V t (B) r f -t y = V t (∞) r f -t y + E V Θ t (B) r f -t y .
Relation (5.19) is proved. Next, using Proposition 5.4.1, we have

V Θ t (B) r f -t y = V Θ t (B) r f = V Θ 1 B B b=1 t (b)
tr ee

(4) = 1 B • V Θ t (1) tr ee ,
where equality (4) follows from the fact that, as detailed in the proof of Proposition 5. This concludes the proof.

Technical lemma 1. Consider the weights of a regression tree as de ned in (5.7). The following hold: i) If there is at least one element per terminal node, then, for all x ∈ R p ,

k ∈S r W k (x) = 1.
ii) The weights of the tree can be seen as a the images of a weight function from R p × R p to [0; 1], that is,

W k (x ) := W (x k , x ) .
iii) If there is at least n 0 elements per terminal node, then the range of W reduces to [0; n -1 0 ].

iv) The weight function is symmetrical in its arguments, that is, for all x, y ∈ R p , W (x, y) = W (y, x) .

Proof. For i), x x ∈ R p . Using the de nition of (5.7), we have

k ∈S r W k (x) = k ∈S r 1 x k ∈ A(x) ∈S r 1 x ∈ A(x) = k ∈S r 1 x k ∈ A(x) ∈S r 1 x ∈ A(x)
= 1.

Point ii) follows directly from the de nition. To prove, iii), write

1 x k ∈ A(x) ∈S r 1 x ∈ A(x) 1 ∈S r 1 x ∈ A(x)
1 n 0v by noting that ∈S r 1 x ∈ A(x) n 0 . To see iv), let x, y ∈ R p . Observe that

W (x, y) =        1 k ∈S r 1 x k ∈ A(y) if x ∈ A(y), 0 otherwise.
Noting that the conditions x ∈ A(y) and y ∈ A(x) are the same, it is enough to split cases to prove the equality. Assuming that x ∈ A(y), it follows that A(y) = A(x), so that

W (x, y) = 1 k ∈S r 1 x k ∈ A(y) = 1 k ∈S r 1 x k ∈ A(x)
= W (y, x) .

In cases where, x A(y), then W (x, y) = 0 and W (x, y) = 0 so that the equality also holds.

In this PhD thesis, I investigated the use of statistical learning procedures for the estimation of population totals in presence of a large number of covariates. Two frameworks were considered: full response and item nonresponse.

In the rst, we assumed full response of the sampled elements and we studied the behavior of model-assisted estimators in presence of a large number of covariates. Convergence rates involving conditions of the ratio of the number of covariates over the sample size were established for linear and penalized linear model-assisted estimators. We also suggested the use of random forests to face high-dimensional data and analyzed their design-based properties.

In the second, we considered a more general framework in which the sampled elements might refuse to respond, thus leading to missing data. We performed a large empirical study aiming at examining which statistical learning predictors are particularly promising for imputation, in a wide variety of scenarios. Finally, we performed an in-depth analysis of regression tree and random forest imputed estimators.

In what follows, I discuss some thoughts on the use of machine learning predictors for survey statistics, I highlight some of the limitations of the work presented in this thesis and introduce a few ideas which might serve as future works.

Some thoughts on the use of machine learning algorithms in surveys

The increasing attention for the eld of statistical learning has permitted the emergence of numerous highly complex predictive models and algorithms, often called machine learning methods. As illustrated with various applications throughout this thesis, the use of such tools may be particularly useful in surveys for the estimation of nite population totals. Yet, an important question arises: is it always wise, pro table even, to incorporate highly complex predictive models, whose e ciency sometimes relies entirely on empirical clues, into survey strategies? It is often objected to statistical learning models that they behave like "black boxes", thus producing results that we do not really know how to explain or interpret. The articles [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF] and [START_REF] Dagdoug | Random forest imputation in surveys and application to data integration[END_REF] presented in this thesis are an attempt at shedding some light towards the intepretability of random forests in surveys. However, the intepretability of random in surveys remains partial only. More generally, additional research is required for a better intepretability of these machine learning methods in a survey framework. Moreover, recent empirical investigations (e.g., Dagdoug et al. (2021a), [START_REF] Larbi | Treatment of unit nonresponse in surveys through machine learning methods: an empirical comparison[END_REF]) showed that highly complex predictive algorithms such as Cubist [START_REF] Quinlan | Learning with continuous classes[END_REF], XGBoost [START_REF] Chen | XGBoost[END_REF], BART [START_REF] Chipman | BART: Bayesian additive regression trees[END_REF]) may be superior in many scenarios to traditional methodologies in surveys. However, these methods should be used with caution as there are still questions regarding the mechanisms that enable them to be that e ective; in particular, they may not always be highly e ective when applied in a survey sampling framework. For example, in the article Dagdoug et al. (2022a), simulation studies put into evidence that the model-assisted estimator based on the original random forest algorithm was biased and thus particularly ine cient in case of informative sampling designs and high-dimensional scenarios. This ine ciency was due to the fact that important design variables might not be selected by the random mechanism used in random forest algorithm used at each split. Forcing the algorithm to consider the design variables with probability one at each split solves the problem and the random forest model-assisted estimator implemented in this way recovers its usual e ciency.

Most often, these unsatisfactory behaviors arise from unexpected interactions between statistical learning methods and survey sampling. Detecting these undesirable phenomena is therefore of great interest and there is a need for both empirical and theoretical in-depth investigations of these methodologies in a survey statistics framework. This thesis has provided answers to several issues related to the use of machine learning techniques with survey data, but there are still many questions which require further investigations. This work has also revealed new phenomena, sometimes unexpected. I discuss and illustrate below several such phenomena as well as some personal thoughts about machine learning methodologies combined with survey statistics.

Estimation in surveys is more than a problem of prediction

Studying the asymptotic properties of nite population total estimators with respect to the sampling design, as we did in Chapter 2 and Chapter 3, can be particularly insightful. Indeed, the fact that some model-assisted estimators converge in L 1 in the design space without restrictions on the number of covariates reveals that, while predictive models can improve substantially the e ciency of point estimators in surveys, estimation of nite population parameters is not a problem of prediction.

A particularly striking example of that statement is given by the Horvitz-Thompson estimator. Indeed, observe that the Horvitz-Thompson estimator is a model-assisted estimator based on the constant function f = 0, that is,

t ma ( f ) := k ∈U f (x k ) + k ∈S y k -f (x k ) π k = k ∈U 0 + k ∈S y k -0 π k = k ∈S y k π k = t π .
Yet, under mild conditions, the Horvitz-Thompson is L 2 consistent for the joint distribution, independently of the true regression function. Of course, it is possible to nd examples of regression function for which the function f = 0 is a particularly poor estimate. This example illustrates that, even in a scenario in which the predictor f used in the model-assisted estimator is a mediocre estimate of the regression function, the estimator might still satisfy the usual square-root consistency.

The objectives of survey statistics and predictions are sometimes not aligned

In some cases, e cient predictions might lead to estimators less e cient than if they were based on "worse" predictions. An example of this situation is provided by the propensity score adjusted (PSA) estimator in presence of nonresponse. Assume the framework of Chapter 4 and Chapter 5, in which the observed data are {(x k ; y k ) ; k ∈ S r } ∪ {x k ; k ∈ S m } and the aim is to estimate t y is presence of nonresponse. One possibility, studied in Chapter 3 and Chapter 4, is to use an imputed estimator to counterbalance the negative e ects of nonresponse. Another possibility is to model the unknown response probabilities [START_REF] Haziza | Imputation and inference in the presence of missing data[END_REF] with propensity score adjusted estimator (PSA) de ned as

t psa ( r) := k ∈S r y k π k p k , 6.2 , 198
where p k := r(x k ) is an estimator of the unknown response probability p k of element k and r : R p → [0; 1] is a predictor for the response probabilities. The rationale behind the PSA estimator follows from the fact that, if the true response probabilities were known, one could use the unbiased estimator of t y t * psa := k ∈S r y k π k p k .

Since the response probabilities {p k } k ∈S r are unknown, the predictor r is used to estimate them. At rst, it would seem that, if r 1 is more e cient for modelling the response probabilities {p k } k ∈S r than r 2 (with respect to some criterion, e.g. mean squared error), then the estimator t psa ( r 1 ) should be more e ective than t psa ( r 2 ) to estimate t y , in terms of mean squared error. However, simulations performed in [START_REF] Larbi | Treatment of unit nonresponse in surveys through machine learning methods: an empirical comparison[END_REF] proved that this statement might not always be true. Indeed, the "best" predictor is the predictor which uses best the information explaining how the response indicators behave, given the covariates. The "best" estimator of t y , however, is the estimator based on the predictor of the response probabilities which best uses the information both related to the response indicators and the survey variable.

Imputation is not more di cult than regression

In the previous paragraphs, we illustrated through practical examples that the goal of survey sampling is the estimation of nite population parameters and not prediction. In some cases, these two goals are even pointing in di erent directions. In some other cases, however, e cient predictions lead to e cient estimation.

In Chapter 5, we established a result formalizing conditions about the predictor m used for imputation such that, whenever satis ed, the resulting imputed estimator t m would be L 2 -consistent for t y with respect to the joint distribution. More precisely, if the predictor m is L 2 -consistent for the regression function m when tted on i.i.d. data and if its L 2 risk is uniformly integrable, then the imputed estimator t m is L 2 -consistent for t y .

This result reveals that, in order to build a consistent imputed estimator, it is enough to use a predictor consistent for the regression function as imputation procedure. In that respect, imputation is not more di cult than regression. A parallel can be made with the problem of binary classi cation, in which one aims at predicting a label, say 0 or 1. In that case, it can be shown that if a predictor m is L 2 consistent for the regression function, then the decision function 1 m>1/2 is consistent in the sense that it minimizes the Bayes risk, see [START_REF] Devroye | A probabilistic theory of pattern recognition[END_REF] for de nitions and details.

We emphasize, however, that the conditions that we found on m for the consistency of t m towards t y are su cient, but probably not required for the consistency of the imputed estimators.

Open questions, extensions and future works

High-dimensional asymptotics for survey data Contrary to the case of linear models studied in high dimensions (see [START_REF] Ta | Generalized regression estimators with high-dimensional covariates[END_REF], [START_REF] Chauvet | Asymptotic e ciency of the calibration estimator in a high-dimensional data setting[END_REF], Dagdoug et al. (2022a)), no condition on the rate of divergence of the number of covariates is required for the design asymptotic properties of model-assisted estimators built upon tree-based methods. This phenomenon seems to be explained by two reasons: 1) as explained above, a model-assisted estimator does not need to be based on a consistent predictor (i.e. consistent for the regression function) to be consistent for t y ; 2) the "asymptotic structure" of a linear estimator (e.g. GREG) versus a tree-based estimator is di erent in essence. Indeed, a linear estimator can be seen as a calibrated estimator (see [START_REF] Deville | Calibration estimators in survey sampling[END_REF] for details) on the p covariates. Therefore, when 6.2 , 199 p = p v is allowed to increase to in nity, it is actually a matter of imposing an ever-increasing number of calibration constraints. This is not the case for estimators built on regression trees: these can also be seen as calibrated estimators, but on the T covariates formed by the indicators of the tree nodes, and not on the p covariates. Typically, T is a function of n 0 (and of n) rather than p, and therefore the number of calibration constraints imposed on an estimator built on a tree remains xed as p tends to in nity.

Several open questions remain in this research area. First, the conditions that we obtained in Chapter 2 for the high-dimensional consistency of model-assisted estimators may not be optimal, in the sense it might be possible to obtain weaker conditions. Whether or not these model-assisted estimators are consistent or unconsistent for t y when the conditions that we found are not satis ed remains an open question. A more re ned asymptotic analysis would be required to bring additional insight towards this question. Moreover, the asymptotic results of Chapter 2 were established for the consistency of point estimators. However, to our knowledge, the asymptotic properties of variance estimators in high-dimensional settings are yet to be determined. Furthermore, while the equivalence of asymptotic distribution holds for the random forest generalized estimator and the random forest model-assisted estimator in high-dimensional settings, a central limit theorem for the di erence estimator in a high-dimensional scenario is yet to be established and might require a substantial amount of additional research. Similar research questions for the case of imputation in presence of a large number of covariates remain open as well.

Variance estimation for model-assisted estimators

In the simulation studies presented in Chapter 3, we discovered that the choice of the minimal number of elements per terminal node of the random forest model-assisted estimator is crucial for the variance estimator

V (B) r f 1 = 1 N 2 v k ∈S v ∈S v π k -π k π π k y k -m (B) r f 1 (x k ) π k y -m (B) r f 1 (x ) π
to be e cient. The simulations performed have shown that, if n 0 is too small, then V (B) r f 1 might su er from an important negative bias. We discovered that the problem encountered is more general and may happen to any model-assisted estimator based on a exible predictor. To illustrate the issue, consider the naive predictor m naive de ned as:

m naive : R p -→ R, x -→ y k 1 {x k ; k ∈S } (x).
It is easily seen that the estimated variance V ma ( t naive ) of t naive is zero:

V ma ( t naive ) = k ∈S ∈S ∆ k π k y k -m naive (x k ) π k y -m naive (x ) π = k ∈S ∈S ∆ k π k y k -y k π k y -y π = 0.
Yet, there is obviously no reason for the true variance of t naive to be zero. The problem follows from the fact that an over tted predictor m will produce a set of underestimated residuals {y km(x k )} k ∈S , which are then used in the traditional variance estimator.

To cope with this issue, in [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF], we proposed to use a general K-fold cross-validated variance estimator, de ned by the following procedure, to estimate the variance of a model-assisted estimator based on a predictor m. We randomly split the sample S into K groups S κ , κ = 1, . . . , K, of 6.2 , 200 approximately equal size. For k ∈ S κ , let m (-κ) (x k ) denote the prediction at the point x k built on S -S κ and (-κ) k = y km (-κ) (x k ) the associated residual. The proposed K-fold variance estimator is given by

V (cv,K) ma := K κ 1 =1 K κ 2 =1 k ∈S κ 1 ∈S κ 2 ∆ k π k (-κ 1 ) k π k (-κ 2 ) π = k ∈S ∈S ∆ k π k (cv) k π k (cv)
π , (6.1)

where (cv) k is the uniquely de ned residual for element k ∈ S. The estimator V (cv,K) ma can be seen as a generalization of the estimator suggested in [START_REF] Opsomer | Selecting the amount of smoothing in nonparametric regression estimation for complex surveys[END_REF] for local polynomial regression. The estimator that we suggest also has connections with the jackknife variance estimators discussed in [START_REF] Duchesne | A note on jackknife variance estimation for the general regression estimator[END_REF] and [START_REF] Valliant | Variance estimation for the general regression estimator[END_REF].

Simulations suggest that, in case of random forests, the estimator V (cv,K) ma is almost unbiased, independently of the minimal number of elements per terminal node. However, the theoretical properties of the estimator (6.1) are yet to be determined, further research in this area would therefore be required. Moreover, since this variance estimator is very general and can be used for the variance estimation of any model-assisted estimator, it is of interest to study the theoretical properties of (6.1) not only for the particular case of random forest, but in a general setting; i.e. to nd conditions on the predictor m which are required for the good behavior of V (cv,K) ma . As suggested by Yves Tillé, another possibility could be to take into account, in the variance estimator a correction based on the degrees of freedom of the predictor used in a given model-assisted estimator. This approach has the advantage of being less computationally intensive than the cross-validated variance estimator as de ned in (6.1); however, this approach also presents the drawback of having to specify the degrees of freedom of a given predictor, which might a delicate task for complex machine learning predictors.

Estimator selection in surveys with full response

The use of complex algorithms such as random forest, boosting, Cubist, neural networks or BART forces statisticians to choose a certain number of hyper-parameters. The choice of some of these parameters may highly in uence the performances of the resulting estimators. More generally, as many estimators have been suggested in the literature (e.g., model-assisted, calibration, ...), the following question arises: among a list of candidates, which estimator should be chosen, given an observed sample? In many areas of statistics, procedures and methodologies have been developed to address this issue; for example, information criteria such as the Akaike Information Criterion (AIC, Akaike (1998)), the Bayesian Information Criterion (BIC, [START_REF] Schwarz | Estimating the dimension of a model[END_REF]) among others, for parametric models, cross-validation for statistical learning, the global Box and Jenkins methodology in time series [START_REF] Box | Time series analysis: forecasting and control[END_REF], to name a few. In surveys, however, there is, to our knowledge, no general methodology to be applied by practitioners. I believe that an important research area for the future in survey statistics is to elaborate a methodology that practitioners can use to guide their choice. Two possibilities come to mind for that purpose: 1) choose the best estimator t * among a list of possible candidates { t 1 , t 2 , ..., t J }; 2) build an aggregate estimator t agg , function of the J candidates. Both options present advantages and drawbacks. Estimator selection is particularly e cient if the estimator selected is itself e cient, but a wrong choice might lead to ine cient strategies. Aggregation has the advantage of being robust as it might use each of J candidates, but as such might su er from additional variations. In a context of imputation, Chen and Haziza (2017) suggested using multiply robust estimators which aggregates a list of predictors to produce a robust estimator, converging towards the parameter of interest if there is one predictor in the list which is correctly speci ed. Simulations of a similar procedure in a context of model-assisted estimation seem to produce e cient estimators as well, but additional research in this 6.2 , 201 area is required.

From an estimator selection point of view, the cross-validated variance estimator (6.1) can also be used in that setting. Initially, it was in that setting that [START_REF] Opsomer | Selecting the amount of smoothing in nonparametric regression estimation for complex surveys[END_REF] proposed a similar estimator. They suggested a weighted version of the usual variance estimator, which corresponds to the estimator in (6.1) with K = n -1; however, the weighted version that they suggested can be used only with linear predictors 1 , whereas the variance estimator in (6.1) can be used with any predictor. The authors proposed to minimize a particular case of the cross-validated variance estimator (6.1) in order to choose the bandwidth of the local polynomial predictor. This idea can be generalized to choose not only an optimal hyper-parameter for a particular model-assisted estimator, but the optimal estimator among a list of model-assisted estimators built on di erent predictors. Consider, as in the previous paragraph, a ( nite) list of candidates { t α 1 1 , t α 2 2 , ..., t α J J }, where α j ∈ R d j is a vector of d j parameters to be chosen for the candidate j. This representation is rather general and includes most model-assisted estimators. For each of the candidates, we suggest choosing the optimal set of parameters solving α * j := arg min

α∈R d j V (cv,K) ma t α 1 .
In practice, the problem should be discretized using a grid of possible values; in case of ties, random selection could be used. We follow this procedure by selecting the best estimator among the list of candidates de ned as the estimator t * satisfying

V (cv,K) ma t * = min V (cv,K) ma t α * j 1
, j = 1, 2, ..., J .

Simulations seem to suggest that the proposed method select the best estimator with high probability. However, rigorous proofs of that statement in a general setting would require substantial additional research, which we believe would be interesting as a future work.

Calibrated matrix completion for covariates imputation in survey sampling

As explained in the introduction, originally, the aim of this PhD thesis was to develop and investigate matrix completion algorithms for imputation in survey statistics. Eventually, other research areas were explored, although, along the way, we did investigate some of the existing algorithms and investigated their use in surveys. To be more precise, some additional notations are required.

Let X S denote the sample restriction of X U , and consider a set of survey variables y 1 , y 2 , ..., y q , concatenated in a population matrix Y U with its sample restriction Y S . We denote by S := [X S , Y S ] ∈ R n×d the sampled data, where d := p + q. In this framework, nonresponse is allowed for both the survey variables and the covariates, and we assume to have access to the population totals of X 1 , X 2 , ..., X p denoted by the vector t x . We denote by Ω S ⊂ {1, 2, ..., n} × {1, 2, ..., d} the set of indexes containing the elements of S which are not subject to nonresponse. That is, z i j = (i, j) ∈ Ω S if and only if S i j , the element in the i-th row and j-th column of S, is observed. Following [START_REF] Candès | The power of convex relaxation: Near-optimal matrix completion[END_REF], denote by P Ω S : R n×d → R n×d the orthogonal projection onto the subspace of rectangular matrices which vanishes outside of Ω S ; that is, for A ∈ R n×d , the matrix P Ω (A) has for coe cients

P Ω (A) i j =            A i j if (i, j) ∈ Ω, 0 otherwise.
(6.2)

1 A predictor m is said to be linear if there exists a set of weights {w k } k ∈S , independent of the survey variable, such that, for all x, m(x) = k ∈S w k y k .

, 202

P Ω|Y and P Ω |X are de ned similarly for the restriction of Ω to the rst q columns, and the next p columns, respectively. With the projection operator above, the information available at the sample level is contained in P Ω (S). Lastly, denote by || A|| F = n i=1 d j=1 A 2 i j the Frobenius norm of A and by || A|| * = min(n,d) j=1 σ j (A) its nuclear norm, with σ j (A) denoting the j-th largest singular value of A.

The original matrix completion problem can be stated as minimize Z ∈R n×d rank (Z) subject to P Ω S (Z) = P Ω S (S) .

(6.3) However, the rank minimization problem in (6.3) is not convex and is known to be NP-hard [START_REF] Candès | The power of convex relaxation: Near-optimal matrix completion[END_REF]; in particular no algorithm is known to solve such a problem in a reasonable time when, say, n 10. In [START_REF] Fazel | A rank minimization heuristic with application to minimum order system approximation[END_REF], the authors considered a convex relaxation of the rank minimization problem in (6.3) by using the nuclear norm, minimize where λ δ is a given constant, depending on δ only. [START_REF] Mazumder | Spectral regularization algorithms for learning large incomplete matrices[END_REF] suggested the soft-impute algorithm, a recursion based on iterative SVDs, to solve this problem. They have shown, among other things, that Soft-Impute is convergent for the solution of (6.6).

In a survey framework, the simulations that we performed using the Soft-Impute algorithm to recover the matrix S and estimate the totals of the survey variables Y 1 , Y 2 , ..., Y q seem to show that this approach leads to severely biased and ine cient total estimators. However, we found that applying the Soft-Impute algorithm to the matrix of covariates X S instead, and then using the imputed design matrix X S with traditional imputation procedures as described in Chapter 4 and Chapter 5 resulted in rather e cient estimators of t y . Yet, this approach does not make use of the totals t x assumed to be known. We thus suggest solving a calibrated matrix completion problem: This optimization problem recovers a low rank matrix, approximating X S at the observed entries and such that, the Horvitz-Thomspon estimators of t x based on X S are close to be calibrated. Using techniques of [START_REF] Cai | A singular value thresholding algorithm for matrix completion[END_REF], [START_REF] Mazumder | Spectral regularization algorithms for learning large incomplete matrices[END_REF] and an iterative algorithm, we could implement an algorithm which seems to converge to the solution of (6.7), but additional research is required to 6.2 , 203 prove the properties of this algorithm. Moreover, an additional extension would be to investigate the properties of survey estimators when the completed matrix X S is used as auxiliary information.

X S = argmin
It seems to me that this approach could be a valuable extension to the work presented in this thesis; indeed, in Chapter 4 and 5, we examined the properties of imputed estimators in a framework in which a set of p covariates were assumed to be fully observed at the sample level. However, in practice, nonresponse might also be present in the covariates. As such, the methodologies detailed in 4 and 5 cannot be applied directly: an adaptation must be made. Matrix completion procedures could be an interesting possibility for this adaptation.

  Superpopulation model (Ω, M, P m , Y ) Population U with {y 1 , y 2 , ..., y N } Sample s with {y (1) , y (2) , ..., y (n) } Parameter space (F, F ) T H θ M = T(P m • Y -1 ) θ U = H(y 1 , y 2 , ..., y N )

Figure 3 :

 3 Figure 3: Estimation of a regression function with a regression tree and a random forest algorithm.

  Result 1.3.1. Consider a sequence of predictors { m} tted on D n r and its population counterparts { m} tted on D N := {(x k , y k ) ; k ∈ U} . If i) The sequence of population predictors { m} satis es lim v→∞ E m(x)m(x) 2 = 0, with a convergence rate denoted γ v .

Figure 4 :

 4 Figure 4: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 1 with SRSWOR (n = 600) and increasing number of auxiliary variables

Figure 5 :

 5 Figure 5: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 2 with SRSWOR, n = 600 and increasing number of auxiliary variables

Figure 6 :

 6 Figure 6: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 3 with SRSWOR, n = 600 and increasing number of auxiliary variables

Figure 7 :

 7 Figure 7: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 4 with SRSWOR, n = 600 and increasing number of auxiliary variables

Figure 8 :

 8 Figure 8: Relative bias of model-assisted estimators t (j)

Figure 10 :

 10 Figure 10: Relative e ciency of model-assisted estimators t (j)

Figure 12 :

 12 Figure 12: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 1 with strati ed without replacement X 2 -proportional to size sampling, n = 600 and increasing number of auxiliary variables

Figure 13 :

 13 Figure 13: Relative e ciency of model-assisted estimators t (j) ma , j = 1, . . . , 12 for the estimation of the total of Y 3 with strati ed without replacement X 2 -proportional to size sampling, n = 600 and increasing number of auxiliary variables

Figure 14 :

 14 Figure 14: Average electricity consumption on each stratum during rst week

Figure 15 :

 15 Figure 15: A regression tree (left) and the corresponding partition of R 2 (right).

Figure 16 :

 16 Figure 16: Evolution of the relative bias with respect to n 0 .

Figure 17 :

 17 Figure 17: Evolution of the e ective coverage with respect to n 0 .

Figure 18 :

 18 Figure 18: Relative e ciency of t r f for the survey variable Y 5 and for several values of n 0 .

Figure 20 :

 20 Figure 20: Relative e ciency of t r f for the survey variable Y 8 and for several values of B.

Figure 21 :

 21 Figure 21: Relative e ciency of t r f for the survey variable Y 8 and for several values of m tr y .

  Similar results can be obtained using other simulation parameters. (C4) The number of subsampled elements n v is such that lim v→∞ n v /n v ∈ (0; 1]. Consistency of the Horvitz-Thompson variance estimator (H18) Assume that lim v→∞ max i,j,k, ∈D 4,Nv |E p I i I j -π i π j (I k I -π k π ) | = 0, where D 4,N v denotes the set of distinct 4-tuples from U v .

  .33) We remark that m * (b) tr ee (x k ) is the Horvitz-Thompson estimator of m * (b) tr ee (x k ). As before, m * (b) tr ee depends on θ (U) b but, for more readability, we drop θ (U) b from the expression of m * (b) tr ee (x k ). We give in the next equivalent expressions of m * (b) tr ee and m * (b) tr ee . Consider for that the B partitions built at the population level: P * U = { P * (b) U } B b=1 . For a given b = 1, . . . , B, the partition P * (b) U build in the b-th stochastic tree is composed by the J * bU disjointed regions: P * (b)

  j = 1, . . . , J * bU . We drop the exponent U from the expression of z * (b) k for more readability. Since P * (b) U is a partition, then x k belongs to only one region of the b-th tree, so the vector z * (b) k will contain only one non-null component. Consider for example that x k ∈ A * (bU) j , then m * (b) tr ee (x k ) is the mean of y-values of individuals for which x ∈ A *(bU)

  41) by assumptions (H13)-(H15) and the fact that ||z * (b) k || 2 2 = 1 for all k ∈ U v and b = 1, . . . , B. From (3.40), (3.41) and assumption (H13), we obtain that it exists a positive constant c2 such that

  44) by assumptions (H13)-(H15) and the fact that ||z * (b) k || 2 2 = 1 for all k ∈ U v and b = 1, . . . , B.

  57), (3.58), (3.59) and (3.60) give then the result.

Figure 22 :

 22 Figure 22: Example of a graph induced by a tree algorithm.

Figure 23 :

 23 Figure 23: Monte Carlo percent relative bias across the scenarios.

Figure 24 :Figure 25 :

 2425 Figure 24: Monte Carlo percent relative e ciency across the scenarios.

Figure 26 :

 26 Figure 26: The e ects of the nonresponse mechanism on the performance of the 10 best imputation procedures.

  (4.40) and (4.41) with t imp replaced with Q γ,imp , t π replaced with Q γ and t y replaced with Q γ .The results are presented in Figures27-29. In each gure, the x-axis corresponds to the median of the Monte Carlo percent relative bias of Q γ,imp computed across the 4 scenarios, whereas the y-axis corresponds to the median of the Monte Carlo relative e ciency. For the purpose of clarity, we have excluded from Figures 27-29 any imputation procedure whose median of the Monte Carlo percent relative bias lied outside the interval[-20; 20] or whose median of the Monte Carlo relative e ciency was above 500.

Figure 27 :

 27 Figure 27: Median performances of the best imputed estimators for the estimation of Q 0.25 .

Figure 28 :

 28 Figure 28: Median performances of the best imputed estimators for the estimation of Q 0.5 .

Figure 29 :

 29 Figure 29: Median performances of the best imputed estimators for the estimation of Q 0.75 .

  Result 5.2.1. Assume (H19) and (H20). Consider a sequence of predictors { m} tted on D n r and its population counterparts { m} tted onD N := {(x k , y k ) ; k ∈ U} . If i)The sequence of population predictors { m} satis eslim v→∞ E m(x)m(x) 2 = 0,with a convergence rate denoted γ v .

  m v ∈ G v := {g(x) = p v j=1 g j (x j ), g j is bounded and Borel measurable, j = 1, 2, ...p v },and ||m v || l 0 = #{ j = 1, 2, ..., p v ; m j non-constant} = o( √ K v ).

  We begin by noting that, from Corollary 4.3 of[START_REF] Klusowski | Universal consistency of decision trees in high dimensions[END_REF], it follows that the sequence { m tr ee,v } of tree predictors tted on D N v is universally consistent in L 2 for m, meaning lim v→∞ E m tr ee (x)m(x) 2 = 0, which is Condition i) of Result 5.2.1. Since we are in a framework in which Y is almost surely bounded, it follows that E m tr ee (x)m(x)

Lemma 7 .

 7 Assume (H19). Let { m} be a sequence of L 2 consistent regression function estimates and let { m} be the corresponding estimates tted on D n r = {(x k , y k ); k ∈ S r }. Then, { m} is such that, In other words, the random variable E m(x)m(x) 2 X := g(X) converges in L 1 towards 0, which implies that g(X) P → 0. Furthermore, note that, under (H20) and (H19), we have almost sure equality of the two random measures P Y |X and P Y |X,I ,r . That is, the nonresponse mechanism and the sampling design are ignorable. Therefore, xing the sample of respondents and using the equality of conditional distribution gives E m(x)m(x)

  (b) } B b=1 . Moreover, in our framework, these are zero mean bounded random variables. To see that, rst note that inf ω ∈Ω Y Y (ω) and sup ω ∈Ω Y Y (ω) are nite constants. Hence, for all b ∈ {1, 2, ..., B} and k∈ S m , inf ω ∈Ω Y Y (ω)sup ω ∈Ω Y Y (ω) m (b) tr ee (x k )m (∞) r f (x k ) sup that inf ω ∈Ω Y Y (ω)sup ω ∈Ω Y Y (ω) < 0, it follows that n m • inf ω ∈Ω Y Y (ω)sup ω ∈Ω Y Y (ω) min k ∈U π k d (b) n m • sup ω ∈Ω Y Y (ω)inf ω ∈Ω Y Y (ω)min k ∈U π k , a.s.

  4.3, conditionally on everything but {Θ b } B b=1 , { t (b) tr ee } B b=1 is a sequence of i.i.d. random variables. Now, for any b ∈ {1, 2, ..., B}, V Θ t (b) tr ee = V Θ

ZF

  ∈R n×p ||Z || * + λ 2 ||P Ω|X (Z) -P Ω|X (X S )|| 2 subject to ||HT π (Z)t x || 2 tolerance γ,and where HT π denotes the Horvitz-Thompson operator HT π : R n×p → R p

Table 2 :

 2 Monte Carlo percent relative bias and relative e ciency of several model-assisted estimators under strati ed simple random sampling with proportional allocation.

  .26) where || • || 2 is the spectral norm matrix de ned for a squared p × p matrix A as ||A|| 2 = sup x∈R p ,||x|| 2 0 ||Ax|| 2 /||x|| 2 . For a symmetric and positive de nite matrix A, we have that ||A||

2 = λ max (A), where λ max (A) is the largest eigenvalue of A. Now, we can write

Table 3 :

 3 The working models Survey variable Vector of explanatory variable X used in the working model

Table 4 :

 4 Monte Carlo percent relative bias (RB) and Monte Carlo e ciency (RE) of several model-assisted estimators for n = 250

	Population		GREG	CART	RF1	RF2	RF3
	Y 1	RB RE	-0.0 3.0	-0.0 3.5	-0.0 3.7	-0.0 3.6	0.0 3.4
	Y 2	RB RE	-0.0 101.0	0.0 37.6	0.0 39.4	0.0 38.3	0.0 35.0
	Y 3	RB RE	0.0 19.6	-0.0 55.2	-0.1 33.8	-0.1 34.0	-0.0 35.4
	Y 4	RB RE	-0.7 81.1	-1.2 61.1	-1.2 49.7	-1.5 49.0	-0.7 53.1
	Y 5	RB RE	-0.1 37.9	0.1 32.7	-0.0 25.8	-0.0 26.5	-0.0 30.7
	Y 6	RB RE	-0.0 105.2	0.3 72.2	-0.0 57.5	-0.0 57.5	-0.0 58.3
	Y 7	RB RE	-0.0 127.6	0.2 84.3	0.1 75.8	0.0 75.5	0.0 76.8
	Y 8	RB RE	0.0 127.0	0.0 135.6	0.0 92.7	0.0 92.5	0.0 95.6

Table 5 :

 5 Monte Carlo percent relative bias (RB) and Monte Carlo e ciency (RE) of several model-assisted estimators for n = 1000.

	Population	GREG CART	RF1	RF2	RF3
	Y 1	RB 0.0 RE 2.8	0.0 3.5	0.0 3.6	0.0 3.5	0.0 3.0
	Y 2	RB 0.0 RE 100.1	0.0 38.7	0.0 40.5	0.0 39.6	0.0 33.3
	Y 3	RB 0.0 RE 20.4	0.0 41.1	-0.1 28.1	-0.1 27.8	0.0 31.6
	Y 4	RB -0.1 RE 78.9	-1.1 52.3	-0.9 36.7	-0.7 36.1	-0.2 44.5
	Y 5	RB -0.0 RE 37.3	0.0 24.5	0.0 20.9	0.0 21.2	-0.0 24.8
	Y 6	RB 0.0 RE 101.1	0.0 65.5	-0.0 49.1	-0.0 49.2	-0.0 50.3
	Y 7	RB 0.0 RE 105.5	0.0 73.2	0.0 63.3	0.0 63.2	0.0 65.0
	Y 8	RB -0.0 RE 166.6	-0.0 137.6	-0.0 96.0	-0.0 95.7	0.0 89.5

  3.39) where || • || 2 is the spectral norm matrix de ned for a squared p × p matrix A by ||A|| 2 = sup x∈R p ,| |x| | 2 0 ||Ax|| 2 /||x|| 2 . For a symmetric and positive de nite matrix A, we have that ||A|| 2 = λ max (A) where λ max (A) is the largest eigenvalue of A. We get, for b = 1, . . . , B :

Table 7 :

 7 . , Y 6 and Y 8 , with simple random sampling without replacement. The results for Poisson sampling used in the case of Y 4 and Y 7 are discussed in Section 4.4.2. Finally, the case of the binary variables Y 9 and Y 10 , whose totals were estimated with simple random sampling without replacement, is discussed in Section 4.4.2. Monte Carlo percent absolute relative e ciency of the imputed estimator: Descriptive statistics over all the scenarios

	Ranking	Model	Min	Q 0.05	Q 0.25	Q 0.5	Q 0.75	Q 0.95	Max
	1	CUBIST3	102	102	111	115	125	158
	2	BART	113	113	116	122	131	154
	3	AMS5	100	101	111	123	147	378
	4	AMS10	100	101	112	123	167	1195
	5	XGB1	101	103	115	129	153	203
	6	CUBIST2	102	103	119	133	187	360
	7	XGB2	102	102	117	133	166	316
	8	CUBIST1	103	105	120	136	182	360
	9	SVR1	94	103	122	141	180	284
	10	SVR3	95	106	122	143	181	269
	11	RF3	115	118	131	149	192	919
	12	RF2	113	118	130	151	202	824
	13	CART	125	134	143	168	248	1498
	14	LR	110	111	114	169	315	823
	15	MWC50	113	114	122	171	205	308
	16	HDWC50	120	120	128	189	240	332
	17	MWC100	116	116	136	191	217	296
	18	NN	101	111	125	194	378	486
	19	XGB3	92	100	128	194	663	1082
	20	HDWC100	123	125	142	213	246	322
	21	RF1	136	137	149	223	375	3656
	22	MWC250	128	130	159	229	279	383
	23	5NN	94	108	123	229	659	775
	24	SVR2	97	102	151	242	1616	3849
	25	SVR4	82	89	117	258	1439	4301
	26	HDWC250	141	143	185	265	325	411
	27	MWC500	151	155	202	269	336	1783

Table 8 :

 8 Best 5 imputation procedures for each survey variable.

Table 10 :

 10 1500);Y 3 = 500 + 2X 4 + 4001 {X 5 >156} -4001 (X 5 156) + 10001 (X 2 > 190)+ 3001 (X 5 > 200) + N (0, 1500); Monte Carlo percent relative e ciency of the imputed estimator: Descriptive statistics for Poisson sampling.

	Ranking	Model	Min	Q 0.5	Max
	1	BART	106	117	139
	2	CUBIST3	111	118	239
	3	XGB1	108	133	207
	4	RF2	114	144	565
	5	RF3	114	145	621
	6	XGB2	110	156	246
	7	SVR3	109	165	198
	8	AMS5	124	168	486
	9	SVR1	109	175	209
	10	CUBIST1	114	175	469
	11	NN	117	178	234
	12	MWC50	125	188	396
	13	MWC100	125	188	363
	14	RF1	122	188	1868
	15	LR	123	189	923
	16	MWC250	128	190	525
	17	CUBIST2	111	193	548
	18	CART	133	198	1224
	19	MWC500	133	198	1346
	20	HDWC50	135	210	409
	21	HDWC100	139	213	381
	22	HDWC250	145	217	539
	23	5NN	120	241	370
	24	XGB3	116	272	441
	25	AMS10	130	313	592
	26	SVR2	142	493	1619
	27	SVR4	141	769	2119

Table 11 :

 11 . Monte Carlo percent relative e ciency of the imputed estimator: Descriptive statistics for the binary survey variables.

		4.4		:	157
	Ranking	Model	Min	Q 0.5	Max
	1	NN	136	144	428
	2	XGB3	153	165	860
	3	XGB2	156	167	827
	4	CUBIST3	156	167	841
	5	XGB1	156	171	932
	6	BART	156	173	1052
	7	5NN	152	174	1191
	8	CUBIST2	163	179	873
	9	CUBIST1	169	191	904
	10	RF2	158	192	1572
	11	RF3	162	198	1769
	12	AMS5	169	219	2453
	13	MWC100	160	221	1120
	14	MWC50	159	222	1067
	15	SVR1	171	222	3196
	16	AMS10	165	223	2472
	17	MWC50	159	223	1061
	8	M100	159	225	1116
	19	CART	176	229	1882
	20	LR	164	230	2707
	21	MWC250	172	244	1460
	22	MWC250	173	246	1471
	23	SVR3	191	280	2899
	24	RF1	190	305	4666
	25	M500	186	365	4977
	26	SVR4	219	409	26429
	27	SVR2	413	1839	17279

Table 12 :

 12 Monte Carlo percent absolute relative bias of the imputed estimator: Descriptive statistics for the binary survey variables.

	Ranking	Model	Min	Q 0.5	Max
		NN	0.0	0.5	3.6
		CUBIST3	0.02	0.7	6.7
		XGB3	0.03	0.8	7.7
		BART	0.1	0.8	8.8
		XGB1	0.14	0.9	7.9
		XGB2	0.0	0.9	6.9
		5NN	0.0	1.0	7.3
		CUBIST2	0.2	1.0	7.0
		CUBIST1	0.0	1.1	6.8
		RF2	0.12	1.5	10.3
		RF3	0.13	1.6	11.0
		AMS5	0.04	1.6	11.9
		AMS10	0.1	1.6	11.9
		SVR1	0.3	1.7	12.0
		LR	0.19	1.8	12.3
		CART	0.18	1.8	11.4
		MWC50	0.0	1.8	7.5
		MWC100	0.0	1.8	7.7
		HDWC50	0.03	1.8	7.5
		HDWC100	0.01	1.8	7.7
		MWC250	0.0	2.0	9.4
		HDWC250	0.0	2.0	9.4
		SVR3	0.43	2.3	11.5
		RF1	0.08	2.7	19.0
		SVR4	0.17	3.0	36.5
		MWC500	0.0	3.2	16.4
		SVR2	1.9	9.5	33.9

Table 13 :

 13 Relative biais (RB) and relative e ciency (RE) of imputation procedures with p = 15 auxiliary variables.

	Variable Dim	Criterion LR	MWC50 RF2 XGB1 NN SVR3 CB3 PCR1 PCR2 PCR3 BART
	Y 1	p=100	RE RB	102 122 0,14 2,1	149 103 4,2 0,3	216 187 6,2 5,1	100 269 0 7,8	226 6,6	151 4,0	105 0,6
	Y 2	p=100	RE RB	115 -23,8 34,3 287	109 100 7,5 0,1	100 340 3,3 26,1	100 100 -0,0 -31,0 -28,9 -32,5 5,8 108 140 127
	Y 3	p=100	RE RB	158 3,2	185 3,9	107 107 1,1 -0,0	354 162 7,0 3,4	108 236 0,9 5,9	224 5,5	196 4,8	129 7,7
	Y 4	p=100	RE RB	140 0,0	141 0,1	151 146 0,7 0,28	243 217 0,4 -1,5	122 120 -0,0 -0,0	120 -0,1	121 -0,1	135 -0,0
	Y 1	p=300	RE RB	120 -0,2 1 215	190 103 5,7 0,6	286 237 7,05 6,7	100 290 0,06 8,3	262 7,7	189 5,7	110 1,3
	Y 2	p=300	RE RB	102 1106 -6,3 89,1	112 100 9,5 0,1	100 405 4,01 35,	100 91 -0,0 -28,4 -25,3 -26,9 4,6 85 109 243
	Y 3	p=300	RE RB	197 1,0	378 6,7	118 107 2,0 0,0	630 180 9,1 4,1	108 350 0,8 6,2	245 6,1	224 5,6	242 6,4
	Y 4	p=300	RE RB	276 0,1	584 2,4	155 143 0,7 0,3	443 214 0,6 -1,5	124 120 0,06 -0,0	120 -0,1	121 -0,1	131 -0,0

Table 14 :

 14 Relative biais (RB) and relative e ciency (RE) of imputation procedures with p = 100 and respectively, p = 300 auxiliary variables.

Table 15 :

 15 below. Summary of the data structure

		π k	X	Y
	S A	Known	Observed Unobserved
	S B Unknown Observed	Observed

  Table 16 and Table 17, respectively.

	Survey variable Design	LR CART NN1 NN5 RF
	Y1	SRSWOR 108 143 STRAT 113 159	131 159 118 130 126 124
	Y2	SRSWOR 122 111 STRAT 120 129	196 299 113 158 215 112
	Y3	SRSWOR 206 168 STRAT 209 153	256 339 160 227 244 145
	Y4	SRSWOR 146 163 STRAT 159 187	175 244 134 161 209 147
	Y5	SRSWOR 139 101 STRAT 144 103	188 257 103 172 210 104

Table 16 :

 16 Relative e ciencies (%) of the imputed estimators.

Table 20 :

 20 Monte Carlo percent relative bias (RB) and Monte Carlo e ciency (RE) of several mass imputation estimators.

Table 21 :

 21 Coverage rates and relative biases of Vrf in percentage was biased for n B = 500 with a value of absolute RB of about 25.1%. Again, the bias decreased as n B increased. For n B = 10000, the absolute RB was approximately equal to 1.5% and the coverage rate was about 94.5%. The results suggest that the coverage rate is close to the nominal rate when n B ≥ n A .

	Sample size	Criterion	n B = 500	n B = 2000	n B = 10000
	n A = 500	RB Coverage	-5.5 94.1	-3.6 94.1	0.0 94.7
	n A = 2000	RB Coverage	-25.1 91.5	-7.7 94.5	-1.5 94.5

  Z ∈R n×d ||Z || * subject to P Ω S (Z) = P Ω S (S) . (6.4) This convex relaxation has been widely studied in the literature, see e.g. Candès and Recht (2009), Candès and Tao (2010). Several authors (e.g. Mazumder et al. (2010)) considered a problem in which it was assumed that the observed values are noisy and thus suggested solving the following problem instead: minimize Z ∈R n×d ||Z || * subject to ||P Ω S (Z) -P Ω S (S)|| 2

	F	δ,	(6.5)

with δ > 0, a given tolerance. Equivalently, (6.5) can be rewritten minimize

Z ∈R n×d ||Z || * + λ δ ||P Ω S (Z) -P Ω S (S)|| 2 F ,

(6.6)

Médiamétrie is leader of media and reference studies in audience measurements in France, see the website https://www.mediametrie.fr/en for more details.

This work is a collaboration which is not part of this PhD and therefore will not be included in this manuscript.

This approximation is valid for high entropy sampling designs, where the entropy I(p) of a design p is de ned by I(p) :=s ∈S p(s) log(s), see[START_REF] Tillé | Sampling and estimation from nite populations[END_REF].

Here, the notation p is used to denote the number of covariates, and not the sampling design.

If the chosen leaf A contains less than 2 × n 0 elements, then A is a terminal node. In this case, return to step 1. for the next node.

For a function f : X → R, we call complexity of f the number of di erent values that f can take, that is, the quantity # ({ f (x); x ∈ X}).

A bootstrap sample of U is a sample of size N, selected from U with replacement.1.2

In a random forest, a resampling mechanism is a process consisting of selecting, with or without replacement, observations from the original data before constructing the trees.

A model-assisted estimator t ma ( m) is said to be a projection estimator if it can be written as the sum of predictions of all population elements, i.e., t ma ( m) = k ∈U m(x k ).

The article is accepted for publication in Journal of Applied Statistics, in Special Issue: Statistical Approaches for Big Data and Machine Learning.

The data are available on request at: https://www.ucd.ie/issda/data/commissionforenergyregulationcer/.2.4

The article is accepted for publication in Journal of the American Statistical Association.

The assumptions presented here are the same as those presented in the main document; they are only recalled here for simplicity of exposition.

The article is accepted for publication in Journal of Survey Statistics and Methodology.

The data are available on request at: https://www.ucd.ie/issda/data/commissionforenergyregulationcer/.

Remerciements Introduction

Nearest-neighbour (NN) imputation corresponds to the limiting case of KNN obtained with K = 1. NN is a donor imputation belonging to the class of hot-deck procedures [START_REF] Chen | Nearest neighbor imputation for survey data[END_REF] since a missing value is replaced by an actual respondent y-value from the same le. NN imputation is especially useful for imputing categorical or discrete Y -variables; e.g., see [START_REF] Chen | Nearest neighbor imputation for survey data[END_REF], [START_REF] Beaumont | Variance estimation when donor imputation is used to ll in missing values[END_REF] and [START_REF] Yang | Nearest neighbor imputation for general parameter estimation in survey sampling[END_REF].

Let N K (i) be the set of K responding units closest to x i . Any distance function in R p may be used to measure the closeness between two vectors x i and x j . In the simulation study presented in Section 4.4, we used the customary Euclidean distance. The KNN imputed value for missing y i is given by

The imputed value y i obtained with KNN can be written as a weighted sum of the respondent y-values:

where w i j = w j 1( j ∈ N K (i))/ j ∈N K (i)∩S r w j for j ∈ S r with j ∈S r w i j = 1. KNN imputation is a locally weighted procedure since the respondents j lying not close enough to unit i with respect to the X-variables are assigned a weight equal to 0; i.e., w i j = 0. The indicator function in the expression of w i j can be replaced by a one-dimensional continuous kernel smoother K h , whose role is to control the size of the weight through a tuning parameter h : the units j lying farther from unit i will be assigned a smaller weight than units lying close to it [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF].

The imputed estimator under KNN imputation tends to be ine cient when the dimension p of x is large. Indeed, as p increases, it becomes more di cult to nd enough respondents around the point at which we aim to make a prediction. This phenomenon is known as the curse of dimensionality (Hastie et al., 2011, Chap. 1) for a more in-depth discussion ok the KNN procedure. Also, it su ers from a model bias which is of order (K/n) 1/p . Nearest-neigbour imputation for missing survey data has been considered in [START_REF] Chen | Nearest neighbor imputation for survey data[END_REF], [START_REF] Beaumont | Variance estimation when donor imputation is used to ll in missing values[END_REF] and [START_REF] Yang | Nearest neighbor imputation for general parameter estimation in survey sampling[END_REF].

B-splines and additive model nonparametric regression

Spline regression is a exible nonparametric method for tting non-linear functions f (•). It can be viewed as a simple extension of linear models. For simplicity, we start with a univariate X-variable supported on the interval [0; 1]. A spline function of order v with κ equidistant interior knots, 0 = ξ 0 < ξ 1 < ... < ξ κ < ξ κ+1 = 1, is a piecewise polynomial of degree v -1 between knots and smoothly connected at the knots. These spline functions span a linear space of dimension of q = v + κ with a basis function given by the B-splines functions:

x and equal to zero, otherwise; see [START_REF] Dierckx | Curves and Surface Fitting with Splines[END_REF][START_REF] Schumaker | Spline Functions: Basic Theory[END_REF]. The B-spline basis is appealing because the basis functions are strictly local: each function B (•) has the knots ξ -v , . . . , ξ with ξ r = ξ min(max(r,0),κ+1) for r =v, . . . , [START_REF] Zhou | Local asymptotics for regression splines and con dence regions[END_REF], which means that its support consists of a small, xed, nite number of intervals between knots. The unknown function f (•) is then approximated by f (•), a linear combination of basis functions {B } q =1

Continuous survey variables and simple random sampling without replacement For simple random sampling without replacement, for each of the twenty-seven imputation procedures, we had seven survey variables and four nonresponse mechanisms, leading to 27 × 4 × 27 = 756 sets of simulation results. For ease of presentation, we present the results in tabular and graphic forms. The displayed statistical analyses were obtained from 4 × 7 = 28 scenarios obtained by crossing all the nonresponse models and the survey variables.

For each imputation procedure, Table 6 andTable 7 display, respectively, some descriptive statistics regarding the Monte Carlo absolute percent relative bias (absolute value of RB) and the Monte Carlo relative e ciency (RE) of t imp calculated across the twenty-eight scenarios. The corresponding sideby-side boxplots obtained from the twenty-eight scenarios are given in Figures 23 and24. In Tables 6 and7, the imputation procedures are ordered from the best to the worst with respect to the median absolute percent RB (the median of the twenty-eight values of absolute RB) and the median percent RE (the median of the twenty-eight values of RE), respectively. Figure 25 shows the distribution of the imputed estimator for the best ten imputation procedures in terms of RE. Finally, Table 8 displays the best ve imputation procedures for each Y -variable.

From Table 6 and Table 7, among the twenty-seven imputation procedures, the best methods were: CUBIST, XGboost, AMS and BART. The performance of CUBIST3 was especially impressive with a median RE of 115%, a value of Q 95 equal to 158% and a maximum value of 211%. The methods XGboost, AMS and BART exhibited similar performances with values of median RE ranging from 122% and 129%. However, for some scenarios, these methods did not perform well. For instance, the procedure XGB2 showed a value of max RE of about 438%, whereas it was equal to 1728% for AM5. Results suggest that additive models with 5 interiors knots perform better than those with 10 interior knots. The next group of imputation procedures includes SVR and RF, with values of median RE ranging from 141% and 151%. Again, for some scenarios, both methods displayed poor performances with values of max RE ranging from 322% to 1138%. The procedure CART was less e cient than RF2 and RF3. The procedure 1-NN did relatively well with a median RE equal to 194%. On the other hand, the procedure 5-NN was rather ine cient with a median RE of 229%, which suggests that KNN with survey data works well only with a small number of neighbour. Turning to mean and random hot-deck imputation within classes, the score method was outperformed by the aforementioned procedures. Among the di erent versions of MCW and HDWC, the procedure MWC50 (which corresponds to 20 classes) led to the best results. This is consistent with the results of [START_REF] Haziza | On the construction of imputation classes in surveys[END_REF]. As expected, the procedure HDWC50 was less e cient than MWC50 as random hot-deck imputation su ers from the imputation variance, arising from the random selection of donors within classes. Finally, for some scenarios, it is worth noting that some of the procedures were better than the complete data estimator. For instance, for SVR4, the minimum value of RE and the value of Q 0.05 were respectively equal to 82% and 89%, respectively (see Table 7). Finally, the results in Table 5 suggest that the best methods were CUBIST, XGBoost, additive models and BART, which is consistent with the discussion above.

For each of the best ten imputation procedures displayed Table 7, Figure 26 displays the distribution of t imp for each nonresponse mechanism. Figure 26 suggests that the nonresponse mechanism may have a considerable impact on the behavior of the imputed estimator. For instance, in our experiments, we note that most of the imputation procedures performed poorly in the case of the nonresponse mechanism (NR1). Notable exceptions were AMS5, BART and Cubist3. In particular, Cubist3 seemed to be insensitive to the nonresponse mechanism, which is a desirable feature.

4.4

: 149

Ranking

where 1 ∼ E(2) and these error terms were centered so as to have a mean equal to zero. We were interested in estimating the population total of the survey variables Y 1 -Y 4 . Again, the simulation was based of R = 5, 000 iterations of the process described in Section 4.4. Samples of size n = 1000 were selected according to simple random sampling without replacement. The missing values to the survey variables Y 1 -Y 4 were generated according to

leading to an average response rate of about 50%.

165

In this paper, the aim is to study a number of random forest algorithms that have been suggested in the literature. In Section 5.2, we begin by introducing a set of two su cient conditions on an imputation model, so that, whenever satis ed, leads to the L 2 -consistency of the resulting imputed estimator. In Section 5.3, we provide an analysis of trees imputed estimators. Finite sample properties are derived through the analysis of the corresponding weighting system. The L 2 -consistency of the tree imputed estimator based on the CART algorithm [START_REF] Breiman | Classi cation and regression trees[END_REF] is established. In Section 5.4, we focus on random forest imputed estimators. We begin by establishing the connection between tree imputed estimators and random forests imputed estimators. As such, random forest estimators inherit many of the properties of tree estimators, minor a few di erences that are highlighted. The L 2 -consistency of forest imputed estimators based on uniform random forests [START_REF] Biau | Consistency of random forests and other averaging classi ers[END_REF], Scornet, 2016a) and Breiman's original algorithm [START_REF] Breiman | Random forests[END_REF] is established. In Section 5.5, using the reverse approach of [START_REF] Shao | Variance estimation for survey data with composite imputation and nonnegligible sampling fractions[END_REF] and the approach of [START_REF] Särndal | Methods for estimating the precision of survey estimates when imputation has been used[END_REF], we suggest two variance estimators that account for the sampling, nonresponse and imputation variability. In Section 5.6, we apply random forest imputation to the case of data integration. Before concluding, we investigate the empirical properties through a simulation study presented in Section 5.7. All proofs and further technical details are relegated to the Appendix.

Mean square consistency of imputed estimators

Consider a nite population U = {1, 2, ..., N } of size N. We are interested in estimating the population total, t y = k ∈U y k , of a survey variable Y . We select a sample S, of size n, according to a sampling design P (S) with rst-order inclusion probabilities {π k } k ∈U and second-order inclusion probabilities {π k } k ∈U ; we shall denote by ∆ k := π k ππ k the sampling covariances, for elements k, ∈ U. The sample S is completely characterized by the vector of sample selection indicators I = (I 1 , . . . , I k , . . . , I N ) , where I k = 1 if k ∈ S and I k = 0, otherwise. A complete data estimator of t y is the well-known Horvitz-Thompson (HT) estimator:

with d k := 1/π k , the sampling weight attached to element k ∈ S. Provided that π k > 0, for all k ∈ U, the estimator (5.1) is design-unbiased for t y .

In practice, the Y variable is subject to missingness. Let r = (r 1 , . . . , r k , . . . , r N ) denote the vector of response indicators such that r k = 1 if y k is observed and r k = 0, otherwise. Let S r = {k ∈ S; r k = 1} denote the set of respondents, of size n r , and S m = {k ∈ S; r k = 0} the set of nonrespondents, of size n m , such that S r ∪ S m = S and n r + n m = n. Let x k = (x 1k , x 2k , ..., x pk ) be the vector of fully observed auxiliary variables attached to unit k and X := {x k } k ∈U . Available to the imputer are the pairs (y k , x k ), for k ∈ S r , and the vector x k for k ∈ S m . In this paper, we restrict our investigations to response mechanisms satisfying the missing at random assumption (MAR, [START_REF] Rubin | Inference and missing data[END_REF]), de ned below.

(H19) The missing data mechanism is such that: a) The random vectors {[r k , y k , x k ] } k ∈U v are independently and identically distributed (i.i.d.).

b) The nonresponse mechanism is missing at random (MAR), that is,

and, for all k ∈ U, P (r k = 1|x k ) > 0.

3. A split is performed in the node G along the axis induced by X j with a location chosen uniformly at random.

The process is repeated K times, with K ∈ N, a parameter chosen by the user.

Example 5.4.3. Breiman's original algorithm.

The algorithm poceeds as follows:

Step 1: Select B bootstrap samples from S r denoted {S r (Θ b )} B b=1 .

Step 2: On each bootstrap sample S r (Θ b ), t a tree predictor m ( • , Θ b ) using the CART algorithm on D n r (Θ b ), where the CART criterion is optimized on p 0 covariates instead of p. The p 0 covariates are chosen uniformly at random (without replacement) among the p covariates available, at each split, according to Θ b .

Uniform random forests are mostly studied in the literature because the partitions of its trees are independent of the observed data, thus making their theoretical analysis simpler. However, because they do not use the data for building the partitions, they are of little practical interest. In practice, Breiman's original algorithm is typically used, but its theoretical analysis is more complicated.

Generally speaking, random forest predictions can be obtained as follows. Let {Θ b } B b=1 denote a sequence of i.i.d. random variables distributed according to some generic random variable Θ and assumed to be independent of the observed data. Let { m tr ee ( • , Θ b )} B b=1 be a sequence of randomized tree predictors. Then, the RF prediction at x is given by (5.11) where S r (Θ b ) = S r if there is no resampling mechanism in the forest. It follows that predictions can be also be written as

with weights given by

k = 0 otherwise. In the sequel, for ease of notation, we suppress the dependence of {Θ b } B b=1 on the predictor and its weight functions in the notations; we write m (B) r f for the predictor, W (B) k for the weight functions, and A b for a node of the b-th tree. Note that m (B) r f is also dependent of D n r , a dependence which is omitted in the notation for readability.

For more details about random forests and their implementation, the reader is referred to [START_REF] Biau | A random forest guided tour[END_REF] and [START_REF] Genuer | Les forêts aléatoires avec R[END_REF].

Random forest imputation

Let m (B) r f be a random forest predictor built on B trees. The random forest imputed estimator t (B) r f of t y is de ned as Overall, the random forest imputed estimator stood out from other models with its good behavior, both in terms of bias and e ciency. It was followed by both the tree and linear regression estimators. Nearest neighbors estimators were less e cient and exhibited (small) negative biases in most scenarios. The conclusions are similar for both strati ed and simple random sampling, as all methods behave in a similar manner in both designs. When the true relationship between the covariates and the survey variable was linear (e.g. Y 1 ), linear regression was the most e cient estimator. For instance, for the estimation of t y1 , the linear regression imputed estimator exhibited a relative e ciency of 108% for SRSWOR and 113% for STRAT, whereas the second best estimator was RF, with 118% and 124%. Thus, even in this scenario, the random forestimputed estimator was e cient and close to the performances of the linear regression imputed estimator. As expected, for non-linear relationships, linear regression was less e cient than random forest. For instance, for the estimation of Y 5 , RF had relative e ciencies of 103% and 104%, where LR had relative e ciencies of 139% and 144%. We emphasize that, as noted in Dagdoug et al. (2022a), in order to make sure that unequal probability sampling designs remain uninformative with a random forest model, the design variables should be considered with high-probability at each split; this is especially true in high-dimensional scenarios.

Performance of variance estimators

We also investigated the performances of variance estimators. To that aim, we generated a larger design matrix X ∈ R N ×p , with N = 100 000, p = 5 and the components X i j were drawn i.i.d. from N (5, 1). We generated the survey variables Y 1 , ..., Y 5 using the same relationships as in the previous section. The nonresponse mechanism was also the same. We used a regression tree imputed estimator with varying node sizes to better understand the impact of this hyper-parameter on variance estimators. We have included the following four di erent variance estimators in our simulations:

• The naive variance estimator (NAIVE) de ned in (5.18); 5.7 183 • The two-phase variance estimator (SAR) de ned in (5.20);

• The two-phase variance estimator (SAR-CV) with σ 2 estimated by means of a cross-validation procedure with K = 5-folds;

• The reverse variance estimator (REV) de ned in (5.22).

For more details on the cross-validation procedure used, the reader is referred to [START_REF] Dagdoug | Model-assisted estimation through random forests in nite population sampling[END_REF].

A Monte-Carlo procedure of R = 10 000 iterations was used to compute the Monte-Carlo relative biases of the variance estimators as well as the 95% coverage that they produce. The results are given in Table 18. For the estimation of Y 5 , all variance estimators exhibited negligible biases and met the required coverage; these results are therefore omitted in Table 18. The relative biases of the naive variance estimators were, for some scenarios, very large. For example, for the estimation of the totals of Y 1 and Y 3 , the naive estimator displayed a relative bias of about 45%. Therefore, as expected, these variance estimators lead to important undercoverages. In most scenarios, the variance estimators SAR and REV behave similarly. For the estimation of Y 1 , the exhibited negative biases ranging between approximately -20% and -10% which lead to coverages between 0.921-0.936. For the estimation of the totals of Y 2 , ..., Y 5 , SAR and REV were close to the 95% required coverage with more than 94% of coverage when enough elements were in the terminal nodes of the tree. We note that, for these two variance estimators, the best results were always obtained for larger values of n 0 , that is, when there were many elements in the terminal nodes. However, independently of the number of elements in the terminal nodes and across all scenarios, the estimator SAR_CV was very e cient, with only negligible positive biases and coverages ranging between 95.1% and 96.1%.

Résumé. Dans cette thèse, nous considérons le problème de l'estimation de totaux en population nie en présence d'un grand nombre de variables auxiliaires. Les scénarios de réponse totale et de non-réponse partielle sont étudiés. Nous examinons les propriétés théoriques et empiriques d'estimateurs assistés par modélisation et d'estimateurs imputés, construits à partir de modèles prédictifs. Les modèles considérés sont des modèles de type machine learning réputés pour être peu sensibles au éau de la dimension, fréquemment étudiés dans la littérature de l'apprentissage statistique.

Dans un cadre de réponse totale, nous examinons les propriétés de di érents estimateurs assistés par modélisation en considérant un cadre asymptotique dans lequel le nombre de covariables tend vers l'in ni. Des conditions su santes sont obtenues pour la convergence d'estimateurs par modélisation assistée basés sur des modèles linéaires et linéaires pénalisés tels que Ridge, Lasso ou Elastic-net. De plus, une nouvelle classe d'estimateurs des totaux par modélisation assistée basée sur des algorithmes de forêts aléatoires est suggérée. Leurs propriétés en échantillons nis et asymptotiques sont étudiées. Des estimateurs de la variance, classique et basé sur la validation croisée, sont également proposés. L'e cacité des estimateurs est testée sur des données simulées et des données réelles d'audience fournies par Médiamétrie.

En présence de nonréponse partielle, nous avons réalisé une large étude par simulation pour comparer des estimateurs imputés basés sur di érents modèles prédictifs provenant de l'apprentissage statistique. Nous avons de plus étudié théoriquement les propriétés des arbres de régression et des forêts aléatoires pour l'imputation. Les propriétés en échantillons nis et asymptotiques de ces modèles ont été examinées et leur e cacité a été testée sur des simulations.

Mots-clés: Théorie des sondages; données manquantes; apprentissage statistique; statistique en grande dimension; forêts aléatoires.

Abstract.

In this thesis, we consider the problem of estimating nite population totals in presence of a large number of auxiliary variables. The scenarios of full response and missing data are both investigated. To that aim, we examine the theoretical and empirical properties of model-assisted and imputed estimators based on statistical learning predictors deemed e cient in high-dimensional scenarios.

In case of full response, we examined the properties of existing model-assisted estimators in a high-dimensional asymptotic framework in which the number of covariates increases to in nity. Conditions for the convergence of model-assisted estimators based on linear and penalized linear models such as ridge, Lasso or Elastic-net are obtained. A new class of model-assisted estimators of nite population totals based on random forest algorithms is also suggested. Their nite sample and asymptotic properties are examined. We also suggested a classic and a cross-validated variance estimators. The performances of the estimators suggested are tested via a large simulation study on simulated and Médiamétrie data.

In presence of item nonresponse, we conducted a large-scale simulation study to compare imputed estimators based on many statistical learning predictors. We also investigated theoretically the use of regression trees and random forests predictors for imputation in surveys. Both their nite sample and asymptotic properties are studied and their properties are investigated by means of simulation studies. Keywords: Survey sampling; missing data; statistical learning; high-dimensional statistics; random forests.