Nadim Ballout

Lola Étiévant

Vivian Viallon
email: viallonv@iarc.fr

On the use of cross-validation for the calibration of the adaptive lasso

Keywords: adaptive lasso, cross-validation, calibration, hyperparameter, tuning parameter, one-step lasso

The adaptive lasso refers to a class of methods that use weighted versions of the L 1 -norm penalty, with weights derived from an initial estimate of the parameter vector to be estimated. Irrespective of the method chosen to compute this initial estimate, the performance of the adaptive lasso critically depends on the value of a hyperparameter, which controls the magnitude of the weighted L 1 -norm in the penalized criterion. As for other machine learning methods, cross-validation is very popular for the calibration of the adaptive lasso, that this for the selection of a data-driven optimal value of this hyperparameter. However, the most simple cross-validation scheme is not valid in this context, and a more elaborate one has to be employed to guarantee an optimal calibration. The discrepancy of the simple crossvalidation scheme has been well documented in other contexts, but less so when it comes to the calibration of the adaptive lasso, and, therefore, many statistical analysts still overlook it. In this work, we recall appropriate crossvalidation schemes for the calibration of the adaptive lasso, and illustrate the discrepancy of the simple scheme, using both synthetic and real-world examples. Our results clearly establish the suboptimality of the simple scheme, in terms of support recovery and prediction error, for several versions of the adaptive lasso, including the popular one-step lasso.

Introduction

High dimensional data are characterized by a number p of variables larger, or at least not significantly lower, than sample size n. They have become ubiquitous in many fields, including biology, medicine, sociology, and economy [START_REF] Giraud | Introduction to high-dimensional statistics[END_REF]. Their analysis raises a number of statistical challenges [START_REF] Fan | Statistical challenges with high dimensionality: Feature selection in knowledge discovery[END_REF]Li, 2006, Hastie et al., 2009), usually summarized under the term "curse of dimensionality". Consequently, it has attracted a lot of attention in the statistical literature over the past decades [START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF][START_REF] Donoho | High-dimensional data analysis: The curses and blessings of dimensionality[END_REF][START_REF] Fan | Statistical challenges with high dimensionality: Feature selection in knowledge discovery[END_REF][START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF][START_REF] Hastie | Statistical learning with sparsity: the lasso and generalizations[END_REF]. In particular, a variety of approaches based on the optimization of penalized versions of the log-likelihood have been developed, to estimate the true parameter vector β * = (β * 1 , . . . , β * p ) T ∈ R p under high-dimensional parametric regression models (Huang et al., 2008a[START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. These approaches use a penalty term whose strength is controlled by a hyperparameter -or tuning parameter -and that is added to the loss-function so that the estimation can take advantage of some expected property of the true parameter vector β * . For example, when β * is expected to be sparse, popular approaches rely on the use of L q penalties, for some q ≤ 1. Among such approaches, the arguably most popular one is the lasso, which uses an L 1norm penalty. Extensions such as the group lasso [START_REF] Jacob | Group lasso with overlap and graph lasso[END_REF], fused lasso [START_REF] Tibshirani | Sparsity and smoothness via the fused lasso[END_REF], generalized fused lasso [START_REF] Viallon | On the robustness of the generalized fused lasso to prior specifications[END_REF], data shared lasso [START_REF] Ballout | Sparse estimation for case-control studies with multiple disease subtypes[END_REF][START_REF] Ballout | Structure estimation of binary graphical models on stratified data: Application to the description of injury tables for victims of road accidents[END_REF][START_REF] Gross | Data shared lasso: A novel tool to discover uplift[END_REF][START_REF] Ollier | Regression modeling on stratified data with the lasso[END_REF], etc., rely on structured sparsity inducing norms, and can be used when some particular structured sparsity is expected in β * .

We will here focus on another extension of the lasso, namely the adaptive lasso (Bühlmann andvan De Geer, 2011, Zou, 2006). It refers to a class of methods where the L 1 -norm β 1 = p j=1 |β j | used in the standard lasso is replaced by a weighted version p j=1 w j |β j |. Weights w j ≥ 0 are typically data-driven, of the form w j = 1/(| βj | + ε), with ε ≥ 0 and ( βj ) 1≤j≤p some initial estimates of the parameters (β * j ) 1≤j≤p . Three popular versions of the adaptive lasso will be considered in this work: (i) the original adaptive lasso introduced by [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF], where weights are derived from Ordinary Least Squares (OLS) estimates; (ii) the version proposed by [START_REF] Bühlmann | Discussion: One-step sparse estimates in nonconcave penalized likelihood models[END_REF], where weights are computed from lasso estimates; and (iii) the version proposed by [START_REF] Zhang | Some two-step procedures for variable selection in high-dimensional linear regression[END_REF], where weights are computed from ridge estimates. In the rest of this article, the original adaptive lasso introduced by [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF] will be referred to as the ols-adaptive lasso, the one proposed by [START_REF] Zhang | Some two-step procedures for variable selection in high-dimensional linear regression[END_REF] as the ridge-adaptive lasso, while we will refer to the method described in [START_REF] Bühlmann | Discussion: One-step sparse estimates in nonconcave penalized likelihood models[END_REF] as the one-step lasso, following their terminology.

The adaptive lasso is very popular in practice. First, it can be implemented very easily and efficiently using algorithms originally developed for the lasso, such as the glmnet R package [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF]. Second, it has been shown to usually outperform the lasso. For example, in the fixed p case, [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF] established that the lasso estimates do not enjoy the asymptotic oracle property (in the sense of [START_REF] Fan | Statistical challenges with high dimensionality: Feature selection in knowledge discovery[END_REF]), while the ols-adaptive lasso estimates do under mild conditions on the value of the hyperparameter that controls the strength of the weighted L 1 -norm penalty. In addition, conditions ensuring support recovery in the non-asymptotic framework (which especially allows the study of the p n case) are weaker for the one-step lasso than for the lasso; see, e.g., Corollaries 7.8-7.9 and Section 2.8.3, in [START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF].

As for other penalized approaches, the theoretical and empirical performance of the adaptive lasso critically depends on the value of the hyperparameter. Its optimal value involves unknown quantities, such as the variance of the noise under linear regression models, but also unknown constants related to the compatibility and irrepresentability conditions of the design matrix [START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF]. Consequently, the practical selection of the tuning parameter, or calibration, also has attracted a lot of attention in the statistical literature [START_REF] Arlot | Minimal penalties and the slope heuristics: a survey[END_REF][START_REF] Chen | Extended bayesian information criteria for model selection with large model spaces[END_REF][START_REF] Chichignoud | A practical scheme and fast algorithm to tune the lasso with optimality guarantees[END_REF][START_REF] Giacobino | Quantile universal threshold[END_REF]. Cross-validation is among the most commonly used strategies for the calibration of the adaptive lasso, and more generally of many statistical and machine learning methods; see, e.g., Chapter 7 in [START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF]. See also [START_REF] Arlot | V -fold cross-validation improved: V -fold penalization[END_REF], [START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF]. Several versions of cross-validation have been described in the literature, but the K-fold version is arguably the most popular one in practice [START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF].

Briefly, K-fold cross-validation first consists in partitioning the original sample D = (y i , x i ) 1≤i≤n into K ≥ 2 balanced folds D (1) , . . . , D (K) , with D = ∪ K k=1 D (k) . Then, it consists of K steps, where at each step k, (i) the fold D (k) is used as an "independent" test sample, while the remaining K -1 folds D \ D (k) are combined and jointly used as the training sample, (ii) the estimator βk is computed on the training sample D \ D (k) , and (iii) its prediction error, say Pred.Err(D (k) , βk ), is evaluated on the test sample D (k) . The cross-validated prediction error is finally defined as the average of these K prediction errors: (1/K)× k Pred.Err(D (k) , βk ). We refer to [START_REF] Bates | Cross-validation: what does it estimate and how well does it do it?[END_REF] for additional details on the estimation of prediction error using cross-validation. This cross-validated prediction error can be used to assess the predictive performance of one estimator, or to compare the predictive performance among a set of estimators, or to select the tuning parameter by considering a set of estimators constructed with different values of the tuning parameter [START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF]. However, for the cross-validation to be valid, it is crucial that the full learning algorithm is run on the training sample only, or, in other words, that data from the test sample are not used during the learning stage at all. For example, it is well known that if some variable selection is involved in the learning algorithm under study, this step has to be performed on the training sample only, and not on the full original sample [START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF].

The adaptive lasso algorithm is made of two steps, where the first one consists in computing the initial estimates ( βj ) 1≤j≤p (through OLS estimation, the lasso, or ridge for example), and the second one consists in computing the adaptive lasso estimate per se with weights derived from these initial estimates. From the principle of the K-fold cross-validation recalled above, both steps have to be performed on the training sample to guarantee the validity of the calibration of the adaptive lasso [START_REF] Krstajic | Crossvalidation pitfalls when selecting and assessing regression and classification models[END_REF][START_REF] Stone | Cross-validatory choice and assessment of statistical predictions[END_REF]. Although this proper cross-validation scheme was described in the literature [START_REF] He | An improved variable selection procedure for adaptive lasso in high-dimensional survival analysis[END_REF][START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF], the discrepancy of a more naive cross-validation scheme, where initial estimates are first derived from the full sample, and then the cross-validation is applied to select the hyperparameter of the adaptive lasso (focusing on the second step only), has not been well described and documented in the literature. In particular, [START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF] do not mention this simple scheme at all, while [START_REF] He | An improved variable selection procedure for adaptive lasso in high-dimensional survival analysis[END_REF] simply note that the it leads to possibly large false discovery rates, without stressing that this scheme is fundamentally inappropriate in this case. Likewise, [START_REF] Bühlmann | Discussion: One-step sparse estimates in nonconcave penalized likelihood models[END_REF] recommend the use of cross-validation for the calibration of the tuning parameters in their onestep lasso for both the initial and final estimators (which is fine if a proper cross-validation scheme is used), but they do not stress the inappropriateness of the simple scheme for calibrating the final estimator, nor do they describe an appropriate scheme. Finally, except for a few exceptions such as the adalasso function of the parcor [START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF] and the cvlasso function of the lassopack package in STATA, proper implementations of the cross-validation for the adaptive lasso are lacking in publicly available adaptive lasso solvers. In particular, when using the popular glmnet R package [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF], it is tempting to implement the simple, and in this case improper, scheme, by simply using the cv.glmnet function to select the hyperparameter of the adaptive lasso after computing the initial estimates (step 1) on the full sample. This improper cross-validation scheme, was used in the illustrative example of Section 2.8.1 of [START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF], and can also be found in the original version of the adapt4pv R package [START_REF] Courtois | New adaptive lasso approaches for variable selection in automated pharmacovigilance signal detection[END_REF]. Consequently, many statistical analysts still seem to apply this improper cross-validation scheme for calibrating the adaptive lasso [START_REF] Chang | Hdmac: a web-based interactive program for high-dimensional analysis of molecular alterations in cancer[END_REF][START_REF] Dessie | A novel mirna-based classification model of risks and stages for clear cell renal cell carcinoma patients[END_REF][START_REF] Pollard | Electrocardiogram machine learning for detection of cardiovascular disease in african americans: the jackson heart study[END_REF]. As will be illustrated in Sections 2 and 3, this generally leads to sub-optimal performance in terms of both support recovery and prediction accuracy.

The rest of the article is organized as follows. In Section 2, we start with a brief overview of the principles of the adaptive lasso, focusing on linear regression models for simplicity. Then, we illustrate the flaw of the simple K-fold cross-validation scheme when used for calibrating the adaptive lasso.

A proper cross-validation scheme in this context is recalled and detailed in Section 3. In Section 4, we present results from a comprehensive simulation study where we empirically illustrate the suboptimality of the simple crossvalidation scheme, in terms of support recovery and prediction accuracy. In Section 5, we present results from an application on single-cell data. Concluding remarks are given in Section 6.

2 The adaptive lasso under the linear regression model

Main notation and working model

As above, we will denote the sample size and the number of covariates by n and p, respectively. For simplicity, we will focus on linear regression models of the form

y = Xβ * + ξ, (1)
where y = (y 1 , . . . , y n ) T ∈ R n is the response vector, X = (x 1 , ...,

x n ) T ∈ R n×p is the design matrix, β * = (β * 1 , ..., β * p ) T ∈ R p
is the p-dimensional vector of unknown parameters to be estimated, and ξ ∈ R n is some random noise. We further denote the support of β * by J = {j : β * j = 0}. For any positive integer d ≥ 1, and any vector u ∈ R d , we will denote the usual Euclidian norm (or L 2 -norm) by u 2 . We will let 0 d and 1 d be the vectors of size d with components all equal to 0 and 1 respectively, and I d be the d × d identity matrix. For any real matrix M = (M 1 , . . . , M d ) ∈ R n×d , and any subset E ⊆ {1, . . . , d}, M E will denote the submatrix composed of the columns (M j ) j∈E . We will further denote the cardinality of E by |E|.

Finally, for any sample D 0 = {y i , x i } i∈I 0 , with I 0 a given set of integers, and any estimator β of β * , we will denote by

Pred.Error(D 0 , β) = 1 |I 0 | i∈I 0 (y i -x T i β) 2
the prediction error corresponding to β evaluated on the sample D 0 .

The lasso and adaptive lasso

For any λ ≥ 0, the lasso estimator βlasso (λ) [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] is defined as any minimizer over β ∈ R p of the penalized criterion

||y -Xβ|| 2 2 n + λ p j=1 |β j |. (2)
The tuning parameter λ controls the amount of regularization through the

L 1 -norm β 1 = p j=1 |β j |.
In practice, an appropriate value for this parameter is needed to guarantee good statistical performance for βlasso (λ), with respect to both support recovery and prediction accuracy. As mentioned above, a popular strategy relies on K-fold cross-validation [START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF] whose pseudo-code is recalled in Algorithm 3 in Appendix A. Let λ CV be the value of λ selected by K-fold cross-validation, and let βCV lasso = βlasso (λ CV ) denote one particular solution of (2) with λ set to λ CV . Now, denote by w = (w 1 , . . . , w p ) ∈ R p ≥0 a given vector of non-negative weights. For any λ ≥ 0, the adaptive lasso estimator βada (λ; w) is defined [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF] as any minimizer over β ∈ R p of the criterion

||y -Xβ|| 2 2 n + λ p j=1 w j |β j |. (3)
The adaptive lasso reduces to the standard lasso for the particular choice of the weight vector w = 1 p : any solution βlasso (λ) is also a solution βada (λ; 1 p ). In practice, weights are usually set to

w j = 1/| βj |, or w j = 1/(| βj | + ε), with β = ( β1 , .
.., βp ) T an initial estimator of β * , and ε some non-negative real number. A positive value for the ε parameter guarantees that every component of the weight vector is finite, so that every component of βada (λ; w) has a chance to be non-zero. If the initial estimator is good enough, then βj is close to 0 for j / ∈ J, and less so for j ∈ J: if, in addition, ε is null or close enough to 0, weights w j are large for j / ∈ J, and less so for j ∈ J. Then, components j / ∈ J of the parameter vector are more heavily penalized than components j ∈ J. Several initial estimates can be used to derive the weight vector w. When p < n, [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF] suggests the use of w OLS = 1/| βOLS |, where βOLS = (X T X) -1 X T y is the OLS estimator. Under mild conditions, and considering the fixed p case, [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF] established the asymptotic oracular property for this ols-adaptive lasso: specifically, the ols-adaptive lasso is consistent in terms of variable selection (or sparsistent), and the distribution of ( βadap,j (λ; w OLS )) j∈J is Gaussian with the same expectation and covariance matrix than that of (X T J X J ) -1 X T J y, the OLS estimator that would be obtained if J were known in advance. The ols-adaptive lasso inherits these good properties from the √ n-consistency of the OLS estimate βOLS in low-dimensional settings. However, the OLS estimate is less attractive when p > n. In these high-dimensional settings, [START_REF] Zhang | Some two-step procedures for variable selection in high-dimensional linear regression[END_REF] proposed the ridge-adaptive lasso, where ridge regression is used for the computation of the weights:

w ridge = 1/| βCV ridge |. More precisely,
βCV ridge denotes the estimator produced by a ridge regression with tuning parameter selected by K-fold cross-validation, and we recall that the ridge regression is a penalized approach similar to the lasso, but with the L 1 -norm in the penalty replaced by the squared L 2 -norm β 2 2 [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF]. Alternatively, [START_REF] Bühlmann | Discussion: One-step sparse estimates in nonconcave penalized likelihood models[END_REF] suggest the use of weights w 1-step derived from βCV lasso . This leads to what they refer to as the one-step lasso. Thanks to the so-called screening property of βCV lasso , the one-step lasso was shown to be sparsistent under weaker irrepresentability conditions than those required for the lasso [START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF]. Other choices for the weights have been proposed in the literature: for example, [START_REF] Huang | Adaptive lasso for sparse high-dimensional regression models[END_REF] suggested the use of univariate OLS estimators.

2.3 Illustration of the defect of the simple K-fold cross-validation scheme for the calibration of the adaptive lasso

As recalled in the Introduction, the cross-validated prediction error (1/K) × k Pred.Err(D (k) , βk ) is widely used to calibrate penalized approaches, including the adaptive lasso. However, many statistical analysts still seem to implement too simple of a scheme for the calibration of the adaptive lasso [START_REF] Chang | Hdmac: a web-based interactive program for high-dimensional analysis of molecular alterations in cancer[END_REF][START_REF] Courtois | New adaptive lasso approaches for variable selection in automated pharmacovigilance signal detection[END_REF][START_REF] Dessie | A novel mirna-based classification model of risks and stages for clear cell renal cell carcinoma patients[END_REF][START_REF] Pollard | Electrocardiogram machine learning for detection of cardiovascular disease in african americans: the jackson heart study[END_REF]. More precisely, they consider the weight vector w ∈ R p ≥0 derived from initial estimates computed on the full original sample as a "given" (fixed) weight vector. Then, they improperly use the simple crossvalidation scheme to compare the predictive performance of the set of estimators ( βada (λ r ; w)) 1≤r≤R , for any given sequence Λ = (λ 1 , . . . , λ R ) of candidate values for the hyperparameter, and finally select the optimal hyperparameter value, say, λ CV (w), and the corresponding optimal estimator βada (λ CV (w); w). See the pseudo-code given in Algorithm 3 in Appendix A for the detailed description of this simple, and in this case improper, cross-validation scheme.

Below, we present results from a first simulation study, whose main objectives are to illustrate (i) the good performance of the simple scheme when used for the calibration of the standard lasso, and (ii) its poor performance when applied for the calibration of adaptive lasso. Results from a more comprehensive simulation study will be presented in Section 4. In this first synthetic example, we generate one sample D = (y i , x i ) 1≤i≤n , made of n = 1, 000 observations under the linear regression model (1) with p = 1, 000. We first set β * = (β * 1 , . . . , β * p ) T with β * j = 0 for all j ≥ 11, and β * j = ι j 0.5 for all j ≤ 10, where ι j is a {-1, 1}-binary random variable, with P(ι j = 1) = 1/2. Then, for each i = 1, . . . , n, we generate a Gaussian random noise ξ i ∼ N (0, 1), a Gaussian vector of covariates x i = (x i,1 , ..., x i,p ) ∼ N (0 p , I p ), and finally the outcome

y i = x T i β * + ξ i .
Similarly, we generate one independent test sample D = (y i , x i ) n+1≤i≤n+N , made of N = 10, 000 observations drawn under the same linear model. Then, for any given weight vector w, the glmnet R package is used to compute the (adaptive) lasso estimator on an appropriate sequence Λ = Λ(D, w) = (λ 1 (D, w), . . . , λ 100 (D, w)) of 100 decreasing values for the tuning parameter. The glmnet R package is used to compute the 10-fold cross-validated prediction error of βada (λ r (D, w); w), for r = 1, . . . , 100. For comparison, the true prediction error of each estimator is estimated on the independent test sample D.

Given the relatively high-dimensional setting of this first simulation study, the ols-adaptive lasso is not considered here. Figure 1 presents the results for the lasso (w = 1 p ), the one-step lasso (w = w 1-step = 1/| βCV lasso |), and the ridge-adaptive lasso (w = 1/| βCV ridge |). For the latter two, we also report the cross-validated prediction error using a proper nested crossvalidation scheme, which we will describe in more details in the next Section. For comparability, the x-axis corresponds to the tuning parameter sequence represented as a fraction of the data-specific maximal value λ 1 (D, w).

First consider the standard lasso (left panel). In this case, the crossvalidated prediction error using the simple cross-validation scheme does a fairly good job in approximating the "true" prediction error (estimated on the test sample) on a wide range of λ-values. In particular, the λ-value at which the cross-validated prediction error is minimized (vertical dotted red line) is very close to that at which the prediction error on the test sample is minimized (vertical dotted blue line). Moreover, the prediction errors evaluated on the test sample at these two λ-values (horizontal dotted red and blue lines, respectively) are indistinguishable: in this example, the lasso estimate βCV lasso selected by cross-validation is therefore nearly optimal with respect to prediction error. However, the simple cross-validation scheme does not perform that well for the two versions of the adaptive lasso. In particular, when using the simple scheme for the one-step lasso, the cross-validated prediction error constantly decreases as the tuning-parameter decreases, a behavior that we observed on many other simulation designs as well (results not shown). Then, the λ-value at which the cross-validated prediction error is minimized (vertical dotted red line) is very different from the minimizer of the prediction error evaluated on the test sample. In this example, the support of the one-step lasso estimator is too large if the tuning parameter is selected via the simple cross-validation scheme. This will also usually be the case in the more comprehensive simulation study presented in Section 4. Moreover, the prediction errors evaluated on the test sample at these two λ-values (horizontal dotted red and blue lines, respectively) differ substantially: the one-step-lasso estimator calibrated using the simple cross-validation scheme is far from optimal with respect to prediction error in this example. A similar albeit less pronounced behavior is observed in the case of the ridge-adaptive lasso, confirming that the simple K-fold cross-validation scheme is not valid for the calibration of the adaptive lasso. Conversely, these defects disappear when using a proper scheme whose principle will be recalled in details in the next Section [START_REF] Ahrens | lassopack: Model selection and prediction with regularized regression in stata[END_REF][START_REF] He | An improved variable selection procedure for adaptive lasso in high-dimensional survival analysis[END_REF][START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF], which is based on nested cross-validation scheme (Krstajic et al., 2014, Varma and[START_REF] Varma | Bias in error estimation when using crossvalidation for model selection[END_REF]. Using this proper scheme, the λ-value at which the corresponding cross-validated prediction error is minimized (vertical dotted green line) is close to the minimizer of lasso one-step-lasso ridge-ada-lasso Figure 1: Comparison between the prediction error estimated on an independent test sample (in blue) and the prediction error estimated via the simple cross-validation scheme (in red), for the lasso (left), the one-steplasso (middle), and the ridge-adaptive lasso (right). For the latter two, the cross-validated prediction error estimated using a proper nested crossvalidation scheme is also presented (in green). Vertical dotted lines represent the value of the tuning parameter for which each particular curve is minimized (red: cross-validated prediction error using the simple scheme; green: cross-validated prediction error using the nested scheme; blue: prediction error estimated on an independent test sample). Horizontal dotted lines represent the value of the estimated true prediction error for these particular values of the tuning parameter.

the true prediction error estimated on the test sample for both the one-step lasso and ridge-adaptive lasso. Moreover, the prediction errors evaluated on the test sample at these two λ-values (horizontal dotted green line) are very close to the optimal prediction error: in this example, the one-step-lasso and the ridge-adaptive lasso estimators calibrated using the proper nested cross-validation scheme are nearly optimal with respect to prediction error.

Proper cross-validation schemes for the adaptive lasso

In this Section, we provide details on the rationale and principle of proper cross-validation schemes for the adaptive lasso, as previously introduced in

Additional notation

For any sample of observations D 0 = {y i , x i } i∈I 0 , with I 0 a given set of integers, any vector of non-negative weights w, and any non-negative λ value, we let Lasso(D 0 , w, λ) denote one particular solution of the adaptive lasso (3), when computed on sample D 0 with weights w and tuning parameter λ.

For any positive integer K ≥ 2, let sCVLasso(D 0 , w, K) be an adaptive lasso estimator computed on D 0 with weights w and the tuning parameter set to its optimal value according to the simple K-fold cross-validation scheme of Algorithm 3 in Appendix A; this scheme is improper unless the weights w are independent from D 0 . On the other hand, nestedCVLasso(D 0 , w, K) will denote an adaptive lasso estimator computed on D 0 with weights w and the tuning parameter set to its optimal value according to an appropriate K-fold cross-validation scheme. This proper scheme will be described in details below. We use the notation nestedCVLasso to emphasize that this cross-validation scheme relies on two nested cross-validations in the case of the one-step lasso and the ridge-adaptive lasso; however, this is not the case for the ols-adaptive lasso, and our notation "nestedCVLasso" is a bit misleading in the case of the ols-adaptive-lasso. On the other hand, we will use the shorthand sCVRidge(D 0 , K) to denote the ridge estimator computed on D 0 with a tuning parameter set to its optimal value according to the simple K-fold cross-validation, and that will be used to compute the weights in the ridge-adaptive lasso.

Rationale and detailed presentation

We start by recalling the rationale for the defect of the simple K-fold crossvalidation when used for the calibration of the adaptive lasso described and illustrated in Section 2.3. As mentioned in the Introduction, the principle of cross-validation is to mimic "independent" test samples. Then, for the crossvalidation to be valid, at each step k of the K-fold cross-validation, the whole estimation procedure should be performed on the training sample D \ D (k) , and should not use any information from the test sample D (k) . However, the weights used in the adaptive lasso are derived from initial estimates computed on the entire original sample D. Therefore, when using the simple cross-validation scheme of Algorithm 3 in Appendix A for the calibration of the adaptive lasso, and considering the estimation of the adaptive lasso estimator as a whole, data from the test samples are used in the first step of the adaptive lasso, which makes the simple cross-validation scheme not valid in this particular framework.

A proper cross-validation scheme for the calibration of the adaptive lasso estimator is described in Algorithm 1 [START_REF] Ahrens | lassopack: Model selection and prediction with regularized regression in stata[END_REF][START_REF] He | An improved variable selection procedure for adaptive lasso in high-dimensional survival analysis[END_REF][START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF]. The main difference with the simple scheme lies in Step 2. Under the simple scheme, this step consists of Step 2-a, which re-lies on sCVLasso(D, w, K), while it is replaced by Step 2-a' in the proper scheme, which relies on nestedCVLasso(D, w, K), detailed in Algorithm 2 in the particular case of the one-step lasso. The key difference between sCVLasso (see Algorithm 3 in Appendix A) and nestedCVLasso is highlighted in blue in Algorithm 2: at each step k of the proper cross-validation scheme, weights used for the adaptive lasso are first recomputed on the training sample D \ D (k) , so that the whole estimation of the adaptive lasso estimator uses information from the training sample only, before computing the corresponding prediction error on the test sample D (k) . In the case of the one-step lasso (or the ridge-adaptive lasso), this leads to a nested crossvalidation scheme (Krstajic et al., 2014, Varma and[START_REF] Varma | Bias in error estimation when using crossvalidation for model selection[END_REF]. More precisely, an appropriate sequence Λ of candidate λ values for the hyperparameter has first to be chosen, which typically depends on the weights computed on the whole original sample (we briefly get back to this point below). Then, at each step k of the "outer" K-fold cross-validation, one "inner" standard cross-validation is performed to compute the optimal lasso (or ridge) estimator on the training sample D \ D (k) , from which the weights are derived, before the corresponding adaptive lasso estimator is computed for each of the λ values of the sequence, and their predictive performance is eventually evaluated on the test sample D (k) . For each λ value of the sequence, the predictive performance is then averaged over the K folds. The optimal value for the hyperparameter is defined as the value that minimizes this averaged criterion. The optimal adaptive lasso estimator finally corresponds to the adaptive lasso computed on D, with weights w also computed on D, and the hyperparameter set to this optimal value.

A first remark is that, in the case of the ols-adaptive lasso, the proper cross-validation scheme still involves the computation of weights on each training sample in the "outer" loop; however, because weights are derived from OLS estimation, no hyperparameter has to be selected for their computation, hence no inner loop is required. In other words, for the ols-adaptive lasso, the proper cross-validation scheme does not involve nested loops. A second important remark concerns the sequence of candidate values for the hyperparameter. Above, we propose to use the data-driven sequence Λ := Λ(D, w) = (λ 1 , . . . , λ R ) computed from the entire original sample, with weights w also computed on the entire original sample. It could be argued that, by doing so, the estimation process is not completely independent from the test samples. We tested alternative strategies for the selection of the sequence Λ in our simulation, including the data-driven sequence Λ(D, 1 p ) computed from the full original sample with weight vector set to 1 p that is used in the adalasso function of the parcor R package [START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF], but the results were virtually unchanged.

Algorithm 1: Cross-validation for the adaptive lasso. Versions (i), (ii) and (iii) of step (1-a) correspond to the one-step lasso, the ols-adaptive Lasso, and the ridge-adaptive lasso, respectively. The simple (and improper) scheme corresponds to the algorithm ran with Step (2-a), while Step (2-a') (detailed in Algorithm 2) should be used instead.

Data: Sample: D = {y i , x i } n i=1 , Version of the adaptive lasso, Number of folds: K, and parameter ε ≥ 0 Result: βCV ada

Step 1: Computation of the initial estimates and weights;

(1-a) Initial estimates: either (i), (ii) or (iii) below, depending on the considered version of the adaptive lasso ; (i): β = sCVLasso(D, 1 p , K) /* one-step lasso */ (ii): β = OLS(D) /* ols-adaptive lasso */ (iii): β = sCVRidge(D, K) /* ridge-adaptive lasso */ (1-b) Weights: w = 1/(| β| + ε);

Step 2: Computation of the final estimates;

(2-a) βCV ada (w) = sCVLasso(D, w, K); (2-a') βCV ada (w) = nestedCVLasso(D, w, K);

Algorithm 2: Details on the nested K-fold cross-validation scheme (nestedCVLasso) for the calibration of the adaptive lasso, in the particular case of the one-step lasso. Data: Sample: D = {y i , x i } n i=1 , Weights: w, Number of folds: K, and parameter ε ≥ 0 Result:

βnCV ada (w) = βada (λ nCV , w, D) Computation of a sequence Λ := Λ(D, w) = (λ 1 , . . . , λ R ); Division of D into K folds: D = ∪ K k=1 D (k) ; for k ∈ {1, . . . , K} do /* Computation of the weights on D \ D (k) */ β(k) = sCVLasso(D \ D (k) , 1 p , K); w k = 1/(| β(k) | + ε); for r ∈ {1, . . . , R} do βr,k = Lasso(D \ D (k) , w k , λ r ); E r,k = Pred.Error(D (k) , βr,k ); end end r * = argmin r K k=1 E r,k ; λ nCV = λ r * ;
βnCV ada (w) = Lasso(D, w, λ nCV ); q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q p: 100 Signal strength q q q q q q q ols-ada-lasso-CV ols-ada-lasso-nestedCV ridge-ada-lasso-CV ridge-ada-lasso-nestedCV one-step-lasso-CV one-step-lasso-nestedCV lasso-CV over the 100 replications we consider for each combination of values for the parameters (p, β).

Figure 2 presents the results. First, focusing on the adaptive lasso estimators, our results illustrate that the proper cross-validation scheme yields better performance than the simple scheme, in terms of both prediction error and support accuracy. In terms of support recovery, the gain actually only concerns precision (or, equivalently, specificity). This observation confirms what we already observed in Section 2.3: the simple cross-validation scheme generally leads to the selection of too small of hyperparameters, hence poor precision/specificity. Conversely, recall (which is related to sensitivity) is usually better with the simple cross-validation scheme, but the balance is clearly in favour of the proper cross-validation scheme, as it usually yields substantially better overall support recovery. We also observe that, when correctly calibrated, the adaptive lasso usually outperforms the lasso in terms of support recovery (again, the gain comes from a better precision/specificity, as expected), and prediction error. However, the two methods achieve very similar performance for low signal strength.

Finally, Figure 3 presents results from an additional simulation study, using a more realistic scenario, where predictors are highly correlated. Specifically, our synthetic data are generated just as above, but this time with q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q p0: 5 Signal strength q q q one-step-lasso-CV one-step-lasso-nestedCV lasso-CV n = 100 and p = 44, and with the variance-covariance matrix Σ of the predictors set to the empirical variance-covariance matrix of 44 of the transciption factors from the real data presented in Section 5. In this case, we make the number of non-zero parameters in vector β * vary in p 0 ∈ {5, 10, 20}. We compute the one-step lasso, calibrated with either the simple crossvalidation scheme or the nested one, as well as the standard lasso. Results from this additional simulation study are mostly consistent with those of the previous simulation study. They confirm that calibration with the nested cross-validation scheme generally leads to better performance of the one-step lasso, in terms of support recovery (in particular precision/specificity) and prediction error. They also highlight that for low signal strengths, the proper nested scheme does not always lead to improved prediction error, especially for large p 0 , compared to the simple (and improper) scheme. Moreover, for low signal strengths, the standard lasso performs a bit better than the one-step lasso in terms of prediction error in this setting.

Application to single-cell data

In this Section, we use single-cell data to illustrate the performance of the one-step lasso calibrated by 10-fold cross-validation, using either the simple or the nested scheme. For comparison, we also implement the lasso calibrated by the simple cross-validation scheme, which is appropriate in this case. We analyse data describing mielocytic leukemia cells undergoing differentiation to macrophage, described in [START_REF] Kouno | Temporal dynamics and transcriptional control using single-cell gene expression analysis[END_REF]; see also [START_REF] Ollier | Regression modeling on stratified data with the lasso[END_REF]. Briefly, expression levels of 45 transcription factors are measured on n = 120 cells at 8 distinct time-points of this differentiation process. The main objective of [START_REF] Kouno | Temporal dynamics and transcriptional control using single-cell gene expression analysis[END_REF] was to determine how associations among the 45 transcription factors vary over time. The authors focused on marginal associations and used univariate analyses, although graphical models, which describe conditional associations, might be better suited. Identification of the structure of a graphical model can be reduced to the identification and description of the neighbourhood of each covariate [START_REF] Meinshausen | High-dimensional graphs and variable selection with the lasso[END_REF]. For illustration, we here focus on the first timepoint, and consider, successively, each of the 45 transcription factors as the outcome of interest and the other 44 factors as the predictors. This leads to the construction of 45 linear regression models, which together can be used to construct a graphical model. Because it is out of the scope of the present work, we do not discuss, nor present, the estimated graphical models here. For each linear regression model, the prediction performance of each method (one-step lasso with the simple or nested cross-validation scheme and the standard lasso) is evaluated by 5-fold cross-validation repeated 100 times; see Figure 4. For 39 of the 45 considered models, the one-step lasso achieves a lower prediction error when calibrated using the proper nested cross-validation scheme. Moreover, even if the prediction error is generally better for the standard lasso compared to the one-step lasso (which may be explained by the low sample size in this application, and possibly low signal too), the one-step lasso outperforms the lasso for 13 models when using the proper nested cross-validation scheme for its calibration, while it never does when using the improper cross-validation scheme. Comparing the sizes of the supports for the parameter vectors produced by each method, the cumulative size of the supports for the 45 models is 464 for the lasso, 395 for the one-step lasso with improper cross-validation, and only 217 when proper cross-validation is used. Overall, these results confirm that the simple cross-validation scheme is generally not appropriate for the calibration of the one-step lasso, and that the nested cross-validation scheme usually yields sparser parameter estimates and lower prediction error.

A Pseudo-code of the simple K-fold cross-validation scheme for the calibration of the (adaptive) lasso

The pseudo-code detailed in Algorithm 3 below describes the function sCVLasso: when applied to the sample D = {y i , x i } n i=1 , with weights w and number of folds K, it produces the adaptive Lasso estimator βCV ada (w) = βada (λ CV , w), with hyperparameter λ CV set to its value minimizing a K-fold cross-validated prediction error. Keep in mind that this simple scheme is not appropriate if weights w are derived from the data, as is the case for the the ols-adaptive lasso, ridge-adaptive lasso and one-step lasso, estimators.

Algorithm 3:

The simple K-fold cross-validation sCVLasso(D, w, K) for the calibration the lasso.

We recall that this scheme is not valid and should therefore not be used in the case of the adaptive lasso. (k) , βr,k ); end end r * = argmin r K k=1 E r,k ; λ CV = λ r * ; βCV ada (w) = βada (λ CV , w); 22 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q NFE2L1 TCF3 MYEF2

  [START_REF] Krämer | Regularized estimation of large-scale gene association networks using graphical gaussian models[END_REF],[START_REF] He | An improved variable selection procedure for adaptive lasso in high-dimensional survival analysis[END_REF] and[START_REF] Ahrens | lassopack: Model selection and prediction with regularized regression in stata[END_REF].

Figure 2 :

 2 Figure 2: Results of the simulation study. Solid lines represent the averaged criteria, while dashed lines represent the corresponding 95% confidence intervals.

Figure 3 :

 3 Figure 3: Results of the additional simulation study inspired by the single cell data. Solid lines represent the averaged criteria, while dashed lines represent the corresponding 95% confidence intervals.

  Data: Sample: D = {y i , x i } n i=1 , Weights: w, Number of folds: K Result: βCV ada (w) Computation of a sequence Λ := Λ(D, w) = (λ 1 , . . . , λ R ); Division of D into K folds: D = ∪ K k=1 {D (k) }; for k ∈ {1, . . . , K} do for r ∈ {1, . . . , R} do βr,k = Lasso(D \ D (k) , w, λ r ); E r,k = Pred.Error(D

Figure 4 :

 4 Figure 4: Prediction performance of the three methods in the 45 considered linear regression models (each model corresponds to the choice of one particular transcription factor as the outcome of interest), evaluated by 5-fold cross-validation repeated 100 times. Each box-plot represents the empirical distribution of the 5-fold cross-validated prediction error over the 100 repetitions.

Availability of data and material

Not applicable

Code availability

All our empirical results can be replicated using our R scripts available on https://github.com/NadimBLT/AdapGlmnet-NestedCv. In particular the Adalasso.Nestedcv function returns the optimal adaptive lasso estimator, with hyperparameter selected by the "nested" cross-validated scheme, in the case of the ols-adaptive lasso, the ridge-adaptive lasso and the one-step lasso.

Simulation study

We now present results from a more comprehensive simulation study, which extends the simple one presented in Section 2.3. As before, we generate a sample D = (y i , x i ) 1≤i≤n , made of n = 500 observations drawn under linear regression model (1). Here, we make the number of covariates p vary in {100, 500, 1000}, and set the number of relevant covariates to p 0 = 10. We randomly select the support J of β * , with |J| = p 0 . Then, we set β * j = 0 for all j / ∈ J, and β * j = ι j β, for all j ∈ J, where ι j is a {-1, 1}-binary variable, with P(ι j = 1) = 1/2. As for the signal strength β, we make it vary in {1/4, 1/2, 1, 3/2}. Then, for each i = 1, . . . , n, we generate a Gaussian random noise ξ i ∼ N (0, 1), a Gaussian vector of covariates x i = (x i,1 , ..., x i,p ) ∼ N (0 p , Σ), with Σ a symmetric p × p matrix, with element (k, j) equal to σ k,j = 0.3 |k-j| , for 1 ≤ k, j ≤ p. Finally the outcome for observation i is generated as

Additionally, we generate one independent test sample D = (y i , x i ) n+1≤i≤n+N , made of N = 10, 000 observations drawn under the same linear model.

We consider the one-step lasso and the ridge-adaptive lasso as before. The ols-adaptive lasso is also considered in the low-dimensional scenario where p = 100. As in the previous study, weights were derived from initial estimates β as w j = 1/| βj |; other choices for the parameter ε ∈ {0, 10 -6 , 10 -4 , 10 -2 } were tested, and led to very similar results (not shown). For each method, the optimal adaptive lasso estimator is computed following either the simple 10-fold cross-validation scheme (Algorithm 1 with Step 2-a; we will refer to these estimators as the one-step lasso CV, the ridgeadaptive lasso CV, and the ols-adaptive lasso CV respectively) or the 10fold "nested" cross-validation scheme (Algorithm 1 with Step 2-a'; we will refer to these estimators as the one-step lasso nested CV, the ridge-adaptive lasso nested CV, and the ols-adaptive lasso nested CV, respectively; keep in mind that for the ols-adaptive lasso nested CV, the terminology is a bit misleading, as the cross-validation scheme does not involve nested loops). For comparison, results from the lasso estimator calibrated by standard 10fold cross-validation (see Algorithm 3 in Appendix A) are presented; we will refer to this estimator as the lasso CV. We used functions from the glmnet R package to compute these different estimators. We evaluate the (signed) support accuracy, the precision, the recall and the prediction error corresponding to each estimator β. More precisely, the signed support accuracy is defined as sACC = { p j=1 1I(sign(β * j ) = sign( βj ))}/p , where 1I is the indicator function, and sign(•) the sign function, that is sign(x) = +1 if x > 0, sign(x) = -1 if x < 0, and sign(x) = 0 if x = 0. Precision and recall are defined as In this article, we described a defect of the simple K-fold cross-validation scheme when applied for the calibration of the adaptive lasso, with emphasis on the ols-adaptive and ridge-adaptive lasso, as well as the one-step lasso. We further described a refined cross-validation scheme, which is appropriate for the calibration of the adaptive lasso. Although well known by specialists, we believe the defect of the simple scheme has not been stressed enough in the literature, and many statistical analysts still overlook it when calibrating the adaptive lasso. In particular, although the general principle according to which the dataset used to train the model should not contain information from hold-out sample used to evaluate prediction performance is clearly violated when using the simple scheme for the calibration of the adaptive lasso, the specific implications of applying the cross-validation after constructing the weights on the full dataset are not as straightforward to appreciate as in other situations, i.e., when applying cross-validation after screening features on the full dataset.

We shall also stress that, although we focused on the K-fold crossvalidation here, our description applies to other versions of cross-validation [START_REF] Arlot | A survey of cross-validation procedures for model selection[END_REF]. Also, we focused on the adaptive lasso under linear regression models for simplicity, but similar defects are expected for the simple cross-validation scheme in the case of the adaptive generalized fused lasso [START_REF] Viallon | On the robustness of the generalized fused lasso to prior specifications[END_REF], the adaptive data shared lasso [START_REF] Ballout | Sparse estimation for case-control studies with multiple disease subtypes[END_REF][START_REF] Gross | Data shared lasso: A novel tool to discover uplift[END_REF][START_REF] Ollier | Regression modeling on stratified data with the lasso[END_REF]), as well as the adaptive lasso under alternative regression models, such as generalized linear models and Cox proportional hazard models [START_REF] He | An improved variable selection procedure for adaptive lasso in high-dimensional survival analysis[END_REF].

Disclaimer

Where authors are identified as personnel of the International Agency for Research on Cancer/World Health Organization, the authors alone are responsible for the views expressed in this article and they do not necessarily represent the decisions, policy or views of the International Agency for Research on Cancer /World Health Organization.

Declarations Funding

Not applicable

Conflicts of interest/Competing interests

The authors declare no conflict of interest