Fadoua Balabdaoui
email: fadoua.balabdaoui@gmail.com

Cécile Durot
email: cecile.durot@gmail.com

Christopher Fragneau
email: fragneau.christopher@parisnanterre.fr

March

On the population least-squares criterion in the monotone single index model

Keywords: AMS 2000 subject classification: 62G05, 62G08 Least-squares, Monotonicity constraint, Non-parametric estimation, Single index model

Monotone single index models have gained over the past decades increasing popularity due to their flexibility and versatile use in diverse areas. Semi-parametric estimators such as the least squares and maximum likelihood estimators of the unknown index and monotone ridge function were considered to make inference in such models without having to choose some tuning parameter. Description of the asymptotic behavior of those estimators crucially depends on acquiring a good understanding of the optimization problems associated with the corresponding population criteria. In this paper, we give several insights into these criteria by proving existence of minimizers thereof over general classes of parameters. In order to describe these minimizers, we prove different results which give the direction of variation of the population criteria in general and in the special case where the common distribution of the covariates is Gaussian. A complementary simulation study was performed and whose results give support to our main theorems.

Introduction

The single index model can be viewed as an attempt of extending the generalized linear model to a more flexible class of models. It provides a simple trade-off between parametric models and fully non parametric models, assuming that the conditional expectation of a 1 real valued response variable, given a covariate vector, depends only on a linear function of the latter. More specifically, the single index model assumes that

E(Y |X) = Ψ 0 (α T 0 X) (1)
almost surely, where Y ∈ R is the response variable, X ∈ R d is a covariate vector, the index α 0 ∈ R d and the ridge function Ψ 0 : R → R are unknown. As opposed to the generalized linear model, the single index model does not assume a specific parametric form for the ridge function, neither does it assume that the conditional distribution of the response variable given the covariate vector belongs to a given exponential family of laws, thereby allowing for more flexibility. The single index model is widely used in the statistical literature and finds applications in areas as diverse as econometrics, biometrics, medicine or marketing (see e.g. [START_REF] Horowitz | Direct semiparametric estimation of singleindex models with discrete covariates[END_REF]; [START_REF] Hardle | Optimal smoothing in single-index models[END_REF]; [START_REF] Foster | Variable selection in monotone single-index models via the adaptive lasso[END_REF]; [START_REF] Naik | Isotonic single-index model for high-dimensional database marketing[END_REF]). Typical generalized linear models assume that the ridge function belongs to a parametric family of monotone functions since in applications, it is often natural to assume a monotone link between the linear predictor α T 0 X and the response variable Y . Hence, even if the single index model does not assume a specific parametric form for the ridge function, it is natural to assume in this non-parametric model that the ridge function is monotone. We point out that because of identifiability considerations, under the assumption that (1) holds with a monotone ridge function Ψ 0 , it can be assumed without loss of generality that the index α 0 belongs to the (d -1)-dimensional unit sphere S d-1 in R d and that Ψ 0 is nondecreasing, see Section 1 in [START_REF] Balabdaoui | Least squares estimation in the monotone single index model[END_REF]. Hence, this is assumed in what follows. The main advantage of the monotonicity assumption is that it allows for estimators (for the index parameter as well as for the ridge function) that do not depend on any tuning parameter. One can consider for instance the least-squares estimator (LSE) developed in [START_REF] Balabdaoui | Least squares estimation in the monotone single index model[END_REF] and defined as follows. Assume that we observe a sample of n independent pairs (X i , Y i ), i = 1, . . . , n, that are all distributed as (X, Y ) and satisfy (1). The LSE of (α 0 , Ψ 0 ) is defined as the minimizer of the least-squares criterion

(α, Ψ) → M n (α, Ψ) := n -1 n i=1 (Y i -Ψ(α T X i )) 2
(2) over the set S d-1 × M, where M denotes the set of nondecreasing functions. Existence of the LSE, together with a characterization, is given in Theorem 2.3 in [START_REF] Balabdaoui | Least squares estimation in the monotone single index model[END_REF]. In case where the conditional distribution of Y given X is assumed to belong to a given exponential family, one can also compute the maximum likelihood estimator (MLE) as in Balabdaoui et al. (2019a). More specifically, assume that with respect to some dominating measure, the conditional distribution of Y given X = x has a density of the form

y → h(y, φ) exp y (Ψ 0 (α T 0 x)) -B • (Ψ 0 (α T 0 x)) φ ( 3
)
where h is a normalizing function, φ > 0 is a possibly unknown dispersion parameter, is a given real valued function with first derivative > 0, and inverse -1 = B . Then, the MLE of (Ψ 0 , α 0 ) is defined as the minimizer of (α, Ψ) → N n (α, Ψ) :

= -n -1 n i=1 Y i [Ψ(α T X i )] -B • [Ψ(α T X i )] (4) over S d-1 × M.
The importance of the aforementioned LSE and MLE resides in the fact that they prevent from bad choices of a tuning parameter as the latter entails wrong statistical conclusions. Note that both estimators are defined as the minimizer of some appropriate empirical criterion over S d-1 × M. The empirical criterion could also be minimized over a restricted set S × M, where S ⊂ S d-1 , in order to take into account e.g. a possible sparsity or any other potential feature of the index (see Section 4 for more details). As can be seen in the Appendix below (Section A), both empirical criterion M n and N n converge pointwise (as n → ∞) to a limiting criterion which shares the same minimizers as

(α, Ψ) → M(α, Ψ) := E[(Ψ 0 (α T 0 X) -Ψ(α T X)) 2 ]. (5)
We call M the population least-squares criterion since it is related to the original criterion (α, Ψ) → E[(Y -Ψ(α T X)) 2 ] through the identity

E[(Y -Ψ(α T X)) 2 ] = M(α, Ψ) + E[(Y -Ψ 0 (α T 0 X)) 2 ]
which can be obtained using the iterative law of expectation and E[Y |X] = Ψ 0 (α T 0 X). The criterion M in (5) shares the same minimizers as the original criterion and has the advantage of involving the only covariate vector X instead of the pair (Y, X).

Because of the aforementioned pointwise limits, it is expected that the minimizers of either (2) or (4) over S ×M, where S ⊂ S d-1 , converge to the minimizer of M over the same set S × M. The first step in the study of convergence of such estimators is to show that for appropriate subsets S of indexes, the limiting criterion M achieves its minimum over S × M. In this paper, we give several insights into M by proving existence of minimizers thereof over general classes of parameters. In order to describe these minimizers, we prove different results which give the direction of variation of M in general and in the special case where the common distribution of the covariate vector is Gaussian. A complementary simulation study was performed and whose results give support to our main theorems. Note that the difficult question about uniqueness of the minimizer of M, as well as that of the convergence of the minimizers of either (2) or (4) over S ×M is still open. The difficulty in proving uniqueness lies in the fact that it is not straightforward to prove that on sets where M is monotone in a sense that will be make clear in Section 2.3, the monotonicity happens in a strict sense. This point is certainly an important research question that is worth of further investigation. The manuscript will be structured as follows. In the next section, we prove under specified conditions existence of a minimizer of the population criterion M over a given subset of parameters S × M, where S is a subset of the unit sphere, and give a precise description of the minimizer of Ψ → M(α, Ψ) for fixed α. The description takes a very simple form in case where the covariate vector have a multivariate Gaussian distribution. In the same section, two variational results about the profile criterion are proved under the same assumption. In Section 3, we present a simulation study where we validate the main results of the previous section and exhibit cases which contradict them in case the Gaussianity assumption is not satisfied. In Section4, we discuss possible applications and extensions of our results that provide avenues for future research. We consider there the open questions of proving convergence of a sparse index in the monotone single index model, and estimating parameters in a convex single index model. Section 5 is devoted to detailed proofs of all the results presented in Section 2. In Appendix A, we give some complementary results on the estimators LSE and MLE as well as the link between the minimizers of the population criterion associated with the MLE and those of M.

Main results

The minimization problem

Let X be a random vector in R d . Let α 0 ∈ S d-1 and Ψ 0 ∈ M, where S d-1 denotes the unit sphere in R d and M denotes the set of all nondecreasing functions on R. The unit sphere is the set of all points α ∈ R d such that α = 1 where • can be chosen to be any norm on R d . One can consider for instance the unit ball in the Euclidean norm, which is the more commonly used norm in R d . However, as we will see later, another interesting norm is the one associated to the covariance matrix Σ of X: if Σ is positive definite, then the corresponding distance between vectors α and β in R d is the one associated with the scalar product

α, β Σ = cov(α T X, β T X) = α T Σβ, (6)
and the corresponding norm of some vector α is given by the standard deviation of α T X.

Note that in the case where the covariance matrix of X takes the form v 2 I d for some v > 0 where I d is the identity function in R d , this distance coincides with the Euclidean distance times v.

We provide below some insights on the criterion M defined in (5), assuming that Ψ 0 is bounded in sup-norm by some c > 0. We first show that when minimizing M(α, Ψ) over classes S × M, where S ⊂ S d-1 , one can restrict attention to bounded and rightcontinuous functions Ψ. Restricting attention to bounded functions will be useful in the proofs to invoke the dominated convergence Lebesgue theorem while restricting attention to right-continuous functions will be useful to invoke Helly's lemma. In the sequel, we will write M c for the set of all nondecreasing functions on R that are right-continuous and bounded in sup-norm by c.

Theorem 2.1. Let S ⊂ S d-1 . Assume that Ψ 0 is bounded in sup-norm by some c > 0 and that for all α ∈ S, the distribution of α T X is absolutely continuous with respect to Lebesgue measure. Then, minimizing M over a class S × M, amounts to minimizing M over the restricted class S × M c . Moreover, it can be assumed without loss of generality that Ψ 0 is right-continuous. Now that the minimization problem is restricted to sets of parameters with ridge function in M c , we show in Theorem 2.2 below continuity of M on such classes for an appropriate topology, and consequently existence of a minimizer over such classes. For this task, we denote by M 2 the set of all functions Ψ ∈ M such that Ψ 2 2 < ∞ where . 2 denotes the L 2 -norm with respect to the distribution F 0 of α T 0 X, which means that for all Ψ ∈ M,

Ψ 2 2 = R Ψ 2 (t)dF 0 (t) = E[Ψ 2 (α T 0 X)].
We equip S d-1 × M 2 with the following distance D and corresponding topology: for all (α, Ψ) and (β, Φ) in

S d-1 × M 2 , D((α, Ψ), (β, Φ)) = max{ α -β , Ψ -Φ 2 }
where . denotes an arbitrary norm in R d . In what follows, we assume without loss of generality that Ψ 0 is right-continuous (see Theorem 2.1).

Theorem 2.2. Let S ⊂ S d-1 be a closed set. Assume that Ψ 0 ∈ M c . Also, assume that the distribution F 0 of α T 0 X is absolutely continuous with respect to Lebesgue measure, and that there exists A > 0 such that for all α ∈ S, the random variable α T X has a density with respect to F 0 that is bounded by A. Then, M is continuous on S × M c equipped with the distance D; and M achieves its minimum (and maximum) over S × M c .

Below, we consider in Section 2.2 the problem of minimizing Ψ → M(α, Ψ) over M c for a fixed α. Then, in Section 2.3 we plug the obtained minimizer Ψ α into the criterion and next consider the problem of minimizing α → M(α, Ψ α ) over S. The rational for this is that if the minimizer α exists, then the minimizer of M over S × M is given by ( α, Ψ α )

Minimization with respect to the ridge function

We consider the problem of minimizing Ψ → M(α, Ψ) over M c for a fixed α, where Ψ 0 is bounded in sup-norm by c. By Theorem 2.1, this is equivalent to minimizing Ψ → M(α, Ψ) over M. Hence, our problem is the population version of Theorem 2.3 of [START_REF] Balabdaoui | Least squares estimation in the monotone single index model[END_REF], which characterizes the minimizer over M of Ψ → M n (α, Ψ), where M n is the empirical least-squares criterion. According to that theorem, the minimizer of the empirical criterion exists and is given by the left-hand slope of the greatest convex minorant (GCM) of a cumulative sum diagram consisting of a finite set of points (the set contains at most n points, with n being the sample size). We prove below that in our population version of the problem, the minimizer also exists and is given by the left-hand slope of an appropriate continuous diagram. The diagram relies on the function

Ψ α : u → E[Ψ 0 (α T 0 X)|α T X = u], (7)
defined on the support of α T X.

Theorem 2.3. Assume that Ψ 0 is bounded in sup-norm by c > 0. Then, a function Ψ α minimizes Ψ → M(α, Ψ) over M c if and only if almost everywhere, Ψ α • F -1 α is the slope of the greatest convex minorant of the graph u,

u 0 Ψ α • F -1 α (t)dt , u ∈ [0, 1] ,
where F α denotes the distribution of α T X and F -1 α its quantile function.

In some situations, the minimizer Ψ α can be characterized in a simpler way. This is the case for instance if X is a Gaussian vector with invertible covariance matrix. To show this, we write the covariance matrix in the form v 2 Σ where v > 0 is a real number and Σ is an invertible matrix, and we assume without loss of generality that S d-1 is the unit sphere in R d equipped with the scalar product in (6). As can be seen from Lemma 5.1 in Section 5.4 below, in that case, the function Ψ α is monotone and the direction of monotonicity depends on the sign of cov(α T X, α T 0 X). We prove in the following theorem that Ψ α coincides with Ψ α if the latter function is nondecreasing, and is a constant function otherwise.

Theorem 2.4. Assume that Ψ 0 is bounded in sup-norm by c > 0, that X is a Gaussian vector with covariance matrix v 2 Σ where v > 0 is a real number and Σ is an invertible matrix, and S d-1 is the unit ball in R d in the norm α = (α T Σα) 1/2 . Then, Ψ → M(α, Ψ) admits a unique minimizer over M c equal to

Ψ α , F α -a.s. if cov(α T X, α T 0 X) > 0, E[Ψ 0 (α T 0 X)] otherwise,
where F α denotes the (Gaussian) distribution of α T X.

Note that our results hold true in particular if S d-1 is the unit ball defined in the Euclidean norm and X is a Gaussian vector with covariance matrix of the form v 2 I d where I d is the identity matrix in R d . When specialized to that case, Lemma 5.1 in Section 5.4 below ensures that the function Ψ α is nondecreasing if α T α 0 > 0. That gives real meaning to Assumption (A3) in Theorem 6.1 of Balabdaoui et al. (2019c). Indeed, the same function Ψ α is considered in that paper, where Assumption (A3) is that there exists δ 0 > 0 such that Ψ α is monotone increasing on {α T x : x ∈ X } when α belongs to a ball with center α 0 and radius δ 0 . Since α T α 0 > 0 for all α ∈ S d-1 such that α is in a small ball with center α 0 , our Lemma 5.1 shows that Assumption (A3) is satisfied for instance if X is Gaussian with covariance matrix of the form v 2 I d . In that case, not only monotonicity of Ψ α holds for α close enough to α 0 , but it holds on the whole half-space of all α's such that α T α 0 > 0.

Minimization with respect to the index

Now, with Ψ α the minimizer of Ψ → M(α, Ψ) for a fixed α, we are interested in minimizing over a subset S ⊂ S d-1 . Hence, we study below the direction of variation of M. First, we consider the restriction of M to any plane in R d that contains α 0 and the origin. We point out that the intersection between S d-1 and such a plane is a one-dimensional unit sphere, that is a circle, that contains α 0 and -α 0 . In the following theorem, we describe the direction of variation of the restriction of M to such a circle: it is non-increasing when α runs from -α 0 to α 0 along the circle (see Figure 1), whence its minimum and maximum are achieved at α 0 and -α 0 respectively. The theorem is illustrated through simulations in Section 3.

α → M(α) := M(α, Ψ α ) = inf Ψ∈M M(α, Ψ) (8)
Theorem 2.5. Assume that Ψ 0 is bounded in sup-norm by c > 0. Assume moreover that X has a convex support and that there exists q > 0 such that for all α ∈ S d-1 , with respect to Lebesgue measure, the variable α T X has a density that is bounded by q. Then, when α runs from -α 0 to α 0 along a circular arc of a one-dimensional unit sphere, the function α → M(α) is non-increasing.

Figure 1 provides a little illustration of the preceding. Note that with • the norm in which S d-1 is defined, "α runs from -α 0 to α 0 "means that α runs on the circular arc defined by a given plane, in such a way that α -α 0 decreases. Hence, on a given circular arc from -α 0 to α 0 , the closer is α to α 0 , the smaller is the corresponding value of M(α). However, the theorem does not allow to compare M(α) and M(β) in the case where α and β do not belong to the same circular arc between -α 0 and α 0 . In particular, it does not allow to compare M(α) and M(β) in the case where α 0 , α and β do not belong to the same plane. Theorem 2.6 below will allow for such comparisons in the case where X is a Gaussian vector.

Similar to Section 2.2, we now consider the specific case where X is a Gaussian vector with invertible covariance matrix, and we write the covariance matrix in the form v 2 Σ where v > 0 is a real number and Σ is invertible. We assume that S d-1 is the unit sphere in R d equipped with the scalar product in (6).

Theorem 2.6. Assume that Ψ 0 is bounded in sup-norm by c > 0. Assume moreover that X is a centered Gaussian vector with covariance matrix v 2 Σ where v > 0 is a real number and Σ is invertible, and

S d-1 is defined in the norm α = (α T Σα) 1/2 . Then, 1. M is constant on the set of all α's in S d-1 such that cov(α T X, α T 0 X) ≤ 0; 2. M(α) = M(β) for all α and β in S d-1 such that α -α 0 = β -α 0 ;
3. For all closed subsets S ⊂ S d-1 , the function M achieves its minimum at some α ∈ S such that

α -α 0 = min α∈S α -α 0 or equivalently, such that cov( α T X, α T 0 X) = max α∈S cov(α T X, α T 0 X).
Theorem 2.5 is useful for minimizing M over sets of the form S ×M, where S is a closed subset of S d-1 . Indeed, under appropriate assumptions, it follows from Theorems 2.1 and 2.2 that M achieves its minimum over S × M. The minimizer (which is not necessarily unique) takes the form (α, Ψ) where Ψ = Ψ α in the notation of Theorem 2.3. This means that ( α, Ψ) can be (theoretically) computed by first computing Ψ α for all α ∈ S, then computing α that minimizes M over S and finally computing Ψ = Ψ α . When computing α, thanks to Theorem 2.5, one can minimize M over the closed subset of S formed by the α's in S such that there is no point of S on the circular arc between α and α 0 in the plane spanned by α and α 0 . This subset can be viewed as a frontier of S in the direction of α 0 . Obviously, in the particular case where α 0 ∈ S, the minimum of M over S is achieved at α 0 and therefore, ( α, Ψ) = (α 0 , Ψ 0 ).

Some simulations

The aim of this section is to give further numerical illustration for Theorem 2.5. Precisely, we aim at drawing the curve representing α → M(α) when α runs on a circular arc of the one-dimensional unit sphere that passes through -α 0 and α 0 and is centered at the origin of R d . As the function M is not in closed form (see (8) where we recall that M is taken from (5) with X a random variable on R d , α 0 ∈ S d-1 and Ψ 0 ∈ M), this requires some preparatory results: instead of drawing the curve representing M on a circular arc, we will draw the curve representing, on the same circular arc, a process M n that is in closed form, is indexed by some n ∈ N and that converges to M, as n → ∞.

The process M n that we will consider is defined below. Let X 1 , . . . , X n be i.i.d. random variables in R d with the same distribution as X. For a given α ∈ S d-1 , under the assumptions of Theorem 2.5, α T X has a continuous distribution so there exists a unique permutation σ of {1, . . . , n} such that

α T X σ(1) < • • • < α T X σ(n) .
Now, let (μ σ(1) , . . . , μσ(n) ) denote the slopes of the greatest convex minorant of the cumulative sum diagram given by the set of points (0, 0), i,

i j=1 µ 0σ(j) , i = 1, . . . , n , (9)
where µ 0i = Ψ 0 (α T 0 X i ) for all i ∈ {1, . . . , n}. The slopes μσ(i) , i = 1, . . . , n are computed using the well-known Pooled Adjacent Violator algorithm (PAVA), see e.g. [START_REF] Barlow | Statistical inference under order restrictions: The theory and application of isotonic regression[END_REF]. Now, we set

M n (α) = 1 n n i=1 μσ(i) -µ 0σ(i) 2 . ( 10
)
Convergence of M n to M is given in the following theorem.

Theorem 3.1. Assume that Ψ 0 is bounded in sup-norm by c > 0. Assume moreover that X has a convex support and that there exists q > 0 such that for all α ∈ S d-1 , with respect to Lebesgue measure, the variable α T X has a density that is bounded by q. Then, for all α ∈ S d-1 , M n (α) converges to M(α) as n → ∞ with probability one.

Based on Theorem 3.1, for fixed α 0 ∈ S d-1 and Ψ 0 ∈ M, and for a fixed distribution of X, our goal is now to draw the curve representing the restriction of M n to a circular arc that passes through -α 0 and α 0 and is centered at the origin of R d . To avoid unecessary technicalities, we consider below the case d = 2 and α 0 is the vector with coordinates (1, 0) in the canonical basis of R 2 . The unit sphere S 1 is defined here in the Euclidean norm. Instead of restricting ourselves to a circular arc as described above, we consider M n on the whole circle, which is the unit sphere in R 2 . We consider two different distributions for X: either X has a centered Gaussian distribution whose components are independent with the same variance 2, or X has a uniform distribution on [-1, 1] 2 . We also consider three different ridge functions Ψ 0 : either Ψ 0 (x) = π -1 (π/2 + arctan(x)) where π is the Archimedes' constant π ≈ 3.14, or Ψ 0 (x) = x, or Ψ 0 (x) = exp(x). Note that the assumptions of Theorem 2.5 about the distribution of X are all fulfilled with the above choices. On the other hand, only the first choice for Ψ 0 satisfies the boundedness assumption of the theorem. For each choice of Ψ 0 and distribution of X, the following steps were implemented:

1. Choose (large) positive integers n and K; 2. Simulate i.i.d. copies X 1 , . . . , X n of X; 3. For k ∈ {0, . . . , K}, compute θ k := 2kπ/K and

M n (α k ) = 1 n n i=1 μσ(i) -µ 0σ(i) 2
where α k is the vector in S 1 with coordinates (cos(θ k ), sin(θ k )) in the canonical basis of R 2 , and (μ σ(1) , . . . , μσ(n) ) denote the slopes of the greatest convex minorant of the cumulative sum diagram given by the set of points in (9) where σ is the unique permutation of {1, . . . , n} (that depends on k) such that α

T k X σ(1) < • • • < α T k X σ(n) .
Note that with k = 0, we have θ k = 0 and (cos(θ k ), sin(θ k )) = (1, 0) so the first point where M n is evaluated is the true vector α 0 . With k = K, we also have α k = α 0 . Similarly, with k = K/2 (where K is an even integer), we have (cos(θ k ), sin(θ k )) = (-1, 0) so the corresponding α k is -α 0 . Note also that α 0 , . . . , α K form a regular grid of the unit circle S 1 , to which corresponds a regular grid θ 0 , . . . , θ K of the interval [0, 2π].

On Figures 2 and3, for fixed Ψ 0 , n, K and distribution of X, the points (θ k , M n (α k )) are drawn for all k ∈ {0, . . . , K}. The integers n and K are chosen sufficiently large to obtain smooth curves. On the curves, the vertical lines with abscissa π/2 and 3π/2 are also drawn. From what precedes, α 0 is represented twice on the horizontal axis by θ 0 = 0 and θ K = 2π(≈ 6.28). The vector -α 0 is represented by θ K/2 = π(≈ 3.14).

Now, for all α ∈ S 1 there exists a unique θ ∈ [0, 2π] such that the coordinates of α in the canonical basis of R 2 are (cos(θ), sin(θ)); so "α runs from -α 0 to α 0 along a circular arc" means that either the corresponding θ runs from π to 0, or from π to 2π. Hence, it can be seen on Figures 2 and3 that when α runs from -α 0 to α 0 along a circular arc, the function α → M n (α) is non-increasing. This illustrate Theorem 2.5 since M n converges to M, see Theorem 3.1.

The figures also illustrate the first claim in Theorem 2.6. To see this, note that the covariance matrix of X is 2I 2 (where I 2 denotes the identity matrix in R 2 ) if X is a centered Gaussian vector with independent components of the same variance 2, whereas it is

(1/3)I 2 if X is uniformly distributed on [-1, 1] 2 . Hence, for all α k we have cov(α T k X, α T 0 X) = v 2 α T k α 0 = v 2 cos(θ k )
where v 2 = 2 in the Gaussian case and v 2 = 1/3 in the uniform case. This implies that we have cov

(α T k X, α T 0 X) ≤ 0 if and only if θ k ∈ [π/2, 3π/2].
Hence, it follows from Theorem 2.6 that in the Gaussian case we consider here, M takes a constant value on the set of all α k with corresponding θ k in the interval [π/2, 3π/2]. This is illustrated on Figure 2 where M n can be seen to be constant on that set. Figure 3 suggests that the first claim in Theorem 2.6 remains valid if X is not a Gaussian vector but is uniformly distributed on a rectangle. On the other hand, replacing the Gaussian assumption by elliptical symmetry with correlated components does not come with the guarantee that this claim holds true. In fact, if we take X to be distributed according to a 2-dimensional centered t-distribution with 4 degrees of freedom and covariance matrix

Σ 0 = 4 2 2 4 , (11)
α 0 = Σ -1/2 0
(1, 0) T , and the true monotone function is given by Ψ 0 (x) = x 3 , then the graph shown in Figure 4 clearly contradicts that claim. Note that in case the claim were true, M would have been constant for those α such that α T Σ 0 α = 1 and

cov(α T X, α T 0 X) = α T Σ 0 α 0 ≤ 0.
For such an α, note that Σ 1/2 0 α belongs to the unit sphere defined with respect to the Euclidean norm so that there exists θ ∈ [0, 2π) such that Σ 1/2 0 α = (cos(θ), sin(θ)) T . More precisely, since α T Σ 0 α 0 is equal to the scalar product between Σ 1/2 0 α and the vector (1, 0) T , there exists a unique θ ∈ [π/2, 3π/2] such that α = Σ To conclude the section, we point out that the curves in Figures 2 and3 show a different range of M n depending on the true ridge function. Also, it seems that the curves in Figure 2 are smoother than those shown in Figure 3. This means that the function M heavily depends on the underlying distribution of X and Ψ 0 .

Discussion

The main goal in this paper is to investigate the variational properties of the population criteria M associated with the LSE and MLE (which minimize (2) or ( 4)) in a monotone single index model. We have proved that in the model, the population criteria admits minimizers over a given closed subset of the parameter space. The motivation behind this study stems from the desire to establish consistency results about the aforementioned estimators.

We foresee that our work would have an important application in the high-dimensional setting where only a few components of the predictors play a role. In that case, the active set A is defined as the subset of {1, . . . , d} that contains j if and only if the j-th component of X plays a role, and the linear predictor α T 0 X takes the form

α T 0 X = j∈A α 0j X j .
Here, α 0j and X j denote the j-th component of α 0 and X respectively. This means that α 0 has at most #A non-zero components. If the active set were known, one would consider estimators that minimize the criterion in (2) or (4) over S × M, with S being the set of vectors in S d-1 with zero j-th component for all j ∈ A. In practice however, the active set is unknown and α 0 is only assumed to be sparse in the sense that at most s 0 of its components differ from zero, with s 0 a positive integer to be chosen by the practitioner.

In that case, typical estimators are the minimizers of ( 2) or (4) over S × M, with

S := {α ∈ S d-1 : #{i : α i = 0} ≤ s 0 }.
Then, we are in either one of the following situations: (1) the well-specified one when α 0 ∈ S, which means that the number of non-zero components of the true index α 0 is actually smaller than or equal to s 0 , (2) the misspecified one when this number is strictly larger than s 0 , and hence, α 0 ∈ S. In the latter situation, the minimizers of ( 2) or (4) over S × M will not converge to (α 0 , Ψ 0 ) since the corresponding estimators of α 0 belong to the closed set S whereas α 0 does not. Instead, under uniqueness of the minimizer of the population criterion, the estimators will converge to the minimizer of the population criterion over the same sub-space of parameters S × M. We view this particular scenario as one of the most interesting ones in view of the results obtained in this paper. However, uniqueness of the minimizer of M, as well as that of convergence of the minimizers of either ( 2) or (4) over S × M, where S ⊂ S d-1 , is difficult to prove. An even more general situation, and which is of great interest in high dimensional settings, is the case where the dimension d of the covariate vector is allowed to depend on the sample size, a situation that is not covered by our theory. We intend to investigate these problems in the future. We would like to add that the monotonicity constraint represents a very important piece for the variational results established in Section 2 to hold. To see this, consider the case shown in Figure 5 where α 0 = (0, 1) T and Ψ 0 is the non-monotone function Ψ 0 (x) = x 2 . If the claims in Theorems 2.5 and 2.6 were true, then θ → M(α), where α and θ are connected through α = (cos(θ, sin(θ)), should be nondecreasing on [0, π/2], constant on [π/2, 3π/2], and nonincreasing on [3π/2, 2π]. This is clearly not supported by the graph of θ → M n (α) on Figure 5. In fact, in situations where Ψ 0 is not monotone but is known to satisfy another shape constraint, the criterion should not be minimized over sets of the form S ×M. Instead, it should be minimized overs a set of the form S ×C with C being the set of all functions that satisfy the relevant shape constraint. For instance, in the convex single index model where Ψ 0 is known to be convex (as in the counterexample of Figure 5), one has to consider C to be the set of all convex functions. In that case, the profile estimator that minimizes Ψ → M(α, Ψ) over Ψ ∈ C will take a form that is very different from the one we obtain in Section 2.2: instead of being connected to the theory of isotonic regression as is the case in the setting we consider, it will be connected to the theory of convex estimation as in [START_REF] Groeneboom | Estimation of a convex function: characterizations and asymptotic theory[END_REF]. We feel that our results from Section 2.3 could be extended to such cases, but the analysis will certainly involve new arguments. Although this is out of the scope of the present paper, it constitutes a very interesting open problem for future research. 5 Proof of the main results

Proof of Theorem 2.1

Since Ψ 0 is bounded in sup-norm by c, for all Ψ ∈ M and all α ∈ S d-1 we have

E[(Ψ 0 (α T 0 X) -Ψ(α T X)) 2 ] ≥ E[(Ψ 0 (α T 0 X) -Ψ c (α T X)) 2 ] ( 12
)
where Ψ c is the truncated version of Ψ defined by Ψ c (t

) = c if Ψ(t) > c, Ψ c (t) = -c if Ψ(t) < -c
, and Ψ c (t) = Ψ(t) otherwise. Hence, the minimization can be restricted to functions Ψ that are bounded in sup-norm by c. Now, a nondecreasing function Ψ on R has at most a countable set of discontinuity points. By assumption, the probability that either α T X or α T 0 X belong to this set is equal to zero and therefore, M(α, Ψ) is unchanged if Ψ and Ψ 0 are replaced by their right-continuous versions, that is the function that coincides with Ψ (and respectively Ψ 0 ) at all its continuity points and is right-continuous at the discontinuity points. This means that we can assume without loss of generality that Ψ 0 is right-continuous, and the minimization can be restricted to functions Ψ ∈ M c .

Proof of Theorem 2.2

We first establish the continuity of √ M. Fix (α, Ψ) in S × M c . It follows from the Minkowsky inequality that for all (β, Φ) in S × M c , we have

| M(α, Ψ) -M(β, Φ)| = |E[(Ψ 0 (α T 0 X) -Ψ(α T X)) 2 ] 1/2 -E[(Ψ 0 (α T 0 X) -Φ(β T X)) 2 ] 1/2 | ≤ E[(Φ(β T X) -Ψ(α T X)) 2 ] 1/2 ,
so it remains to show that the right-hand term tends to zero as (β, Φ) converges to (α, Ψ). It follows from the triangle inequality that this term is at most

E[(Ψ(β T X) -Φ(β T X)) 2 ] 1/2 + E[(Ψ(α T X) -Ψ(β T X)) 2 ] 1/2 . ( 13
)
The density of β T X with respect to F 0 is supposed to be bounded by A, so for the first term we have

E[(Ψ(β T X) -Φ(β T X)) 2 ] ≤ A Ψ -Φ 2 2 .
Now, we turn to the second term in ( 13). For all > 0, let L > 0 be such that P( X > L ) ≤ . Since Ψ is monotone and |(β -α) T X| ≤ β -α × X by the Cauchy-Schwarz inequality, on the event { X ≤ L } we have :

Ψ(β T X) = Ψ(α T X + (β -α) T X) ≤ Ψ(α T X + β -α L ) and similarly, Ψ(β T X) ≥ Ψ(α T X -β -α L ).
It follows that for all > 0,

E[(Ψ(α T X) -Ψ(β T X)) 2 ] ≤ E[(Ψ(α T X + β -α L ) -Ψ(α T X -β -α L )) 2 1 X ≤L ] +E[(Ψ(α T X) -Ψ(β T X)) 2 1 X >L ] ≤ AE[(Ψ(α T 0 X + β -α L ) -Ψ(α T 0 X -β -α L )) 2 ] + 4c 2 ,
using that the density of α T X with respect to F 0 is bounded by A for the first term, and the assumption that Ψ is bounded by c together with the fact that P( X > L ) ≤ for the second term. Combining the above results yields that for all (β, Φ) in S × M c and for all > 0, we have

| M(α, Ψ) -M(β, Φ)| ≤ √ A Ψ -Φ 2 + √ AE[(Ψ(α T 0 X + β -α L ) -Ψ(α T 0 X -β -α L )) 2 ] 1/2 + 2c √ .
Now, F 0 is assumed to be absolutely continuous with respect to the Lebesgue measure, so the probability that α T 0 X belongs to the set of discontinuity points of Ψ (which is countable by monotonicity), equals zero. On the other hand, for continuity points t,

Ψ(t + β -α L ) -Ψ(t -β -α L )
converges to zero as β -α → 0. Since Ψ is bounded by c, combining the preceding arguments with the dominated Lebesgue Theorem yields for all > 0,

E[(Ψ(α T 0 X + β -α L ) -Ψ(α T 0 X -β -α L )) 2 ] → 0 as β -α → 0.
Hence, as (β, Φ) converges to (α, Ψ), we have

lim sup | M(α, Ψ) -M(β, Φ)| ≤ 2c √ ,
for all > 0. Letting → 0 proves that √ M is continuous at (α, Ψ), where (α, Ψ) is arbitrary. Hence, √ M is continuous on S × M c so that M is continuous on the same set, which completes the proof of the first part of Theorem 2.2.

To prove the second part of the theorem, we first show that M c equipped with the L 2 (F 0 )-distance is a compact set. The proof of Lemma 2.5 in [START_REF] Van Der Vaart | Asymptotic statistics[END_REF] (Helly's Lemma) can be adapted to prove that from every sequence (Ψ n ) n∈N of functions in M c , one can extract a pointwise converging sub-sequence, and the limit Ψ belongs to M c . Note that the convergence may hold only at the continuity points of the limit, but the set of discontinuity points is at most countable, hence has F 0 -measure equal to zero and is negligible in the following argument. Now, it follows from the dominated convergence theorem that Ψ n -Ψ 2 2 converges to zero as n → ∞. Hence, from all sequences in M c , one can extract a sub-sequence that converges in the L 2 (F 0 )-distance sense. By the Bolzano-Weierstrass theorem, this implies that M c equipped with the L 2 (F 0 )-distance is a compact set. In addition, any closed S ⊂ S d-1 is a compact set as being a bounded closed set of R d . By Tychonoff's theorem, it follows that S ×M c is a compact set for the product distance D so that by continuity, M achieves its infimum (and supremum) on that set. This completes the proof of Theorem 2.2.

Proof of Theorem 2.3

With Ψ α taken from (7) we have E[Ψ 0 (α T 0 X)|α T X] = Ψ α (α T X) and therefore, conditionning with respect to α T X yields

E[Ψ 0 (α T 0 X)Ψ(α T X)] = E[E(Ψ 0 (α T 0 X)|α T X)Ψ(α T X)] = E[Ψ α (α T X)Ψ(α T X)], which implies that M(α, Ψ) = E[Ψ 0 (α T 0 X) 2 ] + E[Ψ 2 (α T X)] -2E[Ψ α (α T X)Ψ(α T X)] = E[Ψ 0 (α T 0 X) 2 ] -E[Ψ 2 α (α T X)] + E[(Ψ α (α T X) -Ψ(α T X)) 2 ].
Since the first two terms in the latter line do not depend on Ψ, this in turn implies that Ψ → M(α, Ψ) has the same location of minima over M c as

Ψ → M α (Ψ) := E[(Ψ(α T X) -Ψ α (α T X)) 2 ]. Now, with F -1
α the quantile function of α T X and U a random variable uniformly distributed on [0, 1], the variable α T X has the same distribution as F -1 α (U ) and therefore,

M α (Ψ) = 1 0 Ψ • F -1 α (u) -Ψ α • F -1 α (u) 2 du (14) = 1 0 (g(u) -g α (u)) 2 du where g = -Ψ • F -1 α and g α = -Ψ α • F -1 α
are non-increasing. The sup-norm of g α is smaller than or equal to c, whence g α is square-integrable and we can invoke Lemma 2 in [START_REF] Anevski | Monotone spectral density estimation[END_REF] (which still holds with the Archimedes' constant π ≈ 3.14 replaced by 1). From this lemma, it follows that the infimum of the latter integral over the set of functions g that are integrable and monotone non-increasing, is achieved at the derivative of the least concave majorant of the cumulative function

g α (u) = u 0 g α (t)dt, u ∈ [0, 1].
As the derivative takes the form

t → min u<t max v≥t g α (v) -g α (u) v -u ,
it is bounded in sup-nom by c. Hence, the minimizer belongs to the set -M c of monotone right-continuous nonincreasing functions that are bounded in sup-norm by c, which means that it is also the minimizer of the integral over the latter set. Hence, the minimum of

1 0 (f (u) -Ψ α • F -1 α (u)) 2 du
over f ∈ M c is achieved at f α , the derivative of the greatest convex minorant of f α where the latter is defined in the same manner as g α with g α replaced by f α := -g α . Hence, we conclude that

M α (Ψ) ≥ 1 0 ( f α (u) -Ψ α • F -1 α (u)) 2 du
for all Ψ ∈ M c , with equality if and only if for almost all u ∈ [0, 1], Ψ • F -1 α (u) = f α (u). This completes the proof of Theorem 2.3.

Proof of Theorem 2.4

We begin with a lemma.

Lemma 5.1. Assume that X is a Gaussian vector with an invertible covariance matrix v 2 Σ. Then for all α ∈ R d such that α T Σα = 1, the function Ψ α is nondecreasing if cov(α T X, α T 0 X) > 0, and nonincreasing otherwise.

Proof. First, consider the case where cov(α T X, α T 0 X) = 0. Because X is Gaussian, this implies that α T X and α T 0 X are independent variables so that Ψ α (u) = E[Ψ 0 (α T 0 X)] for all u. Hence, Ψ α is constant (hence nonincreasing) which completes the proof for that case. Now, consider the case where cov(α T X, α T 0 X) = 0. Define X = Σ -1/2 X, α = Σ 1/2 α and α0 = Σ 1/2 α 0 . With I d the identity function in R d , write α T 0 X = U + V where U = αT 0 (I d -α αT ) X and V = αT 0 α αT X. Then, for all u in the support of α T X we have

Ψ α (u) = E[Ψ 0 (α T 0 αu + U )|V = αT 0 αu] (15)
where by assumption, αT 0 α = 0. Since αT α = 1, we have cov(U, V ) = 0 so the Gaussian variables U and V are mutually independent. Therefore,

Ψ α (u) = E[Ψ 0 (α T 0 αu + U )] for all u. If cov(α T X, α T 0 X) > 0, that is αT 0 α > 0, the function u → Ψ 0 (α T 0 αu+U
) is nondecreasing so Ψ α is nondecreasing as well. Similarly, Ψ α is nonincreasing if cov(α T X, α T 0 X) < 0, which completes the proof of the lemma. Now, we turn to the proof of Theorem 2.4. If cov(α T X, α T 0 X) > 0, then Lemma 5.1 ensures that Ψ α is nondecreasing and therefore, the function

u → u 0 Ψ α • F -1 α (t)dt, (16)
where F -1 α is the quantile function of α T X, is convex as being the primitive of a nondecreasing function. Hence, it coincides with its greatest convex minorant and it follows from Theorem 2.3 that almost everywhere, Ψ α • F -1 α is the slope of the function in ( 16), that is Ψ α • F -1 α . Using that with U a random variable uniformly distributed on [0, 1], F -1 α (U ) has the same distribution as α T X, we conclude that

P Ψα (α T X) = Ψ α (α T X) = P Ψα • F -1 α (U ) = Ψ α • F -1 α (U ) = 1,
which completes the proof for the case of a positive covariance.

If the covariance is non-positive, then Ψ α is non-increasing by Lemma 5.1, so that the function in ( 16) is concave on [0, 1]. Its greatest convex minorant is the linear function

u → u 1 0 Ψ α • F -1 α (t)dt = uE[Ψ α (α T X)] = uE[Ψ 0 (α T 0 X)],
using the definition of Ψ α . Theorem 2.4 now follows from Theorem 2.3.

Proof of Theorem 2.5

Define

M n (α) = inf Ψ∈M n -1 n i=1 {Ψ 2 (α T X i ) -2Ψ 0 (α T 0 X i )Ψ(α T X i )}
for all α ∈ S d-1 , where X 1 , . . . , X n are independent random variables with the same distribution as X. For arbitrary α we then have

M n (α) = inf µ σ(1) ≤•••≤µ σ(n) n -1 n i=1 {µ 2 i -2µ 0i µ i } (17)
where for all i, µ 0i = Ψ 0 (α T 0 X i ) and σ denotes the almost surely unique permutation (that depends on α) of {1, . . . , n} such that

α T X σ(1) < • • • < α T X σ(n) . (18)
Almost sure uniqueness of the permutation follows from the fact that α T X 1 , . . . , α T X n are independent with a continuous distribution. We show in Lemma 5.2 below that Theorem 2.5 holds with M replaced by M n . The result for M is then derived using convergence of M n to M.

Lemma 5.2. Consider a one-dimensional unit sphere S 1 that contains α 0 . Let α and β be two points on S 1 such that both α and β belong to the circular arc between -α 0 and α 0 , and α belongs to the portion between β to α 0 . Then,

M n (α) ≤ M n (β). ( 19
)
As a consequence, when α runs from -α 0 to α 0 along a circular arc of a one-dimensional unit sphere, the function α → M n (α) is non-increasing.

Proof. For arbitrary (i, j) ∈ {1, . . . , n} with X i = X j , let H + ij and H - ij denote the open half-space of all vectors α ∈ R d such that α T X i < α T X j , and α T X i > α T X j respectively. Assume first that for all i, j, the vectors α and β belong to the closure H + ij of the same half-space H + ij say (the same reasonning holds for H + ij replaced by H - ij ), and denote by σ a permutation that induces α, i.e. such that (18) holds. Then, σ also induces β and because of ( 17), we conclude that M n (α) = M n (β), which proves (19) for that case. Now, assume that there exists i = j such that α and β do not belong to the closure of the same half-space corresponding to i, j. To fix ideas, suppose that α ∈ H + ij and β ∈ H - ij and note that because α belongs to the circular arc between β to α 0 , we must have α 0 ∈ H + ij . For ease of exposition, suppose moreover that there is a unique such pair (i, j). The case of non unique pairs can be handled by reiterating the argument below. Since the pair (i, j) is assumed to be unique, then with σ that satisfies (18) we must have i = σ(k) and j = σ(k + 1) for some k ∈ {1, . . . , n} and

β T X σ(1) ≤ • • • ≤ β T X σ(k-1) ≤ β T X σ(k+1) ≤ β T X σ(k) ≤ β T X σ(k+2) ≤ • • • ≤ β T X σ(n) .
On the other hand, since α and α 0 belong to the same half-space

H + ij we have α T 0 X σ(k) ≤ α T 0 X σ(k+1) . (20)
Now, let π be the permutation on {1, . . . , n} that induces β, that is

π(i) :=      σ(i) for all i / ∈ {k, k + 1} σ(k + 1) if i = k σ(k) if i = k + 1. ( 21
)
Similar to (17) we have

M n (β) = inf µ π(1) ≤•••≤µ π(n) n -1 n i=1 {µ 2 i -2µ 0i µ i } = inf µ π(1) ≤•••≤µ π(n) n -1 n i=1 (µ i -µ 0i ) 2 -n -1 n i=1 µ 2 0i
where the latter sum does not depend on µ. Hence, it follows from standard results in isotonic regression (see e.g. Theorem 1.1 in [START_REF] Barlow | Statistical inference under order restrictions: The theory and application of isotonic regression[END_REF]) that the infimum is achieved and the minimizer (μ 1 , . . . , μn ) in the latter display is unique. Define

μi =    μσ(k) + μσ(k+1) 2 if i ∈ {σ(k), σ(k + 1)} μi otherwise . ( 22
)
By definition, μπ(1)

≤ • • • ≤ μπ(n) so we have μπ(1) ≤ • • • ≤ μπ(n) , (23)
and thanks to (21) we also have

μσ(1) ≤ • • • ≤ μσ(n) . (24)
Furthermore, using the definition ( 22) of μi we have

n i=1 (μ 2 i -2µ 0i μi ) - n i=1 (μ 2 i -2µ 0i μi ) = μ2 σ(k) -2µ 0σ(k) μσ(k) + μ2 σ(k+1) -2µ 0σ(k+1) μσ(k+1) -μ 2 σ(k) + 2µ 0σ(k) μσ(k) -μ2 σ(k+1) + 2µ 0σ(k+1) μσ(k+1) = (µ 0σ(k) -µ 0σ(k+1) )(μ σ(k) -μσ(k+1) ) +2 μσ(k) + μσ(k+1) 2 2 -μ2 σ(k) -μ2 σ(k+1) = (µ 0σ(k) -µ 0σ(k+1) )(μ σ(k) -μσ(k+1) ) - 1 2 (μ σ(k) -μσ(k+1) ) 2 . ( 25
)
By definition, µ 0i = Ψ 0 (α T 0 X i ) for all i where Ψ 0 is nondecreasing. Hence, (20) implies that µ 0σ(k) ≤ µ 0σ(k+1) . On the other hand, (21) together with the definition of μi implies that μσ(k) -μσ(k+1) = μπ(k+1) -μπ(k) ≥ 0 and therefore, the right-hand side in ( 25) is non positive. This implies that

n i=1 (μ 2 i -2µ 0i μi ) ≤ n i=1 (μ 2 i -2µ 0i μi ) = inf µ π(1) ≤•••≤µ π(n) n i=1 (µ 2 i -2µ 0i µ i ).
Since the minimizer in the right-hand side is unique, we conclude from (23) that μi = μi for all i ∈ {1, . . . , n}. Since we also have (24) we conclude from (17) that

M n (α) ≤ n -1 n i=1 (μ 2 i -2µ 0i μi ) = n -1 n i=1 (μ 2 i -2µ 0i μi ).
Therefore, by possibly enlarging the value of c so that the above inequality holds for all ∈ (0, 1) we obtain

1 0 log N B , F, . 2R d < ∞.
Theorem 19.5 in [START_REF] Van Der Vaart | Asymptotic statistics[END_REF] then ensures that F is a P R -Donsker class and therefore, it is P R -Glivenko-Cantelli. This means that, with similar notations as above, we have sup

f ∈F |P Rn f -P R f | → 0
with probability one as n → ∞, where P Rn is the empirical distribution associated with U 1 , . . . , U n . Now, to make the precise connection between the X i 's and the U i 's, we also consider i.i.d. random variables V 1 , . . . , V n that are distributed as X restricted to the complementary of B R in R d , and i.i.d. random variables T 1 , . . . , T n with Bernoulli distribution with parameter P(X ∈ B R ). All the variables U i , V i , T i for i = 1, . . . , n are mutually independent. Then, we define

X i = U i T i + V i (1 -T i ) for all i ∈ {1, . . . , n}.
It is easy to see that X 1 , . . . , X n are i.i.d. with the same distribution as X.

Because U i = X i if T i = 1, and all functions f ∈ F are bounded in sup-norm by 3c 2 , it follows from the triangle inequality that sup

f ∈F |P Rn f -P R f | -sup f ∈F |P n f -P R f | ≤ sup f ∈F |P Rn f -P n f | ≤ sup f ∈F n -1 n i=1 |f (U i ) -f (V i )|1(T i = 0) ≤ 6c 2 n -1 n i=1 1(T i = 0).
By the strong law of large numbers, the random variable on the right-hand side converges to 6c 2 P( X > R) as n → ∞ with probability one, where • denotes the Euclidean norm in R d . We conclude from the two preceding displays that lim sup

n→∞ sup f ∈F |P n f -P R f | ≤ 6c 2 P( X > R) (26)
with probability one. On the other hand, for all f ∈ F we have

|P R f -P f | = |E[(f (V ) -f (U ))(1 -T )]| ≤ 6c 2 E(1 -T ) = 6c 2 P( X > R).
The latter probability can be made arbitrarily small by choice of R. This implies that with > 0 arbitrary, we can find R = R such that the right-hand side in the previous display is smaller than and therefore, (26) yields lim sup

n→∞ sup f ∈F |P n f -P f | ≤ lim sup n→∞ sup f ∈F |P n f -P R f | + ≤ 2
with probability one. Letting ↓ 0 along a sequence of rational numbers completes the proof of Lemma 5.3.

We are now in position to prove Theorem 2.5. In the sequel we assume that Ψ 0 is bounded in sup-norm by c > 0. It follows from Lemma 5.3 that for any subclass G ⊂ F,

sup f ∈G |P n f -P f | → 0, with probability one, from which it follows that inf f ∈G P n f → inf f ∈G P f, (27)
with probability one as n → ∞. Now, let α and β be two points on S d-1 such that both α and β belong to the same circular arc between -α 0 and α 0 , and α belongs to the portion between β to α 0 . For all Ψ ∈ M, let f αΨ be defined on the support of X by

f αΨ (x) = Ψ 2 (α T x) -2Ψ 0 (α T 0 x)Ψ(α T x).
Then, (27) implies that with probability one as n → ∞,

M n (α) = inf Ψ∈M P n (f αΨ ) → inf Ψ∈M P (f αΨ )
since similar to Theorem 2.1, the above infima are equal to the corresponding infima over the set of functions Ψ ∈ M that are bounded in sup-norm by c. Since

P (f αΨ ) = E[(Ψ(α T X) -Ψ 0 (α T 0 X)) 2 ] -E[(Ψ 0 (α T 0 X)) 2 ]
for all (α, Ψ), we conclude from the definition (8) of M that

M n (α) → inf Ψ∈M M(α, Ψ) -E[(Ψ 0 (α T 0 X)) 2 ] = M(α) -E[(Ψ 0 (α T 0 X)) 2 ] (28)
with probability one. A similar convergence holds with α replaced by β, and Lemma 5.2 ensures that M n (α) ≤ M n (β). Passing to the limit yields that M(α) ≤ M(β), and Theorem 2.5 follows.

Proof of Theorem 2.6

It follows from Theorem 2.4 that for all α ∈ S d-1 such that cov(α T X, α T 0 X) ≤ 0 we have

M(α) = E (Ψ 0 (α T 0 X) -E[Ψ 0 (α T 0 X)]) 2 ,
which proves the first claim in Theorem 2.6. Now, let α ∈ S d-1 . We use the same notation α0 = Σ 1/2 α 0 , α = Σ 1/2 α, X = Σ -1/2 X, I d , U and V as in the proof of Theorem 2.4. Recall that U and V are mutually independent centered Gaussian variables. With a = αT 0 α, we have V ar(V ) = v 2 a 2 and V ar(U ) = v 2 αT 0 (I d -α αT )(I d -α αT ) T α0 = v 2 αT 0 (I d -α αT )α 0 , using that αT α = α 2 = 1. The variance is equal to v 2 (1 -a 2 ), so the distribution of the pair (U, V ) depends only on a. If αT 0 α = 0, it follows from (15) that

M(α) = E (Ψ 0 (α T 0 X) -Ψ α (α T X)) 2 = E (Ψ 0 (U + V ) -E[Ψ 0 (U + V )|V ]) 2 ,
which depends only on a. The second claim follows since α -α 0 2 = 2 -2a. To prove the last claim, note that for all α ∈ S d-1 , the Gaussian variable α T X has variance v 2 and therefore, Theorem 2.5 applies. Now, let α and β in S such that α-α 0 ≤ β -α 0 , or equivalently, α T 0 α ≥ α T 0 β. We can find α and β which belong to a circular arc connecting -α 0 and α 0 , and such that α T α 0 = α T α 0 and β T α 0 = β T α 0 . Hence, M(α ) = M(α) and M(β ) = M(β). Moreover, M is monotone on the closed circular arc so it achieves its minimum and M(α ) ≤ M(β ). Theorem 2.6 follows.

Proof of Theorem 3.1

It follows from standard results in isotonic regression (see e.g. Theorem 1.1 in [START_REF] Barlow | Statistical inference under order restrictions: The theory and application of isotonic regression[END_REF]) that with (μ σ(1) , . . . , μσ(n) ) the slopes of the greatest convex minorant of the cumulative sum diagram given by the set of points in (9), we have

n -1 n i=1 μ2 σ(i) -2µ 0σ(i) μσ(i) = inf µ σ(1) ≤•••≤µ σ(n) n -1 n i=1
{µ 2 σ(i) -2µ 0σ(i) µ σ(i) }.

This is equal to M n (α), see (17). Hence,

M n (α) = M n (α) + E n , (29)
where

E n = 1 n n i=1 µ 2 0i .
Now, for any fixed α ∈ S d-1 , the convergence in ( 28) holds with probability one under the assumptions of Theorem 2.5. Moreover, it follows from the strong law of large numbers that E n converges to E[(Ψ 0 (α T 0 X)) 2 ] as n → ∞, with probability one. Combining the two convergences with (29) yields the result.

A LSE and MLE in the monotone single index model

We show below that the empirical criteria in (2) and (4) converge to a criterion which shares the same minimizers as M in (5).

of that the right-hand term in the last line of the previous display is non positive. Hence,

n i=1 (B • (μ i ) -µ 0i (μ i )) ≤ n i=1 (B • (μ i ) -µ 0i (μ i )) = inf µ π(1) ≤•••≤µ π(n) n i=1 (B • (µ i ) -µ 0i (µ i )).
Since the minimizer in the right-hand side is unique, we conclude from ( 23) that μi = μi for all i ∈ {1, . . . , n}. Since we also have ( 24) we conclude from ( 35) and the definition of μ1 , . . . , μn that N n (α) ≤ N n (β). This completes the proof of (36). The last claim in Lemma A.4 easily follows from (36).

Proof of Lemma A.5 Let c be a real number such that Ψ 0 is bounded in sup-norm by c. Let F be the class of real valued functions defined on the support of X by

x → B • [Ψ(α T x)] -Ψ 0 (α T 0 x) [Ψ(α T x)]
for some α ∈ S d-1 and Ψ ∈ M such that Ψ is bounded in sup-norm by c. Using that B and are both of bounded variations (hence are equal to the difference between two monotone functions), it can be proved with similar arguments as for the proof of Lemma 5.3 that sup f ∈F |P n f -P f | → 0 with probablity one, as n → ∞, using the same notation P n f and P f as in that lemma. Hence, for all subclass G ⊂ F we have sup f ∈G |P n f -P f | → 0, from which it follows that inf

f ∈G P n f → inf f ∈G P f, (38)
with probability one as n → ∞. Let α and β be two points on a same circular arc on S d-1 between -α 0 and α 0 , and such that α belongs to the portion between β to α 0 . For all Ψ ∈ M, we denote by f αΨ the function defined on the support of X by

f αΨ (x) = B • [Ψ(α T x)] -Ψ 0 (α T 0 x) [Ψ(α T x)].
Then, (38) implies that with probability one as n → ∞,

N n (α) = inf Ψ∈M P n (f αΨ ) → inf Ψ∈M P (f αΨ ) = N(α)
where N is taken from (34). Similarly, N n (β) → N(β) with probability one as n → ∞, so that Lemma A.4 yields N(α) ≤ N(β). The first claim in Lemma A.5 follows. The second and third claims can be proved with similar arguments as in the proof of Theorem 2.6.

Figure 1 :

 1 Figure 1: Illustration of Theorem 2.5: the function M is non-increasing when moving along the circular arcs connecting -α 0 and α 0 in the direction of the indicated arrows.

Figure 2 :

 2 Figure 2: The graph connecting the points (θ k , M n (α k )) with n = 10000 and θ k = 2kπ/K for k ∈ {0, . . . , K}, α k = (cos(θ k ), sin(θ k )) and K = 1000, X ∼ N (0, 2I 2 ) and α 0 = (1, 0) T . The link functions are Ψ 0 (x) = π -1 (π/2 + arctan(x)) (left), Ψ 0 (x) = x (middle) and Ψ 0 (x) = exp(x) (right). The vertical lines have abscissa π/2 (dashed lines) and 3π/2 (longdash lines).

Figure 3 :

 3 Figure 3: The graph connecting the points (θ k , M n (α k )) with n = 10000 and θ k = 2kπ/K for k ∈ {0, . . . , K}, α k = (cos(θ k ), sin(θ k )) and K = 1000, X ∼ U([-1, 1] 2 ), and α 0 = (1, 0) T . The link functions are Ψ 0 (x) = π -1 (π/2 + arctan(x)) (left), Ψ 0 (x) = x (middle) and Ψ 0 (x) = exp(x) (right). The vertical lines have abscissa π/2 (dashed lines) and 3π/2 (longdash lines).

Figure 4 :

 4 Figure 4: A counter-example for Theorem 2.6 where the predictor X has a 2-dimensional t-distribution with 4 degrees of freedom and covariance matrix Σ 0 given in (11). The true monotone function and index are Ψ 0 (x) = x 3 and α 0 = Σ -1/2 0 (1, 0) T . The graph connects the points (θ k , M n (α k )) with n = 10000, θ k = kπ/K, α k = Σ -1/2 0 (cos(θ k ), sin(θ k )) for k ∈ {0, . . . , K} and K = 1000. The vertical line has abscissa π/2.

Figure 5 :

 5 Figure 5: A counter-example for Theorems 2.5 and 2.6 in case the link function is not monotone. The predictor X ∼ N (0, I 2 ), the true ridge function and index are Ψ 0 (x) = x 2 and α 0 = (1, 0) T . The graph connects the points (θ k , M n (α k )) with n = 10000, θ k = 2kπ/K, α k = (cos(θ k ), sin(θ k )) for k ∈ {0, . . . , K} and K = 1000. The vertical lines have abscissa π/2 (dashed lines) and 3π/2 (longdash lines).

The latter term is equal to M n (β) by definition of (μ 1 , . . . , μn ), so the proof of ( 19) is complete. The last claim in Lemma 5.2 easily follows from (19).

We continue the proof of Theorem 2.5 with a second lemma. Below, for measurable real valued functions f defined on the support of X, we write P n f for the expectation of f under the empirical measure corresponding to X 1 , . . . , X n , P for the distribution of X, and P f for the expectation under the distribution of X. Thus,

Moreover, we denote by F the class of real valued functions defined on the support of X by

x → Ψ 2 (α T x) -2Ψ 0 (α T 0 x)Ψ(α T x) for some α ∈ S and Ψ ∈ M such that Ψ is bounded in sup-norm by c. We prove in the following lemma that F is a P -Glivenko-Cantelli class.

Lemma 5.3. Assume that Ψ 0 is bounded in sup-norm by c > 0. Assume moreover that X has a convex support and that there exists q > 0 such that for all α ∈ S, with respect to Lebesgue measure, the variable α T X has a density that is bounded by q. Then, we can build X 1 , . . . , X n in such a way that

Proof. For all R > 0, we denote by B R the Euclidean ball with radius R in R d . In order to apply results from [START_REF] Balabdaoui | Least squares estimation in the monotone single index model[END_REF], where it was assumed that the support of X is included in B R for some fixed R > 0, an assumption that we do not make here, we consider below random variables U 1 , . . . , U n which are distributed as X restricted to B R , i.e. with density 1(x ∈ B R )/P(X ∈ B R ) with respect to the distribution of X. We denote by P R the common distribution of the U i 's and by • 2R the corresponding L 2 -norm: for all real valued functions f defined on the support of X,

For all > 0 we denote by N B ( ,

and for each f ∈ F, there exists a j ∈ {1, . . . , N } such that f L j ≤ f ≤ f U j . Similar to Lemma 7.6 in [START_REF] Balabdaoui | Least squares estimation in the monotone single index model[END_REF], it can be proved that for some absolute constant A > 0, and all ∈ (0, c), we have

A.1 Convergence of the empirical criteria

We first consider the least-squares criterion M n and show that it converges pointwise to the process M + C, where C is a constant. This means that M n converges pointwise to a process that shares the same minimizers as M. We denote by M ∞ the set of nondecreasing and bounded functions.

Theorem A.1. Let M n be defined by ( 2) where

2. There exists a real number C such that for any fixed (α, Ψ)

We next consider the maximum likelihood criterion N n . We show that it converges pointwise to a process N which, according to Theorem A.3 below, shares the same minimizers as M if X has a centered Gaussian distribution.

Lemma A.2. Assume that with respect to some dominating measure, the conditional distribution of Y given X = x has a density of the form (3) for some

1. For any K ⊂ S d-1 × M, the infimum of N n over K can only be achieved at some

Theorem A.3. Assume that with respect to some dominating measure, the conditional distribution of Y given X = x has a density of the form (3) for some

Assume moreover that X is a centered Gaussian vector with covariance matrix v 2 Σ where v > 0 is a real number and Σ is an invertible matrix, and S d-1 is defined in the norm α = (α T Σα) 1/2 . With N taken from (31), for any compact subset S ⊂ S d-1 , the minimum of N over S × M is achieved at ( α, Ψ) if and only if the minimum of M over S × M is achieved at the same point ( α, Ψ).

A.2 Proof of Theorem A.1

Let c n = max 1≤i≤n |Y i |. For all Ψ ∈ M and α ∈ S d-1 we have

where Ψ n is the truncated version of Ψ defined by

and hence belongs to M ∞ , which proves the first assertion of the theorem. Now, if E(Y 2 ) < ∞, then the law of large numbers entails that for all (α, Ψ)

Conditionning with respect to X and using Assumption (1) then yields

The second assertion of the theorem follows with

A.3 Proof of Theorem A.3

In the sequel, we use the notation

for all α ∈ S d-1 and

where X 1 , . . . , X n are independent random variables with the same distribution as X. For arbitrary α we then have

where for all i, µ 0i = Ψ 0 (α T 0 X i ) and σ denotes the almost surely unique permutation (that depends on α) of {1, . . . , n} such that (18) holds. Uniqueness of the permutation follows from the fact that α T X 1 , . . . , α T X n are independent with a continuous distribution. Similar to Lemma 5.2, we prove in Section A.4 the following Lemma according to which N n is non-increasing on circles passing through α 0 when moving from -α 0 toward α 0 .

Lemma A.4. Consider a one-dimensional unit sphere S 1 that contains α 0 . Let α and β be two points on S 1 such that both α and β belong to the circular arc between -α 0 and α 0 , and α belongs to the portion between β to α 0 . Then,

As a consequence, when α runs from -α 0 to α 0 along a circular arc of a one-dimensional unit sphere, α → N n (α) is non-increasing.

We continue with a lemma on the asymptotic maximum likelihood criterion, which is reminiscent of Theorem 2.5 and Theorem 2.6. See Section A.4 for a proof.

Lemma A.5. Under the assumptions of Theorem A.3 we have:

1. When α runs from -α 0 to α 0 along a circular arc of a one-dimensional unit sphere, α → N(α) is non-increasing.

2. N(α) = N(β) for all α and β in S d-1 such that α -α 0 = β -α 0 ;

3. For all closed subsets S ⊂ S d-1 , the function N achieves its minimum at α such that

Theorem A.3 easily follows by combining Theorem 2.6 and Lemma A.5.

A.4 Proof of the lemmas

Proof of Lemma A.2 Let Ψ ∈ M, c n = max 1≤i≤n |Y i |, and Ψ n be the truncated version of Ψ defined by

otherwise. It follows from convexity of B and monotonicity of that for indices i such that Ψ(α

where B • is the identity function, whence

A similar argument holds for those i's such that Ψ(α T X i ) < -c n which implies that

), which proves the first assertion of the lemma. Now, it follows from the law of large numbers that for all (α, Ψ) ∈ S d-1 × M ∞ , we have

Combining the two previous displays completes the proof.

Proof of Lemma A.4 The proof is similar to the proof of Lemma 5.2 so details are omitted. With the same notation H + ij and H - ij as in that proof, if α and β belong to the closure of the same half-space H + ij or H - ij for all i, j then N n (α) = N n (β), whence (36). Suppose the exists a unique pair (i, j) such that α ∈ H + ij and β ∈ H - ij . The case of non unique such (i, j) can be handled by repeating the argument above. We use that same notation M n , σ and π as in the proof of Lemma 5.2. It follows from Lemma 2.5.3. in [START_REF] Silvapulle | Constrained statistical inference[END_REF] that the infimum in

is uniquely achieved and the minimizer (μ 1 , . . . , μn ) is also the unique minimizer of

the minimizer is also the unique minimizer in the latter display. Define where we used convexity of B for the last inequality. It follows from ( 20) and ( 21) that µ 0σ(k) ≤ µ 0σ(k+1) and μσ(k) -μσ(k+1) = μπ(k+1) -μπ(k) ≥ 0, so we conclude from monotonicity