Fabienne Comte

Adeline Samson

NONPARAMETRIC ESTIMATION OF RANDOM EFFECTS DENSITIES IN LINEAR MIXED-EFFECTS MODEL

Keywords: Linear mixed-effects model, Nonparametric density estimation, Random effect density MSC 2010 subject classification 62G07

We consider a linear mixed-effects model where Y k,j = α k +β k tj +ε k,j is the observed value for individual k at time tj, k = 1, . . . , N , j = 1, . . . , J. The random effects α k , β k are independent identically distributed random variables with unknown densities fα and f β and are independent of the noise. We develop nonparametric estimators of these two densities, which involve a cutoff parameter. We study their mean integrated square risk and propose cutoff-selection strategies, depending on the noise distribution assumptions. Lastly, in the particular case of fixed interval between times tj, we show that a completely data driven strategy can be implemented without any knowledge on the noise density. Intensive simulation experiments illustrate the method.

Introduction

Longitudinal data and repeated measurements along time of a process are classically analyzed with mixed-effects models. This allows taking into account both the inter-subjects and the intra-subjects variabilities. In this paper, we focus on a simple linear mixed-effects model written as follows. Let Y k,j denote the observed value for individual k at time t j , for k = 1, . . . , N , j = 1, . . . , J. The linear mixed-effects model is defined as (1)

Y k,j = α k + β k t j + ε k,j , k = 1, . . . , N j = 1, . . . , J,

where (α k , β k ) represent the individual random variables of subject k, also called random effects, and (ε k,j ) are the measurement errors. We assume that:

[A1] times (t j ) 1≤j≤J are known and deterministic, [A2] measurement errors ε k,j are independent identically distributed (i.i.d.) with a density f ε , such that E(e iuε ) = 0, for any u ∈ R, [A3] variables (α k , β k ) are i.i.d. and we denote by f α and f β the densities of α 1 and β 1 , [A4] the sequence (α k , β k ) 1≤k≤N is independent of the sequence (ε k,j ) 1≤k≤N,1≤j≤J . When the random effects (α k , β k ) and errors (ε k,j ) are normally distributed, the maximum likelihood has been widely studied, the marginal density of Y having a closed form (1) . MAP5, UMR CNRS 8145, Université Paris Descartes, Sorbonne Paris Cité, 45 rue des Saints Pères, 75006 Paris, France.

1

[see e.g. Pinheiro and Bates, 2000, and references therein]. However, the normality assumption of the random effects may be inappropriate in some situations. For example when an important covariate is omitted, a bimodal density may be more pertinent. As recalled by [START_REF] Ghidey | A comparison of methods for estimating the random effects distribution of a linear mixed model[END_REF], wrongly assuming normality of the random effects can lead to poor estimation results. Some authors propose to relax this assumption by developing estimation of the random effects density first four moments [see e.g. Wu and Zhu, 2010, and references therein]. But estimation of the random effects complete density may even be more appropriate, especially if the true density is multimodal. Our aim is therefore to estimate the densities f α and f β .

Several approaches have been proposed for this purpose. [START_REF] Shen | Empirical bayes estimation via the smoothing by roughening approach[END_REF] consider a smoothing by roughening method without any assumption on f ε . Assuming ε Gaussian, [START_REF] Zhang | Linear mixed models with flexible distributions of random effects for longitudinal data[END_REF], [START_REF] Chen | A monte carlo em algorithm for generalized linear mixed models with flexible random effects distribution[END_REF], [START_REF] David | Mixed model analysis of censored longitudinal data with flexible random-effects density[END_REF] propose a semi nonparametric approach based on the approximation of the random effects density by an Hermite series. [START_REF] Verbeke | A linear mixed-effects model with heterogeneity in the randomeffects population[END_REF] develop an heterogeneity model where the random effects have a finite mixture Gaussian density. [START_REF] Ghidey | Smooth random effects distribution in a linear mixed model[END_REF] propose a penalised Gaussian mixture approach. [START_REF] Morris | Wavelet-based functional mixed models[END_REF] use a wavelet-based approach. Non parametric maximum likelihood has also been studied by [START_REF] Laird | Nonparametric maximum likelihood estimation of a mixing distribution[END_REF], [START_REF] Mallet | Nonparametric maximum likelihood estimation for population pharmacokinetics, with application to cyclosporine[END_REF], [START_REF] Kuhn | Maximum likelihood estimation in non linear inverse problems[END_REF], [START_REF] Chafaï | On nonparametric maximum likelihood for a class of stochastic inverse problems[END_REF]. Recently [START_REF] Antic | Comparison of nonparametric methods in nonlinear mixed effects models[END_REF] compare several of these approaches with an intensive simulation study.

In this paper, we consider a different approach based on deconvolution tools. Deconvolution has been widely studied in various contexts. First, the noise density was systematically assumed known and different estimators have been proposed: kernel estimators (e.g. Stefanski and [START_REF] Stefanski | Deconvoluting kernel density estimators[END_REF], [START_REF] Fan | On the optimal rates of convergence for nonparametric deconvolution problems[END_REF]), kernel estimators with bandwidth selection strategies [START_REF] Delaigle | Bootstrap bandwidth selection in kernel density estimation from a contaminated sample[END_REF]), wavelet estimators [START_REF] Pensky | Adaptive wavelet estimator for nonparametric density deconvolution[END_REF]), or projection methods with model selection [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF]). Then several extensions have been considered to relax the assumption about noise density knowledge. [START_REF] Neumann | On the effect of estimating the error density in nonparametric deconvolution[END_REF] first studied the case where the noise density is estimated from a preliminary noise sample. A complete adaptive procedure has been provided by [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF]. Recently, several papers focus on repeated observations, which provide another way to estimate the noise density, see [START_REF] Neumann | Deconvolution from panel data with unknown error distribution[END_REF], [START_REF] Delaigle | On deconvolution with repeated measurements[END_REF], [START_REF] Meister | Deconvolution from non-standard error densities under replicated measurements[END_REF] or Comte et al. [2011].

We propose now a repeated observations strategy applied to mixed-effects models. We describe different estimators whether the noise density is partly or completely unknown. A preliminary estimation of f ε Fourier transform is used, if needed. We study the risk bounds of these estimators and show how the context of longitudinal data affects the variance terms of the bounds. We also propose cut-off selections depending on the noise distribution assumptions. Especially, a completely data driven estimator is proposed in the case of unknown noise density.

The paper is organized as follows. Sections 2 and 3 present the estimators of f α and f β , their risk bounds and the cut-off selection assuming f ε is known. In section 4, the particular case of a Gaussian noise f ε is considered. We propose an optimal cut-off selection in the two cases σ ε known or unknown. Finally, the general case with f ε completely unknown is considered in section 5 and new estimators and cut-off selections are proposed. Performances of the different estimators are evaluated by simulation in section 6. Concluding remarks are proposed in section 7. Proofs are gathered in Appendix.

2. Definition of the estimators with a known noise density 2.1. Model and notations. In this section, we consider model (1) under Assumptions [A1]-[A4] and the additional assumption:

[A5] the noise density f ε is known. For the estimation of f α , we shall distinguish two cases.

-First case, the observation at time 0 is available. For notation simplicity, we will denote by Y k,0 the observation associated to time t 0 = 0, and we will consider J as the number of other observations with t j = 0. -Second case, no observation is available at time 0, then we have t j = 0 for all j = 1, . . . , J. For the estimation of f β , we do not distinguish the two cases. We shall also consider that the time sequence (t j ) 1≤j≤J is in increasing order.

In the following, for notation simplicity and without loss of generality, we will assume that J is even. Then for j = 1, . . . , J/2, we denote by

∆ j = t 2j -t 2j-1
the time step between two successive observations. Assumptions [A1]-[A4] on model (1) imply that for a given j, (Y k,j ) k=1,...,N are i.i.d. Thus we denote by f Y j the density of Y k,j .

We also need few notations related to Fourier transform theory. If f is an integrable function, then we denote by f * (u) = e iux f (x)dx the Fourier transform of f on R. For two real valued square integrable functions f and g, we denote the convolution product of f and g by (f ⋆ g)(x) = f (xy)g(y)dy and we recall that, if f and g are both integrable and square integrable, then (f ⋆ g) * = f * g * . If f is integrable and square integrable we recall that inverse Fourier transform formula yields f (x) = 1/(2π) e -ixu f * (u)du.

2.2.

Estimator of f β . We first remark that by introducing the difference between two successive observations normalized by the length of the time interval, for j = 1, . . . , J/2,

Z k,j = Y k,2j -Y k,2j-1 ∆ j , we have (2) Z k,j = β k + ε k,2j -ε k,2j-1 ∆ j .
For a fixed j, the variables (Z k,j ) k=1,...,N are i.i.d. but the variables Z k,j and Z k,l for j = l are not independent. Let us denote f Z j the density of the variables Z k,j . It follows from (2) and the independence of (β k ) and

(ε k,j ) under [A4], that (3) f Z j = f β ⋆ f (ε k,2j -ε k,2j-1 )/∆ j .
Thus, by noting that, for all j = 1, . . . , J/2,

f * (ε k,2j -ε k,2j-1 )/∆ j (u) = E e iu ε k,2j -ε k,2j-1 ∆ j = E e i u ∆ j ε k,2j e -i u ∆ j ε k,2j-1 = E e i u ∆ j ε E e -i u ∆ j ε = f * ε ( u ∆ j ) 2 ,
we get, by taking the Fourier transform of equality (3), (4)

f * Z j (u) = f * β (u) |f * ε (u/∆ j )| 2 . It follows from (4) that, for all j = 1, . . . , J/2, f * β (u) = f * Z j (u) |f * ε (u/∆ j )| 2 .
In order to exploit all the available observations, we can also write:

f * β (u) = 2 J J/2 j=1 f * Z j (u) |f * ε (u/∆ j )| 2 .

Now, Fourier inversion implies

(5)

f β (x) = 1 2π e -iux f * β (u)du = 1 2π e -iux 2 J J/2 j=1 f * Z j (u) |f * ε (u/∆ j )| 2 du.
This formula allows us to define the estimator of f β based on the natural estimator of

f * Z j (u) (6) f * Z j (u) = 1 N N k=1 e iuZ k,j = 1 N N k=1 e iu Y k,2j -Y k,2j-1 ∆ j
, Plugging (6) in (5) would give a proposal but may induce convergence problems of the integral. Thus, we introduce a cutoff πm in the integral defining the estimator of f β :

f β,m (x) = 1 2π πm -πm e -ixu 2 J J/2 j=1 f * Z j (u) |f * ε (u/∆ j )| 2 du
where f * Z j (u) is given by (6). To summarize, our proposal to estimate f β is:

(7) f β,m (x) = 2 N J N k=1 J/2 j=1 1 2π πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du
Note that under [A5], the estimator can indeed be computed.

2.3.

Estimator of f α . First, we can notice that, if observations for t 0 = 0 are available, then we have

Y k,0 = α k + ε k,0 , k = 1, . . . , N.
This model is a classical deconvolution model. Thus, we propose to estimate f α with the deconvolution estimator proposed by [START_REF] Fan | On the optimal rates of convergence for nonparametric deconvolution problems[END_REF] with specific kernel as in [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF]:

(8) f 0 α,m (x) = 1 2πN N k=1 πm -πm e -ixu e iuY k,0 f * ε (u) du.
Its theoretical properties have been first studied by [START_REF] Stefanski | Deconvoluting kernel density estimators[END_REF], [START_REF] Fan | On the optimal rates of convergence for nonparametric deconvolution problems[END_REF], and then by [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF] for cutoff selection and in more general context of functional regularities (see also [START_REF] Pensky | Adaptive wavelet estimator for nonparametric density deconvolution[END_REF] for such ideas in wavelet framework). Now, we also provide an estimator when observations at t 0 = 0 are not available. In that case, we follow a construction similar to the one used for f β,m . Set, for j = 1, . . . , J/2,

V k,j = Y k,2j t 2j - Y k,2j-1 t 2j-1 .
By definition of Y k,j , we have

V k,j = 1 t 2j - 1 t 2j-1 α k + ε k,2j t 2j - ε k,2j-1 t 2j-1
Remark that for a fixed j, the variables (V k,j ) k=1,...,N are i.i.d. but the variables V k,j and V k,l for j = l are not independent. Let us denote by f * V j the Fourier transform of the density of the variables V k,j and

p j = 1 t 2j - 1 t 2j-1 .
We have, for all j = 1, . . . , J/2,

(9) f * α (u) = f * V j (u/p j ) f * ε (u/(p j t 2j )) f * ε (-u/(p j t 2j-1 ))
.

A natural estimator of f * α (u) would be to compute the mean of the estimators of (9) for j = 1, . . . , J/2. However, this choice can lead to numerical instability because the quantity 1/(p j t 2j ) involved in the denominator of (9) can be large for large values of j. Indeed, when

∆ j = ∆ is fixed, then t j = j∆, 1/(p j t 2j ) = -(2j -1) and f * ε (u/(p j t 2j )) = f * ε (-(2j -1)u). Since f * ε tends to zero near infinity, f * ε (u/(p j t 2j
)) decreases when j increases. Therefore the estimator of f * α based on (9) may artificially take large values for large j. It shall be noted this is not the case for the estimator of f * β (u) which only involves the step size ∆ j . Thus, for numerical reasons, we propose an estimator of f * α (u) which is only based on the first observation V k,1

f * α (u) = 1 N N k=1 e iV k,1 u/p 1 f * ε (u/(p 1 t 2 )) f * ε (-u/(p 1 t 1 ))
Finally, the estimator of f α is defined by . The function f β,m is the function which is in fact estimated by f β,m . We wish to bound the mean integrated squared error (MISE) defined by E ||f βf β,m || 2 . We first remark that the MISE is the sum of the integrated bias and the integrated variance:

(10) f α,m (x) = 1 2πN N k=1 πm -πm e -iux e iV k,1 u/p 1 f * ε (u/(p 1 t 2 )) f * ε (-u/(p
E f β,m -f β 2 = E f β,m -f β 2 + E f β,m -E f β,m 2
We can easily calculate the expectation E f β,m of the estimator f β,m . We have

E f β,m (x) = 2 N J N k=1 J/2 j=1 1 2π πm -πm e -iux E e iuZ k,j |f * ε (u/∆ j )| 2 du = 2 N J N k=1 J/2 j=1 1 2π πm -πm e -iux f * Z j (u) |f * ε (u/∆ j )| 2 du = 1 2π πm -πm e -iux f * β (u) du = f β,m (x)
where the last line follows from (4). Therefore the pointwise bias is

f β (x) -E f β,m (x) = 1 2π e -iux (f * β (u) -f * β,m (u))du = 1 2π |u|≥πm e -iux f * β (u)du,
and the integrated bias is equal to

f β -f β,m 2 = 1 2π |u|≥πm |f * β (u)| 2 du.
To compute the integrated variance we first write, using Parseval formula,

f β,m -f β,m 2 = 2 J 1 2π J/2 j=1 e -iux f * Z j (u) -f * Z j (u) |f * ε (u/∆ j )| 2 1 [-πm,πm] (u)du 2 = 1 2π πm -πm 2 J J/2 j=1 f Z j (u) -f Z j (u) |f * ε (u/∆ j )| 2 2 du = 1 2π πm -πm 2 N J J/2 j=1 N k=1 e iuZ k,j -E(e iuZ k,j ) |f * ε (u/∆ j )| 2 2 du (11) Thus, we get E f β,m -f β,m 2 = 2 πN J 2 πm -πm Var   J/2 j=1 e iuZ 1,j |f * ε (u/∆ j )| 2   du = 2 πN J 2 πm -πm   J/2 j=1 Var(e iuZ 1,j ) |f * ε (u/∆ j )| 4 + J/2 j,j ′ =1,j =j ′ cov(e iuZ 1,j , e iuZ 1,j ′ ) |f * ε (u/∆ j )| 2 f * ε (u/∆ j ′ ) 2   du ≤ 2 πN J 2 πm -πm   J/2 j=1 1 |f * ε (u/∆ j )| 4 + J/2 j,j ′ =1 (1 -|f * β (u)| 2 )   du ≤ 4 N J 2 J/2 j=1 1 2π πm -πm du |f * ε (u/∆ j )| 4 + m N
Note that when the observation times are equally spaced (∆ j = ∆), this reduces to

E f β,m -f β,m 2 ≤ 1 πN J πm -πm du |f * ε (u/∆)| 4 + m N
This result shows that we reduce the variance of a factor 1/J by taking the mean of all J available values of j in (7).

To summarize, the following result holds.

Proposition 1. Consider Model (1) under Assumptions [A1]-[A5] and f β,m the estimator given by (7). If f β is integrable and square-integrable, then

(12) E f β,m -f β 2 ≤ 1 2π |u|≥πm |f * β (u)| 2 du + 4 N J 2 J/2 j=1 1 2π πm -πm du |f * ε (u/∆ j )| 4 + m N
Inequality (12) requires few comments. First, the term

1 2π |u|≥πm |f * β (u)| 2 du
is a squared bias term due to the truncation of the integral. It decreases when m increases, and the rate of decrease is faster when the function f β is more regular. Indeed, classical regularity spaces considered for density f β on R are described by:

[A6] f β ∈ A b (L) = {f β ∈ L 1 ∩ L 2 , |f * β (x)| 2 (x 2 + 1) b dx ≤ L} with b > 1/2, L > 0. Then, under [A6],
we have the following bias order:

(13) f β -f β,m 2 ≤ CL(πm) -2b .
Obviously, the larger the regularity index b of f β , the faster the bias decreases.

The two other terms of inequality ( 12) are variance terms. They clearly increase when m increases. Moreover, the first of these two terms is dominating:

since |f * (u)| ≤ 1, ∀u ∈ R, we have πm -πm du/|f * ε (u/∆ j )| 4 ≥ 2πm and it is usually much larger. For instance, if the noise is Gaussian, πm -πm du/|f * ε (u/∆ j )| 4 is larger than e 2σ 2 ε (πm/∆ j ) 2 ∆ 2 j /(2σ 2 ε πm).
We have therefore to find how to realize a compromise between the bias and the variance terms. This is the purpose of section 3.3 in which a cutoff selection is proposed. Note that we can also provide pointwise risk bounds for the estimator f β,m ; details about this are gathered in Appendix A. given by ( 10). If f α is integrable and square integrable, then we have

(14) E f α,m -f α 2 ≤ 1 2π |u|≥πm |f * α (u)| 2 du + 1 2πN πm -πm du f * ε u p 1 t 2 f * ε u p 1 t 1 2 + m N .
The same comments as for bound (12) apply here.

Cutoff selection.

We propose the following model selection procedure for choosing a relevant cutoff m for the estimators f β,m and f α,m . We define for ω = α or ω = β mω = arg min

m∈M ω,N -f ω,m 2 + pen ω (m) ,
where for ω = β:

(15) pen β (m) = κ β   4 N J 2 J/2 j=1 1 2π πm -πm du |f * ε (u/∆ j )| 4 + m N   ,
and for ω = α,

(16) pen α (m) = κ α    1 2πN πm -πm du f * ε u p 1 t 2 f * ε u p 1 t 1 2 + m N .    .
Here κ β and κ α are constants which are calibrated once for all on preliminary simulation experiments. Moreover, we set

M ω,N = {m ∈ {1, . . . N }, such that pen ω (m) ≤ 1}.
We can prove the following result.

Theorem 1. Consider Model (1) under Assumptions [A1]-[A5] with f ω integrable and square integrable. Assume that the noise is ordinary smooth, i.e. that there exist two constants c ǫ , C ǫ such that, ∀x ∈ R,

(17) c ǫ (1 + x 2 ) δ ≤ 1/|f * ε (x)| 2 ≤ C ǫ (1 + x 2 ) δ . Then, for ω = α, β, (18) E f ω, mω -f ω 2 ≤ C inf m∈M ω,N f ω -f ω,m 2 + pen ω (m) + C ′ N ,
where C and C ′ are constants depending on the problem. (18) shows that the resulting estimator automatically realizes the squaredbias/variance trade-off, up to a multiplicative constant C. For the sake of simplicity, the result is given only in the ordinary smooth case. It can be obtained in general setting of ordinary or super smooth noise, provided that a factor is added in the penalty for the super smooth case, as detailed in [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF]. In practice, in the case of super smooth noise, we multiply the penalty pen β by the factor

Inequality

(19) log   4 N J 2 J/2 j=1 1 2π πm -πm du |f * ε (u/∆ j )| 4   / log(m + 1)
and the penalty pen α by the factor

(20) log    1 2πN πm -πm du f * ε u p 1 t 2 f * ε u p 1 t 1 2    / log(m + 1).
Indeed, it is shown in [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF] that this is a slight over-penalization which has the advantage of being easy to generalize to the unknown noise case.

Special case of Gaussian noise

In this section, we assume that f ε is Gaussian and centered, and that [A6] holds. We distinguish two cases whether σ 2 ε is known or not.

4.1. Cutoff choice for f β,m when σ 2 ε is known. The Fourier transform of f ε is f * ε (u) = exp(-σ 2 ε u 2 /2). Let ∆ min be a constant such that for all j = 1, . . . , J, ∆ min ≤ ∆ j . Then we get πm -πm du |f * ε (u/∆ j )| 4 ≤ πm -πm exp(2σ 2 ε (u/∆ min ) 2 )du ≤ 2πm exp(2σ 2 ε ∆ -2 min π 2 m 2
) which, associated to [A6] and inequality (12), gives the following result.

Proposition 3. Consider Model (1) under Assumptions [A1]-[A6] with f β integrable and square integrable, and assume that ε is Gaussian and ∆ j ≥ ∆ min , ∀j = 1, . . . , J . Then the choice

(21) m 0,β = m 0,β (σ ε ) = κ ′ β log(N J) 4π 2 σ 2 ε ∆ -2 min 1/2 gives the bound E( f β,m 0,β -f β 2 ) = O(1/[log(N J)] b ) provided that κ ′ β < 1.
The consequence is that the convergence rate of the estimator is logarithmic, which is rather slow. Nevertheless, simulation experiments show that deconvolution estimators behave well also in this setting. It is easy to see that the rate will be much improved if f β is also Gaussian or more generally super-smooth. 4.2. Cutoff choice for f α,m when σ 2 ε is known. Now, we assume that ∆ j = ∆ is fixed. The main variance term of Inequality ( 14) with p 1 t 1 = -1/2 and p 1 t 2 = -1 is of order:

2 πN πm -πm exp(5σ 2 ε u 2 )du ≤ 4m N exp(5π 2 σ 2 ε m 2 ).
Under [A6], the bias order is given by ( 13). We deduce that a good choice of m is

(22) m 0,α = m 0,α (σ ε ) = κ ′ α log(N ) 5π 2 σ 2 ε 1/2 with κ ′ α < 1.
It is worth noticing that here and contrary to m 0,β , the choice m 0,α does not depend on the step ∆.

Cutoff selection when σ 2

ε is unknown. The optimal choices m 0,β and m 0,α provided in the two previous sections depend on σ 2 ε . When this variance is unknown, we propose to replace it by an estimator. A natural estimator can be obtained based on the following relations

V ar(Y k,j ) = σ 2 j = σ 2 α + t 2 j σ 2 β + 2t j σ α,β + σ 2 ε (23) cov(Y k,j , Y k,j ′ ) = σ 2 jj ′ = σ 2 α + t j t j ′ σ 2 β + (t j + t j ′ )σ α,β (24)
which hold for all k, j, j ′ and where σ 2 α , σ 2 β and σ α,β are the variances of α k , β k and the covariance of (α 1 , β 1 ), respectively. Set

Y .j = 1 N N k=1 Y k,j . The following quantities σ2 Y.j = 1 N N k=1 (Y k,j -Y .j ) 2 , and σ2 Y.j,Y.j ′ = 1 N N k=1 (Y k,j -Y .j )(Y k,j ′ -Y .j ′ )
are natural estimators of σ 2 j and σ 2 jj ′ , respectively. From equations ( 23) and ( 24), let us define the 4 × 4 -matrix

A A =       1 3 J J/3 j=1 t 2 3j 3 J J/3 j=1 t 3j 1 1 3 J J/3 j=1 t 2 3j-1 3 J J/3 j=1 t 3j-1 1 1 3 J J/3 j=1 t 2 3j-2 3 J J/3 j=1 t 3j-2 1 1 2 J(J-1) J/2 1≤j<j ′ t j t j ′ 2 J(J-1) J/2 1≤j<j ′ (t j + t j ′ ) 0       ,
and the vector

Σ Y =       3 J J/3 j=1 σ2 Y.3j 3 J J/3 j=1 σ2 Y.3j-1 3 J J/3 j=1 σ2 Y.3j-2 2 J(J-1) J/2 1≤j<j ′ σ2 Y.j,Y.j       .
We assume that A is invertible. In this case, we deduce from equations ( 23)-( 24) estimators of σ 2 = (σ 2 α , σ 2 β , σ α,β , σ 2 ε ), which are defined as

σ 2 = σ2 α , σ2 β , σαβ , σ2 ε ′ = A -1 Σ Y As Σ Y is a M-estimator of parameters ( 3 J J/3 j=1 σ 2 3j , 3 J J/3 j=1 σ 2 3j-1 , 3 J J/3 j=1 σ 2 3j-2 , 2 J(J-1) J/2 1≤j<j ′ σ 2 jj ′ ), if E(|Y k,j | 4 ) < ∞, there exists an explicit matrix I such that √ N   Σ Y -   3 J J/3 j=1 σ 2 3j , 3 J J/3 j=1 σ 2 3j-1 , 3 J J/3 j=1 σ 2 3j-2 , 2 J(J -1) J/2 1≤j<j ′ σ 2 jj ′   ′   L ----→ N →∞ N (0, I)
We deduce that

(25) √ N σ 2 -σ 2 L ----→ N →∞ N (0, A -1 IA ′ -1 )
We assume that there exists a known upper bound for the unknown value of σ ε , denoted by σ ε,max . By plugging σ2 ε in the definition (21) of m 0,β , we obtain a random cutoff m:

(26) m 0,β = m 0,β (σ ε )/ √ 2 ∧ m n = log(N J) 2 √ 2σ ε ∆ -1 min π ∧ m n ,
where

m n = m 0,β (σ ε,max )/ √ 2.
The study of f β, m 0,β is complex in that case. We can prove the following upper bound of the integrated risk.

Proposition 4. Consider Model (1) under Assumptions [A1]-[A6], and assume that ε is Gaussian N (0, σ 2 ε ) with unknown σ ε ≤ σ ε,max . Then the estimator f β, d m 0,β with f β,m defined by ( 7) and m 0,β defined by ( 26) is such that

(27) E( f β, d m 0,β -f β 2 ) ≤ C [log(N J)] -b + 1 N .
It follows from Inequality (27) that the estimator f β, m 0,β automatically reaches its best possible rate, without requiring any information on the unknown function.

A similar proposal can be done for f 0 α,m .

Estimators with unknown noise density

In this section, we consider Model ( 1

* ε (u) = 0 imply that f * ε takes values in R + /{0}, i.e. |f * (u)| = f * (u) > 0, ∀u ∈ R.
Furthermore, we restrict to the following assumption on the observations design:

[A7] ∆ j = ∆ for all j, such that t j = j∆ and J ≥ 6.

5.1. Estimator of f * ε . Under Assumptions [A2'] and [A7], we can propose an estimator of the density f ε . Indeed, let us introduce

W k = Z k,2 -Z k,1 = β k + ε k,4 -ε k,3 ∆ -β k - ε k,2 -ε k,1 ∆ = 1 ∆ (ε k,4 -ε k,3 -ε k,2 + ε k,1 )
which implies that

f * W (u) = E(e iuW k ) = E(cos(uW k )) = |f * ε u ∆ | 4 = f * ε ( u ∆ ) 4 . Under Assumption [A2'], we can estimate (f * ε ) 4 via (f * ε ) 4 (u/∆) = 1 N N k=1 cos(uW k ).
Let us define estimators of f β and f α when f ε is unknown. For numerical reasons, we can not directly plug estimators of 1/(f * ε ) 2 and 1/f * ε in ( 4) and ( 9) by considering 1/( (f * ε ) 4 ) 1/2 and 1/( (f * ε ) 4 ) 1/4 . Therefore, following [START_REF] Neumann | On the effect of estimating the error density in nonparametric deconvolution[END_REF], [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF] or Comte et al. [2011], we define a truncated estimator of

1/(f * ε ) 2 1 (f * ε ) 2 (u) = 1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) 1/2
to be plugged in f β,m and a truncated estimator of

1/f * ε 1 f * ε (u) = 1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) 1/4
to be plugged in f 0 α,m and f α,m . The error induced by the truncation is studied in the following lemma, which is an extension of [START_REF] Neumann | On the effect of estimating the error density in nonparametric deconvolution[END_REF]'s lemma for the case we study here.

Lemma 1. Assume that Assumption [A2'] holds.

(1) There exists a constant C 0 such that

(28) E   1 ( f * ε ) 2 (u) - 1 (f * ε ) 2 (u) 2   ≤ 2 |f * ε (u)| 4 ∧ C 0 N -1/2 |f * ε (u)| 8 ∧ C 0 N -1 |f * ε (u)| 12 .
(2) There exists a constant C 1 such that

(29) E   1 f * ε (u) - 1 f * ε (u) 2   ≤ 1 |f * ε (u)| 2 ∧ C 1 min p∈{1,2,3,4} N -p/4 |f * ε (u)| 2+2p .
5.2. Estimator of f β with unknown f ε . We can now define an estimator for f β by plugging (f * ε ) 2 in (4). Under Assumption [A7], J ≥ 6 and the observations used to estimate f * ε can be different from those used to estimate f β . The estimator of f β when f ε is unknown is denoted by f β,m and defined by ( 30)

f β,m (x) = 2 N (J -4) N k=1 J/2 j=3 1 2π πm -πm e -iu(x-Z k,j ) (f * ε ) 2 (u/∆) du.
Applying Lemma 1 shows that the risk bound in the case of estimated noise density is getting more complicated. We denote for any function f integrable and square integrable

D k (m, f ) = 1 2π πm -πm |f * (u)| 2 |f * ε (u/∆)| 2k du.
Then we obtain the following risk bound

Proposition 5. Consider Model (1) under Assumptions [A1], [A2'], [A3], [A4], [A7].
Assume moreover that f β is integrable and square-integrable, then the estimator defined by (30) satisfies

E( f β,m -f β 2 ) ≤ f β,m -f β 2 + 16 D 2 (m, 1) N (J -4) + 4C 0 D 2 (m, f β ) √ N ∧ D 4 (m, f β ) N + 6 m N ,
where C 0 is defined in Lemma 1.

This risk bound implies the usual bias term f β,m -f β 2 and three terms of variance. The terms 16D 2 (m, 1)/[N (J -4)] and 6m/N correspond to the terms obtained in Inequality (12) of Proposition 1 for known f * ε . The additional term, Comte et al. [2011], in a context of repeated measurements, similar variance terms are also obtained and their simulation experiments show that the first variance term remains the dominating one. We conjecture that the same thing happens here, even if the estimation may be more difficult.

4C 0 (D 2 (m, f β )/ √ N )∧(D 4 (m, f β )/N ) comes from estimating f * ε . If |f * β | 2 decreases faster than |f * ε | 8 , it can happen that D 4 (m, f β ) is bounded by a fixed constant. Then this term is negligible compared to D 2 (m, 1)/[N (J - 4)]. Moreover, in
For the adaptive procedure, we customize the proposals of [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF] to the present case. More precisely, we replace f * ε by its estimate in the penalty and in the definition of the collection of cutoffs. For super smooth noise, an additional multiplicative factor has to be added. Therefore, when the noise density is unknown, this multiplicative factor must be systematically added "blindly". Practically, our estimator is f β, mβ defined by (30) with mβ = arg min

m∈ f M β,N -f β,m 2 + pen β (m) with pen β (m) = κβ log( 4π D2 (m,1) J-4 ) log(m + 1) 4π D2 (m, 1) N (J -4) + m N , D2 (m, 1) = 1 2π πm -πm du (f * ε ) 4 ( u ∆ ) and M β,N = {m ∈ {1, . . . , N }, pen β (m) ≤ 1}.
For ordinary smooth noise, this multiplicative factor behaves roughly like a constant. For super smooth noise, it provides a slight overpenalization in the Gaussian case, as required by the theory, see [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF] and [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF].

5.3.

Estimator for f α with unknown f ε . Analogously, we define for the estimation of f α , either, if observations for j = 0, t 0 = 0 are available

(31) f 0 α,m (x) = 1 2π πm -πm e -iux 1 N N k=1 e iuY k,0 f * ε (u) du,
or when observations at time 0 are not available

(32) f α,m (x) = 1 2π πm -πm e -iux 1 N N k=1 e -2i∆uV k,3 f * ε (-u) f * ε (2u) du,
otherwise. In that second case, observations used for the estimation of f * ε are not used for the estimation of the numerator. Now we detail the risk bound of the estimator f 0 α,m deduced from Lemma 1.

Proposition 6. Consider Model (1) under Assumptions [A1], [A2'], [A3], [A4] and [A7].

Assume moreover that f α is integrable and square-integrable and observations for t 0 = 0 are available, then the estimator defined by ( 31) satisfies (33)

E( f 0 α,m -f α 2 ) ≤ f α,m -f α 2 + 3 πN πm -πm du |f * ε (u)| 2 +4C 1 min p∈{1,...,4} N p/4 2π πm -πm |f * α (u)| 2 |f * ε (u)| 2p du , where C 1 is defined in Lemma 1.
The first two terms of the right-hand-side of inequality (33) are the standard terms, which are also obtained for known f * ε . The last term comes from the substitution of f * ε by its estimate. The main difference with the deconvolution estimators studied in [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF] comes from the fact that we replace f * ε by a truncated estimator based on (f * ε ) 4 , while in [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF], the truncated estimator is based on an estimator of f * ε . We notice that a similar (but still different) phenomenon happens for f β,m where we replace (f * ε ) 2 by a truncated estimator based on (f * ε ) 4 . The risk bound for f β,m is in fact similar to the one found in Comte et al. [2011].

If f α is very smooth, and in particular much smoother than f ε , then the integrals

πm -πm |f * α (u)| 2 /(f * ε (u)) 2p
du may be convergent and the last term negligible. For instance, if α is Gaussian and ε is Laplace i.e. ε = σ ε η/ √ 2 where η has density f η (x) = e -|x| /2, then, for p = 1, 2, 3, 4,

πm -πm |f * α (u)| 2 (f * ε (u)) 2p du ≤ κ p and the last term is less than (2C 1 κ 4 /π)/N . If α is Gaussian with variance σ 2
α and ε is Gaussian with variance σ 2 ε , then the same behavior happens provided that σ 2 α > 4σ 2 ε . Practically, our estimator is f 0 α, mα defined by (31) with mα = arg min

m∈ f M α,N -f 0 α,m 2 + pen 0 α (m) with (34) pen 0 α (m) = κ0 α log(2π Î) log(m + 1) 2π Î N , Î = 1 2π πm -πm du (f * ε ) 2 (u) and M α,N = {m ∈ {1, . . . , N }, pen 0 α (m) ≤ 1}.
A similar study can be performed for the estimator f α,m .

σ ε = 1/10 σ ε = 1/4 σ ε = 1/2 Estimator N =

Simulation study

We consider simulations of Model (1) with noise having either Laplace density (σ ε η with

f η (x) = e - √ 2|x| / √ 2)
or Gaussian N (0, σ 2 ε ) density and t j = j∆, ∆ = 2, J = 6. For both α and β distributions, we experiment four possibilities:

• Gaussian, N (0, 1),

• Mixed Gaussian, with 0.3N (-1, (1/4) 2 ) + 0.7N (1, (1/4) 2 ),

• Gamma distribution, γ(25, 1/25)/5, • Mixed Gamma distribution, [0.3γ(2, 1/2) + 0.7γ(20, 1/5)]/ √ 3. All these densities are calibrated so that their variance is approximately 1. We consider three values of σ ε : 1/10, 1/4, 1/2. This means that the ratios of standard deviations of signal over noise are equal to 10, 4 and 2. In the last case, there is a lot of noise in the model, and the robustness of the procedure is really tested. Moreover, to see the improvement due to sample sizes, we take two values for N , N = 200 and N = 2000.

We compare the performances in term of MISE computed over 100 samples of the estimators

• f β,m as given by ( 7) (known noise) with model selection as described in Section 3.3 with constant κ β of penalty ( 15) equal to 0.5 for Laplace errors and 0.001 for Gaussian errors, • f β,m as given by ( 30) (unknown noise) with model selection described in Section 5.2 and constant κβ equal to 0.5, • f β,mopt given by ( 7) (known Gaussian noise) with m opt = m 0,β (σ ε ) given by ( 21) with κ ′ β = 1. Note that the value of κ ′ β does not fulfill the constraint κ ′ β < 1 but other values seemed too small. We also compare

• f 0 α,m
given by ( 8) (known noise) with model selection as described in Section 3.3 with constant κ 0 α equal to 2, • f α,m given by (10) (known noise) and penalization device with constant κ α equal to 0.5 in the Laplace case and 0.0001 in the Gaussian case, • f 0 α,m given by (31) (unknown noise) with model selection described in Section 5.3 and constant κ0 α equal to 2, • f α,m given by (32) (unknown noise) with constant κα equal to 10, • f α,mopt (known Gaussian noise) with m opt = m 0,α (σ ε ) given by ( 22) with κ ′ α = 0.5. The results are gathered in Tables 1 and2.

Clearly, for the estimation of f α , the estimator f 0 α,m based on the observations Y k,0 at t 0 = 0 has better performances than the estimator f α,m based on the other observations. For simple problems, f α,m performs well but fails to recover the two bumps of the bimodal distributions, unless the noise level is very low (σ ε = 1/10). It is worth mentioning that estimating the noise characteristic function often improves the estimation: this has been already observed in [START_REF] Comte | Data-driven density estimation in the presence of additive noise with unknown distribution[END_REF] and in Comte et al. [2011] recently. This may be due to the truncation of f ε which attenuates the small values of f ε involved in the denominator of the estimators of f β and f α . When the true f ε is used, these small values appearing in the denominator are not truncated.

The estimation of f β is very satisfactory and quite stable even for bimodal densities. Globally, increasing the noise level does not degrade too much the results.

As expected, for both functions, increasing the sample size improves the estimation.

In the Gaussian case, we experiment the specific proposals of Section 4. The results are always really interesting, and rather convincing for both sample sizes 200 and 2000; the main exception corresponds to the case where α is bimodal, where the estimator fails to correctly estimate f α even for small noise. This method can be used for estimating f β and more cautiously f α , if one is convinced that the noise is Gaussian.

Conclusion.

In this paper, we consider a linear mixed-effects model with random i.i.d. coefficients α k and β k and we study how to estimate their unknown distributions. We propose several solutions, depending on the available information about noise density. Since it is often assumed to be Gaussian, we show that specific strategies can be developed in this case. In the more realistic case where it is unknown, we also propose general solutions based on deconvolution strategies. All this material is tested on simulation experiments which show the relevance of the methods. These proposals are all new and very different from existing strategies for such models.

Several extensions may be considered in future works. First, we may wish to estimate the joint distribution of α and β, with known or unknown noise density. Secondly, remaining in a linear setting, we may add fixed regressors with constant coefficients to be estimated. Lastly, finding out if such strategies may be successful for more general nonlinear mixedeffects model remains an open question.

Appendix A. Pointwise risk bound of f β

The variance of the estimator is

Var f β,m (x) = 4 N 2 J 2 1 4π 2 N k=1 Var   J/2 j=1 πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du   = 4 N J 2 1 4π 2 J/2 j=1 Var πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du + 4 N J 2 1 4π 2 J/2 j,j ′ =1,j =j ′ cov πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du, πm -πm e -ixz e izZ k,j ′ f * ε (z/∆ j ′ ) 2 dz .
For the first part of the decomposition, we have

Var πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du ≤ E   πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du 2   ≤ E πm -πm πm -πm e -ix(u-z) e i(u-z)Z k,j |f * ε (u/∆ j )| 2 |f * ε (z/∆ j )| 2 du dz ≤ πm -πm πm -πm e -ix(u-z) f * Z j (u -z) |f * ε (u/∆ j )| 2 |f * ε (z/∆ j )| 2 du dz ≤ πm -πm 1 |f * ε (u/∆ j )| 4 du |f * Z j (w)|dw σ ε = 1/10 σ ε = 1/4 σ ε = 1/2 Estimator N =
f α are f 0 α,m , f 0 α,m , f α,m and f α,mopt . Esti- mators of f β are f β,m , f β,m and f β,mopt .
by applying bidimensional Cauchy Schwarz inequality with respect to the measure |f * Z j (uz)|dudz and by using then Fubini. Thus

Var πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du ≤ πm -πm 1 |f * ε (u/∆ j )| 4 du |f * ε (w/∆ j )| 2 dw ≤ f ε 2 ∆ j πm -πm 1 |f * ε (u/∆ j )| 4 du
For the second part of the variance decomposition, we have

|cov πm -πm e -ixu e iuZ k,j |f * ε (u/∆ j )| 2 du, πm -πm e -ixz e izZ k,j ′ f * ε (z/∆ j ′ ) 2 dz | = | [-πm,πm] 2 e -ix(u-z) (f * β (u -z) -f * β (u)f * β (-z))dudz| ≤ 4π 2 m |f * β (w)|dw
By gathering the terms, we obtain

V ar f β,m (x) ≤ 1 π 2 N J 2 J/2 j=1 f ε 2 ∆ j πm -πm 1 |f * ε (u/∆ j )| 4 du + C 4 π m |f * β (w)|dw N .
When the observation times are equally spaced with ∆ j = ∆, we obtain the following bound

V ar f β,m (x) ≤ 2 f ε 2 ∆ πm -πm 1 |f * ε (u/∆ j )| 4 du N J + 4 f * β 1 π m N .
Thus the MSE is bounded by Proposition 7. Consider Model (1) under Assumptions [A1]-[A5] with f β integrable and square integrable, and

f ε ∈ L 2 (R) then, E f β,m (x) -f β (x) 2 ≤ ( 1 2π |u|≥πm |f * β (u)|du) 2 + 2 f ε 2 π 1 N J 2 J/2 j=1 ∆ j πm -πm 1 |f * ε (u/∆ j )| 4 du + 4 f * β 1 π m N . (35) Appendix B. Proofs B.
1. Proof of Proposition 2. We can easily calculate the expectation of the estimator f α,m (x) defined by (10). We have

E f α,m (x) = 1 N N k=1 1 2π πm -πm e -iux E e iV k,1 u/p 1 f * ε u p 1 t 2 f * ε -u p 1 t 1 du = 1 N N k=1 1 2π πm -πm e -iux f * V k,1 (u/p 1 ) f * ε u p 1 t 2 f * ε -u p 1 t 1 du = 1 2π πm -πm e -iux f * α (u) du = f α,m (x)
The integrated bias is therefore given by:

f α -E f α,m 2 ≤ 1 2π |u|≥πm |f * α (u)| 2 du
The integrated variance is

f α,m -f α,m 2 = 1 2π πm -πm e -iux f * V k,1 (u/p 1 ) -f * V k,1 (u/p 1 ) f * ε u p 1 t 2 f * ε -u p 1 t 1 du 2 = 1 2π πm -πm f * V k,1 (u/p 1 ) -f * V k,1 (u/p 1 ) f * ε u p 1 t 2 f * ε u p 1 t 1 2 du
Therefore, as for ( 12),

E f α,m -f α,m 2 ≤ 1 2πN πm -πm 1 f * ε u p 1 t 2 f * ε u p 1 t 1 2 du + m N .
Gathering bias and variance bounds gives the result of Proposition 2.

B.2. Proof of Theorem 1. Let S m = {t ∈ L 1 ∩ L 2 , such that t * = t * 1 [-πm
,πm] } be the subspace of L 2 with functions having Fourier Transforms supported by [-πm, πm]. Now we can notice that f β,m is the minimizer over S m of the contrast

γ N,J (t) = t 2 - 4 N J N k=1 J/2 j=1 1 2π t * (-u) e iuZ k,j |f * ε (u/∆ j )| 2 du since γ N,J (t) = t 2 -2 t, f β,m . Then defining ν N,J (t) = 2 N J N k=1 J/2 j=1 1 2π t * (-u) (e iuZ k,j -E(e iuZ k,j )) |f * ε (u/∆ j )| 2 du
we have the decomposition, for all functions s, t integrable and square integrable,

(36) γ N,J (t) -γ N,J (s) = t -f β 2 -s -f β 2 -2ν N,J (t -s).
By definition of mβ , we have that ∀m ∈ M β,N ,

γ n ( f β, mβ ) + pen β ( mβ ) ≤ γ n (f β,m ) + pen β (m)
and with (36), this yields, ∀m ∈ M β,n ,

f β, mβ -f β 2 ≤ f β -f β,m 2 + pen β (m) + 2ν N,J ( f β, mβ -f β,m ) -pen β ( mβ ) ≤ f β -f β,m 2 + pen β (m) + 2 f β, mβ -f β,m sup t∈Sm+S mβ ν N,J (t) -pen β ( mβ ) ≤ f β -f β,m 2 + pen β (m) + 1 4 f β, mβ -f β,m 2 +4 sup t∈Sm+S mβ ν 2 N,J (t) -pen β ( mβ ) ≤ f β -f β,m 2 + 2pen β (m) + 1 2 f β, mβ -f β 2 + 1 2 f β -f β,m 2 +4( sup t∈Sm+S mβ ν 2 N,J (t) -p( mβ , m))
where 4p(m, m ′ ) ≤ pen β (m) + pen β (m ′ ). We can prove Lemma 2. Under the assumptions of Theorem 1,

E sup t∈Sm+S mβ ν 2 N,J (t) -p( mβ , m) ≤ C ′ N
Applying Lemma 2, we get ∀m ∈ M β,N ,

E( f β, mβ -f β 2 ) ≤ 3 f β -f β,m 2 + 4pen β (m) + C ′ N .
This is the result of Theorem 1.

Proof of Lemma 2. Let us define ϕ(x) = sin(πx)/(πx) and ϕ m,ℓ (x) = √ mϕ(mxℓ) for ℓ ∈ Z. We recall that (ϕ m,ℓ ) ℓ∈Z is an orthonormal basis of S m and it fulfills ℓ∈Z ϕ 2 ℓ ≤ m. Let us denote by

η k,j = ε k,2j -ε k,2j-1 ∆ j .
Then the (η k,j ) k,j are independent. We split ν N,J in two parts and write ν N,J (t) = ν

(1)

N,J (t)+ ν (2) N,J (t): ν (1) N,J (t) = 2 N J N k=1 J/2 j=1 1 2π t * (-u) e iuβ k (e iuη k,j -E(e iuη k,j )) |f * ε (u/∆ j )| 2 du and ν (2) N,J (t) = 1 N N k=1 1 2π t * (-u)(e iuβ k -f * β (u))du = 1 N N k=1 (t(β k ) -t, f β ).
For ν

(2) N,J (t), we find the bounds

sup t∈Sm+S m ′ , t =1 [ν (2) N,J (t)] 2 ≤ sup t∈Sm+S m ′ , t =1 ℓ∈Z a 2 m∨m ′ ,ℓ ℓ∈Z [ν (2) N,J (ϕ m∨m ′ ,ℓ )] 2 ≤ ℓ∈Z [ν (2) N,J (ϕ m∨m ′ ,ℓ )] 2
where m ∨ m ′ = sup(m, m ′ ), and

E sup t∈Sm+S m ′ , t =1 [ν (2) N,J (t)] 2 ≤ ℓ 1 N Var(ϕ m∨m ′ ,ℓ ) ≤ m ∨ m ′ N := H 2 2 .
Moreover [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF]) yield

Var(t(β 1 )) ≤ E(t 2 (β 1 )) ≤ t ∞ E(|t(β 1 )|) ≤ t ∞ t f β and for any t ∈ S m∨m ′ , t ∞ ≤ √ m ∨ m ′ t (see
sup t∈Sm+S m ′ , t =1 Var(t(β 1 )) ≤ √ m ∨ m ′ f β := v 2 and sup t∈Sm+S m ′ , t =1 t ∞ ≤ √ m ∨ m ′ := b 2
Thus Talagrand Inequality implies

E sup t∈Sm+S mβ [ν (2) N,j (t)] 2 -4 m ∨ mβ N ≤ m ′ ∈M β,N E sup t∈Sm+S m ′ [ν (2) N,j (t)] 2 -4 m ∨ m ′ N ≤ m ′ ∈M β,N K 1 ( v 2 N e -K 2 N H 2 2 v 2 + b 2 2 N 2 e -K 3 N H 2 b 2 ) ≤ K 1 N m ′ ∈M β,N ( f β √ m ∨ m ′ e -K 2 √ m∨m ′ / f β + m ∨ m ′ N e -K 3 √ N ) ≤ C ′ N since the sums are convergent or bounded. For ν (1)
N,J we compute the same bounds conditionally to (β k ) 1≤k≤N . Then we proceed analogously to [START_REF] Comte | Adaptive density deconvolution with dependent inputs[END_REF] and [START_REF] Comte | Penalized contrast estimator for adaptive density deconvolution[END_REF], and we get the result. B.3. Proof of Proposition 3. The bias order given in (13) under Assumption [A6], is O(m -2b ) which gives the announced order when choosing m = m 0,β . Moreover, the variance terms are made negligible by this choice. Indeed the integrated variance of f β,m is of order

(πm) -1 2N J exp(2σ 2 ε ∆ -2 min π 2 m 2 ) +
m N which leads to a variance of order

σ ε ∆ -1 min (N J) 1-κ ′ β log(N J) + κ ′ β log(N J) N 2πσ ε ∆ -1 min
and the convergence rate of order (log(N J)) -b . B.4. Proof of Proposition 4. The proof is based on the following decomposition

(37) E( f β, m 0,β -f β 2 ) ≤ E( f β, m 0,β -f β 2 1 | c σ 2 ε -σ 2 ε |≤σ 2 ε /2 )+E( f β, m 0,β -f β 2 1 | c σ 2 ε -σ 2 ε |>σ 2 ε /2 ) First, remark that when | σ 2 ε -σ 2 ε | ≤ σ 2 ε /2, we have 1 2 σ 2 ε ≤ σ 2 ε ≤ 3 2 σ 2 ε . Consequently 1 √ 3 m 0,β ≤ m 0,β ( σ ε )/ √ 2 ≤ m 0,β with m 0,β = m 0,β (σ ε ).
Thus, looking at (11), we get

f β, m 0,β -f β, m 0,β 2 1 | c σ 2 ε -σ 2 ε |≤σ 2 ε /2 ≤ f β,m 0,β -f β,m 0,β 2
and clearly [A6] holds, we have

f β, m 0,β -β 2 1 | c σ 2 ε -σ 2 ε |≤σ 2 ε /2 ≤ CL(π m 0,β ) -2b 1 | c σ 2 ε -σ 2 ε |≤σ 2 ε /2 ≤ C ′ L(πm 0,β ) -2b . Therefore E( f β, m 0,β -f β 2 1 | c σ 2 ε -σ 2 ε |≤σ 2 ε /2 ) ≤ C[log(N J)] -b .
On the other hand, by using that m 0,β ≤ m n , we get

f β, m 0,β -f β, m 0,β 2 ≤ CN and f β, m 0,β -f β 2 ≤ f β 2 , so that E( f β, m 0,β -f β 2 1 | c σ 2 ε -σ 2 ε |>σ 2 ε /2 ) ≤ CN P(| σ 2 ε -σ 2 ε | > σ 2 ε /2).
The following lemma yields the result.

Lemma 3.

(38)

P(| σ 2 ε -σ 2 ε | > σ 2 ε /2) ≤ C/N 2 .
Proof of Lemma 3. We take J = 3 for simplicity. We know that σ2 ε is a linear combination of σ2 Y.j , j = 1, 2, 3 and (say) σ2 Y.2 Y.3 . Therefore as P(|X +Y | > a) ≤ P(|X| > a/2)+P(|Y | > a/2), the result follows if we prove that for j = 1, 2, 3,

P(|σ 2 Y.j -σ 2 .j | > 1 2 c 2 j ) ≤ C j N 2 , and P(|σ 2 Y.2 Y.3 -σ 2 .2,3 | > 1 2 c 2 j ) ≤ C 4 N 2 .
We provide few details for the first term and j = 3. The bound relies on a Rosenthal inequality which states that for independent centered random variables X 1 , . . . , X n admitting moments of order p

E(| n i=1 X i | p ) ≤ c p   n i=1 E(|X i | p ) + n i=1 E(X 2 i ) p/2   .
Consequently

P := P(|σ 2 Y.3 -(σ 2 α + t 2 3 σ 2 β + 2t 3 σ αβ + σ 2 ε )| > λ) = P(| 1 N N k=1 (A 2 k -E(A 2 k )) -( Ā -E( Ā)) 2 | > λ)
where

A k = (α k -E(α k )) + t 3 (β k -E(β k )) + ε k,3 and Ā = N -1 N k=1 A k .
Then by Markov inequality stating that P(|X| > λ) ≤ E(|X| p )/λ p , we get

P ≤ P(| 1 N N k=1 (A 2 k -E(A 2 k ))| > λ/2) + P(( Ā -E( Ā)) 2 > λ/2) ≤ 2 λ 4 E   1 N N k=1 (A 2 k -E(A 2 k )) 4   + 2 λ 4 E(| Ā -E( Ā)| 4 ) ≤ 2 λ 4 E(A 4 1 ) N 2 + 4 λ 2 (N -3 E(A 4 1 ) + N -2 (E(A 2 1 )) 2 )
under a moment condition of order 4 for α, β and ε. This gives the announced result.

B.5. Proof of Lemma 1.

(1) We first prove assertion (1) of the lemma. We denote by

R(u) = 1 ( f * ε ) 2 (u) - 1 (f * ε ) 2 (u)
.

First we write a decomposition:

E(|R(u)| 2 ) = E 1 (f * ε ) 4 (u)<N -1/2 (f * ε ) 4 (u) + 1 (f * ε ) 4 (u) E      1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 (f * ε ) 4 (u) (f * ε ) 4 (u) + (f * ε (u)) 2 2      Then, using that 1/[ (f * ε ) 4 (u) + (f * ε (u)) 2 ] 2 ≤ 1/ (f * ε ) 4 (u), we obtain E(|R(u)| 2 ) ≤ 1 (f * ε ) 4 (u) + N (f * ε ) 4 (u) E (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 ≤ 2 (f * ε ) 4 (u) which is the first term of the bound. (i) if (f * ε ) 4 (u) ≤ 2N -1/2 , we have N -1/2 /(f * ε ) 8 (u) ≤ 2N -1 /(f * ε ) 12 (u).
Moreover, starting in the same way as above, using that 1/

[ (f * ε ) 4 (u) + (f * ε (u)) 2 ] 2 ≤ 1/(f * ε (u)) 4 , we also have E(|R(u)| 2 ) ≤ 1 (f * ε ) 4 (u) + N 1/2 (f * ε ) 8 (u) E (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 ≤ 1 (f * ε ) 4 (u) + N -1/2 (f * ε ) 8 (u) ≤ 3 N -1/2 (f * ε ) 8 (u) (ii) If (f * ε ) 4 > 2N -1/2
, using the Bernstein Inequality yields:

P | (f * ε ) 4 (u)| < N -1/2 ≤ P | (f * ε ) 4 (u) -(f * ε ) 4 (u)| > (f * ε ) 4 (u) -N -1/2 ≤ P | (f * ε ) 4 (u) -(f * ε ) 4 (u)| > (f * ε ) 4 (u)/2 ≤ 2 exp -N (f * ε ) 8 (u)/16 ≤ O N -1 (f * ε (u)) -8
and completing the decomposition above, this yields

E(|R(u)| 2 ) ≤ E 1 (f * ε ) 4 (u)<N -1/2 (f * ε ) 4 (u) + 1 (f * ε ) 8 (u) E      1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 ( f * ε ) 4 (u) + (f * ε ) 2 (u) 2      + 1 (f * ε ) 4 (u) E      1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 (f * ε ) 4 (u) + (f * ε ) 2 (u) 2 1 (f * ε ) 4 (u) - 1 (f * ε ) 4 (u)      Then, using that 1/[ (f * ε ) 4 (u) + (f * ε (u)) 2 ] 2 ≤ 1/(f * ε (u)) 4 , we get E(|R(u)| 2 ) ≤ 1 (f * ε ) 4 (u) P (f * ε ) 4 (u) < N -1/2 + N -1 (f * ε ) 12 (u) + N 1/2 (f * ε ) 12 (u) E (f * ε ) 4 (u) -(f * ε ) 4 (u) 3 ≤ CN -1 (f * ε ) 12 (u) + N -1 (f * ε ) 12 (u) + N 1/2 (f * ε ) 12 (u) N -3/2 ≤ c N -1 (f * ε ) 12 (u)
Thus, in that case where

N -1/2 /(f * ε ) 8 (u) ≥ 2N -1 /(f * ε ) 12 (u), we get E(|R(u)| 2 ) ≤ N -1 |f * ε (u)| 12 .
This ends the proof of assertion (1).

(2) We now prove assertion (2). Set R 0 (u) = 1

f * ε (u) - 1 f * ε (u)
.

First we write:

E(|R 0 (u)| 2 ) = E 1 (f * ε ) 4 (u)<N -1/2 (f * ε ) 2 (u) + 1 (f * ε ) 2 (u) E      1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 (f * ε ) 4 (u) 1/2 (f * ε ) 4 (u) + (f * ε (u)) 2 2 (f * ε ) 4 (u) 1/4 + f * ε (u) 2      ≤ 1 (f * ε ) 2 (u) + N (f * ε ) 2 (u) E (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 ≤ 2 (f * ε ) 2 (u)
which is the first term of the bound.

(i) if (f * ε ) 4 (u) ≤ 2N -1/2 , we have

N -3/4 (f * ε ) 8 (u) ∧ N -1/2 (f * ε ) 6 (u) ∧ N -1/4 (f * ε ) 4 (u) ≤ N -1 (f * ε ) 10 (u)
.

Moreover, starting in the same way as above, we also have

E(|R 0 (u)| 2 ) ≤ 1 (f * ε ) 2 (u) + N 1/4 (f * ε ) 8 (u) ∧ N 1/2 (f * ε ) 6 (u) ∧ N 3/4 (f * ε ) 4 (u) E (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 ≤ 1 (f * ε ) 2 (u) + N -3/4 (f * ε ) 8 (u) ∧ N -1/2 (f * ε ) 6 (u) ∧ N -1/4 (f * ε ) 4 (u) ≤ 3 N -3/4 (f * ε ) 8 (u) ∧ N -1/2 (f * ε ) 6 (u) ∧ N -1/4 (f * ε ) 4 (u)
where the last line follows from the assumption (f * ε ) 4 (u) ≤ 2N -1/2 . (ii) If (f * ε ) 4 > 2N -1/2 , we use as in the proof of (1) that P(| (f * ε ) 4 (u)| < N -1/2 ) ≤ C 1 N -1 (f * ε (u)) -8 , and completing the decomposition above, this yields

E(|R 0 (u)| 2 ) ≤ E 1 (f * ε ) 4 (u)<N -1/2 (f * ε ) 2 (u) + 1 (f * ε ) 4 (u) E      1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 (f * ε ) 4 (u) + (f * ε ) 2 (u) 2 (f * ε ) 4 (u) 1/4 + f * ε (u) 2      + 1 (f * ε ) 4 (u) E      1 (f * ε ) 4 (u)≥N -1/2 (f * ε ) 4 (u) -(f * ε ) 4 (u) 2 (f * ε ) 4 (u) + (f * ε ) 2 (u) 2 (f * ε ) 4 (u) 1/4 + f * ε (u) 2 ×   1 (f * ε ) 4 (u) - 1 (f * ε ) 2 (u)     ≤ 1 (f * ε ) 2 (u) P (f * ε ) 4 (u) < N -1/2 + N -1 (f * ε ) 10 (u) + N 1/2 (f * ε ) 10 (u) E (f * ε ) 4 (u) -(f * ε ) 4 (u) 3 ≤ C 1 N -1 (f * ε ) 10 (u) + N -1 (f * ε ) 10 (u) + N 1/2 (f * ε ) 10 (u) N -3/2 ≤ c N -1 (f *
ε ) 10 (u) Thus, in that case where N -1 /(f * ε ) 10 (u) is smaller than the three other terms found for case (i), we get

E(|R 0 (u)| 2 ) ≤ N -1 |f * ε (u)| 10
. This ends the proof of the lemma. B.6. Proof of Proposition 5. Clearly

f β,m -f β 2 = f β,m -f β,m 2 + f β,m -f β 2 ≤ 2 f β,m -f β,m 2 + 2 f β,m -f β,m 2 + f β,m -f β 2
We already know that E( f β,mf β,m 2 ) ≤ (4/(N (J -4))D 2 (m, 1) + m/N . Moreover,

f β,m -f β,m 2 = 1 2π πm -πm e -iux 2 J -4 J/2 j=3 f * Z j (u)R u ∆ du 2 = 1 2π πm -πm 2 J -4 J/2 j=3 f * Z j (u)R u ∆ 2 du ≤ 1 π πm -πm 2 J -4 J/2 j=3 ( f * Z j (u) -f * Z j (u)) 2 |R u ∆ | 2 du + 1 π πm -πm |f * β (u)| 2 |f * ε (u/∆)| 4 |R u ∆ | 2 du
Then, applying Lemma 1, using the independence of f * Z j (u) and R(u) for j ≥ 3, and that

E   2 J -4 J/2 j=3 ( f * Z j (u) -f * Z j (u) 2   = 1 N 4 (J -4) 2   J/2 j=3
Var(e iuZ j ) + j =j ′ cov(e iuZ j , e iuZ j ′ )

  ≤ 1 N 4 (J -4) 2 J -4 2 + (J -4) 2 4 (1 -|f * β (u)| 2 )|f * ε ( u ∆ )| 4 ≤ 1 N 2 J -4 1 + J -4 2 |f * ε ( u ∆ )| 4 yields E( f β,m -f β,m 2 ) ≤ 1 π πm -πm 1 N 2 J -4 1 + J -4 2 |f * ε ( u ∆ )| 4 du |f * ε ( u ∆ )| 4 + C 0 π πm -πm |f * β (u)| 2 N -1/2 |f * ε ( u ∆ )| 4 ∧ N -1 |f * ε ( u ∆ )| 8 du ≤ 4 N (J -4) D 2 (m, 1) + 2m N + 2C 0 ( D 2 (m, f β ) √ N ) ∧ ( D 4 (m, f β ) N ).
Now, gathering all term gives

E( f β,m -f β 2 ) ≤ f β,m -f β 2 + 16 N (J -4) D 2 (m, 1) + 6 m N + 4C 0 ( D 2 (m, f β ) √ N ) ∧ ( D 4 (m, f β ) N ),
which is the announced result.

3. 2 .

 2 Risk bound for the estimator of f α . Let us define f α,m such that f * α,m = f * α 1 [-πm; πm] . The function f α,m is the function which is in fact estimated by f α,m . Using similar calculations to those used for f β,m , we get the following global MISE bound. Proposition 2. Consider Model (1) under Assumptions [A1]-[A5] and estimator f α,m

  ) under Assumptions [A1], [A3], [A4] and Assumption [A2'] replacing Assumption [A2]: [A2'] Assumption [A2] holds and the measurement errors ε k,j are symmetric.Note that the symmetry of the noise together with the condition f

B. 7 .2

 7 Proof of Proposition 6. The proof of Proposition 6 follows the same line as the proof of Proposition 5 in a somehow simpler setting. Clearlyf 0 α,mf α 2 = f 0 α,mf α,m 2 + f α,m -f α,m 2 + f α,mf α We already know that E( f 0 α,mf α,m 2 ) ≤ (2π) -1 ( πm -πm du/|f * ε (u)| 2 )/N . Moreover Y 0 (u)R 0 (u)| 2 du which yields the result by writing that | f * Y 0 (u)| 2 ≤ 2| f * Y 0 (u)f * Y 0 (u)| 2 + 2|f * α (u)f * ε (u)| 2 , using the independence of f *Y 0 (u) and R 0 and applying Lemma 1.

  Risk bound for the estimator of f β . Let us define f β,m such that f

	1 t 1 ))	du
	3. Risk bounds and cutoff selection with known noise density
	3.1.

* β,m = f * β 1 [-πm; πm]

Table 1 .

 1 Empirical MISE ×100 computed from 100 simulated datasets with ε Laplace. Estimators of f α are f 0 α,m , f 0 α,m , f α,m and f α,m . Estimators of f β are f β,m and f β,m .

			200 2000	200 2000	200	2000
	α Gaussian	f 0 α,m	0.324 0.037 0.402 0.048 0.745 0.105
		f 0 α,m	0.243 0.082 0.266 0.103 0.544 0.113
		f α,m	0.469 0.055 0.520 0.094 2.022 1.014
		f α,m	0.700 0.180 0.736 0.122 1.170 0.228
	β Gaussian	f β,m	0.427 0.044 0.313 0.032 0.402 0.041
		f β,m	0.301 0.032 0.285 0.042 0.535 0.151
	α Gaussian	f 0 α,m	0.335 0.036 0.417 0.048 0.744 0.111
		f 0 α,m	0.238 0.074 0.271 0.105 0.434 0.118
		f α,m	0.469 0.050 0.538 0.097 2.080 1.046
		f α,m	0.448 0.092 0.770 0.162 1.151 0.234
	β Mixed	f β,m	1.500 0.312 2.288 0.425 6.648 1.311
	Gaussian	f β,m	1.407 0.211 6.578 2.107 15.074 10.091
	α Mixed	f 0 α,m	1.690 0.339 6.116 0.684 10.166 5.203
	Gaussian	f 0 α,m	2.088 0.353 6.168 0.703 14.302 6.014
		f α,m	5.673 2.269 22.945 9.926 36.288 34.551
		f α,m	3.328 0.700 14.279 6.555 34.705 31.040
	β Gaussian	f β,m	0.424 0.053 0.283 0.034 0.428 0.043
		f β,m	0.267 0.036 0.297 0.047 0.522 0.156
	α Gamma	f 0 α,m	0.362 0.047 0.483 0.053 0.896 0.125
		f 0 α,m	0.288 0.057 0.348 0.112 0.601 0.173
		f α,m	0.476 0.053 0.618 0.126 2.438 1.221
		f α,m	0.612 0.118 0.730 0.179 1.477 0.307
	β Gamma	f β,m	0.395 0.053 0.351 0.047 0.410 0.080
		f β,m	0.315 0.047 0.335 0.054 0.517 0.183
	α Gamma	f 0 α,m	0.402 0.047 0.412 0.047 0.821 0.113
		f 0 α,m	0.356 0.050 0.313 0.108 0.514 0.164
		f α,m	0.653 0.060 0.586 0.115 2.368 1.232
		f α,m	0.859 0.106 0.876 0.136 1.295 0.289
	β Mixed	f β,m	1.247 0.360 1.228 0.465 2.848 0.639
	Gamma	f β,m	1.194 0.331 1.457 0.689 6.769 1.790