Matthieu Lerasle

Optimal model selection for stationary data under various mixing conditions

Keywords: Density estimation, optimal model selection, resampling methods, slope heuristic, weak dependence. 2000 Mathematics Subject Classification: 62G07, 62G09, 62M99

We build penalized least-squares estimators of the marginal density of a stationary process, using the slope algorithm and resampling penalties. When the data are β or τ -mixing, these estimators satisfy oracle inequalities with leading constant asymptotically equal to 1.

Introduction

The history of statistical model selection goes back at least to Akaike [START_REF] Akaike | Statistical predictor identification[END_REF], [START_REF] Akaike | Information theory and an extension of the maximum likelihood principle[END_REF] and Mallows [START_REF] Mallows | Some comments on c p[END_REF]. They proposed to select among a collection of parametric models the one which minimizes an empirical loss plus some penalty term proportional to the dimension of the models. Birgé & Massart [START_REF] Birgé | From model selection to adaptive estimation[END_REF] and Barron,Birgé & Massart [BBM99] generalize this approach, making the link between model selection and adaptive estimation. They also proved that several estimation procedures as cross-validation (Rudemo [START_REF] Rudemo | Empirical choice of histograms and kernel density estimators[END_REF]) or hard thresholding (Donoho et.al. [START_REF] Donoho | Density estimation by wavelet thresholding[END_REF]) can be interpreted in terms of model selection. More recently, Birgé & Massart [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF], Arlot & Massart [START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF] and Arlot [START_REF] Arlot | Resampling and model selection[END_REF], [START_REF] Arlot | Model selection by resampling penalization[END_REF] arised the problem of optimal model selection. Basically, the aim is to select an estimator satisfying an oracle inequality with leading constant asymptotically equal to 1. Two totally data driven procedures are known to achieve this goal: the slope algorithm, introduced by Birgé & Massart [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] and the resampling penalties defined by Arlot [START_REF] Arlot | Model selection by resampling penalization[END_REF]. Arlot & Massart [START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF] and Arlot [START_REF] Arlot | Model selection by resampling penalization[END_REF] proved that these estimators are efficient to select the best histogram in a general regression framework. In [START_REF] Lerasle | Optimal model selection in density estimation[END_REF], we proved that these procedures are also optimal in density estimation, when the data are independent. There exists a lot of statistical frameworks where the data are not independent. The previous results may therefore not hold. Baraud et.al. [START_REF] Baraud | Adaptive estimation in autoregression or β-mixing regression via model selection[END_REF] proved that penalties proportional to the dimension can also be used when the data are β-mixing (for a definition of the coefficient β, see Rozanov & Volkonskii [START_REF] Volkonskiȋ | Some limit theorems for random functions[END_REF] or Section 2). They worked in a regression framework and Comte & Merlevède [START_REF] Comte | Adaptive estimation of the stationary density of discrete and continuous time mixing processes[END_REF] extended the result to density estimation. In [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF], we proved that the same penalties can also be used with τ -mixing data (the coefficient τ has been introduced by Dedecker & Prieur [START_REF] Dedecker | New dependence coefficients. Examples and applications to statistics[END_REF], see Section 2). The main problem of the algorithm proposed by Comte & Merlevède [START_REF] Comte | Adaptive estimation of the stationary density of discrete and continuous time mixing processes[END_REF] is that the penalty term involves a constant depending on the mixing coefficients (both in the β and τ -mixing cases) which is typically unknown in practice. As in the independent case, we prove that a resampling estimator catches the shape of the ideal penalty with great generality as it "learns" part of the mixing structure of the data (Künsch [START_REF] Künsch | The jackknife and the bootstrap for general stationary observations[END_REF], Liu & Singh [START_REF] Liu | Moving blocks jackknife and bootstrap capture weak dependence[END_REF]). We will also prove that the slope algorithm can be used to calibrate in an optimal way the constant in front of the penalty term. The new penalization procedure is totally data driven. Let us now explain more precisely the problem that we will consider.

Least-squares estimators

We observe n real valued, identically distributed random variables X 1 , ..., X n , defined on a probability space (Ω, A, P), with common law P . We assume that a measure µ on (R, B(R)) is given. We denote by L 2 (µ) the Hilbert space of square integrable real valued functions and by . the associated L 2 -norm. The parameter of interest is the density s of P with respect to µ, we assume that it belongs to L 2 (µ). For all function g in L 1 (P ), we define

P g = R gsdµ = E (g(X)) , P n g = 1 n n i=1 g(X i ),
where X is a copy of X 1 , independent of (X 1 , ..., X n ). s minimizes the integrated contrast t → t 2 -2P t over L 2 (µ). The risk of an estimator ŝ of s is measured with the L 2 -loss, that is sŝ 2 , which is random when ŝ is. The problem of density estimation is a problem of M -estimation. These problems are classically solved in two steps when the data are independent. First, we choose a "model" S m close to the parameter s, which means that inf t∈Sm st 2 is "small". Then, we minimize over S m the empirical version of the integrated contrast, that is, we choose ŝm ∈ arg min t∈Sm t 2 -2P n t.

When the data are mixing, the coupling method is a very powerful tool to extend the methods developed in the independent case. It can be summarized as follows.

Coupling method: Let I 0 , J 0 , ..., I p-1 , J p-1 be a partition of {1, ..., n} satisfying q = min k=0,...,p-1 min(I k+1 )max(I k ) > 0 (for a proper definition of this partition see Section 2). For all k = 0, ..., p -1, let A k = (X l ) l∈I k and let l k be the length of I k . A coupling lemma associates to the sequence (A k ) k=0,...,p-1 independent random variables (A * k ) such that E (d(A k , A * k )) ≤ γ(q), where γ is the mixing coefficient of the data, d is a distance on R l k . Let I = ∪ p-1 k=0 I k and let P A be the empirical process based on the data (X i , i ∈ I), that is P A = i∈I δ X i /|I|. To bound quantities of the form F (P n ), built with the empirical process, we first use algebraic inequalities to obtain

F (P n ) ≤ CF (P A ).
(1)

Then we have F (P A ) ≤ F (P A * ) + |F (P A ) -F (P A * )|.

We can now use the results available for independent random variables to bound F (P A * ) and the mixing properties to bound |F (P A ) -F (P A * )|.

Up to our knowledge, all the model selection procedures proposed for mixing data use the coupling methods. In this scheme, the bounds given on F (P n ) are the same as those given for F (P A ) and the only essentially suboptimal bound is the first one: F (P n ) ≤ CF (P A ).

We extend the procedures developed in the independent case in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF] through the coupling method. As we are looking for optimal results, we will work with the process P A instead of P n , avoiding the lost (1). The counterpart of this choice is that we do not use all the data to build our estimator. In particular, the variance of an oracle built only with the variables (X i ) i∈I is bigger than the one of an oracle built with all the sample when the data are independent. However, we will see in Section 4 that our final estimator improves the previous procedures proposed in a mixing setting.

Let us now define the least-squares estimators by ŝA,m ∈ arg min t∈Sm P A Q(t). The minimization problem defining ŝA,m can be computationally untractable for general sets S m , leading to untractable procedures. However, in density estimation, it can be easily solved when S m is a linear subspace of L 2 (µ) since, for any orthonormal basis (ψ λ ) λ∈m of S m , ŝA,m = λ∈m

(P A ψ λ )ψ λ .
The risk of ŝA,m is decomposed in the classical bias and variance terms thanks to Pythagoras relation. Let s m be the orthogonal projection of s onto S m , then

s -ŝA,m 2 = s -s m 2 + s m -ŝA,m 2 . ( 2
)
The space S m should be chosen in order to realize a trade-off between those quantities.

In [START_REF] Lerasle | Optimal model selection in density estimation[END_REF], we proved a concentration inequality for s m -ŝA,m 2 around its expectation when the data are independent. It proves that

D * A,m = nE( s m -ŝA,m
2 ) is a natural complexity measure of S m and, when the models S m are sufficiently regular, we recovered that the dimension d m of S m has the same order as D * A,m . However, this is not true in general, because there exist simple models (histograms with a small d m ) where D * A,m >> d m and model of infinite dimension where D * A,m behaves nicely (see Birgé [START_REF] Birgé | Model selection for density estimation with l 2 -loss[END_REF] or Section 4).

Model selection

The choice of a "good" model S m is impossible without strong assumptions on s, for example that we have precise information on its regularity. However, if we only assume that s is regular, it is possible to choose a collection of models (S m ) m∈Mn such that one of them realizes an optimal trade-off (see for example Birgé & Massart [BM97] or Barron,Birgé & Massart [BBM99]). Given the projection estimators (ŝ A,m ) m∈Mn associated to this collection, the aim is then to build an estimator m such that the final estimator, s = ŝA, m behaves almost as well as any model m o in the set of oracles

M * n = {m o ∈ M n , ŝA,mo -s 2 = inf m∈Mn ŝA,m -s 2 }.
This is the problem of model selection. More precisely, we want the final estimator s = ŝA, m to satisfy one of the following type of oracle inequalities

∃K > 0, C n > 0, γ > 1, P s -s 2 > C n inf m∈Mn s -ŝA,m 2 ≤ K n γ . ( 3
) ∃K > 0, C n > 0, E s -s 2 ≤ C n E inf m∈Mn s -ŝA,m 2 + K n . (4)
In both cases, the leading constant C n should be as close as possible to 1. In order to build m, remark that, for all m in M n ,

s -ŝA,m 2 -s 2 = ŝA,m 2 -2P A ŝA,m + 2ν A (ŝ A,m ),
where ν A = P A -P . An oracle minimizes s -ŝA,m 2s 2 over M n . As we want to imitate the oracle, we will design a map pen :

M n → R + and choose m ∈ arg min m∈Mn ŝA,m 2 -2P A ŝA,m + pen(m), s = ŝA, m.
(5)

It is clear that the ideal penalty is pen id (m) = 2ν A (ŝ A,m ) and our goal is to design sharp estimators of this quantity as penalty functions.

The key point to obtain oracle inequalities is the following decomposition of the risk of s. For all m in M n , let

p(m) = ν A (ŝ A,m -s m ) = ŝA,m -s m 2 .
For all m in M n ,

s -s 2 = s 2 -2P s + s 2 = s 2 -2P A s + 2ν A s + s 2 ≤ ŝA,m 2 -2P A ŝA,m + pen(m) + (2ν A (s) -pen( m)) + s 2 = s -ŝA,m 2 + (pen(m) -2ν A (ŝ A,m )) + (2ν A (s) -pen( m))
Thus, for all m in M n ,

s -s 2 ≤ s -ŝA,m 2 + (pen(m) -2p(m)) + (2p( m) -pen( m)) + 2ν A (s m -s m ). (6)

Optimal model selection

Let us now precise the definition of the methods that we will use to calibrate the penalty.

The slope algorithm

The "slope heuristic" was introduced by Birgé & Massart [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] in the Gaussian regression framework. It states that there exists a complexity measure ∆ m of S m and a constant

K min such that 1. if pen(m) < K min ∆ m , ∆ m is too large, typically ∆ m ≥ C sup m∈Mn ∆ m , where C is a constant independent of n.
2. if pen(m) ≃ K∆ m for some K > K min , then ∆ m is "much smaller", 3. if pen(m) ≃ 2K min ∆ m , then the risk of the selected estimator satisfies

s -s 2 ≤ C n inf m∈Mn s -ŝA,m 2 , with C n → 1, when n → ∞
in expectation and with large probability.

When both ∆ m and the associated K min are known, point 3 in this heuristic says that pen(m) ≃ 2K min ∆ m is an optimal penalty. This heuristic is classically used when ∆ m is known and K min is unknown. Arlot & Massart [START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF] introduced the following algorithm to calibrate the penalty term in this situation.

Slope algorithm

• For all K > 0, compute the selected model m(K) given by (5) with the penalty pen(m) = K∆ m and the associated complexity ∆ m(K) .

• Find a constant K o such that ∆ m(K) is large when K < K o , and "much smaller" when K > K o .

• Take the final m = m(2K o ).

In [START_REF] Lerasle | Optimal model selection in density estimation[END_REF], we justified the slope heuristic in density estimation with independent data for ∆ m = E( s m -ŝA,m 2 ), K min = 1. This complexity is unknown in practice and has to be estimated. We proposed a resampling estimator and proved that it works without extra assumptions on our collection of models. In this paper, we will extend these results to mixing processes.

Resampling penalties

Data-driven penalties have been studied in density estimation, in particular, cross-validation methods as in Stone [START_REF] Stone | Cross-validatory choice and assessment of statistical predictions[END_REF], Rudemo [START_REF] Rudemo | Empirical choice of histograms and kernel density estimators[END_REF] or Celisse [START_REF] Célisse | Density estimation via cross validation: Model selection point of view[END_REF]. We extend the approach of [START_REF] Lerasle | Optimal model selection in density estimation[END_REF] based on the resampling penalties introduced by Arlot [START_REF] Arlot | Model selection by resampling penalization[END_REF]. We prove that it provides optimal model selection procedures. An important ingredient in the proofs is the coupling properties of mixing processes. The coupling result proved in Viennet [START_REF] Viennet | Inequalities for absolutely regular sequences: application to density estimation[END_REF] for β-mixing processes allows a straightforward extension of the results of [START_REF] Lerasle | Optimal model selection in density estimation[END_REF]. The coupling lemma available for τ -mixing sequences is not so powerful and in that case, we have to develop new methods of proofs.

The paper is organized as follows. In Section 2, we introduce our new estimation procedure and describe our main assumptions. In Section 3, we state our main results, we prove the efficiency of the penalized least-squares estimators based on the slope heuristic and on resampling methods. In Section 4, we compare our new estimators with those given in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF]. The proofs of the main theorems are postponed to Section 5. Section 6 is an Appendix where we recall some probabilistic lemmas proved in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF].

New estimation procedures

Blockwise decomposition of the data

Assume that n is even and let p and q be two integers such that 2pq = n. For all k = 0, ..., p -1, let I k = (2kq + 1, ..., (2k + 1)q), A k = (X l ) l∈I k and I = ∪ p-1 k=0 I k . For all functions t in L 2 (µ) and all x 1 , ..., x q in R, let

L q (t)(x 1 , ..., x q ) = 1 q q i=1 t(x i ), P A t = 1 p p-1 k=0 L q (t)(A k ) = 2 n i∈I t(X i ), ν A (t) = (P A -P )(t).
Let S m be a linear space. The estimator ŝA,m associated to S m , is defined by ŝA,m ∈ arg min

t∈Sm t 2 -2P A t. (7)
Given an orthonormal basis (ψ λ ) λ∈m of S m , classical computations prove that ŝA,m = λ∈m

(P A ψ λ )ψ λ , s m -ŝA,m 2 = λ∈m (ν A (ψ λ )) 2 = sup t∈Bm (ν A (t)) 2 .

Resampling penalties

The first penalization procedure is based on the resampling penalties introduced by Arlot [START_REF] Arlot | Model selection by resampling penalization[END_REF]. The resampling algorithm is slightly modified in order to keep the dependence structure inside the blocks (see Künsh [START_REF] Künsch | The jackknife and the bootstrap for general stationary observations[END_REF], Liu & Singh [START_REF] Liu | Moving blocks jackknife and bootstrap capture weak dependence[END_REF] or Radulovic [START_REF] Radulović | On the bootstrap and empirical processes for dependent sequences[END_REF]). Let W 0 , ..., W p-1 be a resampling scheme, that is, a vector of random variables, independent of X 1 , ..., X n and exchangeable, i.e., for all permutation ξ of {0, ...p -1}, (W ξ(0) , ...., W ξ(p-1) ) has the same law as (W 0 , ..., W p-1 ).

Let P W A and ν W A be the associated resampling empirical processes defined, for all t in L 2 (µ), by

P W A (t) = 1 p p-1 k=0 W k L q (t)(A k ), ν W A (t) = (P W A -Wp P A )(t) = 1 p p-1 k=0 (W k -Wp )L q (t)(A k ), where Wp = 1 p p-1 k=0 W k . For all m in M n , let ŝW A,m = arg min t∈Sm t 2 -2P W A (t) = λ∈m (P W A ψ λ )ψ λ . Setting v 2 W = Var(W 1 -Wp ) and C W = v -2
W , the resampling penalty is defined by

pen(m) = 2C W E W sup t∈Bm (ν W A (t)) 2 = 2C W λ∈m E W (ν W A (ψ λ )) 2 . (8)
Hereafter, for all m in M n and for all function pen, the final estimator is always denoted by s = ŝA, m, where m = arg min

m∈Mn ŝm 2 -2P A ŝA,m + pen(m). ( 9
)
2.3 Some measures of dependence

β-mixing data

The coefficient β was introduced by Rozanov & Volkonskii [START_REF] Volkonskiȋ | Some limit theorems for random functions[END_REF]. For a random variable Y defined on a probability space (Ω, A, P) and a σ-algebra M in A, let

β(M, σ(Y )) = E sup A∈B |P Y |M (A) -P Y (A)| .
For all stationary sequence of random variables (X n ) n∈Z defined on (Ω, A, P), let

β k = β(σ(X i , i ≤ 0), σ(X i , i ≥ k)).
The process (X n ) n∈Z is said to be β-mixing when β k → 0 as k → ∞. Examples of βmixing processes can be found in the books of Doukhan [START_REF] Doukhan | Mixing[END_REF] and Bradley [START_REF] Bradley | Introduction to strong mixing conditions[END_REF]. One of the most important is the following: a stationary, irreducible, aperiodic and positively recurent Markov chain (X i ) i≥1 is β-mixing. Let us recall Lemma 5.1 in Viennet [START_REF] Viennet | Inequalities for absolutely regular sequences: application to density estimation[END_REF].

Lemma: [START_REF] Viennet | Inequalities for absolutely regular sequences: application to density estimation[END_REF]) Assume that the process (X 1 , ..., X n ) is β-mixing and let p, q and A 0 , ..., A p-1 be respectively the integers and the random variables defined in Section 2.1. There exist random variables A * 0 , ..., A * p-1 such that:

1. for all k = 0, ..., p -1, A * k = (X * 2kq+1 , ..., X * (2k+1)q ) has the same law as A k , 2. for all k = 0, ..., p -1, A * k is independent of A 0 , ..., A k-1 , A * 0 , ..., A * k-1 , 3. for all k = 0, ..., p -1, P(A k = A * k ) ≤ β q .

τ -mixing data

The coefficient τ was introduced by Dedecker & Prieur [START_REF] Dedecker | New dependence coefficients. Examples and applications to statistics[END_REF]. For all l in N * , for all x, y in R l , let d l (x, y) = l i=1 |x iy i |. For all l in N * , for all function t defined on R l , the Lipschitz semi-norm of t is defined by Lip l (t) = sup

x =y∈R l |t(x) -t(y)| d l (x, y) .
For all functions t defined on R, we will denote for short by Lip(t) = Lip 1 (t). Let λ 1 be the set of all functions t : R l → R such that Lip l (t) ≤ 1. For all integrable, R l -valued, random variables Y defined on a probability space (Ω, A, P) and all σ-algebra M in A, let

τ (M, Y ) = E sup t∈λ 1 |P Y |M (t) -P Y (t)| .
For all stationary sequences of integrable random variables (X n ) n∈Z defined on (Ω, A, P), for all integers k, r, let

τ k,r = max 1≤l≤r 1 l sup k≤i 1 <..<i l {τ (σ(X p , p ≤ 0), (X i 1 , ..., X i l ))}, τ k = sup r∈N * τ k,r .
The process (X n ) n∈Z is said to be τ -mixing when τ k → 0 as k → ∞. Lemma: [τ -coupling, Claim 1 p17 in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF]] Assume that the process (X 1 , ..., X n ) is τmixing and let p, q and A 0 , ..., A p-1 be respectively the integers and the random variables defined in Section 2.1. There exist random variables A * 0 , ..., A * p-1 such that:

1. for all k = 0, ..., p -1, A * k = (X * 2kq+1 , ..., X * (2k+1)q ) has the same law as A k , 2. for all k = 0, ..., p -1,

A * k is independent of A 0 , ..., A k-1 , A * 0 , ..., A * k-1 , 3. for all k = 0, ..., p -1, E(d q (A k , A * k )) ≤ qτ q .

Main assumptions

Let p, q and A 0 , ..., A p-1 be respectively the integers and the random variables defined in Section 2.1. For all m, m ′ in M n , let

v 2 A,m,m ′ = sup t∈Sm+S m ′ , t ≤1 qVar(L q (t)(A 0 )), D A,m = q λ∈m Var(L q (ψ λ )(A 0 )), b m,m ′ = sup t∈Sm+S m ′ , t ≤1 t ∞ .
For all m in M n , let

R A,m = n s -s m 2 + 2D A,m , e A,m,m ′ = q p b 2 m,m ′ .

We denote by e

A,m = e A,m,m , v A,m = v A,m,m . For all k ∈ N, let M k n = {m ∈ M n , R A,m ∈ [k, k + 1)}.
For all n in N * , for all k > 0, k ′ > 0, for all γ ≥ 0, let [k] denote the integer part of k and let

l n,γ (k, k ′ ) = ln (1 + Card(M [k] n ))(1 + Card(M [k ′ ] n ))(k + 1)(k ′ + 1) + (ln n) γ (10)
The following assumptions generalize Assumptions [V] and [BR] made in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF].

[V']: There exist γ > 1 and a sequence (ǫ n ) n∈N , with ǫ n → 0 such that, for all n in N,

sup (m,m ′ )∈(Mn) 2      v 2 A,m,m ′ R A,m ∨ R A,m ′ 2 ∨ e A,m,m ′ R A,m ∨ R A,m ′   l m,m ′    ≤ ǫ 4 n ,
where, for all m, m

′ in M n , l m,m ′ = l n,γ (R A,m , R A,m ′ ).
[BR'] There exist two sequences

(h * n ) n∈N * and (h o n ) n∈N * with (h o n ∨h * n ) → 0 as n → ∞ such that, for all n in N * , for all m o ∈ arg min m∈Mn R A,m and all m * ∈ arg max m∈Mn D A,m , we have R A,mo D A,m * ≤ h o n , n s -s m * 2 D A,m * ≤ h * n .
3 Main results

Resampling penalties

The first theorem justifies the use of resampling penalties for β-mixing data.

Theorem 3.1 Let X 1 , ..., X n be a strictly stationary sequence of random variables with common density s and let (S m ) m∈Mn be a collection of linear subspaces of L 2 (µ) satisfying Assumption [V']. Let s be the estimator defined in (9) with pen(m) defined in (8). Assume that X 1 , ..., X n are β-mixing, then, there exists a constant C > 0 such that

P s -s 2 > (1 + 110ǫ n ) inf m∈Mn s -ŝA,m 2 ≤ Ce -1 2 (ln n) γ + pβ q . ( 11
)
The second theorem justifies the use of resampling penalties for τ -mixing data.

Theorem 3.2 Let X 1 , ..., X n be a strictly stationary sequence of random variables with common density s and let (S m ) m∈Mn be a collection of linear subspaces of L 2 (µ) satisfying Assumption [V']. Let s be the estimator defined in (9) with pen(m) defined in (8). Assume that X 1 , ..., X n are real valued and τ -mixing, then, there exists an absolute constant C > 0 such that we have

E s -s 2 ≤ (1 + 160ǫ n )E inf m∈Mn s -ŝA,m 2 + C e -1 2 (ln n) γ + τ q M C n , ( 12
)
where the mixing complexity M C n is defined by the following formula:

M C n = m∈Mn   λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ) + s |M n | sup t∈Bm Lip(t)   .
Comments:

• Theorems 3.1 and 3.2 can be compared with Theorem 2.5 in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF]. An extra term pβ q appears in the control of the deviation probability when the data are β-mixing.

In Section 4, it is proved that p and q can be chosen in order to have pβ q ≤ Cn -α for some α > 1 under classical assumptions on the mixing coefficients.

• When the data are τ -mixing, the mixing coefficient τ q must control the mixing complexity M C n . It is clear that M C n = ∞ for many collections of linear spaces (S m ) m∈Mn (as histogram spaces for example). Therefore, the collection M n should be chosen carefully when we deal with τ -mixing data. In Section 4, it is proved that, on wavelet spaces, p and q can be chosen in order to have τ q M C n ≤ Cn -1 under classical assumptions on the mixing coefficient.

• Up to our knowledge, inequalities (11) and ( 12) are the first oracle inequalities obtained for totally data driven PLSE of the density s when the data are mixing. Moreover, this is the first time that the risk of the selected estimator is compared with the risk of an oracle and not with an upper bound.

Slope heuristic

We will now justify the use of the slope heuristic when the data are mixing. The following theorems give point 1 in this heuristic, respectively for β and τ -mixing sequences. In both cases, the complexity ∆ m = D A,m /n can be used with the constant K min = 2.

Theorem 3.3 Let X 1 , ..., X n be a strictly stationary sequence of random variables, with common density s. Let M n be a collection of models satisfying Assumptions [V'], [BR'] and let ǫ * n = ǫ n ∨ h * n . Assume that there exists a constant 0 < δ < 1 such that, for all m in M n ,

0 ≤ pen(m) ≤ (2 -δ)D A,m n .
Let m, s be the random variables defined in (9). Assume that X 1 , ..., X n are β-mixing and let

c n = δ -75ǫ * n 2(1 + 27ǫ n ) .
There exists a constant C > 0, such that, with probability larger than

1 -Ce -1 2 (ln n) γ -pβ q , D A, m ≥ c n D A,m * , s -s 2 ≥ c n h o n inf m∈Mn s -ŝA,m 2 . ( 13
)
Theorem 3.4 Let X 1 , ..., X n be a strictly stationary sequence of random variables, with common density s. Let M n be a collection of models satisfying Assumptions

[V'], [BR'].
Assume that there exists a constant 0 < δ < 1 such that, for all m in M n ,

0 ≤ pen(m) ≤ (2 -δ)D A,m n .
Let m, s be the random variables defined in (9). Assume that X 1 , ..., X n are τ -mixing, let M C n be the mixing complexity defined in Theorem 3.2 and let

c ′ n = δ -h * n 2(1 + 35ǫ n ) .
There exists an absolute constant C > 0 such that

E(D A, m) ≥ c ′ n D A,m * -Cn e -1 2 (ln n) γ + τ q M C n . (14) E s -s 2 ≥ 2 c ′ n h o n E inf m∈Mn s -ŝA,m 2 -C e -1 2 (ln n) γ + τ q M C n . (15)
Comment: When n is sufficiently large, c n ≥ δ/4, c ′ n ≥ δ/4. Hence, when pen(m) is not larger than 2D A,m /n, inequalities (13) and ( 14) ensure that with high probability or in expectation D A, m ≥ cD A,m * , which is as large as possible. Inequalities (13) and (15) show that no optimal oracle inequality can hold. This proves point 1 of the slope heuristic. The following theorems justify the remaining points.

Theorem 3.5 Let X 1 , ..., X n be a stationary sequence of random variables with common density s. Let (S m ) m∈Mn be a collection of models satisfying [V']. For all m in M n , let pen(m) be a penalty function and let s be the estimator defined in (9). Assume that X 1 , ..., X n are β-mixing and that there exist constants δ ≥ δ > -1 and 0 ≤ p ′ < 1 such that, with probability at least 1p ′ , for all m in M n ,

4D A,m n + δ R A,m n ≤ pen(m) ≤ 4D A,m n + δ R A,m n . Let c n = 1+ δ+37ǫn 2(1+δ-27ǫn) if 1 + δ -27ǫ n > 0 +∞ if 1 + δ -27ǫ n ≤ 0 .
There exists a constant C > 0, such that, with probability at least 1-

Ce -1 2 (ln n) γ -pβ q -p ′ , D A, m ≤ c n R A,mo , s -s 2 ≤ 2c n inf m∈Mn s -ŝA,m 2 , ( 16
)
Theorem 3.6 Let X 1 , ..., X n be a stationary sequence of random variables with common density s. Let (S m ) m∈Mn be a collection of models satisfying [V']. For all m in M n , let pen(m) be a penalty function and let s be the estimator defined in (9). Assume that X 1 , ..., X n are τ -mixing and that there exist constants δ ≥ δ > -1 and a sequence (e n ) n∈N , with n∈N e n < ∞ such that

E sup m∈Mn 4D A,m n + δ R A,m n -pen(m) + ≤ e n , E sup m∈Mn pen(m) - 4D A,m n - δ R A,m n + ≤ e n .
Let M C n be the mixing complexity defined in Theorem 3.2 and let

c n = 1+ δ+55ǫn 2(1+δ-85ǫn) if 1 + δ -85ǫ n > 0 +∞ if 1 + δ -85ǫ n ≤ 0 .
There exists a constant C > 0, such that,

E (D A, m) ≤ c n (R A,mo + n(Cτ q M C n + e n )) . ( 17
) E s -s 2 ≤ c n E inf m∈Mn s -ŝA,m 2 + C (τ q M C n + e n ) (18)
Comments:

• D A, m jumps from D A,m * (Theorem 3.3 and 3.4) to R A,mo when pen(m) is around 2D A,m /n. R A,mo is much smaller than D A,m * under Assumption [BR']. This justifies point 2 of the slope heuristic. Point 3 comes from inequalities ( 16) and (18) applied with δ = δ = 0.

• It may be useful to overpenalize a little from a non asymptotic point of view. Imagine that 1 -67ǫ n is very close to 0, then c n is much smaller if δ > 0 than if we take its asymptotic optimal value 0.

• The practical implementation of these algorithms is discussed in general in Arlot & Massart [START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF], see also the discussion for density estimation in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF]. The slope heuristic is very fast to compute and shall be prefered when a shape of the ideal penalty is available. The resampling-based estimators give this shape for more general collections.

Comparison with previous results

In this section, we compare the estimator given by the resampling penalty with those given in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF]. Recall that the estimator was chosen among the collection of least-squares estimators (ŝ m ) m∈Mn , where ŝm = arg min t∈Sm t 2 -2P n t, by a penalization procedure s = ŝ m, where m = arg min

m∈Mn ŝm 2 -2P n ŝm + pen(m). ( 19
)
Mixing assumptions In [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF], we considered two kinds of rates of convergence to 0 of the mixing coefficients. Let γ = β or τ .

[AR(θ)] arithmetical γ-mixing with rate θ: there exists C > 0 such that, for all k in N, 1+θ) , [GEO(θ)] geometrical γ-mixing with rate θ: there exists C > 0 such that, for all k in N, γ k ≤ Ce -θk .

γ k ≤ C(1 + k) -(

β-mixing processes

In Comte & Merlevède [START_REF] Comte | Adaptive estimation of the stationary density of discrete and continuous time mixing processes[END_REF], as well as in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF], the collection of models was assumed to satisfy the following assumptions:

[M 1 ] For all m ∈ M n , S m is a linear space with finite dimension d m ≥ 2 and N n = max m∈Mn d m satisfies N n ≤ n. [M 2 ] There exists a constant Φ such that ∀m, m ′ ∈ M n , ∀t ∈ S m , ∀t ′ ∈ S m ′ , t + t ′ ∞ ≤ Φ dim(S m + S m ′ ) t + t ′ 2 . [M 3 ] d m ≤ d m ′ implies that S m ⊂ S m ′ . From [M 1 ], for all k > n, M k n = ∅ and, from [M 3 ], for all k ≤ n, Card(M k n ) ≤ 1.
Hence, there exists a constant c V such that, for all γ > 1,

l m,m ′ ≤ c V (ln n) 2γ
In order to verify [V'], we need two other assumptions.

[M 4 ] There exists c ′ D > 0 such that, for all n in N * , for all m in M n , D A,m ≥ c ′ D d m .
[M 5 ] There exist γ > 1 and a sequence r n → ∞ such that R n (ln n) -4γ ≥ r n , where

R n = inf m∈Mn R A,m .
Under these assumption, the following result holds:

Corollary 4.1 Let M n be a collection of models satisfying [M 1 ]-[M 5 ].
Assume that the process (X n ) n∈Z is strictly stationary and arithmetically [AR(θ)] β-mixing with mixing rate θ > 1. Let s be the estimator defined in (9) with a resampling penalty (8). Let

ǫ * n = (ln n) -1/4 ∧ r -1/8 n
. There exist constants C > 0 and κ > 0 such that

P s -s 2 2 > (1 + κǫ * n ) inf m∈Mn s -ŝA,m 2 2 ≤ C (log n) 2(θ+2) n θ/2 .
Comments:

• This result can be compared with Corollary 3.1 in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF]. In the independent case, the rate of convergence of the leading constant is always given by (r n ) -1/4 and this rate is often polynomial in n. It is not faster than (ln n) -1/4 in the β-mixing case.

The deviation probability was upper bounded by Ce -1 2 (ln n) γ for some constants C > 0 and γ > 1, it is now polynomial in n.

• [M 5 ] is hard to check in general. Let c -1 d = 2 sup x≥1 (ln x) 8 x -1 . [M 5 ] is satisfied for example, if there is no model in M n that have dimension d m ≤ c d (ln n) 8 and if [M 4 ] is satisfied. In this case, [M 5
] holds with r n = (ln n) 2 and we deduce from our previous computations the following result.

Corollary 4.2 Let M n be a collection of models satisfying [M 1 ]-[M 4 ].
Assume that the process (X n ) n∈Z is strictly stationary and arithmetically [AR(θ)] β-mixing with mixing rate θ > 1. Let s be the estimator defined in (9) with a resampling penalty (8). Then, there exist constants κ > 0, C > 0 such that, with probability larger than 1-Cn -θ/2 (log n) 2(θ+2) ,

s -s 2 2 ≤ 1 + κ (ln n) 1/4 inf m∈Mn,dm≥c d (ln n) 8 s -ŝA,m 2 2 .
Comments: Corollary 4.2 can be compared with Theorem 3.1 in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF].

• Both procedures lead to trajectorial oracle inequalities of type (3).

• The penalty term in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF] depends on a constant c D , which is in general unknown.

On the other hand, in Corollary 4.2, the selection algorithm (X 1 , ...., X n ) → s is totally computable.

ǫ * n = r -1/8 n ∨ n -1/4 (ln n) 1+γ/2 and θ 1 = θ ∧ 1. There exist constants C > 0 and κ > 0 such that P s -s 2 2 > (1 + κǫ * n ) inf m∈Mn s -ŝA,m 2 2 ≤ C n (ln n) 2 e -θ 1 2 (ln n) 2
Comments : Under the stronger assumption that the process is geometrically β-mixing, we almost recover the same results as in the independent case. The rate of convergence is now essentially given by r -1/8 n (it was r -1/4 n in the independent case) and the deviation probability is upper bounded by Cn(ln n) -2 e -θ 1 2 (ln n) 2 (instead of Ce -1 2 (ln n) 2 in the independent case).

τ -mixing processes

Our results for τ -mixing processes do not apply to general collections of models as mentioned before. We give in this section a classical collection where they might be used.

Dyadic Wavelet spaces:

This collection was the one of [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF]. Wavelet spaces are classically considered because the oracle is adaptive over Besov spaces (see for example Birgé & Massart [START_REF] Birgé | From model selection to adaptive estimation[END_REF] or [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF]). Hereafter, r is a real number, r ≥ 1 and we work with an r-regular orthonormal multiresolution analysis of L 2 (µ), associated with a compactly supported scaling function φ and a compactly supported mother wavelet ψ. Without loss of generality, we suppose that the support of the functions φ and ψ is included in an interval [A 1 , A 2 ) where A 1 and A 2 are integers such that A 2 -A 1 = A ≥ 1. For all functions t in L 2 (µ), we denote by t BV its bounded variation semi-norm, that is

t BV = sup l∈N * sup -∞<a 1 <...<a l <+∞ l-1 j=1 |t(a j+1 ) -t(a j )|.
For all k in Z and j in N * , let ψ 0,k : x → √ 2φ(2xk) and ψ j,k : x → 2 j/2 ψ(2 j xk). The family {(ψ j,k ) j≥0,k∈Z } is an orthonormal basis of L 2 (µ). Let us recall the following inequalities: let

K ∞ = ( √ 2 φ ∞ ) ∨ ψ ∞ , K L = (2 √ 2Lip(φ)) ∨ Lip(ψ), K BV = AK L . Then for all j ≥ 0, we have ψ j,k ∞ ≤ K ∞ 2 j/2 , k∈Z |ψ j,k | ∞ ≤ AK ∞ 2 j/2 (20) Lip(ψ j,k ) ≤ K L 2 3j/2 , ( 21
)
ψ j,k BV ≤ K BV 2 j/2 , . (22)
We assume that M n is the following collection.

[W] dyadic wavelet generated spaces: let

J n = [ln(n)/ ln(2)], for all J m = 1, ..., J n , let m = {(j, k), 0 ≤ j ≤ J m , k ∈ Z}
and let S m be the linear span of {ψ j,k } (j,k)∈m .

Hereafter, u denotes the following real number

u = 3 1 + θ ∧ 1.
As in the previous section, we add extra assumptions to prove [V'].

[T4] There exists a constant c ′ D > 0 such that, for all n ∈ N * , for all m in M n ,

D A,m ≥ c ′ D 2 Jm .
[T5] There exist a sequence r n → ∞ and a constant γ > 1 such that,

R n (ln n) -2γ 1-u ≥ r n .
As in the β-mixing case, we deduce the following corollary.

Corollary

= ln n ∧ r 1-u n -1/4 .
There exist constants C > 0 and κ > 0 such that

E s -s 2 2 ≤ (1 + κǫ * n )E inf m∈Mn s -ŝA,m 2 2 + C (ln n) 2(1+θ) n (θ-3)/2 .

Comments:

• With a mixing rate θ > 5, the estimator selected by a resampling penalty satisfies an oracle inequality (4). This result can be compared with Corollary 4.1. When the data are τ -mixing, we do not obtain a trajectorial oracle inequality (3) and the condition on the mixing rate is stronger than in the β-mixing case. However, as mentioned in the introduction, this result is very interesting because there is a lot of examples of processes that are τ -mixing and not β-mixing.

• Assumption [T5] is hard to check in practice but it can be removed as in the βmixing, provided that we only consider models with dimension larger than c M (ln n) η for some well chosen constants c M and η.

• We can get better rates of convergence if we assume that the process is geometrically τ -mixing and if we choose p and q as in Corollary 4.3.

This result can also be compared with Theorem 4.1 in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF].

• As in the β-mixing case, the main improvement of Corollary 4.4 is that the new procedure is totally data driven.

• The risk of s is compared with the oracle in Corollary 4.4 whereas it is compared with an upper bound on inf m∈Mn s -

s m 2 + 2E s m -ŝA,m 2 in [Ler09a].
• [T4] was not necessary in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF] but our new procedure improves the one given in [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF] every time that [T4] or any other Assumption ensuring [V'] holds.

Proofs

Notations

Let us give some notations that we will use repeatedly all along the proofs.

Recall that p and q are integers such that 2pq = n. For all k = 0, ..., p -1, I k = (2kq + 1, ..., (2k + 1)q), A k = (X i ) i∈I k and I = ∪ p-1 k=0 I k . For all functions t in L 2 (µ) and all x 1 , ..., x q in R,

L q (t)(x 1 , ..., x q ) = 1 q q i=1 t(x i ), P A t = 1 p p-1 k=0 L q (t)(A k ) = 2 n i∈I t(X i ), ν A (t) = (P A -P )(t).
The estimator ŝA,m associated to the model S m , is defined as ŝA,m ∈ arg min

t∈Sm P A Q(t).
For all m, m ′ in M n , let

T m = λ∈m (L q (ψ λ ) -P ψ λ ) 2 , U m = 1 p(p -1) p-1 i =j=0 λ∈m (L q (ψ λ )(A i ) -P ψ λ )(L q (ψ λ )(A j ) -P ψ λ ), p(m) = s m -ŝA,m 2 = sup t∈Bm (ν A (t)) 2 = λ∈m (ν A (ψ λ )) 2 . p W (m) = 1 v 2 W λ∈m E W (ν W A (ψ λ )) 2 , δ(m, m ′ ) = 2ν A (s m -s m ′ ). Lemma 6.2 applied with n = p, Λ = m, t λ = L q (ψ λ ), X i = A i-1 , gives p W (m) = 1 p (P A (T m ) -U m ) (23) p(m) -p W (m) = U m , (24)
where P A (T m ) = p-1 k=0 T m (A k )/p. For all functional T = F (A 0 , ..., A p-1 ), let T * = F (A * 0 , ..., A * p-1 ), where the random variables (A * k ) are given by the coupling Lemmas given in Section 2.3. In particular, we will use repeatedly the notations P * A , ν * A , U * m , p * (m), p * W (m), δ * (m, m ′ ). For all functions t of L 2 (µ), for all r in N and all x 1 , ..., x r , y 1 , ...y r in R,

|L r (t)(x 1 , ..., x r ) -L r (t)(y 1 , ..., y r )| ≤ 1 r r i=1 |t(x i ) -t(y i )| ≤ 1 r Lip(t)d r ((x 1 , ..., x r ), (y 1 , ..., y r )).
Thus, for all r in N * , Lip r (L r (t)) ≤ Lip(t)/r. For all k ∈ N,

M k n = {m ∈ M n , R A,m ∈ [k, k + 1)
} and for all n in N and, for all k > 0, k ′ > 0 and γ ≥ 0, let

l n,γ (k, k ′ ) = ln (1 + Card(M [k] n ))(1 + Card(M [k ′ ] n ))(k + 1)(k ′ + 1) + (ln n) γ . For all m, m ′ in M n , let l m,m ′ = l n,γ (R A,m , R A,m ′ ).
From Lemma 6.1 applied with α = α ′ = 0, for all K > 1, there exists a constant C > 0 such that

(m,m ′ )∈(Mn) 2 e -Kl m,m ′ = Ce -K(ln n) γ . Under [V'], sup (m,m ′ )∈(Mn) 2      v 2 A,m,m ′ R A,m ∨ R A,m ′ 2 ∨ e A,m,m ′ R A,m ∨ R A,m ′   l 2 m,m ′    ≤ ǫ 4 n .

Technical Lemmas

Lemmas 5.1 and 5.2 are coupling lemmas. They allow to work with p * (m), p * W (m), δ * (m, m ′ ) instead of p(m), p W (m), δ(m, m ′ ). Lemma 5.3 is a consequence of our study of the independent case. It allows to extend the proofs of [START_REF] Lerasle | Optimal model selection in density estimation[END_REF] to the mixing case. It is the main tool of this paper.

Lemma 5.1 Let X 1 , ..., X n be stationary random variables, real valued and β-mixing. Let p and q be two integers such that 2pq = n and let A * 0 , ..., A * p-1 be the random variables given by Viennet's Lemma in Section 2.3.1. Let (S m ) m∈Mn be a collection of linear spaces of functions. Let (p(m)) m∈Mn , (p

W (m)) m∈Mn , (δ(m, m ′ )) (m,m ′ )∈(Mn) 2 , (p * (m)) m∈Mn , (p * W (m)) m∈Mn , (δ * (m, m ′ )) (m,m ′ )∈(Mn)
2 , be the associated collections defined in Section 5.1. There exists an event Ω C such that P(Ω c C ) ≤ pβ q and such that, on Ω C , for all m, m ′ in M n , we have

p(m) = p * (m), p W (m) = p * W (m), δ(m, m ′ ) = δ * (m, m ′ ). ( 25
)
Proof :

Let Ω C = {∀k = 0, ..., p -1, A k = A * k }.
It comes from Viennet's Lemma that P(Ω c C ) ≤ pβ q and it is clear that, on Ω C , (25) holds.

Lemma 5.2 Let X 1 , ..., X n be stationary random variables, real valued, τ -mixing and with common density s. Let p and q be two integers such that 2pq = n and let A * 0 , ..., A * p-1 be the random variables given by the τ -coupling's Lemma in Section 2.3.2. Let M n be a collection of models. Let

(p(m)) m∈Mn , (p W (m)) m∈Mn , (δ(m, m ′ )) (m,m ′ )∈(Mn) 2 , (p * (m)) m∈Mn , (p * W (m)) m∈Mn , (δ * (m, m ′ )) (m,m ′ )∈(Mn)
2 , be the associated collections defined in Section 5.1. Let M C n be the mixing complexity of M n defined by

M C n = m∈Mn   λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ) + s |M n | sup t∈Bm Lip(t)   . For all m, m ′ in M n , E sup m∈Mn |p(m) -p * (m)| ≤ 4τ q M C n (26) E sup m∈Mn |p W (m) -p * W (m)| ≤ 8τ q p M C n (27) E sup m,m ′ ∈Mn δ(m, m ′ ) -δ * (m, m ′ ) ≤ 4τ q M C n . (28)
Proof : For all m in M n , we have

E sup m∈Mn |p(m) -p * (m)| ≤ m∈Mn E (|p(m) -p * (m)|) .
Moreover, for all m in M n ,

|p(m) -p * (m)| = λ∈m ((P A -P )ψ λ ) 2 -((P * A -P )ψ λ ) 2 = λ∈m ((ν A + ν * A )ψ λ ) ((P A -P * A )ψ λ ) ≤ λ∈m |(ν A + ν * A )ψ λ | 1 p p-1 k=0 |L q (ψ λ )(A k ) -L q (ψ λ )(A * k )| ≤ 4 λ∈m |ψ λ | ∞ sup λ∈m Lip q (L q (ψ λ )) 1 p p-1 k=0 d q (A k , A * k ) ≤ 4 q λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ) 1 p p-1 k=0 d q (A k , A * k ).
We take the expectation in this last inequality and we use the τ -coupling Lemma of Section 2.3.2 to obtain (26). From (23), we have

|p W (m) -p * W (m)| = 1 p |(P A -P * A )(T m ) -(U m -U * m )| .
We have

(P A -P * A )T m = λ∈m 1 p p-1 k=0 (L q (ψ λ )(A k ) -L q (ψ λ )(A * k )) (L q (ψ λ )(A k ) + L q (ψ λ )(A * k ) -2P ψ λ ) , thus
|(P A -P * A )T m | = 4 λ∈m |ψ λ | ∞ sup λ∈m Lip q (L q (ψ λ )) 1 p p-1 k=0 d q (A k , A * k ) ≤ 4 q λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ) 1 p p-1 k=0 d q (A k , A * k ). Moreover U m -U * m = 1 p(p -1) p-1 i =j=0 λ∈m (L q (ψ λ (A j )) -P ψ λ )(L q (ψ λ (A i )) -L q (ψ λ (A * i ))) + 1 p(p -1) p-1 i =j=0 λ∈m (L q (ψ λ (A * i )) -P ψ λ )(L q (ψ λ (A j )) -L q (ψ λ (A * j ))), thus |U m -U * m | ≤ 4 q λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ) 1 p p-1 k=0 d q (A k , A * k ).
Therefore,

E (|p W (m) -p * W (m)|) ≤ 8 pq λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ) 1 p p-1 k=0 E(d q (A k , A * k )) ≤ 8τ q p λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ). Thus E sup m∈Mn |p W (m) -p * W (m)| ≤ m∈Mn E (|p W (m) -p * W (m)|) ≤ 8τ q p m∈Mn λ∈m |ψ λ | ∞ sup λ∈m Lip(ψ λ ).
Finally,

E sup m,m ′ ∈Mn δ(m, m ′ ) -δ * (m, m ′ ) ≤ m,m ′ ∈Mn E |δ(m, m ′ ) -δ * (m, m ′ )| and, for all m, m ′ in M n , E |δ(m, m ′ ) -δ * (m, m ′ )| = 2E (|(P A -P * A )(s m -s m ′ )|) ≤ 2 pq Lip(s m -s m ′ ) p-1 k=0 E (d q (A k , A * k )) ≤ 2τ q Lip(s m -s m ′ ).
For all x, y in R and all m, m ′ in M n ,

(s m -s m ′ )(x) -(s m -s m ′ )(y) ≤ s sup t∈Bm Lip(t) + sup t∈B m ′ Lip(t) d(x, y) Hence, Lip(s m -s m ′ ) ≤ s sup t∈Bm Lip(t) + sup t∈B m ′ Lip(t) , thus E sup m,m ′ ∈Mn δ(m, m ′ ) -δ * (m, m ′ ) ≤ 4τ q s |M n | m∈Mn sup t∈Bm Lip(t).
Let us now derive some consequences of the results of [START_REF] Lerasle | Optimal model selection in density estimation[END_REF].

Lemma 5.3 Let A * 0 , ...,A * p-1 be i.i.d random variables valued in R q , with 2pq = n. Let M n be a collection of models satisfying [V'] and let (p * (m)) m∈Mn , (p * W (m)) m∈Mn , (δ * (m, m ′ )) (m,m ′ )∈(Mn) 2 , (D A,m ) m∈Mn , (R A,m ) m∈Mn be the associated collections defined in Section 5.1. There exists a constant C > 0 such that

P m∈Mn p * (m) - 2D A,m n > 15ǫ n R A,m n ≤ Ce -1 2 (ln n) γ , (29)
P m∈Mn p * (m) - 2D A,m n < -25ǫ n R A,m n ≤ Ce -1 2 (ln n) γ . ( 30
) P m∈Mn p * (m) -p * W (m) > 15ǫ n R A,m n ≤ Ce -1 2 (ln n) γ , (31)
P m∈Mn p * (m) -p * W (m) < -25ǫ n R A,m n ≤ Ce -1 2 (ln n) γ . ( 32
) P   m,m ′ ∈Mn δ * (m, m ′ ) > 12ǫ n R A,m ∨ R A,m ′ n   ≤ Ce -(ln n) γ . ( 33
)
There exists an absolute constant C > 0 such that

E sup m∈Mn p * (m) - 2D A,m n -15ǫ n R A,m n + ≤ Ce -1 2 (ln n) γ . ( 34
) E sup m∈Mn -p * (m) + 2D A,m n -35ǫ n R A,m n + > ≤ Ce -1 2 (ln n) γ . ( 35
) E sup m∈Mn p * (m) -p * W (m) -20ǫ n R A,m n + ≤ Ce -1 2 (ln n) γ . ( 36
) E sup m∈Mn -p * (m) + p * W (m) -35ǫ n R A,m n + ≤ Ce -1 2 (ln n) γ . ( 37
) E sup m,m ′ ∈Mn δ * (m, m ′ ) -20ǫ n R A,m ∨ R A,m ′ n + ≤ Ce -1 2 (ln n) γ . ( 38
)
Proof of the concentration inequalities : p * (m) = sup t∈Bm ((ν * A )(t)) 2 and A * 0 , ..., A * p-1 are independent. Thus

E(p * (m)) = λ∈m Var (L q (ψ λ )(A 0 )) p = 2D A,m n , sup t∈Bm Var (L q (t)(A 0 )) = v 2 A,m q , sup t∈Bm L q (t) 2 ∞ p ≤ e A,m q .
We apply Proposition 6.3 in the Appendix with

B = {L q (t), t ∈ B m }, D = D A,m /q, v 2 = v 2
A,m /q, ǫ = e A,m /q and n = p. For all x > 0 and all m in M n , with probability larger than 1e

-x p * (m) - 2D A,m n ≤ 2D 3/4 A,m (e A,m (19x) 2 ) 1/4 + 6 D A,m v 2 A,m x + 6v 2 A,m x + 2e A,m (19x) 2 n and, with probability larger than 1 -2.8e -x 2D A,m n -p * (m) ≤ 16D 3/4 A,m (e A,m x 2 ) 1/4 + 15.22 D A,m v 2 A,m x + 2e A,m (40.25x) 2 n . (39)
Let K > 0 be a constant to be chosen later, let l m = l n,γ (R m , R m ), and let

x = K 2 l m . From [V'] applied with m = m ′ , since D A,m ≤ R A,m , v 2 A,m x ≤ (Kǫ n ) 2 R A,m , e A,m x 2 ≤ (Kǫ n ) 4 R A,m , D 3/4 A,m (e A,m x 2 ) 1/4 ≤ Kǫ n R A,m , D A,m v 2 A,m x ≤ Kǫ n R m . (40) Let e n (K) = (2 √ 19 + 6)K + 6K 2 ǫ n + 2(19) 2 K 4 ǫ 3 n , from (40), 2D 3/4 A,m (e A,m (19x) 2 ) 1/4 + 6 D A,m v 2 A,m x + 6v 2 A,m x + 2e A,m (19x) 2 n ≤ e n (K)ǫ n R m n .
Thus, from Lemma 6.1, for all K > 1/ √ 2, there exists a constant C > 0 such that

P m∈Mn 2D A,m n -p * (m) > e n (K)ǫ n R A,m n ≤ m∈Mn P p * (m) - 2D A,m n > e n (K)ǫ n R A,m n ≤ m∈Mn e -K 2 lm ≤ Ce -K 2 (ln n) γ . Let K = 11/(2 √ 19 + 6) > 1/ √ 2
n (K) = 31, 22K + 2(40, 25) 2 K 4 ǫ 3 n , from (40), 16D 3/4 A,m (e A,m x 2 ) 1/4 + 15.22 D A,m v 2 A,m x + 2e A,m (40.25x) 2 n ≤ e (2) n (K)ǫ n R A,m n .
We apply inequality (39) with x = K 2 l m . For all K > 1/ √ 2, there exists a constant C > 0 such that

P m∈Mn p * (m) - 2D A,m n < -e (2) n (K)ǫ n R A,m n ≤ m∈Mn P p * (m) - 2D A,m n < -e (2) n (K)ǫ n R A,m n ≤ 2.8 m∈Mn e -K 2 lm ≤ Ce -K 2 (ln n) γ .
Take K = 23/31.22 > 1/ √ 2 and n sufficiently large to have 2(40, 25) 2 K 4 ǫ 3 n ≤ 2, then e

(2) n (K) ≤ 25 and (30) holds for sufficiently large n. It holds then in general, provided that we enlarge the constant C if necessary. From (24), p * (m)p * W (m) = U * m . Therefore, from Lemma 6.4 in the appendix, for all m in M n and all x > 0, with probability larger than 1 -2e -x ,

p * (m) -p * W (m) ≤ 10.62D 3/4 A,m (e A,m x 2 ) 1/4 + 6 v 2 A,m D A,m x + 6v 2 A,m x + 2e A,m (19.1x) 2 n -1 , (41
) and, with probability larger than 1 -3.8e -x ,

p * W (m) -p * (m) > 18D 3/4 A,m (e A,m x 2 ) 1/4 + 15.22 v 2 A,m D A,m x + 2e A,m (40.3x) 2 n -1 . ( 42
) Let K > 0, e (3)
n (K) = (16.62K + 6K 2 ǫ n + 2(19.1) 2 K 4 ǫ 3 n )n/(n -1) and x = K 2 l m , from (40), 10.62D 3/4 A,m (e A,m x 2 ) 1/4 + 6 v 2 A,m D A,m x + 6v 2 A,m x + 2e A,m (19.1x) 2 n -1 ≤ e (3) n (K)ǫ n R A,m n .
We apply (41) with x = K 2 l m . From Lemma 6.1, for all K > 1/ √ 2, there exists a constant C such that

P m∈Mn p * (m) -p * W (m) > e (3) n (K)ǫ n R A,m n ≤ m∈Mn P p * (m) -p * W (m) > e (3) n (K)ǫ n R A,m n ≤ 2 m∈Mn e -K 2 lm ≤ Ce -K 2 (ln n) γ .
Take K = 12/16.62 > 1/ √ 2 and n ≥ 15 such that 6K 2 ǫ n + 2(19.1) 2 K 4 ǫ 3 n ≤ 2, then e

(3) n (K) ≤ 15 and (31) holds for sufficiently large n. It holds in general provided that we enlarge C if necessary. Let K > 0, e (4)

n (K) = (33.22K + 2(40.3) 2 K 4 ǫ 3 n )n/(n -1) and x = K 2 l m . From (40), 18D 3/4 A,m (e A,m x 2 ) 1/4 + 15.22 v 2 A,m D A,m x + 2e A,m (40.3x) 2 n -1 ≤ e (4) n (K)ǫ n R A,m n .
We apply (41) with x = K 2 l m . From Lemma 6.1, for all K > 1/ √ 2, there exists a constant C such that

P m∈Mn p * W (m) -p * (m) > e (4) n (K)ǫ n R A,m n ≤ m∈Mn P p * W (m) -p * (m) > e (4) n (K)ǫ n R A,m n ≤ 3.8 m∈Mn e -K 2 lm ≤ Ce -K 2 (ln n) γ .
Take K = 23.5/33.22 > 1/ √ 2 and n ≥ 25 such that 2(40.3) 2 K 4 ǫ 3 n ≤ 0.5, then e (4) n (K) ≤ 25 and (32) holds for sufficiently large n. It holds in general provided that we enlarge C if necessary. Finally, we apply Lemma 6.5 in the appendix to the functions s ms m ′ , with L = L q and ν n = ν A , we have v 2 ≤ v 2 A,m,m ′ /q and ǫ ≤ e A,m,m ′ /q. For all m, m ′ in M n ,

s m -s m ′ 2 ≤ 2( s m -s 2 + s m ′ -s 2 ) ≤ 4 R A,m ∨ R A,m ′ n ,
thus, for all η > 0, for all x > 0,

P δ * (m, m ′ ) > 4η R A,m ∨ R A,m ′ n + 8v 2 A,m,m ′ x + 4e A,m,m ′ x 2 /9 ηn ≤ e -x . ( 43
) Let K > 0, l m,m ′ = l n,γ (R A,m , R A,m ′ ), x = K 2 l m,m ′ and e (5) n (K) = 2K 2 + K 4 ǫ 2 n /9. From (40), 8v 2 A,m,m ′ x + 4e A,m,m ′ x 2 /9 ≤ 4(e (5) n (K))ǫ n ) 2 R A,m ∨ R A,m ′ ,
thus, for η = e

(5)

n (K))ǫ n , 4η R A,m ∨ R A,m ′ n + 8v 2 A,m,m ′ x + 4e A,m,m ′ x 2 /9 ηn ≤ 8e (5) n (K))ǫ n R A,m ∨ R A,m ′ n .
Hence, for all K > 1, there exists a constant C > 0 such that

P   m,m ′ ∈Mn δ * (m, m ′ ) > 8e (5) n (K))ǫ n R A,m ∨ R A,m ′ n   ≤ m,m ′ ∈Mn P δ * (m, m ′ ) > 8e (5) n (K))ǫ n R A,m ∨ R A,m ′ n ≤ m,m ′ ∈Mn e -K 2 l m,m ′ ≤ Ce -K 2 (ln n) γ .
Take K = 11.4/(8 √ 2) > 1 and n sufficiently large to have 8 K 4 ǫ 2 n /9 ≤ 0.6, then 8e

(5) n (K)) ≤ 12 and (33) holds for sufficiently large n. It holds in general provided that we increase C if necessary.

Proof of the results in expectation

Let K > 0, z > 0, l m = l n,γ (R m , R m ), x = K 2 l m (1 + z) and e (6) n (K, z) = (2 √ 19 + 6)K √ x + 6K 2 ǫ n x + 4(19) 2 K 4 ǫ 3 n x 2 . From (40), 2D 3/4 A,m (e A,m (19x) 2 ) 1/4 + 6 D A,m v 2 A,m x + 6v 2 A,m x + 2e A,m (19x) 2 n ≤ (e (6) n (K, 1) + e (6) n (K, z))ǫ n R A,m n .
Thus, from Proposition 6.3 in the Appendix, for all z > 0 and all m in M n ,

P p * (m) - 2D A,m n -e (6) n (K, 1)ǫ n R A,m n > e (6) n (K, z))ǫ n R A,m n ≤ e -K 2 lm(1+z) .
Let us now briefly explain how to deduce from this concentration inequalities the results in expectation.

[MI]: Integration of the concentration inequality

Let ǫ m = ǫ n R A,m /n and f (m) = p * (m) -2D A,m /n -e (6)
n (K, 1)ǫ m , we have

E sup m∈Mn (f (m)) + ≤ m∈Mn E (f (m)) + = m∈Mn ∞ 0 P (f (m) > y) dy. Since z → g(z) = e (6) n (K, z)) is clearly a C 1 -diffeomorphism of R * + , this last integral is equal to ∞ 0 P (f (m) > ǫ m g(y)) ǫ m g ′ (y)dy
For all K > 0, there exists a constant C > 0 such that g ′ (z) ≤ C(z -1/2 + 1 + z). From Lemma 6.1, for all K > 1, n ≥ 2, there exists a constant C > 0 such that

E sup m∈Mn p * (m) - 2D A,m n -e (6) n (K, 1)ǫ n R A,m n + ≤ C m∈Mn ǫ n R A,m e -K 2 lm ∞ 0 (z -1/2 + 1 + z)e -K 2 lmz dz ≤ Ce -K 2 (ln n) γ .
The last inequality comes from the fact that ǫ n is bounded and K 2 l m ≥ c > 0 for all n ≥ 2, K > 1. Take K = 14.75/(2 √ 19 + 6) > 1 and choose n sufficiently large such that 6K 2 ǫ n + 4(19) 2 K 4 ǫ 3 n ≤ 0.25, then e

n (K) ≤ 15 and (34) holds for all n sufficiently large. It holds for all n provided that we enlarge C if necessary. We obtain (35) with the same arguments. Let us now turn to the result on the resampling estimator of p(m).

Let K > 0, z > 0, l m = l n,γ (R m , R m ), x = K 2 l m (1 + z), e (7) n (K, z) = n n -1 16, 62K √ x + 6K 2 ǫ n x + 4(19.1) 2 K 4 ǫ 2 n x 2 ,
From inequalities (40), we have 10.62D

3/4 A,m (e A,m x 2 ) 1/4 + 6 v 2 A,m D A,m x + 6v 2 A,m x + 2e A,m (19.1x) 2 n -1 ≤ (e (7) n (K, 1) + e (7) n (K, z))ǫ n R A,m n
From inequalities (41) with x = K 2 l m (1 + z) and for all z > 0, for all m in M n and all z > 0

P p * (m) -p * W (m) -e (7) n (K, 1)ǫ n R A,m n > e (7) n (K, z)ǫ n R A,m n ≤ 2e -K 2 lm(1+z) .
We use again the method of integration [MI] to prove that, for all K > 1, there exists a constant C > 0 such that

E sup m∈Mn p * (m) -p * W (m) -e (7) n (K, 1)ǫ n R A,m n + ≤ Ce -K 2 (ln n) γ .
Take K = 17/16.62 > 1 and n ≥ 20 such that 6K 2 ǫ n +4(19.1) 2 K 4 ǫ 3 n ≤ 2, then e

n (K, 1) ≤ 20 and (36) holds for sufficiently large n. It holds in general provided that we enlarge C if necessary. We obtain (37) with the same arguments.

Let K > 0, l m,m ′ = l n,γ (R m , R m ′ ), z > 0, x = K 2 l m,m ′ (1 + z), e (8) n (K, z) = 2K 2 z + 2K 4 ǫ 2 n x 2 /9, e (8)
n (K) = e (8) n (K, 1), g K (z) = (e (8)
n (K, z)) 2 /e (8)
n (K) and η = e (8)

n (K, 1)ǫ n . 4η R A,m ∨ R A,m ′ n + 8v 2 A,m,m ′ x + 4e A,m,m ′ x 2 /9 ηn ≤ 4 2e (8) n (K) + g K (z) ǫ n R A,m ∨ R A,m ′ n .
Thus from (43), for all z > 0, for all m, m ′ in M n and all K > 0, 1+z) .

P δ(m, m ′ ) -8e (8) n (K)ǫ n R A,m ∨ R A,m ′ n > ǫ n R A,m ∨ R A,m ′ n g K (z) ≤ e -K 2 l m,m ′ (
Thus E sup (m,m ′ )∈M 2 n δ * (m, m ′ ) -8e (8) n (K)ǫ n R A,m ∨ R A,m ′ n + ≤ (m,m ′ )∈M 2 n E δ * (m, m ′ ) -8e (8) n (K)ǫ n R A,m ∨ R A,m ′ n + = (m,m ′ )∈M 2 n ∞ 0 P δ * (m, m ′ ) -8e (8) n (K)ǫ n R A,m ∨ R A,m ′ n > x dx Let x = ǫ n R A,m ∨R A,m ′ n g K (z)
. For all K > 0, for all n ≥ 2, there exists a constant C > 0 such that g K (z) ′ ≤ C(1 + z). Thus, from Lemma 6.1, for all K > √ 2, there exists a constant C > 0 such that

E sup (m,m ′ )∈M 2 n δ * (m, m ′ ) -8e (8) n (K)ǫ n R A,m ∨ R A,m ′ n + ≤ C (m,m ′ )∈M 2 n ǫ n R A,m ∨ R A,m ′ n e -K 2 l m,m ′ ∞ 0 (1 + z)e -K 2 l m,m ′ z dz ≤ Ce -K 2 (ln n) γ .
Take K = 17/(8 √ 2) > √ 2 and n sufficiently large to have 8 2K 4 ǫ 2 n /9 ≤ 3, then 8e (8) n (K, 1)) ≤ 20 and (38) holds for sufficiently large n. It holds in general provided that we increase C if necessary. We can now turn to the proofs of the main results of this part.

Proof of Theorem 3.1

Let us first assume that X 1 , ..., X n are β-mixing. Let A * 0 , ..., A * p-1 be the random variables given by Viennet's Lemma in Section 2.3.1. Let (p(m), p W (m), δ(m, m ′ ), p * (m), p * W (m), D A,m , R A,m , δ * (m, m ′ )) (m,m ′ )∈(Mn) 2 be the quantities defined in Section 5.1. Let us define the events

Ω p = m∈Mn -25ǫ n R A,m n ≤ p * (m) - 2D A,m n ≤ 15ǫ n R A,m n , (44) Ωp
= m∈Mn -25ǫ n R A,m n ≤ p * (m) -p * W (m) ≤ 15ǫ n R A,m n Ω d = (m,m ′ )∈Mn δ(m, m ′ ) ≤ 12ǫ n R A,m ∨ R A,m ′ n , (45)
Ω C = m∈Mn {p(m) = p * (m)} ∩ m∈Mn {p W (m) = p * W (m)} ∩   (m,m ′ )∈Mn δ(m, m ′ ) = δ * (m, m ′ )   . (46)
From Lemmas 5.1 and 5.3, there exists a constant C > 0 such that

P(Ω c p ) ≤ Ce -1 2 (ln n) γ , P( Ωc p ) ≤ Ce -1 2 (ln n) γ , P(Ω c d ) ≤ Ce -(ln n) γ , P(Ω c C ) ≤ pβ q . Let Ω = Ω p ∩ Ωp ∩ Ω d ∩ Ω C . Recall that pen(m) = 2p W (m). On Ω, from inequality (6), for all m in M n , s -s 2 ≤ s -ŝA,m 2 + 2 (p W (m) -p(m)) -2 (p W ( m) -p( m)) + δ(m, m) = s -ŝA,m 2 + 2 (p * W (m) -p * (m)) -2 (p * W ( m) -p * ( m)) + δ * (m, m) ≤ s -ŝA,m 2 + 62ǫ n R A,m n + 42ǫ n R A, m n .
On Ω,

R A,m n = s -ŝA,m 2 + 2D A,m n -p * (m) ≤ s -ŝA,m 2 + 25ǫ n R A,m n . If 25ǫ n < 1, on Ω, s -s 2 ≤ 1 + 37ǫ n 1 -25ǫ n s -ŝA,m 2 + 42ǫ n 1 -25ǫ n s -s 2 .
Hence, if 67ǫ n < 1,

P s -s 2 > 1 + 37ǫ n 1 -67ǫ n inf m∈Mn s -ŝA,m 2 ≤ Ce -1 2 (ln n) γ + pβ q .
Take n sufficiently large to have 67ǫ n < 1 and 104/(1 -67ǫ n ) ≤ 110. Then,

1 + 37ǫ n 1 -67ǫ n = 1 + 104 1 -67ǫ n ǫ n ≤ 1 + 110ǫ n
and (11) holds for sufficiently large n. It holds in general provided that we increase the constant C if necessary.

Proof of Theorem 3.2

Let us now assume that X 1 , ..., X n are τ -mixing. Let A * 0 , ..., A * p-1 be the random variables given by the τ -couling Lemma in Section 2.3.2. Let (p(m), p W (m), δ(m, m ′ ), p * (m),

p * W (m), D A,m , R A,m , δ * (m, m ′ )) (m,m ′ )∈(
Mn) 2 be the quantities defined in Section 5.1. From inequality (6), for all m in M n ,

s -s 2 ≤ s -ŝA,m 2 + 2 (p W (m) -p(m)) -2 (p W ( m) -p( m)) + δ(m, m) = s -ŝA,m 2 + 2 p * W (m) -p * (m) -35ǫ n R A,m n + 90ǫ n R A,m n +2 p * ( m) -p * W ( m) -20ǫ n R A, m n + 60ǫ n R A, m n +δ * (m, m) -20ǫ n R A,m ∨ R A, m n + 2(p W (m) -p * W (m)) +2(p * (m) -p(m) + p * W ( m) -p W ( m) + p( m) -p * ( m)) +δ(m, m) -δ * (m, m). For all m in M n , R A,m n = s -ŝA,m 2 1 -35ǫ n + (1 -35ǫ n )R A,m /n -s -ŝA,m 2 1 -35ǫ n = s -ŝA,m 2 1 -35ǫ n + 2D A,m /n -35ǫ n R A,m /n -s m -ŝA,m 2 1 -35ǫ n . (47)
In the control of ss 2 , we replace R A,m /n and R A, m/n by the expressions obtained in (47) in the terms 90ǫ n R A,m /n and 60ǫ n R A, m/n. Assume that 35ǫ n < 1,

1 -95ǫ n 1 -35ǫ n s -s 2 ≤ 1 + 55ǫ n 1 -35ǫ n inf m∈Mn s -ŝA,m 2 + 150ǫ n 1 -35ǫ n sup m∈Mn 2D A,m n -p * (m) -35ǫ n R m n + 4 + 10ǫ n 1 -35ǫ n sup m∈Mn |p * (m) -p(m))| + sup m,m ′ ∈Mn δ * (m, m ′ ) -20ǫ n R A,m ∨ R A,m ′ n +2 sup m∈Mn p * W (m) -p * (m) -35ǫ n R m n + 4 sup m∈Mn |p W (m) -p * W (m)| +2 sup m∈Mn (p * (m) -p * W (m) -15ǫ n f racR m n) + sup m,m ′ ∈(Mn) 2 δ(m, m ′ ) -δ * (m, m ′ ).
We take the expectation in this last inequality and we use inequalities ( 26), ( 27), ( 28), ( 35), ( 36), ( 37) and ( 38) to obtain that, when 95ǫ n < 1, there exists a constant C > 0 such that

s -s 2 ≤ 1 + 55ǫ n 1 -95ǫ n inf m∈Mn s -ŝA,m 2 + C τ q M C n + e -1 2 (ln n) γ
Take n sufficiently large to have 95ǫ n < 1 and 150/(1 -95ǫ n ) ≤ 160. Then,

1 + 55ǫ n 1 -95ǫ n = 1 + 150 1 -95ǫ n ǫ n ≤ 1 + 160ǫ n
and (12) holds for sufficiently large n. It holds in general provided that we increase the constant C if necessary.

Proof of Theorem 3.3

Assume that X 1 , ..., X n are β-mixing and let A * 0 , ..., A * p-1 be the random variables built with Viennet's Lemma in Section 2.3.1. Let (p(m), p W (m), δ(m, m ′ ), p * (m), p * W (m), D A,m , R A,m , δ * (m, m ′ )) (m,m ′ )∈(Mn) 2 be the quantities defined in Section 5.1. Let Ω T = Ω p ∩ Ω d ∩ Ω C where Ω p , Ω d and Ω C are defined respectively in (44), ( 45) and (46). Recall that there exists a constant C > 0 such that

P(Ω c p ) ≤ Ce -1 2 (ln n) γ , P(Ω c d ) ≤ Ce -(ln n) γ , P(Ω c C ) ≤ pβ q .
If c n ≤ 0, there is nothing to prove, hence, we can assume that c n > 0 and thus that 75ǫ * n < δ < 1. m minimizes by definition the following criterion

Crit(m) = ŝA,m 2 -2P A (ŝ A,m ) + pen(m) + s 2 + 2ν A (s mo ) = ŝA,m 2 -2P (ŝ A,m ) + s 2 -2ν A (ŝ A,m ) + 2ν A (s mo ) + pen(m) = ŝA,m -s 2 -2ν A (ŝ A,m -s m ) + 2ν A (s mo -s m ) + pen(m) = ŝA,m -s 2 -2 ŝA,m -s m 2 + δ(m o , m) + pen(m) = s -s m 2 -p(m) + δ(m o , m) + pen(m) since p(m) = s m -ŝA,m 2 = ν A (ŝ A,m -s m ).
m in M n , on Ω T Crit(m) ≥ s -s m 2 - 2D A,m n + 2D A,m n -p * (m) + δ * (m, m o ) ≥ (1 -27ǫ n ) s -s m 2 -(1 + 27ǫ n ) 2D A,m n ≥ -(1 + 27ǫ n ) 2D A,m n Crit(m) ≤ s -s m 2 -(δ -74ǫ n ) D A,m n . If D A,m < c n D A,m * , then Crit(m) ≥ -(1 + 27ǫ n ) 2D A,m n > -(1 + 27ǫ n ) c n 2D A,m * n ≥ -(δ -74ǫ n -h * n ) D A,m * n ≥ Crit(m * ). Since Crit( m) ≤ Crit(m * ), D A, m ≥ c n D A,m * . It follows that, on Ω T , s -s 2 = R A, m n + p( m) - 2D A, m n ≥ (1 -25ǫ n ) R A, m n ≥ (1 -25ǫ n ) 2D A, m n ≥ (1 -25ǫ n )c n 2D A,m * n . Moreover, on Ω T , inf m∈Mn s -ŝA,m 2 ≤ inf m∈Mn R A,m n (1 + 15ǫ n ) ≤ R A,mo n (1 + 15ǫ n ). Thus s -s 2 ≥ (1 -25ǫ n )c n 2D A,m * n ≥ 2c n 1 -25ǫ n 1 + 15ǫ n D A,m * R A,mo inf m∈Mn s -ŝA,m 2 .
Since ǫ n < 1/75, we have 2(1 -25ǫ n )(1 + 15ǫ n ≥ 2(1 -1/3)(1 + 1/5) ≥ 1. This conclude the proof of (13).

Proof of Theorem 3.4

Assume that X 1 , ..., X n are τ -mixing. Let A * 0 , ..., A * p-1 be the random variables given by the τ -couling Lemma in Section 2.3.2. Let (p(m), p W (m), δ(m, m ′ ), p * (m), p * W (m), D A,m , R A,m , δ * (m, m ′ )) (m,m ′ )∈(Mn) 2 be the quantities defined in Section 5.1. In order to prove inequality (14) observe that, for all m in M n , since pen(m) ≥ 0, and

s -s m 2 - 35ǫ n R A,m /n ≥ -35ǫ n D A,m /n Crit(m) ≥ s -s m 2 + -p * (m) + 15ǫ n R A,m n + (p * (m) -p(m)) -35ǫ n R A,m n + δ * (m, m o ) + 20ǫ n R A,m n + (δ(m, m o ) -δ * (m, m o )) = -(1 + 35ǫ n ) 2D A,m n + 2D A,m n -p * (m) + 15ǫ n R A,m n + (p * (m) -p(m)) + δ * (m, m o ) + 20ǫ n R A,m n + (δ(m, m o ) -δ * (m, m o )).
Therefore,

- 2D A, m n (1 + 35ǫ n ) ≤ Crit( m) + sup m∈Mn p * (m) - 2D A,m n -15ǫ n R A,m n + sup (m,m ′ )∈M 2 n δ * (m, m ′ ) -20ǫ n R A,m ∨ R A,m ′ n + sup m∈Mn (p(m) -p * (m)) + sup m,m ′ ∈Mn (δ(m, m ′ ) -δ * (m, m ′ )). ( 48
)
Since, for all m in M n , pen(m)

≤ (2 -δ)D A,m /n, Crit(m) ≤ s -s m 2 + 2D A,m n -p * (m) -δ D A,m n + (p * (m) -p(m)) + δ(m, m o ). ( 49
)
Since Crit( m) ≤ Crit(m * ), from ( 49) and ( 26),

E (Crit( m)) ≤ E (Crit(m * )) ≤ s -s m * 2 -δ D A,m * n + 4τ q M C n ≤ -(δ -h * n ) D A,m * n + 4τ q M C n ≤ -c ′ n (1 + 35ǫ n ) 2D A,m * n + 4τ q M C n .
Take the expectation in (48) and use inequalities ( 26), ( 28), ( 34) and (38) to obtain (14). We deduce from ( 14) that there exists a constant C > 0 such that

E s -s 2 ≥ 2 n E (D A, m) ≥ 2c ′ n D A,m * n -C(e -1 2 (ln n) γ + τ q M C n ) ≥ 2 c ′ n h o n R mo n -C(e -1 2 (ln n) γ + τ q M C n ).
The proof of (15) is conclude since

E inf m∈Mn s -ŝA,m 2 ≤ inf m∈Mn E s -ŝA,m 2 = R mo n + Cτ q M C n .

Proof of Theorem 3.5

If c n = ∞, there is nothing to prove. Thus we can assume that c n < ∞ and thus that 1 + δ -27ǫ n > 0. Let us first assume that X 1 , ..., X n are β-mixing and let A * 0 , ...A * p-1 be the random variables given by Viennet's Lemma in Section 2.3.1. Let (p(m), p W (m), δ(m, m ′ ), p * (m), p * W (m), D A,m , R A,m , δ * (m, m ′ )) (m,m ′ )∈(Mn) 2 be the quantities defined in Section 5.1. Recall that m minimizes over M n the following criterion.

Crit(m) = s -s m 2 -p(m) + δ(m, m o ) + pen(m).
We keep the notations Ω p , Ω d and Ω C defined by ( 44), ( 45), (46). We introduce the event

Ω pen = m∈Mn 4D A,m n + δ R A,m n ≤ pen(m) ≤ 4D A,m n + δ R A,m n and let Ω = Ω p ∩ Ω d ∩ Ω C ∩ Ω pen . Since R A,mo ≤ R A,m , on Ω, Crit(m) ≥ (1 + δ -12ǫ n ) R A,m n + 2D A,m n -p * (m) ≥ (1 + δ -27ǫ n ) R A,m n ≥ (1 + δ -27ǫ n ) 2D A,m n . Crit(m) ≤ (1 + δ + 37ǫ n ) R A,m n . If D A,m > c n R A,mo , Crit(m) ≥ (1 + δ -27ǫ n ) 2D A,m n > 2(1 + δ -27ǫ n )c n R A,mo n ≥ (1 + δ + 37ǫ n ) R A,mo n ≥ Crit(m o ) Since Crit( m) ≤ Crit(m o ), this implies that D A, m ≤ c n R A,mo . Moreover, from (6), for all m in M n s -s 2 ≤ s -ŝA,m 2 + (pen(m) -2p * (m)) + (2p * ( m) -pen( m)) + δ * (m, m) ≤ s -ŝA,m 2 + 2 2D A,m n -p * (m) + ( δ + 12ǫ n ) R A,m n +2 p * ( m) - 2D A, m n + (-δ + 12ǫ n ) R A, m n ≤ s -ŝA,m 2 + (37ǫ n + δ) R A,m n + (27ǫ n -δ) R A, m n .
For all m in M n , on Ω,

s -ŝA,m 2 = R A,m n + p * (m) - 2D A,m n ≥ (1 -25ǫ n ) R A,m n . Assume that 25ǫ n < 1, then, for all m ∈ M n , s -s 2 ≤ s -ŝA,m 2 1 + 37ǫ n + δ 1 -25ǫ n + 27ǫ n -δ 1 -25ǫ n s -s 2 .
This proves (16) for sufficiently large n. (16) holds in general provided that we increase the constant C if necessary.

Proof of Theorem 3.6

Assume that X 1 , ..., X n τ -mixing and let A * 0 , ..., A * p-1 , be the random variables given by the τ -mixing Lemma in Section 2.3.

2. Let (p(m), p W (m), δ(m, m ′ ), p * (m), p * W (m), D A,m , R A,m , δ * (m, m ′ )) (m,m ′ )∈(Mn) 2 be the quantities defined in Section 5.1. Recall that E sup m∈Mn 4D A,m n + δ R A,m n -pen(m) + ≤ e n , E sup m∈Mn pen(m) - 4D A,m n - δ R A,m n + ≤ e n .
For all m in M n , we have,

R A,m n = Crit(m) + p * (m) - 2D A,m n -15ǫ n R A,m n + 4D A,m n -pen(m) + δ R A,m n -δ * (m, m o ) + 20ǫ n R A,m n + (p(m) -p * (m)) +(35ǫ n -δ) R A,m n + (δ * (m, m o ) -δ(m, m o )). Therefore (1 + δ -35ǫ n ) R A, m n ≤ Crit(m o ) + sup m∈Mn p * (m) - 2D A,m n -15ǫ n R A,m n + sup m∈Mn 4D A,m n -pen(m) + δ R A,m n + sup (m,m ′ )∈M 2 n δ(m, m ′ ) -20ǫ n R A,m ∨ R A,m ′ n + sup (m,m ′ )∈M 2 n (δ * (m, m ′ ) -δ(m, m ′ )) + sup m∈Mn |p(m) -p * (m)|. ( 50
)
On the other hand, for all

m in M n , Crit(m) = s -s m 2 -p(m) + δ(m o , m) + pen(m), thus Crit(m o ) ≤ (1 + δ)R A,mo + 2D A,mo n -p * (m o ) + (p * (m o ) -p(m o )) +pen(m o ) - 4D A,mo n - δ R A,mo n . Since E pen(m o ) -4D A,mo /n -δR A,mo /n ≤ e n and 2D A,mo /n = E(p * (m o )), from in- equality (26), there exists a constant C > 0 such that E (Crit(m o )) ≤ (1 + δ) R A,mo n + Cτ q M C n + e n .
For all m in M n , 2D A,m ≤ R A,m . Take the expectation in (50), from inequalities (26), (28), (34) and (38), there exists an absolut constant C > 0 such that

E (D m) ≤ c n R mo + Cn τ q M C n + e -1 2 (ln n) γ + e n .
This proves inequality (17).

From (6), for all m in M n ,

s -s 2 ≤ s -ŝA,m 2 + 2 2D A,m n -p * (m) -35ǫ n R A,m n + δ * (m, m) -20ǫ n R A,m ∨ R A, m n +2 - 2D A, m n + p * ( m) -15ǫ n R m n ) + -pen( m) + 2 2D m n + δ R A, m n + pen(m) -2 2D A,m n - δ R A,m n +(90ǫ n + δ) R A,m n + (50ǫ n -δ) R A, m n +4 sup m∈Mn |p(m) -p * (m)| + sup (m,m ′ )∈M 2 n (δ(m, m ′ ) -δ * (m, m ′ )). (51)
Assume that 35ǫ n < 1, for all m in M n , we have

R A,m n = (1 -35ǫ n )R A,m n - s -ŝA,m 2 1 -35ǫ n + s -ŝA,m 2 1 -35ǫ n ≤ 1 1 -35ǫ n s -ŝA,m 2 + 2D A,m n -p(m) -35ǫ n R A,m n ≤ 1 1 -35ǫ n s -ŝA,m 2 + 2D A,m n -p * (m) -35ǫ n R A,m n + p(m) -p * (m)
We use this expression in the terms (90ǫ n + δ)R A,m /n and (50ǫ nδ)R A, m/n of inequality (51). We deduce that, for all m in M n ,

1 + δ -85ǫ n 1 -35ǫ n s -s 2 ≤ 1 + δ + 55ǫ n 1 -35ǫ n inf m∈Mn s -ŝA,m 2 + 2 + 70ǫ n + δ -δ 1 -35ǫ n sup m∈Mn 2D A,m n -p * (m) -35ǫ n R A,m n + sup m∈Mn pen(m) - 4D A,m n - δ R A,m n + sup m∈Mn 4D A,m n + δ) R A, m n -pen(m) +2 sup m∈Mn p * (m) - 2D A,m n -15ǫ n R A,m n ) + sup (m,m ′ )∈M 2 n δ * (m, m ′ ) -20ǫ n R A,m ∨ R A,m ′ n + 4 + δ -δ 1 -35ǫ n sup m∈Mn |p(m) -p * (m)| + sup (m,m ′ )∈M 2 n (δ(m, m ′ ) -δ * (m, m ′ )).
We take the expectation in this last inequality and we deduce that, for sufficiently large n, (18) comes from (26), (28), (34), ( 35) and (38). It holds in general provided that we enlarge the constant C if necessary.

5.9 Proof of Corollaries 4.1 and 4.3.

In [START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF], we obtained the following inequalities. If there exists θ > 1 such that X 1 , ..., X n are arithmetically [AR(θ)], β-mixing, there exists constants c

v , c e , c D , c M such that, for all m, m ′ in M n , v 2 A,m,m ′ ≤ c v (d m ∨ d m ′ ) 3/4 , b 2 A,m,m ′ ≤ c e (d m ∨ d m ′ ), D A,m ≤ c D d m .
The constants c v and c D depend on the mixing coefficients and are unknown in practice. Without loss of generality, assume that γ ≤ 3/2 in [M 5 ].

Let us first assume that (X 1 , ..., X n ) are arithmetically β-mixing. Choose p ≥ √ n(ln n) 2 /2, q ≥ √ n(ln n) -2 /2 such that 2pq = n. Hence, there exists a constant c M such that

pβ q ≤ c M (log n) 2(θ+2) n -θ/2 . For all m, m ′ in M n , e A,m,m ′ ≤ 2c e d m ∨ d m ′ (ln n) 4 ≤ 2c e c ′ D D A,m ∨ D A,m ′ (ln n) 4 ≤ 2c e c ′ D (ln n) -1 R A,m ∨ R A,m ′ (ln n) 2γ . When d m ∨ d m ′ ≤ r n (ln n) 4γ , then v 2 A,m,m ′ ≤ c v (d m ∨ d m ′ ) 3/4 ≤ c v (r n ) -1/4 R n (ln n) γ ≤ c v (r n ) -1/4 R A,m ∨ R A,m ′ (ln n) γ . When d m ∨ d m ′ ≥ r n (ln n) 4γ , then v 2 A,m,m ′ ≤ c v (d m ∨ d m ′ ) 3/4 ≤ c v c ′ D D A,m ∨ D A,m ′ (d m ∨ d m ′ ) 1/4 ≤ c v c ′ D R A,m ∨ R A,m ′ r 1/4 n (ln n) γ . Therefore, [M 1 ]-[M 5 ] and [AR(θ)] with θ > 1 imply [V'] with ǫ * n = C (ln n) -1/4 ∧ r -1/8 n .
Let us now assume that there exists θ > 0 such that the data X 1 , ..., X n are geometrically [GEO(θ)] β-mixing. We still assume [M 1 ]-[M 5 ] on the models. Let p ≥ n(ln n) -2 /2, q ≥ (ln n) 2 /2 such that 2pq = n. Then there exist constants c e , c M such that, for all m, m ′ in M n , Without loss of generality, assume that γ ≤ 3/2 in [T5] and recall that there exists θ > 2 such that X 1 , ..., X n are arithmetically [AR(θ)] τ -mixing. Choose p ≥ √ n(ln n) 2 /2, q ≥ √ n(ln n) -2 /2 such that 2pq = n. Then, u < 1 and there exists constants c

pβ q ≤ c M n (ln n) 2 e -θ
(2)

T , c e such that

τ q M C n ≤ c (2) T (ln n) 2(1+θ) n (θ-3)/2 , e A,m,m ′ ≤ c e ln n R A,m ∨ R A,m ′ (ln n) 2γ . When 2 Jm∨J m ′ ≤ r n (ln n) 2γ 1-u , v 2 A,m,m ′ ≤ c v r n (ln n) 2γ 1-u 1 2 (1+u) ≤ c v r -1-u 2 n R n (ln n) γ ≤ c v R A,m ∨ R A,m ′ r 1-u 2 n (ln n) γ . When 2 Jm∨J m ′ ≥ r n (ln n) 2γ 1-u , v 2 A,m,m ′ ≤ c v c ′ D D A,m ∨ D A,m ′ r n (ln n) 2γ 1-u 1-u 2 ≤ c v c ′ D r -1-u 2 n R A,m ∨ R A,m ′ (ln n) γ .
We conclude the proof as in the β-mixing case. (q + 1l)|Cov(ψ j,k (X 1 ), ψ j,k (X l ))| ≤ 2 q Jm j=0 k∈Z q l=1 ψ j,k BV E (|ψ j,k (X 1 )|b(σ(X 1 ), X l )) . For all t in B m , qVar(L q (t)(A 0 )) ≤ 2 q l=1 |Cov(t(X 1 ), t(X l ))|.

≤ 2c τ K BV Jm j=0 2 j/2 k∈Z |ψ j,k | ∞ q l=1 τ 1/3 l-1 ≤ 4 c τ AK ∞ K BV
(53)

Let X * l be a random variable, independent of X 1 , with law P , such that

E (|X l -X * l |) ≤ τ l-1 .
This random variable can be defined thanks to the coupling lemma of Dedecker & Prieur [DP05] (section 7.1).

|Cov(t(X 1 ), t(X l ))| = |Cov(t(X 1 ), t(X l )t(X

The last inequality holds since, for all x, y in R there is less than 2A indices k in Z such that |ψ j,k (x)ψ j,k (y)| = 0. Since t belongs to B m , (j,k)∈m a 2 j,k ≤ 1, in particular, for all j, sup k∈Z |a j,k | ≤ 1. Thus, there exists a constant c such that Lip(t) ≤ c2 3Jm/2 . Hence, there exists a constant c such that, for all t in B m and all l in N * |Cov(t(X 1 ), t(X l ))| ≤ c2 5Jm/4 √ τ l-1 .

Remark that we also have |Cov(t(X 1 ), t(X l ))| ≤ t ∞ t s ≤ c2 Jm/2 .

Recall that u = 3/(1 + θ), there exist constants c, which may vary from line to line such that q l=1 |Cov(t(X 1 ), t(X l ))| ≤ c2 Jm/2 ∞ l=1 (2 3Jm/4 √ τ l-1 ∧ 1)

≤ c2 Jm/2 ∞ l=1
(2 3Jm/4 l -(1+θ)/2 ∧ 1) From (54), for all m in M n , there exists a constant c such that sup t∈Bm Lip(t) ≤ c2 3Jm/2 . Since Card(M n ) ≤ ln n/ ln 2, and 2 max m∈Mn Jm ≤ n, there exists a constant c M such that M C n ≤ c M n 2 .

≤ c2 Jm/2
Then the following inequality holds ∀x > 0, P U > 5.31D 3/4 (ǫx 2 ) 1/4 + 3 √ v 2 Dx + 3v 2 x + ǫ(19.1x) 2 n -1 ≤ 2e -x .

∀x > 0, P U < -9D 3/4 (ǫx 2 ) 1/4 + 7.61 √ v 2 Dx + ǫ(40.3x) 2 n -1 ≤ 3.8e -x . Lemma 6.5 Let X, X 1 , ..., X n be i.i.d random variables taking value in a measurable space (X, X ) with common law P . Let µ be a measure on (X, X ) and let (ψ λ ) λ∈Λ be an orthonormal system in L 2 (µ). Let L be a linear functional in L 2 (µ) and let B = {t = λ∈Λ a λ L(ψ λ ), λ∈Λ a 2 λ ≤ 1}, v 2 = sup t∈B Var(t(X)), b = sup t∈B t ∞ and ǫ = b 2 /n. Let s be a function in S, the linear space spanned by the functions (t λ ) λ∈Λ and let η > 0. Then the following inequality holds ∀x > 0, P ν n (L(s)) > η 2 s 2 + 2v 2 x + ǫx 2 /9 ηn ≤ e -x .

  Thus, on Ω T , m minimizes the following criterion Crit(m) = ss m 2p * (m) + δ * (m, m o ) + pen(m) For all m in M n , we have 0 ≤ pen(m) < (2δ)D A,m /n and R A,mo ≤ R A,m . Thus, for all

Lemma 5. 4

 4 Let θ > 2 and assume that X 1 , ..., X n are arithmetically [AR(θ)] τ -mixing and let u = 3/(1 + θ) ∧ 1. Let M n be a collection of regular wavelet spaces [W]. There exist constants c D , c v , c b such that, for all m, m ′ in M n ,D A,m ≤ c D 2 Jm , v 2 A,m,m ′ ≤ c v 2 Jm∨J m ′ 1 2 (1+u) , b 2 A,m,m ′ ≤ c b 2 Jm∨J m ′ . Moreover, M C n ≤ c T n 2 .

5. 11

 11 Proof of Lemma 5.4.Let us first recall the following lemma, obtained in[START_REF] Lerasle | Adaptive density estimation of stationary β-mixing and τ -mixing sequences[END_REF] as a consequence of the covariance inequality proved by Dedecker & Prieur[START_REF] Dedecker | New dependence coefficients. Examples and applications to statistics[END_REF] for τ -mixing sequences.Lemma 5.5 Let X, Y be two identically distributed real valued random variables, with common density s in L 2 (µ). There exists a constant c τ and a random variable b(σ(X), Y ) such that E(b(σ(X), Y )) = c τ (τ (σ(X), Y )) 1/3 such that, for all Lipschitz functions f and all h in BV|Cov(f (X), h(Y ))| ≤ h BV E (|f (X)|b(σ(X), Y )) ≤ c τ h BV f ∞ (τ (σ(X), Y )) 1/3 . (52)It comes from this Lemma and inequalities (20, 21, 22) thatD A,m = 1 q (j,k)∈m Var q i=1 ψ j,k (X i ) ≤ 2 (j,k)∈m q l=1

  we obtain the inequality on D A,m with c D = 4 c τ AK ∞ K BV are nested, we only have to compare, for all m in M n , b 2 A,m and v 2 A,m with 2 Jm . From [T2], b 2 m ≤ Φ 2 2 Jm , this proves the inequality on b 2 A,m,m ′ with c b = Φ 2

λ∈m∞ 2

 2 We deduce the inequality on v 2A,m,m ′ from (53) and this last inequality. It remains to control M C n , recall thatM C n = Lip d (ψ λ ) + s |M n | sup t∈Bm Lip(t)   . From (20, 21), for all m in M n , Jm/2 , sup λ∈m Lip d (ψ λ ) ≤ K L 2 3Jm/2 .

•

  The risk of s in Corollary 4.2 is compared with the best of the risks in the collection M n . It is compared with an upper bound on ss m 2 + 2E s m -ŝA,m

	2 in
	[Ler09a].
	• In [Ler09a], [M 4 ] was not necessary. However, our new estimator improves this pre-vious procedure every time that [M

4 ] (or any other assumption ensuring [V']) holds.

Corollary 4.3 Let M n be a collection of models satisfying [M 1 ]-[M 5 ]. Assume that the process (X n ) n∈Z is strictly stationary and geometrically [GEO(θ)] β-mixing with mixing rate θ > 0. Let s be the estimator defined in (9) with a resampling penalty (8). Let

  4.4 Assume that the process (X n ) n∈Z is strictly stationary and arithmetically [AR(θ)] τ -mixing with mixing rate θ > 2. Let M n be a collection of regular wavelet spaces [W] and assume moreover that [T4], [T5] hold. Let s be the estimator defined in (9) with a resampling penalty (8). Let ǫ * n

  and choose n sufficiently large such that 6K 2 ǫ n + 2(19) 2 K 4 ǫ 3 n ≤ 4, then e n (K) ≤ 15 and (29) holds for all n sufficiently large. It holds for all n provided that we enlarge C if necessary.

	Let e (2)

  2 (ln n) 2 , e A,m,m ′ ≤ c e (ln n) 4 n (d m ∨ d m ′ ) ≤It is a classical result (see for example Birgé & Massart[START_REF] Birgé | From model selection to adaptive estimation[END_REF]) that the collection of wavelet spaces M n satisfies the following assumptions[T1] for all m ∈ M n , 2 Jm ≤ n; [T2] there exists a constant Φ such that ∀m, m ′ ∈ M n , ∀t ∈ S m , ∀t ′ ∈ S m ′ , t + t ′ ∞ ≤ Φ2 (Jm∨J m ′ )/2 t + t ′ 2 ; [T3] |M n | ≤ ln n/ ln 2.Under these assumptions, the following lemma hold.

	c e c ′ D	(ln n) 4+2γ n	R A,m ∨ R A,m ′ (ln n) 2γ	.
	5.10 Proof of Corollary 4.4.

  Since t belongs to B m , t 2 ∞ ≤ Φ 2 2 Jm . Moreover, let a j,k = R tψ j,k dµ, then

	Lip(t) = sup x =y∈R	|t(x) -t(y)| |x -y|	≤	Jm j=0	sup x =y∈R k∈Z	|a j,k |	|ψ j,k (x) -ψ j,k (y)| |x -y|
		Jm
	≤ 2AK L	j=0	2 3j/2 sup
								* l ))|
			≤	Var(t(X 1 ))E (t(X l ) -t(X * l )) 2
			≤	2Var(t(X 1 )) t ∞ E |t(X l ) -t(X * l )|
			≤	2 s t 2

∞ Lip(t)τ l-1 . k∈Z |a j,k |.

Appendix

In this section, we recall some technical lemmas proved in [START_REF] Lerasle | Optimal model selection in density estimation[END_REF]. Lemma 6.1 For all α ≥ 0, K > α + 1,

Then, for all α ≥ 0, α ′ ≥ 0 and all

Lemma 6.2 Let n be an integer and let X 1 , ..., X n be real valued, identically distributed random variables with common law P . Let (t λ ) λ∈Λ be a collection of functions in

Proposition 6.3 Let X, X 1 , ..., X n be i.i.d random variables taking value in a measurable space (X, X ) with common law P . Let B be a symetric class of functions bounded by b. Let Z = sup t∈B (ν n t), ǫ = b 2 /n, v 2 = sup t∈B Var(t(X)), D = E sup t∈B (t(X) -P t) 2 . For all x > 0, we have

Lemma 6.4 Let X, X 1 , ..., X n be i.i.d random variables taking value in a measurable space (X, X ) with common law P . Let µ be a measure on (X, X ) and let (t λ ) λ∈Λ be a set of functions in L 2 (µ). Let B = {t = λ∈Λ a λ t λ , λ∈Λ a 2 λ ≤ 1}, D = E sup t∈B (t(X) -P t) 2 , v 2 = sup t∈B Var(t(X)), b = sup t∈B t ∞ and ǫ = b 2 /n. Let U = 1 n(n -1) n i =j=1 λ∈Λ (t λ (X i ) -P t λ )(t λ (X j ) -P t λ ).


Can Cui

Imran Sheikh

Mostafa Sadeghi

Emmanuel Vincent

END-TO-END JOINT RICH AND NORMALIZED ASR WITH A LIMITED AMOUNT OF RICH TRAINING DATA

Keywords: ASR, rich transcriptions, RNN-T, limited data, streaming ready

Joint rich and normalized automatic speech recognition (ASR), that produces transcriptions both with and without punctuation and capitalization, remains a challenge. Endto-end (E2E) ASR models offer both convenience and the ability to perform such joint transcription of speech. Training such models requires paired speech and rich text data, which is not widely available. In this paper, we compare two different approaches to train a stateless Transducer-based E2E joint rich and normalized ASR system, ready for streaming applications, with a limited amount of rich labeled data. The first approach uses a language model to generate pseudorich transcriptions of normalized training data. The second approach uses a single decoder conditioned on the type of the output. The first approach leads to E2E rich ASR which perform better on out-of-domain data, with up to 9% relative reduction in errors. The second approach demonstrates the feasibility of an E2E joint rich and normalized ASR system using as low as 5% rich training data with moderate (2.42% absolute) increase in errors.

INTRODUCTION

Transcribing speech into rich text with punctuation and capitalization has been an active area of research in automatic speech recognition (ASR) [START_REF] Yi | Adversarial transfer learning for punctuation restoration[END_REF][START_REF] Liao | Improving readability for automatic speech recognition transcription[END_REF][START_REF] Poláček | Online Punctuation Restoration using ELECTRA Model for streaming ASR Systems[END_REF]. Rich transcriptions are essential in applications that expect human-readability of the transcribed text as well in applications that use natural language processing (NLP) systems that rely on punctuation and case information. Joint rich and normalized ASR, that produces transcriptions both with and without punctuation and capitalization, is highly desirable because (a) it simplifies model deployment and maintenance, and (b) it extends compatibility with NLP models which either exploit or discard punctuation and case information.

The conventional approach to rich ASR involves postprocessing the output of a normalized ASR system with a case and punctuation restoration model [START_REF] Yi | Adversarial transfer learning for punctuation restoration[END_REF][START_REF] Liao | Improving readability for automatic speech recognition transcription[END_REF][START_REF] Poláček | Online Punctuation Restoration using ELECTRA Model for streaming ASR Systems[END_REF][START_REF] Huang | Token-level supervised contrastive learning for punctuation restoration[END_REF]. For instance, the authors in [START_REF] Poláček | Online Punctuation Restoration using ELECTRA Model for streaming ASR Systems[END_REF] used a modified Recurrent Neural Network-Transducer (RNN-T) ASR model with stateless prediction network [START_REF] Ghodsi | RNNtransducer with stateless prediction network[END_REF] and a fine-tuned ELECTRA-Small language model (LM) [START_REF] Clark | ELECTRA: Pre-training text encoders as discriminators rather than generators[END_REF]. While this approach does not require rich transcription as ASR training labels, it increases the total model size and inference time. In addition, it does not exploit acoustic information to transcribe punctuation.

The second approach employs an E2E ASR model to directly generate rich transcriptions [START_REF] Futami | Streaming joint speech recognition and disfluency detection[END_REF][START_REF] Tanaka | End-to-end rich transcription-style automatic speech recognition with semi-supervised learning[END_REF]. However, this can lead to poorer word recognition, hence poorer normalized ASR performance. To mitigate this, the study in [START_REF] Nozaki | End-to-end speech-topunctuated-text recognition[END_REF] introduced an auxiliary CTC loss for transcribing non-punctuated text at the output of an intermediate layer of the E2E model. More recently, the study in [START_REF] Ihori | Transcribing Speech as Spoken and Written Dual Text Using an Autoregressive Model[END_REF] proposed three autoregressive decoders that separately output spoken, written, and combined transcriptions. While both of these methods leverage normalized transcriptions to enhance the accuracy of rich transcriptions, the challenges related to lack of training data and increased model size persist.

This paper aims for an E2E joint rich and normalized ASR system which is (a) accurate at rich as well as normalized transcription tasks, (b) trainable with a limited amount of rich labeled data, and (c) ready for streaming applications. We introduce and compare two complementary approaches to train a stateless Transducer-based E2E joint rich and normalized ASR model. The first training approach uses an LM to generate pseudo-rich transcriptions of normalized training data. However, such LMs may not be accurate enough or available for certain domains [START_REF] Sunkara | Robust prediction of punctuation and truecasing for medical ASR[END_REF] and/or languages. We propose a second training approach to address such scenarios. This approach uses a single decoder which is conditioned on the type of the output. The rest of the paper is organized as follows. Section 2 presents the preliminaries. Section 3 presents our training approaches. Section 4 presents our experimental evaluation, followed by the conclusion in Section 5.

PRELIMINARIES

Stateless Transducer-based Rich ASR

An E2E rich ASR system [START_REF] Nozaki | End-to-end speech-topunctuated-text recognition[END_REF] directly transcribes speech into a rich transcription. Given an acoustic feature sequence X ∈ R L×A where L is the sequence length and A the feature dimension, the training objective is to maximize the probability

P (Y R |X) = S s=1 P (y R s |y R [1:s-1] , X), (1)
by generating a sequence Y R ∈ R S R , where R stands for rich transcription, and S R represents the rich sequence length. The loss function can be written as

L R = - Y R ,X log P (Y R |X). (2)
A stateless Transducer-based E2E ASR system, which uses an RNN-T framework [START_REF] Graves | Sequence transduction with recurrent neural networks[END_REF] with a stateless prediction network [START_REF] Ghodsi | RNNtransducer with stateless prediction network[END_REF], can be extended to the rich transcription task (see Fig. 1(a)). This brings the high accuracy and naturally streaming recognition capability of RNN-T to rich transcription.

2-Decoder Joint Normalized-Rich ASR

Inspired by [START_REF] Nozaki | End-to-end speech-topunctuated-text recognition[END_REF][START_REF] Ihori | Transcribing Speech as Spoken and Written Dual Text Using an Autoregressive Model[END_REF], we have designed an ASR system with two sets of Predictors and Joiners (see Fig. 1(b)). Using the output of the same Encoder, these two Decoders generate the rich and normalized transcriptions. Their respective training objectives can be expressed as 1

P (Y N |X) = S N s=1 P (y N s |y N [1:s-1] , X), P (Y R |X) = S R s=1 P (y R s |y R [1:s-1] , X), (3)
where N stands for normalized transcription, and S N , S R represent the normalized, rich sequence length. Therefore, the joint loss function is defined as

L 2-decoder = L N + L R = - Y N ,X log P (Y N |X) - Y R ,X log P (Y R |X). (4)
1 These equations are kept consistent with (1) and do not account for the stateless and/or streaming operation of the Transducer model.

PROPOSED METHODS

Training Rich ASR using Pseudo-rich Transcriptions

Automatically generated pseudo-rich transcriptions can used to train rich ASR models in the absence of human-labeled rich ASR training data. The use of pseudo-rich transcriptions to train rich ASR models remains unexplored in prior works. The work in [START_REF] Tanaka | End-to-end rich transcription-style automatic speech recognition with semi-supervised learning[END_REF] proposed a semi-supervised learning method in which a rich ASR system is first trained on a small amount of human-labelled rich training data and then used to automatically generate pseudo-rich transcripts of a speech corpus. The focus of this approach is rich transcription of speech phenomena such as fillers, laughter and coughs, etc. and hence automatically transcribing the training speech corpus is necessary. In contrast, our rich transcription task requires case and punctuation prediction. Case and punctuation enhanced rich transcriptions can be directly obtained from normalized transcriptions using state-of-the-art case and punctuation restoration models [START_REF] Yi | Adversarial transfer learning for punctuation restoration[END_REF][START_REF] Liao | Improving readability for automatic speech recognition transcription[END_REF][START_REF] Poláček | Online Punctuation Restoration using ELECTRA Model for streaming ASR Systems[END_REF][START_REF] Huang | Token-level supervised contrastive learning for punctuation restoration[END_REF]. Interestingly, this approach can be applied in scenarios where there is no rich labelled training data at all.

Conditioned Predictor ASR

A drawback of the pseudo-rich transcription driven rich ASR training approach, presented in Section 3.1, is that errors made by the case and punctuation prediction model may propagate into the rich ASR model. Moreover, such case and punctuation restoration models may not be accurate enough, or even available, for certain domains [START_REF] Sunkara | Robust prediction of punctuation and truecasing for medical ASR[END_REF] and/or languages. Hence, we propose a second training approach to address such scenarios.

Our approach effectively utilizes the small amount of rich training data and the large amount of normalized training data by using a single conditioned Predictor to handle rich and normalized transcriptions. As shown in Fig. 1 (c), the Predictor is conditioned on the transcription mode ID input which specifies whether we want a normalized (N) or rich (R) output. The mode ID input is inspired by the language ID information used by E2E multilingual ASR models [START_REF] Li | Recent advances in end-to-end automatic speech recognition[END_REF][START_REF] Toshniwal | Multilingual speech recognition with a single end-to-end model[END_REF]. As shown in Fig. 1 (d), the token embeddings are concatenated with the mode embedding before feeding them to the following Predictor layers. The Conditioned Predictor ASR system can use a loss function similar to Equation (4). However, only a portion of the input samples will have a rich transcription reference and will use the corresponding loss. We use the LibriSpeech corpus [START_REF] Panayotov | Librispeech: An ASR corpus based on public domain audio books[END_REF] to conduct our experiments. Following the standard split, our training set includes train-100, train-360 and train-500 subsets. Similarly, the validation sets consist of dev-clean and dev-other, while the test sets are test-clean and test-other. We retrieve the rich transcripts of the LibriSpeech corpus from the original Project Gutenberg texts. 2 In order to evaluate the model's performance on real unseen data we use the independent headset microphone (IHM) recordings of the AMI meeting corpus [START_REF] Carletta | The AMI meeting corpus: A preannouncement[END_REF] test set along with their original rich transcription.

Metrics

The regular Word Error Rate (WER) metric used in ASR evaluation is insufficient to assess the performance in rich transcription tasks. Traditionally, Precision, Recall, and F1 score metrics have been used to evaluate performance of rich transcription [START_REF] Poláček | Online Punctuation Restoration using ELECTRA Model for streaming ASR Systems[END_REF][START_REF] Huang | Token-level supervised contrastive learning for punctuation restoration[END_REF]. However, these prior works present separate performance figures for each of the punctuation marks and lack a standardized metric for casing. In order to have an understanding of the overall performance of prediction of punctuation marks as well as casing, and to facilitate the comparison of error rates across words, cases, and punctuation marks, we introduce the concepts of Case-only Error Rate (CaseER), Punctuation-only Error Rate (PuncER) and Case-Punctuation-aware WER (CP-WER) alongside WER. Their calculation methods are as follows:

CaseER = E np-c -E np-nc N c , PuncER = E p-nc -E np-nc N p , CP-WER = E p-c N p-c , WER = E np-nc N np-nc (5)
where E represents the total number of substitution, deletion and insertion errors, N represents the total number of words and/or punctuation marks. Suffixes c and p stand for presence of case and punctuation marks whereas suffices nc and np stand for absence of case and punctuation marks, respectively. Table 1 shows an example of these metrics computed on a sample reference and prediction.

Model and training setup

We use 80-dimensional log Mel filterbank features as input to all ASR models. Our tokenizer is a SentencePiece model [START_REF] Kudo | Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] with a vocabulary size of 500. All the ASR models (E2E ASR, 2-decoder ASR, Conditioned Predictor ASR) were trained using the recipe for the pruned stateless-Transducer based on the Zipformer [START_REF] Povey | zipformer[END_REF] Encoder in the icefall toolkit. 3For Conditioned Predictor ASR, the Token Embedding and Mode Embedding have 500 and 12 dimensions, respectively. All the models were trained for 40 epochs. We use the Rpunct [START_REF] Nurmanbetov | rpunct[END_REF] model to generate rich transcriptions from normalized transcriptions. This model is derived from the BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] LM after fine-tuning it for English punctuation and case restoration tasks. Given lower-cased text, the model can restore upper-casing, period ("."), exclamation ("!"), question mark ("?"), comma (","), colon (":"), semicolon (";"), apostrophe ("'") and dash ("-"). The generated pseudo-rich transcriptions may differ from the expected rich transcriptions, as illustrated by some sample Librispeech sentences in Table 2.

Table 2. Examples of paired rich transcriptions and pseudorich transcriptions from LibriSpeech dataset.

Rich transcriptions

Pseudo-rich transcriptions See, I could even get in myself! See I could even get in myself. A man said to the universe: Sir, I exist! A man said to the universe sir, I exist. A red-haired mountain of a man, with an apparently inexhaustible store of energy.

A red haired mountain of a man with an apparently inexhaustible store of energy.

Evaluation results

Effectiveness of E2E models

Table 3 presents the (%) error rates for different rich ASR systems trained using the conventional and proposed approaches. The conventional cascaded system, which uses a normalized output ASR followed by a case and punctuation restoration LM, is included as a baseline. Comparison of error rates of the cascaded system and E2E models trained using rich data shows that the latter can lead to significant reductions in CP-WER. For instance, when using the 2-Decoder ASR (R + N) we obtain a relative CP-WER reduction of up to 24% (from 11.7% to 8.85%). Table 3. ASR evaluation results in term of (%) Word Error Rate (WER), Case-only Error Rate (CaseER), Punctuation-only Error Rate (PuncER) and Case-Punctuation-aware WER (CP-WER). All ASR are trained on LibriSpeech train-960 set with different type of transcriptions (Trans.): normalized (N), rich (R) or pseudo-rich (R'). The 2-Decoder ASR and the Conditioned Predictor ASR use both rich (R or R') and normalized transcriptions of the entire train-960 set. When comparing the 2-Decoder ASR with the proposed Conditioned Predictor ASR on the LibriSpeech test set, the latter exhibits a decrease in PuncWER. For example, when using the same R + N training set, the Conditioned Predictor ASR achieves a relative decrease of 3% on test-clean and 5% on test-other in PuncER compared to the former. But 2-Decoder ASR have a better performance in term of case generation, by reducing relatively 24% the CaseER on test-clean.

Rich versus pseudo-rich transcriptions for training

Comparison of different E2E ASR trained using rich transcriptions versus pseudo-rich transcriptions shows that use of pseudo-rich training transcriptions yields higher error rates on in-domain LibriSpeech test-clean and test-other sets. In contrast, the use of pseudo-rich training transcriptions yields lower error rates on the out-of-domain AMI test set. For instance, the 2-Decoder ASR exhibits a CP-WER increase of 2.82% (from 8.85% to 11.67%) and 3.12% (from 11.76% to 14.88%) absolute on LibriSpeech test-clean and test-other sets, respectively, when using pseudo-rich training transcriptions (R'+N) instead of the rich training transcriptions (R+N). However, the same model exhibits a CP-WER reduction of 4.75% (from 50.18% to 45.43%) absolute on the AMI test set when using pseudo-rich training transcriptions.

Conditioned Predictor ASR with limited rich data

Unlike the 2-Decoder ASR, the Conditioned Predictor ASR can be trained with different amount of rich and normalized training data. To evaluate the effectiveness of this capability, we trained the Conditioned Predictor ASR with different proportions of rich and normalized (R + N) training data. Table 4 presents the error rates obtained on LibriSpeech testclean set. It can be observed that the CP-WER and PuncER performances of this model increase by small margins even the amount of rich training data is reduced drastically. Using only 5% of rich training data can still achieve a CP-WER of 11.74%, with only 2.42% absolute increase. This shows that the Predictor-Conditioned ASR can effectively utilize normalized data to enhance the generation of rich labels. Therefore, this model can serve scenarios having limited or severely limited rich training data.

CONCLUSION

In this paper, we introduced two approaches to train a stateless Transducer-based E2E joint rich and normalized ASR with a limited amount of rich labelled data. The first approach uses a language model to generate pseudo-rich transcriptions of normalized training data. Experimental results demonstrate that, compared to the model trained on rich data, the model trained on pseudo-rich data achieves a maximum relative reduction in CP-WER of up to 9% on on out-of-domain data. The second approach uses a single conditioned Predictor. The second approach demonstrates the feasibility of an E2E joint rich and normalized ASR system using as low as 5% rich training data with moderate (2.42% absolute) increase in errors.

ACKNOWLEDGMENTS

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

Fig. 1 .

 1 Fig. 1. (a) Stateless Transducer-based Rich ASR, (b) 2-Decoder Joint Normalized-Rich ASR, (c) the proposed Conditioned Predictor ASR, (d) input layers in the Original and Conditioned Predictors.

Table 1 .

 1 Example of evaluation metrics computed for reference Hi, I am Chloe. and hypothesis hey I am chloe.

	WER	CaseER	PuncER	CP-WER
	1/4 = 0.25 (2 -1)/2 = 0.5 (2 -1)/2 = 0.5 3/6 = 0.5

Table 4 .

 4 (%) error rates of Conditioned Predictor ASR on LibriSpeech test-clean when trained with different proportions of rich and normalized (R + N) training data.

	Type of ASR	# Param. Trans.	LibriSpeech test-clean	LibriSpeech test-other	AMI
				WER CaseER PuncER CP-WER WER CaseER PuncER CP-WER WER CaseER PuncER CP-WER
	Cascaded (ASR+LM) 180 M	N	2.18 35.60	44.33	11.70	5.03 34.82	47.69	14.91 37.46 45.92	74.84	45.99
	Rich only	70 M 70 M	R R'	2.45 28.21 2.35 31.42	30.39 44.43	9.12 11.60	5.73 25.14 5.49 30.14	30.51 47.39	11.94 39.59 38.57 14.94 39.01 37.39	97.02 71.82	50.61 46.33
	2-Decoder	71 M 71 M R' + N 2.28 31.68 R + N 2.33 26.99	29.99 45.10	8.85 11.67	5.45 24.76 5.38 29.53	31.05 47.81	11.76 39.12 39.44 14.88 39.33 33.64	96.48 65.63	50.18 45.43
	Conditioned Predictor	70 M 70 M R' + N 2.25 41.86 R + N 2.35 35.61	29.14 44.60	9.32 12.30	5.49 30.77 5.24 33.99	29.62 46.84	12.01 40.70 38.40 15.42 38.77 35.62	96.40 85.65	51.47 45.24
	(R : N) Proportion WER CaseER PuncER CP-WER
	0.50 : 0.50	2.38 34.86	29.33	9.32
	0.35 : 0.65	2.37 45.44	28.99	9.95
	0.20 : 0.80	2.34 56.37	29.96	10.83
	0.05 : 0.95	2.56 58.58	31.59	11.46

available at https://www.openslr.org/12/

https://github.com/k2-fsa/icefall/blob/master/ egs/librispeech/ASR/pruned_transducer_stateless7