Elisabeth Gassiat

Ramon Van Handel

PATHWISE FLUCTUATIONS OF LIKELIHOOD RATIOS AND CONSISTENT ORDER ESTIMATION

Keywords: AMS 2000 subject classifications: 62G20, 60F15, 60F10, 41A46 uniform law of iterated logarithm, local entropy with bracketing, entropy of finite mixtures, penalized likelihood order estimation, strong consistency

Consider an i.i.d. sequence of random variables whose distribution f ⋆ lies in one of a nested family of models Mq, q ≥ 1. We obtain a sharp characterization of the pathwise fluctuations of the generalized likelihood ratio statistic under entropy assumptions on the model classes Mq. Moreover, we develop a technique to obtain local entropy bounds from global entropy computations, so that these results can be applied in models with non-regular geometric structure. Finally, the results are applied to prove strong consistency and to identify minimal penalties for penalized likelihood order estimators in the absence of prior upper bounds on the model order and the underlying parameter set. Location mixture models, which possess a notoriously complicated geometric structure, are used as a case study throughout the paper, and the requisite geometric analysis is of independent interest.

1. Introduction. Let (X k ) k≥1 be a sequence of random variables whose distribution f ⋆ lies in one of a nested family of models (M q ) q≥1 , indexed (and ordered) by the integers. We define the model order as the smallest index q ⋆ such that the true distribution f ⋆ lies in the corresponding model class. The model order typically determines the most parsimonious representation of the true distribution of the underlying model (for example, it might determine the parametrization of the model which has the smallest possible dimension). On the other hand, the model order often has a concrete interpretation in terms of the modelling of the underlying phenomenon (for example, the estimation of the number of clusters in a data set, or the number of regimes in an economic time series). Therefore, the problem of estimating the model order from observed data is of significant practical, as well as theoretical, interest.

Of course, a satisfactory solution to this problem must provide an estimation method that does not assume prior knowledge on the underlying unknown distribution f ⋆ . In particular, prior bounds on model order and on parameter sets should be avoided. Yet, in this light, even one of the most widely used model selection criteria-the Bayesian Information Criterion (BIC) of Schwarz-is poorly understood. The chief motivation for the use of BIC (as opposed to other model selection criteria, such as Akaike's Information Criterion) is that it is expected to yield a strongly consistent estimator of the model order. However, as is pointed out by Csiszár and Shields [START_REF] Csiszar | The consistency of BIC Markov order estimator[END_REF], almost all existing consistency proofs assume a prior upper bound on the order as well as compactness of the underlying parameter sets. This is hardly satisfactory from the theoretical point of view, and provides little confidence in the basic motivation for this method. More delicate questions, such as the minimal penalty that yields a strongly consistent order estimator in the absence of a prior bound on the order, also remain open (the problem of identifying the minimal penalty, which minimizes the probability of underestimating the model order, was also raised in [START_REF] Csiszar | The consistency of BIC Markov order estimator[END_REF]).

Characterizing strong consistency of penalized likelihood order estimators hinges on a precise understanding of the pathwise fluctuations of the likelihood ratio statistic sup

f ∈Mq ℓ n (f ) -sup f ∈M q ⋆ ℓ n (f ),
as n → ∞, uniformly in the model order q > q ⋆ (here ℓ n (f ) is the likelihood of (X k ) k≤n under the distribution f ∈ M q ). When there is a known upper bound on the order q ⋆ ≤ q max < ∞ and the model classes M q are parametrized by a compact subset of Euclidean space, an upper bound on the pathwise fluctuations can be obtained by classical parametric methods: Taylor expansion of the likelihood and an application of a law of iterated logarithm. This approach forms the basis for most consistency proofs for penalized likelihood order estimators in the litera-ture, for example [START_REF] Hannan | The determination of the order of an autoregression[END_REF][START_REF] Nishii | Maximum likelihood principle and model selection when the true model is unspecified[END_REF][START_REF] Finesso | Consistent estimation of the order for Markov and hidden Markov chains[END_REF][START_REF] Keribin | Consistent estimation of the order of mixture models[END_REF][START_REF] Chambaz | Testing the order of a model[END_REF]. However, such techniques fail in the absence of a prior upper bound: even though each model class M q is finite dimensional, the full model M = q M q is infinite dimensional and, as such, the problem in the absence of a prior upper bound is inherently nonparametric. 1 When the model classes M q are noncompact, one must introduce sieves M n q ⊂ M n+1 q ⊂ • • • ⊂ M q which complicate the problem further (in this case even the parametric theory remains poorly understood, see [START_REF] Hartigan | A failure of likelihood asymptotics for normal mixtures[END_REF][START_REF] Bickel | Asymptotic distribution of the likelihood ratio statistic in a prototypical non regular problem[END_REF][START_REF] Liu | Asymptotics for the likelihood ratio test in a two-component normal mixture model[END_REF]). An entirely different approach based on universal coding theory [START_REF] Finesso | Consistent estimation of the order for Markov and hidden Markov chains[END_REF][START_REF] Gassiat | Optimal error exponents in hidden Markov model order estimation[END_REF][START_REF] Capp É | Inference in hidden Markov models[END_REF][START_REF] Chambaz | A MDL approach to HMM with Poisson and Gaussian emissions. Application to order indentification[END_REF] yields pathwise upper bounds on the likelihood ratio statistic that do not require prior bounds on the order or compactness of the models. However, these bounds are far from tight and cannot even establish consistency of BIC, let alone smaller penalties (this appears to be a fundamental limitation of this approach due to Rissanen's theorem, see [START_REF] Rissanen | Stochastic complexity and modeling[END_REF][START_REF] Barron | The minimum description length principle in coding and modeling[END_REF]).

To our knowledge, the only setting in which the pathwise fluctuations of the likelihood ratio statistic has been studied in the absence of a prior bound on the order is that of higher-order Markov chains, where Csiszár and Shields [START_REF] Csiszar | The consistency of BIC Markov order estimator[END_REF][START_REF] Csiszar | Large-scale typicality of Markov sample paths and consistency of MDL order estimators[END_REF] proved consistency of BIC. The proofs in [START_REF] Csiszar | The consistency of BIC Markov order estimator[END_REF][START_REF] Csiszar | Large-scale typicality of Markov sample paths and consistency of MDL order estimators[END_REF] use delicate estimates specific to Markov chains, and do not yield minimal penalties. However, it was shown in [START_REF] Van Handel | On the minimal penalty for Markov order estimation[END_REF] that a sharp bound can be obtained in the Markov chain case using techniques from empirical process theory, the main difficulty being the dependence structure of Markov chains.

The aim of this paper is to obtain generally applicable upper and lower bounds on the pathwise fluctuations of the likelihood ratio statistic uniformly in the model order q > q ⋆ , in the case of i.i.d. observations (X k ) k≥1 , without a prior bound on the model order and in possibly noncompact parameter spaces. We use empirical process methods as in [START_REF] Van Handel | On the minimal penalty for Markov order estimation[END_REF], but the difficulties to be surmounted in the present setting are of a different nature. Though the Markov chain models in [START_REF] Csiszar | The consistency of BIC Markov order estimator[END_REF][START_REF] Csiszar | Large-scale typicality of Markov sample paths and consistency of MDL order estimators[END_REF][START_REF] Van Handel | On the minimal penalty for Markov order estimation[END_REF] suffer from a lack of independence, geometrically these models are exceedingly simple: the family of qth-order Markov chains endowed with the Hellinger distance is simply a Euclidean ball when viewed in the appropriate parametrization. In contrast, in general order estimation problems, one is often faced with model classes that are geometrically very complex. An important case study that will be considered in this paper are finite mixture models (widely used in practice for clustering), which possess a notoriously complicated non-regular geometry. To obtain sharp bounds in such models, we will develop tools that can be used to obtain local and weighted 1 One of the key issues in this setting is to understand the dependence of the fluctuations of the likelihood ratio statistic on the dimension of the model classes Mq. However, one of the main results of this paper shows that for regular parametric models, the fluctuations of the likelihood ratio statistic uniformly in q > q ⋆ are dimension independent when a prior upper bound is assumed (cf. Remark 2.7), which is certainly not the case in the absence of a prior upper bound. Therefore, we find that the pathwise fluctuations of the likelihood ratio statistic with and without a prior upper bound are qualitatively different. entropy bounds, required for our pathwise fluctuation theorems, in models with non-regular geometric structure. These results are of independent interest: we are not aware of any existing local entropy results for models that possess a nontrivial geometric structure (the difficulty of obtaining local entropy bounds for mixture models is noted, for example, in [START_REF] Genovese | Rates of convergence for the Gaussian mixture sieve[END_REF][START_REF] Maugis | A non asymptotic penalized criterion for gaussian mixture model selection[END_REF]). Finally, we will apply our results to establish strong consistency of BIC and to identify minimal penalties for order estimation for general model classes, in the absence of prior bounds on the order and the underlying parameter set.

The remainder of this paper is organized as follows. Section 2 introduces the general model under consideration, and states our results on the pathwise fluctuations of the likelihood ratio statistic. Section 3 states our general results on local and weighted entropies, and considers also the special case of mixture models. Section 4 derives the consequences for order estimation. Proofs are given in section 5.

Pathwise fluctuations of the likelihood ratio statistic.

2.1. Basic setting and notation. Let (E, E, µ) be a measure space. For each q, n ≥ 1, let M n q be a given family of strictly positive probability densities with respect to µ (that is, we assume that f dµ = 1 and that f > 0 µ-a.e. for every f ∈ M n q ). Moreover, we assume that (M n q ) q,n≥1 is a nested family of models in the sense that M n q ⊆ M n q+1 and M n q ⊆ M n+1 q for all q, n ≥ 1. Let M q = n M n q , M n = q M n q , M = q,n M n q . Consider an i.i.d. sequence of E-valued random variables (X k ) k≥1 whose common distribution under the measure P ⋆ is f ⋆ dµ, where f ⋆ ∈ M q ⋆ \ cl M q ⋆ -1 for some q ⋆ ≥ 1 (here cl M q denotes the L 1 (dµ)-closure of M q ). The index q ⋆ is called the model order. Let us define the log-likelihood function

ℓ n (f ) = n i=1 log f (X i ), f ∈ M.
Evidently ℓ n (f ) is the log-likelihood of the i.i.d. sequence (X k ) k≤n when X k ∼ f dµ. Our aim is to study the pathwise fluctuations of the likelihood ratio statistic

sup f ∈M n q ℓ n (f ) -sup f ∈M n q ⋆ ℓ n (f )
as n → ∞, uniformly over the order parameter q ≥ q ⋆ . Pathwise upper and lower bounds on the likelihood ratio statistic are the key ingredient in the study of strong consistency of penalized likelihood order estimators (see section 4). EXAMPLE 2.1 (Location mixtures). The guiding example for our theory, the case of location mixtures, will be studied in detail in sections 3.2 and 4.2 below. We presently introduce this example in order to clarify our basic setup.

Let E = R d (with its Borel σ-field E) and let µ be the Lebesgue measure on R d . We fix a strictly positive probability density f 0 with respect to µ, and define

f θ (x) = f 0 (x -θ) for x, θ ∈ R d . Fix a sequence T (n) ↑ ∞ and define M n q = q i=1 π i f θ i : π i ≥ 0, q i=1 π i = 1, θ i ≤ T (n) .
Then M q is the family of all q-component mixtures of translates of the density f 0 , while M n q is the subset of the mixtures M q whose translation parameters (θ i ) i=1,...,q are restricted to a ball of radius T (n). The number of components q ⋆ of the true mixture f ⋆ ∈ M can be estimated from observations using the order estimator

qn = argmax q≥1 sup f ∈M n q ℓ n (f ) -pen(n, q) .
Pathwise control of the likelihood ratio statistic allows us to identify what penalties pen(n, q) and cutoff sequences T (n) yield strong consistency of qn (cf. section 4.2). REMARK 2.2. To avoid measurability problems and other technical complications, we employ throughout this paper the simplifying convention that all uncountable suprema (such as sup f ∈M n q ℓ n (f )) are interpreted as essential suprema with respect to the measure P ⋆ . In the majority of applications the model classes M n q will be separable, in which case the supremum and essential supremum coincide.

In the sequel, we will denote by • p the L p (f ⋆ dµ)-norm, that is, g p p = |g(x)| p f ⋆ (x)µ(dx), and we denote by f, g = f (x)g(x)f ⋆ (x)µ(dx) the Hilbert space inner product in L 2 (f ⋆ dµ). Define the Hellinger distance

h(f, g) 2 = ( f - √ g) 2 dµ, f, g ∈ M. It is easily seen that h(f, f ⋆ ) = f /f ⋆ -1 2 .
Finally, we will denote by N(Q, δ) for any class of functions Q and δ > 0 the minimal number of brackets of L 2 (f ⋆ dµ)-width δ needed to cover

Q: that is, N(Q, δ) is the smallest cardinality N of a collection of pairs of functions {g L i , g U i } i=1,...,N such that max i≤N g U i - g L i 2 ≤ δ and for every g ∈ Q we have g L i ≤ g ≤ g U i pointwise for some i ≤ N .
2.2. Upper bound. We aim to obtain a pathwise upper bound on the likelihood ratio statistic that holds uniformly in q > q ⋆ . To this end, define for q, n ≥ 1 and ε > 0 the Hellinger ball

H n q (ε) = { f /f ⋆ : f ∈ M n q , h(f, f ⋆ ) ≤ ε}.
Note that the definition of H n q (ε) depends on f ⋆ (which is fixed throughout the paper). The following result shows that the geometry of the Hellinger balls H n q (ε) controls the pathwise fluctuations of the likelihood ratio statistic. THEOREM 2.3. Suppose that for all n sufficiently large, we have

N(H n q (ε), δ) ≤ K(n)ε δ η(q)
for all q ≥ q ⋆ and δ ≤ ε, with K(n) ≥ 1 and η(q) ≥ q increasing functions. Then

lim sup n→∞ 1 log K(2n) ∨ log log n sup q≥q ⋆ 1 η(q) sup f ∈M n q ℓ n (f ) -sup f ∈M n q ⋆ ℓ n (f ) ≤ C P ⋆ -a.s., where C > 0 is a universal constant.
The proof of Theorem 2.3 is given in section 5.1 below.

The assumption of Theorem 2.3 on the entropy of the Hellinger balls H n q (ε) states, roughly speaking, that the class of densities M n q endowed with the Hellinger distance has the same metric structure as a Euclidean ball of dimension η(q) and radius of order K(n), at least locally in a neighborhood of the true density f ⋆ . The effective dimension η(q) controls the fluctuations of the likelihood ratio statistic as a function of the model order, while the effective radius K(n) controls the fluctuations as a function of time up to a minimal rate of order log log n. In the following section we will see that the minimal log log n rate is indeed optimal.

Let us note that the geometric structure required by Theorem 2.3 is far from obvious in many cases of practical interest. For example, in the case of finite mixtures, the geometry of the parameter sets corresponding to Hellinger balls is notoriously complex and highly non-regular, but we will nonetheless verify the assumption of Theorem 2.3 (see section 3.2). In order to apply Theorem 2.3 in such cases, we therefore need to develop tools to establish local entropy bounds in models that possess nontrivial geometric structure. Section 3 below is devoted to this problem.

Lower bound.

Throughout this section, we specialize to the case that M n q = M q does not depend on n (this implies essentially that M q is compact). In this setting, Theorem 2.3 yields an upper bound of order log log n on the pathwise fluctuations of the likelihood ratio statistic. The aim of this section is to obtain a matching lower bound of order log log n, which shows that the minimal rate in Theorem 2.3 is essentially optimal. For the purposes of a lower bound, the uniformity in q is irrelevant, so that it suffices to restrict attention to some fixed q > q ⋆ . We will in fact obtain a much stronger result in this case, which completely characterizes the precise pathwise asymptotics of the likelihood ratio statistic for fixed q in sufficiently smooth families.

The geometric structure required in the present section is somewhat different than that of Theorem 2.3. Instead of Hellinger balls, we consider the classes of weighted densities D q = {d f : f ∈ M q , f = f ⋆ } and D = q D q , where

d f = f /f ⋆ -1 h(f, f ⋆ ) , f ∈ M, f = f ⋆ .
In addition, we define for ε > 0 and q ≥ 1 the local weighted classes

D q (ε) = {d f : f ∈ M q , 0 < h(f, f ⋆ ) ≤ ε}, Dq = ε>0 cl D q (ε),
where the closure cl D q (ε) is in L 2 (f ⋆ dµ). Evidently Dq is the set of all possible limit points of d f as h(f, f ⋆ ) → 0 in M q . If the neighborhoods of Dq are sufficiently rich, such limits can be taken along a continuous path in the following sense.

DEFINITION 2.4. A point d ∈ Dq is called continuously accessible if there is a path (f t ) t∈]0,1] ⊂ M q \{f ⋆ } such that the map t → h(f t , f ⋆ ) is continuous, h(f t , f ⋆ ) → 0 as t → 0, and d ft → d in L 2 (f ⋆ dµ) as t → 0.
The subset of all continuously accessible points in Dq will be denoted as Dc q .

We can now formulate the main result of this section.

THEOREM 2.5. Let q ⋆ ≤ p < q. Assume that

1 0 log N(D q , u) du < ∞,
and that |d| ≤ D for all d ∈ D q with D ∈ L 2+α (f ⋆ dµ) for some α > 0. Then

lim sup n→∞ 1 log log n sup f ∈Mq ℓ n (f ) -sup f ∈Mp ℓ n (f ) ≥ sup g∈L 2 0 (f ⋆ dµ) sup f ∈ Dc q ( f, g ) 2 + -sup f ∈ Dp ( f, g ) 2 + P ⋆ -a.s.,
as well as

lim sup n→∞ 1 log log n sup f ∈Mq ℓ n (f ) -sup f ∈Mp ℓ n (f ) ≤ sup g∈L 2 0 (f ⋆ dµ) sup f ∈ Dq ( f, g ) 2 + -sup f ∈ Dc p ( f, g ) 2 + P ⋆ -a.s.,
where

L 2 0 (f ⋆ dµ) = {g ∈ L 2 (f ⋆ dµ) : g 2 ≤ 1, 1, g = 0}.
Only the first (lower bound) part of the theorem is needed to conclude optimality of the minimal log log n rate in Theorem 2.3. Indeed, we will obtain as a corollary the following lower bound counterpart to Theorem 2.3. COROLLARY 2.6. Suppose there exists q > q ⋆ such that the following hold.

There is an envelope function

D : E → R such that |d| ≤ D for all d ∈ D q and D ∈ L 2+α (f ⋆ dµ) for some α > 0. Moreover, 1 0 log N(D q , u) du < ∞.

Dc

q \ Dq ⋆ is nonempty. Let η(q) > 0 be an arbitrary positive function. Then

lim sup n→∞ 1 log log n sup q≥q ⋆ 1 η(q) sup f ∈Mq ℓ n (f ) -sup f ∈M q ⋆ ℓ n (f ) ≥ C 0 P ⋆ -a.s.
, where C 0 > 0 is nonrandom but may depend on f ⋆ and η.

The proofs of Theorem 2.5 and Corollary 2.6 are given in section 5.2 below. The fact that the geometric assumptions in Theorem 2.5 and Corollary 2.6 are expressed in terms of weighted classes is not surprising, as the sharp asymptotic expression provided by Theorem 2.5 for the pathwise fluctuations of the likelihood ratio statistic are expressed in terms of a variational problem on the weighted classes. Nonetheless, we are naturally led to ask whether there is any relation between the geometric assumptions imposed in the upper bound Theorem 2.3 and the lower bound Theorem 2.5, which appear to be quite different at first sight. In section 3, we will show that the global entropy of the weighted class is closely related to local entropy, so that the geometric assumptions for the upper and lower bounds are not too far apart. Beside the fundamental value of this observation, the relation between global and local entropies will prove to be an essential tool in order to verify these geometric assumptions in models with a complicated geometry, such as finite mixture models. REMARK 2.7. When Dq and Dp each contain an L 2 (f ⋆ dµ)-dense subset of continuously accessible points (which is typically the case in sufficiently smooth models), then Theorem 2.5 provides the exact characterization

lim sup n→∞ 1 log log n sup f ∈Mq ℓ n (f ) -sup f ∈Mp ℓ n (f ) = sup g∈L 2 0 (f ⋆ dµ) sup f ∈ Dq ( f, g ) 2 + -sup f ∈ Dp ( f, g ) 2 + P ⋆ -a.s.
Beside its intrinsic interest, this result has a surprising consequence. In the case that M q and M p are regular parametric models with dim(M q ) > dim(M p ), one can choose g ∈ Dq which is orthogonal to Dp . As Dq , Dp ⊆ L 2 0 (f ⋆ dµ) (see the proof of Corollary 2.6), it follows easily that in this case the right-hand side of the previous equation display is precisely equal to 1. In particular, we obtain the curious conclusion that in regular parametric models, the magnitude of the fluctuations of the likelihood ratio statistic does not depend on the dimensions dim(M q ) and dim(M p ). In contrast, it is well known that in regular parametric models, the likelihood ratio statistic itself converges weakly to a chi-square distribution with dim(M q ) -dim(M p ). degrees of freedom, so the tails of the distribution of the likelihood ratio statistic do in fact depend strongly on the dimensions dim(M q ) and dim(M p ). Of course, the dimension independence of the pathwise fluctuations will also cease to hold if we are interested in a result that is uniform in the order q, as in Theorem 2.3.

Entropy bounds.

In section 2, we obtained pathwise bounds on the fluctuations of the likelihood ratio statistic in terms of the geometry of the underlying model classes. However, we have required two distinct types of geometric conditions: local entropy bounds for classes of densities, and global entropy bounds for classes of weighted densities. In this section, we will show that the latter implies the former under appropriate conditions, so that a suitable global entropy bound for weighted densities suffices for all the results in section 2. We will subsequently show how the requisite entropy bounds can be obtained for the case of location mixtures (cf. Example 2.1). The latter is significant both as an important application, and as a nontrivial case study in obtaining local entropy bounds in models with a complicated geometry.

3.1.

From global entropy to local entropy. We are going to establish that local entropy estimates for a class of densities M can be obtained from global entropy estimates on the associated weighted class D. To this end, let us consider for the purposes of this section a general class of positive probability densities M with respect to some reference measure µ, a fixed f ⋆ ∈ M, and define the class of weighted densities D = {d f : f ∈ M, f = f ⋆ }. In addition, we define for δ > 0 the Hellinger ball H(δ) = { f /f ⋆ : h(f, f ⋆ ) ≤ δ}. We obtain the following result, whose proof is given in section 5.3. THEOREM 3.1. Suppose that there exist q, C 0 ≥ 1 and ε 0 > 0 such that

N(D, ε) ≤ C 0 ε q for every ε ≤ ε 0 . Let R ≥ sup f |d f | be an envelope function such that R 2 < ∞. Then N(H(δ), ρ) ≤ C 1 δ ρ q+1 for all δ, ρ > 0 such that ρ/δ < 4 ∧ 2 R 2 , where C 1 = 8C 0 (1 ∨ R 2 /4ε 0 ).
Of course, in the setting of section 2, we would apply this result to M n q , D n q , H n q (ε) for given n, q instead of to M, D, H(ε).

3.2.

The entropy of mixtures. We now develop the requisite entropy bounds in the case of mixtures (Example 2.1). In this section, let µ be the Lebesgue measure on R d . We fix a strictly positive probability density f 0 with respect to µ, and consider mixtures of densities in the class

{f θ : θ ∈ R d }, f θ (x) = f 0 (x -θ) ∀ x ∈ R d .
In everything that follows we fix a nondegenerate mixture f ⋆ of the form

f ⋆ = q ⋆ i=1 π ⋆ i f θ ⋆ i .
Nondegenerate means that π ⋆ i > 0 for all i, and θ ⋆ i = θ ⋆ j for all i = j. Let Θ ⊂ R d be a bounded parameter set such that {θ ⋆ i : i = 1, . . . , q ⋆ } ⊆ Θ, and denote its diameter by 2T (that is, Θ is included in some closed Euclidean ball of radius T ). We consider for q ≥ 1 the family of q-mixtures

M q = q i=1 π i f θ i : π i ≥ 0, q i=1 π i = 1, θ i ∈ Θ ,
and define the class of weighted densities as

D q = {d f : f ∈ M q , f = f ⋆ }. Let H 0 (x) = sup θ∈Θ f θ (x)/f ⋆ (x), H 1 (x) = sup θ∈Θ max i=1,...,d |∂f θ (x)/∂θ i |/f ⋆ (x), H 2 (x) = sup θ∈Θ max i,j=1,...,d |∂ 2 f θ (x)/∂θ i ∂θ j |/f ⋆ (x), H 3 (x) = sup θ∈Θ max i,j,k=1,...,d |∂ 3 f θ (x)/∂θ i ∂θ j ∂θ k |/f ⋆ (x)
when f 0 is sufficiently differentiable, and let M = q≥1 M q and D = q≥1 D q . REMARK 3.2. In the setting of Example 2.1, the parameter set Θ = Θ(n) depends on n, and we then write M n q instead of M q , etc. However, as the dependence on n is irrelevant for the entropy computation, we consider a fixed parameter set Θ in this section, and drop the dependence on n in our notation for simplicity.

We can now state the result of this section, whose proof is given in section 5.4. ASSUMPTION A. The following hold:

1. f 0 ∈ C 3 and f 0 (x), (∂f 0 /∂θ i )(x) vanish as x → ∞. 2. H k ∈ L 4 (f ⋆ dµ) for k = 0, 1, 2 and H 3 ∈ L 2 (f ⋆ dµ).
THEOREM 3.3. Suppose that Assumption A holds. Then there exist constants C ⋆ and δ ⋆ , which depend on d, q ⋆ and f ⋆ but not on Θ, q or δ, such that

N(D q , δ) ≤ C ⋆ (T ∨ 1) 1/6 ( H 0 4 4 ∨ H 1 4 4 ∨ H 2 4 4 ∨ H 3 2
2 ) δ 18(d+1)q for all q ≥ q ⋆ , δ ≤ δ ⋆ . Moreover, there is a function D ∈ L 4 (f ⋆ dµ) with

D 4 ≤ K ⋆ ( H 0 4 ∨ H 1 4 ∨ H 2 4 ),
where K ⋆ depends only on d and f ⋆ , such that |d| ≤ D for all d ∈ D.

Let us note that a key aspect of this result is that the dependence of the entropy bound on the order q and on the parameter set Θ is essentially explicit (see Example 3.5 below, for example). However, even for fixed q and Θ, the existence of a polynomial bound on the bracketing number of D q is far from obvious (previous claims [START_REF] Keribin | Consistent estimation of the order of mixture models[END_REF][START_REF] Chambaz | Testing the order of a model[END_REF][START_REF] Azais | The likelihood ratio test for general mixture models with possibly structural parameter[END_REF] that such bracketing numbers are polynomial were stated without proof).

Define the Hellinger ball

H q (ε) = { f /f ⋆ : f ∈ M q , h(f, f ⋆ ) ≤ ε}.
Using Theorem 3.1, we immediately obtain the following result on the local entropy of M q . COROLLARY 3.4. Suppose that Assumption A holds. Then

N(H q (ε), δ) ≤ C Θ ε δ 18(d+1)q+1
for all q ≥ q ⋆ and δ/ε ≤ 1, where

C Θ = L ⋆ (T ∨ 1) 1/6 ( H 0 4 4 ∨ H 1 4 4 ∨ H 2 4 4 ∨ H 3 2 2 ) 5/4
and L ⋆ is a constant that depends only on d, q ⋆ and f ⋆ . EXAMPLE 3.5 (Gaussian mixtures). Consider mixtures of standard Gaussian densities f 0 (x) = (2π) -d/2 e -x 2 /2 , and let Θ(T ) = {θ ∈ R d : θ ≤ T }. Fix a nondegenerate mixture f ⋆ , and define T ⋆ = max i=1,...,q ⋆ θ ⋆ i . Denote by H q (ε, T ) the Hellinger ball associated to the parameter set Θ(T ). Then

N(H q (ε, T ), δ) ≤ C ⋆ 1 e C ⋆ 2 T 2 ε δ 18(d+1)q+1
for all q ≥ q ⋆ , T ≥ T ⋆ , and δ/ε ≤ 1, where C ⋆ 1 , C ⋆ 2 are constants that depend on d, q ⋆ and f ⋆ only. To prove this, it evidently suffices to show that Assumption A holds and that H k 4 for k = 0, 1, 2 and H 3 2 are of order e CT 2 . These facts are readily verified by a straightforward but tedious computation. REMARK 3.6. We have not optimized the constants in Theorem 3.3 and Corollary 3.4. In particular, the constant 18 in the exponent can likely be improved. On the other hand, it is unclear whether the dependence on the diameter of Θ is optimal. Indeed, if one is only interested in global entropy N(H q , δ) where

H q = { f /f ⋆ : f ∈ M q },
then it can be read off from the proof of Theorem 3.3 that the constants in the entropy bound depend on H 0 1 and H 1 1 only, which are easily seen to scale polynomially in T due to the translation invariance of the Lebesgue measure. Therefore, for example in the case of Gaussian mixtures, one can obtain a global entropy bound which scales only polynomially as a function of T , whereas the above local entropy bound scales as e CT 2 . The behavior of local entropies is much more delicate than that of global entropies, however, and we do not know whether it is possible to obtain a local entropy bound that scales polynomially in T .

The proof of Theorem 3.3 is long and rather technical. Nonetheless, there are some key ideas underlying the proof, which we aim to briefly explain here.

(p, θ1, θ2) ≤ 0.05} with N (p, θ1, θ2) = |p(θ1 -0.5)+(1-p)(θ2 -0.5)|+ 1 2 p(θ1 - 0.5) 2 + 1 2 (1 -p)(θ2 -0.5) 2 .

The two plots are related by the local geometry Theorem 5.11, which yields

c ⋆ N (p, θ1, θ2) ≤ h(pf θ 1 + (1 -p)f θ 2 , f ⋆ ) ≤ C ⋆ N (p, θ1, θ2) for all p, θ1, θ2 ∈ [0, 1].
The classical approach to controlling local entropies of a parametric class G = {g ξ : ξ ∈ Ξ} with Ξ ⊂ R d is as follows (cf. [START_REF] Van Der | Asymptotic statistics[END_REF], Example 19.7). Suppose that the square root densities h ξ = g ξ /g ξ ⋆ satisfy the pointwise Lipschitz condition

|h ξ (x) -h ξ ′ (x)| ≤ H(x) |||ξ -ξ ′ |||, ξ, ξ ′ ∈ Ξ,
where H is a function in L 2 and |||•||| is a norm on Ξ. Suppose, moreover, that

h(g ξ , g ξ ⋆ ) ≥ c |||ξ -ξ ⋆ |||, ξ ∈ Ξ. Define Ξ(ε) = {ξ ∈ Ξ : |||ξ -ξ ⋆ ||| ≤ ε} and H(ε) = {h ξ : ξ ∈ Ξ, h(g ξ , g ξ ⋆ ) ≤ ε}. If |||ξ -ξ ′ ||| ≤ δ, then h ξ ′ -δH ≤ h ξ ≤ h ξ ′ + δH.
Therefore, we can control the local bracketing entropy by N(H(cε), 2δ H 2 ) ≤ N (Ξ(ε), δ), where N (Ξ(ε), δ) denotes metric entropy. But the metric entropy of a ball can be controlled by a standard volume comparison argument, yielding

N (Ξ(ε), δ) ≤ ((2ε + δ)/δ) d . Clearly the above properties require c|||ξ -ξ ⋆ ||| ≤ h(g ξ , g ξ ⋆ ) ≤ H 2 |||ξ -ξ ⋆ ||| for all ξ ∈ Ξ.
Therefore, such an approach can only work when the class G endowed with the Hellinger distance has a regular geometry (i.e., equivalent to a subset of a finite dimensional Banach space), at least in a neighborhood of the true parameter. This fails miserably in the case of mixture classes M q , which possess a highly nonregular geometry in a neighborhood of f ⋆ when q > q ⋆ . In fact, it is easily seen that h(f, f ⋆ ) = 0 does not even select a unique set of parameters (π i , θ i ) i=1,...,q , as mixture models are non-identifiable, and consequently the Hellinger balls H q (ε) look nothing like norm-balls when viewed as a subset of the parameters (π i , θ i ) i=1,...,q (cf. Figure 1). Thus we are faced with two basic difficulties:

1. How does one control the subset of parameters (π i , θ i ) i=1,...,q corresponding to the Hellinger balls H q (ε)? 2. How does one control the metric entropy of these sets?

The resolution of the first problem requires us to develop a precise understanding of the local geometry of mixture classes, which is done in Theorem 5.11 below.

One key consequence of this result, for example, is as follows: one can choose sufficiently small neighborhoods A 1 , . . . , A q ⋆ of θ 1 , . . . , θ q ⋆ , respectively, such that the Hellinger distance h(f, f ⋆ ) is bounded above and below up to a constant by the pseudodistance

θ j ∈A 0 π j + q ⋆ i=1 θ j ∈A i π j -π ⋆ i + θ j ∈A i π j (θ j -θ ⋆ i ) + 1 2 θ j ∈A i π j θ j -θ ⋆ i 2 (here f = q i=1 π i f θ i and A 0 = R d \(A 1 ∪ • • • ∪ A q ⋆ ))
. This pseudodistance quantifies precisely (and rather intuitively) the set of parameters with density close to f ⋆ , see Figure 1 for an illustration in the simplest possible case.

As for the second problem, we avoid it entirely by exploiting Theorem 3.1 instead of computing directly the local entropy. Using the local geometry Theorem 5.11 and Taylor expansion, we can approximate the weighted densities d f by linear combinations of their first and second derivatives with coefficients in a Euclidean ball. The entropy of the latter is easily estimated by the Lipschitz argument indicated above. However, the details are somewhat intricate: Taylor expansion should only be applied to parameters θ j that lie close to some θ ⋆ i , which requires careful bookkeeping.

The local geometry Theorem 5.11 and the relation between global entropy of weighted densities and local entropy developed in Theorem 3.1 are key ideas that allow us to obtain local entropy estimates in a geometrically nontrivial model. Let us note that the restriction to location mixtures is only used in the proof of Theorem 5.11. We believe that the same technique is applicable to other classes of mixtures (for example, Poisson mixtures or mixtures of densities in an exponential family) provided that the proof of Theorem 5.11 can be adapted to this setting.

Strongly consistent order estimation.

The goal of this section is to apply the results of sections 2 and 3 to identify what penalties and cutoffs yield strong consistency of penalized likelihood order estimators. We first develop some general consistency and inconsistency results, and then consider specifically the problem of mixture order estimation. 4.1. Consistency and minimal penalties. In this section we consider the general setting introduced in section 2.1. We now suppose, however, that the true model order q ⋆ (as well as the true density f ⋆ ) is not known, so that we must aim to estimate q ⋆ from an observation sequence (X k ) k≥1 . To this end, define the penalized likelihood order estimator as

qn = argmax q≥1 sup f ∈M n q ℓ n (f ) -pen(n, q) ,
where pen(n, q) is a penalty function. Our goal is to show that the penalized likelihood order estimator is strongly consistent, that is, qn → q ⋆ as n → ∞ P ⋆ -a.s., for a suitable choice of the penalty (that does not depend on q ⋆ or f ⋆ ). Let us emphasize that the maximum in the definition of qn is taken over all model orders q ≥ 1, that is, we do not assume that an a priori upper bound on the order is available, in contrast to most previous work on this topic. We obtain the following general result. THEOREM 4.1. Suppose that for all n sufficiently large, we have

N(H n q (ε), δ) ≤ K(n)ε δ η(q)
for all q ≥ q ⋆ and δ ≤ ε, where K(n) ≥ 1 and η(q) ≥ q are increasing functions and we assume that log K(n) = o(n). Let pen(n, q) be a penalty such that

lim n→∞ sup q>q ⋆ η(q){log K(2n) ∨ log log n} pen(n, q) -pen(n, q ⋆ ) = 0, lim n→∞ max q<q ⋆ pen(n, q) n = 0,
and pen(n, q) is increasing in q. Then qn → q ⋆ as n → ∞ P ⋆ -a.s.

Theorem 4.1 is proved in section 5.5 below.

Let us now specialize to the case that M n q = M q does not depend on n, as in section 2.3. In this case, Theorem 4.1 immediately yields the following corollary. COROLLARY 4.2. Suppose that for all q ≥ q ⋆ and δ ≤ ε

N(H q (ε), δ) ≤ Kε δ η(q)
, where K ≥ 1 and η(q) ≥ q is a strictly increasing function. Define the penalty

pen(n, q) = η(q) ̟(n),
where ̟(n) is any function such that

lim n→∞ log log n ̟(n) = 0, lim n→∞ ̟(n) n = 0.
Then qn → q ⋆ as n → ∞ P ⋆ -a.s.

Corollary 4.2 states that, when M n q = M q does not depend on n, the penalized likelihood order estimator is strongly consistent provided the penalty grows faster than log log n and slower than n. Clearly the log log n rate is the minimal one attainable by applying Theorem 4.1. This raises the question whether the log log n rate is indeed minimal, in the sense that smaller penalties yield inconsistent estimators. The following result (which follows easily from Theorem 2.5) shows that this is indeed the case, so that the result of Corollary 4.2 is essentially optimal. COROLLARY 4.3. Suppose there exists q > q ⋆ such that the following hold.

There is an envelope function

D : E → R such that |d| ≤ D for all d ∈ D q and D ∈ L 2+α (f ⋆ dµ) for some α > 0. Moreover, 1 0 log N(D q , u) du < ∞.

Dc

q \ Dq ⋆ is nonempty. Let η(q) > 0 be any strictly increasing function, and define the penalty pen(n, q) = C η(q) log log n.

If the constant C > 0 is chosen sufficiently small, then qn = q ⋆ infinitely often P ⋆ -a.s.

The proof of Corollary 4.3 is given in section 5.5. Let us note that the proof of Corollary 4.3 actually shows that sup f ∈Mq ℓ n (f )-pen(n, q) > sup f ∈M q ⋆ ℓ n (f )pen(n, q ⋆ ) infinitely often P ⋆ -a.s., so the conclusion of Corollary 4.3 is not altered even if we were to impose a prior upper bound on the order.

In conclusion, we have shown that when M n q = M q does not depend on n, penalties growing faster than log log n are consistent while the penalty C η(q) log log n is inconsistent when the constant C is sufficiently small. From the proof of Theorem 4.1, we can also see that the penalty C η(q) log log n is consistent when C is sufficiently large. However, the critical value of C may depend on the unknown parameter f ⋆ , so that this minimal penalty may not be implementable. On the other hand, assuming that η(q) does not depend on f ⋆ (as is typically the case), penalties satisfying the assumptions of Theorem 4.1 obviously do not depend on the unknown parameter f ⋆ and therefore define admissible estimators. When M n q depends on n, larger penalties may be required to ensure consistency, depending on the growth rate of K(n).

4.2. Mixture order estimation. We finally apply the results in the previous section to mixture order estimation. Throughout this section, let E = R d and let µ be the Lebesgue measure on R d . Fix a strictly positive probability density f 0 with respect to µ, and define

M n q = q i=1 π i f θ i : π i ≥ 0, q i=1 π i = 1, θ i ∈ Θ(n) ,
where

f θ (x) = f 0 (x -θ) and • • • ⊆ Θ(n) ⊆ Θ(n + 1) ⊆ • • • ⊂ R d is an increas- ing family of bounded subsets of R d .
We fix f ⋆ ∈ M throughout this section. In the following, we consider two separate cases. The first case is that of a compact parameter set, where Θ(n) = Θ does not depend on n. In this setting, we obtain a general result. Then, we consider the noncompact case in the setting of Gaussian mixtures, and illustrate how Theorem 4.1 can be used to obtain consistency results in this case.

Let us first consider the case of a compact parameter set. Then we obtain a general consistency result under Assumption A (cf. section 3.2). PROPOSITION 4.4. Suppose that the parameter set Θ(n) = Θ is a bounded subset of R d independent of n, and that Assumption A holds. If we choose a penalty of the form

pen(n, q) = q ω(n), lim n→∞ log log n ω(n) = lim n→∞ ω(n) n = 0,
then qn → q ⋆ as n → ∞ P ⋆ -a.s. On the other hand, if we choose the penalty pen(n, q) = C q log log n with a sufficiently small constant C > 0, then qn = q ⋆ infinitely often P ⋆ -a.s.

We therefore find that in the setting of location mixtures with a compact parameter set, the minimal penalty is of order log log n. Moreover, the popular BIC penalty (4.1) pen(n, q) = dq + q -1 2 log n yields a strongly consistent mixture order estimator in this setting, without a prior upper bound on the order. The requisite Assumption A is a very mild one, which highlights the broad applicability of this result. However, the assumption of a compact parameter space can be quite restrictive in practice.

Let us therefore consider a case where the parameter space is noncompact. For simplicity we restrict our attention to Gaussian mixtures, that is, we choose f 0 (x) = (2π) -d/2 e -x 2 /2 , and we choose the restricted parameter sets

Θ(n) = {θ ∈ R d : θ ≤ T (n)} for some sequence T (n) ↑ ∞.
Our aim is to choose the penalty pen(n, q) and cutoff T (n) so that the penalized likelihood order estimator is strongly consistent. In this setting, we obtain the following result.

PROPOSITION 4.5. Let f 0 (x) = (2π) -d/2 e -x 2 /2 and Θ(n) = {θ ∈ R d : θ ≤ T (n)},
and choose a penalty of the form pen(n, q) = q ω(n). If

lim n→∞ log log n ω(n) = lim n→∞ ω(n) n = 0, T (n) = O( log log n),
then qn → q ⋆ as n → ∞ P ⋆ -a.s. On the other hand, the BIC penalty (4.1) yields a strongly consistent order estimator if

T (n) = o( √ log n).
This result illustrates that our theory can establish consistency of the penalized likelihood mixture order estimator without any prior upper bounds on the model order or the magnitude of the true parameters. Let us note that there is nothing particularly special about the Gaussian case: a similar result can be obtained, in principle, for any mixture distribution, as long as one can obtain suitable estimates on the quantities H i 4 that appear in Corollary 3.4 (see Example 3.5 for the Gaussian case).

The proofs of Propositions 4.4 and 4.5 appear in section 5.6 below.

Proofs.

5.1. Proof of Theorem 2.3. The proof of Theorem 2.3 is based on the following deviation bound for the log-likelihood ratio. This bound is essentially from [START_REF] Van De Geer | Applications of empirical process theory[END_REF], Corollary 7.5, but the additional maximum inside the probability is essential for our purposes. THEOREM 5.1. Let M be a family of strictly positive probability densities with respect to a reference measure µ, fix some f ⋆ ∈ M, and define the Hellinger ball

H(ε) = { f /f ⋆ : f ∈ M, h(f, f ⋆ ) ≤ ε} where h(f, g) 2 = ( √ f - √ g) 2 dµ.
Suppose that for some constants

K ≥ 1, p ≥ 1 and all δ ≤ ε N(H(ε), δ) ≤ Kε δ p ,
where N(H(ε), δ) is the minimal number of brackets of L 2 (f ⋆ dµ)-width δ needed to cover H(ε). Let (X i ) i∈N be i.i.d. with distribution f ⋆ dµ. Then

P   max n≤k≤2n sup f ∈M k j=1 log f (X j ) f ⋆ (X j ) ≥ α   ≤ C e -α/C
for all α ≥ Cp(1 + log K) and n ≥ 1, where C is a universal constant.

PROOF. Define f = (f + f ⋆ )/2 for any f ∈ M, and define the empirical process ν n (g

) = n -1/2 n k=1 {g(X k ) -E[g(X k )]}. Using concavity of log x we have k j=1 log f (X j ) f ⋆ (X j ) ≤ 2k 1/2 ν k (log( f /f ⋆ )) -2kD(f ⋆ || f ),
where

D(f ⋆ ||f ) = log(f ⋆ /f )f ⋆ dµ is relative entropy. As D(f ⋆ ||f ) ≥ h(f, f ⋆ ) 2 P   max n≤k≤2n sup f ∈M k j=1 log f (X j ) f ⋆ (X j ) ≥ α   ≤ P max n≤k≤2n sup f ∈M {2k 1/2 ν k (log( f /f ⋆ )) -2kh( f , f ⋆ ) 2 } ≥ α ≤ S s=0 P max n≤k≤2n sup f ∈M:nh( f ,f ⋆ ) 2 ≤α2 s |k 1/2 ν k (log( f /f ⋆ ))| ≥ α2 s-1 ≤ 3 S s=0 max n≤k≤2n P sup f ∈M:h( f ,f ⋆ ) 2 ≤α2 s n -1 |ν k (log({ f /f ⋆ } 1/2 ))| ≥ α2 s-5 n -1/2 ,
where S = min{s : α2 s n -1 > 2}, and we have used Lemma 5.2 below for the last inequality. The remainder of the proof is identical to that of [START_REF] Van De Geer | Applications of empirical process theory[END_REF], Theorem 7.4 provided we show that for

H(ε) = { f /f ⋆ : f ∈ M, h( f , f ⋆ ) ≤ ε} N( H(ε), δ) ≤ 2 √ 2Kε δ p .
To this end, fix δ ≤ ε, and note that

h(f, f ⋆ ) ≤ 4h( f , f ⋆ ) by [25], Lemma 4.2 so that {f ∈ M : h( f , f ⋆ ) ≤ ε} ⊆ {f ∈ M : h(f, f ⋆ ) ≤ 4ε}. By assumption, there exist N ≤ (2 √ 2Kε/δ) p and functions g 1 , . . . , g N , h 1 , . . . , h N such that h i - g i 2 ≤ δ √ 2
for every i, and for every u ∈ H(4ε) there is an i such that

g i ≤ u ≤ h i . But for every f ∈ M such that h( f , f ⋆ ) ≤ ε, we then have for some i 2 -1/2 g 2 i + 1 ≤ f /f ⋆ ≤ 2 -1/2 h 2 i + 1. Moreover, using | √ a + c - √ b + c| ≤ | √ a - √ b| for a, b, c ≥ 0 we obtain 2 -1/2 h 2 i + 1 -2 -1/2 g 2 i + 1 2 ≤ 2 -1/2 h i -g i 2 ≤ δ.
The result now follows directly.

The following variant of Etemadi's inequality was used in the proof. The proof follows closely that of the classical Etemadi inequality, see [START_REF] Billingsley | Convergence of probability measures[END_REF], Appendix M19. LEMMA 5.2. Let Q be a family of measurable functions f : E → R. Then we have for every α > 0 and m, n ∈ N, m ≤ n

P ⋆ max k=m,...,n sup f ∈Q |S k (f )| ≥ 3α ≤ 3 max k=m,...,n P ⋆ sup f ∈Q |S k (f )| ≥ α , where S n (f ) = n 1/2 ν n (f ). PROOF. Define the stopping time τ = inf k ≥ m : sup f ∈Q |S k (f )| ≥ 3α . Then P ⋆ max k=m,...,n sup f ∈Q |S k (f )| ≥ 3α = P ⋆ [τ ≤ n] ≤ P ⋆ sup f ∈Q |S n (f )| ≥ α + n k=m P ⋆ τ = k and sup f ∈Q |S n (f )| < α .
But on the event {τ = k and sup f ∈Q |S n (f )| < α}, we clearly have

2α ≤ sup f ∈Q |S k (f )| -sup f ∈Q |S n (f )| ≤ sup f ∈Q |S k (f ) -S n (f )|.
Therefore, we can estimate

P ⋆ max k=m,...,n sup f ∈Q |S k (f )| ≥ 3α ≤ P ⋆ sup f ∈Q |S n (f )| ≥ α + n k=m P ⋆ τ = k and sup f ∈Q |S n (f ) -S k (f )| ≥ 2α ≤ P ⋆ sup f ∈Q |S n (f )| ≥ α + max k=m,...,n P ⋆ sup f ∈Q |S n (f ) -S k (f )| ≥ 2α ,
where we have used that sup f ∈Q |S n (f ) -S k (f )| and {τ = k} are independent to obtain the last inequality. The proof is now easily completed.

We can now complete the proof of Theorem 2.3.

PROOF OF THEOREM 2.3. By assumption, we have f ⋆ ∈ M n q for all q ≥ q ⋆ when n is sufficiently large. Then by Theorem 5.1, we have for n sufficiently large

P ⋆ max n≤k≤2n sup f ∈M 2n q {ℓ k (f ) -ℓ k (f ⋆ )} ≥ α ≤ C e -α/C
for all α ≥ Cη(q)(1 + log K(2n)) and q ≥ q ⋆ . Using that

M k q ⊆ M 2n q for n ≤ k ≤ 2n and ℓ k (f ⋆ ) ≤ sup f ∈M k q ⋆ ℓ k (f ),
we have for n sufficiently large

P ⋆   max n≤k≤2n sup q≥q ⋆ 1 η(q) sup f ∈M k q ℓ k (f ) -sup f ∈M k q ⋆ ℓ k (f ) ≥ α   ≤ ∞ q=q ⋆ C e -αη(q)/C
for all α ≥ C(1 + log K(2n)). Let β(n) be an increasing function. Then

P ⋆   max 2 n ≤k≤2 n+1 1 β(k) sup q≥q ⋆ 1 η(q) sup f ∈M k q ℓ k (f ) -sup f ∈M k q ⋆ ℓ k (f ) ≥ 2C   ≤ 2C n 2
for all n sufficiently large, provided that β(2 n ) ≥ log K(2 n+1 ) ∨ log log 2 n . The proof is now easily completed using the Borel-Cantelli lemma.

5.2.

Proof of Theorem 2.5. The proof of Theorem 2.5 is based on a sequence of auxiliary results. First, we will need a compact law of iterated logarithm for the Strassen functional

I n (g) = 1 √ 2n log log n n i=1 {g(X i ) -E ⋆ (g(X 1 ))} .
We state the requisite result for future reference.

THEOREM 5.3. Let Q be a family of measurable functions f :

E → R with 1 0 log N(Q, u) du < ∞.
Then, P ⋆ -a.s., the sequence (I n ) n≥0 is relatively compact in ℓ ∞ (Q), and its set of cluster points coincides precisely with the set

K = {f → f, g : g ∈ L 2 0 (f ⋆ dµ)}.
Proofs of this result can be found in [START_REF] Ossiander | A central limit theorem under metric entropy with L2 bracketing[END_REF], Theorem 4.2 or in [START_REF] Ledoux | Comparison theorems, random geometry and some limit theorems for empirical processes[END_REF], Theorem 9. We will also need the following simple result, whose proof is omitted.

LEMMA 5.4. Let (X i ) i≥1 be an i.i.d. sequence of random variables, and suppose

E[|X 1 | p ] < ∞. Then n -1/p max i=1,...,n |X i | → 0 a.s. as n → ∞.
Finally, we will need the following likelihood inequality that relates the loglikelihood ratio ℓ n (f )-ℓ n (f ⋆ ) to the empirical process. Related inequalities appear in [START_REF] Gassiat | Likelihood ratio inequalities with applications to various mixtures[END_REF][START_REF] Liu | Asymptotics for likelihood ratio tests under loss of identifiability[END_REF][START_REF] Chambaz | Testing the order of a model[END_REF], but the following form is perhaps the most natural. LEMMA 5.5. For any strictly positive probability density f = f ⋆ , we have

ℓ n (f ) -ℓ n (f ⋆ ) ≤ |ν n (d f )| 2 , where ν n (g) = n -1/2 n k=1 {g(X k ) -E ⋆ [g(X k )]
} denotes the empirical process.

PROOF. Note that

h(f, f ⋆ ) 2 = 2 -2 f f ⋆ dµ = -2 h(f, f ⋆ ) E ⋆ (d f (X 1 )).
Using log(1 + x) ≤ x, we can estimate

ℓ n (f ) -ℓ n (f ⋆ ) = n i=1 2 log(1 + h(f, f ⋆ ) d f (X i )) ≤ n i=1 2 h(f, f ⋆ ) d f (X i ) = 2 ν n (d f ) h(f, f ⋆ ) √ n -h(f, f ⋆ ) 2 n ≤ sup p∈R 2 ν n (d f ) p -p 2 .
The proof is easily completed.

We can now obtain the following asymptotic expansion of the log-likelihood, which provides a pathwise counterpart to the weak convergence theory in [START_REF] Gassiat | Likelihood ratio inequalities with applications to various mixtures[END_REF][START_REF] Liu | Asymptotics for likelihood ratio tests under loss of identifiability[END_REF]. PROPOSITION 5.6. Let q ≥ q ⋆ . Assume that

1 0 log N(D q , u) du < ∞,
and that |d| ≤ D for all d ∈ D q with D ∈ L 2+α (f ⋆ dµ) for some α > 0. Then

sup f ∈Mq(4 √ log log n/n) 2 I n (d f ) h(f, f ⋆ ) 2n log log n -h(f, f ⋆ ) 2 2n log log n - 1 log log n sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) n→∞ ---→ 0 P ⋆ -a.s.,
where we have defined

M q (ε) = {f ∈ M q : h(f, f ⋆ ) ≤ ε}.
PROOF. We proceed in several steps.

Step 1 (localization). As q ≥ q ⋆ (hence f ⋆ ∈ M q ), clearly

sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) = sup f ∈Mq:ℓn(f )-ℓn(f ⋆ )≥0 {ℓ n (f ) -ℓ n (f ⋆ )} .
Now note that, as in the proof of Lemma 5.5,

ℓ n (f ) -ℓ n (f ⋆ ) ≤ 2 ν n (d f ) h(f, f ⋆ ) √ n -h(f, f ⋆ ) 2 n.
Therefore, we can estimate

sup f ∈Mq:ℓn(f )-ℓn(f ⋆ )≥0 h(f, f ⋆ ) ≤ sup f ∈Mq:ℓn(f )-ℓn(f ⋆ )≥0 h(f, f ⋆ ) + ℓ n (f ) -ℓ n (f ⋆ ) n h(f, f ⋆ ) ≤ 2 √ n sup f ∈Mq:ℓn(f )-ℓn(f ⋆ )≥0 ν n (d f ) ≤ 8 log log n n sup d∈Dq I n (d).
Now note that we can estimate

sup d∈Dq I n (d) ≤ inf g∈L 2 0 (f ⋆ dµ) sup d∈Dq |I n (d) -d, g | + sup d∈Dq sup g∈L 2 0 (f ⋆ dµ) d, g .
The first term on the right converges to zero P ⋆ -a.s. as n → ∞ by Theorem 5.3, while the second term is easily seen to equal sup d∈Dq d -1, d 2 ≤ 1. Therefore

sup f ∈Mq:ℓn(f )-ℓn(f ⋆ )≥0 h(f, f ⋆ ) ≤ (1 + ε)
8 log log n n eventually as n → ∞ P ⋆ -a.s. for any ε > 0. In particular, we find that

{f ∈ M q : ℓ n (f ) -ℓ n (f ⋆ ) ≥ 0} ⊆ f ∈ M q : h(f, f ⋆ ) ≤ 4 log log n/n
eventually as n → ∞ P ⋆ -a.s. This implies that P ⋆ -a.s. eventually as n → ∞

sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) ≤ sup f ∈Mq:h(f,f ⋆ )≤4 √ log log n/n {ℓ n (f ) -ℓ n (f ⋆ )} .
But the reverse inequality clearly holds for all n ≥ 0, so that in fact

sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) = sup f ∈Mq(4 √ log log n/n) {ℓ n (f ) -ℓ n (f ⋆ )} eventually as n → ∞ P ⋆ -a.s.
Step 2 (Taylor expansion). Taylor expansion gives 2 log(1 + x) = 2x -x 2 + x 2 R(x), where R(x) → 0 as x → 0. Thus we can write, for any f ∈ M q ,

ℓ n (f ) -ℓ n (f ⋆ ) = n i=1 2 log(1 + h(f, f ⋆ ) d f (X i )) = 2 h(f, f ⋆ ) n i=1 d f (X i ) + 1 2 h(f, f ⋆ ) -h(f, f ⋆ ) 2 n i=1 (d f (X i )) 2 -n h(f, f ⋆ ) 2 + h(f, f ⋆ ) 2 n i=1 (d f (X i )) 2 R(h(f, f ⋆ ) d f (X i )).
Using that E ⋆ (d f (X 1 )) = -h(f, f ⋆ )/2, we therefore have

1 log log n {ℓ n (f ) -ℓ n (f ⋆ )} = 2 I n (d f ) h(f, f ⋆ ) 2n log log n -h(f, f ⋆ ) 2 2n log log n + R f,n n h(f, f ⋆ ) 2 log log n
where we have defined

R f,n = 1 n n i=1 {1 -(d f (X i )) 2 } + 1 n n i=1 (d f (X i )) 2 R(h(f, f ⋆ ) d f (X i )).
It follows easily that

sup f ∈Mq(4 √ log log n/n) 2 I n (d f ) h(f, f ⋆ ) 2n log log n -h(f, f ⋆ ) 2 2n log log n - 1 log log n sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) ≤ sup f ∈Mq(4 √ log log n/n) |R f,n | n h(f, f ⋆ ) 2 log log n ≤ 16 sup f ∈Mq (4 √ log log n/n) |R f,n | eventually as n → ∞ P ⋆ -a.s.
Step 3 (end of proof ). We can easily estimate

sup f ∈Mq(4 √ log log n/n) |R f,n | ≤ sup f ∈Mq 1 n n i=1 {(d f (X i )) 2 -1} + sup |x|≤4 √ log log n/n max i=1,...,n D(X i ) |R(x)| 1 n n i=1 (D(X i )) 2 .
As N(D q , δ) < ∞ for every δ > 0, the class {d 2 : d ∈ D q } can be covered by a finite number of brackets with arbitrary small L 1 (f ⋆ dµ)-norm and is therefore P ⋆ -Glivenko-Cantelli. Moreover, by construction

E ⋆ [(d f (X i )) 2 ] = 1 for all f ∈ M q .
Therefore, the first term in this expression converges to zero as n → ∞ P ⋆ -a.s. On the other hand, by Lemma 5.4 and the fact that D ∈ L 2+α (f ⋆ dµ), we have P ⋆ -a.s.

log log n/n max i=1,...,n D(X i ) = √ log log n n α/2(2+α) n -1/(2+α) max i=1,...,n D(X i ) n→∞ ---→ 0.
Therefore the second term converges to zero also, and the proof is complete. PROPOSITION 5.7. Let q ≥ q ⋆ . Assume that

1 0 log N(D q , u) du < ∞,
and that |d| ≤ D for all d ∈ D q with D ∈ L 2+α (f ⋆ dµ) for some α > 0. Then

lim inf n→∞ sup d∈ Dq (I n (d)) 2 + - 1 log log n sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) ≥ 0 P ⋆ -a.s.
PROOF. By Proposition 5.6, we have

lim inf n→∞ sup d∈ Dq (I n (d)) 2 + - 1 log log n sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) ≥ lim inf n→∞    sup d∈ Dq (I n (d)) 2 + - sup f ∈Mq(4 √ log log n/n) sup p≥0 2 I n (d f ) p -p 2    = lim inf n→∞    sup d∈ Dq (I n (d)) 2 + - sup f ∈Mq(4 √ log log n/n) (I n (d f )) 2 +    .
Suppose that the right hand side is negative with positive probability. Then there is an ε > 0 and a sequence τ n ↑ ∞ of random times such that (5.1)

sup d∈ Dq (I τn (d)) 2 + - sup f ∈Mq(4 √ log log τn/τn) (I τn (d f )) 2 + ≤ -ε for all n
with positive probability. We will show that this entails a contradiction. By Theorem 5.3 (which can be applied here as N(D q , δ) = N(cl D q , δ) for all δ > 0), the process (I τn ) n≥0 is P ⋆ -a.s. relatively compact in ℓ ∞ (cl D q ) with (5.2) inf

g∈L 2 0 (f ⋆ dµ) sup d∈cl Dq |I τn (d) -d, g | n→∞ ---→ 0 P ⋆ -a.s.
Then there is a set of positive probability on which (5.1) and (5.2) hold simultaneously. We now concentrate our attention on a single sample path in this set.

For any such path, we can clearly find a further subsequence σ n ↑ ∞ such that sup d∈cl Dq |I σn (d) -d, g | → 0 as n → ∞ for some g ∈ L 2 0 (f ⋆ dµ). Therefore Thus we have a contradiction, completing the proof.

( d f , g ) 2 +    = lim inf n→∞    sup d∈ Dq (I σn (d)) 2 + - sup f ∈Mq(4 √ log log σn/σn) (I σn (d f )) 2 +    ≤ -ε. But as d → d, g is continuous in L 2 (f ⋆ dµ) and cl D q (4 log log σ n /σ n ) is com- pact in L 2 (f ⋆ dµ) (which follows from N(D q , δ) < ∞ for all δ > 0),
We now obtain a converse to the previous result.

PROPOSITION 5.8. Let q ≥ q ⋆ . Assume that

1 0 log N(D q , u) du < ∞,
and that |d| ≤ D for all d ∈ D q with D ∈ L 2+α (f ⋆ dµ) for some α > 0. Then

lim sup n→∞ sup d∈ Dc q (I n (d)) 2 + - 1 log log n sup f ∈Mq ℓ n (f ) -ℓ n (f ⋆ ) ≤ 0 P ⋆ -a.s.
PROOF. Suppose that the result does not hold true. By Proposition 5.6, there is an ε > 0 and a sequence τ n ↑ ∞ of random times such that

sup d∈ Dc q (I τn (d)) 2 + - sup f ∈Mq(4 √ log log τn/τn) -h(f, f ⋆ ) 2 2τ n log log τ n + 2 I τn (d f ) h(f, f ⋆ ) 2τ n log log τ n
≥ ε for all n with positive probability. Proceeding as in the proof of Proposition 5.7, we can then show that there is a sequence of times σ n ↑ ∞ and some g ∈ L 2 0 (f ⋆ dµ) such that lim sup

n→∞ sup d∈ Dc q ( d, g ) 2 + - sup f ∈Mq(4 √ log log σn/σn) -h(f, f ⋆ ) 2 2σ n log log σ n + 2 d f , g h(f, f ⋆ ) 2σ n log log σ n ≥ ε.
We will show that this entails a contradiction. Let d 0 ∈ Dq be a continuously accessible point. Then there exists an α 0 > 0 (depending on d 0 ) and a path

(f α ) α∈]0,α 0 ] such that h(f α , f ⋆ ) = α for all α ∈ ]0, α 0 ] and d fα → d 0 in L 2 (f ⋆ dµ) as α → 0. Now choose the sequence α n = {( d 0 , g ) + + σ -1 n } log log σ n 2σ n .
As ( d 0 , g ) + ≤ d 0 2 g 2 ≤ 1, we clearly have 0 < α n < α 0 ∧ 4 log log σ n /σ n for all n sufficiently large. In particular f αn ∈ M q (4 log log σ n /σ n ), so that

sup f ∈Mq(4 √ log log σn/σn) 2 d f , g h(f, f ⋆ ) 2σ n log log σ n -h(f, f ⋆ ) 2 2σ n log log σ n ≥ 2 d fα n , g {( d 0 , g ) + + σ -1 n } -{( d 0 , g ) + + σ -1 n } 2 .
Therefore, we have

lim sup n→∞ sup d∈ Dc q ( d, g ) 2 + - sup f ∈Mq(4 √ log log σn/σn) -h(f, f ⋆ ) 2 2σ n log log σ n + 2 d f , g h(f, f ⋆ ) 2σ n log log σ n ≤ sup d∈ Dc q ( d, g ) 2 + -( d 0 , g ) 2 +
for any continuously accessible element d 0 ∈ Dq . But clearly we can choose d 0 to make the right hand side of this expression arbitrarily small. Thus we have the desired contradiction, completing the proof.

We can now complete the proof of Theorem 2.5.

PROOF OF THEOREM 2.5. We obtain separately the lower and upper bounds. Lower bound. By Propositions 5.7 and 5.8, we have

lim sup n→∞ 1 log log n sup f ∈Mq ℓ n (f ) -sup f ∈Mp ℓ n (f ) ≥ lim sup n→∞ sup d∈ Dc q (I n (d)) 2 + -sup d∈ Dp (I n (d)) 2 + P ⋆ -a.s.
Now fix any g ∈ L 2 0 (f ⋆ dµ). By Theorem 5.3 (which applies here as N(D q , δ) = N(cl D q , δ) ≥ N( Dq , δ) for all δ > 0), there is a sequence τ n ↑ ∞ of random times such that I τn → • , g in ℓ ∞ ( Dq ) P ⋆ -a.s. Therefore

sup d∈ Dc q (I τn (d)) 2 + -sup d∈ Dp (I τn (d)) 2 + n→∞ ---→ sup d∈ Dc q ( d, g ) 2 + -sup d∈ Dp ( d, g ) 2 + P ⋆ -a.s., so that certainly lim sup n→∞ 1 log log n sup f ∈Mq ℓ n (f ) -sup f ∈Mp ℓ n (f ) ≥ sup d∈ Dc q ( d, g ) 2 + -sup d∈ Dp ( d, g ) 2 +
P ⋆ -a.s. But as this inequality holds for every g ∈ L 2 0 (f ⋆ dµ), taking the supremum over g gives the requisite lower bound.

Upper bound. By Propositions 5.7 and 5.8, we have

lim sup n→∞ 1 log log n sup f ∈Mq ℓ n (f ) -sup f ∈Mp ℓ n (f ) ≤ lim sup n→∞ sup d∈ Dq (I n (d)) 2 + -sup d∈ Dc p (I n (d)) 2 + P ⋆ -a.s.
It is elementary that for any d, d ′ ∈ Dq and g ∈ L 2 0 (f ⋆ dµ)

(I n (d)) 2 + -(I n (d ′ )) 2 + ≤ |(I n (d)) 2 + -( d, g ) 2 + | + |(I n (d ′ )) 2 + -( d ′ , g ) 2 + | + ( d, g ) 2 + -( d ′ , g ) 2 + ≤ 2 sup d∈ Dq |(I n (d)) 2 + -( d, g ) 2 + | + ( d, g ) 2 + -( d ′ , g ) 2 + .
Taking > 0.

To this end, note that by direct computation

1, d f = √ f f ⋆ dµ -1 h(f, f ⋆ ) = - h(f, f ⋆ ) 2 . Choose (f n ) n≥0 ⊂ M q \{f ⋆ } such that h(f n , f ⋆ ) → 0 and d fn → d 0 ∈ Dq , then 1, d 0 = lim n→∞ 1, d fn = -lim n→∞ h(f n , f ⋆ ) 2 = 0.
Moreover, it is immediate that d 0 2 ≤ 1. We have therefore shown that Dq ⊂ L 2 0 (f ⋆ dµ). Now choose g ∈ Dc q \ Dq ⋆ . As Dq ⋆ is closed, it follows directly that

sup d∈ Dc q ( d, g ) 2 + = 1, sup d∈ Dq ⋆ ( d, g ) 2 + < 1.
Therefore (5.3) holds, and the proof is complete.

Proof of Theorem 3.1.

PROOF OF THEOREM 3.1. The assumption implies that

N(D, ε) ≤ C 0 ε ∧ ε 0 q for every ε > 0. If ε < R 2 /4, then ε ε ∧ ε 0 ≤ 1 ∨ R 2 4ε 0 . Defining C = C 0 (1 ∨ R 2 /4ε 0 ), we find that N(D, ε) ≤ C ε q for every ε < R 2 /4.
The remainder of the proof is devoted to establishing that

N(H(δ), ρ) ≤ 8Cδ ρ q+1
for all δ, ρ > 0 such that ρ/δ < 4 ∧ 2 R 2 , which is the desired result. Fix ε, δ > 0 and let N = N(D, ε). Then there exist l 1 , u 1 , . . . , l N , u N such that u i -l i 2 ≤ ε for all i and for every f , there is an i such that

l i ≤ d f ≤ u i . Choose f such that r -n δ ≤ h(f, f ⋆ ) ≤ r -n+1 δ (with r > 1). Then there is an i so that (r -n l i ∧ r -n+1 l i ) δ + 1 ≤ f /f ⋆ ≤ (r -n u i ∨ r -n+1 u i ) δ + 1.

Note that

u i r -n δ -l i r -n δ 2 ≤ r -n δε, u i r -n+1 δ -l i r -n+1 δ 2 ≤ r -n+1 δε, u i r -n+1 δ -l i r -n δ 2 ≤ (r -1)r -n δ + r -n+1 δε, u i r -n δ -l i r -n+1 δ 2 ≤ (r -1)r -n δ + r -n+1 δε,
where the latter two estimates follow from

l i ≤ d f ≤ u i , d f 2 = 1, and
(u i -l i ) r -n δ ≤ u i r -n+1 δ -l i r -n δ -d f (r -1)r -n δ ≤ (u i -l i ) r -n+1 δ, (u i -l i ) r -n δ ≤ u i r -n δ -l i r -n+1 δ + d f (r -1)r -n δ ≤ (u i -l i ) r -n+1 δ. As |a ∨ b -c ∧ d| ≤ |a -c| + |a -d| + |b -c| + |b -d|, we can estimate (r -n u i ∨ r -n+1 u i ) δ -(r -n l i ∧ r -n+1 l i ) δ 2 ≤ 2(r -1)r -n δ + 4r -n+1 δε.
Therefore, we have shown that

N({ f /f ⋆ : r -n δ ≤ h(f, f ⋆ ) ≤ r -n+1 δ}, 2(r-1)r -n δ+4r -n+1 δε) ≤ N(D, ε) for arbitrary ε, δ > 0, r > 1, n ∈ N. In particular, N({ f /f ⋆ : r -n δ ≤ h(f, f ⋆ ) ≤ r -n+1 δ}, ρ) ≤ N(D, 1 4 r n-1 ρ/δ -1 2 (1 -1/r))
for every δ > 0, r > 1, n ∈ N, ρ > 2(r -1)r -n δ.

Note that, by finiteness of the bracketing entropies, we can choose an envelope function R ≥ sup f |d f | such that R 2 < ∞. Then we evidently have

1 -r -n δR ≤ f /f ⋆ ≤ 1 + r -n δR whenever h(f, f ⋆ ) ≤ r -n δ. Therefore N({ f /f ⋆ : h(f, f ⋆ ) ≤ r -⌈H⌉ δ}, 2r -H δ R 2 ) = 1
for all δ > 0, r > 1, H > 0. Thus we can estimate

N({ f /f ⋆ : h(f, f ⋆ ) ≤ δ}, 2r -H δ R 2 ) ≤ 1 + ⌈H⌉ n=1 N({ f /f ⋆ : r -n δ ≤ h(f, f ⋆ ) ≤ r -n+1 δ}, 2r -H δ R 2 ) ≤ 1 + ⌈H⌉ n=1 N(D, {r n-H-1 R 2 -(1 -1/r)}/2) whenever δ > 0, r > 1, H > 0 such that R 2 > (1 -1/r)r H . In particular, N({ f /f ⋆ : h(f, f ⋆ ) ≤ δ}, 2r -H δ R 2 ) ≤ 1 + ⌈H⌉ n=1 N(D, r n-H-1 R 2 /4) whenever δ > 0, r > 1, H > 0 such that R 2 ≥ 2(1 -1/r)r H ,
where we have used that the bracketing number is a nonincreasing function of the bracket size. Now recall that

N(D, ε) ≤ C ε q for every 0 < ε < R 2 /4,
where q, C ≥ 1. Thus

N({ f /f ⋆ : h(f, f ⋆ ) ≤ δ}, 2r -H δ R 2 ) ≤ 1 + ⌈H⌉ n=1 r -(n-1)q 8C 2r -H R 2 q whenever δ > 0, r > 1, H > 0 such that R 2 ≥ 2(1 -1/r)r H . But ⌈H⌉ n=1 r -(n-1)q ≤ 1 1 -1/r q ≤ 1 1 -1/r ≤ R 2 2(1 -1/r)r H 4C 2r -H R 2
as r > 1 and q, C ≥ 1. We can therefore estimate

N({ f /f ⋆ : h(f, f ⋆ ) ≤ δ}, 2r -H δ R 2 ) ≤ R 2 2(1 -1/r)r H 8C 2r -H R 2 q+1 whenever δ > 0, r > 1, H > 0 such that R 2 ≥ 2(1 -1/r)r H .
We now fix δ, ρ > 0 such that ρ/δ < 4 ∧ 2 R 2 , and choose

r = 4 4 -ρ/δ , H = log(2 R 2 δ/ρ) log r .
Clearly r > 1 and H > 0. Moreover, note that our choice of r and H implies that

R 2 = 2(1 -1/r)r H and ρ = 2r -H δ R 2 .
We have therefore shown that LEMMA 5.9. It is possible to choose a bounded convex neighborhood A i of θ ⋆ i for every i = 1, . . . , q ⋆ such that, for some linearly independent family u 1 , . . . , u d ∈ R d , the sets { A i , u j : i = 1, . . . , q ⋆ } are disjoint for every j = 1, . . . , d.

N({ f /f ⋆ : h(f, f ⋆ ) ≤ δ}, ρ) ≤ 8Cδ ρ q+1 for all δ, ρ > 0 such that ρ/δ < 4 ∧ 2 R 2 .
PROOF. We first claim that one can choose linearly independent u 1 , . . . , u d such that |{ θ ⋆ i , u j : i = 1, . . . , q ⋆ }| = q ⋆ for every j = 1, . . . , d. Indeed, note that the set {u ∈ R d : |{ θ ⋆ i , u : i = 1, . . . , q ⋆ }| < q ⋆ } is a finite union of (d -1)-dimensional hyperplanes, which has Lebesgue measure zero. Therefore, if we draw a rotation matrix T at random from the Haar measure on SO(d), and let u i = T e i for all i = 1, . . . , d where {e 1 , . . . , e d } is the standard Euclidean basis in R d , then the desired property will hold with unit probability. To complete the proof, it suffices to choose

A i = B(θ ⋆ i , ε/4) with ε = min k min i =j | θ ⋆ i -θ ⋆ j , u k |. A1 A2 A3 θ ⋆ 1 θ ⋆ 2 θ ⋆ 3 u2 u1 A1, u1 A3, u1 A2, u1 A1, u2 A2, u2 A3, u2 FIG 2.
Illustration of the construction of the sets Ai for a mixture with d = 2 and q ⋆ = 3. The sets Ai are chosen in such a way that their projections on some linearly independent vectors u1, u2 are disjoint. Note that the choice of u1, u2 is not arbitrary (e.g., consider the projections on the coordinate axes).

We now fix once and for all a family of neighborhoods A 1 , . . . , A q ⋆ that satisfy the conditions of Lemma 5.9. The precise choice of these sets only affects the constants in the proofs below and is therefore irrelevant to our final result; we only presume that A 1 , . . . , A q ⋆ remain fixed throughout the proofs. Let us also define

A 0 = R d \(A 1 ∪ • • • ∪ A q ⋆ ).
Then {A 0 , . . . , A q ⋆ } partitions the parameter set R d in such a way that each bounded element A i , i = 1, . . . , q ⋆ contains precisely one component of the mixture f ⋆ , while the unbounded element A 0 contains no components of f ⋆ . This construction is illustrated in Figure 2.

Let us define for each finite measure λ on R d the function

f λ (x) = f θ (x) λ(dθ).
We also define the derivatives

D 1 f θ (x) ∈ R d and D 2 f θ (x) ∈ R d×d as [D 1 f θ (x)] i = ∂ ∂θ i f θ (x), [D 2 f θ (x)] ij = ∂ 2 ∂θ i ∂θ j f θ (x).

Denote by P(A) the space of probability measures supported on

D = {(η, β, ρ, τ, ν) : η 1 , . . . , η q ⋆ ∈ R, β 1 , . . . , β q ⋆ ∈ R d , ρ 1 , . . . , ρ q ⋆ ∈ M d + , τ 0 , . . . , τ q ⋆ ≥ 0, ν 0 ∈ P(A 0 ), . . . , ν q ⋆ ∈ P(A q ⋆ )}.
Then we define for each (η, β, ρ, τ, ν) ∈ D the function

ℓ(η, β, ρ, τ, ν) = τ 0 f ν 0 f ⋆ + q ⋆ i=1 η i f θ ⋆ i f ⋆ + β * i D 1 f θ ⋆ i f ⋆ + Tr ρ i D 2 f θ ⋆ i f ⋆ + τ i f ν i f ⋆ ,
and the nonnegative quantity

N (η, β, ρ, τ, ν) = τ 0 + q ⋆ i=1 |η i + τ i | + q ⋆ i=1 β i + τ i (θ -θ ⋆ i ) ν i (dθ) + q ⋆ i=1 Tr[ρ i ] + q ⋆ i=1 τ i 2 θ -θ ⋆ i 2 ν i (dθ).
We now formulate the key result on the local geometry of the mixture class M.

THEOREM 5.11. Suppose that

1. f 0 ∈ C 2 and f 0 (x), D 1 f 0 (x) vanish as x → ∞. 2. [D 1 f 0 ] i /f ⋆ 1 < ∞ and [D 2 f 0 ] ij /f ⋆ 1 < ∞ for all i, j = 1, . . . , d.
Then there exists a constant c ⋆ > 0 such that ℓ(η, β, ρ, τ, ν) 1 ≥ c ⋆ N (η, β, ρ, τ, ν) for all (η, β, ρ, τ, ν) ∈ D.

[The constant c ⋆ may depend on f ⋆ and A 1 , . . . , A q ⋆ but not on η, β, ρ, τ, ν.]

Before we turn to the proof, let us introduce a notion that is familiar in quantum physics. If (Ω, Σ) is a measurable space, call the map λ : Σ → R d×d a state2 if 1. A → [λ(A)] ij is a signed measure for every i, j = 1, . . . , d; 2. λ(A) is a nonnegative symmetric matrix for every A ∈ Σ;

3. Tr[λ(Ω)] = 1.
It is easily seen that for any unit vector ξ ∈ R d , the map A → ξ, λ(A)ξ is a sub-probability measure. Moreover, if ξ 1 , . . . , ξ d ∈ R d are linearly independent, there must be at least one ξ i such that ξ i , λ(Ω)ξ i > 0. Finally, let B ⊂ R d be a compact set and let (λ n ) n≥0 be a sequence of states on B. Then there exists a subsequence along which λ n converges weakly to some state λ on B in the sense that Tr[M (θ)λ n (dθ)] → Tr[M (θ)λ(dθ)] for every continuous function M : B → R d×d . To see this, it suffices to note that we may extract a subsequence such that all matrix elements [λ n ] ij converge weakly to a signed measure by the compactness of B, and it is evident that the limit must again define a state. PROOF OF THEOREM 5.11. Suppose that the conclusion of the theorem does not hold. Then there must exist a sequence of coefficients

(η n , β n , ρ n , τ n , ν n ) ∈ D with ℓ(η n , β n , ρ n , τ n , ν n ) 1 N (η n , β n , ρ n , τ n , ν n ) n→∞ ---→ 0.
Let us fix such a sequence throughout the proof. Applying Taylor's theorem to

u → f θ ⋆ i +u(θ-θ ⋆ i ) , we can write for i = 1, . . . , q ⋆ η n i f θ ⋆ i f ⋆ + β n * i D 1 f θ ⋆ i f ⋆ + Tr ρ n i D 2 f θ ⋆ i f ⋆ + τ n i f ν n i f ⋆ = (η n i + τ n i ) f θ ⋆ i f ⋆ + β n i + τ n i (θ -θ ⋆ i ) ν n i (dθ) * D 1 f θ ⋆ i f ⋆ + Tr ρ n i D 2 f θ ⋆ i f ⋆ + τ n i 2 θ -θ ⋆ i 2 ν n i (dθ) Tr 1 0 D 2 f θ ⋆ i +u(θ-θ ⋆ i ) f ⋆ 2(1 -u) du λ n i (dθ)
where λ n i is the state on A i defined by

Tr[M (θ) λ n i (dθ)] = Tr[M (θ) (θ -θ ⋆ i )(θ -θ ⋆ i ) * ] ν n i (dθ) θ -θ ⋆ i 2 ν n i (dθ)
(it is clearly no loss of generality to assume that ν n i has no mass at θ ⋆ i for any i, n, so that everything is well defined). We now define the coefficients

a n i = η n i + τ n i N (η n , β n , ρ n , τ n , ν n ) , b n i = β n i + τ n i (θ -θ ⋆ i ) ν n i (dθ) N (η n , β n , ρ n , τ n , ν n ) , c n i = ρ n i N (η n , β n , ρ n , τ n , ν n ) , d n i = τ n i 2 θ -θ ⋆ i 2 ν n i (dθ) N (η n , β n , ρ n , τ n , ν n ) for i = 1, . . . , q ⋆ ,
and

a n 0 = τ n 0 N (η n , β n , ρ n , τ n , ν n ) .
Note that

|a n 0 | + q ⋆ i=1 {|a n i | + b n i + Tr[c n i ] + |d n i |} = 1
for all n. We may therefore extract a subsequence such that:

1. There exist

a i ∈ R, b i ∈ R d , c i ∈ M d + , and a 0 , d i ≥ 0 (for i = 1, . . . , q ⋆ ) with |a 0 | + q ⋆ i=1 {|a i | + b i + Tr[c i ] + |d i |} = 1, such that a n 0 → a 0 and a n i → a i , b n i → b i , c n i → c i , d n i → d i as n → ∞ for all i = 1, . . . , q ⋆ . 2.
There exists a sub-probability measure ν 0 supported on A 0 , such that ν n 0 converges vaguely to ν 0 as n → ∞. 3. There exist states λ i supported on cl A i for i = 1, . . . , q ⋆ , such that λ n i converges weakly to λ i as n → ∞ for every i = 1, . . . , q ⋆ . It follows that the functions ℓ(η n , β n , ρ n , τ n , ν n )/N (η n , β n , ρ n , τ n , ν n ) converge pointwise along this subsequence to the function h/f ⋆ defined by

h = a 0 f ν 0 + q ⋆ i=1 a i f θ ⋆ i + b * i D 1 f θ ⋆ i + Tr[c i D 2 f θ ⋆ i ] + d i Tr 1 0 D 2 f θ ⋆ i +u(θ-θ ⋆ i ) 2(1 -u) du λ i (dθ) . But as ℓ(η n , β n , ρ n , τ n , ν n ) 1 /N (η n , β n , ρ n , τ n , ν n ) → 0, we have h/f ⋆ 1 =
0 by Fatou's lemma. As f ⋆ is strictly positive, we must have h ≡ 0.

To proceed, we need the following lemma.

LEMMA 5.12. The Fourier transform F [h](s) := e i x,s h(x)dx is given by

F [h](s) = F [f 0 ](s) a 0 e i θ,s ν 0 (dθ) + q ⋆ i=1 a i e i θ ⋆ i ,s + i b i , s e i θ ⋆ i ,s -s, c i s e i θ ⋆ i ,s -d i e i θ ⋆ i ,s φ(i θ -θ ⋆ i , s ) s, λ i (dθ)s for all s ∈ R d .
Here we defined the function φ(u) = 2(e u -u -1)/u 2 .

PROOF. The a i , b i , c i terms are easily computed using integration by parts. It remains to compute the Fourier transform of the function

[Ξ i (x)] jk = 1 0 [D 2 f θ ⋆ i +u(θ-θ ⋆ i ) (x)] jk 2(1 -u) du [λ i (dθ)] kj .
We begin by noting that

1 0 |[D 2 f θ ⋆ i +u(θ-θ ⋆ i ) (x)] jk | 2(1 -u) du dx |[λ i ] kj |(dθ) = [λ i ] kj TV |[D 2 f 0 (x)] jk | dx < ∞.
We may therefore apply Fubini's theorem, giving

F [[Ξ i ] jk ](s) = -F [f 0 ](s) s j s k e i θ ⋆ i ,s 1 0 e iu θ-θ ⋆ i ,s 2(1 -u)du [λ i (dθ)] kj = -F [f 0 ](s) s j s k e i θ ⋆ i ,s φ(i θ -θ ⋆ i , s ) [λ i (dθ)] kj ,
where we have computed the inner integral using integration by parts.

Let u 1 , . . . , u d ∈ R d be a linearly independent family satisfying the condition of Lemma 5.9. As F [h](s) = 0 for all s ∈ R d , we obtain

Φ ℓ (it) := a 0 Φ ℓ 0 (it)+ q ⋆ i=1 e it θ ⋆ i ,u ℓ a i +it b i , u ℓ -t 2 u ℓ , c i u ℓ -d i t 2 Φ ℓ i (it) = 0
for all ℓ = 1, . . . , d and t ∈ [-ι, ι] ⊂ R for some ι > 0, where we defined

Φ ℓ i (it) = φ(it θ -θ ⋆ i , u ℓ ) u ℓ , λ i (dθ)u ℓ
for i = 1, . . . , q ⋆ , and

Φ ℓ 0 (it) = e it θ,u ℓ ν 0 (dθ).
Indeed, it suffices to note that F [f 0 ](0) = 1 and that s → F [f 0 ](s) is continuous, so that this claim follows from Lemma 5.12 and the fact that F [f 0 ](s) is nonvanishing in a sufficiently small neighborhood of the origin. As all λ i have compact support, it is easily seen that for every i = 1, . . . , q ⋆ , the function Φ ℓ i (z) is defined for all z ∈ C by a convergent power series. The function

Ψ ℓ (it) := Φ ℓ (it)-a 0 Φ ℓ 0 (it) is therefore an entire function with |Ψ ℓ (z)| ≤ k 1 e k 2 |z| for some k 1 , k 2 > 0 and all z ∈ C. But as Φ ℓ (it) = 0 for t ∈ [-ι, ι], it follows from [20], Theorem 7.2.2 that a 0 Φ ℓ 0 (it)
is the Fourier transform of a finite measure with compact support. Thus we may assume without loss of generality that the law of θ, u ℓ under the sub-probability ν 0 is compactly supported for every ℓ = 1, . . . , d, so by linear independence ν 0 must be compactly supported. Therefore, the function Φ ℓ (z) is defined for all z ∈ C by a convergent power series. But as Φ ℓ (z) vanishes for z ∈ i[-ι, ι], we must have Φ ℓ (z) = 0 for all z ∈ C, and in particular (5.4)

Φ ℓ (t) = a 0 Φ ℓ 0 (t) + q ⋆ i=1 e t θ ⋆ i ,u ℓ a i + t b i , u ℓ + t 2 u ℓ , c i u ℓ + d i t 2 Φ ℓ i (t) = 0
for all t ∈ R and ℓ = 1, . . . , d. In the remainder of the proof, we argue that (5.4) can not hold, thus completing the proof by contradiction. At the heart of our proof is an inductive argument. Recall that by construction, the projections { A i , u ℓ : i = 1, . . . , q ⋆ } are disjoint open intervals in R for every ℓ = 1, . . . , d. We can therefore relabel them in increasing order: that is, define (ℓ1), . . . , (ℓq

⋆ ) ∈ {1, . . . , q ⋆ } so that θ ⋆ (ℓ1) , u ℓ < θ ⋆ (ℓ2) , u ℓ < • • • < θ ⋆ (ℓq ⋆ ) , u ℓ .
The following key result provides the inductive step in our proof. PROPOSITION 5.13. Fix ℓ ∈ {1, . . . , d}, and define

Φℓ 0 (t) := a 0 Φ ℓ 0 (t) + q ⋆ i=1 a i e t θ ⋆ i ,u ℓ .
Suppose that for some j ∈ {1, . . . , q ⋆ } we have Φ ℓ,j (t) = 0 for all t ∈ R, where

Φ ℓ,j (t) := Φℓ 0 (t) + j i=1 e t θ ⋆ (ℓi) ,u ℓ t b (ℓi) , u ℓ + t 2 u ℓ , c (ℓi) u ℓ + d (ℓi) t 2 Φ ℓ (ℓi) (t) .
Then d (ℓj) u ℓ , λ (ℓj) (R d )u ℓ = 0, u ℓ , c (ℓj) u ℓ = 0, and b (ℓj) , u ℓ = 0.

PROOF. Let us write for simplicity θ ℓ i = θ ⋆ i , u ℓ , and denote by λ ℓ i and ν ℓ 0 the finite measures on R defined such that f (x)λ ℓ i (dx) = f ( θ, u ℓ ) u ℓ , λ i (dθ)u ℓ and f (x)ν ℓ 0 (dx) = f ( θ, u ℓ )ν 0 (dθ), respectively. For notational convenience, we will assume in the following that (ℓi) = i and ν ℓ 0 ({θ ℓ i }) = 0 for all i = 1, . . . , q ⋆ . This entails no loss of generality: the former can always be attained by relabeling of the points θ ⋆ i , while Φℓ 0 is unchanged if we replace ν ℓ 0 and a i by ν ℓ 0 ( • ∩ R\{θ ℓ 1 , . . . , θ ℓ q ⋆ }) and a i + a 0 ν ℓ 0 ({θ ℓ i }), respectively. Note that

A i , u ℓ = ]θ ℓ- i , θ ℓ+ i [, where θ ℓ- i < θ ℓ i < θ ℓ+ i < θ ℓ- i+1
for all i by our assumptions ( A i , u ℓ must be an interval as A i is convex).

Step 1. We claim that the following hold:

a i = 0 for all i ≥ j + 1 and a 0 ν ℓ 0 ([θ ℓ j+1 , ∞[) = 0.
Indeed, suppose this is not the case. Then it is easily seen that

lim inf t→∞ | Φℓ 0 (t)| e tθ ℓ j+1 > 0,
where we have used that ν ℓ 0 has no mass at {θ ℓ 1 , . . . , θ ℓ q ⋆ }. On the other hand, as φ is positive and increasing and as λ i is supported on cl A i , we can estimate

0 ≤ t 2 e tθ ℓ i Φ ℓ i (t) e tθ ℓ j+1 ≤ t 2 e -t(θ ℓ j+1 -θ ℓ i ) φ(t{θ ℓ+ j -θ ℓ i }) λ ℓ i (R) t→∞ ---→ 0
for i = 1, . . . , j. But then we must have

0 = lim inf t→∞ |Φ ℓ,j (t)| e tθ ℓ j+1 > 0,
which yields the desired contradiction.

Step 2. We claim that the following hold:

d j λ ℓ j ([θ ℓ j , ∞[) = 0, u ℓ , c j u ℓ = 0, and a 0 ν ℓ 0 ([θ ℓ j , ∞[) = 0.
Indeed, suppose this is not the case. As

ν ℓ 0 ({θ ℓ j }) = 0, we can choose ε > 0 such that ν ℓ 0 ([θ ℓ j + ε, ∞[) ≥ ν ℓ 0 ([θ ℓ j , ∞[)/2.
As a 0 , d j ≥ 0, and using that φ is positive and increasing with φ(0) = 1 and that e εt ≥ (εt) 2 /2 for t ≥ 0, we can estimate

a 0 Φ ℓ 0 (t) + e tθ ℓ j t 2 u ℓ , c j u ℓ + d j t 2 Φ ℓ j (t) ≥ t 2 e tθ ℓ j ε 2 4 a 0 ν ℓ 0 ([θ ℓ j , ∞[) + u ℓ , c j u ℓ + d j λ ℓ j ([θ ℓ j , ∞[) > 0
for all t ≥ 0. On the other hand, it is easily seen that

1 t 2 e tθ ℓ j j i=1 e tθ ℓ i a i + t b i , u ℓ + j-1 i=1 e tθ ℓ i t 2 u ℓ , c i u ℓ + d i t 2 Φ ℓ i (t) t→∞ ---→ 0.
But this would imply that

0 = lim t→∞ Φ ℓ,j (t) a 0 Φ ℓ 0 (t) + e tθ ℓ j {t 2 u ℓ , c j u ℓ + d j t 2 Φ ℓ j (t)} = 1,
which yields the desired contradiction.

Step 3. We claim that the following hold:

d j λ ℓ j ([θ ℓ- j , θ ℓ j [) = 0 and a 0 ν ℓ 0 ([θ ℓ- j , θ ℓ j [) = 0.
Indeed, suppose this is not the case. We can compute

0 = d 2 dt 2 Φ ℓ,j (t) e tθ ℓ j = d j e t(θ-θ ℓ j ) λ ℓ j (dθ) + a 0 e t(θ-θ ℓ j ) (θ -θ ℓ j ) 2 ν ℓ 0 (dθ) + j-1 i=1 d 2 dt 2 e -t(θ ℓ j -θ ℓ i ) a i + t b i , u ℓ + t 2 u ℓ , c i u ℓ + d i t 2 Φ ℓ i (t) ,
where the derivative and integral may be exchanged by [START_REF] Williams | Probability with martingales[END_REF], Appendix A16. We now note that as a 0 , d j ≥ 0, we can estimate for t ≥ 0

d j e t(θ-θ ℓ j ) λ ℓ j (dθ) + a 0 e t(θ-θ ℓ j ) (θ -θ ℓ j ) 2 ν ℓ 0 (dθ) ≥ e t(θ ℓ- j -θ ℓ j ) d j λ ℓ j ([θ ℓ- j , θ ℓ j [) + a 0 [θ ℓ- j ,θ ℓ j [ (θ -θ ℓ j ) 2 ν ℓ 0 (dθ) > 0.
On the other hand, as (e x -1)/x is positive and increasing, we obtain for t ≥ 0

e -t(θ ℓ- j -θ ℓ j ) d 2 dt 2 e -t(θ ℓ j -θ ℓ i ) t 2 Φ ℓ i (t) = e -t(θ ℓ- j -θ ℓ j ) × e -t(θ ℓ j -θ ℓ i ) × (θ ℓ j -θ ℓ i ) 2 t 2 φ(t{θ -θ ℓ i }) λ ℓ i (dθ) -2(θ ℓ j -θ ℓ i ) e t(θ-θ ℓ i ) -1 θ -θ ℓ i λ ℓ i (dθ) + e t(θ-θ ℓ i ) λ ℓ i (dθ) ≤ e -t(θ ℓ- j -θ ℓ i ) (θ ℓ j -θ ℓ i ) 2 t 2 φ(t{θ ℓ+ i -θ ℓ i }) + 2 (θ ℓ j -θ ℓ i ) e t(θ ℓ+ i -θ ℓ i ) -1 θ ℓ+ i -θ ℓ i + e t(θ ℓ+ i -θ ℓ i ) λ ℓ i (R),
which converges to zero as t → ∞ for every i < j. It follows that

0 = lim t→∞ d 2 dt 2 Φ ℓ,j (t)/e tθ ℓ j d j e t(θ-θ ℓ j ) λ ℓ j (dθ) + a 0 e t(θ-θ ℓ j ) (θ -θ ℓ j ) 2 ν ℓ 0 (dθ) = 1,
which yields the desired contradiction.

Step 4. Recall that λ ℓ j is supported on [θ ℓ- j , θ ℓ+ j ] by construction. We have therefore established in the previous steps that the following hold:

d j u ℓ , λ j (R d )u ℓ = u ℓ , c j u ℓ = a 0 ν ℓ 0 ([θ ℓ- j , ∞[) = 0, a i = 0 for i > j.
It is therefore easily seen that

0 = lim t→∞ Φ ℓ,j (t) t e tθ ℓ j = b j , u ℓ .
Thus the proof is complete.

We can now perform the induction by starting from (5.4) and applying Proposition 5.13 repeatedly. This yields d j u ℓ , λ j (R d )u ℓ = u ℓ , c j u ℓ = b j , u ℓ = 0 for all j = 1, . . . , q ⋆ and ℓ = 1, . . . , d. As u 1 , . . . , u d are linearly independent and c j ∈ M d + , this implies that b j = 0, c j = 0 and d j = 0 for all j = 1, . . . , q ⋆ , so that a 0 e i θ,s ν 0 (dθ) +

q ⋆ i=1 a i e i θ ⋆ i ,s = 0
for all s ∈ R d (this follows as above by Lemma 5.12, h ≡ 0, F [f 0 ](s) = 0 for s in a neighborhood of the origin, and using analyticity). But by the uniqueness of Fourier transforms, this implies that the signed measure a 0 ν 0 + q ⋆ i=1 a i δ {θ ⋆ i } has no mass. As ν 0 is supported on A 0 , this implies that a j = 0 for all j = 1, . . . , q ⋆ . We have therefore shown that a i , b i , c i , d i = 0 for all i = 1, . . . , q ⋆ . But recall that

|a 0 | + q ⋆ i=1 {|a i | + b i + Tr[c i ] + |d i |} = 1
, so that evidently a 0 = 1. To complete the proof, it remains to note that

ℓ(η n , β n , ρ n , τ n , ν n ) N (η n , β n , ρ n , τ n , ν n ) f ⋆ dµ = q ⋆ i=0 a n i n→∞ ---→ 1.
But this is impossible, as

ℓ(η n , β n , ρ n , τ n , ν n ) N (η n , β n , ρ n , τ n , ν n ) 1 n→∞ ---→ 0
by construction. Thus we have the desired contradiction.

5.4.2.

Proof of Theorem 3.3. The proof of Theorem 3.3 consists of a sequence of approximations, which we develop in the form of lemmas. Throughout this section, we always presume that Assumption A holds.

We begin by establishing the existence of an envelope function.

LEMMA 5.14.

Define S = (H 0 + H 1 + H 2 ) d/c ⋆ . Then S ∈ L 4 (f ⋆ dµ), and
|f /f ⋆ -1| f /f ⋆ -1 1 ≤ S for all f ∈ M.
PROOF. That S ∈ L 4 (f ⋆ dµ) follows directly from Assumption A. To proceed, let f ∈ M q , so that we can write f = q i=1 π i f θ i . Then

f -f ⋆ f ⋆ = j:θ j ∈A 0 π j f θ j f ⋆ + q ⋆ i=1 j:θ j ∈A i π j -π ⋆ i f θ ⋆ i f ⋆ + j:θ j ∈A i π j f θ j -f θ ⋆ i f ⋆ .
Taylor expansion gives

f θ j (x) -f θ ⋆ i (x) = (θ j -θ ⋆ i ) * D 1 f θ ⋆ i (x)+ 1 2 1 0 (θ j -θ ⋆ i ) * D 2 f θ ⋆ i +u(θ j -θ ⋆ i ) (x) (θ j -θ ⋆ i ) 2(1 -u) du.
Using Assumption A, we find that

f -f ⋆ f ⋆ ≤ j:θ j ∈A 0 π j + q ⋆ i=1 j:θ j ∈A i π j -π ⋆ i + j:θ j ∈A i π j (θ j -θ ⋆ i ) + 1 2 j:θ j ∈A i π j θ j -θ ⋆ i 2 (H 0 + H 1 + H 2 ) d.
On the other hand, Theorem 5.11 gives

f -f ⋆ f ⋆ 1 ≥ c ⋆ j:θ j ∈A 0 π j + q ⋆ i=1 j:θ j ∈A i π j -π ⋆ i + j:θ j ∈A i π j (θ j -θ ⋆ i ) + 1 2 j:θ j ∈A i π j θ j -θ ⋆ i 2
.

The proof follows directly.

PROOF. Using

f -f ⋆ TV ≤ 2h(f, f ⋆ ) and | √ x -1| ≤ |x -1|, we find |d f | = | f /f ⋆ -1| h(f, f ⋆ ) ≤ |f /f ⋆ -1| 1 2 f /f ⋆ -1 1 ≤ 2S,
where we have used Lemma 5.14.

Next, we prove that the Hellinger normalized densities d f can be approximated by chi-square normalized densities for small h(f, f ⋆ ).

LEMMA 5.16. For any f ∈ M, we have

f /f ⋆ -1 h(f, f ⋆ ) - f /f ⋆ -1 χ 2 (f ||f ⋆ ) ≤ {4 S 2 4 S + 2S 2 } h(f, f ⋆ ),
where we have defined the chi-square divergence

χ 2 (f ||f ⋆ ) = f /f ⋆ -1 2 2 .
PROOF. Let us define the function R as

f f ⋆ -1 = 1 2 f -f ⋆ f ⋆ + R .
Then we have

f /f ⋆ -1 h(f, f ⋆ ) - f /f ⋆ -1 χ 2 (f ||f ⋆ ) = f /f ⋆ -1 + R f /f ⋆ -1 + R 2 - f /f ⋆ -1 f /f ⋆ -1 2 = (f /f ⋆ -1 + R){ f /f ⋆ -1 2 -f /f ⋆ -1 + R 2 } + R f /f ⋆ -1 + R 2 f /f ⋆ -1 + R 2 f /f ⋆ -1 2 ,
so that by the reverse triangle inequality and Corollary 5.15

f /f ⋆ -1 h(f, f ⋆ ) - f /f ⋆ -1 χ 2 (f ||f ⋆ ) ≤ 2 R 2 S + |R| f /f ⋆ -1 2 .
Now note that for all x ≥ -1

- x 2 2 ≤ - ( √ 1 + x -1) 2 2 = √ 1 + x -1 - x 2 ≤ 0.
Therefore, by Lemma 5.14,

|R| ≤ f -f ⋆ f ⋆ 2 ≤ S 2 f -f ⋆ f ⋆ 2 1 ≤ S 2 f -f ⋆ f ⋆ 1 f -f ⋆ f ⋆ 2 .
The proof is easily completed using f -f ⋆ TV ≤ 2h(f, f ⋆ ).

Finally, we need one further approximation step.

LEMMA 5.17. Let q ∈ N and α > 0. Then for every

f ∈ M q such that h(f, f ⋆ ) ≤ α, it is possible to choose coefficients η i ∈ R, β i ∈ R d , ρ i ∈ M d
+ for i = 1, . . . , q ⋆ , and γ i ≥ 0, θ i ∈ Θ for i = 1, . . . , q, such that

q ⋆ i=1 rank[ρ i ] ≤ q ∧ dq ⋆ , q ⋆ i=1 |η i | ≤ 1 c ⋆ + 1 √ c ⋆ α , q ⋆ i=1 β i ≤ 1 c ⋆ + 2T √ c ⋆ α , q ⋆ i=1 Tr[ρ i ] ≤ 1 c ⋆ , q j=1 |γ j | ≤ 1 √ c ⋆ α ∧ c ⋆ , and f /f ⋆ -1 χ 2 (f ||f ⋆ ) -ℓ ≤ d 3/2 √ 2 3(c ⋆ ) 5/4 { H 3 2 S + H 3 } α 1/4 ,
where we have defined

ℓ = q ⋆ i=1 η i f θ ⋆ i f ⋆ + β * i D 1 f θ ⋆ i f ⋆ + Tr ρ i D 2 f θ ⋆ i f ⋆ + q j=1 γ j f θ j f ⋆ .
PROOF. As f ∈ M q , we can write f = q j=1 π j f θ j . Note that by Theorem 5.11

h(f, f ⋆ ) ≥ c ⋆ 4 q ⋆ i=1 j:θ j ∈A i π j θ j -θ ⋆ i 2 .
Therefore, h(f, f ⋆ ) ≤ α implies π j θ j -θ ⋆ i 2 ≤ 4α/c ⋆ for θ j ∈ A i . In particular, whenever θ j ∈ A i , either

π j ≤ 2 α/c ⋆ or θ j -θ ⋆ i 2 ≤ 2 α/c ⋆ . Define J = i=1,...,q ⋆ j : θ j ∈ A i , θ j -θ ⋆ i 2 ≤ 2 α/c ⋆ .
Taylor expansion gives

f θ j (x)-f θ ⋆ i (x) = (θ j -θ ⋆ i ) * D 1 f θ ⋆ i (x)+ 1 2 (θ j -θ ⋆ i ) * D 2 f θ ⋆ i (x) (θ j -θ ⋆ i )+ R ji (x),
where

|R ji | ≤ 1 6 d 3/2 θ j -θ ⋆ i 3 H 3 . We can therefore write f -f ⋆ f ⋆ = L + q ⋆ i=1 j∈J:θ j ∈A i π j R ji ,
where we have defined

L = q ⋆ i=1 j∈J:θ j ∈A i π j -π ⋆ i f θ ⋆ i f ⋆ + j∈J:θ j ∈A i π j (θ j -θ ⋆ i ) * D 1 f θ ⋆ i f ⋆ + 1 2 j∈J:θ j ∈A i π j (θ j -θ ⋆ i ) * D 2 f θ ⋆ i f ⋆ (θ j -θ ⋆ i ) + j ∈J π j f θ j f ⋆ . Now note that f /f ⋆ -1 χ 2 (f ||f ⋆ ) - L L 2 ≤ |f /f ⋆ -1| f /f ⋆ -1 2 f /f ⋆ -1 -L 2 L 2 + |f /f ⋆ -1 -L| L 2 ≤ f /f ⋆ -1 -L 2 S + |f /f ⋆ -1 -L| L 2 ,
where we have used Lemma 5.14. By Theorem 5.11, we obtain

L 2 ≥ L 1 ≥ c ⋆ 2 q ⋆ i=1 j∈J:θ j ∈A i π j θ j -θ ⋆ i 2 .
Therefore, we can estimate

|f /f ⋆ -1 -L| L 2 ≤ d 3/2 H 3 3c ⋆ q ⋆ i=1 j∈J:θ j ∈A i π j θ j -θ ⋆ i 3 q ⋆ i=1 j∈J:θ j ∈A i π j θ j -θ ⋆ i 2 ≤ 4α c ⋆ 1/4 d 3/2 H 3 3c ⋆
where we have used the definition of J. Setting ℓ = L/ L 2 , we obtain

f /f ⋆ -1 χ 2 (f ||f ⋆ ) -ℓ ≤ d 3/2 √ 2 3(c ⋆ ) 5/4 { H 3 2 S + H 3 } α 1/4 .
It remains to show that for our choice of ℓ = L/ L 2 , the coefficients η, β, ρ, γ in the statement of the lemma satisfy the desired bounds. These coefficients are

η i = 1 L 2 j∈J:θ j ∈A i π j -π ⋆ i , β i = 1 L 2 j∈J:θ j ∈A i π j (θ j -θ ⋆ i ), ρ i = 1 2 L 2 j∈J:θ j ∈A i π j (θ j -θ ⋆ i )(θ j -θ ⋆ i ) * , γ j = π j 1 j ∈J L 2 .
Clearly rank[ρ i ] ≤ #{j :

θ j ∈ A i } ∧ d, so q ⋆ i=1 rank[ρ i ] ≤ q ∧ dq ⋆ . Moreover, L 2 ≥ c ⋆ j:θ j ∈A 0 π j + q ⋆ i=1 j:θ j ∈A i π j -π ⋆ i + j:θ j ∈A i π j (θ j -θ ⋆ i ) + 1 2 j:θ j ∈A i π j θ j -θ ⋆ i 2 by Theorem 5.11. It follows that q ⋆ i=1 Tr[ρ i ] ≤ 1/c ⋆ . Now note that for j ∈ J such that θ j ∈ A i , we have θ j -θ ⋆ i 2 > 2 α/c ⋆ by construction. Therefore L 2 ≥ c ⋆ j ∈J:θ j ∈A 0 π j + 1 2 q ⋆ i=1 j ∈J:θ j ∈A i π j θ j -θ ⋆ i 2 ≥ ( √ c ⋆ α ∧ c ⋆ ) j ∈J π j . It follows that q j=1 |γ j | ≤ 1/( √ c ⋆ α ∧ c ⋆ ). Next, we note that q ⋆ i=1 j∈J:θ j ∈A i π j -π ⋆ i ≤ q ⋆ i=1 j:θ j ∈A i π j -π ⋆ i + j ∈J:θ j ∈A 0 π j . Therefore q ⋆ i=1 |η i | ≤ 1/c ⋆ + 1/ √ c ⋆ α. Finally, note that q ⋆ i=1 j∈J:θ j ∈A i π j (θ j -θ ⋆ i ) ≤ q ⋆ i=1 j:θ j ∈A i π j (θ j -θ ⋆ i ) + 2T j ∈J:θ j ∈A 0 π j . Therefore q ⋆ i=1 β i ≤ 1/c ⋆ + 2T / √ c ⋆ α.
The proof is complete.

We can now complete the proof of Theorem 3.3.

PROOF OF THEOREM 3.3. Let α > 0 be a constant to be chosen later on, and

D q,α = {d f : f ∈ M q , f = f ⋆ , h(f, f ⋆ ) ≤ α}.
Then clearly N(D q , δ) ≤ N(D q,α , δ) + N(D q \D q,α , δ).

We will estimate each term separately.

Step 1 (the first term). Define

M q = {(m 1 , . . . , m q ⋆ ) ∈ Z q ⋆ + : m 1 + • • • + m q ⋆ = q ∧ dq ⋆ }.
For every m ∈ M q , we define the family of functions

L q,m,α = q ⋆ i=1    η i f θ ⋆ i f ⋆ + β * i D 1 f θ ⋆ i f ⋆ + m i j=1 ρ * ij D 2 f θ ⋆ i f ⋆ ρ ij    + q j=1 γ j f θ j f ⋆ : (η, β, ρ, γ, θ) ∈ I q,m,α ,
where

I q,m,α = (η, β, ρ, γ, θ) ∈ R q ⋆ ×(R d ) q ⋆ ×(R d ) m 1 ו • •×(R d ) m q ⋆ ×R q ×Θ q : q ⋆ i=1 |η i | ≤ 1 c ⋆ + 1 √ c ⋆ α , q ⋆ i=1 β i ≤ 1 c ⋆ + 2T √ c ⋆ α , q ⋆ i=1 m i j=1 ρ ij 2 ≤ 1 c ⋆ , q j=1 |γ j | ≤ 1 √ c ⋆ α ∧ c ⋆ .
Define the family of functions

L q,α = m∈Mq L q,m,α
From Lemmas 5.16 and 5.17, we find that for any function d ∈ D q,α , there exists a function ℓ ∈ L q,α such that (here we use that h

(f, f ⋆ ) ≤ √ 2 for any f ) |d -ℓ| ≤ {4 S 2 4 S + 2S 2 } (α ∧ √ 2) + d 3/2 √ 2 3(c ⋆ ) 5/4 { H 3 2 S + H 3 } α 1/4 . Using α ∧ √ 2 ≤ 2 3/8 α 1/4 for all α > 0, we can estimate |d -ℓ| ≤ α 1/4 U, U = 1 + H 3 2 (c ⋆ ) 5/4 + 8 S 2 4 + 4 d 3/2 {S + S 2 + H 3 }, where U ∈ L 2 (f ⋆ dµ) by Assumption A. Now note that if m 1 ≤ ℓ ≤ m 2 for some functions m 1 , m 2 with m 2 -m 1 2 ≤ ε, then m 1 -α 1/4 U ≤ d ≤ m 2 + α 1/4 U with (m 2 + α 1/4 U ) -(m 1 -α 1/4 U ) 2 ≤ ε + 2α 1/4 U 2 . Therefore N(D q,α , ε + 2α 1/4 U 2 ) ≤ N(L q,α , ε) ≤ m∈Mq N(L q,m,α , ε) for ε > 0.
Of course, we will ultimately choose ε, α such that ε + 2α 1/4 U 2 = δ.

We proceed to estimate the bracketing number N(L q,m,α , ε). To this end, let ℓ, ℓ ′ ∈ L q,m,α , where ℓ is defined by the parameters (η, β, ρ, γ, θ) ∈ I q,m,α and ℓ ′ is defined by the parameters (η ′ , β ′ , ρ ′ , γ ′ , θ ′ ) ∈ I q,m,α . Note that

q ⋆ i=1 m i j=1 ρ * ij D 2 f θ ⋆ i f ⋆ ρ ij -(ρ ′ ij ) * D 2 f θ ⋆ i f ⋆ ρ ′ ij ≤ 2d √ c ⋆ H 2 q ⋆ i=1 m i j=1 ρ ij -ρ ′ ij .
We can therefore estimate

|ℓ -ℓ ′ | ≤ H 0 q ⋆ i=1 |η i -η ′ i | + H 1 √ d q ⋆ i=1 β i -β ′ i + H 0 q j=1 |γ j -γ ′ j |+ √ d √ c ⋆ α ∧ c ⋆ H 1 max j=1,...,q θ j -θ ′ j + 2d √ dq ⋆ √ c ⋆ H 2 q ⋆ i=1 m i j=1 ρ ij -ρ ′ ij 2 1/2
.

where we have used that

|f θ -f θ ′ |/f ⋆ ≤ θ -θ ′ H 1 √ d by Taylor expansion. Therefore, writing V = (H 0 + H 1 + H 2 ) d √ dq ⋆ , we have |ℓ -ℓ ′ | ≤ V |||(η, β, ρ, γ, θ) -(η ′ , β ′ , ρ ′ , γ ′ , θ ′ )||| q,m,α ,
where |||•||| q,m,α is the norm on R (1+d)q ⋆ +d(q∧dq ⋆ )+(1+d)q defined by

|||(η, β, ρ, γ, θ)||| q,m,α = q ⋆ i=1 |η i | + q ⋆ i=1 β i + q j=1 |γ j | + 1 √ c ⋆ α ∧ c ⋆ max j=1,...,q θ j + 2 √ c ⋆ q ⋆ i=1 m i j=1 ρ ij 2 1/2 . Note that if |||(η, β, ρ, γ, θ)-(η ′ , β ′ , ρ ′ , γ ′ , θ ′ )||| q,m,α ≤ ε ′ , then we obtain a bracket ℓ ′ -ε ′ V ≤ ℓ ≤ ℓ ′ + ε ′ V of size (ℓ ′ + ε ′ V ) -(ℓ ′ -ε ′ V ) 2 = 2ε ′ V 2 .
Therefore, if we denote by N (I q,m,α , |||•||| q,m,α , ε ′ ) the cardinality of the largest packing of I q,m,α by ε ′ -separated points with respect to the |||•||| q,m,α -norm, then

N(L q,m,α , ε) ≤ N (I q,m,α , |||•||| q,m,α , ε/2 V 2 ) for ε > 0.
But note that, by construction, I q,m,α is included in a |||•||| q,m,α -ball of radius not exceeding (6+3T )/( √ c ⋆ α∧c ⋆ ). Therefore, using the standard fact that the packing number of the r-ball B(r

) = {x ∈ B : |||x||| ≤ r} in any n-dimensional normed space (B, |||•|||) satisfies N (B(r), |||•|||, ε) ≤ ( 2r+ε ε ) n , we can estimate N(L q,m,α , ε) ≤ 4 V 2 (6 + 3T )/( √ c ⋆ α ∧ c ⋆ ) + ε ε (1+d)q ⋆ +d(q∧dq ⋆ )+(1+d)q .
In particular, if ε ≤ 1 and α ≤ c ⋆ , then

N(L q,m,α , ε) ≤ (24 + 12T ) V 2 / √ c ⋆ + √ c ⋆ ε √ α 3(d+1)q .
Finally, note that the cardinality of M q can be estimated as

#M q = q ⋆ + q ∧ dq ⋆ -1 q ∧ dq ⋆ ≤ e q∧dq ⋆ q ⋆ + q ∧ dq ⋆ -1 q ∧ dq ⋆ q∧dq ⋆ ≤ 2 3q ,
where we have used that q ≥ q ⋆ . We therefore obtain

N(D q,α , δ) ≤ m∈Mq N(L q,m,α , δ -2α 1/4 U 2 ) ≤ 24(2 + T ) V 2 / √ c ⋆ + √ c ⋆ (δ -2α 1/4 U 2 ) √ α 3(d+1)q whenever δ ≤ 1 and α ≤ (δ/2 U 2 ) 4 ∧ c ⋆ . Step 2 (the second term). For f, f ′ ∈ M q with h(f, f ⋆ ) > α and h(f ′ , f ⋆ ) > α, |d f -d ′ f | = |( f /f ⋆ -1) f ′ /f ⋆ -1 2 -( f ′ /f ⋆ -1) f /f ⋆ -1 2 | h(f, f ⋆ ) h(f ′ , f ⋆ ) ≤ f ′ /f ⋆ -f /f ⋆ 2 | f /f ⋆ -1| + √ 2 | f /f ⋆ -f ′ /f ⋆ | α 2 ,
where we have used that h(f, f ⋆ ) ≤ √ 2 for any f . Now note that

√ a - √ b 2 ≤ √ a - √ b √ a + √ b = |a -b|
for any a, b ≥ 0. We can therefore estimate

|d f -d ′ f | ≤ (f -f ′ )/f ⋆ 1/2 1 ( √ H 0 + 1) + √ 2 |(f -f ′ )/f ⋆ | 1/2 α 2 ,
where we have used that

| f /f ⋆ -1| ≤ √ H 0 + 1 for any f ∈ M. Now note that if we write f = q i=1 π i f θ i and f ′ = q i=1 π ′ i f θ ′ i , then we can estimate f -f ′ f ⋆ ≤ H 0 q i=1 |π i -π ′ i | + H 1 √ d max i=1,...,q θ i -θ ′ i . Defining W = ( H 0 + 1) H 0 + H 1 √ d 1/2 1 + √ 2 (H 0 + H 1 √ d) 1/2 ,
we obtain

|d f -d ′ f | ≤ W α 2 |||(π, θ) -(π ′ , θ ′ )||| 1/2 q , |||(π, θ)||| q = q i=1 |π i | + max i=1,...,q θ i (clearly |||•||| q defines a norm on R (d+1)q ). Now note that if |||(π, θ) -(π ′ , θ ′ )||| q ≤ ε, then we obtain a bracket d ′ f -ε 1/2 W/α 2 ≤ d f ≤ d ′ f + ε 1/2 W/α 2 of size (d ′ f + ε 1/2 W/α 2 ) -(d ′ f -ε 1/2 W/α 2 ) 2 = 2ε 1/2 W 2 /α 2 . Therefore N(D q \D q,α , δ) ≤ N (∆ q × Θ q , |||•||| q , α 4 δ 2 /4 W 2 2 ),
where we have defined the simplex ∆ q = {π ∈ R q + : q i=1 π i = 1}. We can now estimate the quantity on the right hand side of this expression as before, giving

N(D q \D q,α , δ) ≤ 8(1 + T ) W 2 2 + (c ⋆ ) 4 α 4 δ 2 (d+1)q for δ ≤ 1 and α ≤ c ⋆ .
End of proof. Choose α = (δ/4 U 2 ) 4 . Collecting the various estimates above, we find that for δ ≤ 1 ∧ 4(c ⋆ ) 1/4 (as U 2 ≥ S 1 ≥ 1 by Lemma 5.14)

N(D q , δ) ≤ 768(2 + T ) U 2 2 V 2 / √ c ⋆ + 32 U 2 2 √ c ⋆ δ 3 3(d+1)q + 4 18 (1 + T ) U 16 2 W 2 2 + 4 16 U 16 2 (c ⋆ ) 4 δ 18 (d+1)q ≤ c ⋆ 0 (T ∨ 1) 1/6 ( U 2 ∨ V 2 ∨ W 2 ) δ 18(d+1)q
where c ⋆ 0 = 12(c ⋆ ) -1/12 + 2(c ⋆ ) 1/12 + 4(c ⋆ ) 4/18 + 8. It follows that

N(D q , δ) ≤ C ⋆ (T ∨ 1) 1/6 ( H 0 4 4 ∨ H 1 4 4 ∨ H 2 4 4 ∨ H 3 2
2 ) δ 18(d+1)q for all δ ≤ δ ⋆ , where C ⋆ and δ ⋆ are constants that depend only on c ⋆ , d, and q ⋆ . This establishes the estimate given in the statement of the Theorem. The proof of the second half of the Theorem follows from Corollary 5.15 and H 0 4 ≥ 1. PROPOSITION 5.18. Let M n for n ≥ 1 be a family of strictly positive probability densities with respect to a reference measure µ such that M n ⊆ M n+1 for all n. Define M = n M n , and let f ⋆ be another probability density with respect to µ such that f ⋆ ∈ cl M, where cl M denotes the L 1 (dµ)-closure of M. Let log f (X j ) f ⋆ (X j ) < 0 a.s.

H n = { f /f ⋆ : f ∈ M n },
PROOF. As in the proof of Theorem 5.1, we have

1 n n j=1 log f (X j ) f ⋆ (X j ) ≤ 4n -1/2 ν n (log({ f /f ⋆ } 1/2 )) -2D(f ⋆ || f ).
The following claim will be proved below:

lim n→∞ sup f ∈M n n -1/2 ν n (log({ f /f ⋆ } 1/2
)) = 0 a.s.

Using the claim, the proof is easily completed: indeed, we then have

lim sup n→∞ sup f ∈M n 1 n n j=1 log f (X j ) f ⋆ (X j ) ≤ -2 inf f ∈M D(f ⋆ || f ) < 0 a.s.,
where the last inequality follows from Pinsker's inequality and f ⋆ ∈ cl M.

It therefore remains to prove the claim. To this end we apply [START_REF] Van De Geer | Applications of empirical process theory[END_REF], Theorem 5.11 as in the proof of [START_REF] Van De Geer | Applications of empirical process theory[END_REF], Theorem 7.4 (cf. Theorem 5.1 above), which yields

P sup f ∈M n |n -1/2 ν n (log({ f /f ⋆ } 1/2 ))| ≥ α ≤ C e -nα 2 /C for every α > 0 such that C √ p (1 + log K(n)) ≤ α √ n ≤ 32 √ n and n ≥ 1,
where C is a universal constant. As log K(n) = o(n), we have

n≥1 P sup f ∈M n |n -1/2 ν n (log({ f /f ⋆ } 1/2 ))| ≥ α < ∞
for all 0 < α ≤ 32, so the claim follows from the Borel-Cantelli lemma.

We can now complete the proof of Theorem 4.1.

PROOF OF THEOREM 4.1. By Theorem 2.3 and easy manipulations, P ⋆ -a.s.

lim sup n→∞ sup q>q ⋆ 1 pen(n, q) -pen(n, q ⋆ ) sup

f ∈M n q ℓ n (f ) -sup f ∈M n q ⋆ ℓ n (f ) ≤ lim n→∞ sup q>q ⋆
η(q){log K(2n) ∨ log log n} pen(n, q) -pen(n, q ⋆ ) × lim sup n→∞ 1 log K(2n) ∨ log log n sup

q>q ⋆ 1 η(q) sup f ∈M n q ℓ n (f ) -sup f ∈M n q ⋆
ℓ n (f ) = 0.

Therefore, P ⋆ -a.s. eventually as n → ∞ sup f ∈M n q ℓ n (f ) -pen(n, q) < sup

f ∈M n q ⋆
ℓ n (f ) -pen(n, q ⋆ ) for all q > q ⋆ . It follows that lim sup n→∞ qn ≤ q ⋆ P ⋆ -a.s., that is, the penalized likelihood order estimator does not asymptotically overestimate the order. On the other hand, we note that for every q < q ⋆ lim sup

n→∞ 1 n sup f ∈M n q ℓ n (f ) -sup f ∈M n q ⋆ ℓ n (f ) ≤ lim sup n→∞ sup f ∈M n q 1 n n j=1 log f (X j ) f ⋆ (X j )
which is strictly negative P ⋆ -a.s. by Proposition 5.18, where we have used that log K(n) = o(n) and that N(H n q (2), δ) ≤ N(H n q ⋆ (2), δ) ≤ (2K(n)/δ) η(q ⋆ ) for all δ ≤ 2 and n sufficiently large. As pen(n, q)/n → 0 as n → ∞ for q < q ⋆ lim sup

n→∞ max q<q ⋆ 1 n sup f ∈M n q ℓ n (f ) -pen(n, q) -sup f ∈M n q ⋆
ℓ n (f ) + pen(n, q ⋆ ) < 0 P ⋆ -a.s. In particular, we find that P ⋆ -a.s. eventually as n → ∞ sup f ∈M n q ℓ n (f ) -pen(n, q) < sup

f ∈M n q ⋆
ℓ n (f ) -pen(n, q ⋆ ) for all q < q ⋆ . It follows that lim inf n→∞ qn ≥ q ⋆ P ⋆ -a.s., that is, the penalized likelihood order estimator does not asymptotically underestimate the order. By Theorem 2.5, we have lim sup n→∞ 1 pen(n, q) -pen(n, q ⋆ ) sup

f ∈Mq ℓ n (f ) -sup f ∈M q ⋆
ℓ n (f ) ≥ 1 C{η(q) -η(q ⋆ )} sup P ⋆ -a.s.

Therefore, choosing C < Γ/{η(q) -η(q ⋆ )}, we find that sup f ∈Mq ℓ n (f ) -pen(n, q) > sup

f ∈M q ⋆
ℓ n (f ) -pen(n, q ⋆ ) infinitely often P ⋆ -a.s., so that qn = q ⋆ infinitely often P ⋆ -a.s. To this end, we will need the following lemma characterizing Dq ⋆ (here we adopt the same notations as in section 3.2). LEMMA 5.19. Suppose that Assumption A holds. Then we have

Dq ⋆ = L L 2 : L = q ⋆ i=1 η i f θ ⋆ i f ⋆ +β * i D 1 f θ ⋆ i f ⋆ , η i ∈ R, β i ∈ R d , q ⋆ i=1 η i = 0 .
PROOF. Let (f n ) n≥1 ⊂ M q ⋆ be such that h(f n , f ⋆ ) → 0 and d fn → d 0 ∈ Dq ⋆ . By Theorem 5.11, we may assume without loss of generality that f n = q ⋆ i=1 π n i f θ n i with θ n i → θ ⋆ i and π n i → π ⋆ i for every i = 1, . . . , q ⋆ . Taylor expansion gives

f n -f ⋆ f ⋆ = L n + R n , |R n | ≤ d 2 H 2 q ⋆ i=1 π n i θ n i -θ ⋆ i 2 ,
where

L n = q ⋆ i=1 (π n i -π ⋆ i ) f θ ⋆ i f ⋆ + π n i (θ n i -θ ⋆ i ) * D 1 f θ ⋆ i f ⋆ .
Proceeding as in Lemmas 5.16 and 5.17, we can estimate

d fn - L n L n 2 2 ≤ 2 S 2 4 {2 S 2 + 1} h(f n , f ⋆ ) + { S 2 + 1} R n 2 L n 2 .
But using Theorem 5.11, we find that for n sufficiently large

L n 2 ≥ L n 1 ≥ c ⋆ q ⋆ i=1 π n i θ n i -θ ⋆ i .
Thus we have

R n 2 L n 2 ≤ d H 2 2 2c ⋆ q ⋆ i=1 π n i θ n i -θ ⋆ i 2 q ⋆ i=1 π n i θ n i -θ ⋆ i ≤ d H 2 2 2c ⋆ max i=1,...,q ⋆ θ n i -θ ⋆ i n→∞ ---→ 0.
We have therefore shown that L n / L n 2 → d 0 in L 2 (f ⋆ dµ). Now define

η n i = π n i -π ⋆ i Z n , β n i = π n i (θ n i -θ ⋆ i ) Z n , Z n = q ⋆ i=1 {|π n i -π ⋆ i |+ π n i (θ n i -θ ⋆ i ) }.
As q ⋆ i=1 {|η n i | + β n i } = 1 for all n, we may extract a subsequence such that η n i → η i , β n i → β i , and q ⋆ i=1 {|η i | + β i } = 1. We obtain immediately

d 0 = L L 2 , L = q ⋆ i=1 η i f θ ⋆ i f ⋆ + β * i D 1 f θ ⋆ i f ⋆ .
Clearly q ⋆ i=1 η i = 0. Thus we have shown that any d 0 ∈ Dq ⋆ has the desired form. It remains to show that any function of the desired form is in fact an element of Dq ⋆ . To this end, fix η i ∈ R, β i ∈ R d with q ⋆ i=1 η i = 0, and define f t for t > 0 as

f t = q ⋆ i=1 (π ⋆ i + tη i ) f θ ⋆ i +β i t/π ⋆ i .
Clearly f t ∈ M q ⋆ for all t sufficiently small, and f t → f ⋆ as t → 0. But

f t -f ⋆ t = q ⋆ i=1 π ⋆ i f θ ⋆ i +β i t/π ⋆ i -f θ ⋆ i t + q ⋆ i=1 η i f θ ⋆ i +β i t/π ⋆ i .
Therefore clearly

1 t f t -f ⋆ f ⋆ t→0 --→ q ⋆ i=1 η i f θ ⋆ i f ⋆ + β * i D 1 f θ ⋆ i f ⋆ = L.
Using Lemma 5.16, we obtain

lim t→0 d ft = lim t→0 (f t -f ⋆ )/tf ⋆ (f t -f ⋆ )/tf ⋆ 2 = L L 2 .
Thus any function of the desired form is in Dq ⋆ , and the proof is complete.

REMARK 5.20. The proof of Lemma 5.19 in fact shows that Dq ⋆ = Dc q ⋆ .

We can now complete the proof of Propositions 4.4 and 4.5.

PROOF OF PROPOSITION 4.4. We begin by proving consistency of the penalty pen(n, q) = q ω(n). Note that by Corollary 3.4, the assumption of Corollary 4.2 holds with η(q) = 18(d + 1)q + 1 ≤ 19(d + 1)q. Thus consistency of pen(n, q) = q ω(n) follows from Corollary 4.2 using ̟(n) = ω(n)/19(d + 1).

To prove inconsistency of the penalty pen(n, q) = C q log log n with C > 0 sufficiently small, it suffices to show that Dc q ⋆ +1 \ Dq ⋆ is nonempty. Indeed, if this is the case then we can apply Corollary 4.3 with q = q ⋆ + 1, where the requisite entropy assumption follows immediately from Theorem 4.1.

Fix v ∈ R d , and consider the function f t defined for t > 0 as follows:

f t = π ⋆ 1 2 (f θ ⋆ 1 +vt + f θ ⋆ 1 -vt ) + q ⋆ i=2 π ⋆ i f θ ⋆ i .
Clearly f t ∈ M q ⋆ +1 for all t sufficiently small, f t → f ⋆ as t → 0, and

f t -f ⋆ t 2 = π ⋆ 1 2 f θ ⋆ 1 +vt -2 f θ ⋆ 1 + f θ ⋆ 1 -vt t 2 t→0 --→ π ⋆ 1 2 v * D 2 f θ ⋆ 1 v.
As in the proof of Lemma 5.19, we find that

lim t→0 d ft = lim t→0 (f t -f ⋆ )/t 2 f ⋆ (f t -f ⋆ )/t 2 f ⋆ 2 = v * D 2 f θ ⋆ 1 v v * D 2 f θ ⋆ 1 v 2 = d 0 .
By construction, d 0 ∈ Dc q ⋆ +1 . But by Theorem 5.11, the functions f θ ⋆ i , D 1 f θ ⋆ i , and v * D 2 f θ ⋆ i v (i = 1, . . . , q ⋆ ) are all linearly independent. Together with Lemma 5.19, this shows that d 0 ∈ Dq ⋆ . Thus d 0 ∈ Dc q ⋆ +1 \ Dq ⋆ , and the proof is complete.

PROOF OF PROPOSITION 4.5. By Example 3.5, the assumption of Theorem 4.1 holds with η(q) = 18(d + 1)q + 1 and log K(n) = log C ⋆ 1 + C ⋆ 2 T (n) 2 . The desired consistency results now follow immediately from Theorem 4.1.

FIG 1 .

 1 FIG 1. Let f θ (x) = 2/π e -2(x-θ) 2 and f ⋆ = f0.5, and consider the mixture family M2 = {pf θ 1 + (1 -p)f θ 2 : p, θ1, θ2 ∈ [0, 1]}. The plots illustrate (a) the set of parameters (p, θ1, θ2) corresponding to the Hellinger ball {f ∈ M2 : h(f, f ⋆ ) ≤ 0.05}; and (b) the set of parameters {(p, θ1, θ2) : N (p, θ1, θ2) ≤ 0.05} with N (p, θ1, θ2) = |p(θ1 -0.5)+(1-p)(θ2 -0.5)|+ 1 2 p(θ1 -0.5) 2 + 1 2 (1 -p)(θ2 -0.5) 2 . The two plots are related by the local geometry Theorem 5.11, which yields c ⋆ N (p, θ1, θ2) ≤ h(pf θ 1 + (1 -p)f θ 2 , f ⋆ ) ≤ C ⋆ N (p, θ1, θ2) for all p, θ1, θ2 ∈ [0, 1].

4 √

 4 σn (d)) 2 + -( d, g ) 2 + | ≤ sup d∈cl Dq |I σn (d) -d, g | 2 + 2 sup d∈cl Dq |I σn (d) -d, g | sup d∈cl Dq | d, g | n→∞ ---→ 0,where we have used the elementary estimate|a 2 + -b 2 + | = |a + -b + |(a + + b + ) ≤ |a + -b + |(|a + -b + | + 2b + ) ≤ |a -b|(|a -b| + 2|b|) forany a, b ∈ R, and the fact that sup d∈cl Dq | d, g | ≤ sup d∈cl Dq d 2 g 2 ≤ 1. Thus (5log log σn/σn)

we have sup f ∈Mq( 4 √

 4 log log σn/σn) ( d f , g )

5. 4 . 5 . 4 . 1 .

 4541 Proof of Theorem 3.3. The local geometry of mixtures. Define the Euclidean balls B(θ, ε) = {θ ′ ∈ R d : θ -θ ′ < ε}, denote by u, v the inner product of two vectors u, v ∈ R d , and denote by A, u = { θ, u : θ ∈ A} ⊆ R the inner product of a set A ⊆ R d with a vector u ∈ R d .

  A ⊆ R d , and denote by M d + the family of all d × d positive semidefinite (symmetric) matrices. DEFINITION 5.10. Let us write

COROLLARY 5 .

 5 15. |d| ≤ D for all d ∈ D, where D = 2S ∈ L 4 (f ⋆ dµ).

5. 5 .

 5 Proof of Theorem 4.1. The proof of Theorem 4.1 is based on Theorem 2.3 and the following result.

Finally, let us prove Corollary 4. 3 .

 3 PROOF OF COROLLARY 4.3. It is shown in the proof of Corollary 2.6 that

5. 6 .

 6 Proof of Proposition 4.4. The proofs of the consistency results in Propositions 4.4 and 4.5 follow almost immediately from Theorem 4.1, Corollary 3.4, and Example 3.5. The main difficulty is to establish the condition Dc q \ Dq ⋆ = ∅ of Corollary 4.3, which is needed to prove the inconsistency part of Proposition 4.4.

  the supremum over d ∈ Dq and the infimum over d ′ ∈ Dc p , we find that

	sup d∈ Dq	(I n (d)) 2 + -sup p d∈ Dc	(I n (d)) 2 +
		≤ 2 sup d∈ Dq	|(I n (d)) 2 + -( d, g ) 2 + | + sup d∈ Dq	( d, g ) 2 + -sup p d∈ Dc	( d, g ) 2 +
		≤ 2 sup d∈ Dq	|(I n (d)) 2 + -( d, g ) 2 + |
						+	sup g∈L 2 0 (f ⋆ dµ)	sup d∈ Dq	( d, g ) 2 + -sup p d∈ Dc	( d, g ) 2 + .
	But as this holds for any g ∈ L 2 0 (f ⋆ dµ), we finally obtain
	sup d∈ Dq	(I n (d)) 2 + -sup d∈ Dc p	(I n (d)) 2 + ≤ 2	inf g∈L 2 0 (f ⋆ dµ)	sup d∈ Dq	|(I n (d)) 2 + -( d, g ) 2 + |
							+	sup g∈L 2 0 (f ⋆ dµ)	sup d∈ Dq	( d, g ) 2 + -sup d∈ Dc
	Finally, we now complete the proof of Corollary 2.6
	PROOF OF COROLLARY 2.6. It evidently suffices to prove that
	(5.3)		Γ :=	sup g∈L 2 0 (f ⋆ dµ)	sup d∈ Dc q	( d, g ) 2 + -sup d∈ Dq ⋆	( d, g ) 2 +

p ( d, g ) 2 + .

It follows as in the proof of Proposition 5.7 that the first term in this expression converges to zero P ⋆ -a.s. The requisite upper bound follows immediately.

  and suppose there exist K(n) ≥ 1 and p ≥ 1 so that ≤ 1 and n ≥ 1, where N(H n , δ) is the minimal number of brackets of L 2 (f ⋆ dµ)-width δ needed to cover H n . Let (X i ) i∈N be i.i.d. with distribution f ⋆ dµ. If in addition log K(n) = o(n), then we have

		N(H n , δ) ≤	K(n) δ
	lim sup n→∞	sup f ∈M n	1 n	n j=1

p for all δ

Our terminology is in analogy with the usual notion of a state on the C * -algebraC d×d ⊗ C C (Ω),where Ω is a compact metric space and C C (Ω) is the algebra of complex-valued continuous functions on Ω. Such states are precisely represented by the complex-valued counterpart of our definition.

Acknowledgments. The authors would like to thank Jean Bretagnolle for providing an enlightening counterexample which guided some of our proofs. We also thank Michel Ledoux for suggesting some helpful references.