Leonid Galtchouk

Sergey Pergamenshchikov

Adaptive asymptotically efficient estimation in heteroscedastic nonparametric regression via model selection

The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating a unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimates. 1 2

Introduction

The paper deals with the estimation problem in the heteroscedastic nonparametic regression model y j = S(x j ) + σ j (S) ξ j ,

where the design points x j = j/n, S(•) is an unknown function to be estimated, (ξ j ) 1≤j≤n is a sequence of centered independent random variables with unit variance and (σ j (S)) 1≤j≤n are unknown scale functionals depending on the design points and the regression function S.

Typically, the notion of asymptotic optimality is associated with the optimal convergence rate of the minimax risk (see e.g., Ibragimov, Hasminskii,1981;Stone,1982). An important question in optimality results is to study the exact asymptotic behavior of the minimax risk. Such results have been obtained only in a limited number of investigations. As to the nonparametric estimation problem for heteroscedastic regression models we should mention the papers by [START_REF] Efromovich | Sequential design and estimation in heteroscedastic nonparametric regression[END_REF][START_REF] Efromovich | Sharp -optimal and adaptive estimation for heteroscedastic nonparametric regression[END_REF][START_REF] Galtchouk | Efficient adaptive nonparametric estimation in heteroscedastic regression models[END_REF], concerning the exact asymptotic behavior of the L 2 -risk and the paper by [START_REF] Brua | Asymptotic efficient estimators for non-parametric heteroscedastic model[END_REF], devoted to the efficient pointwise estimation for heteroscedastic regressions.

Heteroscedastic regression models are largely used in financial mathematics, in particular, in problem of calibrating (see e.g., [START_REF] Belomestny | Spectral calibration of exponential Lévy models[END_REF]). An example of heteroscedastic regression models is given by econometrics (see, for example, Goldfeld, Quandt, 1972, p. 83), where for consumer budget problems one uses some parametric version of model (1.1) with the scale coefficients defined as

σ 2 j (S) = c 0 + c 1 x j + c 2 S 2 (x j ) , (1.2)
where c 0 , c 1 and c 2 are some unknown positive constants. The purpose of the article is to study asymptotic properties of the adaptive estimation procedure proposed in [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF], for which a non-asymptotic oracle inequality was proved for quadratic risks. We will prove that this oracle inequality is asymptotically sharp, i.e. the asymptotic quadratic risk is minimal. It means the adaptive estimation procedure is efficient under some the conditions on the scales (σ j (S)) 1≤j≤n which are satisfied in the case (1.2). Note that in [START_REF] Efromovich | Sequential design and estimation in heteroscedastic nonparametric regression[END_REF][START_REF] Efromovich | Sharp -optimal and adaptive estimation for heteroscedastic nonparametric regression[END_REF], an efficient adaptive procedure is constructed for heteroscedastic regression when the scale coefficient is independent of S, i.e. σ j (S) = σ j .

In [START_REF] Galtchouk | Efficient adaptive nonparametric estimation in heteroscedastic regression models[END_REF], for the model (1.1) the asymptotic efficiency was proved under strong the conditions on the scales which are not satisfied in the case (1.2). Moreover in the cited papers the efficiency was proved for the gaussian random variables (ξ j ) 1≤j≤n that is very restrictive for applications of proposed methods to practical problems.

In the paper we modify the risk. We take a additional supremum over the family of unknown noise distributions like to [START_REF] Galtchouk | Asymptotically efficient estimates for nonparametric regression models[END_REF]. This modification allows us to eliminate from the risk dependence on the noise distribution. Moreover for this risk a efficient procedure is robust with respect to changing the noise distribution.

It is well known to prove the asymptotic efficiency one has to show that the asymptotic quadratic risk coincides with the lower bound which is equal to the Pinsker constant. In the paper two problems are resolved: in the first one a upper bound for the risk is obtained by making use of the non-asymptotic oracle inequality from [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF], in the second one we prove that this upper bound coincides with the Pinsker constant. Let us remember that the adaptive procedure proposed in Galtchouk, Pergamenshchikov, 2007, is based on weighted least-squares estimates, where the weights are proper modifications of the Pinsker weights for the homogeneous case (when σ 1 (S) = . . . = σ n (S) = 1) relative to a certain smoothness of the function S and this procedure chooses a best estimator for the quadratic risk among these estimators. To obtain the Pinsker constant for the model (1.1) one has to prove a sharp asymptotic lower bound for the quadratic risk in the case when the noise variance depends on the unknown regression function. In this case, as usually, we minorize the minimax risk by a bayesian one for a respective parametric family. Then for the bayesian risk we make use of a lower bound (see Theorem 6.1) which is a modification of the van Trees inequality (see, [START_REF] Gill | Application of the van Trees inequality: a Bayesian Cramér-Rao bound[END_REF].

The paper is organized as follows. In Section 2 we construct an adaptive estimation procedure. In Section 3 we formulate principal the conditions. The main results are presented in Section 4. The upper bound for the quadratic risk is given in Section 5. In Section 6 we give all main steps of proving the lower bound. In Subsection 6.1 we find the lower bound for the bayesian risk which minorizes the minimax risk. In Subsection 6.2 we study a special parametric functions family used to define the bayesian risk. In Subsection 6.3 we choose a prior distribution for bayesian risk to maximize the lower bound. Section 7 is devoted to explain how to use the given procedure in the case when the unknown regression function is non periodic.

In Section 8 we discuss the main results and their practical importance. The proofs are given in Section 9. The Appendix contains some technical results.

Adaptive procedure

In this section we describe the adaptive procedure proposed in [START_REF] Galtchouk | Asymptotically efficient estimates for nonparametric regression models[END_REF]. We make use of the standard trigonometric basis

(φ j ) j≥1 in L 2 [0, 1], i.e. φ 1 (x) = 1 , φ j (x) = √ 2 T r j (2π[j/2]x) , j ≥ 2 , (2.1)
where the function T r j (x) = cos(x) for even j and T r j (x) = sin(x) for odd j; [x] denotes the integer part of x.

To evaluate the error of estimation in the model (1.1) we will make use of the empiric norm in the Hilbert space L 2 [0, 1], generated by the design points (x j ) 1≤j≤n of model (1.1). To this end, for any functions u and v from L 2 [0, 1], we define the empiric inner product

(u , v) n = 1 n n l=1 u(x l ) v(x l ) .
Moreover, we will use this inner product for vectors in R n as well, i.e. if u = (u 1 , . . . , u n

) ′ and v = (v 1 , . . . , v n ) ′ , then (u , v) n = 1 n u ′ v = 1 n n l=1 u l v l .
The prime denotes the transposition. Notice that if n is odd, then the functions (φ j ) 1≤j≤n are orthonormal with respect to this inner product, i.e. for any 1 ≤ i, j ≤ n,

(φ i , φ j ) n = 1 n n l=1 φ i (x l )φ j (x l ) = Kr ij , (2.2)
where Kr ij is Kronecker's symbol, Kr ij = 1 if i = j and Kr ij = 0 for i = j.

Remark 2.1. Note that in the case of even n, the basis (2.1) is orthogonal and it is orthonormal except the nth function for which the normalizing constant should be changed. The corresponding modifications of the formulas for even n one can see in Galtchouk, Pergamenshchikov,2005. To avoid these complications of formulas related to even n, we suppose n to be odd.

Thanks to this basis we pass to the discrete Fourier transformation of model (1.1):

θ j,n = θ j,n + 1 √ n ξ j,n , (2.3)
where θ j,n = (Y, φ j ) n , Y = (y 1 , . . . , y n ) ′ , θ j,n = (S, φ j ) n and

ξ j,n = 1 √ n n l=1 σ l (S)ξ l φ j (x l ) .
We estimate the function S by the weighted least squares estimator

S λ = n j=1 λ(j) θ j,n φ j , (2.4)
where the weight vector λ = (λ(1), . . . , λ(n)) ′ belongs to some finite set Λ from [0, 1] n with n ≥ 3.

Here we make use of the weight family Λ introduced in Galtchouk, Pergamenshchikov, 2008, i.e.

Λ = {λ α , α ∈ A} , A = {1, . . . , k * } × {t 1 , . . . , t m } , (2.5)
where t i = iε and m = [1/ε 2 ]. We suppose that the parameters k * ≥ 1 and 0 < ε ≤ 1 are functions of n, i.e.

k * = k * n and ε = ε n , such that,    lim n→∞ k * n = +∞ , lim n→∞ k * n ln n = 0 , lim n→∞ ε n = 0 and lim n→∞ n ν ε n = +∞ , (2.6)
for any ν > 0. For example, one can take for n ≥ 3

ε n = 1/ ln n and k * n = k + √ ln n ,
where k is any nonnegative constant.

For each α = (β, t) ∈ A we define the weight vector

λ α = (λ α (1), . . . , λ α (n)) ′ as λ α (j) = 1 {1≤j≤j 0 } + 1 -(j/ω(α)) β 1 {j 0 <j≤ω(α)} . (2.7)
Here

j 0 = j 0 (α) = [ω(α) ε n ] with ω(α) = ω + (A β t) 1/(2β+1) n 1/(2β+1) , (2.8)
where ω is any nonnegative constant and

A β = (β + 1)(2β + 1) βπ 2β .
Remark 2.2. Note that the weighted least squares estimators (2.4) have been introduced by Pinsker, 1981, for continuous time optimal signal filtering in the gaussian noise. He proved that the mean-square asymptotic risk is minimized by weighted least squares estimators with weights of type (2.7). Moreover he has found the sharp minimal value of the mean-square asymptotic risk, which was called later as the Pinsker constant. Nussbaum, 1985, used the same method with proper modification for efficient estimation of the function S of known smoothness in the homogeneous gaussian model (1.1), i.e. when σ 1 (S) = . . . = σ n (S) = 1 and (ξ j ) 1≤j≤n is i.i.d. N (0, 1) sequence.

To choose weights from the set (2.5) we minimize the special cost function introduced by Galtchouk, Pergamenshchikov, 2007. This cost function is as follows

J n (λ) = n j=1 λ 2 (j) θ 2 j,n -2 n j=1 λ(j) θ j,n + ρ P n (λ) , (2.9)
where

θ j,n = θ 2 j,n - 1 n ς n with ς n = n j=l n +1 θ 2 j,n (2.10)
and l n = [n 1/3 + 1]. The penalty term we define as

P n (λ) = |λ| 2 ς n n , |λ| 2 = n j=1 λ 2 (j) and ρ = 1 3 + L n ,
where L n ≥ 0 is any slowly increasing sequence, i.e.

lim n→∞ L n = +∞ and lim n→∞ L n n ν = 0 , (2.11)
for any ν > 0. Finally, we set λ = argmin λ∈Λ J n (λ) and S * = S λ .

(2.12)

The goal of this paper is to study asymptotic (as n → ∞) properties of this estimation procedure.

Remark 2.3. Now we explain why does one choose the cost function in the form (2.9). Developing the empiric quadratic risk for estimate (2.4), one obtains

S λ -S 2 n = n j=1 λ 2 (j) θ 2 j,n -2 n j=1 λ(j) θ j,n θ j,n + S 2 n .
It's natural to choose the weight vector λ for which this function reaches the minimum. Since the last term on the right-hand part is independent of λ, it can be droped and one has to minimize with respect to λ the function equals the difference of two first terms on the right-hand part. It's clear that the minimization problem cann't be solved directly because the Fourier coefficients (θ j,n ) are unknown.To overcome this difficulty, we replace the product θ j,n θ j,n by its asymptotically unbiased estimator θ j,n (see, [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF][START_REF] Galtchouk | Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models[END_REF]. Moreover, to pay this substitution, we introduce into the cost function the penalty term P n with a small coefficient ρ > 0. The form of the penalty term is provided by the principal term of the quadratic risk for weighted least-squares estimator, see [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF][START_REF] Galtchouk | Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models[END_REF].The coefficient ρ > 0 means, that the penalty is small, because the estimator θ j,n approximates in mean the quantity θ j,n θ j,n asymptotically, as n → ∞.

Note that the principal difference between the procedure (2.12) and the adaptive procedure proposed by [START_REF] Golubev | Adaptive spline estimates in a nonparametric regression model[END_REF], for a homogeneous gaussian regression, consists in presence of the penalty term in the cost function (2.9). Remark 2.4. As it was noted at Remark 2.2, Nussbaum, 1985, has shown that the weight coefficients of type (2.7) provide the asymptotic minimum of the mean-squared risk at the regression function estimation problem for the homogeneous gaussian model (1.1), when the smoothness of the function S is known. In fact, to obtain an efficient estimator one needs to take a weighted least squares estimator (2.4) with the weight vector λ α , where the index α depends on smoothness of function S and on coefficients (σ j (S)) 1≤j≤n , (see (5.2) below), which are unknown in our case. For this reason, [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF][START_REF] Galtchouk | Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models[END_REF], have proposed to make use of the family of coefficients (2.5), which contains the weight vector providing the minimum of the mean-squared risk. Moreover, they proposed the adaptive procedure (2.12) for which a non-asymptotic oracle inequality (see, Theorem 4.1 below) was proved under some weak conditions on the coefficients (σ j (S)) 1≤j≤n . It is important to note that due the properties of the parametric family (2.6), the secondary term in the oracle inequality is slowly increasing (slower than any degree of n).

Conditions

First we impose some conditions on unknown function S in the model (1.1).

Let C k per,1 (R) be the set of 1-periodic k times differentiable R → R functions. We assume that S belongs to the following set

W k r = {f ∈ C k per,1 (R) : k j=0 f (j) 2 ≤ r} , (3.1)
where • denotes the norm in L 2 [0, 1], i.e.

f 2 = 1 0 f 2 (t)dt . (3.2)
Moreover, we suppose that r > 0 and k ≥ 1 are unknown parameters. Note that, we can represent the set W k r as an ellipse in L 2 [0, 1], i.e.

W k r = {f ∈ L 2 [0, 1] : ∞ j=1 a j θ 2 j ≤ r} , (3.3)
where

θ j = (f, φ j ) = 1 0 f (t)φ j (t)dt (3.4)
and

a j = k l=0 φ (l) j 2 = k i=0 (2π[j/2]) 2i . (3.5)
Here (φ j ) j≥1 is the trigonometric basis defined in (2.1). Now we describe the conditions on the scale coefficients (σ j (S)) j≥1 .

H 1 ) σ j (S) = g(x j , S) for some unknown function g

: [0, 1] × L 1 [0, 1] → R + ,
which is square integrable with respect to x such that

lim n→∞ sup S∈W k r 1 n n j=1 g 2 (x j , S) -ς(S) = 0 , (3.6)
where ς(S) := 1 0 g 2 (x, S)dx. Moreover,

g * = inf 0≤x≤1 inf S∈W k r g 2 (x, S) > 0 (3.7)
and sup

S∈W k r ς(S) < ∞ . (3.8) H 2 ) For any x ∈ [0, 1], the operator g 2 (x, •) : C[0, 1] → R is differentiable
in the Fréchet sense for any fixed function f 0 from C[0, 1] , i.e. for any f from some vicinity of f 0 in C[0, 1],

g 2 (x, f ) = g 2 (x, f 0 ) + L x,f 0 (f -f 0 ) + Υ(x, f 0 , f ) ,
where the Fréchet derivative L x,f 0 : C[0, 1] → R is a bounded linear operator and the residual term Υ(x, f 0 , f ), for each x ∈ [0, 1], satisfies the following property:

lim f -f 0 ∞ →0 |Υ(x, f 0 , f )| f -f 0 ∞ = 0 , where f ∞ = sup 0≤t≤1 |f (t)|.
H 3 ) There exists some positive constant C * such that for any function S from C[0, 1] the operator L x,S defined in the condition H 2 ) satisfies the following inequality for any function f from C[0, 1]:

|L x,S (f )| ≤ C * (|S(x)f (x)| + |f | 1 + S f ) , (3.9
)

where |f | 1 = 1 0 |f (t)|dt. H 4 ) The function g 0 (•) = g(•, S 0 ) corresponding to S 0 ≡ 0 is continuous on the interval [0, 1]. Moreover, lim δ→0 sup 0≤x≤1 sup S ∞ ≤δ |g(x, S) -g(x, S 0 )| = 0 . Remark 3.1.
Let us explain the conditions H 1 )-H 4 ). In fact,this is the regularity conditions of the function g(x, S) generating the scale coefficients (σ j (S)) 1≤j≤n .

Condition H 1 ) means that the function g(•, S) should be uniformly integrable with respect to the first argument in the sens of convergence (3.6). Moreover, this function should be separated from zero (see inequality (3.7)) and bounded on the class (3.1) (see inequality (3.8)). Boundedness away from zero provides that the distribution of observations (y j ) 1≤j≤n isn't degenerate in R n , and the boundedness means that the intensity of the noise vector should be finite, otherwise the estimation problem hasn't any sens.

Conditions H 2 ) and H 3 ) mean that the function g(x, •) is regular, at any fixed 0 ≤ x ≤ 1, with respect to S in the sens, that it is differentiable in the Frèchet sens (see e.g., [START_REF] Kolmogorov | Elements of the theory of functions and functional analysis[END_REF]) and moreover the Frèchet derivative satisfies the growth condition given by the inequality (3.9) which permits to consider the example (1.2).

Last the condition H 4 ) is the usual uniform continuity the condition of the function g(•, •) at the function S 0 .

Now we give some examples of functions satisfying the conditions H

1 )- H 4 ).
We set

g 2 (x, S) = c 0 + c 1 x + c 2 S 2 (x) + c 3 1 0 S 2 (t)dt (3.10)
with some coefficients c 0 > 0, c i ≥ 0, i = 1, 2, 3. In this case

ς(S) = c 0 + c 1 2 + (c 2 + c 3 ) 1 0 S 2 (t)dt .
The Fréchet derivative is given by

L x,S (f ) = 2S(x)f (x) + 2 1 0 S(t)f (t)dt .
It is easy to see that the function (3.10) satisfies the conditions H 1 )-H 4 ). Moreover, the conditions H 1 )-H 4 ) are satisfied by any function of type

g 2 (x, S) = G(x, S(x)) + 1 0 V (S(t))dt , (3.11)
where the functions G and V satisfy the following the conditions:

• G is a [0, 1] × R → [c 0 , +∞) function (with c 0 > 0) such that lim δ→0 max |u-v|≤δ sup y∈R |G(u, y) -G(v, y)| = 0 (3.12)
and

m 1 = sup 0≤x≤1 sup y∈R |G y (x, y)| |y| < ∞ ; (3.13) • V is a continuously differentiable R → R + function such that m 2 = sup y∈R | V (y)| 1 + |y| < ∞ , where V (•) is the derivative of V .
In this case

ς(S) = 1 0 G(t, S(t))dt + 1 0 V (S(t))dt and n -1 n j=1 g 2 (x j , S) -ς(S) ≤ n j=1 x j x j-1 G(x j , S(x j )) -G(t, S(t)) dt ≤ ∆ n + n j=1 x j x j-1 G(t, S(x j )) -G(t, S(t)) dt ,
where ∆ n = max |u-v|≤1/n sup y∈R |G(u, y) -G(v, y)|. Now to estimate the last term in this inequality note that

G(t, S(x j )) -G(t, S(t)) = x j t G y (t, S(z)) Ṡ(z) dz .
Therefore, from the condition (3.13) we get

|G(t, S(x j )) -G(t, S(t))| ≤ m 1 x j x j-1 |S(z)| | Ṡ(z)| dz ,
and through the Bounyakovskii-Cauchy-Schwarz inequality, for any S ∈ W k r ,

n -1 n j=1 g 2 (x j , S) -ς(S) ≤ n j=1 x j x j-1 G(x j , S(x j )) -G(t, S(t)) dt ≤ ∆ n + m 1 n 1 0 |S(t)| | Ṡ(t)|dt ≤ ∆ n + m 1 n S Ṡ ≤ ∆ n + m 1 n r .
Now, the condition (3.12) implies H 1 ). Moreover, the Fréchet derivative in this case is given by

L x,S (f ) = G y (x, S(x))f (x) + 1 0 V (S(t))f (t)dt .
One can check directly that this operator satisfies the inequality (3.9) with

C * = m 1 + m 2 .

Main results

Denote by P n the family of distributions p in R n of the vectors (ξ 1 , . . . , ξ n ) ′ in the model (1.1) such that the components ξ j are jointly independent, centered with unit variance and max

1≤k≤n E ξ 4 k ≤ l * n , (4.1)
where l * n ≥ 3 is slowly increasing sequence, that is it satisfies the property (2.11).

It is easy to see that, for any n ≥ 1, the centered gaussian distribution in R n with unit covariation matrix belongs to the family P n . We will denote by q this gaussian distribution.

For any estimator S we define the following quadratic risk

R n ( S, S) = sup p∈P n E S,p S -S 2 n , (4.2)
where E S,p is the expectation with respect to the distribution P S,p of the observations (y 1 , . . . , y n ) with the fixed function S and the fixed distribution p ∈ P n of random variables (ξ j ) 1≤j≤n in the model (1.1).

Moreover, to make the risk independent of the design points, in this paper we will make use of the risk with respect to the usual norm in L 2 [0, 1] (3.2) also, i.e.

T n ( S, S) = sup

p∈P n E S,p S -S 2 . (4.3)
If an estimator S is defined only at the design points (x j ) 1≤j≤n , then we extend it as step function onto the interval [0, 1] by setting S(x) = T ( S(x)), for all 0 ≤ x ≤ 1, where

T (f )(x) = f (x 1 )1 [0,x 1 ] (x) + n k=2 f (x k )1 (x k-1 ,x k ] (x) . (4.4)
In [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF][START_REF] Galtchouk | Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models[END_REF] the following non-asymptotic oracle inequality has been shown for the procedure (2.12) . Theorem 4.1. Assume that in the model (1.1) the function S belongs to W 1 r . Then, for any odd n ≥ 3 and r > 0, the estimator S * satisfies the following oracle inequality

R n ( S * , S) ≤ 1 + 3ρ -2ρ 2 1 -3ρ min λ∈Λ R n ( S λ , S) + 1 n B n (ρ) , (4.5)
where the function B n (ρ) is such that, for any ν > 0,

lim n→∞ B n (ρ) n ν = 0 . (4.6)
Remark 4.1. Note that in [START_REF] Galtchouk | Adaptive nonparametric estimation in heteroscedastic regression models. Part 1. Sharp non-asymptotic oracle inequalities[END_REF][START_REF] Galtchouk | Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models[END_REF], the oracle inequality is proved for the model (1.1), where the random variables (ξ j ) 1≤j≤n are independent identically distributed. In fact, the result and the proof are true for independent random variables which are not identically distributed, i.e. for any distribution of the random vector (ξ 1 , . . . , ξ n ) ′ from P n .

Now we formulate the main asymptotic results. To this end, for any function S ∈ W k r , we set

γ k (S) = Γ * k r 1 2k+1 (ς(S)) 2k/(2k+1) , (4.7)
where Γ * k = (2k + 1)

1 2k+1 (k/(π (k + 1))) 2k/(2k+1) .
It is well known (see e.g., [START_REF] Nussbaum | Spline smoothing in regression models and asymptotic efficiency in L 2[END_REF] that the optimal rate of convergence is n 2k/(2k+1) when the risk is taken uniformly over W k r . Theorem 4.2. Assume that in the model (1.1) the sequence (σ j (S)) fulfills the condition H 1 ). Then the estimator S * from (2.12) satisfies the inequalities

lim sup n→∞ n 2k 2k+1 sup S∈W k r R n ( S * , S) γ k (S) ≤ 1 (4.8)
and

lim sup n→∞ n 2k 2k+1 sup S∈W k r T n ( S * , S) γ k (S) ≤ 1 . (4.9)
The following result gives the sharp lower bound for risk (4.2) and show that γ k (S) is the Pinsker constant. In fact the upper bound is true there under some additional condition on the smoothness of the function S, i.e. on the parameter k. In the cited paper this additional condition is not formulated since erroneous inequality (A.6). To avoid using this inequality we modify the estimating procedure by introducing the penalty term ρ P n (λ) in the cost function (2.9). By this way we remove all additional conditions on the smoothness parameter k.

Remark 4.3. In fact to obtain the non-asymptotic oracle inequality (4.5), it isn't necessary to make use of equidistant design points and the trigonometric basis. One may take any design points (deterministic or random) and any orthonormal basis satisfying (2.2). But to obtain the property (4.6) one needs to impose some technical conditions (see [START_REF] Galtchouk | Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models[END_REF]. Note that the results of Theorem 4.2 and Theorem 4.3 are based on equidistant design points and the trigonometric basis.

Upper bound

In this section we prove Theorem 4.2. To this end we will make use of the oracle inequality (4.5). We have to find an estimator from the family (2.4)-(2.5) for which we can show the upper bound (4.8). We start with the construction of such an estimator. First we put

l n = inf{i ≥ 1 : iε ≥ r(S)} ∧ m and r(S) = r/ς(S) , (5.1)
where a ∧ b = min(a, b).

Then we choose an index from the set A ε as

α = (k, t n ) , (5.2)
where k is the parameter of the set W k r and t n = l n ε. Finally, we set S = S λ and λ = λ α .

(5.3)

Now we show the upper bound (4.8) for this estimator.

Theorem 5.1. Assume that the condition H 1 ) holds. Then

lim sup n→∞ n 2k 2k+1 sup S∈W k r R n ( S, S) γ k (S) ≤ 1 .
(5.4)

Remark 5.1. Note that the estimator S belongs to the family (2.4)-(2.5), but we can't use directly this estimator because the parameters k, r and r(S) are unknown. We can use this upper bound only through the oracle inequality (4.5) proved for procedure (2.12). Now Theorem 4.1 and Theorem 5.1 imply the upper bound (4.8). To obtain the upper bound (4.9) we need the following auxiliary result. Lemma 5.2. For any 0 < δ < 1 and any estimate

S n of S ∈ W k r , S n -S 2 n ≥ (1 -δ) T n ( S) -S 2 -(δ -1 -1) r/n 2 , where the function T n ( S)(•) is defined in (4.4).
Proof of this Lemma is given in Appendix A.1. Now inequality (4.8) and this lemma imply the upper bound (4.9). Hence Theorem 4.2.

6

Lower bound

In this section we give the main steps of proving the lower bounds (4.10) and (4.11). In common, we follow the same scheme as Nussbaum, 1985.

We begin with minorizing the minimax risk by a bayesian one constructed on a parametric functional family introduced in Section 6.2 ( see (6.9)) and using the prior distribution (6.10). Further, a special modification of the van Trees inequality (see, Theorem 6.1) yields a lower bound for the bayesian risk depending on the chosen prior distribution, of course. Finally, in section 6.3, we choose parameters of the prior distribution (see (6.10)) providing the maximal value of the lower bound for the bayesian risk. This value coincides with the Pinsker constant as it is shown in Section 9.2.

Lower bound for parametric heteroscedastic regression models

Let (R n , B(R n ), P ϑ , ϑ ∈ Θ ⊆ R l ) be a statistical model relative to the observations (y j ) 1≤j≤n governed by the regression equation

y j = S ϑ (x j ) + σ j (ϑ) ξ j , (6.1)
where ξ 1 , . . . , ξ n are i.i.d. N (0, 1) random variables, ϑ = (ϑ 1 , . . . , ϑ l ) ′ is a unknown parameter vector, S ϑ (x) is a unknown (or known) function and σ j (ϑ) = g(x j , S ϑ ), with the function g(x, S) defined in the condition H 1 ). Assume that a prior distribution µ ϑ of the parameter ϑ in R l is defined by the density Φ(•) of the following form

Φ(z) = Φ(z 1 , . . . , z l ) = l i=1 ϕ i (z i ) ,
where ϕ i is a continuously differentiable bounded density on R with

I i = R φ2 i (u) ϕ i (u) du < ∞ .
Let τ (•) be a continuously differentiable R l → R function such that, for any 1 ≤ i ≤ l,

lim |z i |→∞ τ (z) ϕ i (z i ) = 0 and R l τ ′ i (z) Φ(z)dz < ∞ , (6.2)
where

τ ′ i (z) = (∂/∂z i ) τ (z)
. Let τ n be an estimator of τ (ϑ) based on observations (y j ) 1≤j≤n . For any

B(R n × R l ) -measurable integrable function G(x, z), x ∈ R n , z ∈ R l , we set E G(Y, ϑ) = R l E z G(Y, z) Φ(z) dz ,
where E ϑ is the expectation with respect to the distribution P ϑ of the vector Y = (y 1 , . . . , y n ). Note that in this case

E ϑ G(Y, ϑ) = R n G(v, ϑ) f (v, ϑ) dv , where f (v, z) = n j=1 1 √ 2πσ j (z) exp - (v j -S z (x j )) 2 2σ 2 j (z) . (6.3)
We prove the following result.

Theorem 6.1. Assume that the conditions H 1 ) -H 2 ) hold. Moreover, assume that the function S z (•) with z = (z 1 , . . . , z l ) ′ is uniformly over 0 ≤ x ≤ 1 differentiable with respect to z i , 1 ≤ i ≤ l, i.e. for any 1 ≤ i ≤ l there exists a function

S ′ z,i ∈ C[0, 1] such that lim h→0 max 0≤x≤1 S z+he i (x) -S z (x) -S ′ z,i (x)h /h = 0 , (6.4)
where e i = (0, ...., 1, ..., 0) ′ , all coordinates are 0, except the i-th equals to 1 . Then for any square integrable estimator τ n of τ (ϑ) and any

1 ≤ i ≤ l, E ( τ n -τ (ϑ)) 2 ≥ τ 2 i F i + B i + I i , (6.5)
where

τ i = R l τ ′ i (z) Φ(z)dz, F i = n j=1 R l (S ′ z,i (x j )/σ j (z)) 2 Φ(z)dz and B i = 1 2 n j=1 R l L 2 i (x j , S z ) σ 4 j (S z ) Φ(z)dz , L i (x, z) = L x,S z (S ′ z,i ), the operator L x,S is defined in the condition H 2 ). Proof is given in Appendix A.2.
Remark 6.1. Note that the inequality (6.5) is some modification of the van Trees inequality (see, [START_REF] Gill | Application of the van Trees inequality: a Bayesian Cramér-Rao bound[END_REF] adapted to the model (6.1).

Parametric family of kernel functions

In this section we define and study some special parametric family of kernel function which will be used to prove the sharp lower bound (4.10).

Let us begin by kernel functions. We fix η > 0 and we set

χ η (x) = η -1 R 1 (|u|≤1-η) V u -x η du , (6.6)
where 1 A is the indicator of a set A, the kernel V ∈ C ∞ (R) is such that

V (u) = 0 for |u| ≥ 1 and 1 -1 V (u) du = 1 .
It is easy to see that the function χ η (x) possesses the properties :

0 ≤ χ η ≤ 1 , χ η (x) = 1 for |x| ≤ 1 -2η and χ η (x) = 0 for |x| ≥ 1 .
Moreover, for any c > 0 and ν ≥ 0

lim η→0 sup f : f ∞ ≤c R f (x) χ ν η (x)dx - 1 -1 f (x)dx = 0 . (6.7)
We divide the interval [0, 1] into M equal subintervals of length 2h and on each of them we construct a kernel-type function which equals to zero at the boundary of the subinterval together with all derivatives.

It provides that the Fourier partial sums with respect to the trigonometric basis in L 2 [-1, 1] give a natural parametric approximation to the function on each subinterval.

Let (e j ) j≥1 be the trigonometric basis in L 2 [-1, 1], i.e.

e 1 = 1/ √ 2 , e j (x) = T r j (π[j/2]x) , j ≥ 2 , (6.8)
where the functions (T r j ) j≥2 are defined in (2.1). Now, for any array z = {(z m,j ) 1≤m≤M n , 1≤j≤N n } we define the following function

S z,n (x) = M n m=1 N n j=1 z m,j D m,j (x) , (6.9)
where

D m,j (x) = e j (v m (x)) χ η (v m (x)), v m (x) = (x -x m )/h n , x m = 2mh n and M n = [1/(2h n )] -1 .
We assume that the sequences (N n ) n≥1 and (h n ) n≥1 , satisfy the following conditions.

A 1 ) The sequence N n → ∞ as n → ∞ and for any ν > 0

lim n→∞ N ν n /n = 0 .
Moreover, there exist 0 < δ 1 < 1 and δ 2 > 0 such that

h n = O(n -δ 1 ) and h -1 n = O(n δ 2 ) as n → ∞ .
To define a prior distribution on the family of arrays, we choose the following random array ϑ = {(ϑ m,j ) 1≤m≤M n , 1≤j≤N n } with

ϑ m,j = t m,j ζ m,j , (6.10)
where (ζ m,j ) are i.i.d. N (0, 1) random variables and (t m,j ) are some nonrandom positive coefficients. We make use of gaussian variables since they possess the minimal Fisher information and therefore maximize the lower bound (6.5). We set

t * n = max 1≤m≤M n N n j=1 t m,j . (6.11)
We assume that the coefficients (t m,j ) 1≤m≤M n , 1≤j≤N n satisfy the following conditions.

A 2 ) There exists a sequence of positive numbers (d n ) n≥1 such that

lim n→∞ d n h 2k-1 n M n m=1 N n j=1 t 2 m,j j 2(k-1) = 0 , lim n→∞ d n t * n = 0 , (6.12)
moreover, for any ν > 0, Proposition 6.4. Let the conditions A 1 )-A 4 ). Then, for any ν > 0,

lim n→∞ n ν exp{-d n /2} = 0 . A 3 ) For some 0 < ǫ < 1 lim sup n→∞ 1 h 2k-1 n M n m=1 N n j=1 t 2 m,j j 2k ≤ (1 -ǫ)r 2 π 2k . A 4 ) There exists ǫ 0 > 0 such that lim n→∞ 1 h 4k-2+ǫ 0 n M n m=1 N n j=1 t 4 m,j j 4k = 0 .
lim n→∞ n ν E S ϑ,n 2 1 {S ϑ,n / ∈W k r } + 1 Ξ c n = 0 .
Proposition 6.5. Let the conditions A 1 )-A 4 ). Then for any function g satisfying the conditions (3.7) and H 4 )

lim n→∞ sup 0≤x≤1 E g -2 (x, S ϑ,n ) -g -2 0 (x) = 0 .
Proofs of Propositions 6.2-6.5 are given in Appendix.

Bayes risk

Now we will obtain the lower bound for the bayesian risk that yields the lower bound (4.11) for the minimax risk. We make use of the sequence of random functions (S ϑ,n ) n≥1 defined in (6.9)-(6.10) with the coefficients (t m,j ) satisfying the conditions A 1 )-A 4 ) which will be chosen later.

For any estimator S n we introduce now the corresponding Bayes risk

E n ( S n ) = R l E S z,n ,q S n -S z,n 2 µ ϑ (dz) , (6.13)
where the kernel family (S z,n ) is defined in (6.9), µ ϑ denotes the distribution of the random array ϑ defined by (6.10) in R l with l = M n N n . We remember that q is a centered gaussian distribution in R n with unit covariation matrix.

First of all, we replace the functions S n and S by their Fourier series with respect to the basis

e m,i (x) = (1/ √ h) e i (v m (x)) 1 (|vm(x)|≤1) .
By making use of this basis we can estimate the norm S n -S z,n 2 from below as

S n -S z,n 2 ≥ M n m=1 N n j=1 ( τ m,j -τ m,j (z)) 2 ,
where Moreover, from the definition (6.9) one gets

τ m,j = 1 0 S n (x) e m,j (
τ m,j (z) = √ h N n i=1 z m,i 1 -1 e i (u)e j (u)χ η (u) du .
It is easy to see that the functions τ m,j (•) satisfy the condition (6.2) for gaussian prior densities. In this case (see the definition in (6.5)) we have

τ m,j = (∂/∂z m,j )τ m,j (z) = √ he j (χ η ) ,
where

e j (f ) = 1 -1 e 2 j (v) f (v) dv . (6.14)
Now to obtain a lower bound for the Bayes risk E n ( S n ) we make use of Theorem 6.1 which implies that

E n ( S n ) ≥ M n m=1 N n j=1 he 2 j (χ η ) F m,j + B m,j + t -2 m,j , (6.15)
where

F m,j = n i=1 D 2 m,j (x i ) E g -2 (x i , S ϑ,n ) and
B m,j = 1 2 n i=1 E L 2 m,j (x i , S ϑ,n ) g 4 (x i , S ϑ,n )
with L m,j (x, S) = L x,S D m,j . In the Appendix we show that This means that, for any ν > 0 and for sufficiently large n,

lim n→∞ sup 1≤m≤M n sup 1≤j≤N n 1 nh F m,j -e j (χ 2 η ) g -2 0 ( x m ) = 0 (6.
sup 1≤m≤M n sup 1≤j≤N n F m,j + B m,j + t -2 m,j nhe j (χ 2 η )g -2 0 ( x m ) + t -2 m,j ≤ 1 + ν ,
where x m is defined in (6.9). Therefore, if we denote in (6.15)

κ 2 m,j = nh g -2 0 ( x m ) t 2 m,j and ψ j (η, y) = e 2 j (χ η )y e 2 j (χ 2 η )y + 1
we obtain, for sufficiently large n,

n 2k 2k+1 E n ( S n ) ≥ n -1 2k+1 1 + ν M n m=1 g 2 0 ( x m ) N n j=1 ψ j (η, κ 2 m,j ) .
In Therefore we can write that, for sufficiently large n,

n 2k 2k+1 E n ( S n ) ≥ 1 -ν 1 + ν n -1 2k+1 M n m=1 g 2 0 ( x m ) Ψ N n (κ 2 m,1 , . . . , κ 2 m,N n ) . ( 6
y j j 2k ≤ R .
The parameter R > 0 will be chosen later to satisfy the condition A 3 ). By the Lagrange multipliers method it is easy to find that the solution of this problem is given by y

* j (R) = a * (R) j -k -1 (6.20) with a * (R) = 1 N i=1 i k R + N i=1 i 2k and 1 ≤ j ≤ N .
To obtain a positive solution in (6.20) we need to impose the following condition

R > N k N i=1 i k - N i=1 i 2k . (6.21)
Moreover, from the condition A 3 ) we obtain that

R ≤ 2 2k+1 (1 -ε)r n h 2k+1 n π 2k g 0 := R * n , (6.22)
where

g 0 = 2h n M n m=1 g 2 0 ( x m ) .
Note that by the condition H 4 ) the function g 0 (•) = g(•, S 0 ) is continuous on the interval [0, 1], therefore

lim n→∞ g 0 = 1 0 g 2 (x, S 0 )dx = ς(S 0 ) with S 0 ≡ 0 . (6.23)
Now we have to choose the sequence (h n ). Note that if we put in (6.10)

t m,j = g 0 ( x m ) y * j (R)/ nh n i.e. κ m,j = y * j (R) , (6.24)
we can rewrite the inequality (6.19) as

n 2k 2k+1 E n ( S n ) ≥ (1 -ν) (1 + ν) g 0 Ψ * N n (R) 2h n n -1 2k+1 , (6.25)
where

Ψ * N (R) = N - N j=1 j k 2 R + N j=1 j 2k
.

It is clear that

k 2 /(k + 1) 2 ≤ lim inf N →∞ inf R>0 Ψ * N (R)/N ≤ lim sup N →∞ sup R>0 Ψ * N (R)/N ≤ 1 .
Therefore to obtain a positive finite asymptotic lower bound in (6.25) we have to take the parameter h n as

h n = h * n -1/(2k+1) N n (6.26)
with some positive coefficient h * . Moreover, the conditions (6.21)-( 6.22) imply that, for sufficiently large n,

(1 -ε)r 2 2k+1 π 2k 1 g 0 h 2k+1 * > 1 N k+1 n N n j=1 j k - 1 N 2k+1 n N n j=1 j 2k .
Moreover, taking into account that for sufficiently large n

g 0 1 N n N n j=1 (j/N n ) k -(j/N n ) 2k < (1 + ε) ς(S 0 )k (k + 1)(2k + 1) ,
we obtain the following condition on h *

h * ≥ (υ * ε ) 1/(2k+1) , (6.27)
where

υ * ε = (1 + ε)k c * ε (k + 1)(2k + 1)
and

c * ε = 2 2k+1 (1 -ε)r π 2k ς(S 0 ) .
To maximize the function Ψ *

N n (R) on the right-hand side of the inequality (6.25) we take R = R * n defined in (6.22). Therefore we obtain that lim inf

n→∞ inf S n n 2k/(2k+1) E n ( S n ) ≥ ς(S 0 ) F (h * )/2 , (6.28)
where

F (x) = 1 x - 2k + 1 (k + 1) 2 (c * ε (2k + 1)x 2k+2 + x)
.

Furthermore, taking into account that

F ′ (x) = - (c * ε (2k + 1)(k + 1)x 2k+1 -k) 2 (k + 1) 2 (c * ε (2k + 1)x 2k+2 + x) 2 ≤ 0 we get max h * ≥(υ * ε ) 1/(2k+1) F (h * ) = F ((υ * ε ) 1/(2k+1) ) = (1 + ε ′ )k k + 1 (υ * ε ) -1/(2k+1) ,
where

ε ′ = ε 2k + εk + 1 . (6.29)
This means that to obtain in (6.28) the maximal lower bound one has to take in (6.26)

h * = (υ * ε ) 1/(2k+1) . (6.30)
It is important to note that if one defines the prior distribution µ ϑ in the bayesian risk (6.13) by formulas (6.10), (6.24), (6.26) and (6.30), then the bayesian risk would depend on a parameter 0 < ε < 1, i.e. E n = E ε,n .

Therefore, the inequality (6.28) implies that, for any 0 < ε < 1,

lim inf n→∞ inf S n n 2k/(2k+1) E ε,n ( S n ) ≥ (1 + ε ′ )(1 -ε) 1/(2k+1) (1 + ε) 1/(2k+1) γ k (S 0 ) , (6.31)
where the function γ k (S 0 ) is defined in (4.7) for S 0 ≡ 0. Now to end the definition of the sequence of the random functions (S ϑ,n ) defined by (6.9) and (6.10) one has to define the sequence (N n ). Let us remember that we make use of the sequence (S ϑ,n ) with the coefficients (t m,j ) constructed in (6.24) for R = R * n given in (6.22) and for the sequence h n given by (6.26) and (6.30) for some fixed arbitrary 0 < ε < 1.

We will choose the sequence (N n ) to satisfy the conditions A 1 )-A 4 ). One can take, for example, N n = [ln 4 n] + 1. Then the condition A 1 ) is trivial. Moreover, taking into account that in this case

R * n = 2 2k+1 (1 -ε)r π 2k g 0 υ * ε N 2k+1 n = ς(S 0 ) g 0 k (k + 1)(2k + 1) N 2k+1
n we find thanks to the convergence (6.23)

lim n→∞ R * n + N n j=1 j 2k N k n N n j=1 j k = 1 .
Therefore, the solution (6.20), for sufficiently large n, satisfies the following inequality max

1≤j≤N n y * j (R * n ) j k ≤ 2N k n .
Now it is easy to see that the condition A 2 ) holds with d n = N n and the condition A 4 ) holds for arbitrary 0 < ǫ 0 < 1. As to the condition A 3 ), note that in view of the definition of t m,j in (6.24) we get

1 h 2k-1 n M n m=1 N n j=1 t 2 m,j j 2k = 1 2nh 2k+1 n g 0 N n j=1 y * j (R * n ) j 2k = R * n g 0 N 2k+1 n 2υ * ε = (1 -ǫ)r 2 π 2k .
Hence the condition A 3 ).

Estimation of non periodic function

Now we consider the estimation problem of the non periodic regression function S in the model (1.1). In this case we will estimate the function S on any interior interval [a, b] of [0, 1], i.e. for 0 < a < b < 1.

It should be pointed out that at the boundary points x = 0 and x = 1, one must to make use of kernel estimators (see [START_REF] Brua | Asymptotic efficient estimators for non-parametric heteroscedastic model[END_REF].

Let now χ be a infinitely differentiable [0, 1] → R + function such that χ(x) = 1 for a ≤ x ≤ b and χ (k) (0) = χ (k) (1) = 0 for all k ≥ 0, for example,

χ(x) = 1 η ∞ -∞ V x -z η 1 [a ′ ,b ′ ] (z) dz ,
where V is some kernel function introduced in (6.6),

a ′ = a 2 , b ′ = b 2 + 1 2 and η = 1 4 min(a , 1 -b) .
Multiplying the equation (1.1) by the function χ(•) and simulating the i.i.d. N (0, 1) sequence (ζ j ) 1≤j≤n one comes to the estimation problem of the periodic regression function S(x) = S(x)χ(x), i.e.

y j = S(x j ) + σ j (S) ξ j ,
where σ j (S) = σ 2 j (S) + ǫ 2 ,

ξ j = σ j (S) σ j (S) ξ j + ǫ σ j (S) ζ j .
and ǫ > 0 is some sufficiently small parameter.

It is easy to see that if the sequence (σ j (S)) 1≤j≤n satisfies the conditions H 1 ) -H 4 ), then the sequence ( σ j (S)) 1≤j≤n satisfies these conditions as well with σ j (S) = g(x j , S) = g 2 (x j , S)χ 2 (x j ) + ǫ 2 .

Conclusions

In conclusion, it should be noted that this paper completes the investigation of the estimation problem of the nonparametric regression function for the heteroscedastic regression model (1.1) in the case of quadratic risk. It is proved that the adaptive procedure (2.12) satisfies the non asymptotic oracle inequality and it is asymptotically efficient for estimating a periodic regression function. Moreover, in Section 7 we have explained how to apply the procedure to the case of non periodic function. As far as we know, the procedure (2.12) is unique for estimating the regression function at the model (1.1). Let us remember once more the main steps of this investigation. The procedure (2.12) combines the both principal aspects of nonparametric estimation: non asymptotic and asymptotic. Non-asymptotic aspect is based on the selection model procedure with penalization (see e.g., [START_REF] Barron | Risk bounds for model selection via penalization[END_REF]Massart, 1999, or Fourdrinier, Pergamenshchikov, 2007). Our selection model procedure differs from the commonly used one by a small coefficient in the penalty term going to zero that provides the sharp non-asymptotic oracle inequality. Moreover, the commonly used selection model procedure is based on the least-squares estimators whereas our procedure uses weighted leastsquares estimators with the weights minimizing the asymptotic quadratic risk that provides the asymptotic efficiency, as the final result. From practical point of view, the procedure (2.12) gives an acceptable accuracy even for small samples as it is shown via simulations by Galtchouk, Pergamenshchikov, 2008.

Proofs

Proof of Theorem 5.1

To prove the theorem we will adapt to the heteroscedastic case the corresponding proof from Nussbaum, 1985. First, from (2.4) we obtain that, for any p ∈ P n ,

E S,p S -S 2 n = n j=1 (1 -λ j ) 2 θ 2 j,n + 1 n n j=1 λ 2 j ς j,n (S) , (9.1)
where

ς j,n (S) = 1 n n l=1 σ 2 l (S)φ 2 j (x l ) .
Setting now ω = ω( α) with the function ω defined in (2.8), the index α defined in (5.2),

j 0 = [ ωε n ], j 1 = [ ω/ε n ] and ς n (S) = 1 n n l=1 σ 2 l (S) ,
we rewrite (9.1) as follows

E S,p S -S 2 n = j 1 -1 j= j 0 +1 (1 -λ j ) 2 θ 2 j,n (9.2)
+ ς n (S) n n j=1 λ 2 j + ∆ 1,n + ∆ 2,n with ∆ 1,n = n j= j 1 θ 2 j,n
and ∆ 2,n = 1 n n j=1 λ 2 j ς j,n (S)ς n (S) .

Note that we have decomposed the first term on the right-hand of (9.1) into the sum

j 1 -1 j= j 0 +1 (1 -λ j ) 2 θ 2 j,n + ∆ 1,n .
This decomposition allows us to show that ∆ 1,n is negligible and further to approximate the first term by a similar term in which the coefficients θ j,n will be replaced by the Fourier coefficients θ j of the function S.

Taking into account the definition of ω in (2.8) we can bound ω as

ω ≥ (A k ) 1 2k+1 (nε n ) 1 2k+1 .
Therefore, by Lemma A.1 we obtain

lim n→∞ sup S∈W k r n 2k 2k+1 ∆ 1,n = 0 .
Let us consider now the next term ∆ 2,n . We have Setting

| ∆ 2,n | = 1 n 2 n i=1 σ 2 i (S) n j=1 λ 2 j (φ 2 j (x i ) -1) ≤ σ * n sup 0≤x≤1 n j=1 λ 2 j (φ 2 j (x) -
γ k,n (S) = n 2k 2k+1 j 1 -1 j= j 0 (1 -λ j ) 2 θ 2 j + ς n (S)n -1 2k+1 n j=1 λ 2 j
and applying the well-known inequality

(a + b) 2 ≤ (1 + δ)a 2 + (1 + 1/δ)b 2
to the first term on the right-hand side of the inequality (9.2) we obtain that, for any δ > 0 and for any p ∈ P n ,

E S,p S -S 2 n ≤ (1 + δ) γ k,n (S) n -2k/(2k+1) + ∆ 1,n + ∆ 2,n + (1 + 1/δ) ∆ 3,n , (9.3)
where

∆ 3,n = j 1 -1 j= j 0 +1 (θ j,n -θ j ) 2 .
Taking into account that k ≥ 1 and that

j 1 ≤ ω ε -1 n + (A k ) 1 2k+1 n 1 2k+1 (ε n ) -(2k+2)/(2k+1) ,
we can show through Lemma A. Moreover, by the definition of ( λ j ) 1≤j≤n in (5.3), for sufficiently large n, for which t n ≥ r(S) we find sup

j≥1 n 2k 2k+1 (1 -λ j ) 2 (πj) 2k = π -2k (A k t n ) -2k/(2k+1) ≤ π -2k (A k r(S)) -2k/(2k+1) .
Therefore, by the definition of the coefficients (a j ) j≥1 in (3.

9.2

Proof of Theorem 4.3

In this section we prove Theorem 4.3. Lemma 5.2 implies that to prove the lower bounds (4.10) and (4.11), it suffices to show lim inf

n→∞ inf S n n 2k 2k+1 R 0 ( S n ) ≥ 1 , (9.5)
where

R 0 ( S n ) = sup S∈W k r E S,q S n -S 2 /γ k (S) .
For any estimator S n , we denote by S 0 n its projection onto W k r , i.e. S 0 n = Pr W k r ( S n ). Since W k r is a convex set, we get S n -S 2 ≥ S 0 n -S 2 . Now we introduce the following set

Ξ n = { max 1≤m≤M n max 1≤j≤N ζ 2 m,j ≤ d n } , (9.6)
where (ζ m,j ) are i.i.d. N (0, 1) random variables from (6.10) and the sequence (d n ) n≥1 is given in the condition A 2 ). Therefore, we can write that

R 0 ( S n ) ≥ {z:S z,n ∈W k r }∩Ξ n E S z,n ,q S 0 n -S z,n 2 γ k (S z,n ) µ ϑ (dz) .
Here the kernel function family (S z,n ) is given in (6.9) in which N n = [ln 4 n]+1 and the parameter h is defined in (6.26) and (6.30); the measure µ ϑ is defined in (6.13). Moreover, note that on the set Ξ the random function S ϑ,n is uniformly bounded, i.e.

S ϑ,n ∞ = sup 0≤x≤1 |S ϑ,n (x)| ≤ d n t * n , (9.7)
where the coefficient t * n is defined in (6.11). Thus, we estimate the risk R 0 ( S n ) from below as

R 0 ( S n ) ≥ 1 γ * n {z:S z,n ∈W k r }∩Ξ n E S z,n ,q S 0 n -S z,n 2 µ ϑ (dz) with γ * n = sup S ∞ ≤ √ d n t * n γ k (S) . (9.8)
By making use of the Bayes risk (6.13) with the prior distribution given by formulae (6.10), (6.24), (6.26) and (6.30) for any fixed parameter 0 < ε < 1 we rewrite the lower bound for R 0 ( S n ) as

R 0 ( S n ) ≥ E ε,n ( S 0 n )/γ * n -2 Ω n /γ * n (9.9)
with

Ω n = E(1 {S ϑ,n / ∈W k r } + 1 Ξ c n )(r + S ϑ,n 2
) .

In Section 6.3 we proved that the parameters in chosen prior distribution satisfy the conditions A 1 )-A 4 ). Therefore Propositions 6.3-6.4 and the limit (A.4) imply that, for any ν > 0,

lim n→∞ n ν Ω n = 0 .
Moreover, by the condition H 4 ) the sequence γ * n goes to γ k (S 0 ) as n → ∞. Therefore, from this, (6.31) and (9.9) we get, for any 0

< ε < 1, lim inf n→∞ inf S n n 2k 2k+1 R 0 ( S n ) ≥ (1 + ε ′ )(1 -ε) 1 2k+1 (1 + ε) 1 2k+1
. where ε ′ is defined in (6.29). Limiting here ε → 0 implies inequality (9.5). Hence Theorem 4.3.

Appendix

A.1 Proof of Lemma 5.2

First notice that, for any S ∈ W k r , one has

S n -S 2 n = T n ( S) -S 2 + ∆ * 1,n + ∆ * 2,n , where ∆ * 1,n = 2 n j=1 x j x j-1 ( S n (x j ) -S(x))(S(x) -S(x j ))dx and ∆ * 2,n = n j=1 x j x j-1 (S(x) -S(x j )) 2 dx .
For any 0 < δ < 1, by making use of the elementary inequality

2ab ≤ δa 2 + δ -1 b 2 , one gets ∆ * 1,n ≤ δ T n ( S) -S 2 + δ -1 ∆ * 2,n .
Moreover, for any S ∈ W k r with k ≥ 1, by the Bounyakovskii-Cauchy-Schwarz inequality we obtain that

∆ * 2,n ≤ 1 n 2 n j=1 x j x j-1 Ṡ2 (t) dt = 1 n 2 Ṡ 2 ≤ r n 2 .
Hence Lemma 5.2.

A.2 Proof of Theorem 6.1

For any z = (z 1 , . . . , z l ) ′ ∈ R n we set

̺ i (v, z) = 1 f (v, z)Φ(z) ∂ ∂z i (f (v, z)Φ(z)) .
Note that due to the condition (3.7), the density (6.3) is bounded, i.e.

f (v, z) ≤ (2πg * ) -n/2 .
So through (6.2) we obtain that lim

|z i |→∞ τ (z) f (v, z)ϕ i (z i ) = 0 .
Therefore, integrating by parts yields

E( τ n -τ (ϑ))̺ i = R n+l ( τ n (v) -τ (z)) ∂ ∂z i (f (v, z)Φ(z)) dz dv = R l ∂ ∂ z i τ (z) Φ(z) R n f (v, z)dv dz = τ i .
Now the Bounyakovskii-Cauchy-Schwarz inequality gives the following lower bound E( τ nτ (ϑ)) 2 ≥ τ 2 i / E̺ 2 i . To estimate the denominator in the last ratio, note that

̺ i (v, z) = f i (v, z) + φi (z i ) ϕ i (z i ) with f i (v, z) = (∂/∂ z i ) ln f (v, z) .
¿From (6.1) it follows that

f i (v, z) = n j=1 (ξ 2 j -1) 1 2σ 2 j (z) ∂ ∂ z i σ 2 j (z) + n j=1 ξ j S ′ i (x j ) σ j (z) .
Moreover, the conditions H 2 ) and (6.4) imply

(∂/∂ z i ) σ 2 j (z) = (∂/∂ z i ) g 2 (x j , S z ) = L i (x j , z)
from which it follows

E f i (Y, ϑ) 2 = F i + B i .
This implies inequality (6.5). Hence Theorem 6.1.

A.3 Proof of Proposition 6.2

First note that, for 0 ≤ x ≤ 1, we can represent the lth derivative as

S (l) z,n (x) = 1 h l M n m=1 l i=0 l i χ (l-i) η (v m (x)) Q i,m (z, v m (x)) , (A.1)
where

Q i,m (z, v) = N n j=1 z m,j e (i) j (v) . Therefore S (l) z,n 2 = 1 h 2l-1 n M n m=1 1 -1 l i=0 l i χ (l-i) η (v) Q i,m (z, v) 2 dv
and by the Bounyakovskii-Cauchy-Schwarz inequality we obtain that

S (l) z,n 2 ≤ C * (l, η) h 2l-1 n l i=0 Q i (z) (A.2) with C * (l, η) = max -1≤v≤1 l i=0 l i χ (l-i) η (v) 2 and Q i (z) = M n m=1 1 -1 Q 2 i,m (z, v) dv .
Now we show that, for any 0 ≤ i ≤ k -1 and δ > 0,

lim n→∞ n ν P Q i (ϑ) > δh 2k-1 n = 0 . (A.3)
To this end note that

1 -1 Q 2 i,m (ϑ, v) dv = N n j=1 ϑ 2 m,j 1 -1 (e (i) j (v)) 2 dv ≤ π 2 2i N n j=1 t 2 m,j j 2i ζ 2 m,j .
Therefore, taking into account the definition of the set Ξ n in (9.6), the functions Q i (ϑ) with 0 ≤ i ≤ k -1 can be estimated on this set as

Q i (ϑ) ≤ π 2 2(k-1) d n M n m=1 N n j=1 t 2 m,j j 2(k-1)
and by (6.12) we get, for any δ > 0 and sufficiently large n,

P Q i (ϑ) > δh 2k-1 n ≤ P Ξ c n .
Moreover, for sufficiently large n,

P Ξ c n ≤ M n N n e -d n /2 .
Therefore, the condition A 1 ) implies lim sup n→∞ n ν P Ξ c n = 0 , (A.4)

for any ν > 0. Hence Proposition 6.2.

A.4 Proof of Proposition 6.3

First of all we prove that for ǫ from the condition A 3 )

lim n→∞ n ν P S (k) ϑ,n > (1 -ǫ/4)r = 0 . (A.5)
Indeed, putting in (A.1) l = k we can represent the kth derivative of S z,n as follows

S (k) z,n (x) = S ′ z,n (x) + S ′′ z,n (x) (A.6) with S ′ z,n (x) = 1 h k M n m=1 k-1 i=0 k i χ (k-i) η (v m (x)) Q i,m (z, v m (x))
and

S ′′ z,n (x) = 1 h k M n m=1 χ η (v m (x)) Q k,m (z, v m (x)) .
First, note that, we can estimate the norm of S ′ z,n (x) by the same way as in the inequality (A.2), i.e.

S ′ z,n 2 ≤ C * (k, η) h 2k-1 n k-1 i=0 Q i (z) .
By making use of (A.3) we obtain that, for any p > 0 and for any δ > 0,

lim n→∞ n ν P S ′ z,n > δ = 0 . (A.7)
Let us consider now the last term in (A.6). Taking into account that 0 ≤ χ η (v) ≤ 1 we get Therefore, by the condition A 4 ), for sufficiently large n,

S ′′ z,n 2 = 1 h 2k-1 n M n m=1 1 -1 χ 2 η (v)Q 2 k,m (z, v)dv ≤ π 2
E (Z n ) 2ι ≤ B * (ι) N ι n h ιǫ 0 n E ζ ι * ≤ B * (ι) E (ζ 2 -1) 2ι M n N ι+1 n h ιǫ 0 n ,
where ζ ∼ N (0, 1). Now the condition A 1 ) implies, for sufficiently large n,

E (Z n ) 2ι ≤ n -δ 1 (ιǫ 0 -2) .
Thus, choosing in (A.9) ι > ν/(ǫ 0 δ 1 ) + 2/ǫ 0 we obtain the limiting equality (A.8) which together with (A.6)-(A.7) implies (A.5). Now it is easy to deduce that Proposition 6.2 yields Proposition 6.3.

A.5 Proof of Proposition 6.4

First of all, we recall that, due to the condition A 2 ), A.6 Proof of Proposition 6.5

Taking into account the inequality (9.7) and the condition H 1 ) we obtain

E g -2 (x, S ϑ,n ) -g -2 0 (x) ≤ max S ∞ ≤ √ d n t * n |g -2 (x, S) -g -2 0 (x)| + (2/g * ) P Ξ c n .
Conditions A 2 ) and H 4 ) together with the limit relation (A.4) imply Proposition 6.5.

A.8 Proofs of (6.16) and (6.17) First of all, note that Proposition 6. We recall that the sequence t * n is defined in (6.11). Therefore, property (A.15) implies max

1≤m≤M n max 1≤j≤N n 1 nh n i=1 E(S ϑ,n (x i )D m,j (x i )) 2 = O((t * n ) 2 ) .
As to the second term on the right-hand side of (A. Similarly, D m,j 2 ≤ h and, by (A.10)

E S ϑ,n 2 ≤ h M n m=1 N n j=1 t 2 m,j ≤ (t * n ) 2 .
Therefore, 1 nh max

1≤m≤M n max 1≤j≤N n |B m,j | = O((t * n ) 2 + h n )
and the condition A 1 ) implies (6.17).

Theorem 4 . 3 .T

 43 Assume that in the model (1.1) the sequence (σ j (S)) satisfies the conditions H 2 )-H 4 ). Then the risks (4.2) and (4.3) admit the following asymptotic lower bounds lim inf n ( S n , S) γ k (S) ≥ 1 . (4.11) Remark 4.2. Note that in Galtchouk, Pergamenshchikov, 2005, an asymptotically efficient estimator has been constructed and results similar to Theorems 4.2 and 4.3 were claimed for the model (1.1).

Proposition 6 . 2 .Proposition 6 . 3 .

 6263 Let the conditions A 1 )-A 2 ). Then, for any ν > 0 and for any δ > 0Let the conditions A 1 )-A 4 ). Then, for any ν > 0, lim n→∞ n ν P(S ϑ,n / ∈ W k r ) = 0 .

1 0S

 1 x)dx and τ m,j (z) = z,n (x) e m,j (x) dx .

(y 1

 1 ,...,y N )∈R N + N j=1 ψ j (η, y j ) Ψ N (y 1 , . . . , y N )

  1) . Now by Lemma A.2 and the definition (2.7) we obtain directly the same property for ∆ 2,n , i.e. 2,n | = 0 .

R

  n ( S, S)/γ k (S) ≤ lim sup n→∞ sup S∈W k r γ k,n (S)/γ k (S) and to prove (5.4) it suffices to show that lim sup n→∞ sup S∈W k r γ k,n (S)/γ k (S) ≤ 1 . (9.4) First, it should be noted that the definition (5.1) and the inequalities (3.7)-(3.8) imply directly lim n→∞ sup S∈W k r t n /r(S) -1 = 0 .

π( 1 -

 1 2k (A k r(S)) 2k/(2k+1) (1λ j ) 2 /a j ≤ 1 .Furthermore, in view of the definition (2.7) we calculate directly lim z k ) 2 dz = 0 . Now, the definition of W k r in (3.3) and the condition (3.6) imply the inequality (9.4). Hence Theorem 5.1.

1 .

 1 j 2k ζ 2 m,j .Therefore from the condition A 3 ) we get for sufficiently large n j 2k ζ m,j and ζ m,j = ζ 2 m,j -We show that for any ν > 0 and for any δ > 0lim n→∞ n ν P ( |Z n | > δ) = 0 . (A.8)Indeed, by the Chebychev inequality for any ι > 0P ( |Z n | > δ) ≤ E (Z n ) 2ι /δ 2ι . (A.9)Note now that according to the Burkholder-Davis-Gundy inequality for any ι > 1 there exists a constant B * (ι) > 0 such thatE (Z n ) 2ι ≤ B * (ι) E j 4k ζ * .

  obtain, for sufficiently large n,E S ϑ,n 2 1 {S ϑ,n / ∈W k r } + 1 Ξ c n ≤ max m,j E ζ 2 m,j 1 {S ϑ,n / ∈W k r } + 1 Ξ c n .Moreover, for any 1 ≤ m ≤ M n and 1 ≤ j ≤ N n , we estimate the last term asE ζ 2 m,j 1 {S ϑ,n / ∈W k r } + 1 Ξ c n ≤ n P(S ϑ,n / ∈ W k r ) + n P(Ξ c n ) + 2E ζ 2 1 {ζ 2 ≥n} ,where ζ ∼ N (0, 1). By applying now Proposition 6.3 and the limit (A.4) we come to Proposition 6.4.

f 1 - 1 ff 2 e 2 j

 1122 5, the condition (3.7) and the condition H 4 ) imply that lim n→∞ max 1≤m≤M n sup 0≤x≤11 {|vm(x)|≤1} E g -2 (x, S ϑ,n )g -2 0 ( x m ) = 0 . (A.14)Let us show now that for any continuously differentiable functionf on [-1(v m (x i ))1 {|vm(x i )|≤1} -(v)dv = 0 . (A.15) (v m (x i ))1 {|vm(x i )|≤1}i ) vm(x i-1 ) |f (v m (x i ))f (z)|dz + max |z|≤1 |f (z)|(2v * + v * ),where i * = [n x mnh] + 1, i * = [n x m + nh], v * = ([n x mnh] + 1n x m )/(nh) and v * = ([n x m + nh]n x m )/(nh) .Therefore, taking into account that the derivative of the function f is bounded on the interval [-1, 1] we obtain that∆ n,m ≤ 3 max |z|≤1 | ḟ(z)| + 2 max |z|≤1 |f (z)| nh n .Taking into account the conditions on the sequence (h n ) n≥1 given in A 1 ) we obtain limiting equality (A.15) which together with (A.14) implies(6.16).Now we study the behavior of B m,j . Due to the inequality (3.9) we estimate the Fréchet derivative as| L m,j (x, S ϑ,n )| ≤ C * |S ϑ,n (x)D m,j (x)| + |D m,j | 1 + S ϑ,n D m,j . (A.16)Consider now the fisrt term on the right-hand side of this inequality. We haveE (S ϑ,n (x)D m,j (x)) 2 = E N n l=1 ϑ m,l e l (v m (x)) (v m (x))I 4 η (v m (x)) ≤ N n l=1 t 2 m,l 1 {|v m (x)|≤1} ≤ (t * n ) 2 1 {|v m (x)|≤1} .

1 0 1 - 1

 111 [START_REF] Nussbaum | Spline smoothing in regression models and asymptotic efficiency in L 2[END_REF], we get|D m,j | 1 = |e j (v m (x)) χ η (v m (x))|dx = h |e j (v) χ η (v)|dv ≤ 2h .

  .[START_REF] Stone | Optimal global rates of convergence for nonparametric regression[END_REF] Obviously, to obtain a "good" lower bound for the risk E n ( S n ) one needs to maximize the right-hand side of the inequality(6.19). Hence we choose the coefficients (κ 2 m,j ) by maximization the function Ψ N , i.e.

max y 1 ,...,y N Ψ N (y 1 , . . . , y N ) subject to N j=1

AMS

Subject Classification : primary 62G08; secondary 62G05, 62G20 2 Key words: asymptotic bounds, adaptive estimation, efficient estimation, heteroscedastic regression, nonparametric regression, Pinsker's constant.

Acknowledgements

We are grateful for the comments and stimulating questions by two anonymous Referees which have led to considerable improvements.

A.7 Properties of the trigonometric basis

(A.11)

Proofs of Lemma A.1 and Lemma A.2 are given in Galtchouk, Pergamenshchikov, 2007.

Lemma A.3. Let θ j,n and θ j be the Fourier coefficients defined in (2.3) and (3.4), respectively. Then, for 1 ≤ j ≤ n and n ≥ 2, sup

Proof. Indeed, we have

By making use of the Bounyakovskii-Cauchy-Schwarz inequality we get

The definition of class W 1 r implies (A.13). Hence Lemma A.1.

A.9 Proof of (6.18) Indeed, by the direct calculation it easy to see that, for any N ≥ 1 and for any vector (y 1 , . . . , y N )

,

where the operator e j (f ) is defined in (6.14). Moreover, we remember that 1 -1 e 2 j (v)dv = 1. Therefore, taking into account the property (6.7) we obtain (6.18).


Nasaf Ahmed

Intrusion detection system: A survey and taxonomy

Keywords: computer technicians, cybersecurity, cyber-attack, intrusion detection systems

Over decades of positive research and a series of tests completed within the computing world there have been countless achievements made by computer technicians globally.

With all this success came a wave of unwanted infiltration methods that are used to cause malicious intent on everyday systems. Fortunately, cyber security has always been around to protect all forms of data/information. David Bernstein chairman of the British red Cross once stated, "For every lock there is someone out there trying to pick it or break in", referring to the potential of a cyber-attack on every possible system out there. In the following paper I will write a survey and taxonomy on intrusion detection systems, discussing their types and how they can be successful.

I. INTRODUCTION

Intrusion detection systems (IDSs) have been crucial in detecting malicious activity. IDSs have been studied thoroughly and improved overtime. IDSs are broad and this research paper provides a survey on different types of IDS systems and what influences their true potential. Many other surveys were used to help identify this. As well as a taxonomy on the evasion techniques/how to achieve the highest accuracy for an IDS and the classifications of intrusions.

II. TAXONOMY AND SURVEY OF INTRUSION DETECTIONS SYSTEMS

A. Intrusion detection systems An intrusion detection system (IDS) is a software application or device that checks a network for malicious rule breaks or malicious activity. A violation or malicious activity is usually reported or obtained using a security information system. For example, an activity that would cause the computer services to be unresponsive to users would be recognized as an intrusion. Most of the traditional firewalls are not able to detect malicious activity on a computer. Once the issue is detected it must then be prevented. IPS's (intrusion prevention systems) must eliminate the issue when it is found. However, in this paper I will only cover IDS's. "Failure to prevent the intrusions could degrade the credibility of security services, e.g., data confidentiality, integrity, and availability" [START_REF] Khraisat | Survey of intrusion detection systems: techniques, datasets and challenges[END_REF]. This quote taken from an article written by 'Khraisat, A., Gondal, I., Vamplew, P' has made it clear that without the detection of such attacks, a system will lead to failure.

The two famous types of IDS's can be classified into two categories, Host-based intrusion detection systems (HIDS)

and Network-based intrusion detection systems (NIDS). A Hybrid-based detection system is basically a combination of both the network and host-based systems.

B. Host-based

Host-based intrusion detection systems (HIDS) have a job to examine activities that take place on a specific machine. It can monitor its host in very minute detail. It will only monitor activities taking place on the host it is assigned to. One of the many abilities it has is to dictate the system and find out which users are disobeying guidelines and any users that are involved in malicious activity. A huge difference between Network-based intrusion systems (NIDS) and HIDS is that a host-based can be aware of an outcome caused by an attacker as it can observe the data files and the potentially infected area in the system. They are very similar to application-level intrusion detection systems. Many of the advantages they share are similar however, HIDS are not as capable as those. Majority of the antimalware products available are actively using Host-based IDS. A large amount of the Host-based IDSs rely on string or signature matching to locate threats.

Using simple processes such as this allows them to be infiltrated by altering the signature that will prevent the signature from matching [START_REF] Hammoudeh | Continuous Monitoring in Mission-Critical Applications Using the Internet of Things and Blockchain[END_REF].

An important factor is that the HIDS is incomplete when on its own. When a host is being monitored there are a few things that need to be considered such as any drawbacks that occur. A few of these drawbacks are:  The process of Host-based IDS will consume resources which will reduce the performance of the host.  Any attack taken place on the host will go un-noticed until it has reached the host.  The visibility is also restricted to a single host.

C. Network-based

Network-based intrusion detection systems (NIDS) are devices that are placed inside a chosen network that will assess traffic moving across the devices. Network-based IDS can come in the form of software or hardware-based systems. Most manufacturers will examine the situation and decide which network medium the systems will be joined to. These mediums include: FDDI, Ethernet etc. NIDS usually has two network interfaces that it uses: the first is to operate and report feedback: and the second can listen to network conversations in promiscuous mode (a mode that causes a controller to pass all traffic it obtains to the central processing unit).

Although there happens to be several Network-based IDS vendors, each one of the systems usually operate in two direct ways; Network-based IDS will either be anomaly-based or signature-based systems. Anomaly and signature-based systems detect malicious traffic trying to infiltrate a system. NIDS does in fact have a few drawbacks such as encryption, network data overload, difficulties tuning etc. [START_REF] Ghafir | DNS query failure and algorithmically generated domain-flux detection[END_REF].

Below is an image illustrating NIDS in more detail. The network in the image is using 3 NIDS [START_REF] Ghafir | A Survey on Botnet Command and Control Traffic Detection[END_REF]. They have been positioned in the network system to allow them to observe all the traffic taken place in the circuit. Here is standard security configuration where the NIDS is protecting the network from attacks. In Fig. 1 The firewall is being personally protected by a NIDS to stop any potential attacks that could occur on the firewall. A system can use more than one NIDS to create a more secure system.

D. Hybrid-based

However, Thomas M. Chen writer who wrote 'network and system security' [START_REF] Jyothsna | Anomaly-Based Intrusion Detection System, Computer and Network Security, Jaydip Sen, IntechOpen[END_REF], went into extensive research and conducted many tests and found them to be more effective when used together. In his test he used them separately and then he used them simultaneously. He saw that the detection rate was much better when they were used hand in hand. HIDS can identify malicious data that come for the network whereas NIDS will give a faster response time. He concluded that combining their strengths resulted in a more efficient system [START_REF] Jyothsna | Anomaly-Based Intrusion Detection System, Computer and Network Security, Jaydip Sen, IntechOpen[END_REF].

There are many pros and cons associated with them. Both can be used for different purposes. Most systems will usually use a network-based system as it serves more purpose in the industry [START_REF] Ghafir | Advanced Persistent Threat and Spear Phishing Emails[END_REF].

In fig. 2 NIDS is in contact with the firewalls and HIDS is in contact with the computer itself. Both are working side by side to protect the entire system/server. This shows how each of them operate in a different way [START_REF] Chen | Network and System Security[END_REF].

Both NIDS and HIDS have two types of operation: an anomaly-based and a signature-based. Majority of the IDSs will use both signature and anomaly-based systems. However, there are some systems that will not use both.

These types of systems will clearly be at a disadvantage compared to the systems that use both. I will now go into more detail regarding the anomaly and signature-based systems.

E. Anomaly-based

The behaviour of a network is examined in detail in an anomaly-based system (ABS). The main goal of an anomalybased system is to detect anomalies. There are many different reasons for these anomalies to appear. This includes illegally using a device, harmful malware, suspicious behaviour etc. Information is constantly being gathered to prevent any future attacks [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF]. If there is no suspicion and the rules are being followed, then the transaction will be approved. For a network to know what is acceptable it can gain this information from the network administrator. Sometimes the prerequisites can be set beforehand. If an attack manages to get past the signatured-based method, then the anomalybased system comes into effect.

The anomaly-based system is great at finding out if someone is interfering with a network. An example of this is when someone is attempting to log in too many times and is unsuccessful [START_REF] Ghafir | A Survey on Intrusion Detection and Prevention Systems[END_REF][START_REF] Ghafir | Disguised Executable Files in Spear-Phishing Emails: Detecting the Point of Entry in Advanced Persistent Threat[END_REF]. The system will assume that someone is trying to gain access to the system without permission and usually lock the account.

The IDS should be capable of understanding how protocols work and their objective. Protocol analysis can be costly but despite the cost it can be a liability if this is not covered. Having this will decrease the number of false-positive alerts. The accuracy of a system is vital for a long and efficient lifespan [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF].

In the article 'anomaly based-intrusion detection systems' [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] written by 'Jyothsna and Munivara' a taxonomy was made that stated a couple drawbacks in the anomaly-based system that can lead to many failures in different systems. One of the drawbacks stated was the defining of rule sets. It is crucial that the efficiency of a system is at its best ability. These rule sets will most likely be different depending on who requires the system and what system is being used. They later go on to explain that new rules implemented will have to be tested to ensure efficiency [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF].

F. Signature-based

A signature-based system is also known as a misuse detection system. It is a system that uses patterns it recognizes and then decides if the patterns are suspicious. It stores patterns and checks to see if they are familiar [START_REF] Baumrucker | Cisco Security Professional's Guide to Secure Intrusion Detection Systems[END_REF]. If there is an operation being made on the system with an unusual pattern, it will detect the attack due to previously encountering a similar pattern. All patterns that are encountered are stored in a database so that they can be compared to if need be. The significance of this is that it can quickly check and match a pattern it finds suspicious and see if it may be a threat. This saves a lot of time and makes the system much more efficient [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF]. Which is why it is important systems are updated on a regular basis so it can identify which signatures are new or old.

A huge drawback to signature-based systems stated in 'anomaly based-intrusion detection systems' [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] is that they cannot detect an attack whose pattern does not look familiar. If the pattern of the attack is not located in the database, then it cannot be detected. As soon as an attack is attempted, a signature is made for it. This signature is usually unique as most attacks are different majority of the time. Any attack that can alter its movements could very easily penetrate the signature-based system. It operates best against behaviors that are common/fixed [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF].

When reading 'anomaly based-intrusion detection systems' [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] I concluded that the anomaly and signature-based detection systems can both be very useful when it comes to the security of a system. They operate in different ways and have several purposes. As stated in the article "intrusion detection systems aim to identify intrusion with a low false alarm rate and a high detection rate" [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF]. For this to be achieved the system must be able to perform at the best of its ability which may in the long run be rewarding and costly.

III. ACHIEIVNG THE BEST IDS

In this part of the paper, I will go through the aspects that decide if an IDS will be successful and how it can achieve its best abilities as well as classifying different intrusions.

A. Evasion Techniques

Attackers have a wide arsenal of attacks they can perform on a system. IT departments are not only responsible for knowing these attacks but also preventing them from happening [START_REF] Ghafir | Advanced Persistent Threat Attack Detection: An Overview[END_REF]. For an attack to be pre-determined it can be easy for the department to know how to adjust the IDS system so it cannot be misguided into seeing a malicious attack. Here are a few strategies attacks will use- Avoiding defaults-For instance, a protocol being used at a port can easily be manipulated. The attacker could change ports which will prevent the IDS from detecting the attack.  Proxying/address spoofing: By doing this the attackers can take advantage of a weak proxy server. They can use these weak proxy servers and prevent the IDS from knowing where the attack is coming from.

 Pattern change evasion: as we already know IDSs can detect attacks using the matching of patterns. However, if the blueprint of the pattern is altered, the attack can go unnoticed.

B. Accuracy of IDS's

The accuracy of an IDS is the one of the most important aspects. In total there are four possible outcomes for the activities examined [START_REF] Ghafir | Tor-based malware and Tor connection detection[END_REF]. I will look at the taxonomy [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] that covers these and conclude why they are important. The four types are:

 (TPR) True Positive Rate-This is the ratio between the amount of correctly predicted attacks and the number of attacks. It is very difficult for every single intrusion to be detected however, when this happens the TPR will be 1. For something like this to happen is very scarce and only capable by a very good detection system. TPR can be shown mathematically in the following equation: TPS= TPTP+FNTPR= TPTP+FN  (FPR) False Positive Rate-Is calculated as the ratio between the number of normal instances determined improperly as an attack and the number of normal instances. The equation for FDR is: FPR= FPFP+TNFPR= FPFP+TN  (FNR) False Negative Rate-This is when a detector is not able to recognise an anomaly and it then disregards it and determines it is normal. The equation for FNR is: FNR= FNFN+TPFNR= FNFN+TP  (CR) Classification Rate-The Classification Rate will determine how accurate the IDS is at detecting unusual or normal behaviour in the traffic. This can be described by the percentage of all correctly predicted instances to all the instances. It is shown as: Classification rate (Accuracy)=TP+TNTP+TN+TP+FN [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] The survey [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] written on these matters clearly states that these four situations can dictate the security of a network. The authors then go into detail describing that a network breach is what determines how strong a system is. If a system is not sent through vigorous 'trial and test', it can have severe flaws that can derail a system leaving it open to endless attacks.

Table.1. A confusion matrix on classification

A confusion matrix is the best way to show classification results of an IDS. It depicts true and false results.

C. Challenges of Maintaining an IDS

To manage an IDS requires three demanding tasks. When an IDS Is being chosen, it is important the program prevents these problems from occurring. These can be classified as such:

Identifying false positives: I have already discussed false positives in a previous chapter. The reason false positives must be recognised is because they can create problems for the IT professionals working on the system. From [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] we learnt that false positives must be identified to save the system from any inconvenience. However, using the equation stated in [START_REF] Ghafir | Blacklist-based Malicious IP Traffic Detection[END_REF] we can see how reliable a system is. An IDS that is not updated will find it difficult dealing with false positives.

Staffing: another common problem is staffing. IDS management involves understanding what caused the threat.

In the computing world cyber-attacks are evolving and continue getting better. The attackers are always trying to find flaws in a system, which is why it is crucial for IT technicians to be ahead of the attackers. All organizations will have different goals and requirements. The difficult part with this is locating a professional who has the skills in the required field [START_REF] Damopoulos | Evaluation of anomaly-based IDS for mobile devices using machine learning classifiers[END_REF].

Identifying genuine risks: not identifying a genuine risk can be dreadful. To identify and prevent a risk, the origin of the attack must be known. Letting an attack slip by can be dreadful for any system. Knowing the origin of an attack will decrease the chance of it happening again. Systems used in popular companies usually undergo trial and error before being implemented.

D. IDS's Used in the Industry

In the table below are five of the most popular IDS's used all around the world. As you can see both HIDS and NIDS are used in these detection systems. NIDS however, are used in almost all of them as most companies will require the functions of a network-based system. Companies all around the world put their trust in these software's to protect their systems and keep the personal information of their customers safe.

E. classifications of intrusions-

From the paper so far, a taxonomy can be done to classify intrusions into three different categories, depending on how strong or weak the intrusion is [START_REF] Ghafir | DNS traffic analysis for malicious domains detection[END_REF][START_REF] Axelsson | Intrusion Detection Systems: A Survey and Taxonomy[END_REF]. Well known intrusions These intrusions are common and will usually contain a good pattern. They can be used very easily, and the system can predict them with ease as the pattern is usually within the systems database. Generalized intrusions These may attack a specific part of a system. This is because they patterns may vary depending on the attack. The patterns within these are either weak or great.

Obscure instructions These can be very hard to predict as the system may be confused on what to expect. They are not fully discovered and may alter their attack. A system will usually encounter one of these three types of intrusions. However, it comes down to how well built the system is for it to detect either of these intrusions. [START_REF] Wilhelm | Ninja Hacking[END_REF] CONCLUSION In conclusion of this informative report, surveys of other IDS's were evaluated to see what makes an IDS efficient. It is vital that FP (false positive) and FN (false negatives) are as low as possible. A protocol for an IDS is that malicious traffic will be different to normal traffic. With all the research papers I used within my paper I realized that they all have a similar pattern. They all try to achieve a certain degree of accuracy within an IDS using different techniques. The accuracy and maintenance of an IDS are crucial for it to fulfil its duties. Different systems require different needs. Having a system is one thing, making that system reach its true capabilities is another. Which is why guidelines and protocols are needed whether the vendor using the system is small or large.

Fig. 1 .

 1 Fig.1. Example of network containing NIDS[START_REF] Ghafir | DNS query failure and algorithmically generated domain-flux detection[END_REF]

Fig. 2 .

 2 Fig.2. Example of HIDS and NIDS in the same network[START_REF] Ghafir | DNS query failure and algorithmically generated domain-flux detection[END_REF]

Table 2 .

 2

	Features BRO	Snort	OSSEC	Security	Suricata
				union
	HIDS
	NIDS

A table showing the 5 best IDS software's.