Panagiotis Mantalos

Kyriacos Mattheou

A Karagrigoriou

An Improved Divergence Information Criterion for the Determination of the Order of an AR Process

Keywords: model selection, measure of divergence, AR process, information criterion AR process, Information Criterion, Model selection, Measure of divergence AR process, Information Criterion, Model selection, Measure of divergence

come   L'archive ouverte pluridisciplinaire

RESPONSE TO THE REVIEWER'S COMMENTS

We would like to express our appreciation to the reviewer because his/hers careful reading resulted in identifying a typo in our GAUSS program that was oversighted. When in the original report, the reviewer raised concerns about the simulations, we looked into the programming of HQ, assuming that the error was there. The reviewer's persistence made us look more closely and we have now identified a typo in the programming not of HQ but of SIC. The error, which has partially affected the SIC powers, has been corrected and the correct rates of success for SIC are reported in the present revision (Tables 123456). Note that all other values stay intact. Due to the new values for SIC, some minor changes were made into the text on pages 11-13 (all the changes are highlighted in yellow) which though do not effect in any way our original conclusions. On the contrary, a new special feature of MDIC has been revealed and decided to be included in the present revision:

First note that we included in this revision a brief discussion about MDIC with a equal to that value in [0, 0.50] which maximizes the power (see p. 11, lines 6-9). However, due to insignificant differences between MDIC and MDIC with max power, the powers of the latter for Models 3-6 are suppressed from the Tables (p. 11, lines -5 to -4). Our simulations revealed that especially for small and medium sample sizes, MDIC with 0.25 a = provides an ideal model identification technique for both small and large lag models. As a result, we have chosen to include in the discussion about the choice of a (p. 13, middle paragraph), a statement about the small and medium sample performance of MDIC (p.13,. A similar statement appears also on p. 14, lines 9-13. In these statements, we explicitly state that for small and medium sample sizes, the choice 0.25 a =

is highly recommended since it clearly serves as a fair balance between small and large lag models.

Finally note that throughout the manuscript, the index a has been properly referred to.

Introduction

Consider a discrete-time stationary AR(p) process of the form

1 1 2 2 X a X a X a X e t t t p t p t = + + + + - - - L (1)
where { t e } is a sequence of uncorrelated ( ) 2 0, N σ random variables and all the roots of the autoregressive polynomial are outside the unit circle, that is, ( ) 1 0 1

p A z a z a z p = - -- ≠ L for all z ∈ such that 1 z < .
One of the crucial problems in time series modeling is selecting the optimal model order and specifying a parsimonious model for the data generating process (DGP). Many techniques have been developed for the order selection of linear models, namely a) graphical methods with representative the Box and Jenkins methods (1970) and b) hypothesis test procedures with the pioneering work of Whittle (1952) as representative. Akaike (1974) viewed the problem of model fitting in the context of time series analysis as a multiple decision procedure rather than a hypothesis testing problem. However the same author (Akaike, 1973) initiated the research on automatic model selection techniques by developing the popular Akaike Information Criterion (AIC) which is considered as a significant contribution to statistical modelling where the focus is on the selection of a satisfactory model for a given set of observations. It might appear at first sight that the higher the order of the selected process (model) chosen, the better the fitted model will be. Such a thought may be true for fitting purposes but not for predicting ones. Indeed, the fit may be perfect for the given data but the use of the selected model for predictive purposes may result in gross errors. Numerous selection criteria have been developed over the last 30 years which attempt to prevent overfitting by assigning a penalty to the introduction of an unnecessary

The New Modified Divergence Information Criterion

Measures of Divergence and Selection Criteria

The measures of divergence are used as indices of similarity or dissimilarity between populations and for measuring mutual information concerning two variables and as such they can be used for the construction of model selection criteria. The distance between a candidate model and the true but unknown model could be measured by a measure of divergence. The candidate model for which the measure of divergence is minimized will be selected. The well known Kullback-Leibler measure of divergence (Kullback and Leibler, 1951) was the one used by Akaike (1973) to develop the Akaike Information Criterion (AIC). Akaike proposed the evaluation of the fit of the candidate model using minus twice the mean expected loglikelihood (also known as expected overall discrepancy). Furthermore, he provided an unbiased estimator of the expected loglikelihood so that the resulting criterion is given by where n the sample size, p the order of the candidate model and 2 ˆp σ the estimator of the variance of the p th -order candidate model.

For fitting autoregressive processes, Jones (1975) suggested that AIC has a tendency to overestimate the order of the process and Shibata (1976) showed that the probability of overestimation for a large sample size is nonzero. To correct this tendency, the SIC criterion was proposed independently by Akaike (1978) and Schwarz (1978): Hannan and Quinn, 1979) which has been found to be equivalent to SIC in the sense that they are both consistent (Hannan, 1980).

The problem of avoiding overfitting especially for purely autoregressive processes can also be dealt with the minimization of the final prediction error criterion (FPE) of Akaike (1969). The FPE is an estimate of the one-step ahead prediction mean squared error for a realization of the process independent of the one observed. If we fit AR processes of steadily increasing order p the maximum likelihood estimator (MLE) of the white noise variance will usually decrease with p. However, FPE will decrease to a minimum value and then increase as p will increase.

According to FPE, we choose the order of the candidate process to be the value of p for which FPE is minimized. The FPE is given by 2 ( )

p n p FPE p n p σ + = - .
In all the above cases, any parameter estimation is handled through the maximum likelihood method. A general class of criteria has been introduced by Konishi and Kitagawa (1996) which also estimates the Kullback-Leibler measure where the estimation is not necessarily based on maximum likelihood.

The Development of the MDIC Criterion

One of the most recently proposed measures of divergence is the Basu-Harris-Hjort-Jones power divergence between the candidate model (.) f θ and the true model g (Basu et. al, 1998) which is denoted by BHHJ, indexed by a positive parameter a , and defined as: This family of measures was proposed by Basu et al. (1998) for the development of a minimum divergence estimating method for robust parameter estimation. The index α controls the trade-off between robustness and asymptotic efficiency of the parameter estimators which are the values of θ that minimize the measure over a parametric spaceΘ . It should be also noted that the BHHJ family reduces to the Kullback-Leibler measure of divergence for 0 a ↓ (see Mattheou, 2007) and as it can be easily seen, to the square of the standard L 2 distance between the candidate and the true model for a =1. Mattheou et al. (2008) applying the same methodology used for AIC to the BHHJ divergence developed a new criterion, the Divergence Information Criterion (DIC) which for a set of observations 1 n x ,...,x is given by

1 1 1 1 ( , ) ( ) 1 ( ) ( ) ( ) a a a a I g f f z g z f z g z dz a a θ θ θ ⎧ ⎫ ⎛ ⎞ ⎛ ⎞ + + = -+ + ⎨ ⎬ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ ⎩ ⎭ ∫ . F o
/2 2 /2 ( ) (2 ) (1 ) a p DIC p nQ a p θ π - + = + + , where 1 ˆˆ1 1 1 ( ) 1 ( ) n a a i i Q f z dz f x a n θ θ θ + = ⎛ ⎞ = -+ ⎜ ⎟ ⎝ ⎠ ∑ ∫
and θ a consistent and asymptotically normal estimator of θ . Preliminary simulation studies for regression models (Mattheou, 2007) showed a very good medium sample size performance for DIC for values of α close to zero.

Although the DIC criterion was constructed so that it will be an asymptotically unbiased estimator of the BHHJ divergence measure between the candidate and the true model, the calculation of the first part of Qθ , namely the integral where ( )

1 ˆ1 1 1 ( ) n a i i MQ f x n θ θ α - = ⎡ ⎤ = -+ ⎢ ⎥ ⎣ ⎦ ∑ .
Note that a model selection criterion can be considered as an approximately unbiased estimator of the expected overall discrepancy, a nonnegative quantity which measures the distance between the true unknown model and a fitted approximating model. Observe also that, as it was mentioned in the introduction, a criterion consists of two terms, the first of which is a biased estimator of the expected overall discrepancy. As a result, if we choose the model with the smallest estimator of the expected overall discrepancy we may end up with a selection with an unnecessarily large order. The estimator becomes asymptotically unbiased by introducing the appropriate correction term. The expected overall discrepancy in the present setting is defined by

ˆ( ) ( / ) E W E W θ θ θ θ = = with 1 1 ( ) ( ) a W gzf z d z a θ θ ⎧ ⎫ ⎛ ⎞ = - + ⎨ ⎬ ⎜ ⎟ ⎝ ⎠ ⎩ ⎭ ∫
and θ a consistent and asymptotically normal estimator of θ. Observe that W θ is the targeted divergence ( , ) a I g f θ without the first and the last terms. Notice that the last term remains constant irrespectively of the candidate model f θ while the first term, as stated earlier, is approximately constant for the various candidate models. In that sense, the expected overall discrepancy ( ) E W θ represents the average distance between the true model g and the candidate model f θ up to a constant. The relevant theorem for MDIC is stated below. The proof is omitted since it follows the lines of the proof of Theorem 2.4 (Mattheou et.al, 2009) where the unbiasedness of the original DIC criterion was established.

2 /2 / 2 ( ) * (2 ) (1 ) ˆp a MDIC p n MQ a p π θ + - = + +
where θ is a consistent and asymptotically normal estimator of θ.

Note that although the discussion in Mattheou et. al (2009) is limited to iid random variables, by following the approach by Akaike (1973 and1974) and Billingsley (1961), the same line of discussion can be extended to cover autoregressive models. Recall that in time series models, even under the Gaussian assumption, the evaluation of the log-likelihood cannot be expressed as the sum of the logarithms of the density function of each observation. However, the log-likelihood can generally be expressed by conditional distributions, so that in relatively simple models, like the autoregressive models, we use as an approximation the mean conditional log-likelihood which multiplied by the number of observations is used in place of the log-likelihood in model selection.

In addition to the above, numerous other techniques have been introduced over the years some of which are associated with the order determination and the forecasting performance in time series. Such techniques include approaches based on bootstrapping (Shang and Cavanaugh, 2008), approaches for high dimensional medical data (Koukouvinos et al., 2008), supersaturated designs (Koukouvinos and Mylona, 2008), variations of AIC (Cavanaugh and Johnson, 1999;Bengtsson and Cavanaugh, 2006;Seghouane, 2006;Seghouane et. al, 2005) and time series models (Lee, 1998).

Optimal Choice for the Index a

For practical purposes one has to decide the optimal choice of the positive index a . For this purpose, we simulate a 100 observation series for 5 different models with a between 0.01 and 0.5 and we provide in Figure 1, the power of the selection, namely the proportion of times the correct model is selected as a function of a .

In Figure 1 Figure 1 shows that for small lags (models AR(1) and AR(2)) the power increases as the value of α increases. In all the other cases the power increases up to a value of α and then decreases. More specifically, for the complete AR(3) and AR(4) processes the power stays high approximately up to the value 0.25 a =

. For the AR(5) process the power decreases after the value 0.10 a = .

Although an optimum value of a for all types of autoregressive models may be considered to be the value of 0.10, in the Monte Carlo study of section 3.2 we choose the value of 0.25 since it appears to serve as a fair balance between small and large lag models. 3. Simulations

The Monte Carlo Experiment

In this section we provide the characteristics of the Monte Carlo experiment undertaken. We calculate the estimated "percent" by simply observing how many times the correct AR(p) . We also obtain the results of MDIC with the index a taken to be equal to that value in [0, 0.50] which maximizes the power. According to the analysis in Section 2.3, the maximum powers are obtained for 0.50 a =

(Models 1-5) and 0.10 a = (Model 6). For the implementation of the criteria we use

( ) 2 ˆ2 2 ( ) 1 ( ) exp 2 2 i i i x x f x θ σ πσ - ⎧ ⎫ = - ⎨ ⎬ ⎩ ⎭ ,
where ˆi x the predicted value of i x and 2 σ the estimated variance.

For each time series 20 presample values are generated with zero initial conditions, taking net sample sizes of n = 50,75,100,200,500 in order to cover small, medium and large samples.

The number of replications per model used is 10000. The calculations were performed using GAUSS 8.

Simulation Results

In this section we present the results of the Monte Carlo experiment concerning the percent of correct selected model. Due to insignificant differences between MDIC and MDIC with max power, the powers of the latter for Models 3-6 are suppressed from the Tables.

Table 1 shows the results for Model 1. It is not difficult to see the good performance of the MDIC criterion for all sample sizes. The rate of success of MDIC is almost 98% for the small sample with 50 observations, while the other criteria have a rate of success between 76.9% While in Table 1 MDIC performs quite well with though some competition from the SIC criterion, in the case of the second model, as seen in Table 2, the MDIC is superior to all other criteria for all sample sizes. Observe that MDIC is the best criterion among the competing criteria with the SIC coming second. The worst performance was observed by AIC. The sample effect shows the tendency of the best 2 criteria, namely MDIC and SIC, to approach each other in terms of their rate of success as n increases. Observe the impressive success rate of MDIC even for small sample sizes where for n=75 reaches a remarkable success rate of at least 95%. Our attempt to see if the incomplete AR(2) process misleads the Information criteria does not seem to work since the rate of success is not significantly affected by the particular form of the underlying process.

In case of Models 3 and 5 as Tables 3 and5 show the results are not different from the previous cases. Indeed, MDIC is superior to all other criteria with success rate as high as 95%

for small and medium sample sizes and stays slightly behind SIC for large sample sizes.

In case of Model 4 as Table 4 shows all criteria for small sample sizes (n=50) have a high tendency of selecting the simplest model (AR(1)) which is due to the fact that the coefficient Finally for Model 6, Table 6 shows that for small sample sizes the rate of success is less than 50% with AIC being the best, MDIC being the worse and SIC coming second to last. As the sample size increases, MDIC and SIC have the higher improvement so that for 100 n ≥ they become again the best models with MDIC reaching an impressive success rate of 98.5% for n=200 and 99.1% for n=500.

It should be pointed out that the choice of the index a plays an important role in model selection and controls the rate of success of MDIC. As indicated in Section 2.3 for small lag models, values of a in the interval [0.21, 0.50] appear to maximize the rate. On the other hand, values of a in the interval [0.05, 0.25] produce the best results for large lag models.

Note though that the maximum powers are obtained for values of a in the upper end of the former interval, for small lag models and the lower end of the latter, for large lag models. The value of 0.25 proposed in this work, appears to offer a good balance irrespectively of the type of the underlying AR model. If the candidate set consists exclusively of AR models of one type, we suggest a larger value of a , around 0.35, for small lag models and a smaller one, around 0.10, for large lag models. In reference to the sample size, our simulations clearly show that especially for small and medium sample sizes, the MDIC criterion with 0.25 a = provides an ideal model identification technique for both small and large lag models.

To summarize the findings we could safely conclude that the MDIC criterion performs much better than the other selection criteria for AR processes and for various sample sizes. In particular, the magnitude of the superiority of MDIC is extremely high for AIC and FPE and relatively high for HQ, for all sample sizes. In reference to the SIC criterion, the superiority of MDIC is of much lesser magnitude as compared to the other criteria.

Concluding Remarks

In this paper we have proposed the Modified Divergence Information Criterion (MDIC) for the determination of the order of an autoregressive process and proved its asymptotic unbiasedness. Furthermore, we performed a comparative study of model selection criteria for autoregressive processes for small, medium, and large sample sizes in order to investigate the practical implications of the new criterion. Based on the results of our simulation study we conclude that for regular AR models, the performance of MDIC is excellent with very high rate of success, for all sample sizes. The rate of success of MDIC increases with sample size but usually not as much as the HQ and SIC criteria which is expected though since they are both consistent. It is important to point out that in almost all cases MDIC with 0.25 a = performs extremely well with a high rate of success (approx. equal to 90%) for both small and medium sample sizes and for all types of models. As a result and especially for small and medium sample sizes, the choice 0.25 a =

is highly recommended since it serves as a fair balance between small and large lag models.

For irregular models (like Model 6), MDIC needs a sufficient number of observations for performing well. More specifically, a medium sample size of order 75 or 100 seems to be enough in order to pick up high rates of success.

Finally observe that MDIC is the only criterion that never selects too large models. In fact, in all cases the probability of overestimation is at most 8% and for lags at most 2 or 3 higher than the true lag while other criteria may select lags as large as 7 lags higher than the true one.

A similar behaviour but to a lesser extend is observed by SIC.

Based on the above observations we can conclude that MDIC chooses, at least for regular models, the correct model quite easily even with sample sizes as small as n=50. At the same time, the rate of success appears to plateau as the sample size increases. As it is known, consistency is an attractive asymptotic property. In particular, as n tends to infinity we have 0,769 0,108 0,049 0,022 0,018 0,012 0,013 0,009 SIC 0,909 0,068 0,015 0,003 0,004 0,001 0 0 HQ 0,836 0,097 0,028 0,015 0,009 0,008 0,005 0,002 FPE 0,769 0,11 0,049 0,021 0,017 0,012 0,013 0,009 MDIC 0.25 0,889 0,095 0,014 0,002 0 0 0 0 max 0,974 0,026 0 0 75 obs AIC 0,775 0,124 0,04 0,025 0,016 0,009 0,005 0,006 SIC 0,936 0,050 0,011 0,001 0,001 0,001 0 0 HQ 0,873 0,085 0,024 0,008 0,004 0,002 0,001 0,003 FPE 0,777 0,124 0,04 0,025 0,017 0,008 0,003 0,006 MDIC 0.25 0,894 0,086 0,018 0,001 0,001 0 0 0 max 0,976 0,024 0 0 0 100 obs AIC 0,774 0,107 0,057 0,023 0,019 0,01 0,004 0,006 SIC 0,954 0,034 0,008 0,003 0,001 0 0 0 HQ 0,881 0,071 0,027 0,009 0,009 0,002 0,001 0 FPE 0,774 0,107 0,057 0,023 0,019 0,01 0,005 0,005 MDIC 0.25 0,896 0,082 0,017 0,004 0,001 0 0 0 max 0,982 0,018 0 0 0 200 obs AIC 0,797 0,116 0,038 0,023 0,012 0,007 0,005 0,002 SIC 0,966 0,032 0,001 0,001 0 0 0 0 HQ 0,916 0,062 0,013 0,005 0,001 0,003 0 0 FPE 0,797 0,116 0,038 0,023 0,012 0,007 0,005 0,002 MDIC 0.25 0,896 0,088 0,012 0,002 0,002 0 0 0 max 0,981 0,019 0 0 0 500 obs AIC 0,776 0,117 0,049 0,019 0,016 0,013 0,008 0,002 SIC 0,977 0,022 0,001 0 0 0 0 0 HQ 0,918 0,07 0,01 0,001 0,001 0 0 0 FPE 0,776 0,117 0,049 0,019 0,016 0,013 0,008 0,002 MDIC 0.25 0,893 0,098 0,009 0 0 0 0 0 max 0,977 0,022 0,010

The shading indicates best performance Information Criterion. 0,018 0,748 0,103 0,05 0,026 0,02 0,015 0,02 SIC 0,047 0,865 0,063 0,014 0,006 0,003 0,001 0,001 HQ 0,034 0,805 0,08 0,034 0,02 0,014 0,006 0,007 FPE 0,018 0,751 0,103 0,047 0,026 0,02 0,015 0,02 MDIC 0.25 0,037 0,894 0,06 0,008 0,001 0 0 0 max 0,095 0,902 0,003 0 0 75 obs AIC 0,001 0,777 0,104 0,051 0,028 0,017 0,009 0,013 SIC 0,005 0,947 0,039 0,008 0,001 0 0 0 HQ 0,002 0,878 0,076 0,024 0,01 0,006 0,001 0,003 FPE 0,001 0,777 0,104 0,051 0,028 0,017 0,009 0,013 MDIC 0.25 0,003 0,948 0,041 0,007 0,001 0 0 0 max 0,013 0,985 0,002 0 0 100 obs AIC 0,001 0,775 0,12 0,042 0,031 0,011 0,009 0,011 SIC 0,001 0,944 0,046 0,004 0,005 0 0 0 HQ 0,001 0,874 0,086 0,019 0,013 0,004 0,003 0 FPE 0,001 0,776 0,12 0,042 0,031 0,011 0,008 0,011 MDIC 0.25 0,001 0,932 0,063 0,004 0 0 0 0 max 0,001 0,993 0,006 0 200 obs AIC 0,001 0,814 0,104 0,027 0,032 0,007 0,004 0,011 SIC 0,001 0,974 0,022 0,003 0 0 0 0 HQ 0,001 0,915 0,061 0,013 0,007 0,002 0,001 0 FPE 0,001 0,814 0,104 0,027 0,032 0,007 0,004 0,011 MDIC 0.25 0,001 0,945 0,051 0,003 0 0 0 0 max 0,001 0,996 0,003 0 500 obs AIC 0,001 0,793 0,114 0,042 0,024 0,008 0,011 0,007 SIC 0,001 0,987 0,012 0 0 0 0 0 HQ 0,001 0,915 0,066 0,012 0,005 0,001 0 0 FPE 0,001 0,793 0,114 0,042 0,024 0,008 0,011 0,007 MDIC 0.25 0,001 0,939 0,057 0,003 0 0 0 0 max 0,001 0,996 0,003 0

The shading indicates best performance Information Criterion. 0,001 0,749 0,121 0,052 0,024 0,027 0,015 0,011 SIC 0,001 0,903 0,072 0,017 0,003 0,004 0,001 0,001 HQ 0,001 0,825 0,106 0,031 0,011 0,02 0,004 0,002 FPE 0,001 0,749 0,122 0,052 0,024 0,027 0,015 0,01 MDIC 0,001 0,920 0,072 0,007 0 0 0 0 75 obs AIC 0,001 0,78 0,094 0,059 0,034 0,012 0,01 0,01 SIC 0,001 0,944 0,039 0,008 0,001 0 0 0 HQ 0,001 0,882 0,068 0,024 0,016 0,003 0,003 0,003 FPE 0,001 0,781 0,094 0,058 0,034 0,012 0,01 0,01 MDIC 0,001 0,944 0,048 0,007 0 0 0 0 100 obs AIC 0,001 0,764 0,121 0,046 0,027 0,022 0,013 0,006 SIC 0,001 0,931 0,049 0,007 0,002 0 0 0 HQ 0,001 0,879 0,08 0,019 0,014 0,005 0,002 0 FPE 0,001 0,765 0,121 0,046 0,027 0,022 0,012 0,006 MDIC 0,001 0,933 0,061 0,005 0 0 0 0 200 obs AIC 0,001 0,776 0,116 0,042 0,031 0,018 0,009 0,007 SIC 0,001 0,968 0,029 0,002 0 0 0 0 HQ 0,001 0,913 0,066 0,009 0,009 0,002 0 0 FPE 0,001 0,777 0,116 0,042 0,03 0,018 0,009 0,007 MDIC 0,001 0,949 0,047 0,003 0 0 0 0 500 obs AIC 0,001 0,795 0,095 0,058 0,025 0,012 0,011 0,003 SIC 0,001 0,983 0,014 0,002 0 0 0 0 HQ 0,001 0,931 0,046 0,018 0,004 0 0 0 FPE 0,001 0,795 0,095 0,058 0,025 0,012 0,011 0,003 MDIC 0,001 0,945 0,048 0,006 0 0 0 0

The shading indicates best performance Information Criterion. 0,159 0,607 0,111 0,046 0,026 0,026 0,013 0,012 SIC 0,306 0,622 0,045 0,019 0,005 0,002 0,001 0,001 HQ 0,214 0,643 0,078 0,036 0,009 0,013 0,002 0,005 FPE 0,161 0,61 0,109 0,045 0,026 0,025 0,012 0,012 MDIC 0,268 0,673 0,052 0,007 0 0 0 0 75 obs AIC 0,075 0,71 0,112 0,045 0,03 0,015 0,007 0,006 SIC 0,200 0,751 0,037 0,009 0,001 0 0 0 HQ 0,123 0,756 0,074 0,024 0,014 0,003 0,002 0,004 FPE 0,077 0,71 0,112 0,045 0,029 0,014 0,008 0,005 MDIC 0,144 0,800 0,049 0,007 0 0 0 0 100 obs AIC 0,033 0,739 0,11 0,054 0,021 0,024 0,01 0,009 SIC 0,102 0,846 0,038 0,010 0,002 0 0 0 HQ 0,059 0,818 0,072 0,033 0,009 0,006 0,002 0,001 FPE 0,033 0,739 0,11 0,054 0,021 0,024 0,01 0,009 MDIC 0,072 0,863 0,051 0,013 0,001 0 0 0 200 obs AIC 0,001 0,779 0,114 0,047 0,033 0,014 0,005 0,007 SIC 0,007 0,965 0,019 0,007 0,002 0 0 0 HQ 0,002 0,909 0,06 0,016 0,007 0,005 0,001 0 FPE 0,001 0,779 0,114 0,047 0,033 0,014 0,005 0,007 MDIC 0,003 0,938 0,048 0,009 0,002 0 0 0 500 obs AIC 0,001 0,795 0,106 0,053 0,022 0,013 0,008 0,002 SIC 0,001 0,980 0,016 0,003 0 0 0 0 HQ 0,001 0,917 0,061 0,018 0,003 0 0 0 FPE 0,001 0,795 0,106 0,053 0,022 0,013 0,008 0,002 MDIC 0,001 0,933 0,059 0,007 0 0 0 0

The shading indicates best performance Information Criterion. 0,025 0,712 0,137 0,045 0,027 0,027 0,013 0,014 SIC 0,203 0,699 0,069 0,014 0,010 0,003 0,001 0,001 HQ 0,042 0,782 0,106 0,031 0,012 0,016 0,004 0,007 FPE 0,026 0,716 0,136 0,045 0,026 0,027 0,012 0,012 MDIC 0,058 0,861 0,073 0,007 0,001 0 0 0 75 obs AIC 0,005 0,763 0,127 0,046 0,028 0,014 0,008 0,009 SIC 0,058 0,869 0,062 0,010 0,001 0 0 0 HQ 0,009 0,861 0,086 0,027 0,011 0,003 0,001 0,002 FPE 0,005 0,764 0,127 0,046 0,028 0,014 0,007 0,009 MDIC 0,011 0,927 0,053 0,008 0,001 0 0 0 100 obs AIC 0,001 0,759 0,119 0,055 0,024 0,025 0,009 0,008 SIC 0,019 0,918 0,050 0,006 0,006 0,001 0 0 HQ 0,002 0,859 0,089 0,027 0,013 0,009 0,001 0 FPE 0,001 0,76 0,118 0,055 0,024 0,025 0,009 0,008 MDIC 0,002 0,927 0,062 0,007 0,002 0 0 0 200 obs AIC 0,001 0,775 0,12 0,044 0,033 0,013 0,009 0,005 SIC 0,001 0,966 0,029 0,003 0,001 0 0 0 HQ 0,001 0,898 0,071 0,019 0,004 0,004 0,003 0 FPE 0,001 0,775 0,12 0,044 0,033 0,013 0,009 0,005 MDIC 0,001 0,941 0,05 0,008 0 0 0 0 500 obs AIC 0,001 0,789 0,107 0,052 0,026 0,011 0,01 0,004 SIC 0,001 0,979 0,019 0,001 0 0 0 0 HQ 0,001 0,917 0,055 0,018 0,007 0,002 0 0 FPE 0,001 0,789 0,107 0,052 0,026 0,011 0,01 0,004 MDIC 0,001 0,936 0,056 0,007 0 0 0 0

F

The shading indicates best performance Information Criterion. 0,149 0,02 0,096 0,033 0,498 0,114 0,049 0,041 SIC 0,423 0,025 0,113 0,026 0,358 0,040 0,011 0,004 HQ 0,237 0,022 0,113 0,033 0,466 0,08 0,029 0,02 FPE 0,15 0,02 0,097 0,033 0,497 0,114 0,048 0,041 MDIC 0,495 0,051 0,205 0,026 0,22 0,003 0 0 75 obs AIC 0,059 0,001 0,031 0,014 0,702 0,12 0,047 0,026 SIC 0,238 0,007 0,075 0,015 0,627 0,024 0,011 0,003 HQ 0,121 0,008 0,051 0,013 0,701 0,068 0,026 0,012 FPE 0,059 0,001 0,031 0,014 0,704 0,119 0,046 0,026 MDIC 0,284 0,026 0,146 0,016 0,524 0,004 0 0 100 obs AIC 0,012 0,001 0,008 0,004 0,751 0,139 0,05 0,035 SIC 0,121 0,006 0,035 0,004 0,787 0,036 0,004 0,007 HQ 0,038 0,004 0,013 0,005 0,813 0,087 0,025 0,015 FPE 0,012 0,001 0,008 0,004 0,752 0,139 0,049 0,035 MDIC 0,128 0,015 0,08 0,011 0,758 0,008 0 0 200 obs AIC 0,001 0,001 0,001 0,001 0,801 0,111 0,047 0,037 SIC 0,001 0,001 0,001 0,001 0,961 0,032 0,003 0,001 HQ 0,001 0,001 0,001 0,001 0,901 0,078 0,015 0,002 FPE 0,001 0,001 0,001 0,001 0,801 0,111 0,047 0,037 MDIC 0,002 0,001 0,001 0,001 0,985 0,01 0 0 500 obs AIC 0,001 0,001 0,001 0,001 0,813 0,117 0,034 0,032 SIC 0,001 0,001 0,001 0,001 0,973 0,020 0,002 0,001 HQ 0,001 0,001 0,001 0,001 0,922 0,067 0,005 0,002 FPE 0,001 0,001 0,001 0,001 0,813 0,117 0,035 0,031 MDIC 0,001 0,001 0,001 0,001 0,991 0,005 0 0

F

The shading indicates best performance Information Criterion.

F

p A z a z a z p = - -- ≠ L for all z ∈ such that 1 z < .
One of the crucial problems in time series modeling is selecting the optimal model order and specifying a parsimonious model for the data generating process (DGP). Many techniques have been developed for the order selection of linear models, namely a) graphical methods with representative the Box and Jenkins methods (1970) and b) hypothesis test procedures with the pioneering work of Whittle (1952) as representative. Akaike (1974) viewed the problem of model fitting in the context of time series analysis as a multiple decision procedure rather than a hypothesis testing problem. However the same author (Akaike, 1973) initiated the research on automatic model selection techniques by developing the popular Akaike Information Criterion (AIC) which is considered as a significant contribution to statistical modelling where the focus is on the selection of a satisfactory model for a given set of observations. It might appear at first sight that the higher the order of the selected process (model) chosen, the better the fitted model will be. Such a thought may be true for fitting purposes but not for predicting ones. Indeed, the fit may be perfect for the given data but the use of the selected model for predictive purposes may result in gross errors. Numerous selection criteria have been developed over the last 30 years which attempt to prevent overfitting by assigning a penalty to the introduction of an unnecessary 3 large order and consequently to an unnecessary number of parameters. Schwarz (1978), Hannan and Quinn (1979), Seghouane et. al (2005), Bengtsson and Cavanaugh (2006) and others followed the pioneering work of Akaike. Most of the criteria proposed in the literature consist in minimizing, with respect to the model order, a function of the given observations plus a penalty term for the introduction of additional parameters, which generally depends on the model order and sample size. Thus, according to automatic criteria, the parameters of a variety of competing models are estimated and the model chosen is the one with the smallest criterion value.

In this work we propose a modification of the recently introduced Divergence Information Criterion (DIC) (Mattheou (2007); Mattheou, et. al (2009)) and show that it is an asymptotically unbiased estimator of the expected overall discrepancy, a nonnegative quantity which measures the distance between the true unknown model and a fitted approximating model. Furthermore, we explore the capabilities of the new criterion in selecting the optimal order in autoregressive processes and in general in a time series context. The paper is organised as follows. In Section 2, we first briefly discuss various Information Criteria and their association with measures of divergence and then introduce the Modified Divergence Information Criterion (MDIC) and provide for practical purposes, an optimum choice for the index α which the new criterion depends on. In Section 3 in order to explore the capabilities of the new MDIC criterion, we perform a comparative study of various Information Criteria for different time series examples by using a Monte Carlo method. Finally in Section 4 we summarize our results. For the simulations we make use of the Gauss 8 program.

The New Modified Divergence Information Criterion

Measures of Divergence and Selection Criteria

The measures of divergence are used as indices of similarity or dissimilarity between populations and for measuring mutual information concerning two variables and as such they can be used for the construction of model selection criteria. The distance between a candidate model and the true but unknown model could be measured by a measure of divergence. The candidate model for which the measure of divergence is minimized will be selected. The well known Kullback-Leibler measure of divergence (Kullback and Leibler, 1951) was the one used by Akaike (1973) to develop the Akaike Information Criterion (AIC). Akaike proposed the evaluation of the fit of the candidate model using minus twice the mean expected loglikelihood (also known as expected overall discrepancy). Furthermore, he provided an unbiased estimator of the expected loglikelihood so that the resulting criterion is given by For fitting autoregressive processes, Jones (1975) suggested that AIC has a tendency to overestimate the order of the process and Shibata (1976) showed that the probability of overestimation for a large sample size is nonzero. To correct this tendency, the SIC criterion was proposed independently by Akaike (1978) and Schwarz (1978): Hannan and Quinn, 1979) which has been found to be equivalent to SIC in the sense that they are both consistent (Hannan, 1980).

The problem of avoiding overfitting especially for purely autoregressive processes can also be dealt with the minimization of the final prediction error criterion (FPE) of Akaike (1969). The FPE is an estimate of the one-step ahead prediction mean squared error for a realization of the process independent of the one observed. If we fit AR processes of steadily increasing order p the maximum likelihood estimator (MLE) of the white noise variance will usually decrease with p. However, FPE will decrease to a minimum value and then increase as p will increase.

According to FPE, we choose the order of the candidate process to be the value of p for which FPE is minimized. The FPE is given by 2 ( )

p n p FPE p n p σ + = - .
In all the above cases, any parameter estimation is handled through the maximum likelihood method. A general class of criteria has been introduced by Konishi and Kitagawa (1996) which also estimates the Kullback-Leibler measure where the estimation is not necessarily based on maximum likelihood.

The Development of the MDIC Criterion

One of the most recently proposed measures of divergence is the Basu-Harris-Hjort-Jones power divergence between the candidate model (.) f θ and the true model g (Basu et. al, 1998) which is denoted by BHHJ, indexed by a positive parameter a , and defined as: This family of measures was proposed by Basu et al. (1998) for the development of a minimum divergence estimating method for robust parameter estimation. The index α controls the trade-off between robustness and asymptotic efficiency of the parameter estimators which are the values of θ that minimize the measure over a parametric spaceΘ . It should be also noted that the BHHJ family reduces to the Kullback-Leibler measure of divergence for 0 a ↓ (see Mattheou, 2007) and as it can be easily seen, to the square of the standard L 2 distance between the candidate and the true model for a =1. Mattheou et al. (2008)

1 1 1 1 ( , ) ( ) 1 ( ) ( ) ( ) a a a a I g f f z g z f z g z dz a a θ θ θ       + + = -+ +             ∫ .
= + + , where 1 ˆˆ1 1 1 ( ) 1 ( ) n a a i i Q f z dz f x a n θ θ θ + =   = -+     ∑ ∫
and θ a consistent and asymptotically normal estimator of θ . Preliminary simulation studies for regression models (Mattheou, 2007) showed a very good medium sample size performance for DIC for values of α close to zero.

Although the DIC criterion was constructed so that it will be an asymptotically unbiased estimator of the BHHJ divergence measure between the candidate and the true model, the calculation of the first part of Qθ , namely the integral where ( )

1 ˆ1 1 1 ( ) n a i i MQ f x n θ θ α - =   = -+     ∑ .
Note that a model selection criterion can be considered as an approximately unbiased estimator of the expected overall discrepancy, a nonnegative quantity which measures the distance between the true unknown model and a fitted approximating model. Observe also that, as it was mentioned in the introduction, a criterion consists of two terms, the first of which is a biased estimator of the expected overall discrepancy. As a result, if we choose the model with the smallest estimator of the expected overall discrepancy we may end up with a selection with an unnecessarily large order. The estimator becomes asymptotically unbiased by introducing the appropriate correction term. The expected overall discrepancy in the present setting is defined by

ˆ( ) ( / ) E W E W θ θ θ θ = = with 1 1 ( ) ( ) a W g z f z dz a θ θ     = - +        
∫ and θ a consistent and asymptotically normal estimator of θ. Observe that W θ is the targeted divergence ( , ) a I g f θ without the first and the last terms. Notice that the last term remains constant irrespectively of the candidate model f θ while the first term, as stated earlier, is approximately constant for the various candidate models. In that sense, the expected overall discrepancy ( ) E W θ represents the average distance between the true model g and the candidate model f θ up to a constant. The relevant theorem for MDIC is stated below. The proof is omitted since it follows the lines of the proof of Theorem 2.4 (Mattheou et.al, 2009) where the unbiasedness of the original DIC criterion was established.

2 / 2 / 2 ( ) * (2 ) (1 ) ˆp a MDIC p n MQ a p π θ + - = + +
where θ is a consistent and asymptotically normal estimator of θ.

Note that although the discussion in Mattheou et. al (2009) is limited to iid random variables, by following the approach by Akaike (1973 and1974) and Billingsley (1961), the same line of discussion can be extended to cover autoregressive models. Recall that in time series models, even under the Gaussian assumption, the evaluation of the log-likelihood cannot be expressed as the sum of the logarithms of the density function of each observation. However, the log-likelihood can generally be expressed by conditional distributions, so that in relatively simple models, like the autoregressive models, we use as an approximation the mean conditional log-likelihood which multiplied by the number of observations is used in place of the log-likelihood in model selection.

In addition to the above, numerous other techniques have been introduced over the years some of which are associated with the order determination and the forecasting performance in time series. Such techniques include approaches based on bootstrapping (Shang and Cavanaugh, 2008), approaches for high dimensional medical data (Koukouvinos et al., 2008), supersaturated designs (Koukouvinos and Mylona, 2008), variations of AIC (Cavanaugh and Johnson, 1999;Bengtsson and Cavanaugh, 2006;Seghouane, 2006;Seghouane et. al, 2005) and time series models (Lee, 1998).

Optimal Choice for the Index a

For practical purposes one has to decide the optimal choice of the positive index a . For this purpose, we simulate a 100 observation series for 5 different models with a between 0.01 and 0.5 and we provide in Figure 1, the power of the selection, namely the proportion of times the correct model is selected as a function of a .

In Figure 1, [Dot dash] line is the AR(1) model defined as Model 1 in section 3.2; [Whole] line shows the AR(2) appearing as Model 5 in section 3.2; [Dot close] line refers to the AR(3) process: = + + --+ ; finally [Dot] is the AR(5) process defined as Model 6 in section 3.2.

1 2 3 1 0.2 0.
Figure 1 shows that for small lags (models AR(1) and AR(2)) the power increases as the value of α increases. In all the other cases the power increases up to a value of α and then decreases. More specifically, for the complete AR(3) and AR(4) processes the power stays high approximately up to the value 0.25 a =

. For the AR(5) process the power decreases after the value 0.10 a = .

Although an optimum value of a for all types of autoregressive models may be considered to be the value of 0.10, in the Monte Carlo study of section 3.2 we choose the value of 0.25 since it appears to serve as a fair balance between small and large lag models. The Monte Carlo experiment has been performed by generating data according to the following Data generating processes:

Model 1: . We also obtain the results of MDIC with the index a taken to be equal to that value in [0, 0.50] which maximizes the power. According to the analysis in Section 2.3, the maximum powers are obtained for 0.50 a =

(Models 1-5) and 0.10 a = (Model 6). For the implementation of the criteria we use ( )

2 ˆ2 2 ( ) 1 ( ) exp 2 2 i i i x x f x θ σ πσ -   = -     ,
where ˆi x the predicted value of i x and 2 σ the estimated variance.

For each time series 20 presample values are generated with zero initial conditions, taking net sample sizes of n = 50,75,100,200,500 in order to cover small, medium and large samples.

The number of replications per model used is 10000. The calculations were performed using GAUSS 8.

Simulation Results

In this section we present the results of the Monte Carlo experiment concerning the percent of correct selected model. Due to insignificant differences between MDIC and MDIC with max power, the powers of the latter for Models 3-6 are suppressed from the Tables.

Table 1 shows the results for Model 1. While in Table 1 MDIC performs quite well with though some competition from the SIC criterion, in the case of the second model, as seen in Table 2, the MDIC is superior to all other criteria for all sample sizes. Observe that MDIC is the best criterion among the competing criteria with the SIC coming second. The worst performance was observed by AIC. The sample effect shows the tendency of the best 2 criteria, namely MDIC and SIC, to approach each other in terms of their rate of success as n increases. Observe the impressive success rate of MDIC even for small sample sizes where for n=75 reaches a remarkable success rate of at least 95%. Our attempt to see if the incomplete AR(2) process misleads the Information criteria does not seem to work since the rate of success is not significantly affected by the particular form of the underlying process.

In case of Models 3 and 5 as Tables 3 and5 show the results are not different from the previous cases. Indeed, MDIC is superior to all other criteria with success rate as high as 95%

for small and medium sample sizes and stays slightly behind SIC for large sample sizes.

In case of Model 4 as Finally for Model 6, Table 6 shows that for small sample sizes the rate of success is less than 50% with AIC being the best, MDIC being the worse and SIC coming second to last. As the sample size increases, MDIC and SIC have the higher improvement so that for 100 n ≥ they become again the best models with MDIC reaching an impressive success rate of 98.5% for n=200 and 99.1% for n=500.

It should be pointed out that the choice of the index a plays an important role in model selection and controls the rate of success of MDIC. As indicated in Section 2.3 for small lag models, values of a in the interval [0.21, 0.50] appear to maximize the rate. On the other hand, values of a in the interval [0.05, 0.25] produce the best results for large lag models.

Note though that the maximum powers are obtained for values of a in the upper end of the former interval, for small lag models and the lower end of the latter, for large lag models. The value of 0.25 proposed in this work, appears to offer a good balance irrespectively of the type of the underlying AR model. If the candidate set consists exclusively of AR models of one type, we suggest a larger value of a , around 0.35, for small lag models and a smaller one, around 0.10, for large lag models. In reference to the sample size, our simulations clearly show that especially for small and medium sample sizes, the MDIC criterion with 0.25 a = provides an ideal model identification technique for both small and large lag models.

To summarize the findings we could safely conclude that the MDIC criterion performs much better than the other selection criteria for AR processes and for various sample sizes. In particular, the magnitude of the superiority of MDIC is extremely high for AIC and FPE and relatively high for HQ, for all sample sizes. In reference to the SIC criterion, the superiority of MDIC is of much lesser magnitude as compared to the other criteria.

Concluding Remarks

In this paper we have proposed the Modified Divergence Information Criterion (MDIC) for the determination of the order of an autoregressive process and proved its asymptotic unbiasedness. Furthermore, we performed a comparative study of model selection criteria for autoregressive processes for small, medium, and large sample sizes in order to investigate the practical implications of the new criterion. Based on the results of our simulation study we conclude that for regular AR models, the performance of MDIC is excellent with very high rate of success, for all sample sizes. The rate of success of MDIC increases with sample size but usually not as much as the HQ and SIC criteria which is expected though since they are both consistent. It is important to point out that in almost all cases MDIC with 0.25 a = performs extremely well with a high rate of success (approx. equal to 90%) for both small and medium sample sizes and for all types of models. As a result and especially for small and medium sample sizes, the choice 0.25 a = is highly recommended since it serves as a fair balance between small and large lag models.

For irregular models (like Model 6), MDIC needs a sufficient number of observations for performing well. More specifically, a medium sample size of order 75 or 100 seems to be enough in order to pick up high rates of success.

Finally observe that MDIC is the only criterion that never selects too large models. In fact, in all cases the probability of overestimation is at most 8% and for lags at most 2 or 3 higher than the true lag while other criteria may select lags as large as 7 lags higher than the true one.

A similar behaviour but to a lesser extend is observed by SIC.

Based on the above observations we can conclude that MDIC chooses, at least for regular models, the correct model quite easily even with sample sizes as small as n=50. 0,769 0,108 0,049 0,022 0,018 0,012 0,013 0,009 SIC 0,909 0,068 0,015 0,003 0,004 0,001 0 0 HQ 0,836 0,097 0,028 0,015 0,009 0,008 0,005 0,002 FPE 0,769 0,11 0,049 0,021 0,017 0,012 0,013 0,009 0.25 0,889 0,095 0,014 0,002 MDIC max 0,974 0,026 0 0 0 0 0 0 75 obs AIC 0,775 0,124 0,04 0,025 0,016 0,009 0,005 0,006 SIC 0,936 0,050 0,011 0,001 0,001 0,001 0 0 HQ 0,873 0,085 0,024 0,008 0,004 0,002 0,001 0,003 FPE 0,777 0,124 0,04 0,025 0,017 0,008 0,003 0,006 0.25 0,894 0,086 0,018 0,001 0,001 MDIC max 0,976 0,024 0 0 0 0 0 0 100 obs AIC 0,774 0,107 0,057 0,023 0,019 0,01 0,004 0,006 SIC 0,954 0,034 0,008 0,003 0,001 0 0 0 HQ 0,881 0,071 0,027 0,009 0,009 0,002 0,001 0 FPE 0,774 0,107 0,057 0,023 0,019 0,01 0,005 0,005 0.25 0,896 0,082 0,017 0,004 0,001 MDIC max 0,982 0,018 0 0 0 0 0 0 200 obs AIC 0,797 0,116 0,038 0,023 0,012 0,007 0,005 0,002 SIC 0,966 0,032 0,001 0,001 0 0 0 0 HQ 0,916 0,062 0,013 0,005 0,001 0,003 0 0 FPE 0,797 0,116 0,038 0,023 0,012 0,007 0,005 0,002 0.25 0,896 0,088 0,012 0,002 0,002 MDIC max 0,981 0,019 0 0 0 0 0 0 500 obs AIC 0,776 0,117 0,049 0,019 0,016 0,013 0,008 0,002 SIC 0,977 0,022 0,001 0 0 0 0 0 HQ 0,918 0,07 0,01 0,001 0,001 0 0 0 FPE 0,776 0,117 0,049 0,019 0,016 0,013 0,008 0,002 0.25 0,893 0,098 0,009 MDIC max 0,977 0,022 0,010 0 0 0 0 0

The shading indicates best performance Information Criterion. 0,001 0,749 0,121 0,052 0,024 0,027 0,015 0,011 SIC 0,001 0,903 0,072 0,017 0,003 0,004 0,001 0,001 HQ 0,001 0,825 0,106 0,031 0,011 0,02 0,004 0,002 FPE 0,001 0,749 0,122 0,052 0,024 0,027 0,015 0,01 MDIC 0,001 0,920 0,072 0,007 0 0 0 0 75 obs AIC 0,001 0,78 0,094 0,059 0,034 0,012 0,01 0,01 SIC 0,001 0,944 0,039 0,008 0,001 0 0 0 HQ 0,001 0,882 0,068 0,024 0,016 0,003 0,003 0,003 FPE 0,001 0,781 0,094 0,058 0,034 0,012 0,01 0,01 MDIC 0,001 0,944 0,048 0,007 0 0 0 0 100 obs AIC 0,001 0,764 0,121 0,046 0,027 0,022 0,013 0,006 SIC 0,001 0,931 0,049 0,007 0,002 0 0 0 HQ 0,001 0,879 0,08 0,019 0,014 0,005 0,002 0 FPE 0,001 0,765 0,121 0,046 0,027 0,022 0,012 0,006 MDIC 0,001 0,933 0,061 0,005 0 0 0 0 200 obs AIC 0,001 0,776 0,116 0,042 0,031 0,018 0,009 0,007 SIC 0,001 0,968 0,029 0,002 0 0 0 0 HQ 0,001 0,913 0,066 0,009 0,009 0,002 0 0 FPE 0,001 0,777 0,116 0,042 0,03 0,018 0,009 0,007 MDIC 0,001 0,949 0,047 0,003 0 0 0 0 500 obs AIC 0,001 0,795 0,095 0,058 0,025 0,012 0,011 0,003 SIC 0,001 0,983 0,014 0,002 0 0 0 0 HQ 0,001 0,931 0,046 0,018 0,004 0 0 0 FPE 0,001 0,795 0,095 0,058 0,025 0,012 0,011 0,003 MDIC 0,001 0,945 0,048 0,006 0 0 0 0

The shading indicates best performance Information Criterion. 0,159 0,607 0,111 0,046 0,026 0,026 0,013 0,012 SIC 0,306 0,622 0,045 0,019 0,005 0,002 0,001 0,001 HQ 0,214 0,643 0,078 0,036 0,009 0,013 0,002 0,005 FPE 0,161 0,61 0,109 0,045 0,026 0,025 0,012 0,012 MDIC 0,268 0,673 0,052 0,007 0 0 0 0 75 obs AIC 0,075 0,71 0,112 0,045 0,03 0,015 0,007 0,006 SIC 0,200 0,751 0,037 0,009 0,001 0 0 0 HQ 0,123 0,756 0,074 0,024 0,014 0,003 0,002 0,004 FPE 0,077 0,71 0,112 0,045 0,029 0,014 0,008 0,005 MDIC 0,144 0,800 0,049 0,007 0 0 0 0 100 obs AIC 0,033 0,739 0,11 0,054 0,021 0,024 0,01 0,009 SIC 0,102 0,846 0,038 0,010 0,002 0 0 0 HQ 0,059 0,818 0,072 0,033 0,009 0,006 0,002 0,001 FPE 0,033 0,739 0,11 0,054 0,021 0,024 0,01 0,009 MDIC 0,072 0,863 0,051 0,013 0,001 0 0 0 200 obs AIC 0,001 0,779 0,114 0,047 0,033 0,014 0,005 0,007 SIC 0,007 0,965 0,019 0,007 0,002 0 0 0 HQ 0,002 0,909 0,06 0,016 0,007 0,005 0,001 0 FPE 0,001 0,779 0,114 0,047 0,033 0,014 0,005 0,007 MDIC 0,003 0,938 0,048 0,009 0,002 0 0 0 500 obs AIC 0,001 0,795 0,106 0,053 0,022 0,013 0,008 0,002 SIC 0,001 0,980 0,016 0,003 0 0 0 0 HQ 0,001 0,917 0,061 0,018 0,003 0 0 0 FPE 0,001 0,795 0,106 0,053 0,022 0,013 0,008 0,002 MDIC 0,001 0,933 0,059 0,007 0 0 0 0

The shading indicates best performance Information Criterion. 0,025 0,712 0,137 0,045 0,027 0,027 0,013 0,014 SIC 0,203 0,699 0,069 0,014 0,010 0,003 0,001 0,001 HQ 0,042 0,782 0,106 0,031 0,012 0,016 0,004 0,007 FPE 0,026 0,716 0,136 0,045 0,026 0,027 0,012 0,012 MDIC 0,058 0,861 0,073 0,007 0,001 0 0 0 75 obs AIC 0,005 0,763 0,127 0,046 0,028 0,014 0,008 0,009 SIC 0,058 0,869 0,062 0,010 0,001 0 0 0 HQ 0,009 0,861 0,086 0,027 0,011 0,003 0,001 0,002 FPE 0,005 0,764 0,127 0,046 0,028 0,014 0,007 0,009 MDIC 0,011 0,927 0,053 0,008 0,001 0 0 0 100 obs AIC 0,001 0,759 0,119 0,055 0,024 0,025 0,009 0,008 SIC 0,019 0,918 0,050 0,006 0,006 0,001 0 0 HQ 0,002 0,859 0,089 0,027 0,013 0,009 0,001 0 FPE 0,001 0,76 0,118 0,055 0,024 0,025 0,009 0,008 MDIC 0,002 0,927 0,062 0,007 0,002 0 0 0 200 obs AIC 0,001 0,775 0,12 0,044 0,033 0,013 0,009 0,005 SIC 0,001 0,966 0,029 0,003 0,001 0 0 0 HQ 0,001 0,898 0,071 0,019 0,004 0,004 0,003 0 FPE 0,001 0,775 0,12 0,044 0,033 0,013 0,009 0,005 MDIC 0,001 0,941 0,05 0,008 0 0 0 0 500 obs AIC 0,001 0,789 0,107 0,052 0,026 0,011 0,01 0,004 SIC 0,001 0,979 0,019 0,001 0 0 0 0 HQ 0,001 0,917 0,055 0,018 0,007 0,002 0 0 FPE 0,001 0,789 0,107 0,052 0,026 0,011 0,01 0,004 MDIC 0,001 0,936 0,056 0,007 0 0 0 0

The shading indicates best performance Information Criterion. 0,149 0,02 0,096 0,033 0,498 0,114 0,049 0,041 SIC 0,423 0,025 0,113 0,026 0,358 0,040 0,011 0,004 HQ 0,237 0,022 0,113 0,033 0,466 0,08 0,029 0,02 FPE 0,15 0,02 0,097 0,033 0,497 0,114 0,048 0,041 MDIC 0,495 0,051 0,205 0,026 0,22 0,003 0 0 75 obs AIC 0,059 0,001 0,031 0,014 0,702 0,12 0,047 0,026 SIC 0,238 0,007 0,075 0,015 0,627 0,024 0,011 0,003 HQ 0,121 0,008 0,051 0,013 0,701 0,068 0,026 0,012 FPE 0,059 0,001 0,031 0,014 0,704 0,119 0,046 0,026 MDIC 0,284 0,026 0,146 0,016 0,524 0,004 0 0 100 obs AIC 0,012 0,001 0,008 0,004 0,751 0,139 0,05 0,035 SIC 0,121 0,006 0,035 0,004 0,787 0,036 0,004 0,007 HQ 0,038 0,004 0,013 0,005 0,813 0,087 0,025 0,015 FPE 0,012 0,001 0,008 0,004 0,752 0,139 0,049 0,035 MDIC 0,128 0,015 0,08 0,011 0,758 0,008 0 0 200 obs AIC 0,001 0,001 0,001 0,001 0,801 0,111 0,047 0,037 SIC 0,001 0,001 0,001 0,001 0,961 0,032 0,003 0,001 HQ 0,001 0,001 0,001 0,001 0,901 0,078 0,015 0,002 FPE 0,001 0,001 0,001 0,001 0,801 0,111 0,047 0,037 MDIC 0,002 0,001 0,001 0,001 0,985 0,01 0 0 500 obs AIC 0,001 0,001 0,001 0,001 0,813 0,117 0,034 0,032 SIC 0,001 0,001 0,001 0,001 0,973 0,020 0,002 0,001 HQ 0,001 0,001 0,001 0,001 0,922 0,067 0,005 0,002 FPE 0,001 0,001 0,001 0,001 0,813 0,117 0,035 0,031 MDIC 0,001 0,001 0,001 0,001 0,991 0,005 0 0

The shading indicates best performance Information Criterion.

  above two criteria can be considered as members of the General Information Criterion givenHannan and Quinn criterion (HQ,

  An asymptotically unbiased estimator of n--times the expected overall discrepancy evaluated at θ is given by

Figure 1 :

 1 Figure 1: Optimal choice of the index a (AR(1) -• -AR(2) ___ AR(3) ••••• AR(4)---AR(5) • • • )

  only criterion for which the percent of correct model selection increases significantly is the Hannan and Quinn criterion (HQ). A smaller increase is observed for the SIC criterion. For large sample sizes SIC and MDIC come first with a rate of success of approximately 98% and HQ close second with a success rate almost equal to 92%. At the same time, as the sample size increases, the difference between the MDIC and the remaining Information Criteria decreases, although it is still in favor of MDIC. The good performance of MDIC is evident from the high percent of selecting the correct AR(1) model in repeated samples.

  relatively close to zero (0.35). As the sample size increases the rate of success improves. MDIC is the best criterion in conjunction with HQ, for small sample sizes, and with SIC, for large sample sizes.

  sample size, p the order of the candidate model and 2 ˆp σ the estimator of the variance of the p th -order candidate model.

  Hannan and Quinn criterion (HQ,

  An asymptotically unbiased estimator of n--times the expected overall discrepancy evaluated at θ is given by

Figure 1 :

 1 Figure 1: Optimal choice of the index a (AR(1) -• -AR(2) ___ AR(3) ••••• AR(4)---AR(5) • • • )

  selection is significantly close to 1 even for very small values of n but stays below 1 even for very large sample sizes. This behaviour may not be the best one from the asymptotic (theoretical) point of view but it is ideal from the finite one due to desirable practical implications. It that sense, MDIC may not possess the characteristics of a consistent criterion but the fact that it is superior to all other criteria for finite sample sizes, makes it an excellent selection criterion for practical purposes. Furthermore, another attractive feature of MDIC is the one related to overfitting, where the criterion never selects very large models. Indeed, if k * is the true order of the model and k the order selected by MDIC, we observe that

			F o r F o r
	for	50 n ≥ ,	P 2] 0.005 P k k L L * [ , = + ≥ ≤ P	. In most cases, the probability drops to 0 even for L=3.
			e e
			e r e r
				R R
				e e
				v i e v i e
				w w
				O n l O n l
				y y

that P[correct selection] 1

→ . For MDIC, our results indicate that the probability of correct

Table nr 2

 nr : Model 2

	Lag1	Lag2	Lag3	Lag4	Lag5	Lag6	Lag7	Lag8
	50 obs
	AIC

  applying the same methodology used for AIC to the BHHJ divergence developed a new criterion, the

	Divergence Information Criterion (DIC) which for a set of observations 1 x ,...,x is given by n
	/ 2 (2 ) (1 ) ( a a θ DIC p nQ ) π -	2 / 2 p +	p

  It is not difficult to see the good performance of the MDIC criterion for all sample sizes. The rate of success of MDIC is almost 98% for the small sample with 50 observations, while the other criteria have a rate of success between 76.9% observations, the only criterion for which the percent of correct model selection increases significantly is the Hannan and Quinn criterion (HQ). A smaller increase is observed for the SIC criterion. For large sample sizes SIC and MDIC come first with a rate of success of approximately 98% and HQ close second with a success rate almost equal to 92%. At the same time, as the sample size increases, the difference between the MDIC and the remaining Information Criteria decreases, although it is still in favor of MDIC. The good performance of MDIC is evident from the high percent of selecting the correct AR(1) model in repeated samples.

	F o r
	P
	e
	e r
	R
	e
	v i e
	w
	O n l
	y

  Table4shows all criteria for small sample sizes (n=50) have a high tendency of selecting the simplest model (AR(1)) which is due to the fact that the coefficient relatively close to zero (0.35). As the sample size increases the rate of success improves. MDIC is the best criterion in conjunction with HQ, for small sample sizes, and with SIC, for large sample sizes.

	of lag2 is
	F o r
	P
	e
	e r
	R
	e
	v i e
	w
	O n l
	y
	13

  15selection is significantly close to 1 even for very small values of n but stays below 1 even for very large sample sizes. This behaviour may not be the best one from the asymptotic (theoretical) point of view but it is ideal from the finite one due to desirable practical implications. It that sense, MDIC may not possess the characteristics of a consistent criterion but the fact that it is superior to all other criteria for finite sample sizes, makes it an excellent selection criterion for practical purposes. Furthermore, another attractive feature of MDIC is the one related to overfitting, where the criterion never selects very large models. Indeed, if k * is the true order of the model and k the order selected by MDIC, we observe that

			F o r F o r
	for	50 n ≥ ,	P 2] 0.005 P k k L L * [ , = + ≥ ≤ P	. In most cases, the probability drops to 0 even for L=3.
			e e
			e r e r
			R R
				e e
				v i e v i e
				w w
				O n l O n l
				y y
				At the same
	time, the rate of success appears to plateau as the sample size increases. As it is known,
	consistency is an attractive asymptotic property. In particular, as n tends to infinity we have
	that P[correct selection] 1 → . For MDIC, our results indicate that the probability of correct

Page 43 of 47 URL

: http://mc.manuscriptcentral.com/lssp E-mail: comstat@univmail.cis.mcmaster.

ca Communications in Statistics -Simulation and Computation

	Table nr 5: Model 5
	Lag1	Lag2	Lag3	Lag4	Lag5	Lag6	Lag7	Lag8
	50 obs
	AIC
	F o r
		P
		e
			e r
				R
				e
				v i e
					w
						O n l
							y
								20

Page 44 of 47 URL

: http://mc.manuscriptcentral.com/lssp E-mail: comstat@univmail.cis.mcmaster.

ca Communications in Statistics -Simulation and Computation

	Table nr 6: Model 6
	Lag1	Lag2	Lag3	Lag4	Lag5	Lag6	Lag7	Lag8
	50 obs
	AIC
	F o r
		P
		e
			e r
				R
				e
				v i e
					w
						O n l
							y
								21

Page 45 of 47 URL

: http://mc.manuscriptcentral.com/lssp E-mail: comstat@univmail.cis.mcmaster.

ca Communications in Statistics -Simulation and Computation

	F o r F o r
	P P
	e e
	e r e r
	R R
	e e
	v i e v i e
	w w
	O n l O n l
	y y

Communications in Statistics -Simulation and Computation

URL: http://mc.manuscriptcentral.com/lssp E-mail: comstat@univmail.cis.mcmaster.caCommunications in Statistics -Simulation and Computation

Acknowledgements:

The authors would like to express their appreciation to the Editor and an anonymous referee for valuable comments and suggestions that greatly improved both the quality and the presentation of the manuscript. We especially thank the referee whose careful and thorough reading helped in identifying and correcting a typo in our GAUSS program that was oversighted.

Acknowledgements:

The authors would like to express their appreciation to the Editor and an anonymous referee for valuable comments and suggestions that greatly improved both the quality and the presentation of the manuscript. We especially thank the referee whose careful and thorough reading helped in identifying and correcting a typo in our GAUSS program that was oversighted.

0,018 0,748 0,103 0,05 0,026 0,02 0,015 0,02 SIC 0,047 0,865 0,063 0,014 0,006 0,003 0,001 0,001 HQ 0,034 0,805 0,08 0,034 0,02 0,014 0,006 0,007 FPE 0,018 0,751 0,103 0,047 0,026 0,02 0,015 0,02 0.25 0,037 0,894 0,06 0,008 0,001 MDIC max 0,095 0,902 0,003 0 0 0 0 0 75 obs AIC 0,001 0,777 0,104 0,051 0,028 0,017 0,009 0,013 SIC 0,005 0,947 0,039 0,008 0,001 0 0 0 HQ 0,002 0,878 0,076 0,024 0,01 0,006 0,001 0,003 FPE 0,001 0,777 0,104 0,051 0,028 0,017 0,009 0,013 0.25 0,003 0,948 0,041 0,007 0,001 MDIC max 0,013 0,985 0,002 0 0 0 0 0 100 obs AIC 0,001 0,775 0,12 0,042 0,031 0,011 0,009 0,011 SIC 0,001 0,944 0,046 0,004 0,005 0 0 0 HQ 0,001 0,874 0,086 0,019 0,013 0,004 0,003 0 FPE 0,001 0,776 0,12 0,042 0,031 0,011 0,008 0,011 0.25 0,001 0,932 0,063 0,004 MDIC max 0,001 0,993 0,006 0 0 0 0 0 200 obs AIC 0,001 0,814 0,104 0,027 0,032 0,007 0,004 0,011 SIC 0,001 0,974 0,022 0,003 0 0 0 0 HQ 0,001 0,915 0,061 0,013 0,007 0,002 0,001 0 FPE 0,001 0,814 0,104 0,027 0,032 0,007 0,004 0,011 0.25 0,001 0,945 0,051 0,003 MDIC max 0,001 0,996 0,003 0 0 0 0 0 500 obs AIC 0,001 0,793 0,114 0,042 0,024 0,008 0,011 0,007 SIC 0,001 0,987 0,012 0 0 0 0 0 HQ 0,001 0,915 0,066 0,012 0,005 0,001 0 0 FPE 0,001 0,793 0,114 0,042 0,024 0,008 0,011 0,007 0.25 0,001 0,939 0,057 0,003 MDIC max 0,001 0,996 0,003 0 0 0 0 0

The shading indicates best performance Information Criterion.