Cihan Ay

Johanna Haselböck

Clarissa Laczkovics

Silvia Koder

Prof. Dr Ingrid Pabinger
email: ingrid.pabinger@meduniwien.ac.at

Thrombin generation in patients with a bleeding tendency of unknown origin

Keywords: bleeding, thrombin generation, haemostasis, coagulation assay

come

INTRODUCTION

The von Willebrand´s disease (vWD) and haemophila are the most frequent and best characterized hereditary disorders that lead to an increased bleeding tendency. Also other rare hereditary abnormalities, such as platelet function defects and deficiencies in fibrinogen, factors II, V, VII, X, XI or XIII, are associated with a moderate to severe bleeding tendency. In some cases an increased bleeding risk may occur as a result of acquired deficiencies in clotting factors or abnormalities of other components of the haemostatic system. However, in clinical practice there are still a number of patients with a mild to moderate bleeding tendency, in whom no underlying bleeding disorder can be detected despite thorough clinical and laboratory investigation of all known hereditary and acquired haemostatic abnormalities. This is especially true for persons who exhibit a bleeding tendency that is solely associated with invasive procedures, in some cases even requiring transfusions. To allow for a targeted therapy of bleeding complications, it is essential to identify the underlying cause. Therefore, the management of bleeding complications in patients with unexplained bleeding tendency is still difficult to tackle.

Thrombin is the central enzyme in the plasmatic coagulation system, which leads to the conversion of fibrinogen to fibrin, a process finally resulting in clot formation and maintaining sufficient haemostasis [START_REF] Crawley | The central role of thrombin in hemostasis[END_REF]. Thrombin is also one of the most potent activators of platelets, which are the mainstay of primary haemostasis [START_REF] Brass | Thrombin and platelet activation[END_REF]. The accumulation of thrombin at sites of vascular injury provides an important mechanism for the gathering of platelets and the formation of a growing haemostatic plug. Recent advances in laboratory technologies gave rise to the development of new assays to measure the in-vitro thrombin generation, which reflects the overall coagulation potential [START_REF] Kitchen | New developments in laboratory diagnosis and monitoring[END_REF][START_REF] Nair | Tests of global haemostasis and their applications in bleeding disorders[END_REF][START_REF] Baglin | The measurement and application of thrombin generation[END_REF][START_REF] Van Veen | Thrombin generation testing in routine clinical practice: are we there yet?[END_REF].

In contrast to traditional coagulation tests, such as the prothrombin time (PT) and the activated partial thromboplastin time (aPTT), which correspond to artificially created compartments of haemostasis and do not assess the whole coagulation system, the measurement of thrombin generation has been proposed as a very promising approach to globally estimate an individual´s coagulation potential and to predict a hyper-or hypo-coagulable state [START_REF] Nair | Tests of global haemostasis and their applications in bleeding disorders[END_REF][START_REF] Baglin | The measurement and application of thrombin generation[END_REF][START_REF] Van Veen | Thrombin generation testing in routine clinical practice: are we there yet?[END_REF]. For instance, a high thrombin generation potential has been reported to be associated with an increased risk for thrombotic disorders such as venous thromboembolism [START_REF] Van Veen | Thrombin generation testing in routine clinical practice: are we there yet?[END_REF][START_REF] Pabinger | Biomarkers and venous thromboembolism[END_REF], while a decreased thrombin generation potential was found in patients at risk of bleeding, such as patients with haemophilia A and B, the vWD and other bleeding disorders [START_REF] Dargaud | Evaluation of thrombin generating capacity in plasma from patients with haemophilia A and B[END_REF][START_REF] Rugeri | Thrombin-generating capacity in patients with von Willebrand's disease[END_REF][START_REF] Rugeri | Thrombin generation in patients with factor XI deficiency and clinical bleeding risk[END_REF][START_REF] Dieri | The thrombogram in rare inherited coagulation disorders: its relation to clinical bleeding[END_REF].

Data on the thrombin generation potential in patients with a bleeding tendency of unknown origin have been missing so far. This is why we sought to assess the thrombin generation potential in a casecontrol study of patients with a mild to moderate bleeding tendency of unknown origin compared to age-and sex-matched healthy controls.

METHODS

Study population

This case-control study was conducted between 2004 and 2007 at the outpatient clinic of the Department of Medicine I, Clinical Division of Haematology and Haemostaseology of the Medical University of Vienna, Austria.

Patients with a history of bleeding symptoms of unknown origin who had been referred to our outpatient clinic within a period of five years prior to the beginning of the study (years 2000-2004) were identified and contacted personally, by telephone or by postal mail for participation in this study. All patients in whom an inherited or acquired haemostatic disorder had been established were excluded. Specific exclusion criteria for the study were: haemophilia A and B, vWD, coagulation factor deficiencies and platelet disorders, such as Glanzmann's thromboasthenia or Bernard-Soulier syndrome, or acquired haemostatic deficiencies and defects. Also patients on anti-coagulants, anti-platelet and anti-inflammatory drugs, liver disease (abnormal prothrombin time), renal impairment/insufficiency, thrombocytopenia with a platelet count <100.000/µl, gastro-intestinal (GI) bleeding with a pathologic finding in the GI-tract and cerebral bleeding due to aneurysm were excluded. Further, subjects with overt infection, pregnancy or malignancy were not included in the study.

Eventually, a total of 135 patients with mild to moderate bleeding tendency, in whom all known heritable or acquired bleeding disorders and diseases associated with bleeding risk had been ruled out, were identified to be eligible for the study. The patient population comprised subjects with bleeding conditions associated with trauma, including trauma after surgery or interventions such as dental extraction, a tendency to easy skin bruising, excessive or unprovoked bleeding (e.g. most commonly epistaxis, mucosal and gum bleeding or menorrhagia without a gynaecological/endocrine cause). Of this population 101 patients (74.8%) fulfilling the inclusion and exclusion criteria were recruited. At study inclusion, patients underwent a structured interview on their medical and bleeding history.

Hundred-and-two age-and sex-matched healthy individuals without antithrombotic therapy and without a medical history of a bleeding disorder (inherited and acquired bleeding disorders), without a bleeding tendency and without a history of cardiovascular or thrombotic diseases, who were unrelated to the patients, served as controls. The control group was matched to the patient group by sex and age in a whole-group-and not a pair-wise-manner. Individuals in the control group were unrelated to each other. They were from the same geographic region and ethnic background as the patients.

The study protocol was approved by the Ethics Committee of the Medical University of Vienna in accordance with the Declaration of Helsinki, and written informed consent was obtained from all study participants prior to any study-related procedures.

Evaluation of bleeding tendency

To assess the bleeding tendency, a standardized bleeding questionnaire was used, which was already published previously [START_REF] Rodeghiero | The discriminant power of bleeding history for the diagnosis of type 1 von Willebrand disease: an international, multicenter study[END_REF]. Each study participant had to answer this structured questionnaire about her/his medical history in order to evaluate bleeding tendency. The questionnaire inquired about 10 present and past bleeding symptoms. The results of the questionnaire were summarized in a specific score as previously published [START_REF] Rodeghiero | The discriminant power of bleeding history for the diagnosis of type 1 von Willebrand disease: an international, multicenter study[END_REF]. Specific symptoms included epistaxis, cutaneous symptoms, bleeding from minor wounds, oral cavity bleeding, gastrointestinal bleeding, postpartum haemorrhage, muscle haematomas/haemarthrosis, bleeding after tooth extraction, bleeding after surgery and menorrhagia. The severity of each symptom was subsequently summarized, using a bleeding score system ranging from 0 to 3, as reported in detail previously [START_REF] Rodeghiero | The discriminant power of bleeding history for the diagnosis of type 1 von Willebrand disease: an international, multicenter study[END_REF], and considering for each symptom the most severe occurrence. The scale for the entire bleeding score ranged from 0 (minimum) to 30 (maximum). This structured questionnaire assessing the bleeding tendency was also applied in all controls. All control individuals had a bleeding score of 0.

Blood sampling and laboratory analysis

At study inclusion blood samples were collected by clean and atraumatic venipuncture after overnight fasting using a 21-gauge butterfly needle (0.8 x 19 mm; Greiner Bio-One, Kremsmuenster, Austria) for routine laboratory and clotting analysis. The measurement of factor VIII and factor IX was performed according to routine laboratory methods and as described previously [START_REF] Ay | Determinants of factor VIII plasma levels in carriers of haemophilia A and in control women[END_REF][START_REF] Vormittag | Low-density lipoprotein receptorrelated protein 1 polymorphism 663 C > T affects clotting factor VIII activity and increases the risk of venous thromboembolism[END_REF].

For measurement of thrombin generation venous blood was drawn into a 3.8% sodium citrate Vacuette® tube (Greiner Bio-One Kremsmuenster, Austria; 9 parts of whole blood and 1 part of sodium citrate stock solution at 0.129 mmol/L). Samples were centrifuged to obtain platelet-poor plasma. Aliquots were coded for blind analysis, snap-frozen and stored at -80 °C until analyses were performed in series.

Thrombin generation was measured with a commercially available assay kit (Technothrombin TGA kit, Techonoclone, Vienna, Austria) on a fully automated, computer-controlled microplate reader (BioTek, FLx800) with a specially adapted software (Technothrombin TGA, Vienna, Austria), using the fluorogenic substrate Z-Gly-Gly-Arg-AMC (Bachem, Bubendorf, Switzerland) according to the manufacturer´s instructions and as previously reported [START_REF] Ay | Thrombin generation in morbid obesity: significant reduction after weight loss[END_REF]. The concentration of generated thrombin was registered over time, resulting in a thrombin generation curve. From this curve various parameters could be read out describing thrombin activity, such as the lag phase (time until thrombin burst), peak amount of thrombin generation, time to peak thrombin generation, velocity of thrombin generation, or the total amount of thrombin generated (area under the thrombin generation curve, AUC).

Statistical analysis

Statistical analyses were performed using the Statistical Package for Social Sciences (SPSS version 15.0, SPSS, Chicago, Illinois, USA). Median and interquartile range (25 th to 75 th percentile) of continuous variables are shown. We performed Mann-Whitney U tests to detect differences in metric variables and in the thrombin generation parameters between patients and healthy controls. Comparison of categorical parameters among the groups was done by the chi-square test.

Correlations between metric variables were calculated by the bivariate Spearmantest. A two-tailed p-value of less than 0.05 was considered to indicate statistical significance.

RESULTS

Patient characteristics

Table 1 summarizes demographic, clinical and laboratory characteristics of our patients and the age-and sex-matched healthy controls. No statistically significant differences were observed between the two groups with respect to BMI, aPTT, PT, FVIII activity, FIX activity, fibrinogen, and platelet count. No statistically significant difference was present between the patient and the control group with respect to blood group 0. The median bleeding score in the patient population was 5.0 [range:

1-13].

Distribution of parameters of thrombin generation in patients and controls

Statistic analyses showed no significant difference in the distribution of parameters of thrombin generation between patients and controls (Figure 1). The median [25 th -75 th percentile] lag phase in patients and controls was 14.4 [11.1-18

Correlation of thrombin generation with clinical and laboratory parameters

We evaluated the correlation coefficients between key variables of thrombin generation in the patient group and compared them with the corresponding coefficients in the control group. As the results were broadly similar, the correlation coefficients for the entire study population are presented in Table 2. No relevant correlations were observed between age, BMI, PT, FIX activity or fibrinogen and the lag phase, peak thrombin and AUC. A weak but statistically significant correlation of the lag phase, peak thrombin and AUC was present with aPTT and FVIII activity.

DISCUSSION

Management of a patient with bleeding can be quite challenging in clinical practice, but accurate diagnosis of the cause of bleeding is crucial for devising appropriate therapeutic strategies. Thorough clinical assessment remains the cornerstone in the diagnostic algorithm for bleeding disorders, including medical history, family history, use of medications and physical examination, followed by global tests of haemostasis to screen patients clinically suspected of having a bleeding disorder and to identify the part of the haemostatic system that is aberrant for further diagnostic work-up and for getting deeper insight into the underlying pathogenetic mechanism [START_REF] Greaves | Approach to the diagnosis and management of mild bleeding disorders[END_REF]. Recently, thrombin generation tests have been proposed to assess global haemostasis and its disorders [START_REF] Nair | Tests of global haemostasis and their applications in bleeding disorders[END_REF]. Moreover, thrombin is a key factor in haemostasis, and the measurement of thrombin generation has been applied for investigating the risk of bleeding in patients with a bleeding disorder [START_REF] Van Veen | Thrombin generation testing in routine clinical practice: are we there yet?[END_REF][START_REF] Dargaud | Evaluation of thrombin generating capacity in plasma from patients with haemophilia A and B[END_REF][START_REF] Rugeri | Thrombin-generating capacity in patients with von Willebrand's disease[END_REF][START_REF] Rugeri | Thrombin generation in patients with factor XI deficiency and clinical bleeding risk[END_REF][START_REF] Dieri | The thrombogram in rare inherited coagulation disorders: its relation to clinical bleeding[END_REF].

In the present case-control study we therefore assessed the in-vitro thrombin generation potential in patients with a mild to moderate bleeding tendency in whom no underlying cause could be identified despite comprehensive routine clinical and laboratory investigations, comparing it to age-and sex-matched control individuals without a bleeding or thrombotic disease. We found that parameters measured by the thrombin generation assay in patients did not differ from those measured in controls.

In support of this finding, also traditional coagulation tests exploring the coagulation system, such as the aPTT and the PT, were equally distributed in both groups.

Furthermore, median levels of the FVIII and FIX activity, fibrinogen and platelet count were comparable between patients and the control population.

From the clinical point of view there is a great interest in having tests that assess overall haemostasis and the coagulation potential for screening, diagnosis of a bleeding disorder and evaluation of the individual clinical phenotype of patients with a bleeding tendency. It was reported that parameters of the thrombin generation assay correlate with bleeding tendency in haemophila or rare inherited bleeding disorders [START_REF] Dargaud | Evaluation of thrombin generating capacity in plasma from patients with haemophilia A and B[END_REF][START_REF] Dieri | The thrombogram in rare inherited coagulation disorders: its relation to clinical bleeding[END_REF], however, the correlation with severity and the clinical phenotype of the bleeding disorder is still relatively poor. Previous studies have shown a marked heterogeneity in thrombin generation at similar FVIII levels (reviewed in [START_REF] Van Veen | Thrombin generation testing in routine clinical practice: are we there yet?[END_REF]) and test results seem to be more sensitive to very low levels of coagulation factors [START_REF] Dieri | The thrombogram in rare inherited coagulation disorders: its relation to clinical bleeding[END_REF]. In this context, it has to be pointed out that in our study only patients with a mild to moderate bleeding tendency were included and the presence of severe haemostatic deficiencies was unlikely. Interestingly, in some coagulation disorders, such as FXI, FVII or FXIII deficiency, the relationship between the level of factor deficiency and the clinical phenotype and severity of bleeding tendency is less clear [START_REF] Greaves | Approach to the diagnosis and management of mild bleeding disorders[END_REF]. Another aspect of our study is that symptoms suggestive of a mild bleeding tendency are common in the general population. Although such symptoms are often associated with an underlying bleeding disorder, they may not always reflect the presence of a disorder of the haemostatic system. To evaluate the bleeding tendency, all of our study participants had to answer a standardized and previously published bleeding questionnaire [START_REF] Rodeghiero | The discriminant power of bleeding history for the diagnosis of type 1 von Willebrand disease: an international, multicenter study[END_REF]. The median bleeding score in patients was 5, which was indicative of a mild/moderate bleeding tendency. As expected, all controls had a bleeding score of 0.

Until now, several assays have been developed to measure the in-vitro thrombin generation in plasma. In our study, we used a commercially available assay that has been broadly applied in previously published studies (reviewed in [START_REF] Baglin | The measurement and application of thrombin generation[END_REF]).

However, the use of thrombin generation tests as a tool for identifying a hypocoagulable state and for evaluating the bleeding risk and clinical phenotype in patients with coagulation disorders is still not established in routine practice, because these tests are not yet standardized with respect to preanalytical and analytical conditions [START_REF] Kitchen | New developments in laboratory diagnosis and monitoring[END_REF][START_REF] Dargaud | Standardisation of thrombin generation test--which reference plasma for TGT? An international multicentre study[END_REF][START_REF] Dargaud | Effect of standardization and normalization on imprecision of calibrated automated thrombography: an international multicentre study[END_REF][START_REF] Dargaud | Thrombin generation testing in haemophilia comprehensive care centres[END_REF]. Of particular importance in order to obtain reproducible and meaningful results are adequate sampling conditions, manipulation of plasma samples and performing measurements under standardized conditions. We treated samples of patients and controls in exactly the same way, and measurement of thrombin generation was done by the same person in all study participants under the same conditions.

The main limitations of our study are the relatively small number of study participants and the retrospective case-control design. However, we performed additional coagulation analyses and correlated these results to thrombin generation parameters in patients and controls. As expected, thrombin generation parameters showed a weak, but statistically significant correlation with the aPTT and the FVIII activity. FVIII is considered one of the important determinants of thrombin generation [START_REF] Van Veen | Thrombin generation testing in routine clinical practice: are we there yet?[END_REF]. As patients were measured not during the acute phase of the bleeding symptoms pertinent to their bleeding tendency, but a certain time thereafter, we cannot exclude that thrombin generation might have been disturbed during the acute phase of bleeding.

In summary, parameters of the thrombin generation potential did not differ between patients with a bleeding tendency of unknown origin and healthy controls, which suggests that the thrombin generation potential in patients with a bleeding tendency of unknown origin was not disturbed. Similarly to the traditional coagulation tests, the thrombin generation assay did not provide additional information to explain bleeding in these patients. The difference in the median levels of thrombin generation parameters between patients and controls was not statistically significant.

Legend to

Figure 1 :

 1 Box plot analyses of the distribution of thrombin generation parameters (A: lag phase, B: velocity index, C: peak thrombin, D: time to peak, E: AUC) in patients with a bleeding tendency of unknown origin and healthy controls.

  [135.6-242.6] nM Thrombin vs. 175.1 [261.4-611] nM Thrombin (p=0.576), time to peak 23.6 [18.1-28.6] minutes vs. 22.6 [18.6-27.1] minutes (p=0.790) and the AUC 3496 [3069-3880] nM thrombin vs. 3314 [3045-3750] nM thrombin (p=0.673). Also in separate analyses for women and men no difference in thrombin generation was found between patients and controls (data not shown).

	.1]
	minutes vs. 14.1 [12.1-17.1] minutes (p=0.720), velocity index of thrombin generation
	19.7 [13.0-39.0] nM/minute vs. 22.6 [14.5-36.5] nM/minute (p=0.233), peak thrombin
	179.8

ACKNOWLEDGEMENTS

We thank Tanja Altreiter, MA (Clinical Division of Haematology and Haemostaseology, Department of Medicine I, Medical University of Vienna) for proofreading this manuscript. Furthermore, we are thankful to Technoclone (the late Professor Bernd Binder and Mrs. Veronika Binder) for the donation of the Technothrombin TGA assay kits and for their assistance.

Funding

The study was funded by an unrestricted grant from CSL Behring Austria.


F Alberge
email: alberge@lss.supelec.fr

Z Naja
email: naja@lss.supelec.fr

P Duhamel
email: pierre.duhamel@lss.supelec.fr

FROM MAXIMUM LIKELIHOOD TO ITERATIVE DECODING

Keywords: Maximum likelihood decoding, iterative turbodecoding, BICM

Iterative decoding is considered in this paper from an optimization point of view. Starting from the optimal maximum likelihood decoding, a (tractable) approximate criterion is derived. The global maximum of the approximate criterion is analyzed: the maximum likelihood solution can be retrieved from the approximate criterion in some particular cases. The classical equations of turbo-decoders can be obtained as an instance of an hybrid Jacobi/Gauss-Seidel implementation of the iterative maximization for the tractable criterion. The extrinsics are a natural consequence of this implementation. In the simulation part, we show a practical application of these results.

INTRODUCTION

Bit-Interleaved Coded Modulation (BICM) was first suggested by Zehavi in [START_REF] Zehavi | 8-PSK trellis codes for a Rayleigh fading channel[END_REF] to improve the Trellis Coded Modulation performance over Rayleigh-fading channels. In BICM, the diversity order is increased by using bit-interleavers instead of symbol interleavers. This improvement is achieved at the expense of a reduced minimum Euclidean distance leading to a degradation over non-fading Gaussian channels [START_REF] Zehavi | 8-PSK trellis codes for a Rayleigh fading channel[END_REF], [START_REF] Caire | Bit-interleaved coded modulation[END_REF]. This drawback can be overcome by using iterative decoding (BICM-ID) at the receiver. BICM-ID is known to provide excellent performance for both Gaussian and fading channels. The iterative decoding scheme used in BICM-ID is very similar to serially concatenated turbo-decoders. Indeed, the serial turbodecoder makes use of an exchange of information between computationally efficient decoders for each of the component codes. In BICM-ID, the inner decoder is replaced by demapping which is less computationally demanding than a decoding step. Even if this paper focus on iterative decoding for BICM, the results can be applied to the large class of iterative decoders including serial or parallel concatenated turbo-decoders. The turbo-decoder and more generally iterative decoding was not originally introduced as the solution to an optimization problem rendering the analysis of its convergence and stability very difficult. Among the different attempts to provide an analysis of iterative decoding, the EXIT chart analysis and density evolution have permitted to make significant progress [START_REF] Brink | Convergence behavior of iteratively decoded parallel concatenated codes[END_REF] but the results developed within this setting apply only in the case of large block length. Another tool of analysis is the connection of iterative decoding to factor graphs [START_REF] Kschischang | Factor graphs and the sum-product algorithm[END_REF] and belief propagation [START_REF] Pearl | Probabilistic Reasoning in Intelligent Systems: Network of Plausible Inference[END_REF]. Convergence results for belief propagation exists but are limited to the case where the corresponding graph is a tree which does not include turbo-code. A link between iterative decoding and classical optimization algorithms has been made also in [START_REF] Walsh | Turbo decoding as Iterative Constrained Maximum-Likelihood Sequence Detection[END_REF] where the turbo-decoding is interpreted as a nonlinear block Gauss-Seidel iteration for solving a constrained optimization problem. In [START_REF] Richardson | The geometry of turbo-decoding dynamics[END_REF], the turbo-decoding is interpreted in a geometric setting as a dynamical system leading to new but incomplete results. The failure to obtain complete results is mainly due to the inability to efficiently describe extrinsic information passing. The relation between the optimal maximum likelihood decoding and iterative decoding is not yet fully understood. In this paper, we first review the principle of maximum likelihood decoding. An approximate (and tractable) criterion is derived from an equivalent and convenient formulation of the optimal criterion. We prove that, in specific cases, the global maximum of the approximate criterion yields the maximum likelihood optimum. We then consider the iterative maximization and prove that the choice of a particular scheduling leads to the classical updates used in the iterative (turbo) decoding. In the simulation part, these results are applied to the detection of suspicious solutions ie with a possible large number of errors.

MAXIMUM-LIKELIHOOD DECODING

A conventional BICM system [START_REF] Caire | Bit-interleaved coded modulation[END_REF] is built from a serial concatenation of a convolutional encoder, a bit interleaver and an M-ary bitsto-symbol mapping (where M = 2 m ) as shown in Figure 1. The sequence of information bits b of length n b is first encoded by a convolutional encoder to produce the output encoded bit sequence c of length n which is then scrambled by a bit interleaver (as opposed to the channel symbols in the symbol interleaved coded sequence) operating on bit indexes. Let d = π(c) denote the interleaved sequence. Then, m consecutive bits of d are grouped as a symbol. The complex transmitted signal s k , 1 ≤ k ≤ n/m, is then chosen from an M-ary constellation ψ where ψ denotes the mapping scheme. For simplicity, we consider transmission over the AWGN channel. The received signals can be written as:

y k = s k + n k 1 ≤ k ≤ n m (1)
where n k is a complex white Gaussian noise with independent inphase and quadrature components having two-sided power spectral density σ 2 c . The maximum likelihood sequence detection takes the

where p(y | b) is the likelihood function which results from concatenating the encoder with the channel. Since there is a one-to-one correspondence between the binary message b and the interleaved sequence d, eq. ( 2) is equivalent to searching dMLD as:

dMLD = arg max d∈{0,1} n p ch (y | d)Ico(d) (3)
where p ch (y | d) is the probability of receiving y when the sequence transmitted through the channel is the mapping of d and where Ico(d) is the indicator function of the code meaning that

Ico(d) = 1 if c = π -1 (d)
where ES stands for the set of all possible separable PMFs on d.

A PMF p(d) is separable if p(d) = Q i pi(di)
with pi(di) the probability for bit i to be equal to di. The optimal solution pMLD(d) takes the form

pMLD(d) =  1, d = dMLD 0, otherwise (5)
Any other weighting (with the constraint

P d p(d) = 1)
produces a lower likelihood. The formulation in ( 4) is equivalent to the original problem in [START_REF] Brink | Convergence behavior of iteratively decoded parallel concatenated codes[END_REF]. The practical implementation of this optimal optimization problem is dismissed due to the presence of a random bit interleaver and to the (large) numerical value of n. In the next section, we present a sub-optimal criterion derived from (4) in a constructive way and establish some properties.

A SUB-OPTIMAL MAXIMUM LIKELIHOOD DECODING

Assumptions and approximations

We observed that the optimal maximum likelihood decoding is infeasible due to the interleaver and to the computational complexity involved by the computation and storage of the 2 n taps of the PMF. A solution regarding the interleaver is to consider separately the twoblocks (mapping and coding) in a particular sense to be defined later.

The problem of the computational complexity can be handled by working on the bit-marginals rather than on the PMF of the whole sequence. For that purpose we split the variable p(d) into the product of the two separable PMFs (l(d) and q(d)) and introduce the computation of the bit-marginals into the optimal criterion as

" lMLD(d), qMLD(d) « = arg max l,q∈Es X d k X d:d k Ico(d)p ch (y | d)l(d)q(d) (6)
The double sum above is exactly the same as the some over all the words d. The global maximum is again obtained for the optimal choice of the weights l(d)q(d). Since l(d) and q(d) are PMF, P d l(d)q(d) ≤ 1, and the optimal weighting strategy is again

lMLD(d) = qMLD(d)  1, d = dMLD 0, otherwise (7)
The formulation in ( 6) is then an equivalent form on the original problem since the two solutions lMLD(d) and qMLD(d) both select the optimal sequence dMLD of the maximum likelihood decoding problem. Let CMLD denote the criterion in [START_REF] Walsh | Turbo decoding as Iterative Constrained Maximum-Likelihood Sequence Detection[END_REF]. The direct maximization of CMLD is still untractable. We need to separate the coding part from the mapping and channel part, this can done by replacing the bit-marginals of the product of two PMFs by the product of the bit-marginals of the two PMFs taken separately. This is of course an approximation leading to the new criterion Ck defined as

Ck = " X d k " X d:d k Ico(d)q(d) «" X d:d k p ch (y | d)l(d) «« (8)
This approximation deserves some comments. First, the bitmarginals in Ck are now computable in practice. For example

P d:d k Ico(d)q(d), 1 ≤ k ≤ n, d k ∈ {0, 1}
is exactly the output given by a BCJR [START_REF] Bahl | Optimal Decoding of Linear Codes for Minimizing Symbol Error Rate[END_REF]. Next, the criterion Ck is dependant of k: the quantities involved in the criterion are not the same for two different values of k (whereas CMLD is independent of k). This suggests that criterion Ck should be used for the maximization over the k th bit-marginal. The maximization of the unique criterion CMLD has been turned into a distributed optimization of the n criteria Ck . Last, the criteria CMLD and Ck , 1 ≤ k ≤ n, are the same (meaning there is no approximation) if the two PMFs involved Ico(d)q(d) and p ch (y | d)l(d)) are separable. We can notice that for l(d) = lMLD(d) and q(d) = qMLD(d) (defined in [START_REF] Richardson | The geometry of turbo-decoding dynamics[END_REF]) the two PMFs are indeed separable. This is also true for all the class of "Kronecker" PMFs in which the global optimum is always lying. In the next subsection, we focus on the maximization of the suboptimal criteria Ck and derive some interesting properties.

Sub-optimal criterion and global maximum

We prove in the two propositions below that in some special cases, the criteria Ck yields the same global maxima as the optimal criterion CMLD.

Proposition 1

The maximum of any criterion Ck , 1 ≤ k ≤ n is obtained for q = q and l = l such that

l(d ′ )q(d) =  1, (d, d ′ ) = ( d, d′ ) 0, otherwise (9)
where

( d, d′ ) = arg max d,d ′ ∈{0,1} n p ch (y | d ′ )Ico(d).
Proof: Since q and l are in Es, Ck reads

Ck = X d k " X d:d k X d ′ :d k Ico(d)p(y | d ′ ) Y j qj(dj )lj(d ′ j ) « ( 10
)
and P

d k P d:d k P d ′ :d k Q j qj (dj)lj(d ′ j ) = P d k l k (d ′ k )q k (d k ).
In other words,

1 P d k l k (d ′ k )q k (d k ) X d k X d:d k X d ′ :d k Y j qj (dj)lj(d ′ j ) = 1 (11)
Thus

1 P d k l k (d ′ k )q k (d k ) C k is maximum for q(d ′ ) l(d) = 1 for (d, d ′ ) = ( d, d′
) and q(d) l(d ′ ) = 0 for the other pairs involved in (10). All the pairs (d, d ′ ) have the same value for the bit k, this is true in particular for ( d, d′ ). Then

P d k l k (d ′ k )q k (d k )
and consequently Ck are also maximized by q(d) l(d ′ ).

Let dch = arg max d∈{0,1} n p(y | d). Let suppose that dch is a codeword. This is likely to be so at high SNR. Then ( dch , dch ) = arg max d,d ′ ∈{0,1} n p ch (y | d ′ )Ico(d) and also dch = dMLD = arg max d∈{0,1} n p ch (y | d)Ico(d). Then each criterion Ck has a global maximum at ( lMLD(d), qMLD(d)). We can also remark that the others global maximizers of the individual criterion Ck are not global maximizers of all the others criteria Ci, with 1 ≤ i ≤ n and i = k. As a conclusion, if the channel probability p ch (y | d) reaches its maximum for a particular value of d corresponding to a codeword then the joint global maximization of criteria C k for 1 ≤ k ≤ n yields the same solution (given in [START_REF] Richardson | The geometry of turbo-decoding dynamics[END_REF]) than the maximum likelihood decoding. We can now define a new criterion C = P n k=1 Ck which appears to be a relevant approximation of the optimal criterion C.

Proposition 2 Let suppose that C has a global maximum at

( l C , q C ). If ( l C , q C ) is such that l C q C (d) = 1 at d = d0 and 0 otherwise then d0 = dMLD = arg max d∈{0,1} n p ch (y | d)Ico(d).
Proof: C has a global maximum at ( l C , q C ) then C( l C , q C ) ≥ C(l, q) for any PMF l, q. From the definition of C we have C(l, q) ≥ nCMLD(l, q). Moreover C( l C , q C ) = nCMLD( l C , q C ). Thus CMLD( l C , q C ) ≥ CMLD(l, q) for any PMF l, q.

If we manage to find the global maximum of the sub-optimal criterion C and if the corresponding argument turns out to be a Kronecker PMF then this is also the argument of a global maximum of the optimal criterion CMLD associated with the maximum likelihood decoding. The practical usefulness of criterion C will be emphasized in section 4. In the next subsection, we build an iterative strategy of maximization.

Iterative maximization

We observed in section 3.2 that the sub-optimal criterion Ck was derived from CMLD when dealing with the k th bit-marginal. We propose here to consider a distributed maximization strategy where l k (d k ) and q k (d k ) are chosen in order to maximize Ck as

" lk , qk « = arg max l k ,q k ∈F Ck ( 12
)
where F is the set of all possible PMFs on d k . The solution of ( 12) is given by

lk (d k )q k (d k ) = 1 if f d k (q, Ico)f d k (l, p ch (y | d)) > f d k (q, Ico)f d k (l, p ch (y | d)) lk (d k )q k (d k ) = 0 otherwise (13) where f d k (q, Ico) = P d:d k Ico(d) Q j =k qj(dj ), f d k (l, p ch (y | d)) = P d:d k p ch (y | d) Q j =k lj (dj) and d k = 1 -d k .
An iterative process propagating hard estimates (0 or 1) is likely to get stuck in a local minima. A classical solution is to propagate instead soft-estimates (in [0; 1]) and take hard decisions at the end of the iterative process. For the maximization problem in (12), possible soft estimates are :

lk (d k )q k (d k ) ∝ f d k (q, Ico)f d k (l, p ch (y | d)) lk (d k )q k (d k ) ∝ f d k (q, Ico)f d k (l, p ch (y | d)) (14)
Equation ( 14) characterizes the product lk qk . The individual values of lk and qk depend on the scheduling of the successive updates. Since we have no prior information, a natural choice for the initialization is l

(0) k (d k ) = q (0) k (d k ) = 1 2 for 1 ≤ k ≤ n and d k ∈ {0; 1}. Let consider first the update of variables q k . Follow- ing a Jacobi implementation, q (it) k is obtained from (14) by setting l k (d k ) = l (it-1) k (d k ) for 1 ≤ k ≤ n and qi(di) = q (it-1) i (di) for i = k.
In particular, the update at the first iteration is:

q (1) k (d k ) ∝ X d:d k Ico(d) X d:d k p ch (y | d) (15)
for d k ∈ {0; 1} and for all k ∈ {1, ..., n}. The update of l k also comes from ( 14) by setting li(di) = l

(it-1) i (di) for i = k and q k (d k ) = q (it) k (d k ) for 1 ≤ k ≤ n since q (it)
k (d k ) has just been computed and is available. This is an hybrid Jacobi/Gauss-Seidel implementation. At the first iteration, the update for l k is:

l (1) k (d k ) ∝ P d:d k Ico(d) Q j =k q
(1)

j (dj) P d:d k Ico(d) (16)
The generalization to iteration (it) reads:

q (it) k (d k ) ∝ P d:d k Ico(d) P d:d k p ch (y | d) Q j =k l (it-1) j (dj) l (it) k (d k ) ∝ P d:d k Ico(d) Q j =k q (it) j (d j ) P d:d k Ico(d) (17)
for d k ∈ {0; 1} and for all k ∈ {1, ..., n}. In general, for convolutive codes,

P d:d k Ico = P d:d k Ico.
The iterative updates are then obtained through:

q (it) k (d k ) ∝ P d:d k p ch (y | d) Q j =k l (it-1) j (dj ) l (it) k (d k ) ∝ P d:d k Ico(d) Q j =k q (it) j (dj) (18)
This is exactly the equations that are used in the iterative decoding of BICM where l k (d k ), q k (d k ) are usually called extrinsics and l k (d k )q k (d k ) (after normalization) is the APP (A Posteriori Probability) [START_REF] Alberge | Iterative decoding as Dykstra's algorithm with alternate I-projection and reverse I-projection[END_REF]. The extrinsics are a direct consequence of the maximization of a well-defined criterion and of the choice of a particular scheduling in the computation of the successive updates. The algorithm stops in general when an agreement is reached between the APP computed at the demapper and the APP computed at the decoder. This corresponds with our notations to:

q (it) k (d k )l (it-1) k (d k ) ∝ q (it) k (d k )l (it) k (d k ) (19)
for d k ∈ {0; 1} and for all k ∈ {1, ..., n}. This means that the tentative maximization (via soft estimates) of Ck for 1 ≤ k ≤ n with respect either to l k (d k ) or to q k (d k ) produces exactly the same result regarding the product (APP) l k (d k )q k (d k ). The derivation in this paper are mainly based on the block structure (code, mapping) of the BICM and not on the specificity of each block. The line of arguments followed in this paper and the conclusions apply to a wide range of problems including iterative turbo-decoding. For instance, the serial turbo-decoder is obtained from BICM by replacing the PMF p ch (y | d) with p ch (y | d)Ico1(d) where Ico1(d) is the indicator function of the inner coder involved at the transmitter. Since, no assumptions has been made on p ch (y | d) within this paper, the whole conclusions apply to serial turbo-decoders as well as to any decoder with a structure similar to BICM.

SIMULATIONS

The criterion C = P n k=1 Ck is an approximation of the MLD criterion C. The solution for the approximate criterion C is expected to be close to the MLD solution and hence to achieve low bit error rate (BER). Iterative turbo-decoding is known to provide excellent performance in terms of BER. We can however observe at low SNR a degradation in the performance mainly due to inaccurate solutions of the iterative decoder for some isolated frames. In the following, we use C has a detector to separate the acceptable solutions (few errors expected) from the suspicious ones (many more errors expected). We used a classical transmitter BICM scheme with a (5, 7) convolutional code of rate 1/2. The number of information bits is n b = 400 (a frame). The code bits were passed through a random interleaver and modulated to 16-QAM symbols. The signal to noise ratio is defined as E b N 0 , where E b denotes the energy per information bit and N0 is the noise variance. We consider an environment with varying E b N 0 in the range {4dB, 5dB, ..., 11dB, 12dB}. The values are chosen randomly from a uniform distribution. The iterative decoding is performed using eq. ( 18). For each frame, the iterative process is run until an agreement is reached for all the bits (eq. 19) or the maximum number of iterations is reached. The solutions are qualified of acceptable if P n k=1 log( Ck ) is greater (at the end of the iterative process) than a certain threshold (to be chosen) and are qualified of suspicious if P n k=1 log( Ck ) is under the threshold. We compute BERa resp. BERs the bit error rates of the acceptable solutions resp. the BER of suspicious solutions. We expect BERs to be many more large than BERa. We also evaluate the proportion of suspicious solutions identified (ps) and the false alarm proportion (p s,f alse ) which counts the solutions wrongly qualified of suspicious. We consider that the solutions with less than 6 errors (among the 400 bits) should not have been qualified of suspicious. The algorithm stops when the total number of errors among the acceptable sequences is greater than 200. The results are reported in table The validity of C as a relevant approximation of C is confirmed by this simulation. The BER is strongly correlated with the value of C reached at the end of the iterative process. If the value is close to the global maximum, the solution is also close (or equal) to the optimal MLD solution. At the opposite, a low value of C is often associated with a large number of errors. This is also observable in fig.

(2) where the distribution of the errors (histograms) are plotted for the acceptable solutions and compared to the histogram obtained for the suspicious solutions in the case where the threshold is -20. We can conclude from this experiment that the suspicious solutions are mainly due to local minima ( C is often very far from the threshold) rather than to an inaccuracy of C as an approximate criterion for C. For a practical point of view, the simulation above illustrates how we can guarantee a given performance (in terms of BER) independently of the SNR (at the cost of a rejection/re-emission of the suspicious frames).

CONCLUSION

In this paper, we derived the (turbo) iterative decoding from maximum likelihood decoding. The approximation needed to obtain a tractable solution are clearly stated and some proofs have been established concerning the global maximum of the approximate criterion. It has also be shown that the turbo-decoding follows from an hybrid Jacobian/Gauss-Seidel implementation of the maximization process. The propagation of extrinsics is naturally introduced and is a direct consequence of the scheduling. In the simulation part, a possible application of these results has been presented. In this paper, we proved that iterative turbo-decoding can be interpreted as a distributed optimization strategy reminiscent of mixed cooperative/non cooperative games. The theoretical tools developed within the game theory framework constitute a new open perspective for completing the analysis of turbo-like decoding.

Fig. 1 .

 1 Fig. 1. BICM transmission scheme

Fig. 2 .

 2 Fig. 2. Histogram representation of the error distribution for : (left) acceptable solutions -(right) suspicious solutions

Table 1 .

 1 1. Evaluation of Acceptable/Suspicious solutions

	Threshold	-20	-10	-5
	BERa	8, 78.10 -4	4, 68.10 -4	2, 08.10 -4
	BERs	0, 205	0, 13	9, 28.10 -2
	ps %	6, 4%	10, 8%	14, 8%
	p f alse,s %	2, 5%	36, 4%	53, 83%


Eugenio Della

Vecchia -Silvia Di

Marco -Alain Jean-Marie

Eugenio Della Vecchia

Silvia Di

Alain Jean-Marie

Illustrated review of convergence conditions of the value iteration algorithm and the rolling horizon procedure for average-cost MDPs

Keywords: Réseaux, systèmes et services, calcul Markov decision problems, Value iteration, Heuristic methods, Rolling horizon Processus de décision Markovien, itération de valeurs, méthodes heuristiques, horizon roulant

This paper is concerned with the links between the Value Iteration algorithm and the Rolling Horizon procedure, for solving problems of stochastic optimal control under the long-run average criterion, in Markov Decision Processes with finite state and action spaces. We review conditions of the literature which imply the geometric convergence of Value Iteration to the optimal value. Aperiodicity is an essential prerequisite for convergence. We prove that the convergence of Value Iteration generally implies that of Rolling Horizon. We also present a modified Rolling Horizon procedure that can be applied to models without analyzing periodicity, and discuss the impact of this transformation on convergence. We illustrate with numerous examples the different convergence results.

Introduction 1.Statement of the problem: precision of the rolling horizon procedure

Consider a random dynamical system, observed at discrete times. At each time t ∈ N, the state s t is observed and an action a t is chosen, resulting in an instantaneous gain r t (s t , a t ). From here on, we work with time-homogeneous gains: r t = r, independent on the time. The actions a t chosen at each time t in the respective state s t determine a policy π whose performance is evaluated through a long-run average criterion. More precisely, let

g π (s) := lim inf n→∞ 1 n E π s n-1 t=0 r(s t , a t ). (1)
The objective of the controller is to find (when it exists) the policy that solves, given the current state s: π * (s) = arg max π g π (s) .

However, for a wide class of stochastic control problems in discrete time and infinite horizon, obtaining an optimal policy explicitly is a difficult task. This is why practitioners often use instead a heuristic method called the Rolling Horizon procedure (also, Receding Horizon, Moving Horizon or Model Predictive Control), which works as follows. To the infinite-horizon control problem is associated a finite-horizon problem (FHP): for a given integer n (the horizon length) and a state s, find:

max π E π n-1 t=0
r(s t , a t )|s 0 = s .

(

) 2
Solving this problem results in a sequence of decision rules:

π * n = (d n , d n-1 , . . . , d 2 , d 1 ) (3)
where d 1 (s n-1 ) is the best action to be applied at time t = n -1 when only one step remains to reach the horizon, d 2 is the best decision rule to be applied when two steps remain to get the horizon, at time t = n -2, and so on. In particular, d n (s 0 ) is the best decision rule to be applied to the initial state s 0 .

The Rolling Horizon method (abbreviated as RH from here on), prescribes to repeatedly solve a FHP, taking the current state as initial state. Then, the procedure offers a control sequence where only the first one of them will be applied. Specifically, the procedure to construct a rolling horizon policy is the following one. Fix some integer n.

1. At time t, and for the current state x t , find the value of d n (x t ) in the control problem FHP.

2. Apply a t = d n (x t ).

3. Observe the achieved state at time t + 1: x t+1 .

4. Set t := t + 1 and x t := x t+1 and go to step 1.

RR n°7710

The RH procedure does not specify how to compute the value d n (x t ). Its efficiency is based on the idea that computing the value d n (x t ) alone is usually much easier than solving entirely the FHP, which involves computing the n decision rules in [START_REF] Derman | Finite State Markovian Decision Processes[END_REF]. On the other hand, the performance of the resulting policy is not the optimal one, although the intuition is that when n is "large enough", the performance should be close to the optimal. The practical issue is then to choose n so as to obtain a proper compromise between precision and the computational effort needed to obtain d n (x t ). We address this issue through two formal qualitative and quantitative questions. Let u n (s) be the performance achieved by the RH procedure with horizon length n, starting in state s: Q1 Under which conditions on the problem is it true that lim n→∞ u n (s) = g π * (s)? Q2 Given a state s and ǫ > 0, is it possible to compute n such that |u n (s)g π * (s)| < ǫ?

In this paper, we look at these questions in the context of Markov Decision Processes, through the link it has with the Value Iteration (VI) algorithm and the RH procedure. We focus on two objectives:

• make a review of the results about convergence of the VI algorithm in the literature, including the multichain model,

• analyze the effects of those properties on the convergence of the RH procedure and, in this way, make more practical and wide the use of this method.

We finally propose a modification of the RH procedure that makes convergence easier, and discuss its efficiency.

The paper is organized as follows. We complete this introduction with a brief literature review. In Section 2 and 3, we recapitulate the relationship between RH and the Value Iteration algorithm concerning their convergence concepts. Then in Section 4 we propose and evaluate a Modified Rolling Horizon procedure. Finally, in Section 5, we discuss about stopping rules for both of algorithms and we conclude in Section 6.

Literature review

Markov decision problems have been widely studied during the last sixty years, and the advances and applications have been synthesized in well-known books such as for example [START_REF] Derman | Finite State Markovian Decision Processes[END_REF][START_REF] Ross | Applied Probability Models with Optimization Applications[END_REF][START_REF] Bertsekas | Dynamic Programming: Deterministic and Stochastic Models[END_REF][START_REF] White | Centre de recherche INRIA Sophia Antipolis -Méditerranée 2004[END_REF][START_REF] Puterman | Markov Decision Processes[END_REF][START_REF] Kallenberg | Markov decision processes[END_REF]. The value iteration algorithm is an usual topic in the bibliography, frequently associated to discounted criteria. The analysis of the problem when the performance of the policies is evaluated with the criterion of average rewards over an infinite horizon, in the most general case, presents additional difficulties. This is why this topic has been developed more recently in the literature. For example, it is not present in [START_REF] Derman | Finite State Markovian Decision Processes[END_REF] and few words are devoted to it in [START_REF] Ross | Applied Probability Models with Optimization Applications[END_REF]. In most of the references, convergence of the value iteration algorithm for average rewards criterion is analyzed only for unichain models. Since the distinction between unichain and multichain turns out to be NP-complete to decide (see [START_REF] Kallenberg | Markov decision processes[END_REF]), it should be useful to give convergence results for multichain models which work also for unichain models. We shall consider Markov Decision Processes as described for instance in Puterman [START_REF] Puterman | Markov Decision Processes[END_REF], of whom we adopt the notation. We also refer to the classification of Markov decision problems according to their deterministic stationary policies, i.e. unichain or multichain, as it appears in [START_REF] Kallenberg | Markov decision processes[END_REF][START_REF] Puterman | Markov Decision Processes[END_REF].

The theoretical analysis of RH for MDPs has comparatively received less attention in the literature, where this procedure is often encountered in a heuristic presentation, without precise references to accuracy or convergence. In the discounted case, Puterman [START_REF] Puterman | Markov Decision Processes[END_REF]Theorem 6.3.1,p. 161] proves that both VI and RH converge at the same time (see the definitions of convergence below). Results for the case of average costs include those of Hernández-Lerma and Lasserre, who present in [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF] error bounds for rolling horizon policies in general, stationary and nonstationary, Markov control problems on Borel spaces, with both discounted and average reward criteria. They give a condition (Assumption 5.1 in their work and Condition 5 below in this work) under which the reward of the rolling horizon policy converges geometrically to the optimal reward function, uniformly in the initial state, as the length of the rolling horizon increases. The convergence rate is explicit in their result. Previously, Alden and Smith in [START_REF] Alden | Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes[END_REF] provided an error bound, still for nonstationary MDPs, between a rolling horizon policy and an expected-average optimal policy, considering finite states and finite policies under a Doeblin-like condition (see [START_REF] Meyn | Markov chains and stochastic stability[END_REF]). Guo and Shi, in [START_REF] Guo | Limiting average criteria for non stationary Markov decision processes[END_REF], deal with the limiting average criteria for nonstationary Markov decision processes on Borel state spaces with possibly unbounded rewards . They give conditions under which the existence of both a solution to the optimality equations and the limiting average ε-optimal Markov policies can be derived and also present a rolling horizon algorithm for computing limiting average ε-optimal Markov policies. The proof of the convergence is under a condition similar to those in [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF].

The Value Iteration algorithm for the average reward criterion

In what follows, the state space, S, and the decision set for each s ∈ S, A s are both finite. Also, without losing generality, we consider r(s, a) ≥ 0, ∀ (s, a) ∈ S × A s . We shall focus on stationary policies π = (d) ∞ = (d, d, . . .) where d is a decision rule that maps every state s of S to A s . Every decision rule can be seen as a vector with |S| components. When vectorial notation is possible, for short we write r d for the vector whose components are r(s, d(s)) and P d for the transition matrix where P d (s, s ′ ) = p(s ′ |s, d(s)). Moreover, making an abuse of notation, we use D indifferently for the set of decision rules or the set of stationary policies.

In this finite state space/finite action space setting, it is well-known that there exists an optimal, pure stationary policy, for the infinite-horizon average reward criterion [START_REF] Alden | Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes[END_REF]. Let g * denote the associated optimal gain vector.

Consider now the Value Iteration algorithm:

Value iteration algorithm

1. n = 0, v 0 = 0. 2. Compute v n+1 = max d∈D {r d + P d v n } =: T v n (4)
and some

d n+1 ∈ arg max d {r d + P d v n }. (5)
3. If an adequate stopping rule holds, then go to step 4. Otherwise, set n := n + 1 and go to step 2.

4. Return d n+1 .

When VI stops, for some N , it has computed a sequence of decision rules (d N , . . . , d 1 ) which actually solves the FHP in [START_REF] Bertsekas | Dynamic Programming: Deterministic and Stochastic Models[END_REF]. On the other hand, it has been observed in the literature (e.g. in [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF]) that the RH procedure with horizon n generates decisions precisely according to the stationary policy (d n ) ∞ = (d n , d n , . . .).

Convergence concepts. The usual theoretical and practical challenge for VI is to determine the "adequate stopping rule" of step 3, so that the algorithm does stop at some iteration n such that the policy computed is "good enough".

It is known that v n /n always converges to g * , see [8, Corollary 2.8, p. 49], but this result alone does not help to identify optimal or ǫ-optimal policies. We choose therefore the following, more practical, notion of convergence. The VI algorithm is said to converge if the following limit exists, for some vector h * :1

lim n→∞ v n -ng * = h * . (6)
In addition, the convergence is said to be geometric if there exists N ∈ N, C > 0 and δ < 1 such that, with a suitable norm, ∀n ≥ N ,

||v n -ng * -h * || < Cδ n .
These definitions are motivated by the fact that the (nonstationary but periodic) policy (d N , . . . , d 1 , d N , . . .) has the performance v N /N for the average criterion [START_REF] Alden | Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes[END_REF]. On the other hand, most practitioners are likely to use instead the decision rule d N alone repeatedly (out of simplicity, or the belief that this rule must be the "best" of those computed by VI), thereby implementing effectively a RH procedure with horizon N . The performance obtained by using this stationary policy is u N = g (dN ) ∞ , and in general, u N = v N /N and u N = v N +1v N , nor is there a particular order between these sequences. We illustrate this fact below. As a consequence, any convergence result or stopping rule for VI is not guaranteed to provide a performance bound for RH. This motivates further the need for results concerning specifically RH.

By analogy, the RH procedure is said to converge if

lim n→+∞ g (dn) ∞ = g * .
The convergence is said to be geometric if there exists N ∈ N, C > 0 and δ < 1 such that, with a suitable norm, ∀n ≥ N ,

||g (dn) ∞ -g * || < Cδ n .
This is an abuse of terminology, since the RH procedure itself does not "converge". It merely means that the procedure can be made to perform arbitrarily close to optimal through the choice of a suitable horizon length n.

Example 1.

The example detailed in the Appendix serves to illustrate these issues concerning the convergence of the different sequences involved in the previous discussion.

We have applied to this model the transformation to be described in Section 4 with τ = 0.99. In Figure 1, we show the evolutions of the sequences

v n+1 -v n , v n+1 -v n -g (dn) ∞ , v n /n and v n /n -g (dn) ∞
respectively, evaluated at state 4. Clearly, these sequences do not have a constant sign, and are not monotonously converging.

of v n+1 -v n (top), v n+1 -v n -g (dn) ∞ (top, right), v n /n (bottom, left) and v n /n -g (dn) ∞ (bottom, right), for state 4

Convergence of the VI and RH procedures

The issue of convergence of the VI algorithm has attracted quite some attention in the literature. In Section 3.1, we review some of the conditions that have been proposed for ensuring the convergence of either VI or RH. In Section 3.2, we state the convergence results, including a new one which we prove (Theorem 2). In Section 3.3, we have a look at convergence rates. In Section 3.4, we discuss the relative strength of these conditions.

Convergence conditions

The two following conditions are stated by Schweitzer and Federgruen in [START_REF] Schweitzer | The asymptotic behavior of undiscounted value iteration in Markov decision problems[END_REF].

RR n°7710

Condition 1. There exists a randomized maximal gain policy whose transition probability matrix is aperiodic (but not necessarily unichain) and has R * = {i ∈ S : i is recurrent for some pure maximal gain policy } as its set of recurrent states.

Condition 2. Every optimal (pure) stationary policy gives rise to an aperiodic (but not necessarily unichain) transition matrix.

The following condition, known as weak unichain condition appears in Tijms [17, p. 199] as Assumption 3.3.1.

Condition 3. Every optimal stationary policy has a transition probability matrix unichain and aperiodic

Puterman in [12, p. 370], presents the following one.

Condition 4. Every stationary policy is unichain and gives rise to an aperiodic transition matrix.

The following condition appears in Hernández Lerma and Lasserre [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF] as Assumption 5.1. Condition 1 is the weakest condition under which the convergence of v nng * to h * is guaranteed: it is established in [START_REF] Schweitzer | The asymptotic behavior of undiscounted value iteration in Markov decision problems[END_REF] that this is a necessary and sufficient condition of convergence. Condition 5 for the convergence of RH is related to some ergodicity properties of the chain structure: see Appendix in [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF], or [START_REF] Meyn | Markov chains and stochastic stability[END_REF]Chapter 15]. Other convergence conditions have been proposed in [START_REF] Alden | Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes[END_REF][START_REF] Guo | Limiting average criteria for non stationary Markov decision processes[END_REF]. More precisely, the hypothesis of Alden and Smith in [START_REF] Alden | Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes[END_REF], related to the conditions described above, is similar to a Doeblin condition while in [START_REF] Guo | Limiting average criteria for non stationary Markov decision processes[END_REF], Guo and Shi assume that β := sup{1inf j∈S inf a∈As P n (j|i, a) < 1 : n ≥ 0}. Both assumptions, similarly to Condition 5 (see in Theorem 3 below) imply that the model is unichain.

Convergence results

It is known from the literature that, in general, there is no convergence of the sequence {v n } n , which is unbounded (it actually grows asymptotically linearly, r being positive, since v n /n converges to some g * as mentioned earlier), nor of {v nv n-1 } n . Also, it is proved, see [9, Lemma 5.5, p. 157], that if the sequence {v nng * } n is bounded,

g * = lim n→∞ 1 n v n = lim n→∞ 1 n n k=1 (v k -v k-1 ) , RR n°7710
but this is not enough to compute an ε-optimal policy or an ε-approximation of g * . To have one, the convergence of {v nng * } n is needed, and it is known that it may fail to happen if some of the matrices involved in the MDP are periodic. The analysis of average-cost MDPs often involves the notion of "span" of a function (or vector), defined as: sp(w) := max s∈S (w(s))min s∈S (w(s)). The geometric convergence of the sequence sp(v n+1v n ) to 0 implies the geometric convergence of VI, and the limit of v n+1v n is a vector with zero span, that is, a constant vector.

Theorem 1 (Convergence of VI). The VI algorithm converges geometrically under any of Conditions 1-5.

Proof. Obviously, in view of Remark 1, it is sufficient to prove the result for Condition 1. It is interesting for this review to point out that proofs under specific conditions have been obtained independently, since these may involve different techniques and possibly provide different estimations for the convergence rate. See Section 3.3 below. The convergence under Conditions 1 and 2 is proved in [START_REF] Schweitzer | The asymptotic behavior of undiscounted value iteration in Markov decision problems[END_REF], Theorems 5.1 and 5.5 respectively. The fact that the convergence is geometric is proved by the same authors in [ Finally, the result under Condition 5 is not proved, but commented in [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF]. However, it is not hard to check that Condition 5 is equivalent to Condition a) in [12, Theorem 8.5.3, p. 368] and this theorem, together with Theorems 8.5.1 and 8.5.2, provide the geometric convergence of sp(v n+1v n ) to 0.

The previous theorem means that under the conditions mentioned the sequence {v n -ng * } converges geometrically to h * . Now we ask if under any of these conditions the sequence {g (dn) ∞g * } converges geometrically to zero.

Theorem 2 (Convergence of RH). If the VI algorithm converges geometrically then also does the RH procedure.

Proof. By assumption, ∃N 1 ∈ N, C 1 > 0 and α < 1 such that whenever n ≥ N 1 ,

||v n -ng * -h * || ∞ < C 1 α n . Since v n+1 -v n -g * = (v n+1 -(n + 1)g * -h * ) -(v n -ng * -h * ), with C = 2C 1 we have, for n ≥ N 1 , ||v n+1 -v n -g * || ∞ < Cα n ,
or put differently,

g * -Cα n 1 < v n+1 -v n < g * + Cα n 1 . (7)
Let d n be defined as (see ( 5)):

d n ∈ arg max d {r d + P d v n-1 }.
Denoting with P * dn := lim m→∞ 1 m m k=1 P k dn , the Cesàro limit for P dn , we have g (dn) ∞ = P * dn r dn and g (dn) ∞ = P * dn g (dn) ∞ .

RR n°7710

Moreover, according to [12, Lemma 9.4.3], there exists N 2 such that for all n > N 2 , P dn g * = g * , which implies P * dn g * = g * . Then, since P * dn P dn = P * dn ,

g (dn) ∞ = P * dn r dn = P * dn (r dn + P dn v n -v n ) = P * dn (v n+1 -v n ). (8)
It is clear that for all s ∈ S, g * (s) ≥ g (dn) ∞ (s). Consequently, for n > max{N 1 , N 2 } and any s, (8) and ( 7) imply respectively

P * dn (v n+1 -v n )(s) = g (dn) ∞ (s) ≤ g * (s) (9)
P * dn (v n+1 -v n )(s) > P * dn (g * -Cα n 1)(s) = g * (s) -Cα n . (10)
From ( 9) and ( 10), for n > max{N 1 , N 2 }, it follows that

0 ≤ g * (s) -g (dn) ∞ (s) < Cα n , (11)
which concludes the proof.

Theorems 1 and 2 imply:

Corollary 1. Any of Condition 1 to 5 implies the geometric convergence of the RH procedure.

The reciprocal of Theorem 2 cannot hold: there are MDP models in which RH converges, whereas VI does not. The simplest such example is perhaps that of an uncontrolled, two state, periodic Markov chain (see Example 7 below): since there is only one policy, RH converges, but since the model does not satisfy Condition 1, VI cannot converge.

Convergence rates

Having an estimate of the convergence rate is useful in many practical situations, especially for determining horizon lengths or stopping rules (see Section 5). We discuss this point now.

The question is to find values of N , C and δ such that, for all n ≥ N :

||ng * -v n -h * || < Cδ n (12)
or for the RH procedure,

||g * -g (dn) ∞ || < Cδ n . (13)
As S and D are considered finite in this discussion, taking

δ = 1 -min π1,π2∈D M ρ π1,π2 (M ), (15)
RR n°7710

we have δ < 1 and from [12, Theorem 8.5.2.a., p. 368] T is a M -step contraction operator with coefficient δ and then

0 ≤ g * -g (dnM ) ∞ (s) ≤ sp(g * -v nM ) ≤ C δn , (16)
and C = sp(g *v 0 ). Therefore, [START_REF] Ross | Applied Probability Models with Optimization Applications[END_REF] holds with δ = ( δ) 1/M , at least when n is a multiple of M . In [START_REF] Schweitzer | Geometric convergence of the value iteration in multichain Markov decision problems[END_REF] the authors claim that, for the same value of M , and d ∈ D, min s1,s2∈S j∈S

min P M d (s 1 , j), P M d (s 2 , j) = ρ d (M ) > 0,
and that there is geometric convergence with rate

γ = 1 -min d∈D ρ d (M ). (17)
They do not present a proof but they also refer to [5, Theorem 5]. While it is clear that δ ≥ γ, the fact that γ is also a rate of convergence is not obvious.

In the following example we compute both δ and γ defined in ( 15) and ( 17), in order to illustrate the facts that: these numbers are different, and also that the importance of this result is essentially theoretic. It is clear that the complexity of the task grows exponentially with the number of states and actions.

Example 2.

Let us consider a model with three states S = {s 1 , s 2 , s 3 } and two available action in each state, A si = {a i 1 , a i 2 }, i = 1, 2, 3. The (positive) transitions probabilities are defined as follows:

p(s i+1 |s i , a i 1 ) = p(s i-1 |s i , a i 2 ) = 0.9 and p(s i |s i , a i 1 ) = p(s i |s i , a i 2 ) = 0.1
(where the sum within subindices is modulo 3). Since we have a 3-state model, the value of M is 3. Since there are two actions in each state, there are 8 different stationary policies, and 8 different matrices appearing in the definition of ρ π1,π2 (M ) > 0 in [START_REF] Schweitzer | Iterative solution of the functional equation of undiscounted Markov renewal programming[END_REF].

In the computation of γ, we obtain a value ρ = 0.297 for two policies: (a 1 1 , a 2 1 , a 3 1 ) and (a 1 2 , a 2 2 , a 3 2 ), and a value ρ = 0.487 for the rest. We find then the value γ = 1 -0.297 = 0.703.

In the computation of δ, we obtain a largest value of ρ π1,π2 as 0.244, obtained for instance with π 1 = (a 1 1 , a 2 1 , a 3 2 ) and π 2 = (a 1 1 , a 2 2 , a 3 1 ). We then have δ = 0.756. As established above, δ ≥ γ.

The corresponding convergence rates are respectively δ1/3 ≃ 0.911 and γ 1/3 ≃ 0.889.

In summary, only Condition 5 readily provides bounds on the convergence rate. It is an open question to design algorithms which compute bounds on the convergence rate in an efficient way for generic models. This does not preclude the possibility that bounds be established based on [START_REF] Schweitzer | The asymptotic behavior of undiscounted value iteration in Markov decision problems[END_REF] for specific models.

Discussion and comparison of convergence conditions

Discussion on the practicality of convergence conditions. Conditions 1-4 of Section 3.1 all involve aperiodicity, irreducibility and/or the classification of MDP models. Although determining whether a MDP is irreducible is polynomially solvable, there is no polynomial algorithm to determine whether a MDP is unichain or multichain RR n°7710 (see comments in [9, p. 127]). It is worth mentioning that a simple transformation, discussed below, makes aperiodic any transition matrix of the MDP model without changing the optimization problem. Conditions requiring aperiodicity can therefore be applied without this requirement.

Comparison. Among the five Conditions we have reviewed, Condition 5 is the only one not referring to structural properties of the underlying matrices. We investigate here these structural implications.

The following lemma provides a convenient characterization of cases where Condition 5 does not hold. Remember that, given a probability measure µ, supp(µ) is the smallest set B such that µ(B) = 1. Thus, let us suppose a MDP unichain and periodic for the decision rule d. Denote with γ > 1 the period of P d . Then, there exists B 1 , B 2 , ..., B γ sets as provided in Lemma 2. By construction, for every

Lemma 1. Condition 5 does not hold if and only if there exist

(s, s ′ ) ∈ S × S, a ∈ A s , a ′ ∈ A s ′ , such that supp(p(•|s, a)) ∩ supp(p(•|s ′ , a ′ )) = ∅.
s i ∈ B i , supp(p(•|s i , d(s i ))) ⊂ B i+1
, where it is understood that "γ + 1" means 1. The supports are therefore disjoint for s i and s j if i = j. Therefore, Lemma 1 applies with any

s ∈ B 1 , a = d(s), s ′ ∈ B 2 , a ′ = d(s ′ ).
Finally, to prove the converse statement, we exhibit an aperiodic model for which Condition 5 does not hold.

Example 3. Let

S = {s 1 , s 2 , s 3 }, A s1 = a 1 1 , a 1 2 , A s2 = a 2 1 , A s3 = a 3 1 .

The controlled transition probabilities can be associated to the rules

d 1 = (a 1 1 , a 2 1 , a 3 1 ) y d 2 = (a 1 2 , a 2 1 , a 3
1 ) and the

P d1 =   0 1 0 0 0 1 1/3 1/3 1/3   , P d2 =   1/3 1/3 1/3 0 0 1 1/3 1/3 1/3   .

RR n°7710

Taking k = (s 1 , a 1 1 ) and k ′ = (s 2 , a 2 1 ), we have supp(p(•|k)) = {s 2 } and supp(p(•|k ′ ) = {s 3 }. These sets are disjoint, so that by Lemma 1, Condition 5 fails.

To conclude this section, we discuss the idea of possibly relaxing the aperiodicity assumption. We show through the following example (obtained as a simplification of Example 4 in [START_REF] Lanery | Etude asymptotique des systèmes markoviens à commande[END_REF]) that the RH procedure may not converge on MDPs for which the optimal policy gives rise to an unichain periodic Markov process. The following section will explain how to handle this problem.

Example 4. Let S = {s 1 , s 2 , s 3 } be the state space, and the action sets

A s1 = a 1 1 , a 1 2 , A s2 = a 2 1 , A s3 = {a 3 1 }. The rules we can construct are d 1 = (a 1 1 , a 2 1 , a 3 1 ) and d 2 = (a 1 2 , a 2 1 , a 3
1 ). The transition matrix are P d1 and P d2 . P d2 is periodic of period 2.

P d1 =   1 0 0 0 0 1 0 1 0   , P d2 =   0 1 0 0 0 1 0 1 0   .
We consider the rewards

r(s 1 , a 1 1 ) = 2, r(s 1 , a 1 2 ) = 2, r(s 2 , a 2 1 ) = 5, r(s 3 , a 3 1 ) = 1.
The RH algorithm leads to a sequence where d 2 appears for odd horizons and d 1 for even horizons. When d 2 is considered as stationary policy in the infinite horizon problem, it produces an average value equal to [START_REF] Derman | Finite State Markovian Decision Processes[END_REF][START_REF] Derman | Finite State Markovian Decision Processes[END_REF][START_REF] Derman | Finite State Markovian Decision Processes[END_REF], while d 1 , an average reward equal to [START_REF] Bertsekas | Dynamic Programming: Deterministic and Stochastic Models[END_REF][START_REF] Derman | Finite State Markovian Decision Processes[END_REF][START_REF] Derman | Finite State Markovian Decision Processes[END_REF]. In consequence, the sequence of values does not converge. Moreover, it is immediate that the subsequence corresponding to odd horizons is better than that to even horizons.

Modified Rolling Horizon Procedure

We present a standard transformation under which all policies are perturbed to give rise to aperiodic Markov chains without modifying the corresponding gain. This transformation is similar to what Puterman presents in [START_REF] Puterman | Markov Decision Processes[END_REF]Section 8.5.4] and, up to our knowledge, it is originally due to Schweitzer (see [START_REF] Schweitzer | Iterative solution of the functional equation of undiscounted Markov renewal programming[END_REF]). Although these authors proposed it in the context of unichain models, it is clear that it works also in multichain models.

Transformation and modified procedure

Let 0 < τ < 1. Define S τ = S, A s,τ = A s for all s ∈ S. For all s and all a ∈ A s,τ ,

r τ (s, a) = r(s, a)
and for all j ∈ S,

p τ (j|s, a) = (1 -τ )δ s ({j}) + τ p(j|s, a) , (18)
where δ s (•) is the Dirac measure concentrated at s. Thus, for every decision rule d,

P d,τ = (1 -τ )I + τ P d ,

RR n°7710

Remark 2. Since the state set and the action sets are those of the original problem the policy sets also coincide.

For the sake of completeness we present the following results already proved in [START_REF] Puterman | Markov Decision Processes[END_REF]. We want to highlight that these results are valid also for the multichain case.

Proposition 1. Let us suppose that S finite, so that for each (d) ∞ , a stationary policy, P * d , the Cesàro limit matrix, is stochastic. Then it follows that

g (d) ∞ (s) = P * d r d (s).
Proposition 2. For every decision rule d

P * d,τ = P * d and g (d) ∞ τ (s) = g (d) ∞ (s)
for every s ∈ S.

Proof. We shall use the fact that, since S is finite, given d any decision rule, the limit matrix P * d,τ is the unique one that satisfies

P d,τ P * d,τ = P * d,τ P d,τ = P * d,τ P * d,τ = P * d,τ .
Note that ∀τ ∈ (0, 1], we have Proof. Similar to that of [START_REF] Puterman | Markov Decision Processes[END_REF]Corollary 8.5.9].

P * d P d,τ = (1 -τ )P * d I + τ P * d P d = P *
Our contribution in this direction is to propose to use this transformation as a preprocessing of the problem in order to deal only with aperiodic models.

More precisely, we propose the following procedure. Consider a MDP with state set S, actions A s for s ∈ S, transition probabilities p(j|s, a) for j, s ∈ S, a ∈ A s and rewards r(s, a), s ∈ S, a ∈ A s . Being τ ∈ (0, 1), transform the problem to a new one with S τ = S, A s,τ = A s for s ∈ S τ , r τ (s, a) = r(s, a) s ∈ S τ , a ∈ A s and p τ (•|j, a) given by Equation [START_REF] White | Centre de recherche INRIA Sophia Antipolis -Méditerranée 2004[END_REF].

Modified Rolling Horizon Procedure (MRH)

1. Given 0 < τ < 1, make the transformation described above.

2. Apply RH procedure to the new problem.

RR n°7710

On the theoretical convergence of MRH

The MRH procedure can be applied to MDP whose optimal stationary policy gives rise to chains with several irreducible classes. Through the transformation at Step 1, every finite model becomes, in the most general case, a multichain aperiodic model. Since the number of states and actions are finite and due to aperiodicity, the model satisfies the condition of Schweitzer and Federgruen (Condition 1). Then, by Corollary 1, the geometric convergence of RH is assured. In the particular case where the model is unichain (Condition 4), a parameter associated to the contraction might be computed explicitly, but likely just in small or very simple examples.

Practical convergence of the MRH procedure

Clearly, the transformed transition matrix converges towards original transition matrix when τ → 1. How is the convergence of v n (s)/n toward g * (s) modified when τ increases? We have a quantitative look at the question with the two following examples.

Example 5. Consider again the example with five states described in the Appendix. The optimal gain g * = (2, 2, 4, 4, 4) and it is produced by the stationary policy d = (a 2 , a 2 , a 1 , a 2 , a 1 ),

We have applied the τ -transformation described above. With τ = 0 the model is transformed in an uncoupled one where all the states are absorbing. As it is shown in Figure 2, when τ increases to 1 the periodicity effects are more evident. Example 6. Consider again the example of the Appendix. This model has a multichain and periodic structure. The RH procedure applied directly on this problem produces infinitely (and periodically) many times two policies, one of them is not two policies, (a 2 , a 2 , a 1 , a 1 , a 1 ) and (a 2 , a 2 , a 1 , a 2 , a 1 ). The first one produces a gain g = (2, 2, 3, 3, 3) and then it is not optimal since g * = (2, 2, 4, 4, 4).

When we pre-process the data, the Rolling Horizon procedure gives the optimal policy for the original problem, for any value of τ ∈ (0, 1).

Preservation of Condition 5

Next we investigate the connection of the transformation with Condition 5. Does the transformation preserve this property? Can it be expected obtaining Condition 5 through this transformation? The examples below show that this transformation does not destroy but weakens Condition 5, and that it does not produce it necessarily when it is not initially present, even with uncontrolled Markov chains. This is possibly due to the known fact that Condition 5 is related to some ergodicity property of the Markov chain involved and the transformation proposed by Schweitzer does not produce ergodicity properties, only aperiodicity.

≥ (p(B 1 |s, a) -p(B 1 |s ′ , a ′ )) -(p(B 2 |s, a) -p(B 2 |s ′ , a ′ )) = 1 τ p τ (B 1 |s, a) - 1 -τ τ δ s (B 1 ) - 1 τ p τ (B 1 |s ′ , a ′ ) + 1 -τ τ δ s ′ (B 1 ) - 1 τ p τ (B 2 |s, a) + 1 -τ τ δ s (B 2 ) + 1 τ p τ (B 2 |s ′ , a ′ ) - 1 -τ τ δ s ′ (B 2 ) .
The value of the preceding expression depends on the facts that the states s and s ′ belong or not to the subsets B 1 and B 2 . Since 0 ≤ δ s (•) ≤ 1 for any s, we have the following inequalities:

sp(p(•|s, a) -p(•|s ′ , a ′ )) ≥ 1 τ p τ (B 1 |s, a) - 1 -τ τ - 1 τ p τ (B 1 |s ′ , a ′ ) - 1 τ p τ (B 2 |s, a) + 1 τ p τ (B 2 |s ′ , a ′ ) - 1 -τ τ = 1 τ [p τ (B 1 |s, a) -p τ (B 1 |s ′ , a ′ )] - 1 τ [p τ (B 2 |s, a) -p τ (B 2 |s ′ , a ′ )] -2 1 -τ τ .
Then, as Condition 5 holds, there exists δ, 0 < δ < 1 such that, for any pair of subsets B 1 and B 2 , As expected, Condition 5 holds with a constant δ τ = 1τ + δτ .

2δ ≥ 1 τ [p τ (B 1 |s, a) -p τ (B 1 |s ′ , a ′ )] - 1 τ [p τ (B 2 |s, a) -p τ (B 2 |s ′ , a ′ )] - 2
It is readily checked that δ < δ τ < 1 since 0 < τ < 1. Hence, the transformation makes Condition 5 weaker: the smaller τ is, the weakest is the condition.

To conclude this section, we provide two examples which show that the aperiodicity transformation may or may not produce Condition 5, even for uncontrolled unichain problems (multichain models remain multichain after transformation). It is known that this condition is related to the structure of the Markov chain and not to the policies considered.

Example 7. This example is a case where Condition 5 appears after transformation.

Let us consider a MDP with state space S = {s 1 , s 2 }, and one admissible action in each state, i.e. an (uncontrolled) Markov Chain, where the transition probabilities are given by the matrix

P d = 0 1 1 0 .
This is a periodic and unichain model, and then it does not verify Condition 5.

RR n°7710

After apply the aperiodicity transformation we have the new transition probability matrix

P d,τ = 1 -τ τ τ 1 -τ .
It is not hard to see that Lemma 1 applies to this matrix: supp(p(•|s)) = {s 1 , s 2 } for s = s 1 , s 2 . More precisely,

p τ (s 1 |s 1 , d(s 1 )) -p τ (s 1 |s 2 , d(s 2 )) = (1 -τ ) -τ = 1 -2τ p τ (s 2 |s 1 , d(s 1 )) -p τ (s 2 |s 2 , d(s 2 )) = τ -(1 -τ ) = 2τ -1 sp(p τ (.|s 1 , d(s 1 )) -p τ (.|s 2 , d(s 2 ))) = 2 -4τ if τ ≤ 1/2 4τ -2 if τ ≥ 1/2.
P d =         1/2 1/2 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 1/2 1/2 0 0 0 0 0 0 1 1 0 0 0 0 0         .
The chain is unichain and aperiodic and again Condition 5 does not hold. In fact we can see that as a consequence of Lemma 1, since there exist the pair (s Through transformation for some τ ∈ (0, 1), we have the new transition matrix

P d,τ =         1 -τ /2 τ /2 0 0 0 0 0 1 -τ τ 0 0 0 0 0 1 -τ τ 0 0 0 0 0 1 -τ /2 τ /2 0 0 0 0 0 1 -τ τ τ 0 0 0 0 1 -τ        
where again the pair (s with a known rate, the value of the horizon n can be chosen so that the RH policy is ǫ-optimal.

Assume that some computable δ and C exist such that, for VI,

||ng * -v n -h * || < Cδ n or for the RH procedure, ||g * -g (dn) ∞ || < Cδ n .
Then consider the following stopping rule (or horizon choice rule for RH):

Stopping Rule 1. Stop if n > log(ǫ/C)/ log(δ).
Obviously, when VI stops under this rule, the policy (d n , . . . , d 1 , d n , . . .) is ε-optimal, and when the time horizon for RH is chosen according to this rule, the policy (d n ) ∞ is ε-optimal. We have discussed in Section 3.3 when this rule can be used in practice.

When no explicit convergence bound is known, the following practical convergence rule is proposed in [START_REF] Puterman | Markov Decision Processes[END_REF].

Stopping Rule 2. Stop if sp(v n+1 -v n ) ≤ ǫ.
For the aperiodic irreducible, unichain, communicating and weakly communicating models, or even if any of these properties is required only for optimal policies, all of them with g * a constant vector, Stopping Rule 2 is adequate as proved in [12, Section 8.5.4, p. 370]. More precisely, it is proved that if Stopping Rule 2 applies, the value of the policy is ε-optimal. Indeed, the classical proof of this claim involves passing to the limit in the following inequalities

min s∈S (v n -v n-1 )(s) ≤ g (dn) ∞ (s) ≤ g * (s) ≤ max s∈S (v n -v n-1 )(s).
Clearly, (d n ) ∞ will be ε-optimal for n large enough if

lim n→∞ max s∈S (v n -v n-1 )(s) = lim n→∞ min s∈S (v n -v n-1 )(s).
When we deal with arbitrary MDP's, we do not have a priori information about its structure and, in consequence we cannot guarantee that g * is a constant vector. In this case, Stopping Rule 2 does not provide a suitable stopping criterion, since the span sp(v n+1v n ) fails to converges to zero. Puterman [12, Section 9.4.2, p. 477] states a conjecture about a stopping rule for the multichain case. It involves the following rule.

Stopping Rule 3. Stop if sp(v n+1 -v n ) -sp(v n -v n-1 ) ≤ ǫ.
The next result shows that this rule not appropriate either.

Theorem 5. For any ǫ > 0, there exist MDP models such that, for the sequence {v n } obtained using the VI algorithm, it is possible to have, for some n

∈ N, sp(v n+1 - v n ) -sp(v n -v n-1 ) < ǫ and g * -g (dn) ∞ ≥ ǫ.
Proof. We construct such a model as follows. Consider the two-state model

S = {s 1 , s 2 }, A s1 = {a 1 1 , a 1 2 } y A s2 = {a 2 1 }, with transition probabilities p(s 1 |s 1 , a 1 1 ) = 1; p(s 2 |s 1 , a 1 2 ) = 1; p(s 2 |s 2 , a 2 1 ) = 1 RR n°7710
and gains r(s 1 , a 1 1 ) = 10; r(s 1 , a 1 2 ) = 1; r(s 2 , a 2 1 ) = 10 + ε. In this model, there exist only two stationary policies: d 1 = (a 1 1 , a 2 1 ) and d 2 = (a 1 2 , a 2 1 ). As the gains that they produces are, respectively, (10, 10 + ε) and (10 + ε, 10 + ε), the second policy is the optimal one. The sequence of functions v m obtained with the VI algorithm is such that, for m ≤ N (ε) := 9/ε + 1, v m+1v m = v m /m = 1 m m k=1 (v kv k-1 ) = g (dm) ∞ = (10, 10 + ε). On the other hand, as we have said, for any ε > 0, there exist N (ε) (who tends to infinity as ε tends to zero), such that, if m < N (ε) implies g * (s 1 )g (dm) ∞ (s 1 ) = ε > 0.

Actually, this example serves to show that a larger family of stopping rules, containing Stopping Rule 3, is not adequate.

In where the a ij are arbitrary constants. Theorem 6. There exist finite-state, finite-action MDP models such that, for some n, A(α 1 , α 2 , ..., α m , β 1 , β 2 , ..., β m , γ 1 , γ 2 , ..., γ m , δ 1 , δ 2 , ..., δ m ) < ε, and g *g (dm) ∞ ≥ ε.

As a consequence of Theorem 5, Stopping Rule 3 cannot work in general, and Theorem 6 even shows that no immediate generalization will work either. No alternative seems to be known in the literature, and Kallenberg mentions in [START_REF] Kallenberg | Markov decision processes[END_REF] that formulating a stopping rule for VI without chain analysis for multichain MDPs and provide a valid stopping rule remains an open problem. As it is observed in [17, p. 208], in the proof of Theorem 3.4.1, for models which verify Condition 3 (under which g * a constant vector) we obtain

sp(v n -v n-1 ) min s∈S (v n -v n-1 ) ≤ ε ⇒ 0 ≤ g (dn) ∞ -g * g * ≤ ε,
which induces the following

Stopping Rule 4. Stop if sp(v n+1 -v n ) ≤ ε min s∈S (v n -v n-1 ).
Observe that, in general, this is not an admissible rule, since it is not effective in the cases with non constant optimal rewards. This stopping rule differs form the other ones in the sense that, in this case, when the VI algorithm algorithm terminates iterating, it returns a policy whose relative error is not greater than ε.

Conclusions

We have reviewed in this paper convergence conditions for the Value Iteration procedure, and applied them to the question of "convergence" of the Rolling Horizon procedure.

Our analysis concludes that Condition 1 is a sufficient condition to ensure geometric convergence of the RH procedure in finite models. Condition 5, proposed earlier in [START_REF] Hernández-Lerma | Error Bounds for Rolling Horizon Policies in Discrete-Time Markov Control Processes[END_REF] is less general.

In addition, we introduce for the Rolling Horizon procedure a standard pre-processing of the problem for eliminating periodicities, resulting in the MRH procedure. We show that this transformation does not change the near optimal policies nor their values.

Theorem 2 claims that convergence is geometric, including for Multichain models. However, since a general stopping rule is not available, this result remains mostly theoretical. It remains to find an adequate bound for the error.

The gains can summarize as follows: r(s, a k ) in the matrix below is the gain when at state s, the action a k is chosen.

      1 2 1 2 1 1 3 2 6 6       .
Through the implementation of the MRH procedure the optimal average wealth can be computed: g * = (2, 2, 4, 4, 4). It is produced by the stationary policy associated to the decision rule d = (a 2 , a 2 , a 1 , a 2 , a 1 ) whose transition matrix is:

      0 1
0 0 0 1 0 0 0 0 0 0 0.7 0.3 0 0 0 0 0 1 0 0 0 1 0

      .
Clearly, it is a multichain periodic model. When RH procedure is applied directly, there is no convergence: the procedure gives infinitely (and periodically) many times two policies, (a 2 , a 2 , a 1 , a 1 , a 1 ) and (a 2 , a 2 , a 1 , a 2 , a 1 ). The first one produces a gain g = (2, 2, 3, 3, 3) and then it is not optimal.

Figure 1 :

 1 Figure 1: Evolution of v n+1v n (top), v n+1v ng (dn) ∞ (top, right), v n /n (bottom, left) and v n /ng (dn) ∞ (bottom, right), for state 4

Condition 5 .Remark 1 .

 51 There exists a positive number δ < 1 such that sp(p(.|s, a)p(.|s ′ , a ′ )) ≤ 2δ for every (s, a) and (s ′ , a ′ ) with s, s ′ ∈ S, a ∈ A s , a ′ ∈ A s ′ and for a measure λ, sp(λ) denote the norm sp(λ) In Section 3.4 we show that Condition 5 ⇒ Condition 4. It is easy to see that Condition 4 ⇒ Condition 3 ⇒ Condition 2 ⇒ Condition 1.

Under Condition 5 ,

 5 Inequality (13) holds with N = 1, C = 2||r||/(1δ) and δ given accordingly to [7, Proposition 5.1]. Under Condition 3, [5, Theorem 5] states that for M = 1 2 |S|(|S| -1), and any pair of M -tuples of decision rules π 1 , π 2 ∈ D M , min s1,s2∈S j∈S min P M π1 (s 1 , j), P M π2 (s 2 , j) =: ρ π1,π2 (M ) > 0.

Theorem 3 .Lemma 2 .

 32 Proof. Since the state and action spaces are finite, Condition 5 fails if and only if there exist s, s ′ , a, a ′ such that: sp(λ) = 2, with λ(.) = p(.|s, a)p(.|s ′ , a ′ ). This in turn is equivalent to:sup B λ(B) = 1 and inf B λ(B) = -1,and finally: ∃B, B ′ : p(B|s, a) = 1, p(B ′ |s, a) = 0, p(B|s ′ , a ′ ) = 0 and p(B ′ |s ′ , a ′ ) = 1. The set B can be taken as supp(p(.|s, a)) and B ′ = supp(p(.|s ′ , a ′ )). Next, Condition 5 implies structural properties on the MDP model. Every model where Condition 5 holds is a) unichain and b) aperiodic. The converse is not true. Proof. The proof of the first statement is by contradiction. Consider first part a) of the statement, and assume there is some d, a decision rule associated to a multichain Markov chain with transition probabilities P d , B 1 and B 2 two distinct irreducible recurrent classes. Then choosing s 1 ∈ B 1 , d(s 1 ) ∈ A s1 , s 2 ∈ B 2 and d(s 2 ) ∈ A s2 , we have supp(p(•|s 1 , d(s 1 ))) ⊂ B 1 , supp(p(•|s 2 , d(s 2 ))) ⊂ B 2 , implying that both sets are disjoint. Lemma 1 applies and Condition 5 cannot hold.Next, consider part b) of the statement: we prove that periodic chains do not verify Condition 5. We use for this the following result (see[4, p. 161]): Let X an irreducible Markov chain with recurrent states of period γ. Then states can be divided into γ disjoint sets B 1 , B 2 , . . . , B γ where p(j|i) = 0 unless i ∈ B 1 and j ∈ B 2 or i ∈ B 2 and j ∈ B 3 , . . ., or i ∈ B γ and j ∈ B 1 .

  r d,τ = r d and P d,τ is the transition matrix of a Markov chain with aperiodic recurrent classes.

Corollary 2 .

 2 d and P d,τ P * d = (1τ )IP * d + τ P d P * d = P * d . By well-known results, described for example in [12, Appendix A, p. 595], it follows that P * d,τ = P * d . To establish the second equality, notice that, by Proposition 1, g (d) ∞ τ = P * d,τ r d,τ , so g (d) ∞ τ = P * d,τ r d,τ = P * d r d = g (d) ∞ . The optimal stationary policies d * for the original problem and for the transformed one are the same. In addition, g d * τ (s) = g d * (s) for all s ∈ S and all τ ∈ (0, 1].

Condition 5 is preserved by the aperiodicity transformation Theorem 4 .

 4 If a MDP model satisfies Condition 5, then its transformation according to Section 4.1 still satisfies this condition. The constant "δ τ " of the transformed model can be chosen as: 1τ + δτ . Proof. According to Lemma 1, Condition 5 holds for the MDP if and only if, for all (s, s ′ ) ∈ S × S, all a ∈ A s , all a ′ ∈ A s ′ , supp(p(•|s, a)) ∩ supp(p(•|s ′ , a ′ )) = ∅. From the transformation (18), it is easy to see that supp(p τ (•|s, a)) = supp(p(•|s, a))

Figure 2 :

 2 Figure 2: Evolution of v n (4)/n for n = 1...150, with τ = 0.7 (top left), 0.9 (top right), 0.99 (bottom left) and 1 (bottom right)

1 -τ τ and taking the maximum over B 1

 11 and minimum over B 2 gives 2δ ≥ 1 τ sp (p τ (•|s, a)p τ (•|s ′ , a)) -2 1τ τ , or equivalently: sp (p τ (•|s, a)p τ (•|s ′ , a)) ≤ 2(1τ + δτ ) .

  fact, consider the valuesα m = v m /m, β m = v m+1v m , γ m = 1 m m h=1 v hv h-1 and δ m = g (dm) ∞ , for m = 1, ..., n. Define the rule A : (R |S| ) 4n → R, A(x 1 , x 2 , ..., x 4n ) = 4n i=1 4n j=1 a ij sp(x ix j ) ,

  16, Theorem 4.2]. The claim for Condition 3 is proved in [17, Theorem 3.4.2, p. 209]. We can find the proof of the convergence under Condition 4 in [12, Theorem 8.5.4, p. 370]. The arguments do not include geometric convergence, but this property holds since Condition 4 implies Condition 3 (for this last one, again, see Section 3.3 below).

  Condition 5 therefore holds for the transformed model with a constant δ τ = |1 -2τ | < 1. In this example, Condition 5 does not appear after transformation. Let S = {s 1 , s 2 , s 3 , s 4 , s 5 , s 6 }, and again consider just one action at each state. Transition probabilities are specified in the matrix

	Example 8.

  1 , s 4 ) ∈ S × S, gives us supp(p τ (•|s 1 , d(s 1 ))) ∩ supp(p τ (•|s 4 , d(s 4 ))) = {s 1 , s 2 } ∩ {s 4 , s 5 } = ∅

	and Condition 5 does not hold.
	5 On stopping rules

When a geometric convergence result exists, with a computable convergence bound, a simple stopping rule for VI is easily derived. Likewise, if RH converges geometrically RR n°7710

RR n°7710

Observe the discrepancy with the general notion of convergence of algorithms in Computer Science, which requires that an algorithm stops and returns the correct result.RR n°7710

Appendix

Each month an individual must decide how to allocate his wealth between different consumptions and investments. Each state represents a level of individual's wealth at the start of a month. Wealth levels give access to two different investment opportunities, prudent or risky. Choosing an investment profile at each level results in a probability transition for the next wealth level, as well as an instantaneous gain. The individual's objective is to maximize the average gain.

There are five levels of wealth, ordered from the smallest to the largest. At the medium level, connected to the risky behavior, there exists positive probability to pass to the next inferior level of wealth. It is also possible to cycle among the two inferior levels, but there is no action which permit the access to the three superior levels from the inferior ones. Besides, being at the poorest level, by some external help we achieve level 2. There is a common action space A = {a 1 , a 2 }, where a 1 represents the prudent investment profile and a 2 the risky attitude. We show the data below. P a k (s, j) is the transition probability from the state s to state j when action a k is used, i.e. P a k (s, j) = p(j|s, a k ).