M L Locatelli

S Ortolland

F Lanois

D Planson

T Billon

L Dicioccio

P Lassagne

STUDY OF SILICON CARBIDE HIGH VOLTAGE ABILITY : EXPERIMENTAL RESULTS ON EPITAXIAL-EMITTER AND IMPLANTED-EMITTER DIODES

INTRODUCTION

Silicon carbide is considered as the most promising semiconductor for power electronics, superior to silicon with regard to future application trends in terms of voltage, power, temperature or frequency increases [1]. Very attractive physical properties (high energy bandgap, high breakdown electric field, high saturated velocity and good mobility of carriers, high thermal conductivity), and good recently published electrical performance of experimental structures, both explain this general interest for SiC electronics.

Concerning the high voltage ability of silicon carbide, the availability of better quality thick epilayers has allowed rapid progress in high breakdown voltage diode realisation during the last five years (1000V-Schottky diode [START_REF] Urushidani | 5th Int. Conf. On SiC and Related Materials[END_REF], 2000V- [START_REF] Neudeck | 5th Int. Conf. On SiC and Related Materials[END_REF] and more recently 4500V-bipolar diodes [START_REF] Kordina | Linköping Studies in Science and Technology. Dissertations[END_REF]). These structures confirm the ability of SiC-pn junctions in sustaining much higher reverse voltage than silicon ones, under equivalent conditions of layer doping levels and thicknesses. However, these SiC-high breakdown voltage-diodes still present a destructive breakdown, leading to leaky reverse I(V) characteristics. Moreover, they are limited in maximum forward current, having low active areas due to substrate micropipe densities still relatively high [START_REF] Neudeck | [END_REF] (especially for 6H-SiC polytype). These remaining difficulties in the realisation of high voltage power diodes show the need for further research effort in this field.

This paper presents experimental results we obtained studying reverse electrical characteristics of high voltage bipolar diodes. The influences of junction diameters, and ambient natures (air or silicone oil), on breakdown locus and breakdown voltages are given and discussed, for both epitaxial-and implanted-emitter mesa structures. In the end, we also present more recent experimental results corresponding to planar diodes, with implanted junction termination extension as a peripheral guard.

EXPERIMENTAL STUDY OF MESA STRUCTURES

The mesa diodes have been realised at CEA-LETI, using two 1 inch 6H-SiC wafers (purchased from CREE Research in 1992). Each wafer consisted in a p --epilayer (with specified thickness of 8 µm, and net doping level of few 10 14 cm -3 ) on a p + -substrate, and allows the creation of a n + p --junction using one of the two following techniques : an epitaxial growth by in-situ N-doping CVD, and a multiple N-ion implantation. Circular mesa-devices were patterned on each wafer by plasma etching of its surface through an aluminum mask. 5.7 µm-deep mesa structures, with seven different diameters from 100 µm to 1200 µm, were thus obtained. Passivation of the etched surfaces was realised by deposition of a 6500 Å-SiO 2 -layer. Contact metallizations were made by sputtering tungsten on the top of n +cathodes, and by evaporating aluminum on p + -substrates. Figure 1 presents a schematic view of these mesa structures. About seventy diodes of each type (epitaxial and implanted) were characterised until breakdown voltage at room temperature. The behaviour of the device at breakdown was studied by measuring the Current(Voltage) characteristic evolution (using a curve tracer Tektronix 370A in DC bias) on one hand, and by directly viewing the device under test through a binocular on the other hand, for breakdown locus observation. Breakdown voltages, and leakage current (I R ) versus reverse voltage (V R ), were studied in correlation with diode sizes.

RESULTS AND DISCUSSION ON MESA STRUCTURES

A common observation for both types of studied mesa-diodes was the occurrence of a reverse voltage for which the reverse current suddenly strongly increased. Simultaneously a possible appearance of a bright flash, resulting afterwards in a black mark on the top of the devices, could be observed. We defined this particular reverse voltage as the breakdown voltage of the device. After breakdown, voltage capability of the device was generally reduced, with leakage currents irreversibly higher than before breakdown, for the same applied voltage. Different kinds of pre-breakdown electrical characteristics could be distinguished.

Description of I R (V R ) characteristics before breakdown

Two main different behaviours of I R (V R ) curves before breakdown at 300 K were observed for the two types of studied mesa diodes. The first behaviour corresponds to asdefined "leaky diodes". These diodes are presenting high current versus reverse voltage with discontinuities and slope changes of the I R (V R ) curve. In these cases, breakdown occurs for reverse currents between 100 µA and 5 mA, and for reverse voltages lower than 400 V for epitaxial-emitter diodes and 350 V for implanted-emitter diodes. The second behaviour corresponds to "non-leaky" diodes, with a stable reverse current component lower than 1 µA for increasing voltages, until breakdown. The latter occurs without any precursor sign for voltages higher than 500 V. Figure 2 shows an example of a Current(Voltage) characteristic obtained on a 100 µm-diameter epitaxial-emitter diode, dipped in silicone oil, just before a breakdown happens. 1100 V-breakdown voltage corresponds to the highest value obtained from our diodes at room temperature, all measurements mixed up. Such a value corresponds to 75 % of the theoretical evaluation calculated for a planar asymmetrical n + p --junction with base thickness of 8 µm and a base doping of 8x10 14 cm -3 , and using a critical electric field of 1.7x10 6 V/cm derived from impact ionization coefficients found in [6]. The high voltage drop that can be noticed on figure 2 for high currents in on-state operation is both due to the high resistivities of low-doped p-type base and of ohmic contacts realised on top-emitter and bottom-substrate. In the case of devices with an implanted-emitter, a third typical electrical behaviour before breakdown was also noticed. First the I R (V R ) curve presents low but non-stable currents (with parasitic peaks) which vanish for a given maximum reached voltage (and after several bias scans). A further increase in maximum reverse voltage leads to a higher non stable level of I R (V R ) curve, with a final breakdown for currents lower than 100 µA and voltages higher than 400 V.

A great difference versus device diameters exists between epitaxial-and implantedmesa diodes concerning the measured values of breakdown voltage.

Influence of the device diameter on reverse electrical behaviour

The epitaxial-emitter diodes show a large scattering of breakdown voltage values, from few tens of Volts to 900 Volts in air ambient at room temperature. This result is clearly correlated with the device size, since only 60 % of 100 µm-diodes and 20 % of 200 µmdiodes present breakdown voltages higher than 500 Volts (until 1100 V in silicone oil ambient). These small devices were also the only ones to show very low leakage currents under reverse biases (corresponding to the second type of behaviour described above). All the other diodes present a leaky behaviour, with low breakdown voltage values.

By comparison, the scattering in implanted-emitter diode breakdown voltages is lower, with measured values at room temperature between 400 V and 600 V in air, all diameters mixed up. In fact, very few implanted-diodes (about 10 %) were leaky under reverse biases, all having diameters higher than 600 µm. But most of the implanted-structures had an electrical behaviour under reverse bias of the third type described above, with "progressive" breakdown, equally distributed between the different diameters. The highest voltage capabilities are however obtained from non-leaky devices presenting a sudden and clean breakdown almost exclusively on 100 µm-diameter diodes. The maximum breakdown voltage value obtained with such devices was 780 V in silicone oil. The thickness of the p --base layer, smaller than the specified one (as derived from C(V) measurements) can be one reason for this apparent lower voltage capability of implanted-emitter diodes compared with epitaxial-emitter ones.

Observation of breakdown locus and ambient nature influence

For small-size epitaxial diodes that show the highest breakdown voltages, bright flashes can often be observed, leaving marks on the mesa top, near its periphery. Moreover the measurements realised using silicone oil as the ambient resulted in maximum breakdown voltages 20% higher that the corresponding values obtained using air ambient.

These observations suggest that the breakdowns mainly occur in periphery of these diodes, either in ambient or at SiC/Si0 2 interface. Probable degradation of the passivation, resulting in increased surface leakage could explain the higher reverse current after breakdown. No noticeable degradation of the forward characteristic after breakdown also confirms that the inside junctions is not affected by the breakdown. For the other epitaxialemitter diodes, of the leaky-type, marks in the middle of the top metallization can be observed after breakdown (this one occurring at high current). This fact indicates a breakdown possibly due to structural defects in the junction bulk. Such structural defects, effectively noticed along the surface of the n + -epilayer can also explain the higher leakage currents observed for large devices in comparison with small ones.

In the case of the implanted-emitter diodes, assuming that the absence of mark or the presence of a mark within the middle of a device metallization are related to a bulk phenomenon, and that the existence of a mark near the mesa top edge revealed a peripheral breakdown, then the following observations could be made. Small devices (essentially of 100 µm-and 200 µm-diameters) preferably present a breakdown that occurs at the periphery of the mesa, slightly increased by the use of silicone oil instead of air. On the contrary, leakydiodes preferably show a breakdown in the device bulk, possibly due to the presence of micropipes, which can explain the I R (V R ) curve discontinuities. For the most common behaviour observed, corresponding to "progressive" breakdown in three phases, SiC/Si0 2 interface instabilities can be involved. Peripheral phenomena can lead to breakdown in most of these cases.

RECENT RESULTS ON PLANAR IMPLANTED STRUCTURES

Planar diodes have been realised in collaboration with the Applied Physic Institute in Erlangen. They are based on a CREE 6H-SiC n --epilayer (above n + -substrate) with specified thickness of 10 µm and net doping level of 4x10 15 cm -3 . The circular p + -emitter region was realised by Al-ion implantation. The peripheral guard consists in a pn -junction termination extension ("JTE" protection), that was created by ion implantation too. Total studied device diameters remain lower than 400 µm. Figure 3 schematically describes these structures, having contact metallizations made of aluminium and nickel on p + -emitters and n + -substrate respectively. No passivation was realised on the semiconductor surface.

Typically above 1000 V, up to 1300 V, the breakdown voltages in air ambient occur with reverse currents up to 250 µA at room temperature. Before breakdown, the reverse current values weakly evolve with bias time, and present the shape corresponding to curve n°1 shown on figure 4. The tests in oil ambient lead to the reduction of the leakage current measured for a given reverse voltage after several minute of bias supply. Under these conditions, higher typical breakdown voltages than those obtained in air are measured, reaching more than 1600 V at room temperature. The curve n°2 on figure 4 gives an example of a pre-breakdown I(V) characteristic obtained in oil ambient. After breakdown, either in air or in oil ambients, the I(V) characteristic is transformed into a stable and reversible curve like curve n°3 shown on figure 4. This characteristic presents a rapid increase of leakage current typically occurring between 600 V and 800 V. A visible degraded zone located at the edge of the guard termination is sometimes observed after breakdown. The high voltage capability obtained from these "JTE" 6H-SiC diodes represents an interesting experimental result.

CONCLUSION

6H-SiC-mesa-bipolar diodes were realised using either an epitaxial-growth or a multiple-implantation for the n + p --junction creation, based on two comparable CREE p -epilayers. A difference in maximum reverse voltage scattering has been shown between these two types of diodes. The lower reproducibility in breakdown voltage values of epitaxialemitter structures compared to implanted-emitter ones is probably related to the higher density of structural defects arising from the epitaxial-emitter realisation. Correlatively a stronger effect of device size on the leaky or non-leaky behaviour results for epitaxial-emitter.

Figure 1 :

 1 Figure 1 : Cross-section of 6H-SiC mesa diodes.

Figure 2 :

 2 Figure 2 : Current(Voltage) characteristic of a 100 µm-diameter 6H-SiC mesa diode with a n + -epitaxial emitter, measured at room temperature in silicone oil ambient.

Figure 3 :

 3 Figure 3 : Cross-section of 6H-SiC planar JTE diodes.

Figure 4 :

 4 Figure 4 : Current(Voltage) characteristic of a 6H-SiC p + n -n + diode with a JTE guard, measured at room temperature in air and silicone oil ambients.

For both kinds of mesa-diodes however, the smallest diodes were able to reach the highest reverse voltages with sudden breakdown appearance. The breakdown locus are shown to occur near these device periphery, leading to I(V) characteristic degraded at reverse biases essentially. The respective roles of ambient nature (effective but weak) and passivation remain to be enlightened. More recently, a very interesting reverse voltage capability has been experimentally obtained with 6H-SiC p + n -n + Junction Termination Extension diodes, presenting breakdown voltages up to 1600 V at room temperature.

In conclusion, these experimental results are representative of two main current difficulties in the realisation of SiC high power devices. The first is related to basic-and epitaxial-material quality. The second corresponds to the critical influence of peripheral parameters including ambient natures. A better understanding of electric field distributions under reverse biases, especially along SiC surface, appears to be a third topic of important investigations.

Aknowledgements : CEGELY aknowledges the GIRCEP group, the DRET, and the A2 Research Center of SCHNEIDER ELECTRIC S.A. for the support of this study. CEGELY also greatly aknowledges Professor Helbig, Doctor Pensl and SiC group from the Applied Physic Institute of Erlangen for their strong collaboration in planar diode realization. The authors would like to thank Frederic SEIGNEUR for his help in measurements.


A Thevenet

C Marie

Dominique Guillaumont

V Amendola

Philippe Guilbaud

Perrhenate

Perrhenate and pertechnetate azacryptates in nitric acid solution

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Perrhenate and pertechnetate azacryptates in nitric acid solution

Alexiane Thevenet 1 , Cécile Marie 1 , Christelle Tamain 1 , Dominique Guillaumont 1 , Valeria Amendola 2 , Philippe Guilbaud

Abstract

Technetium is an artificial element produced after fission of uranium in nuclear reactors [1] . 99 Tc is one of the most important fission product found in dissolution solutions of used nuclear fuel [2] . In nitric acid medium, technetium is found at +VII oxidation state as pertechnetate anion TcO 4 -. Due to the large amounts of 99 Tc produced by nuclear fission and to the long half-life of this radionuclide (2.1 x 10 5 years), technetium is an element of particular importance in nuclear waste management [3] . In this work, technetium cryptands in nitric acid medium are studied. The large size and low charge density of the oxoanion make selective recognition in aqueous solution a great challenge. For this purpose, a "polyammonium cage" ligand [4,5] was studied for the pertechnetate anion recognition in nitric acid solution. Coordination studies were performed with its inactive surrogate, the perrhenate anion (ReO 4 -), by Infrared and RAMAN spectroscopies, single crystal XRD (Figure 1) and DFT calculations. As shown in Figure 1, the ligand conformation adapts itself to let one perrhenate anion enter the cavity (cage size expansion, hydrogens of the protonated amines directed towards the inside of the cavity). The complex stabilization is achieved by direct hydrogen bonds formation between the azacryptand and the perrhenate anion.

Figure 1 :

 1 Crystallographic structures a) free ligand H 6 L and b) complex [(ReO 4 )(H 6 L)].(ReO 4 ) 1.5 (NO 3 ) 3.5 (H 2 O) 3


Bruno Schapira
email: bruno.schapira@univ-amu.fr.

LOCALIZATION ON 5 SITES FOR VERTEX REINFORCED RANDOM WALKS: TOWARDS A CHARACTERIZATION

Keywords: Self-interacting random walks; Vertex Reinforced Random Walk, MSC 2010 subject classifications, 60K35

We continue the investigation of the localization phenomenon for a Vertex Reinforced Random Walk on the integer lattice. We provide some partial results towards a full characterization of the weights for which localization on 5 sites occurs with positive probability, and make some conjecture concerning the almost sure behavior.

Introduction

Given a sequence w = (w(n)) n≥0 of positive real numbers, called the weight, one can define a process (X n ) n≥0 on Z, called Vertex Reinforced Random Walk (VRRW) as follows: first X 0 = 0, and then for any n ≥ 0 and x ∈ Z,

(1) P(X n+1 = x ± 1 | F n ) = w(Z n (x ± 1)) w(Z n (x + 1)) + w(Z n (x -1))

,

where F n := σ(X 0 , . . . , X n ) and Z n (y) is the number of visits to site y by the process before time n (see below). This process was introduced by Pemantle [P] on the complete graph and for a linear weight, and then by Pemantle and Volkov on Z, still for the linear weight, who showed that the process localizes on five sites with positive probability, that is with positive probability exactly five sites are visited infinitely often. This result was later improved by Tarrès who showed [T1, T2] that this behavior occurs in fact almost surely.

A few years later, Volkov [V] introduced the model with a general weight sequence, in the same fashion as Davis [Dav] did for Edge Reinforced Random Walks. He proved in particular that for weights of the form w(n) = n α , with α < 1, localization on a finite subgraph is not possible. This was later improved in [CK, Sch, S] in the case α < 1/2, where it was proved that the process visits almost surely all sites infinitely often.

In a previous work in collaboration with Basdevant and Singh [BSS], we managed to completely characterize the nondecreasing weights for which localization on 4 sites occurs with positive probability, or almost surely, in terms of some parameter α c (w) (see below). Our aim here is to analyze the analogous question for the localization on 5 sites. For this we introduce some new parameter β c (w), which should play a similar role as α c (w). To define it, we first extend w as a function on the positive reals by w(t) := w( t ), and then set

W (t) := t 0 1 w(u)
du.

We will assume throughout the paper that

(2)

∞ n=0 1 w(n) = ∞,
which is equivalent to saying that W is a bijection from R + to itself. Note however, that this is not a restrictive hypothesis, since when w is reciprocally summable, it is known [BSS, V] that the process localizes almost surely on two sites. Then we denote by W -1 its inverse, and define for α > 0,

I α (w) := ∞ 0 dx w(W -1 (W (x) + α))
.

When w is nondecreasing, the map α → I α (w) is nonincreasing and one defines

(3) α c (w) := inf{α ≥ 0 :

I α (w) < ∞} ∈ [0, ∞],
with the convention that inf ∅ = ∞. In [BSS] it was proved in particular that localization on 4 sites holds with nonzero probability if, and only if, α c (w) is finite. We now define for β ∈ R,

J β (w) := ∞ 0 dx w(W -1 (2W (x) + β)) ,
with the convention that W -1 (u) = 0, for u < 0, and set

β c (w) := inf{β ∈ R : J β (w) < ∞} ∈ [-∞, +∞].
We make the following conjecture (with R standing for the set of sites which are visited infinitely often):

Conjecture 1.1. Assume that w is nondecreasing and satisfies (2). Assume further that α c (w) = ∞. Then

P(|R | = 5) > 0 ⇐⇒ P(|R | = 5) = 1 ⇐⇒ β c (w) < ∞.
Remark 1.1. As we will later explain further, we also conjecture that in fact β c (w) always belongs to {±∞}.

The hardest part here is the characterization of the almost sure localization, which is a notoriously difficult problem that we will not discuss in this paper; we simply recall that in the case of a linear weight, Tarrès proved that |R | = 5 almost surely [T1, T2]. Proving that the same holds for some other weight function is possibly one of the most challenging problem on this model. Instead we will only be interested here on the easiest part of the conjecture, which is a characterization of the localization with positive probability. Our first result provides one direction of the conjecture: Theorem 1.1. Assume that w is nondecreasing. Then

P(|R | = 5) > 0 =⇒ β c (w) < +∞.
We note that this result was proved in [START_REF] Basdevant | Localization of a vertex reinforced random walk on Z with sub-linear weight[END_REF] (see the proof of Proposition 1.4 there) under some additional hypotheses on w, including the fact that w was a slowly varying function.

Our second result concerns the other direction. However, instead of β c (w) being finite, one needs to assume some slightly stronger condition (which we nevertheless conjecture to be equivalent). Namely, we first define H(x) := x + W -1 (W (x) + 1), and note that H is increasing and continuous; thus it has an inverse which we denote by H -1 . Then set for β ∈ R,

J β (w) := ∞ 0 dx w(H -1 (W -1 (2W (x) + β)))
,

and β c (w) := inf{β ∈ R : J β (w) < ∞} ∈ [-∞, +∞].
Note that H(x) ≥ x and H -1 (x) ≤ x, for all x ≥ 0. Thus for any β ∈ R, J β (w) ≥ J β (w). In particular for any w,

β c (w) ≥ β c (w).
Our second result is the following: Theorem 1.2. Assume that w is nondecreasing and satisfies (2). Assume further that α c (w) = ∞. Then

β c (w) < ∞ =⇒ P(5 ≤ |R | < ∞) = 1, P(|R | ∈ {5, 6}) > 0.
As mentioned above we conjecture that in fact β c (w) = β c (w), for all weights w. We provide some evidence for this fact at the end of the paper, and show that it is true for a large class of weight functions (see Lemmas 5.1 and 5.2).

In particular Lemma 5.1 shows that for any surlinear weight function, such that w(n) = o(n √ log n), one has β c (w) = -∞. This is of course not surprising, regarding the known result for a linear weight, but we stress that prior to this, not much was known for weights with intermediate growth between linear and n log log n. Indeed, in [BSS] it was only proved that for weights satisfying w(n) = o(n log log n), α c (w) = ∞, and localization on 4 or less sites was impossible.

It might look a bit disappointing that we cannot exclude the possibility of a localization on 6 sites in the conclusion of Theorem 1.2, especially since for a linear weight as well as for weights satisfying w(n) ∼ n/ exp(log α n), with α ∈ (0, 1/2), it was proved respectively in [START_REF] Pemantle | Vertex-reinforced random walk on Z has finite range[END_REF][START_REF] Tarrès | Localization of reinforced random walks[END_REF] and [START_REF] Basdevant | Localization of a vertex reinforced random walk on Z with sub-linear weight[END_REF], that localization on 5 sites occurs with positive probability. Let us however observe that in both cases the proofs rely heavily on the explicit form of the weight function and cannot be transposed (at least not directly) to the general setting we are considering here.

Finally we also believe that localization on any even number of sites, larger than or equal to 6, is not possible for any weight function. In contrast it was proved in [START_REF] Basdevant | Localization of a vertex reinforced random walk on Z with sub-linear weight[END_REF] that localization on any odd number of sites -other than one and threeis possible.

The paper is organized as follows. In the next section, we recall some important and elementary facts about the VRRW, and some related martingales attached to each site. Then in Sections 3 and 4 we give the proofs of Theorems 1.1 and 1.2 respectively. The final section is concerned with the computation of the parameters β c (w) and β c (w), and gives some cases where one can show equality between them.

Notation and background

2.1. VRRW. Given some initial distribution of local times C := (z 0 (y)) y∈Z ∈ N Z , we define the C-VRRW as the process (X n ) n≥0 , whose transition probabilities are given by (1), with for any y ∈ Z, Z 0 (y) = z 0 (y), and for any n ≥ 1,

Z n (y) := z 0 (y) + n k=1 1{X k = y}.
We denote by P C the law of the C-VRRW. We call C 0 the configuration with z 0 (y) = 0, for all y = 0 and z 0 (0) = 1. We then simply say that X is a VRRW when its initial local time distribution is given by C 0 , and denote its law by P. We also recall that a C-VRRW can be defined as well on any subgraph of Z, and we refer to [BSS] for details.

The martingales

M n (x). For x ∈ Z, define Z ∞ (x) := lim n→∞ Z n (x).
Recall that R stands for the set of sites visited infinitely often by the walk:

R := {x ∈ Z : Z ∞ (x) = ∞}.
We define for any n ≥ 1, and x ∈ Z,

(4) Y ± n (x) := n-1 k=0 1{X k = x, X k+1 = x ± 1} w(Z k (x ± 1)) ,
and

M n (x) := Y + n (x) -Y - n (x)
. We let also Y ± 0 (x) = 0, and M 0 (x) = 0, and consider the limits:

Y ± ∞ (x) := lim n→∞ Y ± n (x).
An important observation from Tarrès [T1, T2] is that (

M n (x)) n≥1 is a martingale for each x ∈ Z. Moreover, if (5)
∞ n=0 1 w(n) 2 < ∞,
then these martingales are bounded in L 2 , and thus converge almost surely and in L 2 . Moreover, for any C-VRRW, one has

(6) Y + n (x -1) + Y - n (x + 1) = W (Z n (x)) -W (z 0 (x)
). We will also use the following result due to Tarrès (see also [START_REF] Basdevant | Localization on 4 sites for Vertex Reinforced Random Walk on Z[END_REF]Lemma 3.3]):

Lemma 2.1 (Tarrès [T2]). Assume that w is nondecreasing and that (5) holds. Then, for any x ∈ Z, almost surely,

{Y + ∞ (x) < ∞} = {Y - ∞ (x) < ∞} = {Z ∞ (x -1) < ∞} ∪ {Z ∞ (x + 1) < ∞}.
We further use the same notation as in [T2], and write f (n) ≡ g(n), when the sequence (f (n) -g(n)) n converges to some finite real. In particular, it follows from the above discussion that

n≥0 1 w(n) 2 < ∞ =⇒ Y + n (x) ≡ Y - n (x), for all x ∈ Z. (7) 3. Proof of Theorem 1.1
We start the proof with the following lemma: Lemma 3.1. Assume that w is nondecreasing. Then

P(|R | = 5) > 0 =⇒ ∞ n=0 1 w(n) 2 < ∞.
Remark 3.1. This result has the same flavor than some others from [Sch, S, V], which all give different conditions on the weight w, ensuring that localization on any finite subgraph is not possible. In particular the proof in [S] shows that for any weight satisfying lim sup n/w(n) 2 = ∞, the walk cannot localize on any finite subgraph, which is close to imply our result (but not quite).

Proof of Lemma 3.1. We first note that if localization on five sites occurs with positive probability, then there exists some initial configuration C, such that with positive probability the C-VRRW spends all its time in the set {1, 2, 3, 4, 5}, and visits all sites from this set infinitely often. Call E this event. By the conditional Borel-Cantelli Lemma (see Theorem 4.3.2 in [Dur]), one can see that almost surely on the event E, one has Y + ∞ (1) < ∞, since for some constant c > 0 (only depending on C), one has

Y + ∞ (1) ≤ c k≥0 P [X k+1 = 0 | F k ] 1{X k = 1},
where we denote here by F k the sigma field generated by the process X up to its k-th visit to site 1. Then we use that the following process is a martingale (for a very similar reason as for M n (x)):

Y + n (1) - Zn(2)-z0(2) k=1 p k (2, 1) w(k + z 0 (2))
,

where p k (2, 1) denotes the probability to jump to site 1 at k-th visit to site 2. Since this martingale has bounded increments, we know that almost surely, either it converges, or its lim sup as well as its lim inf are both infinite (see Theorem 4.3.1 in [Dur]). However, we have just observed that on the event E, its lim sup is finite, which means that it must converge, and as a consequence on the event E, it holds almost surely

∞ k=1 p k (2, 1) w(k + z 0 (2)) < ∞.
Now by definition of p k (2, 1), one has for some constant c > 0 (depending only on C), and on E,

∞ k=1 1 w(k + z 0 (2))w(Z τ k (3)) ≤ c ∞ k=1 p k (2, 1) w(k + z 0 (2)) < ∞,
where τ k denotes the time of k-th visit to site 2. By symmetry one has as well

∞ k=1 1 w(k + z 0 (4))w(Z τ k (3)) < ∞,
with τ k the time of k-th visit to site 4. Finally observe that for any n, Z n (3) ≤ Z n (2) + Z n (4) + C, with C a constant depending only on C. This implies that for

any k, either Z τ k (3) ≤ 2k + C + z 0 (3) or Z τ k (3) ≤ 2k + C + z 0 (3).
Using that w is nondecreasing, it follows that for some (possibly larger) constant C > 0,

k≥0 1 w(2k + C) 2 < ∞.
The lemma follows, using again that w is nondecreasing.

We next prove the following result.

Lemma 3.2. Let X be a C-VRRW, for some initial local time configuration C. Assume that w is nondecreasing, and satisfies (2) and (5). Then on the event

E = {Z ∞ (0) = Z ∞ (4) = ∞} ∩ {Y + ∞ (0) < ∞} ∩ {Y - ∞ (4) < ∞}, it holds almost surely Z n (2) -max(Z n (1), Z n (3)) → +∞, as n → ∞. Proof. Let for n ≥ 1, N n (y, y ± 1) := n-1 k=0 1{X k = y, X k+1 = y ± 1},
denotes the number of jumps from y to y ± 1 before time n, for any y ∈ Z.

Then (8) Z n (1) ≡ N n (0, 1) + N n (2, 1), and Z n (2) ≡ N n (1, 2) + N n (3, 2). Now observe that W (N n (2, 1)) -Y - n (2) is nondecreasing and that for all n, 0 ≤ W (N n (2, 1)) -Y - n (2) ≤ Y + n (0) + W (z 0 (1)). Since by definition Y + ∞ (0) is finite on the event E, we deduce that W (N n (2, 1)) ≡ Y - n (2)
. By symmetry, one has as well

W (N n (2, 3)) ≡ Y + n (2)
, and since by (7), one also has

Y - n (2) ≡ Y + n (2), we get in fact W (N n (2, 1)) ≡ W (N n (2, 3)). (9)
Moreover, Lemma 2.1 implies that under the hypotheses of the lemma and on the event E, Z ∞ (-1) is finite, and thus Y + ∞ (-1) also. Together with (6), it follows that

W (N n (1, 0)) ≡ W (Z n (0)) ≡ Y - n (1) ≡ Y + n (1). ( 10
)
We claim now that

δ n := W (N n (1, 2)) -Y + n (1) → +∞. (11)
Indeed, on one hand δ n is nondecreasing, and on the other hand its limit and we get (11). By using next that |N n (1, 2) -N n (2, 1)| ≤ 1, together with (9), ( 10) and (11), we obtain

δ ∞ satisfies δ ∞ ≥ Y - ∞ (3). Since Z ∞ (4) = Z ∞ (2) = ∞, Lemma 2.1 shows that Y - ∞ (3) = ∞,
W (N n (2, 3)) -W (N n (1, 0)) → +∞,
which implies that N n (2, 3) -N n (1, 0) → +∞. Using now (8), it follows that Z n (2) -Z n (1) → +∞, almost surely. By symmetry we get as well Z n (2) -Z n (3) → +∞, and the lemma follows.

Let us resume now the proof of Theorem 1.1. Lemma 3.7 in [BSS] shows that there exists some local time configuration C, such that for the C-VRRW, the event

E := {Z ∞ (0) = Z ∞ (4) = ∞} ∩ {Y + ∞ (0) < ∞} ∩ {Y - ∞ (
4) < ∞}, has some positive probability. Moreover, we know by (6) that on E,

W (Z n (1)) -W (Z n (3)) ≡ Y - n (2) -Y + n
(2), and using (7), we deduce that W (Z n (1)) -W (Z n (3)) converges as n → ∞, towards some α ∈ R. Furthermore, Lemma 4.8 in [BSS] shows that almost surely α = 0, and by symmetry we can assume without loss of generality that α > 0. In particular, this gives Z n (1) ≥ Z n (3), for n large enough. Set now

h n (2) := Zn(2)-z0(2) k=1 2p k -1 w(k + z 0 (2))
,

where p k is the probability to jump to site 1 at k-th visit to site 2. As noticed already in the proof of Lemma 3.1, one has

h n (2) ≡ Y + n (1) -Y - n (
3). But since after some time the process has at least probability 1/2 to jump to 1 when it is in 2, we see that for n large enough h n (2) is nondecreasing. In particular there exists some (random) constant γ ∈ R, such that h n (2) ≥ γ, for all n ≥ 0. This implies that for some other constant γ > 0,

Y - n (3) ≤ Y + n (1)
+ γ , for all n ≥ 0. By using also that

W (Z n (0)) ≡ Y - n (1) ≡ Y + n (1) ≡ W (Z n (2)) -Y - n (3)
, we deduce that for some (random) β ∈ R,

W (Z n (2)) ≤ 2W (Z n (0)) + β, for all n ≥ 0. ( 12
)
Together with Lemma 3.2, this yields for some constant c > 0,

Y + ∞ (0) ≡ ∞ n=0 1{X n = 0} w(Z n (1)) ≥ c ∞ n=0 1{X n = 0} w(Z n (2)) ≥ c ∞ n=0 1 w(W -1 (2W (n) + β)) ,
which concludes the proof of the theorem, since Y + ∞ (0) is finite on E.

Proof of Theorem 1.2

We start the proof with some elementary lemma.

Lemma 4.1. Assume that w is nondecreasing. Then

β c (w) < ∞ =⇒ ∞ n=0 1 w(n) 2 < ∞.
Proof. Assume that w(n) ≤ √ n, for some n ≥ 1. Since w is nondecreasing, this implies on one hand W (n + 1) ≥ √ n, and also w(k) ≥ w(0), for all k ≥ 0. The latter implies the existence of a constant c > 0, such that W (k) ≤ √ n/3, for all k ≤ c √ n (namely one can take c = w(0)/3). Assume that n is large enough so that

√ n/3 < ( √ n -β c (w) -1)/2. Then 2W (k) + β c (w) + 1 < √ n, for all k ≤ c √ n. Therefore W -1 (2W (k) + β c (w) + 1) < n + 1
, for all such k, and it follows that

c √ n/2≤k≤c √ n 1 w(W -1 (2W (k) + β c (w) + 1)) ≥ c/2.
In particular, by definition of β c (w), this can only happen for finitely many n, which proves that lim inf w(n)/ √ n ≥ 1. We use now this information to bootstrap the previous argument. Assume that w(n) ≤ n 2/3 for some n ≥ 1, later taken large enough. Note first that this implies W (n + 1) > n 1/3 . Moreover, since lim inf w(m)/ √ m ≥ 1, we can find c > 0 some small enough constant, such that W (cn 2/3 ) ≤ n 1/3 /3, assuming n is large enough. Taking larger n if necessary, one can assume that 2W (k) + β c (w) + 1 ≤ n 1/3 , for all k ≤ cn 2/3 . This implies first W -1 (2W (k) + β c (w) + 1) ≤ n, and then w(W -1 (2W (k) + β c (w) + 1)) ≤ n 2/3 , for all such k. Thus

cn 2/3 /2≤k≤cn 2/3 1 w(W -1 (2W (k) + β c (w) + 1)) ≥ c/2,
from which we deduce that lim inf w(n)/n 2/3 ≥ 1, and the lemma follows.

The next step is the following lemma.

Lemma 4.2. Assume that w is nondecreasing and that β c (w) < ∞. For N ≥ 1 integer, η ∈ (0, 1), and β ∈ R we define C N,η,β , as the set

C N,η,β :=        {z 0 (x)} x∈Z ∈ N Z : z 0 (-1) ≤ z 0 (-2) + z 0 (0) z 0 (-1) ∧ z 0 (0) ≥ N W (z 0 (-2)) ≤ W (z 0 (0)) -η W (z 0 (-3)) ≤ W (z 0 (-1))/2 -β        .
Given C some local time configuration, we denote by P * C the law of the C-VRRW restricted to the set {-3, . . . , 0}. For any η ∈ (0, 1), and β > ( β c (w) + 3η)/2, one has lim

N →∞ inf C∈C N,η,β P * C Y + ∞ (-3) < ∞ = 1.
Proof. Let N ≥ 1, η > 0, and β > ( β c (w) + 3ε)/2 be given. Consider C ∈ C N,ε,β , and define the following stopping times:

T 0 := inf{n ≥ 0 : W (Z n (-2)) ≥ W (N ) -η}, T 1 := inf{n ≥ T 0 : W (Z n (-2)) ≥ W (Z n (0)) -η/2}, T 2 := inf    n ≥ T 1 : W (Z n (-2)) ≥ W (Z n (0)) -η/4 or W (Z n (-2)) ≤ W (Z n (0)) -3η/4    , and
T 3 := inf{n ≥ 0 : W (Z n (-3)) ≥ W (N )/2 -β}.
The main steps of the proof are the following. First we will see that on the event when T 1 is infinite, Y + ∞ (-3) is finite, and thus the main part of the proof is to deal with the event when T 1 is finite. Now after time T 1 we know that the local time in -2 is large, specifically Z T1 (-2) ≥ N := W -1 (W (N ) -1), since η ≤ 1 by hypothesis. This ensures that the fluctuations of the martingale (M n (-1)) n≥0 after time T 1 are small, by Doob's L 2 inequality combined with the fact that the square of w is reciprocally summable by hypothesis on β c (w) and Lemma 4.1 (recall that β c (w) ≤ β c (w)). More precisely, we fix now some ε > 0, and we get that for N large enough,

P * C sup n≥T1 |M n (-1) -M T1 (-1)| ≥ η 10 ≤ 200 η 2 ∞ i=N 1 w(i) 2 ≤ ε. (13)
The next step is to see that necessarily at time T 1 the local time in -3 is also large, and thus that the fluctuations of the martingale (M n (-2)) n≥0 after T 1 are also small for N large enough; see ( 17) below. This is where the role of T 0 comes into play, since we show that the increment of the local time in -3 between T 0 and T 1 goes to infinity as N → ∞. We note that to prove this, we use ( 13). The last step of the proof is to see that with probability close to one T 2 is infinite, and furthermore that the increment of Y + n (-3) between times T 1 and T 2 is dominated by a series which appears in the definition of β c (w), which is why we need this quantity to be finite. This is also where the role of T 3 appears. We show that the process

U (n) := W (Z n (-3)) - W (Z n (-1)) 2 ,
remains bounded between times T 1 ∧ T 3 and T 2 , with probability close to one (and is also bounded up to time T 3 by definition).

Let us now proceed with the details of the argument. First observe that W (Z n (-2))-W (Z n (0)) ≡ Y + n (-3), and thus

{T 1 = ∞} ⊆ {Y + ∞ (-3) < ∞}. (14)
Therefore, one can assume now that T 1 is finite.

The next step is to show that the local time in -3 at time T 1 is large, and for this we show that its increment between T 0 and T 1 is large. Indeed, note first that since the process we consider is reflected in 0, one has for any n ≥ T 0 ,

Y + n (-1) -Y + T0 (-1) = W (Z n (0)) -W (Z T0 (0)
). In addition, (6) gives

Y + n (-3) -Y + T0 (-3) + Y - n (-1) -Y - T0 (-1) = W (Z n (-2)) -W (Z T0 (-2)).
Assume that N = W -1 (W (n) -1) is large enough so that w(N ) ≥ 6/η. By definition of T 0 and T 1 , this implies

W (Z T1 (-2)) -W (Z T0 (-2)) ≥ W (Z T1 (0)) -W (Z T0 (0)) + η 3 .
Then it follows from the last displays and ( 13) that for N large enough,

P * C Y + T1 (-3) -Y + T0 (-3) ≤ η 6 ≤ P * C |M T1 (-1) -M T0 (-1)| ≥ η 6 ≤ ε. (15) Define next N := inf{n ≥ N : W (n) -W (N ) ≥ η 6 }.
Since w is nondecreasing, and since we recall that by definition of T 0 , one has Z T0 (-2) ≥ N , it follows from ( 6) that ( 16)

Y + T1 (-3) -Y + T0 (-3) ≥ η 6 ⊆ {Z T1 (-3) -Z T0 (-3) ≥ N -N } .
However, N -N → ∞, and thus Z T1 (-3) → ∞ as well, when N → ∞. It follows using again Doob's L 2 -inequality, that for N large enough,

P * C sup n≥T1 |M n (-2) -M T1 (-2)| ≥ η 6 ≤ ε. (17)
The last step of the proof is to show that with probability close to one, T 2 is infinite and

Y + T2 (-3) -Y + T1 (-3) is finite. Let h(n) := Zn(-1) k=1 2p k -1 w(k) ,
where p k is the probability to jump to 0 at k-th visit to -1. Recall that

Y - n (0) -Y + n (-2) ≡ h(n)
, and on the other hand (6) and Lemma 2.1 yield

W (Z n (-1)) ≡ Y + n (-2) + Y - n (0), and Y + n (-2) ≡ Y - n (-2) ≡ W (Z n (-3)). As a consequence, U (n) = W (Z n (-3)) - W (Z n (-1)) 2 ≡ - h(n) 2 . Since Z n (-2) ≤ Z n (0), for all n ≤ T 2 , h is nondecreasing up to time T 2 . Note that U (T 1 ∧ T 3 ) ≤ -β + η/2, if N is taken large enough. Also by definition, sup n≤T3 U (n) ≤ -β.
Therefore by using ( 17), and again Doob's L 2 -inequality, we get at least for N large enough,

P * C sup T1∧T3≤n≤T2 U (n) ≥ -β + η ≤ ε. (18)
Remember then that H is defined by H(x) = x + W -1 (W (x) + 1), and thus by using the hypothesis on C, we get that for all

T 1 ≤ n ≤ T 2 , Z n (-1) ≤ Z n (-2) + Z n (0) ≤ H(Z n (-2)).
It follows that on the event {sup T1∧T3≤n≤T2 U (n) ≤ -β + η}, one has

Y + T2 (-3) -Y + T1 (-3) = T2 n=T1 1{X n = -3} w(Z n (-2)) ≤ T2 n=T1 1{X n = -3} w(H -1 (Z n (-1))) ≤ ∞ n=Z T 1 (-3) 1 w(H -1 (W -1 (2W (n) + 2β -2η))) . ( 19
) Using now that β ≥ ( β c (w) + 3η)/2, we can find K ≥ 1 such that ∞ n=K 1 w(H -1 (W -1 (2W (n) + β c (w) + η))) ≤ η 10 .
Then by using ( 15), ( 16), ( 18) and ( 19), we get that if N is large enough,

P * C Y + T2 (-3) -Y + T1 (-3) > η 10 ≤ 2ε. (20)
But by definition of T 1 and T 2 , on the event {T 2 < ∞}, we have for N large enough,

η/5 < {W (Z T2 (-2) -W (Z T1 (-2))} -{W (Z T2 (0) -W (Z T1 (0))} = {Y + T2 (-3) -Y + T1 (-3)} -{M T2 (-1) -M T1 (-1)}.
Therefore ( 13) and ( 20) imply

P * C (T 2 < ∞) ≤ 3ε, and P * C Y + ∞ (-3) -Y + T1 (-3) ≥ 1 ≤ 5ε.
Since ε > 0 can be chosen arbitrarily small, and since we recall that on the event {T 1 = ∞}, one has Y + ∞ (-3) < ∞, this concludes the proof of the lemma.

We can now finish the proof of Theorem 1.2. Fix some η ∈ (0, 1) and β > ( β c (w) + 3η)/2, and consider some initial local time configuration C, such that z 0 (-1) ≥ N , z 0 (0) ≥ N , and z 0 (x) = 0, for x / ∈ {-1, 0}, with N ≥ 1. Note that by definition C ∈ C N,η,β , and thus by Lemma 4.2 one has P * C (Y + ∞ (-3) < ∞) ≥ 3/4, for N large enough. Using the continuous time-line construction of the VRRW (also called Rubin's construction, see [START_REF] Tarrès | Localization of reinforced random walks[END_REF][START_REF] Basdevant | Localization on 4 sites for Vertex Reinforced Random Walk on Z[END_REF]), we can couple the process reflected in -3 and 0, say X, with the process reflected in -3 and 2, say X, and Lemma 3.6 in [BSS] (see also [T2] for a similar result) tells us that Y + ∞ (-3) ≤ Y + ∞ (-3). Applying this argument twice, we see that for N large enough, with probability at least 1/2, one has both Y + ∞ (-3) and Y - ∞ (2) finite. Then using Lemma 3.7 in [BSS], we deduce that for N large enough (say larger than some N 0 ) the unreflected C-VRRW on Z never visits sites -4 and 3, with some positive probability.

Then we can see that the same holds for the C 0 -VRRW, since for any N ≥ 1, with positive probability at time 2N , we have Z N (-1) = N -1, Z N (0) = N , and X 2N = 0, and one can then apply the previous result at time 2N . This proves in particular that |R | ∈ {5, 6}, with positive probability. Now it just remains to show that almost surely the walk visits only a finite number of sites. However, each time the VRRW on Z visits a new site x < 0, two cases may appear. If at this time the local time in x + 1 is not larger than N 0 , then the process has some positive probability (depending only on N 0 ) to jump immediately to x -1, and then to localize on the set {x -7, . . . , x -1} and never come back to x, by the above argument. If instead at this hitting time of x, the local time in x + 1 is larger than N 0 , then necessarily the local time in x + 2 has to be also not smaller than N 0 , and we deduce by using again the above argument, that the process has some (constant) positive probability to never visit x -2. Then the conditional Borel-Cantelli lemma (see Theorem 4.3.2 in [Dur]) shows that almost surely inf n X n > -∞. By symmetry we also get that almost surely sup n X n < ∞, and this concludes the proof of Theorem 1.2. 5. On the values of the parameters β c (w) and β c (w).

Let us first observe that for any nondecreasing w, and any λ > 0, one has β c (λw) = β c (w)/λ, and β c (λw) = β c (w)/λ (which follows from the facts that J β (λw) = J λβ (w)/λ, and J β (λw) = J λβ (w)/λ). Now our aim here is to convince the reader that in most cases (and we believe this is true in fact for any nondecreasing weight function), one has:

β c (w) = β c (w) ∈ {-∞, ∞}. (21)
On one hand we prove in Lemma 5.1 that this is true for any weight function growing at least linearly and not faster than n √ log n. On the other hand, we show in Lemma 5.2 that it holds as well for a large class of sublinear weights. Now recall that one can restrict our attention to weights satisfying (2) and such that α c (w) = ∞, since otherwise we already know the behavior of the process by the results of [BSS]. But it is also proved there that if lim inf w(n)/(n log log n) > 0, then α c (w) is finite; thus the upper bound on w, which is imposed in the hypotheses of Lemma 5.1 below is not a strong restriction.

Lemma 5.1. Let w be some nondecreasing weight function, such that

lim inf n→∞ w(n) n > 0. • If w(n) = o(n log n), then β c (w) = -∞. • If w(n) = o(n √ log n), then β c (w) = β c (w) = -∞. Proof. Assume first that w(n) = o(n log n). Let ε > 0, be such that w(n) ≤ ε 2 n log n,
W (t) + β = W -1 (2W (t)+β) t du w(u)
.

Thus using that w(n) ≥ εn, we get that for t large enough,

W (t) + β ≤ 2 ε log W -1 (2W (t) + β) t .
Combining this with (22), we get,

W -1 (2W (t) + β) ≥ t(log t) 1 4ε
, for all t large enough. Then by choosing ε < 1/8, and using again that lim inf w(n)/n > 0, the first assertion of the lemma follows.

Assume now that w(n) = o(n √ log n), so that for n large enough w(n) ≤ ε 2 n √ log n. Then for x large enough,

1 = W -1 (W (x)+1) x du w(u) ≥ 1 ε 2 W -1 (W (x)+1) x du u √ log u ≥ 2 ε 2 log W -1 (W (x) + 1) -log x .
Thus for x large enough (and ε small enough),

H(x) = x + W -1 (W (x) + 1) ≤ 2x • e ε 2 √ log x .
On the other hand, a similar argument as above shows that for any β ∈ R, for x large enough, W -1 (2W (x) + β) ≥ x exp( log x). In particular, by taking ε small enough, we get that for x large enough, H -1 (W -1 (2W (x) + β)) ≥ x(log x) 2 , and the second assertion of the lemma follows, using again that lim inf w(n)/n is positive.

Our second result is concerned with sublinear weights. Proof. By using a change of variables, we can write for any β > β , for some constant c > 0,

J β (w) = ∞ 0 dt w(W -1 (2W (t) + β)) = ∞ 0 w(W -1 (u)) w(W -1 (2u + β)) du ≥ c ∞ 0 w(W -1 (u)) w(W -1 (2u + β )) du = cJ β (w),
using the two hypotheses of the lemma. This implies that β c (w) ∈ {±∞}. Moreover, by ( 23), one has t ≤ H(t) ≤ Ct, for some constant C > 0, and all t. It follows using (24) that J β (w) ≤ CJ β (w), for some possibly larger C, and all β, and we deduce that β c (w) ∈ {±∞} as well. Now if there exists C > 0, such that w(n) ≤ Cn, for all n ≥ 1, then for any α > 0,

α = W -1 (W (n)+α) n dt w(t) ≥ 1 C W -1 (W (n)+α) n dt t = 1 C log W -1 (W (n) + α) n ,
which proves the second assertion of the lemma.

Let us conclude this section by mentioning that by combining the results of [START_REF] Basdevant | Localization of a vertex reinforced random walk on Z with sub-linear weight[END_REF] with our Theorem 1.1, and the previous lemma, we obtain that any nondecreasing weight function w, such that w(n) ∼ n exp(-(log n) α ), for some α ∈ (0, 1/2) satisfies β c (w) = -∞. Indeed, we know from [START_REF] Basdevant | Localization of a vertex reinforced random walk on Z with sub-linear weight[END_REF], that for such weight localization on 5 sites occurs with positive probability. Then Theorem 1.1 shows that β c (w) is finite, and finally Lemma 5.2 gives that in fact β c (w) = β c (w) = -∞. On the other hand, when w(n) ∼ n exp(-(log n) α ), with α > 1/2, the results of [START_REF] Basdevant | Localization of a vertex reinforced random walk on Z with sub-linear weight[END_REF] show that J 0 (w) = ∞, which imply β c (w) ≥ 0. Observing that w(n) = O(n), and applying Lemma 5.2 gives β c (w) = β c (w) = ∞.

  Lemma 5.2. Let w be some nondecreasing weight function satisfying (2). If the two following conditions hold:lim sup W -1 (W (n) + α)/n < ∞, for any α > 0, (23)and lim sup w(cn)/w(n) < ∞, for any c > 1, (24)then β c (w) = β c (w) ∈ {-∞, ∞}. Moreover, if w(n) = O(n), then (23) holds.

  and w(n) ≥ εn, for all n large enough. Then at least for t large enough, Now by definition, for any β ∈ R, and t large enough,

	(22)	W (t) ≥	1 2ε 2 log log t.

Acknowledgments: I warmly thank an associate editor for his many comments, which helped correct some mistake and improve greatly the readability of the manuscript.


Mayank Shekhar Jha

Philippe Weber

Didier Theilliol

Jean-Christophe Ponsart

Didier Maquin

A reinforcement learning approach to health aware control strategy

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

In the last decade, extensive research has been made in the domain of component-based Prognostics and Health Management (PHM). As almost all the industrial and mission critical systems operate in closed loop, new methods are being sought so that useful life of critical systems can be extended. In this context, health-aware control (HAC) has recently emerged as one of the domains where control synthesis is sought, based upon the state of health (SOH) and/or Remaining Useful Life (RUL) prognostics of critical components [START_REF] Brown | A prognostic health management based framework for fault-tolerant control[END_REF], [START_REF] Escobet | Health aware control and model-based prognosis[END_REF]. Such control paradigms should ensure the stability, while satisfying the desired health-based constraints, plausibly by trading off present system performance at the expense of future health conditions [START_REF] Kacprzynski | Prognostics-enhanced automated contingency management for advanced autonomous systems[END_REF]. Some of the existing methods are developed using reliability-based indicators such as [START_REF] Khelassi | Reconfigurability analysis for reliable fault-tolerant control design[END_REF]. For instance, [START_REF] Salazar | System reliability aware model predictive control framework[END_REF], [START_REF] Salazar Cortés | Reliability importance measures for a health-aware control of drinking water networks[END_REF] proposed reconfiguration strategies in model predictive control (MPC) framework. On the other hand, RUL prognostics is obtained using Degradation models (DMs) that estimate their parameters using actual measurements and adapt the degradation dynamics based upon individual asset. This leads to efficient RUL predictions even in presence of large individual to individual variations in large fleet of assets. The degradation models can be constructed based upon several criteria such as energy loss [START_REF] Rodriguez | An architecture for controlling the remaining useful lifetime of a friction drive system[END_REF], degradation physics [START_REF] Brown | A prognostics enhanced reconfigurable control architecture[END_REF] or statistical analysis of degradation data-base [START_REF] Jha | Particle filter based hybrid prognostics of proton exchange membrane fuel cell in bond graph framework[END_REF].

Reinforcement Learning (RL) based methods have seen a rapid surge in research mainly due to their ability to learn optimal policies (control laws) based upon interaction with the environment, in a model based as well as model-free setting. A general RL framework involves an actor agent (controller) that interacts with the environment (system or dynamic process) and produces a set of control actions. Then, based upon Authors are with Centre de Recherche en Automatique de Nancy (CRAN), UMR 7039, CNRS, University of Lorraine, 54506 Vandoeuvre Cedex, France (*corresponding author, e-mail: mayank-shekhar.jha@univ-lorraine.fr). stimulus (response or reward) received from the environment, it modifies the control actions, eventually leading to learning of optimal control policy (control law). Recently, problems belonging to various scientific domains have been addressed in RL setting including feedback control and optimal adaptive control [START_REF] Lewis | Reinforcement learning and adaptive dynamic programming for feedback control[END_REF]. RL has been applied in various engineering domains including robotics [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] and recently for fault tolerant control [START_REF] Ahmed | Comparison of Model Predictive and Reinforcement Learning Methods for Fault T olerant Control[END_REF]. RL based algorithms can be model based as well as model free. Typically, a Markov Decision Model (MDP) that represents the dynamics of the system is considered to generate the next state as well as reward, in order to learn the optimal control law. However, Q-function based RL algorithms are model-free in that the data obtained fro m process (as samples or trajectories) can be used to learn the inherent dynamics and the optimal control. This contrasts with the usual requirement of an analytical transition model (mathematical expression) in Control Community. Thus, RL based algorithms can be envisaged to provide answers for control synthesis problems in face of partial model knowledge, uncertainties or complete absence of system dynamics model.

For systems under degradation, the DMs of concerned components are usually available. For example, DMs of roller bearings are known to be exponential based upon offlin e processing of degradation data. However, exact knowledge of the global system (usually complex) is usually not available. Moreover, in order to generate health/RUL predictions, a suitable prediction model is required. Such mathematical dynamic (transition) models of RUL are rarely available which makes it difficult for RUL information to be incorporated into the control paradigm. This calls for novel methods that envisage integration of databases (data-driven methods) with model-based knowledge for control learning. In this paper, a novel method is proposed wherein model-based knowledge of system and component degradation, is integrated with RUL prediction data within Reinforcement Learning (RL) framework in order to obtain optimal control policy. To that end, the RUL predictions generated at each step is tracked to a desired value of RUL and the latter is integrated within a cost function (reward), which is maximized to learn the optimal control. Clearly, the scientific interest remains in developing new RL based methods for rendering intelligence to HAC. This is a novel attempt towards integrating model-based approach with model-free database (RUL predictions) to learn the optimal control for HAC. Moreover, it is novel in that this methodology renders intelligence (learning ability) for prognostics-based control. This section is followed by description of problem formulation. Subsequently, a framework is proposed. Finally, simulation results are presented followed by conclusions

A Reinforcement Learning Approach to Health Aware Control Strategy

Mayank S Jha*, Philippe Weber, Didier Theilliol, Jean-Christophe Ponsart, Didier Maquin

II. PROBLEM FORMULATION

The optimal control learning problem can be formalized by using Markov Decision Process (MDP) model [START_REF] Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF] of the dynamical system. The deterministic setting is considered in this paper with X being state space of the process, U being the action space, s f being the state transition function.

Consider a general nonlinear system affine in control in discrete time k:

1 ( ) ( ) k s k k k x f x g x u + = + (1)
where s f describes system evolution (plausibly non-linear)

and g describes the input evolution, with system variables k x X , control input k u U . Then, the control policy ( ) : h X U → is defined as function from state space to control space as:

( )

k k u h x = (2)
Depending on the control action k u , the state transits to 1

k x +
and receives a scalar reward

1 k r + generated by reward function ρ : x X U → as: 1 ρ( , ) k k k r x u + = (3)
Equations ( 1) and ( 3) along with control policy, define the MDP. It is highlighted that given k x , k u , f and ρ , the next In Fig. 1, the global architecture is shown along with the module containing system dynamics and degradation model module.

Assumption 1: System variables including those sensitive to degradation are considered observable (estimated).

Assumption 2:

The system is considered stabilizable on some set x X which implies that there exists a admissible control policy ( )

k h x , such that the closed loop system 1 ( ) ( ) ( ) k s k k k x f x g x h x + = +
is asymptotically stable on .

A. RUL Prediction Module

The degradation dynamics is not embedded into the system dynamic model( 1), but processed in a separate computational block which considers the degradation of critical component/subsystem (chosen a priori). Clearly, this formulation is context/system dependent. It is motivated by the fact that DMs are usually statistically derived using offline controlled environment tests (non-destructive testing) and/or obtained from identification/processing of features in temporal/frequential feature-space. The degradation is considered for a subsystem/component as:

1 ( , , ) k d k k d f d m x + = (4)
where d D represents any degradation index that evolves according to function (plausibly non-linear) D f , depends on a set of known time invariant parameters (plaus ibly estimated beforehand) m M and a set of system variables k x . The initial value of degradation index 0 d is considered known (plausibly estimated beforehand).

Assumption 3: Given that system variables sensitive to DM (4) are stabilizable, the DM is considered stabilizable over some set d D .

B. L-Step Ahead Prediction For RUL

Given fail D as the failure value of degradation index, the RUL at any discrete time instant k can be obtained using l-step ahead prediction. This is done by projecting the DM (4) into future over an infinite horizon [START_REF] Jha | Particle filter based integrated health monitoring in bond graph framework[END_REF], [START_REF] Daigle | Model-based prognostics with concurrent damage progression processes[END_REF], till fail D is reached.

Such a projection involves l-step iteration of ( 1) and (4) into future. In this work, we generate l-step ahead prediction at instant k, by assuming future control inputs , (0,.., )

k i u i l + same as k u . When k l fail d D + =
, the simulation is stopped, and number of steps taken to reach the future equals the RUL (in discrete time) at k. x

k E O L RUL T k Ts = - ( 5
)
where Ts is the sampling time. Thus, the desired RUL at time k,

* k
RUL can be expressed as:

* * x k E O L RUL T k Ts = - (6)
Thus, at any step k, k RUL can be calculated through l-step ahead prediction and * k RUL can be obtained from (6).

Availability of * k

RUL motivates formulation of the problem as an optimal RUL tracking problem. Remark: The knowledge of * EOL T is user based/mission- imposed, which can in turn be used to generate a desired trajectory from We formulate the as a Q-learnin g problem to learn optimal control that assures efficient RUL tracking.

D. Optimal RUL tracking

Given

, ,

k k k
x RUL u , a reward 1 k r + is generated as:

( ) * * 1 1 ( ) ( ) 2 ( , , ) T T T k k k k k k k k k k k k r xS x uR u R U L R U L PR U L R U L x RUL u + = - + + - - = (7)
where S, R and P are invariant positive semi-definite matrices of appropriate sizes. The reward function ( ) is quadratic in nature, inspired from Linear Quadratic regulator type cost function such that argument of ( ) are minimized as reward 1 k r + is maximized over stages. Thus, we seek to minimize the difference * ( )

k k
RUL RUL while assuring minimal energy consumption as well as desired system performance. It is noted that system (1), control policy (2) and reward [START_REF] Rodriguez | An architecture for controlling the remaining useful lifetime of a friction drive system[END_REF], constitute the MDP. Cumulative reward or return ( , , )

h k k k R x RUL u from system state k x , k RUL , following the control policy ( ) k k u h x = , is: 1 ( , , ) ρ( , , ( ))
h i k i k k k k i i i i i k i k R x RUL u r x RUL h x - - + = = = = (8)
where such that 0 1 is the discount factor over the infinite horizon. The return is maximized over an infinit e horizon which allows us to obtain stationary policies. The cost or value associated with a control policy is assessed by two types of 'value' functions: state value functions (V-functions) and state-action value functions (Q-functions) [START_REF] Watkins | Q-learning[END_REF]. Qfunction is defined with respect to both state k

x and action k u , as the function that generates return obtained by applying k u at state k x and following policy ( ) k h x thereafter. The Q- function then becomes:

1 1 1 ( , , ) ρ( , , ) ( , , ( ))
h k k k k k k h k k k Q x RUL u x RUL u R x RUL h x + + + = + (9)
where 1 k RUL + is produced by l-step ahead prediction by following the same policy ( )

h at next system state 1 k x + .
Bellman's equation for Q-function is:

1 1 1 ( , , ) ρ( , , ) ( , , ( ))
h k k k k k k h k k k Q x RUL u x RUL u Q x RUL h x + + + = + (10)
Further, optimal Q-function * Q is the one that maximizes Q- function for any policy, or, * ( , , ) max ( , , )

h u Q x RUL u Q x RUL u = U .
Bellman optimal equation for Q-functions can be obtained as: * * ' ( , ) ρ( , ) max ( ( , ), ')

s u Q x u x u Q f x u u = + (11)
Clearly, the optimal policy * h is the one that selects at each state, an action with largest optimal Q-value, or: ` [START_REF] Ahmed | Comparison of Model Predictive and Reinforcement Learning Methods for Fault T olerant Control[END_REF] Optimality: It has been shown that Bellman equations are fixed point equations, which translates to the fact that starting from an arbitrary 0 Q , a unique fixed point is found upon successive iterations in asymptotic sense as [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] is a contraction mapping with factor 1 in infinity norm; it converges to optimal * Q asymptotically as j → or

* * ( ) argmax ( , , ) u h x Q x RUL u = U
* * 1 || || || || j j Q Q Q Q + - -
Model-free Q-learning: In order to avoid the need of transition function fs, Q-iteration is performed in model free sense using only the observed state transitions and rewards, i.e., data tuples

1 1 1 ( , , , , ) k k k k k k
x RUL u x RUL r

+ + +
. The problem is cast in RL framework under class of model free value iteration algorithms. We use Q-learning algorithm which starts from an arbitrary initial Q-function and updates it as:

1 1 1 1 ' ( , , ) ( , , ) max ( , , ') ( , ) k k k k k k k k k k k k k k k k u current estimate updated estimate temporal difference Q x RUL u Q x RUL u r Q x R U L u Q x u + + + + = + + - (13)
where [0,1] k is the learning rate and the term inside bracket represents the temporal difference, i.e., difference between ,')

1 1 1 ' max ( ,
k k k k u r Q x R U L u + + + +
: the updated estimate of the optimal Q-value of ( , , )

k k k k Q x RUL u ,
k u k k u Q x RUL u exploitation) u u exploration = U ( 14
)
where (0,1) is the exploration probability at step k. In this paper, the epsilon-decay routine is considered such that with each iteration as: 1 (iteration) number of episodic play = .

III. SIMULATIONS We consider a second order DC (direct current) motor model as our system, obtained by discretizing a continuous model with sampling time Ts=0.01s.

k k k k k i i u + + - = + (15)
A simplified model of wear is considered to model localized wearing of shaft due to load and angular rotation. Using Archard equation , the wear rate Hw (m 3 s -1 ) is considered as function of shaft speed as first order differential of wear

where the wear coefficient Cw=0.1 m 2 /N, is assumed known. Thus, a simplified linear degradation model with respect to wear mechanism is considered with a known initial value. Using saturation, the shaft speed is bounded within . Clearly , system configurations are chosen in a way such that all significant variables are within similar scale-ranges for the ease of analysis and simulation.

Figure 2 Open loop characteristics Figure 2 shows the open loop characteristics obtained with input of 10V from one episodic simulation run, which consists of 1000 simulation steps (10s).

Failure value of wear rate is set to Hwfail = 0.02 (m 3 /s), which is attained at around 3.4s under open loop and full loading conditions, i.e. TEOL =3.4s. The objective is to reach a desired T*EOL=9s by learning a suitable control law. As sampling time T*EOL=9s corresponds to 900 simulation steps in time. Thus, the objective becomes learning suitable control law that allows Hw reach not more than 0.02(m 3 /s) at the end of 900 simulation steps, thereby producing 0 RUL at the end of 900 iterations. At

+ = - + + - k T k k k k k k k r i i u R U L R U L (17)
Clearly, the reward structure intends to incentivize (ten times more) the minimizat ion of term * ( )

k k
RUL RUL over the others. For example, a greater negative reward is given when desired RUL is NOT reached at any step. Thus, the agent (via Q-learning) learns to minimize this difference since maximu m of the return (cumulative reward) is considered during Q-learning. For Q-learning, a naïve tabular approach is adopted in this paper such that state-action pair associated with each Q-value is stored in a table. To this end, the state space of Q-function which comprises of state variables of the system and RUL, is partitioned into equal number of bins. Each of the Q-function variable, ,,,, )

k k k k k k x RUL u x RUL r + + +
were used to produce a reward and update the Q-function for all the states visited during that episodic run.

It should be noted that for an episode, at each time step, DC motor model plays the role of a real system by providing the state data (states considered observable), and DM generates the RUL prediction, followed by update of the Q-values (stateaction pairs) stored in a table.

A total of 3000 episodic runs (of 1000 simulation time steps each) were performed with initial values of variables remaining same for each episode (exploring-starts approach was not adopted). Convergence of rewards and Q-values for all states, was detected after approx. 1500 episodic runs as seen in Fig. 3. Fig. 4 shows the system response, degradation levels, the RUL under the control law learnt as well as the learnt control law itself (also see Fig. 6). Clearly, the degradation levels is restricted to Hwfail value at the end of 10s or 1000 simulation steps. Fig. 5 shows the zoomed profile of RUL. As shown, the control law tracks the desired RUL in such a way that approx. zero value is reached at the end of 9s of system functioning (900 simulation steps) leading to the approx. desired RUL.

The simulations were performed on Intel Xeon processor with 32GB of RAM and 3.0GHz of clock frequency. Also, the simulations were programed in Python 3.6.

V. CONCLUSION A novel framework for health aware control is presented in this paper where reinforcement learning is used to learn an optimal control policy in face of component degradation by integrating global system transition data (generated by an analytical model that mimics the real system) and RUL prediction data, in reinforcement learning framework. This leads to learning of suitable control that manages the speed of degradation in a way such that desired RUL is reached.

Clearly, the control is learnt in an offline way. Around 1500 episodic plays (dynamic simulations of 100 steps) were needed in order to converge towards optimal control. Once an appropriate control is learnt it can be applied in online manner.

Moreover, in absence of accurate dynamics of system, this can be achieved by storing system transition data and using experience replay in order to learn the control in offline manner. This paper assumes availability of observable states.

In further works, estimation of hidden states will be investigated. Moreover, this method needs to be developed to handle highly non-linear degradation process which are often stochastic in nature. A naïve tabular approach was adopted in this paper which supports the preliminary results.

The quality of control can be made better using efficien t function approximators for control learning. In future works, author intend to develop more efficient function approximation methods to handle large state spaces and complex system dynamics.

  be determined using the Markov property as the successive state 1 k x + only depends upon k x and k u . MDP model formalization enables us to take advantage of RL based methods for optimal control generation.

Figure 1

 1 Figure 1 Health aware control framework

1 1 ω

 11 [0,10 ]rad s --, output current as ( ) [0,0.1]A i t and input voltage is considered bounded as ( ) [0,10]V u t

  to 50 3 =125000 discretized spaces along the same axis. Then, the input space ( ) [0,10]volts u t is divided into 20 equal bins along second axis leading to a 2-D tabular space of size 125000 X 20. During each iteration of episodic play (simulation of 1000 steps), The Q-value associated with each state-action pair is stored and updated in tabular form. The discount factor 0.95 = is considered and 0.1 = is taken for the -greedy approach. The epsilon values decays with each episodic play (after one simulation run). The Q-values were initialized randomly and for each k within an episode (of 10 seconds

Figure 3

 3 Figure 3 Running Average of Rewards after each 10 iterations (10 episodic plays)

Figure 4 (Figure 5 Figure 6

 456 Figure 4 (a) & (b)System response, (c) degradation state profile, (d) RUL and (e) system input command learnt using the proposed Q-learning procedure