A Doat

F Pellé

A Lebugle

Europium-doped calcium pyrophosphates: Allotropic forms and photoluminescent properties

Keywords: Calcium pyrophosphate, Eu 2+, Eu 3+, Luminescence

In a search for new luminescent biological probes, we synthesized calcium pyrophosphates doped with europium up to an atomic Eu/(Eu+Ca) ratio of 2%. They were prepared by coprecipitating a mixture of calcium and europium salts with phosphate. After heating at 900 °C in air, two phases coexisted, identified as the β calcium pyrophosphate form and EuPO 4 . Heating near 1250 °C in air, during the β→ transformation, europium ions substitute for calcium ions in the * calcium pyrophosphate structure as demonstrated by the spectroscopic study. Europium ions with both valence states (divalent and trivalent) were observed in the samples. Following the synthesis procedure, partial reduction of Eu 3+ took place even in an oxidizing atmosphere. The 0.5%-doped compound could serve as a sensitive probe in biological applications. Depending on the excitation wavelength, the luminescence occurs either in the red or in the blue regions, which discriminates it from parasitic signals arising from other dyes or organelles in live cells.

Introduction

We have recently demonstrated that nanoparticles of biomimetic calcium phosphates with apatitic structure, doped with europium, can be used as luminescent biological probes. Internalization of such probes by living cells was followed by monitoring the europium fluorescence [1]. Among the various calcium phosphates, others with a different composition or structure might also find applications in this field, due to their improved optical or biological characteristics compared to that of the previously studied probes [1]. Calcium pyrophosphates constitute a wide family, which, to our knowledge, has been little explored, although their biocompatibility is well established [2].

Few studies on europium-doped pyrophosphates have been reported in the literature. Most of them concern pyrophosphates doped with divalent europium ions, whose luminescence occurs in the blue range. Wanmaker and Vrugt [3] obtained europium-doped calcium pyrophosphate by a solid-state process, at 1200 °C, but under reducing atmosphere. The synthesis was carried out mainly by a solid-state reaction between monetite CaHPO 4 , diammonium hydrogen phosphate and europium oxide. Slightly reducing conditions (0.1-2% hydrogen) afforded a Eu 2+ -doped compound. Nazarova [4] has indicated the possibility of synthesizing calcium or strontium pyrophosphate doped with Eu 2+ under air atmosphere and at high temperature, but did not provide any details on the experimental conditions.

Few studies have been devoted to pyrophosphates doped with trivalent europium. Pelova and Grigorov [5] reported the synthesis of europium-doped zirconium pyrophosphate (ZrP 2 O 7 :Eu) under air atmosphere and at high temperature. After heating at 900 °C, only Eu 3+ ions were observed in zirconium pyrophosphate, whereas after heating at 1100 °C, Eu 2+ ions were detected. The reduction from Eu 3+ to Eu 2+ ions thus occurred in zirconium pyrophosphate at high temperature, even under air atmosphere. The transformation became total only under slightly reducing atmosphere.

Such abnormal partial reduction of Eu 3+ at high temperature under non-reducing atmosphere has also been described by Chinese workers [6], [7], [8], [9], [10] and [11] in the case of borates and aluminates. Thus, Zeng et al. [6] reported that reduction of Eu 3+ to Eu 2+ only occurred in SrB 6 O 10 :Eu by heating under air atmosphere. In a similar way, Pei et al. [7] and [8] observed the reduction of Eu 3+ to Eu 2+ in Sr 2 B 5 O 9 Cl:Eu and SrB 4 O 7 :Eu by heating at high temperature (900 °C) under air. Peng et al. [9] also observed this reduction by heating at 1400 °C under air in Sr 4 Al 14 O 25 :Eu. Su et al. [10] and Liang et al. [11] obtained the same reduction by heating at 1000 °C under air in SrBPO 5 :Eu and CaBPO 5 :Eu. However, this reduction has not been reported in the case of calcium pyrophosphate.

We investigated here the doping of a calcium pyrophosphate [Ca 2 P 2 O 7 ] by europium ions and the influence of this doping on the phases obtained and luminescence spectra recorded. Dicalcium phosphate dihydrate [CaHPO 4 •2H 2 O] (DCPD) samples synthesized in the presence of europium ions were heated in air at temperatures where the β and forms of calcium pyrophosphates are stable. The different phases present in the heated samples were identified and related to the luminescence spectra observed. Special attention was given to the oxidation state (II or III) of europium ions. These results further the understanding of europium doping in calcium pyrophosphate.

Materials and methods

Synthesis

Europium-doped calcium pyrophosphates were obtained by gradually heating DCPD in the presence of europium ions at different temperatures. DCPD was synthesized at room temperature by double decomposition of ammonium phosphate and calcium nitrate. In order to obtain DCPD in which europium is intimately mixed, europium ions were added to the calcium solution at an atomic ratio R L =Eu/(Eu+Ca) between 0% and 2%, and the synthesis was carried out in a similar way.

Solution A contained a total of 70.6 mmol of calcium nitrate [Ca(NO 3 ) 2 •4H 2 O] (Carlo Erba) and europium nitrate [Eu(NO 3 ) 3 •6H 2 O] (Alfa Aesar) in appropriate proportions, dissolved in 100 mL of deionized water. Solution B contained 62.7 mmol of ammonium dihydrogen phosphate [(NH 4 )H 2 PO 4 ] (Carlo Erba) dissolved in 208 mL of deionized water and 4.2 mL of ammonia (11 M). Solution A was rapidly added to solution B at room temperature while stirring. After 5 min stirring, the precipitate was quickly filtered and washed successively with 167 mL of deionized water and 50 mL of ethanol. The gel obtained was freeze-dried.

In order to obtain β pyrophosphate, samples were progressively heated (4°/min) to 900 °C and left for 1 h at this temperature. All the heat treatments were carried out under air.

To obtain pyrophosphate, the previous samples were further heated at 1250 °C for 15 min under air, and then quenched in liquid nitrogen. In some cases, they were heated at 1275 °C for 30 min. Ca 3 Eu(PO 4 ) 3 was also synthesized using the method described by McCarthy [12]. After quenching in liquid nitrogen, the product was analyzed by X-ray diffraction (XRD) . This compound was prepared for the spectroscopic analysis to interpret the results, as described below.

Chemical analysis

Europium ion content was determined by Induced Coupled Plasma Atomic Emission Spectroscopy analysis (ICPAES) at the Service Central d'Analyse of the CNRS (Vernaison, France) (relative error: 3%). Calcium and europium ions were assayed together by volumetric titration [13] (relative error: 0.5%).

Orthophosphate ions were determined by colorimetry as phosphovanadomolybdenum ( ) (relative error: 0.5%). The (Ca+Eu)/P atomic ratio had a relative error of 1%.

X-ray diffraction (XRD)

XRD patterns were obtained with a CPS 120 INEL diffractometer using the K 1 radiation of a cobalt anticathode ( ).

Scanning electron microscopy (SEM) and energy dispersive analysis of X-rays (EDAX)

The powders were observed with a LEO 435 VP electron microscope. After deposition on an adhesive coating pasted on sample holder, the samples were metallized with silver during 1 min to prevent build-up of surface charge. Observations were carried out either by secondary electron detection, or by retro-diffused electron detection. The brightness produced by the latter increases with atomic number of the atoms. Europium-rich areas (high Z) can be differentiated by their clear appearance. EDAX was also carried out with the same apparatus.

Luminescence

UV excitation was provided either by a high-pressure mercury lamp equipped with a Wood filter or by a XBO 150 xenon lamp with wavelength selection using a HD20 Jobin-Yvon monochromator.

The selective excitation was provided by a tunable frequency dye laser (Coherent 599) pumped by a CW argon laser (Coherent Innova 300), Rhodamine 6G was used as laser dye. The fluorescence was dispersed through a HR1000 Jobin-Yvon monochromator equipped with a 1200 grooves mm -1 diffraction grating. The signal was detected by a RTC 56 TVP photomultiplier and amplified by a PAR 128a lock-in-amplifier with data acquisition on microcomputer. Appropriate filters were set in front of the spectrometer to eliminate parasitic signals from the excitation beam. All spectra were corrected for the response of the experimental set-up and filters.

Results

The composition of the samples before the heating steps was determined. Chemical analysis gave a (Ca+Eu)/P atomic ratio close to unity. Moreover, the Eu/(Eu+Ca) atomic ratio was in agreement with the theoretical atomic ratio R L , used in the synthesis.

The samples heated at 900 and at 1250 or 1270 °C were investigated in more detail.

Samples heated in the stability range of β calcium pyrophosphate, at 900 °C

The XRD patterns of the samples with 0% and 2% atomic Eu/(Eu+Ca) ratio, heated at 900 °C, were recorded (Fig. 1). For the undoped material (R L =0), all the observed reflection peaks corresponded to those of the β pyrophosphate Ca 2 P 2 O 7 phase ). Doping with europium ions (R L =2%) modified the XRD patterns, with additional reflections, and alterations in intensity of some calcium pyrophosphate peaks. A shoulder at 37.1° appeared, the intensity of the ( 202) reflection of the β pyrophosphate, observed at 32.3°, increased, and a low-intensity peak was detected at 55.5°. They correspond, respectively, to the ( 012), ( 200) and ( 212) reflections of europium phosphate EuPO 4 (JCPDS No. 25-1055) whose strongest reflection (( ), d=3.03) is hidden by the (008) peak of the β pyrophosphate at 34.5°. Moreover, a shoulder at 36.1°, and another low-intensity peak at 40.1° were also observed, attributed to the ( 0210) and ( 220) peaks of β tricalcium phosphate Ca 3 (PO 4 ) 2 (β TCP). The samples were observed by SEM. The β pyrophosphate phase, with R L =0%, appeared as large thin sheets (5-8 μm long) (Fig. 2(a)). The sample doped with 2% europium was observed by retrodiffused electron detection (Fig. 2(b)). A second phase in low proportion was seen as a clearer shade than the main β pyrophosphate phase, indicating the presence of europium in higher concentration, which was supported by the EDAX analysis. The EDAX spectrum of this second phase (Fig. 3(b)), indicated a high concentration of europium, whereas the main phase lacked europium (Fig. 3(a)). The luminescence excited in the UV at 253.7 nm or in the visible with the Argon laser line at 465.8 nm enabled observation of narrow spectral features in the visible range of the spectrum, between 570 and 720 nm, corresponding to electronic transitions within the 4f 6 configuration of Eu 3+ ions (Fig. 4). The emission spectrum is dominated by the 5 D 0 → 7 F 2 hypersensitive transition. Upon excitation at 253.7 nm (Fig. 4(a)), the relative intensities of the different transitions and the energy of the different components resembled those of pure EuPO 4 [15]. This is in agreement with the results obtained by XRD. However, on excitation at 465.8 nm, extra emission lines were observed (Fig. 4(b)). Most of these lines matched those in the spectrum of europium-doped β tricalcium phosphate reported by Lazoriak et al. [16]. From these data, the minor phases identified as EuPO 4 and/or Eu-doped Ca 3 (PO 4 ) 2 probably corresponded to the small europium-rich balls observed by SEM at the surface of the crystal. The optical data indicated that the europium ions were all in the trivalent state in this sample.

Samples heated in the stability range of calcium pyrophosphate, at 1250-1270 °C

Samples heated at 1250 °C

The XRD patterns recorded for the samples doped with 0% and 2% Eu/(Eu+Ca) atomic ratio, heated at 1250 °C, are represented in Fig. 5a andb. For the undoped sample (R L =0), only the peaks of pyrophosphate Ca 2 P 2 O 7 were observed. After introduction of europium ions (R L =2), apart from the main pyrophosphate phase, additional low-intensity peaks were observed. Most of them were characteristic of europium phosphate EuPO 4 as previously described. Another diffraction peak was observed at 36.2°, as well as an increase in the (002) reflection intensity of pyrophosphate, observed at 40.1°. These results were ascribed to the presence of β tricalcium phosphate Ca 3 (PO 4 ) 2 whose other main reflection ( 214) is superimposed on the (211) reflection of pyrophosphate at 32.4°. SEM observations of these samples showed that the crystallites were sintering. The sample with R L =2% was observed by retro-diffused electron detection (Fig. 6). A white phase is clearly visible in Fig. 6 in addition to the main grey phase, indicating the presence of europium in higher proportion, supported by the EDAX analysis. This minor phase was attributed to phosphate rich in europium.

Fig. 6. SEM picture (retro-diffused electron detection) of the sample heated at 1250 °C, with R L =2%.

Luminescence study of the sample containing europium (2 at%), after heating at 1250 °C was investigated. The fluorescence emission spectrum recorded at 300 K under non-selective excitation at 253.7 nm is represented in Fig. 7. A wide dissymmetric band was observed peaking at 415 nm which was ascribed to the allowed transition from the lowest component of the 4f 6 5d 1 configuration to the fundamental state 8 S 7/2 of the 4f 7 configuration of the Eu 2+ ions. This oxidation state of europium ions was indicated by the EPR data. The presence of Eu 2+ ions in a sample synthesized at high temperature and under non-reducing atmosphere (air) is of note. Narrow emission lines due to the 5 D 0 → 7 F J (J=0,1,2,3,4) transitions of Eu 3+ ions were also observed, ascribed to Eu 3+ ions in the EuPO 4 phase. However, in this case, excitation using the argon laser line at 465.8 nm produced a more complex emission spectrum revealing the presence of Eu 3+ in other positions in addition to that of the EuPO 4 ones (Fig. 8). The energies and the relative intensities of the additional lines corresponded to those reported by Lazoriak in europium-doped tricalcium phosphate [16]. The presence of two different phases identified by spectroscopy is in agreement with the XRD findings. Another peak of low intensity at 610.4 nm on both spectra should be noted. It will be assigned below. Furthermore, the influence of the heat treatment and its duration and of the europium concentration on the structure of the samples was investigated.

Influence of the heat treatment

The influence of the heat treatment was investigated by heating the sample doped with 2% europium at 1270 °C instead of 1250 °C during 15 min. Additional peaks apart from the pyrophosphate ones were detected by XRD (Fig. 5). In common with the sample heated at 1250 °C, they could be assigned to europium phosphate EuPO 4 and the βTCP phase. However, additional diffraction peaks were recorded and identified as a mixed phosphate Ca 3 Eu(PO 4 ) 3 (JCPDS No. 29-321) of eulytite structure. The appearance of this phase at this temperature is in agreement with the results of the study of Ca 3 Ln(PO 4 ) 3 (Ln=La-Gd) phases with eulytite structure reported by McCarthy et al. [12]. Ca 3 Ln(PO 4 ) 3 was found to decompose below 1250 °C into Ca 3 (PO 4 ) 2 and LnPO 4 (monazite). It is stable at the higher temperature and needs to be quenched in order to be observed.

Luminescence of the sample doped with Eu (2 at%) was dominated by an emission band peaking at 450 nm, ascribed to Eu 2+ ions. On increasing the temperature, the Eu 2+ emission intensity increased compared to that ascribed to Eu 3+ ions (Fig. 7). This was indicative of an increasing concentration of reduced Eu 2+ ions in the samples on increase in temperature of the heat treatment.

The influence of the duration of heating was studied on the sample with R L =2%. The sample was heated at 1270 °C during 30 min instead of 15 min. XR Diffraction showed a progressive disappearance of the β TCP phase, a decrease in amount of EuPO 4 , accompanied by an increase in Ca 3 Eu(PO 4 ) 3 (Fig. 5).

Effect of doping with europium ions

The influence of the europium content was studied on samples heated for 30 min at 1270 °C. The XRD patterns of the different compounds with R L =0.5%, 1% and 2% were recorded (Fig. 5). As noted previously, additional phases appeared next to pyrophosphate. A low-intensity peak at 36.2° was detected with 0.5% europium. From the previous results, it was attributed to β TCP. EuPO 4 and Ca 3 Eu(PO 4 ) 3 were detected with increase in europium concentration.

In addition, the half-width at half-maximum (HWHM) of the (211) of the pyrophosphate reflection peak was measured as a function of the europium concentration. It increased markedly with R L up to R L =0.5% (from 2.2.10 -3 to 2.7.10 -3 rad), and then exhibited a weak dependence on R L up to 2% (2.8.10 -3 rad). The increase observed between 0% and 0.5%, reflects either a reduction in size of the crystallites or a crystallinity with europium in the lattice. This suggests that europium is in the pyrophosphate structure. The slower evolution of the HWHM with increasing europium concentration for R L >0.5% was attributed to a substitution limit of europium in that structure.

On UV excitation (253.7 nm), the luminescence spectrum showed, at all europium concentrations, a wide dissymmetric band peaking at 415 nm as well as narrow emission lines between 570 and 720 nm as observed for samples heated at 1250 °C (Fig. 7). The relative intensities of the wide band and narrow lines were a function of the Eu 3+ concentration in the samples. For the lowest concentration, the fluorescence spectrum was dominated by the wide band peaking at 415 nm. On increase in the europium content, narrow lines observed in the visible region became the most intense (Fig. 7). This will be described in more detail below (Section 3.2.4). Nevertheless, the emission spectra for excitation at 465.8 nm on samples with increasing europium content, exhibited several peaks, assigned to EuPO 4 and Ca 3 Eu(PO 4 ) 3 (Fig. 8). The emission line observed at 610.4 nm was recorded in all the samples irrespective of the excitation wavelength.

Ratio between concentration of europium ions in divalent and trivalent state

The non-selective UV excitation enabled simultaneous excitation of europium ions in the divalent and trivalent states. The ratio between the integrated Eu 3+ and Eu 2+ emissions as a function of the nominal content of europium in the different samples could thus be determined.

The 5 D 0 emitting level was relatively insensitive to the multiphonon relaxation process since it was separated from the lowest 7 F 6 multiplet by a wide energy gap. Furthermore, if the concentration in the sample is low, cross relaxation between coupled ions are unlikely to occur with Eu 3+ . No energy transfer from Eu 2+ to Eu 3+ could be detected. The intensity of the intra 4f 6 configuration 5 D 0 → 7 F 1 (Eu 3+ ) dipolar magnetic transition is independent of the rare earth ion environment. Assuming the absence of non-radiative processes and interactions between ions, the 5 D 0 → 7 F 1 intensity can thus be taken as a reference [17]. From the ratio between the integrated intensity of the 5 D 0 → 7 F 1 (Eu 3+ ) transition and that of the Eu 2+ emission, it is possible to derive the ratio between the relative concentrations of europium in the divalent and trivalent ionization states (Fig. 9(a)). At low concentration, europium ions were present mostly in a divalent state. The ratio between Eu 3+ and Eu 2+ increased with increasing europium concentration as shown on Fig. 7. The increasing ratio between europium concentration in trivalent and divalent states up to the maximum substitution level for Eu 2+ , indicates that europium enters the lattice in a trivalent state filling the other sites as extra lines in the visible range and as EuPO 4 and Ca 3 Eu(PO 4 ) 3 phases. This was supported by the luminescence spectra recorded under UV excitation. From Fig. 10, Eu 3+ in EuPO 4 exhibits a characteristic emission line (noted as (1)). The line observed at 610.4 nm (noted as (2)) belongs to another site and is isolated from those of Eu 3+ in the EuPO 4 phase. The plots of the ratio of the intensities of each of these characteristic lines with integrated emission intensity of Eu 2+ ions is represented in Fig. 9 curves (b) and (c). It can be seen that the concentration of EuPO 4 increases with increase in europium content. The ratio between the emission intensity due to Eu 3+ site at the other centre (610.3) and that of Eu 2+ ions remained constant for different europium contents. This seems to indicate that the concentrations of Eu 2+ and Eu 3+ at the other site vary in the same way with increasing R L . This means that the other site belongs to the same phase. It was ascribed to trivalent europium in the pyrophosphate phase.

Spectral properties of Eu 2+ ions

The excitation at 254 nm produced a dissymmetric wide emission band at 415 nm (Eu 2+ ), in the same position and HWHM for the samples synthesized at 1270 °C with different europium contents. Mathematical decomposition of the Eu 2+ emission band, for the samples treated at 1250 and 1270 °C with R L =2%, with all free parameters (peak value, bandwidth and area) produced two

Gaussian curves peaking at about 23,716 cm -1 (421.7 nm) and 24,206 cm -1 (413.1 nm). Two crystallographic positions are present in pyrophosphate for calcium ions [18]. The presence of two components in the emission spectrum and their dependence on excitation wavelength stem from two different sites for Eu 2+ ions substituting Ca 2+ in the lattice. The relative intensity of both components to the total integrated emission spectrum calculated for the different samples was determined for the 253.7 nm excitation wavelength. The ratio between the two components calculated for samples heated at 1270 °C was not affected by europium content, whereas for the sample heated at 1250 °C, there was a greater contribution from the component peaking at 413.1 nm. For samples heated at 1270 °C during 30 min, there was an equal contribution of both components indicating that Eu 2+ ions split uniformly between the two calcium sites on increase in temperature and time, whereas one calcium site is preferentially occupied for the samples heated at 1250 °C.

Excitation spectra

The excitation spectra of europium ions in the samples treated at 1270 °C were recorded. The peaks of the Eu 2+ emission (415 nm) and the 5 D 0 → 7 F 1 transition of Eu 3+ ions (595 nm) were monitored with optimal excitation wavelenghts of 345 and 400 nm, respectively (Fig. 11). The luminescence of this sample could thus be observed in the blue or the red region of the spectrum depending on the excitation wavelength. This would represent a useful biological probe as it could be readily distinguished from other dyes.

Discussion

Heating dicalcium phosphate samples containing europium at 900 °C leads to β calcium pyrophosphate of tetragonal structure. SEM with retro-diffused electron detection showed the existence of a small proportion of europium-rich areas. These results can be explained by considering that small amounts of EuPO 4 are introduced in solid solution into calcium pyrophosphate during the crystallographic β→ transformation and due to active diffusion at this high temperature. To maintain local electroneutrality, Eu 3+ ions are sited in the vicinity of PO 4 3-ions. The luminescence line at 610.4 nm observed even at low Eu content, could be assigned to Eu 3+ ions in a particular environment within the calcium pyrophosphate phase.

This system is unstable and tends to relax, from an intracrystalline reaction, 2 PO 4 3-ions form P 2 O 7 4-ions:

2(PO 4 ) 3-→(P 2 O 7 ) 4-+1/2O 2 +2e -.

This reaction occurs for two reasons: P 2 O 7 4-will occupy a stable conformation in the crystallographic structure of calcium pyrophosphate, and two neighbouring Eu 3+ ions tend to be reduced into Eu 2+ .

2Eu 3+ +2e -→2Eu 2+ .

In order to find out whether this abnormal reduction of Eu 3+ to Eu 2+ occurred in other matrixes, Eudoped TCP was obtained by heating at 1350 °C under air, apatitic tricalcium phosphate prepared by coprecipitation in the presence of Eu 3+ ions. Luminescence studies showed that europium remained in the trivalent oxidation state: reduction into Eu 2+ did not take place in Eu-doped TCP.

The tricalcium phosphate structure does not appear to be suitable for such reaction, indicating that the matrix plays an essential role in the reduction process under non-reducing atmosphere. Nevertheless, the presence of PO 4 3-ions even as a secondary phase, also appears to be required for this reaction due to their ability to condense into P 2 O 7 4-and provide electrons for the reduction of Eu 3+ .

Finally, with increase in europium content, europium phosphate is still present as the previous reactions are only partial. This compound reacts between 1250 and 1270 °C with the previously formed β TCP leading to a mixed phosphate of eulytite structure.

Conclusion

We described here the different routes to europium-doped calcium pyrophosphate (from brushite) in the presence of europium ions. After heating at 1250 °C or higher, the calcium pyrophosphate phase contains both Eu 3+ and Eu 2+ ions. The presence of europium ions in a divalent state arises from a partial reduction of Eu 3+ occurring at high temperature, and stabilization by the calcium pyrophosphate structure. These compounds are of particular interest as they can exhibit a red or a blue luminescence depending on the excitation wavelength. As biological probes after mechanical grinding, they could be readily distinguished from other dyes and the autofluorescence of proteins and other components of living cells by changing the excitation wavelength.

Fig. 1

 1 Fig. 1. X-ray diffraction patterns of the samples heated at 900 °C, with R L =0% (a) and R L =2% (b).

Fig. 2 .

 2 Fig. 2. SEM picture of the samples heated at 900 °C, with R L =0% (a) and 2% (b) (retro-diffused electron detection).

Fig. 3 .

 3 Fig. 3. EDAX spectra of the sample heated at 900 °C, with R L =2%, carried out on the main phase (a) and on the minor phase (b).

Fig. 4 .

 4 Fig. 4. Luminescence emission spectrum of the sample heated at 900 °C, with R L =2%: (a) excitation at 253.7 nm, (b) excitation at 465.8 nm.

Fig. 5 .

 5 Fig. 5. X-ray diffraction patterns of the samples heated at 1250 °C, with R L =0% (a) and R L =2% (b); at 1270 °C during 15 min, with R L =2% (f); at 1270 °C during 30 min with R L =2% (e), 1% (d) and 0.5% (c).

Fig. 7 .

 7 Fig. 7. Luminescence spectra of the samples recorded under UV excitation at 253.7 nm. Sample heated at 1250 °C (a) 2% Eu, samples heated at 1270 °C (b) 2% Eu; (c) 1% Eu; (d) 0.5% Eu.

Fig. 9 .

 9 Fig. 9. Ratio of Eu 3+ fluorescence intensity taken as: ○ the 5 D 0 → 7 F 1 integrated intensity, • the intensity of line (1), ■ the intensity of line (2), with respect to Eu 2+ integrated intensity.

Fig. 11 .

 11 Fig. 11. Excitation spectra of the luminescence spectrum of the sample heated at 1270 °C (R L =2%). (a) Monitoring the Eu 3+ emission at 595 nm; (b) monitoring the Eu 2+ emission at 415 nm.

  Moreover, luminescence study indicated that divalent europium ions were located in the pyrophosphate and not in the β pyrophosphate. A partial reduction of Eu 3+ to Eu 2+ occurred in spite of the absence of reducing atmosphere. Eu 2+ ions progressively occupied the two calcium sites of pyrophosphate with increase in temperature and duration of the heat treatment. Some Eu 3+ ions however remained in the pyrophosphate structure.

	the following equation:
	3CaHPO 4 +Eu(OH) 3 →Ca 3 (PO 4 ) 2 +Eu(PO 4 )+3H 2 O
	or
	3/2Ca 2 P 2 O 7 +Eu(OH) 3 →Ca 3 (PO 4 ) 2 +Eu(PO 4 )+3/2H 2 O.
	On heating above 1200 °C, β calcium pyrophosphate turns into monoclinic pyrophosphate. X-ray
	patterns and SEM with retro-diffused electron detection showed the presence of minor additional
	phases. They correspond to Eu-doped βTCP and EuPO 4 . At higher temperature (1275 °C),
	Ca 3 Eu(PO 4 ) 3 becomes stable and the following reaction takes place:
	Ca 3 (PO 4 ) 2 +EuPO 4 →Ca 3 Eu(PO 4 ) 3 .

Luminescence study indicated the presence of europium phosphate EuPO 4 and europium-doped β tricalcium phosphate. The formation of tricalcium phosphate can be accounted for as follows: in the conditions of synthesis Eu(OH) 3 is formed, which on heating reacts with dicalcium phosphate CaHPO 4 or pyrophosphate according to It results in a decrease in β TCP and EuPO 4 depending on europium content, temperature and duration of heating.

Acknowledgements References

Acknowledgments

We would like to thank Mr. Thébault for the SEM observations and EDAX analysis.


Eric Baran

Theodorus Visser

Maud Cottet

Anne Tessier

Eric Guerin

Dr Jean Guillard
email: jean.guillard@inrae.fr

A Case Study of Fisheries Governance and Management in a Reservoir of Lao PDR: Perspectives for Improvement

Keywords: Lake, Asia, Fish resource, Management, Tropical

. Our study focuses only on a reservoir fishery, it does not include upstream or downstream fisheries. We set the stage by identifying the various components of a generic tropical reservoir fishery, and then document how much is known and has been done about each component in the case of Nam Theun 2. This highlights strengths and weaknesses in the knowledge base needed for the management of the system. Ultimately, the various degrees of control of managers over the fishery system and its components are identified.

Introduction

Given the increasing demand for electricity, hydropower is promoted to power supply in many developing countries [START_REF] Zarfl | A global boom in hydropower dam construction[END_REF]. In the Mekong basin in particular, massive hydropower development has occurred during the past two decades, at risk of compromising native fishery resources [START_REF] Ziv | Trading-off fish biodiversity, food security, and hydropower in the Mekong River Basin[END_REF]. However, dam development also results in the creation of reservoirs, providing new habitats for fishery development in Africa [START_REF] Ainsworth | A review of major river basins and large lakes relevant to inland fisheries[END_REF][START_REF] Clay | New reservoirs in Africa, 1980-2000[END_REF] India [START_REF] Sugunan | Reservoir fisheries of India[END_REF], South America [START_REF] Petrere | Fisheries in large tropical reservoirs in South America[END_REF] and Asia [START_REF] Silva | Reservoir and culture-based fisheries: biology and management[END_REF]. In Asia, Lao People's Democratic Republic (PDR) has a good case study with 32 hydropower reservoirs operational, 31 under construction and 56 planned [START_REF]Dams in the Mekong River Basin, Laos: commissioned, under construction and planned dams in[END_REF]. In this country, 40% of the fish production originates from reservoirs [START_REF] Phonvisay | An introduction to the fisheries of Lao PDR[END_REF]. Among Laos' existing dams, the Nam Theun 2 project (Figure 1) stands out because the development of a reservoir fishery was one of the objectives of an integrated development approach from the start [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF]. We review the Nam Theun 2 case study using a simple conceptual framework of a fishery as a whole to review achievements and gaps. The framework and the case study are aimed at informing options, needs and constraints for the development of sustainable fisheries in subtropical reservoirs. studies, mainly from international publications and technical reports, academic theses, and presentations at conferences. They are complemented with project planning documents, reports of the panel of experts who monitored the project, project reviews, and studies of fisheries in other Lao reservoirs. In total, 167 documents were reviewed and 61, the most accessible and synthetic are referenced.

Fisheries are social-ecological systems including fish and fishers, but also economic drivers as well as beliefs and behaviors [START_REF] Wiber | Participatory research supporting community-based fishery management[END_REF][START_REF] Pomeroy | Small-scale fisheries management: frameworks and approaches for the developing world[END_REF]. [START_REF] Townsley | A manual on rapid appraisal methods for coastal communities[END_REF][START_REF] Pido | A handbook for rapid appraisal of fisheries management systems. Manila, Philippines[END_REF] assessment frameworks serve as a basis for the present analysis (Figure 2). The fish supply, or yield, results from the interaction between fishers (number, strategies), their fishing effort and the fish stock. Put simply, the stock reflects species (local biodiversity) and their environment, the latter being a combination of water quality, limnological parameters and habitats. Similarly, the fishing effort reflects the number of boats, the number and nature of fishing gears and the intensity of activity. Fishers and their effort are influenced mainly by the level of organization, access to credit and alternatives outside the fishery sector. "Fishing effort" could be seen as a subcomponent of the "Fishers" component, but for clarity purposes, we distinguish social aspects (motivations, organizational modalities, livelihood alternatives, etc.) from purely technical aspects such as gears or boat types. With the exception of a self-consumption fraction, particularly important in the Mekong basin [START_REF] Hortle | Consumption and the yield of fish and other aquatic animals from the Lower Mekong Basin[END_REF], the fish supply is entering the market. Fish marketing consists mainly of value adding, trade, infrastructure and buyers. Value adding corresponds to transformation and quality control. Fish trade is influenced by competition (or a monopoly situation), and access of traders to credit. Infrastructure plays a key role, in particular landing sites and market places, transport options, and preservation options such as ice or smoking facilities. Last, buyers are the often neglected but ultimate drivers of the fishery, with the weight of individual preferences (for wild fish, for some particular species) and also the competition between different fish suppliers or with the aquaculture sector.

The policy environment, expressed through rules and regulations, applies to both fish supply and fish marketing; it defines and controls access rights, creates constraints (e.g., bans, taxes) or facilitates business (e.g., incentives, subsidies). The level of information about each component for the Nam Theun 2 fishery is detailed in the following sections.

The Nam Theun 2 Reservoir

The Nam Theun 2 hydropower dam is a 1070 MW trans-basin project releasing the turbined water from the Nam Theun 2 Reservoir into the Xe Bangfai River through a 27 km long downstream channel (Figure 1) [START_REF] Descloux | Five years of monitoring of zooplankton community dynamics in a newly impounded sub-tropical reservoir in Southeast Asia (Nam Theun 2, Lao PDR)[END_REF]. The Nam Theun 2 reservoir impounded in 2008 is shallow (8 m depth on average) and its surface area varies between 489 km 2 at full supply level to 86 km 2 in the dry season. Sixteen hamlets are established along the reservoir, a part from resettlement, with a total households of more than 2,000 in 2014 and around 85% of that population is involved in fishing [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF].

Fish Supply

Catch monitoring based on landings started in 2008 [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF] and has been regularly implemented since then. The catch varied between 6.2 kg/household/day in 2009 just after impoundment and 2.8 kg/ household/day in 2012-2013, with a rebound after 2013 (4.3 kg/ household/day in 2015; [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]. The annual yield varied between 1027 tons the year following the impoundment and 576 tons in the three following years, then increased again to about 1300 tons in 2016 [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]. The latter corresponds to a low productivity of 27 kg per hectare related to the oligo-mesotrophic status of the reservoir [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF]. Currently the production results from naturally established populations and there is no institutional stocking nor aquaculture program. Overall, the fish component of the Nam Theun 2 reservoir fishery has been subject to extensive monitoring and is well documented [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF][START_REF] Tessier | Low input of offshore areas to fisheries in a large tropical reservoir in Lao PDR[END_REF].

Fishers

Fishers as a social group were mainly characterized through socioeconomic studies linked to resettlement [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF][START_REF] Tessier | Fish population of Nam Theun 2: stock, fisheries and life history traits of the main species. Thonon-les-Bains, France[END_REF]. In 2015, among the 2384 households around the reservoir, 71% were eligible to register as fishing households [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]. Out of these 1682 households, 27% registered to Village Fishing Groups, and 301 households identified themselves as full-time commercial fishers [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]; only 16% of the population entitled to buying a commercial fishing license actually bought one. Despite access limited to local markets, fisheries account for about 38% of resettled communities' source of income [START_REF] Mcdowell | Lao People's Democratic Republic Nam Theun 2 multipurpose project: twenty-fifth report of the international environmental and social panel of experts[END_REF], more than agriculture (20%) that failed to become the main income component [START_REF] Ando | Who is responsible for decision-making in participatory development? -Lessons learned from the social reconstruction plan for the Nam Theun 2 dam project[END_REF]. Fishing is a part-time activity whose intensity varies depending on cost-benefit considerations (in particular the price of fuel) and in relation to alternative income generating activities. Thus, although fishing is repeatedly recognized as a main source of income for villagers living by the reservoir, in particular vulnerable and poor households [START_REF]Nam Theun 2 annual update: project progress during 2013[END_REF], the activity is in fact marginally profitable and constrained by external factors unrelated to the resource. For these reasons, livelihoods based on fisheries tend to be limited to a small number of households characterized by enough means and knowledge to get enough catches, while for the majority this activity only provides occasional income and a supplemental source of protein. Beyond an evaluation of community engagement in fisheries management [START_REF] Phonesavath | Evaluation of community engagement in Nam Theun 2 hydropower project in Lao People's Democratic Republic[END_REF], no social or anthropologic analysis detailed fishers' ambitions (e.g., preferred livelihood options in the long term, investment preferences), priority needs (e.g., subsistence vs. income, ice supply vs. transport improvement), opportunities (such as fishery organization types and modalities) or constraints (for instance access to credit) in relation to the sustainability of the activity. Multiple studies have highlighted the importance of fish in the nutrition and food security of rural populations in Laos [START_REF] Hortle | Consumption and the yield of fish and other aquatic animals from the Lower Mekong Basin[END_REF]. Most resettled households in Nam Theun 2 reservoir fish, at least partly, for subsistence [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]. Nam Theun 2 Power Company [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF] mentions a self-consumption of around 60 kg per person per year, and more recent estimates (unpublished NTPC annual fisheries reports) value fish consumption around the reservoir at 33 kg per person and per year. In 2016, with a population around the reservoir reaching 11,000 people, a total registered catch of 1,347 tons and 17.5% of self-consumption (12.5% fresh and 5% processed), self-consumption could be valued at 21 kg/person/year.

Fishing Effort

Fishing effort was monitored from November 2008 to June 2018 and reported in NTPC's annual Fisheries Reports. Daily catches were related to boat type and their number (from 610 to 646 registered boats). Overall, the number of registered fishing boats decreases, whereas the catch went up between 2013 and 2016. One major constraint in assessing the actual fishing effort lies in the fact that i) only residents of the displaced villages are eligible for a fishing license and ii) only eligible fishing households are allowed to register fishing boats. Non-eligible households' fish without a license or have boats registered in the name of eligible households, and eligible households do not always register all of their boats. Thus, it is believed, based on independent assessment and samples [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF], that only about four out of seven commercial fishing households are registered. This situation reflects the low intensity of law enforcement and the limited risk of being fined when fishing without a license. Fishing is also acknowledged in protected zones, with a correlation between years of higher catch and of poor enforcement in these zones [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF]. Overall, regulation of fishing effort in the Nam Theun 2 reservoir has been recognized weak, with a high potential for improvement [START_REF] Mcdowell | Lao People's Democratic Republic Nam Theun 2 multipurpose project: twenty-fifth report of the international environmental and social panel of experts[END_REF].

Fish Surveys

The reservoir fish stock and its dynamics have been assessed by scientific fish survey monitoring combined with hydro-acoustic surveys [START_REF] Tessier | Low input of offshore areas to fisheries in a large tropical reservoir in Lao PDR[END_REF][START_REF] Tessier | Fish population of Nam Theun 2: stock, fisheries and life history traits of the main species. Thonon-les-Bains, France[END_REF]. Monitoring started in 2008 prior to reservoir impoundment and used experimental gillnet surveys [START_REF] Cottet | Fish population dynamic in the newly impounded Nam Theun 2 Reservoir (Lao PDR)[END_REF]; lengthweight relationships and life traits were also collected [START_REF] Tessier | Length-weight relationships of eight Asian freshwater fish species in Nam Theun 2 Reservoir (Laos PDR)[END_REF][START_REF] Tessier | a) Life history traits and exploitation of Hampala barb (Hampala macrolepidota -Cyprinidae) in a subtropical reservoir (Lao PDR)[END_REF][START_REF] Tessier | Life history traits of the exploited Nile Tilapia (Oreochromis niloticus -Cichlidae) in a subtropical reservoir (Lao PDR)[END_REF]. Results illustrate a typical three-phase process [START_REF] Ba | From Farmers to Fishers: Developing Reservoir Aquaculture for People Displaced by Dams[END_REF], with a trophic explosion and the proliferation of some species (2008-2009), followed by a rapid trophic depletion within a few months (2009-2010) and then a stabilization with a reduced variability and the dominance of very few species: five species make up to 94% of the total catch [START_REF] Cottet | Fish population dynamic in the newly impounded Nam Theun 2 Reservoir (Lao PDR)[END_REF]. Two years after impoundment, the reservoir was characterized by a low annual fish production of about 20 kg/ha [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF]. This productivity is the maximum expected; ensuring fishery sustainability would mainly consist in securing such production level. Using hydro-acoustic data, the fish stock was estimated in the deepest zones of the reservoir and showed a very low fish biomass of around 1 kg/ha [START_REF] Tessier | Low input of offshore areas to fisheries in a large tropical reservoir in Lao PDR[END_REF].

Fish Diversity

Fish biodiversity studies started in 1996 during the environmental impact assessment and the discovery of several dozen new species [START_REF] Kottelat | Fishes of the Nam Theun and Xe Bangfai basins, Laos, with diagnoses of twenty-two new species (Teleostei: Cyprinidae Balitoridae, Cobitidae, Coiidae and Odontobutidae)[END_REF][START_REF] Kottelat | The fishes of the Nam Theun and Xe Bangfai drainages, Laos[END_REF]. Thus, taxonomic surveys triggered by dam construction resulted in an explosion of fish biodiversity records. Within the reservoir, [START_REF] Cottet | Fish population dynamic in the newly impounded Nam Theun 2 Reservoir (Lao PDR)[END_REF] and [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF] showed that the original riverine fish community evolved after impoundment towards a lentic pattern, the formerly dominant native high value river species being replaced by two fast-growing limnophilic low value species, an ubiquist Barb (Hampala macrolepidota) [START_REF] Tessier | a) Life history traits and exploitation of Hampala barb (Hampala macrolepidota -Cyprinidae) in a subtropical reservoir (Lao PDR)[END_REF] and the non-indigenous opportunistic Tilapia (Oreochromis niloticus) [START_REF] Tessier | Life history traits of the exploited Nile Tilapia (Oreochromis niloticus -Cichlidae) in a subtropical reservoir (Lao PDR)[END_REF].

Environmental Parameters

The environment of the reservoir has been subject to multiple studies [START_REF] Descloux | Five years of monitoring of zooplankton community dynamics in a newly impounded sub-tropical reservoir in Southeast Asia (Nam Theun 2, Lao PDR)[END_REF], starting with a hydrodynamic and water quality model based on the monitoring of 15 parameters [START_REF] Chanudet | Application of a three-dimensional hydrodynamic model to the Nam Theun 2 Reservoir (Lao PDR)[END_REF]. A 3D model allows assessing the impact of watershed management on the primary productivity of the reservoir [START_REF] Chanudet | Evolution of the physico-chemical water quality in the Nam Theun 2 Reservoir and downstream rivers for the first 5 years after impoundment[END_REF][START_REF] Chanudet | Hydrodynamic and water quality 3D modelling of the Nam Theun 2 Reservoir (Lao PDR): predictions and results of scenarios related to reservoir management, hydrometeorology and nutrient input[END_REF]. Plankton monitoring indicates the trophic status of the reservoir [START_REF] Descloux | Five years of monitoring of zooplankton community dynamics in a newly impounded sub-tropical reservoir in Southeast Asia (Nam Theun 2, Lao PDR)[END_REF][START_REF] Martinet | Phytoplankton functional groups for ecological assessment in young sub-tropical reservoirs: case study of the Nam-Theun 2 Reservoir, Laos, South-East Asia[END_REF], but results have not been used to predict the evolution of fish stock. Despite the remaining terrestrial vegetation (mainly hard wood, the labile organic matter was decomposed in the first five years after impoundment) the reservoir is meso-oligotrophic, with a limited nutrient concentration, in particular in phosphorus. This evolution and apparent stabilization pattern was rapid when compared to that of other newly created tropical reservoirs [START_REF] Cottet | Fish population dynamic in the newly impounded Nam Theun 2 Reservoir (Lao PDR)[END_REF]. Ten reservoir habitats were monitored to assess the relationships between fish catches and habitat characteristics [START_REF] Cottet | Fish population dynamic in the newly impounded Nam Theun 2 Reservoir (Lao PDR)[END_REF]. Studies conclude that dense submerged forests are favorable to fish, with a number of species and of individuals at least twice higher than in other habitats. [START_REF] Tessier | Fish population of Nam Theun 2: stock, fisheries and life history traits of the main species. Thonon-les-Bains, France[END_REF] determined that inundated habitat has a positive influence on the fish stock, as long as these habitats remain protected from fisheries activities. The reservoir fish stock also benefits from peripheral habitats such as upstream rivers and transition zones of special interest for conservation and resource protection [START_REF] Cottet | Fish population dynamic in the newly impounded Nam Theun 2 Reservoir (Lao PDR)[END_REF].

Fish Marketing

Value adding includes all operations aimed at increasing the fresh fish marketability, such as processing, hygiene and quality control, packaging and labelling. In Lao PDR, processing fish mainly consists of drying fish for trade and making fermented fish sauce for selfconsumption [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF]. Fish processing groups were created in 2015 at Nam Theun 2, with the aim to increase income from the fishery, and new processed products such as fish powder as protein supplement for children were tested [START_REF] Ntpc | Environment and Social Monitoring Report: Annual Progress Report[END_REF]. However, many species have a higher value fresh than processed [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]. Around the reservoir, only 5-10% of the reported catch is processed [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF]. Thus, fish processing is valued only when freshly caught fish cannot be sold, or when there is an oversupply. In a context of challenging fish trading, quality control in relation to hygiene, packaging and labeling standards may increase the value of fishery products and contribute to the economic sustainability of the fishery. However, no quality control was put in place at Nam Theun 2 -even though Village Fishing Groups recently focused on hygiene improvement with for example installation of water system at the fish processing group facilities. The monitoring of landing sites allowed quantifying the quantity, proportion and value of the yield sold, as well as the amount kept fresh or processed -even though that information was never published. Thus, between 2012 and 2016 the number of landing sites decreased by 40%, but the mean catches and the mean value per landing more than doubled. In particular, the value of tilapia went down but that of wild species increased.

Yet, several aspects of the fishery remain undocumented, in particular factors driving fish sale, marketing strategies or trade channels. For instance, informal discussions with fishers indicate that less desirable species are sold at landing sites while high value ones are sold elsewhere; that traders often provide fishing gear to fishers in exchange of exclusivity, and that fish prices are fixed by traders and vary daily, depending on the overall catch of the day. Traders themselves seem to reflect a bottleneck at the level of final buyers, as there is not enough demand for all the fish harvested to be sold at a valuable price. Overall, beyond monitored but little popular landing sites, not much is known about fish sold within communities, at village markets and to wandering fish traders.

Infrastructure (landing sites, ice plants, market places, storage places and access roads) is essential for fish commercialization and subsequently for the sustainability of a fishery. The lack of investment into basic infrastructure and facilities for local fishing communities is considered as one of the reasons for the failure of fishing cooperatives at Nam Ngum reservoir [START_REF] Sommano | Review of fisheries management and development in large hydropower reservoirs of Lao PDR (case study of Nam Ngum 1). Secondary Review of fisheries management and development in large hydropower reservoirs of Lao PDR (case study of Nam Ngum 1). Vientiane[END_REF]. Around Nam Theun 2 reservoir, each relocated hamlet had a fish landing site and market facilities constructed. Few years after the construction, activity varies between landing sites: four of them are operated yearlong while others are only functioning a few months per year [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF]. The relative success of a landing site is mainly influenced by the location in the vicinity of productive fishing grounds. Of the ice plants constructed by NTPC around the reservoir, only one is still functional and operated by the fish processing group of the village. In all other communities, traders who use ice buy it from the district town. Because fish are usually sold a few hours after being caught, there is limited interest in preservation systems. Roads to bigger markets are good, and transport is not an issue for most northern villages and landing sites. The road network is more limited in southern villages but the fish is mainly transported by boat to northern landing sites [START_REF] Ando | Who is responsible for decision-making in participatory development? -Lessons learned from the social reconstruction plan for the Nam Theun 2 dam project[END_REF]. The most significant constraint to transport reported is the price of fuel for boat engines. The marketability of a given fish species depends on its competitiveness in relation to other species, to similar species from closer origin or to aquaculture fish. In the case of Nam Theun 2, fish are subject to competition with fish from other inland fisheries (e.g., from Nam Ngum reservoir), with marine fish (e.g., horse mackerel) and with farmed fish [START_REF] Phonvisay | An introduction to the fisheries of Lao PDR[END_REF]. The disappearance of high value capture species such as snakehead (e.g., Channa striata) and catfish (e.g., Hemibagrus spp.) in landings and the over-reliance on low value species, especially tilapia and hampala barb, limits the profitability of the fishery. Tilapia for instance (73% of catches), is massively farmed along the Mekong River, with a constantly low price. Another issue is the relatively large size of Nam Theun 2 reservoir tilapia, when buyers prefer smaller sizes. While several market options were initially considered, including export to Thailand and Vietnam [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF], the current destinations remain within Laos, and nothing has been initiated nor conceptualized on this part of the fishery. Overall, trade networks and their contribution to the sustainability of the fishery have never been assessed; similarly, interventions in relation to final buyers and consumers have never been considered by any projects in Nam Theun 2 reservoir.

Environment Policy in Relation to Fishing

Administratively, fisheries are under the jurisdiction of the Province and District Agriculture and Forestry Offices (PAFO and DAFO) [START_REF] Visser | Review of regulatory framework for developing fisheries management plan in Nakai Reservoir. Vientiane, Lao PDR[END_REF]. At the local level, fishers must be members of a Village Fishers Group (FVG). Each group elects members to join the Reservoir Fishers Association (RFA), whose representatives are part of the Reservoir Management Coordination Authority (RMCA). DAFO, Resettlement Management Units, the Nam Theun 2 Power Company and the Watershed Management and Protection Authority (WMPA) are also part of the RMCA, ensuring collegial decision-making about fisheries [START_REF] Visser | Review of regulatory framework for developing fisheries management plan in Nakai Reservoir. Vientiane, Lao PDR[END_REF]. The Nam Teun 2 policies and operation are based on the Fisheries National law recently revised in 2016, with only a few exceptions concerning access, as detailed below.

The reservoir fishery was originally structured as a limited access fishery [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF]: exclusive fishing rights have been granted for 20 years, until 2028, to residents of villages bordering the reservoir, resettled households and the second generation of resettled households [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF]. Non-eligible fishers can buy a daily fishing license from either the Village Fishing Groups or the Reservoir Fisheries Association, for a maximum catch of 5 kg/day. In practice, eligibility is flexible, with frequent migration, marriage/divorce and unclear village records of residents. It is also easy to bypass legal requirements for boat registration and membership to village fishing groups. Actually, the number of fishing licenses has steadily declined over the years. Developing the fishery was seen as a way to provide an alternative livelihood and improve income among villagers displaced by the dam project [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF]; this is why fishery access rights are restricted to villagers around the reservoir. Since the creation of the reservoir, policies have been driven by the intention of sustaining or improving the income of fishers. Policy facilitation also included the involvement of Village Fishers Group and of a Reservoir Fishers Association in reservoir management. Acknowledging that only 9% of households originally owned a boat, between May-November 2008 the NTPC resettlement program provided the 1310 eligible households with 660 fishing boats, about 7 meters long, mostly shared between 2 households, as well as a set of fishing gear to each household. The budget also covered construction of Reservoir Fisheries Association offices, construction of ice plants, improvement of road access to landing sites, creation of and financial support to the Reservoir Fisheries Association, establishment of revolving funds and training in fishing and processing [START_REF] Visser | Nakai reservoir fisheries status report. Vientiane, Lao PDR[END_REF]. A Watershed Management Protected Area (WMPA) was established in 2008 and Village Protected Areas were established in 2013-2015. Fishing is banned in these protected areas [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF] and at the connection between three north-east tributaries and the main reservoir (Figure 1).

Zoning was defined by the RMCA and enforcement was expected through peer control, i.e. by fishers themselves -although the risks of such approach were also foreseen [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF]. To date, fishers keep operating in protected areas [START_REF] Cottet | Fish catch and fishing practices in the Nam Theun 2 Reservoir and watershed (Lao PDR)[END_REF][START_REF] Visser | Fisheries status in Nakai reservoir[END_REF], fish protection areas remain to be established [START_REF] Mcdowell | Lao People's Democratic Republic Nam Theun 2 multipurpose project: twenty-fifth report of the international environmental and social panel of experts[END_REF] and access restriction enforcement remains a challenge, linked initially to insufficient patrolling [START_REF] Mcdowell | Lao People's Democratic Republic Nam Theun 2 multipurpose project: twenty-fifth report of the international environmental and social panel of experts[END_REF]. RMCA addressed in 2016-2018 by i) re-imposing manned checkpoints on the entry of selected conservation areas with high ecological significance and ii) improving the co-patrolling systems.

Policy Environment in Relation to Trade

In Nam Theun 2, the policies that facilitate or restrict fish commercialization consist of i) a legal framework (national, regional and communal legislations) and their local implementation; ii) local regulations (such as rules set up among Village Fisheries Groups), and iii) possible interference of local official or unofficial powers. The stability of regulations and to what extent they apply to all is another important parameter: in the case of the Nam Ngum reservoir fishery, the variable environment policy and changing practices of controlling fish market, fish prices and taxation system by the local provincial authorities are considered to have contributed to the failure of fishers' organization [START_REF] Sommano | Review of fisheries management and development in large hydropower reservoirs of Lao PDR (case study of Nam Ngum 1). Secondary Review of fisheries management and development in large hydropower reservoirs of Lao PDR (case study of Nam Ngum 1). Vientiane[END_REF].

In order to operate around the reservoir, fish traders need to register and pay an annual license fee to District Trading Office. Fifteen to twenty fish traders were initially expected and fish trade was not meant to be regulated [START_REF] Ntpc | Nam Theun 2 hydroelectric project, Social development plan. Vientiane, Lao PDR[END_REF]. However, in 2017, fish trade became regulated and licenses became only accessible to villagers living around the reservoir, which resulted in their number dropping.

Initially, facilitation by NTPC consisted of a credit scheme meant to help traders set up their business. That scheme covered main investment costs, e.g. access road improvement and provision of concrete slabs and shelter for fish trade. Assistance to the establishment of a fishing cooperative also involved in trading was also considered. In recent years, new initiatives were implemented to improve market accessibility to resettled communities with the aim of providing additional sources of incomes [START_REF] Mcdowell | Lao People's Democratic Republic Nam Theun 2 multipurpose project: twenty-fifth report of the international environmental and social panel of experts[END_REF][START_REF] Ntpc | Environment and Social Monitoring Report: Annual Progress Report[END_REF]. Rules and regulations around fish trade include ban to selling fish outside official landing sites and 7% taxes on transaction prices to the benefit of Village Fishing Groups and the Reservoir Fisheries Association. Actually, fishers and traders try to escape these taxes: the amount of fish sold at legal landing sites in 2016 is estimated at only 18-25% of the total amount sold, which compromises the funding and functioning of the above organizations [START_REF] Visser | Fisheries status in Nakai reservoir[END_REF][START_REF] Mcdowell | Lao People's Democratic Republic Nam Theun 2 multipurpose project: twenty-fifth report of the international environmental and social panel of experts[END_REF]. A much more aggressive repression of illegal marketing was considered, but in practice, some Village Fishing Groups refrain from collecting taxes because it would reduce the chance of traders showing up. Since monitoring transactions also consumes up to 80% of the tax proceeds, it is questionable if full legal trade would benefit households more than the current situation. Overall, the trade channels are less studied or known than the biological resource [START_REF] Phonvisay | An introduction to the fisheries of Lao PDR[END_REF]. In particular, the pros and cons of facilitation and constraint policy options have never been deeply discussed, and suggestions to liberalize trade, lift fishing licenses requirement and boat registration remain to be considered.

Discussion

The review of ten years of fisheries management at Nam Theun 2 illustrates how the multiple components of a fishery can be more or less covered by the promoters and managers of this fishery. What was not done and questions left unanswered are also aspects essential and are lacking for a comprehensive management for sustainability.

The connection between environmental studies and fishery management has mainly been through the identification of high ecological value areas to be protected. Beyond improving fish stock resilience, the social consequences of setting up conservation zones [START_REF] Stevenson | Socioeconomic consequences of fishing displacement from marine protected areas in Hawaii[END_REF], and the differential impact on the various villages around the lake has not been considered. Yet experience from Marine Protected Areas in Southeast Asia shows that this management approach can produce "successful" biological outcomes but less successful social outcomes [START_REF] Christie | Marine protected areas as biological successes and social failures in Southeast Asia[END_REF]; this calls for use of criteria measuring both biological and social success. Fishing effort, well monitored and documented, focused on registered boats, but field data indicates that close to 60% of the fishing effort originates from unregistered boats -which also affects knowledge of the fish supply. Regulation of the fishing effort remains a weak point. Annual harvest monitoring underline the evolution of the reservoir towards an oligo-mesotrophic status, with a naturally limited fish production. The social component of the fishery is well documented, although not published. The focus has been on socio-economics (income, fishing effort, etc.), in relation to the resettlement program of the company. However, there have been no studies about aspirations or constraints of the fishing community. Socio-cultural drivers of fishers' involvement or behavior are an important component of the sustainability of a fishery [START_REF] Barclay | The importance of qualitative social research for effective fisheries management[END_REF]. In the case of Nam Theun 2 reservoir, questioning the functioning of fisher associations could have flagged representativeness or ownership issues [START_REF] Rangel | Do fisher associations really represent their members' needs and opinions?[END_REF]. Researching why fishers do not comply with regulations (respective role of deterrence, moral support and authority [START_REF] Gezelius | Toward a theory of compliance in state-regulated livelihoods: a comparative study of compliance motivations in developed and developing world fisheries[END_REF][START_REF] Tran | Fishery livelihoods and (non-)compliance with fishery regulations-a case study in Ca Mau Province, Mekong Delta, Viet Nam[END_REF] as well as cultural and psychological dimensions (e.g., degree of concern about management for sustainability in a context of severe environmental modification, skepticism about benefits to be expected from access restriction in protected areas, etc. [START_REF] Carr | It's about seeing what's actually out there": Quantifying fishers' ecological knowledge and biases in a small-scale commercial fishery as a path toward co-management[END_REF] could have suggested management options closely matching fishers' perspectives. More generally, understanding better how changes in a fishery system influence social networks, community cohesion, culture and local institutions [START_REF] Katikiro | Interplay between perceived changes in fishery and social structures in Tanzanian coastal fishing communities[END_REF] could have helped better address villagers' adaptation challenges. Consumption of fish by fishing families ("self-consumption") has been a strong argument in favor of fishery development and nutritional improvement. This theme was covered in several technical reports, but the project has not produced any specific study on this point. Fish prices and the value of the fish officially traded have received specific attention. However, a detailed study of the dominant trading outside official channels, of the demand variability, of trade drivers or of arrangement between fishers and traders remains to be undertaken. More generally, the role of middlemen is usually overlooked, yet experience from marine fisheries shows that credit arrangements, reciprocal agreements and financial guarantees also have stabilizing social effects and contribute to sustainability [START_REF] Crona | Middlemen, a critical socialecological link in coastal communities of Kenya and Zanzibar[END_REF][START_REF] Ferrol-Schulte | Patron-client relationships, livelihoods and natural resource management in tropical coastal communities[END_REF]. Infrastructure development received special attention (investment, documentation). However, it seems that economic factors limit the demand for processing, and this aspect has not been studied. Similarly, the demand for quality control, ice facilities and transport improvement has been limited. Activities have focused on the construction of landing sites and market places. Since the fishing pressure can be predominantly driven by cities and market access [START_REF] Phonvisay | Monitoring of fish markets in Vientiane and Luang Prabang Province, Lao PDR, using logbooks[END_REF], it is important to understand social-ecological linkages in the fishery. Importantly, the demand from buyers has not been subject to any study nor strategic analysis. However, serious constraints apply at this level, in terms of fish species preferences, competitive price per species or competition with other sources of fish including aquaculture and imported marine fish. All these constraints have had a substantial influence on the success and sustainability of the fishery. On the policy side, the regulatory environment in relation to fishing is substantial but sparsely documented, with emphasis in facilitation initiatives (provision of 600 boats, etc.) and access restrictions. However, an analysis of the pros and cons of the various regulatory measures, in a context where enforcement is recognized as a major challenge, remains to be done. The same comments apply to the regulatory environment in relation to fish trade, and the latter is hardly documented. The efficiency of the current policy framework remains disputed, and simplification has been recommended by several sources: the current policy complexity and its implementation challenges do not seem to guarantee the sustainability of the reservoir fishery. The above elements are summarized in Figure 3. [START_REF] Cochrane | A fishery manager's guidebook[END_REF] underline the need to define, for a fishery, quantitative objectives guiding adaptive management (harvest, income, nutrition status or employment of targeted beneficiaries) -which was not developed at Nam Theun 2. [START_REF] Marasco | The role of objectives in fisheries management[END_REF] note that management objectives are ultimately political, and acknowledge that management bodies generally do not establish clear objectives. However, the need to set biological and social objectives to management decisions is widely recognized [START_REF] Mardle | Objectives of fisheries management: case studies from the UK, France, Spain and Denmark[END_REF][START_REF] Brooks | Selecting and assessing social objectives for Australian fisheries management[END_REF]. The challenge in setting measurable and realistic objectives is acknowledged, but the lack of clear biological, social or economic targets is a major risk in developing a fishery. Overall, sustainable management does not require the same high degree of knowledge about each component of the system, the information need being proportional to the degree of decision -making and control over a given component. A map of the extent of possible influence of the Nam Theun 2 fishery managers is proposed below (Figure 4). Nuances reflect the degree of freedom of the fishery actors and the technical feasibility of interventions in a given component.

A comparison of what is known or done (Figure 3) with what can be influenced (Figure 4) highlights the strong points of a fishery (here good knowledge of the environment and of the resource, and investment in infrastructure) and the less managed or manageable ones (here actors and their drivers, marketing and the market demand). For the latter, additional knowledge might not immediately translate into influence over that component. These aspects are part of the challenges of small-scale fisheries governance and open to a contribution to the discussion on governability, i.e. the degree to which a fishery system is deemed governable [START_REF] Jentoft | Interactive governance for small-scale fisheries: global reflections[END_REF][START_REF] Song | Intersectorality in the governance of inland fisheries[END_REF][START_REF] Descloux | Main features of the Nam Theun 2 hydroelectric project (Lao PDR) and the associated environmental monitoring programmes[END_REF].

Figure 1 :

 1 Figure 1: Location map; map data (Google Earth, Landsat).

Figure 2 :

 2 Figure 2: Components of an inland fishery as illustrated by the Nam Theun 2 reservoir fishery.

Figure 3 :

 3 Figure 3: Mapping of the knowledge base about the various components of the Nam Theun 2 reservoir fishery

Figure 4 :

 4 Figure 4: Degree of possible influence of fishery managers over the Nam Theun 2 reservoir fishery system


Francisco Robledo

Vivek Borkar

Urtzi Ayesta

Konstantin Avrachenkov

QWI: Q-learning with Whittle Index

Keywords:

INTRODUCTION

Markov Decision Processes (MDPs) provide a mathematical framework for sequential decision making. Formally, an MDP is a sequential stochastic control process, where a decision maker aims at maximizing its long term reward. The basic setup is as follows, at each time step, a state dependent reward is accrued, the decision maker chooses an action among the available ones, and the process randomly moves to a new state. Due to their broad applicability, MDPs are found in many areas, including artificial intelligence, economics, communications and operations research.

An MDP can be solved via dynamic programming, however, this is a computationally intractable task for realistic model sizes. As a result, classes of MDPs that are analytically tractable or possess good approximations have received a lot of attention. In this extended abstract we focus on one such class, namely, the Restless Multi-Armed Bandit Problem (RMABP), introduced in [START_REF] Whittle | Restless bandits: Activity allocation in a changing world[END_REF]. In an RMABP there are multiple concurrent projects or bandit's arms. The decision maker knows the states of all projects and the reward in every state, and aims at maximizing the long-term reward. At every decision epoch, the decision maker activates one or several projects, and the state of active and passive projects evolve stochastically.

RMABPs have become extremely popular over the years, and have been applied in many contexts, including inventory routing, machine maintenance, health-care systems, network-ing, etc. RMABPs cannot be solved exactly, except for some toy examples. In [START_REF] Whittle | Restless bandits: Activity allocation in a changing world[END_REF] Whittle developed a methodology to obtain heuristics by solving a relaxed version of the RMABP. The obtained heuristics, nowadays known as Whittle's index policy, rely on calculating Whittle's index for each of the projects, and activating in every decision epoch the project with highest Whittle's index. It has been reported on numerous instances that Whittle's index policy provides strikingly good performance, and it has been shown to be asymptotically optimal as the number of projects grows large [START_REF] Weber | On an index policy for restless bandits[END_REF].

Reinforcement Learning refers to the area of machine learning that aims at solving MDPs for which we do not have precise knowledge of the mathematical model. In [START_REF] Watkins | Q-learning[END_REF], it was shown that the well known Q-learning algorithm can solve an MDP, provided that every state-action pair is visited infinitely often. In recent years there has been a technological breakthrough, and the combination of reinforcement learning with deep neural networks has allowed to accomplish large scale problems [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF].

As the main contribution of this paper, we develop a reinforcement learning algorithm QWI, which is based on Q-learning and learns the Whittle Index policy for the total discounted criterion. To illustrate its performance, we consider a benchmark RMABP, restart problem, and compare numerically the performances and convergence times of Q-learning, Q-learning with neural network approximation, and QWI. The numerical results show that our algorithm converges faster than the other algorithms.

RESTLESS MARKOVIAN BANDITS

Problem formulation and relaxation

We consider an RMABP with N projects (or arms) and under the total discounted criterion. We denote by S i n the state of project i at the n-th time step, and let r i (S i n , A i n ) denote the conditional expected reward obtained by the i-th project at step n. The discounting factor is given by γ.

We let An be the vector of actions at time-step n, with elements A i n = a 2 {0, 1}. Since we assume that we can only activate a number M < N of projects at each time-step, the objective is to determine the control policy maximizing the following reward:

E 2 4 1 X n=1 N X i=1 γ n r i (S i n , A i n )
under the constraint:

N X i=1 A i n  M, n ≥ 0. (2)
Following Whittle's development [START_REF] Whittle | Restless bandits: Activity allocation in a changing world[END_REF], we relax the constraint (2) and require that it holds only on average. The constraint can then be added to the objective function:

E 2 4 1 X n=1 N X i=1 γ i ✓ r ⇣ S i n , A i n ⌘ + λ ⇣ 1 -A i n ⌘ ◆ 3 5 , ( 3
)
where λ is a Lagrange multiplier associated to the constrained.

The solution to (3) is obtained by combining the solution to N independent problems. In other words, for each bandit i we need to solve the associated Bellman equation given by:

V i (s) = max a2{0,1} 2 6 4a 0 @ r i (s, 1) + γ • X j p i (j|s, 1)V i (j) 1 A + (1 -a) 0 @ r i (s, 0) + λ + γ • X j p(j, |s, 0)V i (j) 1 A 3 7 5 (4)
where V i (s) is the value function corresponding to the initial state s. The optimal action in (4) will be to activate the bandit if

r i (k, 1) + γ • P j p i (j|k, 1)V i (j) > r i (k, 0) + λ + γ • P j p(j, |k, 0)V i (j)
, while otherwise the optimal action will be to keep that project passive. For this reason, the multiplier λ can be seen as a subsidy for passivity.

Whittle index

We can rewrite the expression in the square brackets of equation ( 4) as a function of the state-action pair in the following manner:

Q i (s, a) =a 0 @ r i (s, a) + γ • X j p i (j|s, a)V i (j) 1 A + (1 -a) 0 @ r i (s, a) + λ + γ • X j p(j|s, a)V i (j) 1 A , ( 5
)
where we have omitted the dependency on project i. Following Whittle's approach, provided that the technical condition known as indexability holds [START_REF] Whittle | Restless bandits: Activity allocation in a changing world[END_REF], the solution to (3) is characterized by a set of indices λ ⇤ (•) known as Whittle indices. A good suboptimal policy will be to activate at every time-step M projects with the largest Whittle indices. Whittle then [START_REF] Whittle | Restless bandits: Activity allocation in a changing world[END_REF] introduced the index as the value of λ ⇤ (k) of λ such that both the passive and active actions are equally preferred for a given state k, namely:

Q(k, 1) -Q(k, 0) = 0. ( 6
)

QWI ALGORITHM DESCRIPTION

We describe now our algorithm QWI that aims at learning both the state-action values in [START_REF] Watkins | Q-learning[END_REF] and the Whittle indices of [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF].

For the calculation of the Q-values we have

Q x n+1 (Sn, An) Q x n (Sn, An) + ↵(n) ⇥ (1 -An)(r(Sn, 0) + λn(x)) + Anr(Sn, 1)+ γ max b2{0,1} Q x n (Sn+1, b) -Q x n (Sn, An) � , ( 7
)
where the learning step must satisfy the usual conditions

P n ↵(n) = 1, P n ↵ 2 (n) < 1.
For the Whittle index, we have

λn+1(x) = λn(x) + β(n) � Q x n (x, 1) -Q x n (x, 0) � , (8)
where

P n β(n) = 1, P n β 2 (n) < 1.
A key requirement is that the algorithm operates in two time scales. On the fast time scale, the Q-values are updated similarly as in the standard Q-learning algorithm, whereas on the slower time scale, the Whittle index is updated. In order to achieve this we require that β(n) = o(↵(n)). In our numerical experiments, and based on the experience reported in [START_REF] Avrachenkov | Whittle index based Q-learning for restless bandits with average reward[END_REF], we will use

↵(n) = 1 d n 5000 e and β(n) = 1 1+d n log n 5000 e
I{n(mod 100) ⌘ 0}. Thus, β 6 = 0 only once every 100 iterations. We make throughout the following assumption of 'sufficient exploration': lim inf n"1

1 n
P n m=1 I{Sm = s, Am = a} > 0, for all s, a. That is, each state-action pair is sampled 'comparably often'. We can now state the main theoretical result of the paper:

Proposition 3.1. Given the two time-scale iterations (7)- (8) with stepsizes {↵(n)} and {β(n)} satisfying P n ↵(n) = 1, P n ↵ 2 (n) < 1, P n β(n) = 1, P n β 2 (n) < 1 and β(n) = o(↵(n))
, under the assumptions of RMABP indexability and sufficient exploration, λn(k) ! λ ⇤ (k), 8k, a.s. as n ! 1.

The proof of the convergence of this two-time scale system is based on the results in [START_REF] Lakshminarayanan | A stability criterion for two timescale stochastic approximation schemes[END_REF]. We also note that an algorithm converging to the Whittle indices for the average long-run reward criterion is given in [START_REF] Avrachenkov | Whittle index based Q-learning for restless bandits with average reward[END_REF]. The proof for the present discounted case will be presented in the full-length version this work.

NUMERICAL RESULTS

To assess the performance of our algorithm we consider the benchmark "restart problem" studied in [START_REF] Avrachenkov | Whittle index based Q-learning for restless bandits with average reward[END_REF] with a state space S = {0, 1, 2, 3, 4}. Active action (a = 1) takes the project to the initial state 0 with probability 1. In the case of the passive action, the transition matrix is given by

P0 = 0 B B B B @
1/10 9/10 0 0 0 1/10 0 9/10 0 0 1/10 0 0 9/10 0 1/10 0 0 0 9/10 1/10 0 0 0 9/10

1 C C C C A .
The conditional expected reward is given by r(s, a) = 0.9 s+1 for a = 0, and r(s, a) = 0 for a = 1. Throughout this section, we have considered the case of homogeneous projects, i.e., all projects have the same dynamics and reward functions. We consider there are 5 projects, and we assess numerically the convergence stated in Proposition 3.1. Figure 1 depicts the evolution of the estimated Whittle indices with respect to the time step. Each of the lines represents the average value of the estimates of all the projects for a given state. We observe that the convergence for state 5 requires many more steps. However, from very early on, the estimated indices are ordered correctly.

We now compare the performance of QWI with respect to the standard Q-learning [START_REF] Watkins | Q-learning[END_REF], and Q-learning with neural network approximation (RLNN) [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. We use a neural network with two hidden layers network, with 16 neurons in each of them. In Figure 2 we consider the case of 5 projects, γ = 0.8 and an exploration parameter ✏ = 0.3 in QWI and RLNN algorithms and ✏ = 1.0 for Q-learning. We note that Qlearning does not converge to the correct value of the value function, but with ✏ = 0.3, the convergence of Q-learning was even slower than in the case of ✏ = 1.0. Both RLNN and QWI algorithms succeed in converging to the optimal value function, but QWI, which takes advantage of the "restless" structure of the problem, converges faster. In Figure 3 we consider the case of 7 projects. We observe that the benefits of QWI with respect to the two other algorithms increases. Once again Q-learning does not converge to the optimal value function, and RLNN performs better than Q-learning, but it does not converge to the optimal values neither. On the other hand, QWI converges to the correct values.

CONCLUSIONS

We have presented QWI, an algorithm that converges to the Whittle indices of any indexable RMABP. An advantage of QWI with respect to standard Q-learning and neuralnetwork based approximations, is that it uses e↵ectively the decomposition structure of RMAB to tackle the curse of dimensionality. In our numerical section we have considered a benchmark RMABP, and showed that QWI indeed outperforms the other algorithms. In future work we plan to investigate in more depth the performance of QWI with more realistic and complex RMABPs.

Figure 1 :

 1 Figure 1: Convergence of the Whittle indices

Figure 2 :

 2 Figure 2: Bellman's Value Function (4) evolution during training for the RLNN, Q-learning and QWI algorithms using 5 projects

Figure 3 :

 3 Figure 3: Bellman's Value Function (4) evolution during training for the RLNN, Q-learning and Whittle index algorithms using 7 projects

,[START_REF] Avrachenkov | Whittle index based Q-learning for restless bandits with average reward[END_REF]