Z B He

J.-L Maurice
email: jean-luc.maurice@polytechnique.edu

C S Lee

D Pribat

C S Cojocaru

Nickel catalyst faceting in plasma-enhanced direct current chemical vapor deposition of carbon nanofibers

Keywords: carbon nanofibers (CNFs), PECVD, Ni catalysts, Ni faceting, TEM

Vertically aligned multi-walled carbon nanofibers (CNFs) were grown by plasmaenhanced chemical vapor deposition with Ni catalysts on the top of nanofibers. Transmission electron microscopy was used to study the morphology and crystallography of Ni catalysts, which are essential for the nucleation and growth of CNFs. A model for the faceted shape of Ni catalytic particles is proposed. It is shown that the exposed polyhedral surfaces of Ni catalytic particles for vertically aligned CNFs are composed of {111}, {110}, and {100}, a

faceting that appears to be characteristic of the growth atmosphere.

I. INTRODUCTION

Since the discovery of carbon nanotubes (CNTs) in 1991 by Iijima [START_REF] Iijima | Helical microtubules of graphitic carbon[END_REF], CNTs as well as carbon nanofibers (CNFs), especially their ordered arrays, have attracted a great deal of attention, not only because of theoretical interest, but also for their superior properties such as high thermal conductivity, excellent electron emission properties and wonderful electron transport with potential commercial applications in electronics, mechanics, chemistry and biochemistry [START_REF] Martel | Singleand multi-wall carbon nanotube field-effect transistors[END_REF][START_REF] Ren | Synthesis of Large Arrays of Well-Aligned Carbon Nanotubes on Glass[END_REF][START_REF] Dresselhaus | Carbon Nanotubes: Synthesis, Structure, Properties and Applications[END_REF][START_REF] Robertson | Realistic applications of CNTs[END_REF][START_REF] Pribat | Carbon nanotubes and semiconductor nanowires for active-matrix backplanes[END_REF][START_REF] Bichoutskaia1 | Nanotube-based data storage devices[END_REF][START_REF] Tsai | Transfer of patterned vertically aligned carbon nanotubes onto plastic substrates for flexible electronics and field emission devices[END_REF]. A multi-walled CNT consists of concentric cylinders of rolled graphene sheets, whereas a CNF is made of stacked cone-shaped graphite planes. The primary difference between CNTs and CNFs is the angle () between the graphite basal planes and the tube axis ( = 0° for CNTs;   0° for CNFs), which can be controlled by, e.g., the amount of hydrogen in the growth atmosphere [START_REF] Nolan | Hydrogen control of carbon deposit morphology[END_REF][START_REF] Nolan | Carbon Deposition and Hydrocarbon Formation on Group VIII Metal Catalysts[END_REF]. Increasing hydrogen concentration would tend to tilt the basal planes away from the cylinder axis to form CNFs [START_REF] Murayama | A novel form of filamentous graphite[END_REF]; while on the contrary, suppressing hydrogen would allow carbon to deposit in the closed forms of shells or CNTs [START_REF] Li | Large-Scale Synthesis of Aligned Carbon Nanotubes[END_REF].

High temperature preparation techniques such as arc discharge or laser ablation were first used to produce CNTs or CNFs and were then replaced by low temperature chemical vapor deposition (CVD) techniques (< 800 °C), since the orientation, alignment and density of CNTs can be precisely controlled in the latter [START_REF] Li | Large-Scale Synthesis of Aligned Carbon Nanotubes[END_REF][START_REF] Meyyappan | Carbon nanotube growth by PECVD: a review[END_REF]. Transition metal nanoparticles, primarily Fe, Co or Ni, are typical catalysts for the growth of CNTs or CNFs using CVD [13-16, 17, 18]. The growth mechanism of the catalytic growth of CNFs have been discussed in some classic works by Baker et al. [START_REF] Baker | Nucleation and growth of carbon deposits from the nickel catalyzed decomposition of acetylene[END_REF][START_REF] Baker | A comparison of the catalytic influence of nickel, iron and nickel-iron on the gasification of graphite in various gaseous environments[END_REF]. In the case of Fe and Ni thin films, forming nanosized particles by high-temperature de-wetting, it was shown that the diameter of the CNTs strongly depended on the starting thickness of catalyst layer [START_REF] Wei | Effect of catalyst film thickness on carbon nanotube growth by selective area chemical vapor deposition[END_REF]. Correlations between the size of metal catalyst nanoparticles and carbon nanotube growth have been found and discussed [START_REF] Kukovitsky | Correlation between metal catalyst particle size and carbon nanotube growth[END_REF]. Besides, the initial stages of the growth of vertically aligned carbon nanofibers were studied by using both scanning and transmission electron microscopy (TEM) [START_REF] Cui | Initial growth of vertically aligned carbon nanofibers[END_REF]. Moreover, Helveg et al. observed directly the formation of CNFs catalyzed by Ni nanoparticles using time resolved, high-resolution in situ TEM [START_REF] Helveg | Atomic-scale imaging of carbon nanofibre growth[END_REF]. The influence of the composition and the morphology of the catalyst nanoparticles on the growth of CNTs by CVD are summarized in a review paper [START_REF] Dupuis | The catalyst in the CCVD of carbon nanotubes-a review[END_REF]. However, the crystallography of catalysts has only been the object of few experimental studies [START_REF] Audier | Crystallographic orientations of catalytic particles in filamentous carbon; Case of simple conical particles[END_REF][START_REF] Kuang | Catalytically active nickel {110} surfaces in growth of carbon tubular structures[END_REF], and the correlations between the catalytically active surfaces of transition metal catalysts and CNFs are still not clear. In this study, we will focus on a TEM investigation of the crystallography of Ni catalysts for the vertical growth of CNFs, which were synthesized by direct current (dc) plasma-enhanced CVD (dc PECVD) in an atmosphere composed of isopropylic alcohol (CH 3 CH(OH)CH 3 ) and water (H 2 O). A three-dimensional (3D) model at the atomic scale of a faceted shape of Ni catalytic nanoparticle is proposed based on the bright-field TEM image observations. The top surface of Ni catalysts is found indeed to be faceted like a Wulf polyhedron, while observations of similar catalysts in other growth conditions exhibited more rounded features and a {110} preferential faceting [START_REF] Kuang | Catalytically active nickel {110} surfaces in growth of carbon tubular structures[END_REF], or flat and exclusively {111} facets [START_REF] Audier | Crystallographic orientations of catalytic particles in filamentous carbon; Case of simple conical particles[END_REF]. Finally, the Ni particle faceting is discussed as a consequence of the growth atmosphere and of the way nucleation and early stages of growth take place.

II. EXPERIMENTS

Vertically aligned CNF arrays were grown from Ni catalysts originating from thin films with different thicknesses. We used bare Si as substrate, and growth was performed by dc PECVD at 600 °C for 85 minutes or 650 °C for 20 minutes, at a pressure of 2 mbars. A mixture of water vapor and isopropylic alcohol with a flow rate of 7.5 sccm and 4.5 sccm, respectively, was used as the reactive gas sources. The gas flows were precisely controlled to get a C/H ratio of ~ 0.26 which avoided the generation of amorphous carbon on the surface of catalysts. In order to obtain well aligned CNFs, the applied voltage between anode and cathode in the chamber was kept constant at 500 V, and with a current intensity of 0.2 A. The extraction voltage and the current were slowly increased up to the optimum conditions at the beginning of the growth process. At the end of the growth process, the chamber was totally pumped out and cooled down. Cooling took around 30 minutes without any gas flow. The length of CNFs was controlled by the growth time and their diameter by the thickness of the starting Ni film. The CNFs were observed using a Hitachi S-4800 FE-scanning electron microscope (SEM) with 25 kV accelerating voltage. The microstructures of CNFs and Ni catalysts were studied by TEM using a Philips CM 30 working at 300 kV, and a Topcon 002B working at 120 kV. between two crystal variants in one Ni catalyst were also found, as seen in Fig. 2(e). However, the intersections of those boundaries with the particle surface do not create visible surface step or discontinuity. And indeed, the graphene layers of CNFs are not affected by the grain boundaries. Sometimes, the Ni particle can be bent and hence change its growth orientation [Fig. 2(f)]. As a result, the corresponding orientation of CNFs at its top is deviated from the initial one. In correlation with that bending, the thickness of the graphite layers on both side surfaces of the top of CNFs in Fig. 2(f) is different. In general, although the morphologies of Ni particles are different, there are some common characteristics: (1), the shape of the catalysts for the growth of CNFs prefer to be pear like (also known as water drop like or tear drop like); (2), each Ni particle consists of two parts: head (free from graphene coverage) and body (covered by graphene layers); (3), the head presents low-index facets while the body does not; (4), the size of the head and the tail of the water drop like Ni catalysts determine the external diameter and the one of the inner hole of CNFs, respectively; (5), neither twins nor grain boundaries affect the growth of CNFs. Figure 3 All the observed Ni particles for vertically aligned CNFs are faceted, with facets corresponding to low-index planes, particularly {111} (Fig. 2). This result is different from other observations [START_REF] Kuang | Catalytically active nickel {110} surfaces in growth of carbon tubular structures[END_REF], which we will discuss in the next section. In order to more precisely analyze the crystallographic surfaces of Ni, we tilted one Ni particle along two mutually perpendicular directions to get three images [Fig. 3]. Figure 3 than that in (a) because it is just a projected image from (a). It should be mentioned that one should be very careful when determining the growth direction of one dimensional nano materials as the sample may be tilted to a relatively large angle from the horizontal, e.g., the schematic illustration in Fig. 3 (c). By combining the information of the experimental results in Fig. 3, we propose a primary 3D structural model for the faceted shape of the Ni particle (Fig. 4). Figure 4 4(c). Note that the same layer was highlighted by the same color in Fig. 4 (a).

III. RESULTS

The foremost plane is (011), the stacking sequence of which controls the growth direction of the Ni particle. It is interesting to note that the carbon-free (1-1-1) plane is larger than the others.

These low-index facet orientations correspond to low surface energy; however, the presence of sharp edges gives a shape which is not yet that of a Wulff polyhedron [START_REF] Muller | For a demonstration of Wulf theorem[END_REF], which is to say that the top surface is still far from thermal equilibrium.

IV. DISCUSSION

The surfaces of Ni catalysts are very significant for the nucleation and growth of CNFs since they play a key role in (1) propan-2-ol dissociation, (2) surface diffusion of carbon atoms, and (3) segregation and crystallization of the latter. Audier et al. showed that carbonfree planes of catalytic transition metal alloys for CNTs were dependent on the crystal structure of body-centred or face-centred cubic (fcc) [START_REF] Audier | Crystallographic orientations of catalytic particles in filamentous carbon; Case of simple conical particles[END_REF]. These authors also concluded that in the case of fcc metals, only {111} facets could be observed. They used catalytic disproportionation of CO at P = 1 atm and 450 °C < T < 650 °C [START_REF] Audier | Crystallographic orientations of catalytic particles in filamentous carbon; Case of simple conical particles[END_REF]. In contrast, Kuang et al. found that the {110} orientation of Ni particles was preferred as the top surfaces [START_REF] Kuang | Catalytically active nickel {110} surfaces in growth of carbon tubular structures[END_REF]. These authors used a mix based on CH 4 , in a hot-wire PE-CVD equipment, at P = 3 kPa, and T = 700 °C. Yang and Chen [START_REF] Yang | Mechanism of carbon filament growth on metal catalysts[END_REF] reported that {110} faces were catalytically active for the dissociation of CH 4 , while {111} was not. Here, we found not only {110} planes as in Ref. [START_REF] Kuang | Catalytically active nickel {110} surfaces in growth of carbon tubular structures[END_REF], and {111} as in Ref. [START_REF] Audier | Crystallographic orientations of catalytic particles in filamentous carbon; Case of simple conical particles[END_REF], but also other low index crystallographic planes such as {100}. Such kind of polyhedral top surfaces reminds one of the equilibrium shapes of Wulf polyhedra, which indicates the tendency of minimizing surface energy by exposing low-index facets. On the contrary, side surfaces mainly consist of high-index planes: They are higher energy surfaces, including a high density of atomic steps, perhaps playing an important catalytic action in the crystallization of graphene layers [START_REF] Helveg | Atomic-scale imaging of carbon nanofibre growth[END_REF]. Our observations are somewhat different from the conclusions of Yang and Chen [START_REF] Yang | Mechanism of carbon filament growth on metal catalysts[END_REF], who had shown that, on the one hand, {100} and {110} surfaces were the most active in the decomposition of CH 4 at 700 °C, and, on the other hand, {111} and {311} where the most efficient for the epitaxy of the graphene layers.

At least three conditions, however, differ between our work and that of Yang and Chen [START_REF] Yang | Mechanism of carbon filament growth on metal catalysts[END_REF]:

(1) theirs was at a pressure of 1 atmosphere while ours is at 1 mbar, (2) their atmosphere was pure CH 4 , while our atmosphere is composed of a mixture of water vapor and isopropylic alcohol, and (3), theirs was standard CVD, while ours is dc PECVD. Thus the dynamic equilibria that establish at surfaces during growth should be different in the two cases.

Figure 5

The increasing number of graphene layers from the head to the foot of Ni particles may explain why the Ni particles have a pear-like shape with a large size at the head, and a small size at the foot. Figure 5(a) is a high resolution electron microscope (HREM) image showing the change of the number of graphene layers at the top of Ni catalyst. There are only three graphene layers at the highest side surface of Ni producing a relative small compressive stress on the Ni particle, but the number of graphene layers increases along an arrow direction to cause a relatively large pressure to compress the Ni particle to a smaller diameter. As a result, the Ni catalyst adopts a pear-like shape with a big size at the position covered with few graphene layers, and elongates along the axis of CNFs. The compressive stress from graphene layers of CNTs could even extract Fe 3 C along the tubes due to a stress as large as 40 gigapascals [START_REF] Sun | Carbon nanotubes as high-pressure cylinders and nanoextruders[END_REF]. Graphene layers of CNFs are always parallel to the side surface of Ni catalysts (Fig. 5b) and, hence the  angle is dictated directly by the slope of the side surfaces of the polyhedral Ni catalyst. Nolan et al. have shown that the  angle could be increased by increasing hydrogen in the chamber because hydrogen was able to saturate the additional surface dangling bonds introduced by the increase of  [START_REF] Nolan | Hydrogen control of carbon deposit morphology[END_REF][START_REF] Nolan | Carbon Deposition and Hydrocarbon Formation on Group VIII Metal Catalysts[END_REF]. With the new cone-shaped graphene layers formed layer by layer from the side surfaces of Ni particle, the CNF becomes longer and longer along the axis of the elongated Ni particle (Fig. 6).

Figure 6 Normally, it is difficult for graphene layers to grow beyond the top of the Ni particle since the edges of graphene layers are easily etched by energetic ions and highly reactive neutral species such as atomic hydrogen, as explained in Ref. 30 (red "x" in Fig. 5a). It seems that the conical Ni particles at the top of CNFs act as moulds to produce cone-shaped layer structure.

Therefore, the inner and external diameters of the top-type CNFs are determined by the size of the tail and head of the elongated Ni particle, respectively.

V. CONCLUSIONS

Vertically aligned CNFs were grown by the tip-growth mechanism, using dc PECVD at low temperatures and H 2 O + CH 3 CH(OH)CH 3 gas mixtures. The exposed surfaces of the Ni particles at the growth front are a combination of low index planes such as {111}, {110}, and {100}, as in a Wulf polyhedron, rather than a unique type of facet such as {111} [START_REF] Audier | Crystallographic orientations of catalytic particles in filamentous carbon; Case of simple conical particles[END_REF] or {110} [START_REF] Kuang | Catalytically active nickel {110} surfaces in growth of carbon tubular structures[END_REF]. Planar defects such as {111} twins with twin boundaries parallel to or almost perpendicular to the CNT were found in the Ni catalysts. Although different faceted shapes of the Ni particles are found, the cone-shaped graphene layers of CNFs are always parallel to the side surfaces of the Ni particles, which in turn have high indices. The synthesis mechanism of the CNF involves nucleation and growth of each graphene layer on the side walls of the Ni particle, followed by their repelling by the next nucleated layer. The resulting structure is a stack of parallel graphene layers "molded" on the side wall of the conical Ni particle and consequently oriented at an angle with respect to the CNF axis. The inner and external diameters of the CNFs are determined by, respectively, the diameters of the tails and of the heads of the elongated Ni catalyst particles.

Figure captions

Figure 1

 1 Figure1shows typical SEM images of top-type CNFs, which are well aligned and oriented

Figure 1 Different

 1 Figure 1

Figure 2 Figure 2

 22 Figure 2

  Fig. 3 are mainly composed of low-index (1-1-1), (011), and (001) planes. The (001) facet is

  Fig. 3, respectively. The exposed top surfaces of Ni are linked by yellow lines, and the side

Fig. 1 .

 1 Fig. 1. SEM images of vertically aligned top-type CNFs grown on different thicknesses of Ni

Fig. 2 .

 2 Fig. 2. (a) Low magnification TEM image of a bunch of CNFs with Ni catalyst particles on

Fig. 3 .

 3 Fig. 3. (a), (b) and (d) A series of TEM bright-field images of the same Ni particle taken at the

Fig. 4 .

 4 Fig. 4. A model for the faceted Ni particle of Fig. 3. (a), (b), and (d): Projections of the 3D

Fig. 5 .

 5 Fig. 5. HREM images of Ni and graphene layers. (a) The head of Ni particle. Note that the

Fig. 6 .

 6 Fig. 6. Schematics of the growth of top-type CNFs. Forbidden growth directions, marked by a

Fig. 3, He et al.

Fig. 6, He et al.

ACKNOWLEDGEMENTS

This work has been supported by the Region Ile-de-France in the framework of C'Nano IdF. C'Nano IdF is the nanoscience competence center of Paris Region, supported by CNRS, CEA, MESR and Region Ile-de-France. It was also partially funded by the European Community project OPTHER (FP 7) and the French ANR project SIMS. One of authors (Z.B. He) thanks Dr. G. Rizza and F. Attouchi at Laboratoire des Solides Irradiés, Ecole Polytechnique, France for useful discussions. Thanks are also due to Dr. J.-E. Bourée for careful reading of the manuscript.


C S Lee

L Baraton

Z B He

Jean-Luc Maurice

D Pribat

C S Cojocaru

L Maurice

Graphene growth directly on functional substrate

HAL is

Figures

Figure 1. General process for plasma-assisted graphene growth Figure 2. TEM images of the samples after growth by PE-CVD at 450°C for 12 min (a,b), and after an additonal annealing treatment in vacuum at 900°C for 18 min (c,d). (a, c) crosssections where graphite blocks can be seen, together with interfacial graphene in the two cases. The latter (a1, c1) appears continuous over several tens of nm, and more so before annealing. (b, d) plan-views obtained after Ni etching, and depositing the remaining layer on a TEM holey-carbon grid. Images recorded in holey regions (no contribution of amorphous carbon). The diffraction paterns (b1 and d1) indicate that the background consists of nanocrystalline graphene, with random orientation of the nano-grains in the plane (see fig. 3). Note the absence of 002 and 004 reflections indicating a good alignment of the graphene layers in the plane. The graphite blocks are small pilars or onions, with (002) planes parallel to the beam before anealing (b2), while they are flat with no (002) planes visible after annealing (d2).

Figure 3 .

 3 Figure 3. Schematic of the graphene structure in the plane (a) and perpendicular to the plane (b). Atomic positions in (a) are given for visualising the in-plane misorientation of four nmsized grains; they do not represent, of course, actual positions.

  Figure 5. SEM images


Thomas Lacoue-Labarthe

Estelle Le Bihan

David Borg

N Kouéta

Paco Bustamante

† Le

Variation of acid phosphatases and cathepsins activities in

Keywords: cephalopod, yolk assimilation, metals, embryonic stage, essential, non-essential

come

Introduction

Among cephalopods, Sepioidea (cuttlefishes) lay singly medium size eggs (3-10 mm) [START_REF] Boletzky | Characteristics of cephalopod embryogenesis[END_REF][START_REF] Boletzky | Cephalopod eggs and egg masses[END_REF] with a large yolk mass, which supplies the needs for a complete embryonic development. Hatchlings are morphologically similar to adults [START_REF] Lemaire | Table de développement embryonnaire de Sepia officinalis L. (Mollusque Céphalopode)[END_REF][START_REF] Boletzky | The "larvae" of Cephalopoda: A Review[END_REF]. This direct development of the telolecithal egg of Sepioidea is indeed characterized by an organogenesis proper (about two third of the development time) followed by a supplementary period of extreme growth during which the embryo size could increase by 80%. Nevertheless, biotic (e.g. yolk quantity) or/and abiotic (e.g. temperature) factors are known to govern the yolk utilization by the embryo and consequently the hatchling size, which could subsequently impact the juvenile recruitment [START_REF] Bouchaud | Etude du développement de l'oeuf de Sepia officinalis L. (Céphalopode, Sepiidae) en conditions expérimentales[END_REF]).

In the yolky eggs of oviparous animals, acid phosphatases (AcP) and cathepsin play a key role in yolk degradation processes as reported in the eggs of molluscs [START_REF] Morrill | Biochemical and electrophoretic analysis of acid and alkaline phosphatase activity in the developing embryo of Physa fontinalis (Gastropoda, Pulmonata)[END_REF][START_REF] Pasteels | Acid phosphatase and thiolacetic esterase activities studied with the electron microscope in the gill epithelium and eggs of Lamellibranchial molluscs[END_REF], echinoderms (e.g. [START_REF] Schuel | Heterogeneous distribution of "lysosomal" hydrolases in yolk platelets isolated from unfertilized sea urchin eggs by zonal centrifugation[END_REF][START_REF] Mallya | Proteolysis of the major yolk glycoproteins is regulated by acidification of the yolk platelets in sea urchin embryos[END_REF], arthropods (e.g. [START_REF] Fialho | Cathepsin D-mediated yolk protein degradation is blocked by acid phosphatase inhibitors[END_REF], fish (e.g. [START_REF] Kestemont | Cathepsin L in eggs and larvae of perch Perca fluviatilis: variations with the developmental stage and spawning period[END_REF][START_REF] Martinez | Digestive enzyme activity during larval development of the Senegal sole (Solea senegalensis)[END_REF][START_REF] Carnevali | Changes of lysosomal enzyme activities in sea bass (Dicentrarchus labrax) eggs and developing embryos[END_REF], amphibians (e.g. [START_REF] Lemanski | Role of acid phosphatase in the breakdown of yolk in developing amphibian embryos[END_REF][START_REF] Fagotto | Changes in yolk platelet pH during Xenopus laevis development correlate with yolk utilization: A quantitative confocal microscopy study[END_REF][START_REF] Komazaki | Degradation of yolk platelets in the early amphibian embryo is regulated by fusion with the late endosomes[END_REF], and birds [START_REF] Gerhartz | Proteolytic enzymes in yolk-sac membrane of quail egg. Purification and enzymatic characterisation[END_REF]. Acid phosphatases are ubiquitous enzymes catalysing the hydrolysis of various phosphate-containing compounds. Cathepsins include various protease forms among which are cystein proteases, i.e. cathepsin B and L, or aspartic proteases, i.e. cathepsin D, have received wider attention regarding the yolk reserve mobilization processes. Both AcP and cathepsins are localized in specialized organelles called yolk platelets that are modified lysosomes containing vitellin reserves, i.e. vitellogenin, phosvitin, lipovitellin, nucleic acids, polysaccharides, lectin and growth factors [START_REF] Fagotto | Yolk degradation in tick eggs: II. Evidence that cathepsin L-like proteinase is stored as a latent, acid-activable proenzyme[END_REF][START_REF] Fagotto | Regulation of yolk degradation, or how to make sleepy lysosomes[END_REF]Komazaki and Himura, 1999). Therefore, the utilization of the maternal food during embryogenesis implies that 1) the lysosomal enzymes are maternally transferred during oogenesis [START_REF] Fausto | The yolk sac in late embryonic development of the stick insect Carausius morosus (Br.)[END_REF] and 2) that the yolk platelets do not degrade their content until specific developmental stages are reached [START_REF] Fagotto | Changes in yolk platelet pH during Xenopus laevis development correlate with yolk utilization: A quantitative confocal microscopy study[END_REF]. Indeed, AcP and cathepsin activation depends on the egg fertilization [START_REF] Fialho | Oocyte fertilization triggers acid phosphatase activity during Rhodnius prolixus embryogenesis[END_REF] and the yolk platelets stimulation by an acidification of these organelles. Once activated, such lysosomal enzymes could interact with other ones: thus, the cathepsin D is cleaved in a most active form by a cystein protease in the yolk platelets of Xenopus laevis [START_REF] Yoshizaki | Cysteine proteinase plays a key role for the initiation of yolk digestion during development of Xenopus laevis[END_REF]. Moreover, in Rhodnius prolixus eggs, AcP inhibitors also blocked the cathepsin D activity disclosing a cooperative action of both enzymes to promote the yolk degradation [START_REF] Fialho | Cathepsin D-mediated yolk protein degradation is blocked by acid phosphatase inhibitors[END_REF].

Different metals such as Cd, Cu and Hg were shown to inhibit AcP activity in the clam Scrobicularia plana [START_REF] Mazorra | Acid and alkaline phosphatase activities in the clam Scrobicularia plana: kinetic characteristics and effects of heavy metals[END_REF] or in the mussel Mytilus galloprovincialis [START_REF] Izagirre | Time-course study of the early lysosomal responses to pollutants in mussel digestive cells using acid phosphatase as lysosomal enzyme marker[END_REF]. In vitro, an inhibitory effect of Cu and Zn has also been demonstrated on the cathepsin activity of the digestive cells of adult Sepia officinalis, whereas Ag led to a stimulation of this enzymatic activity [START_REF] Le Bihan | Development of a bioassay from isolated digestive gland cells of the cuttlefish Sepia officinalis L. (Mollusca Cephalopoda): effect of Cu, Zn, and Ag on enzymes activities and cell viability[END_REF]. When the common cuttlefish S. officinalis migrates in spring to the shallow waters to mate and spawn [START_REF] Boucaud-Camou | The migrations of the cuttlefish (Sepia officinalis L.) in the English Channel[END_REF], the eggs are fixed on hard substrata and therefore potentially exposed to coastal contaminants such as heavy metals. As previously demonstrated (Bustamante et al., 2002a[START_REF] Bustamante | Uptake, transfer and distribution of silver and cobalt in the tissues of the common cuttlefish Sepia officinalis at different stages of its life cycle[END_REF]Lacoue-Labarthe et al., 2008a[START_REF] Lacoue-Labarthe | Contrasting biokinetics of accumulation and distribution of Am, Co, Cs, Mn and Zn by the eggs of the common cuttlefish (Sepia officinalis) during the whole development time[END_REF], dissolved metals accumulate in the embryonic tissues during the embryo growth period independently of their essential (Co, Mn, and Zn) or non essential (Ag, Cd, Hg) character. Such accumulation implies that metals could interact with enzymes and subsequently affect the physiology of the embryo development.

Considering that cephalopods present a short-life span as they die after the reproduction, the population renewal entirely depends of the hatching success of their eggs and the viability of the young cuttlefish during the first weeks of juvenile life.

The aims of the present study were 1) to establish the kinetics of the AcP and cathepsin activities in the whole egg during the direct embryonic development of S. officinalis from the spawning to the hatching time, when the embryo becomes a juvenile morphologically similar to adults; and 2) to determine the potential effects of selected heavy metals on the enzymatic activities and on hatchling weight. In this respect, cuttlefish eggs were exposed to dissolved elements in natural seawater at different concentrations: two non-essential elements, Ag and Cd which are known for their contrasting accumulation capacities in the embryonic tissues (Lacoue-Labarthe et al., 2008a) and one essential element, Cu which is a co-factor of the oxygen carrier protein, i.e. the haemocyanin.

Materials and methods

Biological material and experimental procedure

Cuttlefish eggs were collected on pots from the west coast of Cotentin, France, by local fishermen. Because pots were picked up once in two day and cleaned, sampled eggs were considered as laid in the previous 24-48 h. At the laboratory, eggs were separated for optimal oxygenation and placed into floating sieves in a rearing structure as described by [START_REF] Koueta | Food intake and growth in reared early juvenile cuttlefish Sepia officinalis L. (Mollusca Cephalopoda)[END_REF]. A few days after the field collection, 700 eggs were randomly placed in each out of the 14 tanks containing 11 l seawater (constantly aerated closed circuit; temperature 17°C; 34 p.s.u.; light/dark cycle 12h/12h). Cuttlefish eggs were then exposed to Ag (0.1, 2, 100, 2000 ng AgCl 2 l -1 , i.e. 0.06, 1.2, 60, 1200 ng Ag l -1 ), Cd (50, 100, 500, 1000 ng CdCl 2 l -1 , i.e. 31, 61, 305, 610 ng Cd l -1 ) and Cu (0.5, 5, 50, 500 µg CuCl 2 l -1 , i.e. 0.23, 2.3, 23, 230 µg Cu l -1 ) throughout their development (50 d at 17°C). In parallel, control eggs were incubated in non-contaminated seawater and were used to follow the natural AcP and cathepsin kinetics (one control tank was set up for Ag and Cd experiment and one other control tank for Cu experiment). Stock solutions were prepared in 0.3 N chloridric acid to obtain concentrations allowing the use of spikes ranging between 100 and 1000 µl. Seawater and metal spikes were renewed daily throughout the development time to maintain water quality and metal concentrations as constant as possible.

In each tank, 24 eggs were regularly weight to delineate the embryo development during incubation time and 9 eggs were regularly dissected to follow the developmental stages.

Additionally, 6 eggs were sampled for the enzymatic assays and immediately frozen in liquid nitrogen to be stored at -80°C until being analysed. At the end of development, one hundred hatchlings were weighed for each exposure condition.

Enzymatic assays

Extraction

Three pools of 2 eggs were weighed and homogenized in a potter with a cold extraction buffer containing KCl 1%, EDTA 1 mM (2.5 ml of buffer: 1 mg of egg material). The crude extract was then centrifuged for 60 min at 10 000 g at 4°C [START_REF] Bonete | Acid proteinase activity in fish II. Purification and characterization of cathepsins B and D from Mujil auratus muscle[END_REF][START_REF] Le Bihan | Development of a bioassay from isolated digestive gland cells of the cuttlefish Sepia officinalis L. (Mollusca Cephalopoda): effect of Cu, Zn, and Ag on enzymes activities and cell viability[END_REF], 2006[START_REF] Le Bihan | Effect of different treatments on the quality of cuttlefish (Sepia officinalis L.) viscera[END_REF]. The supernatant liquid was used for enzymatic assays and for quantification of proteins.

Enzymatic assays

Acid phosphatase activity was determined according to [START_REF] Moyano | Characterization of digestive enzyme activity during larval development of gilthead seabream (Sparus aurata)[END_REF] using pnitrophenyl-phosphate 2% as substrate in a 1 M Tris buffer, pH 3. The addition of 100 µl of supernatant sample with 100 µl of substrate started the enzymatic assay. After 30 min of incubation at 25°C, 1 ml of NaOH 1 M was added to stop the reaction. The absorbance was measured at 405 nm and each sample was tested in triplicate. The AcP activity was expressed as specific activity measured in the egg (U mg -1 of protein) where one unit was defined as the amount of enzyme able to produce an increase of 0.01 unit of absorbance.

Cathepsin activity was measured according to the method of [START_REF] Bonete | Acid proteinase activity in fish II. Purification and characterization of cathepsins B and D from Mujil auratus muscle[END_REF] using haemoglobin 2% (w:v) solution as substrate. The method was as follows: 100 µl of sample supernatant were mixed with 50 µl of acetate 0.4 M buffer, pH 4 and 50 µl of substrate. All measurements were performed in triplicate. Following incubation for 60 min at 37°C, the reaction was stopped by addition of 3% trichloroacetic acid (TCA), and after holding for 10 min, the reaction mixture was centrifuged for 10 min at 800 g. The reaction products were assessed by Folin-Lowry methods, using tyrosine as standard. Cathepsin activity was expressed as specific activity in the egg (U mg -1 of protein) where unit of enzyme activity was defined as the amount of enzyme catalyzing the formation of 1 µmol of tyrosine.

The amount of proteins in the extracts was determined by the method of [START_REF] Lowry | Protein measurements with the folin phenol reagent[END_REF] with bovine serum albumin as the standard.

Statistical analysis

A one-way analysis of variance (ANOVA) followed by the Tukey test was applied to analyse the differences of the weights of hatchlings from eggs exposed to metals and control eggs.

Enzymatic activities in all the samples were measured in triplicates. A Kruskall-Wallis test was applied to determine 1) the differences of the AcP and cathepsin activities among the different developmental stages and 2) the significant differences from the control of the cathepsin activity measured in metal exposed eggs at the end of development (day 48). The kinetics of the acid phosphatases activities were described by an exponential equation (AcP t = AcP 0 e kt ) during the last 18 days of the embryonic development, where AcP t and AcP 0 (U mg - 1 ) are the enzyme activities at time t (d) and 0, respectively, and k is the increase rate constant (d -1 ). The effects of metal exposure were determined testing the significant differences (Ftest) between the increase rates (k) of enzyme activities measured, during the last 18 days of development, in eggs reared in control conditions and in eggs exposed to metals. P values lower than 0.05 were used to identify significant differences.

Results

Enzyme activity variations during embryo development

The AcP activity measured in the whole egg increased from 0.37 ± 0.02 to 19.9 ± 3.1 U mg -1 during the embryonic development (Fig. 1.A.). More precisely, this increase comprises three phases: during the first 22 d of development, enzymatic activity increased significantly from 0.37 ± 0.02 to 2.2 ± 0.2 U mg -1 at day 0 and day 22, respectively; p < 0.05. The second phase started by the doubling of AcP activity between day 22 and 27 (p < 0.05), which remained stable from day 36 (from 4.7 ± 0.6 to 4.4 ± 0.3 U mg -1 ). In the third phase, AcP activity increased dramatically until the maximal value was observed few hours before hatching (reaching 19.9 ± 3.1 U mg -1 ).

Concerning cathepsin, the activity kinetic (Fig. 1. B) significantly increased from day 0 to day 27 (from 3.6 ± 0.8 to 7.2 ± 1.0 U mg -1 ; p < 0.05), after which it suddenly dropped to the minimal value at day 36 (2.9 ± 0.3 U mg -1 ; p < 0.05). Finally, cathepsin activity increased again to the maximal value (8.1 ± 2.5 U mg -1 ) at the end of embryonic development.

Metal effect on embryo development and enzyme activities

Exposures to dissolved Ag, Cd and Cu did not induce significant effects on the egg growth (results not shown) and on the egg weight at the end of the development compared to the control (Table 1; p > 0.05), suggesting an unaffected development. However, the hatchlings from the eggs exposed to 0.06, 1.2, 60 and 1200 ng Ag.l -1 were respectively 5, 9, 19 and 9 % lighter than the controls (Fig. 2). Similarly, the weight of the hatchlings from eggs exposed to 305 and 610 ng Cd.l -1 , and to 2.3, 23 and 230 µg Cu.l -1 were 5% and 8%, and 4%, 8% and 9%, respectively, lighter compared to the controls.

Effect of metal exposures on AcP activities was assessed on the exponential activity raise during the last 18 days of development. Comparisons between the AcP activity increase were applied on the increase activity rate (k) determined on the eggs reared in the different exposure conditions (Table 2). Thus, equation parameters revealed that Ag had no significant impact on the AcP activities although at 60 ng Ag.l -1 and 1.2 µg Ag.l -1 the increase rate seemed to be reduced (AcP; 0.092 ± 0.012 and 0.090 ± 0.012 d -1 vs. 0.113 ± 0.014 d -1 in eggs exposed to 60 ng Ag.l -1 and 1.2 µg Ag.l -1 vs. control, respectively). Cd exposure led to a significant inhibition on the AcP activity at 31, 61, 305 and 610 ng.l -1 . Contrasting to this, exposure to Cu at 2.3 µg.l -1 stimulated the AcP activity (0.095 ± 0.004 vs. 0.072 ± 0.006 d -1 in control eggs and eggs exposed to 2.3 µg Cu.l -1 , respectively), the other Cu concentrations having no effect on the AcP activity.

Concerning the cathepsins, lower activity was observed in eggs exposed to 610 ng Cd.l -1 ,at the end of the development (Table 2) while Ag exposure did not provoke any effect. The effect of dissolved Cu was not assessed because of the loss of the egg samples.

Discussion

The degradation of the yolk transferred in the form of precursors from the maternal ovary to the yolk platelets constitutes the only nutrient source for the zygote [START_REF] Fagotto | Changes in yolk platelet pH during Xenopus laevis development correlate with yolk utilization: A quantitative confocal microscopy study[END_REF]. Consequently, the processes that govern its degradation constitute a key function that will subsequently control and determine the embryo development. In this respect, lysosomal enzymes, i.e. acid phosphatase and cathepsin, are directly involved in the vitellus digestion (e.g. [START_REF] Lemanski | Role of acid phosphatase in the breakdown of yolk in developing amphibian embryos[END_REF][START_REF] Carnevali | Yolk formation and degradation during oocyte maturation in seabream Sparus aurata: involvement of two lysosomal proteinases[END_REF]. Nevertheless, contrasting to the "classical" lysosomes which are able to rapidly reduce proteins to free amino acids, the yolk platelet enzymes do not degrade maternal material until specific developmental stages are reached [START_REF] Fagotto | Regulation of yolk degradation, or how to make sleepy lysosomes[END_REF]. In this study, the AcP activity kinetic along the development closely followed the developmental stage course (see Fig. 1.A). Indeed, AcP activity remained at a low level during the first 22 d of development. This period corresponds to 1) the cleavage (blastula) and gastrula phases (until day 13, i.e., stage 17, Table 3) during which the yolk is progressively covered by the yolk sac envelope, and to 2) the patterning phase in the organogenetic zone requiring a convex substrate surface such as the one offered by the uncleaved yolk mass [START_REF] Boletzky | Yolk sac morphology in Cephalopod embryos[END_REF]. After this period, the AcP activity doubled between days 22 and 27 (i.e., stage 25 which corresponds to the end of the organogenesis according to [START_REF] Boletzky | Sepia officinalis[END_REF]; Table 3). This observation strongly suggests that the last stages of the organogenesis require more energy than the previous phases. This is supported by the fact that the embryo grew up from 0.8 mm to 2 mm between the stages 22 and 25 [START_REF] Lemaire | Table de développement embryonnaire de Sepia officinalis L. (Mollusque Céphalopode)[END_REF].

From this time onwards, the post-organogenic phase started. Therefore, an important yolk resorption occurred supplying the nutrient needs for the embryo's growth as suggested by the increasing AcP activity. A similar pattern of AcP activity along the development was already highlighted in frog eggs [START_REF] Fagotto | Changes in yolk platelet pH during Xenopus laevis development correlate with yolk utilization: A quantitative confocal microscopy study[END_REF]. These authors reported an increase of the acidified yolk platelets in which AcP was activated, according to increasing nutrient needs of the embryo. Moreover, during the last two weeks of cuttlefish embryo development, the yolk was transferred from the outer to the inner yolk sack to facilitating its assimilation. In parallel, the nutrient digestion by the syncytium was progressively replaced by the new developed digestive system of the cuttlefish embryo himself. Thus, from the stage 27 (day 32; Table 3) "boules" cells, which are typical digestive gland cells with a well developed lysosomal system appeared in the hepatic epithelium [START_REF] Lemaire | Le foie embryonnaire de Sepia officinalis L. (Mollusque Cephalopode). I. Organogénèse[END_REF]. The occurrence of "boules" cells indicates that an intracellular digestive capacity is already established as reported for juveniles [START_REF] Boucaud-Camou | Digestives enzymes in paralarval cephalopods[END_REF]. Moreover, a previous work on the biochemical characterization of the AcP from the yolk and the embryo revealed the existence of two protein forms (unpublished data). Thus, the strong increase of the AcP activity observed during the last 14 days could be directly due to de novo production of lysosomal enzymes such as AcP by the embryo. Indeed, the digestive system of the future juvenile matured during the last embryonic stages.

As for AcP, cathepsin activity (Fig. 1.B.) varied with time according to the developmental stage. Only few studies reported such variations along the embryonic development [START_REF] Kestemont | Cathepsin L in eggs and larvae of perch Perca fluviatilis: variations with the developmental stage and spawning period[END_REF][START_REF] Carnevali | Changes of lysosomal enzyme activities in sea bass (Dicentrarchus labrax) eggs and developing embryos[END_REF]. Nevertheless, [START_REF] Carnevali | Changes of lysosomal enzyme activities in sea bass (Dicentrarchus labrax) eggs and developing embryos[END_REF] reported the highest cathepsin activity in fish egg during the first developmental stages suggesting that these enzymes were involved in the patterning phase of embryo tissues. Consistently, a similar process seems to occur in cuttlefish eggs as cathepsin activity increased during the first 27 days of development, i.e., during the organogenesis phase. In frog eggs, the progressive increase of cathepsin activity in the early developmental stages was linked to the cleavage of the cathepsin D to a lighter protein form, which was five times more efficient [START_REF] Yoshizaki | Cysteine proteinase plays a key role for the initiation of yolk digestion during development of Xenopus laevis[END_REF]. After this period of development, these authors reported a dramatic decrease of cathepsin activity. In cuttlefish embryos, a similar decrease was observed between days 27 and 36, likely caused by mutual digestion. Interestingly, the cathepsin activity increased again during the last 14 d of development of cuttlefish eggs. This strongly suggests that, as observed for AcP, new proteins were produced de novo in the developing intracellular digestive system of the embryo. Whereas cuttlefish hatchlings are considered as "small" adults, our results complement previous observations highlighting that the transition between the embryonic and sub-adult phases occur from the last days of the embryo development until the first weeks of the juvenile life. For instance, [START_REF] Decleir | The differentiation of blood proteins during ontogeny in Sepia officinalis[END_REF] demonstrated a gradual shift from embryonic to juvenile hemocyanin forms and then to the adult form, with 11 different protein forms occurring between the embryo and the 2 month-old cuttlefish.

Although the lysosomal system of the digestive gland of cephalopod was involved in the trace elements storage and detoxification processes (e.g. [START_REF] Tanaka | Subcellular distribution and binding of heavy metals in the untreated liver of the squid; comparison with data from the livers of cadmium and silver exposed rats[END_REF]Bustamante et al., 2002b[START_REF] Bustamante | Subcellular distribution of Ag, Cd, Co, Cu, Fe, Mn, Pb, and Zn in the digestive gland of the common cuttlefish Sepia officinalis[END_REF], to the best of our knowledge only one study focused on the impact of metals on the intracellular digestion enzymes in cephalopods [START_REF] Le Bihan | Development of a bioassay from isolated digestive gland cells of the cuttlefish Sepia officinalis L. (Mollusca Cephalopoda): effect of Cu, Zn, and Ag on enzymes activities and cell viability[END_REF]. These authors found an inhibition of the cathepsin activities from digestive gland cells exposed in vitro to Cu and Zn at high concentration (i.e., 1.2 and 1.3 mg.l -1 , respectively). However, Ag and Zn at lower concentrations (i.e. 2.2 and 1.3 µg.l -1 ) stimulated the cathepsin activity. Our study highlighted the inhibitory effect of Cd (and the potential effect of Ag) and the positive impact of Cu on the AcP and cathepsin activities during the embryo growth period, suggesting that 1) dissolved metals could globally disturb the embryogenesis conditions and then affect indirectly the digestive system maturation and/or 2) that the accumulated metal fraction in the embryonic tissues leads to biochemical interactions with the enzymes produced de novo throughout the maturation of the digestive gland. First, no direct cause-consequence relationship was demonstrated between the metal response of the enzymatic activities involved in the yolk digestion and the hatchling weight at the end of embryonic development.

For example, the lighter juveniles hatched from eggs exposed to Ag at 50 ng.l -1 were related to no significant AcP and cathepsin inhibition at this concentration. One explanation is that metals could affect other physiological functions such as the immune system [START_REF] Establier | Efecto del cadmio y el cobre sobre el desarrollo de los huevos de Sepia officinalis Linneo[END_REF][START_REF] Lacoue-Labarthe | Phenoloxidase activation in the embryo of the common cuttlefish Sepia officinalis and responses to the Ag and Cu exposure[END_REF], the osmoregulation processes (e.g. [START_REF] Wu | Effects of cadmium and zinc on oxygen consumption, ammonium excretion, and osmoregulation of white shrimp (Litopenaeus vannamei)[END_REF][START_REF] Bianchini | Mechanism of acute silver toxicity in marine invertebrates[END_REF] or the acid-base balance [START_REF] Bielmyer | The effects of metals on embryo-larval and adult life stages of the sea urchin, Diadema antillarum[END_REF]. In the same way, this observation supports the idea that the measured inhibition or activation of the AcP and cathepsin activities could be the indirect consequence of potential metals impacts on other biological traits of the embryo. Secondly, considering the accumulated metals could have interfered with digestive enzymes, the higher sensibility of AcP and cathepsin activities for Cd compared to Ag was noteworthy taking into account the fact that cuttlefish embryo showed much higher accumulation capacities for Ag compared to Cd during the last month of embryo development (Lacoue-Labarthe et al., 2008a). This result may arise for two reasons: first, the immature embryo could not already have developed efficient detoxification processes such as binding of the Cd to specific proteins into the cytosol [START_REF] Tanaka | Subcellular distribution and binding of heavy metals in the untreated liver of the squid; comparison with data from the livers of cadmium and silver exposed rats[END_REF][START_REF] Finger | Molecular association of Cu, Zn, Cd and 210 Po in the digestive gland of the squid Nototodarus gouldi[END_REF]Bustamante et al., 2002b) whereas Ag was sequestered in the insoluble fraction, i.e. associated to cellular organelles and/or granules [START_REF] Bustamante | Subcellular distribution of Ag, Cd, Co, Cu, Fe, Mn, Pb, and Zn in the digestive gland of the common cuttlefish Sepia officinalis[END_REF]. Subsequently, free cytosolic Cd may cause lysosomal membrane destabilisation leading eventually to a leakage of the enzymes into the cytosol [START_REF] Viarengo | Metallothioneins and lysosomes in metal toxicity and accumulation in marine mussels: the effect of cadmium in the presence and absence of phenanthrene[END_REF] and consequently to their inhibition. Further studies should be carried out to confirm these results and determine the subcellular distribution of these metals in the embryonic tissues in order to assess the toxicity mechanism of Ag and Cd towards the enzymatic activities. Conversely, Cu stimulated the AcP activity in eggs exposed to 2.3 µg.l -1 probably in accordance with its essential character in the cephalopod metabolism, e.g. as a haemocyanin co-factor [START_REF] Decleir | A study of the blood proteins in Sepia officinalis L. with special reference to embryonic hemocyanin[END_REF]. However, this positive effect disappeared at higher level exposure, suggesting that the accumulated metal in the egg reached a threshold value from which potentially deleterious Cu effects may be started [START_REF] Paulij | The effect of copper on the embryonic development and hatching of Sepia officinalis L[END_REF][START_REF] Lacoue-Labarthe | Phenoloxidase activation in the embryo of the common cuttlefish Sepia officinalis and responses to the Ag and Cu exposure[END_REF].

Finally, on one side, this study highlighted the variations of AcP and cathepsin activities during the embryogenesis and suggested a progressive setting up of a lysosomal system in the maturing digestive gland during the last day of the embryonic life. On the other side, in our experimental conditions, these results gave first insight on the effect of metals on these enzymatic activities and pointed to the potential sensibility of the less than one-month old juveniles as the metals they accumulated via both seawater and dietary pathways could also disturb or enhanced their intracellular digestion efficiency. Whereas no metal effect on the AcP and cathepsin activities was observed when the eggshell protects the embryo against metal penetration, i.e. the first month of development (Bustamante et al., 2002b[START_REF] Bustamante | Uptake, transfer and distribution of silver and cobalt in the tissues of the common cuttlefish Sepia officinalis at different stages of its life cycle[END_REF]Lacoue-Labarthe et al., 2008a[START_REF] Lacoue-Labarthe | Contrasting biokinetics of accumulation and distribution of Am, Co, Cs, Mn and Zn by the eggs of the common cuttlefish (Sepia officinalis) during the whole development time[END_REF], further studies should verify the impact of the metals that are maternally transferred such as Ag or Zn (Lacoue-Labarthe et al., 2008b), and that could potentially induce an inhibition or a delay on the AcP and cathepsin activation contained in the yolk platelets. with the main embryonic events. The developmental stages were defined according to [START_REF] Lemaire | Table de développement embryonnaire de Sepia officinalis L. (Mollusque Céphalopode)[END_REF]. Captions to figures

Fig 1 .

 1 Fig 1. Sepia officinalis. Acid phosphatases (A) and cathepsin (B) specific activities (r) in the whole egg along the development (mean ± SEM, U mg -1 of protein, n = 3 pools of 2 eggs). Different letters denote statistically significant differences (for p < 0.05) between the sampling times (a ≠ b ≠ c ≠ d ≠ e ≠ f ≠ g)

Table 1 .

 1 Sepia officinalis. Wet weight of the eggs at the end of embryonic development (mean ± SEM, g ; n = 8) following exposure to five metal concentrations (Control, Conc. 1, Conc. 2, Conc.3, Conc. 4), which correspond to 0.06, 1.2, 60, 1200 ng Ag l -1 , to 31, 61, 305, 610 ng Cd l -1 and to 0.23, 2.3, 23, 230 µg Cu l -1 , respectively.

	Concentration	Ag	Cd	Cu
	Control	4.90 ± 0.21	4.90 ± 0.21	4.03 ± 0.15
	Conc. 1	5.14 ± 0.36	4.88 ± 0.24	4.31 ± 0.16
	Conc. 2	4.95 ± 0.28	4.94 ± 0.40	4.16 ± 0.19
	Conc. 3	5.50 ± 0.23	5.17 ± 0.42	4.35 ± 0.24
	Conc. 4	4.89 ± 0.31	5.08 ± 0.24	4.52 ± 0.20

Table 2 .

 2 Sepia officinalis. Parameters of the exponential equations describing the kinetics of the acid phosphatases (U mg -1 prot; n = 3 pools of 2 eggs at each sample time) activities during the last 18 days of the development and cathepsin activity at the end of development (d48), measured in the whole eggs exposed to Ag, Cd and Cu all along the development time

	Acid Phosphatases (last 18 days)	Cathepsin (d48)

eggs exposed to Cd during the development

	Control	0.085 ± 0.056	0.113 ± 0.014		0.677		6.199 ± 1.975
	31 ng.l -1	0.621 ± 0.199	0.062 ± 0.007	*	0.703	0.007	5.962 ± 1.662		1.000
	61 ng.l -1	1.356 ± 0.284	0.047 ± 0.005	*	0.772	< 0.001	6.362 ± 0.682		0.297
	305 ng.l -1	0.417 ± 0.118	0.073 ± 0.006	*	0.847	0.030	7.090 ± 2.048		0.387
	610 ng.l -1	0.760 ± 0.190	0.057 ± 0.006	*	0.803	0.004	4.380 ± 1.047	*	0.019
	(c)

eggs exposed to Cu during the development

	Control	0.256 ± 0.075	0.072 ± 0.006		0.863		-
	0.23 µg.l -1	0.127 ± 0.042	0.085 ± 0.007		0.910	0.150	-	-
	2.3 µg.l -1	0.092 ± 0.016	0.095 ± 0.004	*	0.972	0.002	-	-
	23 µg.l -1	0.151 ± 0.089	0. 084 ± 0.012		0.699	0.383	-	-
	230 µg.l -1	0.220 ± 0.051	0.072 ± 0.005		0.914	0.980	-	-
	* Significance difference from control, p < 0.05

Table 3 .

 3 Sepia officinalis. Timetable of the embryonic development of the eggs reared at 17°C

Acknowledgments

Authors thank MM. Lapie & Lapie, Mr. Guenon, Mr. Le Monnier and Mr. Longuet, fisherman of the West-Cotentin coast, Normandy, France, for providing the cuttlefish eggs needed for this study. Authors are grateful to A. Coulon, A. Meslon, A. Montcuit, H. Viala and G. Safi for their technical assistance during experiments. Part of this work was conducted in CREC (Centre de Recherche sur les Ecosystèmes Côtiers) at Luc sur Mer (France).


Sarah Filippi
email: filippi@telecom-paristech.fr

Olivier Cappé
email: cappe@telecom-paristech.fr

Garivier Aurélien
email: garivier@telecom-paristech.fr

Ltci

Optimism in Reinforcement Learning and Kullback-Leibler Divergence

Keywords: Reinforcement learning, Markov decision processes, Modelbased approaches, Optimism, Kullback-Leibler divergence, Regret bounds

), which follows this strategy, has recently been shown to guarantee near-optimal regret bounds. In this paper, we strongly argue in favor of using the Kullback-Leibler (KL) divergence for this purpose. By studying the linear maximization problem under KL constraints, we provide an efficient algorithm, termed KL-UCRL, for solving KL-optimistic extended value iteration. Using recent deviation bounds on the KL divergence, we prove that KL-UCRL provides the same guarantees as UCRL2 in terms of regret. However, numerical experiments on classical benchmarks show a significantly improved behavior, particularly when the MDP has reduced connectivity. To support this observation, we provide elements of comparison between the two algorithms based on geometric considerations.

Introduction

In reinforcement learning, an agent interacts with an unknown environment, aiming to maximize its long-term payoff [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. This interaction is modelled by a Markov Decision Process (MDP) and it is assumed that the agent does not know the parameters of the process and needs to learn directly from observations. The agent thus faces a fundamental trade-off between gathering experimental data about the consequences of the actions (exploration) and acting consistently with past experience in order to maximize the rewards (exploitation).

We consider in this article a MDP with finite state and action spaces for which we propose a model-based reinforcement learning algorithm, i.e., an algorithm that maintains running estimates of the model parameters (transitions probabilities and expected rewards) [START_REF] Even-Dar | Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems[END_REF][START_REF] Kearns | Near-optimal reinforcement learning in polynomial time[END_REF][START_REF] Strehl | An analysis of model-based interval estimation for Markov decision processes[END_REF][START_REF] Tewari | Optimistic linear programming gives logarithmic regret for irreducible MDPs[END_REF]. A well-known approach to balance exploration and exploitation, followed for example by the well-know algorithm R-MAX [START_REF] Brafman | R-max-a general polynomial time algorithm for near-optimal reinforcement learning[END_REF], is the so-called optimism in the face of uncertainty principle. It was first proposed in the multi-armed bandit context by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], and has been extended since then to several frameworks: instead of acting optimally according to the estimated model, the agent follows the optimal policy for a surrogate model, named optimistic model, which is close enough to the former but leads to a higher long-term reward. The performance of such an algorithm can be analyzed in terms of regret, which consists in comparing the rewards collected by the algorithm with the rewards obtained when following an optimal policy. The study of the asymptotic regret due to [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] in the multi-armed context has been extended to MDPs by [START_REF] Burnetas | Optimal adaptive policies for Markov decision processes[END_REF], proving that an optimistic algorithm can achieve logarithmic regret. The subsequent works [START_REF] Auer | Logarithmic online regret bounds for undiscounted reinforcement learning[END_REF][START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF][START_REF] Bartlett | REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs[END_REF] introduced algorithms that guarantee non-asymptotic logarithmic regret in a large class of MDPs. In these latter works, the optimistic model is computed using the L 1 (or total variation) norm as a measure of proximity between the estimated and optimistic transition probabilities.

In addition to logarithmic regret bounds, the UCRL2 algorithm of [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF] is also attractive due to the simplicity of each L 1 extended value iteration step. In this case, optimism simply results in adding a bonus to the most promising transition (i.e., the transition that leads to the state with current highest value) while removing the corresponding probability mass from less promising transitions. This process is both elementary and easily interpretable, which is desirable in some applications.

However, the L 1 extended value iteration leads to undesirable pitfalls, which may compromise the practical performance of the algorithm. First, the optimistic model is not continuous with respect to the estimated parameters -small changes in the estimates may result in very different optimistic models. More importantly, the L 1 optimistic model can become incompatible with the observations by assigning a probability of zero to a transition that has actually been observed. Moreover, in MDPs with reduced connectivity, L 1 optimism results in a persistent bonus for all transitions heading towards the most valuable state, even when significant evidence has been accumulated that these transitions are impossible.

In this paper, we propose an improved optimistic algorithm, called KL-UCRL, that avoids these pitfalls altogether. The key is the use of the Kullback-Leibler (KL) pseudo-distance instead of the L 1 metric, as in [START_REF] Burnetas | Optimal adaptive policies for Markov decision processes[END_REF]. Indeed, the smoothness of the KL metric largely alleviates the first issue. The second issue is completely avoided thanks to the strong relationship between the geometry of the probability simplex induced by the KL pseudo-metric and the theory of large deviations. For the third issue, we show that the KL-optimistic model results from a trade-off between the relative value of the most promising state and the statistical evidence accumulated so far regarding its reachability.

We provide an efficient procedure, based on one-dimensional line searches, to solve the linear maximization problem under KL constraints. As a consequence, the numerical complexity of the KL-UCRL algorithm is comparable to that of UCRL2. Building on the analysis of [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF][START_REF] Bartlett | REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs[END_REF][START_REF] Auer | Near-optimal regret bounds for reinforcement learning[END_REF], we also obtain logarithmic regret bounds for the KL-UCRL algorithm. The proof of this result is based on novel concentration inequalities for the KL-divergence, which have interesting properties when compared with those traditionally used for the L 1 norm. Although the obtained regret bounds are comparable to earlier results in term of rate and dependence in the number of states and actions, we observed in practice significant performance improvements. This observation is illustrated using benchmark examples (the RiverSwim and SixArms environments of [START_REF] Strehl | An analysis of model-based interval estimation for Markov decision processes[END_REF]) and through a thorough discussion of the geometric properties of KL neighborhoods.

The paper is organized as follows. The model and a brief survey of the value iteration algorithm in undiscounted MDPs are presented in Section 2. Section 3 and 4 are devoted, respectively, to the description and the analysis of the KL-UCRL algorithm. Section 5 contains numerical experiments and Section 6 concludes the paper by discussing the advantages of using KL rather than L 1 confidence neighborhoods.

Markov Decision Process

Consider a Markov decision process (MDP) M = (X , A, P, r) with finite state space X , and action space A. Let X t ∈ X and A t ∈ A denote respectively the state of the system and the action chosen by the agent at time t. The probability to jump from state X t to state X t+1 is denoted by P (X t+1 ; X t , A t ). Besides, the agent receives at time t a random reward R t ∈ [0, 1] with mean r(X t , A t ). The aim of the agent is to choose the sequence of actions so as to maximize the cumulated reward. His choices are summarized in a stationary policy π : X → A.

In this paper, we consider communicating MDPs, i.e., MDPs such that for any pair of states x, x ′ , there exists policies under which x ′ can be reached from x with positive probability. For those MDPs, it is known that the average reward following a stationary policy π, denoted by ρ π (M) and defined as

ρ π (M) = lim n→∞ 1 n E M,π n t=0 R t ,
is state-independent [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. Let π * (M) : X → A and ρ * (M) denote respectively the optimal policy and the optimal average reward:

ρ * (M) = sup π ρ π (M) = ρ π * (M) (M)
. The notations ρ * (M) and π * (M) are meant to highlight the fact that both the optimal average reward and the optimal policy depend on the model M. The optimal average reward satisfies the so-called Bellman optimality equation: for all x ∈ X ,

h * (M, x) + ρ * (M) = max a∈A r(x, a) + x ′ ∈X P (x ′ ; x, a)h * (M, x ′ ) ,
where the |X |-dimensional vector h * (M) is called a bias vector. Note that it is only defined up to an additive constant. For a fixed MDP M, the optimal policy π * (M) can be derived by solving the optimality equation and by defining, for all x ∈ X ,

π * (M, x) ∈ argmax a∈A r(x, a) + x ′ ∈X P (x ′ ; x, a)h * (M, x) .
In practice, the optimal average reward and the optimal policy may be computed, for instance, using the value iteration algorithm [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF].

The KL-UCRL algorithm

In this paper, we focus on the reinforcement learning problem in which the agent does not know the model M beforehand, i.e. the transition probabilities and the distribution of the rewards are unknown. More specifically, we consider model-based reinforcement learning algorithms which estimate the model through observations and act accordingly. Denote by Pt (x ′ ; x, a) the estimate at time t of the transition probability from state x to state x ′ conditionally to the action a, and, by rt (x, a) the mean reward received in state x when action a has been chosen. We have:

Pt (x ′ ; x, a) = N t (x, a, x ′ ) max(N t (x, a), 1) rt (x, a) = t-1 k=0 R k 1 {X k =x,A k =a} max(N t (x, a), 1) , (1)
where

N t (x, a, x ′ ) = t-1 k=0 1 {X k =x,A k =a,X k+1 =x ′ }
is the number of visits, up to time t, to the state x followed by a visit to x ′ when the action a has been chosen, and similarly,

N t (x, a) = t-1 k=0 1 {X k =x,A k =a} .
The optimal policy in the estimated model M t = (X , A, Pt , rt ) may be misleading due to estimation errors: pure exploitation policies are commonly known to fail with positive probability. To avoid this problem, optimistic model-based approaches consider a set M t of potential MDPs including M t and choose the MDP from this set that leads to the largest average reward. In the following, the set M t is defined as follows:

M t = {M = (X , A, P, r) : ∀x ∈ X , ∀a ∈ A, |r t (x, a) -r(x, a)| ≤ ǫ R (x, a, t)
and d( Pt (.; x, a), P (.; x, a)) ≤ ǫ P (x, a, t)} , where d measures the difference between the transition probabilities. The radius of the neighborhoods ǫ R (x, a, t) and ǫ P (x, a, t) around, respectively, the estimated reward rt (x, a) and the estimated transition probabilities Pt (.; x, a), decrease with N t (x, a).

In contrast to UCRL2, which uses the L 1 -distance for d, we propose to rely on the Kullback-Leibler divergence, as in the seminal article [START_REF] Burnetas | Optimal adaptive policies for Markov decision processes[END_REF]; however, contrary to the approach of [START_REF] Burnetas | Optimal adaptive policies for Markov decision processes[END_REF], no prior knowledge on the state structure of the MDP is needed. Recall that the Kullback-Leibler divergence is defined for all n-dimensional probability vectors p and q by KL(p, q) = n i=1 p i log pi qi (with the convention that 0 log 0 = 0). In the sequel, we will show that this choice dramatically alters the behavior of the algorithm and leads to significantly better performance, while causing a limited increase of complexity; in Section 6, the advantages of using a KL-divergence instead of the L 1 -norm are illustrated and argumented.

The KL-UCRL algorithm

The KL-UCRL algorithm, described below, is a variant of the efficient modelbased algorithm UCRL2, introduced by [START_REF] Auer | Near-optimal regret bounds for reinforcement learning[END_REF] and extended to more general MDPs by [START_REF] Bartlett | REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs[END_REF]. The key step of the algorithm, the search for the optimistic model (Step 8), is detailed below as Algorithm 2.

Algorithm 1 KL-UCRL 1: Initialization: j = 0, t 0 = 0; ∀a ∈ A, ∀x ∈ X , n 0 (x, a) = 0, N 0 (x, a) = 0; initial policy π 0 . 2: for all t ≥ 1 do 3: Observe X t 4: if n j (X t , π j (X t )) ≥ max(N tj (X t , π j (X t )), 1) then 5:
Begin a new episode: j = j + 1, t j = t,

6:

Reinitialize: ∀a ∈ A, ∀x ∈ X , n j (x, a) = 0

7:

Estimate Pt and rt according to (1)

8:

Find the optimistic model M j ∈ M t and the related policy π j solving equation (2) and using Algorithm 2

9: end if 10: Choose action A t = π j (X t ) 11:
Receive reward R t

12:

Update the count within the current episode:

n j (X t , A t ) = n j (X t , A t ) + 1 13:
Update the global count:

N t (X t , A t ) = N t-1 (X t , A t ) + 1 14: end for
The KL-UCRL algorithm proceeds in episodes. Let t j be the starting time of episode j; the length of the j-th episode depends on the number of visits N tj (x, a) to each state-action pair (x, a) before t j compared to the number of visits n j (x, a) to the same pair during the j-th episode. More precisely, an episode ends as soon as n j (x, a) ≥ N tj (x, a) for some state-action pair (x, a). The policy π j , followed during the j-th episode, is an optimal policy for the optimistic MDP M j = (X , A, P j , r j ) ∈ M tj , which is computed by solving the extended optimality equations: for all x ∈ X

h * (x) + ρ * = max P,r max a∈A r(x, a) + x ′ ∈X P (x ′ ; x, a)h * (x ′ ) (2)
where the maximum is taken over all P, r such that ∀x, ∀a, KL( Ptj (.; x, a), P (.; x, a))

≤ C P N tj (x, a) , ∀x, ∀a, |r tj (x, a) -r(x, a)| ≤ C R N tj (x, a) ,
where C P and C R are constants which control the size of the confidence balls.

The transition matrix P j and the mean reward r j of the optimistic MDP M j maximize those equations. The extended value iteration algorithm may be used to approximately solve the fixed point equation (2) [13, 1].

Maximization of a linear function on a KL-ball

At each step of the extended value iteration algorithm, the maximization problem (2) has to be solved. For every state x and action a, the maximization of r(x, a) under the constraint that |r tj (x, a)r(x, a)| ≤ C R / N tj (x, a) is obviously solved taking r(x, a) = rtj (x, a) + C R / N tj (x, a), so that the main difficulty lies in maximizing the dot product between the probability vector q = P (.; x, a) and the value vector V = h * over a KL-ball around the fixed probability vector p = Ptj (.; x, a):

max q∈S |X | V ′ q s.t. KL(p, q) ≤ ǫ , (3)
where V ′ denotes the transpose of V and S n the set of n-dimensional probability vectors. The radius of the neighborhood ǫ = C P /N tj (x, a) controls the size of the confidence ball. This convex maximization problem is studied in Appendix A, leading to the efficient algorithm presented below. Detailed analysis of the Lagrangian of (3) shows that the solution of the maximization problem essentially relies on finding roots of the function f (that depends on the parameter V ), defined as follows: for all ν ≥ max i∈ Z V i , with Z = {i :

p i > 0}, f (ν) = i∈ Z p i log(ν -V i ) + log   i∈ Z p i ν -V i   . (4)
In the special case where the most promising state i M has never been reached from the current state-action pair (i.e. p iM = 0), the algorithm makes a tradeoff between the relative value of the most promising state V iM and the statistical evidence accumulated so far regarding its reachability.

Algorithm 2 Function MaxKL

Input A value function V , a probability vector p, a constant ǫ Output A probability vector q that maximizes (3)

1: Let Z = {i : p i = 0} and Z = {i : p i > 0}. 2: Let I * = Z ∩ argmax i V i 3: if I * = ∅ and there exists i ∈ I * such that f (V i ) < ǫ then 4:
Let ν = V i and r = 1exp(f (ν)ǫ).

5:

For all i ∈ I * , assign values of q i such that i∈I * q i = r .

6:

For all i ∈ Z/I * , let q i = 0. 7: else

8:

For all i ∈ Z, let q i = 0. Let r = 0.

9:

Find ν such that f (ν) = ǫ using Newton's method. 10: end if 11: For all i ∈ Z, let q i = (1-r)qi i∈ Z qi where qi = pi ν-Vi .

In practice, f being a convex positive decreasing function (see Appendix B), Newton's method can be applied to find ν such that f (ν) = ǫ (in Step 9 of the algorithm), so that numerically solving (3) is a matter of a few iterations. Appendix B contains a discussion of the initialization of Newton's algorithm based on asymptotic arguments.

Regret bounds 4.1 Theoretical results

To analyze the performance of KL-UCRL, we compare the rewards accumulated by the algorithm to the rewards that would be obtained, on average, by an agent playing an optimal policy. The regret of the algorithm after T steps is defined as in [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF]:

Regret T = T t=1 (ρ * (M) -R t ) .
We adapt the regret bound analysis of the UCRL2 algorithm to the use of KL-neighborhoods, and obtain similar theorems. Let

D(M) = max x,x ′ min π E M,π (τ (x, x ′ )) ,
where τ (x, x ′ ) is the hitting time of x ′ , starting from state x. The D(M) constant will appear in the regret bounds. For all communicating MDPs M, D(M) is finite. Theorem 1 establishes an upper bound on the regret of the KL-UCRL algorithm with C P and C R defined as

C P = |X | B + log B + 1 log(T ) 1 + 1 B + 1 log(T ) where B = log 2e|X | 2 |A| log(T ) δ and C R = log (4|X ||A| log(T )/δ) 1.
99 .

Theorem 1 With probability 1δ, it holds that for T > 5, the regret of KL-UCRL is bounded by

Regret T ≤ CD(M)|X | |A|T log(log(T )/δ) ,
for a constant C ≤ 24 that does not dependent on the model.

It is also possible to prove a logarithmic upper bound for the expected regret. This bound, presented in Theorem 2, depends on the model through the constant ∆(M) defined as ∆(M) = ρ * (M)max π,ρ π (M)<ρ * (M) ρ π (M). ∆(M) quantifies the margin between optimal and suboptimal policies.

Theorem 2 For T > 5, the expected regret of KL-UCRL is bounded by

E(Regret T ) ≤ CD 2 (M) |X | 2 |A| log(T ) ∆(M) + C(M) ,
where C ≤ 400 is a constant independent of the model, and C(M) is a constant which depends on the model (see [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF]).

Elements of proof

The proof of Theorem 1 is inspired from [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF][START_REF] Bartlett | REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs[END_REF]. Due to the lack of space, we only provide the main steps of the proof. First, the following proposition enables us to ensure that, with high probability, the true model M = (X , A, P, r) belongs to the set of models M t at each time step.

Proposition 1 For every horizon T ≥ 1 and for δ > 0,

P (∀t ≤ T , M ∈ M t ) ≥ 1 -2δ.
The proof relies on the two following concentration inequalities due to [START_REF] Garivier | Context tree selection: A unifying view[END_REF][START_REF] Garivier | On upper-confidence bound policies for nonstationary bandit problems[END_REF]: for all x ∈ X , a ∈ A, any C P > 0, and C R > 0, it holds that P ∀t ≤ T, KL( Pt (.; x, a), P (.; x, a)) > C P N t (x, a)

≤ 2e(C P log(T ) + |X |)e -C P |X | = 1 - δ |X ||A| (5)
P ∀t ≤ T, |r t (x, a) -r(x, a)| ≤ C R N t (x, a) ≤ 4 log(T )e -1.99CR = 1 - δ |X ||A| .
Then, summing over all state-action pairs, Proposition 1 follows. Using Hoeffding's inequality, with high probability, the regret at time T can be written as the sum of a regret in each of the m(T ) episodes plus an additional term C e (T, δ)) = T log(1/δ)/2:

Regret T ≤ (x,a) N T (x, a)(ρ * (M) -r(x, a)) + C e (T, δ) ≤ m(T ) k=1 (x,a) n k (x, a)(ρ * (M) -r(x, a)) + C e (T, δ)
Let P k and π k denote, respectively, the transition probability matrix of the optimistic model and the optimal policy in the k-th episode (1 ≤ k ≤ m(T )). It is easy to show that (see [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF] for details), with probability 1δ,

Regret T ≤ m(T ) k=1 x∈X n k (x, π k (x)) + (r k (x, π k (x)) -r(x, π k (x))) (P k (.; x, π k (x)) -P (.; x, π k (x))) ′ h k + (P (.; x, π k (x)) -e x ) ′ h k + C e (T, δ) ,
where h k is a bias vector, e x (y) = 1 if x = y and e x (y) = 0 otherwise. We now bound each of the three terms in the previous summation. Denote by π k k the row vector such that n π k k (x) = n k (x, π k (x)) and by r π k k (resp. r π k the column vector such that r

π k k (x) = r k (x, π k (x)) (resp. r π k (x) = r(x, π k (x))). Similarly P π k k (resp. P π k
) is the transition matrix if the policy π k is followed under the optimistic model M k (resp. the true model M). If the true model M ∈ M t k , we have for all x ∈ X , for all a ∈ A,

n π k k (r π k k -r π k ) ≤ 2 (x,a) n k (x, a) C R N t (x, a) (6)
Using Pinsker's inequality, and the fact that h k ∞ ≤ D [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF],

n π k k (P π k k -P π k )h k ≤ (x,a) n k (x, a) P k (.; x, a) -P (.; x, a) 1 h k ∞ ≤ 2D √ 2 (x,a) n k (x, a) C P N t k (x, a) . (7)
The third term n π k k (P π k -I)h k may be written as follows:

n π k k (P π k -I)h k ≤ t k+1 -1 t=t k (P (.; X t , A t ) -e Xt+1 )h k + h k (X t+1 ) -h k (X t ) ,
where e x is the all 0's vector with a 1 only on the x-th component. For all ∈ [t k , t k+1 -1], note that ξ t = (P (.; X t , A t )-e Xt+1 )h k is a martingale difference upper-bounded by D. Applying the Azuma-Hoeffding inequality, we obtain that

m(T ) k=1 n π k k (P π k -I)h k = T t=1 ξ t + m(T )D ≤ D T log(1/δ) 2 + m(T )D (8)
with probability 1δ. In addition, Auer and al [START_REF] Auer | Near-optimal regret bounds for reinforcement learning[END_REF] proved that

m(T ) k=1 x,a n k (x, a) N t k (x, a) ≤ ( √ 2 + 1) |X ||A|T and m(T ) ≤ |X ||A| log 2 8T |X ||A| .
Combining all the terms completes the proof of Theorem 1. The proof of Theorem 2 follows from Theorem 1 using the same arguments as in the proof of Theorem 4 in [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF].

Numerical experiments

To compare the behavior of algorithms KL-UCRL and UCRL2, we consider the benchmark environments RiverSwim and SixArms proposed by [START_REF] Strehl | An analysis of model-based interval estimation for Markov decision processes[END_REF] as well as a collection of randomly generated sparse environments. The RiverSwim environment consists of six states. The agent starts from the left side of the row and, in each state, can either swim left or right. Swimming to the right (against the current of the river) is successful with probability 0.35; it leaves the agent in the same state with a high probability equal to 0.6, and leads him to the left with probability 0.05 (see Figure 1). On the contrary, swimming to the left (with the current) is always successful. The agent receives a small reward when he reaches the leftmost state, and a much larger reward when reaching the rightmost state -the other states offer no reward. This MDP requires efficient exploration procedures, since the agent, having no prior idea of the rewards, has to reach the right side to discover which is the most valuable state-action pair. The SixArms environment consists of seven states, one of which (state From the initial state, the agent may choose one among six actions: the action a ∈ {1, . . . , 6} leads to the state x = a with probability p a (see Figure 2) and let the agent in the initial state with probability 1p a . From all the other states, some actions deterministically lead the agent to the initial state while the others leave it in the current state. Staying in a state x ∈ {1, . . . , 6}, the agent receives a reward equal to R x (see Figure 2), otherwise, no reward is received.

We compare the performance of the KL-UCRL algorithm to UCRL2 using 20 Monte-Carlo replications. For both algorithms, the constants C P and C R are settled to ensure that the upper bounds of the regret of Theorem 1 and Theorem 2 in [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF] hold with probability 0.95. In the SixArms environment, the received rewards being deterministic, we slightly modify both algorithms so that the agent knows them beforehand. We observe in Figure 3 and 4 that the KL-UCRL algorithm accomplishes a smaller average regret than the UCRL2 algorithm in those benchmark environments. In both environments, it is crucial for the agent to learn that there is no action leading from some states to the most promising one: for example, in the RiverSwim environment, between one of the first four states and the sixth state.

In addition to those benchmark environments, a generator of sparse environments has been used to create 10-states and 5-actions environments with random rewards in [0, 1]. In these random environments, each state is connected with, on average, five other states (with transition probabilities drawn from a Dirichlet distribution). We reproduced the same experiments as in the previous environments and display the average regret in Figure 5.

Discussion

In this section, we expose the advantages of using a confidence ball based on the Kullback-Leibler divergence rather than an L 1 -ball, as proposed for instance in [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF][START_REF] Tewari | Optimistic linear programming gives logarithmic regret for irreducible MDPs[END_REF], in the computation of the optimistic policy. This discussion aims at explaining and interpreting the difference of performance that can be observed in simulations. In KL-UCRL, optimism reduces to maximizing the linear function V ′ q over a KL-ball (see (3)), whereas the other algorithms make use of an

max q∈S |X | V ′ q s.t. p -q 1 ≤ ǫ ′ . (9)

Continuity

Consider an estimated transition probability vector p, and denote by q KL (resp. q 1 ) the probability vector which maximizes Equation (3) (resp. Equation ( 9)).

It is easily seen that q KL and q 1 lie respectively on the border of the convex set {q ∈ S |X| : KL(p, q) ≤ ǫ} and at one of the vertices of the polytope {q ∈ S |X| : pq 1 ≤ ǫ ′ }. A first noteworthy difference between those neighborhoods is that, due to the smoothness of the KL-neighborhood, q KL is continuous with respect to the vector V , which is not the case for q 1 .

To illustrate this, Figure 6 displays L 1 -and KL-balls around 3-dimensional probability vectors. The set of 3-dimensional probability vectors is represented by a triangle whose vertices are the vectors (1, 0, 0) ′ , (0, 1, 0) ′ and (0, 0, 1) ′ , the probability vector p by a white star, and the vectors q KL and q 1 by a white point. The arrow represents the direction of V 's projection on the simplex and indicates the gradient of the linear function to maximize. The maximizer q 1 can vary significantly for small changes of the value function, while q KL varies continuously.

Unlikely transitions

Denote i m = argmin j V j and i M = argmax j V j . As underlined by [START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF], q 1 im = max(p imǫ ′ /2, 0) and q 1 iM = min(p 1 iM + ǫ ′ /2, 1). This has two consequences:

1. if p is such that 0 < p im < ǫ ′ /2, then the vector q 1 im = 0; so the optimistic model may assign a probability equal to zero to a transition that has actually been observed, which makes it hardly compatible with the optimism principle. Indeed, an optimistic MDP should not forbid transitions that really exists, even if they lead to states with small values; 2. if p is such that p iM = 0, then q 1 iM never equals 0; therefore, an optimistic algorithm that uses L 1 -balls will always assign positive probability to transitions to i M even if this transition is impossible under the true MDP and if much evidence has been accumulated against the existence of such a transition. Thus, the exploration bonus of the optimistic procedure is wasted, whereas it could be used more efficiently to favor some other transitions.

This explains a large part of the experimental advantage of KL-UCRL observed in the simulations. Indeed, q KL always assigns strictly positive probability to observed transitions, and eventually renounces unobserved transitions even if the target states have a potentially large value. Algorithm 2 works as follows: for all i such that p i = 0, q i = 0; for all i such that p i = 0, q i = 0 except if p iM = 0 and if f (V iM ) < ǫ, in which case q iM = 1exp(f (V iM )ǫ). But this is no longer the case when ǫ becomes small enough, that is, when sufficiently many observations are available. We illustrate those two important differences Figure 6: The L 1 -neighborhood {q ∈ S 3 : pq 1 ≤ 0.2} (left) and KLneighborhood {q ∈ S 3 : KL(p, q) ≤ 0.02} (right) around the probability vector p = (0.15, 0.2, 0.65) ′ (white star). The white points are the maximizers of equations ( 3) and ( 9) with V = (0, 0.05, 1) ′ (up) and V = (0, -0.05, 1) ′ (down). in Figure 7, by representing the L 1 and KL neighborhoods together with the maximizers q KL and q 1 , first if p im is positive by very small, and second if p iM is equal to 0. Figure 8 also illustrates the latter case, by representing the evolution of the probability vector q that maximizes both ( 9) and (3) for an example with p = (0.3, 0.7, 0) ′ , V = (1, 2, 3) ′ and ǫ decreasing from 1/2 to 1/500.

A Linear optimization over a KL-ball

This section explains how to solve the optimization problem of Equation (3). In [START_REF] Nilim | Robust control of Markov decision processes with uncertain transition matrices[END_REF], a similar problem arises in a different context, and a somewhat different solution is proposed for the case when the p i are all positive. As a problem of maximizing a linear function under convex constraints, it is sufficient to consider the Lagrangian function q 2 q 3

L(q, λ, ν, µ 1 , . . . , µ N ) = N i=1 q i V i -λ N i=1 p i log p i q i -ǫ -ν N i=1 q i -1 + N i=1 µ i q i .
Figure 8: Evolution of the probability vector q that maximizes both (3) (top) and (9) (bottom) with p = (0.3, 0.7, 0) ′ , V = (1, 2, 3) ′ and ǫ decreasing from 1/2 to 1/500

If q is a maximizer, there exist λ ∈ R, ν, µ i ≥ 0 (i = 1 . . . N ) such that the following conditions are simultaneously satisfied:

                         V i + λ p i q i -ν + µ i = 0 (10) λ N i=1 p i log p i q i -ǫ = 0 (11) ν N i=1 q i -1 = 0 ( 12
)
µ i q i = 0 (13)
Let Z = {i, p i = 0}. Conditions [START_REF] Kearns | Near-optimal reinforcement learning in polynomial time[END_REF] to [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF] imply that λ = 0 and ν = 0. For i ∈ Z, Equation [START_REF] Kearns | Near-optimal reinforcement learning in polynomial time[END_REF] implies that q i = λ pi ν-µi-Vi . Since λ = 0, q i > 0 and then, according to [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF], µ i = 0. Therefore,

∀i ∈ Z , q i = λ p i ν -V i . (14)
Let r = i∈Z q i . Summing on i ∈ Z and using equations ( 14) and ( 12), we have

λ i∈ Z p i ν -V i = i∈ Z q i = 1 -r . (15)
Using ( 14) and ( 15), we can write i∈ Z p i log pi qi = f (ν)log(1r) where f is defined in (4). Then, q satisfies condition (11) if and only if f (ν) = ǫ+log(1-r) .

Consider now the case where i ∈ Z. Let I * = Z ∩ argmax i V i . Note that, for all i ∈ Z \ I * , q i = 0. Indeed, otherwise, µ i should be zero, and then ν = V i according to [START_REF] Kearns | Near-optimal reinforcement learning in polynomial time[END_REF], which involves a possible negative denominator in [START_REF] Strehl | An analysis of model-based interval estimation for Markov decision processes[END_REF]. According to [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF], for all i ∈ I * , either q i = 0 or µ i = 0. The second case implies that ν = V i and r > 0 which requires that f (ν) < ǫ so that (A) can be satisfied with r > 0. Therefore,

• if f (V i ) < ǫ for i ∈ I * , then ν = V i and the constant r can be computed solving equation f (ν) = ǫlog(1r); the values of q i for i ∈ I * may be chosen in any way such that i∈I * q i = r;

• if for all i ∈ I * f (V i ) ≥ ǫ, then r = 0, q i = 0 for all i ∈ Z and ν is the solution of the equation f (ν) = ǫ.

Once ν and r have been determined, the other components of q can be computed according to ( 14): we have that for i ∈ Z, q i = (1-r)qi i∈ Z qi where qi = pi ν-Vi .

B Properties of the f function

In this section, a few properties of function f defined in Equation ( 4) are stated, as this function plays a key role in the maximizing procedure of Section 3.2.

Proposition 2 f is a convex, decreasing mapping from ] max i∈ Z V i ; ∞[ onto ]0; ∞[. Proof 1 Using Jensen's inequality, it is easily shown that the f function decreases from +∞ to 0. The second derivative of f with respect to ν is equal to

- i p i (ν -V i ) 2 + 2 i pi (ν-Vi) 3 i pi ν-Vi - i pi (ν-Vi) 2 2 i pi ν-Vi 2 .
If Z denotes a positive random value such that P Z = 1 ν-Vi = p i , then

f ′′ (ν) = 2E(Z 3 )E(Z) -E(Z 2 )E(Z) 2 -E(Z 2 ) 2 E(Z) 2 .
Using Cauchy-Schwartz inequality, we have E(Z 2 ) 2 = E(Z 3/2 Z 1/2 ) 2 ≤ E(Z 3 )E(Z).

In addition E(Z 2 ) 2 ≥ E(Z 2 )E(Z) 2 . These two inequalities show that f ′′ (ν) ≥ 0.

As mentioned in Section 3.2, Newton's method can be applied to solve the equation f (ν) = ǫ for a fixed value of ǫ. When ǫ is close to 0, the solution of this equation is quite large and an appropriate initialization accelerates convergence. Using a second-order Taylor's-series approximation of the function f , it can be seen that, for ν near ∞, f (ν) = σp,V 2ν 2 +o( 1 ν 2 ), where σ p,V = i p i V 2 i -( i p i V i ) 2 . The Newton iterations can thus be initialized by taking ν 0 = σ p,V /(2ǫ).

Figure 1 :

 1 Figure 1: RiverSwim Transition Model: the continuous (resp. dotted) arrows represent the transitions if action 1 (resp. 2) has been chosen.

Figure 2 :

 2 Figure2: SixArms Transition Model 0) is the initial state. From the initial state, the agent may choose one among six actions: the action a ∈ {1, . . . , 6} leads to the state x = a with probability p a (see Figure2) and let the agent in the initial state with probability 1p a . From all the other states, some actions deterministically lead the agent to the initial state while the others leave it in the current state. Staying in a state x ∈ {1, . . . , 6}, the agent receives a reward equal to R x (see Figure2), otherwise, no reward is received.We compare the performance of the KL-UCRL algorithm to UCRL2 using 20 Monte-Carlo replications. For both algorithms, the constants C P and C R are settled to ensure that the upper bounds of the regret of Theorem 1 and Theorem 2 in[START_REF] Jaksch | Near-optimal Regret Bounds for Reinforcement Learning[END_REF] hold with probability 0.95. In the SixArms environment, the received rewards being deterministic, we slightly modify both algorithms so that the agent knows them beforehand. We observe in Figure3and 4 that the KL-UCRL algorithm accomplishes a smaller average regret than the UCRL2

Figure 3 :

 3 Figure 3: Comparison of the regret of the UCRL2 and KL-UCRL algorithms in the RiverSwim environment.

Figure 4 :

 4 Figure 4: Comparison of the regret of the UCRL2 and KL-UCRL algorithms in the SixArms environment.

Figure 5 :

 5 Figure 5: Comparison of the regret of the UCRL2 and KL-UCRL algorithms in randomly generated sparse environments.

Figure 7 :

 7 Figure 7: The L 1 (left) and KL-neighborhoods (right) around the probability vector p = (0, 0.4, 0.6) ′ (up) and p = (0.05, 0.35, 0.6) ′ (down). The white point is the maximizer of the equations (3) and (9) with V = (-1, -2, -5) ′ (up) and V = (-1, 0.05, 0) ′ (down). We took, ǫ = 0.05 (up), ǫ = 0.02(down) and ǫ ′ = √ 2ǫ.

* This work has been partially supported by Orange Labs under contract n o 289365.