Laura Silviana

Stéphane Diaz

Nicolas Doly

Sébasatien Narboux-Nême

Pierre Fernández

Sophie Marie Mazot

Katia Banas

Imane Boutourlinsky

Arnauld Moutkine

Anne Belmer

Roumier

Phane Doly

Nicolas Narboux-N

Pierre Mazot

Sophie Marie Banas

Katia Boutourlinsky

Imane Moutkine

Arnauld Belmer

Anne Roumier

Luc Maroteaux

5-HT(2B) receptors are

Keywords: 8-Hydroxy-2-(di-n-propylamino)tetralin, adverse effects, Analysis of Variance, Animals, Bromodeoxyuridine, metabolism, Cell Differentiation, drug effects, Chromatography, High Pressure Liquid, Exploratory Behavior, Fluoxetine, pharmacology, Gene Expression Regulation, Hippocampus, Hypothermia, chemically induced, Ki-67 Antigen, Male, Mice, Mice, Inbred C57BL, Mice, Knockout, Microdialysis, Neurogenesis, Reaction Time, Receptor, Serotonin, 5-HT2B, deficiency, physiology, Serotonin, Serotonin Plasma Membrane Transport Proteins, Serotonin Receptor Agonists, adverse effects, Serotonin Uptake Inhibitors, Time Factors, Transcription Factors serotonin levels, 5-HT receptors 2B, antidepressants, neurogenesis

teaching and research institutions in France or abroad, or from public or private research centers.

Introduction

The most widely used antidepressants, the selective serotonin reuptake inhibitors (SSRIs), induce an increase in extracellular serotonin (5-hydroxytryptamine, 5-HT) concentrations by acutely blocking the specific serotonin transporter (SERT) ( , ). The SERT regulates 1 2 the extracellular serotonin concentration by removing 5-HT from the synaptic cleft ( ), and diverse molecules modulate this activity, 3

including the A adenosine receptor ( ) and kinases such as PKG/p38 MAPK ( ). Previous studies have suggested a functional interaction 3 4 5 between SERT and the 5-HT receptor a receptor implicated in 5-HT-dependent phenotypes, including impulsivity, aggressivity and 2B suicidality ( ). studies have indicated that 5-HT receptors might participate in the control of SERT in raphe neurons ( ), while 6 Ex vivo 2B 7 studies further confirmed that 5-HT receptors contribute to the behavioral and physiological effects of the SERT-targeting 5-HT in vivo 2B releasers, MDMA (the club-drug ecstasy) and dexfenfluramine (

). Furthermore, we recently reported that the acute response to 8 -10 SSRIs in the forced swimming test (FST), a classical test for antidepressant activity, is absent in mice lacking 5-HT receptors ( ). 2B 11 Conversely, a 5-HT receptor agonist induced an antidepressant-like action in the FST, suggesting that this receptor is required for acute 2B SSRI effects and might modulate serotonergic tone.

Whereas blockage of SERT and increase of 5-HT levels are immediately attained after SSRI administration, therapeutic effects are only observed after weeks of treatment. The delay before the onset of clinical effects in depressive individuals appears to rely on the time required for stabilization of monoamine levels and other neuroadaptations, including neurogenesis ( ). As well, regulation of 5- HT 12 receptors appears to be required for either the acute or chronic effects of SSRIs, as it was largely demonstrated for 5-HT , 5-HT , 5-HT 1A 2A

, and 5-HT receptors ( ). Importantly, fewer than 50 of all patients with depression show remission with optimized available

The 5-HT mice used in all experiments are in a 129S2/SvPas (129S2) background ( ), while SERT and Pet-1 mice are on C57Bl/6NCrl background ( , ). Adult (7 9 week-old) male 5-HT , SERT and Pet-1 mice, and their respective wild type 20 21

- 2B / -- / -- / --
(WT) control mice (originally obtained from Charles River Laboratories, L Arbresle, France), were derived from heterozygous crosses and ' were bred at our animal facility. All mice were maintained according to the EC directive 86/609/CEE. Forced Swimming Test (FST) was conducted as already described ( ). The NSF test was basically conducted as already described ( ), with some modifications (See 22 17 ). Suppl. information

Chronic treatments, Proliferation and Survival assays and Tissue Preparation

For all the drugs employed in this study, the lowest acute effective dose was selected based on our previous data from WT mice ( ). . All these drugs were injected once daily, and at the end of the chronic treatments (24 h and 4 weeks after BrdU, respectively for proliferation and survival assays), mice were anesthetized and their brains were removed, fixed, and processed for immunohistochemistry. See supplemental information.

Immunohistochemistry & Immunofluorescence

Newborn cells were detected by classical peroxidase immunostaining of 5-bromo-2 deoxyuridine (BrdU), an exogenous marker of cell ′ division, or Ki67, an endogenous cell cycle marker. Identification of newly formed cells was performed by double-labeling cells with neuronal or astroglial markers and BrdU or Ki67, followed by confocal microscopy (see ). supplemental information

Microdialysis in freely moving mice

Microdialysis and high-pressure liquid chromatography (HPLC) were performed as already published by our laboratory (

). All 8 -10 measurements were obtained from freely moving mice following injections of RS127445 (0.5 mg/kg, i.p.) and/or paroxetine (2 mg/kg, i.p.) (see ). supplemental information

Single-cell RT-PCR

Pet-1-EGFP mice, generated from a cross of Pet-1-CRE ( ) with RCE:loxP ( ) mice, were used for single-cell RT-PCR 23 24 experiments. Mice were anesthetized and decapitated, and the brain was rapidly dissected out. Coronal slices (250-m thick) were μ prepared using a vibratome, in artificial cerebral spinal fluid aCSF supplemented with sucrose. The methodology involving harvesting of cytoplasmic content and subsequent single cell PCR amplification has been described elsewhere ( ). 25

5-HT receptor quantitative PCR

1A

Raphe nucleus and hippocampus were dissected and homogenized in 1 ml of RNAsolv. Genomic DNA was removed by digestion with Amplification Grade DNase I (Sigma-Aldrich). First-strand cDNA was synthesized by reverse transcription of 3 g of total RNA with μ Superscript-II reverse transcriptase (Invitrogen) according to standard protocols. Reverse transcriptase was omitted in some samples as negative control. Relative expression levels of 5-HT mRNA were determined by real time RT-PCR using Absolute SYBR Green Mix 1A (ABgene) and a set of primers specific for 5-HT receptor. 5-HT expression was normalized to mouse CyclophilinB mRNA expression.

1A 1A

Data were analyzed with the 2 DCt method, and values are expressed as the mean of four separate experiments performed in duplicate.

-

In Situ Hybridization

Coronal sections from the same experimental subjects used in immunohistochemistry studies were employed for in situ hybridization, which was performed as previously described ( ) on a pool of two coronal free-floating sections per mouse from each experimental 26 group (WT-Veh; WT-Flx; 5-HT -Veh and 5-HT -Flx). 5-HT (IMAGE:8861702) dig-UTP-labeled probe was used and

2B / -- 2B / -- 1A
detected using an antibody coupled to alkaline phosphatase (1/2000, Roche). AP activity was revealed using NBT/BCIP.

8-OH-DPAT-induced-hypothermia

Mol Psychiatry . Author manuscript Page / 3 10

Body temperature was measured intrarectally using a lubricated probe inserted ~2 cm and a digital thermometer (CEM advanced thermometer; DT-610B) ( ). Mice were moved to the behavioral room and two baseline temperature measurements were taken. After 10 27 min, animals received 8-OH-DPAT 0.5mg/kg, i.p., and body temperature was recorded every 15 min for a total of 45 min.

Statistical analysis

To determine differences between the experimental groups, responses were analyzed by either an unpaired Student s t test or a ' two-way analysis of variance (ANOVA) with genotypes and treatments as main factors, depending on the experimental design. Previously, normal distributions and homoscedasticity were verified by Shapiro-Wilk s test and Levene s test, respectively. Bonferroni s or Dunnett s ' ' ' '

test was used for comparisons. The 5-HT extracellular levels and hypothermic responses in both genotypes were compared by post hoc two-way repeated measures ANOVA. In all cases, p < 0.05 was considered statistically significant. A summary of tests employed and statistics are presented in . Table S1 Results

Statistical analyses of the data are summarized in . Table S1 5-HT receptors in the chronic behavioral effects of SSRIs

2B

Although the FST remains a key screen for acute antidepressant activity with good reliability and predictive validity, this test cannot reveal the neurobiological mechanisms underlying antidepressant therapeutic effects that are only observed after weeks of treatment. We thus studied the putative participation of 5-HT receptors in chronic behavioral effects triggered by SSRIs using the novelty-suppressed

feed (

). Then, we chronically treated WT and 5-HT mice with either Flx or Prx for 4 weeks and conducted an NSF test. Fig. 1a 2B / --Interestingly, even though Flx and Prx chronic treatments significantly reduced the latency to feed in WT mice, no differences were observed between control and SSRI-treated 5-HT mice ( ). These mutant mice also failed to respond in the NSF after 2B / --Fig. 1b injections of 1 or 20 mg/kg Flx ( ). To eliminate putative bias in the NSF test, food intake in the home cage was measured during Fig. S1a the 15 minutes immediately following the NSF test. No significant changes were registered according to either genotype or treatment (Fig.

). S1b

To verify that the results obtained in 5-HT mice were not due to developmental alterations, we also tested a group of WT mice 2B / -after simultaneous chronic treatment with RS127445 and Flx or vehicle. In the NSF test, chronic pharmacological blockade of 5-HT 2B receptors mimicked the effects of 5-HT ablation ( ), including the reduced basal latency and the lack of SSRI-antidepressant 2B Fig. 1c action.

Since acute injection of BW723C86 mimicked the effect of SSRIs in WT mice in the FST ( ), we tested the effect of chronic direct 11 activation of 5-HT receptors. A 4-week treatment with BW723C86 induced a significant decrease in the latency to feed in WT mice, 2B with no effect on 5-HT mice ( ). Thus, direct chronic activation of 5-HT receptors appears sufficient to induce chronic 2B / --Fig. 1d 2B SSRI-like effects in the NSF test.

Pro-neurogenic and plastic effects induced by SSRIs

The well-established neurogenic effect of chronic antidepressant treatment ( ) has been proposed to be necessary for behavioral 28 responses to SSRIs ( ). Therefore, cell proliferation and survival were studied in mice in which 5-HT receptors were genetically or 17 2B pharmacologically abolished. Chronic administration of either Flx or Prx failed to induce a significant increase in the incorporation of the exogenous DNA tracer, BrdU, in the hippocampal subgranular zone (SGZ) in 5-HT mice, whereas an SSRI neurogenic effect was 2B / -clearly observed in WT mice ( ). These results were confirmed by the endogenous cell cycle progression marker Ki-67 ( ). Fig. 2a Fig. S2a Also, no alterations in cell proliferation in either WT or mutant mice were observed in the subventricular zone, confirming that the neurogenic effect of SSRIs is specific to the SGZ of the dentate gyrus (

). Moreover, chronic Flx did not increase cell proliferation Fig. S2b in WT mice following chronic pharmacologic inactivation of 5-HT receptors (

). Altogether, these results confirm that a lack of 2B Fig. 2b functional 5-HT receptors is sufficient to abolish the chronic actions of SSRIs at the cellular level. increase in cell proliferation ( ), confirming that the observed Flx effects in WT mice depend on SERT. Fig. S2c Cell survival was also analyzed after administration of Flx for 8 weeks (BrdU was injected after the first 4 weeks), and a significant increase, dependent on the maintenance of Flx, was only observed in WT but not in 5-HT mice ( ). Regarding the fate of these 2B / --Fig. 2d newly generated cells, colabeling of BrdU with neuronal and astroglial markers was conducted in proliferation (24 h after BrdU) and survival (4 weeks after BrdU) conditions, and no difference in cell differentiation was found between genotypes ( ). Fig. S3 5-HT receptors are required for SSRI-induced extracellular 5-HT accumulation

2B

Although the neurobiological mechanisms mediating the behavioral and neurogenic effects of SSRIs are not completely understood, it is clear that responses to SSRIs do not occur after 5-HT depletion ( ). We therefore evaluated extracellular hippocampal 5-HT levels, by 30 means of microdialysis assays, after acute administration of SSRIs to freely moving mice. In WT mice, a single administration of Prx induced a maximal increase in 5-HT concentrations 60 min after injection, and levels remained elevated for at least 2 h ( ). In 5-HT Fig. 3a mice, as in RS127445-pretreated WT mice, the maximal increase in 5-HT concentrations following a single Prx injection was 2B / -significantly lower (around a quarter of the increase in WT mice). This particular response in mice with non-functional 5-HT receptors 2B could not be attributed to decreased basal levels of 5-HT, given that 5-HT concentrations in both genotypes are similar in the hippocampus, and even slightly higher in the raphe nuclei, of 5-HT mice ( ), consistent with the lack of major alterations in requires both SERT and serotonergic neurons results that are consistent with a presynaptic role for the 5-HT receptor.

-2B

5-HT receptors are not altered by the absence of 5-HT receptors 1A 2B

The 5-HT autoreceptor is involved in chronic SSRI effects ( , ). The proposed mechanism for SSRI action is a temporary 1A 17 32 increase in serotonergic tone that downregulates presynaptic 5-HT expression ( , ). We wondered if the absence of 5-HT ). In addition, the response Fig. 4d in the FST to 8-OH-DAT is mediated by post-synaptic receptors, and at higher doses of this 5-HT agonist, mice develop a serotonin 1A syndrome. Again, for this test, mice from both genotypes had similar responses ( ), indicating no differences in functionality for this Fig. 4e receptor. Therefore, the lack of extracellular 5-HT accumulation, behavioral or neurogenic responses after SSRI treatment in 5-HT 2B / -mice cannot be explained by alterations in 5-HT receptor expression or functionality.

1A

Discussion

Based on the results presented herein, the activation of 5-HT receptors is necessary for acute and chronic SSRI actions, and chronic 2B stimulation with a 5-HT receptor agonist is sufficient to mimic SSRI effects in WT mice. Furthermore, the moderate increase in 2B extracellular 5-HT levels induced by SSRIs in the absence of functional 5-HT receptors might be insufficient to trigger behavioral and 2B neurogenic actions. These data, together with the single-cell RT-PCR results showing expression of 5-HT receptors in raphe serotonergic 2B neurons, are consistent with a presynaptic action of 5-HT receptors mediating the acute and chronic SSRI effects.

2B

Although many data have been published on the topic, the exact role of hippocampal neurogenesis in the clinical response to antidepressants remains equivocal ( , ). Neuroplastic adaptations induced by chronic antidepressants appear to result from slow and 18 34 fine regulations allowing a re-establishment of monoamine levels ( ) and 5-HT receptor adaptation ( ). In this context, considering the 35 12

strong reduction in the SSRI-induced increase in extracellular 5-HT both in 5-HT mice and RS127445-treated WT mice, the lack of 2B / --SSRI-induced proliferative and survival effects in these mutant animals is not unexpected. The neurogenic actions of antidepressants have been proposed to be responsible for the behavioral effects induced by chronic treatment ( ). Accordingly, our results show that the 17 absence of neurogenic effects induced by chronic SSRIs in mice with pharmacologically or genetically abolished 5-HT receptors 2B correlates with a lack of a response in the NSF test. Conversely, chronic agonist stimulation of the 5-HT receptor is associated with 2B neurogenic effects and a positive response in the NSF test, supporting a link between neurogenesis and behavior.

Previous studies from our laboratory demonstrate that the absence of 5-HT receptors impairs the response to drugs targeting the 2B 5-HT system. In particular, 5-HT receptors are necessary for the 5-HT releasing effect of SERT-targeting drugs, like MDMA (the 2B club-drug ecstasy) and dexfenfluramine ( ). In line with these results, we shown here that acute SSRI administration to 5-HT 8 -10 A similar lack of acute and chronic effects were obtained in mice in which the 5-HT receptor was pharmacologically blocked or 2B genetically ablated, confirming that effects seen in mutant mice are not a consequence of developmental compensation. Notably, the baseline response in the NSF test is altered in both groups of mice and might result from specific settings established as a consequence of chronically inactive 5-HT receptors. The SSRI-reduced latency to feed in the NSF is associated with anxiolytic-like effects. The ability 2B of 5-HT mice to show anxiolysis clearly persists, as evidenced by their response to diazepam. The fact that both 5-HT receptor 2B / --

2B

agonist and antagonist induced a similar decrease in the latency to feed appears paradoxical. Nevertheless, all the results obtained herein for BW723C86 are consistent with antidepressant-like actions, whereas the response induced in the NSF test by RS127445 may be a consequence of the impulsive phenotype already described for these animals ( ) rather than an anxiolytic effect. Moreover, the fact that 6 basal neurogenesis is unaltered in either WT mice chronically treated with RS127445 or in 5-HT mice is more consistent with an 2B / -effect of impulsivity in the NSF in mice with non functional 5-HT receptors than an antidepressant-like action.

2B

The recent report that a single-nucleotide polymorphism (SNP) introducing a stop codon at the beginning of the human 5-HT 2B receptor (Q20 ) is associated with psychiatric diseases marked by high impulsivity and suicidal behavior ( ) supports a possible action of * 6 5-HT receptors on serotonergic tone. Herein, we show that the presence of a functional 5-HT receptor is required for behavioral and 2B 2B neurogenic SSRI actions. One of the unanswered questions about antidepressant effects remains the lack of explanation for the limited percentage of patients responding to chronic treatment ( ). Interindividual variation in antidepressant efficacy is thought to be at least 1 partly genetic. A number of studies have tested the association between allelic variations in candidate genes of the serotonergic pathway involved in the synthesis, transport, recognition or degradation of 5-HT and the response to various SSRIs. However, even in positive studies (i.e., long/short alleles in the SLC6A4/SERT promoter) ( ), these polymorphisms do not explain all of the variance seen in 38 clinical responses, suggesting the involvement of variants within other genes. To date, none of these studies investigated the potential role of the 5-HT receptor. We believe that our data pinpoint the need to investigate the association between treatment response to SSRIs and 2B SNPs in the 5-HT receptor gene. Interestingly, 70 of the Q 20 male cases displayed impulsive suicidal behavior, and 66 had at least 2B % * % one life-threatening suicide attempt by age 33.5 ( ). Thus, it would be interesting to collect and analyze clinical data related to SSRI 6 responses from these cases. Moreover, 5-HT receptor mutant mice appear to be a useful animal model for studying the clinical resistance 2B to SSRI antidepressant effects. The contribution of 5-HT receptors to the effects of SSRI antidepressants is undoubtedly demonstrated by the results presented 2B above, and agonist stimulation of this receptor induces SSRI-like effects. However, it is clear that 5-HT receptor agonists cannot be used 2B as antidepressant molecules, given established harmful cardiopulmonary peripheral effects of such compounds ( , ). Despite notable 39 40 advances, many patients with anxiety/depressive disorders fail to adequately respond to existing pharmacologic treatments. Augmenting antidepressant efficacy with other drugs has generated considerable clinical interest. Our study demonstrating that behavioral and neurogenic SSRI effects are abolished after pharmacologic inactivation of 5-HT receptors should be taken into account when designing 2B polypharmacology treatment strategies, given that actions mediated by this receptor might hinder the desired effects. As novel antidepressants are being developed to both ameliorate efficacy of treatments and increase the number of responders, the positive contribution of the 5-HT receptor to serotonergic homeostasis should be considered.

2B

Acknowledgements:

We thank Drs. Patricia Gaspar and Fiona Francis for critical reading of the manuscript, Sarah Rogan for English editing of the manuscript, as well as C. G. Bonelli (Institute of Pharmacological Research-CONICET, Argentina) for her excellent assistance on HPLC techniques. 32 . Richardson-Jones JW , Craige CP , Guiard BP , Stephen A , Metzger KL , Kung HF . 5-HT1A autoreceptor levels determine vulnerability to stress and response to

Pet-1-CRE and

  2Bfeeding (NSF) test, one of the few behavioral tests sensitive to chronic antidepressant treatments. Consistent with the impulsive phenotype shown by 5-HT mice ( ), their basal latency to feed in the NSF test was decreased compared to WT mice ( )mice are able to respond in this test, we injected the anxiolytic drug diazepam and observed a further decrease in the latency to 2B / --

  2BAgain, as in behavioral tests, chronic treatment with the 5-HT receptor agonist BW723C86 induced an antidepressant-like action 2B only in WT mice, evidenced by a significant increase in BrdU incorporation (). We verified that the chronic cellular effects of Fig.2cSSRIs are mediated by SERT and not by direct action at 5-HT receptors, as suggested by a previous study ( ) in spite of the low 2B affinity of Flx for mouse 5-HT receptors (Ki 5.65 0.52) ( ). After a 4-week treatment with Flx, SERT mice did not shown any 2B

  Fig. S4a tissue 5-HT levels in 5-HT or chronically RS127445-treated WT mice after Flx treatment ( ). Moreover, SERT binding 2B /--

  Fig. 4a c -). Moreover, we evaluated the functionality of these receptors by two different tests. Acute administration of the 5-HT agonist 1A 8-OH-DPAT induces a decrease in body temperature which is mediated by presynaptic 5-HT receptors ( ). Mice from both genotypes 1A

  a reduced (1/4) increase in hippocampal 5-HT levels compared to WT mice. Furthermore, we demonstrate for the first time that 5-HT receptors are expressed by serotonergic neurons of the raphe nuclei, which is consistent with a positive regulatory role for 2B these receptors in synaptic 5-HT homeostasis. This inhibition of SSRI-induced extracellular 5-HT accumulation in mice with no functional 5-HT receptors likely reflects a lack of positive control exerted by 5-HT signaling in raphe neurons. Indeed, microdialysis2BIn vivo experiments performed with the agonist BW723C86 confirm that local 5-HT receptor activation in the raphe nuclei is sufficient to induce 2B extracellular 5-HT accumulation ( ). Moreover, stimulation of 5-HT receptors appears to be presynaptic since BW723C86-induced 9 2B acute behavioral effects are abolished in Pet-1 mice (a genetic model of 5-HT depletion) or in SERT mice. data identify, for the first time, 5-HT receptors as possible positive 5-HT autoreceptors, acting in an opposing 2B manner to the 5-HT and 5-HT receptors, which are well established as negative autoreceptors ( ). This concept is consistent with 1A 1B 36 previous electrophysiology studies showing reduced or abolished 5-HT-dependent depolarization of serotonergic neurons by 5-HT 2 receptor antagonists (ketanserin or mesulergine) in the presence of the 5-HT receptor antagonist (WAY100635) ( ), but extensive 1A 37 electrophysiological studies are required to fully test this hypothesis. Moreover, given that no specific anti-mouse 5-HT receptor 2B antibody is currently available (each giving different results as well as residual labeling in 5-HT mice), we are not currently able to 2B /-evaluate whether these autoreceptors are localized presynaptically or in the soma of the raphe serotonergic neurons. This last point could have important physiological consequences in the mode of action of the receptor.

Figure 1 bFigure 2

 12 Figure 1The 5-HT receptor is necessary for the chronic behavioral effects of SSRIs 2B Chronic behavioral responses to antidepressants were evaluated by the Novelty-Suppressed Feeding test (NSF). Given the decreased basal (Veh) latency shown by 5-HT mice in the NSF test ( , ), their ability to respond in this paradigm was evaluated by acute 2B / --

3

 3 Figure 3 SSRI-induced increase in extracellular 5-HT concentration is blunted when 5-HT receptors are absent 2B After 4 h of equilibration, the SSRI Prx (2 mg/kg, i.p.) was administered at time zero (0): hippocampal samples obtained by microdialysis were collected each 20 minutes for 3 h, and 5-HT concentrations were measured by HPLC ( ). The peak observed at one hour after Prx a injection was significantly higher in WT mice than in 5-HT mice or RS127445-pretreated WT mice (15 min before Prx injection). Inset 2B / -represents the area under the curve (AUC) for each experimental group. Basal extracellular serotonin levels were (mean SEM, n 3 4) 0.28 ± = -0.09 nM in WT mice, 0.23 0.09 nM in 5-HT mice, and 0.3 0.1 nM in RS127445-pretreated WT mice ( ). See also Figure S3. ( ) ± ±

Figure 4

 4 Figure 4 Expression, affinity, and functionality of 5-HT receptors are similar in WT and 5-HT mice 1A 2B / --( ) Expression of the 5-HT receptor was evaluated by qPCR in raphe nucleus and hippocampus and no significant difference was found a 1A between WT and 5-HT mice. ( ) As well, binding assay with the 5-HT agonist 8-OH-DPAT on membranes prepared from the raphe 2B / --

  Alzet model 2004) for 4 weeks in addition to the daily i.p. injections of Flx or Veh. The last day of this ® 4-week experimental protocol, mice received 2 injections of 150 mg/kg BrdU (2-h interval between injections). Then, 24 hours after BrdU administration, a pool of mice was perfused and brains were recovered for cell proliferation studies. Another pool of mice continued receiving the SSRI or vehicle for 4 more weeks after BrdU administration and was finally perfused and the brains recovered for cell survival studies. Similar experimental protocols were conducted for cell proliferation induced by paroxetine (2 mg/kg) or BW723C86[START_REF] Schafer | How do antidepressants work? Prospects for genetic analysis of drug mechanisms[END_REF]

	11
	Fluoxetine (3 mg/kg/day in 129S2 or 10 mg/kg/day i.p. in C57Bl6/6NCrl mice) or vehicle (0.9 NaCl) was injected once daily for 4 %
	weeks. WT mice chronically treated with RS127445 continuously received the 5-HT antagonist at 1 mg/kg/day or vehicle (50 DMSO)

2B

% via subcutaneous osmotic pumps (mg/kg/day in 129S2 or 10 mg/kg/day i.p. in C57Bl6/6NCrl mice)

  Pet-1 mice were kindly provided by Dr. Evan Deneris (Case Western Reserve University, Cleveland, USA), whereas RCE:loxP mice were kindly obtained from Dr. Fishell (Smilow Research Center, New York University, USA). This work has been supported by funds from the Centre National de la Recherche Scientifique, the Institut National de la Sant et de la Recherche M dicale, the Universit Pierre et Marie Curie, and by grants from the Fondation de France, the Fondation pour la Recherche M dicale, the French ministry of research é (Agence Nationale pour la Recherche), and the European Commission (FP7-health-2007-A-201714-DEVANX). S. Diaz is supported by fellowships from IBRO and from Region Ile de France DIM STEM and S. Doly by a LeFoulon-Lalande fellowship. Casta ñé A , Semakova J , Santana N , Alvarado G , Cort s é R . Selective siRNA-mediated suppression of 5-HT(1A) autoreceptors evokes strong anti-depressant-like effects . Mol Psychiatry . 2011 ; Aug 2 In press 28 . Malberg JE , Eisch AJ , Nestler EJ , Duman RS . Chronic antidepressant treatment increases neurogenesis in adult rat hippocampus . J Neurosci . 2000 ; Dec 15 20 : (24 ) OF , Bechtholt AJ , Crowley JJ , Hill TE , Page ME , Lucki I . Depletion of serotonin and catecholamines block the acute behavioral response to different classes of antidepressant drugs in the mouse tail suspension test . Psychopharmacology (Berl) . 2007 ; Jun 192 : (3 ) 357 -371 31 . Kim DK , Tolliver TJ , Huang SJ , Martin BJ , Andrews AM , Wichems C . Altered serotonin synthesis, turnover and dynamic regulation in multiple brain regions of mice lacking the serotonin transporter . Neuropharmacology . 2005 ; Nov 49 : (6 ) 798 -810

	27 . Bortolozzi A , 9104 -9110
	29 . Li B , Zhang S , Zhang H , Nu W , Cai L , Hertz L . Fluoxetine-mediated 5-HT2B receptor stimulation in astrocytes causes EGF receptor transactivation and ERK
	phosphorylation . Psychopharmacology (Berl) . 2008 ; Dec 201 : (3 ) 443 -458
	30 . O Leary
	é	é	é
	Footnotes:
		Page / 7 10
	Mol Psychiatry . Author manuscript

/

--'

[START_REF] Banas | Deconstructing antiobesity compound action: requirement of serotonin 5-HT2B receptors for dexfenfluramine anorectic effects[END_REF]

[START_REF] Banas | Deconstructing antiobesity compound action: requirement of serotonin 5-HT2B receptors for dexfenfluramine anorectic effects[END_REF]

[START_REF] Banas | Deconstructing antiobesity compound action: requirement of serotonin 5-HT2B receptors for dexfenfluramine anorectic effects[END_REF]

[START_REF] Banas | Deconstructing antiobesity compound action: requirement of serotonin 5-HT2B receptors for dexfenfluramine anorectic effects[END_REF]

[START_REF] Banas | Deconstructing antiobesity compound action: requirement of serotonin 5-HT2B receptors for dexfenfluramine anorectic effects[END_REF]

Conflict of interest

The authors declare no conflict of interest.


M Chalony

A Kastberg

B Klappauf

D Wilkowski

Doppler cooling to the Quantum limit

Keywords: numbers: 37, 10, De, 37, 10, Gh

Doppler cooling on a narrow transition is limited by the noise of single scattering events. It shows novel features, which are in sharp contrast with cooling on a broad transition, such as a non-Gaussian momentum distribution, and divergence of its mean square value close to the resonance. We have observed those features using 1D cooling on an intercombination transition in strontium, and compared the measurements with theoretical predictions and Monte Carlo simulations. We also find that for a very narrow transition, cooling can be improved using a dipole trap, where the clock shift is canceled.

Laser cooling of atoms is a technique widely used, often as a first cooling stage on the road to quantum degeneracy. In the framework of Doppler cooling, a moving atom is cooled because of the difference in absorption probabilities, induced by the Doppler effect, between quasi-resonant red-detuned counterpropagating laser beams [1,2]. At each scattering event the momentum is, in average, modified by a recoil unit k, with k the wave number. If the steady state root mean square (rms) of the momentum distribution,

σ p ≫ k , (1)
is much larger than the recoil momentum. Doppler cooling can be expanded into a semi-classical theory, where the momentum evolution follows a damped Brownian motion with a pure friction force. In the low intensity regime the distribution is thermal-like with,

σ 2 p = mk b T = 7m 80|δ| (4δ 2 + Γ 2 ) , (2)
for a standing wave in 1D. δ is the laser detuning, Γ the linewidth of the transition and m the atomic mass. From the relation (2), the inequality (1) can be reformulated as

ω r = k 2 2m ≪ Γ , (3)
where ω r is the recoil frequency. Thus the semiclassical Doppler theory, discussed above, is valid only for atoms having a broad transition, i.e. fulfilling the inequality (3).

In the 1980s, with the progress of laser cooling and trapping techniques, in parallel with precise measurements of the momentum distribution, Doppler theory was found to be too crude to explain the steady state regime of atoms with complex internal structure. The ground state Zeeman manifold turned out to play a crucial role leading to "sub-Doppler cooling" [3][START_REF] Shevy | Proceedings of the Conference on Spin Polarized Quantum Systems[END_REF][START_REF] Dalibard | Proceedings of the 11th Conference on Atomic Physics[END_REF][START_REF] Dalibard | [END_REF][7]. However, Doppler theory remains valid for a two-level atom and for J = 0 → J = 1 systems. Several attempts were made to quantitatively test the theory of a broad transition, using spin polarized alkaline atoms [8], or more recently with spinless, ground state, bosonic two-electrons atoms [9][10][11][12]. However, most of those measurements show higher temperatures than predicted by the theory. Explanations suggested include spurious heating effects coming from spatial fluctuations of the laser intensity [10], and excited state coherences [13].

Two-electron atoms remain an ideal testing ground for Doppler theory. In addition to broad transitions, narrow intercombination transitions also exist, where the inequality (3) no longer holds. Those later cases lead to new features of the steady state regime, theoretically studied by Castin and co-authors using a full quantum approach of the Doppler theory [14]. In particular, close to the resonance, but still on the red side of the transition, the momentum distribution has a non-Gaussian shape characterized by long tails, leading to a divergence of the rms momentum. Hence the minimum rms momentum -lower bounded by the recoil momentum-is red shifted in frequency, with respect to the δ = -Γ/2 value for broad transitions. The use of two-electron atoms in laser cooling with narrow intercombination lines is growing, and several groups already reported laser cooling on narrow transitions with temperature at or close to the recoil limit [15][16][17][18][19][20][21][22]. Most of those works were carried on in a magneto-optical trap (MOT) where the cooling dynamic and steady state regime are modified by the presence of the magnetic field gradient [17,19]. Moreover, the gravity plays an important role displacing the position of the cloud and thus changing its magnetic environment.

In this letter, we present some experiments done using a dipole trap where the clock shift is canceled and we discuss the first quantitative test of this important theory of quantum Doppler cooling.

As described in [19], our 88 Sr cold atomic sample is produced as follow; after a loading and a precooling stage on the broad 1 S 0 → 1 P 1 dipole-allowed transition (Γ/2π = 32 MHz, ω r /2π = 10.6 kHz), the atoms are transferred to a MOT operating on the narrow 1 S 0 → 3 P 1 intercombination transition at 689 nm (Γ/2π = 7.5 kHz, ω r /2π = 4.7 kHz). This final cooling stage lasts for 130 ms and leads to a cold gas containing about 2 × 10 7 atoms at a temperature of T = 2 µK. A few tens of milliseconds before switching off the MOT, a far-off resonant dipole trap, centered on the atomic gas, is turned on. This dipole trap consists of a single focused laser beam at 780 nm. The laser power is 120 mW for a beam waist of 17 µm, corresponding to a potential depth of T 0 ≃ 20 µK. The radial and longitudinal trap frequencies are respectively ω ⊥ = 670 Hz and ω = 8 Hz. Because of the weak overlap between the dipole trap and the initial cold cloud, at best 1% of the atoms are transferred into the dipole trap. 50 ms after the MOT stage, a counterpropagating pair of beams, aligned with the long axis of the dipole trap and red-detuned with respect to the 1 S 0 → 3 P 1 transition is turned on for 450 ms (Fig. 1). We carefully balance the intensity of the cooling beam minimizing the displacement of the gas center of mass in the dipole trap. The cloud's spatial distribution is recorded by absorption imaging on the broad line at 461 nm.

A 0.3 G magnetic bias field (B) is applied during the 1D cooling experiment, for two important reasons. First, the Zeeman degeneracy of the excited state is lifted so that the lasers interact only with a two-level system made out of the m = 0 → m = 0 transition, which is unsensitive to residual magnetic field fluctuations. Second, the orientation of B, with respect to P 780 , the linear polarization of the dipole trap beam, is tuned to cancel the clock (or transition) shift induced by the dipole trap on the transition of interest [23]. The variation of the clock shift with respect to the angle θ between P 780 and B is given in Fig. 1b, and is in good agreement with the predicted value. The relative accuracy of the clock shift cancellation, for the whole trapping potential, is about 4 kHz, i.e., below the bare transition linewidth Γ. At this precision, one can ignore the small spatial dependency of the laser detuning induced by the dipole trap over the confined atomic gas. The position of the resonance has been measured with a precision of ±Γ/2 using absorption spectroscopy on the cold cloud. For technical reasons, the linear polarization P 689 of the cooling lasers is not aligned with the bias B-field. Thus the effective coupling intensity is reduced by a factor cos 2 α ≃ 1/15, where α is the angle between P 689 and B. Taking into account this reduction factor, the effective intensity I f is in the range 0.03 -0.1I s , where I s = 3 µW/cm 2 is satu- The dashed line corresponds to the red curve shifted in θ to show that the small disagreement between the experimental point and the predicted value is most likely due to a systematic error in the angle calibration.

ration intensity. At those intensities, the damping times of the momentum are long, in the range 0.02 -0.2 s. As a consequence, the steady state regime is usually not reached in a free falling experiment, which validates the use of the trapping potential.

In Fig. 2 we show the experimental mean square momentum (black full circles) in temperature and recoil units. The comparison with the analytical prediction of the full quantum approach [14] (green curve) shows an excellent agreement. Signatures of a quantum nature of Doppler cooling are found, e.g., the mean square momentum has a minimum below the recoil, and it shows a divergence close to the resonance, but at a detuning clearly larger negative than δ = -Γ/2 predicted by the broad transition Doppler theory (see [Eq. ( 2)] and the corresponding plot: blue, dashed curve in Fig. 2). Spurious heating effects reported for broad transitions [10,13] are not observed, most likely because we are using a twolevel system with a long damping time.

Several steps are necessary in order to validate the experiment and theory comparison of the preceding paragraph. First, the value of the mean square momentum can not be deduced from the spatial expansion of the cloud in the trap at the steady state regime, because of the presence of light induced collective effects. For a FIG. 2: Mean square momentum in temperature and recoil units as function of cooling laser detuning. The experimental data (black, full circles), extrapolated for N → 0 (noninteracting gas), are compared to the full quantum approach developed by Castin et al. [14] (green, solid line). The blue, dashed curve shows the prediction for broad transition Doppler theory. The red points (green stars) correspond to the MC simulation performed in (without) the dipole trap. Inset: Measured mean square momentum as function of the number of atoms in the trap with δ = -20 kHz and I f = 0.06Is (red circles) with linear fit (dashed line).

standard 3D magneto-optical trap, the cloud would inflate due to the repulsive multiple scattering force [24]. In contrast, we observe a compression of the cloud because of the dominance of the attractive shadow force in the 1D configuration [25,26]. The mean square momentum is then deduced by measuring the evolution of the cloud over half of the trap period, after switching off the 1D cooling laser. For that purpose, we use the relationship σ 2 z (t) = σ 2 z (0) cos 2 (ωt) + (σ p (0)/mω) 2 sin 2 (ωt) , linking the rms value of the cloud size in the 1D harmonic trap, σ z (t), to the initial values (at the switching off of the lasers) in real space and momentum space.

Moreover, the mean square momentum depend on the number of atoms, showing that collective effects also induce extra heating. Regardless of the exact origin of this extra heating, one example of this dependency is shown in the inset of Fig. 2. We extrapolate the mean square momentum value to the noninteracting limit (vanishing number of atoms) using a linear fit. The data points for the mean square momentum in Fig. 2 are deduced in this way.

Finally, even if the trap is loose along the cooling axis (ω ≪ ω r ), it is not clear that it does not affect the cooling process. Later on we will show that the trapping indeed has a major impact when the transition is very narrow, ω r ≫ Γ. To explore the influence of the trap, we use a Monte Carlo (MC) simulation comparing cooling with and without the trap. The MC simulation is based on a rate equation describing the scattering events FIG. 3: Upper panels: Raw data, false color image of the atomic cloud after 1D cooling in free space, and 50 ms time of flight. The cooling laser are along the horizontal axis. From left to right the laser detunings are respectively δ = -33 kHz, δ = -21 kHz and δ = -10 kHz, and the laser intensity is around 0.5Is. The small horizontal asymmetry is likely due to imperfect balance of the cooling beams intensity, which were not precisely adjusted like for the cloud in the Dipole trap case. Middle panels: Normalized spatial distribution along the cooling axis extracted from the upper images. Lower panels: Normalized momentum distribution extracted from the MC simulation. The laser detuning is the same as in the experiment, but the simulation is performed at the low intensity limit. These plots show a qualitative agreement with the experiment at higher intensity without the added trap (green open circles), as well as in the trap (red dots). The resonance lines correspond to the vertical, dashed lines.

where the external degrees of freedom are treated classically. This approach, neglecting the quantum nature of the external degrees of freedom, is known to be consistent with the full quantum approach in free space [14]. This point is also confirmed here, where the results of the MC simulation in free space are plotted in Fig. 2 (green stars). In the trap, the dynamic is more subtle because of the presence of the trapping force. It turns out, however, that for the strontium intercombination transition with ω r ≃ 0.6Γ the trap does not significantly modify the mean square momentum in the steady state (red dots in Fig. 2). We will see later that the condition ω r ≫ Γ leads to different conclusions.

Other signatures of the quantum nature of Doppler cooling can be found in the shape of the momentum distribution. In the broad transition semiclassical picture, the momentum distribution is essentially Gaussian since it remains very well confined far from the two ±δm/k resonance lines. With a narrow transition, a single scattering event might be enough to bring the atom out of resonance. As a consequence, the momentum distribu- tion is not Gaussian anymore and shows out-of-resonance long tails and dips at the resonance lines [14,27]. A precise measurement of the momentum distribution has been done for the case of free space 1D cooling on clouds with large number of atoms, to improve the signal to noise ratio. The laser intensity was increased to 0.5I s in order to reach the steady state regime during the laser interaction time. This is a likely explanation to why there is only a qualitative agreement found between the experiment and the MC simulations done for the low intensity limit (see Fig. 3). The experimental momentum distributions (Fig. 3, middle panels) might be decomposed into two domains; between and beyond the two ±δm/k resonance lines (red, dashed, vertical lines). The distribution on the inside is in very good agreement with the MC simulation since it depends only on the laser detuning. For the contribution outside the resonance lines, the dependence more pronounced for the experimental cases, that extra heating is at play.

So far, we have discussed Doppler Cooling on the narrow strontium intercombination transition with Γ ≃ ω r ≃ kσ p /m. We have shown that the trap has no major impact on the steady state regime. We will now consider the case of a very narrow transition, i.e. Γ ≪ ω r ≃ kσ p /m, where the laser excitation is well localized in the momentum space. In the trap, the laser excitation frequency is broaden by the oscillation of the atom and occurs above an energy threshold corresponding to |δ|. This configuration has strong similarity with the broadband cooling proposal discussed in Ref. [28]. It is expected to be more efficient than single frequency cooling, since an atom outside the resonance lines can still be cooled and brought back to the central region. As a consequence the long tails are reduced in the trap with respect to the free space case. This effect is in fact evident also for the strontium transition (see lower-right panel on Fig. 3). However, it will be more pronounced with a very narrow transition. As an example, MC simulations were performed for the calcium intercombination transition (λ = 657 nm, Γ/2π = 400 Hz and ω r /2π = 11 kHz), using the same dipole trap parameters as previously described (see Fig. 4). We observed that the momentum distribution is more confined in the central region, leading to a reduction of its mean square value by more than a factor of 3, with a minimum closer to the resonance line. More systematic studies are left for future investigation.

To conclude, we have found an excellent agreement between cooling on the narrow intercombination transition in strontium, and the quantum theory of Doppler cooling developed in [14]. As a major feature of cooling on narrow transitions, the momentum distribution can be decomposed into a cold part at lower momenta than one corresponding to the laser resonance line, and a hotter part at higher momenta. This latter component can be strongly reduced by using a very narrow transition in a dipole trap, where the clock shift is canceled.

FIG. 1 :

 1 FIG.1:(a): Outline of the 1D experimental set-up. The cold atomic gas, at the center of the picture, is held by the focused far-off resonance laser beam at 780 nm. The 1D cooling lasers at 689 nm are superimposed on the 780 nm beam. Along a perpendicular axis, an imaging system on the 461nm broad transition records the spatial distribution of the cloud. A 0.3 G magnetic bias field is adjusted in angle θ with respect to the polarization of the 780 nm beam in order to cancel the clock shift on the 1D laser system. (b): Variation of the clock shift as a function of θ. The circles are experimental data points, whereas the full red curve is the predicted behavior. The dashed line corresponds to the red curve shifted in θ to show that the small disagreement between the experimental point and the predicted value is most likely due to a systematic error in the angle calibration.

FIG. 4 :

 4 FIG.4: MC simulation on the calcium intercombination transition. (a): Mean square momentum in recoil units, as function of the cooling laser detuning. The red points (green crosses) correspond to the MC simulation performed in (without) the dipole trap. The green, full line and the blue, dashed line correspond, respectively, to the analytical full quantum expression[14], and broad transition limit ([Eq. (2)]). (b): Normalized momentum distribution without trap (green open circles), and in the trap (red dots). From left to right the laser detunings are respectively δ = -3.5ωr, δ = -2.6ωr and δ = -1.6ωr. Black arrows indicate those points in (a). The resonance conditions correspond to the dashed vertical lines.


est destinée au dépôt

  Influence of patients' emotional state on the recovery processes after a transient global amnesia. Audrey Noël, Peggy Quinette, Jacques Dayan, Bérengère Guillery-Girard, Pascale Piolino, Alice Pélerin, Vincent de La Sayette, Fausto Viader, Béatrice Desgranges, Francis Eustache To cite this version: Audrey Noël, Peggy Quinette, Jacques Dayan, Bérengère Guillery-Girard, Pascale Piolino, et al.. Influence of patients' emotional state on the recovery processes after a transient global amnesia.: Emotional and memory disorders after TGA. Cortex, 2011, 47 (8), pp.981-91. 10.1016/j.cortex.2010.10.003. inserm-00652848 1

Table 3 :

 3 Scores of both patients and controls on the psychopathological scales in the two sessions (four months and one year).

	Patients	Controls

Session 1: four months; Session 2: one year after the episode STAI: State-Trait Anxiety Inventory; BDI: Beck Depression Inventory


Antoine Salomon
email: salomona@imagine.enpc.fr

Jean-Yves Audibert
email: audibert@imagine.enpc.fr

Issam El

Regret lower bounds and extended Upper Confidence Bounds policies in stochastic multi-armed bandit problem

Keywords: stochastic bandits, regret bounds, selectivity, ucb policies

This paper is devoted to regret lower bounds in the classical model of stochastic multiarmed bandit. A well-known result of Lai and Robbins, which has then been extended by Burnetas and Katehakis, has established the presence of a logarithmic bound for all consistent policies. We relax the notion of consistence, and exhibit a generalisation of the logarithmic bound. We also show the non existence of logarithmic bound in the general case of Hannan consistency. To get these results, we study variants of popular Upper Confidence Bounds (ucb) policies. As a by-product, we prove that it is impossible to design an adaptive policy that would select the best of two algorithms by taking advantage of the properties of the environment.

Introduction and notations

Multi-armed bandits are a classical way to illustrate the difficulty of decision making in the case of a dilemma between exploration and exploitation. The denomination of these models comes from an analogy with playing a slot machine with more than one arm. Each arm has a given (and unknown) reward distribution and, for a given number of rounds, the agent has to choose one of them. As the goal is to maximize the sum of rewards, each round decision consists in a trade-off between exploitation (i.e. playing the arm that has been the more lucrative so far) and exploration (i.e. testing an other arm, hoping to discover an alternative that beats the current best choice). One possible application is clinical trial, when one wants to heal as many patients as possible, when the latter arrive sequentially and when the effectiveness of each treatment is initially unknown [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF]. Bandit problems has initially been studied by [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF], and its interest has then been extended to many fields such as economics [START_REF] Lamberton | When can the two-armed bandit algorithm be trusted?[END_REF][START_REF] Bergemann | Bandit problems[END_REF], games [START_REF] Gelly | Exploration exploitation in go: UCT for Monte-Carlo go[END_REF], optimisation [START_REF] Kleinberg | Nearly tight bounds for the continuum-armed bandit problem[END_REF][START_REF] Coquelin | Bandit algorithms for tree search[END_REF][START_REF] Kleinberg | Multi-armed bandits in metric spaces[END_REF][START_REF] Bubeck | Online optimization in X-armed bandits[END_REF],... Let us detail our model. A stochastic multi-armed bandit problem is defined by:

• a number of rounds n,

• a number of arms K ≥ 2,

• an environment θ = (ν 1 , • • • , ν K ), where each ν k (k ∈ {1, • • • , K}) is a real-valued measure that represents the distribution reward of arm k.

We assume that rewards are bounded. Thus, for simplicity, each ν k is a probability on [0, 1]. Environment θ is initially unknown by the agent but lies in some known set Θ of the form Θ 1 × . . . × Θ K , meaning that Θ k is the set of possible reward distributions of arm k. For the problem to be interesting, the agent should not have great knowledges of its environment, so that Θ should not be too small and/or contain too trivial distributions such as Dirac measures. To make it simple, each Θ k is assumed to contain the distributions pδ a + (1 -p)δ b , where p, a, b ∈ [0, 1] and δ x denotes the Dirac measure centred on x. In particular, it contains Dirac and Bernoulli distributions. Note that the number of rounds n may or may not be known by the agent, but this will not affect the present study. Some aspects of this particular point can be found in [START_REF] Salomon | Deviations of stochastic bandit regret[END_REF].

The game is as follows. At each round (or time step) t = 1, • • • , n, the agent has to choose an arm I t in the set of arms {1, • • • , K}. This decision is based on past actions and observations, and the agent may also randomize his choice. Once the decision is made, the agent gets and observes a payoff that is drawn from ν It independently from the past. Thus we can describe a policy (or strategy) as a sequence (σ t ) t≥1 (or (σ t ) 1≤t≤n if the number of rounds n is known) such that each σ t is a mapping from the set {1, . . . , K} t-1 × [0, 1] t-1 of past decisions and outcomes into the set of arm {1, . . . , K} (or into the set of probabilities on {1, . . . , K}, in case the agent randomizes his choices).

For each arm k and all times t, let T k (t) = t s=1 ½ Is=k denote the number of times arm k was pulled from round 1 to round t, and X k,1 , X k,2 , . . . , X k,T k (t) the corresponding sequence of rewards. We denote by P θ the distribution on the probability space such that for any k ∈ {1, . . . , K}, the random variables X k,1 , X k,2 , . . . , X k,n are i.i.d. realizations of ν k , and such that these K sequences of random variables are independent. Let E θ denote the associated expectation. Let µ k = xdν k (x) be the mean reward of arm k. Introduce µ * = max k∈{1,...,K} µ k and fix an arm k * ∈ argmax k∈{1,...,K} µ k , that is k * has the best expected reward. The agent aims at minimizing its regret, defined as the difference between the cumulative reward he would have obtained by always drawing the best arm and the cumulative reward he actually received. Its regret is thus

R n = n t=1 X k * ,t - n t=1 X It,T I t (t) .
As most of the publications on this topic, we focus on expected regret, for which one can check that:

E θ R n = K k=1 ∆ k E θ [T k (n)], (1)
where ∆ k is the optimality gap of arm k, defined by ∆ k = µ * -µ k . We also define ∆ as the gap between the best arm and the second best arm, i.e. ∆ := min k =k * ∆ k .

Previous works have shown the existence of lower bounds on the performance of a large class of policies. In this way [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] proved a lower bound of the expected regret of order log n in a particular parametric framework, and they also exhibited optimal policies. This work has then been extended by [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]. Both papers deal with consistent policies, meaning that all the policies considered are such that:

∀a > 0, ∀θ ∈ Θ, E θ [R n ] = o(n a ).
(2)

The logarithmic bound of Burnetas and Katehakis is expressed as follows. For all environment θ = (ν 1 , • • • , ν K ) and all k ∈ {1, . . . , K}, let us set

D k (θ) := inf νk ∈Θ k :E[ν k ]>µ * KL(ν k , νk ),
where KL(ν, µ) denotes the Kullback-Leibler divergence of measures ν and µ. Now fix a consistent policy and an environment θ ∈ Θ. If k is a suboptimal arm (i.e.

µ k = µ * ) such that 0 < D k (θ) < +∞, then ∀ε > 0, lim n→+∞ P T k (n) ≥ (1 -ε) log n D k (θ) = 1.
This readily implies that:

lim inf n→+∞ E θ [T k (n)] log n ≥ 1 D k (θ)
.

Thanks to Equation (1), it is then easy to deduce a lower bound on the expected regret. One contribution of this paper is to extend this bound to a larger class of policies. We will define the notion of α-consistency (α ∈ [0, 1]) as a variant of Equation ( 2), where equality

E θ [R n ] = o(n a )
only holds for all a > α. We show that the logarithmic bound still holds, but coefficient θ) . We also prove that the dependence of this new bound in the term 1 -α is asymptotically optimal when n → +∞ (up to a constant). As any policy achieves at most an expected regret of order n (because the average cost of pulling an arm k is a constant ∆ k ), it is also natural to wonder what happens when expected regret is only required to be o(n). This notion is equivalent to Hannan consistency. In this case, we show that there is no logarithmic bound any more. Some of our results are obtained thanks to a study of particular Upper Confidence Bound algorithms. These policies were introduced by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]: it basically consists in computing an index at each round and for each arm, and then in selecting the arm with the greatest index. A simple and efficient way to design such policies is to choose indexes that are upper bounds of the mean reward of the considered arm that hold with high probability (or, say, with high confidence level). This idea can be traced back to [START_REF] El | Sample mean based index policies with o(log n) regret for the multi-armed bandit problem[END_REF], and has been popularized by [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF], who notably described a policy called ucb1. For this policy, each index is defined by an arm k, a time step t, and an integer s that indicates the number of times arm k has been pulled before stage t. It is denoted by B k,s,t and is given by:

1 D k (θ) is turned into 1-α D k (
B k,s,t = Xk,s + 2 log t s ,
where Xk,s is the empirical mean of arm k after s pulls, i.e. Xk,s = 1 s s u=1 X k,u . To summarize, ucb1 policy first pulls each arm once and then, at each round t > K, selects an arm k that maximizes B k,T k (t-1),t . Note that, by means of Hoeffding's inequality, the index B k,T k (t-1),t is indeed an upper bound of µ k with high probability (i.e. the probability is greater than 1 -1/t 4 ). Note also that a way to look at this index is to interpret the empiric mean Xk,T k (t-1) as an "exploitation" term, and the square root as an "exploration" term (as it gradually increases when arm k is not selected). The policy ucb1 achieves the logarithmic bound (up to a multiplicative constant), as it was shown that: [START_REF] Audibert | Exploration-exploitation tradeoff using variance estimates in multi-armed bandits[END_REF] studied some variants of ucb1 policy. Among them, one consists in changing the 2 log t in the exploration term into ρ log t, where ρ > 0. This can be interpreted as a way to tune exploration: the smaller ρ is, the better the policy will perform in simple environments where information is disclosed easily (for example when all reward distributions are Dirac measures). On the contrary, ρ has to be greater to face more challenging environments (typically when reward distributions are Bernoulli laws with close parameters). This policy, that we denote ucb(ρ), was proven by Audibert et al. to achieve the logarithmic bound when ρ > 1, and the optimality was also obtained when ρ > 1 2 for a variant of ucb(ρ). [START_REF] Bubeck | Bandits Games and Clustering Foundations[END_REF] showed in his PhD dissertation that their ideas actually enable to prove optimality of ucb(ρ) for ρ > 1 2 . Moreover, the case ρ = 1 2 corresponds to a confidence level of 1 t (in view of Hoeffding's inequality, as above), and several studies [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF][START_REF] El | Sample mean based index policies with o(log n) regret for the multi-armed bandit problem[END_REF][START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF][START_REF] Audibert | Exploration-exploitation tradeoff using variance estimates in multi-armed bandits[END_REF][START_REF] Honda | An asymptotically optimal bandit algorithm for bounded support models[END_REF] have shown that this level is critical. We complete these works by a precise study of ucb(ρ) when ρ ≤ 1 2 . We prove that ucb(ρ) is (1 -2ρ)-consistent and that it is not α-consistent for any α < 1 -2ρ (in view of the definition above, meaning that expected regret is roughly of order n 1-2ρ ). Not surprisingly, it performs well in simple settings, represented by an environment where all reward distributions are Dirac measures. A by-product of this study is that it is not possible to design an algorithm that would specifically adapt to some kinds of environments, i.e. that would for example be able to select a proper policy depending on the environment being simple or challenging. In particular, and contrary to the results obtained within the class of consistent policies, there is no optimal policy. This contribution is linked with selectivity in on-line learning problem with perfect information, commonly addressed by prediction with expert advice such as algorithms with exponentially weighted forecasters (see, e.g., [START_REF] Cesa-Bianchi | Prediction, learning, and games[END_REF]). In this spirit, a closely related problem to ours is the one of regret against the best strategy from a pool studied by [START_REF] Auer | The nonstochastic multiarmed bandit problem[END_REF]: the latter designed a policy in the context of adversarial/nonstochastic bandit whose decisions are based on a given number of recommendations (experts), which are themselves possibly the rewards received by a set of given algorithms. To a larger extent, model selection have been intensively studied in statistics, and is commonly solved by penalization methods [START_REF] Mallows | Some comments on cp[END_REF][START_REF] Akaike | Information theory and an extension of the maximum likelihood principle[END_REF][START_REF] Schwarz | Estimating the dimension of a model[END_REF]. Finally, we exhibit expected regret lower bounds of more general ucb policies, with the 2 log t in the exploration term of ucb1 replaced by an arbitrary function. We obtain Hannan consistent policies and, as mentioned before, lower bounds need not be logarithmic any more.

∀θ ∈ Θ, ∀n ≥ 3, E θ [T k (n)] ≤ 12 log n ∆ 2 k and E θ R n ≤ 12 K k=1 log n ∆ k ≤ 12 log n ∆ .
The paper is organized as follows: in Section 2 we give bounds on the expected regret of ucb(ρ) (ρ < 1 2 ). In Section 3 we study the problem of selectivity. Then we focus in Section 4 on α-consistent policies, and we conclude in Section 5 by results on Hannan consistency by means of extended ucb policies. Throughout the paper ⌈x⌉ denotes the smallest integer which greater than the real x, and Ber(p) denotes the Bernoulli law with parameter p.

Bounds on the expected regret of ucb(ρ), ρ < 1 2

In this section we study the performances of ucb(ρ) policy, with ρ lying in the interval (0, 1 2 ). We recall that ucb(ρ) is defined by:

• Draw each arm once,

• Then at each round t, draw an arm

I t ∈ argmax k∈{1,...,K} Xk,T k (t-1) + ρ log t T k (t -1)
.

Small values of ρ can be interpreted as a low level of experimentation in the balance between exploration and exploitation, and present literature has not provided precise regret bound orders of ucb(ρ) with ρ ∈ (0, 1 2 ) yet. We first study the policy in simple environments (i.e. all reward distributions are Dirac measures), where the policy is supposed to perform well. We show that its expected regret is of order ρ log n ∆ (Proposition 1 for the upper bound and Proposition 2 for the lower bound). These good performances are compensated by poor results in complexer environments, as we then prove that the overall expected regret lower bound is roughly of order n 1-2ρ (Theorem 3).

Proposition 1 Let 0 ≤ b < a ≤ 1 and n ≥ 1. For θ = (δ a , δ b ), the random variable T 2 (n) is uniformly upper bounded by ρ ∆ 2 log(n) + 1. Consequently, the expected regret of ucb(ρ) is upper bounded by ρ ∆ log(n) + 1.
Proof Let us prove the upper bound on the sampling time of the suboptimal arm by contradiction. The assertion is obviously true for n = 1 and n = 2. If the assertion is false, then there exists t ≥ 3 such that

T 2 (t) > ρ ∆ 2 log(t) + 1 and T 2 (t -1) ≤ ρ ∆ 2 log(t -1) + 1.
Since log(t) ≥ log(t -1), this leads to T 2 (t) > T 2 (t -1), meaning that arm 2 is drawn at time t. Therefore, we have a

+ ρ log(t) t-1-T 2 (t-1) ≤ b + ρ log(t) T 2 (t-1) , hence ∆ ≤ ρ log(t)
T 2 (t-1) , which implies T 2 (t -1) ≤ ρ log(t) ∆ 2 and thus T 2 (t) ≤ ρ log(t) ∆ 2 + 1. This contradicts the definition of t, which ends the proof of the first statement. The second statement is a direct consequence of Formula (1).

The following shows that Proposition 1 is tight and allows to conclude that the expected regret of ucb(ρ) is equivalent to ρ ∆ log(n) when n goes to infinity.

Proposition 2 Let 0 ≤ b < a ≤ 1, n ≥ 2 and h : t → ρ ∆ 2 log(t) 1 + 2ρ log(t) (t-1)∆ 2 -2
. For θ = (δ a , δ b ), the random variable T 2 (n) is uniformly lower bounded by

f (n) = n 2 min h ′ (s), 1 ds -h(2).
As a consequence, the expected regret of ucb(ρ) is lower bounded by ∆f (n).

Straightforward calculations shows that h ′ (s) ≤ 1 for s large enough, and this explains why our lower bound ∆f (n) is equivalent to ∆h(n) ∼ ρ ∆ log(n) as n goes to infinity.

Proof First, one can easily prove (for instance, by induction) that T 2 (t) ≤ t/2 for any t ≥ 2. Let us prove the lower bound on T 2 (n) by contradiction. The assertion is obviously true for n = 2. If the assertion is false for n ≥ 3, then there exists t ≥ 3 such that T 2 (t) < f (t) and T 2 (t -1) ≥ f (t -1). Since f ′ (s) ∈ [0, 1] for any s ≥ 2, we have f (t) ≤ f (t -1) + 1. These last three inequalities imply T 2 (t) < T 2 (t -1) + 1, which gives T 2 (t) = T 2 (t -1). This means that arm 1 is drawn at time t. We consequently have

a + ρ log(t) t -1 -T 2 (t -1) ≥ b + ρ log(t) T 2 (t -1) , hence ∆ ρ log(t) ≥ 1 T 2 (t -1) - 1 t -1 -T 2 (t -1) ≥ 1 T 2 (t -1) - √ 2 √ t -1 .
We then deduce that

T 2 (t) = T 2 (t -1) ≥ h(t) ≥ f (t).
This contradicts the definition of t, which ends the proof of the first statement. Again, the second statement results from Formula (1). Now we show that the order of the lower bound of the expected regret is n 1-2ρ . Thus, for ρ ∈ (0, 1 2 ), ucb(ρ) does not perform enough exploration to achieve the logarithmic bound, as opposed to ucb(ρ) with ρ ∈ ( 1 2 , +∞).

Theorem 3 For any ρ ∈ (0, 1 2 ), any θ ∈ Θ and any β ∈ (0, 1), one has

E θ [R n ] ≤ k:∆ k >0 4 log n ∆ k + 2∆ k log n log(1/β) + 1 n 1-2ρβ 1 -2ρβ .
Moreover, for any ε > 0, there exists θ ∈ Θ such that

lim n→+∞ E θ [R n ] n 1-2ρ-ε = +∞.
Proof Let us first show the upper bound. The core of the proof is a peeling argument and makes use of Hoeffding's maximal inequality. The idea is originally taken from [START_REF] Audibert | Exploration-exploitation tradeoff using variance estimates in multi-armed bandits[END_REF], and the following is an adaptation of the proof of an upper bound in the case ρ > 1 2 which can be found in S. Bubeck's PhD dissertation. First, let us notice that the policy selects arm k such that ∆ k > 0 at step t only if at least one of the three following equations holds:

B k * ,T k * (t-1),t ≤ µ * , (3)
Xk,t ≥ µ k + ρ log t T k (t -1) , (4)
T k (t -1) < 4ρ log n ∆ 2 k . (5)
Indeed, if none of the equations holds, then:

B k * ,T k * (t-1),t > µ * = µ k + ∆ k ≥ µ k + 2 ρ log n T k (t -1) > Xk,t + ρ log t T k (t -1) = B k,T k (t-1),t .
We denote respectively by ξ 1,t , ξ 2,t and ξ 3,t the events corresponding to Equations (3), ( 4) and ( 5).

We have:

E θ [T k (n)] = E n t=1 ½ It=k ≤ 4 log n ∆ 2 k + E   n t=⌈4 log n/∆ 2 k ⌉ ½ {It=k}\ξ 3,t   ≤ 4 log n ∆ 2 k + E   n t=⌈4 log n/∆ 2 k ⌉ ½ ξ 1,t ∪ξ 2,t   ≤ 4 log n ∆ 2 k + n t=⌈4 log n/∆ 2 k ⌉ P(ξ 1,t ) + P(ξ 2,t ).
We now have to find a proper upper bound for P(ξ 1,t ) and P(ξ 2,t ). To this aim, we apply the peeling argument with a geometric grid over the time interval [1, t]:

P(ξ 1,t ) ≤ P ∃s ∈ {1, • • • , t}, Xk * ,s + ρ log t s ≤ µ * ≤ log t log(1/β) j=0 P ∃s : {β j+1 t < s ≤ β j t}, Xk * ,s + ρ log t s ≤ µ * ≤ log t log(1/β) j=0 P ∃s : {β j+1 t < s ≤ β j t}, s l=1 X k * ,l -µ * ≤ -ρs log t ≤ log t log(1/β) j=0 P ∃s : {β j+1 t < s ≤ β j t}, s l=1 X k * ,l -µ * ≤ -ρβ j+1 t log t .
By means of Hoeffding-Azumas inequality for martingales, we then have:

P(ξ 1,t ) ≤ log t log(1/β) j=0 exp   - 2 β j+1 tρ log t 2 β j t    = log t log(1/β) + 1 1 t 2ρβ ,
and, for the same reasons, this bound also holds for P(ξ 2,t ). Combining the former inequalities, we get:

E θ [T k (n)] ≤ 4 log n ∆ 2 k + 2 n t=⌈4 log n/∆ 2 k ⌉ log t log(1/β) + 1 1 t 2ρβ (6) ≤ 4 log n ∆ 2 k + 2 log n log(1/β) + 1 n t=⌈4 log n/∆ 2 k ⌉ 1 t 2ρβ ≤ 4 log n ∆ 2 k + 2 log n log(1/β) + 1 n 1 1 t 2ρβ dt ≤ 4 log n ∆ 2 k + 2 log n log(1/β) + 1 n 1-2ρβ 1 -2ρβ .
As usual, the bound on the expected regret then comes formula (1). Now let us show the lower bound. The result is obtained by considering an environment θ of the form Ber( 12 ), δ 1 2 -∆ , where ∆ > 0 is such that 2ρ(1 + √ ∆) 2 < 2ρ + ε. We set

T n := ⌈ ρ log n ∆ ⌉, and define the event ξ n by:

ξ n = X1,Tn < 1 2 -(1 + 1 √ ∆ )∆ .
When event ξ n occurs, for any t ∈ {T n , . . . , n} one has

X1,Tn + ρ log t T n ≤ X1,Tn + ρ log n T n < 1 2 -(1 + 1 √ ∆ )∆ + √ ∆ ≤ 1 2 -∆,
so that arm 1 is chosen no more than T n times by ucb(ρ) policy. Thus:

E θ [T 2 (n)] ≥ P θ (ξ n )(n -T n ).
We shall now find a lower bound of the probability of ξ n thanks to Berry-Esseen inequality. We denote by C the corresponding constant, and by Φ the c.d.f. of the standard normal distribution. For convenience, we also define the following quantities:

σ := E X 1,1 - 1 2 2 = 1 2 , M 3 := E X 1,1 - 1 2 3 = 1 8 . Using the fact that Φ(-x) = e -x 2 2 √ 2πx β(x) with β(x) ----→
x→+∞ 1, we are then able to write:

P θ (ξ n ) = P θ X1,Tn -1 2 σ T n ≤ -2 1 + 1 √ ∆ ∆ T n ≥ Φ -2(∆ + √ ∆) T n - CM 3 σ 3 √ T n ≥ exp -2( ρ log n ∆ + 1)(∆ + √ ∆) 2 2 √ 2π(∆ + √ ∆) √ T n β 2(∆ + √ ∆) T n - CM 3 σ 3 √ T n ≥ n -2ρ(1+ √ ∆) 2 exp -2(∆ + √ ∆) 2 2 √ 2π(∆ + √ ∆) √ T n β 2(∆ + √ ∆) T n - CM 3 σ 3 √ T n .
Previous calculations and Formula (1) gives

E θ [R n ] = ∆E θ [T 2 (n)] ≥ ∆P θ (ξ n )(n -T n )
and the former inequality easily leads to the conclusion of the theorem.

Selectivity

In this section, we address the problem of selectivity in multi-armed stochastic bandit models. By selectivity, we mean the ability to adapt to the environment as and when rewards are observed. More precisely, it refers to the existence of a procedure that would perform at least as good as the policy that is best suited to the current environment θ among a given set of two (or more) policies. Two mains reasons motivates this study.

On the one hand this question was answered by Burnetas and Katehakis within the class of consistent policies. Let us recall the definition of consistent policies.

Definition 4 A policy is consistent if

∀a > 0, ∀θ ∈ Θ, E θ [R n ] = o(n a ).
Indeed they show the existence of lower bounds on the expected regret (see Section 3, Theorem 1 of [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]), which we also recall for the sake of completeness.

Theorem 5 Fix a consistent policy and θ ∈ Θ. If k is a suboptimal arm (i.e. µ k < µ * ) and if 0 < D k (θ) < +∞, then ∀ε > 0, lim

n→+∞ P θ T k (n) ≥ (1 -ε) log n D k (θ) = 1. Consequently lim inf n→+∞ E θ [T k (n)] log n ≥ 1 D k (θ)
.

Remind that the lower bound on the expected regret is then deduced from formula (1). Burnetas and Katehakis then exhibits an asymptotically optimal policy, i.e. which achieves the former lower bounds. The fact that a policy does as best as any other one obviously solves the problem of selectivity.

Nevertheless one can wonder what happens if we do not restrict our attention to consistent policies any more. Thus, one natural way to relax the notion of consistency is the following.

Definition 6 A policy is α-consistent if ∀a > α, ∀θ ∈ Θ, E θ [R n ] = o(n a ).
For example we showed in the former section that ucb(ρ) is (1 -2ρ)-consistent for any ρ ∈ (0, 1 2 ). The class of α-consistent policies will be studied in Section 4. Moreover, as the expected regret of any policy is at most of order n, it seems simpler and relevant to only require it to be o(n):

∀θ ∈ Θ, E θ [R n ] = o(n),
which corresponds to the definition of Hannan consistency. The class of Hannan consistent policies includes consistent policies and α-consistent policies for any α ∈ (0, 1). Some results on Hannan consistency will be provided in Section 5.

On the other hand, this problem has already been studied in the context of adversarial bandit by [START_REF] Auer | The nonstochastic multiarmed bandit problem[END_REF]. Their setting differs from our not only because their bandits are nonstochastic, but also because their adaptive procedure takes only into account a given number of recommendations, whereas in our setting the adaptation is supposed to come from observing rewards of the chosen arms (only one per time step). Nevertheless, there are no restrictions about consistency in the adversarial context and one can wonder if an "exponentially weighted forecasters" procedure like Exp4 could be transposed to our context. The answer is negative, as stated in the following theorem.

Theorem 7 Let à be a consistent policy and let ρ be a real in (0, 0.4). There are no policy which can both beat à and ucb(ρ), i.e.:

∀A, ∃θ ∈ Θ, lim sup n→+∞ E θ [R n (A)] min(E θ [R n ( Ã)], E θ [R n (ucb(ρ))]) > 1.
Thus there are no optimal policy if we extend the notion of consistency. Precisely, as ucb(ρ) is (1 -2ρ)-consistent, we have shown that there are no optimal policy within the class of α-consistent policies (which is included in the class of Hannan consistent policies), where α > 0.2. Moreover, ideas from selectivity in adversarial bandits can not work in the present context. As we said, this impossibility may also come from the fact that we can not observe at each step the decisions and rewards of more than one algorithm. Nevertheless, if we were able to observe a given set policies from step to step, then it would be easy to beat them all: it is then sufficient to aggregate all the observations and simply pull the arm with the greater empiric mean. The case where we only observe decisions (and not rewards) of a set of policies may be interesting, but is left outside of the scope of this paper.

Proof Assume by contradiction that

∃A, ∀θ ∈ Θ, lim sup n→+∞ u n,θ ≤ 1, where u n,θ = E θ [Rn(A)] min(E θ [Rn( Ã)],E θ [Rn(U CB(ρ))]) . One has E θ [R n (A)] ≤ u n,θ E θ [R n ( Ã)],
so that the fact that à is a consistent policy implies that A is also consistent. Consequently the lower bound of Burnetas and Katehakis has to hold. In particular, in environment θ = (δ 0 , δ ∆ ) one has for any ε > 0 and with positive probability (provided that n is large enough):

T 1 (n) ≥ (1 -ε) log n D k (θ) .
Now, note that there is simple upper bound of D k (θ):

D k (θ) ≤ inf p,a∈[0,1]:(1-p)a>∆ KL(δ 0 , pδ 0 + (1 -p)δ a ) = inf p,a∈[0,1]:(1-p)a>∆ log 1 p = log 1 1 -∆ .
And on the other hand, one has by means of Proposition 2:

T 1 (n) ≤ 1 + ρ log n ∆ 2 .
Thus we have that, for any ε > 0 and if n is large enough

1 + ρ log n ∆ 2 ≥ (1 -ε) log n log 1 1-∆
Letting ε go to zero and n to infinity, we get:

ρ ∆ 2 ≥ 1 log 1 1-∆ .
This means that ρ has to be lower bounded by

∆ 2 log( 1 1-∆ )
, but this is greater than 0.4 if ∆ = 0.75, hence the contradiction.

Note that the former proof give us a simple alternative to Theorem 3 to show that ucb(ρ) is not consistent if ρ ≤ 0.4. Indeed if it were consistent, then in environment θ = (δ 0 , δ ∆ ), T 1 (n) would also have to be greater than

(1-ε) log n D k (θ)
and lower than 1 + ρ log n ∆ 2 , and the same contradiction would hold.

Bounds on α-consistent policies

We now study α-consistent policies. We first show that the main result of Burnetas and Katehakis (Theorem 5) can be extended in the following way.

Theorem 8 Fix an α-consistent policy and θ

∈ Θ. If k is a suboptimal arm and if 0 < D k (θ) < +∞, then ∀ε > 0, lim n→+∞ P θ T k (n) ≥ (1 -ε) (1 -α) log n D k (θ) = 1. Consequently lim inf n→+∞ E θ [T k (n)] log n ≥ 1 -α D k (θ) .
Recall that, as opposed to [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], there are no optimal policy (i.e. a policy that would achieve the lower bounds in all environment θ), as proven in the former section.

Proof We adapt Proposition 1 in [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] and its proof, which one may have a look at for further details. We fix ε > 0, and we want to show that:

lim n→+∞ P θ T k (n) log n ≥ (1 -ε)(1 -α) D k (θ) = 0. Set δ > 0 and δ ′ > α such that 1-δ ′ 1+δ > (1 -ε)(1 -α). By definition of D k (θ), there exists θ such that E θ[X k,1 ] > µ * and D k (θ) < KL(ν k , νk ) < (1 + δ)D k (θ), 1
1. In [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], D k (θ) is denoted Ka(θ) and KL(ν k , νk ) is denoted I(θ a , θ ′ a ). The equivalence between other notations is straightforward.

where we denote θ = (ν 1 , . . . , ν K ) and θ = (ν 1 , . . . , νK ). Let us define I δ = KL(ν k , νk ) and the sets

A δ ′ n := T k (n) log n < 1 -δ ′ I δ , C δ ′′ n := log L T k (n) ≤ 1 -δ ′′ log n ,
where δ ′′ is such that α < δ ′′ < δ ′ and L j is defined by log

L j = j i=1 log dν k dν k (X k,i ) . We show that P θ (A δ ′ n ) = P θ (A δ ′ n ∩ C δ ′′ n ) + P θ (A δ ′ n \ C δ ′′ n ) -----→ n→+∞ 0.
On the one hand, one has:

P θ (A δ ′ n ∩ C δ ′′ n ) ≤ e (1-δ ′′ ) log n P θ(A δ ′ n ∩ C δ ′′ n ) (7) ≤ n 1-δ ′′ P θ(A δ ′ n ) = n 1-δ ′′ P θ n -T k (n) > n - 1 -δ ′ I δ log n ≤ n 1-δ ′′ E θ[n -T k (n)] n -1-δ ′ I δ log n (8) ≤ l =k n -δ ′′ E θ[T l (n)] 1 -1-δ ′ I δ log n n -----→ n→+∞ 0,
where ( 7) is consequence of the definition of C δ ′′ n , (8) comes from Markov's inequality, and where the final limit is a consequence of the α-consistence.

On the other hand we set b n := 1-δ ′ I δ log n, so that we have:

P θ (A δ ′ n \ C δ ′′ n ) ≤ P max j≤⌊bn⌋ log L j > (1 -δ ′′ ) log n ≤ P 1 b n max j≤⌊bn⌋ log L j > I δ 1 -δ ′′ 1 -δ ′ .
This term then tends to zero, as a consequence of the law of large numbers. Now that P θ (A δ ′ n ) tends to zero, the conclusion comes from the following inequality:

1 -δ ′ I δ > 1 -δ ′ (1 + δ)D k (θ) ≥ (1 -ε)(1 -α) D k (θ) .
The former lower bound is asymptotically optimal, as claimed in the following proposition.

Proposition 9 There exists θ ∈ Θ and a constant c > 0 such that, for any α ∈ [0, 1), there exists an α-consistent policy and k = k * such that:

lim inf n→+∞ E θ [T k (n)] (1 -α) log n ≤ c.
Proof By means of Proposition 1, the following holds for ucb(ρ) in any environment of the form θ = (δ a , δ b ) with a = b:

lim inf n→+∞ E θ T k (n) log n ≤ ρ ∆ 2 ,
where k = k * . As ucb(ρ) is (1 -2ρ)-consistent (Theorem 3), we can conclude by setting c = 1 2∆ 2 and by choosing the policy ucb( 1-α 2 ).

Hannan consistency and other exploration functions

We now study the class of Hannan consistent policies. We first show the necessity to have a logarithmic lower bound in some environments θ, and then a study of extended ucb policies will prove that there does not exist a logarithmic bound on the whole set Θ.

The necessity of a logarithmic regret in some environments

A simple idea enables to understand the necessity of a logarithmic regret in some environments. Assume that the agent knows the number of rounds n, and that he balances exploration and exploitation in the following way: he first pulls each arm s(n) times, and then selects the arm that has obtained the best empiric mean for the rest of the game. If we denote by p s(n) the probability that the best arm does not have the best empiric mean after the exploration phase (i.e. after the first Ks(n) rounds), then the expected regret is of the form

c 1 (1 -p s(n) )s(n) + c 2 p s(n) n. (9)
Indeed if the agent manages to match the best arm then he only suffers the pulls of suboptimal arms during the exploration phase, and that represents an expected regret of order s(n). If not, the number of pulls of suboptimal arms is of order n, and so is the expected regret. Now we can approximate p s(n) , because it has the same order as the probability that the best arm gets an empiric mean lower than the second best mean reward, and because

X k * ,s(n) -µ * σ s(n) (where σ is the variance of X k * ,1
) approximately has a standard normal distribution by the central limit theorem:

p s(n) ≈ P θ (X k * ,s(n) ≤ µ * -∆) = P θ X k * ,s(n) -µ * σ s(n) ≤ - ∆ s(n) σ ≈ 1 √ 2π σ ∆ s(n) exp   - 1 2 ∆ s(n) σ 2   ≈ 1 √ 2π σ ∆ s(n) exp - ∆ 2 s(n) 2σ 2 .
Then it is clear why the expected regret has to be logarithmic: s(n) has to be greater than log n if we want the second term p s(n) n of Equation ( 9) to be sub-logarithmic, but then first term (1 -p s(n) )s(n) is greater than log n.

This idea can be generalized, and this gives the following proposition.

Proposition 10 For any policy, there exists θ ∈ Θ and such that

lim sup n→+∞ E θ R n log n > 0.
This result can be seen as a consequence of the main result of Burnetas and Katehakis (Theorem 5): if we assume by contradiction that lim sup n→+∞ E θ Rn log n = 0 for all θ, the considered policy is therefore consistent, but then the logarithmic lower bounds have to hold. The reason why we wrote the proposition anyway is that our proof is based on the simple reasoning stated above and that it consequently holds beyond our model (see the following for details).

Proof The proposition results from the following property on Θ: there exists two environments θ = (ν 1 , . . . , ν K ) and θ = (ν 1 , . . . , νK ) and k ∈ {1, . . . , K} such that • k has the best mean reward in environment θ,

• k is not the winning arm in environment θ,

• ν k = νk and there exists η ∈ (0, 1) such that

ℓ =k dν ℓ dν ℓ (X ℓ,1 ) ≥ η P θ -a.s. ( 10
)
The idea is the following: in case ν k = νk is likely to be the reward distribution of arm k, then arm k has to be pulled often for the regret to be small if the environment is θ, but not so much, as one has to explore to know if the environment is actually θ (and the third condition ensures that the distinction can be tough to make). The lower bound on exploration is of order log n, as in the sketch in the beginning of the section.

The proof actually holds for any Θ that has the above-mentioned property (i.e. without the assumptions we made on Θ, i.e. being of the form Θ 1 × . . . × Θ K and/or containing distributions of the form pδ a + (1 -p)δ b ). In our setting, the property is easy to check. Indeed the three conditions hold for any k and any pair of environments θ = (ν 1 , . . . , ν K ), θ = (ν 1 , . . . , νK ) such that each ν ℓ (resp. νℓ ) is a Bernoulli law with parameter p ℓ (resp. pℓ ) and such that

• ∀ℓ = k, pk > pℓ , • ∃ℓ = k, p k < p ℓ ,
• pk = p k and p ℓ , pℓ ∈ (0, 1) for any ℓ = k.

It is then sufficient to

set η = min p 1 p1 , . . . , p k-1 pk-1 , p k+1 pk+1 , . . . , p K pK , 1 -p 1 1 -p1 , . . . , 1 -p k-1 1 -pk-1 , 1 -p k+1 1 -pk+1 , . . . , 1 -p K 1 -pK K-1
, as dν ℓ dν ℓ (X ℓ,1 ) equals p ℓ pℓ when X ℓ,1 = 1 and 1-p ℓ 1-p ℓ when X ℓ,1 = 0.

We will now compute a lower bound of the expected regret in environment θ. To this aim, we set

g(n) := 2E θ R n ∆ .
In the following, ∆k denotes the optimality gap of arm k in environment θ. Moreover the switch from θ to θ will result from Equality (10) and from the fact that event

ℓ =k T ℓ (n) ≤ g(n) is measurable with respect to X ℓ,1 , . . . , X ℓ,⌊g(n)⌋ (ℓ = k) and to X k,1 , . . . , X k,n . That enables us to introduce the function q such that ½ { ℓ =k T ℓ (n)≤g(n)} = q (X k,s ) s=1..n , (X ℓ,s ) ℓ =k, s=1..⌊g(n)⌋
and to write:

E θR n ≥ ∆k E θ[T k (n)] ≥ ∆k (n -g(n))P θ (T k (n) ≥ n -g(n)) = ∆k (n -g(n))P θ   ℓ =k T ℓ (n) ≤ g(n)   = ∆k (n -g(n)) q (x ℓ,s ) ℓ =k, s=1..t , (x k,s ) s=1..n ℓ = k s = 1..⌊g(n)⌋ dν ℓ (x ℓ,s ) s=1..n dν k (x k,s ) ≥ ∆k (n -g(n))η ⌊g(n)⌋ q (x ℓ,s ) ℓ =k, s=1..t , (x k,s ) s=1..n ℓ = k s = 1..⌊g(n)⌋ dν ℓ (x ℓ,s ) s=1..n dν k (x k,s ) ≥ ∆k (n -g(n))η g(n) P θ   ℓ =k T ℓ (n) ≤ g(n)   = ∆k (n -g(n))η g(n)   1 -P θ   ℓ =k T ℓ (n) > g(n)     ≥ ∆k (n -g(n))η g(n)   1 - E θ ℓ =k T ℓ (n) g(n)   ≥ ∆k (n -g(n))η g(n)   1 - E θ ℓ =k ∆ ℓ T ℓ (n) ∆g(n)   ≥ ∆k (n -g(n))η g(n) 1 - E θ R n ∆g(n) = ∆k n -g(n) 2 η g(n) ,
where the very first inequality is a consequence of Formula (1).

We are now able to conclude. Indeed, if we assume that E θ Rn log n -----→ n→+∞ 0, then one has g(n) ≤ min n 2 , -log n 2 log η for n large enough and:

E θR n ≥ ∆k n -g(n) 2 η g(n) ≥ ∆k n 4 η -log n 2 log η = ∆k √ n 4 .
In particular, we have To finish this section, note that a proof could have been written in the same way with a slightly different property on Θ: there exists two environments θ = (ν 1 , . . . , ν K ) and θ = (ν 1 , . . . , νK ) and k ∈ {1, . . . , K} such that

• k has the best mean reward in environment θ,

• k is not the winning arm in environment θ,

• ν ℓ = νℓ for all ℓ = k and there exists η ∈ (0, 1) such that dν k dν k (X k,1 ) ≥ η P θ -a.s.

The dilemma is then between exploring arm k or pulling the best arm of environment θ.

There are no logarithmic bound in general

We extend our study to more general ucb policies, and we will find that there does not exist logarithmic lower bounds of the expected regret in the case of Hannan consistency.

With "ucb", we now refer to an ucb policy with indexes of the form:

B k,s,t = Xk,s + f k (t) s
where functions f k (1 ≤ k ≤ K) are increasing.

To find conditions for Hannan consistency, let us first show the following upper bound.

Lemma 11 If arm k does not have the best mean reward, then for any β ∈ (0, 1) the following upper bound holds: t) ,

E θ [T k (n)] ≤ u + n t=u+1 1 + log t log( 1 β ) e -2βf k (t) + e -2βf k * (
where u = 4f k (n) ∆ k .
Proof We adapt the arguments leading to Equation (6) in the proof of Theorem 3. We begin by noticing that, if arm k is selected, then at least one of the three following equations holds:

B k * ,T k * (t-1),t ≤ µ * , Xk,t ≥ µ k + f k (t) T k (t -1) , T k (t -1) < 4f k (n) ∆ 2 k ,
and the rest follows straightforwardly.

We are now able to give sufficient conditions on the f k for ucb to be Hannan consistent.

Proposition 12 If f k (n) = o(n) for all k ∈ {1, . . . , K}, and if there exists γ > 1 2 and N ≥ 1 such that f k (n) ≥ γ log log n for all k ∈ {1, . . . , K} and for any n ≥ N , then ucb is Hannan consistent.

Proof Fix an index k of a suboptimal arm and choose β ∈ (0, 1) such that 2βγ > 1. By means of Lemma 11, one has for n large enough:

E θ [T k (n)] ≤ u + 2 n t=u+1 1 + log t log( 1 β )
e -2βγ log log t , where u = 4f k (n) ∆ k . Consequently, we have:

E θ [T k (n)] ≤ u + 2 n t=2 1 (log t) 2βγ + 1 log( 1 β ) 1 (log t) 2βγ-1 . ( 11
)
Sums of the form n t=2 1 (log t) c with c > 0 are equivalent to n (log n) c as n → +∞. Indeed, on the one hand we have

Now, by means of Equation ( 11), there exists C > 0 such that

and this proves Hannan consistency.

The fact that there is no logarithmic lower bound then comes from the following proposition (which is a straightforward adaptation of Propostion 1).

ucb is Hannan consistent and the expected regret is of order log log n in all environments of the form (δ a , δ b ). Hence the conclusion on the non-existence of logarithmic lower bounds.