Emmanuelle Courtiol

Edenia Menezes

Catia M Teixeira
email: catia.teixeira@nki.rfmh.org

Serotonergic regulation of the dopaminergic system: implications for reward-related functions

Keywords: Serotonin, dopamine, reward, ventral tegmental area, Raphe nuclei, development

Introduction

"We would like provisionally to name it serotonin, which indicates that its source is serum and its activity is one of causing constriction" [START_REF] Rapport | Crystalline Serotonin[END_REF]. Serotonin (5hydroxytryptamine,, first named after the study of [START_REF] Rapport | Crystalline Serotonin[END_REF], is now a classic and perhaps one of the most described neuromodulators. The functions of this crystalline substance go well beyond the vasoconstriction effect described in 1948.

Indeed, although secreted to a minor extent in the brain compared to the massive amount found in the gastrointestinal tract [START_REF] Bertrand | Serotonin release and uptake in the gastrointestinal tract[END_REF], 5-HT has a major role in neuronal function.

Primarily found in the raphe nuclei (RN), subdivided into the Dorsal Raphe Nucleus (DRN) and Median Raphe Nucleus (MRN), 5-HT neurons compose 30-50% of these nuclei [START_REF] Belin | Immunohistochemical evidence for the presence of gamma-aminobutyric acid and serotonin in one nerve cell. A study on the raphe nuclei of the rat using antibodies to glutamate decarboxylase and serotonin[END_REF][START_REF] Descarries | The serotonin neurons in nucleus raphe dorsalis of adult rat: a light and electron microscope radioautographic study[END_REF][START_REF] Huang | Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task[END_REF]. The function of this monoaminergic system starts early in development and 5-HT has been shown to have a key role in neural development [START_REF] Khozhai | Participation of serotonin in the mechanisms of the formation of trigeminal motor nucleus[END_REF][START_REF] Lavdas | Serotonin promotes the differentiation of glutamate neurons in organotypic slice cultures of the developing cerebral cortex[END_REF][START_REF] Shah | Abnormal serotonin levels during perinatal development lead to behavioral deficits in adulthood[END_REF][START_REF] Vichier-Guerre | Impact of selective serotonin reuptake inhibitors on neural crest stem cell formation[END_REF]. In addition, the massive distribution of 5-HT fibers to practically all parts of the brain combined with the diversity of receptors (over 14 subtypes) explain the involvement of 5-HT in a long list of physiological and behavioral functions [START_REF] Berger | The expanded biology of serotonin[END_REF][START_REF] Linley | Pattern of distribution of serotonergic fibers to the orbitomedial and insular cortex in the rat[END_REF][START_REF] Linley | Pattern of distribution of serotonergic fibers to the amygdala and extended amygdala in the rat[END_REF][START_REF] Rodriguez | Serotonergic projections and serotonin receptor expression in the reticular nucleus of the thalamus in the rat[END_REF][START_REF] Vasudeva | Functional organization of the dorsal raphe efferent system with special consideration of nitrergic cell groups[END_REF][START_REF] Vertes | A PHA-L analysis of ascending projections of the dorsal raphe nucleus in the rat[END_REF][START_REF] Vertes | Pattern of distribution of serotonergic fibers to the thalamus of the rat[END_REF][START_REF] Zhou | Induced homotypic sprouting of serotonergic fibers in hippocampus. II. An immunocytochemistry study[END_REF] including respiration, reproduction, cardiovascular regulation, sleepwake cycle, locomotion, emotion, learning, and memory [START_REF] Charnay | Brain serotonergic circuitries[END_REF][START_REF] Gervasoni | Role and origin of the GABAergic innervation of dorsal raphe serotonergic neurons[END_REF][START_REF] Jacobs | 5-HT and motor control: a hypothesis[END_REF][START_REF] King | A role for the 5-HT(1A), 5-HT4 and 5-HT6 receptors in learning and memory[END_REF][START_REF] Lucki | The spectrum of behaviors influenced by serotonin[END_REF][START_REF] Teixeira | Hippocampal 5-HT Input Regulates Memory Formation and Schaffer Collateral Excitation[END_REF][START_REF] Teran | Serotonin neurons and central respiratory chemoreception: where are we now?[END_REF][START_REF] Urbain | Electrophysiological diversity of the dorsal raphe cells across the sleep-wake cycle of the rat[END_REF]. Among all the roles of the 5-HT system, one that is still being dissected is its role in reward (For review: [START_REF] Hayes | 5-HT receptors and reward-related behaviour: a review[END_REF][START_REF] Hu | Reward and Aversion[END_REF][START_REF] Kranz | Reward and the serotonergic system[END_REF]Luo et al., 2015)). In animals, it was notably shown that electrical stimulation of the RN can be reinforcing (Miliaressis, 1977;Miliaressis et al., 1975;Van Der Kooy et al., 1978). In humans, using fMRI, the DRN is activated during a reward-related task [START_REF] Tanaka | Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops[END_REF].

The RN has strong connections to brain areas involved in reward processing including the nucleus accumbens (NAc), the amygdala, the medial prefrontal cortex and the olfactory tubercle [START_REF] Beart | 5-Hydroxytryptamine and 5-hydroxytryptaminergicdopaminergic interactions in the ventral tegmental area of rat brain[END_REF][START_REF] Berridge | Affective neuroscience of pleasure: reward in humans and animals[END_REF][START_REF] Brown | Dual serotonin (5-HT) projections to the nucleus accumbens core and shell: relation of the 5-HT transporter to amphetamine-induced neurotoxicity[END_REF][START_REF] Macoveanu | Serotonergic modulation of reward and punishment: evidence from pharmacological fMRI studies[END_REF][START_REF] Moore | Serotonin neurons of the midbrain raphe: ascending projections[END_REF][START_REF] Paton | The primate amygdala represents the positive and negative value of visual stimuli during learning[END_REF][START_REF] Steinbusch | The nucleus raphe dorsalis of the rat and its projection upon the caudatoputamen. A combined cytoarchitectonic, immunohistochemical and retrograde transport study[END_REF][START_REF] Vertes | A PHA-L analysis of ascending projections of the dorsal raphe nucleus in the rat[END_REF]. But the most important interconnection linking the serotonergic system with reward function is the one with the dopaminergic system, dopamine (DA) being the well-recognized neurotransmitter involved in reward processing [START_REF] Schultz | Dopamine neurons and their role in reward mechanisms[END_REF][START_REF] Schultz | A neural substrate of prediction and reward[END_REF][START_REF] Wise | Addictive drugs and brain stimulation reward[END_REF]. The RN project massively to the ventral tegmental area (VTA) and to the substantia nigra [START_REF] Herve | Serotonin axon terminals in the ventral tegmental area of the rat: fine structure and synaptic input to dopaminergic neurons[END_REF][START_REF] Miller | Anatomical and electrophysiological identification of a projection from the mesencephalic raphe to the caudate-putamen in the rat[END_REF][START_REF] Vertes | A PHA-L analysis of ascending projections of the dorsal raphe nucleus in the rat[END_REF][START_REF] Watabe-Uchida | Whole-brain mapping of direct inputs to midbrain dopamine neurons[END_REF]. The RN also receive, although to a lesser extent, projections back from the dopaminergic system [START_REF] Kalen | Projections from the ventral tegmental area and mesencephalic raphe to the dorsal raphe nucleus in the rat. Evidence for a minor dopaminergic component[END_REF]) [START_REF] Ogawa | Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems[END_REF][START_REF] Vertes | Efferent and afferent connections of the dorsal and median raphe nuclei in the rat[END_REF]. Different views have emerged on the respective role of DA and 5-HT in reward functions either opposing them or making them allies [START_REF] Boureau | Opponency revisited: competition and cooperation between dopamine and serotonin[END_REF][START_REF] Cools | Serotonin and dopamine: unifying affective, activational, and decision functions[END_REF][START_REF] Daw | Opponent interactions between serotonin and dopamine[END_REF]. The complexity of this relationship notably lies in the fact that there is a heterogeneous expression of 5HT receptors in the VTA on both DA and non-DA neurons [START_REF] De Deurwaerdere | Serotonergic modulation of the activity of mesencephalic dopaminergic systems: Therapeutic implications[END_REF]. The heterogeneous expression of receptors also holds true in other structures involved in reward processing [START_REF] Hayes | 5-HT receptors and reward-related behaviour: a review[END_REF]. That may explain why manipulation of 5-HT levels has complex effects on reward processing. For example, in humans, there is a disparity in the effects of acute tryptophan depletion on reward-related processes and on the activation of reward-related brain areas [START_REF] Cools | Tryptophan depletion disrupts the motivational guidance of goal-directed behavior as a function of trait impulsivity[END_REF][START_REF] Cools | Acute tryptophan depletion in healthy volunteers enhances punishment prediction but does not affect reward prediction[END_REF][START_REF] Faulkner | The role of serotonin in reward, punishment and behavioural inhibition in humans: insights from studies with acute tryptophan depletion[END_REF][START_REF] Finger | The impact of tryptophan depletion and 5-HTTLPR genotype on passive avoidance and response reversal instrumental learning tasks[END_REF][START_REF] Macoveanu | Serotonergic modulation of reward and punishment: evidence from pharmacological fMRI studies[END_REF][START_REF] Rogers | Tryptophan depletion alters the decision-making of healthy volunteers through altered processing of reward cues[END_REF][START_REF] Tanaka | Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum[END_REF].

To clearly decipher the role of the serotonergic system in reward and on the dopaminergic system, there is a need to target specific circuits. This can notably be achieved through the use of recent tools including opto-and chemo-genetics allowing to specifically tag different neuronal subsets. In this review, we will first describe the evidence linking 5-HT to reward, discuss the conflicting evidence emerging on the effects of the serotonergic system over the dopaminergic system and then focus on recent studies specifically tagging the RN to VTA pathway in the context of reward.

The serotonergic system and reward: a complex relationship

In this section, we will describe some of the evidence linking 5-HT to reward. We will start with the "classical" involvement of 5-HT in reinforcing paradigms, specifically intracranial self-stimulation (ICSS) and conditioned place preference (CPP). In Table 1, we summarize ICSS and pharmacological studies and its effects on behavior and dopaminergic activity. We will then discuss the encoding of reward in RN neurons and recent studies specifically tagging and stimulating RN 5-HT neurons during different reward-related tasks.

2.1.

Reinforcing paradigms implicating the serotonergic system in reward ICSS has been widely used to determine the brain areas involved in reward function and drug addiction. In this operant paradigm, the animals perform an action (e.g., nose poke, lever press) to receive a brief intra-cerebral electrical stimulation. The electrical stimulation of a brain area involved in reward will induce higher rates of "action".

Mapping of brain areas involved in reward using this approach had a massive bump in the seventies. Notably, stimulation of the VTA and the lateral hypothalamus stimulations was found to be reinforcing [START_REF] Miliaressis | Self-stimulation versus food reinforcement: comparative study of two different nervous structures, the lateral hypothalamus and the ventral tegmental area of the mesencephalon[END_REF]. Many studies demonstrated the critical importance of the dopaminergic system in reward function through this paradigm [START_REF] Phillips | The role of dopamine in maintaining intracranial selfstimulation in the ventral tegmentum, nucleus accumbens, and medial prefrontal cortex[END_REF][START_REF] Routtenberg | Brainstem pathways of reward[END_REF].

Regarding the serotonergic system, several studies did demonstrate increased rates of self-stimulation (SS) with electrodes located in the DRN and the MRN (Gratton, 1982;Margules, 1969;Miliaressis, 1977;Miliaressis et al., 1975;Rompre and Boye, 1989;Van Der Kooy et al., 1978). However, there were some discrepancies regarding the neurochemical basis of this effect. Margules (1969) observed that SS was increased by D-amphetamine (reuptake inhibition increasing DA and Noradrenaline), blocked by chlorpromazine (DA2 receptor antagonist), and not altered by the administration of parachlorophenylalanine (PCPA; potent tryptophan hydroxylase inhibitor), leading the author to conclude that SS from the DRN was due to the stimulation of noradrenergic fibers.

However, high rates of SS were later reported with electrodes located in the MRN or DRN and this effect was decreased following PCPA, suggesting the involvement of 5-HT (Miliaressis, 1977;Miliaressis et al., 1975;Van Der Kooy et al., 1978). On the contrary, Simon et al. (1976) reported that PCPA increased the SS rate with electrodes located in the DRN. This result aligns with experiments showing that muscimol infusion (GABAa agonist) in the MRN and DRN induces higher rates of lever press, suggesting that the RN neurons act on the reward circuit through a tonic-inhibition of the DA-dependent reward circuit (Liu and Ikemoto, 2007).

To conclude, several bodies of evidence point to higher rates of RN SS, however the neurochemical basis of this effect seems to be complex. The complexity and contradictory effects are even more prominent when looking at 5-HT implication in brainstimulation paradigms. For example, depletion of brain 5-HT, using PCPA (intragastric route), with electrodes located in the lateral hypothalamus increases the rate of SS (Phillips et al., 1976;Poschel and Ninteman, 1971;van der Kooy et al., 1977). Similarly, another drug reducing the 5-HT level, 5-HT1A agonist 8-hydroxy-2-(di-n-propylamino) tetralin injection in the MRN, increases lateral hypothalamus stimulation (Fletcher et al., 1995). This increase was also found with electrodes located in the medial forebrain bundle and VTA [START_REF] Gibson | Effect of selective inhibitors of tyrosine and tryptophan hydroxylases on self-stimulation in the rat[END_REF]Poschel and Ninteman, 1971). These results

suggest that a decrease of 5-HT potentiates sensitivity to rewarding stimuli. However, the effect is not ubiquitous as it was found that PCPA decreases SS with electrodes located in the hippocampal formation (van der Kooy et al., 1977) and the neostriatum (Phillips et al., 1976). In addition, local infusion of 5-HT in the ventral midbrain increases SS with electrodes located in the medial forebrain bundle (Redgrave and Horrell, 1976). Fletcher et al. (1993) injected the 8-hydroxy-2-(di-n-propylamino) tetralin in the MRN and DRN, leading to a decrease of 5-HT transmission. They demonstrated a preference for the compartment where the animals were injected with this selective agonist, and the effect was more robust when the drug was injected in the DRN compared to the MRN.

CPP was also observed when injecting baclofen, a GABAb receptor agonist leading to an inhibition of 5-HT neurons, in the DRN and MRN, or muscimol in the MRN (Liu and Ikemoto, 2007;Shin and Ikemoto, 2010). These reports, as well as others [START_REF] Higgins | Behavioural and biochemical consequences following activation of 5HT1-like and GABA receptors in the dorsal raphe nucleus of the rat[END_REF][START_REF] Invernizzi | Administration of 8hydroxy-2-(Di-n-propylamino)tetralin in raphe nuclei dorsalis and medianus reduces serotonin synthesis in the rat brain: differences in potency and regional sensitivity[END_REF] lead to the view that decreasing 5-HT activity facilitates the reward circuitry. In this line of evidence, [START_REF] Marona-Lewicka | Reinforcing effects of certain serotonin-releasing amphetamine derivatives[END_REF] showed that the selective 5-HT releasing agent, 5-methoxy-6methyl-2aminoindan, induces place aversion in the CPP test. However, and again, the picture is not so simple. Indeed, CPP induced with amphetamine was blocked by a pretreatment with a selective 5-HT uptake blocker [zimelidine- (Kruszewska et al., 1986)] but was also blocked with a treatment decreasing 5-HT transmission [ritanserin, (Nomikos and Spyraki, 1988)]. In addition, it was shown that fluoxetine, increasing the level of 5-HT, can induce CPP in DA-deficient mice (Hnasko et al., 2007).

Why is it so difficult to get a unified picture on the role of the serotonergic system in reward functions? The broad distribution of serotonergic fibers in the brain, the heterogenous properties and distribution of 5-HT receptors, and therefore, the electrode location, the specificity, dose and route of administration of the pharmacological agents used are some of the factors that can account for the contradictory results observed in the literature [START_REF] Kranz | Reward and the serotonergic system[END_REF][START_REF] Muller | The role of serotonin in drug use and addiction[END_REF]Van Der Kooy et al., 1978).

Importantly, these inconsistent results are also observed in human studies, notably using acute tryptophan depletion in reward-related tasks. Indeed, acute tryptophan depletion has been shown to either leave reward processes unaltered [e.g. reward prediction;

choice between immediate and delayed reward; [START_REF] Cools | Acute tryptophan depletion in healthy volunteers enhances punishment prediction but does not affect reward prediction[END_REF][START_REF] Crean | Effect of tryptophan depletion on impulsive behavior in men with or without a family history of alcoholism[END_REF][START_REF] Tanaka | Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum[END_REF]] or to have an impact on reward processes such as discrimination and response to different magnitudes of expected rewards or delayed reward discounting [START_REF] Aquili | The Role of Tryptophan and Tyrosine in Executive Function and Reward Processing[END_REF][START_REF] Cools | Tryptophan depletion disrupts the motivational guidance of goal-directed behavior as a function of trait impulsivity[END_REF][START_REF] Faulkner | The role of serotonin in reward, punishment and behavioural inhibition in humans: insights from studies with acute tryptophan depletion[END_REF][START_REF] Finger | The impact of tryptophan depletion and 5-HTTLPR genotype on passive avoidance and response reversal instrumental learning tasks[END_REF][START_REF] Rogers | Tryptophan depletion alters the decision-making of healthy volunteers through altered processing of reward cues[END_REF][START_REF] Schweighofer | Low-serotonin levels increase delayed reward discounting in humans[END_REF]. In addition, the administration of anti-depressants has been shown to have distinct effects on brain regions responding to reward [START_REF] References Abler | Neural correlates of antidepressant-related sexual dysfunction: a placebocontrolled fMRI study on healthy males under subchronic paroxetine and bupropion[END_REF][START_REF] Del-Ben | The effect of citalopram pretreatment on neuronal responses to neuropsychological tasks in normal volunteers: an FMRI study[END_REF][START_REF] Macoveanu | Serotonergic modulation of reward and punishment: evidence from pharmacological fMRI studies[END_REF][START_REF] Macoveanu | Playing it safe but losing anyway--serotonergic signaling of negative outcomes in dorsomedial prefrontal cortex in the context of risk-aversion[END_REF][START_REF] Marutani | Functional magnetic resonance imaging study on the effects of acute single administration of paroxetine on motivation-related brain activity[END_REF][START_REF] Mccabe | Diminished neural processing of aversive and rewarding stimuli during selective serotonin reuptake inhibitor treatment[END_REF][START_REF] Vollm | Serotonergic modulation of neuronal responses to behavioural inhibition and reinforcing stimuli: an fMRI study in healthy volunteers[END_REF]. For example, the ventral striatal response to an appetitive reward is decreased following citalopram administration [START_REF] Mccabe | Diminished neural processing of aversive and rewarding stimuli during selective serotonin reuptake inhibitor treatment[END_REF] while striatal response in a monetary reward task was increased following two-week administration of duloxetine [START_REF] Ossewaarde | Two-week administration of the combined serotonin-noradrenaline reuptake inhibitor duloxetine augments functioning of mesolimbic incentive processing circuits[END_REF].

It clearly appears from these studies that the serotonergic system is involved in reward processing however, there is a need to specifically tag 5-HT neurons (leaving the other neuromodulatory system such as noradrenergic fibers unaffected), as well as specific pathways. To go further in the understanding of the contribution of 5-HT neurons in reward processing, we will next review the encoding of reward in putative and genetically identified 5-HT neurons.

Encoding of reward processing in the DRN

The encoding of reward has been well described in the dopaminergic system, especially in the VTA [START_REF] Cohen | Neuron-type-specific signals for reward and punishment in the ventral tegmental area[END_REF][START_REF] Schultz | Dopamine neurons and their role in reward mechanisms[END_REF][START_REF] Schultz | A neural substrate of prediction and reward[END_REF]. In humans, the DRN is activated during a reward-related task using fMRI, specifically when participants learn to obtain large future rewards while sustaining small immediate losses [START_REF] Tanaka | Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops[END_REF]. DRN neurons were also recorded in awake animals performing reward-related tasks. Most of the studies classified putative DRN 5-HT neurons based on their electrophysiological properties [START_REF] Aghajanian | Intracellular recordings from serotonergic dorsal raphe neurons: pacemaker potentials and the effect of LSD[END_REF]. However, these criteria might be insufficient to clearly identify 5-HT neurons as demonstrated by juxtacellular experiments [START_REF] Allers | Neurochemical and anatomical identification of fast-and slow-firing neurones in the rat dorsal raphe nucleus using juxtacellular labelling methods in vivo[END_REF][START_REF] Hajos | Neurochemical identification of stereotypic burst-firing neurons in the rat dorsal raphe nucleus using juxtacellular labelling methods[END_REF][START_REF] Kocsis | Serotonergic neuron diversity: identification of raphe neurons with discharges time-locked to the hippocampal theta rhythm[END_REF]. We will thus also highlight recent evidence obtained with recordings of genetically identified 5-HT neurons during reward-related tasks.

One of the first studies evaluating the activity of DRN neurons in a reward-related task was performed in monkeys [START_REF] Nakamura | Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus[END_REF]. This study, using saccade tasks with a biased reward schedule, revealed that putative DRN 5-HT neuron activity is modulated during the delay period (before reward) but also after reward onset. In rats performing a two-alternative odor discrimination task, DRN neurons are also significantly modulated during the task with a majority responding during the reward phase of the task [START_REF] Ranade | Transient firing of dorsal raphe neurons encodes diverse and specific sensory, motor, and reward events[END_REF]. This effect was further confirmed in studies recording genetically identified 5-HT neurons [START_REF] Cohen | Serotonergic neurons signal reward and punishment on multiple timescales[END_REF][START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Ren | Anatomically Defined and Functionally Distinct Dorsal Raphe Serotonin Sub-systems[END_REF]. The common features of all these reports were that DRN neuron activities are diverse and modulated during reward-related tasks with their activity locked to different events (e.g., nose poke, moving to a reward port, the different reward-related periods).

Response to reward-predicting cues: Responses to reward-predicting cues were observed in a subset of 5-HT neurons and correlated to the stimulus value/expected reward size [START_REF] Cohen | Serotonergic neurons signal reward and punishment on multiple timescales[END_REF][START_REF] Hayashi | Appetitive and aversive information coding in the primate dorsal raphe nucleus[END_REF][START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Matias | Activity patterns of serotonin neurons underlying cognitive flexibility[END_REF]. In this line of results, it was also shown, in humans with acute tryptophan depletion, an altered ability in providing adaptive response to incentivemotivational cues signaling reinforcement certainty [START_REF] Cools | Tryptophan depletion disrupts the motivational guidance of goal-directed behavior as a function of trait impulsivity[END_REF].

Delay period: During the delay period, most 5-HT neurons increase their firing rate [START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF]Miyazaki et al., 2011a). In a sequential food-water navigation task in rats, 5-HT firing increases during waiting for delayed reward (Miyazaki et al., 2011a;[START_REF] Miyazaki | Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience[END_REF]. Confirming the contribution of 5-HT neurons during the delay period, Zhong et al.

(2017) demonstrated a ramp-up activity pattern in response to the conditioning stimulus during the delay to reward period and increasing across learning. This result paralleled the result obtained by DRN recordings showing tonic spike firing during the delay period [START_REF] Bromberg-Martin | Coding of task reward value in the dorsal raphe nucleus[END_REF][START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Nakamura | Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus[END_REF].

These observations lead some of the authors to conclude that putative 5-HT neurons were involved in waiting (Miyazaki et al., 2011a;Miyazaki et al., 2011b). Interestingly, in humans it was found that dietary tryptophan depletion increases the delayed reward discounting with a preference for small immediate rewards rather than larger but delayed rewards [START_REF] Schweighofer | Low-serotonin levels increase delayed reward discounting in humans[END_REF].

Response to reward: Several studies have observed a response of DRN neurons to the reward itself [START_REF] Bromberg-Martin | Coding of task reward value in the dorsal raphe nucleus[END_REF][START_REF] Hayashi | Appetitive and aversive information coding in the primate dorsal raphe nucleus[END_REF][START_REF] Huang | Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task[END_REF][START_REF] Nakamura | Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus[END_REF]. Interestingly, [START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF] demonstrated that DRN 5-HT neurons display a change of activity in response to natural/ecological rewards: when the mouse looks for and acquires sucrose, food, sex, and social interaction. This finding thus extends the previous results to different qualities of reward.

Importantly, activity of DRN neurons is modulated as a function of the received reward magnitude, and these neurons can keep track of the reward value with an activity dependent on reward probability [START_REF] Bromberg-Martin | Coding of task reward value in the dorsal raphe nucleus[END_REF][START_REF] Hayashi | Appetitive and aversive information coding in the primate dorsal raphe nucleus[END_REF][START_REF] Huang | Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task[END_REF][START_REF] Nakamura | Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus[END_REF]. This effect was further confirmed with singleunit/fiberphotometry recordings of identified DRN 5-HT neurons [START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF]. Finally, [START_REF] Zhong | Learning and Stress Shape the Reward Response Patterns of Serotonin Neurons[END_REF] demonstrated that 5-HT activity tracks reward availability and it has also been proposed that DRN neurons have a role in evaluating future reward and tracking progress toward future delayed reward [START_REF] Bromberg-Martin | Coding of task reward value in the dorsal raphe nucleus[END_REF][START_REF] Huang | Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task[END_REF]. Indeed, [START_REF] Bromberg-Martin | Coding of task reward value in the dorsal raphe nucleus[END_REF] showed that most of DRN neurons have their firing activity at task onset correlated with their response to the reward cues and outcomes.

Paralleling the encoding of reward magnitudes in the DRN, it was shown in humans, with dietary tryptophan depletion, a diminished discrimination between different magnitudes of expected rewards [START_REF] Rogers | Tryptophan depletion alters the decision-making of healthy volunteers through altered processing of reward cues[END_REF]. In addition, in humans, tryptophan depletion alters the representation of reward outcome value [START_REF] Seymour | Serotonin selectively modulates reward value in human decision-making[END_REF] and impairs response to rewarded stimuli [START_REF] Finger | The impact of tryptophan depletion and 5-HTTLPR genotype on passive avoidance and response reversal instrumental learning tasks[END_REF].

Activity in the absence of reward: It was observed by [START_REF] Ranade | Transient firing of dorsal raphe neurons encodes diverse and specific sensory, motor, and reward events[END_REF], using reward omission, that a very small population of DRN neurons (4/32) changed their firing rate around the expected time of reward, however, this modulation of DRN activity when the reward was omitted was not found by (Miyazaki et al., 2011a). This difference might be ascribable to the small percentage of units displaying a significant change of activity in the absence of reward and that this subpopulation was not recorded in Miyazaki et al. (2011a). This discrepancy was still present when recording genetically identified 5-HT neurons. [START_REF] Zhong | Learning and Stress Shape the Reward Response Patterns of Serotonin Neurons[END_REF] demonstrated that reward omission led to a progressive decrease of 5-HT neuron response to reward predicting cues. However, Matias et al.

(2017) observed increased 5-HT activity when a fully predicted reward was omitted.

Indeed, using a reversal, a more sudden alteration of the cue-outcome, they showed that 5-HT neurons increase activity to positive and negative reward prediction error which corresponds to unsigned prediction error [START_REF] Matias | Activity patterns of serotonin neurons underlying cognitive flexibility[END_REF].

Encoding of aversive stimulus-Timescale view: Different studies found phasic response to aversive events during a conditioning task [START_REF] Cohen | Serotonergic neurons signal reward and punishment on multiple timescales[END_REF][START_REF] Matias | Activity patterns of serotonin neurons underlying cognitive flexibility[END_REF], however, this response to aversive events was not observed by [START_REF] Li | Serotonin neurons in the dorsal raphe nucleus encode reward signals[END_REF].

Nonetheless, an interesting, and perhaps key, point raised by the study of [START_REF] Cohen | Serotonergic neurons signal reward and punishment on multiple timescales[END_REF] is that there are different timescales of 5-HT response during the reward/punishment conditioning task, with a tonic firing variation depending on state value (long-lasting), and a phasic/transient activity in response to reward-predicting cues, reward or punishment. This important notion of timescale was also found by [START_REF] Hayashi | Appetitive and aversive information coding in the primate dorsal raphe nucleus[END_REF] demonstrating temporal firing rate variation between appetitive and aversive contexts. Notably, they found that 5-HT neurons encode the aversive blocks via a tonic activity but rather on a shorter time scale during the appetitive blocks. These results are paralleled with the ones performed in humans where it was suggested and then demonstated that 5-HT is involved in reward prediction at different timescales [START_REF] Doya | Metalearning and neuromodulation[END_REF][START_REF] Kranz | Reward and the serotonergic system[END_REF][START_REF] Tanaka | Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum[END_REF]. Indeed, [START_REF] Tanaka | Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum[END_REF] demonstrated that 5-HT, through dietary manipulation of tryptophan level, differentially controls the time scale of reward prediction within the striatum.

To conclude, all these studies point to the role of DRN 5-HT neurons in reward, however, there is a lot of diversity in neuronal response. To gain insight into the functions of DRN 5-HT neurons in reward, the next section will focus on the consequences of activation of genetically-tagged 5-HT neurons during reward-related tasks.

2.3.

Specific DRN 5-HT neurons stimulations: to be or not to be rewarding?

The development of tools including opto-and chemo-genetics allows to specifically tag different neuronal subsets, to test the role of DRN 5-HT neurons in reward-related tasks and to have good temporal resolution. In Table 2, we summarize recent studies using optogenetic stimulation to assess the interaction between the serotonergic system, reward and the dopaminergic system. Using different knock-out mice combined with optogenetics and pharmacological agents, they showed that both 5-HT and glutamate are critical for reward signaling in DRN 5-HT neurons. This study strongly supports RN serotonergic neural activity as a key signal for reward. At the same time and using different paradigms as well as targeting the 5-HT population with a different promoter (tryptophan hydroxylase 2 promoter), it was shown that activating 5-HT neurons rather than being rewarding by itself, promotes waiting for future rewards [START_REF] Miyazaki | Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience[END_REF][START_REF] Miyazaki | Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards[END_REF]. Notably, the authors used a sequential tone-food waiting task, and showed that DRN-5-HT photostimulation 1) is not rewarding as it does not reinforce spontaneous nose poking, 2) decreases the tone wait errors while the mice are waiting for the tone, 3) decreases the reward wait errors while the mice are waiting during the variable delay period for a reward, this is specifically true for high expectation reward (75% vs 25% or 50%) and when the timing of the reward is 2). In the rest of the review, we will focus on a specific pathway, namely the RN to VTA to disentangle the effect of the serotonergic system in reward function.

The serotonergic system and the dopaminergic system : a love/hate relationship

The VTA is an integral piece of the reward puzzle. Its contribution to reward processing has been well described using single-unit recordings and pharmacological approach [START_REF] Morales | Ventral tegmental area: cellular heterogeneity, connectivity and behaviour[END_REF][START_REF] Schultz | Dopamine neurons and their role in reward mechanisms[END_REF][START_REF] Schultz | A neural substrate of prediction and reward[END_REF][START_REF] Wise | Dopamine, learning and motivation[END_REF].

Optogenetics manipulations have also revealed that phasic photoactivation of VTA DA neurons promotes place preference in the CPP test and positive reinforcement [START_REF] Kim | Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement[END_REF][START_REF] Tsai | Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning[END_REF].

The anatomical connection between the RN and the VTA, the impact of RN on VTA DA activity as well as the specific stimulation of the RN to VTA pathway will be discussed in this section.

Anatomical relationship between the RN AND VTA

It has been demonstrated that the RN project massively to the VTA [START_REF] Herve | Serotonin axon terminals in the ventral tegmental area of the rat: fine structure and synaptic input to dopaminergic neurons[END_REF][START_REF] Miller | Anatomical and electrophysiological identification of a projection from the mesencephalic raphe to the caudate-putamen in the rat[END_REF][START_REF] Vertes | A PHA-L analysis of ascending projections of the dorsal raphe nucleus in the rat[END_REF][START_REF] Watabe-Uchida | Whole-brain mapping of direct inputs to midbrain dopamine neurons[END_REF]. Both the MRN and DRN project to the VTA and substantia nigra but the DRN projections are more intense than the MRN (Watabe-Uchida et al., 2012).

Importantly, RN to VTA projecting neurons have been found to release not only 5-HT, but also glutamate. In fact, different populations have been identified that transmit 5-HT only, glutamate-only or co-transmit 5-HT and glutamate [START_REF] Qi | A glutamatergic reward input from the dorsal raphe to ventral tegmental area dopamine neurons[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF]; possibly having different implications in behavioral expression. various way of regulation of the serotonergic system over the VTA activity [START_REF] Doherty | Ultrastructural localization of the serotonin 2A receptor in dopaminergic neurons in the ventral tegmental area[END_REF].

The RN also receive, although to a lesser extent, projections back from the dopaminergic system: DRN and MRN 5-HT neurons receive projections from the substantia nigra, the VTA as well as the retrorubral field [START_REF] Kalen | Projections from the ventral tegmental area and mesencephalic raphe to the dorsal raphe nucleus in the rat. Evidence for a minor dopaminergic component[END_REF]) [START_REF] Ogawa | Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems[END_REF][START_REF] Vertes | Efferent and afferent connections of the dorsal and median raphe nuclei in the rat[END_REF]. [START_REF] Ogawa | Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems[END_REF] demonstrated that VTA and substantia nigra project strongly to DRN 5-HT neurons. Interestingly, the majority of VTA neurons projecting to DRN 5-HT neurons are non-dopaminergic. MRN 5-HT neurons also receive input from the VTA but much more moderate inputs from substantia nigra. It is interesting to note that DRN 5-HT neurons and VTA DA neurons share many monosynaptic inputs, especially from the hypothalamus, amygdala, and basal ganglia [START_REF] Beier | Circuit architecture of VTA dopamine neurons revealed by systematic input-output mapping[END_REF][START_REF] Ogawa | Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems[END_REF][START_REF] Watabe-Uchida | Whole-brain mapping of direct inputs to midbrain dopamine neurons[END_REF].

To conclude, hardwire between the serotonergic system and the dopaminergic system is heavy, specifically there are intense projections from the DRN to the VTA and back and those two areas share many similar inputs from a wide variety of brain regions.

Therefore, understanding this specific pathway is critical to gain insight in reward-related functions given the major role of VTA DA in reward.

Importantly, a recent work, using rabies virus-based and cell-type-specific monosynaptic tracing techniques, in addition to whole brain trans-synaptic tracing experiments was able to reveal the input-output relationship of VTA DA neurons [START_REF] Beier | Circuit architecture of VTA dopamine neurons revealed by systematic input-output mapping[END_REF]. This study has uncovered a strong disynaptic circuit linking the DRN, VTA, and NAc with DRN input to NAc projecting VTA DA neurons. This is particularly interesting in the context of reward and drug addiction, reviewed in [START_REF] Day | The nucleus accumbens and Pavlovian reward learning[END_REF].

Suppressing or activating? Impact of DRN 5-HT neurons on the VTA dopaminergic system

It has long been known both in-vitro and in-vivo that 5-HT-like drugs can modulate activity of dopaminergic neurons and the concentration of DOPAC in the VTA [START_REF] Beart | 5-Hydroxytryptamine and 5-hydroxytryptaminergicdopaminergic interactions in the ventral tegmental area of rat brain[END_REF]. This supports the mechanism by which anti-depressants can modulate mesolimbic dopaminergic circuits in humans [START_REF] Mccabe | Diminished neural processing of aversive and rewarding stimuli during selective serotonin reuptake inhibitor treatment[END_REF][START_REF] Ossewaarde | Two-week administration of the combined serotonin-noradrenaline reuptake inhibitor duloxetine augments functioning of mesolimbic incentive processing circuits[END_REF].

The effect of 5-HT on DA neuronal activity seems to be complex, depending on the type of the receptors targeted and the specificities of administration of the pharmacological agent (route, localization, acute or chronic). De Deurwaerdere and Di Giovanni (2017)

thoroughly reviewed the effects of different 5-HT modulation (raphe lesion; receptor agonist or antagonist) on DA neuronal activity [for a full list see table 2 of (De Deurwaerdere and Di Giovanni, 2017)].

Along with the longstanding view that increasing 5-HT level inhibits reward circuits, it was shown that various selective 5-HT reuptake inhibitors induce a significant decrease of the basal firing of VTA DA cells (Di Mascio et al., 1998;Prisco et al., 1994;[START_REF] Prisco | Chronic treatment with DAU 6215, a new 5-HT3 receptor antagonist, causes a selective decrease in the number of spontaneously active dopaminergic neurons in the rat ventral tegmental area[END_REF], leading to the interpretation that VTA DA neurons are under the tonic inhibitory control of DRN 5-HT terminals (Prisco et al., 1994). Moreover, in-vivo systemic injection of 5-HT1a receptor agonist induces an increase of VTA neuron firing rate, and specifically their bursting, but this was not observed when the agonist was infused directly in the VTA. In addition, it was shown that ritanserin (5-HT2 receptor antagonist, systemic administration) increases the firing rate and the bursting of VTA and substantia nigra DA neurons (Ugedo et al., 1989). Congruent with this observation, it was shown that lesions of the DRN AND MRN increase the dopaminergic metabolites (DOPAC and HVA) but the level of DA level remained unchanged in the striatum [START_REF] Juorio | The effect of raphe nuclei lesions on striatal tyramine concentration and dopamine turnover in the rat[END_REF]. Consistently, [START_REF] Morelli | Chronic 5-HT transporter blockade reduces DA signaling to elicit basal ganglia dysfunction[END_REF] demonstrated in a model of mice with chronic 5-HT transporter blockade that DA signaling (DOPAC and HVA metabolite level) is decreased in the nigrostriatal system.

However, once again there are ambiguities (table 1). Indeed, it was shown by Guan and McBride (1989) that infusion of 5-HT directly in the VTA increases the DA release in the NAc via 5-HT1B receptors leading the authors to conclude that 5-HT innervations in the VTA have an excitatory action. An excitatory effect of 5-HT (highly concentrated) on VTA DA neurons was also found in vitro [START_REF] Pessia | Actions of 5hydroxytryptamine on ventral tegmental area neurons of the rat in vitro[END_REF]. The authors showed that 5-HT depolarized almost half of the DA cells recorded in vitro. This depolarization was blocked by ketanserin and reproduced by local application of a 5-HT2 receptor agonist, meaning that the increase of DA neuron firing rate was mostly mediated through 5-HT2 receptors.

There is an interesting alternative route by which 5-HT can regulate DA activity -via modulation of GABAergic cells -adding another layer of complexity to the DA-5-HT interaction. As mentioned earlier, 5-HT2 receptors are expressed in the VTA in both DA and GABAergic neurons [START_REF] Doherty | Ultrastructural localization of the serotonin 2A receptor in dopaminergic neurons in the ventral tegmental area[END_REF][START_REF] Nocjar | Localization of 5-HT(2A) receptors on dopamine cells in subnuclei of the midbrain A10 cell group[END_REF]. These receptors have been shown to mediate a modest depolarization leading to increased DA activity and DA release [START_REF] Pessia | Actions of 5hydroxytryptamine on ventral tegmental area neurons of the rat in vitro[END_REF]. However, 5-HT2A/C receptor activation in substantia nigra GABAergic neurons may indirectly inhibit DA neuron activity [START_REF] Demireva | 5-HT2C receptor blockade reverses SSRI-associated basal ganglia dysfunction and potentiates therapeutic efficacy[END_REF][START_REF] Esposito | Serotonin-dopamine interaction as a focus of novel antidepressant drugs[END_REF][START_REF] Tepper | GABAA receptor-mediated inhibition of rat substantia nigra dopaminergic neurons by pars reticulata projection neurons[END_REF].

It is clear that 5-HT impacts VTA DA activity, however, the effects are complex [START_REF] De Deurwaerdere | Serotonergic modulation of the activity of mesencephalic dopaminergic systems: Therapeutic implications[END_REF][START_REF] De Deurwaerdère | Serotonin in Health and Disease[END_REF].

Another report in-vivo in anesthetized rats notably highlighted this complexity. Gervais and Rouillard (2000) electrically stimulated the DRN while recording extracellular single unit activity of DA neurons in the substantia nigra and VTA, and they revealed a complex response to DRN stimulation. They showed two populations of VTA DA cells; 63% that were initially excited followed by an inhibition, and 34% displayed an inhibition-excitation response. Importantly, this former inhibition-excitation response was abolished by PCPA but not the excitation-inhibition responses. This suggests that the DRN-5-HT inputs to VTA DA neurons are mainly inhibitory but also that some DRN non-5-HT inputs can impact the activity of VTA DA cells. To go further in the understanding of the impact of DRN 5-HT neurons and non-5-HT neurons on VTA, it is necessary to specifically tag each cellular type and record VTA DA neuron activity.

Activation of the DRN to VTA pathway in reinforcing paradigms: contribution of serotonin and glutamate

As described in section 2.3, whether or not the stimulation of DRN 5-HT neurons was reinforcing is still under debate with some studies showing no reinforcing effect (Fonseca et al., 2015;[START_REF] Mcdevitt | Serotonergic versus nonserotonergic dorsal raphe projection neurons: differential participation in reward circuitry[END_REF][START_REF] Miyazaki | Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience[END_REF][START_REF] Miyazaki | Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards[END_REF] and others demonstrating a reinforcing effect [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Nagai | The Role of Dorsal Raphe Serotonin Neurons in the Balance between Reward and Aversion[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF].

To refine the analysis of the role of DRN 5-HT neurons in reward functions, the DRN 5-HT neurons to VTA DA pathway can be specifically tagged and modulated (see table 2).

Consistent with previous studies showing either that DRN 5-HT stimulation was not reinforcing or that DRN stimulation reinforcing effect was not 5-HT dependent (Fonseca et al., 2015;[START_REF] Miyazaki | Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards[END_REF]Simon et al., 1976), it was shown that photostimulation of DRN 5-HT neuron projections to VTA when associated to a low dose of citalopram decreases operant response to saccharine, similar to what the authors found when stimulating 5-HT neurons in the DRN [START_REF] Browne | Dorsal raphe serotonin neurons inhibit operant responding for reward via inputs to the ventral tegmental area but not the nucleus accumbens: evidence from studies combining optogenetic stimulation and serotonin reuptake inhibition[END_REF]. Therefore, this report shows that DRN 5-HT projections to VTA have an inhibitory effect. In addition, [START_REF] Mcdevitt | Serotonergic versus nonserotonergic dorsal raphe projection neurons: differential participation in reward circuitry[END_REF] did not observe that DRN 5-HT fibers to VTA stimulation was reinforcing.

However, [START_REF] Nagai | The Role of Dorsal Raphe Serotonin Neurons in the Balance between Reward and Aversion[END_REF] found the opposite effect. Indeed, they showed that activation of DRN 5-HT neurons to VTA pathway increases self-stimulation behavior and conditioned place preference. In addition, these authors also showed that inhibiting the These discrepancies might come from the fact that the DRN is composed of different cellular types: serotonergic, GABAergic, as well as non-serotonergic glutamatergic neurons projecting to the VTA. In addition, it has also been demonstrated that DRN 5-HT neurons, notably projecting to the VTA, can also co-express vesicular glutamate transporter type 3 [START_REF] Amilhon | VGLUT3 (Vesicular Glutamate Transporter Type 3) Contribution to the Regulation of Serotonergic Transmission and Anxiety[END_REF][START_REF] Gras | A Third Vesicular Glutamate Transporter Expressed by Cholinergic and Serotoninergic Neurons[END_REF][START_REF] Ren | Anatomically Defined and Functionally Distinct Dorsal Raphe Serotonin Sub-systems[END_REF]. Importantly, [START_REF] Mcdevitt | Serotonergic versus nonserotonergic dorsal raphe projection neurons: differential participation in reward circuitry[END_REF] found that while DRN 5-HT fibers to VTA stimulation was not reinforcing, stimulating DRN non-serotonergic glutamatergic projections to the VTA was.

They also showed that stimulation of this non-serotonergic DRN to VTA pathway induces monosynaptic glutamatergic currents and action potentials in DA cells.

Congruently, [START_REF] Qi | A glutamatergic reward input from the dorsal raphe to ventral tegmental area dopamine neurons[END_REF] shown that a downregulation of GluN2C NMDA subunit (siRNA injection in the VTA) leads to a reduction of nose pokes in rats with stimulating electrodes implanted in the DRN [START_REF] Hernandez | Glutamate NMDA receptors containing GluN2C subunit relay the reward signal of the ventral tegmental area upon dorsal raphe stimulation[END_REF].

Altogether, these studies let [START_REF] Mcdannald | Serotonin: waiting but not rewarding[END_REF] to propose that DRN serotonergic cells were involved in patience while the glutamatergic DRN to VTA pathway was mediating reward (Fonseca et al., 2015;[START_REF] Mcdevitt | Serotonergic versus nonserotonergic dorsal raphe projection neurons: differential participation in reward circuitry[END_REF][START_REF] Miyazaki | Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience[END_REF][START_REF] Miyazaki | Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards[END_REF]. But, as always with the serotonergic system, this conclusion must be put in perspective with a recent cohort of studies.

Different groups have recently demonstrated that stimulation of 5-HT cells can be reinforcing and that this effect is mediated through activation of VTA-DA cells via glutamatergic co-transmission [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Luo | Reward processing by the dorsal raphe nucleus: 5-HT and beyond[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF]. Indeed, it was demonstrated that DRN neurons express VGlut3 and that both 5-HT only and 5-HT-glutamatergic neurons from DRN project to VTA neurons [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF][START_REF] Hioki | Vesicular glutamate transporter 3-expressing nonserotonergic projection neurons constitute a subregion in the rat midbrain raphe nuclei[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF]. Paralleling these results, a co-transmission of 5-HT and glutamate in the basal amygdala and orbitofrontal cortex following 5-HT fibers stimulation was demonstrated [START_REF] Ren | Anatomically Defined and Functionally Distinct Dorsal Raphe Serotonin Sub-systems[END_REF][START_REF] Sengupta | Control of Amygdala Circuits by 5-HT Neurons via 5-HT and Glutamate Cotransmission[END_REF].

In-vitro recordings of VTA DA neurons combined with specific stimulation of DRN-5-HT neurons revealed that these neurons release 5-HT and glutamate on VTA DA neurons [START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF]. Optical stimulation of RN 5-HT fibers can evoke EPSP in VTA DA neurons that are blocked by AMPA receptor antagonists [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF]. In addition, DRN 5-HT-glutamatergic neurons make excitatory synapses on VTA DA neurons targeting the NAc, and this effect is mediated through both AMPA and 5-HT3 receptors [START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF]. Moreover, stimulation of 5-HT fibers in the VTA led to a release of DA in the NAc mediated through AMPA and 5-HT3 receptors [START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF]. The results from these three laboratories lead to the conclusion that DRN 5-HT projections to the VTA can be excitatory through a co-transmission of glutamate and led to a release of DA in the NAc. This mechanistic and hardwire evidence explains the results obtained during reinforcing paradigms. Indeed, activation of DRN-5-HT fibers in the VTA led to a strong reinforcement effect in place preference mediated through AMPA and 5-HT3 receptors [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF].

Finally, [START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF] made a clever comparison by stimulating either the DR SERT+ fibers, which can co-transmit 5-HT and glutamate, or the DR glutamatergic (VGlut3 -only) fibers to the VTA. They showed that both induce CPP, but the effect is weaker but lasts longer when stimulating SERT+ fibers compared to DR VGLUT3-only fibers. This is particularly interesting in the light of different timescales encoding found in DRN 5-HT neurons in rodents [START_REF] Cohen | Serotonergic neurons signal reward and punishment on multiple timescales[END_REF][START_REF] Hayashi | Appetitive and aversive information coding in the primate dorsal raphe nucleus[END_REF] as well as the involvement of 5-HT, in humans, in the control of the time scale of reward prediction within different subparts of the striatum [START_REF] Tanaka | Serotonin differentially regulates short-and long-term prediction of rewards in the ventral and dorsal striatum[END_REF].

The different results obtained by the stimulation of the DRN to VTA pathway and whether the reinforcing effect is mediated through DRN glutamatergic only or DRN serotonergic and glutamatergic co-transmission are probably not exclusive from each other and are probably dependent on the context, the tasks used, their timeline, the cellular type tagged as well as the network of cells recruited during each specific task. In our opinion, the critical point from all these studies is that stimulation of DRN to VTA inputs can be rewarding, either by 5-HT via 5-HT3 receptor or/and glutamate (Figure 1).

Finally, in addition to the routes or neurotransmitters by which RN can modulate DA activity, a loop can also take place between the RN and the VTA. Indeed, the VTA projects to the DRN [START_REF] Ogawa | Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems[END_REF] and this circuit, as well as the feedback activity it may convey, can also be critical for reward-related functions. Recently, it was found that VTA inputs to the DRN were primarily GABAergic. Importantly, there were spatially segregated population-projections with the rostral VTA projecting to DRN GABAergic neurons and the caudal VTA projecting to DRN 5-HT neurons [START_REF] Li | Rostral and Caudal Ventral Tegmental Area GABAergic Inputs to Different Dorsal Raphe Neurons Participate in Opioid Dependence[END_REF]. The functions of those two sub-circuits are distinct, with the rostral VTA GABAergic projection to DRN GABAergic neurons leading to a disinhibition of 5-HT neurons, while the caudal VTA GABAergic neurons led to an inhibition of 5-HT neurons. This was further tested in real-time place preference or real-time place aversion tasks. Photoactivation of the rostral VTA fibers to the DRN lead to an avoidance of the chamber, while the activation of the caudal VTA fibers to the DRN led to a preference for the stimulated-paired chamber [START_REF] Li | Rostral and Caudal Ventral Tegmental Area GABAergic Inputs to Different Dorsal Raphe Neurons Participate in Opioid Dependence[END_REF]. The results of this study are in line with results demonstrating that DRN 5-HT neurons are not reinforcing [START_REF] Browne | Dorsal raphe serotonin neurons inhibit operant responding for reward via inputs to the ventral tegmental area but not the nucleus accumbens: evidence from studies combining optogenetic stimulation and serotonin reuptake inhibition[END_REF][START_REF] Mcdevitt | Serotonergic versus nonserotonergic dorsal raphe projection neurons: differential participation in reward circuitry[END_REF] .

But as we highlighted, the complexity of the serotonergic system and the numerous subpopulations of the RN, as well as the subpopulations from the dopaminergic systems might give rise to different behavioral outcomes when stimulated. In any case, this last study highlights the importance of studying sub-circuits, neuronal types, and different timescales in a large cohort of reward-related behaviors. For example, it has recently shown a contribution of DRN DA neurons in encoding the motivational salience of cues and prediction errors [START_REF] Cho | Dorsal raphe dopamine neurons signal motivational salience dependent on internal and external states[END_REF].

RN to VTA sensitivity to high levels of 5-HT during development

In rodents, 5-HT neurons appear on embryonic day 12 (E12) [START_REF] Lauder | Ontogeny of monoamine neurons in the locus coeruleus, Raphe nuclei and substantia nigra of the rat. I. Cell differentiation[END_REF].

5-HT begins to be released by day 13 [E13 (Lambe et al., 2000;Lidov and Molliver, 1982a, b)], reaching adult levels by P15 [START_REF] Hohmann | Transient postnatal elevation of serotonin levels in mouse neocortex[END_REF]. Compared to the serotonergic system, the dopaminergic system develops more slowly. Dopaminergic neurons emerge between E12 and E15 [START_REF] Birnie | Plasticity of the Reward Circuitry After Early-Life Adversity: Mechanisms and Significance[END_REF][START_REF] Olson | Early prenatal ontogeny of central monoamine neurons in the rat: fluorescence histochemical observations[END_REF], but dopaminergic innervation gradually increases until P60 [START_REF] Kalsbeek | Development of the dopaminergic innervation in the prefrontal cortex of the rat[END_REF]. The earlier developing serotonergic system sends strong projections to the dopaminergic system, enabling the serotonergic system to modulate dopaminergic development [START_REF] Suri | Monoamine-Sensitive Developmental Periods Impacting Adult Emotional and Cognitive Behaviors[END_REF]. Given the earlier apparition of the 5-HT system and its strong inputs to the dopaminergic system, the question arises: How does change in 5-HT levels during the perinatal period alter the dopaminergic system?

It is now well documented that abnormal 5-HT levels during perinatal development can have a strong impact on emotional and cognitive functions in adulthood [START_REF] Shah | Abnormal serotonin levels during perinatal development lead to behavioral deficits in adulthood[END_REF]. Interestingly, [START_REF] Yu | Dopamine and serotonin signaling during two sensitive developmental periods differentially impact adult aggressive and affective behaviors in mice[END_REF] artificially increased 5-HT signaling during adolescence by blocking either the 5-HT transporter or monoamine oxidase A. They demonstrated that increased 5-HT signaling during development lead to decreased aggression during adulthood. Moreover, they showed that aggression in the adult was correlated with dopaminergic activity, together suggesting a possible blunting of the dopaminergic activity resulting from postnatal/adolescent exposure to high levels of 5-HT.

Different studies have demonstrated the importance of the DRN to VTA control in adult animals (see section 3.3). Notably, it has been shown that the DRN 5-HT neurons modulate VTA DA activity through glutamatergic co-transmission [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF]. Given the strong impact of abnormal 5-HT levels during development on adult behaviors, a recent study explored the effect of high-levels of 5-HT on this specific pathway [i.e.: RN→VTA; [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF]]. In this study, pups (P2 to P11) received a daily treatment of fluoxetine (a selective 5-HT reuptake inhibitor (SSRI)). The authors found that modulating 5-HT during early life has an effect on the serotonergic to VTA dopaminergic pathway with a decrease of excitatory input through glutamate co-transmission. Importantly, this early life manipulation had long lasting consequences in dopaminergic function and in adult behavior [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF]. The question that is still ongoing is how the effect of early life abnormal 5-HT levels on the RN to VTA pathway can impact reward-related functions in the adult. A recent study suggests that early-life manipulation of the serotonergic system may result in the disruption of reward mechanisms. Indeed, adult mice exposed to a SSRI from postnatal day 2 to 11 presented reduced effort-related motivation in the progressive ratio in operant conditioning, associated with dopaminergic deficits [START_REF] Menezes | Reduced motivation in perinatal fluoxetine treated mice: a hypodopaminergic phenotype[END_REF]. This reduction could be reversed by the administration in adulthood of a DA/norepinephrine reuptake inhibitor [START_REF] Menezes | Reduced motivation in perinatal fluoxetine treated mice: a hypodopaminergic phenotype[END_REF].

Conclusion

Certainty is a concept that is difficult to reach with the serotonergic system and its contribution to different functions, especially reward-related. It seems clear that the serotonergic system is involved in reward processing and recent evidence clarifies its contribution but there are still many steps to reach a conclusion on all the mechanisms possibly involved. Many points of divergence between the studies looking at the involvement of the serotonergic system in reward-related functions can probably be explained by the variety, heterogeneity and expression of 5-HT receptors combined with the specificity of the DRN cellular type activated or/and the use of 5-HT agents directly.

From our point of view, there are some points of convergence of all these studies that differ from the classical view that DRN neurons inhibit the dopaminergic system:

projections from the DRN either coming from non-serotonergic neurons or/and from serotonergic neurons co-transmitting glutamate [START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF][START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF][START_REF] Mcdevitt | Serotonergic versus nonserotonergic dorsal raphe projection neurons: differential participation in reward circuitry[END_REF][START_REF] Wang | Dorsal Raphe Dual Serotonin-Glutamate Neurons Drive Reward by Establishing Excitatory Synapses on VTA Mesoaccumbens Dopamine Neurons[END_REF] can excite VTA DA neurons which in turn release DA in the NAc leading to a reinforcement learning in reward tasks (Figure 1).

Critically, we also described that abnormal levels of 5-HT during development can have lifelong effect on the specific RN to VTA pathway and possibly implication in rewardrelated functions.

More studies are necessary to fully dissect this circuit. Understanding how 5-HT and DA interact in the context of reward is critical to comprehend certain aspects of neuropsychiatric disorders such as deficits in motivation, anergia and apathy [START_REF] Kapur | Serotonin-dopamine interaction and its relevance to schizophrenia[END_REF][START_REF] Salamone | Mesolimbic dopamine and the regulation of motivated behavior[END_REF].

Manipulation

Location of optical fiber

Effects on rewardrelated behaviors

Biomarker effect References

5-HT neurons (Tph2-tTA promoter [START_REF] Gratton | Time course analysis of para-chlorophenylalanine induced suppression of self-stimulation behavior[END_REF][START_REF] Margules | Noradrenergic rather than serotonergic basis of reward in the dorsal tegmentum[END_REF]; SERT promoter [3]; Pet-1 [4]) DRN NOT reinforcing but Enhances patience for future rewards [START_REF] Gratton | Time course analysis of para-chlorophenylalanine induced suppression of self-stimulation behavior[END_REF][START_REF] Margules | Noradrenergic rather than serotonergic basis of reward in the dorsal tegmentum[END_REF][3][4] 5-HT neurons (Pet1 promoter) DRN SS Firing activity during reward mediated by 5-HT and glutamate

VTA

Operant response [8] DRN glutamatergic projection to VTA (Vglut3 promoter)

VTA

Operant response [4,9] 5-HT neurons (SERT promoter Via : 5-HT3R GlutR

(1) (2)

(3)

  A critical point highlighted by those different studies is the importance of the timescale and the possibility of separate DRN circuits and pathways for different reward functions (McDannald, 2015). Specifically, McDevitt et al. (2014) revealed different patterns of projection of DRN 5-HT neurons vs. DRN non-5-HT neurons. In addition, in mice trained to lever press for a sucrose solution, Ren et al. (2018) elegantly demonstrated that 5-HT neurons projecting to the orbitofrontal cortex and 5-HT neurons projecting to the central amygdala have similar responses to reward (activation after the onset of licking), however, their activity differs prior to lever press and they have opposite response to foot shock (inhibition of 5-HT neurons projecting to OFC and activation of 5-HT neurons projecting to central amygdala). Therefore, these results highlight the importance of not only recording identified 5-HT neurons, but also recordings from projection-specific populations (see section 3.3).

  [START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF] demonstrated that photo-stimulation of DRN 5-HT neurons induces operant reinforcement for self-administration and, preference for a place or a lesspreferred drink (with Channelrhodopsin 2 being expressed under the control of the Pet-1 promoter). This evidence points to DRN 5-HT neurons involvement in reinforcing paradigms. This reinforcing effect was confirmed in another study using the tryptophan hydroxylase 2 promoter, DRN 5-HT neuron stimulation induced self-stimulation and conditioned place preference[START_REF] Nagai | The Role of Dorsal Raphe Serotonin Neurons in the Balance between Reward and Aversion[END_REF]. In addition,[START_REF] Cunha | Perinatal interference with the serotonergic system affects VTA function in the adult via glutamate co-transmission[END_REF] also confirmed that stimulation of RN 5-HT neurons can elicit place preference and hyperlocomotion similar to what was obtained when stimulating VTA DA neurons.Importantly,[START_REF] Liu | Dorsal raphe neurons signal reward through 5-HT and glutamate[END_REF] demonstrated that 5-HT neurons also release glutamate.

more uncertain and 4 )

 4 photostimulation alone was not sufficient to impact in any direction the operant response to saccharine. However, the association of low dose of citalopram to DRN 5-HT stimulation significantly reduced the operant response to saccharine[START_REF] Browne | Dorsal raphe serotonin neurons inhibit operant responding for reward via inputs to the ventral tegmental area but not the nucleus accumbens: evidence from studies combining optogenetic stimulation and serotonin reuptake inhibition[END_REF].

Critically, there are different 5 -

 5 HT receptors present in the VTA and their expression depends on the neuronal type (i.e. DA vs non-DA neurons; for extensive reviews, see[START_REF] De Deurwaerdere | Serotonergic modulation of the activity of mesencephalic dopaminergic systems: Therapeutic implications[END_REF][START_REF] Hayes | 5-HT receptors and reward-related behaviour: a review[END_REF]). 5-HT1A, 5-HT2A, 5-HT2C and 5-HT3 are mostly expressed in DA neurons while 5HT1A; 5-HT1B; 5-HT2A; 5HT2C; 5HT4-3; 5HT 6 are expressed on non-DA cells, presumably GABAergic cells in the VTA. Interestingly, the expression of either different receptors in DA and non DA-cells or the expression of 5-HT2A receptors on both DA and non-DA cells offer

DRN 5 -

 5 HT neurons to VTA pathway induced conditioned place aversion. The authors conclude that DRN 5-HT to VTA pathway is thus a key element in the balance between reward and aversion. Interestingly, Cunha et al. (2020) demonstrated that optogenetic stimulation of 5-HT terminals in the VTA results in place preference and hyperlocomotion similar to what was obtained when directly stimulating the VTA-DA neurons.

  also demonstrated DRN glutamatergic input to VTA and the expression of VGlut3. They showed that stimulation of the DRN VGLUT3 to VTA pathway induces preference to the laser associated chamber, AMPA-mediated excitatory currents in VTA DA neurons and release of DA in NAc. Recently, an NMDA component in the DRN to VTA connection has also been proposed. Indeed, it was

  neurons projection to VTA (Pet1 promoter) citalopram (5 or 10 mg/kg i.p.).

Figure 1 :

 1 Figure 1: RN to VTA projections can be serotonergic, glutamatergic or can co-transmit glutamate and 5-HT. Serotonergic release via 5-HT3 receptors, glutamatergic transmission (Glut) and glutamatergic-serotonergic co-transmission have been associated with reward in operant (either by nose poke or lever pressing) and place preference tasks and increased DA release into the nucleus accumbens. 5-HT3R: activation via 5-HT3 receptors, GlutR: activation via glutamatergic receptors. These papers show effects on behavior of activation of the following pathways: (1) DRN-Serotonergic only : Nagai et al., 2020; Wang et al., 2020 ; (2) DRN-Glutamatergicserotonergic co-transmission: Liu et al., 2014; Wang et al., 2020; Cunha et al., 2020; (3) DRN-glutamatergic only: McDevitt et al., 2014; Qi et al., 2014.

2.1.2. Conditioned place preference

	Another behavioral paradigm testing the reward function is CPP. CPP is a behavior
	combining Pavlovian learning and motivation, where an animal is placed in a box with
	two compartments. An unconditioned stimulus (e.g. drug, stimulation/inhibition of a brain
	area) is associated with only one of the compartments. The amount of time spent in
	each compartment is measured to assess preference. The rewarding or aversive value
	of the unconditioned stimulus is then assessed as a function of preference. Different
	authors have tested either the value of 5-HT as an unconditioned stimulus or how 5-HT
	was able to modulate CPP induced by other drugs.

Table 2 . Effect of optostimulation on behavioral expression and neural modulation. Abbreviation: Self

 2 -stimulation: SS; excitatory postsynaptic potential: EPSP; dopamine: DA; Nucleus accubems: NAc; Ventral tegmental area: VTA; Dorsal Raphe Nuclei: DRN; Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience. Nat Commun, 2018. 9(1): p. 2048. 2. Miyazaki, K.W., et al., Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards. Curr Biol, 2014.

		Effect on 5-HT	Drug and target	Electrode location	Effects on	Biomarker effect	References
		transmission		(stimulation or	behavior
				recording)
				ICSS
	ICSS			DRN and MRN	SS		[1-6]
	ICSS +	5-HT	PCPA (400 mg/kg i.p [3, 4] ;400	DRN and MRN	SS		[3, 4, 6]
	Pharmacological		mg/kg intragastric injection [6])
	(inhibitor of
	TPH)
	ICSS +	5-HT	PCPA (350 mg/kg i.p)	DRN	SS		[7]
	Pharmacological
	(inhibitor of
	TPH)
	ICSS +	5-HT	PCPA (316 mg/kg i.p.)	DRN	SS not altered		[2]
	pharmacological
	(inhibitor of
	TPH)
	Pharmacological	5-HT	Baclofen (0.1-2.5mM	DRN and MRN	SS and Place	Tonic inhibition of	[8, 9]
	(GABA receptor		intracranial injection		preference	the DA dependent
	agonist)		[8])/Muscimol (50 and 100 μM			reward circuitry
			intracranial injection [9])
	ICSS		Downregulation of GluN2C	DRN	Nose poke		[10]
			NMDA subunit (siRNA in
			VTA)
	ICSS +	5-HT	8-OH-DPAT (0.2-5 ug	Lateral hypothalamus	SS		[11]
	Pharmacological		intracranial injection) in MRN	stimulation
	(5HT1A
	receptor agonist)
	ICSS +	5-HT	PCPA (500 mg/kg i.p.)	Medial forebrain and	SS		[12]
	Pharmacological			VTA
	(inhibitor of
	TPH)
	ICSS +	5-HT	PCPA (400 mg/kg intragastric	Lateral hypothalamus	SS		[12-14]
	Pharmacological		injection. [13, 14]; 500 mg/kg
	(inhibitor of		i.p. [12])
	TPH)

24(17): p. 2033-40.

Acknowledgments

This work is supported by the National Institute of Child Health and Human Development (R01 HD095966 and R03 HD094978, CMT) and the Neurodis Foundation.

Conflict of interest statement

The authors declare no competing financial interest. Electrical stimulation of DRN Substantia nigra VTA DA cells 63% initially excited followed by inhibition and 34% inhibition followed [START_REF] Gervais | Dorsal raphe stimulation differentially modulates dopaminergic neurons in the ventral tegmental area and substantia nigra[END_REF] Table 1 Effect of ICSS and pharmacological manipulations on behavior and neural activity. Abbreviation: Intracranial self-stimulation: ICSS; Self-stimulation: SS; intra-peritoneal injection: i.p.; intravenous i.


Reinforcement Learning Enhancements to CQSim

Naunidh Singh and Dustin Favorite Illinois Institute of Technology

INTRODUCTION

Recently, there has been some attention given to reinforcement learning agents in cluster scheduling. [START_REF] Fan | Deep Reinforcement Agent for Scheduling in HPC[END_REF][START_REF] Zhang | Rlscheduler: Learn to schedule hpc batch jobs using deep reinforcement learning[END_REF][START_REF] Fan | DRAS-CQSim: A Reinforcement Learning based Framework for HPC Cluster Scheduling[END_REF] Cluster schedulers are low-level applications in high performance computing systems that decide when and which user jobs are allocated to system resources. Traditionally, cluster schedulers are designed and tuned by hand to ensure some performance outcomes like high system utilization and low job starvation. However, as HPC systems become increasingly complex and handle more diverse workloads, the difficulty of designing and tuning their schedulers has become prohibitive. There is an increasing need for a scheduling system that maintains the speed and simplicity of a traditional cluster scheduler but also provides hands-off optimization. One such solution is a reinforcement learned cluster scheduler. Over the past decade, reinforcement learning (RL) has proved to be the state-of-the-art approach to dynamic decision making problems. Fan et al. present DRAS, a reinforcementlearned agent for cluster scheduling. [START_REF] Fan | Deep Reinforcement Agent for Scheduling in HPC[END_REF] DRAS yields performance gains over traditional scheduling approaches by up to 45% and demonstrates that RL scheduling may in fact be a reasonable answer to this new scheduling dilemma. To substantiate the claims made in the DRAS paper, a full investigation into RL applications for HPC scheduling is in order. In this paper we present RL enhancements to CQSim, a trace-based HPC scheduling simulator. [START_REF] Fan | Application Checkpoint and Power Study on Large Scale Systems[END_REF][START_REF] Yang | Integrating dynamic pricing of electricity into energy aware scheduling for hpc systems[END_REF][START_REF] Fan | Scheduling Beyond CPUs for HPC[END_REF][START_REF] Allcock | Experience and Practice of Batch Scheduling on Leadership Supercomputers at Argonne[END_REF] The goal of these enhancements is to reduce the complexity and cost of implementing a RL agent for HPC scheduling and open up this topic for further analysis. Here we detail the implementations of these enhancements along with some supplemental findings and future work.

The remainder of this paper is organized as follows. We start by introducing background and related work in §2. §3 presents the implementation of enhancement. We conclude the paper in §4 and the future work in §5.

BACKGROUND

This section introduces OpenAI in §2.1 and RLlib §2.2. Then, we introduce CQSim in §2.3.

OpenAI -Gym.Env

OpenAI's gym is a package to create custom reinforcement learning agents. The Gym toolkit is used for developing and comparing Reinforcement Learning algorithms. It is implemented in Python and R and can be used to make the code for training RL models short, simple and easy-to-read. The environment must contain all the necessary functionality to run an agent and allow it to learn. A gym environment will basically be a class with 4 functions. The first function is the initialization function of the class, which will take no additional parameters and initialize a class. It also sets the initial state of the RL model. The second function is the step function, that will take an action variable and will return following variables: the next state, the reward for the current state, a boolean representing whether the current episode of our model is done. The other functions are reset, which resets the state and other variables of the environment to the start state and render, which gives out relevant information about the behavior of our environment so far. Each custom environment must implement the following gym interface:

1 import gym 2 from gym import spaces ...

RLlib

RLlib is a reinforcement learning API. Built upon Ray, a framework for building distributed applications, RLlib seeks to provide scalable learning to reinforcement learning methods through high-level API calls that handle distributing the training workload. RLlib uniquely provides RL routines that otherwise would have to be written by hand in either Tensorflow or PyTorch. We theorized that RLlib could increase the portability of the simulator and also provide avenues to develop distributed learning applications. In practice, RLlib does not increase the simulator's ease of use. Using RLlib requires domain knowledge that adds to the cost of implementation. We would not recommend using RLlib for any application that is not explicitly focused on distributed learning.

CQSim

We compare these scheduling policies through trace-based simulation. Specifically, a trace-based, event-driven scheduling simulator called CQSim is used in our experiments. [START_REF] Fan | Trade-Off Between Prediction Accuracy and Underestimation Rate in Job Runtime Estimates[END_REF][START_REF] Li | The Effect of System Utilization on Application Performance Variability[END_REF][START_REF] Fan | Exploiting Multi-Resource Scheduling for HPC[END_REF][START_REF] Fan | ROME: A Multi-Resource Job Scheduling Framework for Exascale HPC System[END_REF][START_REF] Fan | Hybrid Workload Scheduling on HPC Systems[END_REF] CQSim contains a queue manager and a scheduler that can plug in different scheduling policies. It emulates the actual scheduling environment. A real system takes jobs from user submission, while CQSim takes jobs by reading the job arrival information in the trace. Rather than executing jobs on system, CQSim simulates the execution by advancing the simulation clock according to the job runtime information in the trace. [START_REF] Fan | Job Scheduling in High Performance Computing[END_REF] 3. IMPLEMENTATION

Implementation by refactoring 3.1.1 Overview

This implementation focuses on changing the control flow of CQSim to match the chronology required by the Gym interface. While the primary data structures of CQSim can remain unchanged the simulation logic would be completely refactored. Figure 1 describes the proposed control flow. The primary design goals are:

1. The simulator should enter step upon receiving a new job to be scheduled, and return from step to the runner when requesting a scheduling decision from the agent.

2. All other scheduling events (job submissions and completions) occur in correct sequence between scheduling decisions. 3. Simulation output should be identical between the original and new simulators for a static agent like first come, first served.

Code Structure

The class CqSim Env inherits the Gym.Env interface and implements all the abstract methods. It holds the data structure classes JobTrace and NodeStruc as self.jobs and self.nodes, respectively. Cqsim sim.py is replaced by CQSimEnv. Code from other modules (Backfill, Basic algorithm) is extracted and have methods in CQSimEnv.

Implementation

1. Once CqsimEnv is initialized, it proceeds to either cqsim main.train or cqsim main.infer. (e) Repeat.

3. This control loop continues until the workload trace has been exhausted. 4. Final results are written to file and console, and reward results are printed to screen.

Experimental Results

Table 1 details the results of the validation experiment for the PG trainer in DRAS and in this implementation. All hyperparameters (learning rate, gamma, neural net size and shape, loss) are identical. Both weights were trained on the same 10k jobs from the Theta system at ANL. The expected outcome is that performance results would be nearly identical given identical training parameters. However, the DRAS version is overall more performant. Inconsistencies in performance results indicate that this system has some point of failure in the simulation environment. It is recommended that the simulation is debugged before any further use.

Implementation using multithreading 3.2.1 Overview

This section discusses an alternate approach to couple CqSim Simulator with OpenAi Gym Interface to implement a Custom Gym Environment which can enable training of Reinforcement Learning Models using Libraries such as Tensorflow. The notion behind the implementation is similar to the Producer-Consumer problem where the CqSim behaves as Consumer and Gym.Env.Step provides the Producer. Both CqSim and Gym.Env run on 2 separate threads and interact using shared variables.

Code Structure

The class CqSim Env inherits the Gym.Env interface and implements all the abstract methods. It holds the CqSim.Cqsim sim class object as its variable -self.simulator.

From the original implementation in CQSim Code, the class Cqsim.Cqsim sim is altered in the following ways:

1. Cqsim sim class inherits the classes Thread and Pause.

2. Thread class enables implementation of the object to be run on a parallel thread.

3. The Thread.run method from the parent class is overwritten and executes the class method -Cqsim sim.cqsim sim().

Which is the main function that coordinates the different components of the simulator.

4. Class Pause() is implemented to control the synchronization between the Cqsim Env main thread and Cqsim Env.simulator thread.

Implementation

1. Once Cqsim Env is initialized, it initiates the thread for Cqsim sim.

2. qsim sim thread runs through the simulation steps:

(a) Imports submit event.

(b) Reaches event job().

(c) Initiates start scan().

3. At this point the Cqsim Env thread is notified and Cqsim sim thread gets paused and waits for the notification for resuming from Cqsim Env.step() 4. Cqsim Env thread is initially paused and waits for the notification from Cqsim sim thread.

Once Cqsim

Env is resumed, it takes the current state of Cqsim sim using the shared variable Cqsim Env.simulator.simulato Figure 5: A screenshot of the Rendered Info at a random step while running GymEnv.

6. This state is passed to the Model for making the selection of the job, and the shared variable is updated.

7. At this point the Cqsim Env thread is paused and the simulator thread resumes and reaches to the next event or next state.

Thread Pause

The Pause class helps implement the pausing and resuming of the Cqsim Env (Producer) and Cqsim sim (Consumer) threads and synchronize their communication.

The Pause class makes use of python Conditional Variables. It maintains 2 conditional variables -Pause.prod cv and Pause.cons cv.

In order to implement the Producer-Consumer approach, when one condition variable is paused the other one is resumed and vice versa.

The Pause class also maintains a special variable -intial check (boolean). This variable is used to pause the Cqsim env.get state() method to the point when the Cqsim sim thread is initiated and the first set of jobs is loaded.

Step Function Components

1. Gym State: A separate class has been created to maintain the necessary variables for managing a complete state information required for the RL model. An object of this class is instantiated at each event when a decision is required to be made by the RL model.

State Reward:

In order to compute the reward a for the reward at a particular step, the function 'get reward(action)' is defined as part of the GymState class. The function expects as an argument the action suggested by the RL model and then using the state information calculates the reward.

3. Render Info: An important feat of this implementation is to provide a visual representation of the current state. The render function accesses the current state information and using the graphing library Matplotlib, is able to provide a comprehensible set of plots describing the important parameters which can be visually examined during the run time of the RL Model training. This can help the user understand how effectively the model is being trained and can also analyze the updates on the CQsimulator state, environment and RL model parameters during runtime.

Models

We first seek to validate our new simulator by reimplementing a RL agent used in DRAS. The policy gradient (PG) model is a reinforcement learning agent that outputs the likelihood of selecting each job awaiting execution given the state of the environment. By either sampling from these probabilities (as is done in training) or simply selecting the determined best candidate by the highest probability (as is done in testing) the environment receives a job to begin scheduling and can proceed to the next state. The model is illustrated in Figure 6.

Among the RL models we intend to develop as part of the future works, we proposed an approach to merge neural-nets for the inference stage. Since we aim to tackle the scenario where the input to the neural network is of fundamentally 2 different types -Jobs info and Node info, and contain information at different scales and ranges, a model with two networks that may have different architectures can possibly handle the 2 different data types. This method aligns the layers of the original networks and merges them into a unified model by sharing the representative codes of weights. The shared weights are further re-trained to fine-tune the performance of the merged model. The proposed method effectively produces a compact model. As it preserves the general architectures and leverages the co-used weights of well-trained networks, a substantial training overhead can be reduced to shorten the system development time. [START_REF] Chou | Unifying and merging well-trained deep neural networks for inference stage[END_REF]

CONCLUSION

In our investigation of reinforcement learned cluster schedulers we found many areas that necessitate greater inspection. We hope that our work illuminates areas that require more attention, guides future work, and creates an easier environment for implementing those studies.

FUTURE WORK

The most pressing item for our simulator is beginning further analysis of reinforcement learning agents. Experiments should be designed to test for optimizations in the observation, reward, neural net, and training methodology. The following are steps for evaluating training methods:

• A2C: advantage actor-critic improves upon vanilla PG by integrating a second model into the training routine. The critic provides a reward estimate that can be used for more potent and accurate training.

• PPO: PPO is another actor-critic routine that introduces a novel clipping operation to the loss. The goal of PPO is to reduce the impact of saturated gradients upon the model.

We also propose the following topics should be individually studied:

• Deep learning models: the models presented here and in previous work have much room for optimization.

• Reward: Properly studying the effect of different reward functions on training and performance results could reveal much for use in real world application.

• Observation: Concerning the information observed from the environment and the input to the deep learning models there is much to study in terms of features, their shape, and normalization.

• Multi-resource: The current design schedules one type of resources. As the HPC systems becomes more complicated. For example, burst buffer, IO and SSD are incorporating to the next generation HPC systems. [START_REF] Qiao | Preliminary Interference Study About Job Placement and Routing Algorithms in the Fat-Tree Topology for HPC Applications[END_REF][START_REF] Yu | System-wide Trade-off Modeling of Performance, Power, and Resilience on Petascale Systems[END_REF][START_REF] Qiao | Joint Effects of Application Communication Pattern, Job Placement and Network Routing on Fat-Tree Systems[END_REF] Scheduling multiple resources could potentially improves scheduling performance.

Additionally, we do not believe that our enhancements to CQSim are final. There are further improvements to be made to increase ease of use.

3 4 5 " 7 8

 357 class CustomEnv ( gym . Env ) : "" Custom Environment that follows gym interface """ 6 metadata = { ' render . modes ': [ ' human ' ]} def init ( self , arg1 , arg2 , ...) : 9 super ( CustomEnv , self ) . _init_ () # Define action and observation space # They must be gym . spaces objects # Example when using discrete actions : self . action_space = spaces . Discrete ( N_DISCRETE_ACTIONS ) # Example for using image as input : self . observation_space = spaces . Box ( low =0 , high =255 , shape =( HEIGHT , WIDTH , N_CHANNELS ) , dtype = np . uint8 ) def step ( self , action ) : ... return observation , reward , done , info def reset ( self ) : ... return observation # reward , done , info can 't be included def render ( self , mode = ' human ') : ... def close ( self ) :

Figure 1 :

 1 Figure 1: Proposed protocol for scheduling jobs in Gym environment.

2 .

 2 Training or inference begins with an initial observation from CQSimEnv.reset and then proceeds through the rest of the scheduling scenario by: (a) Providing the observation to the model and getting an output. (b) Providing the actionable output to the environment. (c) Receiving a new observation. (d) If training, the agent will attempt model.train

Figure 2 :

 2 Figure 2: CQSimEnv Class structure.DRAS PG New PG Metric 10k-40k 40k-70k Average 10k-40k 40k-70k Average Max Wait 612609 1527674 1070141.5 988297 3650872 2319584.5 Avg. Wait 22413 53545 37979. 446343 174666 310504.5 Utilization 88.7% 91.8% 90.25% 86.40% 81.41% 83.91% Table 1: Results from validation experiment.

Figure 3 :

 3 Figure 3: Overview of the flow for the multithreading approach.

Figure 4 :

 4 Figure 4: CqSim Env Class structure.

Figure 6 :

 6 Figure 6: PG neural network.

Figure 7 :

 7 Figure 7: An overview of the Merged DNN model approach.

Table 1 :

 1 Results from validation experiment.

	5 988297	3650872 2319584.5