Modelling of Individual Variability in the Pavlovian Conditioned Approach Task

I would like to start this thesis by thanking all the people who have helped me in the past four years and to apologise to those I am about to forget in the following lines. Firstly, I would like to thank all my former teachers who first kindled my love of science, especially Mrs Marsh, Miss Harridge, M. Weiller, M. Bertrand, M me Saux, M me Defosseux, M me Lenoble, and M. Demont. Secondly, I would like to thank my friends and co-workers for providing the perfect environment to accomplish this work with the right balance of fun and studiousness. Special thanks go to the wise elders Erwan, Guillaume, Arthur, Thibaud, Nassim, Carlos, Romain, and Marwen, the supportive peers Léni and Pierre, and best of luck to the next generation Paul, Matthieu, Rémi, Astrid, Maud, Giussepe, and Loïc. Thirdly, I would like to thank my supervisors Benoît and Mehdi who complemented each other extraordinarily well, Benoît with his rigorous analyses, always probing and keeping me on my toes, and Mehdi with his inextinguishable enthusiasm for even the smallest results. They accepted to take me on as a master student in spite of my modest background in neuroscience and computational modelling, and then extended their welcome for an extra three years. After twice failing to obtain a PhD grant, they stuck by me and found an alternative source of funding for which I am extremely grateful. They gave me a great deal of freedom during my years under their supervision, and I am very proud to have been their student. Fourthly, I must thank my parents for providing me with such an excellent education, and for humouring my curious career choices. Finally, none of this would be possible without the support of my dear Julie who has bravely put up with my work hours and occasionally sullen mood.

English Abstract

In this work, I will discuss two main topics concerning learnt behaviour in Rats: firstly meta-learning, i.e. the regulation of learning and decision-making parameters; secondly, inter-individual variability in the strategies used in a simple Pavlovian conditioning experiment. In both cases, I will adopt a computational standpoint using reinforcement learning algorithms to model experimental data while also attending to related dopamine functions in the Rat brain.

If environmental access to food and reproductive opportunities evolves at a relatively stable pace, the learning abilities of an organism should keep track of this evolution and enable appropriate behaviour in response to these changes, but, should the environment change unexpectedly, an additional process of meta-learning might be required to cope with this change. In particular, controlling the learning rate or speed with which state, stimulus or action values are updated in response to discrete environmental feedback, and balancing exploitation of what seems to be the best option with exploration of potentially better ones, could constitute two powerful meta-learning strategies when facing a volatile environment. I will start my investigation of meta-learning by analysing the results of a three-armed bandit task with pharmacological inhibition of dopamine, a neurotransmitter suspected of regulating the exploration-exploitation trade-off by [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF]. After this, I will assess how well different models with meta-learning mechanisms regulating either the learning rate or exploration-exploitation trade-off can explain long-term changes in behaviour during the control sessions of the same three-armed bandit task.

Finally, in a Pavlovian conditioning task in which the appearance of a lever predicts food delivery, it is well known that two kinds of behaviour can appear in a rat population [START_REF] Flagel | Individual differences in the propensity to approach signals vs goals promote different adaptations in the dopamine system of rats[END_REF]. On the one hand, so-called sign-trackers become strongly attracted to the lever which they will approach and nibble, while goal-trackers will prefer to immediately go to the site of reward delivery. In parallel, there are differences in the associated dopamine signals, sign-trackers presenting a classical reward prediction error pattern, i.e. a burst of phasic activity which shifts from the time of reward delivery in the early stages of the task to the time the lever appears in later stages, contrary to goal-trackers whose dopamine signals are mostly stable throughout the task. A model aimed at explaining these behavioural and neurological results was previously proposed by Lesaint et al. (2014a), and I will apply this model to new experimental findings based on a task with different inter-trial interval durations. This will result in adjustments to the previous model and propositions for going forward.

Chapter 1

Introduction

Spurn not the men of Science, They sob beneath your sneers As with their large thermometers They test their burning tears, Because they rend the rock and flower To prove, is this their sin, Nature, the good King's daughter All glorious within

The Coloured Lands, G.K. Chesterton

If the purpose of the brain and nervous system is to produce adaptive behaviour, as stressed in [START_REF] Rosenblueth | Behavior, Purpose and Teleology[END_REF] and [START_REF] Weiskrantz | Experiments on the r.n.s. (real nervous system) and monkey memory[END_REF], then the natural starting point of our investigation into the mechanisms of decision-making should be behavioural [START_REF] Krakauer | Neuroscience Needs Behavior: Correcting a Reductionist Bias[END_REF]. Adaptive behaviour, in other terms behaviour which adapts responses of the organism to environmental stimuli with regards to its internal drives [START_REF] Rosenblueth | Behavior, Purpose and Teleology[END_REF], falls into three different categories according to [START_REF] Tinbergen | The study of instinct[END_REF]: firstly, there is behaviour that has become adaptive over several generations and which is somehow encoded in a species' genetic makeup, in other words innate or instinctive behaviour [START_REF] Lorenz | The Foundations of Ethology[END_REF]; secondly, there is behaviour which has become adaptive over the course of an individual's lifetime, through different processes of learning, either by direct reinforcement of specific behaviours or by building a mental representation of the world which allows anticipation of the outcome of potential actions and the planning of novel strategies [START_REF] Dickinson | Actions and Habits: The Development of Behavioural Autonomy[END_REF]; thirdly, there is insightful behaviour (Bird and Emery, 2009a) in which an animal elaborates a radically novel solution to a problem it is facing. Importantly, these types of behaviour are not parallel, distinct repertoires but are functionally related and more or less overlapping.

Firstly, the goals of learned and insightful behaviour are defined by the instinctive drives of that particular organism: the drives to eat and drink from certain sources of nutrients, to reproduce in a certain way sometimes at unique times of the year, to respond aggressively to certain visual stimuli, etc. For instance, in the different conditioning experiments which I will present in the following chapters, the rats are conditioned by food rewards which answer to the innate need to feed Introduction themselves. This drive is so fundamental to achieving correct conditioning that it is even sharpened by keeping the animals' weight at 90% of its usual value when food is provided ad libitum. Other conditioning experiments use reproductive signals as reinforcers, such as visual presentation of potential partners [START_REF] Hollis | Classical conditioning provides paternity advantage for territorial male blue Gouramis (Trichogaster trichopterus)[END_REF] or access to sexually receptive females [START_REF] Domjan | Learning with arbitrary versus ecological conditioned stimuli: Evidence from sexual conditioning[END_REF]. Furthermore, according to [START_REF] Tinbergen | The study of instinct[END_REF] and [START_REF] Lorenz | The Foundations of Ethology[END_REF], learned behaviour not only relies on innate drives to define its goals but is in itself never anything more than a modification of innate responses, insofar as you can only teach an animal sequences of movements which are already present in its innate repertoire but not radically novel behaviour. In other words, animals can learn when to act but the how is predetermined. There are also fixed boundaries to what a particular species is capable of learning about, something Tinbergen called "localized dispositions to learn". For instance, in its natural environment, a seagull will learn to distinguish its offspring from other chicks in a matter of days, despite the very strong resemblance between seagull chicks from a human perspective, but is totally incapable of recognising its eggs in spite of the very varied colour patterns they present. Another example of such specific limits to learning is given in a laboratory setting by the attempt to condition pigeons to refrain from pecking visual cues predicting a food reward [START_REF] Williams | Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement[END_REF]. Even if pecking the illuminated response key prevented the delivery of reward, the subjects could not refrain from responding in this way to the illumination of the key; this result can be explained by the fact that pecking is an innate and irrepressible response to visual food cues for this species.

Secondly, the relationship between learned and insightful behaviour is also far from trivial. Indeed, the ability to plan ahead and anticipate the result of a strategy or sequence of actions is obviously dependent on some form of generalisation of past experience, and Machine Learning specialists would probably like to argue that insightful behaviour is just a special kind of learning called model-based learning in which the agent creates his own internal representation or model of how the environment works allowing him to anticipate the outcome of actions he hasn't actually tested yet. The classical example of this is the ability to follow a different path towards a reward in a maze after reconfiguration of its layout [START_REF] Tolman | Insight" in rats[END_REF][START_REF] Krausz | Dual credit assignment processes underlie dopamine signals in a complex spatial environment[END_REF]. However, there is an extra ingredient which justifies putting insightful behaviour into its own separate category [START_REF] Tinbergen | The study of instinct[END_REF]). Sticking to the example of navigation in a maze, although thanks to his internal model the animal is capable of mentally testing and evaluating actions it has never actually done, these actions are of the same kind as the actions previously tested in this context: they are all trajectories. Contrary to this, insightful behaviour results from the ability to make entirely novel connections between actions and the environment. Maybe another example will help to defend this position. Bird and Emery (2009b) presented rooks with a problem inspired by Aesop's fable "The Crow and the Pitcher", in which a thirsty crow, unable to quench his thirst by reaching the surface of the water in a jug, raises the water level by dropping stones into the jug (incidentally, the moral of the story, Necessity is the mother of invention, is particularly relevant to the argument that this behaviour is creative). In the modern day study, rooks, which are not known for using tools in the wild, had to raise the water level in a large tube so as to reach a floating worm, a situation which they were highly unlikely to have ever encountered before. All test subjects were successful in solving this task in the expected manner, and preferred large stones to small stones and a tube filled with water rather than sawdust, thus showing a relatively good understanding of the mechanics of the task: the animals were not randomly chucking anything in any tube containing a worm. They were also capable of predicting how many stones would be required from the onset of the task, only pausing to retrieve the worm when the correct number of stones had been dropped. Although the animals had experience of using stones as tools in a previous study (Bird and Emery, 2009a), this was the first time they were required to raise the water level in such a way, which is why it is difficult to label this kind of behaviour as learning. This study was replicated and extended with wild New Caledonian Crows by [START_REF] Jelbert | Using the Aesop ' s Fable Paradigm to Investigate Causal Understanding of Water Displacement by New Caledonian Crows[END_REF].

If we are to believe [START_REF] Tinbergen | The study of instinct[END_REF] and [START_REF] Lorenz | The Foundations of Ethology[END_REF], then learned and insightful behaviour really are the exception rather than the rule in the Animal Kingdom, [START_REF] Lorenz | The Foundations of Ethology[END_REF] going so far as to circumscribe learned behaviour as an exclusive prerogative of higher organisms such as Birds and Mammals, a conclusion the famous nineteenth century entomologist Fabre who dedicated much of his work to demonstrating that Insects are incapable of any kind of "reasoned" behaviour [START_REF] Fabre | Les merveilles de l'instinct chez les insectes[END_REF] would wholeheartedly subscribe to. More recent research in the field of Insect cognition would probably disagree with this judgement as we now know that Wasps are capable of recognising individual members of their colony, that Bumble Bees can learn which flowers to visit by observing conspecifics, and that Honey Bees can be trained to follow abstract rules such as sameness and difference [START_REF] Avarguès-Weber | Visual cognition in social insects[END_REF]. Whatever the case may be concerning Insects, it is widely admitted that learned behaviour is especially prevalent among Mammals and specifically Primates and Rodents, and probably explains in large part the evolutionary success of these species, and this thesis will now focus exclusively on that type of behaviour.

Reinforcement learning, the ability to find the most rewarding actions available to an agent or an organism in his environment [START_REF] Sutton | Reinforcement learning : an introduction[END_REF], is an adaptive process which could itself be adaptable as different environments can be more or less rewarding and volatile [START_REF] Behrens | Learning the value of information in an uncertain world[END_REF]. This adaptability of learning and decision-making processes is called meta-learning and will be the subject of Chap. 4 and 5. In the rodent brain, reinforcement learning is hypothesised to be implemented in the dopaminergic system which innervates different areas of the prefrontal cortex and the striatum where it has the ability of modulating the plasticity of cortico-striatal synapses thereby enabling learning [START_REF] Daw | Value Learning through Reinforcement[END_REF]. However, it has also been suggested that dopamine could control the expression of learning by setting the level of stochasticity in the action selection process, which is why in Chap. 4, I model an experiment of pharmacological inhibition of dopamine signalling. In doing so, I show that this intervention is indeed consistent with the idea that dopamine enhances the tendency to exploit what the agent cur-Introduction rently believes to be the best option. In Chap. 5, I endeavour to explain long-term changes in rat behaviour between experimental sessions through different models of meta-learning in which behavioural parameters are controlled by different potential signals, such as accumulated reward rate or convergence of learning. Finally, in Chap. 6, I tackle the subject of inter-individual differences in the responses to a Pavlovian conditioning experiment in which a lever predicts reward delivery. Before all this however, I will make a brief account of the current state of the art regarding behavioural, computational and neurological aspects of learned behaviour in Chap. 3. Finally, in compliance with the rules of the Sorbonne Université, an extended summary in French will also be included immediately after this introduction. comportementaux qui déterminent la prise de décision; deuxièmement, une variabilité inter-individuelle quant à la stratégie employée dans le cadre d'une expérience de conditionnement pavlovien. Pour chacun de ces deux sujets, j'adopterai un point de vue computationnel ancré dans les techniques de l'apprentissage par renforcement afin de modéliser des données expérimentales, tout en dressant des parallèles avec les fonctions dopaminergiques censées être associées à ces processus.

Si les capacités d'apprentissage d'un être vivant peuvent suffire à assurer sa survie dans un environnement stable, un environnement dans lequel les ressources alimentaires et les occasions de se reproduire varient de façon imprévisible nécessite une capacité supplémentaire de méta-apprentissage permettant d'ajuster ses paramètres d'apprentissage et de prise de décision à ces changements inattendus. La régulation de la vitesse avec laquelle l'estimation subjective de la valeur d'une action, d'un stimulus ou d'un événement est mise à jour, et celle de l'équilibre entre exploitation de ce qui semble être la meilleure option et exploration d'options alternatives qui pourraient s'avérer meilleures, constituent deux levier d'action particulièrement aptes à répondre aux défis posés par un environnement instable. Je commencerai mon investigation par l'analyse d'une tâche d'apprentissage sur un bandit manchot à trois bras avec une inhibition pharmacologique de la dopamine et démontrerai que les résultats sont en accord avec l'hypothèse avancée par [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF] selon laquelle ce neurotransmetteur régulerait le taux d'exploration. Dans un second temps, j'évaluerai la capacité de différents modèles originaux de méta-apprentissage soit du taux d'exploration soit de la vitesse d'apprentissage pour rendre compte des changements comportementaux observés sur le long terme entre sessions expérimentales.

Enfin, dans le cadre d'un conditionnement classique ou pavlovien au cours duquel le stimulus conditionnel est l'apparition d'un levier prédisant une récompense alimentaire, il a été établi que deux types de réponses peuvent émerger au sein d'une cohorte de rats [START_REF] Flagel | Individual differences in the propensity to approach signals vs goals promote different adaptations in the dopamine system of rats[END_REF]. Une première catégorie d'individus, appelés "sign-trackers" en anglais, vont préférentiellement se diriger vers le stimulus conditionnel et interagir avec lui, en le mordillant par exemple, tandis que la seconde catégorie, constituée de "goal-trackers", se rend directement à l'emplacement où la récompense sera livrée. En parallèle, ces deux catégories d'individus présentent une disparité des signaux dopaminergiques produits: chez les sign-trackers, l'activité dopaminergique suit le profil classique d'une erreur de prédiction, c'est-àdire une forte activation au moment de la réception de la récompense en début de tâche qui s'atténue progressivement pour se reporter au moment de l'apparition du levier, tandis que chez les goal-trackers, les signaux dopaminergiques sont stables au cours de la tâche. Un modèle expliquant ces différences comportementales et neurologiques a déjà été proposé par Lesaint et al. (2014a), et au cours de cette thèse, j'évaluerai sa capacité à rendre compte de nouvelles données expérimentales obtenues sur une tâche avec des intervalles entre chaque essai de durées différentes et proposerai des ajustements du modèle en conséquence. Des projections corticales et dopaminergiques convergent sur les neurones épineux moyens du striatum. La dopamine est responsable de la régulation de la plasticité des connexions entre cellules corticales et striatales. Tiré de [START_REF] Hyman | Addiction and the brain: The neurobiology of compulsion and its persistence[END_REF].

Contrôle dopaminergique du taux d'exploration 2.2.1 Contexte

La dopamine est un neuromodulateur issu principalement de noyaux mésencéphaliques, la substantia nigra pars compacta (SNc) et l'aire tegmentale ventrale (VTA), qui est notamment libéré dans le cortex préfrontal où elle orienterait les comportements d'exploration dits dirigés, et dans le striatum où elle modulerait la plasticité des connexions cortico-striatales et l'excitabilité des neurones épineux moyens (figure 2.1). Le striatum constitue la porte d'entrée des ganglions de la base qui sous-tendent les mécanismes d'apprentissage et de prise de décision et il est généralement admis que la modulation des connexions cortico-striatales correspond à l'apprentissage de la valeur des différentes actions. Par ailleurs, [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF] ont proposé que la modulation de l'excitabilité des neurones épineux moyens servait quant à elle à réguler le taux d'exploration, c'est-à-dire la tendance plus ou moins forte à sélectionner une action qui ne paraît pas la meilleure. Il s'agit dans ce cas d'un mécanisme d'exploration aléatoire dans lequel la sélection de l'action se fait de façon plus ou moins bruitée sans tenir compte de facteurs additionnels tels que l'incertitude liée au résultat des différentes actions. Afin de tester cette hypothèse, une équipe d'expérimentateurs bordelais, avec qui nous collaborions, ont effectué une expérience d'apprentissage instrumental chez des rats tout en manipulant pharmacologiquement la signalisation dopaminergique grâce à l'injection intrapéritonéale de différentes doses de flupenthixol, un inhibiteur compétitif de la dopamine (figure 2.2). La tâche consistait à déterminer lequel de trois levier était le plus souvent récompensé, sachant que l'identité de ce bon levier changeait régulièrement, tous les 24 essais. Par ailleurs, il y avait deux niveaux de difficulté en fonction de la probabilité de récompense des différents leviers: dans le niveau facile ou bas risque, le bon levier était récompensé avec une probabilité de 7/8 tandis que les deux leviers restants l'étaient avec une probabilité de 1/16 chacun, alors que dans le niveau difficile ou haut risque, le bon levier n'était récompensé qu'avec une probabilité de 5/8 et les deux autres avec une probabilité de 3/16. Quatre conditions pharmacologiques furent testées: une condition contrôle avec injection d'une solution de NaCl, et trois dosages de flupenthixol, à savoir 0.1, 0.2 et 0.3 mg/kg.

Résumé en français

Résultats expérimentaux

Le comportement des rats fut analysé à travers deux mesures principales moyennées entre les différents blocs de 24 essais: la performance, c'est-à-dire la proportion d'essais corrects, et le win-shift, c'est-à-dire le nombre de fois où, après un essai correct récompensé, l'animal change malgré tout son choix à l'essai suivant. Comme indiqué sur la figure 2.3.a., la performance des animaux progresse au cours des blocs, divisés en sous-blocs de 4 essais, et est affectée par le niveau de difficulté mais aussi par la condition pharmacologique. Plus précisément, l'augmentation de la dose de flupenthixol entraîne une détérioration de la performance principalement marquée en fin de bloc.

Quant au win-shift (figure 2.4.a.), qui constitue une mesure approximative de la tendance à explorer des actions alternatives, il présente lui aussi d'importants effets liés à l'inhibition de la dopamine. En effet, plus la dose de flupenthixol augmente, plus le niveau global de win-shift augmente, indépendamment du sousbloc en question. Cette dernière observation, ainsi que le fait que la pente initiale des courbes de performance n'est pas affectée par le flupenthixol, plaident en faveur d'une augmentation du taux d'exploration indépendamment de tout effet sur l'apprentissage (voir la figure 4.6 et le texte anglais pour plus de détails sur ce point).

Interprétation computationnelle

Afin d'apporter un éclairage plus rigoureux sur ces effets expérimentaux, j'ai modélisé la tâche en question à l'aide d'un algorithme d'apprentissage par renforcement avec mécanisme d'oubli. A chaque nouvel essai produisant une récompense r t de 1 ou 0 (en cas d'échec), une valeur correspondant à l'estimation du taux moyen de récompense produit par l'action en question est mise à jour:

Q(a t ) ← Q(a t ) + α(r t -Q(a t )) (2.1)
avec α, le taux d'apprentissage. Il est important de noter que dans les théories classiques sur le rôle de la dopamine dans l'apprentissage, il est admis que celle-ci signale l'erreur de prédiction permettant la mise à jour des valeurs d'action, c'està-dire r t -Q(a t ) dans l'équation précédente. Par ailleurs, les valeurs des autres actions sont oubliées de la façon suivante:

Q(a i = a t ) ← (1 -α 2 )Q(a i ) (2.2)
avec α 2 , le taux d'oubli. Une fois qu'il dispose de ces valeurs d'action, le modèle est en mesure de prendre une décision à l'essai suivant en tirant aléatoirement dans une distribution softmax définie de la façon suivante:

P (a t+1 = a i ) = e βQ(a i ) 3 j=1 e βQ(a j )
(2.3) avec β la température inverse qui détermine le niveau de bruit dans le processus de sélection de l'action. Lorsque ce paramètre est élevé, les différences entre les valeurs d'action sont exacerbées en sorte que le modèle choisira plus volontiers l'action avec la plus haute valeur, tandis que si ce paramètre s'approche de 0, la sélection de l'action deviendra de plus en plus équiprobable et indépendante des valeurs apprises. Ces probabilités permettent de définir la vraisemblance de l'ensemble des choix effectués par un animal au cours d'une session expérimentale, son historique H, en fonction de l'ensemble des paramètres θ:

P (H|θ) = T t=1 P (a t |θ) = T t=1
e βQ(at) j e βQ(a j )

(2.4) C'est en maximisant cette vraisemblance que j'ai pu optimiser les paramètres de chaque individu pour les différentes doses de flupenthixol. Une fois optimisé, j'ai simulé ce modèle afin d'en extraire des nouvelles courbes de performance et de win-shift que j'ai pu comparer avec les données expérimentales. Comme on peut le voir sur les figures 2.3.b. et 2.4.b., les simulations ont un comportement très proche de celui des données expérimentales, validant ainsi ce choix de modèle.

Il m'était alors permis d'utiliser ce modèle pour essayer de déterminer la cause du changement de comportement. Pour ce faire, je me suis tourné vers les valeurs des paramètres optimisés séparément pour chaque dose (figure 2 alors une diminution significative de la température inverse, tandis que les taux d'apprentissage et d'oubli restent constants. Ceci signifie que l'inhibition de la dopamine produit bien une augmentation du taux d'exploration aléatoire sans perturber la vitesse d'apprentissage.

Évolution du comportement sur le long terme et méta-apprentissage

Contexte

Ayant démontré que l'inhibition de la dopamine produit une augmentation du taux d'exploration aléatoire, la question qui se pose naturellement à nous est de savoir si ce résultat est un simple artefact expérimental ou si cette relation entre dopamine et exploration est bel et bien employé par le cerveau du Rat. Un tel mécanisme constituerait un cas de méta-apprentissage, c'est-à-dire de régulation d'un paramètre comportemental. Par ailleurs, une éventuelle faculté de méta-apprentissage pourrait également s'appliquer à la vitesse d'apprentissage, en augmentant celle-ci dans un environnement instable et en le diminuant dans un environnement plus statique, ce qui a d'ailleurs était découvert chez l'être humain par [START_REF] Behrens | Learning the value of information in an uncertain world[END_REF]. Pour dépister un éventuel méta-apprentissage, je me suis intéressé aux sessions expérimentales précédant les sessions pharmacologiques de l'expérience précédente. En effet, ces sessions, au nombre de 24, permettent de voir un changement sur le longterme du comportement susceptible de s'expliquer en terme de méta-apprentissage.

Amélioration sur le long-terme du comportement

L'expérience est la même que précédemment (figure 2.2) excepté l'injection de flupenthixol ou de solution saline qui n'a pas lieu. Si l'on observe l'évolution sur

Évolution du comportement sur le long terme et méta-apprentissage 13

Sessions le long-terme de la performance, en la découpant en quatre périodes de six sessions successives, on observe une augmentation significative du taux de performance (figure 2.6). Contrairement à ce qui a pu être observé dans le cas de l'inhibition dopaminergique, cette amélioration s'effectue non seulement par une augmentation du taux final de performance mais aussi par un redressement progressif des courbes, laissant à penser que nous sommes en présence d'une augmentation de la vitesse d'apprentissage. Cette amélioration de la performance est accompagnée par ailleurs d'une modification progressive du win-shift qui diminue au cours de l'expérience (figure 2.6).

Évolution des paramètres d'un modèle simple

Afin d'avoir une meilleure idée des potentiels mécanismes en jeu, j'ai adopté une stratégie similaire à celle utilisée pour analyser l'inhibition de la dopamine. Plus précisément, j'ai optimisé le même modèle d'apprentissage par renforcement avec oubli sur chaque période de six sessions, mais en ne permettant qu'à un seul paramètre, soit α soit β de varier, puis j'ai regardé comment le paramètre en question évoluait de période en période (figure 2.7).

Dans le cas du modèle où seul α est autorisé à varier d'une période à l'autre, on observe une augmentation statistiquement significative des valeurs de ce paramètre suggérant que les rats mettent à jour leurs valeurs d'action de plus en plus rapidement. De même, lorsque β est le seul paramètre variable, on observe une augmentation statistiquement significative ce qui voudrait dire que les rats adoptent un comportement de moins en moins exploratoire au cours du déroulement de l'expérience. Ce résultat est néanmoins à nuancer car si les trois paramètres sont réoptimisés à chaque période, on observe alors une augmentation des taux d'apprentissage et d'oubli, mais une diminution de la température inverse.

Proposition de modèles mécanistes

Ayant démontré que le changement du comportement des animaux admettait une explication en terme de variation des paramètres, j'ai ensuite cherché différentes règles pour réguler ces variations. Je vais retenir ici une seule de ces règles, à savoir la régulation soit de α, soit de β par un taux moyen de récompense η t qui indiquerait à l'animal à quel point sa performance s'améliore:

η t+1 = η t + α η (r t -η t ) (2.5)
avec α η la vitesse d'intégration des récompenses reçues. Comme on peut le remarquer, ce taux moyen est analogue à une valeur d'action, sauf qu'au lieu de

Évolution du comportement sur le long terme et méta-apprentissage 15

collecter seulement les récompenses qui suivent une action particulière, cette valeurci collecte toutes les récompenses obtenues. On peut donc considérer cette grandeur comme une estimation de la valeur de la stratégie globale de l'individu, et mon intuition était que quand cette valeur augmenterait, l'individu pourrait soit cristalliser sa stratégie actuelle en augmentant β, soit moduler sa vitesse d'apprentissage probablement en la réduisant pour diminuer sa sensibilité aux essais non récompensés. J'utilise ensuite ce taux de récompense global pour moduler soit β:

β t+1 = β 0 + (β 1 -β 0 )η t (2.6)
soit α:

α t+1 = α 0 + (α 1 -α 0 )η t (2.7)
avec β 0 et α 0 , les valeurs de α et β lorsque la récompense moyenne vaut 0 et α 1 et β 1 les valeurs maximales de ces paramètres. Suite à l'optimisation de ces modèles, je les ai simulés pour en extraire des nouvelles courbes de performance et de win-shift afin de les comparer aux courbes expérimentales. Contrairement à un modèle d'exploration dirigée également testé [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF], ces deux modèles se révèlent capable de répliquer plus ou moins grossièrement les données expérimentales (figures 2.8 et 2.9).

Résumé en français

Évolution du comportement sur le long terme et méta-apprentissage 17

Sessions • une composante FMF (Feature Model-Free) qui apprend différentes valeurs attribuées à divers éléments de l'environnement, notamment le levier, la mangeoire où la récompense est livrée, la nourriture elle-même et l'environnement global. Cette composante va attribuer une valeur plus forte aux éléments les plus fortement associés à la récompense, et repose sur une mise à jour des valeurs par apprentissage par renforcement en utilisant des erreurs de prédiction qui correspondraient aux pics d'activité dopaminergique.

• une composante MB (Model-Based) qui s'attache à représenter les différentes transitions entre les états du monde: lorsque le levier apparaît, que se passerat-il si le rat s'approche du levier? ou bien de la mangeoire? Le but de cette composante est de déterminer le chemin le plus court pour obtenir la récompense.

Ces deux composantes sont ensuite combinées pour déterminer la probabilité des différentes actions possibles au moment de l'apparition du levier (figure 2.11), contrairement à d'autres modèles à plusieurs composantes, tel que celui de [START_REF] Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF] où les deux systèmes sont en compétition pour contrôler le comportement.

Dans le cadre de ce modèle, les individus sign-trackers font davantage usage de la composante FMF qui va favoriser le levier dans la mesure où celui-ci est un prédicteur fiable de la récompense, puisque son apparition est toujours suivie d'une récompense. La mangeoire par contre pourra être visitée même lorsqu'elle est vide pendant la période inter-essai, en sorte que sa valeur est plus faible que celle du levier poussant ainsi ces individus à prêter plus d'attention à ce dernier. Les individus goal-trackers correspondent au contraire à une stratégie davantage dominée par le système model-based qui, comprenant l'enchaînement des différents états du monde, ne considère le levier que comme un indicateur de l'état actuel. Ces animaux cherchent ensuite le moyen le plus rapide d'obtenir la récompense, c'est-à-dire se diriger vers la mangeoire sans perdre de temps.

Une des prédictions de ce modèle est que diminuer l'intervalle de temps séparant chaque essai devrait diminuer les différences entre les valeurs du levier et de la mangeoire dans la composante FMF du modèle et ainsi diminuer le comportement de type sign-tracking au profit de celui du goal-tracking. Je me suis attaché à vérifier computationnellement cette prédiction qui avait été faite sans simulation et à la comparer avec les données expérimentales de [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] qui se sont justement attachés à tester cette hypothèse à l'aide d'une expérience où deux groupes d'individus étaient soumis à cette même tâche mais avec des délais d'attente entre chaque essai de longueur différente .

Effets du délai inter-essais sur le comportement du modèle Lesaint

Afin d'illustrer l'effet du délai séparant chaque essai et d'insister sur le fait que cet effet est uniquement dû à la composante FMF du modèle, j'ai repris les paramétrages proposés dans Lesaint et al. (2014a) et simulé deux types de modèle avec

MB FMF ω (1-ω) T A R Integration Action selection V ( f 3 ) V ( f 2 ) V ( f 1 ) A ( a 1 ) A ( a 2 )
A ( a 3 ) P(a 1 ) P(a 2 ) P(a 3 )

a t V δ + Figure 2
.11: Représentation schématique du modèle de Lesaint et al. (2014a) qui combine deux modules d'apprentissage différents pour déterminer la meilleure action à prendre dans l'état s1 de l'expérience. deux durées de délais différentes: une version du modèle n'incorpore que les valeurs issues de la composante FMF dans le processus de sélection de l'action, tandis que la deuxième version n'utilise que les contributions de la composante MB. J'ai ensuite extrait de ces simulations les probabilités d'approcher soit le levier soit la mangeoire, comme représentées dans la figure 2.12. D'après l'examen de ces figures, il est apparent que seule la composante FMF du modèle serait sensible à un éventuel changement de la durée de l'intervalle entre chaque essai. En effet, augmenter cette durée stimule les comportements de type sign-tracking (figure 2.12.a-b.), tandis qu'un modèle dépourvu de composante FMF reste totalement insensible à la manipulation de ce facteur (figure 2.12.cd.). Cela sous-entendrait que dans une population comportant des individus avec une forte influence d'une composante MB, il resterait un noyau dur de goal-trackers quelle que soit la durée séparant chaque essai, tandis que les individus avec une forte composante FMF auraient un comportement plus flexible qui les permettraient d'être catalogués tantôt comme des sign-trackers tantôt comme des goal-trackers.

Variabilité inter-individuelle dans l'approche

Modélisation des données de Lee et al. (2018)

Afin de modéliser les données expérimentales de [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF], j'ai simulé une population d'individus avec des contributions des différentes composantes qui, tout en étant variables, étaient biaisées en faveur de la composante FMF en raison de la plus grande prévalence documentée de sign-trackers dans une population typique. Suite à cela, j'ai calculé les probabilités d'approche des différents stimuli et également calculé l'indice PCA (Pavlovian Conditioned Approach) qui est une métrique caractérisant les comportements sign-tracking/goal-tracking. J'ai ainsi pu répliquer l'effet de l'augmentation des durées d'intervalle entre essais sur l'ensemble de la population en montrant une augmentation du profil sign-trackers (figure 2.13).

Conclusion

Au cours de ce travail, j'ai démontré que l'inhibition de la dopamine provoquait une augmentation de l'exploration aléatoire, que les changements du comportement sur le long terme admettait une explication en terme de méta-apprentissage sans qu'il soit possible de trancher définitivement ni sur le mécanisme de régulation ni sur l'identité du paramètre régulé, et que le modèle des comportements de sign-tracking et goal-tracking de Lesaint et al. (2014a) était bien sensible aux manipulations de la durée de l'intervalle entre chaque essai, comme observé expérimentalement par [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF]. Chapter 3

Background

The dwarf sees farther than the giant, when he has the giant's shoulders to mount on

The Friend, S. T. Coleridge

Introduction

In this chapter, I will present the main results from previous studies concerning learning and decision-making from three different standpoints: behavioural, computational and neurological.

Behavioural results

This first section is devoted to describing the different kinds of learned behaviour, namely Pavlovian conditioning, operant conditioning and goal-directed behaviour.

Background

Pavlovian conditioning

Origin

Classical or Pavlovian conditioning takes its name from Ivan Pavlov, a Russian scientist, whose experiments on dogs, collected in [START_REF] Pavlov | Conditioned reflexes[END_REF], demonstrated the possibility of shifting an innate response (or unconditioned response, UR) to an unconditioned stimulus (US), to a previously neutral stimulus (the conditioned stimulus, or CS). Initially, Pavlov's interest lay in the physiological aspects of salivation, namely how different environmental stimuli, saltiness, quantity, etc., impacted the production of saliva. In his attempts to study this process, he noticed how sensitive this response was to stimuli caused by the experimenters such as the sound of footsteps or conversations, the movement of their shadows, etc. to such a point that specially isolated installations had to be built to suppress these influences. Intrigued, Pavlov went on to devote much of his work to understanding how such apparently neutral stimuli could cause what he called a signal or conditioned reflex, and tested a wide range of possible stimuli such as the sound of a metronome ticking, a thermal stimulus applied to the skin, and even electric shocks which would normally produce a painful reaction, and came to the conclusion that in order to be effective, a conditioned stimulus must overlap in time with the unconditioned stimulus: a stimulus coming after the unconditioned stimulus would never produce increased salivation. Additionally, Pavlov tested an aversive unconditioned stimulus, an acid solution which also causes salivation, and found that stimuli preceding its administration could become conditioned stimuli. He also showed that there were certain limits to what could be used as a conditioned stimulus. For example, a stimulus already associated to a certain reflex, notably pain, would only come to produce a different reflex if the intensity of the new reflex was greater than the initial one. For instance, a dog could learn to salivate after an electric shock or pricking of the skin but not after an electric current applied near to a bone. Clearly, the defence reflex is too strong to override when bone injuries are possible, but it might be in the case of minor injuries, something which Pavlov found normal for a predator capable of disregarding minor skin injuries in the hunt for food. Pavlov's choice of the word reflex is revealing of how he understood this phenomenon. On the one hand, every organism is born with a set of inborn or unconditioned reflexes which are set off by predetermined unconditioned stimuli, such as a foreign body inserted in the mouth of puppies; these are inborn, or species (because they are shared by all individuals of a given species), or unconditioned reflexes. In the course of their individual lifetimes, organisms will encounter these unconditioned stimuli in association with other stimuli, such as specific tastes or smells or sounds, and these gradually come to produce the same kind of reflexes, which are called signal, acquired, individual, or conditioned reflexes. From a neurological point of view, Pavlov hypothesised that unconditioned reflexes were dependent on hardwired connections between sensory features of unconditioned stimuli and the motor response corresponding to the reflex, while conditioned reflexes were possible thanks to a rewiring of nervous connections between sensory features of the new Figure 3.1: Dependence between the CR and the type of CS employed, adapted from [START_REF] Rescorla | Pavlovian Conditioning It's Not What You Think It Is[END_REF]. The graph on the left shows the number of pecks a pigeon will make in response to either a light or a tone CS followed by food. The light CS causes an increasing number of pecks as training goes on, while the tone does not. However, as shown in the graph on the right, if a coloured light stimulus X or Y (red or green, counterbalanced between individuals) predicts the previously conditioned light or tone respectively, then the number of pecks in response to these coloured cues increases at similar rates to similar levels, proving that the light and tone stimuli are equally reinforcing.

stimuli towards the same motor response.

Pavlovian conditioned approach

Although Pavlov himself noted that salivation was not the only conditioned reflex he could produce, observing that the animals also grew physically restless and excited when food was expected, according to [START_REF] Flagel | Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction[END_REF], it was [START_REF] Zener | The Significance of Behavior Accompanying Conditioned Salivary Secretion for Theories of the Conditioned Response[END_REF] who showed how much individual variation Pavlovian conditioning could produce by training unrestrained dogs. This resulted in certain animals tentatively approaching the origin of the bell sound before retrieving the food while other animals would just glance towards the bell and then fix the food tray in which reward was expected. These two different kinds of behaviour were later termed sign-tracking [START_REF] Hearst | Sign-tracking: The Stimulus-reinforcer Relation and Directed Action[END_REF] and goal-tracking [START_REF] Boakes | Performance on learning to associate a stimulus with positive reinforcement[END_REF] and it will be the focus of the results presented in Chap. 6. In the same vein, the review of [START_REF] Rescorla | Pavlovian Conditioning It's Not What You Think It Is[END_REF] reports on great variability in the form of the CR both between individuals for a same CS, but also for different CS and identical US, thus stressing the importance of the sensory characteristics of the predictive stimulus.

As stressed by [START_REF] Rescorla | Pavlovian Conditioning It's Not What You Think It Is[END_REF], the simple picture of a CS gradually eliciting a

Background

CR in anticipation of a US conceals a great variety of phenomena. For instance, the form of the CR may be different for a same US, depending on the nature of the CS.

A localized light predicting the arrival of food to pigeon subjects can cause pecking movements at the location of the signal, while an auditory signal predicting the same event will cause an increase in general arousal and activity but no pecking (see Fig.

3.1).

Conversely, a study by [START_REF] Jenkins | The form of the auto-shaped response with food or water reinforcers[END_REF] demonstrated that the CR could be dependent on the US, by observing that pigeons conditioned to expect either food or water after the appearance of a lighted key button, would respectively either peck it the way they would food, or respond to it with distinct drinking-like pecks. This procedure of allowing the animal to develop his own response to a CS is called auto-shaping or Pavlovian Conditioned Approach (PCA) and it is this procedure which allowed the discovery of sign-and goal-tracking behaviours.

The typical present-day experiment, as described in [START_REF] Flagel | Individual differences in the propensity to approach signals vs goals promote different adaptations in the dopamine system of rats[END_REF], consists in the presentation of an illuminated lever or press-button for a period of eight seconds, followed by the delivery of a food pellet in a different location as the lever is retracted. This procedure allows for variation in terms of number of CS, type of CS (e.g. [START_REF] Meyer | The form of a conditioned stimulus can influence the degree to which it acquires incentive motivational properties[END_REF] used both levers and auditory tones as cues), and type of US. The recorded data usually consists in the number of times the animal touched the lever, entered the magazine, as well as the latencies for both of these actions. For instance, in Fig. 3.2, reproduced from [START_REF] Flagel | Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction[END_REF], the evolution between training blocks of these three different measurements with regards to either the lever or the food cup is depicted. We see that some individuals labelled as sign-trackers increasingly approach the lever, touch it and do so faster from block to block, while another group labelled as goal-trackers do not change their behaviour towards the lever and show little interest for it. By contrast, this same group of animals show increased interest for the food cup, while that of the sign-trackers decreases. Interestingly, these opposite behaviours are acquired at similar rates. These measurements are then usually collapsed into a PCA index first defined by [START_REF] Meyer | Quantifying individual variation in the propensity to attribute incentive salience to reward cues[END_REF] which averages probability of approach to the lever or the food dispenser, number of interactions with either of these stimuli, and normalized latencies between lever appearance and the first interaction with either stimuli. This PCA index ranges between -1 (exclusively goal-tracking behaviour) and 1 (exclusively sign-tracking behaviour). Based on this quantitative score, the animal population can be split into groups of goal-trackers, sign-trackers and occasionally intermediates, either by a ranking method (one third of the population with the highest PCA scores are labelled as sign-trackers, one third with the lowest PCA scores as goal-trackers, and the rest as intermediates, this is the strategy of [START_REF] Flagel | Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction[END_REF]) (see Fig. 3.2) or by setting arbitrary boundaries in the range of possible PCA scores (for instance, classifying all animals with a score higher than 0.5 as sign-trackers as done by [START_REF] Meyer | Quantifying individual variation in the propensity to attribute incentive salience to reward cues[END_REF]). This diversity in responses to a CS has been explained as resulting from two different learning strategies which different individuals may favour for purely idiosyncratic reasons (for instance, they are equated with genetic variability in [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF]). However, contrary to this theory, the study of [START_REF] Patitucci | The origins of individual differences in how learning is expressed in rats: A general-process perspective[END_REF] showed that animals which sign-track or goal-track for a particular cue and reinforcer combination do not necessarily keep this same strategy for a different cue-reinforcer combination. They instead suggest that the difference between goal-and sign-tracking correspond to individual differences in the value attributed to the reinforcer, more precisely, that goal-tracking is used when the reinforcer is particularly valued by the animal.

The study of these individual differences in PCA behaviour is particularly crucial because of the links which have been established between these behavioural phenotypes and susceptibility to drug addiction. Indeed, addicts are known for their sensitivity to cues previously associated to drug-taking, such as particular people or places. When he encounters these cues, an addict, even earnestly determined to giving up drugs, may suddenly feel a craving for the drug, meaning that the cue is not merely predictive of future drug use but actually induces the desire for drugs [START_REF] Flagel | Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction[END_REF]. This phenomenon is similar to what seems to happen to sign-trackers with regards to the CS according to [START_REF] Saunders | Individual variation in resisting temptation: Implications for addiction[END_REF] insofar as the CS is no longer just a predictive cue but has seemingly acquired some motivating qualities of the US: it is attractive as shown by PCA, desirable as it can serve as secondary reinforcers, and can produce a general increase in motivation, an effect known as Pavlovian-instrumental transfer (PIT, see [START_REF] Everitt | Neural systems of reinforcement for drug addiction: from actions to habits to compulsion[END_REF]). In the technical jargon of the field, the CS has acquired incentive salience, something I will come back to more extensively later. Furthermore, [START_REF] Flagel | An Animal Model of Genetic Vulnerability to Behavioral Disinhibition and Responsiveness to Reward-Related Cues : Implications for Addiction[END_REF] were able to selectively breed rats with a tendency to either goal-track or sign-track proving that the tendency to attribute more or less incentive salience to Pavlovian cues is potentially a genetic trait, which would suggest that different susceptibilities to drug addiction are themselves partly genetic.

Operant conditioning

While Pavlovian conditioning concerns the association between two stimuli or events, operant or instrumental conditioning deals with the association between a stimulus or situation and the appropriate behavioural response of the animal. For this reason, it is sometimes referred to as stimulus-response (S-R) learning, a framework which, as explained later, has also been used to explain Pavlovian conditioning. This highlights how these two types of conditioning can overlap and why there is an ongoing debate about whether Pavlovian conditioning isn't just a form of operant conditioning [START_REF] Daw | Multiple Systems for Value Learning[END_REF]. Research in this field is in fact anterior to Pavlov and goes back to the American psychologist Edward Thorndike at the end of the nineteenth century. Thorndike's main interest was whether animals could behave by reasoning and planning, as humans seemingly do. His most famous experiments consisted in putting cats, dogs, or chicks into cages or 'puzzle-boxes' which could be opened from the inside thanks to different relatively elaborate procedures [START_REF] Thorndike | Animal intelligence: An Experimental Study of the Associate Processes in Animals[END_REF]. Using this setup, he found that, when first introduced in the cage, a cat would usually chance upon the opening mechanism by clawing desperately all around the cage, and then in subsequent tests gradually refine its attempts to escape by acting less and less randomly and focusing on the mechanism. This observation led him to his famous Law of Effect which states that if a response in presence of a given stimulus (e.g. the cage environment or the latch mechanism) leads to 'satisfaction', then the link between this stimulus and the response would be strengthened making the response more likely to occur again in the same circumstances [START_REF] Daw | Multiple Systems for Value Learning[END_REF]. Vice versa, a response which produced 'discomfort' would be weakened.

Thorndike's ideas were picked up and developed by two other prominent American psychologists, J.B. Watson and B.F. Skinner usually considered the main founders and proponents of behaviourism, an extremely influential school of thought in psychology. In a seminal article [START_REF] Watson | Psychology as the behaviorist views it[END_REF], Watson stated that the goal of psychology should be to explain animal and human behaviour without reference to internal conscious states but rather in terms of the responses different stimuli elicit and rejects any recourse to introspection as a valid investigatory method. As for Skinner, in his 1938 book The Behavior of Organisms: An Experimental Analysis [START_REF] Skinner | The Behavior of Organisms: An Experimental Analysis[END_REF], he refined Thorndike's Law of Effect by shedding it of any subjective element such as 'satisfaction' and 'discomfort' in favour of a more operational vocabulary : for example, a reinforcer becomes a stimulus which can admit and strengthen an S-R association. Skinner is also credited with considerably expanding the boundaries of operant conditioning, testing positive and negative reinforcers, using different rewarding schedules, and studying the effect of these factors on both the acquisition of behaviour and its extinction across different species, and achieved extremely impressive feats with these procedures such as successfully training a couple of pigeons to play table tennis [START_REF] Skinner | Two "synthetic social relations[END_REF]. Despite its many achievements, the behaviourist school of thought is based on the rather limiting hypothesis that ultimately, behaviour is nothing else than the product of a more or less complex set of reflexes, meaning that it is entirely determined by external stimuli. These reflexes are either innate or learned, as is the case for stimulus-response behaviour, but they leave no room for planning and goal-directed behaviour which I will now present.

Goal-directed behaviour

Once firmly established, a stimulus-response behaviour adapts slowly to either an environmental change, such as when a previously rewarded stimulus ceases to be so, or a change in desirable outcomes, such as when an animal is no longer hungry. An everyday human example of this inflexibility occurs when driving repeatedly along a particular route, from home to work for instance, until it becomes so automatic that, when a change in itinerary is required to pick up the groceries, the driver forgets to do so. For this reason, this type of behaviour is sometimes qualified as 'habitual' [START_REF] Dickinson | Actions and Habits: The Development of Behavioural Autonomy[END_REF] and is particularly adapted to describing such pathological behaviour as drug addiction, a situation in which the mere appearance of a cue associated to drug use, such as a particular place or paraphernalia, will induce a sudden craving for drug, despite the negative consequences the drug user may have previously experienced [START_REF] Everitt | Neural systems of reinforcement for drug addiction: from actions to habits to compulsion[END_REF].

Background

However, although maladaptive cases do persist, animals, ourselves included, can frequently override a habit and flexibly adjust their behaviour to a change in goal, meaning that other mechanisms controlling behaviour must also exist. The existence of such mechanisms -which produce actions in the hope of a certain result and are based on so-called action-outcome (A-O) relations [START_REF] Dickinson | Motivational Control of Goal-directed Action[END_REF] -was demonstrated by two findings in a landmark article by [START_REF] Tolman | Cognitive maps in rats and men[END_REF]. Firstly, rats which had been allowed to explore a maze without any reward, would demonstrate "one-shot" learning by going straight back to the location where food was provided for the first time, without the usually slow acquisition process of operant conditioning. Secondly, the animals would look for an alternative route towards the same location if the maze were changed, despite these new paths never having been reinforced either. Tolman called this effect 'latent learning' as it was only visible once a reward was at stake and introduced the concept of cognitive map to explain it. A more remarkable experiment was published in [START_REF] Tolman | Insight" in rats[END_REF] which I was unable to retrieve but thankfully, it is reviewed by [START_REF] Anthony | The Tolman and Honzik Insight Situation[END_REF]. In this experiment (see Fig. 3.3), rat subjects were trained to seek food in a maze with three different paths leading to the food from the starting point: a short direct route (path 1), a medium route which merged with an important section of path 1 (path 2), and a long route (path 3). As a result of these different lengths, the animals who were in a hurry to get to the goal, should have learnt to prefer path 1 to path 2 and path 2 to path 3. After sufficient training and exploration of the maze, the experimenter would place an obstacle blocking the section of the maze common to both paths 1 and 2 (location B in Fig. 3.3). After running down path 1 and finding it blocked, the animals would immediately go down path 3 which went round the obstacle instead of path 2, despite path 2 having been more strongly reinforced than path 3, proving that the rats generalised their discovery that path 1 was blocked to their knowledge about path 2 and pointing to the existence of a mental map of the maze layout guiding the animals' decisions.

Initially, Tolman's cognitive map was quite literally a map or a representation of the spatial configuration of the maze and reward location, but it came to mean the broader representation of the different contingencies of a task, i.e. the outcomes of different possible actions. Importantly, the outcomes in question are not merely a scalar reward signal as is the case in conditioning, but a richer representation of the world in terms not only of rewards but also of environmental states: e.g. if I move to the right, I will be in this position or if I send this motor command, my hand will do this particular movement.

To discriminate habitual from so-called goal-directed behaviour, one of the simplest approach is a devaluation experiment. For example, in [START_REF] Dickinson | Actions and Habits: The Development of Behavioural Autonomy[END_REF], two groups of rats were trained to press a lever to receive one type of food, the contingent food, either sugar or a mixed diet food pellet, while the other type of food was delivered non-contingently, independently of lever pressing. After training, one group of rats experienced devaluation of the reward associated to the lever by injecting lithium chloride which causes mild feelings of illness after food ingestion, while the other group experienced devaluation of the non-contingent food using the Figure 3.3: Maze used in the experiment by [START_REF] Tolman | Insight" in rats[END_REF], reproduced from [START_REF] Anthony | The Tolman and Honzik Insight Situation[END_REF]. Rats learned to go down three different paths to retrieve a food reward: a short path 1, a medium-length path 2 which merges with path 1, and a long path 3. When blocked in location B, the animals immediately go down path 3 instead of path 2 despite it being less strongly reinforced, proving that they have some kind of mental representation of the environment which can be used to guide behaviour.

Background

same procedure. The two groups of rats were then given the opportunity to lever press under extinction (the lever is no longer rewarded), and the group for which the contingent food had been devalued did indeed press the lever less often than the other group, meaning that in this case, behaviour was goal-directed. However, the same article also tells us about a similar experiment by [START_REF] Adams | Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation[END_REF], which trained one group of rats for a total of 100 lever-presses and a second over-trained group for a total of 500 lever-presses. The result of this procedure was that the over-trained group had become insensitive to reward devaluation, and the conclusion from this study was that rats shifted from a goal-directed behaviour in early training stages to a habitual one as training is pursued. [START_REF] Dickinson | Actions and Habits: The Development of Behavioural Autonomy[END_REF] goes on to propose an interesting explanation for this phenomenon: as training increases, there is little variation in response rate or performance and in reward rate, so that the relationship between response rate and reward rate becomes less strong. In a sense, the rats become less aware of the relationship between their actions and their consequences. This explanation is corroborated by the fact that different kinds of reward schedules will favour either habitual or goal-directed behaviour. As reported in [START_REF] Dickinson | Actions and Habits: The Development of Behavioural Autonomy[END_REF] and [START_REF] Yin | The role of the basal ganglia in habit formation[END_REF], ratio schedules, in which responses result in a reward with a certain fixed probability, tend to produce goal-directed behaviour. This is supposedly because this type of schedule allows for a strong instrumental contingency: the more the rat responds the more rewards it reaps. Contrary to ratio schedules, in interval schedules, a response is rewarded only if a certain amount of time has elapsed since the last reward, meaning that there comes a point where accelerating the rate of response does not increase the rate of rewards. In this case, subjects are known to develop habitual strategies instead. [START_REF] Daw | Model-based influences on humans' choices and striatal prediction errors[END_REF] proposed a more sophisticated task to discriminate habitual from goal-directed behaviour in human subjects which takes advantage of the cognitive map supposedly used in goal-directed behaviour. This task consisted in two sequential stages as depicted in Fig. 3.4 a. in which the human participant had to choose between two different visual stimuli (neutral tibetan symbols in [START_REF] Daw | Model-based influences on humans' choices and striatal prediction errors[END_REF], or pictures of spaceships and aliens in [START_REF] Sharp | Dopamine selectively remediates 'model-based' reward learning: A computational approach[END_REF]). Choosing a stimulus in stage 1 led stochastically to one of two possible stage 2's with their own different pairs of stimuli. For each stimulus in stage 1, there was a common transition (70% of the time) to one possible stage 2 and a rare transition to the other possible stage 2. Choosing a stimulus in stage 2 led to a possible reward which serves as a reinforcer. Now if we consider a reinforced trial in which a rare transition occurred between stages 1 and 2, operant conditioning predicts that the reinforced stimulus in stage 1 is whichever was chosen that particular time, leading to an increased likelihood in repeating that choice. On the other hand, a goal-directed strategy, which is aware of the fact that the transition was a rare one would instead favour the alternative stimulus knowing that it is the most likely to lead to the same stage 2. These differing predictions are shown in Fig. 3.4 b. and can be used as criteria to determine whether participants are following a goal-directed or habitual strategy. For example, [START_REF] Sharp | Dopamine selectively remediates 'model-based' reward learning: A computational approach[END_REF] used this paradigm to show are commonly used to model habitual and goal-directed behaviour respectively. In the case of a habitual system, the probability of 'staying', i.e. repeating the same choice in phase 1 as in the preceding trial, is independent of whether the transition to phase 2 was rare or common. Contrary to this, goal-directed behaviour is sensitive to transition probabilities and prefers to stay after a reward if the transition was common or stay after not being rewarded if the transition was rare.

that Parkinson Disease patients, who present decreased dopaminergic signalling, were more reliant on habitual than goal-directed decision-making. However, the ability of this task to discriminate between habitual and goal-directed behaviour has come under question in [START_REF] Akam | Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task[END_REF].

Computational insights

In the following paragraphs, I will present the most popular theories proposed for the types of behaviour I have just described. Most of these theories are computational in nature, with the exception of the incentive salience theory of Pavlovian conditioning. I will conclude this section by introducing the concept of meta-learning.

The Rescorla-Wagner model

According to the review by [START_REF] Mackintosh | Pavlov and Associationism[END_REF], the first theoretical account of Pavlovian conditioning was given by [START_REF] Konorski | Conditioned reflexes and neuron organization[END_REF] who hypothesised that the CS activated a representation of the US which in turn activated the UR . This stimulus-substitution (Stimulus-Stimulus or S-S) theory is often opposed to the stimulus-response (S-R) theory. In this view, the CS acts as stimulus which elicits a response, the CR, by virtue of its predictive value only. In support of this second theory, it is often noted that the CR is not the same as the UR, which proves that the animal is not completely fooled or blinded to the distinct properties of the CS.

Background

This is the theory adopted in [START_REF] Rescorla | A theory of pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement[END_REF] to propose a mechanism to account for the effect of reinforcement of a compound stimulus AX on the value of the isolated stimulus X, while the value of A is manipulated in different possible ways. In this framework, value is an operative concept which denotes how effective the conditioned stimulus is in producing the CR which is why the authors prefer to use the concept of associative strength, rather than an estimate of future rewards which will be the point of view adopted by reinforcement learning models (see section 3.3.3). It starts by supposing that the associative strength of the compound is simply the sum of the associative strength of each individual stimulus:

V AX = V A + V X (3.1)
If this compound stimulus is followed by a rewarding US, then the values of the individual cues A and X are updated according to the following rule:

∆V A = α A β U S (λ U S -V AX ) (3.2) ∆V X = α X β U S (λ U S -V AX ) (3.3)
with α the different saliences of the stimuli, to reflect the fact that different stimuli may be more or less effective at supporting learning, β U S the salience of the US, since different US may also be more or less effective at inducing learning, and λ U S , the maximum level of responding that the US can produce. Independently of the specific case of compound cues which constituted the main interest of Rescorla and Wagner, the key idea of this early model consists in adjusting a certain value by the difference between this value and the observed outcome after each trial, a process which has since been called the Rescorla-Wagner rule (e.g. in the standard textbook by [START_REF] Dayan | Theoretical neuroscience : computational and mathematical modeling of neural systems[END_REF]). In this way, the Rescorla-Wagner model becomes a direct precursor of reinforcement learning models in which V is explicitly defined as expected reward rate following the appearance of the CS, the product of the CS and US saliences becomes equivalent to a learning rate and λ to the value of the reward. This model was further refined by [START_REF] Pearce | A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli[END_REF] who proposed that learning requires that the subject pays attention to the cue, and that the amount of attention an animal is willing to spend is determined by the predictive accuracy of the CS, anticipating what would later be called meta-learning.

Incentive salience theory of Pavlovian conditioning

This theoretical outlook contrasts with the incentive salience theory of Pavlovian conditioning which insists on the fact that the CS does not merely predict the US but acquires motivational properties attached to the US which may fluctuate through time. In a sense, it is a more elaborate version of the stimulus-substitution theory. More precisely, a stimulus has incentive salience if it is presently desired. According to [START_REF] Berridge | From prediction error to incentive salience: mesolimbic computation of reward motivation[END_REF], incentive salience results from the interaction between a learning process which reveals the contingencies between different events or a. b.

Figure 3.5: a. Illustration of the Partial Reinforcement Acquisition Effect adapted from [START_REF] Gottlieb | Acquisition with partial and continuous reinforcement in pigeon autoshaping[END_REF]. Pigeons develop a key pressing CR more slowly under partial reinforcement than under continuous reinforcement, but ultimately respond more vigorously in later sessions. b. Dead Sea Salt experiment from [START_REF] Robinson | Instant transformation of learned repulsion into motivational "wanting[END_REF]. The value of a lever associated to a mouth injection of a very salty water solution is low until a state of salt appetite is induced pharmacologically. In this case, the animals immediately start approaching the lever which they previously shunned in accordance with the incentive salience theory and contrary to theories of reinforcement which predict a more gradual change in behaviour.

successive stimuli, and a certain motivational state of the animal. For instance, an animal may have perfectly learned that a light predicts the delivery of food, but this knowledge is more or less important if the animal is sated or hungry, meaning that the incentive salience of the light may vary between different experimental settings.

In practice, most Pavlovian conditioning tasks try to maintain the animals in the same motivational state, usually by controlling their weight or making sure there is a constant time delay since the last meal if food is a reward. As a result, these experiments are not well equipped for disentangling learning itself from any potential "wanting" effect, but one such experiment was made by [START_REF] Robinson | Instant transformation of learned repulsion into motivational "wanting[END_REF]. In this experiment, dubbed the "Dead Sea salt experiment" [START_REF] Dayan | Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation[END_REF], rats initially learnt that the brief insertion of a lever in the test cage predicted the administration of a very salty solution of water directly into their mouths. In a normal physiological state, such an unpleasant US is repulsive and rats did indeed display their repulsion by turning away whenever this lever appeared, while they would willingly approach a distinct sugar-predicting lever. However, in a state of salt appetite, induced artificially through drug injection, the animals' repulsion was immediately transformed to attraction when the salt lever was presented in this new state (Fig. 3.5 b.)

Another argument in favour of the incentive salience theory is the previously mentioned fact that for some individuals at least, so-called sign-trackers [START_REF] Anselme | Incentive salience attribution under reward uncertainty: A Pavlovian model[END_REF][START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF], the CS can act as a "motivational magnet", thereby revealing that the CS has acquired motivational value of its own. To prove that this value is not merely predictive, an experiment reviewed in Berridge (2012) used a Background serial CS, a sequence CS1-CS2-US. In this scenario, the CS1, which is not predicted by any other is the most informative, while the CS2 is redundant. Despite this, the CS2 is the one carrying the highest levels of incentive value as shown by the greater eagerness to approach it.

A computational model, derived from reinforcement learning which I will present extensively later, was provided for this perspective by [START_REF] Zhang | A neural computational model of incentive salience[END_REF] and consists in modulating the reward function guiding behaviour by a certain κ factor which represents the current motivational state of the subject. The reward function itself was estimated by a standard reinforcement learning algorithm (see later sections), but this need not be the case according to [START_REF] Berridge | From prediction error to incentive salience: mesolimbic computation of reward motivation[END_REF]. The exact form of modulation by κ can vary; for example, a multiplicative modulation can be used to adjust the incentive value of a food reward given the animal's physiological state. As such, this model is apt to explain how the physiological state of an animal can change the effect of different reinforcers, but it doesn't really address the essential difference between learning and wanting which is supposed to be the core of the incentive salience theory. Indeed, in this setting, the two are implicitly still equivalent, a stimulus is desirable if its value is high with the added sophistication that value is also dependent on the inner physiological state.

A more radical model is proposed by [START_REF] Anselme | Incentive salience attribution under reward uncertainty: A Pavlovian model[END_REF] which also underlines the inadequacy of standard associative models of Pavlovian conditioning in explaining the Partial Reinforcement Acquisition Effect or PRAE (see Fig. 3.5 a.). Briefly, if instead of always being followed by a reward (continuous reinforcement), the CS is only occasionally rewarded (partial reinforcement), the development of the sign-tracking CR, though initially slower, may eventually overtake that of a CS which was continuously rewarded (see [START_REF] Gottlieb | Acquisition with partial and continuous reinforcement in pigeon autoshaping[END_REF]; [START_REF] Anselme | Reward uncertainty enhances incentive salience attribution as sign-tracking[END_REF] for a thorough description of this experimental effect). [START_REF] Anselme | Incentive salience attribution under reward uncertainty: A Pavlovian model[END_REF] explains this by supposing that the CR, or approach to the cue, is motivated by a certain level of wanting W, which is itself the sum of two components: W N or need-induced wanting which corresponds to estimated reward value modulated by the internal state of the animal as proposed by [START_REF] Zhang | A neural computational model of incentive salience[END_REF], and W c contrast-induced wanting which arises from "hoping" to receive an uncertain reward. Fundamentally, however, it is unclear whether a simple process of adding an uncertainty bonus to the learned value of cues will not replicate these effects in a standard reinforcement learning framework. In any case, the most important takeaway for us from this debate surrounding the mechanisms of Pavlovian conditioning is the potentially dual nature of the CS, as a simple predictor, or as a desirable object in its own right, a duality which may reflect a duality in the underlying learning processes.

Reinforcement learning

In this section, I will present the framework of reinforcement learning which is a family of computational solutions for learning and decision-making in order to achieve a goal formalised as maximisation of rewards based on direct interactions between the learning agent and the environment. As illustrated in Fig. 3.6 a., the reinforcement learning problem is based on the interaction between an agent and his environment. Time is usually treated discreetly and will be throughout this work, and in each time step, the agent receives two kinds of inputs from the environment, the current states and potentially a numerical reward (which can be negative), and in turn produces actions which lead to a new state and potential reward, either deterministically or stochastically. Any method that aims to solve this problem can be considered a reinforcement learning method.

Markov decision processes

Markov Decision Processes (MDPs) are a particular family of reinforcement learning tasks which satisfy the Markov property that state signals contain all the relevant information for future action. Practically, this contrasts with situations where events preceding the current instant are still relevant for the next decision. For example, the game of chess satisfies this property: at any point of the game, the configuration of the board is all that matters to decide the next move; it is history independent. When the state and action spaces are finite, these tasks are called finite MDPs and they can be characterised by two functions:

• the transition probability function, P a ss , the probability of being in state s' after executing action a in state s;

• the expected value of the next reward, R a ss .

Together, these functions define the possible interactions between an agent and his environment, and this information can be summarized in a graph form of the type depicted in Fig. 3.6 b. which shows the possible state transitions and rewards given the agent's choices. The issue of how an agent comes to know the state 40 Background representation in the first place goes beyond reinforcement learning and is still the subject of much research [START_REF] Doncieux | Open-Ended Learning: A Conceptual Framework Based on Representational Redescription[END_REF].

In this framework, an agent can start building up a mapping between the different states he visits and his actions which is called a policy and noted π. Typically and throughout the rest of this work, π is probabilistic, hence the expression π(a|s) in the following equations. Given this current policy and a discount factor γ, a state-value function for this policy, V π (s), is defined at instant t as the expected return of discounted future rewards starting from this state and if the agent follows his policy:

V π (s) = E π T -t k=0 γ k r t+k+1 s t = s (3.4)
with T the total number of time-steps in this task. If T is infinite, the task is called continuing, otherwise it is episodic, which will be the case throughout the rest of this work. Additionally, we can define the action-value function which gives the expected return of discounted future rewards after taking action a in state s and then following policy π for the remainder of the interaction:

Q π (s, a) = E π T -t k=0 γ k r t+k+1 s t = s, a t = a (3.5)
If the transition and reward functions of the MDP are available, then we can expand Eq. 3.4 and 3.5 to obtain recursive formulations as definitions of V π and Q π , known as the Bellman equations for V:

V π (s) = E π r t+1 + γ T -(t+1) k=0 γ k r (t+1)+k+1 s t = s = a π(a|s) s P a ss [R a ss + γV π (s )] = a π(a|s)Q π (s, a) (see next equation) (3.6)
and for Q:

Q π (s, a) = E π r t+1 + γ T -(t+1) k=0 γ k r (t+1)+k+1 s t = s, a t = a = s P a ss [R a ss + γV π (s )] (3.7)
Solving an RL task becomes equivalent to finding the policy π * that gives the most rewards over time, in other words finding the policy that leads to V * (s) = max π {V π (s)} and Q * (s, a) = max π {Q π (s, a)}.

Temporal-difference learning (TD learning)

There are a great many different algorithms with different advantages and drawbacks designed to tackle this problem. For instance, dynamic programming methods which, starting with an arbitrary value function and policy, alternate between evaluating V π and improving the policy, are guaranteed to eventually compute the optimal policy, but require a perfect model of the environment; in other words, they need to know P a ss and R a ss beforehand. Contrary to dynamic programming, Monte Carlo methods do not need complete knowledge of the environment but instead, use experience to evaluate the value of a state, by simply averaging the amount of rewards obtained after visiting this state. This can also be done for the action-value function by averaging the amount of rewards following each action in each state. Contrary to dynamic programming which bootstraps the value functions, Monte Carlo methods will evaluate the value of each state and each state-action pair separately without taking advantage of the Bellman equations; e.g. knowledge about V (s t+1 ) is not used when calculating V (s t ).

Temporal-difference learning combines the main advantages of both of these methods: it learns from experience without needing complete knowledge of the environment, similar to Monte Carlo methods, but also makes use of the Bellman equations to bootstrap its estimates of the value functions. An example of a temporal difference algorithm which we will use throughout Chap. 4 and 5 is Q-learning which is used to estimate the optimal Q-values independently of the current policy, which is why it is called off-policy learning:

Q(s t , a t ) ← Q(s t , a t ) + α[r t+1 + γ max a Q(s t+1 , a) -Q(s t , a t )] (3.8)
with α the learning rate between 0 and 1 which determines how fast the Q-values converge to the expected return of Eq. 3.5. Examining Eq. 3.8 tells us that the update of the Q-value is based partly on the observation of the received reward, as it would be in a Monte Carlo method, and on an estimate of the next best Q-value, which is the bootstrapping component of the model. In addition to this update of the chosen action, several papers [START_REF] Barraclough | Prefrontal cortex and decision making in a mixed-strategy game[END_REF][START_REF] Ito | Validation of decision-making models and analysis of decision variables in the rat basal ganglia[END_REF][START_REF] Niv | Reinforcement learning in multidimensional environments relies on attention mechanisms[END_REF] include a forgetting of the value of the other actions, a mechanism I also incorporated in many of my own models:

Q(s t , a i = a t ) ← (1 -α 2 )Q(s t , a i ) (3.9)
with α 2 the forgetting rate. This way, unselected actions tend to regress back to 0, the value they were initialised at. According to [START_REF] Katahira | The relation between reinforcement learning parameters and the influence of reinforcement history on choice behavior[END_REF], this process is in fact responsible for switching or persistent behaviour independently of rewards. More precisely, if the forgetting rate α 2 is smaller than the learning rate α, then the agent has a tendency to switch from the previous action, while if it is greater, there is a tendency to repeat the previous choice, independently of whether it was rewarded or not. To understand this, consider the case of an unrewarded trial: the Q-value of the chosen action decreases proportionally to 1 -α, while the other Background actions decrease proportionally to 1 -α 2 . As a result, when α 2 is greater than α, the decrease in value of the alternative actions is relatively more important that that of the action that was just tested and the agent is more likely to stick with the same choice notwithstanding the negative outcome. On the other hand, if α 2 is smaller than α, then the decrease in Q-value of the chosen action will be even greater compared to the other actions and the subject will be more inclined to shift. Another view is given by [START_REF] Niv | Reinforcement learning in multidimensional environments relies on attention mechanisms[END_REF], who use a different implementation of this mechanism, according to which, the forgetting process could in fact be capturing selective attention by which an agent focuses on what seems to be the most appropriate stimulus.

Learning the optimal Q-values is only the first part of the problem though, and the next stage is deciding what to do with them. A naive answer would be to simply take whichever action currently has the best value. This is exploitation or a greedy strategy and although it makes perfect sense once the Q-values have converged, it would be catastrophic to apply it from the start, and in most natural environments, which change over time preventing the Q-values from ever definitively converging, it would never be viable. Instead, to ensure a good sampling of all possible actions, it is important for the agent to explore options which appear less profitable and the regulation of this exploration-exploitation dilemma will constitute a major part of the next chapters. The most common solution in computational neuroscience, since the study by [START_REF] Daw | Cortical substrates for exploratory decisions in humans[END_REF] established it as the most likely candidate for the way the human brain tackles this issue, is called softmax action selection (more rarely Boltzmann selection as in [START_REF] Doya | Metalearning and neuromodulation[END_REF] and [START_REF] Doya | Modulators of decision making[END_REF]). This method assigns a different probability for each action based on the current estimate of its value and is parameterised by the inverse temperature β which determines how noisy the process will be:

P (a t = a i ) = e βQ(a i ) j e βQ(a j ) (3.10)
If β is high, then the highest Q-value will tend to crush the others and the agent will greatly favour exploitation, while a low β will make action selection more equiprobable and independent from the relative size of the learned values. However, the question of how to decide what precisely is a good value for β is an open one which I will address in Chap. 4 and 5.

Model-based learning and its interactions with model-free learning

We have seen how one of the drawbacks of dynamic programming was the requirement of perfect knowledge of the environmental dynamics. But what if the environmental characteristics, the state transition probability and the reward function, could also be learned? This is the key idea of model-based learning which is one of the main ingredients of the model by Lesaint et al. (2014a) which I will focus on in Chap. 6. Although it has occasionally been used to explain Pavlovian condition-ing such as when [START_REF] Dayan | Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation[END_REF] applied it to the Dead Sea experiment of [START_REF] Robinson | Instant transformation of learned repulsion into motivational "wanting[END_REF] to explain how a state of salt appetite instantaneously transformed the desirability of a salt-predicting cue, model-based learning is most commonly used to explain goal-directed behaviour insofar as goal-directed behaviour is supposed to imply a representation of the different environmental contingencies [START_REF] Dickinson | Motivational Control of Goal-directed Action[END_REF] in the form of a cognitive map [START_REF] Tolman | Cognitive maps in rats and men[END_REF]. One of the most popular algorithm of this kind is Dyna-Q which builds a deterministic model of the world by simply recording the next state and reward after choosing a certain action in a certain state, and uses this model to generate simulated experiences and update the Q-values as though the simulations were real [START_REF] Sutton | Reinforcement learning : an introduction[END_REF]. An example of how to build a non-deterministic model of the world instead will be developed later in Chap. 6. Given the experimental evidence that animals possess both model-free and model-based learning systems, the crucial question is how do these two systems interact? The most common answer to this question is that there is a winner-takesall competition between these two arbitrated by mechanisms which are still being debated. [START_REF] Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF] proposed that the criterion for arbitration was based on the uncertainty of each system, and found that such an arbitration mechanism could explain how behaviour in an operant conditioning task might start out as model-based or goal-directed behaviour before gradually becoming model-free and habitual as training pursued. However, it seems this result is heavily dependent on assuming a certain level of constant uncertainty in the model-based system which the authors attribute to the 'computational noise' of having to explore the different branches of the tree of state transitions.

Another proposal came from [START_REF] Keramati | Speed/accuracy trade-off between the habitual and the goal-directed processes[END_REF] which suggested using a trade-off between the need for accuracy and the extra computation time required by model-based calculations. More precisely, the model-based system, which is assumed to have perfect knowledge of the world, is selected when the gain which can be expected from having a perfect value estimate is superior to the amount of reward lost during the time required to calculate that perfect estimate. In order to have an estimate of this gain without having to actually compute the model-based Q-values, the arbitration mechanism uses the readily available model-free Q-values to compute an expected gain of having perfect information, the Value of Perfect Information or VPI. If the VPI is above the current average reward rate, then it is worthwhile to use the model-based system. This model was also successful in replicating the effect of extensive training on animal behaviour with the shift from goal-directed to habitual behaviour, as well as data from [START_REF] Pessiglione | An effect of dopamine depletion on decision-making: the temporal coupling of deliberation and execution[END_REF], but the requirement that the model-based system have perfect knowledge of state transitions is an important drawback, unless prior latent learning has taken place, which is certainly not the case in many task designs.

A final arbitration mechanism worth mentioning here is developed in [START_REF] Dollé | Path planning versus cue responding: a bio-inspired model of switching between navigation strategies[END_REF][START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF] which deals with choosing what strategy -a model-free stimulusresponse strategy, a model-based strategy using place cells, or a random exploration strategy -to apply in different navigation tasks. The solution they propose is based 44 Background on a gating network which learns how well the different strategies did in the past in similar circumstances, and then cedes entire control to the most successful control system. This mechanism could be viewed as a form of meta-learning because the gating network learns which learning strategy is the most appropriate for this task.

All of these proposed models have in common that they assume that modelbased and model-free systems are competing in the brain, that the organism somehow decides to hand over full control to one or the other, but there is another possibility illustrated in the work of Lesaint et al. (2014a) which is based on cooperation between these two systems. In this particular model of Pavlovian conditioned approach to either a lever CS or the food cup, there are two subsystems:

• a model-free component which attaches different values to the main features of the environment which as a result become attractive and orient decisionmaking towards them;

• a model-based component which gradually builds a representation of the transitions between different events and favours the fastest path to getting a reward.

Instead of choosing either one of these mechanisms exclusively, the model in question combines the outputs of each module into weighted sums which are then used to determine behaviour. There is still an element of competition in so far as the weighted sum represents the relative importance of each system for determining final behaviour, but it is no longer a winner-takes-all paradigm.

Meta-learning

Learning, whether model-free or model-based, is supposed to explain the adaptation of behaviour over time, but isn't it possible that the learning mechanisms themselves adapt? Such a potential mechanism is called meta-learning, and usually consists in the dynamic adjustment of behavioural parameters. For instance, [START_REF] Sutton | Reinforcement learning : an introduction[END_REF] argue that even in a stationary environment with stochastic rewards (stationary meaning that the reward probabilities are constant), Q-values will be unable to converge if the learning rate is constant and will instead fluctuate in response to the most recently acquired rewards. A simple normative solution to this issue is to have a learning rate α t which decreases as a function of time so that more recent rewards impact the Q-values less and less compared to the total sum of previous experience.

Similarly, a simple solution to the exploration-exploitation dilemma is to simply program the exploration rate to decrease, assuming once again that the environment is static and that once the agent has explored every option to a satisfactory degree, there is no need to maintain exploration. An example of this strategy being applied to the temperature of a softmax decision-making process can be found in [START_REF] Sandholm | Multiagent reinforcement learning in the Iterated Prisoner's Dilemma[END_REF] in which the temperature is made to gradually decrease over time, a process called annealing by analogy with the metallurgical technique with the same name consisting in heating metal to a high temperature and then carefully controlling its cooling down. Of course, the major drawback of both these methods is that they are specifically designed for stationary environments, a situation which is rarely if ever met in real life scenarios.

As a result, more elaborate meta-learning methods have been proposed. To start with, there have been several theoretical attempts such as [START_REF] Tokic | Value-difference based exploration: Adaptive control between epsilon-greedy and softmax[END_REF] who applied a Value-Difference Based Exploration or VDBE algorithm regulating the ε of an ε-greedy mechanism given the magnitude of the last TD error to multistate MDPs, and [START_REF] Schweighofer | Meta-learning in reinforcement learning[END_REF] who use a comparison between mid-term and long-term reward rates to decide in what direction to change their parameters, the basic idea being that if the mid-term reward rate is larger than the long-term reward rate, meaning that performance has improved, then it is worth increasing β to avoid unnecessary exploration and decreasing α to stabilise current action values.

An early meta-learning model was proposed by [START_REF] Pearce | A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli[END_REF] who suggested that the "associability", which is more or less equivalent to a learning rate, between a CS and a US should be modulated by the discrepancy between the maximum level of reinforcement caused by the US, λ, and the current level of reinforcement (see Rescorla-Wagner's model, Eq. 3.2). From a reinforcement learning perspective, this means regulating α based on current reward prediction errors, a hypothesis I actually tested in Chap. 5. More recently, [START_REF] Behrens | Learning the value of information in an uncertain world[END_REF] also tackled the question of how to regulate the learning rate and proposed to do so by estimating the volatility of the environment, reasoning that if an environment is volatile, in other words reward rates of different actions are changing fast, then the learning rate had better be large so as to adapt quickly to these changes, and vice versa for an environment with low volatility. This theory was applied to a decision-making task with brain imaging in humans using functional Magnetic Resonance Imaging (fMRI) which found that BOLD signals in the anterior cingulate cortex of human subjects significantly correlated with the model estimate of volatility. Similarly, [START_REF] Jepma | Catecholaminergic Regulation of Learning Rate in a Dynamic Environment[END_REF] argued that the learning rate of human subjects was modulated by prior uncertainty about beliefs and posterior unexpectedness of the last outcome, and found interesting evidence that this control was dependent on catecholamine (dopamine and/or norepinephrine) activity in the cortex. This ties in interestingly with the work by [START_REF] Yu | Uncertainty, neuromodulation, and attention[END_REF] which proposed that two different neurotransmitters, acetylcholine and norepinephrine, were responsible for signalling expected and unexpected uncertainty, and that the balance between these two signals would be useful in determining whether to explore or not.

Concerning the regulation of exploration rate, [START_REF] Khamassi | Role of the frontal cortex in solving the exploration-exploitation trade-off: model-based analysis of single-unit recordings[END_REF][START_REF] Khamassi | Robot cognitive control with a neurophysiologically inspired reinforcement learning model[END_REF] proposed a neural network model, based on different areas of the monkey brain, which had the property of adapting β in response to the reward history. They successfully replicated animal behaviour in two different visual decision-making task and also showed its applicability to a robotic implementation. Another recent line of research into the question has focused on the question of directed exploration, that is exploration which is specifically oriented towards actions which are less Background known [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF][START_REF] Zajkowski | A causal role for right frontopolar cortex in directed, but not random, exploration[END_REF][START_REF] Cogliati Dezza | Learning the value of information and reward over time when solving explorationexploitation problems[END_REF], 2019) as opposed to random exploration. [START_REF] Gershman | Dopaminergic genes are associated with both directed and random exploration[END_REF] found evidence, by comparing the behaviour of human subjects with particular genotypes, that dopamine, an important neurotransmitter I will discuss about later, controlled both types of exploration, thus uncovering a potential mechanism of meta-learning, although his study was focused on inter-subject differences while meta-learning per se is an individual process.

Finally, [START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF] have gone even further by developing the concept of meta-reinforcement learning using deep neural networks, whereby a neural network, meant to represent the prefrontal cortex, is innervated by a global error signal analogous to the hypothetical RPE signals of dopamine which shapes within it a task-specific reinforcement learning agent that eventually becomes independent of dopamine error signals as it generates its own prediction errors.

Neurological parallels

Having seen the different ways an animal can learn and what computational processes might underpin these adaptations, we will now look at neural activity in the brain to try and find neural correlates of these processes. In contrast to a pure behaviourist approach, the idea is that looking into brain processes can help us understand the more precise mechanisms which underlie behavioural adaptation and the reasons why animal behaviour sometimes deviates from the normative behaviour predicted by computational models. In this section, I will review the main brain areas and neuromodulators that are involved in Pavlovian and instrumental conditioning, and how their manipulation has been found to alter behaviour. To start off, it is now well established that cortical connections from the cortex to the basal ganglia play an important role in the regulation of learned behaviour, be it Pavlovian, habitual or goal-directed [START_REF] Yin | The role of the basal ganglia in habit formation[END_REF]. Importantly, the striatum, which is the input nucleus of the basal ganglia is innervated by dopamine, a neuromodulator which affects the plasticity of cortico-striatal synapses [START_REF] Reynolds | A cellular mechanism of reward-related learning[END_REF] and the excitability of striatal neurons [START_REF] Surmeier | D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons[END_REF], which is why dopamine is of particular interest when studying acquisition of behaviour.

The history of dopamine research

The discovery that dopamine (or 3,4-Dihydroxyphenetylamine) is a neurotransmitter is attributed to Arvid Carlsson [START_REF] Yeragani | Arvid Carlsson, and the story of dopamine[END_REF] for which he was rewarded with the 2000 Nobel Prize in Medicine. Carlsson's own interest in dopamine was mainly focused on pathological dysfunctions of the dopaminergic system. Indeed, his initial discovery came about when studying the antagonistic effect of L-Dopa, a chemical precursor of dopamine, on reserpine, a drug used for treating schizophrenia [START_REF] Carlsson | 3,4-Dihydroxyphenylalanine and 5-Hydroxytryptophan as reserpine antagonists[END_REF]. Until then, dopamine was viewed simply as the precursor of noradrenaline/norepinephrine (Fig. 3.7 shows the similarity between these two chemical substances), another catecholamine neurotransmitter, but Carlsson found no sign that norepinephrine concentrations increased after L-Dopa administration, leading him to the conclusion that reserpine was in fact targeting dopamine [START_REF] Carlsson | Nobel prize acceptance speech: A half century of neurotransmitter research: impact on neurology and psychiatry[END_REF]. This led researchers to the "schizophrenia theory of dopamine", according to which this psychological condition was due to abnormally high levels of dopaminergic activity.

Parkinson's disease was also linked very early on to dopamine when the brain of dead patients was found to have abnormally low levels of dopamine [START_REF] Ehringer | Verteilung von Noradrenalin und Dopamine (3-Hydroxytyrammin) im Gehirn des Menschen und Ihr Verhalten bei Erkrankungen des ExtraPyramidalen Systems[END_REF]. We now know that this disease is indeed caused by the death of dopaminergic neurons, and studying the normal functions of dopamine through the careful comparison of Parkinson's disease patients on and off medication is a very common method of investigation (e.g. Dopamine dysfunction is also related to drug abuse and addiction. Addiction is defined by an uncontrollable desire for drugs which is often sparked by drug-related cues, such as people, places, or paraphernalia previously associated with drug-taking experience, even when drugs are no longer desirable and the known cause of negative effects. A common feature of most addictive drugs is that they activate dopamine signalling in a way which has been likened to biologically relevant rewards such as food and sex, but to an even stronger degree [START_REF] Hyman | Addiction and the brain: The neurobiology of compulsion and its persistence[END_REF]. For example, it is established that the stimulating effects of amphetamine can be countered by pimozide which blocks dopamine receptors [START_REF] Beninger | Pimozide Blocks Establishment but not Expression of Amphetamine-Produced Environment-Specific Conditioning[END_REF]. Finally, dopamine stimulating drugs are effective in treating attention deficit hyperactivity disorder (ADHD) suggesting that this disorder also has a dopaminergic origin [START_REF] Iversen | Dopamine: 50 years in perspective[END_REF].

One of the first functional accounts of dopamine, which is still prevalent in popular representations of dopamine, is given by the "anhedonia hypothesis of neuroleptic action" put forward by [START_REF] Wise | Neuroleptics and operant behavior: The anhedonia hypothesis[END_REF]. This theory is based on the observation that neuroleptics, anti-psychotic drugs which target dopamine receptors, cause a gradual decrease in activity to obtain either drug [START_REF] Wise | Neuroleptics and operant behavior: The anhedonia hypothesis[END_REF] or food [START_REF] Wise | Catecholamine theories of reward: A critical review[END_REF] 48 Background [START_REF] Wise | Pimozide attenuates acquisition of leverpressing for food in rats[END_REF]. Importantly, the effect of the neuroleptic pimozide, a dopamine antagonist, was dependent on the fact that animals experienced the food under this drug condition [START_REF] Wise | Catecholamine theories of reward: A critical review[END_REF] as though the response to the food was being extinguished, which led Wise to the conclusion that the effect was due to a decrease in the rewarding value of this specific reinforcer rather than a general deficit in motivation or motor control. Furthermore, subjective human reports of dysphoria during neuroleptic treatment [START_REF] Wise | Dopamine and reward: the anhedonia hypothesis 30 years on[END_REF] convinced Wise that at least one of dopamine's functions is to signal reward and regulate feelings of pleasure. However, the similarity between drug-induced weakening of responding and extinction, a conditioning procedure which consists in weakening a previously learned stimulus-response association by no longer rewarding it, has been challenged, and this reward theory of dopamine is now largely discredited [START_REF] Salamone | Beyond the reward hypothesis: alternative functions of nucleus accumbens dopamine[END_REF].

Dopaminergic pathways and functions

Dopamine pathways originate from two distinct nuclei in the midbrain called the ventral tegmental area (VTA) and the Substantia Nigra pars compacta (SNc) (see Fig. 3.8). There are four major dopaminergic pathways [START_REF] Beaulieu | The Physiology, Signaling, and Pharmacology of Dopamine Receptors[END_REF], which are:

• the mesolimbic pathway between the VTA and the Nuccleus Accumbens in the ventral striatum. This pathway is usually linked with the prediction error function of dopamine, as this region is known to encode rewards.

• the nigrostriatal pathway between the SNc and the dorsal striatum. This pathway is most clearly linked with motor control, but also with some aspects of reward learning which overlap with the mesolimbic pathway. A framework which may explain this apparent reduncancy is provided by special reinforcement learning models called actor-critic models. In this framework, the ventral striatum would be analogous to a critic learning the value of states, while the dorsal striatum would serve as the actor component in charge of devising a policy based on these same RPEs [START_REF] O'doherty | Dissociable Role of Ventral and Dorsal Striatum in Instrumental Conditioning[END_REF]). An alternative explanation is that distinct regions of the striatum support different types of behaviour: the Nucleus Accumbens would be used for S-S Pavlovian associations, the dorsolateral striatum for S-R habitual strategies and the dorsomedial striatum (DMS) for goal-directed behaviour [START_REF] Everitt | Neural systems of reinforcement for drug addiction: from actions to habits to compulsion[END_REF][START_REF] Yin | The role of the basal ganglia in habit formation[END_REF].

• the mesocortical pathway between the VTA and different regions of the prefrontal cortex. This pathway has been linked with the control of the level of exploration-exploitation [START_REF] Ellwood | Tonic or phasic stimulation of dopaminergic projections to prefrontal cortex causes mice to maintain or deviate from previously learned behavioral strategies[END_REF], and with directing exploration towards uncertain cues [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF].

• the tuberoinfundibular pathway between the hypothalamus and the pituitary gland which regulates production of the prolactin hormone and which will not concern us any further. There are two distinct modes of dopamine activity from the VTA and SNc: a slow, single-spike tonic activity at about 5 Hz, which is driven by pacemaker mechanisms intrinsic to dopaminergic neurons [START_REF] Grace | Regulation of firing of dopaminergic neurons and control of goal-directed behaviors[END_REF], and phasic bursts of 2 to 8 spikes within an interval of roughly 80 ms [START_REF] Paladini | Dopamine controls the firing pattern of dopamine neurons via a network feedback mechanism[END_REF] which are dependent on excitatory glutamatergic signals [START_REF] Grace | Regulation of firing of dopaminergic neurons and control of goal-directed behaviors[END_REF]. Moreover, in the striatum, dopamine transmission is probably segregated into distinct compartments: the intrasynaptic environment which is the main target of phasic bursts of activity and in which the concentration of dopamine changes very rapidly, and an extrasynaptic pool of "tonic" dopamine whose concentration changes much more slowly and targets receptors outside of the synaptic buttons, including potential autoreceptors found on the dopaminergic neurons [START_REF] Paladini | Dopamine controls the firing pattern of dopamine neurons via a network feedback mechanism[END_REF].

The RPE hypothesis of phasic dopamine

Focusing on the fast bursts of dopaminergic activity, known as phasic dopamine, the RPE hypothesis states that these bursts of activity signal a reward prediction error similar to the ones used in TD-learning. Initial support for this theory came from [START_REF] Schultz | A neural substrate of prediction and reward[END_REF] in which recorded dopaminergic activity during a simple Pavlovian conditioning experiment in Monkeys was seen to shift from the reward to the predictive cue (see Fig. 3.9). This suggests that initially, the delivery of reward being a surprising positive event caused the increased activity, but that gradually, as the association between US and CS was established, the reward was no longer surprising and the prediction error decreased to 0. Meanwhile, the CS which had gradually taken a strong rewarding value was not predicted by any environmental feature meaning that its appearance was now responsible for an RPE and a phasic burst of dopamine. Furthermore, when the CS-US association was firmly established, the authors would occasionally withhold rewards after CS Once the CS-US association is learnt, the peak of dopamine activity has shifted from the reward to the CS. c. In the case where an expected reward is omitted, there is an interruption of tonic dopamine firing which is consistent with a negative prediction error.

appearance. This produced a period of decreased dopamine firing consistent with a negative reward prediction error. This result was confirmed by [START_REF] Fiorillo | Discrete coding of reward probability and uncertainty by dopamine neurons[END_REF] who also recorded dopaminergic activity consistent with RPE signals in monkeys undergoing a Pavlovian conditioning task. An important study by [START_REF] Bayer | Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal[END_REF] set out to determine the precise relationship between bursts of phasic dopamine and past rewards. They found that dopamine activity, at the time of reward, did indeed encode a reward prediction error between the current reward and a weighted average of past rewards, but only when the reward was better than expected. More precisely, the weighted average of past rewards consists in a sum of past rewards with decreasing weights for each rewards as we go further back in time, which is consistent with a TD-learning rule in which the most recent rewards have more weight in determining current state or action value. As for the inability of the dopamine system to reliably signal negative reward prediction errors, this could be compensated for by an additional serotonin system responsible for learning from punishments and losses [START_REF] Ouden | Dissociable effects of dopamine and serotonin on reversal learning[END_REF].

The RPE hypothesis which was initially introduced in a Pavlovian conditioning framework was successfully extended to operant conditioning notably by [START_REF] Morris | Midbrain dopamine neurons encode decisions for future action[END_REF] who showed that the dopamine response in monkeys undergoing operant conditioning corresponded to the expected reward of the action about to be chosen. This response would be consistent with a type of TD-learning algorithm known as SARSA, which, contrarily to Q-learning (see Eq. 3.8), uses an on-policy method by basing the reward prediction error on the value of the next chosen action a t+1 instead of the best possible action:

Q(s t , a t ) ← Q(s t , a t ) + α[r t+1 + γQ(s t+1 , a t+1 ) -Q(s t , a t )] (3.11)
Surprisingly, the study by [START_REF] Roesch | Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards[END_REF] which used mice as animal subjects recorded dopaminergic signals which were similar to the RPE of a Q-learning algorithm instead (see Eq. 3.8). This was confirmed by a computational study [START_REF] Bellot | Which Temporal Difference Learning algorithm best reproduces dopamine activity in a multi-choice task?[END_REF] which compared the different RPE signals of Q-learning, SARSA and Actor-Critic reinforcement learning algorithms simulated on the same task as [START_REF] Roesch | Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards[END_REF] and which came to the conclusion that the recorded pattern of dopamine signals most closely resembled those predicted by a Q-learning algorithm. This study also pointed at the fact that there remained significant activity at the time of reward as though learning never finished converging. One possible explanation for the discrepancy between the studies of [START_REF] Roesch | Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards[END_REF] and [START_REF] Morris | Midbrain dopamine neurons encode decisions for future action[END_REF], besides inter-species differences, is the recording site of dopaminergic neurons. While the study of [START_REF] Morris | Midbrain dopamine neurons encode decisions for future action[END_REF] recorded dopaminergic neurons in the substantia nigra pars compacta (SNc), [START_REF] Roesch | Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards[END_REF] preferred to record neurons from another nucleus of dopaminergic neurons, the ventral tegmental area (VTA). This suggests that there is a potentially heterogeneous quality of the signal emitted by distinct dopaminergic populations.

A relatively popular counter-theory is the incentive salience theory of phasic dopamine which is the neurological counterpart of the incentive salience theory of Pavlovian conditioning previously discussed, and which asserts that phasic bursts of dopamine serve to indicate desirable stimuli [START_REF] Salamone | Beyond the reward hypothesis: alternative functions of nucleus accumbens dopamine[END_REF]. Evidence in favour of this theory can be found in the fact that, in some experimental tasks, dopamine depletion inhibits performance without affecting learning itself [START_REF] Beninger | Pimozide Blocks Establishment but not Expression of Amphetamine-Produced Environment-Specific Conditioning[END_REF], which is explained by supposing that the animal no longer cares about stimuli whose value can still be learned. Alternatively, it is possible that there are other learning systems in the brain which can compensate for a failure of the dopaminergic system. For example, an intriguing solution could perhaps be found in the recent theoretical paper by [START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF], which proposes that dopamine is necessary to shape a reinforcement learning agent within the prefrontal cortex suited to a particular task at hand which can then operate more or less autonomously. In other words, dopaminergic signals could be necessary to the establishment of learning, but once a learning system dedicated to the task at hand is set up in the prefrontal cortex, learning changes which take place within the task's boundaries (i.e. changes in reward contingencies but not changes in the nature of the task) could become independent of dopamine RPEs. A computational model aiming to reconcile the RPE theory with the incentive salience theory was advanced by [START_REF] Mcclure | A computational substrate for incentive salience[END_REF] in which dopamine bursts of activity serve both to update a state value and to bias the action selection process to signal that the cue is "wanted". However, the incentive salience theory of dopamine, although very compelling when it comes to its accounts of addiction, pointing out that this type of behaviour is inconsistent with simple learning phenomena, does not come without its own deficiencies. It notably fails, as far as I can tell, to explain the gradual decrease in dopaminergic activity at US times and potential dips in activity in case of reward omission. If dopamine really signals "wanted" targets, there does not seem to be any reason for learning to affect how desirable the US is.

Aside from the incentive salience theory, there have been several recent studies challenging the idea that phasic dopamine represents an RPE. Firstly, although it had already been noted by [START_REF] Bayer | Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal[END_REF] that there was constant phasic dopamine activity at the sound of the tone announcing a trial, there is 52 Background an increasingly growing realisation that dopaminergic bursts are not exclusive to rewarding stimuli or cues, as a variety of other events produce activity in different subpopulations of neurons, such as novel stimuli [START_REF] Kakade | Dopamine: Generalization and bonuses[END_REF] or the initiation of movement [START_REF] Syed | Action initiation shapes mesolimbic dopamine encoding of future rewards[END_REF]. [START_REF] Hamid | Mesolimbic dopamine signals the value of work[END_REF] also found an important effect of phasic dopamine on motivation, a function generally attributed to tonic dopamine. Furthermore, as argued very convincingly by [START_REF] Coddington | The timing of action determines reward prediction signals in identified midbrain dopamine neurons[END_REF], it is in fact very difficult to dissociate an RPE signal from what would be a signal of the value of the current state. Work which took place in this very same laboratory [START_REF] Bellot | Modélisation computationnelle du rôle de la dopamine dans les boucles cortico-striatales dans l'apprentissage et la régulation de la sélection de l'action[END_REF], also advanced the possibility that what phasic dopamine really encodes is a mixture between value and RPE, rather than a pure RPE signal.

Tonic dopamine, motivation and exploration

The most common role attributed to tonic dopamine, the slow-changing concentration of dopamine outside of synaptic buttons, is regulation of motivation. Motivation covers numerous aspects of behaviour, namely the delay with which an animal initiates actions, the speed of his actions once initiated, and the rate of response. All these factors seem to increase with the average tonic level of dopamine which, for this reason, is said to "energise" behaviour [START_REF] Niv | Cost, benefit, tonic, phasic: What do response rates tell us about dopamine and motivation?[END_REF]. [START_REF] Ellwood | Tonic or phasic stimulation of dopaminergic projections to prefrontal cortex causes mice to maintain or deviate from previously learned behavioral strategies[END_REF] report that mimicking tonic activity by optogenetic stimulation of the mesocortical pathway increases persistence of learned cue-response associations despite changes in reward contingencies. This result is probably linked to the supposed role of prefrontal dopamine in controlling directed exploration. [START_REF] Panigrahi | Dopamine Is Required for the Neural Representation and Control of Movement Vigor[END_REF] showed that MitoPark mice, a model of Parkinson's disease which presents a gradual extinction of midbrain dopamine neurons, present significant declines in both the number of movements (akinesia) and in the speed of these movements (bradykinesia) despite retaining the ability of adjusting movement speed if required to obtain a water reward in a self-paced task. Thanks to localized recordings, as well as optogenetic and pharmacological manipulations, the authors were able to pinpoint this effect on vigour to the striatum. An interesting theory was put forward by [START_REF] Niv | Tonic dopamine: Opportunity costs and the control of response vigor[END_REF], according to which tonic dopamine represents an average of past phasic bursts of dopamine, meaning that it would correspond, according to the RPE theory of dopamine, to an average reward rate which could explain why increased tonic dopamine should lead to increased motivation. I took inspiration from this view of tonic dopamine when designing a meta-learning model in which the average rate of RPEs is used to control the inverse temperature β of a softmax decision process in Chap. 5.

Additionally, tonic dopamine has previously been linked to regulation of the exploration rate, a topic I will present in greater depth in Chap. 4. On the one hand, research by [START_REF] Beeler | Tonic dopamine modulates exploitation of reward learning[END_REF][START_REF] Beeler | Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources[END_REF] defends the thesis that dopamine promotes exploration. They reach this conclusion by comparing the behaviour of a hyperdopaminergic mouse breed genetically engineered to present increased dopamine levels with wild-type mice. The two types of mice were studied in a special homecage environment so that they could work to receive food at their own pace. In this setup, food could be obtained by repeated lever pressing, with a highcost lever and a low cost lever, depending on the number of presses required before food delivery. The main result from this study [START_REF] Beeler | Tonic dopamine modulates exploitation of reward learning[END_REF] is that despite being sensitive to lever costs, hyperdopaminergic mice were more willing to choose the high cost lever, and fitting a reinforcement learning model to the experimental data revealed an increased tendency to explore randomly without any effect on learning. To explain this, [START_REF] Beeler | Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources[END_REF] suggested that dopamine indicates available internal energy, so that high dopamine is interpreted by the organism as high energy levels making it worthwhile to explore. Low dopamine on the other hand would mean that internal energy is low and the animal should focus on the best possible actions to correct this imbalance.

On the other hand, [START_REF] Eisenegger | Role of dopamine D2 receptors in human reinforcement learning[END_REF] came to the opposite conclusion regarding the direction of dopaminergic regulation of random exploration. This study used an antagonist of D2 receptors called sulpiride on human subjects and found an increased tendency to choose randomly, independently of any effect upon learning, suggesting that decreased dopamine levels produce exploration. This is also in line with theoretical predictions based on a network model of the basal ganglia [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF] which found that the effects of dopamine on medium spiny neurons excitability mediated specifically by D1 receptors should sharpen the initial discrepancy between unequally strong cortical inputs.

Alternatively, there have been several suggestions that the regulation of exploration is in fact regulated by a different neurotransmitter. For instance, setting the level of exploration is expected to be dependent on the current level of unexpected uncertainty [START_REF] Cohen | Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration[END_REF], uncertainty that arises from totally novel events which violate expectations, relative to expected uncertainty, the possibly well-known variance with which rewards are delivered for example, and [START_REF] Yu | Uncertainty, neuromodulation, and attention[END_REF] attribute such signals to norepinephrine and acetylcholine respectively. [START_REF] Doya | Metalearning and neuromodulation[END_REF] who proposes different meta-learning functions to the main neuromodulators in the human brain also subscribes to the opinion that it is norepinephrine which regulates the exploration-exploitation trade-off, while dopamine would be left with its classical RPE function.

Dopamine receptors and the cortico-striatal synapse

Dopamine receptors are metabotropic receptors coupled to a heterotrimeric G protein [START_REF] Puig | Dopamine modulation of learning and memory in the prefrontal cortex: insights from studies in primates, rodents, and birds[END_REF][START_REF] Beaulieu | The Physiology, Signaling, and Pharmacology of Dopamine Receptors[END_REF], which means that the response of the neuron to their activation is mediated by the dissociation of the subunits of G proteins, the α subunit on the one hand, and the βγ dimer on the other hand, which activate distinct signalling cascades. Five different dopamine receptors are currently known -D1, D2, D3, D4 and D5 receptors -and these fall into two main subtypes called the D1 and D2 receptor types depending on the type of G-protein they activate. D1-type receptors -which comprise D1 and D5 receptors -are associated to Gα s/olf proteins which stimulate the production 54 Background of cAMP, while D2-type receptors -which include D2, D3 and D4 receptorsinhibit cAMP production by activating Gα i/o subunits. In addition, dopamine receptors also modulate the level of intracellular calcium which is another important secondary messenger [START_REF] Nakano | A model-based prediction of the calcium responses in the striatal synaptic spines depending on the timing of cortical and dopaminergic inputs and post-synaptic spikes[END_REF]. These different signalling cascades have complex and contrasted consequences which go far beyond the scope of this work so I will focus on just two of these.

To start with, Paul Greengard, who shared the 2000 Nobel prize with Carlsson, established that dopamine is a slow neurotransmitter, or in other words a neuromodulator, meaning that it modulates the efficiency of other fast neurotransmitters such as glutamate [START_REF] Reynolds | A cellular mechanism of reward-related learning[END_REF]. Modulation of the efficiency of transmission between two neurons is called synaptic plasticity, and in the case of medium spiny neurons, which are the targets of dopaminergic projections in the striatum, this plasticity is implemented through the regulation of the number of glutamate AMPA receptors in the post-synaptic membrane [START_REF] Greengard | Nobel prize acceptance speech: The neurobiology of dopamine signaling[END_REF][START_REF] Nakano | A kinetic model of dopamine-and calcium-dependent striatal synaptic plasticity[END_REF]. In addition, MSNs are the input neurons of the basal ganglia, a functional group of neurons involving the striatum which control action selection, and are organised into distinct sub-populations or channels corresponding to different actions or stimuli. This architecture could help explain how reinforcement learning is implemented by dopamine RPEs: the connection between the cortical inputs, which could represent different stimuli or actions, is strengthened or weakened by the arrival of positive or negative dopamine signals which are delayed with respect to the initial cortical input [START_REF] Pawlak | Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity[END_REF][START_REF] Gurney | A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface[END_REF], and the strength of this connection is representative of the value of the corresponding action or stimulus. More precisely, according to [START_REF] Centonze | Dopaminergic control of synaptic plasticity in the dorsal striatum[END_REF], D1 receptors promote long term potentiation while D2 receptors tend to inhibit it, while long term depression requires the cooperation of both receptor types.

Besides this effect on plasticity, activation of dopamine receptors also directly impacts the excitability of medium spiny neurons by controlling different synaptic and intrinsic ion channels [START_REF] Moyer | Effects of Dopaminergic Modulation on the Integrative Properties of the Ventral Striatal Medium Spiny Neuron[END_REF][START_REF] Surmeier | D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons[END_REF][START_REF] Humphries | Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models[END_REF]. The exact effects are too complex to detail here -for instance D1 receptors can enhance both a hyperpolarizing potassium current and a depolarizing calcium current [START_REF] Humphries | Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models[END_REF] -but according to the review by [START_REF] Surmeier | D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons[END_REF], a simple and acceptable picture is that activation of D1-type receptors depolarizes the dendritic membrane, making medium spiny neurons more excitable, while D2-type receptors have the opposite effect. This is also the point of view adopted by [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF]. This study is based on a model of the basal ganglia in which dopaminergic modulation of medium spiny neurons is captured by simply multiplying cortical glutamatergic signals by a gain factor greater or smaller than 1 for D1 and D2 receptors respectively. The authors then determined that this modulation was responsible for setting the entropy or random exploration levels of subsequent action selection processes, a result which serves as starting point for our investigation into the effects of dopaminergic inhibition on behaviour in Chap. 4.

Chapter 4

Dopamine control of random exploration

Not all those who wander are lost

The Lord of the Rings, J.R.R. Tolkien

The results and text of this chapter have been partially published in Scientific Reports (Cinotti et al., 2019a)

Introduction

All organisms need to make choices for their survival while being confronted to uncertainty in their environment. Animals and humans tend to exploit actions likely to provide desirable outcomes, but they must also take into account the possibility that environmental contingencies and the outcome of their actions may vary with time. Behavioural flexibility is thus needed in volatile environments in order to detect and learn new contingencies [START_REF] Behrens | Learning the value of information in an uncertain world[END_REF]. This requires a delicate balance between exploitation of known resources and exploration of alternative options that may have become advantageous. How this exploration/exploitation dilemma may be resolved and regulated is still a subject of active research in the fields of Neuroscience and Machine Learning [START_REF] Daw | Cortical substrates for exploratory decisions in humans[END_REF][START_REF] Schweighofer | Meta-learning in reinforcement learning[END_REF][START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF].

Dopamine holds a fundamental place in contemporary theories of learning and decision-making. The temporal evolution of phasic dopamine signals across learning has been extensively replicated, and is most of the time considered as evidence of a role in learning [START_REF] Schultz | A neural substrate of prediction and reward[END_REF][START_REF] Schultz | Updating dopamine reward signals[END_REF]. Indeed, dopamine reward prediction error (RPE) signals have been identified in a variety of instrumental and Pavlovian conditioning tasks [START_REF] Bayer | Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal[END_REF][START_REF] Morris | Midbrain dopamine neurons encode decisions for future action[END_REF][START_REF] Roesch | Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards[END_REF], they affect plasticity and action value learning in cortico-basal networks [START_REF] Centonze | Dopaminergic control of synaptic plasticity in the dorsal striatum[END_REF][START_REF] Reynolds | A cellular mechanism of reward-related learning[END_REF][START_REF] Izhikevich | Solving the distal reward problem through linkage of STDP and dopamine signaling[END_REF] and have been directly related to behavioural adaptation in a number of decision-making tasks in humans, non-human primates [START_REF] Costa | Dopamine modulates novelty seeking behavior during decision making[END_REF] and rodents [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF]. Accordingly, it is commonly assumed that manipulations of dopamine activity would affect the rate of learning. However, besides learning, dopamine is also thought to be directly involved in the control of behavioural performance, although theoretical accounts for this are far less well-developed. Dopamine is known to modulate incentive choice (the differential weighing of costs and benefits) [START_REF] Salamone | Beyond the reward hypothesis: alternative functions of nucleus accumbens dopamine[END_REF][START_REF] Berridge | From prediction error to incentive salience: mesolimbic computation of reward motivation[END_REF] as well as other motivational aspects such as effort and response vigour [START_REF] Niv | Tonic dopamine: Opportunity costs and the control of response vigor[END_REF]. Because dopamine is one of the key factors that may encode success or uncertainty, it might modulate decisions by biasing them toward options that present the largest uncertainty [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF][START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF]. This would correspond to a "directed" exploration strategy [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF][START_REF] Zajkowski | A causal role for right frontopolar cortex in directed, but not random, exploration[END_REF][START_REF] Cogliati Dezza | Learning the value of information and reward over time when solving explorationexploitation problems[END_REF]. Alternatively, success and failure could affect tonic dopamine levels and control random exploration of all options, as recently proposed by [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF]. This form of undirected exploration, which is difficult to disentangle from poor performance, may be viewed as "noise" in the choice process [START_REF] Sutton | Reinforcement learning : an introduction[END_REF][START_REF] Doya | Modulators of decision making[END_REF][START_REF] Khamassi | Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters[END_REF]. It is nevertheless known as a classical and efficient exploration strategy in machine learning [START_REF] Sandholm | Multiagent reinforcement learning in the Iterated Prisoner's Dilemma[END_REF]. Previous computational analyses of behavioural data in stochastic tasks have yielded mixed results, some suggesting a promotion of random [START_REF] Beeler | Tonic dopamine modulates exploitation of reward learning[END_REF] or directed [START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF] exploration by dopamine with possibly an effect on learning [START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF], others a reduction of random exploration [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF][START_REF] Lee | Injection of a Dopamine Type 2 Receptor Antagonist into the Dorsal Striatum Disrupts Choices Driven by Previous Outcomes, But Not Perceptual Inference[END_REF][START_REF] Eisenegger | Role of dopamine D2 receptors in human reinforcement learning[END_REF] and still others an effect on learning only [START_REF] Krugel | Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions[END_REF].

In the following paragraphs, I will first describe new experimental results from a study addressing this hypothesis of dopamine regulation of random exploration. This experiment was conducted by collaborators (Virginie Fresno, Alain Marchand and Étienne Coutureau) from the Institut des Neurosciences Cognitives et Intégratives d'Aquitaine (INCIA) in Bordeaux who also carried out the first analyses of the data. After this first section, I will attempt to dissociate what might be distinct contributions of learning and exploration to the observed changes in behaviour using a computational approach.

Experimental results

Experiment description

The experiment consisted in a three-armed bandit task (see Fig. 4.1) during which rats selected a lever by pressing on it, which resulted in the retraction of the lever and the probabilistic delivery of a reward in the food tray. Each lever was assigned a different reward rate and rats were expected to try and find the most rewarded option. Two different reward contingencies were tested, one in which the correct lever was rewarded with probability 7/8 and the other two with a probability equal to 1/16, and the other in which the correct lever was rewarded with a probability of 5/8 and the other two levers 3/16. These different contingencies affected the difficulty of finding the correct lever. Each experimental session was composed of twelve fixed-length blocks of twenty-four trials defined by the position of the correct lever and the reward distribution. Block transitions were not signalled and were always followed by a change of the correct lever and sometimes of the risk level. Block order within a session was pseudo-random so that each correct lever and risk level combination was presented twice to the rat. Rats were tested in four different pharmacological conditions: a control condition during which a saline solution was injected, and three conditions of dopamine blockade by flupenthixol, a D1/D2 receptor antagonist which is expected to cause a decrease of both tonic and phasic sensitivity, injected at 0.1mg/kg, 0.2 mg/kg and 0.3mg/kg.

Dopamine blockade affects exploratory behaviour

In this section, I will present the experimental results obtained with this task by averaging performance, win-shift and lose-shift between blocks for each rat, and then averaging these individual curves. Trials were grouped into six bins of four trials for simplification, and statistical tests were carried out based on these binned averages, so that any trial effect should be understood as a bin of trials effect. I used repeated-measures ANOVA with three within-factors which are risk, trial and

Comparison

Estimated difference (%) Standard error (%) p-value saline -0.1 mg/kg 0.7 1.0 0.89 saline -0.2 mg/kg 2.9 1.1 0.08 saline -0.3 mg/kg 4.7 1.4 0.02 0.1 mg/kg -0.2 mg/kg 2.2 0.9 0.11 0.1 mg/kg -0.3 mg/kg 3.9 1.4 0.05 0.2 mg/kg -0.3 mg/kg 1.7 1.5 0.65 Table 4.1: Post hoc comparison of average experimental performance between the different pharmacological conditions using Bonferroni tests. pharmacological condition. A fourth within-factor will be included in section 4.3.2 to account for the experimental or simulated origin of the data when evaluating simulations. Because the sphericity assumption was violated for performance, winshift and lose-shift (as indicated by Mauchly's tests with p < 0.0001 in all three cases), I applied the Greenhouse-Geisser correction to adjust the degrees of freedom of the F-statistic (ε < 0.75 in every case which is why I did not use the Huynd-Feldt correction). Overall, the rats were able to identify the correct lever over the course of a block as performance (Fig. 4.2 a.) increased quickly in the first few trials before stabilising to an asymptotic performance level in both low and high risk conditions (significant trial effect: F (1.1, 24.8) = 186.7, p < 0.0001), with better performance being observed under low risk (significant risk effect: F (0.2, 5.0) = 148.2, p < 0.0001). There was also a significant interaction between trial and risk (F (1.1, 24.8) = 26.3, p < 0.0001). Dopaminergic blockade by flupenthixol, a D1-D2 dopamine receptor antagonist, produced a decrease in performance (significant flupenthixol effect: F (0.7, 14.9) = 5.61, p = 0.0039, post hoc Bonferroni test results are reported in Table 4.1) without any interaction with either trial number or risk (p ≥ 0.16).

I then studied exploration by using a win-shift index limited to shifts from the current target lever only (Fig 4.2.a.). This index decreased significantly within blocks (trial effect: F (1.1, 17.9) = 27.7, p < 0.0001) but increased with risk (F (0.2, 3.6) = 25.7, p < 0.0001). Contrary to performance, there was no significant interaction of trial and risk on win-shift (F (1.1, 17.9) = 1.7, p = 0.18). Dopaminergic blockade dose-dependently elevated win-shift at all stages (F (0.6, 10.8) = 14.5, p < 0.0001), without interacting with trial or risk (p ≥ 0.19), and it seems that winshift is more sensitive to flupenthixol than performance given the greater number of positive post hoc Bonferroni tests shown in Table 4.2. Because win-shift in the early stages of blocks may not reflect exploration, but rather a return to a previously reinforced lever, a second analysis of win-shift was limited to the last 8 trials when behaviour has stabilised and the correct lever presumably identified (Fig. 4.4.a.). In this case also there were significant main effects of risk (F (0.6, 13.2) = 41.8, p < 0.0001) and flupenthixol dose (F (1.8, 39.5) = 13.5, p < 0.0001) without any significant interaction (F (1.8, 39.5) = 0.6, p = 0.52). Post hoc Bonferroni tests (see Table 4.3) confirm previous observations that increasing dopamine inhibition produces an increase in average win-shift.

Another index of shifting behaviour, lose-shift (Fig 4.5.a.), which may denote a correction strategy, is remarkably unaffected by either risk (F (0.3, 6.4) = 2.7, p = 0.12) or pharmacological condition (F (0.9, 19.2) = 1.8, p = 0.17), with a significant trial effect (F (1.5, 32.0) = 6.8, p < 0.0001) driven by lower lose-shift levels in the very first bin of trials (see table 4.4), probably due to persistence, i.e. reluctance to shift from the previous target despite the fact it is no longer rewarded.

The question that naturally emerges from observing these effects of flupenthixol on performance and win-shift is how does dopamine inhibition produce these effects. On the one hand, the classical theory of phasic dopamine signalling reward prediction errors suggests that the learning process should be affected, but on the other hand there are theories linking dopamine, either phasic or tonic, to the regulation of exploration. It is not clear experimentally how we could disentangle these two effects from our behavioural data, but luckily, reinforcement learning provides us with a framework in which these two processes are distinct. Indeed, a typical Q-learning algorithm (like the one presented in more detail in 4.3.1) works in two stages: first a learning stage, controlled by the learning rate α, second a decisionmaking stage in which an action is picked randomly given the learned action values and a parameter called the inverse temperature which determines the level of greediness or exploration. We can thus rephrase the question as "does dopamine affect the learning rate α or the inverse temperature β?". It has already been shown in such studies as the one of [START_REF] Averbeck | Motivational neural circuits underlying reinforcement learning[END_REF] how manipulating either of these two factors impacts performance in different ways, an effect replicated in Fig. 4.6 alongside the previously unknown effects of parameter manipulation on win-shift. Briefly, the learning rate controls the initial slope of the performance curve, as well as the initial level of win-shift, but given enough time, both these curves tend to converge to the same level. Intuitively, this makes perfect sense: the Q-values take more time to be learned, but once they are, they are used in exactly the same way independently of α. In other words, we should expect a significant interaction between trial and dopamine inhibition on performance and win-shift, as the initially separate curves gradually converge. By contrast, the inverse temperature, by setting the amount of noise in the action selection process, determines the asymptotic level of performance, and affects win-shift levels at all block stages, i.e. there should not be any interaction between trial and pharmacological condition for win-shift. A qualitative comparison between these simulations and the behavioural data shown in Fig. 4.2.a. and 4.3.a. plead for a change in exploration rate as the more likely explanation, as performance curves do not seem to converge to the same level between doses, in low risk blocks at least, and increasing flupenthixol simply shifts win-shift curves upwards without in either case any trial × dose interaction effect which we would expect from a learning rate manipulation. Furthermore, we have already seen how the win-shift rate in the last eight trials of blocks was still significantly affected by flupenthixol dose (Fig. 4.4.a.). However, this persistent difference in the curves at the end of each block could perhaps be explained simply by the inability of rat behaviour to reach convergence in the limited number of trials provided to them in each block. Additionally, the fact that these behavioural curves are obtained by averaging several individuals with potentially very different parameters while the simulations of Fig. 4.6 are carried out with a single set of parameters should also be taken into account, and we still have to ensure that this reinforcement learning framework can be applied successfully to the experimental data, which is why a full answer to this question can only come from direct modelling of the experimental data.

Modelling results

Model description and optimisation

In order to gain insight into what mechanisms could cause these behavioural effects, I optimised a Q-learning model extended with a forgetting mechanism. Q-learning aims to represent the expected reward rate following each possible action. To do this, at each trial the model computes a reward prediction error:

δ t = r t -Q(a t ) (4.1)
with r t the reward obtained after selecting this action, equal to 1 or 0 if a reward is delivered or not respectively, and Q(a t ) the current value of the chosen action. This prediction error is used to update the corresponding Q-value:

Q(a t ) ← Q(a t ) + α × δ t (4.2)
with α the learning rate controlling how fast the Q-values update. The forgetting mechanism which I implemented in parallel consists in devaluing the two other levers back to their initial value of 0:

Q(a i = a t ) ← (1 -α 2 ) × Q(a i ) (4.3)
with α 2 the forgetting rate. This mechanism [START_REF] Barraclough | Prefrontal cortex and decision making in a mixed-strategy game[END_REF]; Ito and Doya ( 2009)) is necessary because, as illustrated in B.2, a Q-learning model without this mechanism overestimates the rates of win-and lose-shift. [START_REF] Katahira | The relation between reinforcement learning parameters and the influence of reinforcement history on choice behavior[END_REF] explains that this mechanism is actually necessary to adequately represent the dependence of action selection on previous choices independently of their outcome; in other words, it represents choice persistence or switching tendencies depending on whether α 2 is smaller or greater than α. A simple way to see this, is by considering the case of an unrewarded trial with a forgetting rate which is larger than the learning rate: if this is the case, the Q-value of the chosen action will decrease by a factor equal to (1 -α) but the other two actions will decrease by a greater amount since the factor (1 -α 2 ) is smaller meaning that action selection is biased in favour of the previous choice despite the lack of reinforcement.

Conversely, if the forgetting rate is smaller than the learning rate, then the effect of any reward omission on the difference between the Q-value of the previous action and the other Q-values will increase even more, making a switch more likely.

Once updated, the Q-values were used to determine the probability of each action using a softmax mechanism:

P (a t+1 = a i ) = e βQ(a i ) j e βQ(a j ) (4.4)
with β the inverse temperature determining the exploration-exploitation balance in such a way that increasing its value causes more exploitation by exacerbating the difference between the highest Q-value and the others; conversely, in the extreme case of β = 0, actions are chosen equiprobably with no regard for their expected value. Once defined, and given a set of parameters θ, this model can give us the likelihood of the history, H, of choices made by a given animal between the first and last trial T:

Parameter Minimum Maximum Initial values α 0 1 0.1 0.5 0.9 β 0 + inf 1 5 20 α 2 0 1 0.1 0.5 0.9 Table 4.5: Characteristics of the parameters of the forgetting Q-learning model as used for likelihood optimisation.

P (H|θ) = T t=1 P (a t |θ) = T t=1
e βQ(at) j e βQ(a j ) (4.5)

In practice, parameters were optimised by minimising the negative log-likelihood using the fmincon function in MATLAB. To avoid falling into local minima, different initialisation points in the parameter space were used, three for each parameter, meaning that for this particular model -which has three parameters -there were 3 3 = 27 different initialisations. Once the model had been optimised for each initialisation, I kept the set of parameter values with the greatest likelihood. The different initial values and the bounds given to each parameter are shown in Table 4.5.

Model simulations

Once optimised and before drawing any conclusions from a subsequent parameter analysis, I had to verify that this model was appropriate to explain the animals' behaviour. I did so in a twofold process, firstly by examining the quality of the optimisation using "constrained" simulations, secondly by examining the fit of "free" simulations to the original data.

Constrained simulations

Constrained simulations are simulations in which the choices of the model are identical to those of the rat. This allows me to extract the Q-values and action probabilities the model predicts in each trial, so that I then compare these action probabilities with the animals' actual choices. On each trial the animal can obviously only make a single choice, while the model provides a probability distribution. In order to compare the animals choices with the model probabilities, I decided to label each possible action within blocks as follows:

• the target lever

• the previous target lever

• the third action Model probabilities are obtained by averaging the softmax probability in a given bin of four trials within a block. Each point represents an average for a given combination of rat, 4-trial bin, risk level, dose and action. A linear regression curve without an intercept was fitted to these points: y = 0.99 × x and is plotted in black.

In the case of the very first block of a given action, I simply used the target of the very last block as previous target, so that the labels of each lever were counterbalanced in each session (provided of course that the session was completed). I then averaged the probability of selecting each possible action -target, previous target, third choice -between blocks for separate block uncertainties and for bins of four trials, similar to the standard adopted in Fig. 4.2, 4.3 and 4.5. Finally, I computed the average probability given by the model for each of these points, plotted it against the average frequency as shown in Fig. 4.7 and found a very good correspondence between model predictions and experimental data. A linear regression curve without an intercept parameter was fitted to the data and was found to be almost equal to the identity function: y = 0.99 × x. The same analysis was applied separately to different pharmacological and risk conditions and gave similarly good results in every case (data not shown).

Free simulations

Free simulations are simply simulations obtained by applying the optimised parameter sets to the same sequence of blocks as the individual subject but otherwise letting the model determine its own choices by sampling from the probability distribution computed on each trial. Obviously, this means I also simulate the reward process which is quite straightforward. In this section, I report results from comparing experimental data with a single set of simulations, while the published paper (Cinotti et al., 2019a) uses an average of 100 simulations.

A large number of simulations ensures that the average simulated behaviour has fully converged, but the reduced variability in simulations means that significant differences between simulated and experimental data emerge which is one of the reasons I prefer to compare like for like in this thesis manuscript, a single artificial dataset with a single experimental dataset, rather than an average of 100 datasets with a single experimental one. In addition, choosing 100 simulations was purely arbitrary and, when I wanted to justify this number, I looked at how adding an extra simulation affected average performance, win-shift or lose-shift curves by comparing the average for n simulations with the average of n+1 simulations for n between 1 and 99. Doing this, I found that around 30 simulations per individual was quite sufficient for the average curves to converge.

When experimental and simulated data (shown in Fig. 4.2.b., 4.3.b., and 4.5.b.) were pooled together for repeated-measures ANOVA with four within factors (simulated/experimental data, risk, pharmacological condition and trial bin), there was no significant main effect of simulations on within-block evolution of either performance (F (0.1, 3.1) = 2.7, p = 0.12), win-shift (F (0.1, 1.6) = 1.1, p = 0.31) or lose-shift (F (0.2, 3.8) = 3.9, p = 0.06). However, a significant interaction between simulations and trial did emerge in the case of performance (F (0.7, 15.3) = 5.20, p = 0.0003) driven by differences emerging towards the end of blocks in which simulations do a better job at choosing the correct option (see Table 4.6). Crucially however, no interaction involving simulations and flupenthixol dose could be detected (smallest p = 0.0665 which was for the interaction between flupenthixol and simulations on lose-shift curves). Analysis of the simulations of this model separately from the experimental data using a repeated-measures ANOVA with just three factors, risk, trial and dose, yields comparable results to the experimental data:

• for average performance: significant main effects of dose (F (0.7, 15.7) = 4.6, p0.01), risk (F (0.2, 5.2) = 309.6, p < 0.0001), and trial (F (1.2, 26.2) = 201.1, p < 0.0001) and a significant interaction of trial and risk (F (1.2, 26.2) = 32.6, p < 0.0001);

• for average win-shift: significant main effects of dose (F (0.6, 11.6) = 8.8, p = 0.002), risk (F (0.2, 3.9) = 23.4, p = 0.0001) and trial (F (1.0, 19.3) = 46.8, p < 0.0001) on win-shift;

• for average lose-shift: significant main effects of trial (F (1.4, 30.6) = 15.3, p < 0.0001), but also effects of dose (F (0.8, 18.4) = 11.6, p < 0.0001) and risk (F (0.3, 6.1) = 8.8, p = 0.007) which are absent in the experimental data.

As for win-shift rates in the last eight trials of blocks (see Fig. 4.4.b.), there is no significant main effect of simulations (F (0.3, 7.5) = 0.3, p = 0.62) or of any interaction involving simulations (smallest p = 0.051 for the interaction of all three

Model optimisation dissociates random exploration from learning

To disentangle the effects of flupenthixol on learning versus performance, I then examined the values of the different parametersα, β and α 2 -across pharmacological conditions (see Fig. 4.8). This is a standard technique used for instance in the study by [START_REF] Eisenegger | Role of dopamine D2 receptors in human reinforcement learning[END_REF]. Because the distribution of values of the inverse temperature violated the assumption of normality required for a repeatedmeasures ANOVA, Friedman ANOVAs were used to study the potential effect of flupenthixol on parameters. Flupenthixol had no discernible effect on the learning rate α (χ 2 (3) = 4.04, p = 0.26) or on the forgetting rate α 2 (χ 2 (3) = 1.38, p = 0.71), but clearly decreased the exploration parameter β (χ 2 (3) = 15.1, p = 0.0018). Post hoc tests revealed that β for 0.2 and 0.3 mg/kg was significantly smaller than for 0 mg/kg (p = 0.012 and p = 0.0024 respectively). Thus, the only parameter of the model significantly affected by dopaminergic blockade was the inverse temperature which decreased (i.e. exploration increased) as dopaminergic inhibition increased.

Next, I generalised this result by testing the optimised parameters of a range of different models which include a standard Q-learning model (Supplementary ). In all of these cases, the only parameter that significantly varied with dose condition was the one responsible for controlling random exploration which flupenthixol invariably increased.

Finally, because of the strong interaction between learning rate and inverse temperature [START_REF] Daw | Trial-by-trial data analysis using computational models[END_REF], I checked that this methodology was able to distinguish variations in inverse temperature β from variations in the learning rate α, by applying full parameter optimisation (leaving all three parameters free) to an artificial data set generated with either α or β varying between doses while the other two parameters remain constant. The values of α and β which were used to generate the simulations had themselves been optimised on the experimental data so as to capture the maximum variability allowed by the data for this specific parameter. In both cases, we found that there were indeed significant variations (p<0.0001 in both cases, data not shown here) in the optimised values, a fact which suggests that variations of one parameter could be at least partially captured by variations of the other. In fact, when only the learning rate α varied (Fig. 4.9.a.), the full optimisation on the simulated data set was capable of identifying this effect on α (χ 2 (3) = 16.1, p = 0.0011), without confusion with the two other parameters (χ 2 (3) < 1.54, p > 0.67). Conversely, on an artificial data set where only the inverse temperature β varied (Fig. 4.9.b.), the subsequent optimisation correctly identified this parameter as the only varying parameter (p = 0.0007 and p > 0.12 for the other two parameters). These results clearly show that the computational analysis can disentangle the effects of dopamine manipulations on the inverse temperature from possible effects on learning rate despite their interdependence. .9: Verification procedure to determine if optimisation of a forgetting Qlearning model can distinguish variations of the learning rate from variations of the inverse temperature. The first step in this process was to optimise two versions of the model, one where α was allowed to vary between doses, while the other parameters were not, and one where only β was allowed to vary. This way, the maximum variation in the parameter of interest allowed by the experimental data was captured, and I indeed found that if only α was allowed to vary, it would significantly decrease (not shown here), proving that there is indeed a relative tradeoff between learning rate and inverse temperature. The second step consisted in creating an artificial dataset by simulating these two versions of the model. Finally, I optimised a model with all three parameters allowed to vary between doses on each dataset a. and b. and verified that in each case, the only parameter whose optimised values varied significantly was the one originally allowed to vary.

Relationship between RPEs and exploration

The last paragraphs have led us to the conclusion that inhibition of dopamine leads to a decrease in the inverse temperature of a forgetting Q-learning model without affecting the learning rate. This appears to validate the theory of [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF], but in the following lines, I will show that such an effect is to be expected within the classical theory that phasic dopamine signals reward prediction errors which support learning, without requiring any additional functionality of tonic dopamine control of the exploration rate through modulation of the excitability of striatal medium spiny neurons. In the following lines, I would like to show that inhibition of positive reward prediction errors only is mathematically equivalent to changing the value of the inverse temperature, thus providing a mechanism for our conclusion that dopamine inhibition affects exploration rate instead of learning. The model of inhibition we have selected takes the following form:

δ t =    1 -Q t -f 0 -Q t (4.6)
In other terms, positive reward prediction errors are dampened by a flupenthixol effect f while negative reward prediction errors are left intact. It is immediately evident that this is actually equivalent to defining a new reward function R = {0, 1 -f } which is in fact simply the original reward function R = {0, 1} multiplied by (1 -f ), thanks to the fact that absence of rewards is arbitrarily defined as worth 0. We will now show that this means that Q-values calculated with an inhibition factor f, which we will call Q , are simply proportional to Q-values without an inhibition factor, called Q. First, a simple but tedious demonstration by recurrence based on the learning and forgetting equations (Eq. 4.2 and 4.3) and provided that Q-values are initialised at 0 as was the case, will provide us with the following definition of the different Q-values at any given trial t and for action a i :

Q t (a i ) = t k=1 δ k,i .α.(1 -α) n i,k -1 .(1 -α 2 ) t-n i,k -1 .r k (4.7)
with δ k,i = 1 if action i was selected at trial k and 0 otherwise, and n i,k the number of times action a i was selected since trial k. This equation simply means that the Q-value of a given action is the sum of past rewards received after selecting this action weighted by (1) the learning rate for the very trial it was received, ( 2) (1-α) for every trial the same action was selected since that reward, and (3) (1-α 2 ) for every trial where this action was not selected since. In the case of Q , if the value of rewards is equal to r = (1 -f ).r instead, then we can simply factor (1 -f ) out of the sum and get:

Q t (a i ) = (1 -f ).Q t (a i ) (4.8)
Notice that the factor (1-f) is independent of trial t and action a i so that when the Q-values are plugged into the softmax function we get:

Discussion

73

P (a t+1 = a i ) = e β.Q t (a i ) j e β.Q t (a j ) = e β .Qt(a i ) j e β .
Qt(a j ) (4.9)

with β = (1 -f ).β for every action and at every trial. This means that a model where β is allowed to vary between doses is strictly equivalent to a model with fixed parameters and a flupenthixol effect applied as described in Eq. 4.6. It is important to bear in mind that this equivalence is true only because non-rewarded trials are arbitrarily given a value of 0, allowing us to set r = (1 -f ).r. If we use an alternative convention such as unrewarded trials being worth -1, then the equivalence no longer holds. For this reason, I tested a further two forgetting Q-learning models, with unrewarded trials being worth -1 (and rewarded trials1), or -5 (with rewarded trials worth 10) in the hope of being able to disentangle this effect of RPEs on the inverse temperature, by optimising a model with an explicit flupenthixol effect on RPEs (Eq. 4.6) and a model with a varying β parameter. Unfortunately, even with these new conventions, the two models were too close to convincingly separate the conflicting hypotheses. As a result, this question remains unsettled.

Discussion

This chapter presents a formal demonstration that in simple reinforcement learning models, a reduction of the amplitude of positive reward prediction errors directly translates into changes in random exploration levels. In rats tested on a probabilistic choice task, we experimentally observed that systemic administrations of flupenthixol, a D1-D2 antagonist, dose-dependently increased random exploration, and only indirectly affected performance. Dopamine blockade increased win-shift behaviour under both low and high risk conditions and noticeably late in a block when the rats had acquired the correct response. I reproduced behavioural data using unconstrained simulations and showed under a variety of models that exploration rate was the only parameter significantly affected by dopamine blockade in this task.

Previous experimental and computational work by [START_REF] Costa | Dopamine modulates novelty seeking behavior during decision making[END_REF] has demonstrated through blockade of dopamine transporter that increasing dopamine signalling increases novelty seeking which is a form of directed exploration. However, contrary to our results, they found no drug-related differences on the inverse temperature which controls undirected or random exploration. Our own task is not specially well-designed to tackle the question of novelty-driven exploration, as the only occurring change is the reward rates of the three levers, but I nonetheless optimised a model of directed-exploration [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF]) and still found an effect on random instead of directed exploration. Future work should focus on clearly disentangling these two processes of exploration and the role of dopamine in each of them [START_REF] Gershman | Dopaminergic genes are associated with both directed and random exploration[END_REF]). In the present study, the effects of various doses of dopamine antagonist in the same individual indicate that normal dopamine levels limit undirected exploration in this task and therefore bias performance toward exploitation without affecting the learning rate, a result which is consistent with that of [START_REF] Lee | Injection of a Dopamine Type 2 Receptor Antagonist into the Dorsal Striatum Disrupts Choices Driven by Previous Outcomes, But Not Perceptual Inference[END_REF] using a different animal model, the Macaque, and a different decision-making task. Undirected exploration may reflect several factors, and flupenthixol could for instance have reduced motivation, response vigour [START_REF] Niv | Cost, benefit, tonic, phasic: What do response rates tell us about dopamine and motivation?[END_REF]) or attention. However, the observation that learning rates were unchanged argues in favour of a selective effect on random exploratory choices.

In the task, performance improved within a block as one of the levers was gradually identified as target. Concurrently, win-shift decreased as learning progressed. At the beginning of each block, performance dropped to chance levels or below, while win-shift increased. These high levels of win-shift in the absence of drug correspond to moments when the values of the various options have not been well identified and the rat's uncertainty is high. Indeed, in the high risk condition, identifying the correct lever was more difficult and was associated with both lower performance and higher win-shift levels. Under dopamine blockade, the dose-dependent increase in win-shift appeared independently of uncertainty, since it did not interact with either trial number within a block or risk level. This is consistent with the notion that dopamine unconditionally scales action values and controls noise in the last stage of decision-making, where action values are converted to actual choices [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF]). As to the effect of flupenthixol on lose-shift, the simulations (based on 100 runs) did show a slight increase corresponding to increased random exploration. This effect did not reach significance in the actual behavioural data, where it was probably more difficult to detect close to the ceiling value (the probability of lose-shift when choices are random is roughly 0.67).

Several past studies [START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF]; [START_REF] Beeler | Tonic dopamine modulates exploitation of reward learning[END_REF][START_REF] Beeler | Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources[END_REF]) have focused on the effects on choice of inter-individual differences in dopaminergic function. My results thus stand in contrast with those of [START_REF] Beeler | Tonic dopamine modulates exploitation of reward learning[END_REF][START_REF] Beeler | Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources[END_REF] who observed that hyperdopaminergic mice allocate more time and energy to expensive options. On the basis of computational modelling, the authors interpreted their data as an increase in undirected exploration. However, data from primates and human subjects instead indicate increases in exploration with reduced dopaminergic activity, in agreement with my results. In particular, in an investigation of the behavioural disparities between human subjects due to genes controlling prefrontal and striatal dopamine function, [START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF] concluded that COMT alleles associated with lower dopamine levels increased the exploration component of their reaction time model. [START_REF] Eisenegger | Role of dopamine D2 receptors in human reinforcement learning[END_REF] found in a between-subject design that a strong dose of sulpiride, a dopamine D2 receptor antagonist, in healthy human subjects favoured exploration without affecting learning in a probabilistic task. These inter-individual studies supposed a fixed exploration level, and were inadequate to describe changes in exploration in the same subject. However, a similar result was achieved within subjects in an oculomotor decision task by [START_REF] Lee | Injection of a Dopamine Type 2 Receptor Antagonist into the Dorsal Striatum Disrupts Choices Driven by Previous Outcomes, But Not Perceptual Inference[END_REF] who demonstrated that injections of a dopamine type 2 receptor antagonist in the dorsal striatum of two macaques deteriorated the animals' performance in a manner best explained by an increase of noise in the decision-making process rather than an effect on learning.

These experimental results, together with theoretical approaches [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF], suggest that the behavioral effects of dopamine antagonists result from their action on the striatum, and [START_REF] Lee | Injection of a Dopamine Type 2 Receptor Antagonist into the Dorsal Striatum Disrupts Choices Driven by Previous Outcomes, But Not Perceptual Inference[END_REF]' s results point to a role of D2 receptors. However, [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF]'s model predicts that changes in D1 receptor activation are the main determinants of the exploration-exploitation trade-off. The use of flupenthixol, which blocks both D1 and D2 receptors, as well as the systemic mode of administration do not allow me to clarify this issue. I can only speculate that DA receptor blockade by flupenthixol might exert its effect on random exploration through direct effects on postsynaptic activity, mimicking a reduction in dopaminergic activity. By contrast, exploration directed toward novel objects [START_REF] Costa | Dopamine modulates novelty seeking behavior during decision making[END_REF] might be facilitated by increased dopamine levels if they enhance a dopamine-mediated novelty bonus [START_REF] Kakade | Dopamine: Generalization and bonuses[END_REF].

In the three-armed bandit task used in the present study, I modelled learning of the correct action using an extended Q-learning model with forgetting, and I modelled choice behaviour using a softmax mechanism. This model was sufficient to account for behavioural performances as shown by the high similarity of the simulated (unconstrained) and experimental behavioural data (Fig. 2.3, 4.3, 4.4 and 4.5) and the high correlation between the modelled value (constrained by the rats' choices) of the different levers and actual choice probability (Fig. 4.7), even during periods of low value when the target lever was not yet identified. I showed that a forgetting mechanism is required to adequately account for the rats' behaviour as a simple Q-learning mechanism appeared unable to cope with the multiple reversals occurring throughout the task. In this model, forgetting is important to reduce the value of competing actions even when these actions are not chosen any more, unlike simple Q-learning which only adjusts the value of actions actually performed. The observation that the forgetting rate is generally larger than the learning rate (Fig. 4.8) implies that the rats tended to persist on a choice even in the absence of reward. This process stands in contrast with some theories of directed exploration which predict that unchosen options become attractive as uncertainty about their outcome increases.

The model does not include any mechanism to track uncertainty about action values, unlike several models of choice behaviour in humans [START_REF] Nassar | An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment[END_REF]; [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF]; [START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF]; [START_REF] Behrens | Learning the value of information in an uncertain world[END_REF]; [START_REF] Jepma | Catecholaminergic Regulation of Learning Rate in a Dynamic Environment[END_REF]). The simulations furthermore show that the gradual reduction in win-shift within a block does not reflect a dynamic adaptation of the model parameters since it is reproduced in the simulations where these parameters are kept constant. Instead, this decrease is a consequence of the interaction between value learning and the softmax mechanism. Choice is more variable when actions values are relatively similar, and becomes less variable as the values of the various actions are better differentiated. We did observe a significant effect of risk on performance and win-shift in the behavioural data, but because the same effect was present in the simulations, it is attributable to a slower acquisition of value in the high risk situation due to increased stochasticity.

As expected from the formal analysis, dopamine was found to specifically control the exploration parameter β (inverse temperature) which represents undirected exploration or random noise in the choice process converting values to actions [START_REF] Zajkowski | A causal role for right frontopolar cortex in directed, but not random, exploration[END_REF]; [START_REF] Doya | Modulators of decision making[END_REF]; [START_REF] Khamassi | Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters[END_REF]), rather than directed exploration driven by uncertainty [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF]; [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF][START_REF] Frank | The neurogenetics of exploration and exploitation: Prefrontal and striatal dopaminergic components[END_REF][START_REF] Cogliati Dezza | Learning the value of information and reward over time when solving explorationexploitation problems[END_REF]). Furthermore, this result is still valid with other models such as the standard Q-learning, the ε-greedy version of Q-learning, a meta-learning model, and even a directed exploration model. This result agrees with the theoretical proposal by [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF] that tonic dopamine in the basal ganglia could modulate the exploration-exploitation trade-off during decisionmaking, On the basis of a prior, biologically inspired model of the basal ganglia, they showed that changing simulated tonic dopamine levels had similar effects as changes in the β parameter. The mechanism they proposed rested on a different mechanism though, namely the direct modulation of the excitability of striatal neurons instead of a change in the learning mechanism as we propose here. This simply highlights the fact that our study cannot disentangle these two possibilities: either dopamine inhibition diminishes neuron excitability which in turn increases noisy exploration, or dopamine inhibition causes a down-revision of the reward value which has the same downstream effect on decision-making. Computationally at least, I have proved that these two possibilities are in fact indistinguishable.

On the other hand, a hypothesis that my work can successfully rule out is that dopamine controls the learning rate. Indeed, my study highlights a common misconception that equates the well-established role of dopamine in learning [START_REF] Schultz | Neuronal Reward and Decision Signals: From Theories to Data[END_REF]) with an effect on learning rate. To the extent that learning is based on reward prediction errors and action selection on a softmax mechanism, as is typically assumed in model-free reinforcement learning, my formal analysis indicates that the inverse temperature, the parameter controlling random exploration, is the only parameter affected by simple manipulations of the reward prediction error signal. Notably, in our task, the fitted learning rate was unaffected by flupenthixol, and behavioural performance was largely preserved, while the win-shift index markedly increased. In a probabilistic learning task, [START_REF] Pessiglione | Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans[END_REF] showed that administration of L-DOPA, a chemical precursor of dopamine known for enhancing dopaminergic functions, improved performance in accumulating gains compared to subjects under a dopamine antagonist. This improvement was attributed to an increase in learning from positive reward errors, but increased dopamine could also have reduced exploration. Similarly, [START_REF] Krugel | Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions[END_REF] reported that COMT alleles increasing dopamine levels were associated with better performance in a rewardbased learning task with reversals, and they explained their results by a modulation of learning rate. In contrast, there are reports in humans and monkeys [START_REF] Costa | Dopamine modulates novelty seeking behavior during decision making[END_REF]) that probabilistic learning is not impaired by dopamine antagonists [START_REF] Shiner | Dopamine, salience, and response set shifting in prefrontal cortex[END_REF][START_REF] Smittenaar | Decomposing effects of dopaminergic medication in Parkinson's disease on probabilistic action selection -learning or performance[END_REF]. Our results call for careful modelling of the impact of dopaminergic manipulations in behavioural tasks as changes in random exploration rates could easily be mistaken for changes in learning rate.

As fluctuations in tonic dopamine levels track the average reward rate [START_REF] Niv | Tonic dopamine: Opportunity costs and the control of response vigor[END_REF]), it seems natural to use such a signal to regulate the exploration-exploitation trade-off [START_REF] Khamassi | Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters[END_REF]): high reward rates suggest that the current policy is appropriate and the subject could crystallise its behaviour by exploiting more. Conversely, sudden drops in reward rate leading to tonic dopamine decreases may lead to increased exploration of the environment in search for better options. Dopamine levels could thus contribute to dynamically regulating exploratory choices in volatile environments where option values change with time. Here, we show that dopamine blockade affects undirected exploration independently from the changes in uncertainty levels within blocks. Dopaminergic regulation of exploration appears to occur at a longer time scale than that of a few trials, which would constitute a form of meta-learning adapting behaviour to the general characteristics of the task rather than to immediate events, a topic we will now turn our attention to in the very next chapter.

in Chap. 4 whereby I will optimise a forgetting Q-learning model on separate stages of the experiment and analyse parameter evolution, and finally propose and compare a variety of potential meta-learning models, as well as a model of directed exploration.

Experimental results

Between session improvements in performance

The experiment lasting several weeks, an interesting aspect of behaviour is its longterm, session-to-session evolution. To study this aspect, I looked at average performance, win-shift, and lose-shift curves separately for successive groups of six sessions. A qualitative inspection of Fig. 5.1 informs us that performance of rats increases between sessions in both low and high uncertainty conditions. At the beginning of a block, performance lies below chance around 28% as rats are unaware of the block change and probably persist in selecting the previous target lever. The different curves then increase at different speeds resulting in different final performance levels: in the early stages of the experiment (sessions 1-6), rats reach final performance levels of only 45% and 40% in low and high risk blocks respectively, compared to 63% and 48% in the latest sessions (sessions 19-24). Unsurprisingly, the low risk condition allows for better performance as previously reported in Chap. 4. A repeated-measures ANOVA with three within-subjects factors confirms these different observations by uncovering statistically very significant effects of trials (F (5, 115) = 112.1, p < 0.0001), session group (F (3, 69) = 29.4, p < 0.0001) and risk level (F (1, 23) = 98.4, p < 0.0001). There are also significant effects of interactions between all possible combinations of main factors (p < 0.0463). Post hoc Bonferroni tests on low risk blocks showed in particular that performance in the first six sessions was significantly worse than all following groups of sessions (p < 0.0097) and this was also the case for sessions 7-12 when compared with the following two groups of sessions (p < 0.0012). As for high risk blocks, performance in sessions 1-6 is also significantly worse than in all other sessions (p < 0.0061), which is also the case for sessions 7-12 compared to sessions 19-24 (p = 0.0242). The difference in performance level is most obvious in later trials which is why I also focused my analysis on the last eight trials of blocks in Fig. 5.2.a., in which case I also found significant session (F (3, 69) = 24.7, p < 0.0001), and risk (F (1, 23) = 94.6, p < 0.0001) effects as well as a significant interaction between these two factors (F (3, 69) = 4.8, p = 0.0043). In summary, there is clear improvement in performance between sessions in both high and low risk blocks. It is noteworthy that this improvement is not limited to the very first sessions as significant differences were also found when comparing sessions 7-12 to later ones. A potential explanation for this improvement is that rats gradually learned to expect a switch in target from one block to the next, a hypothesis that is all the more attractive that blocks are of fixed length (24 trials). However, this possibility is ruled out by comparing the length of persistent streaks at the beginning of new blocks. A persistent streak is simply defined as a sequence of uninterrupted selections of the previous target and if the rats learn to expect a block change, we would expect these streaks to shorten. Because persistence is likely to be influenced by the level of reinforcement the previous target has received, persistent streaks following low risk and high risk blocks are shown separately in Fig. 5.3. Instead of decreasing, this measure actually increased significantly following low risk blocks. A repeated-measures ANOVA finds a significant main effect effect of the previous block risk (F (1, 23) = 52.2, p < 0.0001), no significant main effect of sessions (F (3, 69) = 2.6, p = 0.059) but a significant interaction effect between previous block risk and session (F (3, 69) = 4.6, p = 0.005), and persistent streaks after low risk blocks are indeed significantly shorter in the first six sessions than in any other session group according to post hoc Bonferroni tests (p < 0.02). As rats improved their overall performance, it seems they actually had a greater tendency to persist with the previous target after a transition from a low risk block meaning that the improvement in performance cannot be due to growing expectation that the target is about to change. In fact, the increase in persistence is certainly a side-effect of the higher level of reinforcement received by the previous target, as the rats improved their performance by increasingly exploiting it.

Long-term trends in exploration

In fact, another hypothesis is that rats require less and less exploration to find new target levers, something we can investigate by studying win-shift which clearly drops from the first group of sessions to all subsequent ones (see Fig. 5.1 cd.). As for performance, I also found very significant main effects of trials, risk and sessions (p < 0.0001) but the only interaction involving sessions was with risk (F (3, 69) = 6.1, p = 0.0021). In low risk blocks, win-shift in the first group Similarly to the analysis of the win-shift curve, post hoc Bonferroni tests show that win-shift in the first group of sessions is significantly higher than in other groups (p < 0.0019). These results indicate a between-session decrease in exploration in accordance with the improvement in performance. On the other hand, lose-shift curves (Fig. 5.1.e-f.) which track the probability of choosing a different lever after an unrewarded trial, are not affected by any main or interaction effect involving sessions (p > 0.13) and lie quite stagnant between 50 and 60% with the exception of a slight dip at the beginning of blocks (lose-shift in the first bin of trials is significantly smaller than all other trial bins p < 0.0131) probably due to the persistent selection of the previous target. Interestingly, the level of lose-shift is close to 60% which is similar to what we would expect if the animal was choosing his actions randomly, i.e. with a probability of 1 3 suggesting that from a computational point of view a single unrewarded trial is sufficient for all Q-values to be close to equal, probably thanks to the forgetting mechanism which also pulls non-selected action values down to 0.

Parameter evolution of a forgetting Q-learning model

To summarise the experimental results, there were two consistent long-term trends in the rats' behaviour: performance in later trials of blocks improved between ses-sions, while the win-shift index, assumed to at least partially reflect exploration, decreased throughout block duration. This occurred in spite of the fact that the task did not change between sessions, and that the target systematically changed from one block to the next in a pseudo-random manner, so that all levers were equally reinforced by the end of each session. Reinforcement learning theory (Sutton and Barto, 1998) provides a normative framework in which to interpret these data. I made use of a particular kind of reinforcement learning algorithms known as Q-learning which relies on three basic parameters:

• a learning rate α which determines how quickly the system learns from observed outcomes: low values ensure that action values are relatively stable, changing only slowly over time. In the present task where the correct action periodically changes, higher learning rates should allow a rapid increase in performance within a block, at the cost of increasing sensitivity to the stochastic nature of reinforcement.

• a random exploration parameter β, known as the inverse temperature, which sets the level of noise in the choice process based on action values. Low values of β result in almost equiprobable action selection, in spite of their respective values, while high values of β exacerbate the differences between action values and thus increase bias towards the action with highest expected value.

• a forgetting rate α 2 , which determines how much the value of an unselected action is lost or forgotten. This is an optional mechanism used in such studies as those by [START_REF] Barraclough | Prefrontal cortex and decision making in a mixed-strategy game[END_REF]; [START_REF] Ito | Validation of decision-making models and analysis of decision variables in the rat basal ganglia[END_REF] and studied extensively by [START_REF] Katahira | The relation between reinforcement learning parameters and the influence of reinforcement history on choice behavior[END_REF] who linked it to staying/switching strategies as previously discussed. As with the pharmacological experiment, we found this mechanism to be necessary (see annex chapter B).

If the value of β were to increase between sessions, this would directly impact exploration and performance, since rats would become increasingly biased towards the action identified as best, which might increase the number of persistent mistakes at the beginning of blocks but also exploitation of the correct lever in later trials. In parallel, an increase in the α parameter could produce similar results by allowing the correct action value to reach higher values faster, making it easier to discriminate from the other actions. Thus, within a reinforcement learning framework, the long-term effects that we observed may be due to changes in learning or exploration rate, or to a combination of the two. To illustrate this important point, I optimised two different models on the experimental data: the first model has fixed inverse temperature and forgetting rates throughout the experiment but allows for the possibility of having different learning rates for the different groups of six sessions while the second model tests the possibility of having different inverse temperatures while keeping other parameters constant. These two models were compared with a model with fixed parameters by applying paired-t-tests to the distributions of individual Akaike Information Criterion (AIC) and Bayesian The MSE scores of the α-free and β-free models are both significantly smaller than those of the fixed-parameters model (p < 0.0009). There is no statistically significant difference between the MSE scores of the α-and β-free models (p = 0.15).

Information Criterion (BIC); these measures allow for model comparison of likelihood scores while compensating for different numbers of parameters [START_REF] Lebarbier | Le critère BIC : fondements théoriques et interprétation[END_REF]. The BIC score is known to be more stringent than the AIC score. I found that both models with free α or β parameters have significantly improved AIC (p < 0.0010) but BIC scores are equivalent to those of the model with fixed parameters (p > 0.23). However, if we take a look at the generative power of these models by comparing the fit of simulations to individual experimental performance and win-shift curves [START_REF] Palminteri | The Importance of Falsification in Computational Cognitive Modeling[END_REF], through the mean-squared errors (MSE) of individual fits of simulated performance and win-shift curves (see Fig. 5.4), there is a marked improvement when α is allowed to change between groups of sessions compared to a fixed parameter set model for both behavioural measurements (p < 0.0001) as is also the case for the model with β free (p < 0.0009). In addition, the model with a variable learning rate has significantly smaller MSE when fitting the performance curves than the model with β free, suggesting that this might be a better model. This resonates with the fact that the improvement in performance is visible from the very first trials and not limited to final performance levels (see reminder of how variations of the learning rate cause different performance profiles than variations of the inverse temperature). It must be noted, however, that even in the case of a free learning rate, there are significant main and interaction effects of simulations when comparing experimental versus simulated sets of data with a four-factor (simulations, sessions, trials, and risk) repeated-measures ANOVA.

Meanwhile, if we take a look at parameter variations underlying these improvements in model fit shown in Fig. 5.5, we find that these also follow long-term trends as there is a significant session effect on α (non-parametric Friedman ANOVA: χ 2 (3) = 26.7, p < 0.0001) and on β (χ 2 (3) = 13.2, p = 0.0043). Post hoc Bonferroni tests show that α has increased significantly between the first group of sessions and the other three (p < 0.0020) and that there is also a significant increase of β between the first group of sessions and the second and fourth groups (p < 0.0152). It seems then that the improvement in performance can be explained either by a faster updating of Q-values or by an increased tendency to exploit the best action, although model comparison favours the first hypothesis. These conclusions should be treated cautiously, given the relatively poor fit of simulations to data and the known interdependence between learning rate and inverse temperature which means that variations of one parameter can be compensated to a relative degree by variations of the other [START_REF] Daw | Model-based influences on humans' choices and striatal prediction errors[END_REF]. In fact, if I allow all parameters to be free between sessions, we find a decreasing rather than an increasing tendency for β (not shown). In any case, allowing parameter variations is a promising approach to explaining the long term changes in behaviour and in the following paragraphs I will expound on these results by proposing two classes of meta-learning models characterised by a regulation of either α or β. These models will also be confronted with a recently proposed model of directed exploration.

Mechanistic models

In order to find an explanation for the long-term changes in behaviour, I confronted three different classes of models representing different possible strategies. We just saw how changing the learning rate and/or the inverse temperature was a promising idea, but the first class of models I tested are models of directed exploration which have generated a lot of interest in the recent literature [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF]) and are typically based on the notion that actions which are uncertain are more interesting to try than well-tested actions; these more interesting actions are therefore given an uncertainty bonus [START_REF] Daw | Cortical substrates for exploratory decisions in humans[END_REF]; [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF]). Because of the recent surge in interest for these models [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF][START_REF] Cogliati Dezza | Learning the value of information and reward over time when solving explorationexploitation problems[END_REF][START_REF] Zajkowski | A causal role for right frontopolar cortex in directed, but not random, exploration[END_REF], I decided to test a few of these alongside the two other classes of models which are meta-learning models adjusting either the learning rate or the inverse temperature based on different average signals. In the following sections, I will describe these three sets of models in more detail before confronting them using metrics of their predictive and generative abilities.

Model descriptions

Models of directed exploration

There are two sorts of uncertainties when considering an action:

• subjective uncertainty which arises from the scarcity of information acquired from a certain action, e.g. if an action is selected rarely, an action can be interesting by virtue of not having been tested very often. This is the point of view adopted by the work of [START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF][START_REF] Cogliati Dezza | Should We Control ? The Interplay Between Cognitive Control and Information Integration in the Resolution of the Exploration-Exploitation Dilemma[END_REF]; [START_REF] Zajkowski | A causal role for right frontopolar cortex in directed, but not random, exploration[END_REF] in which a bonus is directly applied to actions which have been sampled less often.

• the underlying variability in outcome of a given action. This sort of uncertainty is the one dealt with in the studies of risk aversion or risk attractiveness and consists in the idea that actions, which are possibly well-known, but whose reward outcome variance is large are, all other things equal, more (or less) appealing than actions which are more certain [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF].

These two types of uncertainties are tightly intertwined since the objective uncertainty of an action must be learned which supposes a certain level of sampling which will affect subjective uncertainty, but they are not completely reducible: an action can have been sampled repeatedly to the point that its reward distribution is perfectly well-known and yet retain objective uncertainty from the fact that the reward outcome is probabilistic instead of deterministic. In looking for a model of directed exploration, I adopted a model representative of the second tendency in which the agent learns a representation of the expected reward rate combined with the variance of rewards: E(a i ) + φσ 2 (a i ) with φ a parameter determining how attractive uncertainty is. To achieve this, the agent estimates the expected reward rate E(a i ) using a classical Q-learning algorithm accumulating reward prediction errors δ t , and estimates the variance σ 2 (a i ) by accumulating uncertainty prediction errors ξ t . If the action i was selected at trial t, we get:

ξ t = δ 2 t -σ 2 (a i ) (5.1)
This uncertainty prediction error is then used to update the current estimate of the variance in a similar manner to what is done with reward prediction errors and Q-values:

σ 2 (a i ) ← σ 2 (a i ) + α φ ξ t (5.2)
with α φ the specific learning rate for estimating expected uncertainty. The Q-value and expected uncertainty are then combined and plugged into the softmax selection function:

P (a t+1 = a i ) = e β(Q(a i )+φσ 2 (a i )) j e β(Q(a j )+φσ 2 (a j )) (5.3)
In an extension of this model, [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF] proposed that the Q-values also be modulated by the unexpected uncertainty so that Q(a i ) + φσ 2 (a i ) + ξ i,t is plugged into the softmax but the relatively poor performance of this alternative which I also tested convinced me to focus on the simpler version of this model which I will henceforth refer to as the DE (Directed Exploration) model. Finally, given that forgetting significantly improves the fit of simulations to experimental data, I also added this mechanism to the Q-values (but not to expected uncertainties σ 2 (a i )).

Meta-learning on β models

The DE model is opposed to so-called random exploration which is computationally simpler and consists in controlling the relationship between learning and decisionmaking. For example, if we take an ε-greedy decision rule in which the best action is selected with probability (1 -ε) while the remaining actions are chosen with probability ε (N -1) with N the total number of actions, then the level of random exploration is directly given by the value of ε which when low favours exploitation and when high favours equiprobability of action selection. Similarly, when using the softmax decision-making mechanism, the value of β determines how much influence the ranking of the Q-values will exert on final decision-making. While the DE model allows a finer degree of control on exploration, random exploration is probably simpler to implement as it is not based on any additional information besides action values. For this reason, and because I suspect the between-session improvements to be at least partly due to an effect on exploration, it is particularly interesting to try and confront these two hypotheses directly. Furthermore, having previously shown in Chap. 4 that dopamine inhibition increases exploration without impacting the learning rate, regulation of the inverse temperature is a very promising avenue of inquiry. In the following models, I test the idea that the long-term changes in behaviour are to be accounted for by a regulation of random exploration using different methods which are all based on the calculation of a running average, called η, of a particular quantity which can be:

• rewards as a reflection of current performance for the Reward-based Meta-Learning on β (R-ML-β) model. The more rewards the agent accumulates the more likely he is to have identified the correct lever making exploration a pointless costly strategy. Conversely, when the animal is hitting a rough patch, he should probably explore to try and relocate the correct action:

η ← η + α η (r t -η) (5.4)
• reward prediction errors for the RPE-based Meta-Learning on β (RPE-ML-β) model. The idea behind this mechanism derives directly from the finding in Chap. 4 that dopamine inhibition increased exploration. Tonic dopamine, the background concentration of dopamine, which [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF] and [START_REF] Niv | Tonic dopamine: Opportunity costs and the control of response vigor[END_REF] hypothesise as responsible for controlling exploration, could perhaps correspond to a running average of phasic activity [START_REF] Mcclure | A computational substrate for incentive salience[END_REF], which encodes reward prediction errors:

η ← η + α η (δ t -η) (5.5)
• squared reward prediction errors as an estimate of global uncertainty for the Uncertainty-based Meta-Learning on β (U-ML-β) model. When the agent experiences large prediction errors (positive or negative) denoting high global undertainty, he should explore, and conversely focus on the best option if he experiences low global uncertainty:

η ← η + α η (δ 2 t -η) (5.6)
Once updated, η is used to modulate β for the following trial. Different forms of modulation are possible; for example [START_REF] Khamassi | Robot cognitive control with a neurophysiologically inspired reinforcement learning model[END_REF] used a sigmoid function but this would be a relatively complex model with three additional parameters, so I settled for a simple linear function:

β ← β 0 + (β 1 -β 0 )η (5.7)
In total, these different models have six parameters: the learning rate α, the forgetting rate α 2 , α η , β 0 and β 1 .

Meta-learning on α models

As previously shown in section 5.3, the between session changes in behaviour can be captured by a series of independent Q-learning models for which the varying parameters are either the learning rate, the inverse temperature (Fig. 5.5), or perhaps even a combination of the two. In fact, when looking at the progress in performance and win-shift curves, the between-session trends more closely resemble an increase in learning rate than one in exploitation as illustrated in Fig. 4.6, with an increase in the slope of performance curve but no apparent effect on the final convergence values. This motivated me into testing α-metalearning models despite hoping to find a meta-learning on β instead, given the results previously described in Chap. 4. Similar to what was tested with the inverse temperature, three metalearning on α models were based on calculating the running average η of either rewards (R-ML-α), reward prediction errors (RPE-ML-α) or squared reward prediction errors (U-ML-α) and then using that average to adjust α with a simple linear function.

α ← α 0 + (α 1 -α 0 )η (5.8)
In all cases, in the course of a single trial, the regulation of the learning rate was the first step followed by the updating of the Q-values using these newly updated learning rates. I also tested the alternative (updating Q-values first then updating the learning rate for the next trial) for some of these models and found practically the same results (not shown here). This problem of step order does not present itself for the inverse temperature, since the animal first selects an action, then observes its result before it can update the β parameter which is then necessarily applied to the next trial.

Model comparison

Model optimisations

I followed the same methodology as in Chap. 4 and in section 5.3 to optimise each model. Briefly, models were optimised separately for each rat by maximisation of the log-likelihood using standard gradient descent, implemented by the fmincon function of MATLAB. Three different initial points were used for each parameter to have enough coverage of the parameter space to avoid falling into local maxima, meaning that for n parameters there was a total of 3 n different initialisations for each individual.

Comparison of the models' predictive abilities

A first comparison method is to compare the distributions of individual loglikelihood scores (which will be equivalent to the AIC and BIC scores since the same number of parameters and trials is used for all models). Using right-tailed paired t-tests to test the hypothesis that the mean log-likelihood score of a first model is greater than that of another (Table 5.1), it is apparent that U-ML-α is significantly worse than all others (p < 0.02) and that its counterpart U-ML-β is itself worse than the two other meta-learning on β models (p < 0.02). More importantly, the DE model has significantly better log-likelihood scores than all other models except for R-ML-β (p only just equal to 0.05) especially when compared to the meta-learning on α models. In summary, some differences in likelihood scores are detectable statistically and favour the DE model. However, these differences are generally quite modest in size as evidenced when looking at supplementary Fig. B.1 which plots the individual AIC scores of these models and other variants not discussed here and which shows no clear difference between models except when comparing models with or without the forgetting mechanism.

R-ML-β RPE-ML-β U-ML-β R-ML-α RPE-ML-α U-ML-α DE R-ML-β 0.

Comparison of the models' generative abilities

Meta-learning vs. directed exploration A more important criterion of selection when comparing models is what the work of [START_REF] Palminteri | The Importance of Falsification in Computational Cognitive Modeling[END_REF] calls the 'generative' capability, that is the ability when simulated to produce similar behaviour to the original experiment. In order to carry out this comparison, I simulated 100 agents per rat and then averaged performance, win-shift and loseshift of each model in early (sessions 1-8), middle (sessions 9-16) and late (sessions 19-24) sessions. Because simulations of the various meta-learning models were practically indistinguishable, I shall restrict my analysis to the R-ML-β, R-ML-α and DE models. Average curves of the meta-learning models are shown in Fig. 5.6 and 5.7 while those of the DE model are represented in Fig. 5.8. As evidenced from these curves, despite having significantly better optimisation scores, the DE model crucially fails to replicate the effect we are most interested in, namely the between-session improvement in performance, but it does strangely present the desired decrease in win-shift. In contrast, both meta-learning models are capable of replicating both of these key characteristics of the experimental data. From the point of view of the generative abilities of these models, it thus seems that a metalearning process on either random exploration or learning rate is a more fruitful approach than an explanation through directed exploration. However, this does not rule out this process altogether, as it is possible that it captures other aspects of the data which are not immediately visible in these inter-session averages.

Meta-learning on α vs. β The observation of the long-term trends in behaviour successfully rules out the directed exploration models, without favouring either meta-learning model. To try and solve this dilemma, a good idea is to look at the models' reaction after an unexpected reward, that is a reward on a lever which is not the current block target. In the meta-learning on β models, an unexpected reward should cause an increase in β leading to an increased tendency to exploit and a shift back to the target despite the reward. Unfortunately, this effect is partly counterbalanced by the increased value of the chosen action and the decreased value of the other action due to the learning and forgetting mechanisms. By contrast, if an unexpected reward is given to a meta-learning on α model, this will cause an increase in learning rate and should enhance the tendency to stay on this same choice. To test this idea, I looked at the win-stay probabilities after being rewarded on either of the non-target levers in the last twelve trials of blocks, in order to give animals time to learn which is the target lever. Since we might expect the target lever to be more strongly differentiated in low risk than high risk blocks, average win-stay was calculated separately for these two types of blocks (see Fig. 5.9). In fact, there was no discernible differences between experimental data and simulations of either one of the two meta-learning models, and Friedman ANOVA tests were negative in both low (p = 0.88) and high risk blocks (p = 0.58). In summary, I was unable to find any experimental support to distinguish between the two possible meta-learning models.

Discussion

In this chapter, I have attempted to find evidence of meta-learning abilities in Rats by looking at long-term changes in behaviour. A simple approach consisting in separate optimisation of a forgetting Q-learning model for different stages of the experiment showed that there were indeed changes in parameter values, namely increases in the learning rate and in the inverse temperature leading to increased sensitivity to recent trial outcomes and increased exploitation. I then moved on to propose different meta-learning models based on the regulation of either of these parameters by a variety of different feedback signals, corresponding to current performance levels or convergence of learning, and pitted these models against a model of directed exploration which gives an uncertainty bonus to actions with surprising outcomes. Unfortunately, I was unable to discriminate between the different meta-learning models which all proved equally capable of replicating the long-term changes in behaviour, but was able to reject directed exploration as a possible explanation for this effect. However, the fact that this category of models presented better likelihood, AIC and BIC scores in spite of its poor performance from a simu- lation viewpoint, may serve as a warning for careful modelling which should always include a process of simulation as a validation step. Of course, this result does not invalidate directed exploration in itself, but simply points to its inability to explaining long-term changes which could apparently be dependent on regulation of random exploration.

Long-term improvements of behaviour and meta-learning

An important weakness in this study is the nature of the task which is not particularly suited to the question at hand. Indeed, because of the regularity of the task structure, with its blocks of fixed length and its limited risk settings (low or high with fixed probabilities), it is doubtful that meta-learning is a particularly crucial mechanism for solving the task. In this present experiment, the animals only really need to tune their parameters in the early stages of the task. In fact, when I optimised forgetting Q-learning models separately for different stages of the experiment, as reported in section 5.3, the only significant difference in parameter ranking concerns the comparison between early stages and later ones, with no further evolution detected. This means that when looking for a potential regulatory signal such as accumulated reward, time itself is a confounding factor. There is a slight element of variability in the task in the way different levels of risk or uncertainty alternate but I was unable to detect any indication of meta-learning from the raw data. However, when adopting the same simple strategy as in Chap. 4 and section 5.3, I optimised a model with different sets of parameters for high risk and low risk blocks respectively, I did manage to find a significant drop in the values of the learning rate from high to low risk blocks and no difference on β (see Fig. 5.10), but the magnitude of the effect is very small in any case. This promising result calls for future experiments with a more variable structure to properly disentangle the different potential mechanisms. An example for such a task is to be found in the work of [START_REF] Behrens | Learning the value of information in an uncertain world[END_REF] which included periods of different reward volatility and successfully demonstrated the adaptation of the learning rate. the other parameter was exclusive, but I do not think there is any good reason for this to be the case. My intent in using this approach was to find the simplest possible model to account for the data, and it seems that regulation of the learning rate has a slight edge here. A more elaborate model with regulation of both parameters was in fact tested but did not lead to improved fitting of the simulations which is why I chose not to include it here.

Introduction

Individual differences in response to Conditioned Stimuli (CS) have elicited much interest in the recent years as a model of differential susceptibility to drug addiction [START_REF] Saunders | Individual variation in resisting temptation: Implications for addiction[END_REF]. In a Pavlovian appetitive task where a CS cue invariably predicts the occurrence of a biologically relevant event or Unconditioned Stimulus (US) such as a food reward, animals could be expected to direct their response towards the location of food delivery (goal-tracking or GT), but a sub-population of subjects instead focus on the cue itself, as if it had acquired incentive properties similar to those of the reward [START_REF] Meyer | Quantifying individual variation in the propensity to attribute incentive salience to reward cues[END_REF]. The latter form of behaviour, termed sign-tracking (ST), has been reported in several species, including Pigeons [START_REF] Jenkins | The form of the auto-shaped response with food or water reinforcers[END_REF] as well as Rats [START_REF] Robinson | Dissociating the Predictive and Incentive Motivational Properties of Reward-Related Cues Through the Study of Individual Differences[END_REF] in a Pavlovian task where the CS is the presentation of an inactive lever. Sign-tracking is thought to be a stable trait of individuals (Sign-trackers, STs) that are more prone to display an automatic behavior towards reward-predicting cues, in the sense that the same animals may be less sensitive to extinction of conditioning Approach Task [START_REF] Ahrens | Rats that sign-track are resistant to Pavlovian but not instrumental extinction[END_REF] or to devaluation of the reward [START_REF] Morrison | Sign tracking, but not goal tracking, is resistant to outcome devaluation[END_REF][START_REF] Nasser | Individual variability in behavioral flexibility predicts sign-tracking tendency[END_REF][START_REF] Patitucci | The origins of individual differences in how learning is expressed in rats: A general-process perspective[END_REF] than individuals exhibiting goal-tracking (known as Goal-trackers, or GTs) but see [START_REF] Derman | Signtracking is an expectancy-mediated behavior that relies on prediction error mechanisms[END_REF]. Many interpretations of sign-tracking have been proposed, including potentiation of orienting responses dependent on CS-US pairing [START_REF] Holland | CS-US interval as a determinant of the form of Pavlovian appetitive conditioned responses[END_REF], or instrumental conditioning due to adventitious response-reinforcer associations [START_REF] Davey | Autoshaping in the rat: Effects of omission on the form of the response[END_REF]. Nevertheless, sign-tracking has been more recently construed as the acquisition of incentive motivation by the conditioned cue [START_REF] Berridge | From prediction error to incentive salience: mesolimbic computation of reward motivation[END_REF]. One of the arguments in favour of this interpretation is that the acquisition of sign-tracking, but not goal-tracking, is dependent upon dopaminergic signalling, specifically in the core of the nucleus accumbens [START_REF] Saunders | The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses[END_REF][START_REF] Scülfort | Dopamine antagonism does not impair learning of Pavlovian conditioned approach to manipulable or nonmanipulable cues but biases responding towards goal tracking[END_REF][START_REF] Fraser | Long-lasting contribution of dopamine in the nucleus accumbens core, but not dorsal lateral striatum, to sign-tracking[END_REF][START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF]. Moreover, goal-tracking and sign-tracking are associated with distinctive patterns of dopamine signalling in this brain region [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF].

Several models have been proposed to explain the motivational role of dopamine during acquisition and/or expression of sign-tracking [START_REF] Zhang | A neural computational model of incentive salience[END_REF][START_REF] Kaveri | Dual reward prediction components yield Pavlovian sign-and goal-tracking[END_REF][START_REF] Anselme | Incentive salience attribution under reward uncertainty: A Pavlovian model[END_REF], but few of them account for individual differences in the form of conditioned responses, namely the existence of separate populations of STs and GTs [START_REF] Kaveri | Dual reward prediction components yield Pavlovian sign-and goal-tracking[END_REF]. One of the hypotheses proposed is the existence of parallel learning processes relying on different error signals and providing different degrees of behavioural flexibility. In the present chapter, I examine the ability of one such model (Lesaint et al., 2014a,b) to replicate new experimental findings [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] from an experiment specially designed to test predictions the model had made in [START_REF] Lesaint | Experimental predictions drawn from a computational model of sign-trackers and goal-trackers[END_REF].

Description of the original experiment and of the original FMF-MB model

The model proposed by Lesaint et al. (2014a) was aimed at explaining a Pavlovian conditioning experiment designed by [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF] in which the CS was the 8s-presentation of a retractable lever which would be immediately followed by the delivery of a food pellet US in a nearby magazine. Computationally, the structure of this task is represented by a Markovian Decision Process (MDP) consisting of seven different states (Fig. 6.1) defined by the environmental conditions, such as the presence of the lever or of the food, and the current position of the animal (close to the food tray or to the lever). There are six different actions (explore the environment or goE, approach the lever or goL, approach the magazine or goM, wait, engage the closest stimulus and eat the reward), and state transitions given a selected action are deterministic. The decision-making model itself (Fig. 6.2) consists of a Model-Based (MB) and of a Feature-Model-Free (FMF) learning system which output respectively an advantage function and a value function for each possible action in the current state. These two functions are then combined into a weighted mean defined by the ω parameter representing how much each system contributes to final behaviour. A high ω gives more importance to the value function computed by the FMF system, while a low ω favours the MB system. Finally, the weighted averages are given to a softmax function representing the action selection mechanism (Fig. 6.

Description of the original experiment and of the original FMF-MB model 101

2).

The originality of the FMF system compared to other reinforcement learning algorithms is to assign a value representing future expected reward not to states or actions but to the features which the different actions are focused on. This allows for a generalization of values between different states, e.g. when the animal goes towards the magazine in state 1 or engages the magazine in state 4, it is the same feature value V(M) which is called upon. The feature which the animal's current action is aimed at is defined by the feature-function f (see Table 6.1). After each action, observing the delivery of the reward r (equal to 1 or 0 if delivery takes place or not, respectively) and the new state s t+1 , the animal updates the value the last action was aimed at using the following rules:

δ = r + γ max j (V (f (s t+1 , a j ))) -V (f (s t , a t )) (6.1) V (f (s t , a t )) ← V (f (s t , a t )) + αδ (6.2)
We do this with all features (E: environment, L: lever, M: magazine), except for food, F, which is constantly equal to 1, the value of the reward. The initial formulation in Lesaint et al. (2014a) made the hypothesis that the value of F also had to be learned but this leads to an artefact in theoretical dopaminergic activity (see Fig. 7C of Lesaint et al. (2014a)): instead of decreasing from the very first session, RPEs in the model spike in session 2 before decreasing, as initially the value of food is also small compared to that of the previous feature; by fixing the value of F, this artifact disappears. This could mean that the value of the sensory features of the food reward do not need to be learned or that they are learned almost immediately after just a few encounters contrary to other features, something which has already been suggested by [START_REF] Rescorla | A theory of pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement[END_REF] which proposes that the learning rate for different cues depends on their individual characteristics (see Eq. 3.2).

In addition, because the animal is likely to visit the magazine and explore the environment during the ITI, the values of these features are revised between states 7 and 0:

102 Modelling of Individual Variability in the Pavlovian Conditioned Approach Task

V (M ) ← (1 -u IT I ) × V (M ) V (E) ← (1 -u IT I ) × V (E) (6.3)
with u IT I the ITI update factor ranging between 0 and 1, which, crucially, is dependent on the duration of the ITI; the longer this duration, the more opportunities the animal has to negatively revise these values, which is why I modelled two different ITI durations by using a small (0.01) and large (0.1) value of this update factor. Meanwhile, the MB system relies on learned transition T and reward R functions which have the following learning rules:

T (s t , a t , s j ) ←    (1 -α) × T (s t , a, s j ) + α if s t+1 = s j (1 -α) × T (s t , a, s j ) otherwise (6.4) R(s t , a t ) ← R(s t , a t ) + α(r -R(s t , a t )) (6.5)
The initial values of T and R are set to 0. Using these functions, the agent computes an action-value function for each possible action in the current state:

Q(s t , a i ) = R(s t , a i ) + γ j (T (s t , a i , s j ) max k (Q(s j , a k )) (6.6)
Finally, these Q-values are compared to each other so as to compute the Advantage function A:

A(s t , a i ) = Q(s t , a i ) -max j Q(s t , a j ) (6.7)
Once the FMF and MB systems have outputted the feature values and the advantages of the possible actions, these are integrated into a weighted sum:

P (s t , a i ) = (1 -ω)A(s t , a i ) + ωV (f (s t , a i )) (6.8)
with ω the integration factor comprised between 0 and 1, which sets the relative importance of each system in determining behaviour. These integrated values are then plugged into a softmax function to compute the probability of selecting each action:

p(a t = a i ) = e βP (st,a i ) j e βP (st,a j ) (6.9) with β the inverse temperature. In total this model has five parameters: the learning rate α, the discounting factor γ, the inverse temperature β, the ITI update factor u IT I , and the integration factor ω. Note that the learning rate and discounting factor are shared by the MB and FMF systems despite the fact that they could very conceivably be different in reality. Originally, this was done to simplify the optimization of this model on experimental data (Lesaint et al., 2014a)

Predicted effects of ITI duration on the FMF-MB model

I first set out to verify the claim made in Lesaint et al. (2014a) that by decreasing the ITI duration, goal-tracking would be favoured. This prediction, which was made without model simulations, is based on the fact that for the FMF module, the lever is preferable to the food cup because the latter is present during the ITI and consequently subject to down-revision of its value as the rat visits it without receiving a reward. Hence, if we reduce the duration of ITI and allow less time to visit the unrewarded food cup, the feature value of the food cup should be higher and the advantage of the lever should decrease.

To illustrate this, I present simulations of a model with only the FMF or the MB system being used to determine action by setting ω to either 1 (FMF system alone) or 0 (MB system only) while keeping other parameters constant, and with two different values of the ITI update factor. I plotted the average number of ST and GT choices between sessions in Fig. 6.3 a.,b.,e., and f.. As predicted, when food cup value revision is low, as should be the case during short ITIs, the FMF-only model shows a definite increase in goal-tracking, although sign-tracking is clearly still dominant. If I decompose action probabilities into their FMF and MB contributions in Fig. 6.3c., we can verify that there is indeed an increase in the feature value of the food cup, and also an increase in the model-based advantage of going towards the magazine which is probably due to the fact that calculation of the transition function is biased by the animal's choices and if the animal preferentially sign-tracks he will learn the transition from state 1 to state 3 with more difficulty. These differences straightforwardly impact the downstream outputs of the softmax function as evidenced in Fig. 6 Lesaint et al. (2014a). There are six possible actions leading deterministically from one state to the next: exploring the environment (goE), approaching the lever (goL), approaching the magazine (goM), waiting, engaging with the closest stimulus and eating the reward. Each of these actions focuses on a specific feature indicated in brackets: the environment (E), the lever (L), the magazine (M) and the food (F). These are the features used by the FMF learning component. The red path corresponds to sign-tracking behaviour and the blue path to goal-tracking behaviour. b. Corresponding timeline of lever and food appearances.

MB FMF

ω (1-ω) T A R Integration Action selection V ( f 3 ) V ( f 2 ) V ( f 1 ) A ( a 1 ) A ( a 2 )
A ( a 3 ) P(a 1 ) P(a 2 ) P(a 3 ) a t V δ + Figure 6.2: Schematic representation of the FMF-MB decision-making model adapted from Lesaint et al. (2014a). The model combines a Model-Based learning system which learns the structure of the MDP and then calculates the relative advantage of each action in a given state, with a Feature-Model-Free system which attributes a value to different features of the environment which is generalized across states (e.g. the value of the magazine is used both in states 1 and 4). The advantage function and value function are weighted by ω, their relative importance determining the sign-vs goal-tracking tendency of the individual and then passed to the action selection mechanism modelled by a softmax function. Approach Task (Fig. 6.3 e.,f.), despite the fact that there is a large significant change in the FMF value of the food cup (Fig. 6.3 g.). In parallel, the advantage function remains unchanged. It is interesting to note how in the particular case of the FMF-only model, the change in ITI duration affect both the FMF and MB system, while only the FMF system is affected in the MB-only model. This is because, when the FMF system controls behaviour, it will constantly bias action in a particular direction and thus affect the transition function. This highlights how deceptively complex the interaction between the two systems is, and how the dominant system can affect calculations of the second system. From these simple simulations, we can conclude that the initial claim is indeed correct: decreasing food cup value down-revision should favour goal-tracking. However, it is important to point out that this effect is dependent on the FMF system only: an agent using only or predominantly MB learning will not show this effect, and as such it does not constitute hard evidence for the combination of FMF and MB learning.

If we take the opposite perspective of increasing food cup value down-revision by increasing the duration of ITI, and suppose that we are dealing with a mixed population of goal-trackers and sign-trackers, the main prediction of the model is that there should be an increase in sign-tracking behaviour on average over the entire population of animals. Importantly, this prediction has been verified by [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF]. Now we can look deeper into what are the ingredients that compose this computational prediction, something which had not been done before. Fig. 6.3 a.-b. shows that agents simulated with a FMF-only model do sharply increase their sign-tracking tendency in the long ITI condition. Strikingly, simulations with a MBonly model show that there is no impact of the increase of ITI duration on the extent to which simulated agents approach the food cup or the lever. In other words, this predicts that animals exclusively or mostly reliant on model-based learning should be insensitive to food cup down-revision and keep goal-tracking, meaning that a more or less significant portion of the population should goal-track no matter what. Note that in the original STGT model aimed at accounting for the experimental results of [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF], there were no pure MB agents, but simulated goaltrackers had a very small parameter ω < 0.05, which means that the FMF system was contributing to less than 5% of their behaviour (Lesaint et al., 2014a[START_REF] Lesaint | Experimental predictions drawn from a computational model of sign-trackers and goal-trackers[END_REF] and that any impact on this system should be small. Interestingly, while the study by [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] shows that at least a few individuals can be categorised as goaltrackers in the long ITI condition, at the level of the behaviour of the population the average goal-tracking behaviour seems quite weak compared to sign-tracking behaviour. Moreover, the percentage of contacts with the food cup during the CS period (i.e. goal-tracking behaviour) appears to progressively decrease sessions after sessions in the long ITI condition. In computational terms, this could either mean that there are too few animals using a predominantly MB strategy or that the influence of the MB system is unstable, similarly to its previously reported decline over long training in an instrumental conditioning paradigm, as goal-directed behaviour is replaced by habitual behaviour [START_REF] Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF]. In the long ITI condition, simulated rats increasingly approach the lever and avoid the magazine, while the opposite holds for the short ITI group (Fig. 6.4 b. and c.). Because I kept the same parameters in the long and short ITI conditions, I could apply a repeated-measures ANOVA with two within factors corresponding to sessions and ITI conditions (instead of a mixed-design ANOVA with ITI as a between factor) on the proportion of trials where the simulated animal approached the lever and the magazine. I found very significant effects of sessions, ITI duration, and of their interaction on both proportions (p > 0.0001). Post hoc t-tests comparing the number of trials where the lever was chosen in each session produced significant differences for every session except the very first (smallest mean difference = 6.75 +/-0.84, p < 0.0001). Conversely, the number of times the magazine was approached was significantly higher in the short ITI group for every session, including the first (smallest mean difference = 2.55 +/-0.82, p < 0.0059). In summary, shortening the ITI produces an increase in goal-tracking choices. [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] found similar results but preferred to report them through the distributions of behavioural session scores which included the response bias, the difference in probability of approach to the lever and the food cup, a latency of response index, and a Pavlovian Conditioning Approach (PCA) score [START_REF] Meyer | Quantifying individual variation in the propensity to attribute incentive salience to reward cues[END_REF] which consists in the average of the three other scores. These different indices are designed to range between -1 and 1 such that sign-tracking behaviour corresponds to scores closer to 1 and vice-versa for goal-tracking. These indices also allow for the possibility that an animal will interact repeatedly with one or both of the stimuli. Using these metrics, the experimenters found a significant increase in the tendency to sign-track when comparing all four score distributions of the short and long ITI groups. Unfortunately, the original model Lesaint et al. (2014a) allows only for a single interaction with only one of the stimuli during the CS period of a trial, and does not attempt to model the latencies of responses either. This means that the only index at our disposal is the normalised difference in the probability of approach: P CA = P (goL) -P (goM ) P (goL) + P (goM ) (6.10)

Replication of dopaminergic patterns found by Lee et al. (2018) 109

The normalisation was necessary given that the model also has the possibility to explore instead of interacting with either cue. Given that the model provides direct access to the probability of each action at each trial through the softmax function, we used these probabilities (Fig. 6.4 d.) as well as the frequency of different choices made in the simulations (Fig. 6.4 e.), which is closer to the experimental methodology. The distributions of these two scores are significantly biased towards goal-tracking for the short ITI group (Wilcoxon's signed rank test: µ < -0.74, p < 0.0001); to the contrary, the distributions corresponding to the long ITI group are significantly biased towards sign-tracking (Wilcoxon's signed rank test: µ > 0.51, p < 0.0001). Direct comparison of the short and long ITI groups also produced significant differences for both scores (Wilcoxon's signed rank test for long minus short ITI: µ > 1.26, p < 0.0001). If we look for the origin of this effect by examining the average feature values and advantages in Fig. 6.4 f. top, we see that shortening the ITI has a significant positive effect on both the feature value and the advantage of going towards the magazine (Welch's t-test: p < 0.0001) and conversely a significant negative effect on the feature value and advantage of going towards the lever (Welch's t-test: p < 0.0001). These effects directly translate into a significant increase in the probability of going towards the magazine (Fig. 6.4 f. bottom, Welch's t-test: t(27.6) = 28.12, p < 0.0001) and of avoiding the lever (Welch's t-test: t(27.7) = -25.44, p < 0.0001). In conclusion, shortening the ITI causes an increase in both magazine value and advantage of going towards the magazine which both result in promoting goal-tracking, as found experimentally by [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF].

Replication of dopaminergic patterns found by Lee et al. (2018)

The study by [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] also included measurements of dopamine release in the Nucleus Accumbens core (NAc) using fast scan cyclic voltammetry. These measurements found bursts of dopaminergic activity hypothesized to represent the RPEs computed by the FMF learning system. A first result is that, after averaging over all sessions, dopamine release at the time of the CS was significantly greater for the long ITI group than for the short ITI group. This result had not been explored by previous model simulations, but it is in fact predicted by the model (Fig. 6.5 a., Welch's t-test: t(25.68) = -43.36, p < 0.0001). Indeed, the RPE computed by the model at CS presentation is the RPE corresponding to transition from state 0 to state 1:

δ 0 = γ × max(V (M ), V (L)) -V (E) (6.11)
Additionally, although it was only implied in the original paper by Lesaint et al. (2014a) but not discussed in [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF], the value of the environment is also revised during the ITI (see Eq. 6.3), meaning that it is smaller after a long ITI than after a short ITI. This increased activity simply reflects the fact that in Approach Task a. Approach to the lever for different ITI durations (mean +/-s.e.m.). c. Approach to the food cup for different ITI durations (mean +/-s.e.m.). d. Distribution of differences in softmax probability of approach to lever and magazine for the two ITI durations. There is a significant bias towards goal-tracking choices in the short ITI group and a significant bias towards sign-tracking choices in the long ITI group. e. Distribution of differences in average simulated number of approaches to lever and magazine for the two ITI durations. As expected from the differences in softmax probabilities, there is a significantly higher number of goal-tracking than sign-tracking trials in the short ITI condition and vice-versa in the long ITI condition. f. Top: Effect of down-revision of food cup value during ITI of different durations on average FMF-values and MB action advantages. Bottom: Average softmax probabilities of engaging with either the lever or the food cup during the CS period for different ITI durations.

Replication of dopaminergic patterns found by Lee et al. (2018) 111

a setting where rewards are less frequent, signals that predict rewards will cause more positive surprise.

On the other hand, the authors report that dopamine activity at US delivery is greater in the long ITI group than in the short ITI group and they expect this to be predicted by the model on the basis that this activity would correspond to the discrepancy between the reward and the value of the magazine which is smaller for the long ITI group. However, the original MDP is designed in such a way (Lesaint et al., 2014a) that this discrepancy is not necessarily calculated on each trial because the value the reward is compared to is not always that of the food cup. Indeed, as shown in Fig. 6.1a., if the animal followed a sign-tracking strategy, then the RPE between states 2 and 5 which corresponds to the arrival of the US is:

δ 2 = γ × V (F ) -V (L) (6.12)
because the model hypothesises that when the rat goes to the magazine in state 5 (which is the only available action), the feature it is comparing the reward to is the one it was previously focusing on, i.e. the lever and not the food cup (see Table 6.1). It might be tempting to propose an alternative feature function in which the animal focuses on the food cup when going from state 5 to 7, but this would make the value of the lever dependent on the value of the magazine rather than the food, meaning that increased ITI duration would affect both lever and magazine values. Alternatively, we could say that the RPE at US delivery should be measured when the reward is actually eaten, that is the RPE generated between states 7 and 0. However, in this case the RPE is:

δ 7 = r -V (F ) = 0 (6.13)
For this reason, I applied a simple correction to the feature function by supposing that the feature the animal focuses on when it eats is not the food but the food cup. In this case, we do indeed get an RPE equal to the difference between reward and value of the magazine at the cost of losing "real-world" significance of the action-feature function. However, a significant advantage of this new formulation is that it allows the updating of the food cup value even in trials during which the animal sign-tracked. Indeed, under the previous conventions, if the animal signtracked it would never perform an action focused on the food cup feature and the corresponding value would remain the same despite the fact that the reward really is retrieved from the food cup. As a consequence, there is an increased tendency to goal-track using this convention compared to the previous version of the model (comparison not shown here). Using this new model convention, RPEs at the US are significantly greater in the long ITI group than in the short ITI group (Fig. 6.5 a., Welch's t-test: t(21.80) = -252.87, p < 0.0001), a result which is consistent with [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF].

As learning progresses and the feature values change, the RPEs should evolve between early and late sessions. [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] that in the long ITI group, dopamine release to lever presentation was significantly greater in late sessions than in early sessions, reflecting the increase in lever value, while dopamine activity at reward delivery remains important due to the positive surprise of receiving reward from a low-value magazine. In simulations of the long ITI group (Fig. 6.5 b.), there is indeed a very significant increase in average RPE value between early and late sessions at CS presentation (paired t-test: t(19) = -56.20, p < 0.0001), but also a very significant decrease at US delivery (paired t-test: t(19) = 65.86, p < 0.0001). Although highly significant, this decrease is modest in size (Fig. 6.5 b.) which might contribute to explaining why it is not detected experimentally. Additionally, there is little variability between rats in the simulations (as evidenced from the standard errors of the mean in Fig. 6.5) probably because they share the same parameters except for ω, which increases the likelihood of detecting such small effects.

Concerning the short ITI group, the experimental results show that dopamine release during the CS period did not significantly change, while there was a significant decrease during the US period. Similarly in my simulations (Fig. 6.5 c.) I found a significant decrease in RPEs during the US period (paired t-test: t(19) = 78.55, p < 0.0001), but there is also a small but significant decrease at CS presentation (paired t-test: t(19) = 9.66, p < 0.0001). It is interesting to note that the simulations point to a reversal in DA activity at CS time between short and long ITI groups, with rats in the long ITI group seeing a significant increase and rats in the short ITI group a significant decrease, an effect that could perhaps be verified experimentally with a greater number of subjects. Because RPEs at this point correspond to the difference between most valuable feature and the environment (see Eq. 6.11), this effect is certainly due to the fact that the value of the environment is greater in the short ITI group. 6.6. Detailed analysis of behaviour during the CS presentation 113 6.6 Detailed analysis of behaviour during the CS presentation [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] also published a result which cannot be accounted for by the original version of the FMF-MB model as presented in Lesaint et al. (2014a). Indeed, this model was designed to account for a single behavioural response per CS period of a trial. In contrast, in late sessions, [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] found that when the 8s CS period is divided into two, while in the long ITI group sign-tracking dominates throughout the entire CS period, in the short ITI group after learning, goal-tracking is limited to the first 4s of the CS period, while sign-tracking takes over in the last 4s. This result is at first glance very surprising as it seems to stand in contradiction with previous reports that food cup responses are usually concentrated in the last seconds of CS presentation [START_REF] Holland | Conditioned stimulus as a determinant of the form of the Pavlovian conditioned response[END_REF]. It also stands in contradiction with what we would expect from the original model. Indeed, if goal-trackers are predominantly using a MB strategy then, if we allow the model to make a second decision during the CS period, it is plausible that the simulated goal-trackers shift from sign-tracking during early CS period to goal-tracking during late CS period, as they search for the shortest possible route to the reward. The reverse on the other hand undermines the MB strategy by adding an intermediate step between sign-tracking and reward consumption. To prove this, I ran simulations of the original model with additional intermediate steps within the CS period (Fig. 6.6 a.) and found a significant increase in goal-tracking in the last 4 seconds compared to the first 4s (Fig. 6.6 b., Wilcoxon signed-rank test: W = 21.5, p = 0.0018).

A plausible explanation of the result by [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] might be found in the relative feature values of the lever and the magazine. Indeed, if we turn back to the original single-action model and plot the average FMF value of the magazine and the lever we find that in the case of the long ITI, both values stabilise and stay far apart, while in the short ITI group, the value of the lever keeps increasing despite not being selected very often (Fig. 6.6 c.). The stability for the long ITI group could be viewed as an absence of uncertainty about estimated feature values, and thus a good reason to simply select one feature to focus on during the CS period: the Lever. This is consistent with the stable sign-tracking behavior observed after learning throughout the entire CS period in the long ITI group in [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF]. In contrast, the instability of the lever value for the short ITI group might cause a growing uncertainty as to which feature to focus on. The fact that the value of the magazine remains greater might be the reason why the animals first visit it. Then, after visiting the magazine, rather than waiting for the end of the CS period, the rats might be attracted by the lever because of the high uncertainty associated to its estimated value, which could be viewed as a form of directed exploration [START_REF] Daw | Cortical substrates for exploratory decisions in humans[END_REF][START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF]. This might also explain some reports that goal-tracking is an unstable phenotype (see [START_REF] Derman | Signtracking is an expectancy-mediated behavior that relies on prediction error mechanisms[END_REF], and Fig. 2 in [START_REF] Nasser | Individual variability in behavioral flexibility predicts sign-tracking tendency[END_REF]), which sometimes disappears as conditioning goes on. Indeed, looking at the curves, it seems very likely that the feature value of the lever will eventually catch up with that of the magazine, as shown in simulations with an extended number of sessions in the supplementary Fig. D.3. This would predict that extending the number of acquisition sessions could enable a slow learning process to be revealed, which could lead some rats to ultimately sign-track, probably depending on the strain and the supplier of these rats. Consistently with this interpretation, some reports show some shifting in the distribution of responding from goal-to sign-tracking even across only 5 days of acquisition [START_REF] Meyer | Quantifying individual variation in the propensity to attribute incentive salience to reward cues[END_REF].

Discussion

In this chapter, I presented novel simulations of the STGT model of Lesaint et al. (2014a). The goal was to assess whether the model can account for all behavioural and neurophysiological results reported in the recent paper by [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF]. The latter study was designed to test some specific predictions of the STGT model, namely that manipulating the duration of the inter-trial interval (ITI) so that animals have either more or less time to visit the unrewarded magazine during that period would change the relative proportions of sign-versus goal-tracking behaviour in the population, as well as the dopamine response pattern. More precisely, the STGT model assumed a down revision of magazine value each time it is visited but unrewarded during the ITI, which constituted a possible explanation why the dopamine response at CS in goal-trackers does not reflect an increase in reward expectancy [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF]. This led the model to predict that shortening the duration during which animals could visit the unrewarded magazine during the ITI (or simply making the magazine inaccessible during ITI) would lead to an increase of goal-tracking behaviours as well as a restoration of a reward prediction error-like dopamine response pattern [START_REF] Lesaint | Experimental predictions drawn from a computational model of sign-trackers and goal-trackers[END_REF]. [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] tested both an increase and a decrease of the ITI and found that the former led to increased sign-tracking and increased dopamine phasic response to the US, while the latter led to increased goal-tracking and decreased dopamine phasic response to the US (even in animals individually categorised as goal-trackers). While these results are consistent with the model predictions, [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] also found behavioural and neural responses which had not been explored by previous modelling work and which I addressed here.

First, after averaging over all sessions, [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF] found that dopamine release at the time of the CS was significantly greater for the long inter-trial interval group than for the short inter-trial interval group. This result had not been explored by previous model simulations, but I have shown here through novel simulations that it is consistent with the STGT model. Second, the authors report that dopamine activity at US delivery is greater in the long ITI group than in the short ITI group, which cannot be accounted for by the original STGT model in the cases where the simulated sign-trackers focus on the feature value of the lever (rather than the food cup) at US delivery. Here I showed that the increase of dopamine activity at US from the lever to the food cup and vice-versa in states 2 and 3. In addition, the possibility of exploring the environment was removed for simplicity. b. Probability of approach to the food cup during the first and last four seconds of the CS presentation period. Bar plot represents mean probability +/-s.e.m. and grey lines individual probabilities. c. Average feature values of the lever and magazine in the short and long ITI conditions across sessions. In the long ITI group, the value of the less favourable feature, which is the food cup, is constant, while in the short ITI the value of the lever keeps increasing, causing possible ambiguity which could explain unstable behaviour during the CS period. Approach Task delivery can be accounted for by extending the STGT model so that the feature the animal focuses on when it eats is not the food but the food cup. Third, after learning the authors report a stable sign-tracking behaviour throughout the entire CS period in the long ITI condition, as opposed to initial goal-tracking during the first 4s of the CS period followed by later sign-tracking in the short ITI condition. This last result goes beyond the original STGT model which had been designed to account for a single behavioural response (Lesaint et al., 2014a). Here I showed that the uncertainty associated to learned feature values in the model is stable after learning in the long ITI condition, while the uncertainty associated to the lever increases in the short ITI condition due to the continuous increase of the lever value. The former would be consistent with the stability of sign-tracking behaviour in the long ITI condition while the latter could provide an explanation why animals in the short ITI condition are first attracted to the food cup (which has the highest value in the model) and then by the lever (which has the highest uncertainty in the model). The model further predicts that extending the number of experimental sessions would cause GT behaviour to progressively diminish in both conditions. These results have important implications for the understanding of the possible neural mechanisms underlying individual differences in Pavlovian autoshaping. They further confirm the computational interpretation that sign-trackers may rely more on MF learning processes while goal-trackers may rely more on MB learning processes (Lesaint et al., 2014a;[START_REF] Dayan | Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation[END_REF]. They bring detailed analyses of the respective contributions of MB and MF learning mechanisms which may explain specific results of [START_REF] Lee | Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release[END_REF], while remaining consistent with experimental results previously accounted for by the model [START_REF] Robinson | Dissociating the Predictive and Incentive Motivational Properties of Reward-Related Cues Through the Study of Individual Differences[END_REF][START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF][START_REF] Saunders | The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses[END_REF]. They moreover lead to testable predictions, which could lead to future experiments aiming at further assessing the present computational hypotheses. In particular, the model predicts that animals whose learning process can be modelled as pure MF should be sensitive to down-revision of food cup value through both increase and decrease of ITI duration. In contrast, the model predicts that animals whose learning process can be modelled as pure MB should be insensitive to this manipulation of the protocol.

The present work also highlights a characteristic of simulated ST and GT behaviours which had not been addressed by previous models (Lesaint et al., 2014a;[START_REF] Kaveri | Dual reward prediction components yield Pavlovian sign-and goal-tracking[END_REF]: the tendency to goal-track less and less along training. This result is of particular importance since several studies have reported that goal-tracking is an unstable phenotype [START_REF] Derman | Signtracking is an expectancy-mediated behavior that relies on prediction error mechanisms[END_REF][START_REF] Nasser | Individual variability in behavioral flexibility predicts sign-tracking tendency[END_REF], which sometimes may eventually disappear as conditioning goes on. Some reports show some shifting in the distribution of responding from goal-to sign-tracking even across just 5 days of acquisition [START_REF] Meyer | Quantifying individual variation in the propensity to attribute incentive salience to reward cues[END_REF]. In contrast, [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF] found that both ST and GT behaviours were stable and robust between sessions. Because the present computational model accounts for the experimental results of [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF] while at the same time predicting a tendency to goal-track less along training, the model has the potential to reconcile these different studies by suggesting that small differences in protocol can lead to variability and instability in ST and GT behaviours. It suggests that the behavioural phenotype may be affected by not only the duration of training, but also precise timing between events of the task, duration of the ITI, and any manipulation which can affect the values and uncertainties associated to stimuli. Similarly to the apparent discrepancy between studies showing similar proportions of STs and GTs [START_REF] Flagel | A selective role for dopamine in stimulus-reward learning[END_REF], a prevalence of STs or on the contrary a majority of GTs [START_REF] Morrison | Sign tracking, but not goal tracking, is resistant to outcome devaluation[END_REF], it is worth noting that the same computational model has been also applied to Pigeon negative automaintenance paradigm [START_REF] Lesaint | Accounting for negative automaintenance in pigeons: A dual learning systems approach and factored representations[END_REF] where it was able to explain the discrepancy between different studies showing different proportions of individuals either able (putatively model-based) or not (putatively model-free) to refrain from pecking a light when this action prevented the delivery of reward.

Interestingly, the progressive decrease in GT behaviour cannot here be directly linked to a relative change in influence of the MB system over behaviour. The relative contribution of MB and MF modules to decision-making in the model are indeed determined by a parameter ω which is fixed in an individual and stable in time. In contrast, computational models for the coordination of MB and MF reinforcement learning tackling experimental data in the instrumental conditioning and navigation paradigms have commonly assumed a progressive shift from MB to MF across learning to explain animals' tendency to develop behavioural habits [START_REF] Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Dollé | Path planning versus cue responding: a bio-inspired model of switching between navigation strategies[END_REF][START_REF] Keramati | Speed/accuracy trade-off between the habitual and the goal-directed processes[END_REF][START_REF] Viejo | Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning[END_REF][START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF]. In future work, it would be interesting to extend the present STGT model so that it progressively shifts control over decision-making from MB to MF, and study whether this can further expand the model's explanatory power with respect to experimental data.

Finally, the present set of results have important implications for the understanding of the mechanisms underlying drug addiction. The present paradigm assessing individual differences in response to CS had indeed initially been proposed as a model of differential susceptibility to drug addiction [START_REF] Saunders | Individual variation in resisting temptation: Implications for addiction[END_REF]. In this paradigm, the fact that some individuals (sign-trackers) develop a strong attraction towards stimuli that invariably predict the occurrence of a biologically relevant event such as a food reward, can be seen as a model for individuals that develop a strong attraction for stimuli that predict drug rewards. Sign-trackers have indeed been found to be more sensitive to drug-predicting stimuli than goaltrackers [START_REF] Flagel | Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction[END_REF]. The fact that in the present model sign-trackers rely more on inflexible MF behaviours than on flexible MB decisions could provide a computational basis for further understanding why individuals who become addicted are unable to shift their thoughts and actions away from drugs and drugassociated stimuli. Interestingly and consistently, sign-trackers are more prone to display an automatic behaviour towards reward-predicting cues, in the sense that the same animals may be less sensitive to extinction of conditioning [START_REF] Ahrens | Rats that sign-track are resistant to Pavlovian but not instrumental extinction[END_REF] or to devaluation of the reward [START_REF] Morrison | Sign tracking, but not goal tracking, is resistant to outcome devaluation[END_REF][START_REF] Nasser | Individual variability in behavioral flexibility predicts sign-tracking tendency[END_REF][START_REF] Patitucci | The origins of individual differences in how learning is expressed in rats: A general-process perspective[END_REF] than goal-trackers. The present model could thus add to a growing computational literature addressing drug addiction in terms of MB and MF learning mechanisms [START_REF] Simon | Dual-System Learning Models and Drugs of Abuse[END_REF].

Summary of results

In the past chapters, I have tackled questions of meta-learning in an instrumental three-armed bandit task and of inter-individual variability in a Pavlovian Conditioned Approach task. First of all, I showed that the effects of dopamine inhibition on performance and win-shift in a three-armed bandit task are best explained by an increase in random exploration rather than any other effect on the learning rate for instance. Moreover, I have mathematically proved that this effect on random exploration parameter was directly equivalent to a reducing effect of dopamine blockade on positive reward prediction errors. Thus, the observed effect on random exploration is consistent with the traditional RPE theory of dopamine. Nevertheless, a downside to this result is that it means that a theory according to which it is tonic dopamine which controls random exploration is not necessary. Careful experiments and computational analyses are still needed to fully disentangle these two possible explanations.

Secondly, considering how performance and win-shift changed between sessions of the long training period preceding the pharmacological experiment, I put forward the theory that these changes were caused by a process of meta-learning based on a modulatory signal such as the current rate of rewards which might either diminish random exploration or sensitivity to recent rewards. I compared such models of meta-learning to a model of directed exploration which I found to be inadequate to replicate these long-term changes, despite having better log-likelihood scores, thus highlighting the importance of unconstrained model simulations when 120 Conclusion assessing different models. However, I was unable to decide which kind of metalearning on either the inverse temperature, which controls random exploration, or the learning rate was the best to explain the experimental data. It is suggested that this experiment is not particularly suited to the investigation of meta-learning, as it is relatively static, which is why individuals only need to adjust their parameters in the beginning phases of the experiment. A more volatile environment would be better suited in my opinion.

Finally, I pursued previous computational work surrounding the question of signand goal-tracking behaviour in a Pavlovian Conditioned Approach task which uses a lever as conditioned stimulus, by studying in detail the impact of the inter-trial interval on the model's behaviour and by comparing it to new experimental data. I found that the model was generally quite capable of replicating the experimental results, but that there are additional findings, obtained when analysing behaviour during the CS presentation window more precisely, which go beyond the expectations of the model and which call for a more detailed and fine-grained model of this type of behaviour.

Closing thoughts about modelling

Because of the central role which computational modelling has played in this work, I would like to finish this thesis with a few considerations on the strengths and weaknesses of such an approach. In his book on the history of astronomy from Ancient Greeks to Galileo [START_REF] Duhem | Sozein ta phainomena : essai sur la notion de theorie physique de Platon a Galilee[END_REF], Pierre Duhem contends that Ancient Greek and medieval astronomers viewed the aim of their many various astronomical theories of celestial motion as to merely copy the appearance of reality. Their aim was to save appearances (σoζειν τ α φαινoµενα), to design increasingly elaborate mechanisms which could replicate past observations and predict future apparent motion of the stars, believing that the real movement of the stars, which according to most had to be circular in nature, was beyond our reach and comprehension. This puts us in the uncomfortable situation of asking whether the same might not be true of our sophisticated computational models. Is the value of reinforcement learning only to produce behaviour similar to natural behaviour or is there really a shared mechanism between models and the brain?

Whether or not they are perfectly faithful to reality, models have a heuristic value insofar as by proposing a mechanical explanation of data, they can offer insight and disentangle hypotheses which experiments alone cannot. Such an example was presented in Chap. 4 where we saw that dopaminergic inhibition caused a decrease in performance and an an increase in win-shift. At such a limited empirical level, there was no way of knowing whether this effect was simply due to an impairment in learning which led to greater difficulty in distinguishing the value of different levers or whether it was due to an increase in random exploration, meaning that the animals were still capable of learning the value of the different levers but were less "focused" on accumulating rewards. Indeed, if learning was impaired, we would expect both performance to decrease and win-shift to increase as rats failed to recognise the correct lever. Inversely, if random exploration had increased so that rats cared less about what they learned, their performance would likewise diminish and win-shift increase. However, a computational model in which these two processes are clearly distinguished is unambiguous about what changes need to be made to replicate the experimental results, namely, it is β the inverse temperature that controls random exploration which is affected by dopamine manipulation rather than α. The experimenter may want to argue that he could have reached this conclusion without the help of modelling by looking at the initial rate of performance and final convergence levels as these two indicators are different in the two scenarios (see Fig. 4.6) but this begs the question of how he would have come to know these two criteria and their significance without the insight of modelling.

Two other examples of the usefulness of computational modelling came in Chap. 5, when we asked the question of whether the rats might be using meta-learning to adjust their parameters between sessions (Fig. 5.5) and also between blocks of different risk levels (Fig. 5.10). Both of these cases are interesting because of their artificiality: the models we optimised supposed that entirely different sets of parameters were used for different sessions or different risk levels, as though the animal switched completely from one set to another, which seems extremely unrealistic. Despite this, these models were extremely useful in analysing the data and providing the groundwork with which the more "mechanistic" models of metalearning were built.

I will now turn to some of the drawbacks of computational modelling, and to start off, it is important to remember the specific limits within which a model operates. Some of these limits are quite obvious, for instance, reinforcement learning provides answers to questions of what choices an animal makes, but, in its traditional form, is silent about the rate with which an animal responds. In our particular case, there was additional experimental data in the form of the different response times to the events of the three-armed bandit task which were not interpreted computationally simply because the tools we had chosen were not meant for such data. But there are also more subtle limits to what a model can be used for. In Chap. 4, I gave an example of such a limit when we found that it was not possible to disentangle reward magnitude from the inverse temperature. Another example is found in Chap. 5 where I discussed about models of directed exploration and the surprising result that despite the better optimisation scores these models failed at replicating what seemed a quite striking property of the experimental data, namely the between-session changes in behaviour. One possible explanation lies in the optimisation process itself: when optimising a model, one looks at the probability of selecting the same action as the animal at each trial given that past choices are exactly the same. In other words, model optimisation is based on constraining the model to the history of the animal and this might potentially conceal any divergence between the mechanics of the model and those directing the behaviour of the animal. Once the model is simulated however, these divergences may become apparent. Another explanation which I think is more relevant for our case, is that Conclusion how well different models replicate experimental data is inextricably linked to how you choose to look at the data. In our case, data was analysed in terms of performance and win-shift which we averaged separately for different groups of sessions, the aim being to uncover the long-term changes in behaviour. Because these longterm changes are what motivated our research efforts, we used these as criteria for model selection, but there is no reason to suppose that if we had chosen to look at our data differently, we would not have found a different aspect of behaviour which meta-learning would fail to replicate while directed exploration would succeed. This brings us again to the observation that models are tailored for a specific task, and begs the question of whether there is or could be such a thing as a "true" model of behaviour. As famously put by George E. P. Box: "All models are wrong but some are useful".

Another danger of computational modelling is that certain ideas, when backed by a computational framework, seem to crystallise and become entrenched. One such idea is that the habitual versus goal-directed systems are equivalent to a modelfree versus model-based distinction. Yet, despite this widespread idea, we have successfully used model-free Q-learning, which supposedly causes habitual behaviour, to replicate data from a three-armed bandit task in which there were multiple changes in contingencies to which the animals reacted perfectly well. Far from being a slow and inflexible learning system, model-free Q-learning adjusts quite well so long as its parameters are adapted to the task and that this task is relatively simple. Proponents of the habitual = model-free theory are in fact arguing their case on the basis of more complex tasks, in which model-free learning does indeed become inefficient and slow, but in a simple task, in which habitual behaviour is also quite capable of emerging, it seems to be a misconception to say that modelfree will always be inefficient. In this way, a computational model which is too successful can paradoxically get in the way of potentially better answers. [START_REF] Niv | Reinforcement learning in multidimensional environments relies on attention mechanisms[END_REF] briefly proposed that forgetting is due to a passive decline in the strength of cortico-striatal synapses, and I wanted to test this hypothesis in a simplified molecular model of this synapse during a three-week project at the Okinawa Computational Neuroscience Course with the help of Dr. Andrew Gallimore. This model was implemented in STEPS, a python package designed for modelling chemical reactions on a molecule by molecule basis and is derived from the model of [START_REF] Nakano | A kinetic model of dopamine-and calcium-dependent striatal synaptic plasticity[END_REF]. It includes 63 different chemical species involved in 110 reactions. Briefly (see Fig. B.5), dopamine activation of D1 receptors stimulates the production of cAMP which activates a kinase known as PKA. PKA has two different targets, the DARPP enzyme which inhibits a phosphatase PP1 which is responsible for dephosphorylation of AMPA receptors, leading to their storage in the cytosol, and AMPA receptors themselves. Crucially, phosphorylation of AMPA receptors by PKA is not sufficient to promote their exocytosis and this requires a calcium signal originating from a concurrent activation of glutamate NMDA receptors. Co-activation of glutamate and dopamine receptors thus leads to an increase in the number of AMPA receptors in the membrane. The question I wanted to tackle using this model was whether after such an increase in the number of membrane AMPA receptors, this number would stay stable or progressively decline in the absence of any other events, a decline which could be interpreted as a forgetting of the corresponding action value. This model was simulated in a scenario where a peak of dopamine and glutamate was provided while the number of AMPA receptors was being recorded (shown in Fig. B.6). As expected, this event produces a surge in the number of AMPA receptors present in the membrane, and in the absence of any further activity, this was followed by a decline. Under the hypothesis that the strength of the cortico-striatal synapse, which is dependent, among other potential mechanisms, upon the number of glutamate receptors in the membrane, represents the value of an action then this decline can be interpreted as a forgetting of the corresponding action value when it is no longer activated.

B.3 Proposed synaptic mechanism

C.2 ε-greedy model with forgetting

This model computes the Q-values in the same way as the Q-learning with forgetting model, but uses a different decision process. In this case, the model simply selects the greedy action, i.e. the one with a highest value, with probability 1-ε, and selects the remaining actions with probability ε N -1 with N the total number of available actions. Thus, in this framework, the higher the value of ε, the more random the model is in his choices. After optimisation of this model, the only parameter affected by flupenthixol was indeed ε (Friedman ANOVA: chi 2 (3) = 34.7, p < 0.0001), and this parameter increased with the level of dopamine inhibition in accordance with my main results (

C.3 Directed exploration

Models of directed exploration, in which action selection is biased towards actions with a high uncertainty, are attracting increasing levels of interest [START_REF] Wilson | Inferring Relevance in a Changing World[END_REF][START_REF] Wilson | Humans use directed and random exploration to solve the explore-exploit dilemma[END_REF][START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF][START_REF] Zajkowski | A causal role for right frontopolar cortex in directed, but not random, exploration[END_REF][START_REF] Cogliati Dezza | Learning the value of information and reward over time when solving explorationexploitation problems[END_REF] which is why I felt the need to test this model on the experimental data to see if the result really was limited to random exploration. It should be noted however that the task at hand is not specially well-designed for investigating these questions. The model I chose is taken from [START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF] which is a standard Q-learning model which adds an uncertainty bonus σ 2 i to the Q-value of each action a i . This uncertainty bonus requires that the model calculate an uncertainty prediction error ξ, analogous to the reward prediction error:

ξ t = δ 2 t -σ 2 i (C.1)
with σ 2 i the current value of the uncertainty bonus, initialised at 0. This uncertainty prediction error is then used to update the current estimate of the variance in a similar manner to what is done with reward prediction errors and Q-values:

σ 2 i ← σ 2 i + α φ ξ t (C.2)
with α φ a special learning rate dedicated to learning uncertainty bonuses. The Qvalue and expected uncertainty are then combined to be plugged into the softmax selection function:

P (a t+1 = a i ) = e β(Q i +φσ 2 i ) j e β(Q j +φσ 2

Figure 2

 2 Figure 2.1: Le système dopaminergique. a. Le système dopaminergique dans le cerveau du Rat tiré de Puig et al. (2014). La dopamine est émise depuis les noyaux du VTA et du SNc en direction du striatum et du cortex préfrontal. b. Le système dopaminergique dans le cerveau humain, également tiré de Puig et al. (2014) c. Des projections corticales et dopaminergiques convergent sur les neurones épineux moyens du striatum. La dopamine est responsable de la régulation de la plasticité des connexions entre cellules corticales et striatales. Tiré de Hyman and Malenka (2001).

Figure 2 . 2 :

 22 Figure 2.2: Représentation schématique de l'expérience d'apprentissage instrumental sous différentes conditions d'inhibition dopaminergique.

Figure 2

 2 Figure 2.3: a. Performance expérimentale moyenne (moyenne +/-erreur standard de la mesure ou e.s.m.) en fonction de la dose de flupenthixol et du niveau de difficulté. b. Performance extraite des simulations du modèle.

Figure 2 .

 2 Figure 2.4: a. Win-shift moyen des rats (moyenne +/-e.s.m.) en fonction de la dose de flupenthixol et du niveau de difficulté. b. Win-shift extrait des simulations du modèle.

Figure 2 . 5 :

 25 Figure 2.5: Évolution des paramètres du modèle en fonction des différentes doses de flupenthixol. Les traits gris relient les valeurs des paramètres d'un même individu. a. Variations du taux d'apprentissage. b. Variations de la température inverse. c. Variations du taux d'oubli.

Figure 2 . 6 :

 26 Figure 2.6: Évolution du comportement des rats sur le long terme. a. Performance moyenne des rats dans les blocs de difficulté faible. b. Performance moyenne des rats dans les blocs de difficulté élevée. c. Win-shift moyen des rats dans les blocs de difficulté faible. d. Win-shift moyen des rats dans les blocs de difficulté élevée.

Figure 2 .

 2 Figure 2.8: Évolution du comportement des simulations du modèle de métaapprentissage sur β. a. Performance moyenne des simulations dans les blocs de difficulté faible. b. Performance moyenne des simulations dans les blocs de difficulté élevée. c. Win-shift moyen des simulations dans les blocs de difficulté faible. d. Win-shift moyen des simulations dans les blocs de difficulté élevée.

Figure 2 . 9 :

 29 Figure 2.9: Évolution du comportement des simulations du modèle de métaapprentissage sur α. a. Performance moyenne des simulations dans les blocs de difficulté faible. b. Performance moyenne des simulations dans les blocs de difficulté élevée. c. Win-shift moyen des simulations dans les blocs de difficulté faible. d. Win-shift moyen des simulations dans les blocs de difficulté élevée.

Figure 2 .

 2 Figure 2.10: a. Représentation schématique des différents étapes de la tâche. C'est cette structure-ci que la composante MB du modèle apprend à trouver. Le chemin en bleu indique celui emprunté lors d'une stratégie de goal-tracking, tandis que le chemin rouge correspond à du sign-tracking. b. Chronologie des différentes apparitions des stimuli associée à ces étapes.

Figure 2 .

 2 Figure 2.12: Réaction des deux variantes du modèle à une manipulation de la durée entre chaque essai. a. Probabilité d'approche du levier pour le modèle dont la composante MB a été désactivée. b. Probabilité d'approche de la mangeoire pour ce même modèle. c. Probabilité d'approche du levier pour le modèle dont la composante FMF a été désactivée. d. Probabilité d'approche de la mangeoire pour ce même modèle.

Figure 2 .

 2 Figure 2.13: Simulations du comportement d'une population aléatoire d'individus effectuant la tâche de conditionnement pavlovien. a. Distribution du paramètre ω responsable de la contribution relative des deux composantes du modèle. Plus ce paramètre est élevé, plus la composante FMF est importante. Les valeurs de ce paramètre sont biaisées vers 1 en raison des nombreux rapports convergents sur la prépondérance du comportement de sign-tracking. b. Probabilité d'approche du levier. c. Probabilité d'approche de la mangeoire. d. Distribution des différences de probabilités softmax de s'approcher soit du levier soit de la mangeoire. Il y a un biais pour les comportements de type goal-tracking lorsque les délais entre essais sont courts, et vice-versa lorsqu'ils sont longs. e. Distribution des différences de fréquences relatives des différents choix d'approche effectivement simulés. Le même constat s'applique. f. Haut: Effet du délai sur les valeurs des contributions relatives de chaque composante. Bas: Probabilité moyenne de s'approcher du levier (goL) ou de la mangeoire (goM) en fonction du délai d'attente.

Contents 3 . 1

 31 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Behavioural results . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2.1 Pavlovian conditioning . . . . . . . . . . . . . . . . . . . . . . 26 3.2.2 Operant conditioning . . . . . . . . . . . . . . . . . . . . . . 29 3.2.3 Goal-directed behaviour . . . . . . . . . . . . . . . . . . Rescorla-Wagner model . . . . . . . . . . . . . . . . . . . 35 3.3.2 Incentive salience theory of Pavlovian conditioning . . . . . . 36 3.3.3 Reinforcement learning . . . . . . . . . . . . . . . . . . . . . 38 3.3.4 Meta-learning . . . . . . . . . . . . . . . . . . . . . . . . history of dopamine research . . . . . . . . . . . . . . . . 46 3.4.2 Dopaminergic pathways and functions . . . . . . . . . . . . . 48 3.4.3 Dopamine receptors and the cortico-striatal synapse . . . . . 53

BackgroundFigure 3 . 2 :

 32 Figure 3.2: Behavioural results from Flagel et al. (2009) illustrating the differences between goal-and sign-trackers. A third intermediate group is also depicted. Data are expressed as mean + s.e.m. and are averaged between different blocks of 50 trials (2 sessions). A Probability of approaching the lever during the 8-s CS presentation interval. B Probability of approaching the food cup. C Average number of lever contacts. D Average number of food cup contacts. E Average latency before first lever contact, 8s being the maximum possible value. F Average latency before first food cup contact.

Figure 3

 3 Figure3.4: Two-step task from[START_REF] Sharp | Dopamine selectively remediates 'model-based' reward learning: A computational approach[END_REF]. a. Task structure: in the first phase, subjects chose a spaceship which would fly to one of two possible planets with different probabilities (common or rare transitions). Once on the planet (phase 2), the subject chooses an alien in the hope of being rewarded. b. Different predictions made by model-free and model-based learning strategies, which, as we will see later, are commonly used to model habitual and goal-directed behaviour respectively. In the case of a habitual system, the probability of 'staying', i.e. repeating the same choice in phase 1 as in the preceding trial, is independent of whether the transition to phase 2 was rare or common. Contrary to this, goal-directed behaviour is sensitive to transition probabilities and prefers to stay after a reward if the transition was common or stay after not being rewarded if the transition was rare.

Figure 3

 3 Figure 3.6: a. Depiction of the reinforcement learning problem. An agent interacts with his environment by producing actions and in turn receiving state and reward information as feedback. His goal is to find which actions to take in each state, a policy, to maximise the amount of rewards. b. Schematic representation of a Markov Decision Process with different actions leading to different states. These transitions may be stochastic.

Figure 3

 3 Figure 3.7: Chemical structure of dopamine (left) and noradrenaline (right). These neurotransmitters are catecholamines characterized by a catechol cycle (benzene with two adjacent hydroxyl functions) and an amine group. Initially, dopamine was thought to be nothing more than the chemical precursor of noradrenaline.

  Sharp et al. (2016); Smittenaar et al. (2012); Shiner et al. (2012)).

  Figure 3.8: Dopamine circuits in the brain. a. The dopaminergic system in the Rat brain from Puig et al. (2014). Dopamine from the VTA and SNc innervate the striatum and the prefrontal cortex. b. Dopaminergic system in the human brain also from Puig et al. (2014). c. Cortical and dopaminergic projections converge in the striatum onto medium spiny neurons, allowing dopaminergic inputs to directly modulate the strength of the cortico-striatal connection. Taken from Hyman and Malenka (2001).

Figure 3

 3 Figure3.9: Raster plots adapted from the study of[START_REF] Schultz | A neural substrate of prediction and reward[END_REF]. a. Before conditioning, unexpected reward delivery produces a peak in dopaminergic activity. b. Once the CS-US association is learnt, the peak of dopamine activity has shifted from the reward to the CS. c. In the case where an expected reward is omitted, there is an interruption of tonic dopamine firing which is consistent with a negative prediction error.

  description and optimisation . . . . . . . . . . . . . . 63 4.3.2 Model simulations . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3.3 Model optimisation dissociates random exploration from learning 69 4.4 Relationship between RPEs and exploration . . . . . . . . . 72 4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

  Figure 4.1: Three-armed bandit task.

Figure 4 . 2 :Figure 4 . 3 :

 4243 Figure 4.2: Effects of flupenthixol on performance. a. Experimental data. b. Simulated data.

Figure 4

 4 Figure 4.4: Effects of flupenthixol and risk on the number of win-shifts in the last 8 trials of blocks. a. Experimental data. b. Simulated data.

Figure 4 . 5 :

 45 Figure 4.5: Effects of flupenthixol on lose-shift. a. Experimental data. b. Simulated data.

Figure 4

 4 Figure 4.6: Distinguishable impacts of learning rate and inverse temperature manipulations on performance and win-shift curves. These curves were obtained by running simulations of a Q-learning algorithm without forgetting with different parameterisations onto sessions of low-risk blocks only. Parameter values were set arbitrarily to best illustrate the effects I wanted to highlight. a. Simulated performance curves when holding β constant and varying α. b. Simulated win-shift curves when holding β constant and varying α. c. and d. Same as a. and b. respectively when holding α constant and varying β.

Figure 4

 4 Figure4.7: Relationship between model action probabilities and experimental frequencies. Model probabilities are obtained by averaging the softmax probability in a given bin of four trials within a block. Each point represents an average for a given combination of rat, 4-trial bin, risk level, dose and action. A linear regression curve without an intercept was fitted to these points: y = 0.99 × x and is plotted in black.

  Fig. C.1), an ε-greedy model of action selection (Supplementary Fig. C.2), a model of directed exploration (Supplementary Fig. C.3) taken from the literature[START_REF] Naudé | Nicotinic receptors in the ventral tegmental area promote uncertainty-seeking[END_REF] which includes an uncertainty bonus to bias decision-

Figure 4 . 8 :

 48 Figure 4.8: Distribution of the optimised parameter values of the Q-learning with forgetting model for the different flupenthixol doses. Grey lines connect the parameter values of a single individual. a. Distribution of the learning rate values. b. Distribution of the inverse temperature values. c. Distribution of the forgetting rate values.

Figure 4

 4 Figure4.9: Verification procedure to determine if optimisation of a forgetting Qlearning model can distinguish variations of the learning rate from variations of the inverse temperature. The first step in this process was to optimise two versions of the model, one where α was allowed to vary between doses, while the other parameters were not, and one where only β was allowed to vary. This way, the maximum variation in the parameter of interest allowed by the experimental data was captured, and I indeed found that if only α was allowed to vary, it would significantly decrease (not shown here), proving that there is indeed a relative tradeoff between learning rate and inverse temperature. The second step consisted in creating an artificial dataset by simulating these two versions of the model. Finally, I optimised a model with all three parameters allowed to vary between doses on each dataset a. and b. and verified that in each case, the only parameter whose optimised values varied significantly was the one originally allowed to vary.

Figure 5 . 2 :

 52 Figure 5.2: a. Performance in the last eight trials of blocks at different stages of the experiment. b. Win-shift in the last eight trials of blocks at different stages of the experiment.

Figure 5

 5 Figure 5.3: Average length of uninterrupted sequences of perseverative choices at the start of blocks.

Figure 5 . 4 :

 54 Figure5.4: Individual mean-squared errors of simulations of the forgetting Qlearning model with α free between groups of sessions, with β free, or with all three parameters optimized on the entirety of the experiment. Grey lines connect the MSEs for the simulations of a single individual. a. MSE on individual performance curves. The α-free model has significantly smaller MSEs than both the β-free (p = 0.004) and fixed-parameters model (p < 0.0001), while the MSE scores of the β-free model are also significantly smaller than those of the fixed-parameters model (p = 0.0009). b. MSE on individual win-shift curves. The MSE scores of the α-free and β-free models are both significantly smaller than those of the fixed-parameters model (p < 0.0009). There is no statistically significant difference between the MSE scores of the α-and β-free models (p = 0.15).

Fig

Figure 5 . 5 :

 55 Figure 5.5: Evolution of the parameters of models with either the learning rate or the inverse temperature allowed to change between sessions. Grey lines connect the parameter values of a same individual. In the case of the inverse temperature, the natural logarithm of parameter values was applied because of the great interindividual variability. Statistical tests reported in the text were applied on the raw values of β nonetheless. a. Evolution of the learning rate. b. Evolution of the inverse temperature.

Figure 5

 5 Figure 5.6: Between session changes in simulations of the R-ML-β model. a.b. Trial-average performance in low risk and high risk blocks respectively improves from early to late sessions. c.-d. Trial-average win-shift, in low and high risk blocks respectively, decreases from early to late sessions. e.-f. Trial-average lose-shift in low and high risk blocks respectively does not change between sessions.

Figure 5 . 7 :Figure 5 . 9 :

 5759 Figure 5.7: Between session changes in simulations of the R-ML-α model. a.b. Trial-average performance in low risk and high risk blocks respectively improves from early to late sessions. c.-d. Trial-average win-shift, in low and high risk blocks respectively, decreases from early to late sessions. e.-f. Trial-average lose-shift in low and high risk blocks respectively does not change between sessions.

Figure 5

 5 Figure 5.10: Parameter values of the forgetting Q-learning models with separate parameter sets for high risk and low risk blocks. As usual, grey lines connect the parameter values of a same individual. I represented the natural log of the parameter values for readibility purposes, but statistical tests were applied to raw values. The learning rate is significantly affected by risk (p = 0.0002), as is the forgetting rate (p = 0.04), but not the inverse temperature (p = 0.08) according to Wilcoxon signed-rank tests.

Figure 6

 6 Figure 6.1: a. MDP representation of a single trial from the original experiment in Flagel et al. (2011) adapted fromLesaint et al. (2014a). There are six possible actions leading deterministically from one state to the next: exploring the environment (goE), approaching the lever (goL), approaching the magazine (goM), waiting, engaging with the closest stimulus and eating the reward. Each of these actions focuses on a specific feature indicated in brackets: the environment (E), the lever (L), the magazine (M) and the food (F). These are the features used by the FMF learning component. The red path corresponds to sign-tracking behaviour and the blue path to goal-tracking behaviour. b. Corresponding timeline of lever and food appearances.

Figure 6 . 3 :

 63 Figure 6.3: Behaviour of FMF-only (ω = 1, a-d) and MB-only (ω = 0, e-h) models. For each graph, I have plotted the mean +/-s.e.m. a. Approach to lever of simulations of the FMF-only model for different ITI durations. b. Approach to the food cup of simulations of the FMF-only model for different ITI durations. c. Effect of down-revision of food cup value on FMF and MB values of the FMF-only model. d. Average softmax probabilities of engaging with either the lever or the food cup during the CS period of the FMF-only model for different ITI durations. e. Approach to lever of simulations of the MB-only model for different ITI durations. f. Approach to the food cup of the MB-only model for different ITI durations. g. Effect of down-revision of food cup value on action probabilities of the MB-only model. h. Average softmax probabilities of engaging with either the lever or the food cup during the CS period of the MB-only model for different ITI durations.

Figure 6 . 4 :

 64 Figure 6.4: Simulations of the behaviour of a population with random ω parameter values. a. Distribution of the ω parameters sampled from a β distribution which were then used for the simulations. The same values of ω were used in both short and long ITI condition. Inset: probability density function of the original distribution is biased towards 1 in accordance with the reported prevalence of sign-trackers. b.Approach to the lever for different ITI durations (mean +/-s.e.m.). c. Approach to the food cup for different ITI durations (mean +/-s.e.m.). d. Distribution of differences in softmax probability of approach to lever and magazine for the two ITI durations. There is a significant bias towards goal-tracking choices in the short ITI group and a significant bias towards sign-tracking choices in the long ITI group. e. Distribution of differences in average simulated number of approaches to lever and magazine for the two ITI durations. As expected from the differences in softmax probabilities, there is a significantly higher number of goal-tracking than sign-tracking trials in the short ITI condition and vice-versa in the long ITI condition. f. Top: Effect of down-revision of food cup value during ITI of different durations on average FMF-values and MB action advantages. Bottom: Average softmax probabilities of engaging with either the lever or the food cup during the CS period for different ITI durations.

Figure 6 . 5 :

 65 Figure 6.5: Reward prediction errors of the model at CS and US presentation for short and long ITIs. a. Reward prediction errors averaged across all sessions. b. Reward prediction errors for the long ITI simulations averaged in early and late sessions. c. Reward prediction errors for the short ITI simulations averaged in early and late sessions.

  Figure B.5: Main interactions leading to the trafficking of AMPA receptors in cortico-striatal synapses under the influence of dopamine and glutamate. Not depicted here is the detailed calcium pathway which relies on the activation of calmodulin.

Figure B. 6 :Figure C. 1 :

 61 Figure B.6: Number of AMPA receptors present in the membrane as a function of time. At t=750, there was a burst in both dopaminergic and glutamate inputs leading to the exocytosis of new receptors. However, in the absence of further excitation, the number of AMPA receptors present in the membrane steadily declines following this event.

FigFigure C. 2 :

 2 Figure C.2: Parameter variations for the ε-greedy model. Gray lines connect parameter values of a same individual and the bold lines plot average parameter values. Box plots represent the median, interquartile and furthest values not considered as outliers.

Figure D. 1 :

 1 Figure D.1: Simulations of the behaviour of a population with random ω parameter values sampled from a uniform distribution. a. Distribution of the ω parameters sampled from a uniform distribution over the range [0, 1]. b.Approach to the lever for different ITI conditions. c. Approach to the food cup for different ITI conditions. d. Distribution of differences in probability of approach to lever and magazine for the two ITI conditions using the output of the softmax function. e. Distribution of differences in average simulated number of approaches to lever and magazine for the two ITI conditions. (f. Top: Effect of food cup devaluation on FMF and MB values of the FMF model. Bottom: Average probabilities of engaging with either the lever or the food cup during the CS period for different devaluation levels.

2.4 Variabilité inter-individuelle dans l'approche d'un stimulus conditionnel 2.4.1 Contexte

	Au cours d'une tâche de simple conditionnement Pavlovien dans laquelle
	l'apparition d'un levier pendant 8 secondes annonçait la livraison d'une récom-
	pense alimentaire (figure 2.10), Flagel et al. (2011) ont décrit deux types de com-
	portement différents parmi les individus d'une même population. D'un côté certains
	individus, surnommés "sign-trackers" en anglais, s'approchent préférentiellement du
	levier prédictif et interagissent de diverses manières avec lui, en le mordillant par
	exemple. D'un autre côté, d'autres individus, surnommés "goal-trackers", appren-

nent à se diriger d'emblée vers le lieu où la récompense sera livrée. En outre, tandis que les sign-trackers présentent un profil de signalisation dopaminergique typique d'une erreur de prédiction, avec une forte activité initialement associée à l'arrivée du stimulus inconditionnel qui se propage au cours de l'expérience au moment de l'apparition du levier, les goal-trackers présentent des signaux dopaminergiques relativement insensibles aux principaux événements de la tâche.

Pour expliquer ces résultats,

Lesaint et al. (2014a)

proposa un modèle d'apprentissage avec deux composantes:

2.4. Variabilité inter-individuelle dans l'approche d'un stimulus conditionnel 19

	a.				s 2	eng <L>	s 5
			goL <L>
	s 0	goE <E>	s 1	goE <E>	s 3	wait <E>	s 6	goM <F>	s 0
						goM <F>
			goM <M>	s 4	eng <M>	s 7	eat <F>
	b.
	Lever (CS)
				8s
	Food (US)

Table 4 . 2 :

 42 Post hoc comparison of average experimental win-shift between the different pharmacological conditions using Bonferroni tests. Post hoc comparison of average experimental win-shift in the last eight trials of blocks between the different pharmacological conditions using Bonferroni tests.

	Comparison	Estimated difference (%) Standard error (%)	p-value
	saline -0.1 mg/kg	-2.7	1.3	0.05
	saline -0.2 mg/kg	-9.2	1.4	<0.0001
	saline -0.3 mg/kg	-12.6	2.7	0.0003
	0.1 mg/kg -0.2 mg/kg	-6.4	1.8	0.002
	0.1 mg/kg -0.3 mg/kg	-9.9	2.2	0.0003
	0.2 mg/kg -0.3 mg/kg	-3.4	3.0	0.26
	Comparison	Estimated difference (%) Standard error (%) p-value
	saline -0.1 mg/kg	-1.9	1.3	0.84
	saline -0.2 mg/kg	-5.9	1.5	0.003
	saline -0.3 mg/kg	-10.7	2.1	0.0002
	0.1 mg/kg -0.2 mg/kg	-4.0	1.7	0.17
	0.1 mg/kg -0.3 mg/kg	-8.8	1.8	0.0004
	0.2 mg/kg -0.3 mg/kg	-4.7	2.4	0.37
	Table 4.3:

Table 5 .

 5 1: Right-tailed paired t-tests on the log-likelihood scores of each model after optimisation indicating whether the log-likelihood of the row model is smaller than that of the column model.

			94	0.99	0.67	0.78	0.99	0.05
	RPE-ML-β	0.06		0.98	0.60	0.73	0.99	0.03
	U-ML-β	0.01	0.02		0.51	0.63	0.98	0.03
	R-ML-α	0.33	0.40	0.49		0.90	1.0	0.02
	RPE-ML-α	0.22	0.27	0.37	0.10		1.0	0.008
	U-ML-α	0.01	0.01	0.02	<0.0001	<0.0001		0.0004
	DE	0.95	0.97	0.97	0.98	0.99	0.99

Table 6 .

 6 1: Feature function mapping state-action pairs to features. For the state action pair s7-eat, we used either the original feature mapping to food or the mapping to magazine when trying to explain dopaminergic activity, as explained in the main text.

	States	s 0	s 1	s 1	s 1	s 2	s 3	s 4	s 5	s 6	s 7
	Actions goE goL goE goM eng wait eng goM goM	eat
	f(s,a)	E	L	E	M	L	E	M	F	F	F or M

3. Predicted effects of ITI duration on the FMF-MB model 103Table 6 .

 6 by reducing the number of parameters, and I kept this convention as my own parameter values 6.2: Parameter values of the model were adapted from previous work byLesaint et al. or hand-tuned; ω was either set to 1 or 0 for the FMF-only and MB-only models, or sampled from a probability distribution as explained in the main text. Because this particular work is interested in looking at how different contributions of the FMF and MB systems can produce different behaviours independently of other parameter effects, I fixed α, β and γ for all the simulations, used two different values of u IT I to model a short and long ITI (see Table6.2), and varied ω as appropriate for our purposes.

		α	γ	β	u IT I r
	short ITI 0.03 0.8 6.7 0.01 1
	long ITI	0.03 0.8 6.7 0.1 1
	are derived from that original study.

  .3 d.. In the case of the MB-only model, things are much simpler with little or no behavioural effect of down-revision of food cup value on either type of behaviour Approach Task

					s 2	eng <L>	s 5
			goL <L>
	s 0	goE <E>	s 1	goE <E>	s 3	wait <E>	s 6	goM <F>	s 0
						goM <F>
			goM <M>	s 4	eng <M>	s 7	eat <F>
	Lever (CS)
				8s
	Food (US)

Acknowledgements

Chapter 5

Long-term improvements of behaviour and meta-learning

Though this be madness, yet there's method in it Hamlet, William Shakespeare

Introduction

Having previously established that altering dopamine levels through flupenthixol inhibition of D1 and D2-type receptors produces an increase in exploration without affecting learning rate, the natural next step is to ask whether animals actually use this relationship between dopamine and the exploration-exploitation trade-off, i.e. whether this reflects a true biological function or is nothing more than an experimental artefact with no functional significance. To begin to answer these questions, we can turn towards the analysis of the preceding training sessions of the pharmacological interventions we have previously focused upon. This training sequence was sufficiently long for the animals to show long-term improvements in their behaviour, in the form of increased performance levels and lower win-shift levels in later sessions compared to earlier ones. In this chapter, I will first present the raw experimental results pleading in favour of some form of meta-learning, then adopt a simple computational approach similar to what has already been done

Appendix A

Simulations of the forgetting model in the flupenthixol experiment

The object of this appendix is simply to present trial average performance, win-shift, and lose-shift curves from the pharmacological experiment of Chap. 4 side-byside with the simulations of the forgetting Q-learning model that chapter focuses on. These plots are merely a rearrangement of

Appendix B

The forgetting mechanism

In this annex chapter, I will discuss further the forgetting mechanism by showing how essential it is to improve model optimisation scores and simulations as well as propose a modest explanation for the origin of this mechanism.

B.1 Optimisation

During this PhD, I have optimised a great number of models on the training sessions of the three-armed bandit task, some of which are presented in Chap. 5 which could on each occasion be declined into forgetting and non-forgetting versions. After model optimisation, a standard method of comparison is to compute the AIC and BIC scores which are smaller for better models. We did this for each model and each individual rat, and Fig. B.1 represents the distribution of these scores for each model and clearly shows that forgetting models are always better than non-forgetting counterparts. This advantage is all the more evident that any other difference between models, such as meta-learning versus directed exploration models, is indiscernible in this figure.

B.2 Simulations of the standard Q-learning model on the flupenthixol experiment

The aim of this section is to illustrate the need for forgetting by comparing the simulations of a Q-learning model without forgetting with experimental data. Insofar as there was no effect of simulations of this model without forgetting on performance curves (interaction between trial and simulations: F (0.6, 12.7) = 1.6, p = 0.22), this model is actually slightly better than the model with forgetting used in Chap. 4 which presented significant differences in the later stages of blocks (see Table 4.6), but this advantage is squandered by the much larger differences with original data observed in win-shift and lose-shift. Indeed, there is a significant main effect of simulations on win-shift (F (0.1, 1.8 = 47.7, p < 0.0001) and on loseshift (F (0.2, 3.8) = 61.0, p < 0.0001). Post hoc Bonferroni tests (Table B.1) and side-by-side representations of simulations and experimental data (Fig. B.3 and B.4) that there is a constant overestimation of shifting tendencies. This is in line with the explanations of [START_REF] Katahira | The relation between reinforcement learning parameters and the influence of reinforcement history on choice behavior[END_REF] regarding the link between forgetting and shifting/persistence and the conclusion that a forgetting rate which is inferior to the learning rate (which is the case of a model without forgetting for which α 2 = 0) produces a shifting tendency.

Appendix C

Parameter evolution of alternative models in response to dopaminergic inhibition

In this section, I will briefly present the parameter variations of alternative models to the forgetting Q-learning model which I chose to focus upon in Chap. 4. This will illustrate the robustness of the main conclusion that dopamine inhibition primarily affects whichever parameter controls random exploration. An additional four models were optimised separately on each pharmacological condition:

C.1 Standard Q-learning model

As explained in the annex Chap. B, this model was not used in the main text because of its very poor fit to experimental win-shift and lose-shift curves. However, although there is a global increase in win-shift levels compared to experimental data, this model is also sensitive to flupenthixol in a similar way: as the dose increases so does win-shift behaviour. If we turn to the values of the two parameters, α and β, we find the same result as for the forgetting model (

C.4 Meta-learning on β model

Finally, since it seems that dopamine plays a role in regulating β, I was also interested how one of the meta-learning on β models from Chap. 5 would react to flupenthixol inhibition of dopamine. I chose the model which uses the current rate of reward prediction errors as a regulatory signal η t : with α η , a special learning rate for integrating this regulatory signal. The inverse temperature is then determined simply by a linear function of η:

with β 0 and β 1 the minimum and maximum values allowed for β. After optimisation, the only parameters found to significantly vary between doses was β 1 , the maximum value of β which decreased significantly, although the statistical effect is in fact quite modest (Friedman ANOVA:

Appendix D

Demonstration that even in the absence of ITI value revision US RPEs cannot converge to 0

This demonstration serves to explain the effect observed in supplementary Fig. D.2 in which it is observed that contrary to experimental data, there is residual activity at US delivery time. I will first demonstrate that the value of the magazine from one trial to the next follows an arithmetico-geometric progression ("suite arithmético géométrique" in French), the limit of which we will then determine and prove to be different from the value of the food reward, even in the case where there is no ITI down-revision of the magazine value, thus resulting in a constant error signal. If we take a GT trial in which the animal progresses from state 0 to state 1, from state 1 to state 4, from state 4 to state 7 and finally from state 7 to state 0 of the next trial, then the value of the magazine is revised three times:

• from state 1 to state 4:

• from state 4 to state 7:

• from state 7 to state 0 of the next trial:

Combining these three equations we get: V s 0 ,t+1 = (1-α)(1-α+αγ) 2 ×V s 0 ,t +α×r which is indeed an arithmetico-geometric progression. Because the absolute value of the term in front of V s0,t is inferior to 1, we are assured that this sequence converges, and that its limit will be α×r 1-(1-α)(1-α+αγ) 2 which is different from r. Thus there will always be a persistent difference between the reward and the value of the magazine as illustrated in Fig. D.2. This persistent difference would disappear if the discounting factor was set to 1, i.e. if no discounting takes place, as

would be equal to 1.

Demonstration that even in the absence of ITI value revision US RPEs cannot converge to 0

If we had taken the ITI into account, then we would have to include the following transition :

and the value of the magazine would then converge to 1 -u IT I which is closer to 1 as the ITI duration diminishes, in the absence of discounting (γ = 1).

Demonstration that even in the absence of ITI value revision US RPEs cannot converge to 0 a. In these simulations, having found that the FMF value of the lever seemingly keeps increasing, we continued simulations for a much greater number of sessions to see whether the FMF value of the lever would eventually catch up with that of the foodcup and the effect this would have on behaviour. a. FMF values of the lever and food cup in the short ITI condition. Contrary to the long ITI condition (see main text and Fig. 6.6) where FMF values converge quite rapidly, the value of the lever keeps increasing and given enough time overtakes the value of the food cup which stays high. b. In the short ITI condition, approach to the lever initially drops as the difference between magazine and lever values increases, but eventually should increase, contrary to what we observe in the long ITI condition where behaviour converges towards sign-tracking quite rapidly and stays stable. c. In parallel, the opposite should be true for the approach to the magazine, with an initially strong goal-tracking tendency in the short ITI condition progressively weakening, but remaining high nonetheless.