Keywords: Contrôle cognitif, Neurosciences computationnelles, apprentissage par renforcement, Cortex Préfrontal Computational modeling Cognitive Control, Computational Neuroscience, Reinforcement learning, Prefrontal Cortex

Modélisation du contrôle cognitif pour le comportement guidé par les règles Résumé : Le contrôle cognitif est la capacité générale d'un organisme à inhiber le comportement dominant en faveur d'une réponse pertinente selon des objectifs internes et en lien avec des facteurs environnementaux et/ou motivationnels. Diverses études expérimentales ainsi que des modèles computationnels ont tenté de mettre en évidence les mécanismes et les structures neuronales sous-jacents qui autorisent un comportement à la fois flexible et adaptatif. Néanmoins, une théorie unifiée qui tiendrait compte de l'ensemble de ces mécanismes reste insaisissable, notamment en ce qui concerne le degré d'adaptabilité qui varie entre les humains et les animaux non humains. Dans ce travail, nous souhaitons caractériser cette gradation du contrôle cognitif afin de poser un cadre conceptuel nous permettant de concevoir des modèles informatiques biologiquement plausibles à même de mettre en évidence les étapes clés du contrôle cognitif.

Dans une première approche, et sur la base d'études chez la souris, nous utilisons un modèle acteur-critique standard afin de montrer comment le comportement naturel d'exploration de la souris doit être inhibé afin de permettre au modèle d'apprendre une règle simple dans un labyrinthe radial. Au travers d'une série de tâche de complexité croissante, nous montrons alors la nécessité de posséder des systèmes de mémoire de travail et épisodique, en adéquation avec la littérature sur la prise de décision chez les rongeurs. Cela est notamment réalisé en étendant le modèle précédent avec une modélisation fonctionnelle de ces deux systèmes de mémoire, nous permettant ainsi de caractériser les contributions respectives de ces deux systèmes, en accord avec les études chez les rongeurs.

Dans un troisième temps, nous mettons en évidence la nécessité de former des représentations explicites du contexte à partir de règles acquises implicitement, et ceci, afin de pouvoir acquérir un comportement spécifique vis à vis d'un contexte particulier. Enfin, pour comprendre comment le cortex préfrontal soutient cet apprentissage contextuel et autorise une pleine capacité du contrôle cognitif chez l'Homme, nous proposons un modèle hiérarchique global qui explique notamment le rôle de l'attention sélective dans l'apprentissage de règles abstraites. Notre hypothèse étant que cette capacité d'attention permet la sélection des règles concrètes les plus appropriées ainsi que la manipulation des représentations sous-jacentes. Tout cela étant réalisé en assurant le monitoring des ces représentations ainsi que les erreurs de prédiction.

A

I owe my deepest gratitude to my thesis supervisors, Fréderic Alexandre, and Nicolas Rougier, for their guidance over the last three years. The lessons I have learnt from you are too many to list, but rst and foremost, you have taught me that the research journey is about asking the right questions, to look at both the big picture, and the minutest details. You have let me arrive at them in my own time and pace, and encouraged me to think for myself. No discussion with you has ever left me feeling less than inspired, and invigorated about this incredibly interesting topic that has become my thesis. Thank you for your patience, your detailed feedback and your constant encouragement. More than anything, thank you for believing in me. It has been a pleasure and honor to do research under your wings.

My work would not have been complete without the collaboration with Christopher. Discussions with you have been very in uential in my understanding of experimental behavioral work, and has sharpened my sense of where we, as computational modelers t in, and what we are seeking to explain. Special thanks to Naomi -research as a PhD student can be lonely, and co-working with you was inspiring, and motivated me to resolve the nitty gritty details of the computational process. I extend my thanks also to Nathan and Xavier, who helped us with digging into the "reservoir" with their expertise.

I would also like to acknowledge the contribution of the "Controlled Contextual Learning" group in my work -Thierry and Jianyong who led the group and initiated excellent scienti c discussions. Thanks to Hugo, with who I was able to discuss the larger perspective of PFC-hippocampus interactions, and Chloé, for the symbolic processing perspective.

Although not directly related to my research, there are a number of people without who, this journey would not have been possible, and to who I will always be indebted.

I would like to extend my sincerest gratitude to the former team of Mnemosyne -Anthony, Thalita, Pramod, Ikram, Bhargav, and beloved room mate Silvia -who made me feel so at home in this city from the rst day, and throughout the lockdown days of covid. We didn't get to spend much time in the same city, and yet, your absence was always felt. v To the entire team of Mnemosyne and Neuroprosthetics -Nikos, Melodie, Ankur, Axel. The years have passed by in a blink in between the co ee breaks, the memes, and the laughs.

To my favorite colocs -Giuliano and Fjola -you both made a "home". My sanity was preserved because of our daily dinners, our evening chats. Thank you for being my companions, for celebrating my small wins, and lifting me up when I was down.

To Remya, my constant over a decade, through all the ups and downs. You have made every hurdle simpler, a little easier to cross. You are my rst reach out in times of trouble, that is su ce to say.

To my friends in London -Yash, Saumya, Garg (and Drushti in Amsterdam), thank you for keeping your doors open for me always, for being my cheerleaders and my much needed respite. Nam, Simran, Jigar, and Arwa -thank you for pampering me, I will nd my way to your couch again. You have made a new meaning for home. My friends across the seas -you know who you are, but specially Arnav and Sudhanshu -thank you for your support.

Finally, to my one rock solid support system, my north stars -Siddharth, Vrinda.

I

The science of AI is concerned with the study of intelligent forms of behavior, in computational terms. But even to de ne the term intelligence is notoriously dicult. Is AI able to tell us when a good semblance of behavior can be achieved using cheap tricks that seem to have little to do with what we intuitively imagine intelligence to be ? Take ChatGPT for example. That it can automatically produce what me might consider to be humanlike text is remarkable. But roughly speaking, such AIs take huge amounts of data, search for patterns in it and become increasingly pro cient at generating statistically probable output. So then, are these intuitions wrong, and is intelligence really just a bag of tricks ? Or are the philosophers right, and is a behavioral understanding of intelligence simply too weak ? [ ] To begin asking speci c questions about behavior, we need to rst de ne what sort of behavior we care about. Di erent researchers will quite naturally focus on di erent aspects. The behavior may or may not depend on perceptual or motor skills. It may or may not include learning. It may or may not be grounded in emotional responses, or in social interactions. From the perspective of neuroscience, if the question is "How does the brain lead to behavior", the pertinent question to rst ask is why is the brain performing this behavior and then asking how is it doing it [ ]. Moreover, understanding something is not the same as just describing it or knowing how to intervene to change it -an aspect on which AI and neuroscience have signi cantly diverged.

Consider the decisions we make on a daily basis about how to get around. Our preferred mode of transport is based on multiple individual, social and situational factors. Seemingly on daily recurrent journeys, it is a behavior we rarely intentionally think about. Depending on how one has grown up, the city one lives in, and how far one has to travel, we usually default to certain habitual patterns, or past repetitive behaviors. Yet, certain external or internal levers seem to be able to modify this choice, leading to deliberation and possibly the choice of an alternate behavior. These contextual factors could be available infrastructure (maybe there is a strike), weather conditions (biking or walking on a rainy day is unappealing), Contents time constraints (you have an appointment and are running on a tight schedule), or even internal motivational factors (you want to hit your target of , steps a day) . Decision making is rarely ever as simple as choosing between two options. In the complex environment we live in, amending our behaviors is an explicit, deliberative, voluntary process that requires Cognitive Control.

For the purpose of this manuscript, it is this kind of adaptive behavior that is the spotlight of our study, or more precisely Cognitive Control over rule guided behavior. Many cognitive abilities in human and non human primates, such as inferential reasoning, planning, social interaction and exibility in adapting to novel situations are known to be strongly dependent on the formation and implementation of rules, and impairments in such cognitive processes have been reported in a range of neuropsychological disorders.

There has been much focus on systems neuroscience in the past few decades, with the advent of advanced recording techniques pushing the envelope on the microscopic scale at which we can record neural data. It remains the case though that it is very hard to infer the mapping between the behavior of a system and its lower-level properties by only looking at the lower level properties. To quote Marr on the inadequacy of a strictly neurophysiological approach to understanding : trying to understand perception by understanding neurons is like trying to understand a bird's flight by studying only feathers. It just cannot be done

In other words, understanding what process X is -what computations it embodies -can rarely be done at the neural level alone. Even when looking at neural data alone, some authors argue [ ] that a mathematical bias by researchers may actually hinder our understanding of brain and cognition, the interpretations of said data, or even what data to look for in the rst place. Using the impact of mathematics in the eld of decision making and as explained in [ ], they quote an example : nearly all theories of decision, from expected utility theory through prospect theory and even modern reinforcement learning algorithms have shared the notion that in order to choose, the di erent attributes of each option must at some point be converged, however idiosyncratically, incompletely and imperfectly, into a single value for the actual process of comparison. They argue that this notion of value as the common currency equated to decisions may be a mathematical bias since mathematics o ers no other way of comparing two unrelated objects.

On the behavioral science side, many have argued that examining behavior itself is more valuable that looking at individual neurons or neuronal assemblies [ ].

Contents

This idea is explained well through an example: A rat is trained to run through a Tshaped maze to reach food, and at the junction of the T, it must decide whether to turn right or left. The rat may use either internal cues (turning right relative to its own body) or external cues (turning towards a speci c location in space) to make this decision. However, it is quite a challenge to determine which strategy the rat is using by recording brain activity alone. To investigate this behavior, Packard and McGaugh (

) [ ] turned the maze around and observed the rat's behavior. If the rat continued to turn right, it would indicate an egocentric strategy, but if it turned left, it must be following external cues. This manipulation allowed for a better understanding of the neural strategies used by the rat and led to followup experiments to determine the conditions under which rodents use allocentric rather than egocentric strategies. Ultimately, this research led to computational models that speci ed the types of data that the rat must learn and use in each condition, and the computations that may support transitioning from one strategy to another.

In the eld of computational modeling of cognitive science, the discourse has moved from the connectionist framework that previously held much promise as an explanatory model of the brain, to a bayesian modeling of cognition [ ]. Most proposals is these frameworks start from the same computational principle, that a goal function must be optimized. This may be an error function (to be minimized) in the PDP tradition, a value function (to be maximized) in the RL tradition, or a posterior probability function (to be maximized) in the bayesian tradition. Examining the claims of deep learning models of neuroscience, Schaeffer and colleagues in [ ] have demonstrated that the results of these models are more strongly driven by particular, non-fundamental, and post-hoc implementation choices than fundamental truths about neural circuits or the loss function(s) they might optimize. These authors conclude that such models thus cannot be expected to produce accurate models of the brain without the addition of substantial amounts of inductive bias, and as such don't hold any explanatory sense for neuroscience.

Clearly, in trying to ask the right questions to investigate what mechanisms lead to adaptive behavior, one needs to traverse this rich and complex academic landscape, with regards to what normative approach to use, what assumptions to make, what modeling tradition to follow, what level of description is most useful etc. This manuscript is hence divided in two parts : Part I lays out the groundwork and the current understanding of Cognitive Control from the perspective of cognitive science, neuroscience, and computer science.

In chapter , we begin by presenting a conceptual overview that serves as the foundation for the rest of this manuscript. We explain classical behavioral theo-Contents ries that have formed our understanding of learning, and subsequently we motivate the need for Cognitive Control. The second half of the chapter then invokes conceptual notions associated with Cognitive Control -in terms of constructs, rules, context, abstraction, hierarchy, and in terms of mechanisms, working memory, attention and performance monitoring. Agnostic of the species, we present a conceptual view of Cognitive Control as a gradient, with di erent kinds of tasks requiring di erent degrees of control.

In chapter , we then provide the neuroscience evidence for the mechanisms involved, and discuss how di erences in the cortex anatomy of di erent species can underlie the di erence in complexity of cognitive control observed in their behaviors. Further, we gather evidence on the functional subdivisions in the primate prefrontal cortex and the roles they might subserve. We put forth a functional sketch of how these di erent brain regions interact with one another in learning, and cognitive control.

In chapter , we review computational models of cognitive control, analysing to what extent these existing models are able to explain and integrate the computational principles and biological mechanisms identi ed with cognitive control. At the end of this chapter, we identify some open questions with regard to the synthesis of the literature reviewed in the rst three chapters.

Part II then details our particular contribution to the story of Cognitive Control, using di erent kinds of computational models. In chapter , we present a series of computational models to identify the roles of the working and episodic memory learning systems, default behavior and context, in the kind of cognitive control at work in rodents.

Finally, in chapter , we present a hierarchical model, in which superior layers in the hierarchy deploy selective attention to select among multiple possible options of responses at the lower layer.

We conclude this manuscript by a summary of our main ndings and an introduction to some fascinating perspectives it has opened.

P I C C

A comprehensive literature review C I In , Skinner wrote the "Principles of Behavior" to present in an objective, systematic manner the primary, or fundamental, molar principles of behavior, with the assumption that all behavior, individual and social, moral and immoral, normal and psychopathic, is generated from the same primary laws [ ]. While we have come a long way in the study of behavior since the publication of that book, the motivation remains the same and a long line of studies continue to deepen the work of unraveling the principles of behavior and cognition.

In that regards, a central and ubiquitous element of behavior is the capacity to make choices, that is, a commitment to a proposition among alternatives that arises through a process of deliberation. A series of decisions makes a pattern of behavior and consequently, the inner mechanisms of decision making provide a window on cognition. Many cognitive abilities in human and non human primates, such as inferential reasoning, planning, social interaction and exibility in adapting to novel situations are contingent on complex decision making. These kind of executive functions are higher level functions for the control of cognition. Over the last few decades, scholars from economics, psychology, cognitive science, evolutionary biology, computer science and neuroscience have tried to provide a framework for investigating the neural mechanism of choice behavior, poised to investigate decision making at the theoretical, algorithmic and implementation levels [ ].

This chapter aims at gathering information towards an important and di cult question : How does the brain choose e ciently and adaptively among available options to ensure coherent, goal-directed behavior ? Hidden behind this question are many problems that necessitate a multidisciplinary approach. Indeed, to understand how humans and other animals solve this problem, we need answers from researchers versed in anatomy, traditional psychology, learning theory, neuroimaging and mathematical modeling. Central to nearly all de nitions of exec-

Conceptual overview

utive function are two concepts : rules and control. The rules that guide human behavior -and the behavior of many other but not all organisms are abstract and exible. Control processes allow us to engage rules appropriate to a particular context. These two aspects -( ) creating and modifying rules for behavior and ( ) engaging the appropriate rule for a particular context-represent the highest levels of a taxonomy for executive function and thus re ect major sections of this chapter.

T

In the formalism of reinforcement learning, any kind of decision making rule or strategy consists of "a mapping from perceived states of the environment to actions to be taken when in those states" [ ]. Historically, how reinforcement learning conceptualizes this mapping, is rooted in experimental and animal psychology. The study of animal conditioning is broadly divided into two main areas : Classical or Pavlovian and instrumental or operant conditioning.

First, described by Ivan Pavlov, Classical, Pavlovian or sensory (respondent) conditioning focuses on involuntary, automatic behaviors. It is a process that involves pairing a previously neutral (conditioned) stimulus (e.g., a bell) with an appetitive unconditioned (or biologically signi cant) stimulus (e.g., food); after repeated exposure to the food following the bell, the bell starts to elicit re exive responses normally reserved for the food (e.g., salivation). It is the expression of innate knowledge (in other words, the decision making structures presumably evolved to handle natural rewards (reinforcers) such as food, water, and intrinsic threats) of what responses are usually appropriate when certain types of events are observed to be correlated in the environment. In the pavlovian scheme, responses are thus stereotyped (also called pavlovian re exes) and are consequences of learned associations. The Rescorla-Wagner (RW) [ ] model, arguably the most in uential model of animal learning to date, provided the mathematical formalism that could capture this phenomenon by postulating that learning occurs only when events violate expectations. For instance, in a conditioning trial in which conditional stimuli CS 1 and CS 2 (say a light and a tone) were presented, as well as an a ective stimulus (food), the unconditional stimulus U S, the model postulates that the associative strength of each of the conditioned stimuli V (CS i ) will change according to

V new (CS i ) = V old (CS i ) + η(CS i , U S) × [λ(U S) -i V old (CS i )]
Whereas, based on CS, pavlovian conditioning passively predicts the US to occur and prepares the body for this event, another more active learning scheme called Operant or instrumental conditioning focuses on using reinforcement (positive or negative) to increase or decrease the strength of a voluntary behavior. Notably, Thorndike (

) rst presented the concept of Law of E ect, which is the idea that a response will be triggered more (or less) frequently when observed to lead to a positive (or negative) consequence. Alternatively, a response will be triggered more frequently if it leads to the avoidance or removal of a negative stimulus. The early work of behaviorist experimentalist and theorist B.F. Skinner used the terminology positive reinforcers and negative reinforcers to describe these two. Computationally, this kind of learning is treated as attempting to optimize the consequences of actions in terms of some long term measure of total obtained rewards (and/or avoid punishments).

Since here, the response is voluntary, it is possible to consider the corresponding level of need, devaluate the outcome and refrain from acting if the motivation is low. Whereas pavlovian conditioning simply de nes how much an outcome is liked (thus described as Stimulus-Response associations), instrumental conditioning considers how much it is currently wanted and chooses to trigger responses taking motivations into account (hence described as Response-Outcome associations). These can be extrinsic motivations, to get a desired (external or extrinsic) outcome satisfying fundamental needs, or more complex internal representations, expressing intrinsic motivations, as described in more detail in [ ]. They are related to a more abstract need of (intrinsic) information, to obtain from the exploration of the complex world and from the monitoring of internal activity, as is the case with curiosity and attention towards novelty.

This categorization of behavior was motivated by the observable responses an animal made or could be made to exhibit through experimentation. On the other hand, another categorization of behavior has been motivated by the driving factors of behavior, i.e. about the internal or external environmental triggers for particular behaviors. Hence, external and stimulus-driven on the one hand or internal and goal-directed on the other, where a goal is any mental representation of a desired activity (which need not be a biologically signi cant stimulus). Instrumental conditioning can be performed under the control of (or conditional to) stimuli also called occasion setters, that can become conditional reinforcers, leading to chaining in complex behavioral goal-directed sequences towards primary reinforcers (respectively de ned as subgoals and goals in planning). Conversely, these associations can be transformed in habits through extensive learning, where the conditional stimuli directly elicit responses without references to the outcomes to be obtained. More generally, this refers to the dichotomy (cf section . . ) between

Conceptual overview

goal-driven behavior (where the behavior is driven by internal goals and can adopt complex schemes) and stimulus driven behavior (where the agent mainly reacts to perceived stimuli).

G

We have seen above that it is important to identify stimuli in the world as possible goals of behavior (emotional or pavlovian learning) and relate them to the corresponding need they can satisfy, to decide if it is worth triggering responses to get them (motivational or instrumental learning) [ ]. Often the formulation of perception and action is considered as the beginning and end of a linear process, a stimulus -response arc. Organisms however have goals and agency, with behavior being more akin to a control loop, with inputs modifying outputs that in turn modify the next set of inputs to achieve an ecological life sustaining goal ("Those which are most useful to the organism"). Here, we refer to behavior as the internally coordinated response (external or internal actions) of living organisms to internal and external stimuli. With such consideration of behavior -as that which leads to responses monitoring a transition from one state to another, of the type Situation -Action -Situation (S -A-S ), any behavior can be viewed as initially, and essentially goal driven. S can be interpreted as the initial condition eliciting A as the possible response and S as the consequence that can be anticipated if A is preactivated. Conversely, if S is the desired state, A is the response that has to be activated to reach S , which is possible if S is compatible with the current state. Else, A can display a sustained activity, as in working memory (cf section . . ), and remain actively waiting until S is satis ed.

More generally, throughout this thesis, we elaborate on behavior through these key ingredients :

Sensations : which can be internal or external; local or global (contextual), referred to as Stimulus for simplicity The sensory cortex encodes this sensory information (called posterior cortex for simplicity, since most of them are posterior to the central sulcus

Response : which can be external or internal, in which case it is referred to as Action (thus this could be an internal action of updating the working memory)

The motor and premotor cortex encode information related to external responses, while the limbic frontal regions are responsible for internal responses (e.g. selection of goal and motivation)

Outcome : which relates to the consequences of actions, and refers to the reinforcing aspects of the pain / pleasure circuitry These are encoded in the insula or medial temporal lobe, depending on the emotional and motivational aspects.

To illustrate this point, let us try to examine the ingredients of the basic behavior of hunger satisfaction (the brain circuits mostly involved in triggering this behavior are mentioned in parenthesis, as it will be more clear in Chapter ), as di erent brain regions participating in the answer to one of four fundamental questions :

• What is the goal of my behavior ?

• Why should I spend energy satisfying the corresponding need (and upto which level) ?

• Where is this goal ?

• How should I behave (which response should I trigger) to get it ?

The perception or feeling of hunger (in the insula, hypothalamus) activates the motivation to eat (in the dorso medial prefrontal cortex (dmPFC; pgACC)). In order to trigger an appropriate action, rst the progress towards the desired goal needs to be checked. If food is already present in the mouth, the animal needs to eat until satiety. This is checked by the ventro medial prefrontal cortex (vmPFC), which also represents the value of the expected reward. If the check is yes, then the sequence of actions for eating is triggered (masticating and swallowing in the motor cortex); if not, then it becomes a desired goal (in the lateral orbito frontal cortex (lOFC)). In the next sequence of actions, an intermediary goal is to get the food, and the cost to get the food is evaluated (in the vmPFC (sgACC)). The dmPFC (pgACC) compares the reward and cost to decide to act. In this case, food as a desired goal in the lOFC sets attention on all food like stimuli, sequentially evaluating the reward and cost until a choice is made. The cost, a ordance or the goal itself is linked to comparing the strength of the goal, the urgency of the action and the con dence in the outcome.

In the example presented above, we see that even for the simplest of goals, a number of di erent regions of the prefrontal cortex (PFC) need to collaborate to achieve the said goal. We will focus on the role of each of the mentioned areas in the second chapter. However, all of these regions are not usually involved when we eat because more often than not, such behaviors occur under a normal, stable context, and this habitually used behavior is activated without much supervision. Nevertheless, the role of the prefrontal cortex goes beyond simply the ability to Conceptual overview navigate the sequence of reward and cost evaluation; its role is fundamentally to allow for the manipulation of existing behaviors.

On one hand, in simple and stable worlds, the most obvious solution becomes the dominant or default behavior, becoming strengthened after each success. Furthermore, after enough repetitions, certain actions are compiled together as chunks and directly trigger cached values, becoming stimulus driven. After a su ciently long training period, they might become habitual, and rigid, where the current state is enough to directly trigger the response with no need to refer to a priori model of sensorimotor transitions or to the value of the outcome. On the other hand, in certain contexts, several close solutions to the same goal might be possible, or the dominant behavior might become erroneous due to a volatile and changing environment. This necessitates the need for cognitive control (cf section . ) in order to resolve the con ict between several possible responses, and the continuous need of an explicit deliberative system, that is able to track performance on multiple time scales, signalling the need to switch to an alternate or exploratory strategy. Said simply, to nd the best global solution, contingencies between local decisions and their consequences must be evaluated, and corresponding reinforcements must be compared. This can also be associated to the domain of planning, with the classical steps of deciding for goal, motivation, strategy and execution, and of backtracking in the hierarchy when one step is impossible. We know that the brain has the capacity to assign control to the appropriate regions for each of these kinds of behaviors. In the following sections, we elaborate on the precise de nitions of these behaviors, and review the current view in the eld.

S G B

Behavior has been traditionally divided into two categories : external, stimulus driven on the one hand, and internal, goal directed on the other hand. This traditional viewpoint induces two competing requirements -(i) to respond quickly to familiar situations while being able to (ii) adapt to novel ones and plan for the future.

Quickly responding to the immediate environment is accomplished when familiar stimuli activate well established neural pathways, producing stereotypical behaviors. These behaviors are executed quickly because they are based on stable, "concrete" rules ( . . ), i.e. they are grounded on speci c stimulus-response associations. Even though the acquisition of such associations is a slow and incremental learning process, the triggering of these associations is done with little need for control nor oversight. In this kind of behavior, stimuli (or states of the world) trigger actions, which then produce outcomes (new states of the world).

In contrast, sophisticated, goal directed behavior requires the ability to act on, or predict future events, rather than just react to the environment. Goals, or mental representations of desired future states of the world determine actions, lead to new states of the world, which can be compared with those envisioned as goals.

The aforementioned distinction does not constrain or formalize the neural or cognitive mechanisms that disassociate the two systems. In the eld of machine learning, and more precisely, reinforcement learning (RL), this dual system theory of behavior has been formalized as a mapping to model free (MF) and model based (MB) forms of RL. While RL algorithms can be categorized along many dimensions, MB vs MF algorithms are contrasted based on the extent to which they represent the environment or not, and how they respond to changes in the environment or the agent's goals. MB algorithms maintain a representation of the problem beyond the state and action space, usually by including the transition and reward function. This internal model enables the agent to guide it's decisions by considering the consequences of its actions. These algorithms thus adapt more readily by leveraging the task model to dynamically plan toward an arbitrary goal. However, maintaining the task model and computing an action plan can quickly become intractable. On the other hand, MF algorithms do not maintain such an explicit model. Instead, they store a set of estimates, each representing the aggregated reward history of choices made by the agent in the past. While such algorithms cannot adapt as easily due to their strategy of integrating reward history into a single value estimate, they o er an e cient approach to learning and decision making [ ].

It may be considered natural to identify the value of mapping model free RL to stimulus driven behavior since it is expressed in the stimulus response outcome form, and to map model based RL to goal directed behavior due to the internal model or cognitive style representation of the world. A MB strategy involves prospective cognition and assessment of the consequences of taking particular actions and supports the computation of value transformations when relevant conditions change -all features of goal directed behavior. Similarly, like stimulus driven behavior, MF systems are reactive, making future estimates only based on reward values encountered in the past and caching information about the utilities of those outcomes. However, such clear distinction is not always possible, and the two models might overlap. For example, standard MF algorithms augmented with additional computational machinery like a working memory, can mimic a MB planning strategy [ ].

To the extent of nding neural correlates of these systems in the brain, it was the temporal di erence (TD) learning mechanism, that RL is based on, which sparked a turning point in the understanding of dopamine function in the brain.

Conceptual overview

The phasic ring patterns of dopamine neurons in the VTA were shown to mirror the characteristics of a TD-RL reward prediction error. This has provided a bridge between behaviorally descriptive models and a functional understanding of a learning algorithm that may be supported by the brain. Thus, model based and model free (and similarly stimulus driven and goal directed) algortihms (in the context of learning and decision making), are better understood as the two ends of a high level process that emerges through the coordination of many separable sub-computations. However, these dichotomies may not be helpful in identifying those unique and separable mechanisms underlying behavior.

The aforementioned distinctions are useful in explaining behavior from the point of view of the observed input and response (SD vs GD), or as a computational formalism (MF vs MB). Nevertheless, they are insu cient to unravel computational and mechanistic primitives supporting behavior. In the next section, we review accounts of behavior in terms of processing. Shifting the focus to this axis may provide a more clear path towards nding computational primitives liking brain and behavior.

A

C P H C B

The view that human cognition may comprise two di erent types of processing, automatic and controlled, has been a theme in the psychology literature for a long time.

In earlier papers [ ], an automatic process was de ned as the activation of a sequence of nodes that nearly always becomes active in response to a particular input con guration, and that is activated automatically without the necessity of active control or attention by the subject. In general, automatic processes are thought to operate through a relatively permanent set of associative connections, requiring an appreciable amount of consistent training to fully develop. According to these authors, controlled processes are de ned as those processes that rely on attention for execution, while automatic processes are those processes that can be carried out without attention [ ]. For example, in the Stroop task ( . . ) (where participants are cued to determine either the ink color (color naming) or the color name (word reading) associated with a word stimulus), color naming was considered to be controlled because it relies on attention. Without attention to color, subjects will read the word by default. Furthermore, the color has no impact on word reading, even when it con icts with the word being read. Conversely, word reading is automatic because it does not appear to rely on attention. Even when asked to name the color, if a con ict word is present, it slows the response to the color.

Conceptual overview

However, there are problems with this simple dichotomous distinction. First, it is not clear that any cognitive process can occur entirely independent of attention. For example, although an individual is reading the words on this page, presumably they are not reading the words out loud. Second, it is not clear that color naming is always dependent on attention and control. In essence, such accounts propose that processes that demand control are distinguished from automatic processes, which involve associations su ciently strong as to be resistant to distraction or interference [ ].

All processes rely on attention to some degree, and this may vary in a graded fashion. Thus, some processes are more automatic than others, and processes vary in their automaticity based on the context in which they occur. This axis of automaticity starts from coordinating con icting responses demanding contextual control on one end, to readily triggered default behaviors that can turn into rigid unchangeable habits on the other end.

Default behavior : Routine sensory conditions tend to elicit some kind of default response, which might either arise due to some kind of innate knowledge about the world, resulting from an evolutionary process (think about the re ex action of ee or freeze when a prey encounters a predator), or due to prior learning which has been reinforced repeatedly.

In the initial stages of habit learning, behaviors are not automatic. They are goal directed, such as for example, an animal working to obtain a food reward. When goal directed behavior consistently produces the same behavior in response to the same stimulus, that behavior is likely to become a habitual response. Once a habit is formed, behavior can become in exible in the face of changes to the environment that render it no longer desirable. In experimental settings, with extended training or training with interval schedules of reward, animals typically come to perform the behaviors repeatedly, on cue, even when the value of the reward to be received is reduced so that it is no longer rewarding (for example, if the animal is tested when it is sated or if its food reward has been repeatedly paired with an aversive outcome). Habits are promoted in situations where the contingencies between action and outcomes are weak. Particularly, when the values of actions are divorced from outcomes, habitual control takes over. Dickinson de ned the goal oriented, purposeful, non habitual behavior as action outcome (A-O) behaviors and labeled the habitual behaviors occurring despite reward devaluation as stimulus response behaviors (S-R). From simple motor actions to choice of meals, travels and exercise routines, behaviors become more automatic (faster, more accurate, less susceptible to interferences) the more often those behaviors are performed in the presence of a particular set of cues.

In the consciousness literature, the study of habits invokes this dichotomy between the conscious, voluntary control over behavior, which is considered the essence of higher order deliberative behavioral control on one hand, and lower order behavioral control that is scarcely available to consciousness, on the other hand. To consider the de ning characteristics of habits, through this lens, four key points become evident [ ]. First, habits are largely learned i.e. they are acquired via experiment dependent plasticity. Second, habitual behaviors occur repeatedly over the course of days or years, and can become rigid and xed. Third, fully acquired habits are performed almost automatically, allowing attention to be focused elsewhere. Fourth, they tend to invoke an ordered, structured action sequence that is prone to being elicited by a particular context or stimulus. Thus, these characteristics suggest that habits are sequential, repetitive, motor or cognitive behaviors elicited by external or internal triggers that once released, can go to completion without constant conscious oversight. In addition, habits can be de ned experimentally as being performed not in relation to a current or future goal but rather in relation to a previous goal and the antecedent behavior that most successfully led to achieving that goal.

Coming to the experimental literature, this distinction between action-outcome vs stimulus-response systems is an ongoing debate [ ] [ ]. Evidence suggests that these are not independent "systems". For example, after training that produces habitual behavior in rats, goal oriented behavior can be reinstated if the infralimbic prefrontal cortex is inactivated [ ]. This nding has led researchers to believe that the circuits controlling goal oriented behavior may be actively suppressed when behavior becomes habitual. The idea that there is a dynamic balance between control systems governing exible cognitive control and more nearly automatic control of behavior responses supports the long standing view from clinical studies that frontal cortical inhibitory zones can suppress lower order behaviors. This view has become especially important in models of system level interactions [ ].

The distinction described above and reiterated in scienti c literature, between controlled and automatic processes is useful in shedding light on two important mechanisms that are involved in cognition -(i) attention to relevant features (or on the contrary, ignoring irrelevant ones), and (ii) strength of action -outcome associations based on context . Yet, it still does not provide a clear picture of how basic learned behaviors might be manipulated and coordinated.

The study of learning and decision making, in both human and nonhuman animals has broadly been studied under the idea that behavior is governed by two separable controllers. Behavior has thus been dichotomized against several dimensions, including emotion (Hot/Cold), action selection (habitual/goal-directed), judg-Conceptual overview ments (associative/rule-based) and reinforcement learning (model-free/model-based) [ ]. Critics have pointed to the multitude of dual processing accounts, the vagueness of their de nition, and the lack of coherence and consistency in the proposed cluster of attributes for the dual system accounts. Largely, these dichotomies are subsumed under the terms System and System thinking. In the above sections we have tried to disentangle the de ning versus the correlated features in some of these accounts. In the following section, we focus only on the control aspect of behavior -how it might arise, its minimal de ning characteristics, and the neural underpinnings that support the implementation of such control.

C C

To grasp an intuitive understanding of how Cognitive Control is implicated in behavior, let us look at a few examples : two based on daily life, and two based on laboratory tasks. Consider the sequence of actions required for driving. Any experienced driver understands and follows standard rules without much thought or e ort -when to stop at tra c signals, when to accelerate and break and what side of the road to overtake from. Until this driver visits the UK on holiday, and now needs to modify some of these rules for driving because the context (environment) has changed -the driver needs to change the side of the road (s)he drives on, while still following most of the other basic rules for driving (the right sensorimotor actions for changing gears, etc). The driver also understands, or in able to predict the consequences of not adapting the driving rule : one might need to pay a hefty ne, risk getting into an accident, or even be questioned by the police, and hence there is a signi cant cost attached to the failure to adapt. Alternatively, consider our instinctual response of looking for our phones and swiping towards the green button when we hear a phone ring (which may or may not belong to us); until we nd ourselves in a meeting and this behavior needs to be suppressed. Paying attention to a phone ring is in fact a habitual behavior, even if answering it might be the dominant behavior. In either case, it requires an active suppression of the habitual tendency, which we know to be maladaptive in certain social contexts. In controlled, experimental conditions, we can consider the standard Wisconsin card sorting test (WCST; cf section . . ), where one has to go on following an underlying rule of matching the presented card to a number, color or shape, until the experimenter changes the task criteria. The participant will subsequently start to make mistakes and will need to reconsider the rule to be applied. This requires not only inferring a rule that is not clearly speci ed (as a latent state), but also an active monitoring of one's performance, such as to be able to detect when there is sudden drop and must be attributed to external factors. Yet another example is the Go -NoGo task in which the subject has to mostly make the same response unless the stimulus presented has a certain property.

In each of these cases, there seems to be a default or dominant way of doing things and performing actions, until something in the environment changes and the underlying default (or usual) rule needs to be adapted or suppressed in order to cope with the change in the environment. The adaptation requires one to maintain the new context in mind (working memory), selectively attend to a few features in the environment (selective attention) and monitor one's belief about the current state in the world (performance monitoring). This ability of inhibition or suppression of a default response, in favor of an alternative behavior more suited to the context and environmental changes is generally termed as "cognitive control". It is an ability that both humans and non humans demonstrate ubiquitously, and perform with much ease, all the time. This kind of exibility, at the most basic level requires coordination between motor control and a higher level cognitive control. Here, we refer to motor control as the ability to plan and select speci c actions while cognitive control is the ability to coordinate a set of responses. At the neuroanatomical level, it can be thought of as the di erent roles of the agranular and granular frontal cortex [ ], with the agranular cortex implementing the default behaviors and responses and the granular acting on top to selectively inhibit these responses.

In summary, goal directed behavior requires the selection of task relevant information and the suppression of task irrelevant noise. A prominent characteristic (as seen from the examples above) of most current theoretical models of cognitive control is that this process is mediated by higher level control signals that bias the state of lower level neural processing. The origin of these control signals is commonly seen as originating in the prefrontal cortex. In the context of action selection, this top down control is particularly needed during situations of response con ict, where a predominant response (default behavior) needs to be inhibited in favor of an alternative response or no response at all. This corresponds to the view of cognitive control that the rest of this thesis is based upon : a set of cognitive processes which act not on speci c stimulus response contingencies but on the performances of previously learned, more basic behavioral modules (what we call Rules, cf section . ).

A core problem here is what is referred to in literature, as the stability-plasticity dilemma, which is the degree to which a new instance (rule) should alter existing knowledge about a class of instances (schema, or here, Task Set) without destabilizing such knowledge. The cognitive tasks we perform at each moment, and the e cacy with which we perform them is a result of a complex interplay of deliberate

Conceptual overview

intentions that are governed by internal goals (endogenous control) and the availability, frequency and recency of the alternative tasks a orded by the environment and its context (exogenous in uences). E ective cognition requires a delicate, 'just enough' calibration of endogenous control that is su cient to protect an ongoing task from disruption but does not compromise the exibility that allows the execution of other tasks when appropriate.

Some authors have de ned cognitive control as the general capacity to use an internal contextual representation to guide full pathways of thought and action, in accord with goals [ ] [ ]. In particular, it has been described as the ability to perform task relevant processing in the face of distractions or in the absence of environmental support, speci cally by active maintenance and exible updating of task representations over time, in order to pursue task relevant objectives and

goals [ ] [ ].
Elsewhere, this ability or capacity for control is interpreted as a set of superordinate functions which include working, semantic and episodic memory; perceptual attention; and action selection and inhibition [ ]. The state of control at any instant can be characterized in terms of (a) its direction, the speci c task objectives toward which control is directing subordinate systems; and (b) its intensity, the strength of its top-down input to those systems.

It is evident that allocation of control depends on the circumstances of cognitive demand (which can be interpreted as goals, context, or task depending on the granularity of the de nition) and as such requires the dynamic recruitment of available cognitive processes that can appropriately meet these demands. In this view, cognitive control can be though of as having the following features :

Regulation : The capacity to govern or in uence lower level information processing mechanisms Speci cation : A decision on which controlled tasks should be undertaken and on how intensively they should be pursued Monitoring : The system must have access to information about current circumstances and how well it is serving task demands. It refers to the three criteria to monitor once a goal has been selected : the outcome to reach, the action to perform and a measure of progress

Having established the de ning features of cognitive control, and the general set of functions it uses, there are three focal questions that a computational modeling study of cognitive control might address [ ] :

• How do control functions in uence information processing ? What are the top down regulative e ects of control ?

• How might control emerge from learning and experience ? What are the bottom up factors that govern the selection and modulation of top down control signals ?

• How does the architecture and representation of the control system come to assume its speci c form or functional organization ?

This thesis is an attempt to answer each of these three questions systematically, through biologically plausible computational modelling and coming to a consensus by reviewing the extensive neuroscienti c literature on cognitive control and its associated mechanisms and constructs. Figure . explains this gradient of control we have described thus far, schematically. Speci cally, the full capacity of control that this gure describes requires incrementally such constructs as Goal, Context, Plan, and monitoring. Some of these of have been elaborated in previous sections. In the next section, we begin by delving a little deeper into the constructs and mechanisms that support e ective complex, hierarchical control (as seen in primates) i.e. working memory, attention and performance monitoring.

To talk about cognitive control, we need to rst talk about what this control is exerted over. In any scenario, there are possibly in nite kind of actions that can be taken, any number of di erent behaviors that may be demonstrated. Only some, if not of those is ultimately chosen, while the others are suppressed. Since we carry within us a repertoire of possible behaviors, it is reasonable to start with the smallest currency of any well de ned behavior, by decomposing it into simple routines for selecting actions (or thoughts) -a rule.

After introducing conceptually, the structures and mechanisms needed to understand cognitive control, in the next section, we try to narrow down the denition of a "rule" and its related aspects and categorizations (concrete vs abstract, implicit vs explicit), from the view point of how these constructs are implicated in cognitive control. In the second part of that section we discuss the conceptualisation of abstraction, and theories of how the brain might be organized to support such abstraction with hierarchical organization. We also look at behavioral evidence of rules in animals, and how the de nition of a rule in cognitive science is di erent from that in computer science and logic. Slightly more exible behaviors require the organism to make contextual representations (external as in environmental cues or internal latent states) of the learned associations such that a change in the context is able to trigger a change in the expression of an underlying rule (contextual control leaning). Planned behaviors further involve the organization of sequential behavior in time, that can be triggered by goals (or sub goals) such that once initiated, will reach completion without explicit oversight. Finally, a complete and complex manifestation of cognitive control requires all the above ingredients, in addition to a contextual monitoring of motivations, goals, errors, and self performance so as to recognize uncertainties in the environment and link them to the correct causes.

Conceptual overview

M C

Studying cognitive control from a computational frame of reference, calls for oft used terms in the literature. Below, we brie y describe such constructs, and emphasize the usefulness of such concepts.

G

Goals in general refer to desired or anticipated outcomes or end states. These goals can be the consequence of physiological needs such as thirst and hunger, as well as various other needs or motives such as making friends, acquiring knowledge, etc. Goals are critical because the decision process is assumed to be intended, and perhaps even optimized, to achieve them. Indeed, optimality must be assessed only in the ecological context of a goal. Thus, behavior that is "suboptimal" with respect to certain objective goals such as maximizing accuracy might in fact be optimal with respect to the idiosyncratic goal(s) of the decision maker.

S

At the root of computational models (especially RL) is the concept of state representations, an abstract representation of the task that describes its underlying structure. States can be tied to external stimuli, or they can include internal information that is not available in the environment and must be retained in memory or inferred, such as one's previous actions, or the context of the task.

V

Humans and animals make predictions about the rewards they expect to receive in di erent situations known as value representations. They drive choice : expected value of available options are compared to one another and the best is selected. They support learning : expected values are compared to rewards actually received, and future expectations are updated accordingly. This assumes the notion of common currency to be able to compare things.

Value is the subjective costs and bene ts that can be attributed to each of the potential outcomes (and associated courses of action) of a decision process. Value can be manipulated by giving explicit feedback or monetary rewards to human subjects or preferred food or drink to nonhuman subjects. Value can also re ect more implicit factors such as the costs associated with wasted time, e ort, and resources.

Conceptual overview

C

Contexts encompass internal (cognitive or a fective), interoceptive and external settings. Thus context frames and contains a priori information about reward.

Consider the example of the at vs hierarchical rule discussed in [ ]. When the rule is at, all the cues that play a role in the decision play the same role and are combined at the same level. With a hierarchical rule, we make a di erence between the lower cues which are the main cues and the higher cues which are put at a secondary level and can be considered as context. Context is not a property per se, but depends on the situation; of course, of this 'secondary' cue corresponds to the background which is e ectively often secondary in the decision process. In this case, it is also possible to propose the fact that context is more to de ne the kind of task rather than the main cue that will trigger the action.

U S V

Uncertainty typically arises in a situation that has limited or incalculable information about the predicted outcomes of behavior. Uncertainty can be induced not only by lowering the probability of stimulus-response-outcome contingencies, but also by fundamental changes in these contingencies that forces a modi cation of previous beliefs. It can arise from : Stochasticity or expected uncertainty [ ]; it is inherent in the decision making process; S-R-O rules learned from past events are weak predictors of the outcomes of future actions and this unreliability is known and stable. Volatility or unexpected uncertainty; it arises from fundamental changes in the S-R-O contingencies of the environment that invalidate predictions based on previous experience; it is a variation in the frequency of the changes in existing S-R-O contingencies across time T S

In daily life, humans are constantly confronted with a lot of information from the environment, but only part of this information actually gains access to the cognitive system. Furthermore, we also have a very large behavioral repertoire of likely responses we can make, given even this partial information. Consider the standard example of standing in your kitchen, prepped to prepare a meal. First, you inadvertently only a few objects of focus -one of them being the vegetables on the kitchen counter. Now given this ltered information from the environment, your likely response is to chop the vegetables. On the other hand, in the context of a grocery store, the object of information from the environment -the "vegetables" -your response is very di erent, ie, it is to put them in your grocery basket and head to the checkout counter. It is evident that depending on the context (and in the example described, the context is the task), we pre-select all the behaviors (or rules) that would be pertinent here, and then user ner processing to select one among them. Furthermore, at least most of the time, it is relevant information that is attended to, whereas irrelevant information can successfully be ignored. Two questions thus arise : rst, what decides which information is relevant and thus gains access to further processing, or which information is to be discarded as irrelevant.

And second, what structures might support the sub-selection of valid behaviors from the repertoire of all behaviors in long term memory ?

One possible answer to these questions is the conception of "task sets" or task representations. Such representations can form the basis of modulating which stimulus information is processed and which is not, by narrowing down the focus of attention towards the relevant set of behavorial responses. Thus, a task set can be thought of as a set of all the stimulus -action -outcome associations that are pertinent for the task at hand.

In the experimental literature, to perform a speci c task, humans are thought to enter a task dependent cognitive state, mode, or set that is maintained for the duration of the task [ ]. A task-set (TS) is a con guration of cognitive processes that is actively maintained for subsequent task performance. It is the representation of a mental state corresponding to any currently used rule-mapping in order to perform a given task [ ]. When subjects are given instructions prior to a task, after practice for several trials, the task information is maintained as a con guration of perceptual, attentional, mnemonic, and motor processes necessary to perform the task. When subjects are then asked to perform another task, they have to establish a new task set in a form distinct from the previous one. The stimulus set and response set may be same between di erent tasks, but the rules of association between the S-R might di er. A TS has to be speci c in the sense that it represents a rule of a speci c task to be performed. It is nonspeci c in the sense that it can be applied to any stimulus as long as it belongs to a task-relevant stimulus set. Action sets -the di erent ways to do the same thing (eg cutting the vegetable with this or that tool, with the right or left hand etc)

The prefrontal cortex is associated with a variety of executive functions, including this ability to exibly change cognitive con gurations (ie task sets) to newly relevant task demands. Theoretically, there may be two di erent aspects to the process of establishing a task set. First, the relevant rules must be activated (eg : [ ], [ ]). Secondly, interference from competing task sets has to be minimized, possibly through active inhibition [ ]. The dissociation of these two functions Conceptual overview however is fairly complicated [ ], and we talk about it in more detail in later sections.

The updating of a task set may be a self-regulatory mechanism emerging from interactions and competitions among rule representations. The key feature of a task set is its prospective and predictive nature. In this sense, the task set is a crucial concept in elucidating the causal mechanisms of the brain in creating complex behaviors and abstract thoughts. Task-sets not only help in avoiding mistakes in behavior, the bene t of task set is facilitation of subsequent task performance. Many task-sets which are initially acquired through instruction or trial and error, are stored in our memories. The more we practice a task, or the more recently we have practiced it, the easier it becomes to reenable that task set. To investigate the neural correlates of a task set, experimenters typically record brain activity while subjects are required to perform a speci c task; the idea being that such task set activity likely determines the activity in areas involved in task execution. Single unit studies have shown that representing, updating, and implementing task sets are subserved by interactions among di erent sets of neurons in the same region of the prefrontal cortex. By contrast, imaging studies have shown that each of these processes is subserved by distinct regions in the prefrontal cortex and other areas. We discuss these ndings in greater detail in Chapter , but these results suggest parallel processing at di erent levels of brain organization ie at the neural level, the implementation of task rules, task items and responses are distributed.

In the following section, we elucidate on three core mechanisms that enable cognitive control : attention, working memory, and the monitoring of errors.

A

Real world choices are typically guided by multiple shifts in attention between the choice and alternatives. Without the ability to narrow down focus to a subset of environmental features (as input) and of behavioral responses (as actions), conscious awareness would be completely swamped by irrelevant information, or one would never be able to resist inappropriate urges. Given that cognitive control is primarily the ability to inhibit behavioral sets, attentional control is the ability to lter information that enters the decision circuit. Attention might be the brain's solution to the problem of living in an informationally dense world with a limited capacity for processing information [ ]. Only some environmental information, thoughts, and sensory inputs can be processed at any point in time and attention is the mechanism by which processing priority is accomplished. Not all sensory input is behaviorally relevant and processing irrelevant information can be energetically very costly. Attentional selection is the process of selecting information for prioritized processing in order to demonstrate adaptive behavior that is improved upon as we learn information about our environment that relates to our goals, and thus learning and attention are tightly linked.

Selective attention can be directed by two types of mechanisms : voluntary (topdown) and involuntary (bottom up), which is the distinction between goal directed (top down) and salience directed (bottom up) mechanisms. In the absence of explicit instructions, which is normal experience in everyday environments, the control of selective attention needs to rely on internal mechanisms that dynamically track the relevance of sensory information in the environment.

According to the representation account, attention is the modulatory in uence that representations of one type have on selecting which (or to what degree) representations of other types (for example, the rules to be triggered or inhibited) are processed, that is, how representations on one type (for example, the representation of active goals) guide the ow of activity among other types [ ]. An attentional set (like a task set or action set) is a de nition of the representations of the advanced information involved in selecting task relevant stimuli and responses. By determining which information enters the decision circuit, attention a ects the temporal dynamics of several decision related computations, including stimulus identi cation, valuation, comparison to previously attended alternatives and action selection.

One view of attention is that it consists of separable, yet interconnected brain networks that in uence computational priority, controlling what information enters conscious awareness. According to another theory, the objective of cognitive control is the prioritization of computations of speci c input information so that uncertainty can be minimized, and attentional functions operate to serve cognitive control in the reduction of uncertainty in temporal, spatial and process/response domains. Alerting increases the predictability in time of the upcoming information that is to be processed. Orienting acts to select the most relevant and important information, in space, to be processed. Orienting and executive control of attention are di erentiated in that orienting acts at the input stage to lter (or attenuate) task irrelevant information whereas executive control acts to bias the task relevant process at the processing and response stages when there is competition between processes.

W

Working memory, on the other hand, is the sustained attention or active representation of a limited amount of currently relevant information so that it is available

Conceptual overview

for use; the key word here being sustained. This sustained activity can be used to maintain attention for a long time, solving the problem of holding a 'goal, task or context' in mind till the time the lower order processing required for action is nished. The term 'working memory' has traditionally been associated with the maintenance of speci c stimulus features, such as phonological, visual or spatial information. However, the concept of working memory and the mechanisms that enable active maintenance of stimulus features can be extended to other, more abstract types of information. For example, sustained activation of an individual's current environmental context and task goals is also necessary for exible behavior and higher cognitive processing.

Originally coined by Newell and Simon (

) in the context of computer science, the term working memory was introduced into cognitive pyschology by G. A. Miller, Galanter, and Pribram ( ) who used it for the idea of holding goals and subgoals in mind in the service of planning and executing complex behaviors (Cowarn, ) [ ]. A central idea behind most neurobiologically based computational models of WM is that neural activity can be sustained through mutual excitation, where populations of interconnected neurons send each other excitatory activity in a self sustained manner (recurrent activity). Computationally, this corresponds to a stable attractor in a dynamical system : a state that remains constant over time once the system enters the vicinity of that state. This mechanism of working memory is speci cally described as robust active maintenance. Functionally, the ability to robustly maintain activity over time must also be complemented by an ability to rapidly update to encode new information into working memory. These two demands are mutually contradictory, and the concept of gating has been introduced as a way to dynamically switch between robust maintenance, versus rapid updating [ ]. There are other proposed mechanisms, like the possibility for certain neurons to keep a long plateau activity thanks to internal mechanisms. Sustained activity can be seen not only in the PFC but also in higher perceptual cortical regions and in other neuronal structures like the cerebellum.

Working memory dynamics, and models thereof, include how sequences of events are temporarily stored in ventrolateral and dorsolateral prefrontal cortex, how these sequences are unitised, or chunked, into cognitive plans and how interactions of prefrontal regions with other brain regions enables predictions and actions to be chosen that are most likely to succeed based on sequences of previously rewarded experiences.

P M

Recent years have seen the emergence of a wave of in uential theories that highlight the predictive nature of cognition. A common denominator of these theories is that they paint a picture of the mind wherein our mental representations of the world become active before we engage with reality (top down processing); this view contrasts with traditional perspectives that assumed that our representation of the current state of the world emerged only after we have acquired evidence from our sense organs (bottom up processing) [ ].

A prominent theory within this framework is the Predictive Processing approach that argues that every encounter we have with reality is akin to scienti c hypothesis testing. For example, a person who is about to open the fridge already has prior representations of what they are about to see (a milk carton); to the extent that this representation successfully predicted the event to come, there is no need for much additional cognitive processing; however, when a discrepancy between the prior representation and bottom up inputs is detected (if there is no milk left, a prediction error), then there is a need to update the mental representation in light of the new evidence.

Missing from the picture until now, however, is the process by which organisms detect environmental change and begin the process of either switching between or learning new behavioral strategies. Such a process should successfully handle changes in both the statistical parameters of the environment (alterations in volatility, outcome probability and outliers) and its contingency structure (changes in the state space, its transition properties, and the introduction of new events).

Con ict between Task Sets : Task switching is the process of selecting between competing task sets. Because a task set is a con guration of perceptual, attentional, mnemonic, and motor processes, the con ict can occur at various stages of cognitive processes depending on the di erences between taks, which could be associated with activation in di erent brain areas. Compared with con icts at stimulus and response processing stages, con icts at a conceptual rule-processing stage occur regardless of the overlap among stimuli and among responses.

W R

"Rule" is a commonly used word in everyday language : ranging from rules of a game to imperceptible social rules like being polite to acquaintances. Given the intuitiveness of its use, in this review, we hope to provide an operational de nition of "Rule" in the context of understanding the neuro-computational properties that lead to behavior (animal or human), starting from the most generic and proceed-

Conceptual overview

ing to the most speci c. We will borrow and use literature from logic, psychology and neuroscience to get a rounded view of what rules are, how they might be implemented and processed in the brain and how we adaptively use them in our lives.

A note on nomenclature : There are laws (or rules) of physics (eg, gravity ie an object released from a height will fall to the ground) and social and moral laws (eg, being polite is requested to be successful in certain circumstances). Such rules exist outside of us and our in uence. We don't invent them and yet our behavior conforms to such rules and they play an important role when we de ne our (internal) rules that guide our behavior ie behavior that we have some agency over.

More precisely, we take into account the rules of physics when we de ne our own sensorimotor behavioral rules (in the motor and premotor cortex) and the natural and social laws when we de ne our emotional or motivational behavioral rules (in the limbic prefrontal cortex). The rst kind of rules are factual or objective, i.e, they can be categorized as right or wrong while the latter rules are subjective, i.e., they can only be categorized as good or bad. On the other hand, more abstract rules such as the rule of addition, and other mathematical axioms, or syntactical rules of language are also beyond the scope of this study. Instead, we focus on simpler studies and tasks (despite the complexity of human behavior) simply because much of what we know about the encoding of rules in the brain comes from animal studies, which in itself require several constraints.

The Merriam Webster dictionary describes Rule[s] as prescribed guide[s] for action. In logic, rules can be transformations, as in , or simply expressions that specify a particular set of relations. For example, in propositional logic, modus ponens is a rule of inference that states "If P, then Q. P is true, therefore Q must also be true." To put it more generally, a rule then has two parts : a condition that must be satis ed to trigger the rule, and an action that follows once the rule has been triggered. The same holds true in the real world, and depending on the speci cations of the triggering condition, some situations will trigger the rule in a clear prototypical fashion while others will partially match the conditions and will result in a slow and uncertain application of the rule.

In modern psychology, rules derived from formal logic are combined according to a reasoning program for using the schemas; a basic universal routine and a set of acquired strategies to account for individual di erences. In the domain of neuroscience, we can think of rules as instances of repetition of spatio-temporal rela-. What are Rules? tionships between discrete object features, events and actions present as statistical regularities in the environment. Through repeated experience of these regularities, animals and humans learn to generalize and link these events and objects together as associations. Consequently, a triggering condition or situation can then extract and apply these rules and responses.

In a broad, general sense, rules are those beliefs, strategies, stimulus-response behaviors, in uenced by memory and learning, reward-expectation based probabilities that guide decision making on the basis of prior learning recalled, via perception of current environmental states as a guide to action. First you store rules for sensori-motor (in the general sense, on the premotor and limbic sides) behavior. Then this memory provides a representation that can be used in anticipation (strategies) or be associated to their level of con dence (beliefs).

To make things more precise, in this section, we try to break down these di erent components of what constitutes the formulation of such "rules".

C

A R

In the simplest case, when an object's properties, usually through a discrimination of one or more features of the stimulus, directly indicate a response or category, such associations are referred to as concrete rules [ ]. They describe simple spatio-temporal links between objects, events and actions. These links are known as stimulus-response or stimulus-outcome associations [ ]. They can discriminate on a single feature or a subset of features shared across a group of sitmuli. They can also discriminate using multiple features of a stimulus. For example, when I encounter a red tra c light while driving, I should press the brake. In this case, there are two relevant stimulus features (tra c signal, red light) and one state based condition (while driving). The formation of concrete rules is experience dependent, meaning that they are typically learnt gradually across multiple instances of experienced positive and/or negative reinforcement.

Abstract rules are rules that do not assign speci c responses to stimuli but instead are used to select a set of concrete rules from all possible set of concrete rules that could be applied to the stimulus. For example, while sorting coins, concrete rules are the individual rules of sorting by size or by value, while the contextual decision of using the 'sorting by size' rule is an abstract rule. Abstract rules are complex and applicable to multiple exemplars. They describe interactive and causal associations between objects, events and responses. They require linking together di erent concrete rules and integrating them with instructions and information for achieving a particular goal, and under a certain context [ ].

Conceptual overview

A R

Humans and nonhuman animals show the ability to learn about and act on the perceptual relations between events, properties and objects in the world, but humans go a step beyond since they are able to grasp the higher order relation between these perceptual relations in a structurally, systematic and inferentially productive fashion. In that context, analogical reasoning is a fundamental and ubiquitous aspect of human thought. It is at the core of creative problem solving, scienti c heuristics, causal reasoning and poetic metaphor. Humans form general categories based on structural rather than perceptual criteria, nd analogies between perceptually disparate relations, draw inferences based on the hierarchical or logical relation between relations, cognize the abstract functional role played by constituents in a relation as distinct from the constituents' perceptual characteristics, or postulate relations involving un-observable causes such as mental states and hypothetical physical forces [ ].

The understanding of abstraction from both philosophical and cognitive theoretical perspectives has evolved over time. In his Essay Concerning Human Understanding, John Locke distinguishes between particular and general ideas. Particular ideas are constrained to speci c contexts in space and time. General ideas are free from such restraints and thus can be applied to many di erent situations. In Locke's view, abstraction is the process in which "ideas taken from particular beings become general representatives of all of the same kind" by dint of the mind's removing particular circumstances from an idea [ ]. A theory advanced by Barsalou [ ] advocates for a theory wherein connections between concrete and abstract concepts are direct and nonmetaphorical. In this theory, concepts take the form of simulators, which are semantic clusters that can generate in nite further examples of a concept. As we encounter examples of objects, we encode their perceptual features and store them in our memories. These features will form into clusters, which eventually become simulators, with a frame of previously encountered common features and a set of in nite possible simulations that the frame can generate. For instance, our perceptual experience with various chairs has helped us form a concept of '"chair". We can now use this concept to simulate in nite further examples of chairs.

In more recent studies, neuroscienti c evidences of abstraction has been found. Several studies on nonhuman primates have shown that prefrontal cortex plays a key role in abstract rule-guided behaviors [ ], [ ], [ ] By de ning abstraction as the ability to move away, in a graded fashion from perceptual stimuli to mental constructs, it is useful to look at a few examples where several di erent types of "abstraction" can be seen . Contexts can generalize over more rules (policy abstraction), more dimensions may need to be integrated to make a decision (relational integration) or contexts may need to be sustained over longer periods of time while lower order decisions are made (temporal abstraction) [ ] :

• Contexts can generalize over more rules, called policy abstraction

If you are asked to raise your left hand when a red stimulus appears and to raise your right hand when a green stimulus appears, the rule is the speci c association between the stimulus and the response. Alternately, the rule can be more abstract. For example, if you are asked to press the left button when two pictures are the same and to press the right button when the two pictures are di erent, the rule is not associated with any particular feature of the sensory stimuli. This "same or di erent" kind of abstract rule is often studied in primate experiments.

• We may be required to integrate multiple dimensions to make a decisionrelational integration Consider two sequences of words of objects : little-medium-big and lightshaded-dark. A rst order relation in this case might be represented as bigger(x,y) and darker (a,b). A second order relation linking these two together could be represented as greater(size(x), size(y)) and greater(shading(a), shading(b)).

• In most everyday tasks, contexts (or goals) may need to be sustained over long periods of time while lower order decisions are made -this kind of abstraction can be called temporal abstraction There are tasks that require an abstract hierarchy of goals, which need to be processed either sequentially (in the case of making co ee, with sub-goals being to get a cup, add sugar etc..), or need processing resources between concurrent tasks such as listening and reading, and remembering and returning to where you left o reading

Coming back to the hierarchical representation evoked above, contexts are more easy to de ne : At the rst level of the hierarchy, corresponding to concrete rules, the conditions correspond to sensory cues. At the other higher levels, they correspond to contexts (as manipulated in abstract rules for cognitive control). The types of abstraction mentioned above, can also be understood in terms of building in the PFC, di erent kinds of abstract rules, ie the what, where and when, mentioned above. Since contexts, in this de nition, correspond to the 'sensory cues' used for abstract rules in the PFC for cognitive control, we propose that contexts could also correspond to the context of tasks, as they are mapped in this PFC, and particularly in the vmPFC and vlPFC when we are looking for the semantic representation of this context.

One theory by [ ] suggests that there is a learning continuum in which a gradual abstraction occurs. On one end, there are initial conservative, fully concrete mappings (context speci c representations), which paves the way for a kind of analogical mapping in which a relational structure is imported to a new domain with no support from object matches to a fully abstract mapping in which base domain contains variables and the target contains objects. Other researchers have argued that human subjects possess a qualitatively distinct system for reinterpreting sameness and di erence in a logical and abstract fashion that generalizes beyond any particular source for stimulus control [ ].

If two objects are not distinguishable in any sense by the observer, there is only a single object in mind, hence there is no abstraction. The requirement of distinguishability means that abstraction involves having at least two dimensions in mind : one dimension on which the stimuli di er, and another dimension on which they will be considered identical. Thus, when performing an act of abstraction, one makes a decision on which dimension is central, and by doing so, one designates other dimensions as secondary or irrelevant in the current context. Because abstraction entails selecting / attending to one dimension and disregarding other dimensions that might be salient, many acts of abstraction likely rely on cognitive operations often referred to as cognitive control and selective attention.

A C

Compositionality is the idea that new representations can be constructed through the combination of primitive elements [ ]. An in nite number of representations can be constructed from a nite set of primitives, just as the mind can think an in nite number of thoughts and learn new concepts from a seemingly in nite space of possibilities. Structural description models represent visual concepts as compositions of parts and relations. Because the parts and relations are themselves a product of previous learning, their facilitation of the constructions of new models is also an example of learning to learn. For example, new spoken words can be created through a combination of phonemes or a new gesture or dance move can be created through a combination of more primitive body movements.

G

Generalization is the capability of transferring learned behavior about a previous stimulus to novel stimuli. Generalizing newly acquired behavior is an important part of learning and allows an organism to respond quickly and adaptively. Generalization might be based on the perceptual features of stimuli. For example, when a tone (stimulus A) is followed by a shock, conditioned fear will generalize to another tone (stimulus B) to the extent that A and B are perceptually similar. If generalization is based on the perceptual features of stimuli, then it can be said that it is feature based. The second hypothesis is rule-based. Humans can spontaneously create rules, not easily reducible to perceptual features, which allow for e cient generalization of what is learned to novel situations [ ]. Take another example : an infant desires milk. An instinct causes it to put various objects in its mouth. A perceptual pattern, which -from an external perspective -we call "mother", repeatedly of whether mother is wearing a tickling sweater or a smooth t-shirt. Once this substitutability is represented as a new entity in the infant's mental system, it has performed an act of abstraction. Such acts of abstraction are often discussed under the term generalization.

C

In the context of rules, complexity is di erent from complexity in information theory. It also di erent from the concrete / abstract dichotomy evoked above. What we generally evaluate as 'complex', refers to the fact that there are several conditions (hierarchy of cognitive control) and/or a complex processing in time.

I E R

Rules are explicit constructs but we can learn them either explicitly as in the case of arbitrary symbols such as road signs that are associated with speci c meanings or implicitly as in the case of unspoken rules for social interaction.

There has long been a theory of dual processing in the brain, cemented by ndings of multiple brain systems that support learning and memory. These memory systems have di erent operating characteristics, acquire di erent kinds of knowledge and depend on di erent brain structures and connections for their operations. Two main strategies are generally reported for the learning and selection of behavior, owing to the two types of functionally disassociable memories : declarative and explicit, or non-declarative and implicit.

Conceptual overview

In many situations, learning does not proceed in an explicit or goal directed manner. A lot of knowledge and skills are in fact acquired in an incidental and unintentional manner. Implicit or non-declarative memory generally refers to nonconcsious, procedural memory, responsible for skill based kinds of learning. Skill learning, habit formation, simple classical conditioning, priming are all thought to rely on this kind of memory. It is the knowledge that is expressed through performance rather than recollection. It is elaborated by a slow learning processes can generate a rigid behavior, robust in stable worlds, easy to generate but di cult to quickly adapt to changes. Implicit learning is generally presented as sub-symbolic, associative and statistics based. It refers to the non conscious e ects that prior information processing may exert on subsequent behavior. It is implemented in associative sensorimotor procedural learning and also in model-free reinforcement learning with biologicical counterparts in the motor and premotor cortex and basal ganglia. On the other hand, explicit memory manipulating models of the world can be used for the prospective and explicit exploration of possible behaviors, yielding a exible and rapidly changing strategy, where behavioral rules can be associated to contexts and selected quickly as the evironment or task demand changes. Explicit or declarative memory is the conscious recollections of facts and events. Explicit learning on the other hand is presented as symbolic, declarative and rule based. It is associated with consciousness or awareness, and to the idea of building mental representations that can be used for exible behavior, involving the prefrontal cortex and hippocampus.

An important point to note here is despite this dichotomy, when one learns to explicitly select a non dominant behavior by cognitive control, repeatedly, it can eventually become a routine, implicit behavior. In other words, the explicit rule can become implicit because it is transferred in the (agranular) regions of concrete rules.

A similar dichotomy or distinction in psychology has long been that between Rules vs Similarity [ ]. Rules should eventually, after su cient training, apply equally well to familiar and novel stimuli. All de nitions of the notion of rule take this feature as a de ning one. In this context, some investigators believe that learning based on co-occurence statistics between a set of elements, associative learning can only give rise to similarity knowledge, not abstract knowledge. In a network with only implicit representations, knowledge may be represented and activated in response to external stimuli, but it is not available for use by any other part of the system [ ]. Only the formation of explicit representations provides a system with . What are Rules? a kind of exibility and generality such that they can become objects of cognitive manipulation transportable to other tasks.

H

The notion of hierarchy has become common in the study of the human cognition in recent time, stating it as a way of how rule represenatations might be implemented in the brain. Another outlook states that cognitive reasoning often involves making hierarchically organized decisions, and also that such decisions then require a causal inference about errors [ ], ie reasoning about one's failures by assessing self con dence. Some researchers also believe that there exists a hierarchical organization in motor movements such as reaching (bringing the hand and arm to target). In such accounts, information is processed in a hierarchical system where at the rst level, motor related information (for eg, which arm to use) is generated and the target is selected. At the second level, this information is collected and integrated to 'plan' the reaching movement. Finally, at the third level, the planned movement is prepared and executed [ ].

There is signi cant evidence in the neuroscience literature to suggest that the prefrontal cortex is organized hierarchically, and can support such hierarchical processing of information. Precisely how this hierarchy emerges and how it processes and represents information is unclear. There are two common accounts of understanding prefrontal organization -a representation based and a process based account [ ]. The rst group, as exempli ed by Goldman-Rakic, suggests that the functional division of the frontal cortex is on the basis of the di erent kinds of representational content (eg object vs spatial representations) they encode, while performing essentially the same kind of processing function (eg working memory). On the other hand, the other group suggests that the contribution of di erent areas is based on di erent processing functions (eg inhibition vs selective attention [ ], or maintenance vs complex processing [ ].)

Process based accounts appear to be most useful in the context of incremental extraction of information from the environment, whereby the products of one mental process are used by another mental process. Higher levels exert control over lower levels, for example by controlling the ow of information or by setting the agenda for lower levels. Long term planning problems can be solved using hierarchy because higher levels have a bird's eye view of the problem and can identify appropriate sub-goals while lower levels have a ner temporal resolution and are able to reach each subgoal.

Conceptual overview

Representation based accounts on the other hand appear to be more useful in the context of attentional and cognitive control processes that require the maintenance of particular information in working memory for the purpose of biasing representations in other regions. Higher levels form abstractions over lower levels, such that lower levels contain concrete, sensory and ne-grained information whereas higher levels contain general, conceptual and integrated information. Cognitive exibility and task switching can be achieved using a hierarchy over strategies : A high level strategy is in charge of selecting one of several lower level strategies, and low-level strategies guide actual behavior. Whereas the high level strategy is trained to identify the best lower-level strategy for each context, the low-level strategies are trained to optimize behavior within each context.

While the process based and representation based accounts are theories of what is organized hierarchically, there are other theories that explain how these organizations are implemented. One such theory [ ] proposes this dichotomy on a What vs How axis or ventral vs dorsal pathways. The ventral hierarchy plays a role in guiding the selection and retrieval of semantic/linguistic knowledge while the dorsal hierarchy carries out the processing of the sensory information to guide action outputs. This idea can also be thought of in terms of gradients of generalization on the one hand and complexity on the other. Generalization refers to having broader categories (eg: color vs red vs brick red) or otherwise being more distantly removed from concrete physical objects (eg: beauty vs sunset). By contrast, complexity refers to the number of di erent elements that must be taken into account (ie multiple conditionals) to generate a task-appropriate response. For example, the rule "hit left button if the previous stimulus was an A and the current one is an "X" requires two items to be integrated (A and X) to determine the response.

B R

Initially much of animal learning was thought to be associative or hebbian learning ie hebbian type rules that appear to ensure that jointly activated synapses reinforce each other, and this mechanism may su ce for an ordered separation of pathways and projections, such as those controlling exion and extension of a muscle. But what remains to be secured is the choice of the correct, adaptive behavioral pattern in response to speci c stimulus con guration. This choice could be achieved by a general reward system that would condition the animal to select the correct behavior, but it could also be determined genetically at the structural level. Whereas the second mechanism may be the norm among invertebrates, it has less commonly been demonstrated in vertebrates.

Foraging and social information seeking are two prototypical types of behaviors displayed by many animals. In natural environments, animals have to face many di cult decision making problems, posed by the details of their habitat and social system. Such problems can include when, where, and for what to forage; with whom to mate and where to nest; whether to ee or to ignore a potential predator. In many primate social groups, males do not mate with all females, because to do so would risk reprisals from dominant males. Yet the presence of a sexually receptive female is among the most potent natural stimuli in the animal's sensory world. That the mating between sensation and behavior is exible enough to take into account such complex and uid information as the present state of a group's dominance hierarchy argues against a simple view of stimulus response mappings and for a richer, more nuanced view phrased in terms of decisions.

Most widely replicated test of relational concept learning -same-di erent task. Another one -relational match-to-sample (RMTS) task, subject must select the choice display in which the perceptual similarity among elements in the display is the same as the peceputal similarity among elemnts in the sample stimulus. For example -AA (sample), subject should select BB rather than CD. Success on such tasks has been reported in chimpanzees, parrots, dolphins, baboons and pigeons.

The ability to make systematic inferences about unobserved transitive relations has been taken as a litmus test of logical relational reasoning. Male pinyon jays can anticipate their own subordinance relation to a stranger after having witnessed the stranger win a series of confrontations with a familiar but dominant conspeci c.

Regarding the neural correlates found in the brain, especially the Prefrontal cortex (PFC) and other relatively older brain structures like basal ganglia (BG) and amygdala, the neural processes behind several of the above discussed paradigms in these brain regions will be explored in the following chapter.

In this Chapter, we have introduced and precisely de ned some notions and concepts integral to the systematic study of Cognitive Control. While widely studied, Cognitive Control still remains poory understood. In this conceptual overview, we have attempted to tie together the concepts and mechanisms that will be explored through computational modeling in the second part of this manuscript. Particularly, we have begun with the hypothesis that control is a gradient, and that the allocation of control then necessarily depends on the particular cognitive demands evoked by the task at hand. Di erent conditions will elicit the need for the di erent mechanisms discussed. In chapter , we illustrate this point through the computational modeling of increasingly complex tasks, thus highlighting the minimal principles and algorithms needed for the implementation of cognitive control. At the rst step of this gradient, in two di erent studies, we illustrate the need of a context to manipulate both innate behaviors, and implicitly acquired behaviors. At the second step, we add a working memory component that can hold the context until the task is completed. Under multiple contexts, we show how an episodic memory component is needed, thus enabling planned and exible behavior. Finally, at the last and fully elaborated level of Cognitive Control, in chapter , we describe the hierarchical elaboration of contexts, which in tandem with the prediction and monitoring of errors, enable adaptive behavior. More generally, the underlying approach in this thesis has been motivated by the development approach of software engineering -to build systems that progressively increase complexity, such that points of failure can then be easily identi ed, and thus allow for systems that balance complexity and exibility. We believe that computational neuroscience can bene t from this approach, and could be applied to several questions in the eld. P C Anatomy is not tedious; it is fundamental -Richard Passingham I One cannot understand the neural basis of control processes without carefully considering the anatomy of the brain. Understanding the neural mechanisms of control requires delineating speci c functional roles of individual neural structures and consequently their functional relationships. The frontal cortex is called so being situated at the anterior end of the brain (anterior to the central sulcus). Comparative studies of the frontal cortex and behavior to understand cognitive abilities can be considered from two perspectives: across the evolution of cognitive abilities over the eons (phylogeny) i.e. across di erent species; and the development of cognitive abilities during the maturation of an individual member of a species (ontogeny). From this perspective, higher order control over behavior has traditionally been seen as the function of the prefrontal cortex (PFC) [ ], which reaches its greatest elaboration and relative size in the primate, especially human brain (Fuster,

)[ ]. The subcortical and cortical regions work in tandem, creating loops broadly classi ed under the sensori-motor, limbic and associative labels, with the latter particularly engaged for cognitive control. In humans and non-human primates, these are reported to be in the PFC, which is considered to be the granular part of the frontal cortex [ ], i.e., the associative regions and parts of the limbic regions. In rodents, there is no granular part, but some parts of the agranular regions play the role in cognitive control (Box ??).

Thus, the PFC seems anatomically well situated to play a role in the creation and implementation of abstract rules in primates, and being the hub of cortical processing, be able to exert control over much of the cortex. Moreover, the development of the PFC is also linked to cognitive development, with the rate of brain growth di ering for di erent brain regions [ ] [ ]. For example, regions that control primary functions, such as motor and sensory systems develop rst, followed by the temporal and parietal cortices associated with language and spatial attention. Of note is that the last brain regions to mature are the prefrontal and lateral temporal cortices involved in the integration of sensory-motor processes, the modulation of attention and critical aspects of decision making and exible behavior. These cognitive functions are also the latest to develop according to behavioral studies. Much of what we know today about the functional subdivisions in the PFC, comes from lesion and neurophysiological studies in animals, and imaging studies in humans.

The interest in functional properties of the cerebral cortex can be traced back to the early nineteenth century. By that time, it was clear that damage to the cerebrum from war wounds and other causes had a variety of behavioral e ects. The anatomist Franz Joseph Gall proposed that di erences among individuals in their cognitive functions and personality traits were associated with di erent parts of the cerebral cortex. His theory gave rise to phrenology, a new approach to studying brain function. Hence, throughout the rst half of the nineteenth century, detailed maps were created that classi ed various aspects of the cortex into categories including memory, color vision, vanity, morality and many others. Today, the concept that distinct areas of the brain are involved in di erent types of information processing is referred to as localization of function. Then came the initial explosion of connectionist research, which held the view of the brain as governed by distributed representations and tuning of synaptic strengths. These fully distributed models became dominant in connectionist psychological modeling. More commonly now, the idea of the brain as an "entangled system" is being promoted [ ] where "an entangled system is a deep context-dependent one in which the function of parts (such as a brain region, or a population of cells within a region) must be understood in terms of other parts : an interactionally complex system". An integrated view of the brain, and particularly the prefrontal cortex and how it executes complex cognition may well lie somewhere in between these poles. In any case, some insight about the functional subdivision is quite important to build a comprehensive, well rounded framework for cognitive control.

This chapter provides an overview of the general neuroanatomy of the prefrontal cortex (in humans), its similarities or dissimilarities with other species, and the functional subdivisions, in relation to control processes that have come out after decades of clinical studies. In this section, but also in the wider literature, the focus is primarily on the prefrontal cortex and the basal ganglia. However, other areas are thought to play important roles in cognitive control as well. The presupplementary motor area (pre-SMA), inferior frontal junction (IFJ), intraparietal cortex and insula have all been suggested to play an important role in control mechanisms. In the second section, we provide a general overview of the experimental tasks typically used in the neuroscience literature to probe and test cognitive control.

Prefrontal Cortex

The PFC across species Brodmann initially identi ed a large Regio frontalis in several monkey species [ ], including areas occupying the anterior-most lateral, dorsal, medial and orbital surfaces of the hemisphere. These areas together came to be termed as the "PFC", "granular frontal cortex", "frontal association cortex", or a combination of these titles. Brodmann's ndings have since sparked a debate since they imply that the granular PFC, and thus the higher level cognitive activities it supports, is missing in the most often used neuroscience model : rodents. Rodents possess homologs of the agranular medial frontal (MF) and agranular orbital areas of primates but lack homologs of the granular cortex that makes up the largest parts of PFC in most primate species. The frontal proisocortex, which is sandwiched between the core isocortex and the periallocortex is comprised of the agranular MF cortex (aMFC), consisting of area (the anterior cingulate area), area (the prelimbic area) and area (the infralimbic area), as well as parts of the orbital and insular cortex adjacent to the isocortex. Primates, but not rodents, have subdivisions of area that are dysgranular as well as agranular, but it is convenient to refer to this cingulate region collectively as aMFC. Similarly, primate orbitofrontal cortex (OFC) is a component of the PFC that includes posterior agranular and dysgranular components, as well as anterior, granular divisions, whereas rodent OFC is exclusively agranular [ ]. Furthermore, if the granular PFC is a primate specialization, we should expect it to have features that rodents and other mammals do not have. Such properties, both structural and functional do exist. The granular PFC is part of a wider system of association areas in animals that contain traits speci c to primates. Macaque granular PFC neurons, for example, encode correlations between acoustic stimuli and abstract behavior guiding rules [ ], as well as associations between color-shape stimuli and abstract problem solving strategies [ ]. Lesions of speci c parts of the granular PFC in macaques cause profound impairments in rapid learning of arbitrary associations between color-shape stimuli and behavioral goals [ ], whereas lesions of the aMFC cause no impairment [ ] and even facilitate early stages of learning in these associations in rats performing a similar task [ ]. The picture is complex [ ], but to draw a simpli cation, it is reasonable to conclude that rodents have no granular regions, but some agranular regions play the role and act for Cognitive Control. Moreover, the evolutionary addition of granular regions in the primate cortex allows the prospect for contextual control. Further, the di erence in the control abilities of human and non-human primates can be attributed to the evolution of the granular cortex itself, with an augmentation in the size of some PFC areas (dlPFC, FPC, anterior granular OFC), without speci c new PFC areas per se.

. Cortico -Basal Ganglia loops

C B G

There are a number of theoretical (and computational) models that elaborate the role of the basal ganglia in decision making, and action selection (and inhibition).

Evidence for the function of the BG in action selection initially came from human patients with damage or dysfunction to this area. For example, Parkinson's disease and Huntington's disease, both cause serious behavioral de cits, ranging from motor (e.g., di culty in initiating volitional movement) to cognitive (e.g., di culty in switching tasks).

The main components of the primate basal ganglia are the striatum, the global pallidus (GP) and the subthalamic nucleus (STN) in the forebrain, and the subthalamic nigra (SN) in the midbrain. The globus pallidus contains two subdivisionsthe internal and external segments (GPi and GPe respectively), while the substantia nigra contains distinct areas designated compacta (SNc) and reticulata (SNr). In vertebrates, the BG are interconnected with the cerebral cortex and the thalamus. Across species, many aspects of motor function like movements, learning and habituation of actions are believed to be modulated by the processes in BG. Lesions of the striatum, in animal models have shown to produce impairments in learning new operant behaviors (or concrete rules

) [ ] [ ].
A number of computational accounts explain the functional architecture and the possible intrinsic computations in its interaction with cortex and thalamus that drive action selection. Most models of the basal ganglia dissociate two major projection systems : the direct pathway from cortex via striatum to the globus pallidus ineterna (Gpi)/substantia nigra (SN), and the indirect pathway from cortex via striatum, globus pallidus externa (GPe), subthalamic nucleus (STN) and GPi/SN. These two pathways are thought to interact, by achieving approximately opposite behavioral e ects, via opposing e ects on BG output nuclei, to produce successful response selection. More recently, there also has been evidence of a hyperdirect pathway, implicated in action inhibition, in which the striatum is bypassed, such that the STN forms the input to the basal ganglia.

In general terms, the basal ganglia enables actions by the release of inhibition. In the direct pathway, projections from the striatum release gamma-aminobutyric acid (GABA), thereby inhibiting the downstream neurons in the GPi/SNr, thereby disinhibiting the thalamocortical circuitry to promote movement. In contrast, in the indirect pathway, the striatum counteracts this e ect by inhibiting the in-Prefrontal Cortex hibitory GPe. Thus e ectively, the SNr ring is increased by the STN-mediated excitation, ultimately inhibiting downstream circuitry and suppressing movement.

The reward circuit is a central component of the network that drives incentive based learning, appropriate responses to stimuli and accurate decision making.

Several parallel and segregated functional loops exist in the basal ganglia. The frontal cortex, the basal ganglia (BG) and the thalamus are associated into several functional loops known as cortex basal ganglia loop (CBG loop). Primarily, based on their functional subdivisions, these are segregated into loops, as described below. However, Alexander et al [ ] propose a more precise division in loops.

Di erent basal ganglia circuits appear to operate predominantly in relation to di erent types of cognitive and motor actions. These parallel loops, going through distinct regions of the BG, are classi ed intro three major classes : limbic, sensorimotor and associative, shown schematically in Figure . .

• The sensori-motor loops (on the left of the gure), involving the dorsolateral striatum, originate in the sensorimotor and premotor cortices, and process exteroceptive information. They are organized around the motor behavior allowing to reach the goal, according to its spatial position (orientation) or according to the physical characteristics involved (handling).

• The limbic loops (on the right of the gure) originate in the orbitomedial prefrontal cortex (generally comprised of OFC and the Anterior Cingulate Cortex (ACC)), through amygdala, hypothalamus and the subdivisions of ventral striatum and end back in the medial PFC. Beside processing external information, these loops are based on interoceptive information. They are organized around the selection of the goal of the behavior, according to its motivational value [ ], in response to perceived needs or according to its hedonic value [ ].

• Finally, the associative loops (in the middle of the gure) involve the lateral prefrontal cortex and dorsomedial striatum . Also called cognitive loops, these loops are implied in cognitive control, related to the ability to manipulate abstract rules.

The names of these subregions of the striatum are taken from the literature in rodents. The analagous regions sometimes have di erent names in primates, but the idea of the subdivision remains the same

. Motor/Pre-motor regions M P

According to the proposed function distinction by Fulton [ ], the motor cortex could be divided into a primary motor area (M , Broadmann area ) and a premotor area (area ), which is now further distinguished by its dorsal (PMd) and ventral parts (PMv). Furthermore, the caudal PMd has strong connections with M and is well positioned to in uence the generation of movements. In contrast, the rostral PMd has strong connections with the prefrontal cortex and selects responses based on arbitrary and spatial cues [ ]. In general, when a stimulus is represented in the sensory cortex, candidate actions are generated in the premotor cortices, with both of these regions projecting to the striatum. There also exist oldest agranular and limbic regions, responsible for the simple selection of actions, motivations and goals. The next, more rostral area to these is the pre-PMd (area , or the FEF in primates), responsible for attentional processes [ ]. This region can also be seen as the most caudal region of the lateral PFC.

O

The more recent granular regions of the prefrontal cortex are the ones involved in cognitive control. The prefrontal cortex comprises of those parts of the frontal lobe that are anterior to the motor and premotor cortices. Typically, the PFC is subdivided into several functional regions, as illustrated in Figure . . Its lateral surface, as spanned by the inferior, middle, and superior frontal gyri, is usually called lateral prefrontal cortex. In some topographical systems, this is further separated into upper and lower parts, termed the dorsolateral and ventrolateral prefrontal cortex respectively. The ventral surface of the frontal lobes is often called the orbitofrontal cortex (i.e. the part of the brain above the orbit of the eyes), and regions along the ventral midline are speci cally called ventromedial prefrontal cortex. The medial surface of the prefrontal cortex can be roughly divided into anterior and posterior parts, with the posterior, dorsal parts collectively called dorsomedial prefrontal cortex. Finally, the most anterior parts of the prefrontal cortex are often called frontopolar cortex [ ].

Below, we discuss the functional contribution of the medial and lateral parts of the PFC in cognitive control.

M PFC

The medial prefrontal cortex (mPFC) is not a homologous unique region, but refers instead to multiple subregions elaborated in parallel. What is referred to as the mPFC in this section, includes the ventral part -ventromedial PFC (vmPFC), the orbital PFC (OFC, the medial aspect of which belongs to vmPFC), the dorsal part -dorsomedial PFC (dmPFC) and nally the anterior cingulate cortex (ACC, which belongs to the dmPFC). The ACC and parts of the OFC are found in many mammals, but they are especially extensive in primates. The idea that these areas of the OFC and brain areas on the medial surface of the frontal cortex, such as the ACC and vmPFC guide decision making, is bolstered by a series of investigations that show that their activity re ects the value of The medial frontal cortex is more speci cally involved in motivating behaviors by monitoring motivationally salient events such as errors, con ict situations, rewards and penalties. In this section, we review the neuroscienti c evidence of these claims.

OFC

. Organization of the prefrontal cortex Lesion studies in rats and monkeys have consistently implicated the OFC in the guidance of exible behavior. There are two consistent impairments found in such lesion studies : perseveration of previously rewarded choices following the reversal of deterministic stimulus-outcome associations [ ] [ ], and insensitivity to outcome devaluation [ ] [ ]. One proposal that followed these results was that there seemed to be an insensitivity to negative reinforcement (errors) [ ], or an inability to inhibit previously rewarded actions [ ]. More recently, studies [ ] [ ] have shown that monkeys with OFC lesions performing -alternate choice tasks switch as frequently as controls, rendering the account unlikely. An alternative proposal is that the OFC plays a role in representing outcome expectations predicted by cues in the environment [ ]. Generally, animals with OFC lesions fail to reduce responses to devalued cues (ie when previously rewarded cues are 'devalued'). This suggests that the OFC may contain information about the subjective value of speci c outcomes. However, owing to its connectivity with several other specialized brain regions, pinning down a speci c and unique functional role has proved di cult. In general, it is implied in a ective decision making.

Prefrontal Cortex

PFC PFC

Decision neuroscience studies have long suggested a disassociation between the ventral and dorsal medial prefrontal cortex (vmPFC and dmPFC), which provide estimates of goal value and action cost, respectively. Everyday life often requires arbitrating between pursuing an ongoing action plan by possibly adjusting it versus exploring a new action plan instead, or deciding to either choose between currently available options or foraging for other, possibly better ones. Resolving this so called exploitation exploration dilemma involves the mPFC, and shows a di erence in the functional role of the ventral and dorsal parts.

In typical studies attempting to dissociate the functional role of these parts, choice or preference tasks are performed. A standard view emerged through these studies [ ], that during decision making, vmPFC/mOFC acts as a choice option comparator. It represents potential choice options, computes their comparison, and turns them into actual choices in the frame of reference currently relevant for guiding actions [ ]. It re ects the relative evidence for taking one choice over another. Activity in the vmPFC has also been shown [ ] to correlate with the expected monetary value (reward probablility x reward magnitude) associated with a stimulus even in the absence of choice. Extensive evidence also indicates that vmPFC activity scales with the value of the chosen option during decision mak-

ing [ ] [ ] [ ] .
In the study by [ ], FMRI results suggest the role of the vmPFC as a generic valuation system, its activity increasing with reward value and decreasing with e ort cost. In contrast, more dorsal regions are not concerned with attributes of options but with metacognitive estimates, con dence level being computed in mPFC and deliberation time in the dmPFC.

Some interpretations insist that the dual role of the vm/dmPFC is based on the estimation of costs and bene ts [ ], implicating the vmPFC in the estimation of reward values and the dmPFC in the estimation of cost e ort Other accounts are based on the comparison between options during choice, suggesting that both regions estimate decision values, but in an opposite manner. For example, for value di erence, the vmPFC activates while the dmPFC deactivates. Closely linked to this view is the account of vmPFC activity signalling the value of the default or dominant behavior (in other words, the reliability of the current strategy, or exploitation), and the dmPFC activity signalling the value of alterna-. Organization of the prefrontal cortex tive options (the need for exploration / switching) [ ]. Human neuroimaging studies have shown that activation in the vmPFC re ects the subjective value of the ongoing plan according to action outcomes, whereas the dmPFC exhibits activation when this value drops and the plan is abandoned for exploring new ones This idea was elaborated in a study done by Domenech et al [ ]. The authors discovered that the neural activity in the vmPFC infers and tracks the reliability of the ongoing action plan in order to proactively encode the upcoming action outcomes as either learning signals or potential triggers to explore new plans. By contrast, they claim that the dmPFC exhibits neural responses to action outcomes, which results in either improving or abandoning the ongoing plan. Thus, their account suggests that the mPFC solves the exploitation -exploration dilemma by a two stage predictive process : the rst being the proactive ventromedial stage, followed by a reactive dorsomedial stage.

In summary, a key question to ask is : what precise computational role does the vmPFC play during RL and value based choice ? Four non mutually exclusive proposals have been emerging : the vmPFC compares the evidence between competing options, provides input to a decision making integrator elsewhere in the brain, encodes the subjective value of the chosen option and transmits the chosen value predictions to DA neurons for prediction error computation

ACC

The anterior cingulate cortex (ACC) commonly refers to the cytoarchitectonic areas and . In the neuroscience literature of cognitive control, the dorsal ACC (dACC) has an almost ubiquitous presence. It has been implicated in a diversity of functions, from reward processing and performance monitoring, to the execution of control and action selection.

Converging experimental evidence shows that dACC activity ( ) encodes the distribution of opportunities across time as well as space, ( ) assesses the value of disengaging from the present course of action and ( ) regulates switching between periods of exploiting such knowledge and seeking more information [ ]. Both neuroimaging studies in humans and single neuron recording studies in macaques demonstrate that the dACC simultaneously holds multiple representations of value with di erent time constants. When lesions are made in the ACC, macaques can only adjust their behavior in response to the most recent outcome, but the in uence of the longer history of reward and choice is lost. Neuroimaging studies also con rm that the dACC has a preeminent role in information seeking; its activity re ects a person's uncertainty about the choice that they are taking when they are actively exploring options to obtain information rather than when they are simply randomly responding. In sum, the dACC represents the distribution of opportunities in the environment, it computes recent and long term value, and on these bases, it determines whether a person or other animal should engage with a current option or explore the environment, including driving speci c information seeking activity.

L PFC

The lateral part of the prefrontal cortex is engaged in cognitive control, related to the ability of the PFC to manipulate abstract rules when the selection criteria is required to be more elaborated. Especially dorsolateral prefrontal cortex (dlPFC) in primates has been found crucial for the most exible, complex, and expectationoriented behaviors that need to be organized, planned and produced The developmental literature makes similar observations, suggesting that the growth of knowledge sometimes proceeds faster than the ability to control behavior. Conventionally, the lateral prefrontal cortex is divided into the dorsolateral prefrontal cortex (dlPFC) and the ventro lateral prefrontal cortex (vlPFC). These two areas are separated by the inferior frontal sulcus.

The dorsolateral prefrontal cortex (dlPFC), along with the posterior parietal cortex (PPC) has been implied in the encoding of rules. Functional neuroimaging studies of action-rule switching typically report activation in a distributed network encompassing the dlPFC, vlPFC, premotor cortex (PMC), pre-SMA, and the PPC. The ventrolateral prefrontal cortex (vlPFC) is viewed to be more involved with the visualization with more precision, possibly owing to its connections with the sensory areas in the inferotemporal cortex and the auditory temporal gyrus. This region has also been implicated in rule learning [ ], retrieval form long term memory and on line maintenance during task preparation. Although vlPFC has been regarded to play a role in learning stimulus outcome associations as does the orbitofrontal cortex (OFC), an interesting disassociation has been pointed out that OFC is necessary for updating associations that signal desirability whereas the vlPFC is necessary for updating associations that signal availability [ ]. The vlPFC also has outputs to dlPFC, and taken together, the ndings suggest that it is possible that vlPFC de nes a goal, and governs the processes in the dlPFC, transforming the information from stimulus to behavior. Or on the contrary, for more complex Prefrontal Cortex and planned behaviors that have subgoals, then the rules in the dlPFC activate the vlPFC which maintains subgoals.

H

According to the research discussed thus far, the lateral PFC plays a critical role in cognitive control. Left unresolved are questions pertaining to the wider functional organization of the PFC, i.e., are the di erent functional roles ascribed to di erent subregions organized into any sort of topography ? For a long time, neurologists have recognized that the more posterior regions of the frontal lobes are more closely linked to motor activity, while the more anterior regions support processes related to reasoning and mental simulation [ ]. Furthermore, the posterior frontal regions mature relatively early during human development and also share more similarities with nonhuman primates, whereas anterior regions like the frontopolar cortex develop late. Therefore, it makes sense to hypothesize that the PFC is organized in rostral-caudal manner locally (i.e., within the dlPFC or dmPFC), in accordance with the increasing abstraction of cognitive control, with anterior regions supporting complex functions related to higher-order behavioral goals and posterior regions supporting simple functions linked to matching behavior to stimuli. As discussed in Section . . , two main theories have been proposed.

The rst theory by Koechlin, Ody, and Kouneiher [ ] [ ] contends that executive functions are organized according to the degree of temporal abstraction. The lateral PFC is involved in cognitive control by forming a hierarchy of top down selection processes from posterior to anterior regions for selecting appropriate behaviors, according to the temporal structure of events involved in action selection, which de nes the crucial levels of cognitive control. Speci cally, sensory control involved in selecting motor actions in response to currently observed stimuli is subserved by lateral premotor regions. Contextual control is subserved by more caudal lPFC regions which is involved in selecting S-R associations depending on contextual signals that accompany stimulus occurrences. Finally, episodic control is subserved by rostral lPFC regions involved in selecting caudal lPFC representations (task sets) that relate to ongoing internal goals for behavior. Posterior lPCF regions subserve transient control by selecting sensori motor associations for immediate action according to information conveyed by concomitant contextual signals. In contrast, middle lPFC regions subserve sustained control over behavioral episodes by adjusting selection in posterior lPCF regions according to information conveyed by temporally remote events [ ]. Further, these authors propose a similar hierarchical organization in the mPFC.

According to the other theory, the authors Badre, Ho man, Cooney, and D'esposito [ ] [ ] propose policy abstraction as a substitute for traditional organizational structures. They concur with Koechlin et al, that posterior regions facilitate the creation and application of straightforward rules that connect stimuli to behavior (such as "Click the button when you see a red square"). However, they argue that the more anterior regions support higher-order behavioral policies, which are required to choose which of a number of straightforward rules applies in the given situation. Their research suggests that executive functions are organized in a hierarchy, such that more complex processing of the anterior regions shapes the functioning of the posterior regions. They do this by incorporating several levels of increasingly complex rules into a single experiment, ranging from simple stimulusresponse mapping to meta-rules that govern how a task should be approached.

The question then is whether there is an overarching principle of functional organization for the PFC, engaged and responsible for cognitive control. Based on the literature reviewed in this section, we propose a functional organization as illustrated in Figure . . We make a distinction between rule learning on the one hand, and rule representation and implementation on the other. These learned values can then be used to select the best concrete rule from the situation (stimulus driven) or depending on a current goal (goal driven). Typically, the same rule is selected in certain contexts (default or habitual behavior).

Prefrontal Cortex

(b) Rule representation and implementation :

There is another region of the dmPFC (same for vmPFC), which can detect con icts (several close possible solutions), predict outcomes, and compute errors of prediction; this region can inhibit the default behavior and explore to trigger another, more adapted concrete rule. This is the basis of cognitive control. When the contextual condition to select another concrete rule is too complex, an abstract rule must be created to represent it. This is done in lateral PFC, to pay more attention to some cues in some context, with vlPFC to de ne such complex hierarchical cues (goals and subgoals) and dlPFC to create a sequence of actions to get these cues (strategies or abstract rules). This lateral region of PFC in loop with associative region of striatum, can learn internal actions of WM monitoring (input and output gating); Abstract rules can be hierarchical, both in lPFC and mPFC. In lPFC, the lower level of rule corresponds to PrePMd (FEF in primates) for attention monitoring. In mPFC, this di erence of level is between PreSMA (con ict monitoring at the lower level) and dACC (error monitoring at higher levels)

Prefrontal Cortex

E T

Cognitive control is studied using a wide variety of experimental paradigms that measure switch costs, perseveration costs, anti-saccade latencies, percent recall, and stimulus response compatibility e ects.

W M

n-back : Paradigms that test working memory typically require processing the properties of current stimulus and making response related decisions, while simultaneously maintaining and updating information held in working memory.

An example is the N-back task, in which participants are shown a series of stimuli (words, numbers) and are required to respond to the current stimulus if it is identical to a stimulus presented one, two or more (n-back) trials ago. Cognitive control load is manipulated by increasing the length of the sequence that needs to be maintained in working memory.

Matching and non matching rules : Delayed matching to sample (DMS) and delayed non-matching to sample (DNMS) tasks require rule-based comparisons of the sameness or di erence between stimuli that can be generalized to multiple exemplars, including novel items. Generally, monkeys are trained to apply DMS and DNMS rules to visual items with a familiar cue at the start of each trial indicating the relevant rule to be applied. Monkeys need to be able to maintain the rule information across a delay period to be able to apply the rule when the sample and test items are shown.

C

These paradigms introduce interference that results in a con ict between task relevant and task irrelevant stimulus properties and/or stimulus -response associations. Cognitive control is engaged to monitor this con ict. The con ict manipulation is achieved by presenting stimuli that contain features associated with different responses.

Eriksen anker task : This task requires participants to respond to the central stimulus in an array, which is anked by non-target stimuli which correspond to either the same directional response as the target (congruent ankers, or compat-ible condition), to the opposite response (incongruent ankers, or incompatible condition), or to neither (neutral ankers). Subjects need to suppress responses that inappropriate given the context, and hence the task measures the ability of the subjects to engage cognitive control in the inhibition of competing responses.

Stroop task : In this task, participants must attend to one dimension of the presented stimulus (eg the color in which the word is displayed) and ignore a competing but prepotent dimension (the word itself). This task thus tests the capacity to inhibit the irrelevant or interfering stimulus or response representation.

Hierarchical discrimination tasks : In the hierarchical discrimination tasks reported by [ ], [ ], there are typically or more rst order stimulus response associations based on speci c dimensions of the stimuli (for e.g., vowel/consonant discrimination and upper/lower case discrimination) and the con ict between these lower order responses is resolved based on another dimension of the stimulus (eg, color -red means task vowel/consonant, blue means task upper/lower case) R These paradigms involve monitoring con ict between a prepotent tendency to emit a response and the need to withhold that response under certain circumstances.

Anti-saccade task : This task requires the subjects to stop a re exive eye movement to a brief cue and instead look voluntarily in the opposite direction to identify a brie y appearing target stimulus before it is masked.

Go/NoGo task : Generally, this kind of task involves a primary task that requires a response (eg, a -choice decision task such as "press left for X and right for O") as well as an infrequent contingency condition that requires this response to be withheld (eg, do not respond to red stimuli). The properties of the stimulus determine whether to implement the S-R association, select and execute a response or alternatively, to interrupt this process and wait for the next trial.

Prefrontal Cortex

AXCPT and AX task : These are both continuous performance tasks which mix elements of working memory tasks with those of go -nogo and hierarchical tasks and therefore tax multiple aspects of cognitive control. In the AXCPT, a response needs to be given to X on a trial, but only if X was preceded by an A on the immediately preceding trial. The elaborated, conditional AX task stresses the sequential interaction between the contents of the working memory and direct stimulus input. The task is to respond only for X of "AX" in the case that the most recent number has been a " " and the Y of "BY" if the most recent number had been a " ".

T

Wisconsin Card Sorting Test (WCST) : The most established paradigm that involves non-cued selection of abstract rules and exible shifting between such rules is the WCST, routinely used in neuropsychological assessments to test cognitive exibility. In the WCST, the relevant rule for matching (color, shape and number matching) is changed without explicitly cueing when correct performance under the current rule meets a predetermined criterion, after which participants need to discover the new rule by trial and error. The key measure that indicates the ability to switch quickly are the number of perseverative errors the participant displays (the tendency to stick to the current rule). Children typically persevere more than adults.

Intra and extra dimensional shifting and reversal learning task : Select shape or line stimuli based on rules that are inferred based on feedback. After some number of correct trials, the rule changes. Shift trials require applying old rules in new stimuli in the same shape category (intradimensional) or the other line category (extradimensional). Reversals require selecting the previously ignored stimuli within or between categories.

Task-Set Switching : Subjects are typically required to switch between two (or more) tasks. For example, when two tasks, task A and B need to be performed, the sequence of tasks may switch from one to the other at random : AAABBBAA. Task-set switching is the process of selecting between the two competing task sets.

In this chapter, in light of the evidence from neuroscience, we have proposed a functional organization of the PFC that supports cognitive control. In the second section, we have identi ed commonly used experimental paradigms that are used to test it. This functional description will form the basis of our computational modeling e orts (Part II of the manuscript). In the next chapter, we present the current state of the art in computational modeling, and discuss the extent to which the existing models explain or contradict the evidence from literature. Further, we also place an emphasis on models that take into account the architectural or mechanistic constraints that are highlighted by the literature reviewed here.

C M

A number of computational models have played a prominent role in the development and understanding of cognitive control and its underlying mechanisms. These models can be roughly partitioned into two categories : computational models and cognitive models. Cognitive models generally provide a higher level abstraction using symbolic approaches that may provide some hints but in the meantime, they rarely explain the neural basis and for this reason, we will discard them in the present review. On the computational side, there are also a very large number of models that provide explanation and implementation of di erent aspects of cognitive control. Since the number of such models is also very large, we will concentrate in this review on the most comprehensive and contemporary models of cognitive control, which have focused on some key functions integral to cognitive control : namely working memory, monitoring and top-down control. Computationally, these translate into three core challenges :

• How does the brain determine what information is relevant to be maintained during an ongoing task goal, and when this information should be updated with newer information ?

• How is the current demand for control evaluated and what is the necessary relevant information that underlies this neural computation ?

• How can higher level goals constrain and implement a lower level goal (or rule)

In the following sections, we review existing models that model each of these functions, and the challenges associated with each of them. A summary of these models is listed in table . .

W

As de ned in the section . . , working memory refers to that process which allows an agent to store, update and retrieve information that may be relevant for an ongoing task, and thus supports abstract, goal directed behavior. Beyond the question of the neural mechanisms responsible for the storage/update and retrieval of Computational Models this memory, there are also questions related to the content and timing. More precisely,

• Which information needs to be stored ?

• When does the information need to be stored ?

• When does this information need to be updated ?

• When does this information need to be retrieved A number of models have attempted to simulate this process, in a biologically plausible manner. These models attempt to put forward a computational theory that answers the question,

Early neurocomputational models [ ] [ ] [ ] [

] used attractor models to investigate the mechanisms through which working memory could be actively maintained against irrelevant distractors. These models characterize how stable, but exible representations can occur in biologically plausible neural networks (for eg, some forms of recurrent neural networks), which lead to stable attractor states, resembling the brain's neural activity during working memory maintenance. However, most of these models lacked a mechanism for updating the working memory when newer, task relevant information was presented. In fact, this contradiction between the two main working memory functions -that of active maintenance vs exible updating -is di cult to reconcile, since maintenance increases the resistance to distractors, while updating makes the system vulnerable to them. Thus, this problem poses a fundamental computational problem in learning when to maintain context representations and when to rapidly update information into working memory, which is also a core problem of cognitive control.

The most well established model for working memory that was able to present a solution to the problem, from the connectionist or neural network tradition, is the prefrontal cortex and basal ganglia working memory (PBWM) model developed by O'Reilly and Frank [ ]. In this model, the prefrontal cortex (PFC) and basal ganglia (BG) interact to solve the maintenance versus updating problem by implementing a exible working memory system with an adaptive gating mechanism. The working memory is insulated from irrelevant sensory input when the gating mechanism is closed, and is receptive to updating information when the gating mechanism is open. At the biological level, the model proposes that the PFC facilitates the maintenance of information while the BG performs the dynamic gating via disinhibition. Dopaminergic "Go" neurons in dorsal striatum re to disinhibit PFC to enable updating of working memory representations, while "NoGo" neurons support maintenance of PFC working memory representations.

Another model Todd, Niv, and Cohen [ ] provides an abstraction of this model by framing the concept of working memory as an internal (hidden) state representation, using a partially observable markov decision problem (POMDP) framework. The model learns when to update the internal working memory element by online temporal di erence methods, by using a tabular version of the RL algorithm.

Yet another, recently proposed model, Working Memory Through Attentional Tagging (WorkMATe) [ ] combines the gating mechanism insight in LSTM and PBWM models, takes a simple, biologically plausible learning algorithm of the AuGMEnt model [ ], and uses abstract stimulus representations. This model takes longer to train on tasks compared to other models, but is able to generalize across previously unobserved stimuli as compared to the simpli ed PBWM by Todd et al.

Although these models capture certain aspects of reinforcement learning and working memory in a biologically realistic way, they typically need very long timescales to train, and also do not capture the resource limited capacity of working memory in both humans and animals.

M

Several theories regarding the interpretation of the ACC's computational role in cognitive control have implicated the detection of error signals, con ict monitoring, error likelihood detection and calculating uncertainty as likely candidates.

A prominent example of the framework of control is the con ict model proposed by Botvinick, Braver, Barch, Carter, and Cohen [ ] . The conflict monitoring model identi ed the role of ACC as a con ict monitor that increases in activation as a function of con ict between available responses. According to this account, incongruent stimuli, ie stimuli whose response is incompatible on two (or more) dimensions (eg, word reading and ink color in the Stroop task (section . . )) will activate competing responses. Such competition leads to a con ict in response selection, where con ict is de ned as the multiple activity of the competing channels. The model proposes that the ACC resolves this con ict by tracking the evidence for a need to increase cognitive control, and sends this information to the dlPFC. The dlPFC then exerts control over the processing in posterior brain areas, and favors one solution with an attentional process. However, lesion studies have shown that ACC lesions do not always impair the cognitive control adjustments that are likely to follow after con ict detection, according to this theory.

Computational Models

A related model was later proposed by Brown and Braver [ ], called the error likelihood model. They also posited that the ACC activity regulates the activity of other structures involved in implementing cognitive control. This account proposed that ACC associates errors to stimulus-context in which they occur, thus e ectively trying to explain both the con ict and error activity. While subsequent experiments have veri ed the critical aspects of this model, it was unable to simulate the e ects of unexpected errors, ie errors that are committed in contexts with low error likelihood.

A rather di erent model has been proposed by Holroyd and Coles [ ], [ ], ascribing a role in action selection to the ACC, shifting it from the evaluation domain role. In this view, the ACC acts as a "motor control lter" with action policies learned via a dopaminergic teaching signal (reward prediction error). There is evidence for ACC encoding action values in uncertain environments; however this model only explains a mechanism for action selection, but does not make speci c predictions about how these reward and error signals can regulate behavior.

The theory put forward by Behrens, Woolrich, Walton, and Rushworth [ ] proposed that the ACC is sensitive to the volatility of environmental outcomes. According to this theory, the ACC is responsible for detecting how rapidly reward contingencies in the environment are changing over time. The model puts forth a mechanism for exibly adapting the rate at which the current knowledge of the world is updated with new information : the volatility measure computed by the ACC is used to dynamically adjust this learning rate. What the model also insists on, is that the volatility signal is separable from the prediction error signals (mentioned in the previous model) and that these two can coexist within the ACC. On the other hand, this model does not explain the contribution of the ACC to action selection.

Recent studies have started to investigate the role of the ACC in processes that require e ort, showing that the ACC activity increases when subjects prepare for di cult or e ortful tasks, even without the presence of con ict, error or choice. For example, ACC lesions have been shown to be associated with motivational impairments and apathy in humans, and in non human animals, they impair decisions that evaluate trade-o s between e ort and reward value. These ndings, led to the proposal that the ACC might encode for choice difficulty (con ict between choice options). The adaptive e ort allocation model by Verguts, Vassena, and Silvetti [ ] proposes a "boosting" mechanism implemented by the ACC, which is able to bias behavior towards e ortful options when they are predicted to return a large enough reward. In line with this conception of the ACC role, the expected A more recent model [ ] called PRO (prediction response outcome) provides a reconciliation to several of the competing accounts of mPFC function mentioned above. The PRO model assumes the medial PFC learns a forward model predicting multiple likely outcomes of chosen actions, and tracks the discrepancies between actual and predicted or expected outcomes, and uses this error signal to re ne future predictions. Interestingly, the prediction error signal in this model also signals a "negative surprise", when an expected outcome does not occur. Using these disassociated signals, the model posits that the prediction signal increases reliably immediately prior to when the most likely outcome will occur, while the negative surprise signal activates reliably after the action that produces the unexpected outcome has occurred. The model is able to explain most empirical ndings (for both neuroimaging and single cell data) supported by previous account of the mPFC (con ict, error, reward prediction errors), making it a useful generalisable computational framework for ACC's role in cognitive control.

M

The notion of applying hierarchical structure to parse complex systems into subordinate and interrelated systems has long been established, with subsystems being further subdivided into 'elementary' units [ ].

The information cascade model proposed by Koechlin, Ody, and Kouneiher [ ] argues that control signals used to guide behavioral actions, based on internal plans and goals, can be subdivided into sensorimotor, contextual and episodic control. According to this model, the division of control is based on the temporal aspect of when control is implemented, with actions selected based on temporally proximal stimulus lower on the hierarchy and actions selected based on past information higher on the hierarchy. At the neural level, this model gives an account of hierarchical organization along the anterior-posterior axis of the lateral PFC for storing hierarchical rules, with more hierarchically superior control signals being represented in the more anterior cortical regions. Kouneiher, Charron, and Koechlin [ ] have shown that there is a similar anterior-posterior division in the medial PFC for control.

Ribas-Fernandes, Solway, Diuk, McGuire, Barto, Niv, and Botvinick [ ]'s hierarchical model extends standard RL approaches to address the higher level organization of behavior in lateral PFC. According to this framework, RL principles operate simultaneously at the levels of temporally extended behavioral sequences (options) and the lower level actions that make up those sequences. A central claim of this theory is that achieving the subgoal speci cally by a particular option should act as a reinforcing event in the absence of explicit reward. E ectively, it is argued that the PFC is able to hijack the brain's basic reward mechanisms to reinforce behaviors consistent with the organisms' high level goals. This hierarchical framework provides a formal model of the emergence of high level task structure in lateral PFC through reinforcement learning, but does not directly identify the speci c contributions of medial PFC in the RL process.

This idea of hierarchical organization of action sequences has also been implemented by the HRL-ACC model by Holroyd and McClure [ ], to account for how the ACC selects and motivates the execution of extended behaviors, rather than encoding moment to moment changes in behavior following con icts and errors. The idea was implemented using a three level hierarchical reinforcement learning model, to explain rodent behavior. In particular, the 'high level' module (rostral ACC) selects the 'meta task' for execution, which applies a control signal to the middle (caudal ACC) module based on e ort cost. This module then selects the task for execution and further applies a control signal to the lower module which implements the action selection mechanism. While on the one hand, this

Computational Models

Figure . : The information cascade model by Koechlin et al

The model posits that cognitive control operates according to three nested levels of control processes (branching, episodic, contextual), which are implemented as a cascade from anterior to posterior prefrontal regions.

Figure taken from [ ]

model accounts for the e ects of ACC damage observed on rodent behavior, and is compatible with the electrophysiological evidence of reward prediction errors, and neuroimaging data from e ort and control studies, nevertheless, it does not account for the surprise and error signals observed in the ACC.

On the other hand, in the model proposed by Frank and Badre [ ], the authors take the view that cortico-striatal-thalamic (loops between the PFC and BG) circuits are central in supporting the network interactions required for hierarchical control. They propose that these circuits are organized hierarchically, and in each circuit, the basal ganglia gate frontal actions, with some units gating input into the PFC and others gating outputs to facilitate response selection. This functionality of a nested output gating, with higher order information represented and maintained in rostral regions, which conditionalize attentional selection in more caudal regions, thus allows this hierarchical version of the PBWM model to learn conditional if-then problems.

More recently, Eckstein and Collins [ ] have proposed a hierarchical RL model that consists of learning distinct selective models composed of stimulus action associations, as well as learning 'selective models of selective models' composed of associations between contextual cues and selective models, to arbitrate between the subordinate selective models and drive adaptive behavior across changing contexts. Collins and Frank [ ] present a model that combines selective, predictive, and contextual models within task sets. This model implies that the selective model learns through model free RL and selects actions within the actor task set, while predictive models learn to predict action outcomes from selected responses to stimuli. Contextual models learn environmental cues to forecast proactively the reliability of predictive models. The reliability of task sets is computed using rst order probabilistic inferences about the reliability of predictive models, both proactively and reactively based on actual outcomes. As long as the actor task set is judged reliable, it serves as a guide for behavior. In parallel, several other task sets are monitored and if one of them is regarded as more reliable, that task set is chosen to serve as the actor. If none of the monitored task sets are deemed reliable, a new task set is created by combining the task sets stored in long term memory.

This notion of reliability is not new. Doya, Samejima, Katagiri, and Kawato [ ] proposed a multiple model based RL model that conceptualizes how several internal models within the actor task set might be weighted to drive behavior concurrently. Their model is made up of many pairs of selective and predictive models, called controllers. Predictive models are utilised to determine the reliability (called responsibility signal by the authors) of each controller, which is then used to (i) identify the weights or relative contribution of each selective model to action selection and (ii) adjust the updating of internal models across all controllers in re-

Computational Models

sponse to action outcomes. The model assumes that all controllers are identical in nature, although it might be modi ed to include controllers that operate at di erent time scales.

The Hierarchical Error Representation (HER) model by Alexander and Brown [ ] (a hierarchical extension of the PRO model), explains cognitive control in terms of the interaction between the dlPFC (dorsolateral prefrontal cortex) and the mPFC (medial part of the PFC). The dlPFC learns to maintain representations of stimuli that reliably co-occur with outcome prediction error and these error representations are used by the mPFC to re ne predictions about the likely outcome of actions. The error is broadcasted through the PFC in a bottom up manner, and modulated predictions from top-down facilitate selection of an appropriate response. Owing to its recursive architecture, this model can elaborate hierarchical rules on the basis of learning by weight updating, both to select pertinent stimuli and to map a representation inspired by the principles of predictive coding. In addition to its hierarchical structure, the model proposes to decompose the functioning of the PFC between, on the one hand, the prediction of the outcome and the monitoring of the error of prediction, and on the other hand, the elaboration of contextual (and possibly hierarchical) rules to compensate for errors. This distribution of functions has also been reported between respectively the medial and lateral parts of the PFC, as discussed in the previous chapter.

A computational model proposed by Khamassi, Lallée, Enel, Procyk, and Dominey [ ] simulates cellular activity in both ACC and lPFC, on the basis that the feedback related signals in ACC (con ict or surprise) modulate the exploitation-exploration tradeo in lPFC during decision making. Another model proposed by them focuses on the 'meta control', they provide a theory for how the interactions between the lateral and medial PFC play a role in the tuning of the meta learning parameters of the model, hence providing a solution to the problem of when to abandon an ongoing plan in favor of an alternate one.

Models of cognitive control that focus on ACC need to be integrated into more comprehensive accounts that explain how the lateral and medial parts of the PFC interact with one another and other brain regions. This thesis takes some of these models as the launch pad to investigate questions that existing models fail to account for.

Over the last two decades, many neuroscientists have become increasingly convinced that the brain works along the lines of bayesian logic. The idea being that the brain represents sensory information probabilistically, in the form of probability distributions. The existence of something like Bayesian predictions taking place within the nervous system to explain perception seems certain. For the moment, . Modeling uncertainty the theoritical generalization of this assumption to explain the whole of the brain remains speculative. [ ] M One missing aspect in the aforementioned models are the methods by which organisms notice environmental change and start the process of either switching between, or learning whole new behavioral strategies. Unexpected events and variable contexts also activate behavioral adaptation mechanisms in animals with highly developed nervous systems, including adjustments in foraging, action planning, motivational drive, valuation strategy, and the rate of picking up new information, such as rules and higher order statistical models of the environment. Future rewards are uncertain for a variety of reasons. Some rewards are risky or essentially probabilistic, which makes them uncertain (e.g., coin toss).

Such a process should be capable of adjusting to changes in the environment's statistical parameters (variability, outcome probability, and outliers), as well as its contingency structure (changes in the state space, its transition properties, and the introduction of new events). In typical RL algorithms, agents start with either a model of the world (model based algorithms) or just a collection of results seen in di erent states (model free algorithms), and they adjust the values assigned to the states and actions by a percentage proportional to discrepancy between their estimated and experienced values, the reward prediction error (RPE). Under very general assumptions, such algorithms are theoretically capable of locating at least a local optimum for the decision policy. Naive implementations may still need thousands of observations to converge on stable behavior due to the incremental udate process. However, such RL agents may thus nd themselves perpetually playing catch up in an environment that rapidly alternates between numerous xes, unique reward structures, unable to do more than gradually adapt in reaction to abrupt transitions. The ability of many species, including humans, to successfully adopt a large array of behavioral strategies, each of which may be independently altered and deployed with little switching costs, is obviously misrepresented by this situation. Consequently, despite the fact that many traditional theories of conditioning propose surprise (formalized as the absolute value of RPE) or similar violations of expectation as a way to dynamically adjust reward rates and promote new learning, such models are still predicated on the notion of a single policy that is subject to gradual updates. For instance, the performance of RL learning rules can be enhanced by adding extra meta learning parameters based on estimates of uncertainty, particularly since the values of its actions are changing. If agents adjust Computational Models their learning rates to re ect the environment, they can learn considerably more quickly. Contrast this with the di culty of change detection, where it is necessary to distinguish between predicted variance in results and a real change in the environment's underlying structure [ ]. In these situations, agents may perform better by learning a wide variety of behavioral strategies (or a meta approach with a limited number of fast adjustable parameters) and having the option of switching between them when a signi cant enough shift in the environment is recognized. Such a model operates similarly to more advanced models of conditioning that use bayesian mechanisms to dynamically alter learning rates, and it agrees well with the observed quickness of behavioral adjustment in the face of abrupt changes in the reward structure of the environment. In this case, outcomes would be monitored to assess whether the environment has altered signi cantly enough to call for a change in the existing strategy (and learning rates within that strategy). The change detection system, which would combine bayesian inference and a variety of innate or derived models of the world, would then function as a subprocess within reinforcement learning.

When uncertainty is high, it may be essential to extend the learning time scale so that full sequences of behavioraly important events are taken into account. This may require episodic memory and facilitate processes like transfer and one shot learning. Agents may place a special emphasis on "learning how to learn", maybe in part by synaptic adjustments that in uence how brain circuits react to unexpected occurrences, or by altering their behavioral approach, such as, for example, facilitating various attentional modes. There have been suggested algorithmic approximations that depend on more clearly identifying a change point in latent states. Nassar, Wilson, Heasly, and Gold [ ] condensed a Bayesian change detection model to a delta rule model, which modi es the in uence of new results in accordance with the uncertainty and likelihood of change points. To address learning under volatility, there are additional Kalman lter inspired models that monitor both the estimates state of the system and the variance of these estimates.

Model

Model

D

This chapter exempli es some of the principal computational approaches to cognitive control in current research, including those seeking neuroscience applications of established formalisms from the machine learning literature, those that detail biologically inspired neural network models of interacting control mechanisms, and those deriving rational models of decision making and control from optimality constraints. Computational approaches can be used in both a reductionist and emergent manner : deconstructing the homunculus' mysterious intelligence into hopefully more understandable "dumb" neural subcomponents, while also demonstrating how complex control functions emerge from the dynamic interactions among these multiple sub components of cognitive control. A key aspect of this approach when applied to the domain of cognitive control is that these models view control as arising from the interactions of multiple relatively simple elements (e.g" neurons or neural assemblies that perform local processes within a single brain system or unit). Thus, these models emphasize how cognitive control functions emerge from a network of brain regions activated interactively and in parallel, rather than the more historical approach of localizing cognitive function to a single brain region. Nevertheless, the models reviewed in this Chapter either explain the mechanisms of just one of multiple aspects of cognitive control (as in the case of working memory and monitoring (ACC) models) while ignoring their interactions with other computations involved in control. Or, in other cases (top-down or hierarchical models), they explain and implement more global constructs agnostically (task sets or hierarchical organization), while being speci c only about a few of them (hierarchical loops of gating). In each of these models, we have found (i) one or more aspects of control (as detailed in Chapter , namely working memory, selective attention, hierarchical organization, and performance monitoring) missing or (ii) built on simplifying assumptions and partial explanations of behavioral data, or in some cases (iii) implementations that do not yet have a grounding mechanistic link in literature (backpropogation, bayesian models)

A key theme of the theory of 'method of minimal anatomies' is that a theory of an entire brain (or in the case of this thesis, Cognitive Control) cannot be derived in one step. Rather, this is done incrementally, in stages. A 'minimal model' is one for which, if any of the model's mechanisms is removed, then the surviving model can no longer explain a key set of previously explained data. In Chapters and , taking inspiration from this theory, and from our conceptualization of Cognitive Control as a gradient, we implement and analyze computational models, increasingly appending them with the key ingredients needed for Cognitive Control.

In the rst three chapters, we have walked through the current understanding of Cognitive Control through the lens of three elds : Cognitive Science (Psychology), Neuroscience and Computer Science. This has highlighted that a global framework for Cognitive Control, and hence any computational model that seeks to explain it, and further make predictions about behavior must have the following constituents, as per Marr's three levels of analysis :

• At the computational level Why is the computation that the cognitive system must perform important ? to deal with the challenges of catastrophic forgetting, the stability-plasticity dilemma, rule switching

• At the algorithmic level What is the algorithm for the transformation from inputs to outputs ? through mechanisms like working memory, selective attention, task sets, contexts, and hierarchical representations

• At the implementation level How do neural systems perform and carry out the cognitive functions described ? through biologically plausible mechanisms (prediction errors), rooted in neuroscienti c data about connections between di erent brain regions (lateralmedial and anterior-posterior gradient)

In the following two chapters, we build computational models, that incrementally are able to bring in the constituents mentioned above, and this increasing gradient explains the gradient of cognitive control explained in Chapter , as seen in the complexity of observable behavior across di erent species.

P II C M C C M M L S
In this Chapter, we start providing the building blocks for understanding adaptive behavior and cognitive control in terms of di erent learning strategies that link stimuli, actions and outcomes to guide behavior. The concepts of di erent memory systems (working and episodic), performance monitoring, context, and default behavior that were invoked in Chapter , and their need and involvement for cognitive control, are detailed here, in a step by step manner, through simple computational models. The rst section lays the groundwork for illustrating the di erent learning systems an organism (agent) has at its disposal, and the following two sections provide addendums to these systems, di erent aspects such as default behavior and context, that can either help or interfere with learning.

In the rst section, we look at how adaptability arises from speci c interactions between multiple learning and memory systems. We use an actor critic model as the base, to illustrate the capacity and limits of such a model, through increasingly complex decision making tasks taken from the rodent literature. Basic rules that link stimulus to a response or action (what we call concrete rules) are straightforward enough to learn through valued guided decision making. Particularly, the actor critic architectures for reinforcement learning have long been proposed as models of dopamine like reinforcement learning mechanisms in the rat's basal ganglia (together with learning mechanisms based on other subcortical structures like the midbrain and amygdala). Thus, this framework provided a solid foundation for the purpose of our investigation. Very quickly though, as soon as an element of time is brought into the picture, one faces what is called the temporal and structural credit assignment problem in RL. We show how the basic architecture we started with, can then be appended with a very minimal abstraction of a "working memory" like state to get around this issue. By the nature of its implementation, and its capacity limited resources, this yet again remains insu cient for more temporally extended tasks or ones that require the maintenance of several contexts, at

Cognitive Control over Multiple Memory and Learning Systems

which point, an approximation of a long term or "episodic" memory is needed. Finally, we highlight that even with these systems in place, the model has limitations on its ability to explain complex behavior, and thus requires some mechanism to detect abrupt changes in the environment.

In addition to the role of the multiple learning systems noted above, organisms are not "naive", but have certain pre-learned "innate" behaviors, due to their unique social and ecological landscapes. Taking this into account, the computational processing challenge for an organism is not to optimize learning in a xed evironment, but rather the challenge of lifelong, online learning, that must continuously cope with already learned behavior. In the second section, we highlight the in uence of "default" behaviors in learning simple stimulus-response associations in a seemingly complex experimental setting. As explained in the conceptual overview, at its simplest, cognitive control is the ability to override default or dominant behaviors. However, most computational models in literature begin learning with either a blank slate, or with randomly initialised action values. Using the same actor-critic architecture as in section , we show the in uence of innate behaviors in learning even basic concrete rules, or SR associations, given a su ciently complex task setting. We use the ndings and learning curves from an experimental task to validate the results of our model, thus highlighting an organism's need to "cope" with innate behaviors to demonstrate adaptability. We discuss the interpretations of the experimental result, and the di erent hypotheses about the neural mechanisms of inhibiting default behaviors. While competing accounts are proposed by di erent researchers, a unifying theme that emerges out of this study, is the need for a "context" representation for behaviors that do not cooperate but rather compete with the default behavior of an organism. Thus, in the nal section, we highlight the importance of context in learning and cognitive control. Even in situations where the context is overtly present, one needs to learn to identify the said stimulus as a "context". In cases where the context is covert rather than overt, this requires inferring it as a latent state. Using the same spatial alternation paradigm introduced in the rst section, albeit in a continuous state space, we show that an agent can solve the alternation task without an explicit working memory, using a reservoir computing framework. Analysis of the model's internal activity reveals that the memory is encoded inside the dynamics of the network. However, such dynamic working memory remains inaccessible such as to bias the behavior of the agent into one of the two attractors (left or right). To do so, external cues are fed to the network such that the agent can follow arbitrary sequences, instructed by the cue. The model highlights that procedural learning and its internal representations can be dissociated, with the former being insu -. Deciphering the contributions of episodic and working memories in increasingly complex decision tasks cient to allow for an explicit and ne grained manipulation required for cognitive control.

D

Learning and decision making are fundamental aspects of cognition and are closely linked. They have long been studied in animals through various levels of complexity in behavioral tasks. Reinforcement Learning (RL) provides a theoretical framework for modeling tasks in which agents interact with their environment and learn rules by receiving reward signals upon taking actions, and has an undeniable biological basis [ ]. It is nonetheless constrained by the Markovian property, stating that the decision can be directly made from the present state, not consistent with known characteristics of animal behavior in real world situations or even in cognitive tasks.

This class of partially observable Markov decision problems (POMDPs) has been solved by extending the present state (representing the state of the environment) with internal representations [ ] that might correspond to memories built from previous experiences. A basic version of this principle has been proposed in [ ] and related to biological basis, with the distinction between a working memory (associated with the prefrontal cortex) [ ], where a given cue can be kept present in memory for some time, even if it disappears from the experienced world, and an episodic memory (associated with the hippocampus), where a previous episode (series of steps) can be recalled by similarity from the present state and manipulated as a virtual state.

On the computational and experimental neuroscience sides, a biological neural network framework was proposed [ ] to explain how working memory representations in the PFC may be updated and maintained. This concept of gating models was also used to study rule acquisition in rats [ ], by comparing the ability of two RL algorithms to replicate rat behavior. The ability and limitations of such a model in a common human behavioral task has also been demonstrated [ ]. More recently, a simpli cation of this model has been extended [ ] to include a bias that better explains the performance of rats in a spatial alternation task.

This article was presented at IJCNN, [ ]

Cognitive Control over Multiple Memory and Learning Systems

On the machine learning side, learning and exploiting these forms of memory have been adapted to RL [ ], introducing complex representational and computational mechanisms that have also been compared in more details with human brain circuitry, thus introducing meta-RL [ ] and episodic-RL [ ] as new paradigms for addressing non Markovian problems. But this impressive level of performance is at the price of complex computations, requiring an often prohibitive training time (and correspondingly corpus size) and resulting in obscure computing phenomena, di cult to interpret and analyse in terms of functional contributions of the respective memory mechanisms.

What we propose here is to design a study where a basic RL agent is extended with a minimal version of working memory and of episodic memory, that can be trained quickly and without hyper-parameters, and to de ne tasks where the usefulness of each memory can be analysed in details. Particularly, what we want to understand and share with our experimental neuroscienti c colleagues are the conditions where one or the other kind of memory are needed and where they are not su cient and should be complemented with more complex mechanisms. In other words, this explanatory study is the premise for predictions to be con rmed in forthcoming experimental studies in neuroscience and for precise speci cation to be implemented in more powerful learning algorithms for machine learning.

M T C B RL

A tabular, actor critic temporal di erence learning architecture with greedy exploration was used. The agent maintains a table of state values V , where V (s) is the agent's current estimation of expected, temporally discounted reward that will follow state s. The agent also maintains a table of action values Q, where Q(s, a) is the value of taking action a in state s. The actor part of the architecture follows a simple policy where the agent picks the action with the highest Q-Value, except with a probability 0 ≤ ≤ 1 the agent selects an action at random. In the critic part of the architecture, the TD error δ is computed when the agent takes an action a in state s and transitions to state s after receiving a scalar reward r :

δ = r + γV (s ) -V (s) ( . )
where 0 ≤ γ ≤ 1 is a temporal discounting factor. The old state value and action value are then updated as :

. Deciphering the contributions of episodic and working memories in increasingly complex decision tasks

V (s) ← V (s) + αδ ( . ) Q(s, a) ← Q(s, a) + αδ ( . )
where α is a learning rate parameter.

The agent acts on the state space S = S L where S L are all the possible location states an agent can nd itself in (numbered squares in a T-shape grid world), by taking motor actions A = up, down, right, lef t which can result in a change of the location state.

RL

W M : To include a working memory representation, the state space was augmented with an extra memory element. A factored state space representation was used : S : S L XS W M with each tuple of the form (S L , S W M )(grid world location, working memory contents) having its own set of action values. The total number of possible WM states is one more than the number of location states (one extra for empty memory at the start of the episode) i.e.

|S W M | = |S L | + 1.
The memory element is "hidden" as the agent can only access this state using the update action -which sets the working memory state to the current sensory location state, until it is overwritten when the next memory action is taken.

RL E M :

We include an abstraction of the episodic memory system in the model, which is content addressable and temporally indexed. The model maintains a history of the agent's n most recently visited states, in order. After each action that changes the agent's location state, the previous state is added to the end of the list, and the oldest state in the list is removed (no state is removed for the rst n steps). The model now has a element tuple for state representation. The factored state space S = S L XS EP XS W M . The episodic memory state S EP can take either one state from the episodic memory list or an additional state representing "nothing recalled". To interact with this memory system, the agent can take two actions -"cue retrieval" to nd the most recent instance of the agent's current state in the list (and set S EP to that state) or "advance retrieval" to set S EP to the state following its current state in the list. This kind of abstraction allows the model to retrieve a speci c episode from its past and replay the memory from the retrieved point.

T T

A D In the rst, tactile discrimination task, the agent receives a sensory stimulus or a "cue" at the starting state. In this version of the task, it was a tactile cue about the surface, which could be rough on the right or left (represented by di erent states, as in Figure . .B) and was indicative of the rewarding arm i.e. if the surface was rough on the right, the reward was placed at the end of the right arm while if it was rough on the left, the reward was placed at the end of the left arm. Thus, the agent's choice depended on learning this contextual or sensory rule. Another version of this task could be one where instead of a tactile stimulus, the agent could receive an auditory or odor stimulus in the starting state [ ]. The important point is that this kind of sensory cue allows the rat or agent to form a distinct representation of the state

T B A
The second task is a spatial alternation task, in the environment as shown in Figure . .A. This class of tasks is widely used to study hippocampal and working memory (PFC) functions [ ]. The agent begins in the bottom of the central hallways (square marked with a black dot) and proceeds up to the choice point. On the rst (sample) trial, the agent can either turn left or right and receives a positive reward at the end of the arm (squares marked highlighted in black). The agent then continues along the return arm and back to the starting point (where it is prevented from entering the side hallway by a barrier). Following the rst trial, the agent receives a positive reward if it chooses the opposite direction as on the previous trial, otherwise it receives a negative reward.

T R M In this task, there are three task conditions or contexts as represented in Figure . (left). In each task condition, the agent has the option of choosing only between arms, with the rest of the arms blocked (the visual barriers being the contextual or sensory cue). For each of the contexts (A,B,C or D), the agent has to learn the alternating rule. In the trial phase, each context is presented once -A, B then C. The agent is free to go into any of the open arms, and receives a reward at the end of the arm. Following the trial phase, the contexts are presented at random and the agent only receives a reward if it chooses the arm that it had not picked in the previous trial of the same context.

. Deciphering the contributions of episodic and working memories in increasingly complex decision tasks In the displayed setup, the environment is a classical T-maze where the agent starts from the bottom location in order to reach A or B. The rst choice is free and the reward is obtained after each alternation between A and B. After each trial, the agent restarts from the initial location.

On this example, the trials can be written as ABBAB and only transitions AB and BA are rewarded. This task implies for the agent to remember its previous choice across trials. B The discrimination task requires for an agent to learn which cue (out of two) is associated with a reward. The agent has to choose between the two di erent options (A & B) depending on a cue that is presented at the entrance of the T-Maze. This task implies for the agent to remember the initial cue until it reached the corresponding target during a single trial. C One example for the discrimination task where the agent has learned to memorize the location at the entrance. D One example for the alternation task where the agent has learned to memorize the location after its choice has been made.

Cognitive Control over Multiple Memory and Learning Systems

C 0 A 0 C 1 A 1 D 0 B 0 D 1 B 1 Task A (State: A 1 ) Task D (State: D 0 ) Task C (State: C 1 )
Task B

(State: B 0 )

A 1 A 0 A 1 B 0 B 1 C 1 C 0 C 1 A 0 A 1 A 0 B 0 B 1 D 1 D 0 D 1 switch switch switch switch switch switch Hold A 1 Hold A 1 Hold B 1 Hold C 1 … START A 1 A 0 A 1 B 0 B 1 C 1 C 0 C 1 A 0 A 1 D 1 D 0 D 1 B 0 B 1 A 0 1 0 1 0 1 1 0 1 0 1 1 0 1 0 1 0

Trials

Radial maze task (inter-task working memory) start

Figure . :

The radial maze task corresponds to a contextualized alternation task with four di erent contexts (A, B, C, D) as illustrated on the left part of the gure. At any time, only one context X is open such that the agent has only to choose between X 0 or X 1 . The di culty however is to maintain a memory for a given context when the context is changed. To be able to successfully solve this global task, the agent has thus to maintain simultaneously four di erent memories corresponding to the four di erent states of the context.

R N

We rst show, individually through the discrimination and alternation tasks, that the simple RL agent is unable to learn these tasks, remaining at chance performance for the discrimination task, and below chance for the alternation task. Performance in this task is plotted in Figure . (a) and (b). The RL agent with a working memory reaches an optimal performance by appropriately updating its internal memory at the right time. The agent learned di erent policies for each item that may be stored in the working memory. Accordingly, this can be interpreted as learning behaviors for a given context. In the discrimination task, the agent does this by bu ering the sensory cue it received at the start state (or central arm), and maintaining it in memory until the choice point, thus disambiguating the trial type. In the alternation task, the agent had to bu er a location following the choice, and maintain this in memory until the choice point in the succeeding trial.

. Deciphering the contributions of episodic and working memories in increasingly complex decision tasks

In the simulations, we could often observe that the agent could nd the corresponding strategy and could learn to update and maintain the memory with the previous choice (for alternation) or the cue (for discrimination), whether left or right (accordingly, we call this strategy "remember both") (Figure . Right, Figure . Right). Interestingly enough, we could also observe sometimes the elaboration of another strategy, which is valid even if based on a side e ect. In this "remember one" strategy, it is su cient for the agent to remember only one choice (for alternation) or one cue (for discrimination) (Figure . Left, Figure . Left) and to simply label the other case by clearing the working memory. What is important at the end is to learn to associate the good action with a non ambiguous encoding of the state. The strategy adopted by the agent as a percentage over runs is shown in Figure . . Next, we demonstrate the agent's behavior when the task is made more complex, and the agent has to learn the same behavioral rule of alternating between its choices, but under di erent contexts. We use the radial maze environment and show that a simple augmentation of working memory is not enough to learn this task. While in a task with only two contexts, the RL agent with one working memory element may be able to perform the task at slightly above chance performance, increasing the number of tasks quickly deteriorates the learning. A separate memory mechanism, or the episodic memory is needed to learn the presented task, as shown in Figure . (c). To perform this task correctly, the agent depends on its episodic memory to retrieve the last presentation of the current context, advance one step in the memory to discover which direction it had previously chosen, and then choose the opposite direction. It may be possible to solve this task by increasing the number of memory elements, but this would exponentially scale the state space, making value learning increasingly di cult. Learning to use working memory is an implicit kind of learning, which evolves over time as knowing what and knowing when to update and maintain while making the use of episodic memory is an explicit learning of recall. . Deciphering the contributions of episodic and working memories in increasingly complex decision tasks The proportion of trials in the nal block in which the previous choice (alternation) or the cue (discrimination) was in memory at the choice point was calculated. A threshold of / . was used, for example, (a) the number of trials in which cue was in memory / number of trials with cue was greater than threshold, strategy was "remember cue " (similarly for cue ) and (b) number of trials is which left was in memory / number of trials when previous choice was left was greater than threshold, strategy was "remember left" (similarly for right); if proportions for both were above threshold, the strategy was "remember both"; and if both were below threshold, it was "remember other". We analyzed the ability of the agent to adapt in a non-stationary environment. We tested if the agent is able to acquire the two distinct rules of task A (discrimination) and task B (alternation) by switching the underlying rule from one to the other, starting with task A, in the order ABAB. We show that after learning the rst task,

Cognitive Control over

Cognitive Control over Multiple Memory and Learning Systems

the agent is able to learn the second task (alternation), but not to an optimal level of performance (as shown in Figure . ). This is due to contextual interference from the rst task, in which the agent learned to 'pay attention' (i.e store in memory) to the presented cues. Nonetheless, due to the variation in the type of strategy used by the agent, it is able to reach a sub-optimal, but above chance level of performance.

In the two rules we consider, the state spaces are only partially overlapping, and the di erence in the policies is about when to update, thus the agent's performance doesn't drastically drop after the second switch. However in a situation where the rules are reversed [ ] (for example, for the discrimination task -initially if cue rewards the right choice and cue rewards the left choice, a rule reversal would mean cue rewards the left choice and cue rewards the right choice), using the presented model would show such a sudden drop. This is because (a) the same set of Q-values are learned and updated continuously as the rules change, and (b) the model uses the same agent for learning motor and memory policies (as opposed to some multi-agent RL models). In any case, this model has the limitation of being unable to recall a previously learned rule. Hence, the best it can do is to identify a change in the environment, 'forget' its currently held policy, or adapt its exploitation/exploration, and relearn its action preferences.

. Deciphering the contributions of episodic and working memories in increasingly complex decision tasks

D

The current study explored the ability and constraints of a RL model, augmented with working and episodic memories, to model rule learning under increasing levels of task complexity. Our focus for this work was to identify the limits of a minimal RL model, and increasingly complement it with biologically plausible mechanisms to explain learning behavior. Even though the three tasks considered here are not really ecological, they are indeed experimented with rodents by our neuroscienti c colleagues and could be used to replicate the present ndings and explore possible predictions. Particularly, they allow us to illustrate the necessity of having multiple memory systems (working memory and episodic memory) in this context. For the simple discrimination task, which shares a number of similarities with a regular delay match to sample (DMTS) task, we explained that the Markovian Property forbids a basic RL agent to solve the task and an additional hidden state (working memory) is necessary to

Cognitive Control over Multiple Memory and Learning Systems

reach the optimal behavior. The alternating task shares a lot of features with the discrimination task but the main di erence concerning the working memory is not "what to store" but "when to store". We also showed that it is necessary for the model to store the state after the decision has been taken, that is, when the agent is close to one of the two end states. We also noticed that the model can eciently learn to update and maintain working memory representations which can be intra-trial, inter-trial and inter-task.

Finally, the radial maze setup implies (in our implementation) a di erent kind of memory (episodic memory) such as to be able to "upload" the right context to the working memory. An alternative implementation could have been to have four di erent working memories, one for each context. But even in such case, the core "routing" problem remains the same: depending on the context, the agent needs to store the memory at the right place.

If we now go back to behavior, we think that the radial maze task is actually quite representative of our day to day behavior. An agent already engaged in a given task may "pause" it in favor of another task provided it perceives some cue indicating that a new task can be advanced or completed. This kind of behavior requires in fact the temporal organization of behavior. Furthermore, even though the task is already quite complex, we nonetheless keep it simple by explicitly cuing the agent with the unambiguous identi cation of the context, contrarily to tasks such as the Wisconsin Card Sorting Task (WCST) [ ] that require an e ective cognitive e ort to monitor performance to decide if the context has changed or not. It thus comes as no surprise that our human brains possess dedicated areas in the frontal cortex to monitor, select and instantiate such high level rules.

Much work has been done on dealing with non-stationary environments, in model-free algorithms [ ] in multi-arm bandits, or model-based RL frameworks such as that proposed by [ ] but the neural underpinnings for these are not yet clear. On the other hand, models for rule-learning and rule-switching have been proposed that rely on bayesian inference [ ] [ ], and while they provide a unifying theory for these concepts, their complexity makes them infeasible for guiding and testing fundamental hypothesis in experiments. Coming back to biological inspiration and building upon working memory and episodic memory a biologically inspired cognitive control [ ] [ ] could be an interesting way to de ne a more exible decision making agent, able to quickly adapt in an uncertain and changing world, as we will study in the near future.

. Cognitive Control over Default Behaviors

C C D B

The case of spatial alternation in mice, discussed above, merits special examination because it falls under what Watkins [ ] refers to as "innate knowledge" (and what we have described as default behavior). It is a dependable spontaneous behavioral tendency that has likely been preserved through evolution due to some reproductive advantage it delivers to the organism. "What kinds of innate knowledge do animals have, and how does this innate knowledge contribute to learning?" is the central query posed by Watkins, in the context of learning and the assessment of learning rates. Furthermore, it is equally crucial to position this idea of "innate knowledge" in relation to the behavioral a ordances o ered by a certain context. It is highly relevant also in relation to cognitive control, since our understanding and explanation of cognitive control, rst and foremost, is based on the ability of an organism to overcome default behaviors when they prove undesirable in a given environment. For instance, mice tend to spatially alternate in a T or Y maze, even when the behavior is not positively reinforced, leading to the conclusion that this particular state action policy can manifest itself even in the absence of a clear environmental reinforcer. Recent research has even demonstrated that mice spatially alternate in a T-maze even after they have already developed a preference for a reward that is only present in one of the arms [ ].

Through reinforcement learning, we get formalisms for accounting for both the creation and revision of policies. In both instances, it is believed that this is the result of the agent assessing results (which may be xed or dynamic) from actions taken while in a speci c state, associatively storing these state-action-outcome evaluations, and using them to guide subsequent action while in the same (or similar) state. As a result, the term "state-action" policy enables us to group together under a single abstract idea all cognitive material that is known to control how an organism (or agent) chooses to act in a given situation (where beliefs, guidelines, attitudes, correlations between stimuli and responses etc comprise cognitive content).

According to the animal exploration theory, the principle cognitive drive underpinning exploratory behavior, is global information gain, over foraging [ ].

Through this interpretation, it seems more plausible to explain the spontaneous spatial alternation behavior of mice as the need to explore. In fact then, it is the physical constraints of the maze (T or Y) environment settings that allow for this exploration to be interpreted as what researchers identify as "spontaneous spatial alternation". In terms of reinforcement learning, spatial alternation can be simply viewed as the most e ective or ideal strategy for navigating the T or Y maze. On the other hand, since the environmental conditions of the tactile discrimination task explicitly reward non-exploratory behavior, this learning can be interpreted as requiring a context-dependent adjustment of the inherent exploratory policy rather than as the initial creation of a novel policy. This is speci cally what we investigate in this section, using the experimental framework of Stevens [ ], and the modeling approach introduced in the rst section. We show how the established default behavior of alternating (alternatively interpreted as foraging, curiosity, or information seeking), observed in rodents, interferes with the learning and subsequent expression of a tactile rule. While on one hand, this is a critique of most existing models, on the other hand, it provides an illustration of how a simple model can explain some of the behavioral results observed in more "ecological" settings.

M T C M

The same actor critic model, as described in the previous section was used, with an internal state or 'working memory' representation. The agent maintains a table of state values V , and a table of action values Q, as described.

For including the e ects of the 'default behavior', or the spontaneous alternation, or the curiosity drive, or exploration, we used an innately learned policy H(s, a), which was sensitive only to the history of selected actions, and not to the outcomes of those actions. To account for the preference of the agent to alternate in the maze, we added a bias to the propensities of the action values in H(s, a) :

H(s, a) = bias * H(s, a) ( . )
The bias function reduces the probability of selecting that action (or choice) which was made in the previous trial in the following manner :

bias(a) = 1 if s = s c or a trial-1 = a i 0.7 if s = s c and a trial-1 = a i ( . )
Finally, an action is selected by computing the weighted sum of action values from the learned and innate policies :

D(s, a) = w * H(s, a) + (1 -w) * Q(s, a) ( . )
The weight w that governs the relative in uence of each policy on the nal action chosen, at each trial, remained xed for the whole duration of the simulation. T Figure . : The radial maze task corresponds to pairs of arms, of which only one within each pair has a surface associated with a reward (arms marked in yellow). The maze is modeled as discrete states, with the beginning state always in the central platform ( -), as illustrated. The agent nds itself in a di erent starting state, depending on the relative presentation of the surfaces (state , in pair A, when the rewarding surface, S , is on the left, or state , when the rewarding surface is on the right, and so on for other pairs .

The radial maze task is modelled as in gure . . There are three task contexts, in which one pair of arms (A, B, or C) is open at a time, and the agent has the option of choosing only between those arms, with the rest of the arms blocked.

Cognitive Control over Multiple Memory and Learning Systems

Each pair of arms has one rewarding surface (represented in yellow) S and one unrewarding surface S . These surfaces are " xed" in one session or episode i.e. if on session one, S was the left arm in pair A, on session it might be on the right and so on for each pair). The agent only receives a reward at the terminal states of the arms. On the central platform, the agent can thus nd itself in one of the six di erent states ( / when the context is A, / when the context is B, and / when the context is C). For each context, there is a binary state representation in the central (or starting) platform to represent S being on the right arm or left. Thus, ultimately, the policy the agent must learn is to go right when in state , , or and to go to the left when in state , , or .

R E T S

Acquisition and expression of a binary choice-based tactile discrimination of foraging rule

Mice were trained under a tactile discrimination based reward location association rule (R ) to choose, trial after trial, between the two contiguous radial maze arms of a sequence of arm-pairs, each arm of which was covered with one of two distinct surface types, one predictive of reward location (S ), one predictive of absence of reward (S ). The training was conducted in conditions of zero or almost zero visibility, i.e. without any extra-maze spatial cues, thereby constituting a classical stimulus-response (S-R) task where the stimulus in question was tactile. Across trials, the relative left and right position of S and S was counter balanced. Training consisted of either one or two sessions per day, with each session composed of between and trials. In early sessions, sequences were composed of a combination of repeated consecutive presentations of a same pair plus pseduo-randomized presentations of all available pairs. The aim of the repeated sequences was to explicitly lead the animal to inhibit its innate drive to alternate. As the animals approached criterion level across sessions, the arm pair presentation sequences became progressively pseudo-random. The nal sessions of tactile discrimination

. Cognitive Control over Default Behaviors

training were therefore fully pseudo-random trial sequences. Performance criterion was xed as follows : animals had to attain either an average of at least % correct responses across two of the nal pseudo-random sessions. In experiments with control groups, control animals were rewarded on every trial regardless of which surface, S or S , they chose.

C M

Simulation :

The agent was trained to learn the simple tactile discrimination task as described in the previous section, from a blank slate ie with Q values initialized to . The performance is shown in gure . (b).

Simulation : The agent was pre-trained as on the radial maze, as in the control condition (ie when there was a reward at the end of the arm, regardless of the surface chosen), so as to obtain a policy for the default behavior H(s, a) (with equal preference for either surface), and this policy was then biased (as described in the previous section) such as to reduce the probability of repeating the previous choice, thus mimicking an alternation bias. This default behavior was assumed to be rigid, and when the model was subsequently trained on the tactile task, only the new policy values Q(s, a) were updated with the TD error. The performance is shown in gure . (c). The results show that a model that is biased by a default behavior is able to better capture the behavioral results from the experimental study. Secondly, this model is able to capture the expression of alternation behavior observed in the initial training sessions. Simulation : The biased agent was trained as in Simulation , but the bias was removed mid way through the training. The results ( gure . ) show that the model is in fact learning by "coping with" the bias of the default behavior. When this bias is removed at the point when the model has learned the tactile task with a % probability, the performance of the agent drops, proving that the tactile policy it has learned is

Cognitive Control over Multiple Memory and Learning Systems

Cognitive Control over Multiple Memory and Learning Systems

a ected strongly by the default policy. This result qualitatively captures the experimental results which show that rule learning is disassociated from rule expression, with the latter strongly a ected by innate or default behaviors. This e ect is evidently not observed in a model that starts from scratch, with only one policy to learn, where the learning and expression of the rule occurs in parallel.

D

The study by Stevens et al, used in this section, was speci cally designed to frame a behavioral training paradigm for rodents, which was purposefully antagonistic with respect to their spontaneous drive to explore. In so far as innate or "naive" animal behaviors can be viewed as evolutionarily maintained state action policies, of which "exploration" was the one rodents were expected to express in the particular task setting, the task can be regarded as a straightforward state action policy revision.

It is believed that innate behavioral traits of complex animals that have survived through evolution, like curiosity, re ect the kinds of cognitive responses most likely to provide an adaptive advantage in the variety of contexts in which a species has developed. When compared to innate, default behaviors, novel, learned behaviors can be either cooperative or competitive. These behaviors are acquired as . Cognitive Control over Default Behaviors a result of contingent particularities of the environment the animal actually nds itself in. In light of this, cognitive plasticity or exibility, or more generally cognitive control can be conceived as a trait of complex creatures, allowing them to speci cally inhibit and go beyond innate behaviors, in acquiring and expressing new, learned behaviors such as to best exploit unpredictable environments.

A possible hypothesis which supports the results we show with our model, is the Heksor theory [ ], which proposes the idea of a negotiated equilibrium between networks implementing newly acquired and old behaviors. The problem of "catastrophic interference", which is now prevalent in the research on arti cial neural networks (ANN), illustrates the di culties that a widely plastic system faces in learning a new behavior while simultaneously preserving existing ones (originally described by Grossberg [ ] as the 'stability plasticity' dilemma). A second behavior may render an ANN incapable of producing the rst behavior it had already learned. An adaptive behavior is based on a network of neurons and synapses that are constantly changing to improve performance and reduce errors, with the changes guided by feedback during the behavior and its outcomes.

While we don't make strong claims on the basis of this study, regarding the speci c mechanisms that contribute to how shared neurons and synapses reach a negotiated equilibrium that allows for new and old behaviors to coexist, it is a possible explanation as to how various behavioral schemas, in the absence of explicit context, may coexist in the PFC.

The need to overcome default behaviors is obvious, and the role of inhibition has been highlighted, for context control learning. We have discussed in previous chapters, the critical need of inhibition for enabling controlled behavior : bad habits, unfamiliar situations, and dangerous environments in an animal's life often require that default behaviors be inhibited and more context appropriate actions performed. However, e ective inhibitory control not only requires actually stopping unwanted actions, thoughts, or emotions -it also requires the e cient detection of those contexts that indicate the need for those forms of stopping. However, in the absence of an overt context, an animal must still cope with its innate tendencies. This is what we show in our model.

In the next section, we take this further and show how in the absence of explicit contexts, an agent can nd itself unable to overcome its learned behavior, and provide an illustration for such contexts may then be developed internally.

F

Suppose you want to study how an animal, when presented with two options A and B, can learn to alternatively choose A then B then A, etc. One typical lab setup to study such alternate decision task is the T-maze environment where the animal is confronted to a left or right turn and can be subsequently trained to display an alternate choice behavior. This task can be easily formalized using a block world as it is regularly done in the computational literature. Using such formalization, a simple solution is to negate (logically) a one bit memory each time the model reaches A or B such that, when located at the choice point, the model has only to read the value of this memory in order to decide to go to A or B. However, as simple as it is, this abstract formalization entails the elaboration of an explicit internal representation keeping track of the recent behavior, implemented in a working memory that can be updated when needed.

But then, what could be the alternative? Let us consider a slightly di erent setup where the T-Maze is transformed into a closed -Maze (see gure . -Left). Suppose that you can only observe the white area when the animal is evolving along the arrowed line (both in observable and non-observable areas). From the observer point of view, in the central corridor, the animal is turning left one time out of two and turning right one time out of two. Said di erently, the observer can infer an alternating behavior because of its partial view of the system. The question is: does the animal really implement an explicit alternate behavior or is it merely following a mildly complex dynamic path? This is not a rhetorical question because depending on your hypothesis, you may search for neural correlates that actually do not exist. Furthermore, if the animal is following such mildly complex dynamic path, does this mean that it has no explicit access to (not to say no consciousness of) its own alternating behavior?

This question is tightly linked to the distinction between implicit learning (generally presented as sub-symbolic, associative and statistics-based) and explicit learning (symbolic, declarative and rule-based). Implicit learning refers to the non-conscious e ects that prior information processing may exert on subsequent behavior [ ]. It is implemented in associative sensorimotor procedural learning and also in modelfree reinforcement learning, with biological counterparts in the motor and premotor cortex and in the basal ganglia. Explicit learning is associated with conscious- ness or awareness, and to the idea of building explicit mental representations [ ] that can be used for exible behavior, involving the prefrontal cortex and the hippocampus. This is what is proposed in model-based reinforcement learning and in other symbolic approaches for planning and reasoning. These strategies of learning are not independent but their relations and interdependencies are not clear today. Explicit learning is often observed in the early stages of learning whereas implicit learning appears on the long run, which can be explained as a way to decrease the cognitive load. But there is also a body of evidence, for example in sequence learning [ ] or arti cial grammar learning studies [ ], that suggests that explicit learning is not a mandatory early step and that improvements in task performance are not necessarily accompanied by the ability to express the acquired knowledge in an explicit way [ ].

Coming back to the task mentioned above, it is consequently not clear if we can learn rules without awareness and then to what extent can such implicit learning be projected to performance in an unconscious way? Furthermore, without turning these implicit rules into an explicit mental representation, is it possible to manipulate the rules, which is a fundamental trademark of exible adaptable control of behavior?

Using the reservoir computing framework generally considered as a way to implement implicit learning, we rst propose that a simple alternation or sequence learning task can be solved without an explicit pre-encoded representation of memory. However, to then be able to generate a new sequence or manipulate the rule learnt, we explain that inserting explicit cues in the decision process is needed. In a second series of experiments, we provide a proof of concept still using the reservoir computing framework, for the hypothesis that the recurrent network forms con-

Cognitive Control over Multiple Memory and Learning Systems

textual representations from implicitly acquired rules over time. We then show that these representations can be considered explicit and necessary to be able to manipulate behaviour in a exible manner.

In order to provide preliminary interpretation of what is observed here, it is reminded that recurrent networks, particularly models using the reservoir computing framework, are a suitable candidate to model the prefrontal cortex[ ], also characterized by local and recurrent connections. Given their inherent sensitivity to temporal structure, it also makes these networks adaptable for sequence learning. This approach has been used to model complex sensorimotor couplings [ ] from the egocentric view of an agent (or animal) that is situated in its environment and can autonomously demonstrate reactive behaviour from its sensory space[ ], as we also do in the rst series of experiments, for learning sensorimotor couplings by demonstration, or imitation. In the second series of experiment, we propose that the prefrontal cortex is the place where explicit representations can be elaborated when exible behaviors are required.

M T

The objective is the creation of a reservoir computing network of type Echo State Network (ESN) that controls the movement of a robot [ ], [ ], which has to solve a decision-making task (alternately going right and left at an intersection) in the maze presented in gure . .

M A E S N
An ESN is a recurrent neural network (called reservoir) with randomly connected units, associated with an input layer and an output layer, in which only the output (also called readout) neurons are trained. The neurons have the following dynamics:

x[n] = (1 -α)x[n -1] + αx[n]
( . ) ) where x(n) is a vector of neurons activation, x(n) its update, u(n) and y(n) are respectively the input and the output vectors, all at time n. W , W in , W out are respectively the reservoir, the input and the output weight matrices. The notation [.; .] stands for the concatenation of two vectors. α corresponds to the leak rate. tanh corresponds to the hyperbolic tangent function and f to linear or piece-wise

x[n] = tanh(W x[n -1] + W in [1; u[n]]) ( . ) y[n] = f (W out [1; x(n)]) ( .
. From implicit learning to explicit representations linear function.

The values in W , W in , W out are initially randomly chosen. While W , W in are kept xed, the output weights W out are the only ones plastic (red arrows in Figure . ). In this model, the output weights are learnt with the ridge regression method (also known as Tikhonov regularization):

W out = Y target X T (XX T + βI) -1 ( . )
where Y target is the target signal to approximate, X is the concatenation of , the input and the neurons activation vectors: [1; u(n); x(n)], β corresponds to the regularization coe cient and I the identity matrix.

E U

The class of tasks called spatial alternation has been widely used to study hippocampal and working memory functions [ ]. For the purpose of our investigation, we simulated a continuous version of the same task, wherein the agent needs to alternate its choice at a decision point, and after the decision, it is led back to the central corridor, in essence following an -shaped trace while moving (see gure . -Left). This alternation task is widely believed to require a working memory such as to remember what was the previous choice in order to alternate it. Here we show that the ESN previously described is su cient to learn the task without an explicit representation of the memory.

T

In order to generate data for learning, we implemented a simple Braintenberg vehicle where the robot moves automatically with a constant speed and changes its orientation according to the values of its sensors. At each time step the sensors measure the distance to the walls and the bot turns such as to avoid the walls. At each timestep, the position of the bot is updated as follows:

θ(n) = θ(n -1) + 0.01 i α i s i ( . ) p(n) = p(n -1) + 2 * (cos(θ(n)) + sin(θ(n))) ( . )
where p(n) and p(n+1) are the positions of the robot at time step n and n+1 , θ(n) is the orientation of the robot, calculated as the weighted sum (α i ) of the values of the sensors s i . The norm of the movement is kept constant and xed at . The black arrows are xed while the red arrows are plastic and are trained. The reservoir states are used as the input to a classi er which is trained to make a prediction about the decision (going left or right) of the robot. A left (L) and right (R) cue can be fed to the model depending on the experiment (see Methods).

Cognitive Control over Multiple Memory and Learning Systems

S[0] S[i] S[n] W in ξ S[i] W out W ••• •••

Sensors

. From implicit learning to explicit representations Figure .

: Generation of the -shape pathway with the addition of walls at the intersection points

T

The ESN is trained using supervised learning, containing samples from the desired -shaped trajectory. Since the Braitenberg algorithm only aims to avoid obstacles, the robot is forced into the desired trajectory by adding walls at the intersection points as shown in gure . . After generating the right pathway, the added walls are removed and the true sensor values are gathered as input. Gaussian noise is added to the position values of the robot at every time step in order to make the training more robust. Approximately , time steps were generated (equivalent to complete -loops) and separated into training and testing sets.

H

The ESN was built with the python library Reser-voirPy [ ] with the hyper-parameters presented in table . , column "Without context". The order of magnitude of the hyper-parameters was rst found using the Hyperopt python library [ ], then these were ne tuned manually. The ESN receives as input the values of the sensors and output the next orientation.

M

The performance of the ESN has been calculated with the Normalized Root Mean Squared Error metrics (NRMSE) and the R square (R 2 ) metrics, de ned as follows :

N RM SE = n i=1 (yi-ŷi) 2 n σ ( . ) R 2 = 1 - (y i -ŷi ) 2 (y i -ȳ) 2 ( . )
where y i , ŷi and ȳ are respectively the desired output, the predicted output and the mean of the desired output.

R

In this section the reservoir states are analysed such as to inspect to which extent they form an internal and hidden representation of the memory. To do so, we use Principal Component Analysis (PCA), a dimensionality reduction method enabling the identi cation of patterns and important features of the processed data. PCA is carried out on the reservoir states for each position of the robot during the -shape trajectory. We continued the analysis by doing a classi cation of the reservoir states. We made the assumption that it is possible to know the future direction of the robot observing the internal states of the reservoir. This implies that the reservoir states can be classi ed in two classes: one related to the prediction of going left and the other related to the prediction of going right. Two standard classi ers, the KNN (K-Nearest Neighbors) and the SVM (Support Vector Machine) were used. They take independently as input, the reservoir state at each position of the bot while executing the -shape and predict the decision the robot will take at the next intersection (see gure . ). Since the classi ers are trained using supervised learning, the training data were generated in . From implicit learning to explicit representations the central corridor of the maze (yellow area in gure . -Right), assuming that it is where the reservoir is in the state con guration in which it already knows which direction it will take at the next intersection.

data points were generated and separated into training and testing sets.

E M T C I
In this experiment, we fed the reservoir with two additional inputs that represent the next decision, one being related to a right turn (R) and the other to a left turn (L) (see gure . ). They are binary values, switched to a value of only when the bot is known to take the corresponding direction. We thus built a second ESN with the hyper-parameters presented in TABLE . , column "With context". The network is similar to the previous one, except that the contextual inputs are added with a di erent input scaling than the one used for the sensors inputs. During the data generation, the two additional inputs are set to everywhere in the maze, except in the central corridor.

R M

We rst show that a recurrent neural network like the ESN can learn a rule-based trajectory in the continuous space, without an explicit memory or feedback connections. The score of the ESN is shown in TABLE . and the results for the trajectory predicted by the ESN are presented in gure . and in the top panel in gure . . At each period of about steps, a behavior or decision switch takes place, which is evident from the crests and troughs in the y-axis coordinates. It can be seen that the ESN correctly predicts the repeated alternating choice in the central arm of the maze. In addition to switching between the left and right loops, the robot also moves through the environment without colliding into obstacles.

Performance of the ESN for simulations

R S P

Next, we show that even a simple classi er such as SVM or KNN can observe the internal states of the reservoir and learn to predict the decision (whether to go left or right) of the network. The results of the predictions are presented in the top part of gure . . As expected, there is a periodicity of choice in line with the position of the bot in the maze, showing that the classi cation is relevant. At each time step, both classi ers output the same prediction with a small discrepancy in time. The accuracy score obtained for both classi ers is . In the bottom part of gure . , we can observe that the robot knows quite early which decision it will take at the next loop while we could expect that it would take its decision in the yellow corridor in gure . . Here, we see that if the robot just turned right, the reservoir switches its internal state to go left next time only a few dozen time steps after. We tried the same classi ers but instead of the reservoir states as input, we used the sensors values. The results are shown in the gure . . As expected, the classi cations fail with an accuracy score of . for SVM and . for KNN; this randomness can be seen in both gures. Thus, we showed that by simply observing the internal states of the reservoir, it is possible to predict its next prediction. In essence, this is a proof of concept to show that second-order or observer networks, mimicking the role of the regions of the prefrontal cortex implementing contextual rules, can consolidate information linking sensory information to motor actions, to develop relevant contextual representations.

Since the state space of the dynamic reservoir is high-dimensional, using the Principal Component Analysis (PCA) on the states, we investigated if it is possible to observe sub-space attractors. The result for the PCA analysis is presented in gure . , where PCA was applied for time steps, which corresponds to -loops. The gure shows two symmetric sub-attractors, which are linearly separable, that correspond to the two parts of the -shape trajectory. E Finally, we demonstrate that although the ESN can learn a motor sequence without contextual inputs, it is limited by its internal representation to learn more complex sequences which may require a longer memory. Adding contextual or explicit information about the rule (which we propose are representations developed by the prefrontal cortex over time) can then bias the ESN to follow any arbitrary trajectory as in . . With the additional contextual inputs, the ESN is able to reproduce the standard sequence (the performance is shown in table . ) but can also achieve more complex tasks by sending to it the proper contextual inputs. One ex-

D

Using a simple reservoir model that learns to follow a speci c path, we have shown how the resulting behavior could be interpreted as an alternating behavior by an external observer. However, we've also shown that from the point of view of the model and in the absence of associated cues, this behavior cannot be interpreted as such. Instead, the behavior results from the internal dynamics of the reservoir (and the learning procedure we implemented). Without external cues, the model is unable to escape its own behavior and is trapped inside an attractor. Only the cues can provide the model with the necessary and explicit information that in turn allows to bias its behavior in favor of option A or option B.

From a neuroscience perspective, as developed in more details in [ ], it can be proposed that the reservoir model in the rst experiment implements the premotor cortex learning sensorimotor associations in the anterior cortex. In the rst experiment, this is made by supervised learning in a process of learning by imitation. In a di erent protocol, this is also classically done by reinforcement learning, involving another region of the anterior cortex, the anterior cingulate cortex, manipulating prediction of the outcome. Whereas both regions of the anterior cortex are present in mammals, [ ] reports that another region, the lateral prefrontal cortex, is unique in primates and has been developed to implement the learning of contextual rules and to possibly act in a hierarchical way in the control of the other regions. We have proposed an elementary implementation of the lateral prefrontal cortex in the second experiment, adding explicit contextual inputs as a basis to form contextual rules. It was accordingly very important to observe that it was then possible to explicitly manipulate the rules and form exible behav-

Cognitive Control over Multiple Memory and Learning Systems

ior, whereas in the previous case, rules were implicitly present in the memory but not manipulable.

This simple model shows that the interpretation of the behavior by an observer and the actual behavior might greatly di er even when we can make accurate prediction about the behavior. Such prediction can be incidentally true without actually revealing the true nature of the underlying mechanisms. Based on the reservoir computing framework which can be invoked for both premotor and prefrontal regions, we have implemented models which are structurally similar (as it is the case for that regions) and we have shown that a simple di erence related to their inputs can orient then toward implicit or explicit learning as respectively observed in the premotor and lateral prefrontal regions. It will be important in future work to see how these regions are associated to combine both modes of learning and switch from on to the other depending on the complexity of the task.

D

The approach towards the modeling e orts presented in this chapter was to investigate the basic constituents of cognitive control, using simple computational frameworks. In this respect, the contribution of this work are three fold : In the rst section, we illustrate the necessity of having multiple memory systems (working memory and episodic memory). As discussed in chapter , we highlighted the di erence between the rodent and primate cortex as that of the absence or presence of the granular cortex. The limbic and agranular regions can be considered homologous for interpreting our model. In that aspect, we demonstrate that learning simple associations involves the striatum, while learning contextual associations may involve a basic version of a working memory system, and switching between contextual associations may involve the episodic memory system. We proposed earlier that some parts of the agranular cortex in rodents may play the role of regulating control. Following from this, what we show in our results is that a basic form uncertainty monitoring algorithm may be su cient to describe change detection in the environment, and signal the need to explore (foraging).

In the second section, we show how adding a bias akin to a 'default behavior' to the same framework can better explain experimental behavioral results that use a more ecological setting than the restrictive T or Y maze. The question as to how the default behavior is overcome in absence of an explicit context in rodents -whether by creating a latent state representation of the context, or by simply coping with pre-learned associations remains unresolved for now and may need speci c experiments to probe further. Nevertheless, our results strengthen our initial view of cognitive control as the ability to over come default behaviors, an ability not special to primates.

In the third section, we use the reservoir computing framework which can be thought of as analogue for the motor and premotor regions, to show that a simple di erence related to their inputs can orient then toward implicit or explicit learning. This is also consistent with the functional organization we have presented previously, with the motor and premotor connected to the preSMA in both rodents and primates. This experiment also highlights the question raised in the previous section, about the importance of explicit contexts for e cient manipulation of behavior.

The aim of this Chapter was two fold : . To con rm and illustrate our view of cognitive control as a gradient, through the computational modelling of increasingly complex tasks, . To investigate the role of Cognitive Control is resolving the interference that may arise from these di erent learning and memory systems. To this end, our work highlighted the importance of a "context" in switching between behaviors, and in the next chapter, we posit that the lateral PFC is particularly well suited for the elaboration of contextual behaviors.

In the next chapter, we show how the architectural addition of the lateral PFC, along with its hierarchical organization, allows for an elaborate representation and maintenance of contextual rules over time, thus endowing primates and humans with the ability to learn, and switch between abstract rules.

C R

In Chapter , we explored the basic constituents of cognitive control, particularly concluding on the role of contexts in executing e cient cognitive control. While previously, we focused on studies and experimental results reported in the rodent literature, here we study the experimental paradigms used to probe more complex Cognitive Control in humans. As discussed in Chapter , the distinguishing feature of the primate brain is the granular cortex, in particular the lateral part of the prefrontal cortex. While the models presented in Chapter focused on the role of the medial PFC (and more generally the limbic regions of the frontal cortex) in learning the values of goals and actions, this Chapter elaborates on the role of the lateral part of the prefrontal cortex (or the associative regions) in learning, representing and processing abstract rules. The interactions between the medial and lateral regions are discussed, with the latter being activated by the former for contextual control. Finally, while the previous chapter was about the mediation between di erent learning strategies an animal might use, this chapter aims to describe the PFC's role in the selection, maintenance and manipulation of learned behaviors.

Flat learning methods learn only one set of stimulus action mappings (referred to as a Rule Set, and several Rule Sets make up a Task Set). As a result, each stimulus corresponds to a single ideal action. This is limited as a model of human cognition since the best optimal action given a certain input typically relies on the current goal or context. While changing contexts, a at learning agent would have to repeatedly override or relearn this information (called catastrophic forgetting in case of reversal learning). Biological agents though, must have the ability to cope with, and bene t from continuous, lifelong, online learning. A hierarchical architecture poses a natural solution to the storage of numerous task sets and adaptive switching between them in response to signi cant changes in the stimulus action reward contingencies. In such architectures, several rule sets are learned at the low-Cognitive control over contextual and abstract Rules est level (concrete rules) and the agent chooses when to transition between various rule sets at the upper level (abstract rules). At even higher levels of the hierarchy, i.e, more complex or abstract rules, the decision can be to switch between task sets. Thus, the role of rule and task sets is to provide protection from forgetting the established, previous learning.

To reiterate the distinction between concrete and abstract rules (presented in . . ), in the simplest case, this means learning rules de ned as associations between an object's properties and a direct response. Such rules can be called concrete, while more complex or abstract rules may involve the learning of second order relations on top of the rst-order rules. The prefrontal cortex (PFC) is believed to provide the ability to contextualize concrete rules that leads to the acquisition of abstract rules [ ]. Considering the number of contexts we encounter every day and the ease with which we select appropriate strategies for each, some relevant questions arise : How do we represent these strategies or rules and how do we determine which one is appropriate ? An important way of understanding how the PFC supports contextual learning and implements cognitive control is thus to understand how its representations are organized and manipulated [ ].

There is su cient evidence to suggest that the PFC is organized hierarchically [ ] with more caudal areas learning rst-order associations and more rostral areas putting them in context to facilitate learning of abstract rules. The hierarchy itself can be at two levels : a hierarchy between concrete and abstract rules, and a hierarchy of abstract rules, represented in increasingly rostral areas. The selection of the appropriate rule, depending on the context, is done by top-down modulation in the PFC, which underlies the ability to focus attention on task-relevant stimuli and ignore irrelevant distractors, in two ways in models : either as a result of weight changes in modulated pathways and predictions, or through activation-based biasing provided by a working memory system. Theoretically, despite agreement on the presence of an anterior-posterior (or rostral-caudal) gradient in the PFC, there is substantial ambiguity regarding the representational and/or processing demands that underlie this organization [ ]. Two major hypotheses have been proposed :

The information cascade hypothesis [ ]: The anterior-posterior gradient of lateral PFC is organized in a such a manner as to when a cue stimuli (or context) reduces uncertainty (i.e. provides information useful for) in the action selection process. In order to respond selectively to task signals that are temporally removed from the action selection process, anterior portions of the lateral PFC must be maintained for a long period of time (i.e. across multiple trials). On the contrary, it is hypothesized that posterior areas are receptive to both -cues that are important across trials and cues that arise in close temporal proximity to action selection (i.e. in the same trial).

The levels of abstraction hypothesis [ ]:

The anterior-posterior gradient of PFC is organized according to the level of abstraction (or hierarchical nesting) of the cues required to guide action selection. Thus, the processing of more abstract information about actions (such as collections of stimulus-response mappings) is selectively to anterior parts of the lateral PFC, whereas the processing of more speci c information about actions is thought to be linked to posterior areas of the PFC (e.g. individual stimulus-response mappings).

These two hypotheses make similar predictions under many circumstances, and have been explored in a computational model -the Hierarchical Error Representation (HER) [ ] model, which explains cognitive control in terms of the interaction between the dlPFC (dorsolateral prefrontal cortex) and the mPFC (medial part of the PFC). The dlPFC learns to maintain representations of stimuli that reliably co-occur with outcome prediction error and these error representations are used by the mPFC to re ne predictions about the likely outcomes of actions. The error is broadcasted through the PFC in a bottom-up manner, and modulated predictions from top-down facilitate selection of an appropriate response. Thanks to its recursive architecture, this model, presented in more details below, can elaborate hierarchical rules on the basis of learning by weight updating, both to select pertinent stimuli and to map a representation inspired with principles of predictive coding [ ].

In addition to its elegant recursive mechanism, proposing an original computational mechanism to account for the hierarchical structure of the PFC, the HER model is also very interesting because it proposes to decompose the functioning of the PFC between, on the one hand, the prediction of the outcome and the monitoring of the error of prediction and, on the other hand, the elaboration of contextual (and possibly hierarchical) rules to compensate errors. The context under this framework, is built from error representations in the dlPFC. This distribution of functions has also been reported between respectively the medial and lateral parts of the PFC [ ], yielding more importance to the biological plausibility of the HER model. For these reasons, the HER model could be presented as a more elaborated and accurate model of the PFC, except for two points of discussion that we put forward here.

In the work presented here, we seek to answer speci c questions about the nature of top-down modulation and selective attention, through the lens of hierarchical learning and representations. In the rst section, we start from the implementation of the HER model with a reinforcement learning training signal for the gating mechanism instead of the back-propagated error signal reported in the original paper. We report the benchmark performance of this model on two di erent kinds of hierarchical tasks (corresponding to the information cascade and level of Cognitive control over contextual and abstract Rules abstraction hypotheses). Further, we use the model to study a task in which individual rst-order rules can be learned alone or associated within speci c contexts to form second-order rules. In the second section, we evaluate the performances of the HER model in the di erent cases in such a task, in which the model fails to perform, and compare them with a case where an attentional mechanism should be deployed to facilitate and orient its learning. As discussed in the concluding part, we observe that the attentional mechanism should be considered not only for the processing of information but also for the learning of rules, particularly in the hierarchical and contextual case.

T H E R HER M M This section rst summarizes the HER model algorithm and equations, as described in the original paper [ ] and the subsequent section presents the tasks that we have chosen for our study.

M D HER

W M G

At each level of the hierarchy, external stimuli presented to the model may be stored in WM based on the learned value of storing that stimulus versus maintaining currently active WM representations. External stimuli are represented as a vector s, while internal representations of stimuli are denoted by r. The value of storing the stimulus represented by s in WM versus maintaining current WM representation r is determined as :

v = X T s ( . )
where X is a matrix of weights associating the external stimuli (s) with corresponding WM representations (r). The value of storing stimulus s i (v i ) is compared to the value of maintaining the current contents r j of WM (v j ) using a softmax function : probability of storings i = (exp βvi + bias) (exp βvi + bias) + exp βvj

( . )

. The Hierarchical Error Representation (HER) Model O P Following the update of WM, predictions regarding possible responses and outcomes are computed at each hierarchical layer, using a simple feedforward network :

p = W T r ( . )
where p is a vector of predictions of outcomes and W is a weight matrix associating r and p

T D M

Beginning at the top of the hierarchy, predictions are used to modulate weights at inferior layers and modulated predictions are computed, as shown with the red arrows in gure . . For a given layer, the prediction signal p' additively modulates stimulus-speci c predictions p generated by the lower layer. In order to modulate predictive activity, p' is reshaped into a matrix P' and added to W in order to generate a modulated prediction of outcomes :

m = (W + P ) T r ( . )
These modulated predictions are then used to modulate predictions of additional inferior layers (if any exist)

m = (W + M ) T r ( . ) R S
Actions are learned as response-outcome conjunctions at the lowest layer of the hierarchy. In fact, a weakness of functional description of the model is that these concrete rules are simply represented at the lowest level of the hierarchy, while according to our functional description, these rules must be learned or represented in the premotor cortex, and thus not be part of the prefrontal hierarchy per se, a distinction we clarify in our proposed model. To select a response, the model Cognitive control over contextual and abstract Rules compares the modulated prediction of correct feedback to the prediction of error feedback, for each candidate response :

u response = m response/correct -m response/error ( . )
This is then used in a softmax function to determine a response :

P rob(u i ) =
exp γui exp γu ( . ) B P Following the model's response, it receives feedback regarding its performance and two error signals are computed at the bottom most hierarchical layer, one comparing the unmodulated predictions to the outcome :

e = a(o -p) ( . )
and another comparing the modulated predictions to the outcome :

e = a(o -m) ( . )
where o is the vector of observed outcomes and a is a lter that is for outcomes corresponding to unselected actions and everywhere else.

The outer product of the rst error signal and the current contents of the WM at the bottom level is used as the feedback signal for the immediately superior layer where this process is repeated ( gure . ).

O = re T ( . )
E ectively, at the second layer, the outcome matrix is a conjunction of stimuli, actions and outcomes. This matrix is reshaped into a vector o' and used to compute the prediction error at the superior layers :

e = a (o -p ) ( . ) W U
. The Hierarchical Error Representation (HER) Model

The second error signal is used to update weights within the bottom-most hierarchical layer, it updates the weights connecting the WM representation to prediction units (W), as well as weights in the WM gating mechanism (X):

X t+1 = X t + (e T t W t • r t )d T t ( . )
An eligibility vector d is used instead of the stimulus vector s. When a stimulus i is presented, the value of d i is set to , indicating a currently observed stimulus and at each iteration of the model, d is multiplied by a constant decay parameter indicating gradually decaying eligibility traces.

The above equation uses a backpropagated error to train the associative weights X, a learning mechanism considered neurally implausible.

New learning rule:

We have proposed (and use for the rest of this work), a more biologically plausible training of the gating mechanism, using a scalar reinforcement learning signal. On each simulated trial, the model receives feedback regarding its performance : error or correct. In a reinforcement learning framework, such feedback constitutes a binary reward signal which takes the value of for correct feedback, and for error feedback. In RL, learning is driven by the di erence between a received reward and a predicted reward; however the HER model learns predictions of multiple possible outcomes based only on the likelihood of observing the outcome and without regard for its a ective valence. In order to model reward predictions, the output of a softmax function for the selected behavior is used as a proxy for reward prediction, with a temperature parameter set to in the simulations.

Value = exp γui exp γu ( . )
When correct feedback is highly probable, the output of the softmax function will be close to for the response associated with receiving correct feedback. Following feedback, a scalar error term is calculated as the outcome feedback (correct/error) minus the proxy reward prediction.

δ t = reward t -V alue t ( . )
Weights in the WM mechanism are then updated according to :

X t+1 = X t + α t r t d T t ( . )

Cognitive control over contextual and abstract Rules

Finally, the prediction weights in the model are updated according to Stimuli are presented beginning with a context cue ( or ), followed by a pattern cue (A or B) and then with a target cue (X or Y). The context cue indicates which pattern cue -target sequence is valid (AX or BY). Sequences of stimuli are organized as inner and outer loops. This task is an extension of the AXCPT task. It is an example of a learning paradigm, where the hierarchy is present in the form of temporal abstraction (the higher level context cue is to be maintained until a valid target sequence is observed). It tests the working memory of the subjects. In the notional task, as depicted in gure . , each compound stimulus consists of two dimensions, shape and orientation, with dimensionalities of two and three respectively. This results in a task with three possible responses and two response mappings. It should be noted that structured tasks of this sort can be solved using a generic backpropagation network with a single hidden layer. Although such networks are not hierarchically structured, they are able to learn hierarchical tasks through learning conjunction of features, represented by activity in hidden units, which can then be used to generate appropriate responses.

W t+1 = W t + α(e t r T t ) ( . ) W 1 W 0 W 2 r0

H C T

In the hierarchical task reported by Badre, an individual stimulus conjunction consisted of one of shapes, at one of orientations, inside a box that was one of colors, for a total of unique stimuli ( shapes x orientations x colors). The arrangement of response mappings for these stimuli was such that a second order relationship existed. In the context of one colored box, only the shape dimension Cognitive control over contextual and abstract Rules was relevant to the response, with each of the unique shapes mapping to one of the button responses regardless of orientation. Conversely, in the context of the other colored box, only the orientation dimension was relevant to the response. Thus, this task permitted learning of an abstract conditional rule that speci ed how one dimension (color) determined which of the other dimensions (shape or orientation) would provide a context for selecting a response.

R B P S AX C P T

The HER model was simulated on a version of the -AX task as described in O'Reilly and Frank ( ) in which each outer loop consisted of one to four inner loops and the probability of observing a valid sequence for each inner loop was . . There were six inputs to the model, corresponding to the six relevant cues in the task. At each cue, the model made a response to indicate whether the current sitmulus was a target (or not Of the simulations, the model successfully met criterion % of times. On average, the number of cue presentations to criterion was . trials (approximately epochs). According to the other criterion, on average, the number of cue presentations to criterion was . trials (approximately epochs)

Cognitive control over contextual and abstract Rules

S S T

The model was simulated performing the task for trials, and simulations were performed. To assess performance, the model was considered to have completed learning the task on the rst of consecutive trials in which no errors were committed. The results are show in gure . (a). The more complicated version of this task, in which one dimension provides the context for which one of the other dimensions must be attended to is unable to meet the convergence criteria. Because of the speci city of representation at each hierarchical layer, the HER model is unable to generalize associations learned in one context to a novel one. Once representations are formed, prediction related activity in mPFC may be sufcient to elicit associated activity in dlPFC that may then be dynamically mapped to external stimuli. It is unlikely that detailed information carried by the error signal in the HER model is used to train a gating mechanism as implemented in real brains. Rather, it is more likely that such training involves a scalar signal, typically associated with dopaminergic activity.

I M

A hierarchical extension of the PBWM model [ ] [ ] proposes that hierarchical control can arise from multiple nested frontostriatal loops (loops between the PFC and the BG). The system adaptively learns to represent and maintain higher order information in rostral regions which conditionalize attentional selection in more caudal regions. All the adaptations of the HER model are made through learning by weight modi cations, whereas the property of working memory of the PFC, as it is for example exploited in the PBWM model, is often presented as a key mechanisms for its adaptive capabilities. These two mechanisms re ect two very di erent ways of learning: one purely cortical and slower way of building representations on the cortical surface on the longer term, and the other based on subcortical interactions that learn when to gate information (using the cortical representations) on a faster timescale. These two mechanisms on the surface seem incompatible, and di cult to compare. An important question is consequently to determine up to which point working memory and attentional modulations are necessary for the learning of hierarchical rules in cognitive control. Through a careful examination of these mechanisms, we conclude that they in fact are complementary, and we propose a model, and an experimental paradigm that is able to reconcile these two views of learning into a an integrative framework.

One way to learn e ciently in complicated surroundings is to use selective attention, which can narrow down the task's complexity. This means that attention is focused on a few environmental aspects that are relevant to the task at hand, while others are ignored or generalized over. By doing this, the number of stimulus con gurations of di erent states that need to be considered is reduced. However, attention should only be directed to dimensions of the environment that are important for the task, i.e., dimensions that predict reward, in order to provide a suitable representation of the task's state for the learning process. It is not always clear which dimensions are relevant to a particular task and may need to be learned through experience, a mechanism thought to be implemented by the use of an output gate.

O G

Working memory may not always contain information that is pertinent to behavior at any given time. Instead, it is also adaptive to regulate which WM representations can a ect attention and action selection, and when. Such selection from inside the working memory, or singling out of the WM representations is resource demanding and PFC dependent. A possibility is that selection from within WM can be conceived of as a gating function, similar to that outline for WM updating. According to this perspective, an output gate can regulate the information ow within the WM, between an actively maintained but inert state, to one that is capable of exerting a top down in uence on behavior. To put it another way, when the output gate is closed, information would be maintained, but not have any modulatory top-down e ect on behavior. Conversely, when the output gate is

. Integrative Model open, the maintained information would provide a top-down contextual signal (illustrated in gure . ). This issue of selection from within WM by output gating has been proposed as a solution by many computational models []. In the model with output gating and selective attention, instead of using modulated predictions P from the superior layer, additively in the lower layer, the agent uses the prediction mapping, to rather select which stimulus dimension needs to be attended to, out of the ones maintained at the lower layer r 0,i p m,i = max(P T r 0,i ) ( . )

Default behavior

Cognitive control over contextual and abstract Rules

These predictions are then used in a softmax function to determine which stimulus dimension (or rule set) will nally be selected to determine the response at the lower layer P rob(r 0,i ) = exp pm,i exp p m,i ( . ) The third one (Block in gure . (b)) is a random mix of trials from the other two blocks. This framework is interesting because, whereas rules T and T in blocks and require the subject to attend to a single dimension of the stimulus, block requires to pay attention to both and to decide which rule to apply based on the third (contextual) dimension. Let us also mention here that, while there is no apparent di culty with such tasks, it is actually harder than it appears depending on the way a task is learnt. During block , one can either learn the rule : "green means case and black ignore" or the rule: "black ignore, else case". The same is true for block with sound. If we now consider block and depending on how a subject learnt the rst two blocks, she may succeed or fail immediately. In this latter case, this means block cannot exploit previous learning and has to be (re)learnt.

The original task was cued by instruction and corresponding performances were reported in the paper [ ]. Here, we wish to explore the inherent capability of a model to learn an abstract and hierarchical rule task without instructional cues, as in the paradigm reported by [ ] and also to consider how the hierarchy can be learnt, depending on how information is represented in the model. We used two types of learning paradigms for the simulations : the rst paradigm in which rules T and T were learned one after the other, and the performance of the model was then tested on random trials interleaved from rule T and T (to say it di erently, we apply successively block , and ). In the second paradigm, an entire abstract rule that we call T , corresponding to the selection on rules T and T depending on the contextual cue 'color' was directly learned (block applied rst) and performance of the model was subsequently tested on rule T and T (blocks and ). In the next section, we report performances observed with the HER model and with an adapted version that we propose subsequently. E,I,O,a,e,i,o,C,G,K,P,c,g,k

Cognitive control over contextual and abstract Rules

X X X T2 T2 X X X T1 T1 X X X T1 T2 X : {A,

R S S

We have rst studied how the HER model, as it has been designed (cf section . . ), can address the tasks de ned above, under the two mentioned paradigms (cf section . . ). Due to the design of the HER model, each layer can only map or process one stimulus value, thus requiring as many layers as there are stimulus dimensions. The mapping in the model is also highly sensitive to the stimulus dimensions relative to one another, particularly higher-dimensional stimulus are preferentially mapped onto the lowest hierarchical layer. This rests on the assumption that stimulus dimensions better able to predict and reduce uncertainty about the response are mapped to lower layers. This may not always be the case in real life situations though. We often have to adapt and generalize the same rules over several di erent contexts. In the task we consider as well, the context is determined by the color, which has possible values -one of which always maps to the same response (to ignore) and the other determine the response based on other stimulus dimensions.

L C

Performance observed for the rst and second learning paradigms are reported in gures . (a) and (b) respectively. We see in the gure . (b) that due to its hier-. Integrative Model archical structure, when there is an underlying abstract rule to learn (rule T ), the model is able to use the hierarchical information to acquire the rule while retaining performance in each of the sub-rules (Rule T and T ). It does so by monitoring an "error of errors" at each hierarchical layer, broadcasting this error to superior layers (bottom-up processing) that put it in context with the stimulus feature being attended to and nally sends this prediction information to the lower layers (top-down modulation) which are able to then select the appropriate response. In the gure . (a), we show that when the composite rules are rst learnt sequentially, the model is not able to compose them into a single rule, but instead has to relearn its representations to reach optimal performance. Next we show that due to the design of the model, a task which has only one level of hierarchy, such as the one considered here, can not be learnt with a model with layers. In gure . (c) we see that with layers, the model is able to learn the subparts of the rule (rules T and T ), but performance on the composite rule T saturates at %. By exploiting the gating mechanism, each sub-rule can be learnt individually by gating the relevant feature dimensions at the layers (color, vowel/consonant for rule T and color, lower/upper case for rule T ). However, in the third rule T when the relevant features change from trial to trial to determine the correct response, the model fails to learn, since the contextual stimulus features don't provide top-down information about "which" other stimulus feature to attend to at the lower layer.

G

In the model, the gating weights determine both, when to update or maintain a stimulus feature, and also which of the stimulus features is to be gated. We observed the adjusted weights after each rule that is learned. In the rst block, vowel, consonant and black have high values of getting updated at the lowest layer, while in rule T all the "lower level" cues have high values of getting updated. In such a case, there is again competition between which one of them to gate, and both can win with close probabilities, in the absence of any information from the superior layers. Depending on what is gated into the top two layers, any of those mappings could emerge.

P

The prediction weights at layer are Stimulus-Action-Outcome conjugations and the gating mechanism determines which stimulus and in turn which actionoutcome association is to be selected. The selected associations are then modulated by superior layers and used the determine the response. At layer , the prediction errors of layer are contextualized to make SxSxAxO conjugations and so on.

In the task considered for all our simulations, there are concrete rules or S-A-O predictions to learn : Black -Action , Vowel, Lower case -Action and Consonant, Upper case -Action ( gure . (b)). In gure . , we present examples of how a model with layers selects a response by additive prediction modulation. We observed that elaborating a mapping between the stimulus and what is gated into the internal representation (r) at di erent layers could be done in di erent ways, including randomly, as long as these mappings led to orthogonal and mu-. Integrative Model tually exclusive activations of predictions (in W). For example, in gure . (e), in Block , the color red was not gated into the internal representation, but the random gating of the other dimensions still led to an appropriate modulated prediction that could initiate the correct response. (c) (d) (e) . We tested the ability of the HER model with layers, to use this information and contextualize it to learn the abstract rule. The bottom layer of the model was initialized to the predictions previously learned and moreover, it was "frozen" such that no learning happened at this level, implying that these behaviors were rigid. At the upper layer, the gating weights were biased to update the internal representation with the context, which was the color in this case, implying saliency to previously unattended cues. As expected, the model failed to learn the abstract rule with these modi cations. With the modi ed model, we used the same protocol ie the bottom layer was kept frozen, and there was a bias added to the upper layer to encourage gating of the color. However, instead of an independent gating at the bottom layer, we included an output gating from the upper layer, which used the prediction errors at the upper layer to select which stimulus dimension was going to be gated into the bottom layer ( gure . (a)). To put it more generally, the bottom layer was responsible for response selection while the upper layer was responsible for action-set selection through targeted attention (cf [ ] for more details about the structuring concept of action-set and its role in PFC information processing). Our modi ed model achieved optimal performance fairly quickly, as shown in . (c).

R1/C R1/E R2/C R2/E R3/C R3/E L 2 -C L 1 -W L 0 -W L 0 -m o d u
R1/C R1/E R2/C R2/E R3/C R3/E L 2 -U L 1 -G L 0 -V L 0 -m o d u
R1/C R1/E R2/C R2/E R3/C R3/E L 2 -V L 1 -G L 0 -G L 0 -m o d u
R1/C R1/E R2/C R2/E R3/C R3/E L 2 -V L 1 -U L 0 -U L 0 -m o d u
R1/C R1/E R2/C R2/E R3/C R3/E L 2 -U L 1 -R L 0 -V L 0 -m o d u
.

D

The PFC plays a major role in cognitive control and particularly for learning, selecting and monitoring hierarchical rules. For example, in experimental paradigms, discrimination or categorization tasks can be considered as rst-order rules which could be learned individually. However, when con icting stimuli are presented si-Cognitive control over contextual and abstract Rules multaneously, a contextual cue is needed to identify which of the rst order rules is to be applied, thus forming second-order rules.

The inner mechanisms of the PFC have been studied in computational models and among them, the property of working memory used for biasing by selective attention in the PBWM model and, more recently in the HER model, the separation between outcome prediction error monitoring, and hierarchical rule learning. Considering the indisputable progress brought by the design of the HER model, we questioned whether it was now a standalone model of the PFC to be used in any circumstances or if the contribution of certain mechanisms like selective attention was still to be considered in some cases and possibly added to the general framework of PFC modeling. More speci cally, considering the deployment of cognitive control in realistic behavioral tasks and considering that most hierarchical representations arise from the intersection between agents and the problems they face, and are created over time in a learning process, in a rapid and exible way, our question was to know if the HER model could account for this kind of process.

Using a task elaborated along two paradigms, we show that, when concrete rules are already learnt and need to be contextualized, the use of a biasing selective attention mechanism is more e ective than modulated weights changes in displaying e ective cognitive control. When concrete rules are acquired rst, superior layers must learn to select the appropriate concrete rule by targeted attention, rather than by relearning representations. We observe that a subject can perform optimally on a given task even though she uses a di erent rule representation compared to the official one. On a single task, this has no consequence and there is actually no way to know which exact rule is used internally. However, when this rule needs to be composed with another rule such as to form a new rule, this may pose problem and lead to bad performance. This has been illustrated on the task: if a subject uses any of the alternative rules for tasks T or T , she'll be unable to solve task T even though this task is merely made of a mix of T or T trials. The reason for the failure of the HER model in this case is to be found in the failure to attend the relevant dimension of the task, here, color, thus claiming for considering and incorporating this mechanism to a versatile PFC model. Analyzing these results in a more general view, we can remark that most experimental paradigms that study hierarchy break down the complexity of a task by providing instructional cues to the participant. Even in studies with rodents and non-human primates, shaping is used in learning paradigms to enable the learning of complex or abstract rules. In developmental learning, this kind of shaping is called curriculum learning. It is evident that such breaking down of complexity must facilitate the acquisition of . Discussion and Perspectives abstract rules, and hence modeling approaches must demonstrate these behavioral results.

It is thus important for a model of the PFC to exploit both views, suggesting to incorporate an attentional mechanism for the exible and controlled design of hierarchical rules from previously learned concrete rules, as we proposed in the new model sketched here.

D P

In this Chapter, we have described a new integrative computational model of cognitive control in the PFC, delineating the roles of (i) mPFC and dlPFC, (ii) posterior and anterior PFC, and (iii) nested CBG loops, building on previous models of hierarchical representations and learning.

Mapping the model to brain regions : In the model, the division of labor between representing abstract rules, or in other words maintaining predictions of possible events versus monitoring errors is assigned to dlPFC and mPFC respectively. Similarly, the distinction between learned concrete rules, and implementing abstract rules, is assigned on the posterior-anterior axis, with the premotor cortex responsible for the former, and more anterior regions (prePMd, mid-dlPFC) for the latter. Further, we make a clear distinction between learning cortical associations in dorsal-medial PFC, versus learning to select the relevant representations through gating with the CBG loops.

Our model proposes a number of mechanisms regarding the interaction of different brain regions. In the HER model, error signals generated by the mPFC are used to train weights that govern how and when a stimulus may be stored in working memory. This is signi cantly di erent from the proposed role for BG in WM updating. Consequently, the more plausible reinforcement learning mechanism that we propose is more consistent with the second account, and with the known anatomy of CBG loops. As in the HER model, the medial PFC provides prediction error signals that train the lateral PFC regarding potential errors associated with di erent if-then rules. The architectural di erences included in our model, as borrowed from the theory of nested cortico-striatal loops, where selected abstract rule representations in rostral regions, constrain the selection of concrete rules in caudal regions (through output gating) is a signi cant di erence that allows the system to build on previously acquired information, and thus allows the possibility to generalize learning over multiple contexts (a limitation of the originally proposed HER model). Moreover, the gating mechanism in our model does not restrict the dlPFC's role to maintaining representations over extended periods Cognitive control over contextual and abstract Rules of time, but also allows transient activity in the dlPFC to in uence rule selection at lower layers.

Thus, the model synthesizes a number of ideas that have previously appeared in the neuroscienti c and modeling literature. In light of the evidence we have provided in this chapter, our results con rm that a hierarchically organized system allows individuals to form abstractions that can be applied in a variety of di erent contexts. Moreover, once discovered, these abstractions or chunks can be applied to novel situations, and help in generalization or transfer learning. Our model provides an architecture that substantiates the intuition that breaking apart a complex task into a multitude of smaller ones will automatically establish a form of hierarchical structure. In the particular experiments we simulate, a set of base competencies is provided (concrete rules, or otherwise policies in RL) upon which more complex learning is based. In this context, our results suggest a bidirectional interaction between attention and learning : attention constrains learning to relevant dimensions of the environment, while the model learns what to attend to via trial and error.

We also highlight that the ability to selectively update some contents of WM while leaving others intact is a process that might be fundamental to hierarchical behavior, because the nature of task representations may have, by de nition, different temporal dynamics. Larger goals are relevant over longer periods of time, and thus should not be updated once one sub goal is completed and another is begun.

R

Neurophysiology : To extent of nding the particular brain systems responsible for deploying selective attention, there remains some discrepancy. In the fMRI study by Nee and Brown [ ], on the AX CPT task described and modeled in previous sections, the authors observed that updating the WM with higher level contextual information involved the anterior PFC and BG. By contrast, updating WM with lower level contextual information involved the posterior PFC and the posterior parietal cortex (PPC). Based on these ndings, the authors suggest that responses to lower level context updates may be better modeled as attention shifts than changes in the WM content. Our proposal in the model presented in this chapter is consistent with this account that the presentation of contexts serves to limit the set of relevant rules and engages selective attention towards the currently relevant rules. Interestingly, regions involved in attention are located just anterior to those involved in motor preparation, and consequently the results from this . Discussion and Perspectives study could re ect a mixture of motor preparatory processes caudally and attention process rostrally. This could also be due to the fact that at lower levels, the function has already been overtrained and consequently there is no observable activation of the caudal CBG loop, but rather the habitual execution of behavior [ ].

Hierarchical organization : Reynolds, O'Reilly, Cohen, and Braver [ ] provide a third possible account of the hierarchical organization of the lateral cortex (adaptive context maintenance hypothesis), as a middle ground or compromise between the information cascade hypothesis and the levels of abstraction hypothesis. This theory postulates that both the posterior and anterior regions of PFC are reliably engaged in task conditions requiring active maintenance of contextual information, and that the temporal dynamics of activity in these regions exibly track the duration of task demands. Areas of both mid-dlPFC and posterior PFC displayed sustained activity when information had to be maintained across multiple trials and more transient activity when information had to be updated frequently.

Experimental Paradigm : To test the demands on input and output gating, Chatham, Frank, and Badre [ ] designed an experimental protocol based on hierarchical control tasks requiring the use of conditional rules. In the task, subjects had to base their responses on one of two possible letters, or one of two possible symbols, depending on the identity of a number cue. Thus the number acted as the higher-order context that speci ed which set of lower order items (letters, or symbols) would be relevant for the response at that trial. This can be seen as a variation on the temporal AX task, and the hierarchical categorization tasks described previously, with abstraction at both the temporal, and policy levels. Of note was that each stimulus was presented in an unpredictable serial order on each trial, with the higher order context being presented either before (context rst CF) or after (context last CL) the lower order contexts had appeared. Thus, the CF conditions would allow the subjects to use an input gating strategy while conversely, under the CL conditions, subjects would need to input each lower order item into the WM and then on the presentation of the context in the last position, select from among the items maintained in the WM that which would in uence the response. Thus, the CL condition required the subjects to use an output gating strategy. The fMRI results of the experiment showed that activation was greater for contexts presented last relative to rst in the lateral frontal and partietal cortices, peaking in prePMd. The posterior parietal cortex was also more strongly recruited for CL than for CF conditions. Consistent with the results from Nee and Brown, the authors failed to observe corticostriatal connectivity during input gating of the lower level context. Instead, corticostriatal coupling increased during the output gating of lower level items. Figure . illustrates a schematic description of how and when input and output gating acts to solve this task in the two conditions.

Computational modeling : We have already discussed the ideas borrowed in our model from the HER and hierarchical PBWM models. In a di erent account, the PROBE model [ ], models abstract rules as task sets, with each task set represents one strategy stored in long term memory, comprising of . a selective mapping Q(s, a) which encodes the stimulus-response associations . a predictive mapping γ(o, s, a) which encodes the expected action outcomes given stimuli

. a contextual mapping F (i|C) which encodes external cues predicting the task set reliability

The selection happens at the level of the task set rst, then at the level of actions within task sets. Analogous to these computations, in our model, the prediction

Cognitive control over contextual and abstract Rules

weight matrix W at each layer can be thought of as a predictive mapping, and the action values derived from these mappings can be thought of as a selective mapping. However, while the PROBE and C-TS model [ ] use RL formulations for learning concrete rules in conjunction with bayesian approaches for updating beliefs about task sets, the model presented here uses the same mechanism to compute error and drive learning.

Another prevalent theory states that quantities related to prediction and calculation of value might constitute the common currency under which the functions of brain regions must be interpreted. The open question then is whether predictive coding in general and the HER model in particular might be expanded to account for the function of additional regions of the PFC without reference to explicit value signalling .

In chapter , we have discussed the key ndings from neuroscience about the functional subdivisions of the PFC. In this chapter, we focused on the question -"How might the computational modeling of Cognitive Control be in uenced by the architectural and algorithmic constraints imposed by the neurobiology of the brain ?" We have shown through our work in this chapter, that a general principle of predictive coding, organized hierarchically is consistent with the neuroscience ndings, and that such hierarchy aids in, and may be necessary for the elaboration of contextual or abstract rules. The same general algorithms, with di erent interpretations of the input and output conditions can explain a lot of the internal information processing likely to be implemented in the brain. In particular, we have highlighted the role of selective attention in working memory, and how this architectural condition can aid in the selection of lower order rules from a possibly large repertoire of learned behaviors.

C

We began this manuscript with an apparently simple question : How does the brain choose e ectively and adaptively among available options to ensure coherent, goal directed behavior ?

The emerging answers to this question from a range of theoretical perspectives have homed in on a consistent set of key computational principles, some of which are shared across species, that emphasize the shaping of behavior at di erent levels of organization. These principles are implemented in a core set of neural structures that support the valuation, comparison, and selection of behavioral options. The manuscript presented here highlights the achievements and missing elements in the study of cognitive control and proposes a systemic description of the gradient of cognitive control. We described a three fold criteria as the objective of this thesis:

. to highlight the need for recruiting di erent mechanisms of cognitive control (to deal with catastrophic forgetting, dealing with the stability-plasticity dilemma and exibly switching between rules) under di erent tasks and conditions

. to algorithmically describe all the mechanisms that fall under the purview of cognitive control in both rodents and primates (working memory, attention, prediction of errors)

. to model the aforementioned algorithms in a biologically plausible manner (hierarchical organization along the posterior-anterior axis, rule representation versus monitoring along the lateral-medial axis, working memory maintenance and selective attention through CBG loops).

Here, we summarize the contributions of this work, with regards to successfully meeting the criteria identi ed as missing elements.

In chapter , we use the actor critic framework to describe the interaction of multiple learning systems in rodent models, and determine the way that slight differences in task demands can change the demands on these di erent subsystems in the brains of rodents. We describe how in the simplest case, as permitted by the agranular frontal areas, an immediate decision is su cient to trigger the behavior Cognitive control over contextual and abstract Rules in a stable world, from the selective and predictive models. We hypothesised that mice do possess a "working memory" like capacity, that lets them express contextual behavior. We highlight an oft ignored aspect of cognitive control in computational models -the presence of an innate or "default" behavior i.e. the behavior or actions that occur automatically in response to certain stimuli or contexts. We show that this behavior is not always optimal or desirable, and models need to account for the necessity to override these behaviors in such cases. Our analysis from the three studies is in line with experimental ndings from rodent literature, which implicate the dorsal striatum in learning S-R associations and action selection, the mPFC acts as the cognitive control center, implementing a working memory like system, which along with the episodic memory from the hippocampus helps in contextualizing learning. Particularly, in this chapter, we show the limits of cognitive control as seen in rodents, due to the organization and complexity of their brains.

In chapter , we show how as permitted by the granular frontal areas, sensory representations must be modi ed by cognitive control to be adapted to the situation, after an internal deliberation possibly exploiting episodic and semantic memory. We use the HER model as the basis of our computational framework, which suggests that error calculation and representation serve as the common code underlying neural activity and communication. In our study, we investigate the function of speci c cerebral structures. Our analysis suggests that the basal ganglia (BG) should be seen as a modulatory system that provides the frontal cortex with exible gating signals, rather than as a procedural learning system that directly encodes stimulus-response associations, which is the prevailing idea. The evidence supporting this comes from the observation that BG lesions have a greater e ect on learning than on behavioral performance. Further, we show how selective attention is deployed by the PFC to guide rule selection at lower levels. We describe not only the interactions between the frontal cortex and the basal ganglia, but also the interactions between the medial and lateral part of the PFC.

However, there is a risk in computational neuroscience of constructing models of speci c neural structures in isolation, without considering the broader information ows. In fact, the fundamental biological mechanisms underlying cognitive control have been implemented in various models, but only in isolation and not in an integrative manner. This fractionation of interpretations by specialized subelds may result in an incomplete understanding of the neural mechanisms underlying human behavior. Our description of cognitive control is thus in width rather than depth, and hence evokes a variety of sensorimotor loops and levels of repre-sentation, from pavlovian to instrumental conditioning, from goal directed to habitual behavior, form episodic to semantic memory, from simple to complex rules, that can coexist and act in competition or in synergy. We propose steps towards integrating key methods and mechanisms to overcome the challenge of developing a uni ed cognitive architecture.

There is a growing trend in computational neuroscience to develop new models that aim to explain neural processes from scratch, rather than building on existing models and re ning them. While this approach can lead to novel insights, it can also result in a proliferation of redundant and overlapping models, which can be di cult to integrate, reconcile and compare. A potential problem with reinventing the wheel is that it can lead to a lack of theoretical coherence, hence diluting the explainability of these models, and eventually making it a challenge to develop a comprehensive understanding of the brain and its functions. We chose in this work, to build on existing models to be able to rst, leverage the insights gained from previous research and to be able to address our speci c research questions and hypotheses by re ning on them. Further, by re ning and improving on these models, we are better able to contribute to the ongoing development to the eld of cognitive control.

Our presented model of Cognitive Control is by no means complete. For example, uncertainty is de ned both as stochasticity (i.e expected uncertainty) and volatility (i.e unexpected uncertainty), and while we give plausible bayesian inference methods to deal with volatility, our model and analysis is limited to deterministic tasks, in which S-R contingencies are strongly de ned. In tasks like the multi arm bandit task, in which the reward contingencies behind di erent actions is probabilistic, and especially in cases in which this probability itself might gradually change over time, it is unclear how our model would be able to deal with such scenarios. A meta learning mechanism may be needed. For example, in the paper by Wang et al [ ], the authors show how their RNN is able to tune metaparameters such that after su cient training, the model "learns how to learn" i.e., it is able to converge on the optimal strategy for the bandit task fairly quickly.

There is evidence of how neurotransmitters like acetylcholine and norepinephrine may play a role in signalling these uncertainty computations [ ] [ ], and these mechanisms may be integrated into our proposed model at a later stage. Further, we also don't delve into the particular roles of tonic and phasic dopamine in the prediction of errors. There may also be a lateralization e ect in the networks associated with cognitive control [ ], an avenue which was beyond the scope of this thesis.

Cognitive control over contextual and abstract Rules

Second, it is unclear how and when humans decide to create new rules (or task sets) and hence this problem remains untackled in the model as well. At what point do we decide that none of previously learned knowledge is useful for the task at hand and we must learn a new strategy ? P On the machine learning side, in recent years the emphasis has been on Deep Networks, which are powerful for executing individual tasks, but lack exibility in several other aspects. State of the art methods remain inferior to human learners in their ability to transfer knowledge to new domains, to capture compositional or systematic structure, to play e ciently and to reason abstractly. The need for using PFC-inspired principles to make deep learning architectures more adaptable to realistic learning times has been identi ed [ ] and in fact the use of attention has become an increasingly popular approach in many tasks. Although the Neural Turing Machines and Long Short-Term Memory architectures have been suggested for the processing of structured and temporal data, they still rely on the same slow and data-intensive learning principles using di erentiable functions. Moreover, traditional deep reinforcement learning is slow and incompatible with biological system observations in similar tasks. As a result, episodic reinforcement learning and meta reinforcement learning have been de ned, taking inspiration from the hippocampus and prefrontal cortex respectively. However, what is still lacking and could bene t from the current framework is how both learning techniques are linked and interact in their development.

Learned rules are continually re ned and updated through experience to become ever more e ective in guiding adaptive behavior. In fact, one could argue that the primary job of learning is to extend and enrich internal causal models of the world. This kind of continuous lifelong, online learning can be useful in the context of AI systems that need to learn in real time and adapt their knowledge and skills based on new experiences and data. Compositionality and learning to learn might be ingredients that make this type of rapid learning possible. When humans or machines make inferences that go beyond the available data, it is likely that strong prior knowledge or inductive biases are compensating for the missing information. Learning to learn is a term used to describe how people acquire prior knowledge through previous learning experiences related to a particular task or concept, closely related to the machine learning notions of transfer learning, multi-task learning or representational learning. This prior knowledge can accelerate the learning process of a new task or concept. Hierarchical bayesian model-ing involves sharing a general prior on concepts among multiple speci c concepts, which is learned over time as speci c concepts are learned. This approach has been used to explain human learning in various cognitive domains. In machine vision, deep convolutional networks or other discriminative methods can learn-to-learn by sharing features between old and new objects or tasks. Neural networks can also optimize their hyperparameters over a set of related tasks.

Another interesting research perspective is three stages of the development of cognitive control as proposed by Munakata, Snyder, and Chatham [ ]. First, children develop the ability to break away from habits and engage in cognitive control in response to environmental stimuli as they grow. The development is linked to their ability to retain information relevant to their goals, starting with concrete information, such as remembering the location of a hidden toy, and gradually moving towards more abstract information, like task rules. These actively maintained representations help support their goal directed thoughts and behaviors. As they continue to develop, children start recruiting cognitive control more proactively, anticipating the need for it instead of reacting to the moment. They also become more self directed in their cognitive control, relying less on external signals and more on their internal representations.

A complete picture of cognitive control is also incomplete without reference to the interactions between the PFC and the hippocampus. Evidence suggests that abstracted representations in the medial prefrontal cortex (mPFC) guide the reactivation of related memories during new encoding events, which promotes the integration of related experiences in the hippocampus. Recent studies have also shown that integrated memories are used during novel situations to facilitate a range of behaviors such as spatial navigation and imagination [ ]. Moreover, information that is congruent with existing knowledge (a schema) is usually better remembered than less congruent information and thus, another learning system may be needed to overcome interference resulting from multiple medial temporal lobe (MTL) instances sharing common elements. The function of the mPFC may then be to select the most relevant elements of an experience during both encoding and retrieval, and suppress activity in representations inconsistent with the dominant schema while amplifying activity in congruent representations [ ]. The ability to imagine hypothetical situations before they happen can increase the accuracy of predicting their outcome. Remembering past experiences is a reconstructive process where memories are recreated from their component parts. Construction may therefore crucial for planning the future and recalling the past [ ].

Figure . :

 . Figure . : Model-based and model-free actions in a simpli ed maze task. (a) A simple maze with three states (S , S and S ) from which the animal has to make leftright decisions, with the terminal states yielding outcomes of cheese, nothing, water or carrots. (b) The values of these outcomes under three di erent motivational states : hunger, thirst, and cheese devaluation. (c) A tree-based model of the state-action environment, which can be used to guide decisions at each state by a model-based controller. (d) The cached values available to a modelfree, habitual controller. Immediately after cheese devaluation, these values do not change (in contrast to the model-based controller). It is only after direct experience with the devalued cheese that the value associated with Left (S ), and subsequently Left (S ), is reduced. Figure taken from [ ]

Figure . :

 . Figure . : Schematic gure illustrating the main cortico-basal ganglia thalamocortical circuits within the human brain. Red : motor loops, Green : associative loops, Blue : limbic loops (Figure taken from [ ])

  choices [ ] [ ], the process of decision making [ ], the representation of motivation [ ] [ ], outcome and error [ ][ ] and the value of the course of action pursued [ ] [ ].

  Figure . : Major brain regions that support executive control The brain regions that mediate executive control are (A) lateral structures including the lateral prefrontal cortex (PFC), which is split into dorsal and ventral aspects, and the posterior parietal cortex; (B) midline and inferior structures including the ventromedial PFC, orbitofrontal cortex (which continues to the ventral surface of the frontal lobes, not shown), and dorsomedial PFC, which includes the anterior cingulate gyrus (Figure taken from [ ])

Figure . :

 . Figure . : Medial and orbital frontal cortex in rodents, macaques, and humansFive functional regions, dorsal anterior cingulate cortex (dACC), perigenual anterior cingulate cortex (pgACC), dorsomedial prefrontal cortex (dmPFC), anterior medial frontal cortex (amPFC), and ventromedial prefrontal cortex (vmPFC/mOFC) are shown in relation to cytoarchitectonic maps of rat (left), macaque (center), and human (right) medial and orbital frontal cortex. The color scheme indicates that the orbital region of rodents has some functional features shared with primate dmPFC, amFC, and vmPFC/mOFC but that it does not correspond in a simple way to any of them. Thus, while the extent to which regions are homologous across humans and other primates, such as macaques, is relatively clear, correspondences between primates and rodents are more contentious and unlikely to be one-to-one in nature. (as illustrated in [ ])

  Studies show that individual neurons in the region re categorically different responses depending on the speci c or abstract rule used [ ][ ] [ ]. It has also been shown that this region plays an important role in working memory [ ] [ ]. The dlPFC has long been considered to house representations that guide temporally integrated, goal-directed behavior. [ ] [ ]. Recent work has re ned this idea by demonstrating that dlPFC neurons play a direct role in representing task sets. [ ] [ ] [ ]

  Figure . : Functional organization of PFC (a) Rule Learning : Sensory representations are encoded in the posterior cortex, with two kinds of information : what in the posterior temporal cortex andwhere/how in the posterior parietal cortex; the motor-premotor cortex encodes concrete sensorimotor rules, and transforms sensory representations into actions; in the limbic regions of the frontal cortex, values of goals (vmPFC) and values/cost of actions (dmPFC) are learned, along with interactions with the striatal regions. These learned values can then be used to select the best concrete rule from the situation (stimulus driven) or depending on a current goal (goal driven). Typically, the same rule is selected in certain contexts (default or habitual behavior).(b) Rule representation and implementation :There is another region of the dmPFC (same for vmPFC), which can detect con icts (several close possible solutions), predict outcomes, and compute errors of prediction; this region can inhibit the default behavior and explore to trigger another, more adapted concrete rule. This is the basis of cognitive control. When the contextual condition to select another concrete rule is too complex, an abstract rule must be created to represent it. This is done in lateral PFC, to pay more attention to some cues in some context, with vlPFC to de ne such complex hierarchical cues (goals and subgoals) and dlPFC to create a sequence of actions to get these cues (strategies or abstract rules). This lateral region of PFC in loop with associative region of striatum, can learn internal actions of WM monitoring (input and output gating); Abstract rules can be hierarchical, both in lPFC and mPFC. In lPFC, the lower level of rule corresponds to PrePMd (FEF in primates) for attention monitoring. In mPFC, this di erence of level is between PreSMA (con ict monitoring at the lower level) and dACC (error monitoring at higher levels)

Figure . :

 . Figure . : Frank and O'Reilly's Prefrontal Basal Ganglia and Working Memory (PBWM) model (a) Gating mechanism which switches between active maintenance and exible updating of working memory in incorporate task-relevant information (b) Neural Network model implementation of PBWM. Figure taken from [ ]

Figure . :

 . Figure . : Alexander and Brown's Prediction-Response Outcome (PRO) modelThe model learns predictions of multiple possible outcomes of various chosen actions using an error likelihood signal.Figure taken from [ ]

  Figure.: A The alternation task requires for an agent to alternate its choice between two di erent options (A & B). In the displayed setup, the environment is a classical T-maze where the agent starts from the bottom location in order to reach A or B. The rst choice is free and the reward is obtained after each alternation between A and B. After each trial, the agent restarts from the initial location.On this example, the trials can be written as ABBAB and only transitions AB and BA are rewarded. This task implies for the agent to remember its previous choice across trials. B The discrimination task requires for an agent to learn which cue (out of two) is associated with a reward. The agent has to choose between the two di erent options (A & B) depending on a cue that is presented at the entrance of the T-Maze. This task implies for the agent to remember the initial cue until it reached the corresponding target during a single trial. C One example for the discrimination task where the agent has learned to memorize the location at the entrance. D One example for the alternation task where the agent has learned to memorize the location after its choice has been made.

  Figure . : Performance of the agent in the (a) alternation task -a simple RL agents performs worse than chance as repeated visits to the same arm result in a negative reward (b) discrimination task (c) radial maze task. Each plot corresponds to the mean performance of individual agents.

Figure . :

 . Figure . : Percentage of strategy types learned in the Alternation and Discriminationtasks. The proportion of trials in the nal block in which the previous choice (alternation) or the cue (discrimination) was in memory at the choice point was calculated. A threshold of / . was used, for example, (a) the number of trials in which cue was in memory / number of trials with cue was greater than threshold, strategy was "remember cue " (similarly for cue ) and (b) number of trials is which left was in memory / number of trials when previous choice was left was greater than threshold, strategy was "remember left" (similarly for right); if proportions for both were above threshold, the strategy was "remember both"; and if both were below threshold, it was "remember other".

  Figure . : Examples of "remember one"(left) and "remember both"(right) strategies learned by the model in the discrimination task. (a) and (b) Probability over blocks of di erent possible memory contents (cue / , central arm or empty) at the choice point. Since on average, half of the trials begin with an observation of cue and the other with an observation of cue , the maximum proportion of trials that either of these can be in memory is around % (c) and (d) Probability over blocks of choosing to turn right or left as a function of di erent possible memory contents. Probabilities are derived from Qvalues at the end of each block. (e) and (f ) Probability over blocks of updating or maintaining memory contents conditional on ( ) observing the cue (at the start state) or ( ) having it already in memory (before the choice point). Probabilities are derived from Q-values at the end of each block

Figure

  Figure . : The agent learns task A (discrimination) for the rst episodes, then task B(alternation) for the next episodes, followed by a switch to task A and then back to task B. The presentation of the cues interferes with the learning of the tas k. A simple method of performance monitoring to identify a rule change can be maintaining a running mean of the rewards obtained over the last n episodes

  This work was done in collaboration with Christopher Stevens (on the experimental side) during his PhD in Neuroscience, in the Inserm Magendie research unit, on the Bordeaux NeuroCampus. The experimental results have been published in [ ] Cognitive Control over Multiple Memory and Learning Systems

  Figure . : (a) % correct tactile discrimination responses over repeated training sessions, displayed as rolling averages over sessions. R trained animals (n= ) represented in red, control animals (n= ) represented in blue. Controls were rewarded on all trials, regardless of surface chosen. Curves represent mean population score, dots represent individual performances. Figure taken from [ ] (b) Performance curves of the model with no bias, in the control (in blue) and tactile trained (in red) simulation conditions. (c) Performance curves of the model with default behavior bias, in the control (in blue) and tactile trained (in red) simulation conditions. All the performance curves are averaged over individual simulations of the agent. Each "session" is comprised of blocks of trials, and the mean of these trials gives the % correct responses on the y axis. Dots represent individual simulations.

Figure

  Figure . (b) shows the expression of surface alternation of the model with an added default behavior component, during learning. The model was trained on the tactile discrimination rule protocol. As in the results from the experiment ( gure . (a)), initially the agent displays a greater than chance expression of the alternation behavior, in both the control and tactile trained simulation conditions, which eventually drops below chance over extensive training.The results show that a model that is biased by a default behavior is able to better capture the behavioral results from the experimental study. Secondly, this model is able to capture the expression of alternation behavior observed in the initial training sessions.

  Figure . : (a) Initial surface alternation behavior averages across rst sessions in both experimental and control populations. Figure taken from [ ]. (b) Expression of the surface alternation behavior, during learning in the model biased with default behavior, in trained and control simulation conditions.

Figure . :

 . Figure . : Expression of the tactile rule when the bias provided by the default behavior is removed after episodes

  This work was done with Naomi Chaix-Eichel, another PhD student in the Mnemosyne team and is available as a preprint [ ] . From implicit learning to explicit representations

Figure . :

 . Figure . : Left: An expanded view of a T-Maze. An observer can infer an alternating behavior because of her partial view (white area) of the system. Right: -maze with cues. A cue (left or right) is given only when the bot is present inside the yellow area.

Figure

  Figure . : Model Architecture with sensor inputs, and a motor output (orientation).The black arrows are xed while the red arrows are plastic and are trained. The reservoir states are used as the input to a classi er which is trained to make a prediction about the decision (going left or right) of the robot. A left (L) and right (R) cue can be fed to the model depending on the experiment (see Methods).

Figure . :

 . Figure . : The trajectory of the robot following the -trace in the cartesian map.

Figure . :

 . Figure . : Prediction from sensors during time steps. Top gure shows the prediction of the KNN and SVM classi er, bottom gure shows the SVM prediction along the trajectory.

Figure . :

 . Figure . : Prediction from reservoir state during time steps. Top gure shows the predictions of the KNN and SVM classi er. Bottom gure shows the SVM prediction along the trajectory.

Figure . :

 . Figure . : The rst two principal components of the reservoir state space after applying PCA on the reservoir states. On the bottom right is the corresponding map of the positions of the robot in the maze.

  Figure . : The coordinates of the agent for timesteps in the prediction phase. The plots in blue show the x axis coordinates while the ones in red show the y axis coordinates. The gure on top shows the results for the standard sequence [ABABAB..], the gure at the bottom shows the results for a randomly generated sequence [AABAABBBABBAABAB], where 'A' is the left loop and 'B' is the right loop.

  Figure . : Model Schematics : Figure adapted from [ ]

  Figure . : Task Schematics : (a) AX continuous performance task (b) Hierarchical categorization task, Figures taken from []

  Figure . : (a) Percent correct trials for simulations of the HER model on the AX task. Values re ect the running average for a moving trial window. (b Performance of the HER model in relation to simulations of the backprop version, PBWM and LSTM models using the criteria of epochs without error.The RL version of the model, as reported here takes more time to converge than the version trained using backprop.

  Figure . : The memory weights X (averaged over simulations) in the gating mechanism re ect the value of maintaining versus updating WM representations following learning. At the lowest hierarchical layer, weights are con ned to the diagonal, indicating the mechanism tends to update WM on each trial, with weights corresponding to X and Y particularly strong indicating that these are the representations updated every time they are presented. In layer , the weights re ect the tendency to maintain representations of context variables A and B, rather than gating in X and Y representations, as seen by the positive o diagonal weights. Finally, weights in layer strongly favor the representation of and , regardless of what stimulus is presented.

  Figure . : (a) The model's performance on the structured task described. (b) As reported in the HER model, increasing the dimension of the stimulus leads to a signicant inability of the model to converge. Hence, the more complicated version of the structured task (Hierarchical Categorization Task ) does not meet the convergence criteria.

Figure . :

 . Figure . : Illustration of output gating : access to WM is controlled via the operation of an input gate that determines whether a stimulus is updated into WM. An output gate controls whether or not information within WM can in uence behavior. Figure adapted from [ ]

  Figure . : The modi ed model with output gating from layer . The gating weights in layer (X 1 ) learn over time to gate the context into r 1 . The selected prediction units from layer (p 1 ) are then used to make a decision on which value of the stimulus s is gated into r 0 (the output gate).

  Figure . : Task Schematics, as described in the section . .

  Figure . : Performance of the model with layers for the two paradigms (a, b), plotted as an average over runs, only for the runs that reached convergence criteria. The convergence criteria was de ned as having a performance greater than % in the last trials. (c) Performance for the model with layers on the rst learning paradigm.

  Figure . : Examples of how the model solves di erent cases of stimuli. The matrix shows the prediction values at di erent layers ( rst rows), given the internal representation of the stimulus, and how they are modulated additively (row ) to give the nal Action-Outcome predictions that are used for response selection. a, b show the case when the stimulus is black, in rules and respectively. d, e show the case when the stimulus is Green, Vowel (rule T ) and Red, Upper case (rule T ). c, f show the case for Green, Vowel and Red, Upper case in rule T

  Figure . : (a) Prediction weights (W 0 ) for the concrete rules at layer . These weights are pre-learned by training the model with rules T and T , independently. (b) Performance of the original model compared to the modi ed model over a runs, when layer is xed to the weights in gure (b) and only layer prediction weights (W 1 ) and gating weights (X 1 ) are learned. (c) (Left) Model performance on the rst trials, as compared to the performance reported by [ ] (Right)

  Mom, Dad -everything I do, I owe to you. Implicit vs Explicit Rules . . . . . . . . . . . . . . . . . . . Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . Behavioural evidence of Rules in animals . . . . . . . . . Basal Ganglia loops . . . . . . . . . . . . . . . . . . . . . Motor/Pre-motor regions . . . . . . . . . . . . . . . . . . . . . . Organization of the prefrontal cortex . . . . . . . . . . . . . . . . . . Medial PFC . . . . . . . . . . . . . . . . . . . . . . . . . . . Lateral PFC . . . . . . . . . . . . . . . . . . . . . . . . . . . Hierarchical organization . . . . . . . . . . . . . . . . . . Experimental Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . Working Memory paradigms . . . . . . . . . . . . . . . . . . Con ict paradigms . . . . . . . . . . . . . . . . . . . . . . . Response inhibition paradigms . . . . . . . . . . . . . . . . Task switching paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . From implicit learning to explicit representations . . . . . . . . . . . Methods and Task . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HER) Model . . . . . . . . Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results : Benchmark Performance . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . Integrative Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion and Perspectives . . . . . . . . . . . . . . . . . . . . .

		Contents
	C	R
	.	The Hierarchical Error Representation (
	C
	I
	I C P . . Working memory models II C C C Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M . M C C M M L S Cortico -C . Deciphering the contributions of episodic and working memo-
	vi viii

C . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Theories of behavior and learning . . . . . . . . . . . . . . . . . . . General loop of behavior . . . . . . . . . . . . . . . . . . . . Stimulus driven vs Goal directed Behavior . . . . . . . . . . Automatic vs Controlled Processing, or Habitual vs Contextual Behavior . . . . . . . . . . . . . . . . . . . . . . . Cognitive Control . . . . . . . . . . . . . . . . . . . . . . . . . . Mechanisms and structures involved . . . . . . . . . . . . . . . . . . Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . Uncertainty : Stochasticity vs Volatility . . . . . . . . . . . . Task Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . Working memory . . . . . . . . . . . . . . . . . . . . . . . Prediction of errors / Monitoring of errors . . . . . . . . . What are Rules? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Concrete vs Abstract Rules . . . . . . . . . . . . . . . . . . Abstraction of Rules . . . . . . . . . . . . . . . . . . . . . . vii Contents . . Models of monitoring . . . . . . . . . . . . . . . . . . . . . . . . . Models of top-down control . . . . . . . . . . . . . . . . . . . . . Modeling uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ries in increasingly complex decision tasks . . . . . . . . . . . . . . . Methods and Tasks . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . Cognitive Control over Default Behaviors . . . . . . . . . . . . . . . Methods and Task . . . . . . . . . . . . . . . . . . . . . . C B ix

Schematic gure illustrating the gradient of cognitive control

						Monitoring
			Goal	Context Goal	Context Goal	Context Goal	Prefrontal
					Plan	Plan
	Reflex (default	Stimulus	Stimulus	Stimulus	Stimulus	Stimulus
	mode), habits,
	model free,
	implicit	Response	Action Response	Action Response	Action Response	Action Response	Posterior
	Goal directed,
	model based,
	explicit
			Outcome	Outcome	Outcome	Outcome
		Classical	Operant	Flexible	Planned	Cognitive
		conditioning	conditioning	behavior	Behavior	Control
		(rodents, primates)	(rodents, primates)	(rodents, primates)	(rodents, primates)	(primates)
	Figure . : Starting from Stimulus-Response associations acquired through pavlovian
		conditioning which can be seen as unchanging behaviors (habits or default be-
		haviors) that need to be overcome by exerting control. Here, a distinction is
		made between Responses (external actions which are triggered by the external
		environment), and Actions (voluntary responses that can be internally or ex-
		ternally directed). Next, are Action-Outcome associations learned through in-
		strumental learning paradigms, which are sensitive to reward outcomes, gener-
		ally controlled by certain goals that are intrinsically rewarding to the organism.

  [ ] [ ]. A related view is that the vmPFC signals values in a space of goods, whereas the dmPFC encodes values in a space of actions. While this view is supported by empirical evidence in a number of studies [ ] , other researchers have found e ort cost representations in vmPFC activity [ ] [ ] and reward value in dmPFC activity [ ].

  [ ].Lateral PFC is connected to wide range of secondary sensory regions like Frontal Eye Fields (FEF), secondary visual cortex, parietal cortex, supplementary motor cortex and pre-motor cortex. When the selection is not trivial and requires memory, context, and abstract rules combining them, lPFC complements additionally the other PFC systems and the downstream selection mechanisms, and hence seems to be a critical area for the representation of rules [ ] . Patients with lateral PFC damage report correctly on what the appropriate task rule is, even while being unable to implement it correctly [ ].

Table . :

 . Parameter con guration for the ESN

	Parameter	Without context With context
	Input size
	Output size
	Number of units
	Input connectivity	.	.
	Reservoir connectivity .	.
	Reservoir noise	e-	e-
	Input scaling		(sensors), .	(cues)
	Spectral Radius	.	.
	Leak Rate	.	.
	Regularization	. e-	e-

Table . :

 . N RM SE and R 2 score of the ESN with inputs Cognitive Control over Multiple Memory and Learning Systems

	N RM SE			R 2
	Mean	.		Mean	.
	Variance .	e-	Variance .	e-

Table . :

 . N RM SE and R 2 score of the ESN with the two additional contextual inputs ample can be seen in gure . : the top graph shows the positions of the bot while making the standard sequence [ABABABAB...], the bottom one shows that the bot was able to achieve a more complex sequence [AABBAABBAABB...].

  ). Feedback to the model indicated correct or incorrect performance. One hundred simulations of the HER model performing the task

	were conducted for	outer loop sequences (equivalent to	training epochs
	in O'Reilly & Frank,	, where epoch was outer loops, or approximately
	,	individual cue presentations). The model was considered to have success-
	fully learned the task on the rst of	consecutive cue presentations (approx-
	imately seven epochs) in which no response errors were made. The more lenient
	criterion as described in O'Reilly & Frank,	was two consecutive epochs with
	no errors (or approximately	cue presentations).

D

  In the predictive coding framework, top-down processes provide predictions from superior hierarchical levels to inferior levels, while residual prediction errors, ie, input that cannot be accounted for by the predictions supplied by top-down processes, are carried from inferior levels to superior levels. At a single level, the HER model suggests that error signals computed in the mPFC can be used to train representations of the error signal in dlPFC. Error representations learned by dlPFC are associated with task stimuli that reliably precede error prediction signals generated by mPFC such that, on subsequent stimulus presentations, error representations maintained in dlPFC may be deployed to reduce prediction errors in mPFC. Previous functional imaging results have suggested a correspondence between medial and lateral PFC hierarchies, arguing that medial PFC provides motivational signals, while lateral PFC provides selection, that is cognitive signals. The HER model di ers from this proposal in that rather than motivational signals, the mPFC provides prediction error signals that train the lateral PFC regarding what information must be maintained for successful task performance. The representation scheme proposed by the HER model suggests that individual neurons in lPFC should code for components of a distributed error representation, with single units signaling the identity and likelihood of observing a particular error. The working memory activations constitute a de facto representation of inferred states and thus provide a context dependent pattern of activity that minimises prediction error.

Context

Subjects are presented with a sequence of items -a number, a symbol, and a letter. The numbers represent the context, based on which the relevant items to attend to are either the symbols, or the letters. CF trials are those in which the context appears rst, and CL trials are those in which the context appears last. (b) Schematic gure explaining how our hierarchical model with selective attention would solve the context last task. Without an output gate to select from within the WM items, the model would be unable to select between the two maintained items in WM. The circles represent the BG input gates, with red signifying a closed gate, and green an open gate. The grids represent the stimulus-action contingencies learned in the lower layer. The task of the higher layers then is to select the appropriate mapping, based on the context.