Hugo Chateau-Laurent
email: hugo.chateau-laurent@inria.fr

Frédéric Alexandre
email: frederic.alexandre@inria.fr

Augmenting Machine Learning with Flexible Episodic Memory

Keywords: Explicit Memory, Episodic Memory, Prospective Memory, Hippocampus, Prefrontal Cortex

A major cognitive function is often overlooked in artificial intelligence research: episodic memory. In this paper, we relate episodic memory to the more general need for explicit memory in intelligent processing. We describe its main mechanisms and its involvement in a variety of functions, ranging from concept learning to planning. We set the basis for a computational cognitive neuroscience approach that could result in improved machine learning models. More precisely, we argue that episodic memory mechanisms are crucial for contextual decision making, generalization through consolidation and prospective memory.

INTRODUCTION

Despite recent progress in machine learning, the field of artificial intelligence has not yet given rise to generally intelligent and autonomous agents endowed with autobiographical memory and capable of defining their own goals and planning their way to reach them. As a result, machine learning models are mainly confined to domain-specific processing. Complementary and interacting memories have been argued to be a key component of autonomous learning [START_REF] Alexandre | Beyond machine learning: Autonomous learning[END_REF]. We claim that current techniques in artificial intelligence mirror implicit memory, while topics related to building explicit memory systems remain largely unaddressed. The view we depict here is mostly consistent with the one proposed in [START_REF] Botvinick | Reinforcement learning, fast and slow[END_REF] in that they both propose to draw inspiration on both the prefrontal cortex and the hippocampus to develop more powerful algorithms. Our work further expands this proposal by considering additional mechanisms inspired from anatomical and electrophysiological studies.

We first argue that models need complementary memories, then set the focus on the organization of episodic memory in biological organisms. Finally, we propose a roadmap to implement these mechanisms in artificial agents.

THE NEED FOR COMPLEMENTARY MEMORIES

Consider a task where you have different colored shapes displayed on a table and you can press one among a set of buttons to get rewards. There are hidden rules controlling reward delivery that you can discover by trial and error. An example rule would be that if a red square is on the left, pressing the left button is rewarded. This kind of task can be learned very easily and efficiently with classical machine learning techniques like reinforcement learning (RL). Similarly, the subtask of object identification is prototypical of gradient-based forward layered architectures. Through learning, these models build what is called in cognitive science an implicit memory. This corresponds to a slow and procedural learning of the underlying regularities. If information (like position or color of the shapes) is encoded in a topographical way, the model builds an overlapping representation, which enables it to generalize. In RL, this learning can be done with a model-free (MF) approach, meaning that the values of configurations are learned without explicit knowledge of the task structure (hence the term implicit memory). These computational approaches have been paralleled with known principles in neuroscience. For instance, layered architectures of deep networks have been compared to the hierarchical structure of the sen-sory posterior cortex. RL models have been very beneficial to better understand learning principles of the most ancient regions of the frontal cortex (i.e. the part of the cortex that is anterior to the central sulcus) that are also called agranular cortex, corresponding to the motor, premotor and lateral orbitofrontal cortices [START_REF] Domenech | Executive control and decision-making in the prefrontal cortex[END_REF]. In association with the sensory cortex, these regions are involved in decision making via action selection from perceptual cues (associated to actions in motor and premotor cortex) and reward values (learnt in the orbitofrontal cortex). Loops associating these cortical regions with the basal ganglia and modulated by dopaminergic projections carrying reward prediction errors have been carefully studied, and corresponding models have been compared to MF-RL [START_REF] Joel | Actor-critic models of the basal ganglia: new anatomical and computational perspectives[END_REF]. It is also reported in cognitive neuroscience that, after a long period of repetition, this reward-based behavior is transformed into a purely sensorimotor behavior (also called habitual) simply associating the sensory and motor cortices, with no sensitivity to reward change [START_REF] Boraud | A natural history of skills[END_REF]: after a long practice, you automatically press the left button when you see a red square on the left without even thinking of an anticipated reward.

The need for episodic memory

The functions of these cortical regions are consequently tightly associated with implicit learning, as performed by the most famous and efficient models in artificial intelligence today, like deep networks and MF-RL agents. Nevertheless, these models suffer from a series of well-identified weaknesses. Layered architectures with overlapping representations are not good for learning data without structure or regularities. They learn very slowly, from global aspects to details and are sensitive to what is called catastrophic forgetting: if you first learn a relation (e.g. pressing the left button when seeing the red square on the left), then, when learning another one (e.g. pressing the left button when seeing the blue circle on the left), the previous association will be forgotten in the absence of regular recall.

This problem has been clearly identified and addressed in the framework of the complementary learning systems (CLS) [START_REF] Mcclelland | Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory[END_REF], introducing the role of the hippocampus (HPC). This neuronal structure has been originally studied in navigation tasks, demonstrating major learning capabilities and exhibiting place cells that respond to specific places in the environment [START_REF] Stachenfeld | The hippocampus as a predictive map[END_REF]. Later on, the HPC has been associated with more complex and general functions. It is thought to receive a summary sketch of the current cortical ac-tivation and perform functions like associative memory and recall. Thanks to its very sparse coding and an advanced function of pattern separation [START_REF] Kassab | Pattern separation in the hippocampus: distinct circuits under different conditions[END_REF], the HPC is able to learn very quickly, possibly in one shot, an arbitrary binding of any distributed cortical representation with a minimized risk of interference. Consequently, it cannot generalize well. However, it also performs pattern completion, namely the ability to rebuild a complete pattern from partial cues, and replay it in (i.e. send it back to) the cortex. Implemented in attractor networks, this function is useful for doing reinforcement learning in the real world with high-dimensional states and noisy observations [START_REF] Hamid | Reinforcement learning and attractor neural network models of associative learning[END_REF]. It can consequently complement the cortex by learning rare patterns very quickly and binding a distributed cortical representation that would be hard to associate with a motor response at the cortical level. At each moment, when the HPC receives a cortical pattern, it has to decide whether it is novel and must be learned, or if it is reminiscent of a previously stored activation and must be completed (recalled) and sent back to the cortex. This is done by a clever mechanism described in the next section. It must also be stressed that what is manipulated by the HPC (originating from current cortical activation) does not represent a single event but rather, thanks to several mechanisms of temporal coding of the cortex evoked below, a full behavioral episode that can last several seconds. This faculty of storing and recalling specific episodes is termed episodic memory. It is a kind of explicit memory because information can here be explicitly questioned. In humans, this is often associated with a conscious declarative process.

We have now discussed principles of these complementary learning systems, with the cortex performing a slow statistical learning using distributed overlapping representations, and the HPC performing rapid arbitrary binding of cortical activation to store new episodes in a sparse associative memory and recall them later on. Interestingly, the capacity of the HPC to send back previously stored episodes to the cortex ( Ólafsdóttir et al., 2018) is very useful because it can solve the problem of catastrophic forgetting. Thanks to the replay of virtual (i.e. not actually experienced) episodes, the HPC indeed has the capacity of gradually inserting examples of a new rule and interleaving them with examples of the previously stored rules not to forget them. This process of replay is also very useful for another learning mechanism called consolidation. We have explained above that it is difficult for the cortex to manipulate and for example associate by learning a concept which is distributed in distant regions of its surface. Conversely, it is not possible for combinatorial reasons, to represent locally in associative regions of the cortex, any combination of elementary patterns that might constitute the premises of a behavioral rule. By this replay mechanism, if it appears that the HPC has to frequently send back an arbitrary binding of pattern to answer non tractable situations, the cortex will be able to slowly learn this combination of activities replayed by the HPC on its surface and to link units in an associative area to this distributed activation, thus forming locally a new concept to be employed in future behaviors, without the help of the HPC. Under this view, it can be said that the HPC is a supervisor of cortical learning. If the stored episodes represent a very rare case (or a particular case or an exception), it can continue and be treated by the HPC.

The need for cognitive control

Another series of weaknesses associated with implicit learning in the cortex is linked to the fact that, sometimes, and particularly in a complex and dynamic world, it is not sufficient for the behavior to be only reactive, stimulus driven, but it also need to be goal driven, guided by internal states like motivations, intentions or to obey external instructions providing a new rule. Consider cases where the rule associating the red square on the left with the left button changes for the right button and comes back to the left button (so-called reversal learning): a reactive stimulus-driven behavior would slowly unlearn the previous rule and learn the new one and unlearn it again, whereas it is rather expected to have a more flexible behavior including an internal deliberation to switch rapidly from one rule to the other. You can also consider cases (like in the Wisconsin Card Sorting Test [START_REF] Bock | The Wisconsin Card Sorting Test: Theoretical analysis and modeling in a neuronal network[END_REF]) where several rules are proposed but at a given moment only one is valid or cases where the new rule is given by instruction from a fellow creature. Here also you should give up immediately the previously learned rule and adopt a new one provided by internal process. You generally wait for errors (difference between anticipated and actual reward) to understand that rule has changed and that you have to find a new one. Sometimes, you can also discover that changes are associated with a contextual cue. In this case, you can learn contextual rules and change your behavior before making errors [START_REF] Koechlin | An evolutionary computational theory of prefrontal executive function in decisionmaking[END_REF]. Consider now cases where the rule becomes more complex and relies on non visible cues (for example press the left button if you see a red square on the left and if on the previous trial the red square was on the right). Here also a reactive behavior guided by observed cues is not sufficient and should be completed by the possibility to decide from internal cues corresponding here to the memory of recently observed cues. Other internal cues to consider for biasing the decision could be also related to emotion or motivation associated to the present situation. All these cases share common principles [START_REF] Miller | An integrative theory of prefrontal cortex function[END_REF]. We are here in cases where the default behavior learned by a slow stimulus-driven process must be inhibited and replaced with a new behavior adapted to the present context. To ensure flexibility and reversibility, this process must be explicit and manipulate knowledge about the world, as in Model Based approaches in RL. Not only based on the perception of external stimuli, this system must rely on a internal source of activity to bias the default behavior suggested by stimuli and impose the new behavior in a top-down way. At the same time, it must be flexible but also robust to distractions from external stimuli.

It has been demonstrated by neuroscience that all these properties are ensured by the prefrontal cortex (PFC), a more recent (also called granular) region of the frontal cortex, particularly developed in primates [START_REF] Fuster | The prefontal cortex. Anatomy, physiology and neurophysiology of the frontal lobe[END_REF]. This structure, organized according to the kind of task [START_REF] Domenech | Executive control and decision-making in the prefrontal cortex[END_REF] implements contextual rules which can ensure a topdown control of behavior. It has a modulatory role and does not replace reactive rules slowly acquired by implicit learning. Instead, it is able to inhibit the default (and not adapted here) behavior and trigger an attentional process to favor task-relevant stimuli and consequently elicit a previously learnt rule more adapted to the present context [START_REF] O'reilly | Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control[END_REF]. This modulatory role implies that, in case of a lesion, the behavior is not stopped but comes back to the (non appropriate) default case, yielding classical deficits of PFC damage like perseveration. All these mechanisms are implemented by the unique mechanism of working memory, where neuronal populations of the PFC exhibit a sustained activity to represent, maintain and impose this contextual activity. In addition to its role in learning, the role of the dopamine has been shown central to regulate the balance between maintenance and updating of the sustained activity [START_REF] Braver | On the control of control: The role of dopamine in regulating prefrontal function and working[END_REF].

In contrast to the classical weight-based learning (as presented above with the HPC), what we describe here is an activity based control but if it is frequently exploited in certain context, it can become a default behavior by slow implicit learning.

The need for prospective memory

It is now time to describe what could be called a side effect of the interactions between these two kinds of explicit memories, considering that the HPC and the PFC are interconnected [START_REF] Simons | Prefrontal and medial temporal lobe interactions in long-term memory[END_REF] and apply their respective functions one on the other. In short, the HPC can replay episodes of cortical activation, including prefrontal activation; reciprocally, the PFC is informed about contextual aspects of the situation by the HPC and can control the generation of adapted episodes and their storage. These episodes can be manipulated explicitly by the PFC, the same way it creates new rules adapted to the situation. Altogether, this gives rise to prospective memory [START_REF] Schacter | Remembering the past to imagine the future: the prospective brain[END_REF], also called memory of the future [START_REF] Fuster | The prefontal cortex. Anatomy, physiology and neurophysiology of the frontal lobe[END_REF], the unique capacity of imagination, of explicitly anticipating the future from an internal model of the world and of making a decision informed by the potential outcomes of possible future scenarios and deliberating on their respective interest. This is among the most powerful capabilities of human cognition, also associated to planning and reasoning, clearly going beyond reactive automatic behavior and enabled by the combination of the two kinds of explicit memory evoked above, episodic and working memory. An alternative computational account of how these memories interact has been proposed by [START_REF] Zilli | Modeling the role of working memory and episodic memory in behavioral tasks[END_REF]). Yet, this framework is rather elementary and its limitations have been highlighted by [START_REF] Dagar | Deciphering the contributions of episodic and working memories in increasingly complex decision tasks[END_REF], who argued that more complex mechanisms are needed in realistic cases, in the same vein as what we propose here.

Models for implicit learning are today well mastered and employed in artificial intelligence. Models for cognitive control by the PFC have been proposed for some time in computational neuroscience [START_REF] O'reilly | Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control[END_REF] and their principles begin to be transferred to machine learning [START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF]. Episodic learning has also been considered recently for machine learning [START_REF] Ritter | Episodic Control as Meta-Reinforcement Learning[END_REF] but for the moment, the corresponding models propose a rather superficial view of this cognitive function and of its multiple roles in cognitive control, consolidation and prospective memory, as we have evoked here. In the forthcoming section, we explain that integrating information about episodic learning from different sources in neuroscience can help understand and give a more precise computational account of this cognitive function in order to define an integrated explicit learning involving both the HPC and the PFC.

ORGANIZATION OF EPISODIC MEMORY

H ip p oc a m p us

Memory storage in the HPC

As mentioned above, the HPC is able to store and replay sequences of experienced states, as well as construct novel ones. These sequences are embedded in two coupled oscillatory regimes: a theta rhythm (4-10 Hz) and faster gamma rhythm (20-80 Hz). According to their respective frequency, approximately seven gamma cycles are nested in each theta cycle. The entorhinal cortex (EC) constitutes the main input to the HPC by its superficial layers that we call ECs. They are thought to provide one state per theta cycle [START_REF] Jensen | Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer[END_REF]. The HPC then either engages its gamma regime in a retrospective code supporting short-term memory and episodic storage, or in a prospective code with which it tries to predict future states [START_REF] Lisman | Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the socratic model to account for normal and aberrant effects of dopamine[END_REF]. In both cases, sequences are represented in a compressed way, with one state represented per gamma cycle. In the retrospective (or learning) code, the sequence of gamma states reflects the order at which past states have just been experienced, with the current state being represented at the end of the theta cycle. The main computational reason behind having such a buffer is that the gamma timescale is more appropriate at a biophysical level (relating to constants of time of molecular processes) for learning state transitions than the behavioral timescale at which events naturally occur [START_REF] Lisman | Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the socratic model to account for normal and aberrant effects of dopamine[END_REF], especially given the fact that the most anterior parts of the HPC learn transitions between temporally distant events as we will discuss below. In the prospective (or recall) code, the item provided by the cortex is followed by one item per gamma cycle. The order and identity of these subsequent items reflect a prediction of upcoming states, potentially based on previous learning through retrospective coding. Alternation between these two mechanisms is presumably based on the ability (or inability) of the prospective code to accurately predict future states. Under this view, the HPC begins by considering the current episode as already known and attempts to predict the next states until errors indicate that the current episode is too different from stored situations and must consequently be considered novel and learned accordingly. In fact, a crucial part of the HPC called CA1 is often linked to novelty detection, namely computation of the mismatch between predictions and incoming sensory information [START_REF] Duncan | Evidence for area ca1 as a match/mismatch detector: A high-resolution fmri study of the human hippocampus[END_REF]. When unpredicted sequences of stimuli are presented to the HPC, a dopaminergic signal is used to switch from the prospective code to the learning retrospective code in order to update the internal model [START_REF] Lisman | Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the socratic model to account for normal and aberrant effects of dopamine[END_REF].

Retrospective and prospective codes are expanded with mechanisms of pattern separation and pattern completion [START_REF] Kassab | Pattern separation in the hippocampus: distinct circuits under different conditions[END_REF]. The HPC indeed tries to complete each gamma state (whether being actual or predicted) by relating it to states it has experienced in the past. This is the task of a recurrent region called CA3. Conversely, if the state does not match any stored pattern, its representation is modified in such a way that it becomes as dissimilar (orthogonal) as possible to any other pattern, to avoid interference in this very fast learning process. This is performed in the largest region of the HPC, the Dentate Gyrus (DG), thus encoding what makes each state unique, with a very sparse coding. In sum, the HPC performs a succession of auto-associations (encoding states) and hetero-associations (encoding their sequence, whether being retrospective or prospective) [START_REF] Lisman | Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the socratic model to account for normal and aberrant effects of dopamine[END_REF].

The mechanisms just described are thought to underlie the learning of individual experienced sequences of states (i.e. episodes). They mainly involve the trisynaptic pathway : ECs -→ DG -→ CA3 -→ CA1. CA1 also receives information from a more direct monosynaptic pathway: ECs -→ CA1 -→ ECd. While fast learning enabling the trisynaptic pathway to learn to link items represented in successive gamma cycles is obviously useful for one-shot sequence learning, slower learning deployed in the monosynaptic path-way may serve additional roles. We have seen that what is encoded in DG and CA3 is not the full description of a state but rather what makes it different from the others, through orthogonalization processes. When the HPC sends information back to the cortex for reasons we will detail in the next part, this particular encoding cannot be used as is. In fact, the role of CA1 is to act as a translator between the sparse representation of the HPC and the dense overlapping representation used in the cortex [START_REF] Schapiro | Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning[END_REF]. Learning mechanisms in the monosynaptic pathway ensure CA1 is tuned in such a way that what is reconstructed in deep layers of EC (ECd) corresponds to the original information sent by ECs. This tuning can be done at the moment of learning, when both the original information and its reconstruction are present. At the moment of prediction, the same comparison mechanism serves as a novelty proxy for switching between recall and learning regimes, like previously mentioned. A side effect of this slower learning mechanism is the involvement of CA1 in gathering statistics over multiple episodes [START_REF] Schapiro | Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning[END_REF], thus explaining why the activity of some hippocampal units like place cells reflect statistical regularities [START_REF] Stachenfeld | The hippocampus as a predictive map[END_REF]. Computer simulations reveal the importance of such a mechanism for discovering associations not directly experienced in for example transitive inference, namely the discovery that if A is associated to B and B is associated to C, A is indirectly associated to C [START_REF] Schapiro | Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning[END_REF]. The role of episodic aggregation has also been described for building a kernel function in the context of episodic reinforcement learning [START_REF] Gershman | Reinforcement learning and episodic memory in humans and animals: an integrative framework[END_REF]. Finally, these statistics might serve as an intermediate step for the transfer between episodic and cortical memory systems [START_REF] Kumaran | What learning systems do intelligent agents need? complementary learning systems theory updated[END_REF]. We explain this in more detail in the next part.

Interfacing between the HPC and the cortex

A constant dialog between the HPC and the cortex is ensured by multiple pathways, including a major hub: the entorhinal cortex. This highly integrated cortical region receives inputs from most regions of the entire cortex in its superficial layers and buffers them in episodes to be sent to the HPC. Reciprocally, its deep layers receive replays from the HPC and distribute them back to the cortex. Other minor paths for the specific communication with the PFC will also be evoked below. While the precise description of these pathways is outside our scope, we would like to highlight a few key characteristics before addressing the computational role of this dialog.

Connectivity with the cortex is not homogeneous throughout the HPC. In humans, the HPC runs along an anterior-posterior longitudinal axis. Along this axis, genetic, electrophysiological, and anatomical differences underlie differences in functionality, and in particular, decreasing degrees of spatio-temporal abstraction (see [START_REF] Strange | Functional organization of the hippocampal longitudinal axis[END_REF] for a review). In the spatial domain, the gradient of abstraction translates into place cells being sensitive to larger portions of the environment in the anterior HPC compared to place cells located in the posterior HPC [START_REF] Jung | Comparison of spatial firing characteristics of units in dorsal and ventral hippocampus of the rat[END_REF]. More generally, it is hypothesized that on the one hand, more posterior parts of the HPC which are mostly connected to posterior (and sensory) parts of the cortex are concerned with specifics (details of a scene, precise location,...). On the other hand, more anterior parts which are mostly interacting with anterior parts of the cortex (dealing with cognitive and behavioural control but also emotional and motivational aspects) represent more abstract and general information (context, larger scale location, semantic information, mood...). Intermediate portions are thought to mediate interactions between the two ends of the longitudinal axis using intermediate representations [START_REF] Strange | Functional organization of the hippocampal longitudinal axis[END_REF]. Organisation in trisynaptic and monosynaptic pathways is preserved all along, although the dentate gyrus is more prominent in posterior HPC while CA1-3 are more prominent in its anterior part [START_REF] Malykhin | In vivo quantification of hippocampal subfields using 4.7 t fast spin echo imaging[END_REF]. Hence, the same mechanisms for learning detailed perceptual state transitions in the posterior HPC are used for learning transitions between more spatially and temporally distal cognitive cues in the anterior HPC. Interestingly, this also constitutes a potential substrate for transitive inference, since items that are not associated at the posterior scale of functioning can potentially be associated at the anterior scale of temporal abstraction [START_REF] Strange | Functional organization of the hippocampal longitudinal axis[END_REF].

Let us now discuss candidate functions of hippocampo-cortical dialog, beginning with topics involving communication through the entorhinal cortex. Firstly, hippocampal pattern completion can be taken advantage of for the recall of missing information in the cortex [START_REF] Rolls | The mechanisms for pattern completion and pattern separation in the hippocampus[END_REF]. It is obviously helpful in the noisy domain of perception, in which objects must be identified while exhibiting a constantly changing aspect, for example dependent on light conditions in the case of vision. As mentioned before, similar mechanisms are used with more abstract representations in intermediate and anterior regions of the HPC. Consequently, pattern completion might also be useful for retrieving more abstract information related to context, emotional states, and even behavioral plans, a requirement for prospective memory. Secondly, a fur-ther function of the dialog is to use knowledge about state transitions stored in the HPC. This has been shown to be useful for predictive coding in the visual cortex [START_REF] Hindy | Linking pattern completion in the hippocampus to predictive coding in visual cortex[END_REF] and cognitive anticipation [START_REF] Gershman | Reinforcement learning and episodic memory in humans and animals: an integrative framework[END_REF]. Thirdly, the complementary learning systems theory suggests that the HPC acts as a temporary store for memories to be transferred to the cortex [START_REF] Mcclelland | Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory[END_REF]. More precisely, dense representation and slow learning are deployed for the cortex to learn high-level generalizations while avoiding catastrophic forgetting respectively, resembling implicit memory of classical deep learning models. The other side of the coin is that the cortex is not able to learn from a single exposure to a piece of information. Conversely, the HPC deploys fast learning and sparse representation for enabling one-shot acquisition. Since sparse encoding is not suitable for generalization, the HPC alone is not sufficient either. In fact, it is really the conjunction and interaction of these two systems that enable humans to learn fast and exploit acquired knowledge efficiently. During resting periods, the HPC is thought to "teach" the cortex by sending recently acquired information interleaved with more established knowledge. This process is called interleaved learning and is used to incorporate new information into cortical knowledge without interference [START_REF] Mcclelland | Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory[END_REF]. Note that the aforementioned hypothesis regarding the complementarity of the monosynaptic and trisynaptic pathways was introduced as an analogous and intermediate step to this consolidation process [START_REF] Schapiro | Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning[END_REF]. Indeed, intermediate statistics discovered in the monosynaptic pathway might preferentially be sent to the cortex. There would therefore be a trisynaptic-monosynaptic-cortical gradient of learning abstraction, in the perpendicular direction of the anterior-posterior axis of representational abstraction. To our knowledge, understanding the differential contributions of these two axes to functions like transitive inference remains an open and unaddressed question. As a concluding remark, the fact that the monosynaptic pathway might play a more dominant role in the anterior HPC and the trisynaptic pathway a more dominant role in the posterior one suggests that these two axes might not be completely orthogonal.

Concerning specific paths of communication between the PFC and the HPC, several principles must be mentioned. In short, a unidirectional loop of communication has been described [START_REF] Eichenbaum | Memory: Organization and Control[END_REF]. The anterior HPC is hypothesized to send contextual information about the retrieved episode to the PFC, which uses it to better select appropriate behavioral rules. Reciprocally, the PFC controls episodic recall and encoding, either directly by modulating ac-tivity in the posterior HPC [START_REF] Rajasethupathy | Projections from neocortex mediate top-down control of memory retrieval[END_REF] or indirectly by biasing the activity of the sensory and entorhinal cortices. As a result, hippocampal recall and its prospective code are oriented towards task-relevant episodes, whereas encoding and its retrospective code are modulated to control memory organization. The PFC would therefore need a model of what is stored in the HPC, a function that is sometimes referred to as metamemory. The direction of these information flows are presumably controlled by an additional communication pathway linking the PFC and the HPC through the thalamus, the mechanisms of which are still to be understood in more details. To better understand the crucial role of these recently discovered and yet to model interactions, let us consider the example task of deciding how to dress. While learning to wear warm clothes when perceiving rain and lighter clothes when perceiving sunlight is sufficient in some situations, elements of controlled episodic memory can help us making a better decision. The HPC might be able to recall and send information to the PFC concerning a weather forecast inconsistent with your current perception. The PFC could then predict that the weather is likely to change in the near future. It might in turn inhibit the automatic recall of an alternative but fictional weather forecast you saw in a movie. This situation illustrates how interactions between the HPC and the PFC harness memory of the past to make decisions in the present by predicting the future.

ROADMAP FOR THE FUTURE

In this paper, we have shown that important limitations in today's machine learning are due to a lack of explicit memory and we have reported that, in the brain, the HPC and the PFC play a fundamental role in creating and managing this memory. We have also explained that this form of memory plays a central role in concept learning, planning, reasoning and other major cognitive functions. Although this global view begins to be understood and exploited in machine learning, the specific role of episodic memory and its substrate in these functions is still not well understood and corresponding working models could be augmented with additional mechanisms.

We attempted to report results and hypotheses coming from different domains of neuroscience in order to provide an increasingly precise view of hippocampal circuitry and its relationship with posterior and prefrontal cortices. On the one hand, we think that the role of the HPC in supervising the sensory cortex to allow for consolidation without interference could be investigated in more details and give rise to better balanced neural networks capable of forming new conceptual knowledge online. On the other hand, even if several pieces of information are still missing, preliminary studies about the involvement of interactions between the HPC and the PFC in forming a prospective memory should also be considered for improving decision making algorithms. Finally, we believe that explicit cognition is a key ingredient for creating artificial agents capable of deliberating and explaining their actions.

Figure 1 :

 1 Figure 1: Connectivity within the hippocampus and between the hippocampus and the frontal, posterior and entorhinal cortices. Connections of the trisynaptic pathway are shown in blue while connections of the monosynaptic pathway are shown in green.


Hugo Chateau
email: hugo.chateau-laurent@inria.fr

Laurent Mnemosyne

Alexandre Frédéric
email: frederic.alexandre@inria.fr

Mnemosyne

Towards a Computational Cognitive Neuroscience Model of Creativity

Keywords: creativity, explicit memory, episodic memory, prospective memory, cognitive control, imagination, computational cognitive neuroscience, artificial intelligence

Recent progress in AI has expanded the boundaries of the cognitive functions that can be simulated, but creativity remains a challenge. Neuroscience sheds light on its mechanisms and its tight relationship with episodic memory and cognitive control, while machine learning provides preliminary models of these mechanisms. We present these lines of research and explain how they can be exploited in the domain of computational creativity in order to further expand the capabilities of AI.

I. INTRODUCTION

Creativity is a fundamental feature of human information processing but a historically challenging task for a machine. It is generally agreed that it includes two defining characteristics, namely coming up with a novel idea and verifying that this idea is appropriate for the task at play [START_REF] Dietrich | The cognitive neuroscience of creativity[END_REF]. They are respectively termed divergent and convergent thinking [START_REF] Jung | The structure of creative cognition in the human brain[END_REF].

Despite tremendous progress made with machine learning, artificial intelligence models suffer from a series of weaknesses including but not limited to overfitting and poor transferability [START_REF] Marcus | Deep learning: A critical appraisal[END_REF]. Thinking through analogy and imagination would be key ingredients for solving these problems. Unfortunately, such creative functions seem to remain a challenge for the machine. Yet, it is declared in [START_REF] Boden | Computer Models of Creativity[END_REF] that "creativity isn't magical. It's an aspect of normal human intelligence". In accordance with M. Boden, we argue that creativity results from the functioning of the human brain and should consequently be better understood by studying its physical substrate. As a consequence, we propose to relate computational creativity (CC) to computational cognitive neuroscience (CCN). It has already been suggested in [START_REF]Creativity explained by computational cognitive neuroscience[END_REF] that current modeling work in CCN carried out to remedy current limitations of artificial intelligence could be directly beneficial to CC by unravelling the brain mechanisms responsible for divergent and convergent thinking. Here we propose to extend this view by providing more details on recent experiments in neuroscience and recent modeling efforts in that direction. In that perspective, we briefly sketch the mnemonic architecture of the human brain, going in more details on mechanisms reported to be involved in imagination and cognitive control, participating in creativity.

Then we report recent modeling efforts to implement these mechanisms and discuss how they might be adapted to CC.

II. THE HUMAN MNEMONIC ARCHITECTURE

A. Implicit memory

In cognitive neuroscience, a major distinction is made between implicit and explicit memory, also called nondeclarative and declarative memory respectively [START_REF] Squire | Declarative and nondeclarative memory: multiple brain systems supporting learning and memory[END_REF]. The former corresponds to the slow learning of regularities present in the world. This learning is well captured in current machine learning models, as they rely on statistical methods. This is for example the case with the layered architecture of deep networks [START_REF] Lecun | Deep learning[END_REF] that has demonstrated high performance in pattern identification and is sometimes compared to the architecture of the sensory cortex. It is also the case with the most classical approaches of reinforcement learning, the actorcritic architectures [START_REF] Joel | Actor-critic models of the basal ganglia: new anatomical and computational perspectives[END_REF], which are today among the best models for action selection and decision making. These approaches are paralleled with the loops between the basal ganglia and the most ancient parts of the frontal cortex, namely the motor and premotor cortex (standing for the actor) and the orbitofrontal cortex (standing for the critic). These models perform implicit learning because they simply learn from a statistical analysis of trials and errors, regularities in labelled patterns, rewarding value of states or sensorimotor associations, with no explicit knowledge of the world structure [START_REF] Marcus | Deep learning: A critical appraisal[END_REF]. After learning, these values cached in weights are exploited implicitly to respond to new cases. As it can be seen with the recent success of machine learning, these approaches are very successful when large training corpora are available, but they suffer from several weaknesses, precisely because their learning is implicit. Two sets of weaknesses have been studied in the context of two different mechanisms that might remedy them, both corresponding to building an explicit memory: episodic learning and cognitive control.

B. Explicit memory: episodic learning

A first set of weaknesses is observed in deep networks, where learning takes place in a layered architecture with distributed, integrated and overlapping representations. This is the reason why these models can generalize, at the price of a very slow learning. Conversely, they are not good for learning data without structure nor regularities. They learn slowly, from global aspects to details, and they forget the individual cases they have received. They are also sensitive to catastrophic forgetting: if they first learn to identify a certain pattern and then learn to identify another one, they will forget the first kind of identification if they don't have a regular recall of the previously learnt association, thus adding to the slowness of learning.

It is explained in [START_REF] Mcclelland | Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory[END_REF] how the hippocampus which implements episodic learning could complement the cortex in order for the mammalian brain to remedy that kind of weaknesses. This cerebral structure in the medial temporal lobe binds information from most regions of the cortex and is able to learn the current cortical activation in one shot. Due to mechanisms of sustained activity in the cortex evoked below, this activation stands for the current episode processed by the cortex, hence the term episodic learning. This memory is explicit because, in contrast to the cortex that evokes relations acquired from many episodes, this learning manipulates individual episodes explicitly. This is associated with specific characteristics of hippocampal information processing: an elaborated function of pattern separation based on sparse coding [START_REF] Kassab | Pattern separation in the hippocampus: distinct circuits under different conditions[END_REF] to minimize interference when learning a new episode and a mechanism of recall enabling the hippocampus to rebuild an originally stored pattern from a partial cue thanks to its unique recurrent architecture. Recall is associated with a mechanism called replay [START_REF] Stoianov | The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning[END_REF] to reactivate a stored episode in the cortex. This process helps solving the problem of catastrophic interference because, when you are learning a new ability in the cortex, it has been shown that during sleep, the hippocampus replays other kinds of examples to balance learning [START_REF] Stoianov | The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning[END_REF]; and what is replayed here are explicit and unique episodes and not a statistical implicit model. This illustrates an interesting duality in our memories between knowledge (I know that) and skill (I know how).

C. Explicit memory: cognitive control

Let us now introduce another set of weaknesses associated with implicit learning. Reinforcement learning models are very powerful in stable worlds and can assign a value to states and actions and guide behavior accordingly, with no explicit knowledge of the rules of the world. This is no longer the case if the behavior to be learnt is context-dependent, or in other words when the rules change depending on certain perceptual cues or on internal information (instructions given or simply emotions or motivations). An implicit model could unlearn the old behavior and learn the new one but this would come at a cost of time. This is not compatible with the flexibility of human behaviour that can exhibit rapid changes from one situation to an other. In addition, we can see that this kind of change is explicit and deliberative. In this case, the solution would be to not only rely on the default behavior proposed by the implicit model from external information but to explicitly identify the context, inhibit the default behavior and instead promote the appropriate one which has been stored from previous learning. This is what we call cognitive control: whereas simple decision making relies on perception to control actions and promote a certain behavior, cognitive control relies on external and internal contextual cues to control behavior in a top-down way.

It has long been known in neuroscience that the prefrontal cortex (PFC) is the structure responsible for this function [START_REF] Fuster | The prefontal cortex. Anatomy, physiology and neurophysiology of the frontal lobe[END_REF]. This structure, particularly developed in humans (30% of the total surface of the cortex), is characterized by its ability to show a long-lasting sustained activity, also called working memory. In certain circumstances, it is able to inhibit a maladapted default behavior and promote a more appropriate one for an extended period of time. This is done by biasing the activity of the cortex, i.e. by artificially imposing an increased activity to some sensory regions of the cortex through an attentional process [START_REF] O'reilly | Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control[END_REF]. In this way, a nondominant behavior can become eligible and active. When rule changes, the working memory can be updated to promote another behavior. Deciding if the rules have changed (from the monitoring of errors measured between the anticipated and observed outcomes) and learning what contextual cues to maintain in working memory are the roles of the medial and lateral PFC respectively [START_REF] Alexander | Hierarchical error representation: a computational model of anterior cingulate and dorsolateral prefrontal cortex[END_REF]. In these regions, the neuronal populations are organized according to the nature of the task [START_REF] Domenech | Executive control and decisionmaking in the prefrontal cortex[END_REF]. It has been shown that they are organized along several dimensions on the cortical surface [START_REF] O'reilly | The What and How of prefrontal cortical organization[END_REF]. From rostral to more caudal portions, more and more abstract rules can be observed. There is also a dorsal-ventral axis, with rules predominantly concerned with the semantic meaning of the task on the ventral side and rules controlling the syntactic organization of the task on the dorsal side. When errors indicate that the rule has changed, it is possible to immediately test other rules which have been proven efficient for this task, else to look for rules applied in similar (i.e. sharing characteristics) tasks. Another proposed strategy is to build new rules by mixing old ones [START_REF] Collins | Reasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making[END_REF]. As discussed below, these are key mechanisms for creativity.

III. DELIBERATING IN THE FUTURE A. Prospective memory

We have explained that the hippocampus is able to bind distributed activities of the cortex to explicitly create a memory of a specific episode and that the PFC is able to bias the activity of the cortex to inhibit the default implicit behavior and explicitly promote a behavior more appropriate in the current context [START_REF] Cohen | A Preliminary Theory of the Interactions Between Prefrontal Cortex and Hippocampus that Contribute to Planning and Prospective Memory[END_REF]. What is fundamental to understand here is that the hippocampus also receives pieces of information from the PFC and binds them within episodes which are also sent back to the PFC thus participating to its training and activation. Reciprocally, the PFC receives contextual information elaborated in the hippocampus and can bias the activity of the hippocampus in order to control both the storage and retrieval of episodes [START_REF] Eichenbaum | Prefrontal-hippocampal interactions in episodic memory[END_REF]. Furthermore, it is now widely accepted that memory is a (re)constructive process, and that nearly the same circuitry is used for not only remembering (or reconstructing) the past, but also imagining the future [START_REF] Schacter | Remembering the past to imagine the future: the prospective brain[END_REF]. In summary, the hippocampus and the PFC both form a generative model by learning an explicit model of the world. By sampling from various episodes and anticipating the consequences of one's actions, they are able to internally simulate and evaluate candidate strategies. This is what we call imagination and what is realized in the process of thinking [START_REF] Pezzulo | Thinking as the control of imagination: a conceptual framework for goal-directed systems[END_REF].

B. What about creativity ?

Neuroscientific studies about creativity are in accordance with the general picture given here and provide additional details. [START_REF] Milivojevic | Insight Reconfigures Hippocampal-Prefrontal Memories[END_REF] report an interesting fMRI experiment demonstrating the role of the medial PFC and hippocampus in reorganizing memory when new information is presented. In the experiment, a series of narrative videos is shown, including clues linking some of them and stimulating insight in participants (i.e. the sudden discovery of a solution to a problem). It is observed that these moments are associated with the emergence of new mnemonic representations while irrelevant events are pruned out. The paper reports individual events activating the hippocampus and mismatch response in the anterior hippocampus being sent to the medial PFC. [START_REF] Mayseless | Generating original ideas: The neural underpinning of originality[END_REF] report the role of the medial PFC in inhibiting automatic common ideas through its dorsal part and allowing for flexibility and original ideas generation through its ventral part. The medial PFC can detect the co-existence of several solutions including non-dominant solutions and can switch attention to favor one of them [START_REF] Kounios | The cognitive neuroscience of insight[END_REF], with the level of originality in creativity rather linked to broadened and not focused attention [START_REF] Takeuchi | Originality of divergent thinking is associated with working memory-related brain activity: Evidence from a large sample study[END_REF]. Concerning divergent thinking, insight is defined as the reinterpretation of a situation to produce non-dominant interpretation [START_REF] Kounios | The cognitive neuroscience of insight[END_REF] and characterized by the activation of the medial PFC (for conflict monitoring) and of the medial temporal lobe including the hippocampus (associated with semantic integration before insight). In that paper, the role of mood is also underlined, associated with the broadening of semantic processing. This might correspond to the difference between spontaneous creativity monitored by such bottom-up information, opposed to deliberate creativity monitored in a top-down way by the PFC [START_REF] Dietrich | The cognitive neuroscience of creativity[END_REF]. Concerning convergent thinking, verifying the semantic and syntactic pertinence of a newly proposed behavior is part of the top-down control of the PFC observed during creativity, shown to activate the ventral and dorsal regions respectively [START_REF] Jung | The structure of creative cognition in the human brain[END_REF]. Interestingly, in prospective memory, this verification can be done internally using the learnt generative models, with no need for real action.

IV. COMPUTATIONAL ASPECTS

A. Models of explicit memory

Whereas recent achievements in machine learning are mainly related to models of implicit memory, more attention has recently been given to models of explicit memory in order to provide more realistic and flexible models. In reinforcement learning, this is for example the case of episodic and meta reinforcement learning [START_REF] Botvinick | Reinforcement Learning, Fast and Slow[END_REF], introduced to accelerate deep reinforcement learning with the same kind of arguments as provided above. In the case of supervised learning, the principle of conceptors [START_REF] He | Overcoming Catastrophic Interference using Conceptor-Aided Backpropagation[END_REF] has also been introduced as an attentional process by restricting processing to a sub-region of the data space. Beyond these computational principles, other models have a deeper biological grounding. Let us first mention a model of the hippocampus able to store and recall episodes [START_REF] Schapiro | Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning[END_REF], with the emphasis set on interactions between various inner mechanisms. Particularly, in addition to the learning of specific episodes, this model is also able to discover statistical regularities in the episodes, which is shown to be a potential substrate for transitive inference (i.e. the discovery that if A is associated to B and B is associated to C, then A is associated to C), an important ingredient of creativity. Other computational studies have extended the traditional reinforcement learning paradigm to account for hippocampal sequence generation. For example, a recently proposed model provides an explanation as to why and how the hippocampus is capable of not only replaying experienced sequences of states, but "imagining" unexperienced ones as well [START_REF] Khamassi | Modeling awake hippocampal reactivations with model-based bidirectional search[END_REF]. Concerning cognitive control, [START_REF] Alexander | Hierarchical error representation: a computational model of anterior cingulate and dorsolateral prefrontal cortex[END_REF] implements a neuronal model based on the principles of predictive coding. Errors detected between the anticipated and actual outcomes are mapped and associated with their context to subsequently trigger a more adapted behavior when the same context is detected.

B. What about computational creativity ?

To our knowledge, there are presently very few CCN models dealing explicitly with creativity, with the notable exceptions of [START_REF] Collins | Reasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making[END_REF] and [START_REF] Thagard | The aha! experience: Creativity through emergent binding in neural networks[END_REF]. The former model mainly focuses on the action of the medial PFC in creativity. It proposes a biologically informed computational account of how we can flexibly switch from one set of rules to another and explains that we might create a new set of rules from combining existing ones. In this model using simple symbolic tasks, the role of the hippocampus is not considered, which leaves considerable room for improvement. In the latter study, a computational account of creativity emphasizing the importance of binding information from multiple sources is given. In our opinion, what is currently lacking is a computational model of the interactions between the hippocampus (arbitrary binding and episodic learning) and the PFC (control of encoding, recall and imagination) dedicated to creativity. Beyond implementing controlled episodic memory, we believe that such a model would shed light on the ability of the "construction system of the brain" [START_REF] Hassabis | The construction system of the brain[END_REF] to creatively think about novel situations and verifying their task relevance (Fig. 1).

V. CONCLUDING REMARKS

In this paper, we attempted to lay down the foundations for a CCN approach to creativity based on the combination of cognitive control and episodic learning. We have explained that several models already propose mechanisms participating to creative processes in the brain but several characteristics need to be studied in more depth. For example, the role of hemispheric differences is often mentioned in neuroscientific experiments about creativity [START_REF] Kounios | The cognitive neuroscience of insight[END_REF] whereas this concept of lateralization is mostly absent in models. We believe that an accurate model of how creativity is implemented in the brain would explain experimental findings, such as the deactivation of the lateral PFC and activation of the medial PFC that have been observed during music improvisation [START_REF] Jung | The structure of creative cognition in the human brain[END_REF]. We can hypothesize that the medial PFC is responsible for tracking errors in the improvisation, while the constrained control of behavior implemented by the lateral PFC is being inhibited. We also believe that better understanding the computational mechanisms associated to creativity would greatly augment machine intelligence considering their role in imagination, planning, decision-making, navigation, and arguably all other cognitive abilities. The development of a detailed computational framework will likely lead to the demystification of creativity as a unique human capability. Even more striking is the possibility that creativity might be reduced to certain aspects of control, memory and decision making.

Fig. 1 .

 1 Fig. 1. The proposed framework combines cognitive control and episodic learning for solving flexible episodic memory and creative planning.