Vasil Penchev
email: vasildinev@gmail.com

Quantity in Quantum Mechanics and the Quantity of Quantum Information

Keywords: quality, quantity, quantum information, qubit Hilbert space, space-time I

The paper interprets the concept "operator in the separable complex Hilbert space" (particalry, "Hermitian operator" as "quantity" is defined in the "classical" quantum mechanics) by that of "quantum information". As far as wave function is the characteristic function of the probability (density) distribution for all possible values of a certain quantity to be measured, the definition of quantity in quantum mechanics means any unitary change of the probability (density) distribution. It can be represented as a particular case of "unitary" qubits. The converse interpretation of any qubits as referring to a certain physical quantity implies its generalization to non-Hermitian operators, thus neither unitary, nor conserving energy. Their physical sense, speaking loosely, consists in exchanging temporal moments therefore being implemented out of the space-time "screen". "Dark matter" and "dark energy" can be explained by the same generalization of "quantity" to non-Hermitian operators only secondarily projected on the pseudo-Riemannian space-time "screen" of general relativity according to Einstein's "Mach's principle" and his field equation.

However in the "classical" quantum mechanics, "quantity" has been defined partly in a compromising manner, "middle-of-the-road": indeed as a transformation of the probability (density) distribution, but allowing for the transformation to be only unitary and accordingly, the the corresponding operator defined on the separable complex Hilbert space of their characteristic functions to be only Hermitian. That restriction limits all admissible transformations, being interpreted physically, to those conserving energy, and thus, the quantity of time makes sense after them. In other words, time is saved not only as a fundamental physical quantity, but furthermore as an absolute condition for anything claiming to be physical or studible by physics.

The "slap glued on the chick" of the "classical" quantum mechanics due to that compromise is both "dark energy" and "dark matter", at that, about twenty times more tnah the "visible" ones in the universe. The magnitude of that "darkness" is the measure of our ignorance as well as for the falsity of human cognition, not only physical.

II QUANTUM QUANTITY AS INFORMATION BY QUANTUM INFORMATION

The exhaustive reformulation of quantum mechanics as the theory of quantum information undertaken at the end of the last century allowed particularly for the compromise to "quantity" to be removed thoroughly in favor of quantum mechanics due to its "completeness' (including the corresponding theorems: Neumann 1932; [START_REF] Kochen | The problem of hidden variables in quantum mechanics[END_REF] as well as due to all phenomena of entanglement corroborated experimentally very well.

The concept itself of quantum information as a quantity measured in qubits allows for it to be considered as invariant to a new and generalized "quantum relativity", on the basis of which both entangled and non-entangled systems can be unified (Penchev 2021 June 8) unlike that of Hermititian operator relevant only to non-entangled systems therefore clearing the pathway for a new, more fundamental and properly informational understanding of "quantity", which can be illustrated unambiguously by a single qubit of any physical quality, respectively quantity:

"𝑄𝑢𝑏𝑖𝑡" (𝑑𝑒𝑓) ⇔ α⊥ 0 + β⊥ 1 𝑓𝑜𝑟: α, β ∈ 𝐶: α | | 2 + β | | 2 = 1 ; ⊥ 0 , ⊥ 1 ⊆ 𝐻: (⊥ 0 ) ⊥ ( ⊥ 1 )
Or written down by words: "Qubit" is any normed superposition by complex coefficients of two subspaces of the separable complex Hilbert space, orthogonal to each other, but not necessarily convex to each other (i.e. their intersection can be a nonempty set). If a qubit is interpreted as any representation of a three-dimensional unit vector as the vector sum of two orthogonal two complex vectors, a bit of information is the particular case where the complenes of the two vectors are to be summed are whole positive numbers (that is: either "0" or "1").

The definition of quantity in the "classical" quantum mechanics as a unitary change of probability distribution or respectively, a Hermitiam operator can be rewritten by quantum information or by qubits in a few ways, one of which is the following: are ⊥ 0 , ⊥ 1 ⊆ 𝐻 interpreted as relevant to state correspondingly "before" and "after" the Hermitian transformations: Then, the condition of unitarity (i,e the operator to be Hermitian) can be visualized as the particular case of to be real numbers furthermore: a condition, which each qubit refers α, β ∈ 𝐶 to a Hermitian operator ought to satisfy. The real axis implies a well-ordering, which can be "materialized" as a special physical quantity what time is, and naturally accompanied by the "conservation of energy conservation" in quantum mechanics [START_REF] Penchev | Quantum-Information Conservation. The Problem About 'Hidden Variables[END_REF].

If one considers the general case for the condition of unitarity not to be necessary, this corresponds for not to be necessarily real: the well-ordering of the real axis is α, β ∈ 𝐶 cancelled, and the quantity of time is irrelevant in general. The definition of quantity after quantum information can be represented concisely: any qubit (rather than that obeys the condition of unitarity) can be a value of a quantity. If all quantities in classical physics as ratios of integers (since they are always rational numbers representable as ratios of integers), quantum mechanics after quantum information considers the generalized case where all quantities as relations (i.e. relevantly generalized "ratios") are qubits: the fundamental "unit" of any quantum quantity is a qubit.

III QUANTUM QUANTITY GENERALIZED BY QUANTUM INFORMATION

There exists an implicit fundamental equation shared by any classical experimental science, which is not articulated, seeming to be trivial and absolutely obvious. However, it will be written expressly now, because quantum mechanics modifies it radically: 𝑃ℎ𝑦𝑠𝑖𝑐𝑎𝑙 𝑞𝑢𝑎𝑛𝑡𝑖𝑡𝑦 = (𝑅𝑎𝑡𝑖𝑜𝑛𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟)` 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑖𝑒𝑑 𝑏𝑦 (𝑡ℎ𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑖𝑛𝑔 𝑢𝑛𝑖𝑡 𝑓𝑒𝑎𝑡𝑢𝑟𝑖𝑛𝑔 𝑡ℎ𝑒 𝑞𝑢𝑎𝑙𝑖𝑡𝑦 𝑜𝑓 𝑡ℎ𝑒 𝑞𝑢𝑎𝑛𝑡𝑖𝑡𝑦)

For example, "A certain distance is 4.23 meters" means:

'𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 = (4. 23) 𝑋 (𝑡ℎ𝑒 𝑢𝑛𝑖𝑡 𝑜𝑓 𝑚𝑒𝑡𝑒𝑟)'
This kind of representation is applicable to each physical quantity and each measurement. At first glance, the generalization of quantum mechanics is not so radical, only substituting eventually the rational number (e.g. "4.23") by a relevant wave function; at that, any single measurement in quantum mechanics keeps the same form as a classical one.

However, the radicality becomes even shocking if one admits that "wave function" and the "quality" of the quantity at issue are complementary and thus interpretable as orthogonal to each other. In other words, the statement that the measuring wave function and the quality of the physical quantity constitutes in turn a pair, to which the concept and quantity of wave function are applicable: the corresponding quality of which is quantum information, and its unit is a qubit, therefore remaining physically dimensionless. Furthermore, quantum information can be considered not only as a universal physical substance more general than energy, but as the ultimate physical substance in virtue of the completeness of quantum mechanics (respectively, the "no hidden variables" theorems in it): indeed, the wave function measuring any physical quantity can be identified as the same with the meta-wave function able to represent the relation of the measuring wave function and the physical quantity at issue. Furthermore, one can introduce the conservation of quantum information to express the above identification of the wave function on the level of physical quantity and it on the corresponding metalevel. The eventual and admissible, mutual transformations of "quantity" and "quality" (absolutely forbidden in classical science and even qualified as anti-scientific) obey quantum-information conservation, which regulates them excluding many classes of them as physically impossible. All phenomena of entanglement allow for such transformations between "quality" and "quantity". For example the change of any quantity due to entanglement can be interpreted as the transformation of the corresponding quality of that quantity into the change of wave function.

IV NON-TEMPORAL BOTH QUANTUM MECHANICS AND PHYSICS

The mutual transformations of "quantity" and "quality" cannot happen as processes in time and consequently they are not pictorable on the "temporal screen" of classical science or human empirical experience. The phrase "physical phenomena out of space-time" does not make sense for classical physics and even science: one of the most fundamental scientific postulates states that any natural phenomenon can happen only as a process in space-time.

Quantum mechanics weakens it, declaring that space-time is necessary only for measurement, which can really occur only in space-time, but the problem about its applicability to the physical phenomena by itself remains open: for example, "Copenhagen interpretation" suggests that the question does not sense therefore being a pseudo-problem reflecting only the prejudice of classical physics. Practically, Pauli's solution consisting in not assigning an operator to time (unlike all other physical quantities) or the rejection of the "four uncertainty" of energy and time by some authors [START_REF] De Broglie | Heisenberg's Uncertainties and the Probabilistic Interpretation of Wave Mechanics: with Critical Notes of the Author (Series: Fundamental Theories of Physics[END_REF][START_REF] Mehra | Niels Bohr's discussions with Albert Einstein, Werner Heisenberg, and Erwin Schrödinger: The origins of the principles of uncertainty and complementarity[END_REF][START_REF] Costa De Beauregard | Bohr's discussion of the fourth uncertainty relation revisited[END_REF] follows the "Copenhagen" track (even rejecting it).

However, quantum correlations or the phenomena of entanglement confirmed experimentally [START_REF] Camilleri | A history of entanglement: Decoherence and the interpretation problem[END_REF][START_REF] Ladyman | Entanglement and non-factorizability[END_REF][START_REF] Cohen-Tannoudji | Quantum Mechanics. Volume 3: Fermions, Bosons, Photons, Correlations, and Entanglement[END_REF] cannot fit in the paradigm of the "Copenhagen agnosticism" rather than only in that of classical physics, since what a quantum system is before measurement (i.e. "by itself" following Kant's idea of "Ding-an-sich") can be measured anyway as the change of probability distribution as a relation of at least two series of quantum measurements and situated at an arbitrary distance from each other. That change of probability distribution is not unitary definitively therefore suggesting that the intersection of the investigated remote probability distributions is not empty, though it should be such for, otherwise, it would imply the "spooky action at a distance" (in Einstein's eloquent enough metaphor) of entanglement.

Well, entanglement and thus quantum correlations are corroborated well enough nowadays: consequently there exists that "spooky action at a distance" (though rejected by Einstein in favor of the universality of space-time and implying the alleged "incompleteness of quantum mechanics": [START_REF] Einstein | Can Quantum-Mechanical Description of Physical Reality Be Considered Complete?[END_REF].

Anyway, the violation of the "unconditional" space-time condition of classical physics seemed to be restricted only to quantum phenomena commensurable with the Planck constant and thus, not imposing any change of the fundamental understanding of science as it has been built relying on human empirical experience and only conservatively extended by relevant, though more and more complicated experiments during the period of classical science and physics and ended by the occurrence of quantum mechanics. In fact, the latter doubts the universal validity of "space-time screen" also to the macroscopic phenomena investigated by classical physics:

For example, the famous metaphor of "Schrödinger's cat" called to represent quantum phenomena by our usual experience utilizes the dissipative mechanism of many quantum measurements. It can be easily modified into two "entangled cats" situated anywhere in the universe interacting in virtue of quantum correlations and thus out of space-time: their fates will be entangled in a rigorous scientific meaning as the death of the one of them would change the chance of survival of the other instantaneously.

The converse implication can serve for confirming phenomena of macroscopic entanglement: if one observes any jump-like change in the probability distribution of any natural process which cannot be ascribed to any known cause, this can be explained in two alternative ways: (1) the action of some unknown cause yet not investigated by science; (2) macroscopic entanglement.

The peculiarity of the former hypothesis consists in its unfalsifiability (and consequently its non-scientificity in Popper's sense) if the latter conjecture is not researched: however, the law of teleportation needing two additional bits of classical information prevents its experimental corroboration (at first glance):

Alice's "Schrödinger's cat" and Bob's one are entangled, which is equivalent to the instantaneous transmission of a qubit between them. However, the fact that the former is observed to be either alive or dead is just that one classical bit of information transferable only in a subliminal channel. Bob waits for it. Meaning the distance to Alice, it calculates the corresponding moment in the past of his cat, but he is not able to implement any observation in a past moment, post factum Nonetheless, an experiment able to confirm macroscopic entanglement of the remote cats seems to be possible. Bob and Alice conduct simultaneously two series of experiments on two sets of Schrödinger's cats, sending the results immediately. Waiting enough, each of them can notice the correlative coincidences just post factum in the corresponding probability distributions (if any). If those correlations violate the [START_REF] Bell | On the Einstein Podolsky Rosen Paradox[END_REF] equations (being a sufficient, but not necessary condition for entanglement), they would confirm macroscopic entanglement. If they do not violate them, both kinds of explanations remain possible1 .

One can visualize the direct transformation of "quality" and "quantity" by those two entangled Schrödinger cats considering the mutual correlations of the two bits "dead / alive" for each of both cats and granting for both "alive" and "dead" to be qualities. In each moment of time, the quality "dead" (respectively alive") is transformed in the quantity of a qubit, just as a quantity "transferred instantly" (or "properly mathematically"), and restored again by a correlative quality (i.e. properly physical) elsewhere. Particularly, classical physics suggests for itself, mathematics, and philosophy to be absolutely divided from each other, but quantum mechanics and information needs their unification, an expression of which is the mutual transformation of "quantity" and "quality".

The Schrödinger thought experiment suggests that the deterministic dissipative mechanism, which is able to unambiguously link quantum phenomena to macroscopic experience, is explicit. However, Bob and Alice might "bracket" it, admitting for that mechanism whether to be implicit, or even that it does not exist, therefore restricting themselves in a inherent "Copenhagen maner" to searching for coinciding simultaneous correlations in probability distribution of the same macroscopic phenomena distinguishing them from each other only in space-time localization. Such kind of "macroscopic quantum correlations" would confirm "macroscopic entanglement".

VI A NON-TEMPORAL SOLUTION OF THE PROBLEM ABOUT "DARK MATTER" AND "DARK ENERGY"

However, "macroscopic entanglement" definiable also experimentally as above would imply direct physical effects as in Alice's reference frame as in Bob's one not needing any communication between them, necessary for quantum correlations to be established experimentally:

The jump-like change of probability distribution observed by Bob or Alice implies a Hermitean operator, a physical quantity and a certain mass (respectively, energy) of some unknown something (which can be called "dark" metaphorically), to which that quantity due to the probability distribution change is to be ascribed. The total sum of all those changes once integrated all over the universe would result in "dark mass" and "dark energy" (independent of each other in virtue of the independence of mass and energy in general relativity and the Einstein field equation).

So, entanglement suggests to be an admissible hypothesis about the origin of "dark masss" and "dark energy" (Penchev 2021 October 20;[START_REF] Penchev | The Generalization of the Periodic Table: The 'Periodic Table' of 'Dark Matter[END_REF]. According to it, they or measured quantity. So, the necessary and sufficient condition of entanglement is: those probability distributions overlap each other, which is impossible in classical science for no probability distribution can correspond to any quantity in its framework, but only real and thus certain values; or by reflecting philosophically: entanglement refers to physical interaction of possibilities which cannot and will not be transformed in reality. They have to interact "in advance" (but "in advance" here means "before time or space-time to appear") to avoid any inconsistency in the ultimate solution what reality represents after decoherence and as a result of the "quantum calculation of the universe" (the quotation marks here are only an expression of diplomacy, tact, and courtesy). The ratio of dark matter and energy to visible ones (≈ 20:1) recalcultes quantum information into mass and energy unambiguously, therefore implicitly admitting their mutual transformation. would appear as "projections" of quantum information (being out of space-time by itself) on the space-time "screen", only on which general relativity is able to observe any physical phenomenon, however necessarily violating "Mach's principle" suggested by [START_REF] Einstein | Prinzipielles zur allgemeinen Relativitätstheorie[END_REF]: it and postulatws that only energy and mass can be the source of gravitational field. On the contrary, quantum information turns out to be another source needing "dark mass" and "dark energy" to legitimize its existence since only the quantities of mass and energy are claimed to be admissible until now.

VI CONCLUSIONS, OR DARK MASS AND DARK ENERGY BY THE UNIFICATION AND MUTUAL TRANSFORMATION OF QUANTITY AND QUALITY

The sense of the paper consists in the interpretation of deducing the existence of "dark matter" and "dark matter" from the postulate of the totality (as in another paper: Penchev 2020 October 20) by the philosophical concepts of "quantity" and "quality" relying on the conception of "scientific transcendentalism" in turn interpreting "philosophical transcendentalism" in a falsifiable (thus non-metaphysical) way:

The area of quantity (e.g. understood in a contemporary and wider meaning as "mathematical structure") suggests the smooth transformation in its framework if and only if the quality of the quantity at issue remains the same. On the contrary, the area of quality supposes the discrete, jumplike transformation of any quality into another.

Quantum mechanics in physics and category theory in mathematics overcome or were forced to overcome that absolute opposition of quality and quantity featuring classical science and particularly as physics as mathematics motivated accordingly by the necessity: (1) to unify classical smooth physics and quantum phenomena to conserve quantum mechanics as an objective and experimental science; (2) to justify the consistent completeness of mathematics internally, i.e. admitting a mathematical proof of consistent completeness to be able to be demonstrated.

In the final analysis, the isomorphism of the consistent completeness of both quantum mechanics and category theory can be deduced relying as on the separable complex Hilbert space of quantum mechanics as on the space of all categories transformable between each other by functors in turn belonging to categories necessarily as the category of functors.

Both result into the qubit Hilbert space, which is able to deliver also a formal model of Husserl's phenomenology furthermore, therefore being able to unify (at worst, at least formally) a physical theory (such as quantum mechanics), a mathematical theory (such as category theory) and a philosophical doctrine (such as Hussel's phenomenology) thus all the three underlain by the same mathematical structure, namely the qubit Hilbert space.

The present paper is limited to the particular objectivity to explicate how the philosophical assumption of mutual direct transformation of quality and quantity implies the existence of dark matter and dark energy by the mediation of the qubit Hilbert space being furthermore in the foundations of the just cited three regions of cognition absolutely different according to classical science:

If one admits quality and quantity to be transferable mutually, the quantitative (i.e. as a single quantity) expression of the nonzero area of both quantity and quality is meant necessarily in physics as dark matter and dark energy (also under the implicit condition for general relativity to be valid in order to be independent of each other):

Indeed, any quality in classical physics features a certain quantity furthermore needing to be a property of some physical entity belonging to this property. Being just physical, it shares a certain amount of energy (and an amount of mass being independent of the former amount of energy according to general relativity and e.g. obvious in the structure of the Einstein field equation) directly corresponding or being due to the quality of the property, the carrier of which is the entity with that energy:

If one changes the quality, this implies to be changed the entity and thus its energy (as the universal quantity shared by all physical entities) in general, resulting in the final analysis into the transformation of any quality represented in physics as a relevant quantity into an amount of the universal physical quantity of energy: thus being necessarily mapped on the absolute distinction of quality and quantity (in particular, "confessed" by classical science).

The change of the paradigm and corresponding worldview can be illustrated by the new gestalt for realizing the "Big Bang". All experimentally established quantitative properties and relations featuring the "first seconds of the universe after the Big Bang" are conserved, but interpreted otherwise: as the cumulative projection backwards in time of the entire decoherence, i.e. the quantity of decoherence determining a "field of decorence", further integrated throughout the space-time area of all the universe.

Though the two interpretations of the Big Bang are rather similar to each other, they can be distinguished experimentally anyway (and thus the one of them to be rejected in virtue of the factual mismatch). The explanation by the cumulative effect of the decoherence field implies a very small, but nonzero probability of "astronomical objects older than the universe" to be observed in virtue of quantum uncertainty since decoherence is a quantum phenomenon. That probability is smaller and smaller in the course of time featuring the "age of the universe", but remains enough to be discovered somewhere in the more and more expanding universe, especially in its "early stages" becoming observable only recently.

The new worldview, in which the transformation of quantity and quality is not only possible, but prevailing (e.g. in virtue of the predominance of dark matter and dark energy according to the experiments) seems so, figuratively:

A homogeneous "ocean" of quantum information being more general concept and physical quantity than space-time generates by itself and by mathematical necessity out of time, the space-time itself of the universe due to decoherence (e.g. representable as the process of dividing of the local and global space of the Standard model from each other): space-time is relevant only to the degree of absolute decoherence or respectively, to zero entanglement.

If one postulates space-time to be an absolute condition for anything claiming to be physical, that permanent process of decoherence (generating the space time itself) remains hidden and observable only: (1) as dark matter and dark energy; (2) particularly resulting in the concept of the "Big Bang" as a real phenomenon rather than a cumulative and probabilistic projection of all decoherence accompanying constality the generation of space-time.

In addition, that absolute understanding of the "Big Bang" implies an obvious inconsistency: itself is a huge violation of energy conservation, not only established as an absolute natural law after it, but necessary for it to be inferred interpolating the observed astronomical phenomena backwards in time: therefore relying on a "vicious circle" logically.

The physical sense of violating the Bell equations consists in the following: those are correlations in the separable complex Hilbert space rather than in space-time. So, their violation means immediately that there exist corresponding physical phenomena, which are out of space-time. However, that corollary is so shocking for the scientific common sense, that it dare not make it (just as all people who watch the "Nude King", but dare not tell the truth). Any entangled wave functions are characteristic functions of probability distributions which overlap each other by nonzero segments of the variable of the investigated


Mehdi Khamassi
email: mehdi.khamassi@sorbonne-universite.fr

Adaptive coordination of multiple learning strategies in brains and robots

Engineering approaches to machine learning (including robot learning) typically seek for the best learning algorithm for a particular problem, or a set problems. In contrast, the mammalian brain appears as a toolbox of different learning strategies, so that any newly encountered situation can be autonomously learned by an animal with a combination of existing learning strategies. For example, when facing a new navigation problem, a rat can either learn a map of the environment and then plan to find a path to its goal within this map. Alternatively, it can learn sequences of egocentric movements in response to identifiable features of the environment. For about 15 years, computational neuroscientists have searched for the mammalian brain's coordination mechanisms which enable it to find efficient, if not necessarily optimal, combinations of existing learning strategies to solve new problems. Understanding such coordination principles of multiple learning strategies could have great implications in robotics, to enable robots to autonomously determine which learning strategies are appropriate in different contexts. Here, we review some of the main neuroscience models for the coordination of learning strategies and present some of the early results obtained when applying these models to robot learning. We moreover highlight important energy costs which can be reduced with such bio-inspired solutions compared to current deep reinforcement learning approaches. We conclude by sketching a roadmap for further developing such bio-inspired hybrid learning approaches to robotics.

Introduction

The mammalian brain combines multiple learning systems whose interactions, sometimes in a competitive way, sometimes in a cooperative way, are thought to be largely responsible for the high degree of behavioral flexibility observed in mammals [START_REF] John | The hippocampus as a cognitive map[END_REF][START_REF] James L Mcclelland | Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory[END_REF][START_REF] Nakahara | Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences-a computational approach[END_REF][START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Khamassi | Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies[END_REF][START_REF] Keramati | Speed/accuracy tradeoff between the habitual and the goal-directed processes[END_REF][START_REF] Khamassi | Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies[END_REF][START_REF] Pezzulo | The mixed instrumental controller: using value of information to combine habitual choice and mental simulation[END_REF][START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF]. For instance, the hippocampus is a brain region playing an important role in the rapid acquisition of episodic memories -the memory of individual episodes previously experienced, such as sequences of visited places while visiting a new city [START_REF] Burgess | The human hippocampus and spatial and episodic memory[END_REF][START_REF] Leutgeb | Independent codes for spatial and episodic memory in hippocampal neuronal ensembles[END_REF][START_REF] Eichenbaum | Prefrontal-hippocampal interactions in episodic memory[END_REF]. Together with the prefrontal cortex, the hippocampus can link these episodes so as to store in long-term memory a mental representation (or 'model') of statistical regularities of the environment [START_REF] Paul | The organization of recent and remote memories[END_REF][START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF][START_REF] Stoianov | The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning[END_REF]. In the spatial domain, such a mental model can take the form of a 'cognitive map' [START_REF] John | The hippocampus as a cognitive map[END_REF]. Even if it constitutes an imperfect and incomplete representation of the environment, it can be used to mentally explore the map [START_REF] Anoopum S Gupta | Hippocampal replay is not a simple function of experience[END_REF][START_REF] Cazé | Hippocampal replays under the scrutiny of reinforcement learning models[END_REF], or to plan potential trajectories to a desired goal before acting [START_REF] Johnson | Neural ensembles in ca3 transiently encode paths forward of the animal at a decision point[END_REF][START_REF] Marcelo | Prioritized memory access explains planning and hippocampal replay[END_REF]. Such a model-based strategy enables to rapidly and flexibly adapt to changes in goal location, since the map can be updated instantaneously with the new goal location so that the animal can plan the new correct trajectory in a one-trial learning manner [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF]. Nevertheless, such a flexibility comes at the expense of time-and energyconsuming planning phases (the larger the map, the longer it takes to find the shortest path between two locations). This is typically observed when a human or an animal takes longer decision time after task changes, putatively implying a re-planning phase [START_REF] Johnson | Neural ensembles in ca3 transiently encode paths forward of the animal at a decision point[END_REF][START_REF] Viejo | Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning[END_REF].

In contrast, the basal ganglia, and especially its main input region called the striatum, is involved in the slow acquisition of procedural memories [START_REF] Steven P Wise | The role of the basal ganglia in procedural memory[END_REF][START_REF] Mark | Learning and memory functions of the basal ganglia[END_REF][START_REF] Khamassi | Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies[END_REF]. This type of memories is typically acquired through the repetition of sequences of egocentric movements (e.g., turn left, go straight, turn right) or sequences of stimulus-triggered responses (e.g., start moving in response to a flash light, then stop in response to a sound) which become behavioral 'habits' [START_REF] Mandar | Building neural representations of habits[END_REF][START_REF] Dezfouli | Habits, action sequences and reinforcement learning[END_REF]. These habits are known to be inflexible, resulting in slow adaptation to both changes in the environment (e.g., change in goal location) and changes in motivation (e.g., an overtrained rat habitually presses a food-delivering lever, even when it is satiated) [START_REF] Dickinson | Motivational control of goal-directed action[END_REF]. Nevertheless, when these habits have been well acquired in a familiar environment, they enable the animal to make fast decisions and to perform efficient action sequences without relying on the time-consuming planning system [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Khamassi | Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies[END_REF]. Such a learning strategy is thus called model-free because making a decision does not require the manipulation of an internal model to mentally represent the potential long-term consequences of the actions before acting. In contrast, it is the perception of a stimulus or the recognition of a familiar location which triggers the execution of a habitual behavioral sequence.

It is fascinating how lesion studies have highlighted some degree of modularity of the organization of learning systems within the brain. Lesioning the hippocampus impairs behavioral flexibility as well as behavioral strategies relying on a cognitive map (see [START_REF] Khamassi | Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies[END_REF] for a review). As a consequence, hippocampus-lesioned animals only display stimulus-response behaviors in a maze and do not seem to remember the location of previously encountered food. In contrast, animals with a lesion to what is called the dorsolateral striatum have an intact map-based behavioral strategy and perform less egocentric movements during navigation (see [START_REF] Khamassi | Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies[END_REF] for a review). Nevertheless, the modularity is not perfect and an important degree of distributed information processing also exists. For instance, lesions to what is called the ventral striatum seem to impair representations of reward value which are required for both model-based and model-free learning strategies (again see [START_REF] Khamassi | Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies[END_REF] for a review). Moreover, some brain regions do not seem to play a specific role in learning one particular strategy, but rather a role in the coordination of these strategies. For instance, lesions to the medial prefrontal cortex only impairs the initial acquisition of model-based strategies, but not their later on expression [START_REF] Sean | Lesions of medial prefrontal cortex disrupt the acquisition but not the expression of goal-directed learning[END_REF]. Indeed, it seems that the medial prefrontal cortex plays a central role in the arbitration between model-based and model-free strategies [START_REF] Killcross | Coordination of actions and habits in the medial prefrontal cortex of rats[END_REF]. As a consequence, lesioning a subpart of the medial prefrontal cortex can even restore flexible model-based strategies in overtrained rats [START_REF] Coutureau | Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats[END_REF].

This paper is particularly aimed at illustrating how neuroscience studies of decision-making have progressively helped understanding (and are still currently investigating) the neural mechanisms underlying animals' ability to adaptively coordinate model-based and model-free learning, and to illustrate how this biological knowledge can help towards developing behaviorally flexible autonomous robots. Since the computational models of these processes largely rely on the reinforcement learning theoretical framework, the next section will first describe the employed formalism. Then the third section will briefly review some of the neuroscience results which contribute to deciphering the principles of this coordination, and how this coordination was mathematically formalized by computational neuroscience models. The fourth section will then review some of the experimental tests of these principles in robotic scenarios. We will finally discuss the perspectives of this field of research, and how it could not only contribute to improving robots' behavioral flexibility, but also to reducing the computational cost of machine learning algorithms for robots (by enabling to skip model-based strategies when the robot autonomously recognizes that model-free strategies are sufficient).

2 Formalism adopted to describe model-based and model-free reinforcement learning

For simplicity, existing computational models are most often framed within standard Markov Decision Problem (MDP) settings, where an agent visits discrete states s ∈ S, using a finite set of discrete actions a ∈ A. They can encounter reward scalar values r ∈ R after performing some actions a in some states s, which they have to discover. And there is a transition probability function T (s, a, s ) : (S, A, S) → [0; 1], which is a generative model underlying the statistics of the task that the agent will face, and which basically determines what is the probability of ending up in a state s after performing an action a in state s.

In navigation scenarios, states represent unique locations in space. In neuroscience modeling studies, these states are usually equally spaced on a square grid, an information expected to be provided by place cell activity in the hippocampus, which are neurons that participate in the estimation of the animal's current location within the map. In the robot navigation experiments that will be presented later on, the state space discretization process is autonomously performed by the robot after an initial exploration of the environment. As a consequence, different states have different sizes and are unevenly distributed. It is important to note that these models can easily be extended to more distributed or con-tinuous state representations [START_REF] Arleo | Cognitive navigation based on nonuniform gabor space sampling, unsupervised growing networks, and reinforcement learning[END_REF], for instance when facing Partially Observable Markov Decision Process (POMDP) settings [START_REF] Anthony R Cassandra | Acting optimally in partially observable stochastic domains[END_REF]. The models reviewed here can also be generalized to more continuous representations of space and actions (e.g., [START_REF] Doya | Reinforcement learning in continuous time and space[END_REF][START_REF] Khamassi | Robot fast adaptation to changes in human engagement during simulated dynamic social interaction with active exploration in parameterized reinforcement learning[END_REF]). Nevertheless, we stick here to the discrete state representation for the sake of simplicity and because it has a high explanatory power.

As classically assumed within the Reinforcement Learning (RL) theoretical framework, the agent's goal is here considered to be the maximization of the expected value of the long-term reward, that is the maximization of the discounted sum of future rewards over a potentially infinite horizon:

E [ ∞ t=0 γ t r(s t , a t )],
where γ (γ < 1) is a discount factor which basically assigns a weaker weights to long-term rewards than to short-term rewards. In order to meet this objective, the agent will learn a state-action value function Q : (S, A) → R which evaluates the total discounted sum of future rewards that the agent expects to receive when starting from a given state s, taking the action a and then following a certain (eventually learned) behavioral policy π:

Q π (s, a) = E t γ t r(s t , a t ) | s 0 = s, a 0 = a, a t = π(s t ), s t+1 ∼ T (s t , a t , .) (1)
Saying that the agent adopts a model-based RL strategy means that the agent will progressively try to estimate an internal model of its environment. Conventionally, this model is the combination of the estimated transition function T (s, a, s ) and the estimated reward function R(s, a) that aims at capturing the rules that the human experimenter chooses to determine which (state,action) couples yield reward in the considered task.

Various ways to learn these two functions exist. Here, we will simply consider that the agent estimates the frequencies of occurrence of states, actions and rewards from its observations. Then, the learned internal model can be used by the agent to infer the value of each action in each possible state. This inference can be a computationally costly process, especially when all (s, a) are to be visited multiple times before reaching an accurate estimation of the stateaction value function Q. Some heuristics exist to simplify the computations, or to make it less costly, like trajectory sampling [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF] or prioritized sweeping [START_REF] Peng | Efficient learning and planning within the Dyna framework[END_REF][START_REF] Atkeson | Prioritized sweeping: Reinforcement learning with less data and less time[END_REF], which we review in [START_REF] Cazé | Hippocampal replays under the scrutiny of reinforcement learning models[END_REF]. Some alternatives to full model-based strategies exist, like the successor representation [START_REF] Dayan | Improving generalization for temporal difference learning: The successor representation[END_REF], which provides the agent with more flexibility and generalization ability than a pure model-free strategy, a smaller computational cost than a pure model-based strategy, and at the same time can contribute to describe some neural learning mechanisms in the hippocampus [START_REF] Kimberly L Stachenfeld | The hippocampus as a predictive map[END_REF][START_REF] Momennejad | Learning structures: Predictive representations, replay, and generalization[END_REF]. Nevertheless, for the sake of simplicity, here we will consider that the inference process in the model-based (MB) RL agent is performed through a value iteration process [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF]:

Q (t+1) M B (s, a) = R(t) (s, a) + γ s T (t) (s, a, s ) max k∈A Q (t) M B (s , k) (2)
In contrast, an agent adopting a model-free RL strategy will not have access to nor try to estimate any internal model of the environment. Instead, the agent will iteratively update its estimation of state-action value function through its interactions with the environment. Each of these interactions consist in performing an action a in a state s and observing the consequence in terms of the reward r that this yields and the new state s of the environment. Again, if we address a navigation problem, the possible actions are typically movements towards cardinal directions (North, South, East, West) and the new state s is the new location of the agent within the environment after acting. A classical and widely used model-free RL algorithm is Q-learning [START_REF] Christopher | Q-learning[END_REF]:

Q (t+1) M F (s t , a t ) = Q (t) M F (s t , a t ) + α(r t + γ max k∈A Q (t) M F (s t+1 , k) -Q (t) M F (s t , a t )) (3)
where α ∈ [0; 1] is the learning rate, and the term between parentheses, often written δ t is called the temporal-difference error [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF] or the reward prediction error [START_REF] Schultz | A neural substrate of prediction and reward[END_REF] because it constitutes a reinforcement signal which compares the new estimation of value (r t + γ max k∈A Q (t) M F (s t+1 , k)) after performing action a t in state s t , arriving in state s t+1 and receiving a reward r t , with the expected value Q (t) M F (s t , a t ) before executing this action. Any deviation between the two is used as an error signal to correct the current estimation of the state-action value function Q.

Finally, for the decision-making phase, no matter if the agent is model-free or model-based, the agent selects the next action a to perform from a probability distribution over actions in the current state s computed from the estimated state-action value function

x (t) (s, a) (with x (t) (s, a) = Q (t) M B (s, a) if the agent is model-based, or x (t) (s, a) = Q (t) M F (s, a)
if the agent is model-free), using a Boltzmann softmax function:

P (t) (a|s) = e βx (t) (s,a) k∈A e βx (t) (s,k) (4)
where β is called the inverse temperature which regulates the exploration/exploitation trade-off by modulating the level of stochasticity of choice: the closer β is to zero, the more the contrast between Q-values will be attenuated, the extreme being for β = 0 which produces a flat action probability distribution (random exploration); in contrast, the larger the value of β, the more the contrast between Q-values will be enhanced, which makes the probability of the action with the highest Q-value close to 1 when β tends towards ∞ (exploitation).

Neuroscience studies of the coordination of model-based and model-free reinforcement learning

Reinforcement learning models (initially only from the model-free family) have started to become popular in neuroscience in the mid 90s, when researchers discovered that a part of the brain called the dopaminergic system (because it innervates the rest of the brain with a neuromodulator called dopamine) increases its activity in response to unpredicted reward, decreases its activity in response to the absence of a predicted reward, but does not respond to predicted ones, as can be modeled when δ t , the right part between parentheses in Equation 3, is positive, negative or null, respectively [START_REF] Schultz | A neural substrate of prediction and reward[END_REF]. This discovery was followed by a large set of diverse neuroscience experiments to verify that other parts of the brain could show neural activity compatible with other variables of reinforcement learning models like state-action values (see examples of comparisons of neuroscience results with RL models' predictions in [START_REF] Bellot | Which temporal difference learning algorithm best reproduces dopamine activity in a multi-choice task?[END_REF][START_REF] Bellot | Dopamine neurons activity in a multi-choice task: reward prediction error or value function[END_REF]; and see [START_REF] Khamassi | Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies[END_REF] for a review).

More important for the topic of this paper, since about 10 years, an increasing number of neuroscience studies have started to investigate whether human and animal behavior during reward-based learning tasks could involve some sort of combination of model-based (MB) and model-free (MF) learning processes, and what are the neural mechanisms underlying such a coordination.

The simplest possible way of combining MB and MF RL processes is to consider that they occur in parallel in the brain, and that any decision made by the subject results from the simple weighted sum of MB and MF state-action values (i.e., replacing

x (t) (s, a) in Equation 4 by (1 -ω)Q (t) M B (s t , a t ) + ωQ (t) M F (s t , a t ),
with ω ∈ [0; 1] a weighting parameter). This works well to have a first approximation of the degree with which different individual subjects, whether human adults [START_REF] Nathaniel D Daw | Model-based influences on humans' choices and striatal prediction errors[END_REF], human children [START_REF] Johannes H Decker | From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learning[END_REF], or animals [START_REF] Lesaint | Modelling Individual Differences in the Form of Pavlovian Conditioned Approach Responses: A Dual Learning Systems Approach with Factored Representations[END_REF], rely on a model-based system to make decisions while considering the ensemble of trials made by the subject during a whole experiment. This has for instance helped understand that children make more model-free decisions than adults because their brain area subserving model-based decisions (their prefrontal cortex) takes years to mature [START_REF] Johannes H Decker | From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learning[END_REF]. This has also helped better model why some subjects are more prone than others to be influenced by reward predicting stimuli (which has implication to understand stimulus-triggered drug-taking behaviors in humans): roughly because their model-based process contributes less to their decisions [START_REF] Lesaint | Modelling Individual Differences in the Form of Pavlovian Conditioned Approach Responses: A Dual Learning Systems Approach with Factored Representations[END_REF].

Nevertheless, a systematic weighted sum of MB's and MF's decisions has the disadvantage of systematically requiring the (potentially heavy) computations from both learning systems. In contrast, it is thought that relying on habitual behaviors learned by the model-free system when the environment is stable and familiar is useful to avoid the costly inference-related computations of the modelbased system [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Khamassi | Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies[END_REF]. There could thus be some evolutionary reasons why humans do not always perform rational choices as could be (more often) the case if they were relying more on their model-based system [START_REF] Kahneman | Thinking, fast and slow[END_REF]: namely that they would not be able to make fast decisions in easy familiar situations. Instead, they would always need to make long inferences with their internal models before deciding. Thus, they would be exhausted at the end of the day if they had to think deeply for all the simple decisions they have to make everyday, like whether they should drink a coffee before taking a shower or the opposite, whether they should wear a blue or a red shirt, where to go for lunch, etc.

Alternatively, early neuroscience studies of the coordination of MB and MF process hypothesized a sequential activation of the two systems: humans and animals should initially rely on their MB system when facing a new task, so as to figure out what are the statistics of the task and what is the optimal thing to do; and as they repeat over and over the same task and get habituated to it (making it become familiar), they should switch to the less costly MF system which hopefully will have had time to learn during a long repetition phase. Moreover, if suddenly the task changes, they should restart using their MB system (and thus break their previously acquired habit) in order to figure out what has changed and what is the new optimal behavioral policy. And then again after many repetitions with the new task settings, they can acquire a new behavioral habit with the MF system.

An illustrative example is the case where someone has to visit a new city. In that case, people usually look at a map, which is an allocentric representation of the city, and try to remember the parts of the maps that are useful to reach a desired location. And then, once people walk through the city, if they suddenly find themselves in front of some landmark that they thought would not be encountered during their planned trip (e.g., a monument), they can close their eyes and try and understand where they might actually be located within their mental map, and which change in their trajectory they should operate. This is typically a MB inference process. In contrast, when one always takes the same path from their home until their workplace, they rarely perform MB inference, and rather let their body automatically turn at the right corner and lead them to their usual arrival point. This works well even if one is discussing with a friend while walking, or is not fully awake. We thus think that in such a case the brain has shifted its decisions to the MF system. This permits to free other parts of the brain which can be used to think while we walk about the last book we read, or to try and solve the maths problem we are currently addressing.

One initial computational proposal for the coordination of MB and MF learning systems which can well capture this dynamics consists in comparing the uncertainty of the MB and MF systems and relying on the most certain one [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF]. This can be achieved if a Bayesian formulation of RL is adopted where the agent does not simply learns point estimates of state-action value functions, but rather full distributions over each (state,action) pair value. In that case, the precision of the distribution can be used to represent the level of uncertainty. In practice, when facing a new task, the uncertainty in both systems is high. But the uncertainty in the MB system decreases faster with learning (i.e., after less observations made following interactions of the agent with the world, even if these observations are processed during a long inference phase). As a consequence, the agent will rely more on the MB system during early learning. In parallel the uncertainty in the MF system slowly decreases, until the MF system is sufficiently certain to take control over the agent's actions. When the task changes (e.g., the goal location changes, or a path is now obstructed by an obstacle), uncertainty re-increases in both systems, but again it decreases faster in the MB system, so that again a sequence of MB decisions followed by MF decisions after a long second learning phase can be produced.

However, systematically monitoring uncertainty in the MB system can be computational heavy, and does not really permit the avoidance of the costly computations of the MB system when the MF system is currently leading. Alternatively, a more recent computational neuroscience model proposes to only monitor uncertainty within the MF system, and considers that the MB system is by default providing perfect information, so that it should be chosen when the MF system is uncertain, and avoided only when the MF system is sufficiently certain [START_REF] Keramati | Speed/accuracy tradeoff between the habitual and the goal-directed processes[END_REF]. This works well in a number of situations and enables to well capture the behavior of animals in a number of experiments. However, there are situations where this assumption cannot be true. In particular, as we will illustrate with some robotic tests of this kind of models in the next section, if the agent has an inaccurate internal model, it is better to rely on the more reactive MF system even when it is still uncertain [START_REF] Caluwaerts | A biologically inspired meta-control navigation system for the psikharpax rat robot[END_REF][START_REF] Renaudo | Respective advantages and disadvantages of model-based and model-free reinforcement learning in a robotics neuro-inspired cognitive architecture[END_REF]. Less costly alternative estimations of uncertainty can be used to permanently monitor both the MB and the MF system. For instance, the degree of instability of Q-values (MB or MF) before convergence is reached can be a good proxy to uncertainty [START_REF] Cazé | Hippocampal replays under the scrutiny of reinforcement learning models[END_REF][START_REF] Khamassi | Modeling awake hippocampal reactivations with model-based bidirectional search[END_REF]. Choice confidence can also give a relatively good proxy to choice uncertainty in simple situations, by measuring the entropy of the probability distribution over all possible actions in a given state and comparing MB and MF estimations of this measure [START_REF] Viejo | Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning[END_REF]. This moreover enables to well capture not only choices made by human subjects during simple decision-making tasks, but also their decision time (the more uncertain they are, the more time they need to make their decision). Finally, this type of mechanism also enables to explain why the ideal MB-to-MF sequence is not always true, since early choices of human subjects can sometimes significantly rely on the MF system because their MF system might initially be overconfident [START_REF] Viejo | Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning[END_REF].

Another important current question is whether uncertainty alone is sufficient to arbitrate between MB and MF systems [START_REF] Philip | Why and how the brain weights contributions from a mixture of experts[END_REF], or whether, when the two systems are equally uncertain, the agent should rely on the least computationally costly one [START_REF] Dromnelle | How to reduce computation time while sparing performance during robot navigation? a neuro-inspired architecture for autonomous shifting between model-based and model-free learning[END_REF]. If we want the agent to be initially agnostic about which system is more costly, and if we even want the agent to be able to potentially arbitrate between N different learning systems with different a priori unknown computational characteristics, then one proposal is simply to measure the average time taken by each system when it has to make decisions [START_REF] Dromnelle | How to reduce computation time while sparing performance during robot navigation? a neuro-inspired architecture for autonomous shifting between model-based and model-free learning[END_REF]. In some of the robotic experiments that we will describe in the next section, we found that this principle works robustly, enables to produce the ideal MB-to-MF sequence, not only during initial learning but also after a task change. We will come back to this later.

Finally, other current outstanding questions are whether the two systems shall always be in competition, or whether they shall sometimes also cooperate (as can be achieved with the weighted sum of their contribution described above); and whether an efficient coordination mechanism shall arbitrate between MB and MF at each timestep from the current available measures (e.g., uncertainty, computational cost, etc.), or whether it is sometimes more efficient to learn and remember that the MF system is usually better in situation type A while the MB system is better in situation type B. The latter could enable the agent to instantaneously rely on the best memorized system without needing

Model-based Strategy

Fig. 1. The generic architecture for the coordination of multiple learning strategies applied to navigation proposed by Dollé and colleagues [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF]. Three main learning strategies are considered here (but the paper tests other variants, such as the combination of multiple instances of the same strategy, which would correspond to a case of ensemble learning [START_REF] Wiering | Ensemble algorithms in reinforcement learning[END_REF]): a model-based planning strategy; a model-free habitual strategy; and a distinct random exploration strategy in order to avoid cumulating the exploratory decisions of the two other strategies. A so-called 'meta-controller' performs the high-level strategy coordination process. This process can consists in different manners of coordinating strategy, such as giving the hand to the least uncertain one [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF]. Nevertheless, in [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF] the meta-controller learns which strategy yields the largest amount of reward in different locations of the environment or in the presence of different stimuli. Adapted from [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF].

to fully experience a new situation identified as belonging from a recognized type. This issue relates to current investigations in subfields of machine learning known as transfer learning, life-long learning and open-ended learning [START_REF] Thrun | Lifelong learning algorithms[END_REF][START_REF] Ruvolo | Ella: An efficient lifelong learning algorithm[END_REF][START_REF] Doncieux | Dream architecture: a developmental approach to open-ended learning in robotics[END_REF].

One solution to this coordination memory problem consists in adopting a hierarchical organization where a second, higher-level, learning process (in what Dollé and colleagues call a 'meta-controller' ) learns which strategy (model-based, model-free or random exploration) is the best (in terms of the amount of reward it yields) in different parts of the environment [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF] (Fig. 1). This model learns through RL which strategy is the most efficient in each part of the environment. It can moreover learn that a certain equilibrium between MB and MF processes is required for good performance, thus resulting in cooperation between systems. It can even learn to change through time the weight of the contribution of each system, as learning in the MF system progresses, thus producing something that looks like the ideal MB-to-MF sequence.

With these principles in hand, the Dollé model [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF] can explain a variety of rat navigation behaviors experimentally observed, including data that initially appeared as either contradictory to the cognitive map theory, or contradictory to the associative learning theory which approximately considers that navigation behaviors shall all be learned through model-free reinforcement learning. Finally, it is worthy of note that performing offline replay of the MB system can result in learning by observation in the MF system, so that the two somehow cooperate [START_REF] Cazé | Hippocampal replays under the scrutiny of reinforcement learning models[END_REF], as inspired by the now classical DYNA architecture [START_REF] Richard S Sutton | Integrated architectures for learning, planning, and reacting based on approximating dynamic programming[END_REF].

Overall, this short review highlights that the investigation of the principles underlying the adaptive coordination of model-based and model-free reinforcement learning mechanisms in humans and animals is currently an active area of research in neuroscience.

Robotic tests of bio-inspired principles for the coordination of model-based and model-free reinforcement learning

The importation of these bio-inspired ideas to robotics is quite recent, and is still an emerging field of research. Nevertheless, a few studies and their outcomes deserve to be mentioned here.

To our knowledge, the first test with a real robot of a bio-inspired algorithm for the online coordination of model-based and model-free reinforcement learning has been presented in [START_REF] Caluwaerts | A biologically inspired meta-control navigation system for the psikharpax rat robot[END_REF]. This work included an indoor robot navigation scenario with the Psikharpax rat robot [START_REF] Meyer | The psikharpax project: Towards building an artificial rat[END_REF] within a 2mx2.5m arena (Fig. 2). The robot first explored the environment to autonomously learn a cognitive map of the environment (hence a mental model used by its model-based learning strategy). In addition, the robot could use a model-free reinforcement learning strategy to learn movements in 8 cardinal directions in response to perceived salient features within the environment (i.e., stimuli in the vocabulary of psychology). The latter MF RL component of the model was later improved in [START_REF] Caluwaerts | Neuro-inspired navigation strategies shifting for robots: Integration of a multiple landmark taxon strategy[END_REF] to make it able to learn movements away from visual features when needed. The proposed algorithm for the online coordination of MB and MF RL was based on the computational neuroscience model of Dollé and colleagues [START_REF] Dollé | Analyzing interactions between navigation strategies using a computational model of action selection[END_REF][START_REF] Dollé | Path planning versus cue responding: a bio-inspired model of switching between navigation strategies[END_REF][START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF], which has been presented in the previous section and sketched in Fig. 1.

The first important result of this robotic work is that the algorithm could autonomously learn the appropriate coordination of MB and MF systems for each specific configuration of the environment that was presented to the robot. For a first configuration associated to an initial goal location (where the robot can obtain a scalar reward), the algorithm learned that the MB strategy was appropriate to guide the robot from far away towards the goal, and that the MF strategy was appropriate to control the fine movements of the robot when closer to the goal. This was an emergent property of the coordination that was not designed by human. Instead, it was autonomously learned by the algorithm in response to the specific environment where the robot was located. The reason is that the robot had less explored the area around the initial goal location. Fig. 2. Robotic experiments presented in [START_REF] Caluwaerts | A biologically inspired meta-control navigation system for the psikharpax rat robot[END_REF] and aiming at testing the performance of a bio-inspired algorithm for the online coordination of model-based and model-free reinforcement learning. The algorithm itself is based on the computational neuroscience model of Dollé and colleagues [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF], which is presented in Fig. 1. The photo shows the Psikharpax rat robot [START_REF] Meyer | The psikharpax project: Towards building an artificial rat[END_REF] within a 2mx2.5m indoor arena. Salient features are displayed on the surrounding walls and objects because the purpose of this research work was not on vision, but how to make the robot use the noisy perception of these features with its cameras in order to learn a simple cognitive map that can be used for model-based navigation. The green and red dots superimposed on the photo show the location of the place cells of the cognitive map learned by the robot. The blue dot (invisible to the robot) shows the current location of the goal, where the robot receives a scalar reward during the first part of the experiment. Later on this goal location is moved without informing the robot, so that it needs to detect this change and learn a new appropriate coordination of model-based and model-free strategies to quickly reach a good performance. In the last part of the experiment, the goal is again moved back to its initial location, to show that the coordination algorithm can, after detecting this new change, instantaneously retrieve its memory of the first appropriate coordination and thus quickly re-display a good performance without re-learning. Finally, the green dots are labelled 'taxon' (a taxon strategy in neuroscience consists in learning actions in response to visual cues), which corresponds to areas where the algorithm considers the model-free strategy as the best strategy, while red dots are labelled 'planning' because the algorithm considers that the model-based strategy is the most appropriate there. Adapted from [START_REF] Caluwaerts | A biologically inspired meta-control navigation system for the psikharpax rat robot[END_REF].

Thus, its cognitive map was less precise there. As a consequence, a pure MB version of the algorithm could learn to approach the robot near the goal, but could not learn to precisely reach it (because of the imprecision in the map). As a consequence, the autonomous coordination algorithm found out that the MF system could compensate for this lack of precision. From this simple example we can learn two things: first, that in contradiction to the assumption made by some previously discussed computational models that the MB system has access to perfect information, the map (i.e., model) learned by the MB system can be imperfect, and the coordination algorithm has to cope with it. More generally, we think that when experimenting with robots, there will always be a situation where the map cannot be accurately learned, because of noisy perceptions, problems with the light, etc. So, rather than endlessly trying to refine the MB system to make it appropriate for each new situation at hand, it might be better to let the coordination algorithm autonomously find out what is the appropriate alternation between MB and MF for the present situation. The second thing that we can learn from this example is that a simple coordination algorithm which puts MB and MF systems in competition, and selects the most efficient one, can sometimes produce a sort of cooperation between them. In this particular example, a learned trajectory of the robot to the goal can be hybrid, involving a first use of the MB strategy when far away from the goal, and then a shift to the MF strategy when getting closer. This enables us to draw a model-driven prediction for neuroscience: that sometimes animals solving these types of task may display a trajectory within a maze or an arena that is not the result of a single learning system, but rather a hybrid byproduct of the coordination of multiple systems.

Another important result of this robotic work relates to its ability to learn context-specific coordination patterns, which can relate to what people call episodic control. This occurred when we changed the goal location after some time, and let the robot adapt to the new configuration. What happened is that the algorithm first detected the change because of the different profile of reward propagation through its mental map that this induced. Then the algorithm decided to store in memory the previously learned coordination pattern between MB and MF, and to learn a new one. After a new learning phase, the algorithm found a new coordination pattern adapted to the new condition, thus producing good performance again. Finally, we suddenly moved the goal location back to its initial location. The algorithm could detect the change and recognize the previous configuration (again thanks to the profile of reward propagation through its mental map). As a consequence, the algorithm retrieved the previously learned coordination pattern, which enabled the robot to instantaneously restore the appropriate behavior without re-learning.

Nevertheless, some limitations and perspectives of this seminal work ought to be mentioned here. First, the coordination component of the algorithm (which is called the meta-controller in [START_REF] Dollé | Interactions of spatial strategies producing generalization gradient and blocking: A computational approach[END_REF][START_REF] Caluwaerts | A biologically inspired meta-control navigation system for the psikharpax rat robot[END_REF][START_REF] Caluwaerts | Neuro-inspired navigation strategies shifting for robots: Integration of a multiple landmark taxon strategy[END_REF]) slowly learns through MF RL (in addition to the MF RL mechanism used within the MF system dedicated to the MF strategy) which strategy is the most appropriate in each part of the environment (In other words, the model involves a hierarchical learning process in addition to the parallel learning process between MB and MF strategies). While this is good for the robot to be able to memorize specific coordination patterns for each context (i.e., for each configuration of the goal location within the arena), this nevertheless requires a long time to achieve a good coordination within each context. Thus, it would be interesting to also test coordination mechanisms based on instantaneous measures such as uncertainty, as discussed in the previous section. A second limitation is that this experiment involved a specific adaptation of a coordination model to a simple indoor navigation task, with a small map, a small number of states to learn, and an action repertoire which is specific to navigation scenarios. A third limitation is at the technical level, involving an old custom robot. Thus, it it not clear if these results could be generalized to other robotic platforms facing a wider variety of tasks, and sometimes more complex tasks involving a larger number of states. Red numbers are starting location indexes; blue numbers are some states where some changes in the configuration of the environment can occur. Adapted from [START_REF] Chatila | Toward self-aware robots[END_REF].

A more recent series of robotic experiments with the same research goal (i.e., assessing the efficience and robustness of bio-inspired coordination principles of MB and MF learning) has been presented in [START_REF] Renaudo | Design of a control architecture for habit learning in robots[END_REF][START_REF] Renaudo | Which criteria for autonomously shifting between goal-directed and habitual behaviors in robots?[END_REF][START_REF] Renaudo | Respective advantages and disadvantages of model-based and model-free reinforcement learning in a robotics neuro-inspired cognitive architecture[END_REF] and later in [START_REF] Chatila | Toward self-aware robots[END_REF][START_REF] Dromnelle | How to reduce computation time while sparing performance during robot navigation? a neuro-inspired architecture for autonomous shifting between model-based and model-free learning[END_REF][START_REF] Dromnelle | Coping with the variability in humans reward during simulated human-robot interactions through the coordination of multiple learning strategies[END_REF]. First, [START_REF] Renaudo | Design of a control architecture for habit learning in robots[END_REF][START_REF] Renaudo | Which criteria for autonomously shifting between goal-directed and habitual behaviors in robots?[END_REF][START_REF] Renaudo | Respective advantages and disadvantages of model-based and model-free reinforcement learning in a robotics neuro-inspired cognitive architecture[END_REF] compared different coordination principles, including methods coming from ensemble learning [START_REF] Wiering | Ensemble algorithms in reinforcement learning[END_REF] in several different simulated robotic tasks. They found again that the MB system was not always the most reliable system, especially in tasks with hundreds of states, where the MB system requires long inference durations to come up with a good approximation of the state-action value function Q. These experiments highlighted the respective advantages and disadvantages of MB and MF reinforcement learning in a variety of simulated robotic situations, and concluded again for the added value of coordinating them. In [START_REF] Chatila | Toward self-aware robots[END_REF][START_REF] Dromnelle | How to reduce computation time while sparing performance during robot navigation? a neuro-inspired architecture for autonomous shifting between model-based and model-free learning[END_REF][START_REF] Dromnelle | Coping with the variability in humans reward during simulated human-robot interactions through the coordination of multiple learning strategies[END_REF], simulated and real robotic experiments were presented, some involving navigation with a Turtlebot, and others involving simulated tasks with the PR2 robot and the Baxter robot (Fig. 3).

The first important result of these new series of experiments to highlight is that the coordination of MB and MF RL was efficient in a variety of tasks, including navigation tasks with detours, non-stationarity of the configuration of the environment (i.e., sudden introduction of obstacles obstructing some corridors), but also simple human-robot interaction tasks. In the latters, the human and the robot had to cooperate to clean a table by putting objects in a trashbin. Importantly, some objects were reachable by the human, some by the robots, thus forcing them to communicate and cooperate. In that case, the model-based system could compute joint action plans where actions by the robot and actions by the humans alternated. In all these situations, the robot could autonomously learn the task and reach good performance.

The second important result to highlight is that instantaneous measures of uncertainty in MB and MF systems allow a quicker reaction of the coordination mechanism to changes in the environment. Nevertheless, this does not permit memorization nor episodic control, which the work of [START_REF] Caluwaerts | A biologically inspired meta-control navigation system for the psikharpax rat robot[END_REF] did. Thus, the results are in good complementarity and in the future it would interesting to test combinations of these two principles.

The last important result to highlight here is that an efficient coordination mechanism proposed by [START_REF] Dromnelle | How to reduce computation time while sparing performance during robot navigation? a neuro-inspired architecture for autonomous shifting between model-based and model-free learning[END_REF][START_REF] Dromnelle | Coping with the variability in humans reward during simulated human-robot interactions through the coordination of multiple learning strategies[END_REF], and successfully applied to robot navigation and human-robot interaction scenarios, consists in taking into account not only the uncertainty but also the computational cost of each learning system. In practice, the proposed algorithm monitored the average time taken by each system to make its inference phase before deciding. It learned that the MB system takes on average 10 times longer than the MF system, in these specific tasks, before making a decision. As a consequence, the coordination algorithm gave the lead to the MF system in cases of equal uncertainty, and even in cases of slightly higher uncertainty in the MF system. As a result, the algorithm mostly relied on the MF system but transiently and efficiently gave the lead to the MB system, only when needed. This occurred during initial learning as well as after task changes. As a consequence, the algorithm could reproduce the nice MB-to-MF sequence that we discussed in previous sections, both during initial learning and after task changes. Moreover, with this new coordination principle, the robot could achieve the same optimal performance as a pure MB system (which was optimal in these cases) while requiring a cumulated computational cost which was closer to that of a pure MF system (which achieves a lower bound on computational cost in these experiments). Thus, the coordination algorithm not only allowed for an efficient and flexible behavior of the robot in these non-stationary tasks, but it also permitted to reduce the computational cost of the algorithm controlling the robot. Finally, the authors also compared their algorithm with a state-of-theart deep reinforcement learning algorithm. They found that the latter requires a very large number of iterations to learn, much more than their proposed MB-MF coordination algorithm.

Finally, it is interesting to mention that in the meantime, several other research groups throughout the world have also started to test hybrid MB/MF algorithms for robot learning applications [START_REF] Jauffret | From self-assessment to frustration, a small step toward autonomy in robotic navigation[END_REF][START_REF] Maffei | An embodied biologically constrained model of foraging: from classical and operant conditioning to adaptive real-world behavior in dac-x[END_REF][START_REF] Santos-Pata | Navigate the unknown: Implications of grid-cells 'mental travel' in vicarious trial and error[END_REF][START_REF] Hangl | Skill learning by autonomous robotic playing using active learning and creativity[END_REF][START_REF] Llofriu | A computational model for a multi-goal spatial navigation task inspired by rodent studies[END_REF][START_REF] Marcela Rojas-Castro | Rhizome architecture: An adaptive neurobehavioral control architecture for cognitive mobile robots' application in a vision-based indoor robot navigation context[END_REF]. In particular, the deep re-inforcement learning community is showing a growing interest for such hybrid learning algorithms [START_REF] Chebotar | Combining model-based and model-free updates for trajectory-centric reinforcement learning[END_REF][START_REF] Nagabandi | Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning[END_REF][START_REF] Muhammad Burhan Hafez | Curious meta-controller: Adaptive alternation between model-based and model-free control in deep reinforcement learning[END_REF]. This illustrates the potentially broad interest that this type of hybrid solutions to reinforcement learning can have in different research communities.

Conclusion

This paper aimed at first illustrating current outstanding questions and investigations to better understand and model neural mechanisms for the online adaptive coordination of multiple learning strategies in humans and animals. Secondly, the paper reviewed a series of recent robot learning experiments aimed at testing such bio-inspired principles for the coordination of model-based and model-free reinforcement learning strategies.

We discussed the respective advantages and disadvantages of different coordination mechanisms: on the one hand, mechanisms relying on instantaneous measures of uncertainty, choice confidence, performance, as well as computational cost; on the other hand, mechanisms relying on hierarchical learning where a high-level meta-controller autonomously learns which strategy is the most efficient in each situation.

The robotic experiments discussed here showed that this type of coordination principle can work efficiently, robustly and at a reduced computational cost in a variety of robotic scenarios (navigation, human-robot interaction). This is of particular importance at a time where energy saving is a critical issue for the planet and to slow down global warming. In contrast, many current machine learning techniques, especially those relying on deep learning, require tremendous amounts of energy and long pre-training phases.

Finally, the paper aimed at also illustrating the interest of testing neuroinspired models in real robots interacting with the real world so as to generate novel model-driven predictions for neuroscience and psychology. In the particular case of the adaptive coordination of model-based and model-free reinforcement learning strategies, we showed that some situations can induce cooperation between learning strategies. We moreover showed that not only taking into account the uncertainty of each learning system but also its computational cost could work efficiently in a variety of task. This raises the prediction that the mammalian brain may also monitor and memorize the average computational cost (for instance in terms of the duration required for inference) of different learning strategies in different memory systems, in order to favor those which cost less when they are equally efficient. This paves the way for novel neuroscience experiments aimed at testing these new model-driven predictions and understanding the underlying neural mechanisms.

Fig. 3 .

 3 Fig. 3. (Left) Human-Robot Interaction task tested in [60]: the human and the robot collaborate to put all boxes in a trashbin. (Right) Navigation task autonomously mapped and discretized by the robot during exploration [60, 50]. The red area indicates the goal location whereas the green areas indicate starting locations of the robot.Red numbers are starting location indexes; blue numbers are some states where some changes in the configuration of the environment can occur. Adapted from[START_REF] Chatila | Toward self-aware robots[END_REF].

Acknowledgments

The author would like to thank all his collaborators who have contributed through the years to this line of research. In particular, Andrea Brovelli, Romain Cazé, Ricardo Chavarriaga, Laurent Dollé, Benoît Girard, Agnes Guillot, Mark Humphries, Florian Lesaint, Olivier Sigaud, Guillaume Viejo for their contribution to the design, implementation, test, and analysis of computational models of the coordination of learning processes in humans and animals. And Rachid Alami, Lise Aubin, Ken Caluwaerts, Raja Chatila, Aurélie Clodic, Sandra Devin, Rémi Dromnelle, Antoine Favre-Félix, Benoît Girard, Christophe Grand, Agnes Guillot, Jean-Arcady Meyer, Steve N'Guyen, Guillaume Pourcel, Erwan Renaudo, Mariacarla Staffa for their contribution to the design, implementation, test and analysis of robotic experiments aimed at testing neuro-inspired principles for the coordination of learning processes.

Funding

This work has been funded by the Centre National de la Recherche Scientifique (CNRS)'s interdisciplinary programs (MITI) under the grant name 'Hippocampal replay through the prism of reinforcement learning'.

Disclosure/Conflict-of-Interest Statement

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.