Souheïl Hanoune

Philippe Gaussier

Mathias Quoy

Cortical chunks learning for action selection in a complex task

For low level behaviors, navigational trajectories can be encoded as attraction basin resulting from associations between visual based localization and directions to follow. The use of other sensory information such as contexts for modifying the behavior needs a specialized learning. In this paper, we propose a minimal model using multimodal contexts, and a mechanism for obtaining a better generalization of the contexts and creation of chunks. We briefly present the bases of the sensory-motor architecture, and explain the neurobiological principal inspiring this model. We also evaluate the proposed improvement on simulated signals and in a robotic navigational experiment.

I. INTRODUCTION

Action selection is wide and complex field. An "action" can represent notions ranging from high level abstract procedures ("pour water in a glass") to low level motor commands ("move arm joint with chosen speed").

Before the 80's, the action selection problem was solved by executing a sequence of steps, obtained by decomposing the plan or the path to achieve the goal. After the mid 80's, a shift for reactive intelligence and behavior based models [START_REF] Brooks | A robust layered control system for a mobile robot[END_REF] occurred. This module based approach depends on the fast reactivity of the modules. The organization of these modules can be in parallel [START_REF] Maes | Learning to Coordinate Behaviors[END_REF], or hierarchical [START_REF] Tyrrell | Computational mechanisms for action selection[END_REF]. The resolution of an action selection problem switches to a module selection problematic that can be solved by specializing the design of the hierarchy between modules [START_REF] Bryson | Hierarchy and sequence vs. full parallelism in action selection[END_REF] depending on the experiment or the studied situation. However, this limitation could be overpassed by learning the activations of the different modules. In [START_REF] Maes | Learning to Coordinate Behaviors[END_REF], the modules controlling the behavior encode the action to perform, the condition to perform this action, and the outcome of the action. The action is learned, by associating the direction with the sensory inputs configuration, when the correlation between the sensory inputs and the outcome is learned. Moreover, by integrating the activities of the inputs for multiple occurrences of the same condition, a better generalization of the condition can be achieved.

Another paradigm for the action selection problem is reinforcement learning (RL) [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. Relying on neurobiological results, RL is based on a computational modeling of corticobasal loops (especially the cortico-striatal loops). The properties of dopamine neurons indicates that they could implement some kind of reinforcement learning mechanism [START_REF] Schultz | A neural substrate of prediction and reward[END_REF]. The Basal Ganglia, organized in parallel loops (potentially coding action's evaluation) with mutual inhibition, can perform the competition necessary for the action selection [7] [8]. The different levels of cortico-basal loops, from sensorimotor to more cognitive loops [START_REF] Alexander | Parallel organization of functionally segregated circuits linking basal ganglia and cortex[END_REF], even suggest that this very structure could manage the different actions mentioned above. However, both the reinforcement learning and the correlation learning are quite slow to learn. During an interaction, the behavior should be adapted quickly.

In this paper, we are interested in the task of Figure 1 where a robot must navigate to different places depending on its situation. The robot takes small or big objects in the picking place P and has to choose between two dropping places (A or B), in order to release the objects. In a previous work, we proposed an architecture to solve such an approach [START_REF] De Rengervé | Building specific contexts for on-line learning of dynamical tasks through non-verbal interaction[END_REF]. The robot uses place cell/action associations, inspired by a model of the hippocampus in order to obtain visual place cells (PC) [START_REF] Okeefe | The hippocampus as a cognitive map[END_REF] that allowed controlling mobile robots for visual navigation tasks [START_REF] Giovannangeli | Robust mapless outdoor vision-based navigation[END_REF]. Those associations allow the robot to create attraction basins, and thus, shape a trajectory allowing the navigation using visual cues and the orientation information 1 .

The robot selects the adequate actions (moving directions i.e. low level actions) according to the current sensory inputs. The behavior's learning is based on an interaction with a human teacher, that corrects the actions when wrong, allowing a better and faster acquisition of the behavior as shown in [START_REF] Giovannangeli | Interactive teaching for visionbased mobile robots: A sensory-motor approach[END_REF].

Because the correction is provided only when the action is wrong, the robot must capture this sensory configuration in order to avoid the wrong action in the future. In this article, the focus is put on the categorization of the multimodal sensory information, and on the creation of contexts enabling the action selection. We define a context as the combination of sensory information at a particular moment, i.e. an association between the actual PC (what the robot sees), the griper information and the ultrasound profile (what the robot "feels like touching").

We propose a bio-inspired neural architecture of context learning and generalization to solve the action selection problem. It is based on the improvement of the model presented in [START_REF] De Rengervé | Building specific contexts for on-line learning of dynamical tasks through non-verbal interaction[END_REF] by allowing the system to have a better adaptation for the contexts. In the past architecture, the sensory inputs configuration describing a context is learned once and never adapted afterwards. By adapting the context, we provide a better description of the discriminant information, and thus, the contexts can be reused and adapted if the sensory information varies in time (due to noise in the sensors, or a modification in the environment conditions). In section II, the sensorymotor model for the navigation is described and the context learning is explained. In section III, the bio-inspired model is presented and the computational details of the algorithm are exposed. In section IV, an example of the system behavior for a simplified simulated simulation is presented, and a behavioral presentation of the robotic simulation is given. Finally, section V discusses the limitations of the model and proposes future improvements. The navigational model is based on sensory motor associations, i.e. PC-action association. Following a trajectory can be encoded as an attraction basin emerging from multiple placecell/action pairs as in fig 2 . When the robot moves too far from the desired trajectory, it is corrected (redirected) to get back on the desired trajectory by forcing its orientation2 . This corrected orientation is associated with a new learned PC, completing the encoded attraction basin. In order to learn the task of Figure 1, the object size is the critical information for discriminating the context in which the correction occurs. However, the system has no prior knowledge that the size of the object being relevant to the action selection.

II. SENSORY-MOTOR MODEL FOR NAVIGATION AND MUTIMODEL INHIBITORY CONTEXTS FOR ACTION

The robot uses the visual information for the localization (by creating its PCs). The action selection is done on this level: by detecting a wrong action, the context is learned in order to inhibit the winning PC. By doing so, the associate action is no longer expressed, and other actions can be implemented. When the PCs are no longer sufficiently recognized, a new PC (and also a new PC-action association) is learned. In the figure 3, the "wrong action detection" module detects when the actual action (direction) is wrong (by evaluating the difference between this action and the desired one). When the difference is over a prefixed threshold β, a recruitment signal is sent to the context management module, in order to evaluate the PCs inhibitions and/or recruitment. If no context is recognized, the sensory input configuration is associated with a newly recruited context, and the winning PC is inhibited. The "Action to inhibit" signal is then sent to the PC-action associations in order to inhibit the winning PC (more details on the model are available in [START_REF] De Rengervé | Building specific contexts for on-line learning of dynamical tasks through non-verbal interaction[END_REF]).

III. CORTICO-BASAL BASED MODEL OF CHUNCKING AND CONTEXT LEARNING

The theory of chunking was first introduced in the 1950s by DeGroot [START_REF] De Groot | Thought and choice in chess[END_REF] and Miller [START_REF] Miller | The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information[END_REF]. The main idea is that a chunk collects pieces of information in order to obtain a higher level of information coding. It was applied in lowlevel visiomotor learning [START_REF] Kuniyoshi | From visuo-motor self learning to early imitation -a neural architecture for humanoid learning[END_REF], and in high level interactions, like talking or singing [START_REF] Yamashita | Developmental learning of complex syntactical song in the bengalese finch: A neural network model[END_REF]. In a previous work [START_REF] Hanoune | An architecture for online chunk learning and planning in complex navigation and manipulation tasks[END_REF], we suggested that a modified version of Schmajuk's and DiCarlo's learning of conditioning [START_REF] Schmajuk | Stimulus configuration, classical conditioning, and hippocampal function[END_REF] could model the cortico-basal loop with associative conditioning for chunks learning. Here, we use recruited neurons to encode a categorization. Moreover, population coding as in dynamic field activity [START_REF] Durán | A dynamic field architecture for the generation of hierarchically organized sequences[END_REF] can be used to store the categorization.

As shown in [START_REF] Hanoune | An architecture for online chunk learning and planning in complex navigation and manipulation tasks[END_REF], a limitation in the chunk creation mechanism is the generalization of the categories. To overcome this barrier, we use an estimation of the inputs variance to adapt the learned contexts. This is an implementation of the cortical neurons capacity, allowing this kind of estimation by population coding [START_REF] Stevens | Input synchrony and the irregular firing of cortical neurons[END_REF] [START_REF] Lukashin | Modeling of directional operations in the motor cortex: a noisy network of spiking neurons is trained to generate a neural-vector trajectory[END_REF]. By evaluating the changes in the input neurons, the system can estimate the stable sensory information, and ignore the most variant ones. This estimation is done in a time window, allowing the learning to adapt to different variations in the categories.

The chunk creation is based on the cortico-basal loops described in [START_REF] Alexander | Parallel organization of functionally segregated circuits linking basal ganglia and cortex[END_REF]. The hippocampus takes sensory information as inputs and provides the basal ganglia with sensory situations inducing the actions selection. The prefrontal cortex creates context (when needed), capturing a certain input configuration, in order to bias the action selection on the basal ganglia level (see fig: 4).

A. Variance estimation for stable generalization

The contextual chunk learning is inspired from the Adaptative Resonance Theory [START_REF] Carpenter | Adaptive resonance theory[END_REF]. The basic idea is that the knowledge is stored in the neurons or their weights based on the inputs. When a change in the inputs configuration is detected (by an error estimation of some sort) a new category, i.e. a neuron, is recruited to encode this new configuration. In our case, the learning of a new category is achieved by recruiting a new neuron and associating the input activities with the corresponding weights ω i,j connecting this recruited to the inputs. When the maximum activity is below a vig-ilance level 3 , the recruitment is triggered. When a neuron is recruited, knowing that the neuron is connected to all the neurons of the previous group, the activities on each input neuron is copied on the input link. Thus, the value on the link between the recruited neuron j and the neuron i of the previous group (with the activity E i ) is:

ω i,j = E i (1)
note that the neuron j is connected to all the neurons of the previous group. The modification of the weights for each neuron is done by computing ∆ω, the amount by which the weight modified:

∆ω i,j = ǫ • (E i -ω i,j ) • (1 -A j ) (2)
where ǫ is a learning rate for the adaptation, and A j the activity of the present neuron j in this group. The modification of the weights is computed by the equation 3:

ω i,j = ω i,j + ∆ω i,j (3)
This modification depends on the activity of the winning neuron that is going to be adapted. The output activation of the neuron A j is computed in equation 4:

A j = 1 -dist vari (4)
The activity of the neuron is maximal when the weights values are equals to the inputs neurons values. The term var i div i

Stable input occurence

Unstable input occurence

Variation zone 0 1 τ Fig. 5. Sketch explaining the application of the exponential decrease. The stable red input induce a divergence div i close to 0, and thus, maintains its importance for the activity of the neuron. The blue input is less stable, and the divergence is important enough so that it implication is mostly lost, i.e. its weight drops towards 0. dist vari represents the normalized distance of the neuron according to the accumulated variation on all its input links. The accumulated distance dist of the neuron is evaluated in eq 5:

dist = (var i • |ω i,j -E i |) (5)
The basic idea is that the estimation is based on the distance between the actual weight value and the input value (the initial weight learned). The inputs that fluctuate a lot (and thus not informative) produce a big error, thus their implication in the activity of the neuron activity should be minimized. To obtain such a behavior, instead of using directly the divergence from the expected input configuration div i (see equation6, the variation var i of each link is used by passing the divergence through a Gaussian with a standard variation ofτ , as presented in equation7.

div i = div i + ζ • (|A j -ω i,j | -div i ) (6)
Where ζ is a variance rate, i.e. a parameter specifying the importance of the distance from the inputs for each step.

var i = e - div 2 i 2•τ (7)
The result is, when an input fluctuates with an important variation, the variance estimation var i decreases quickly to zero (depending on the τ parameter). Therefore, the link weight drops down, and the importance of the modality associated to the neuron connected to the link (eq4) decreases. The figure 5 explains how the application of the exponential decrease affects the weights.

IV. VISUALIZATION OF CONTEXT GENERALIZATION AND BEHAVIORAL EXPERIMENT

In this section we evaluate the algorithm behavior in a simulated situation and analyze the system's outputs. In a second evaluation, behavioral results regarding a navigational experiment are presented. The evaluation is based on the behaviors and the emergent capacities of the model.

A. Context learning and adaptation based on sensory inputs variance

A simplified simulation presents the capacities of the model. For this test, we provide simplified simulated signals related to the desired test (the navigational experiment). In figure 6, simulated signal of the sensory inputs are fed to the model, and the responses are highlighted. The first subplot presents simulated activities for PCs, when a robot is navigating in a round or a square trajectory. The second and third plots are the gripper activity (stable all along the simulation) and the ultrasound activity. The three other plots represent, respectively, the output activity of the context, the variance (i.e. the error) of each modality and the normalized contribution of these modalities. The figure exposes the fast adaptation of the signal to correct values. The gripper activity (color magenta) being the more stable (with the less variance within the fifth subplot), its importance is quickly the most important and is stabilized that way. The sudden variation in the activity of the ultrasound values induce a small drop in the output activity (the same color in the last subplot). The adaptation is perceptible where the system drops the importance of the ultrasound until the activity is stabilized again.

Because this is simplified, and the number of inputs is low, the importance of each input signal is amplified. In a real situation, the number of the sensory inputs being more important, the stable inputs tend to have the most importance (i.e. their weights are maintained). On the other hand, if an input has a big variance, its weights are decreased, and its relevance for the output computation is reduced.

B. Behavioral results for the navigational experiment

The goal of this implementation is to perform a better generalization over the learning of the chunks. The evaluation of the navigational experiment is performed with a simulated robot, running with the Promethe simulator [START_REF] Lagarde | Learning New Behaviors : Toward a Control Architecture Merging Spatial and Temporal Modalities[END_REF] under the Webots (Cyberbotics) 3D environment. The figure 7 exposes the learned PC, contexts and directions for the task described in figure 1. It is clear that the algorithm enables the resolution of the task. As in [START_REF] Hanoune | An architecture for online chunk learning and planning in complex navigation and manipulation tasks[END_REF], PC are presented and contexts inhibiting certain PCs are described in the figure. The algorithm maintains the same performance, regarding the learning phase, i.e. this result is obtained after fifteen rounds with the small object and fourteen with the big one.

Another behavioral point is the emergence of attraction basins depending on the size of the object held by the robot. In figure 8, a learning phase in performed, and the robot starting point is randomly chosen in the environment. The robot can hold a big, a small or no object at all. The behavior is correct. However, it is highly dependent on the learning phase. Because the robot has no explicit knowledge about the environment (other than PC-action associations), when the starting point is unknown, the behavior can be extremely random. The objective was not to solve this question, nevertheless, it is important to expose this limitation. Another example of limitation is that the robot could choose the wrong dropping place (also relatively to the starting point). The red trajectories in the however, the robot chose the dropping place B twice. This is the result of an unlearned situation. Because in the learning phase, the robot takes the objects always in P, the robot was never corrected to the dropping place A when in the right side of the environment. It is also the case when the object is a big one, and the robot starts on the left side of the environment. To summarize, the system maintains the abilities inherited by the past architecture, when the improvements done on the context adaptation allow learning to remain continuous. In the past architecture, the contexts where learning with a one shot learning, and no adaptation was done afterwards. With the allowed adaptation, the system can continue to adapt the contexts in a continuous manner, realizing a chunk creation procedure based on the generalization of the contexts over the variance of the sensory inputs. The arrow starting from the dots indicates the orientations that are associated to the place-cell. Due to the effect of the chunks, the orientation may not be followed as the neuron selecting this orientation is inhibited. Fig. 8. Change in behavior according to the object's size. After the learning phase, the robot is lunched from a random position. The blue lines are the trajectories followed when the robot does not hold an object. The red trajectories are in the case of a small object. The gray ones are in the case of a big object.

V. DISCUSSION

In this paper we presented an improvement for an action selection model based on contexts guiding of sensory-motor associations. This approach allows us to extend the placecell/action model [START_REF] Giovannangeli | Robust mapless outdoor vision-based navigation[END_REF] to solve the action selection problem and the model presented in [START_REF] De Rengervé | Building specific contexts for on-line learning of dynamical tasks through non-verbal interaction[END_REF] for a better context learning.

In [START_REF] Giovannangeli | Interactive teaching for visionbased mobile robots: A sensory-motor approach[END_REF], the authors showed the influence of the teacher on the learned trajectory by comparing different of methods of correction. The contexts acquisition in [START_REF] De Rengervé | Building specific contexts for on-line learning of dynamical tasks through non-verbal interaction[END_REF] was rigid; and as in [START_REF] Giovannangeli | Robust mapless outdoor vision-based navigation[END_REF], the teacher's expertise is expected to have a big impact on the learning. The generalization model proposed in this article should reduce this dependency, as the the system adapts itself to the signals on an analysis level, i.e. by evaluating their importance, more than just expecting the teacher to be always correct. This point would be expressively addressed in future work, where the changes of the teacher would be evaluated. Also, for this work, the experiment was done on a simulated environment; testing on a real robot would evaluate the robustness of the system in a real dynamical environment.

The model presented in this paper does not completely solve the action selection dilemma. The goal is to propose a study to improve interactive learning with a teacher, by a neuro-biologically inspired model explaining fast adaptations occurring in the cortex, and contributing the cortico-basal loops for the action selection. The results show that the proposed principles are efficient to some extent. Currently, the robot must face the different situations that can occur to guarantee that it will behave correctly in any of these situations. The assumption is that the initial behavior encoding its learned actions (the ones that can be inhibited) are learned well enough for a correct generalization most of the time. A more robust learning should be achieved by associating the fast learning of our model with a slower one, assuring a better generalization over the variant situations during the learning.

Hints on future improvements for robotic systems can be deduced from the many neurobiological studies dedicated to the action selection. A more bio-inspired modeling of the cortico-baso-thalamo-cortical loop, exposed in [START_REF] Prescott | Introduction. modelling natural action selection[END_REF], is a logical conclusion for our model. Other models of the basal ganglia were presented for the action selection, especially the dynamical system approach as in [START_REF] Gurney | A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour[END_REF], or the CBTC model [START_REF] Girard | Integration of navigation and action selection functionalities in a computational model of cortico-basal-ganglia-thalamo-cortical loops[END_REF] that explicits the internal connectivity in the basal ganglia.

Fig. 1 .

 1 Fig.1. An example of a contextual action selection in a navigation task. By encoding the sensory context in the particular situations (picking place P, dropping places A and B), the created contexts can bias the action selection.

Fig. 2 .

 2 Fig. 2. Left Sketch of place/action associations. The association of multiple place/action associations can create attraction basins for a navigation task. Right Example of typical trajectory learned through interaction as an attraction basin emerging from several place-cell/orientation couples (black arrows).

Fig. 3 .Fig. 4 .

 34 Fig.3. Model of context recruitment mechanism and action inhibition. When a wrong action is detected (by computing the difference between the actual action and the wanted one) a recruitment/adaptation signal is sent the context management. If no active context is already responding, a new one is recruited and association to the sensory inputs. Otherwise, the context is adapted in order to obtain better generalization.

Fig. 6 .

 6 Fig. 6. Plot of the different signals used in the simulation of the model. The three first plots represent the sensory inputs: the PCs activities (for four of them), the gripper activation and the Ultrasounds activity. The fourth plot is the output activity of the context studied in this case. The two last ones are the variance and the normalized contribution of each modality. The colors in the variance plot and the normalized importance plot are correlated with the colors of the different input signals (the context plot is not an input signal).

figure 8

 8 figure 8 are supposed to converge to the dropping place A. however, the robot chose the dropping place B twice. This is the result of an unlearned situation. Because in the learning phase, the robot takes the objects always in P, the robot was never corrected to the dropping place A when in the right side of the environment. It is also the case when the object is a big one, and the robot starts on the left side of the environment. To summarize, the system maintains the abilities inherited by the past architecture, when the improvements done on the context adaptation allow learning to remain continuous. In the past architecture, the contexts where learning with a one shot learning, and no adaptation was done afterwards. With the allowed adaptation, the system can continue to adapt the contexts in a continuous manner, realizing a chunk creation procedure based on the generalization of the contexts over the variance of the sensory inputs.

Fig. 7 .

 7 Fig. 7. Trajectory (gray line) during reproduction of the task (several times consecutively). The colored dots are the learned place-cells. The chunks associated with the place-cells are represented around the corresponding dots.The arrow starting from the dots indicates the orientations that are associated to the place-cell. Due to the effect of the chunks, the orientation may not be followed as the neuron selecting this orientation is inhibited.

The visual information is provided by a pan-tilt camera mounted on the robot, the orientation information is given by an electronic compass.

In the experiments of this article, a joystick is used considering that it simulates the action of a leash

The vigilance is a neuromodulation level, when high the system is reactive and recruits easily, when low the system does not recruit and reacts by adapting the winning category

ACKNOWLEDGMENT

This work was supported by the NEUROBOT project ANR-BLAN-SIMI2-LS-100617-13-01.