Olivier L Georgeon
email: olivier.georgeon@liris.cnrs.fr

Christian Wolf
email: christian.wolf@liris.cnrs.fr

Simon Gay
email: simon.gay@liris.cnrs.fr

An Enactive Approach to Autonomous Agent and Robot Learning

Keywords: Enaction, self-motivation, constructivist learning. I

A novel way to model an agent interacting with an environment is introduced, called an Enactive Markov Decision Process (EMDP). An EMDP keeps perception and action embedded within sensorimotor schemes rather than dissociated. Instead of seeking a goal associated with a reward, as in reinforcement learning, an EMDP agent is driven by two forms of self-motivation: successfully enacting sequences of interactions (autotelic motivation), and preferably enacting interactions that have predefined positive values (interactional motivation). An EMDP learning algorithm is presented. Results show that the agent develops a rudimentary form of self-programming, along with active perception as it learns to master the sensorimotor contingencies afforded by its coupling with the environment.

INTRODUCTION

In cognitive science, there has been a customary and traditional tripartite division of the mind between perception, the control system, and motor action. This view has been nicely dubbed the "classic sandwich model" by Susan Hurley [START_REF] Hurley | Consciousness in action[END_REF]. There have, nonetheless, been many attempts to challenge this traditional picture, particularly in the field of robotics [START_REF] Brooks | New approaches to robotics[END_REF] [START_REF] Krüger | Tracking in object action space[END_REF], but also from a more psychological and theoretical perspective [START_REF] Shanahan | Embodiment and the Inner Life[END_REF] [START_REF] Ziemke | The construction of reality in the robot: Constructivist perspective on situated artificial intelligence and adaptive robotics[END_REF]. In particular, the idea emerged that it might be a mistake to consider sensation independently from action and that we should design cognitive systems on the basis of low-level sensorimotor loops that represent sensorimotor patterns of interaction. Here, we introduce a modeling approach that goes a step beyond the notion of low-level sensorimotor loops by simply considering sensorimotor patterns-also called sensorimotor schemes by Piaget [START_REF] Piaget | The psychology of intelligence[END_REF]-as the atomic elements manipulated by our algorithms.

Furthermore, the theory of enaction [START_REF] Varela | The embodied mind: Cognitive science and human experience[END_REF] stresses the importance of constitutive autonomy-the capacity of the system to "self-constitute its identity" [START_REF] Froese | Enactive artificial intelligence: Investigating the systemic organization of life and mind[END_REF]. Our modeling approach offers a way to address the problem of constitutive autonomy by supporting a form of autonomous sensorimotor self-programming through interaction with the environment.

II. ENACTIVE MARKOV DECISION PROCESS (EMDP)

We introduce a new way to model an agent interacting with an environment called an Enactive Markov Decision Process (EMDP). An EMDP conflates action and observation within a single entity called an interaction. Based on the EMDP model, we define EMDP problems consisting of designing agents that learn to master their sensorimotor contingencies [START_REF] O'regan | A sensorimotor account of vision and visual consciousness[END_REF] so as to exhibit self-motivation. The EMDP model allows defining two forms of self-motivation: the motivation to be in control of one's own activity by seeking to successfully enact interactions, and the motivation to enact interactions that have predefined positive values and to avoid enacting interactions that have predefined negative values. We call the former autotelic motivation in reference to Steel's autotelic principle [START_REF] Steels | The Autotelic Principle[END_REF], and the latter interactional motivation [START_REF] Georgeon | Interactional motivation in artificial systems: between extrinsic and intrinsic motivation[END_REF].

A. EMDP Formalism compared to POMDP

Formally, we define an EMDP as a tuple (S, I, q, v) in which S is the set of environment states; I is the set of primitive interactions offered by the coupling between the agent and the environment; q is a probability distribution such that q(s t+1 |s t ,i t ) gives the probability that the environment transitions to state s t+1 ∈ S when the agent chooses interaction i t ∈ I in state s t ; and v is a probability distribution such that v(e t |s t ,i t ) gives the probability that the agent receives the input e t ∈ I after choosing i t in state s t . We call i t the intended interaction because it represents the sensorimotor scheme that the agent intends to enact at the beginning of step t; and e t the enacted interaction because it represents the sensorimotor scheme that the agent records as actually enacted at the end of step t. If the enacted interaction equals the intended interaction (e t =i t ) then the attempted enaction of i t is considered a success, otherwise, it is considered a failure. The EMDP cycle evolves as follows and is illustrated in Fig. 1.

1. Let t = 0 and let s 0 ∈ S denote the initial environment state. 2. Choose intended interaction i t ∈ I. 3. Transit to new state s t+1 ∈ S with probability q(s t+1 |s t ,i t ). 4. Generate actually enacted e t ∈ I with probability v(e t |s t ,i t ). 5. t = t + 1.

Goto 2.

There are three major differences between an EMDP and a Partially Observable Markov Decision Process (POMDP) [START_REF] Kaelbling | Planning and acting in partially observable stochastic domains[END_REF]: (a) the cycle does not start from the environment but from the agent, implying a conceptual inversion of the perception/action cycle (as some other authors have also proposed, e.g. [START_REF] Pfeifer | From perception to action: The right direction?[END_REF]); (b) The agent's input and output belong to the same set I rather than two different sets (observations O and actions A); (c) There is no reward defined as a function of the MDP's state.

To pursue the comparison, it would be possible to decompose the interaction set as the product space of a POMDP's action set and observation set (I = A × O). Yet, the philosophy of an EMDP differs in that the observation set would contain too little information about the state s to allow traditional reinforcement learning techniques to generate interesting behaviors. To illustrate this difference, we present an experiment in which the equivalent observation set is reduced to a single bit, while the interaction set if fairly sophisticated. The lack of observational information is counterbalanced by the fact that the observation's meaning depends on the previous action. Moreover, the EMDP goal is not of maximizing a reward function but rather of exhibiting a more complex form of self-motivation that we present next.

B. Self motivation

Now that we have formally defined a successful enaction, we can define autotelic motivation as the tendency to learn to successfully enact interactions. This tendency involves neither a reward nor a goal; it is intrinsically encoded in the algorithm through discovering, recording, and re-enacting sequences of interactions that capture regularities in the coupling with the environment, as we will explain in Section III.

Additionally, we define interactional motivation as a way to associate an "innate" intrinsic value with some interactions, for example: a positive value with interactions representing ingesting food, and a negative value with interactions representing getting hurt. Interactional motivation is meant to underdetermine the agent's behavior so that the agent can use neutral interactions to place itself in situations in which it can successfully enact positive interactions and to stay away from situations in which negative interactions cannot be avoided. We specify interactional motivation through a function r: I → ℝ that associates a scalar value r(e) with each primitive interaction e ∈ I. Interactional motivation relates to intrinsic motivation (e.g., [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]) in that it is defined independently of the environment's state.

To fulfill both its autotelic and interactional motivation, we expect an EMDP agent to discover, memorize, and exploit regularities that exist in its coupling with its environment. Section II.C formally defines a regularity of interaction as a series of primitive interactions that can be learned and enacted as a whole sequence.

C. Self-programming EMDP agents

We define a composite interaction as a sequence of two interactions i c = 〈i pre , i post 〉, where i pre and i post may be primitive interactions or other composite interactions. We refer to i pre as i c 's pre-interaction, also denoted pre(i c ), and to i post as i c 's post-interaction, also denoted post(i c ). We define K t as the set of composite interactions known by the agent at time t, and Jt = I ∪ K t as the set of all interactions, primitive or composite, at time t. Interactions in J t are thus hierarchically organized in a pairwise manner, all the way down to primitive interactions. We extend r to be a function from J t → ℝ that gives the motivational value of a composite interaction as the sum of the values of its primitive interactions, meaning that enacting a composite interaction has the same motivational value as enacting all of its primitive interactions.

A self-programming agent can choose to enact any interaction in J t , primitive or composite. We call the mechanism that chooses an interaction the decisional mechanism, the point in time t d when this choice is made a decision time, and the time lapse during which an interaction is enacted a decision cycle. Decision steps thus do not occur on each time step but rather between each decision cycle. Trying to enact a composite interaction i c consists of sequentially trying to enact the series of the k primitive interactions 〈i p1 … i pk 〉 that compose i c over the next k time steps t d +1, … t d +k. If all the primitive interactions are successfully enacted, then i c is successfully enacted, and the decision cycle ends at time t d +k. If the enaction of the j th element of i c fails, then the decision cycle is interrupted at time t d +j and the actually enacted composite interaction is constructed from the series of the j actually enacted primitive interactions 〈e p1 … e pj 〉.

We describe such agents as self-programming because composite interactions work as programs that the agent learns and subsequently executes. Rather than being written in a conventional programming language, such programs are written in the "agent's programming language" in the sense that they are made of sequences of instructions that the agent knows how to execute.

III. INTERACTIONAL MOTIVATION SYSTEM (IMOS)

We implemented a new algorithm to control EMDP agents, called Interactional MOtivation System (IMOS), based on a previous algorithm [START_REF] Georgeon | An intrinsically-motivated schema mechanism to model and simulate emergent cognition[END_REF]. Section III.A provides a new explanation of the main mechanisms of the previous algorithm in the light of the EMDP formalism introduced in this paper: the learning mechanism and the context construction mechanism. We refer the reader to the previous paper [START_REF] Georgeon | An intrinsically-motivated schema mechanism to model and simulate emergent cognition[END_REF] for a more thorough explanation. Section III.B presents the new decisional mechanism implemented in IMOS.

A. Learning and context construction mechanism

The agent's current situation is represented by a set of interactions C d ⊂ J d referred to as the context (from now on, all "X t " at decision time t d is denoted "X d "). C 0 = ∅ and J 0 = I. At the end of decision cycle t d , the agent records or reinforces the composite interactions whose pre-interaction belongs to C d and whose post-interaction is the actually enacted interaction e d . The set of learned or reinforced composite interactions is thus

L d = {〈i pre , e d 〉 | i pre ∈ C d }.
Each learned or reinforced composite interaction thus records the experience that, in a context in which i pre was enacted, e d was subsequently enacted. The set of composite interactions known by the agent at decision cycle t d+1 thus becomes

K d+1 = K d ∪ L d . Let wei t (i)
∈ ℕ be the weight of composite interaction i at time t. The weight of a new learned composite interaction is initialized to 1. Reinforcing an already-known composite interaction means incrementing its weight.

If i d 's enaction failed (e d ≠ i d ), then the agent adds e d to the set of i d 's alternative interactions alt d (i d ) ⊂ J d known at time td. Later, the decisional mechanism uses alt d (i d ) to evaluate the odds that other interactions may be actually enacted when trying to enact i d again, as we further explain in Section III.B.

Before making the next decision at time t d+1 , the agent computes the context C d+1 as the set of interactions that were enacted at the end of decision cycle t d and that are sufficiently reinforced. We define a threshold η, and let Lη d = {l ∈ L d | wei d (l) > η} be the set of composite interactions in L d whose weight is greater than the threshold η at time t d .

C d+1 = Lη d ∪ {e c } ∪ {post(e c )} (1)
Equation ( 1) implies that C d+1 contains interactions of various lengths that have just been enacted at the end of decision cycle t d . Since, over time, the agent tends to enact interactions that capture sequential regularities afforded by the environment, C d tends to represent the agent's situation in terms of the sequences of interactions that are the most representative of the situation at time t d .

In the experiments reported in Sections IV and V, we set η = 5, meaning that a sequence had to be encountered 5 times before being eligible for inclusion in the context. Lower values of η tend to favor faster learning of possibly less positive behaviors; greater values of η tend to favor slower learning of possibly more positive behaviors. Automatically adjusting η is still an open question. Limiting the size of the context to interactions sufficiently reinforced prevents combinatorial growth of the number of interactions learned over time. With this mechanism, experiments show that the number of interactions learned after each decision cycle remains limited (less than 10 in the experiment reported in Section 5).

B. New decisional mechanism

At decision cycle t d , let A d = {a ∈ K d | pre(a) ∈ C d } be the set of composite interactions whose pre-interaction belongs to the current context, called activated interactions. Let P d = {p ∈ J d | ∃ a ∈ A d , p = post(a)} be the set of interactions that constitute a post-interaction of an activated interaction. We call this set the proposed interactions. The agent thus anticipates that proposed interactions might be successfully enacted in the current context based on the regularities the agent has learned thus far. On decision cycle t d , the agent chooses the next intended interaction i d from amongst the proposed interactions in P d , as we explain next.

For each proposed interaction p in P d , let A pd ⊂ A d be the set of activated interactions that proposed p. The decisional mechanism computes a proclivity value proc d (p) as the sum of the weight of all activated interactions a in A pd multiplied by the value r(p). The agent thus has a proclivity to try to enact the interactions that are proposed by the most reinforced activated interactions and that have the highest motivational values. Interactions that have a negative value create a negative proclivity, which restrains the agent from trying to enact them.

proc d (p) = ∑ a∈Apd wei d (a) × r(p) (2)
Additionally, the decisional mechanism computes an alternative proclivity value proc d (e) for each alternative interaction e ∈ alt d (p) that may result from the attempt to enact p. For each alternative interaction e, we let A ed denote the set of activated interactions that proposed e. proc d (e) = ∑ a ∈Aed wei d (a) × r(e)

(3)

The final proclivity proc_final d (p) is then set equal to the sum of the proclivity of p and of the proclivities of all of its alternative interactions. Alternative interactions of p that have a negative motivational value thus restrain the agent from trying to enact p.

proc_final d (p) = proc d (p) + ∑ e∈altd(p) proc d (e) (4)
The decisional mechanism then chooses the intended interaction i d as the interaction in P d that has the highest final proclivity. As a result, the agent intends to enact the interaction i d that has the highest positive motivational value r(i d ), weighted by the agent's confidence that i d will be successfully enacted in the current context C d , and balanced by the weighted "fear" that it may result into a negative value r(e d ) < 0 if i d 's enaction fails. Because composite interactions may contain interactions with negative values, this mechanism does not drive the agent to the highest immediate value but rather allows the agent to enact sequences of negative interactions to reach even greater positive interactions, and to avoid immediate positive interactions that likely would lead subsequently to even more negative interactions.

The main novelty of IMOS resides in the fact that it takes into account the various alternative interactions that may result from intended interactions, whereas the previous algorithm only considered their success or failure. The algorithm is also clearer in terms of the new EMDP formalism.

IV. EXPERIMENT IN A SIMULATED ENVIRONMENT

Cognitive theories have often advocated that hierarchical sequence learning is a key feature of cognition (e.g., [START_REF] Sun | Sequence Learning -Paradigms, Algorithms, and Applications[END_REF]). Accordingly, we designed an EMDP problem that offers hierarchical sequential regularities of interaction, called the Small Loop Problem (SLP) [START_REF] Georgeon | Demonstrating sensemaking emergence in artificial agents: A method and an example[END_REF] (Fig. 2). The possibilities of interaction are summarized in Fig. 3a. Examples hierarchical regularities of interactions are provided in Fig. 3b (induced by the loop-shaped pathway that constrains the agent's behavior).

The feel interactions especially illustrate the enactivist approach: the agent only receives information about the environment through the enaction of interactions. For example, trying to enact i 7 informs the agent about the absence (if it succeeds) or the presence (if it fails and results in i 8 ) of a wall in front, but the agent initially ignores this meaning, as well as the position of interactions and the existence of walls. In contrast, the turn interactions (i 3 and i 4 ) always succeed when the agent tries to enact them, and, therefore, provide no information (hence the need for representing the situation not only upon the last enacted interaction but upon longer sequences).

The SLP is deterministic, making it a particular EMDP in which q and v implement no stochasticity. The agent is nonetheless confronted with uncertainty because it cannot initially predict the consequences of its intended interactions until it starts learning the regularities afforded by the environment. For example, when circling the loop counterclockwise, if it feels a wall in front, it can often feel an empty cell to the left, but not always. The agent's algorithm is also deterministic, meaning that two runs lead to the same behavior. Different behaviors can nonetheless be observed by starting the agent from different initial positions.

The experiment can be run interactively online 1 . A video showing an example run is also available 2 , corresponding to the trace reported in Fig. 4. Results show that the agent generally learns to avoid bumping into walls by adopting the behavior of feeling in front before trying to move forward within the first 300 steps (Bump Count graph in Fig. 3). Then, when the agent feels a wall in front, it progressively learns to feel to the side before deciding on which direction to turn. This behavior generally leads the agent to start to indefinitely circle the loop after approximately 400 steps, as reported in the example in Fig. 4. In some instances, it learns suboptimal Meaning (ignored by the agent) !

i 1 (5) i 2 (-10) ! ! ! step forward, bump !! b) Example sequential regularities: ! (reg1) After i 7 , attempting i 1 or i 2 more likely results in i 1 than i 2 .! (reg3) After !i 9 , i 3 , i 1 , i 8 " , sequence !i 4 , i 7 , i 1 " can" often be enacted. ! (reg2) After i 8 , sequence !i 9 , i 3 , i 1 "
can often be enacted.

! i 3 (-3) i 4 (-3) turn left, turn right! i 7 (-1) i 8 (-1)
feel front empty, feel front wall! i 5 (-1) ! i 6 (-1) feel right empty, feel right wall! i 9 (-1) i 10 (-1) feel left empty, feel left wall! Fig. 3. Interactions offered by the SLP. a) The agent has 10 primitive interactions at its disposal, which have scalar values set by the experimenter (in parentheses). Interactions that are afforded by empty cells are represented in white and those afforded by walls in green and red, but the agent ignores this distinction. To the agent, interactions only differ by their values. b) The coupling offers hierarchical sequential regularities of interactions. For example, we expect the agent, in discovering and exploiting (reg1), to choose interaction i7, and if this effectively results in i7, to subsequently choose i1 so as to safely enact i1 which has a positive value, thus avoiding i2 which has a very negative value. Regularities have a hierarchical structure: for example, 〈i9, i3, i1〉 in (reg2) is a subsequence of the (reg3) sequence 〈i9, i3, i1, i8〉. The agent can begin by discovering and exploiting lower-level regularities, then learn higher-level regularities from sequences of lower-level regularities.

behaviors such as moving back and forth along a single edge of the loop, repeatedly making U-turns at each end of the edge. We observed that IMOS was generally faster at learning to avoid bumping into walls than the previous algorithm (e.g., 10 bumps in the example in Fig. 4 vs 18 bumps in the example reported in [START_REF] Georgeon | Demonstrating sensemaking emergence in artificial agents: A method and an example[END_REF]).

V. EXPERIMENT WITH AN E-PUCK ROBOT

We ran a similar experiment with an e-puck robot [START_REF] Mondada | The e-puck, a robot designed for education in engineering[END_REF] in the Box world shown in Fig. 5. The set of primitive interactions I was the same as those in the experiment in Section IV, with the same value function r. The interaction i 1 (step forward) consisted of moving approximately 5 cm. If the robot bumped into a wall during this displacement, the movement was interrupted and the interaction i 2 was actually enacted. The three infrared sensors available in the front, front-left, and front-right directions were used to detect bumping (using a range of approximately 0.5 cm). The interactions i 3 (turn left) and i 4 (turn right) consisted of spinning approximately 90° to the left or to the right. The feel interactions (to the front, left, or right) were implemented using the three infrared sensors available on the front, left, and right sides of the robot, using a range of approximately 5 cm. Trying to enact a feeling interaction i 5 , i 7 , or i 9 consisted of switching on the corresponding infrared LED and reading the infrared detection signal. Interactions i 5 , i 7 , i 9 ended up actually enacted if no wall was detected, and i 6 , i 8 , i 10 were actually enacted if a wall was detected within the approximate 5 cm range respectively to the right, front, and left.

Similar to the experiment in Section IV, this experiment showed that the robot learned to avoid bumping by using its front sensor to check for the absence of a wall before moving forward. A video of an example run is available online 3 . The first hundred steps are reported in the trace in Fig 6 .   This experiment illustrates how an EMDP algorithm can be transferred to a robot interacting with the real world. The algorithm worked well with imprecise displacements and noisy sensors. In some cases, spurious composite interactions may be learned (e.g., feel front empty then bump) but they do not degrade the robot's behavior as long as correct composite interactions have dominant weights. The algorithm can be 3 http://e-ernest.blogspot.fr/2012/04/ernest-7-in-e-puck.html Fig. 5. The e-puck robot in the Box world. Fig. 4. First 500 steps of an example run of an IMOS agent in the SLP. Tape 1 represents the primitive interactions actually enacted over time: step forward (white triangles), bump (red triangles), turn (half-circles), feel empty (white squares), feel wall (green squares). Tape 2 represents the values of the enacted primitive interactions as a bar graph (green when positive, red when negative). Tape 3 represents the level of the enacted composite interaction in the hierarchy (gray when the enaction of the primitive interaction was successful, black when it failed, thus interrupting the decision cycle). The second-order composite interaction feel front empty -move forward (exploiting reg1) was successfully enacted for the first time on steps 57-58, then again on steps 59-60. The agent unsuccessfully tried to enact it on step 52 but the unexpected feeling of a wall caused its interruption, preventing the agent from bumping into the wall (black, second-order segment in Tape 3). Because the agent makes no assumptions on the meaning of primitive interactions, it also tries composite interactions that seem absurd to the observer (such as feeling twice to the front on steps 97-98) but subsequently abandons them because they prove useless to satisfy its interactional motivation. A third-order composite interaction was successfully enacted for the first time on steps 208-210: turn right -feel front empty -step forward, and its counterpart to the left on steps 404-406. The fourth-order composite interaction feel left wall -turn right -feel front emptystep forward was enacted on steps 441-444. From that point, the agent had learned enough regularities to circle the loop counterclockwise indefinitely by appropriately feeling in front and to the right to prevent bumping and turning to the wrong direction, thus also minimizing mildly negative turn interactions. used as such to control the robot because the robot's motivation is defined in terms of interactions that are known to the robot itself, rather than requiring states of the world to be explicitly specified, as in POMDPs.

Note that this experiment rested on the fact that the ranges of the sensors were tuned to approximately correspond to the distance covered by a step forward move (5 cm). If these distances are too different, the robot learns that the sensors are useless. In this case, the robot moves forward without using the sensors and does not avoid bumping. Nonetheless, if we reduce the cost of turning to a small cost (r(i 3 ) = r(i 4 ) = -1), the robot learns to avoid bumping by alternating moving forward and turning, which causes it to circle around continuously in the box without hitting the walls.

VI. CONCLUSION

The EMDP model offers a framework for conceptualizing an agent's coupling with its environment without separating perception from action. EMDP problems consist of designing agents that tend to successfully enact sequences of interactions, and, simultaneously, seek to enact interactions that have predefined positive values while avoiding interaction that have strong negative values. While this objective involves a scalar value function, it differs from the reinforcementlearning objective by the fact that maximizing this value is not the only goal.

We introduce a new algorithm to address EMDP problems called IMOS. By learning sequences of behaviors, IMOS relates to adaptive history methods (e.g. [START_REF] Mccallum | Learning to use selective attention and short-term memory in sequential tasks[END_REF]), used in reinforcement learning, but differs by the fact that the learning is not driven by a reward function of the MDP's state. By involving bottom-up chaining of behaviors, it relates to means-end analysis approaches (e.g., [START_REF] Drescher | Made-up minds, a constructivist approach to artificial intelligence[END_REF]), but differs by the fact that it is not based upon triples 〈pre-observation, action, post-observation〉, but rather couples 〈pre-interaction, post-interaction〉, allowing the recursive learning of hierarchies of sequences, and rudimentary self-programming.

We propose a method to assess EMDP agents through behavioral analysis. In our experiments, IMOS agents exhibit autotelic motivation and interactional motivation, in particular by learning to use some sensorimotor interactions (feeling) as a form of active perception to subsequently choose adapted behaviors (although the specific meaning of the interactions is initially unknown to the agent). Fig. 6. First 100 steps of an example run of an e-puck robot controlled by IMOS in the Box world, using the same symbols as in Fig. 4. Tape 1 shows that the robot learned to avoid bumping (red triangles) after step 36. Tape 2 shows that the robot received more positive values more regularly from step 42 on. Tape 1 with Tape 3 show that the composite interaction feel front empty -step forward was enacted for the first time on steps 27-28. On step 64, 74, and 92, this composite interaction was interrupted (black second-order segment in Tape 3) due to the fact that the feel front empty intended primitive interaction resulted in the feel front wall interaction being actually enacted, preventing the robot from bumping and inciting it to turn left instead.

1 http:

 1 //e-ernest.blogspot.fr/2012/03/small-loop-challenge.html 2 http://www.youtube.com/watch?v=yliUE39l2R4

Fig. 2 .

 2 Fig. 2. The Small Loop Problem (SLP) in NetLogo. The environment (left) is the loop of white squares surrounded by green walls. The agent (brown arrowhead) can try to move one cell forward, turn to the left or to the right, feel in front, to the left or to the right, but it ignores the meaning of interactions. The experimenter can preset the values of the primitive interactions using the slider controls (center). The Interaction-Value window shows ASCII codes representing the primitive interactions enacted by the agent over time next to their values. The Bump Count graph (right) displays the number of times the agent bumps into a wall (cumulative total in blue), showing that the agent gradually learns to avoid bumping into walls. When the agent touches/feels a cell, the cell flashes yellow, and when the agent bumps into a wall, the wall flashes red, making the agent's behavior intelligible to the experimenter.

  Fig.1. Diagram of an Enactive Markov Decision Process (EMDP). At time t, the agent chooses an intended primitive interaction it (the agent's output). The attempt to enact it generates a transition of the environment (represented by the Markov Decision Process, MDP) from state st to st+1. The agent then receives the enacted primitive interaction et ∈ I (the agent's input). If et = it, then the attempt to enact it is considered a success, otherwise, a failure. The agent's "perception of its environment" is an internal construct Ct rather than the input et.

	MDP
	s t i t	q(s t+1 |s t , i t )
	state s t+1	state s t
	v(e t |s t ,i t )
		Agent
	e t ! I	r(e t ) ! Context C t	i t ! I

This work was supported by the French Agence Nationale de la Recherche (ANR) contract ANR-10-PDOC-007-01. We gratefully thank James Marshall and Riccardo Manzotti for their contribution to this report.


Olivier L Georgeon
email: olivier.georgeon@liris.cnrs.fr

James B Marshall
email: jmarshall@sarahlawrence.edu

DEMONSTRATING SENSEMAKING EMERGENCE IN ARTIFICIAL AGENTS: A METHOD AND AN EXAMPLE

Keywords: sensemaking, self-motivation, emergence, behavioral analysis

We propose an experimental method to study the possible emergence of sensemaking in artificial agents. This method involves analyzing the agent's behavior in a test bed environment that presents regularities in the possibilities of interaction afforded to the agent, while the agent has no presuppositions about the underlying functioning of the environment that explains such regularities. We propose a particular environment that permits such an experiment, called the Small Loop Problem. We argue that the agent's behavior demonstrates sensemaking if the agent learns to exploit regularities of interaction to fulfill its self-motivation as if it understood (at least partially) the underlying functioning of the environment. As a corollary, we argue that sensemaking and selfmotivation come together. We propose a new method to generate self-motivation in an artificial agent called interactional motivation. An interactionally motivated agent seeks to perform interactions with predefined positive values and to avoid interactions with predefined negative values. We applied the proposed sensemaking emergence demonstration method to an agent implemented previously, and produced example reports that suggest that this agent is capable of a rudimentary form of sensemaking.

Introduction

We address the problem of demonstrating the potential capability of an agent to make sense of information when it interacts with its environment. We refer to this problem as the sensemaking demonstration problem. The term sensemaking is used here according to its common definition of giving meaning to experience or understanding experience. We consider the problem of machine sensemaking as a particular aspect of the broader problem of machine consciousness because being able to make sense of information is an important part of what it is to be conscious.

Here we address the issue of rudimentary sensemaking as it may be exhibited by rudimentary systems as opposed to higher-level sensemaking such as understanding human language. Rudimentary sensemaking refers to the capability of an agent to make sense of basic elements of its experience interacting with the environment. Because the study of rudimentary sensemaking focuses on the study of cognition through the agent's experience of interaction, it falls within the field of situated cognition [e.g., [START_REF] Suchman | Plans and situated actions[END_REF] and embodied cognition [e.g., [START_REF] Pfeifer | How the body shapes the way we think: A new view of intelligence[END_REF].

The sensemaking demonstration problem is different from the problem of designing an agent that has sensemaking capabilities; it is the problem of assessing the capabilities of an existing agent. To address this problem, we acknowledge the fact that sensemaking is an emergent property of an agent interacting with an environment. As an emergent property, sensemaking can potentially be observed in the agent as a whole but cannot be traced to the agent's components. Because the demonstration of sensemaking is based on the observation of the whole agent rather than on the analysis of its components, it belongs to the experimental sciences. Our approach consists of taking an alreadydesigned agent as an object of study and empirically investigating whether or not this agent is capable of some form of sensemaking.

In essence, we seek to produce evidence to support or contradict the claim that a given agent is capable of giving meaning to its experience; and, if it is, to clarify the form and the extent of this capability on the basis of explicit arguments. We argue that addressing this problem entails three aspects: one aspect consists of defining what may constitute acceptable evidence of sensemaking for the scientific community-typically, the cognitive science, artificial intelligence, and machine consciousness communities. Another aspect consists of designing an experiment that offers opportunities for the agent to demonstrate its sensemaking capabilities. A third aspect consists of generating acceptable evidence that the agent did indeed exhibit sensemaking during the experiment. These three aspects are intertwined: designing experiments will help clarify what we expect from a sensemaking agent, and reporting evidence will help clarify what kind of evidence is acceptable to the scientific community.

Applying experimental science to an object that was designed by humans raises the difficulty of separating design from experiment. This difficulty was captured a long time ago by John von Neumann: "if you will tell me precisely what it is that a machine cannot do, then I can always make a machine that will do just that" [e.g., cited by Samsonovich, 2012, p105]. This remark applies to the sensemaking demonstration problem: if we define precisely an experiment and specify precisely the results that we seek to observe, then one can always design an algorithm that will generate just these results. In this case, these results will not constitute evidence of sensemaking because they will be generated by a specifically designed algorithm for this precise experiment rather than arising out of the algorithm's execution in an initially unknown experiment. This remark suggests that emergence is not only a property but also a requirement of sensemaking: the experimental results should be emergent in the sense that it should not be possible to track their causes down to the algorithm's logic. Evidence that the results are emergent can be given by a description of the algorithm provided by the designer. This description must show that the algorithm was not specifically designed to generate these results in this specific experiment but rather that the algorithm has no direct causal efficacy to explain the results.

Philosophers of mind have been discussing sensemaking for a long time and some of their results can help specify the type of observations that may constitute evidence of sensemaking. Specifically, we subscribe to pragmatic epistemology [e.g., [START_REF] James | Pragmatism[END_REF][START_REF] Wittgenstein | Philosophical Investigations[END_REF] that suggests that meaning is given by usage. This idea has been summarized by Wittgenstein's famous quote "meaning is use". Following this idea, we acknowledge that the possibility for an observer to consider that an agent makes sense of an experience comes together with the possibility for the observer to consider that the agent has its own motivation. Based on this conception of sensemaking coming from pragmatic epistemology, we address the problem of the emergence of sensemaking jointly with the problem of self-motivation in artificial systems. We seek to produce evidence that the agent understands experiences based on evidence that the agent has a form of self-motivation, and on evidence that the agent is capable of using such experiences for the purpose of fulfilling its self-motivation.

Several solutions for implementing self-motivation in artificial systems have been proposed. For example, [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] proposed a motivational system based on curiosity; [START_REF] Blank | The multiple roles of anticipation in developmental robotics[END_REF] proposed the search for predictability and control; [START_REF] Schmidhuber | Formal theory of creativity, fun, and intrinsic motivation[END_REF] proposed the search for simplification of knowledge and compressibility of data. These authors have contributed to defining the term intrinsic motivation as opposed to extrinsic motivation, which comes from an external goal defined by the designer. Indeed, specifying a final goal to reach in the environment makes the agent's motivation appear (from the observer's perspective) as if it were specified by the designer rather than coming from the agent itself. To provide the opportunity for the agent to exhibit self-motivation, we designed an experiment in which the agent's behavior is driven by preferences of interactions rather than by a final goal to reach.

For the agent to exhibit sensemaking in addition to self-motivation, we expect the agent to demonstrate a form of progressive organization of behavior through its experience of interacting with the environment. Indeed, demonstrating that the agent makes sense of its experience implies that the agent initially doesn't understand its experience but actively engages in the process of understanding it. Offering the opportunity for sensemaking implies providing the agent with an environment that affords regularities of interaction that the agent can discover, memorize, and exploit to fulfill its self-motivation.

We make the hypothesis that providing the agent with an environment that affords hierarchies of sequential regularities of interaction for the agent to discover and exploit is suitable to demonstrate a rudimentary form of sensemaking. This hypothesis is supported by studies in artificial cognition that suggest that hierarchical sequence learning is a key feature of cognition [e.g., [START_REF] Sun | Sequence Learning -Paradigms[END_REF], and, more broadly, by constructivist epistemology that suggests that knowledge arises from the discovery and exploitation of sensorimotor regularities of interaction with the world [e.g., [START_REF] Piaget | The psychology of intelligence[END_REF]. Section 2 presents the general formalism of such experiments, and Sections 3 and 4 present an experiment involving a specific agent as an example.

Hierarchical sequential regularity learning based on interactional motivation

To study rudimentary sensemaking, we place the agent in the situation of interacting with an environment that affords hierarchical sequential regularities of interaction to discover and exploit. Additionally, we offer the possibility for the experimenter to attribute scalar values to the possible interactions afforded to the agent by the environment. Values associated with interaction patterns allow the experimenter to define innate preferences: seeking interactions with positive values and avoiding interactions with negative values, without defining a final goal state to reach. We call this form of self-motivation interactional motivation [Georgeon et al., 2012] because it is neither purely internal to the agent (as is curiosity) nor external (as is a goal state to reach in the environment). Fig. 1 illustrates the Hierarchical Sequential Interactional Motivation Learning problem (HSIML) with an example.

Set of n moves

U = {u 1 , u 2 , … u n }, e.g., !u 1 ! !u 2 ! !u 3 ! ! !u 4 ! ! u 5 ! !u 6 ! Set of m effects Y = {y 1 , y 2 , … y m }, e.g., !y 1 ! !y 2 ! Set of n!m interactions I = {i=(u, y) | u U, y Y} with value r(i) ! e.g.,! i 1 !(-1) ! ! !i 3 ! !(-1) ! ! !i 5 !(-1) ! ! !i 7 !(5) ! ! !i 9 ! (-3) ! ! !i 11 ! (-3)! i 2 !(-1) ! !i 4 !(-1) ! !i 6 !(-1) ! !i 8 !(-10)! !i 10 ! (-3) ! !i 12
! (-3)! The environment offers hierarchical sequential regularities of interactions to discover and exploit, e.g., ! -(reg1) After i 3 ! , move u 4 results more likely in i 7 than in i 8 .! -(reg3) After sequence !i 5 , i 9 , i 7 , i 4 " , sequence !i 11 , i 3 , i 7 " can most likely be enacted. ! -(reg2) After i 4 , sequence !i 5 , i 9 , i 7 " can most likely be enacted. ! etc. ! Fig. 1: Example of Hierarchical Sequential Interactional Motivation Learning (HSIML) problem. On each decision cycle t, the agent chooses a move ut from amongst the n possible moves in the set U, represented by grey symbols. As a result of the chosen move ut and the state of the environment, the agent receives an effect yt from amongst the m possible effects in the set Y, represented by a color. Each pair i = (u,y) forms a primitive interaction (i ∈ I = UÍY), represented by a symbol whose shape represents the move u and whose color represents the effect y. A scalar value r(i) is attached to each primitive interaction, represented by values in parentheses next to the interactions. The agent seeks to enact interactions with positive values (in this example: only i7 (5)) and to avoid interactions with negative values as much as possible. The environment offers hierarchies of sequential regularities to which the agent is sensitive. We expect the agent to discover, memorize, and exploit these regularities so that it can learn to enact interactions with positive values, or slightly negative values that subsequently lead to more positive interactions, and to avoid interactions with larger negative values. For example, we expect the agent, in discovering and exploiting (reg1), to choose move u2, and, if this results in interaction i3, to subsequently choose move u4 so as to safely enact i7 which has a positive value, thus avoiding i8 that has a very negative value. Regularities have a hierarchical structure: for example, the learned sequence 〈i3, i7〉 used to exploit (reg1) is a subsequence of the (reg3) sequence 〈i11, i3, i7〉, and the sequence 〈i5, i9, i7〉 in (reg2) is a subsequence of the (reg3) sequence 〈i5, i9, i7, i4〉. In turn, sequences in (reg3) can constitute subsequences involved in higher-level regularities. The agent can begin by discovering and exploiting lowerlevel regularities, then learn higher-level regularities from sequences of lower-level regularities.

The diagram in Fig. 2 represents the formalism for the HSIML problem in a form similar to Partially Observable Markov Decision Processes (POMDPs) [e.g., Aström, 1995] to allow for comparison.

Agent

r (u t , y t ) v(y t |s t ,u t ) (s t , u t ) u t r t y t Q(s t+1 |s t , u t ) MDP Fig. 2: Formal diagram
of the Hierarchical Sequential Interactional Motivation Learning (HSIML) problem. In contrast to the traditional reinforcement learning cycle, the interaction cycle starts with the agent rather than the environment. At time t, the agent chooses a move ut, then receives an effect yt that depends on the state of the environment st and on the move ut, according to the probability distribution v(yt|st,ut). The Markov Decision Process (MDP) represents the environment, which is partially hidden to the agent by the probability distribution v, as it is in a Partially Observable Markov Decision Process (POMDP) problem. A crucial difference, however, is that yt does not depend only on st but also on ut. The pair (ut,yt) is called a primitive interaction. A value function r(ut,yt) is associated with the interaction (ut,yt) rather than with the environment state as in the POMDP problem. The move ut changes the environment state from st to st+1 according to the distribution Q(st+1|st,ut).

Formally, the HSIML is defined as follows:

-|S| states S = {s 1 , ... ,s |S| } of the environment; -n moves U = {u 1 , … , u n } available to the agent; -m effects Y = {y 1 , ... ,y m }; -A value function r(u,y) ∈ ℝ for each primitive interaction (u,y) ∈ UÍY.

An HSIML evolves in a similar way as a POMDP, but we start the description from the agent's move because the effect conceptually results from the move, as opposed to a POMDP where the action conceptually results from the observation. Notably, this conceptual inversion finds some resonance with [START_REF] Pfeifer | From perception to action: The right direction?[END_REF] questioning of the traditional perception-action cycle. An HSIML evolves as follows:

1. Let t = 0 and let s 0 denote the initial environment state. 2. At time step t, generate move u t according to the agent's policy µ. 3. Generate a new effect y t with probability v(y t |s t ,u t ). The interaction i t =(u t ,y t ) is called the enacted interaction at time t. 4. Generate the satisfaction value r t = r(u t ,y t ) associated with the enacted interaction. 5. Generate a new environment state s t+1 with probability Q(s t+1 |s t ,u t ). 6. t = t + 1. 7. Goto 2.

The fact that the environment affords sequential regularities of interaction means that the probability distributions Q and v are such that pairs of sequences of interactions 〈σ 1 ,σ 2 〉 exist such that, when σ 1 is enacted, it is probable that σ 2 can subsequently be enacted. σ 1 and σ 2 are sequences of primitive interactions (e.g., σ 1 = 〈i 1 ,i 2 ,…,i k 〉). Enacting a sequence of interactions means sequentially enacting all of its primitive interactions. Enacting a primitive interaction i=(u,y) means performing move u and obtaining effect y. Notably, the agent may intend to enact primitive interaction i but end up enacting a different primitive interaction if the resulting effect differs from the expected effect y. The fact that sequential regularities have a hierarchical structure means that a regularity σ = 〈σ 1 ,σ 2 〉 can be part of a higher-level regularity (e.g., σ 5 = 〈σ 3 ,σ 4 〉 with σ 3 = 〈i 1 ,…, σ,… ,i k 〉). We define HSIML problems as any problem that affords such hierarchical sequential regularities of interactions and that can be represented by the formalism presented above (Fig. 2). Section 3 presents an agent that we developed to address this class of problems.

A sensorimotor self-programming agent

We designed an agent called Ernest that addresses HSIML problems as presented in Section 2. A detailed description of Ernest's algorithm is provided elsewhere [START_REF] Georgeon | An intrinsically-motivated schema mechanism to model and simulate emergent cognition[END_REF]. In accordance with the principles of the sensemaking demonstration problem presented in the introduction, we present here the principle of Ernest's algorithm in order to show that Ernest is, by design, capable of addressing HSIML problems without presuppositions about the underlying functioning of the environment that causes the regularities of interaction.

Conceptually, Ernest relies on a model of cognition based on sensorimotor patterns rather than on the traditional "cognitive loop" that separates perception, cognition, and action. This model offers a framework for conceptualizing the agent's ability to perform a form of sensorimotor self-programming. We use the term sensorimotor self-programming because Ernest records sequences of sensorimotor patterns, and learns to recognize the context in which these sequence can subsequently be executed as programs. The context is encoded in the form of sensorimotor patterns, which requires no assumption about the underlying structure of the environment. This learning is bottom-up, meaning that Ernest initially learns short sequences of behaviors adapted to short sequential regularities afforded by the environment. Once Ernest masters the short-term regularities, it can discover and learn higher-level regularities that consist of sequences of shorter-term regularities. This hierarchical organization helps Ernest deal with the combinatorial complexity of the particular HSIML problem with which it is confronted. Fig. 3 illustrates the sensorimotor self-programming model. the input to the agent is considered the agent's "perception"; the agent interprets its perception in order to choose its next action. Top right: the input to a sensorimotor self-programming agent is an "effect" produced by performing a "move" in the environment (the effect depends not only on the state of the environment but also on the actuator that was moved). The pair (move, effect) is called a primitive sensorimotor scheme. The agent's perception is an internal structure that is gradually constructed as sensorimotor schemes are enacted. The agent represents its current situation in terms of schemes, that is, in terms of possibilities of interaction. The agent chooses the next sensorimotor schemes to enact based on its anticipation of the resulting effects. Bottom: the Sensorimotor Self-Programming Model: the sensorimotor loop is "unfolded" to display two time scales: the activity time and the developmental time. Primitive schemes, i.e., (move, effect) pairs, that are enacted during the activity time are represented by short arrows at the bottom of the diagram. As the developmental time goes on, the agent learns higher-level schemes (longer arrows, higher in the diagram).

Higher-level schemes are hierarchical compositions of lower-level schemes matched to regularities discovered along the activity time. As the agent develops, it engages in increasingly complex activities made possible by the high-level schemes that were previously learned. Enacting a high-level scheme consists of sequentially activating and enacting its sub-schemes, all the way down to predefined primitive schemes. Therefore, schemes work as executable programs that are autonomously learned by the agent.

Ernest's algorithm implements two motivational principles. The first principle is to record and test sequences of interactions, in search of those that capture regularities afforded by the environment. This motivational principle can be understood as a tendency to be in control of its activity [START_REF] Steels | The autotelic Principle[END_REF]. The second principle is to seek interactions with positive values and to avoid interactions with negative values. This principle implements interactional motivation as presented in Section 2. Because of this dual motivational principle, Ernest does not learn optimal behaviors in terms of the value of the interactions. Nonetheless, it learns satisfying behaviors adapted to the regularities it discovers through its singular experience of interacting with the environment. For a complete description of Ernest's algorithm, we refer the reader to our paper [START_REF] Georgeon | An intrinsically-motivated schema mechanism to model and simulate emergent cognition[END_REF]. The present paper focuses on analyzing Ernest's behavior from an observer's perspective, which we do next.

The Small Loop Problem (SLP)

We designed a particular instance of an HSIML problem that allows an external observer to easily study the agent's behavior. We call it the Small Loop Problem (SLP) because the environment can be visualized as a small loop around which the agent can move [Georgeon & Marshall, 2012]. We implemented the Small Loop environment in NetLogo [Wilensky, 1999] to provide the interface shown in Fig. 4. showing that the agent gradually learns to avoid bumping into walls. When the agent touches/feels a cell, the cell flashes yellow, and when the agent bumps into a wall, the wall flashes red, making the agent's behavior intelligible to the experimenter.

In the SLP, the set of possible moves U contains the 6 following moves: feel to the right (u 1 ), feel to the front (u 2 ), feel to the left (u 3 ), try to move one cell forward (u 4 ), turn 90° left (u 5 ), turn 90° right (u 6 ). The set of possible effects Y contains two elements, making 10 possible primitive interactions: feel an empty cell to the right/front/left i 1 =(u 1 ,y 1 ), i 3 =(u 2 ,y 1 ), i 5 =(u 3 ,y 1 ); feel a wall to the right/front/left i 2 =(u 1 ,y 2 ), i 4 =(u 2 ,y 2 ), i 6 =(u 3 ,y 2 ); step forward i 7 =(u 4 ,y 1 ); bump i 8 =(u 4 ,y 2 ); turn left i 9 =(u 5 ,y 1 ); turn right i 11 =(u 6 ,y 1 ). This environment is deterministic, meaning that, in this particular HSIML, the corresponding probability distributions Q and v (presented in Fig. 2) implement no stochasticity.

The experimenter presets the values of primitive interactions before running the agent (using the controls shown in Fig. 4). We specify the following reference values: step: 5; bump: -10; turn: -3; feel (empty or wall): -1. Note that these primitive interactions and values correspond to the example HSIML shown in Fig. 1, with the exception that turns always produce effect y 1 , which makes 10 interactions rather than 12 (i.e., the interactions i 10 and i 12 presented in Fig. 1 do not exist in the SLP). The agent, however, ignores the meaning of these interactions, and even ignores the spatial organization of the environment.

Note that the SLP affords the regularities (reg1, reg2, reg3) presented in Fig. 1. Although the agent ignores their meaning, (reg1) means that feeling an empty cell in front indicates that it is possible to step forward. (reg2) means that, when a wall is felt in front, it is often possible to feel an empty cell on the left and to turn left and step forward. This regularity can only be discovered after the agent has started circling the loop in the counterclockwise direction. If the agent starts circling the loop in the clockwise direction, then it can discover the counterpart regularity based on right turns. (reg3) characterizes the passage in the upper right area of the loop when circling the loop in the counterclockwise direction.

Because Ernest can learn regularities of interactions, we expect it to learn a behavior that suggests that it progressively understands the possible interactions that are afforded by the environment, jointly with understanding the structure of the environment. Typically, we expect Ernest to appropriately use i 1 , i 2 , i 3 , i 4 , i 5 , and i 6 to effectively feel its surrounding environment, and to use i 9 and i 11 to effectively turn towards empty cells. This behavior should emerge as if Ernest enjoyed moving forward and disliked bumping, and progressively understood what the various available interactions correspond to.

Behavioral analysis

Ernest's behavior can be observed in the NetLogo platform available online1 . One can choose different values for primitive interactions and reconfigure the environment by clicking on the grid in order to observe Ernest's adaptation to various situations. The NetLogo source code can also be downloaded so developers can change the ASCII characters that encode the different moves and effects generated by the environment without changing Ernest's algorithm, to verify that the algorithm does not exploit predefined semantics associated with these codes.

The experiment shows that, with the values defined in Section 4, Ernest learns to use feel interactions to avoid bumping into walls and to turn towards empty cells. First, Ernest learns to feel in front and to move forward if it feels an empty cell, and to not move forward if it feels a wall. Then, after a while, when it feels a wall in front, it learns to feel to the sides and turn towards empty cells so as to subsequently move forward. This behavior makes Ernest usually learn to avoid bumping into walls after approximately 300 steps. When engaging in circling the loop, Ernest often has difficulties in the upper right area because of the inverted corner. This is because Ernest learns only sequential regularities, which do not reflect the two-dimensional spatial organization of the environment. Yet, after a while, Ernest discovers the sequential regularities that characterize the upper left area (reg3), which allows it to reach a stable satisfying behavior that most often consists of circling the loop after approximately 600 steps. Notably, in some instances, Ernest learns suboptimal behaviors such as moving back and forth along a single edge of the loop, repeatedly making U-turns at each end of the edge.

Fig. 5 shows the trace of a representative run. This trace represents the primitive interactions enacted by Ernest over time with symbols similar to those used in Fig. 1. For easier reading, however, a few differences were introduced: i 8 is represented as a red triangle rather than a green triangle to more intuitively evoke bumping into walls. Feel-to-the-sides interactions (i 1 , i 2 , i 5 , i 6 ) are not represented by trapezoids but by squares that are placed above or below the line to represent, respectively, feeling to the left and to the right. feel empty (white squares), feel wall (green squares), turn (half-circles), step forward (white triangles), bump (red triangles); the upper part represents interactions to the left, the lower part interactions to the right. Tape 1 shows that Ernest learned to feel in front before trying to move forward from step 190 on, but still got puzzled in the upper right area of the loop around step 220 and 270. Ernest finally learned to avoid bumping after step 276. Tape 2 represents the interaction values as a bar graph (green when positive, red when negative); it shows that Ernest got more consistently positive interactions from step 290 on, meaning that it learned to fulfill its interactional motivation. Tape 3 represents the level of the enacted sequence in the hierarchy of learned sequences; it shows that Ernest gradually exploited higher-level sequences; black segments show sequences of interactions that are interrupted due to an unexpected effect when trying to enact a constituent primitive interaction. For example, on steps 62 and 69, Ernest expected to enact the sequence "feel front empty -move forward", but this sequence was interrupted due to the unexpected feeling of a wall in front. Together, the three tapes show that Ernest learned to characterize left corners (situations with walls in front and on the left) by the sequence that leads to them when circling the loop counterclockwise: "feel empty to the left, turn left, move forward, feel wall in front". In this left corner context, Ernest learned to choose turn right (steps 318, 354, 390), which allowed it to engage in full tours of the loop. The average value obtained after behavior stabilization was of 5×12 (move forward) -3×6 (turn) -1×17 (touch), corresponding to 0.71 points/step.

A different choice of motivational values shows that sensemaking depends on the motivation that drives the agent. For example, if the experimenter gives positive values to the turn interactions, then the agent leans to spin in place, and exhibits poor sensemaking. This behavior has been observed in Experiment 2 illustrated by the example trace in Fig. 6. shows that Ernest gradually learns sequences that consist of enacting a series of right turns (a series of 2 right turns from step 42 on, then a longer series not represented in the figure). In this run, Ernest never tried to turn to the left, so it never learned that it could receive even higher values (10) by turning to the left. This shows again that Ernest's learning is suboptimal (in terms of interactional motivation). Higher probability of finding optimal behaviors could be obtained by adding other forms of intrinsic motivation such as boredom avoidance, curiosity, or variable interactional motivation depending on internal states.

We propose these two behavior analyses as two complementary pieces of evidence supporting the claim that Ernest is capable of some form of sensemaking and selfmotivation, while acknowledging that such sensemaking and self-motivation are still rudimentary.

Experiment 1 (Fig. 5) shows that Ernest appears to be driven by the motivation of moving forward and avoiding bumping, as if it enjoyed moving forward and disliked bumping into walls. Ernest learns to appropriately use the available possibilities of interaction to gain information to fulfill this motivational principle. It learns to engage in active perception (feeling) even though feel interactions were not initially defined as perception, and to use the resulting information to engage in appropriate subsequent behaviors (turning to the appropriate direction and moving forward) that fulfills its motivational principles.

Experiment 2 (Fig. 6) shows an example in which Ernest is motivated to turn, as if it enjoyed turning. This second example shows that Ernest's algorithm was not specifically designed to generate the sensemaking behavior demonstrated in Experiment 1 but rather to fulfill its interactional motivation whatever it may be. Therefore, Experiment 2 supports the premise that the sensemaking exhibited in Experiment 1 emerged out of a particular configuration of the agent's motivation and of the environment rather than being predefined in the algorithm.

Notably, the sensemaking exhibited in Experiment 1 is still rudimentary. For example, the agent does not understand that the environment contains other useful concepts such as "long edge of the loop" which would allow the agent to safely move three times forward without the need to feel in front before each step. Experiment 2 suggests that other forms of intrinsic motivation, such as boredom avoidance or curiosity, may help achieve higher-level sensemaking. We expect future versions of the agent to be able to engage in such higher forms of sensemaking and to exhibit more complex forms of self-motivation.

Conclusion

We propose an experimental method to study the sensemaking capabilities of an artificial agent. This method aims at producing evidence to support or contradict the claim that a particular agent is capable of some form of sensemaking. We present an application of this method that suggests that evidence of an agent's possible sensemaking capabilities can be found in the agent's behavior, through behavioral analysis. Our results also support the notion that observing sensemaking in an agent's behavior is linked to observing a form of self-motivation.

We propose a new class of problems that offers the opportunity for the agent to simultaneously exhibit rudimentary sensemaking and self-motivation, called Hierarchical Sequential Interactional Motivation Learning problems (HSIML). HSIML problems make it possible to investigate a form of self-motivation called interactional motivation. An interactionally motivated agent is an agent that seeks to enact interactions with positive values and to avoid interactions with negative values. Notably, interactional motivation can coexist with intrinsic motivation. In principle, an HSMIL problem in which all the values of interactions are set to zero can be used to demonstrate intrinsic motivation (such as curiosity), while setting interactions to nonzero values can generate supplementary interactional motivation. HSIML problems, however, contrast with most existing benchmarks in unsupervised learning [e.g. [START_REF] Rohrer | Accelerating progress in Artificial General Intelligence: Choosing a benchmark for natural world interaction[END_REF] in that HSIML problems do not involve extrinsic motivation defined as a final goal to reach. The purpose of an HSIML problem is to offer the opportunity for the agent to learn to organize its behaviors appropriately, rather that to learn to reach a final goal.

We defined a particular instance of an HSIML problem called the Small Loop Problem (SLP) as a practical benchmark to study self-motivation and sensemaking in artificial agents. The SLP is available as a NetLogo project to other researchers interested in studying the sensemaking capabilities of their own agents.

We presented an application of the SLP to study the behavior of an agent that we developed in a previous study, and performed a behavioral analysis of this agent that suggests that it exhibits some form of rudimentary sensemaking and self-motivation. In summary, the behavioral analysis shows that the agent actively engaged in seeking information that it subsequently used to perform appropriate sequences of behaviors that fulfilled its interactional motivation, even though the semantics of such information and behaviors was not encoded in the algorithm. However, the agent was not able to find the optimal behaviors in terms of interactional motivation values, nor did it appear to seek such optimal behavior.

Overall, this study suggests that the demonstration of sensemaking relies on three types of evidence: (a) a description of the agent's algorithm that shows that the agent does not exploit a predefined semantics adapted to the experiment and encoded by the programmer; (b) an interactive demonstration of the agent's behavior in an experiment in which the experimenter can witness the gradual organization of behavior and modify the parameters to ensure that the experiment was not set up to work only in a predefined case; and (c) a set of behavioral reports that describe the kinds of behaviors that the agent is capable of.

Rather than providing absolute proof of sensemaking in an agent, the proposed method produces a set of detailed pro and con arguments for consideration by the research community. Continued work on refining this method and these arguments will provide a pathway toward more effectively evaluating the ability of autonomous agents to understand their environment. In turn, this will help us design agents that have better sensemaking capabilities and that exhibit more interesting forms of self-motivation.

Fig. 3 :

 3 Scheme

Fig. 4 :

 4 Fig. 4: The Small Loop platform in NetLogo. The environment (left) is the loop of white squares surrounded by green walls. The brown arrowhead represents the agent. The experimenter can preset the values of the primitive interactions using the slider controls (center). The Interaction-Value window shows the sequence of the primitive interactions enacted by the agent, with ASCII characters encoding the moves and effects. The Bump Count graph (right) displays the number of times the agent bumps into a wall (cumulative total in blue),showing that the agent gradually learns to avoid bumping into walls. When the agent touches/feels a cell, the cell flashes yellow, and when the agent bumps into a wall, the wall flashes red, making the agent's behavior intelligible to the experimenter.

Fig. 5 :

 5 Fig.5: Example trace of Ernest in Experiment 1. Tape 1 represents the primitive interactions enacted over time: feel empty (white squares), feel wall (green squares), turn (half-circles), step forward (white triangles), bump (red triangles); the upper part represents interactions to the left, the lower part interactions to the right. Tape 1 shows that Ernest learned to feel in front before trying to move forward from step 190 on, but still got puzzled in the upper right area of the loop around step 220 and 270. Ernest finally learned to avoid bumping after step 276. Tape 2 represents the interaction values as a bar graph (green when positive, red when negative); it shows that Ernest got more consistently positive interactions from step 290 on, meaning that it learned to fulfill its interactional motivation. Tape 3 represents the level of the enacted sequence in the hierarchy of learned sequences; it shows that Ernest gradually exploited higher-level sequences; black segments show sequences of interactions that are interrupted due to an unexpected effect when trying to enact a constituent primitive interaction. For example, on steps 62 and 69, Ernest expected to enact the sequence "feel front empty -move forward", but this sequence was interrupted due to the unexpected feeling of a wall in front. Together, the three tapes show that Ernest learned to characterize left corners (situations with walls in front and on the left) by the sequence that leads to them when circling the loop counterclockwise: "feel empty to the left, turn left, move forward, feel wall in front". In this left corner context, Ernest learned to choose turn right(steps 318, 354, 390), which allowed it to engage in full tours of the loop. The average value obtained after behavior stabilization was of 5×12 (move forward) -3×6 (turn) -1×17 (touch), corresponding to 0.71 points/step.

Fig. 6 :

 6 Fig.6: Example trace of Ernest in Experiment 2, with positive values for turning and negative values for all other interactions (turn right: 5, turn left: 10, step forward -5). This trace uses the same symbols as Fig.5. Tape 1 shows that Ernest tries different interactions before finding that it can keep turning to the right to constantly receive positive values. Tape 2 shows that Ernest receives constant positive values from step 34 on. Tape 3 shows that Ernest gradually learns sequences that consist of enacting a series of right turns (a series of 2 right turns from step 42 on, then a longer series not represented in the figure). In this run, Ernest never tried to turn to the left, so it never learned that it could receive even higher values (10) by turning to the left. This shows again that Ernest's learning is suboptimal (in terms of interactional motivation). Higher probability of finding optimal behaviors could be obtained by adding other forms of intrinsic motivation such as boredom avoidance, curiosity, or variable interactional motivation depending on internal states.

Olivier L. Georgeon, James B. Marshall

Olivier L. Georgeon, James B. Marshall

http://liris.cnrs.fr/ideal/demo/small-loop/

Acknowledgments

This work was supported by the French Agence Nationale de la Recherche (ANR) contract ANR-10-PDOC-007-01 and by a research fellowship from the Collegium de Lyon.