Olivier L Georgeon
email: olivier.georgeon@liris.cnrs.fr

Mark A Cohen
email: mcohen@lhup.edu

Amélie V Cordier
email: amelie.cordier@liris.cnrs.fr

A Model and Simulation of Early-stage Vision as a Developmental Sensorimotor Process

Keywords: Cognitive development, Intrinsic Motivation, Artificial Intelligence, Cognitive Science, Intelligent agents, Machine learning, Computer simulation

Theories of embodied cognition and active vision suggest that perception is constructed through interaction and becomes meaningful because it is grounded in the agent's activity. We developed a model to illustrate and implement these views. Following its intrinsic motivation, the agent autonomously learns to coordinate its motor actions with the information received from its sensory system. Besides illustrating theories of active vision, this model suggests new ways to implement vision and intrinsic motivation in artificial systems. Specifically, we coupled an autonomous sequence learning mechanism with a visual system. To connect vision with sequences, we made the visual system react to movements in the visual field rather than merely transmitting static patterns.

Introduction

We address the question of how autonomous agents can learn sensorimotor contingencies-contingencies between the agent's motor actions and the signal received from the sensors. We propose a model that learns such contingencies in rudimentary settings. The agent has primitive possibilities of interaction in a twodimensional grid, and distal sensors that reflect some remote properties of the grid. This novel learning process is driven by intrinsic motivations hard coded in the agent, and results in the agent gradually improving its capacity to exploit distal sensory information to orient itself toward targets in the grid.

The idea that visual perception is actively constructed through interaction was proposed by theories of active vision [e.g., 1]. Specifically, O'Regan and Noë [START_REF] O'regan | A sensorimotor account of vision and visual consciousness[END_REF] proposed the sensorimotor hypothesis to vision. They used the metaphor of a submarine controlled from the surface by engineers, but with connections that have been mixed up by some villainous marine monster. They argue that the engineers would have to learn the contingencies between the commands they send and the signals they receive. O'Regan and Noë's sensorimotor hypothesis to perception postulates that making sense of perception precisely consists of knowing these contingencies. Following these views, we seek to demonstrate that our agent can autonomously learn such contingencies. From an observer's viewpoint, the agent would seem to learn to make sense of its perceptions with regard to its intrinsic motivations.

To address this problem we rely on our previous work on intrinsically-motivated hierarchical sequence learning [START_REF] Georgeon | An Algorithm for Self-Motivated Hierarchical Sequence Learning[END_REF]. In this previous work, we implemented an original algorithm that learned regularities of interaction. We demonstrated that an agent could use this algorithm to learn sequential contingencies between touch interactions and move interactions to avoid bumping into obstacles. In this paper, we report an extension of this algorithm that allows the agent to learn contingencies when perception is not a direct feedback from motion. For example, unlike touch in our previous agent, vision does not directly result from motion. Yet, because our previous algorithm succeeded in learning "touch/motor" contingencies, we expect it to prove useful for learning "visio/motor" less direct contingencies. Specifically, we envision coupling the algorithm with a complementary sensory mechanism as suggested by theories of dual process [e.g., 4].

More broadly, this work seeks to model and simulate ab-nihilo autonomous learning, sometimes referred to as bootstrapping cognition [START_REF] Dennett | Brainchildren. Essays on designing minds[END_REF]. We relate this developmental approach to Piaget's [START_REF] Piaget | The construction of reality in the child[END_REF] notion of an early stage in human's ontological development (pre-symbolic). For this work, though, this early-stage notion can also fit the framework of phylogenetic evolution of animal cognition, as discussed for example by Sun [START_REF] Sun | Desiderata for cognitive architectures[END_REF].

Because this work focuses on autonomous cognitive development, we do not follow traditional methods of task modeling that use either a symbolic cognitive modeling approach [START_REF] Newell | Unified Theories of Cognition[END_REF] or a sub-symbolic approach (e.g., [START_REF] Brooks | Intelligence without representation[END_REF]). Instead, we situate ourselves in the area of studies on intrinsic motivation [START_REF] Blank | Bringing up robot: Fundamental mechanisms for creating a self-motivated, self-organizing architecture[END_REF][START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] that seek to implement agents whose behavior organizes autonomously with no pre-assumed goal.

Background

This work implements an intrinsically-motivated schema mechanism [START_REF] Georgeon | An Algorithm for Self-Motivated Hierarchical Sequence Learning[END_REF][START_REF] Georgeon | An Intrinsically-Motivated Schema Mechanism to Model and Simulate Emergent Cognition[END_REF] coupled with a distal sensory system (Figure 1). In essence, the intrinsically-motivated schema mechanism implements Piaget's [START_REF] Piaget | The construction of reality in the child[END_REF] views that perception and action should be kept embedded in sensorimotor schemas rather than separated in the traditional perceptioncognition-action loop. We use the terms schemas, interaction patterns, and sensorimotor behaviors interchangeably, and we represent them as small white rectangles (primitive schemas) and gray rectangles (learned composite schemas) in Figure 1.

The mechanism records past sequences of interaction as hierarchically organized interaction patterns (episodic memory in Figure 1). The mechanism also maintains a representation of the agent's current situation (in what we have labeled working memory). The representation of the current situation takes the form of interaction patterns, in compliance with Gibson's [START_REF] Gibson | The ecological approach to visual perception[END_REF] ecological theory, according to which agents represent the world in the form of affordances, i.e., possibilities of interaction. The consistency of knowledge representation across the two memory systems makes it possible to match past episodes against the current situational representation to select forthcoming behavior.

Through interaction, the mechanism learns composite behaviors, in a bottom-up way, as the agent experiences their corresponding sequences through its activity. Composite behaviors consist of hierarchically-organized subsequences of lower-level behaviors, all the way down to pre-defined primitive behaviors (the sequential relation in the schema hierarchy is represented by arc arrows between the branches of the hierarchy in Figure 1). This schema mechanism implements intrinsic motivation by associating satisfaction values with schemas. The modeler initializes the agent with inborn satisfaction values associated with primitive schemas. The agent then selects newlylearned schemas based on their expected satisfaction in specific contexts, balanced with the expectation of success in such contexts. That is, satisfaction values operate in a proactive way during the selection of forthcoming behavior. In turn, the agent makes a selection of composite schemas based on the satisfaction that effectively results from enacting these schemas.

In this paper, we report an experiment where the agent has six primitive behaviors. Each primitive behavior consists of the association of primitive actuators with binary feedback. These six primitive interaction patterns are listed in the first six lines of Table 1. Similar to our previous work [START_REF] Georgeon | An Algorithm for Self-Motivated Hierarchical Sequence Learning[END_REF], the binary feedback corresponds to a proximal sense that can be thought of as touch. In our experiment, if the agent tries to move forward, he can either succeed and touch no wall, which leaves him indifferent; or bump into a wall, which he dislikes. When the agent turns, he receives tactile information about the adjacent square that he turned towards (touched a wall: dislike, or not touched: indifferent).

The Model

Now that we have presented the intrinsically motivated schema mechanism, we can examine the issue of implementing an agent that learns to coordinate actions with sensory inputs, when both are not a priori connected together within sensorimotor schemas.

As a start, we have implemented the rudimentary distal sensory system depicted in Figure 2. This system consists of two eyes that detect the blue color in the environment (target in Figure 2). Each eye has a visual field that covers 90° of the agent's surrounding environment. The two visual fields overlap in the line straight in front of the agent, including the agent's location. Each eye generates a single integer value that indicates the amount of blue color detected, also reflecting the distance to a blue square if there is only one. This distal sensory system can thus be seen as a rudimentary monochromic visual system with a resolution of two pixels.

We want the visual system to forward visual information to the agent's situational representation in working memory to inform the selection of behavior. Inspired by the dual process argument, we first considered an iconic approach that would incorporate the two-pixel icon provided by the visual system as an element of context in our existing schemas. This approach proved inefficient because it added too much random combination in the schema mechanism and the agent was unable to learn the contingency between the perceived icons and the actions. Moreover, the combinatorial growth would be prohibitive with wider icons.

To better support contingency learning, we looked for inspiration from studies of biological primitive organisms. We found useful insights from the limulus (horseshoe crab), an archaic arthropod whose visual system has been extensively studied [START_REF] Shuster | The american horseshoe crab[END_REF]. Specifically, we retained the two following principles: 1) Sensibility to movement: the signal sent to the brain does not reflect static shape recognition but rather reflects changes in the visual field. A horseshoe crab's "eye is highly sensitive to images of crab-size objects moving within the animal's visual range at about the speed of a horseshoe crab (15 cm/s)" [15, p. 172].

2) Visio-spatial behavioral proclivity: male horseshoe crabs move toward females when they see them with their compound eyes, whereas females move away from other females.

Similar to our agent, horseshoe crabs' eyes are fixed to their body and have poor resolution (roughly 40*25 pixels).

From these insights, we modeled the visual system so that it updated the schema mechanism only when a change occurred in the visual field. We identified four different signals that each eye would generate: appear, closer, reached, and disappear. Additionally, to generate a visio-spatial behavioral proclivity, the schema mechanism receives an additional intrinsic satisfaction associated with each of these signals. These four signals are listed with their satisfaction values in the last four lines of Table 1. For example, an eye sends the signal closer when the amount of blue color has increased in this eye's visual field over the last interaction cycle, meaning the square has gotten closer (with this regard, our agent's visual acuity is more than two pixels because the agent can detect the enlargement of the target's span in the visual field). We associate the closer signal with a positive inborn satisfaction value [START_REF] Blank | Bringing up robot: Fundamental mechanisms for creating a self-motivated, self-organizing architecture[END_REF] to generate the proclivity to move toward blue squares.

With these settings (as reported in Table 1), we expect our agent to learn to coordinate its actions with its perception and orient itself toward the blue square. We must note that nothing tells the agent a priori that moving would, in some contexts, get it closer, or that turning would shift the blue color in the visual field. These are the kind of contingencies that the agent will have to learn through experience.

After an initial learning phase, we expect to see different behaviors emerge. One possible behavior is the diagonal strategy depicted in the left side of Figure 3. This behavior consists of alternatively moving forward and turning toward the blue square until the agent becomes aligned with the blue square. At this point, the agent will continue to move forward.

Another possible behavior is the tangential strategy depicted in the right side of Figure 3. The tangential strategy consists of approaching the blue square in a straight line. The trick with the tangential strategy is that the agent cannot accurately predict when he should turn toward the blue square before he passed it. The tangential strategy thus consists of moving on a straight line until the blue square disappears from the visual field, then returning one step backward, and then turning toward the blue square. Of course, such specific strategies have probably little to do with real horseshoe crabs. These strategies would only arise due to the coupling of our environment's topological structure with our agent's sensorimotor system, intrinsic motivations, and learning skills.

The Experiment

We use the vacuum environment implemented by Cohen [START_REF] Cohen | Teaching agent programming using custom environments and Jess[END_REF] as an experimental test bed for our agent. This environment is based on Russell and Norvig's [START_REF] Russell | AI: A modern approach[END_REF] general vacuum cleaner environment. Figure 3 shows the agent in this environment. Filled squares around the grid are walls that the agent will bump into if he tries to walk through them. The agent's eyes are represented by quarters of circles that turn a blue color when they detect a blue square; the closer the blue square, the more vivid the eye's color. When both eyes send a reached signal, this signal triggers an additional systematic eating behavior (with no additional satisfaction value) that makes the agent "eat" the blue square and makes the blue square disappear. The observer can thus see the agent's behavior as a quest for food. The observer can click on the grid to insert a new blue square when the agent has eaten the previous one.

We provide online videos of different runs of the agent . At the beginning, these videos show the agent acting frantically because it has not yet learned the contingency between its actions and its perceptions. The agent picks random behaviors when it has no knowledge of what to do in a specific context. It learns to categorize contexts in terms of possibilities of behavior, in parallel with learning interesting composite behaviors. After the initial pseudo random activity, the videos show the agent more often orienting itself toward the blue square. After eating one or two blue squares, the agent starts to stick to a specific strategy. Our website shows example videos where the agent learned the diagonal strategy 1 and where it learned the tangential strategy 2 . Traces of these runs are reported in Table 2 and Table 3, and discussed below to provide a deeper view of the learning process.

In the traces, the numbers indicate the interaction-cycle counter also displayed in the bottom-right corner of the video. The symbols used to represent the agent's primitive behaviors are ^ (turn left), > (try to move forward), v (turn right). The symbols that represent the eyes' signals are * (appear), + (closer), x (arrived), o (disappear). The eye signals are represented on each side of a | character, the left eye signal being on the left and the right eye signal on the right. When none of the eyes sends a signal, the | character is omitted. The entire sensorimotor pattern is displayed within parentheses when a true feedback is received and within angle brackets when a false feedback is received. For example, the agent bumped a wall on step 8 and 9, and turned toward an adjacent wall on steps 11 and 24. Simultaneously, a blue square appeared in the left eye's visual field on step 10, and disappeared on step 11. 2 shows pseudo random behavior. On steps 21 to 23, the agent learned the sequence consisting of getting closer twice, then loosing the detection. On step 36 to 39, it learns a step of the stairstep progression necessary for the diagonal strategy: turn-right, move-forward, turn-left, move-forward, with the appropriate visual signals associated. It eats its first blue square on step 59. By clicking on the grid, we inserted the second blue square on step 69. From then on, the agent enacted a perfect diagonal strategy and reached the second blue square on step 82. The video shows that the agent continues enacting the diagonal strategy to reach subsequent blue squares.

The beginning of the trace in Table

Table 3 reports the trace of a run where the agent learned the tangential strategy. In this run, the agent learned the behavior consisting of moving forward and getting closer to the blue square (on the right eye) from the beginning (from step 2 to 6). Due to the positive satisfaction experienced, it sticks to this behavior until it loses the detection on step 7. A partial component of the returning sequence is then discovered on steps 7 to 11, but the agent then gets away from the blue square. This partial returning sequence is then enacted again in steps 47 through 51, and followed by a move forward and getting closer in both eyes on step 52, but then, the agent moved away again. The agent finally learns to stick to moving forward when in alignment with the blue square during steps 60 to 62. It finally eats its first blue square on step 62. By clicking on the grid, we added the second blue square on step 75. From then on, the agent was able to put together all the subsequences of the tangential strategy that it has learned. It enacts a perfect tangential strategy to reach the second blue square on step 88. The video shows that the agent then keeps enacting the tangential strategy to reach subsequent blue squares. Other runs show that the agent always learns a strategy within the first hundred steps, and that the most frequently found strategy is the diagonal strategy, with the settings defined in Table 1. The experiment therefore demonstrates that the agent always succeeds in learning sensorimotor contingencies.

The pre-defined primitive satisfaction values impact the frequency of the resulting strategies; for example, a negative satisfaction value for turning would favor the tangential strategy because it incites the agent to minimize turning. In addition, the location where the experimenter introduces new blue squares influences the resulting strategy. For example, placing blue squares next to walls disfavor the tangential strategy because the agent would often bump the wall, which it does not like. The experimenter thus plays a role in training the agent.

Results and discussion

This work demonstrates a technique for implementing vision as an embodied process in an artificial agent. In this technique, the visual system does not send static images to the central system, but rather sends signals denoting change in the visual field. Notably, this technique allows the agent to see static objects, because changes in the visual field can result from the agent's own movements. This technique is consistent with studies of horseshoe crabs that show male horseshoe crabs can orient themselves toward static females because their visual system reacts to female-like objects that appear to be moving (relatively to their own speed) in a uniform background (a sandy shallow ocean bottom or beach) [START_REF] Shuster | The american horseshoe crab[END_REF].

By succeeding in learning the contingencies between actions and sensors, our model illustrates a sensorimotor theory of vision [START_REF] O'regan | A sensorimotor account of vision and visual consciousness[END_REF]. O'Regan and Noë argue that this approach would also be consistent with complex visual systems, including human vision. This argument rests upon experiments that show that human vision requires eye movements [e.g., 18]. In future studies, we will further investigate this theory by endowing our agent with the capacity to move its eyes independently from its body, and with better eye resolution, so it can learn contingencies between eye saccades and the received visual signals.

This work also demonstrates that our intrinsically motivated schema mechanism can be used as a general framework to model perception and action. The principles of this study can be followed to model other senses, based on the detection of features other than basic changes in the visual field. Moreover, this framework helps model aspects of ontogenetic development of cognition in a novel way because:

a) The agent's goals emerge from low-level drives. The agent does not encode strategies or task procedures defined by the programmer, but rather autonomously constructs a strategy, as opposed to traditional cognitive models [START_REF] Newell | Unified Theories of Cognition[END_REF]. This accounts for theories of cognition that see natural cognitive systems not as being born with an explicit final goal to achieve, but rather pushed by primitive drives [e.g., 6].

b) The agent's instances are capable of "individuating" themselves through their experience, i.e., acquiring their own cognitive individuality that was not encoded in their "genes". This accounts for the role of individual experience in cognitive development.

Few studies have investigated the sensorimotor hypothesis to vision. Moller & Wolfram [START_REF] Moller | Bootstrapping Cognition from Behavior--A Computerized Thought Experiment[END_REF] proposed a model that learned to distinguish between dead-ends and corridors in visual scenes. Their agent internally simulated its own way through the scene, based on its own experience previously acquired through interaction, to discover a dead-end or a corridor. Moller and Wolfram's objectives, however, differ from our own in that they focused on visual processing of complex images rather than on initial sensorimotor contingency learning. Their work, nevertheless, motivates us to endow our agent with the capacity to simulate courses of action internally to more efficiently distinguish scenes and choose forthcoming behaviors.

Other studies refer more loosely to the sensorimotor hypothesis of vision to categorize images of objects through representing the objects' functions [e.g., [START_REF] Bogoni | More than just shape: a representation for functionality[END_REF][START_REF] Stark | Achieving generalized object recognition through reasoning about association of function to structure[END_REF].

In the domain of modeling the visual system of natural organisms, Barlow and coauthors [START_REF] Barlow | Limulus vision in the marine environment[END_REF] proposed a computerized model of the horseshoe crab's visual neural system. This model, however, did not simulate the horseshoe crab's behavior, nor did it functionally represent the connection between the animal's perceptions and its behaviors.

Conclusion

This work demonstrates that it is possible to implement a sensorimotor theory of vision in rudimentary settings. To do so, we proposed a novel intrinsically motivated schema mechanism coupled with a visual system that detects dynamic features in the visual field. The agent has primitive drives that push it to learn to coordinate motor actions with sensory information to enact behaviors that, in turn, increase the agent's fulfillment of its primitive drives. This work opens the way to more complex models where the eye's resolution will be increased and where the agent will have the capacity to move its eyes independently from its body. Such developments inform our understanding of visual systems in natural organisms and suggest new techniques to implement vision in autonomous robots.

Figure 1 .

 1 Figure 1. The intrinsically-motivated schema mechanism coupled with the distal sensory system.

Figure 2 .

 2 Figure 2. The agent's vision of its environment.

Figure 3 .

 3 Figure 3. Two possible strategies: diagonal strategy (left) and tangential strategy (right).

11 2 (

 2 > |+) 3(v*|o) 4(vo| ) 5(v) 6(^) 7(>) 8[>] 9[>] 10(^*| ) 11[vo| ] 12(^*| ) 13(^o|*) 14(^ |o) 15(>) 16(v |*) 17(v*|o) 18(>+| ) 19(^o|*) 20(v*|o) 21(>+| ) 22(>+| ) 23(>o| ) 24[v] 25[>] 26(^) 27(>) 28[v] 29(v |*) 30(v*|o) 31(>+| ) 32(^o|*) 33(v*|o) 34(>+| ) 35(^o|*) 36(v*|o) 37(>+| ) 38(^o|*) 39(> |+) 40(^ |o) 41(v |*) 42(> |+) 43(v*| ) 44(^o| ) 45(> |o) 46(^) 47(>) 48(v) 49(^) 50(v) 51(v |*) 52(> |+) 53(v*|o) 54(>+| ) 55(^ |*) 56(>+|+) 57(^o| ) 58(v*| ) 59(>x|x) 60(vo|o) 61(v) 62(v) 63(v) 64(v) 65(v) 66(v) 67(v) 68(v) 69(v |*) 70(> |+) 71(v*|o) 72(>+| ) 73(^o|*) 74(> |+) 75(v*|o) 76(>+| ) 77(^o|*) 78(> |+) 79(v*|o) 80(>+| ) 81(^ |*) 82(>x|x) 83(>o|o) ...

1 2 (

 2 > |+) 3(> |+) 4(> |+) 5(> |+) 6(> |+) 7(> |o) 8(v |*) 9(v*|o) 10(>+| ) 11(^ |*) 12(^o| ) 13(^ |o) 14(^*| ) 15(>o| ) 16(>) 17(>) 18(^*| ) 19(vo| ) 20(v) 21(^) 22(>) 23(^*| ) 24(^o|*) 25(^ |o) 26(^) 27(v) 28(v |*) 29(^ |o) 30(^) 31(>) 32(^*| ) 33(^o|*) 34(v*|o) 35(>+| ) 36(^o|*) 37(v*|o) 38(>+| ) 39(^ |*) 40(v |o) 41(>o| ) 42(>) 43(^*| ) 44(^o|*) 45(> |+) 46(> |+) 47(> |o) 48(v |*) 49(v*|o) 50(>+| ) 51(^ |*) 52(>+|+) 53(v |o) 54(vo| ) 55(v |*) 56(v*| ) 57(v |o) 58(^ |*) 59(>+|+) 60(>+|+) 61(>+|+) 62(>x|x) 63(>o|o) 64(v) 65(v) 66(v) 67[v] 68(^) 69[v] 70(v) 71(v) 72(v) 73[v] 74[>] 75(^*| ) 76(^o|*) 77(> |+) 78(> |+) 79(> |+) 80(> |+) 81(> |+) 82(> |o) 83(v |*) 84(v*|o) 85(>+| ) 86(^ |*) 87(>+|+) 88(>x|x) 89(^o|o) ...

Table 1 .

 1 Primitive actuators and sensors.

	Symbols Actuators Sensors	Description	Intrinsic
					satisfaction
	^ (^)	Turn left	True	Turn 90° left toward adjacent empty square	0 (indifferent)
	[^]		False	Turn 90° left toward adjacent wall	-5 (dislike)
	> (>)	Forward	True	Move forward	0 (indifferent)
	[>]		False	Bump wall	-8 (dislike)
	v (v)	Turn right True	Turn 90° right toward adjacent empty square	0 (indifferent)
	[v]		False	Turn 90° right toward adjacent wall	-5 (dislike)
	*		Appear	Target appears in distal sensor field	15 (love)
	+		Closer	Target approaches in distal sensor field	10 (enjoy)
	x		Reached	Target reached according to distal sensor	15 (love)
	o		Disappear Target disappears from distal sensor field	-15 (hate)

Table 2 .

 2 Trace of diagonal strategy learning.

Table 3 .

 3 Trace of tangential strategy learning.

http://e-ernest.blogspot.com/2011/01/ernest-82-can-find-his-food.html

http://e-ernest.blogspot.com/2011/01/tengential-strategy.html

Acknowledgments. This work was supported by the Agence Nationale de la Recherche (ANR) contract RPDOC-2010-IDEAL. We gratefully thank Pr. Alain Mille for his useful comments.


Olivier L Georgeon

David W Aha
email: david.aha@nrl.navy.

Haris Dindo

James Marshall

Giovanni Pezzulo

The Radical Interactionism Conceptual Commitment

Keywords: constructivist learning, sensorimotor modeling, self-programming, constitutive autonomy, self-motivation, Enaction

We introduce Radical Interactionism (RI), which extends Franklin et al. 's (2013) Cognitive Cycles as Cognitive Atoms (CCCA) proposal in their discussion on conceptual commitments in cognitive models. Similar to the CCCA commitment, the RI commitment acknowledges the indivisibility of the perception-action cycle. However, it also reifies the perception-action cycle as sensorimotor interaction and uses it to replace the traditional notions of observation and action. This complies with constructivist epistemology, which suggests that knowledge of reality is constructed from regularities observed in sensorimotor experience. We use the LIDA cognitive architecture as an example to examine the implications of RI on cognitive models. We argue that RI permits selfprogramming and constitutive autonomy, which have been acknowledged as desirable cognitive capabilities in artificial agents.

Introduction

In their paper "Conceptual Commitments of the LIDA Model of Cognition", Stan Franklin, Steve Strain, and Ryan McCall invite Artificial General Intelligence (AGI) researchers to discuss which conceptual commitments are essential for AGI agents. In response to this invitation, we wish to further discuss their fourth commitment: Cognitive Cycles as Cognitive Atoms (CCCA). On page 9, they write: "we hesitate to propose it as important for the AGI research in general, since to our knowledge, no other system-level cognitive architecture makes such a commitment". We make this commitment in our Enactive Cognitive Architecture (ECA, Georgeon, Marshall, and Manzotti, 2013). Moreover, we extend this commitment into a more radical one called Radical Interactionism (RI).

In Section 4.4, Franklin et al. define a cognitive cycle as a cycle that begins by sampling (sense) the environment and ends by selecting an appropriate response (action), traversing various phases including perception, understanding, consciousness, and learning. Modeling such a cognitive cycle as a cognitive atom entails considering it as indivisible, as opposed to dividing it into a sequence of separate tasks. To our understanding, LIDA acknowledges this indivisibility by implementing cognitive cycles through asynchronous distributed processes. Since LIDA's design imposes no synchronicity on the distributed processes that generate the cognitive cycle, it is indeed not divided into a predefined sequence. In our view, this approach embodies CCCA in which "atom" denotes a sense of indivisibility. We introduce a more radical view that additionally uses "atom" as a primitive notion for designing the model.

Our Radical Interactionism conceptual commitment invites designers of cognitive models to consider the notion of sensorimotor interaction as a primitive, instead of perception and action. This is analogous to a mathematical system's primitives, which are used in axioms and theorems to define more complex structures, but which are themselves undefined within the system. Traditional cognitive models take perceptions and actions as primitive notions, and derive the notion of sensorimotor interaction from them. The RI conceptual commitment recommends doing the opposite. Although the CCCA commitment takes a first step in the RI direction by considering cognitive cycles as indivisible, Franklin et al. still define a cognitive cycle using the primitive notions of observation and action. The RI commitment eliminates these as primitives and instead frames each cognitive cycle as a sensorimotor interaction.

Intuitively, a sensorimotor interaction fits Franklin et al.'s description of a cognitive cycle: "Each cycle constitutes a unit of sensing, attending and acting. A cognitive cycle can be thought of as a moment of cognition, a cognitive moment" (p4). Within constructivist epistemology, sensorimotor interactions can be viewed as representing a Piagetian (1955) sensorimotor scheme, from which the subject constructs knowledge of reality. At a philosophical level, this view relates to phenomenology (e.g., [START_REF] Dreyfus | Why Heideggerian AI failed and how fixing it would require making it more Heideggerian[END_REF], which argues that knowledge of the self and of the world derives from regularities in phenomenological experience. Accordingly, a sensorimotor interaction may be understood as a chunk of phenomenological experience, making the stream of sensorimotor interactions the primitive material from which the agent constructs all of its knowledge.

RI Impact on cognitive modeling

As an example to illustrate the RI commitment, we examine how it would impact the LIDA model. It would imply modifying the left-side part of Figure 1 The algorithm begins with a predefined set of sensorimotor interactions I, called primitive interactions. At a given time t, the agent chooses a primitive interaction i t that it intends to enact, from among I. The agent ignores this enaction's meaning; that is, the agent has no rules that would exploit knowledge of how the designer programmed the primitive interactions through actuator movements and sensory feedback (such as: "if a specific interaction was enacted then perform a specific computation"). As a response from the tentative enaction of i t , the agent receives the enacted interaction e t , which may differ from i t . The enacted interaction is the only data available to the agent that carries some information about the external world, but the agent ignores the meaning of this information. As an example, the primitive interaction i t may correspond to actively feeling (through touching) an object in front of the agent, involving both a movement and a sensory feedback. The tentative enaction of i t may indeed result in feeling an object, in which case e t = i t , or may result in feeling nothing, if there is no object in front of the agent, in which case the enacted interaction e t corresponds to a different interaction: moving while feeling nothing. The agent constructs knowledge about its environment and organizes its behavior through regularities observed in the sequences of enacted interactions.

Within the RI commitment, LIDA would not have to construct action schemes defined "as an action together with its context and expected result" (Franklin et al., p5, citing Drescher, 1991). Procedural and episodic memories would instead directly process sensorimotor interactions. The learning mechanism would construct higher-level sensorimotor interactions, called composite interactions, as pairs of a context interaction and an intention interaction: i composite = 〈i context , i intention 〉. Such a learning mechanism would be bottom-up, and would generate a hierarchy of composite interactions where each level would contain lower-level composite interactions, except the bottom level, which would be made of primitive interactions. At any time, the agent would represent its current situation as a set of interactions, called interactional context. The LIDA behavior selection mechanism would be modified to activate composite interactions i composite whose context interaction i context belongs to the interactional context. The intention interactions i intention of the activated composite interactions i composite would compete to be the one selected as the next intended interaction. Since the intended interaction can be a primitive interaction or a previously learned composite interaction, this mechanism allows asynchrony between the architecture's behavior-selection mechanism and the primitive interactional mechanism, as illustrated in Figure 2. The coupling between the cognitive architecture and the environment (represented by the Internal & External Environment Constructed at Time t in Figure 2) evolves as the agent learns increasingly sophisticated patterns of behaviors, while simultaneously representing the world in terms of increasingly sophisticated affordances. We refer the reader to other publications for a more thorough description of this mechanism [START_REF] Georgeon | An intrinsically-motivated schema mechanism to model and simulate emergent cognition[END_REF], its implications in a cognitive architecture [START_REF] Georgeon | ECA: An enactivist cognitive architecture based on sensorimotor modeling[END_REF], and demonstrations in which primitive interactions consist of active feeling (e.g., Georgeon and Marshall, 2013) and rudimentary active vision [START_REF] Georgeon | A model and simulation of early-stage vision as a developmental sensorimotor process[END_REF].

Conclusion

Rather than offering a method to address the traditional perception-action problem, Radical Interactionism presents a profoundly different formulation of this problem. Indeed, many studies define the perception-action problem as learning a mapping between a predefined action space and a predefined observation space. For example, these include studies in reinforcement learning (e.g., [START_REF] Sutton | Reinforcement learning: An introduction[END_REF], means-end analysis (e.g., [START_REF] Drescher | Made-up minds, a constructivist approach to artificial intelligence[END_REF], and robotics (e.g., [START_REF] Pierce | Map learning with uninterpreted sensors and effectors[END_REF]. In contrast, the RI approach begins with a predefined interactional space, and focuses on the problem of generating learning effects that are considered important in AGI agents, namely self-programming (e.g., [START_REF] Thórisson | Approaches and Assumptions of Self-Programming in Achieving Artificial General Intelligence[END_REF] and constitutive autonomy, which [START_REF] Froese | Enactive artificial intelligence: Investigating the systemic organization of life and mind[END_REF] define as an agent's ability to "self-constitute its identity", which they argue is a prerequisite for autonomous sense-making. RI agents realize selfprogramming because they learn increasingly long sequences of interactions and re-enact these in appropriate contexts. They have constitutive autonomy because their coupling with the environment evolves through their individual experience of interaction.

Moreover, RI allows implementing a type of self-motivation called interactional motivation [START_REF] Georgeon | Interactional motivation in artificial systems: between extrinsic and intrinsic motivation[END_REF]. To implement interactional motivation, a designer attaches a predefined numerical valence to some primitive interactions, and biases the behavior selection mechanism to select sequences of interactions that have the highest total valence. Interactional motivation provides a way to specify inborn preferences (some primitive interactions that the agent innately likes or dislikes) but does not specify the agent's goal; the agent thus engages in open-ended learning to find its own way to fulfill its inborn preferences of interaction. This view implements Glasersfeld's (1984, p29) idea that goals "arise for no other reason than this: a cognitive organism evaluates its experiences, and because it evaluates them, it tends to repeat ones and avoid others".

Associated with interactional motivation, the RI commitment attacks the problem of designing self-motivated agents that learn to master lawful regularities in the flow of sensorimotor contingencies (SMCs, O'Regan and Noë, 2001). Notably, RI agents do not have to learn the first category of SMCs identified by Buhrmann, Di Paolo, and Barandiaran (2013): the sensorimotor environment, which they "innately" know through the set of primitive interactions I. However, they must construct intentional actions-actions that they can choose knowingly of their anticipated effects (e.g., [START_REF] Engel | Where's the action? The pragmatic turn in cognitive science[END_REF]. Concurrently, they must learn the existence of possibly persistent entities in the environment that they can observe, displace, transform, or use through intentional actions.

With regard to epistemological theories, RI relates to Glasersfeld's radical constructivism, whose "radical difference [from traditional conceptualizations] concerns the relation of knowledge and reality. Whereas in the traditional view of epistemology, as well as of cognitive psychology, that relation is always seen as a more or less picture-like (iconic) correspondence or match, radical constructivism sees it as an adaptation in the functional sense" (Glasersfeld, 1984, p20). Accordingly, RI does not implement a "picture-like" relation in the input received by the model from the environment. In contrast with observations in traditional models, the input received by an RI model from the environment only consists in the enacted interaction, which does not directly represent the environment. Because the input does not directly represent the environment, RI agents can be designed without ontological assumptions about the environment, which is critical for agents that perform open-ended learning in the real world.

  in Franklin et al.'s paper as shown in our Figure 1 below. Since perception and action do not exist in RI, the Sensory Memory and the Motor Plan Execution modules would be removed, as well as the Sensory Stimulus, Actuator Execution, and Dorsal Stream connections. Instead, the Sensory Motor Memory module would directly connect to the Internal & External Environment module through the Intended Interaction and the Enacted Interaction connections. As a result of these changes, stimuli and actions would be eliminated from the architecture and replaced by a single type of primitive object: sensorimotor interactions.

  Figure 1. A diagrammatic adaptation of LIDA to Radical Interactionism. Sensory Motor Memory directly interacts with Internal & External Environment through the Intended Interaction i t and the Enacted Interaction e t . The Other Modules of the Cognitive Model box encompasses the remaining modules of LIDA. At time t, the agent chooses the sensorimotor interaction i t that it intends to enact from among the set of sensorimotor interactions I. The attempt to enact i t may change the environment. The agent then receives the enacted sensorimotor interaction e t .

	Intended Interaction i t ! I
	Internal & External Environment	Sensory Motor Memory	Other Modules of the Cognitive Model
	Enacted Interaction e t ! I

  Levels of interaction in Radical Interactionism. Over time, the agent constructs composite interactions, which implement a series of primitive interactions. At time t, the behavior selection mechanism may select a previously constructed composite interaction i ct ≈ 〈i p1 … i pn 〉 to try to enact. The tentative enaction of i ct is delegated to the Primitive Sensory Motor Memory, which controls the enaction of the sequence of the n primitive interactions i p1 to i pn , and returns the enacted composite interaction e ct to the Composite Sensory Motor Memory. The rest of the cognitive architecture sees the primitive loop as an Internal & External Environment Constructed at Time t, with which it interacts through higher-level interactions. Since the environment that the cognitive architecture interacts with evolves as the agent develops, the agent can recursively learn increasingly complex behaviors.

	Internal & External
	Environment Constructed at Time t
	i p1 # I	Intended Composite
	… i pn # I	Interaction i ct = !i p1 … i pn "!
	Internal & External Environment	Primitive Sensory Motor Memory	Composite Sensory Motor Memory	Other Modules of the Cognitive Model
	e p1 # I
	… e pk # I	Enacted Composite
		Interaction e ct = !e p1 … e pk "
	Figure 2.

Acknowledgements

This work was supported by the French Agence Nationale de la Recherche (ANR) contract ANR-10-PDOC-007-01, and NRL. The views and opinions contained in this paper are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of NRL or the DoD.


Olivier L Georgeon
email: olivier.georgeon@liris.cnrs.fr

Ilias Sakellariou
email: iliass@uom.gr

Designing Environment-Agnostic Agents

Keywords: I.2.6 [Artificial Intelligence]: Learning. I.2.11 [Artificial Intelligence]: Distributed Artificial Intelligence -Intelligent agents Algorithms, Measurement, Design, Experimentation Intrinsic motivation, Autonomous learning, Cognitive development, Enactive cognition, Affordances, Constructivism

We present autonomous agents that are designed without encoding strategies or knowledge of the environment in the agent. The design approach focuses on the notion of sensorimotor patterns of interaction between the agent and the environment rather than separating perception from action. The agent's motivational system is also interaction-centered in that the agent has inborn proclivities to enact certain sensorimotor patterns and to avoid others. Such motivations result in the agent autonomously discovering, learning, and exploiting regularities of interaction afforded by the environment, and constructing operative knowledge of the environment. Because such agents have no predefined goals, we propose a set of behavioral criteria to both judge and demonstrate the agents' capacities, rather than performance measurement. A design platform based on NetLogo is presented. Results show that these agents demonstrate interesting behavioral properties such as hedonistic temperance, active perception, and individuation.

INTRODUCTION

This paper presents an argumentation in favor of the concept of environmental agnosticism as an original and useful concept to think about autonomous agents. We introduce this concept to capture the idea that we do not encode our knowledge of the environment in the agent's decisional mechanism. A typical example of such encoded knowledge would consist of a set of logical rules that specify the agent's behavior in response to specific information received from the environment (sensory input). Another example would consist of modeling the agent's environment as a predefined problem space in which reward values would be associated with predefined problem states and such values propagated across states [START_REF] Laird | The Soar User's Manual Version 9[END_REF] 1 . Instead, we expect environment-agnostic agents to learn the semantics of sensorimotor information and the ontological structure of their world by themselves. 1 Our motivation for studying environment-agnostic agents is both theoretical and practical. On the theoretical level, we believe that this study can shed some light on how knowledge emerges in artificial agents and becomes meaningful to them. This question relates to the symbol grounding problem [START_REF] Harnad | The symbol grounding problem[END_REF] and the study of developmental cognition (e.g., [START_REF] Weng | Artificial intelligence -Autonomous mental development by robots and animals[END_REF]). On the practical level, environment-agnostic agents will facilitate the development of agent-based simulations by unburdening the modeler from encoding knowledge in the agent.

As we intend to show, designing an environment-agnostic agent raises the crucial question of defining the agent's drives. We employ the term drive to emphasize the difference from traditional approaches that employ the terms task or goal. Indeed, we argue that programming an agent to perform a given task or to reach a given goal generally implies specifying how the agent interprets its world. For example, behavioral rules presuppose the semantics of input data, and traditional reward values presuppose goal assessment criteria. Such presuppositions conflict with the environment-agnosticism principle because environment-agnostic agents are precisely expected to learn by themselves how to interpret their world.

An intuitive distinction between drives and goals may be expressed by the fact that drives enforce a bottom-up approach starting from inborn behavioral tendencies toward the possible construction of higher-level goals, while goals follow a top-down approach through the decomposition of a problem into sub-goals. This conception of drives can also be related to the concept of intrinsic motivation (e.g., [START_REF] Blank | Bringing up robot: Fundamental mechanisms for creating a self-motivated, self-organizing architecture[END_REF][START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]) in that the motivation does not come from an external reward. Yet, we acknowledge that the nuance may still seem vague and better pertaining to the domain of philosophy than computer science. At a philosophical level, let us only note here that we find some resonance with Dennett's inversion of reasoning argument [START_REF] Dennett | Darwin's ''strange inversion of reasoning[END_REF]. The purpose of this paper is not to pursue this philosophical discussion any further but to show that this shift of viewpoint is not mere jargon and rhetoric but can have a strong impact on the way we develop autonomous agents.

Our intuition in developing environment-agnostic agents is to put the focus on the interaction between the agent and the environment rather than only on the agent. When designing autonomous agents, we, indeed, must presuppose the possible range of interactions between the agent and the environment. For example, in the case of robots, engineers define the robot's interactions when designing sensors and effectors. In the case of natural organisms, phylogenetic evolution selected the organism's sensorimotor system. In a similar manner, we predefine environment-agnostic agents' interactions in a given environment. We, however, neither presuppose nor specify how the agent should interpret such interactions. Instead, the agent has behavioral drives that define preferred courses of action in the world. The agent learns regularities of interactions and exploits such regularities in turn to better fulfill its drives. This mechanism is explained in more detail in Section 2.

The fact that the agent has no predefined goals raises the question of how to assess its behavior. This question is discussed in Section 3, and illustrated by experiments in Section 4. Section 5 presents the design/simulation platform employed for implementing environment-agnostic agents. Finally, the conclusion discusses the implications and limitations of the concept of environmental agnosticism for future work.

ENVIRONMENTAL AGNOSTICISM

Fundamentally, we believe designing environment-agnostic agents entails considering individual sensorimotor patterns as the atomic elements of cognition, without making an initial distinction between perception and action. This assumption is supported by many theories in cognitive science that argue that perception, cognition, and motion are entangled (e.g., [START_REF] Dennett | Consciousness explained[END_REF][START_REF] Hurley | Consciousness in action[END_REF][START_REF] O'regan | A sensorimotor account of vision and visual consciousness[END_REF][START_REF] Pfeifer | From perception to action: The right direction? In From Perception to Action[END_REF]). Specifically, Piaget [START_REF] Piaget | L'épistémologie génétique[END_REF], proposed the term scheme to refer to sensorimotor patterns. In the rest of this paper, we simply refer to sensorimotor patterns by the term elementary interactions.

Figure 1 illustrates the elementary interactions along the interaction timeline. Elementary interactions are represented by letters (A, B, C, A, D, A …). As introduced in Section 1, these interactions have no associated semantics implemented in the agent. Because of the absence of implemented semantics, these interactions can correspond to anything in the environment, which is why we characterize the agent as environmentally agnostic. As opposed to traditional artificial agents, this agent has no channel by which it would directly "perceive" its environment, but can only discover the structure of the environment through the regularities experienced while enacting these interactions.

Intrinsic drives

In addition to focusing on elementary interactions, we propose implementing inborn proclivity values associated with such interactions (in parenthesis in Figure 1). Accordingly, we provide the agent with a mechanism that tends to seek interactions with positive proclivity values, and to avoid interactions with negative proclivity values. Such a mechanism results in the implementation of primitive intrinsic drives because, subjectively, the agent seems to simply enjoy interactions that have positive proclivity values, and to dislike interactions that have negative proclivity values. As we present later, the difficulty resides in that the agent needs to learn what interactions will result from its choices.

Proclivity values are related to the notions of intrinsic reward (e.g., [START_REF] Singh | Intrinsically motivated reinforcement learning[END_REF]) and value systems (e.g., [START_REF] Pfeifer | From perception to action: The right direction? In From Perception to Action[END_REF][START_REF] Sporns | Embodied cognition[END_REF][START_REF] Sutton | Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction[END_REF]) in reinforcement learning. Because of this relation, our approach can be considered a form of discrete time decision process consisting of learning a policy function that tends to maximize a reward function over time. Traditional algorithms of discrete time decision processes, however, require assumptions incompatible with the agnosticism principle. For example, Markov Decision Process (MDP) algorithms require that the temporal dependency be known a priori, and that the environment be modeled in the form of states that the agent can directly recognize. Partially Observable Markov Decision Process (POMDP) [START_REF] Aström | Optimal control of Markov processes with incomplete state information[END_REF] algorithms offer a way toward eliminating these hypotheses but they require a state evaluation function to assesses a believed state from observational data. The The agent's activity is represented along the interaction timeline. Letters (A, B, C, etc.) represent primitive interactions. Primitive interactions are associated with proclivity values pre-defined by the designer (e.g., (0), (-1), (5), etc.). Through its activity, the agent learns hierarchical sequences of interactions (composite interactions) that capture hierarchical regularities of interaction with a given environment (e.g., (A(DA))(BC)). The agent represents its current situation in the form of these hierarchical composite interactions. Over time, learned composite interactions allow the agent to predict the consequences of its choices depending on the current situation, and, therefore to select interactions that have the best chance to maximize the agent's proclivity in a given environment (e.g., BC). The agent can learn to enact unsatisfying interactions (e.g., B(-1)) to reach situations where it can enact even more satisfying interactions (e.g., C(5)).

state evaluation function needs to be known a priori, which remains incompatible with the agnosticism principle.

Here, we propose the term proclivity because this term conveys the idea that interactions are enacted for their own sake, while the term reward suggests that interactions are enacted for the sake of their outcome. Enacting interactions for their own sake removes the need for a pre-assumed model of the world. With proclivity, knowledge follows from action, while, with reward, action follows from knowledge. Our approach based on proclivity values relates more to continuous case-based reasoning [START_REF] Ram | Continuous case-based reasoning[END_REF] or tracebased reasoning [START_REF] Mille | From case-based reasoning to traces-based reasoning[END_REF] (but unsupervised, with no presupposed knowledge) than to traditional reinforcement learning, as we will develop in Section 2.2.

Notably, we consider this proclivity value mechanism only as an initial value system. In the future, we imagine implementing more elaborated drives, for example drives that may vary according to the agent's internal status (e.g., simulating a form of hunger varying over time).

Knowledge representation

An agent's stream of interaction with a given environment depends both on its decisions and on the unfolding of the environment. That is, the agent may well decide to try to enact an interaction with a high value, but this attempt may result in an actual interaction with a low value, due to unexpected environmental conditions (e.g., the agent may try to enact an interaction consisting of moving forward, but this attempt may result in bumping into a wall). This effect will be further explained in Section 4 and Figure 2. Because of this, the agent needs to learn to predict the interactions that result from its decisions. This raises the question of how to encode the knowledge that the agent learns.

We recommend representing the agent's knowledge as sequences of interactions. This view conforms with Gibson's [START_REF] Gibson | The ecological approach to visual perception[END_REF] notion of affordance. Gibson suggests that the world is not known "objectively" but is rather known in terms of possibilities of interaction, called affordances. Encoding the agent's knowledge as sequences of interactions also follows from the fact that our agent has no other source of information about the environment anyway.

We devised an algorithm in compliance with these principles, called the Intrinsically Motivated Schema mechanism (IMOS) [START_REF] Georgeon | An Intrinsically-Motivated Schema Mechanism to Model and Simulate Emergent Cognition[END_REF]. A significant difference from most existing decision process algorithms is that IMOS does not require Markov's hypothesis that the duration of the temporal dependency be known a priori. It can learn arbitrarily long episodes of interest by autonomously finding their beginning and end points. As opposed to partially observable Markov decision process (POMDP), IMOS does not require that the set of possible hidden states be defined a priori. IMOS recursively organizes episodes in a hierarchy of subepisodes, a higher-level episode being a sequence of lower-level episodes. The proclivity value of a higher-level episode is set equal to the sum of the proclivity values of its sub-episodes, all the way down to predefined primitive proclivity values. The ability to perform such learning stems from the fact that, in the environmentally agnostic approach, the criteria for selecting interesting episodes are incorporated within the learning mechanism in the form of proclivity values. The agent can test hypothetical episodes and progressively select the most satisfying episodes with regard to their value. Figure 1 illustrates this principle by representing learned hierarchical episodes of interaction in the agent's memory ((A(DA))(BC)). The activity at hand reactivates previously learned episodes that match the current situation (A(DA)) which in turn, triggers the subsequent interactions that are the most likely to result in satisfying interactions (as far as the agent can tell at this point in its development). This matching is possible because of the consistency between the representation of the situation and the representation of agent's procedural experience (a form of homoiconicity).

Again, we consider this mode of knowledge representation only as a starting point from which more elaborated representational structures can be derived. In particular, this mode of representation can be coupled with additional structures proposed by other researchers in Piagetian mechanisms, such as synthetic elements (e.g., [START_REF] Perotto | Constructivist anticipatory learning mechanism (CALM): Dealing with partially deterministic and partially observable environments[END_REF]) or bare schemas (e.g., [START_REF] Guerin | A Piagetian model of early sensorimotor development[END_REF]). When implementing such structures, however, the assumptions these structures make about the environment should be explicitly stated. Notably, the purely sequential representations that we suggest here have the advantage of remaining compliant with the principle of environmental agnosticism because the agent has no a priori knowledge of how to represent the world "as such". The agent only knows and learns interactions.

EXPERIMENTAL PARADIGM

As introduced in Section 1, environment-agnostic agents are not designed to improve their performance over time with regard to a predefined problem set, task, or goal. Performance, as traditionally defined, is, therefore, not a property that appropriately accounts for the expected qualities of such agents. Instead, environment-agnostic agents are developed for their qualitative behavioral properties or for their theoretical implications on the study of developmental cognition. To study these agents, researchers need to agree on such expected properties.

This section proposes an initial list of expected properties based on our experience implementing environment-agnostic agents and on existing literature. In particular, Oudeyer, Kaplan, and Hafner [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] noted similar needs to characterize the properties of intrinsically motivated robots. These authors distinguished between three categories of criteria: (a) evolution of internal variables that account for the robot's learning (e.g., accuracy of anticipation or level of detail of learned categories); (b) evolution of external variables that characterize the robot's behavior (e.g., efficiency in the interaction with the environment); (c) evidence of reaching certain well-known developmental stages with regard to psychological or ethological theories.

Internal evaluation criteria

Category (a), the evolution of internal variables, has the advantage of objectivity because these criteria are based on variables implemented in the system. The drawback, however, is that each system has its specific variables, which complicates the comparison across systems. With these criteria, the authors need to clearly explain the significance of the variables.

A typical internal criterion is the growth of the variable that represents the system's satisfaction, as measured by its value system. We can formulate this criterion as: a.1 Principle of objective hedonism.

For example, with the value system introduced in Section 2.1, the agent's objective hedonism is demonstrated by the agent's increasing ability to perform interactions with high values, and avoid interactions with negative values.

Notably, the principle of objective hedonism does not require the agent to reach the optimum value but only good-enough values (notion of bounded rationality [START_REF] Simon | A behavioral model of rational choice[END_REF]). Because good-enough values cannot be precisely defined, this principle should be complemented with qualitative principles that reflect the agent's decisional mechanisms more precisely. In particular, the agent should not simply react towards the highest immediate value. This can be expressed in the form of the corollary principle: a.2 Principle of hedonistic temperance.

The agent should learn to enact negative interactions when such interactions can lead to even more positive interactions. Conversely, the agent should learn to refrain from enacting positive interactions when such interactions would lead to more negative interactions.

Behavioral criteria

Category (b), behavioral criteria, has the advantage of supporting comparisons across systems, because these criteria are based on the external observation of the system's behavior. The expected behavior can, however, vary across studies, raising the need for defining general principles. Assessing the agent's development with regard to general principles is the point of the third category listed by Oudeyer and coauthors (c). Yet, principles proposed by theories in psychology and ethology appear too vague, and out of reach for current artificial systems [START_REF] Guerin | A Piagetian model of early sensorimotor development[END_REF]. In the current state of the art, we need precise behavioral principles that account for the very beginning of the developmental process.

Surveys in developmental robotics (e.g. [START_REF] Blank | Bringing up robot: Fundamental mechanisms for creating a self-motivated, self-organizing architecture[END_REF][START_REF] Lungarella | Developmental robotics: a survey[END_REF][START_REF] Weng | Artificial intelligence -Autonomous mental development by robots and animals[END_REF]) suggest three widely acknowledged principles:

b.1 Principle of situational categorization.

The agent should exhibit the capacity to categorize aspects of its situation and to adjust its behavior according to such categories.

b.2 Principle of situational disambiguation.

The agent should distinguish between different situations that generate the same sensory stimuli (perceptual aliasing, [START_REF] Crook | Learning in a state of confusion: Perceptual aliasing in grid world navigation[END_REF]).

b.3 Principle of graceful readaptation.

The agent should readapt gracefully to novel situations rather than experiencing catastrophic forgetting [START_REF] French | Catastrophic forgetting in connectionist networks: Causes, consequences and solutions[END_REF].

Moreover, the interaction-centered approach inspires two additional principles:

b.4 Principle of active perception.

The agent should learn to enact interactions not only because of their direct proclivity but also to update its representation of the current situation so as to take better decisions.

b.5 Principle of individuation.

Different executions of the same system should possibly lead to individualized instances exhibiting different habits. This principle accounts for the intuitive distinction between drives and goals according to which drives should leave room to individual choices. Individuation can occur through an "en habitus deposition" (De Loor [START_REF] De Loor | Enactionbased artificial intelligence: Toward co-evolution with humans in the loop[END_REF] citing Husserl).

In summary, the criteria to assess environment-agnostic agents generally involve temporal analysis-either quantitative (category a) or qualitative (category b). This indicates a need for implementation platforms that generate activity traces and support activity trace analysis. The next section illustrates these principles by presenting example experiments.

EXAMPLE IMPLEMENTATIONS

When implementing an environment-agnostic agent in a specific environment, the designer chooses the meaning that he or she assigns to the primitive interactions. For example, he or she may design a two-dimensional grid where interactions may consist of moving, turning, bumping into obstacles, touching objects, etc. An agnostic agent, however, may even ignore that its world has two dimensions. The designer implements the execution of the interactions in an interface layer, as depicted in Figure 2.

The designer also chooses the primitive interactions' proclivity values to generate interesting behaviors. For instance, if a negative value is associated with the interaction turn, and a positive value with the interaction move forward, then the agent tends to move forward, and turns only to ensure even more moving forward or to avoid subsequent even more negative interactions (such as bumping a wall). Conversely, with positive values associated with turning, the agent would learn to spin in place.

Interesting behaviors come from that an agent's possibilities of interaction and their proclivity values are adapted to a specific environment. In the case of natural organism, we assume that such adaptation results from phylogenetic evolution. Notably, this motivational mechanism is neither purely extrinsic (as would be a reward associated with an object in the environment), neither purely intrinsic (as curiosity in [START_REF] Blank | Bringing up robot: Fundamental mechanisms for creating a self-motivated, self-organizing architecture[END_REF][START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]). Instead, it is interactioncentered and results from the pairing of an agent with an environment.

Simple sequential environment

We first demonstrated the intrinsically motivated schema mechanism in the experiment shown in Figure 3 [START_REF] Georgeon | An Intrinsically-Motivated Schema Mechanism to Model and Simulate Emergent Cognition[END_REF]. In this case, The interface between the agent and the environment generates the actions in the environment (e.g., try to move) from the intended interactions (e.g., B: move), and either confirms the intended interaction (B) or generates a contradictory interaction (e.g., E: bump). The meaning of the interaction is not implemented in the agent's decisional algorithm.

The agent had 6 possible choices: try to move forward, turn left, turn right, touch in front, touch left, and touch right. The environment generated a single bit in return, which resulted in the 12 possible primitive interactions listed in Figure 3 (center). Again, the agent had no other way of apprehending its environment than interaction (no traditional "perceptual system"). This experiment can be seen online 2 .

An analysis of this experiment based on the criteria listed in Section 3 leads to the following findings. This agent met the principle of objective hedonism (a.1) by learning to enact interactions of higher value, in particular by learning to avoid bumping into walls. Yet, the agent demonstrated temperance (a.2) because it learned to enact turn and touch interactions to ensure safer moves. The agent also demonstrated its capacity to identify and discriminate between situations (b.1, b.2) by representing the situation in the form of sequences rather than with the current feedback received from the environment (a single bit at each single point in time). The agent showed active perception (b.4) by adopting the habit of touching ahead before moving forward, and not moving forward if it touched a wall. This result is original because nothing initially differentiated perceptive interactions from motion interaction from the agent's viewpoint, except their cost (value). In essence, the agent learned to use cheap interactions to gain a better representation of the situation to ensure safer high-value interactions, which grounded the meaning of the constructed perception in the agent's activity.

This agent, however, had difficulties when the environment was not shaped as a linear route but was rather an open space, because the agent's sequential mechanism had trouble capturing spatial regularities. To address this difficulty, we implemented the experiment reported next.

Simple open space environment

In the experiment reported in Figure 4, we implemented interactions that were sensitive to remote properties of the environment [START_REF] Georgeon | A Model and simulation of Early-Stage Vision as a Developmental Sensorimotor Process[END_REF]. This implementation was inspired by the visual system of an archaic arthropod, the limulus (horseshoe crab). In particular, two notable properties of the limulus were reproduced: (a) sensitivity to movement: the limulus's eye responds to To replicate these behaviors, we simply implemented an interface that sent dynamic visual features to the agent. In this case, the primitive interactions were generated from the agent's action in the environment (move and turn), associated with the dynamic features resulting from the changes in the agent's visual field (target appear, closer, reached, and disappear). The behavioral proclivity toward targets was implemented by giving positive values to interactions in which the target appeared or enlarged in the visual field, and negative values when the target disappeared from the visual field (Figure 4, center). Note that the agent had no way to accurately predict when the target would disappear from its visual field as it moved forward.

This experiment demonstrated hedonist temperance (a.2) and active vision (b.4). In some instances, the agent learned to move forward until the target disappeared from its visual field as the agent passed it (active vision, b.4), then made a U-turn (hedonist temperance, a.2) and turned to realign itself with the target (in bold in Figure 4, right). In other instances, the agent learned to move toward the target in stair steps until it aligned itself with the target (trail shown in Figure 5). Once learned, an agent instance kept the same strategy when new targets were introduced in the board. This demonstrates the principle of individuation (b.5).

Videos of these behaviors are available online 3 . These behaviors can also be reproduced in the simulation platform described in Section 5.

In this experiment, the agent exhibited adaptation to a spatial environment by simply capturing sequential regularities, but without capturing spatial regularities. For example, the agent did not learn the persistence of objects in space: it stopped pursuing a target in configurations where the target became hidden behind walls. Additionally, this agent was unable to adapt its behavior to different objects in the environment, for example, seeking food when hungry and water when thirsty. Such issues of spatial regularity learning and object persistence learning constitute a topic of research that we are currently exploring [START_REF] Georgeon | Early-Stage Vision of Composite Scenes for Spatial Learning and Navigation[END_REF]. Trace:

1! 2\ 3!! 4\ 5\ 6/! 7\

DESIGN/SIMULATION PLATFORM

Evaluating agnostic agents using the principles and criteria described in Sections 2 and 3 demands a flexible agent simulation platform to easily set up and run experiments. Such a platform should provide the means to parameterize experiments as well as offer mechanisms to detect patterns of agent behaviors in order to assess the agent's performance. We decided to use NetLogo [START_REF] Wilensky | NetLogo. Center for Connected Learning and Computer-Based Modeling[END_REF] because we find it one of the simplest, most powerful, and most widely used agent simulation platforms available.

In a typical NetLogo simulation, the world consists of patches, i.e. components of a grid, and turtles that are agents that "live" and interact in that world. Modeling complex agents and environments is greatly facilitated by the fact that each entity participating in the simulation can carry its own state, stored in a set of system and/or user defined variables, allowing great flexibility in the simulations that can be implemented. Moreover, a simulation domain specific programming language permits implementation of any "sensors" or "actuators" that the experiment requires. Thus, a large range of worlds and a variety of agents can be simulated in order to evaluate design choices and test ideas in agent systems.

The intrinsically motivated schema mechanism [START_REF] Georgeon | An Intrinsically-Motivated Schema Mechanism to Model and Simulate Emergent Cognition[END_REF] was integrated in NetLogo through an appropriate Java module, that we called the IMOS extension (Interaction MOtivation System). Thus, the environment and the interface (as defined in Section 4 and illustrated in Figure 2) are implemented in standard NetLogo, whereas the agents' decision/learning mechanism is handled by the IMOS extension. This architecture facilitates implementation of an unlimited number of different experimental settings by simply encoding the agent/environment related models, as well as effortlessly changing the agent parameters (e.g. proclivity values) in order to see how these parameters affect the agent's behavior. Finally, the NetLogo primitives offer an easy visualization of the agent's motion in the world (trails) that leads to clear detection of patterns of interaction in the agent's behavior (Figure 5). Although, all the above could be implemented in any programming language, the simulation Domain Specific Language (DSL) that NetLogo offers makes the task of encoding simulation environments including controls, monitors etc., simple.

The implementation of the open space environment described in Section 4.2 is depicted in figure 5. Through the interface, the user can change the full set of experimental parameters in order to investigate their impact on the learned behavior. Such experimentation can lead to rather interesting observations. For example, using the platform, we were able to investigate how proclivity values affect the agent's interaction with the environment and also to investigate the impact of initial conditions, i.e. the initial locations of the agent and the target, on the different "strategies" learned by the agent in order to approach the target.

In the experiment in Section 4.2, the agent had a rather limited way of navigating through the environment, simply by moving between grid positions and turning by 90 degrees each time. In order to assess the algorithm's robustness and performance in a higher resolution environment, we simply changed some of the sensors and effectors (turning angle, cone of vision and movement capabilities) of the agent in the NetLogo part of the simulation, with proclivity values and the learning algorithm remaining the same as in the original experiment. As seen in figure 6, the agent managed to learn a similar strategy to approach the target, supporting further the argument regarding the robustness of the learning algorithm and the approach. Thus, in more than one respect, NetLogo proved to be an excellent platform for evaluation and testing of the agnostic agent approach presented in this paper. The whole experimental platform, together with the Java library module, is available online4 .

CONCLUSION

We propose an approach to designing autonomous agents with minimal preconception of their environment. We characterize such agents as environmentally agnostic. This approach focuses on the notion of sensorimotor patterns of interaction between the agent and the environment rather than on the usual notions of perception and action.

The sensorimotor approach allows the implementation of an interaction-centered motivational mechanism. With this mechanism, the agent has inborn proclivities to enact sensorimotor patterns with high values and to avoid sensorimotor patterns with negative values. To do so, the agent needs to discover, learn, and exploit regularities of interaction afforded by the environment, which results in the autonomous construction of operative knowledge.

The interaction-centered motivational mechanism contrasts with traditional problem-solving or reinforcement-learning approaches in that it implements drives rather than goals. We argue that the fulfillment of drives should be assessed through activity analysis rather than performance measurement. We propose a list of developmental principles that should be observed in the agent's activity, in particular: objective hedonism, hedonistic temperance, active perception, and individuation.

Experiments show agents that meet the developmental principles in rudimentary settings. We provide the algorithm as a NetLogo extension to demonstrate that the agent's decisional process is independent from the environment that the designer chooses to implement. These experiments constitute an initial investigation of the principles of environment-agnosticism but the resulting behaviors are still rudimentary and many issues remain. In particular, we are now studying how environment-agnostic agents can learn spatial regularities and knowledge of persistent objects in the environment.

Figure 1 .

 1 Figure 1. Learning and decisional mechanism of an environment-agnostic agent.

Figure 2 .

 2 Figure 2. The interface agnostic agent/environment.

2 http:

 2 //e-ernest.blogspot.com/2010/12/java-ernest-72-invacuum.html movement, and the limulus has to move to "see" immobile things; (b) behavioral proclivity toward targets: male limulus move toward females, based on vision.

  3 http://e-ernest.blogspot.com/2011/01/tengential-strategy.html

Figure 3 .

 3 Figure 3. Example environment-agnostic agent in a sequential environment (adapted from [10]). Left: The agent (triangle) in the environment. Filled cells are walls which the agent can bump into. Center: List of the 12 primitive interactions with their proclivity values. Right: Activity trace of an example run. Steps 116 through 120: the agent has learned how to recognize and deal with a corner: touch the wall on the left -touch the wall in front -touch the empty square on the right -turn right -move forward.

5

 5 to empty square (v) 0 Turn 90° right to adjacent wall [v] -5 Turn 90° left to empty square (^) 0 Turn 90° left to adjacent wall [^] -Additional > |+) 3(> |+) 4(> |+) 5(> |+) 6(> |+) 7(> |o) 8(v |*) 9(v*|o) 10(>+| ) 11(^ |*) 12(^o| ) 13(^ |o) 14(^*| ) 15(>o| ) 16(>) 17(>) 18(^*| ) 19(vo| ) 20(v) 21(^) 22(>) 23(^*| ) 24(^o|*) 25(^ |o) 26(^) 27(v) 28(v |*) 29(^ |o) 30(^) 31(>) 32(^*| ) 33(^o|*) 34(v*|o) 35(>+| ) 36(^o|*) 37(v*|o) 38(>+| ) 39(^ |*) 40(v |o) 41(>o| ) 42(>) 43(^*| ) 44(^o|*) 45(> |+) 46(> |+) 47(> |o) 48(v |*) 49(v*|o) 50(>+| ) 51(^ |*) 52(>+|+) 53(v |o) 54(vo| ) 55(v |*) 56(v*| ) 57(v |o) 58(^ |*) 59(>+|+) 60(>+|+) 61(>+|+) 62(>x|x) 63(>o|o) 64(v) 65(v) 66(v) 67[v] 68(^) 69[v] 70(v) 71(v) 72(v) 73[v] 74[>] 75(^*| ) 76(^o|*) 77(> |+) 78(> |+) 79(> |+) 80(> |+) 81(> |+) 82(> |o) 83(v |*) 84(v*|o) 85(>+| ) 86(^ |*) 87(>+|+) 88(>x|x) 89(^o|o) ...

Figure 4 .Figure 5 :

 45 Figure 4. Example environment-agnostic agent in an open space environment (adapted from [8]). Left: The agent in the environment. The agent's visual system is made of two pixels that can only detect blue cells (targets). Each pixel covers a 90° span. Center: primitive interactions: move, or turn 90°, plus dynamic features possibly returned by each eye: appear, closer, reached, disappear. Right: Activity trace. An interaction consists of associating the agent's action with the signal sent by the eyes, separated by the symbol "|". Step 62: the agent finds and "eats" the first target. Step 75: a second target is inserted in the environment. Steps 77 to 88 (bold) demonstrate the learned behavior: Steps 77-81: the agent goes on a straight line with the target enlarging in its right eye's field. Step 82: the target disappears from the agent's right eye's field as the agent passes it. Steps 83-86: the agent makes a U-turn, returns back one step, and turns left towards the target. Step 87: the agent is aligned with the target and moves forward. Step 88: the agent reaches the target. Once learned, this "strategy" is repeated to reach other targets that the experimenter randomly introduces in the environment.

The Soar architecture offers an emblematic illustration of these two types of examples as a rule-based system extended with reinforcement learning (Soar-RL). We credit the Soar team for acknowledging this knowledge-oriented bias in both cases.

http://users.uom.gr/~iliass/projects/NetLogo/Ernest/index.html Figure 6: Learning to reach the target in a continuous environment

ACKNOWLEDGMENTS

This work was supported by the Agence Nationale de la Recherche (ANR) contract ANR-10-PDOC-007-01. We gratefully thank Jonathan Morgan and James Marshall for their review of this article.