Bhargav Teja Nallapu
email: bhargav.teja-nallapu@inria.fr

Frédéric Alexandre
email: frederic.alexandre@inria.fr

Cognitive Architecture and Software Environment for the Design and Experimentation of Survival Behaviors in Artificial Agents

Keywords: Cognitive Architecture, Cerebral Systems, Survival

We discuss here the characteristics of a software environment appropriate for the development of a bio-inspired cognitive architecture, which can emulate the behavior of autonomous intelligent agents. First, it is reminded that, while the focus is often set on the more abstract aspects of cognitive abilities, studying the fundamental bases of intelligence that allow for autonomy is a prerequisite for well defined intelligent systems. Secondly, we highlight functional loops associating cerebral structures including the basal ganglia in the brain of most species along the evolution. They are dedicated to the organization of behavior under the constraint of reinforcement, corresponding in their simplest expression to the selection of action for survival. Lastly, concerning the simulation of such models, we describe a software environment to study such relations in a more controlled way than hardware implementations, by adapting a platform built on the top of a video game for the development of classical artificial intelligence models. We explain here how our neuronal model exhibits bodily and internal characteristics necessary for survival tasks and how these characteristics are plugged in the simulation platform. Some scenarios of survival are reported as an illustration of this environment.

INTRODUCTION

Autonomous behavior and ability to survive forms a key prerequisite for any cognitive agent. High level cognitive agents, on the other hand, are often characterized to solve specific complex problems while their capacities to survive and evolve autonomously are often disregarded. We describe a bio-inspired approach that (i) forms the basis of understanding these capacities and (ii) defines the characteristics of a software environment in which the problem of survival is demonstrated using a virtual agent.

From a simple organism such as C-elegans with 302 neurons to a human brain with billions of neurons, the changes that led to more complex behavior are quite challenging to understand. So is extracting the invariants that underlie the survival capacities of these species. Within this scope, at all stages of phylogeny, two processes can be mentioned: signaling and regulation. The first indicates that, by various means, the organism is informed about the state of the environment and its own (bodily and mental) state. The second emphasizes that the organism has different ways (motor, chemical, decisional) of responding to adapt (its body and mental analysis) to the situation. These processes are central in the concept of autopoiesis introduced by [START_REF] Maturana | Autopoiesis and Cognition: The Realization of the Living[END_REF], as the property of a system to produce itself, permanently and in interaction with its environment, and thus to maintain its organization despite the changes. First introduced to characterize living cells, it was then extended to organisms and became equally important in cognitive science [START_REF] Varela | The Embodied Mind: Cognitive Science and Human Experience[END_REF].

In vertebrates, the existence of central and peripheral nervous systems makes it possible to better emphasize this distinction between the environment, body and brain and to introduce two types of perception, exteroception for sensations coming from the environment (e.g., visual or auditory) and interoception for those coming from the body (including pleasure, pain and needs), as well as different types of responses aimed at controlling these two classes of signals [START_REF] Craig | Interoception: the sense of the physiological condition of the body[END_REF].

The basal ganglia (BG) are brain structures that play a central role in the selection of responses adapted to internal and external states [START_REF] Redgrave | The basal ganglia: A vertebrate solution to the selection problem?[END_REF]. These structures are present in all vertebrates and an homologous structure has even been found in arthropods [START_REF] Strausfeld | Deep Homology of Arthropod Central Complex and Vertebrate Basal Ganglia[END_REF], with similar neuronal activities and connectivity allowing behavioral regulation. This structural consistency across species makes them core structures within survival neural loops and we will therefore present these structures in more detail in the next section.

Developing models of bio-inspired neuronal architectures for survival functions is challenging at multiple levels because the focus is usually set in neuronal models on local perceptual analyses or cognitive functions associated with direct performance measures. It is sometimes difficult to distinguish the general principles that aim at survival of autonomous systems. The same is true for the software and hardware systems used to implement and evaluate these architectures. They are generally oriented towards solving specific complex and sometimes abstract problems, particularly in the context of Artificial Intelligence [START_REF] Stoeter | Autonomous stair-climbing with miniature jumping robots[END_REF].

In the rest of this paper, we present a bio-inspired neuronal model of loops involving the BG and its implementation and experimentation using a software platform dedicated to video games. The key point, here, is to demonstrate to which extent this arrangement is relevant to account for some fundamental mechanisms of agent behavior related to survival and to observe and manipulate them experimentally. The joint use of these systems showcases several survival scenarios and provides a road-map for future works.

A FUNCTIONAL DESCRIPTION OF THE BASAL GANGLIA LOOPS

The basal ganglia (BG) are a set of interconnected sub-cortical nuclei, organized in loops with many other brain structures [START_REF] Parent | Functional anatomy of the basal ganglia. I. The cortico-basal gangliathalamo-cortical loop[END_REF], as elaborated here. These loops are described as parallel and segregated [START_REF] Alexander | Parallel organization of functionally segregated circuits linking basal ganglia and cortex[END_REF] because they correspond to distinct and related territories of the structures involved, and because they are structurally similar (in terms of involved neural populations and connectivity), suggesting that the same kind of processing is applied generically to different information.

The association of (interoceptive or exteroceptive) sensations with (internal or external) responses is sometimes straightforward and a simple sensorimotor structure is enough to trigger the response. But the involvement of BG is essential when the selection of the response (e.g., goal or action) is based on ambivalent or uncertain criteria [START_REF] Floresco | The nucleus accumbens: an interface between cognition, emotion, and action[END_REF].

In the experiments reported in this paper, we consider four loops as identified in primates in [START_REF] Alexander | Parallel organization of functionally segregated circuits linking basal ganglia and cortex[END_REF], involving different regions of the Striatum, the largest nucleus of the BG. The first two loops are called limbic and are based on interoceptive information. They are organized around the selection of the goal of the behavior, according to its motivational value, in response to perceived needs or according to its hedonic value. The other two are called sensori-motor and they process exteroceptive information. They are organized around the motor behavior allowing to reach the goal, according to its spatial position (orientation) or according to the physical characteristics involved (handling). We refer to these four loops by the question each loop attempts to answer, detailed as follows.

1. The Why loop selects the current motivation (satisfying hunger or thirst in our task) from the interoception of needs and possibly the costs of actions. The motivation is expressed in the anterior cingulate cortex (ACC) and the loop also associates the ventral striatum (the core of the nucleus accumbens), lateral hypothalamus and insula for interoception.

2. The What loop selects the goal according to the preferences (e.g., gustative preferences, quantity), innate or acquired and represented in the amygdala. Preferences are expressed in the orbitofrontal cortex (OFC) and the loop also combines the ventral striatum (the shell of the nucleus accumbens), amygdala and insula for gustative interoception. The goal object can be consumed if it is directly available, otherwise it will become the goal for the spatial and temporal organization of the behavior.

3. The Where loop considers the spatial location of the goal and selects the orientation behavior relevant to face it, which can concern eye movement as well as body orientation, as also observed in the superior colliculus. The orientation strategy is expressed in the Frontal Eye Field (FEF) in the frontal cortex and the loop also combines the dorsolateral striatum, the parietal cortex and the superior colliculus.

4. The How loop supports the latest postural adjustments when the goal is attainable, by simply reducing the distance or possibly manipulating the object before consuming it. This concerns the motor areas, the parietal cortex and the dorsolateral striatum.

This functional description highlights that the generic processing of response selection by the BG is ascribed in a generic loop, also associating the frontal cortex, sub-cortical and cortical sensory structures as shown in figure 1.

We now describe the implementation of the loops in a bio-inspired neuronal model and use them to emulate a survival behavior in a simulation platform. From a technical point of view, the description of the tasks accomplished by each loop reveals certain characteristics that are not conventionally considered in neural models (e.g. notions of motivation or goal). Furthermore, implementation in a video game simulator requires dedicated specifications (e.g. bodily characteristics, bodily needs and biological constraints). We address these requirements in the design and implementation of the system.

In addition, we have so far discussed each behavioral loop individually, whereas describing the possible association and interactions of these loops is a major topic in neuroscience [START_REF] Haber | Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum[END_REF]. In the case of survival tasks like the one we demonstrate, we can particularly wonder how to model the functional interaction between these loops and if different forms of survival strategies (e. g. goal-driven or stimulusdriven) can be performed on this basis. The latter question forms one of the open problems in computational neuroscience [START_REF] Daw | Uncertaintybased competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF], thus motivating our digital experiments.

IMPLEMENTING A MODEL OF BG LOOPS

As highlighted in the previous section, although each loop addresses a different issue, the principle behind the operations of these loops appears quite generic in terms of their physical connectivity and computational dynamics. In this perspective, several generic computational neuronal models of BG loops have been proposed [START_REF] Gurney | A computational model of action selection in the basal ganglia. i. a new functional anatomy[END_REF][START_REF] Guthrie | Interaction between cognitive and motor cortico-basal ganglia loops during decision making: A computational study[END_REF][START_REF] Hazy | Banishing the homunculus: making working memory work[END_REF]. These neuronal models exploit several pathways observed between the nuclei of the BG to implement the decisional process, with a globally excitatory (direct) pathway for selecting the best response and other inhibitory pathways (hyperdirect or indirect) that will penalize inappropriate responses. The BG implementation in our model description is directly inspired from the 'Go-No Go' process implemented in [START_REF] Hazy | Banishing the homunculus: making working memory work[END_REF] for the decision process. Above mentioned models also agree on the critical role of neuromodulators, in particular dopamine, in updating contextual associations according to the prediction errors. We do not consider this aspect in the current implementation and will address it in further work.

Based on this computational formalism, we im-plemented the four loops discussed in section above, each in the form of a generic loop. A loop is formed between 3 components (see Figure 1), with the input information coming to the sensory module of the loop (blue component). This sensory information will elicit possible actions in the frontal module (green component). These actions will compete until one is selected and triggered. Basically, this selection is made by BG (red component), also informed by the sensory context. We will see below that the selection process might also involve the influence of other loops. When an action is triggered, it remains active with a sustained activity until some sensory information is received, informing about the end of the action.

From a more practical point of view, in each loop, the processing happens in the following stages, in a given small time intervalinformation acquisition, action evaluation and selection and sustained activation by feedback control. For each loop (i) acquire sensory information through exteroception and interoception, (ii) evaluate alternative responses, select the most appropriate one and set the corresponding goal, and finally (iii) sustain the activation of the response by a constant feedback until the goal has been achieved.

This generic mechanism of response maintenance and goal monitoring is an important aspect of our computational model, implemented in each loop. Selecting a response to be executed means defining a sensory state that must be achieved (the goal). As a consequence, the rule of response execution is implemented as a sustained activation of the response which terminates, thanks to a feedback mechanism, when the goal is met. As it is elaborated in the section Scenarios, the goal is not always reached simply by activating the response, but sometimes requires other responses and secondary goals to be defined, still within the same generic mechanism. To implement this, we define a desired state of activation for goals that asks for additional responses until it becomes actual.

To give a more concrete understanding of the information processed in each loop, we stick to the very simple example that will be developed below, describing the connection of the model to an experimentation platform and the unfolding of survival scenarios. In this example, the agent is given two needs (representing thirst and hunger), each as a variable to maintain between bounds to survive. It can also detect and possibly reach objects (representing food and drinks).

In that context, the two limbic loops can be defined as follows. The Why loop is responsible for the selection of the need. It receives sensory information about the levels of need through interoception and about the kinds of objects perceived by exteroception. Responses it can trigger correspond to the decision to go for food or drink, until the need is satisfied (by consuming upon reaching). The What loop is responsible for the selection of an object. It receives sensory information about the levels of preference through interoception and about the identity of the objects perceived by exteroception. Responses it can trigger correspond to the decision to select one object until the object is reached.

Similarly, the two sensori-motor loops can be defined, still using the same framework, as follows. The Where loop is responsible for the orientation of the agent in space. It receives the azimuth of each object perceived by exteroception and when one is selected, triggers a movement of orientation which stops when the agent is facing the object. The How loop is responsible for the reaching of an object. It receives the distance to each object it is facing by exteroception and when one is selected, it moves forward until the object is reached.

Before describing how these loops can be associated in organized behavior, we present the Malmo platform used for the experimentations and describe the connection of the platform to the cerebral model of loops.

THE MALMO PLATFORM

Minecraft is a well-known video game, with a block based 3D world, allowing virtual exploration, resource gathering, and including survival task scenarios, this for a single or multiple players. It has been adapted for a systemic neuroscience simulation platform called 'Virtual Enaction'. Later, an experimentation platform called Malmo was built on the top of Minecraft [START_REF] Johnson | The malmo platform for artificial intelligence experimentation[END_REF], and is dedicated to support research in various AI related areas. Malmo allows to incorporate various models of reinforcement learning, planning and related problems into the Minecraft game environment, ranging from a basic Q-learning algorithms on a single agent to more collaborative and competitive strategies among multiple agents.

The advancements in the research on intelligent systems and their behavior requires to be able to test, study and visualize the models in a more elaborative setting, as opposed to the traditional symbolic representations and numerical experimentations. Given the complexity of the survival task that we target to demonstrate, it is considerably difficult to choose the right kinds and number of attributes to be encoded in the model. Malmo, exploiting the power of Minecraft environment, precisely provides such great convenience to study our model of generic loops. Here, since our goal is to explain the dynamics of emergence of the behavior using and organizing the loops defined in the model presented above, we use only a specific set of Malmo features that are adapted according to the task and the model. We have designed on the top of Malmo a minimal software layer to accommodate our adaptations concerning the interoceptive and exteroceptive attributes relevant to the survival task.

Particularly, Malmo provides a set of attributes representing the vital characteristics of the agent. We have built on them more precise variables relevant to the task together with their functional dynamics, as a part of the software layer. Similarly, the adaptations related to the agent's vision and the sensori-motor responses can be conveniently implemented using the related features of Malmo. These attributes in conjunction with our adaptations concisely explain the generic dynamics in the loops as a result of which several behaviors emerge. Furthermore, it is interesting to observe that Malmo invariantly supports demonstrating these different behaviors with no or minimal changes to itself but only from the changes in the state of the agent or that of the environment.

In the rest of this section, we explain the attributes of Malmo that we have used in the context of the survival task. In the subsequent section, we describe the additional software layer with the adaptations that also demonstrates the embodiment of the agent.

World. This is the environment in which an agent is free to move around and explore, besides other objects (items) present in it. It is a simplified environment of the Minecraft world, designed to have a complete control on the external objects, and simple enough to understand the causal relationships with respect to the simulated agent behavior. It is 3 dimensional, allowing the items to be at a height above the ground and allowing the agent(s) to jump if necessary. The ground (floor) is defined in terms of blocks which have properties like texture, type and color. Such block properties like the color play the role of the environment context. In behavioral scenarios like fear learning or fear extinction, the context is a useful attribute because it adds an extra dimension to processing the stimulus information and attributes a preferential relevance to it (either from previous learning or from memory).

Agent. Malmo allows multiple agents to interact simultaneously in a given environment. An agent, at any given point of time, has access to its vital variables like life, its current position and its current orientation with respect to the World. Like in the case of an animal, the variables are affected by the external world -for e.g, components like fire or an attack could reduce life. In the context of our task, we consider only a single agent.

Items Malmo provides a list of items that can be procedurally placed in the environment. When the items are in the configured vicinity of the agent, the positions and the orientations of the items are available for the agent. Each item can be configured with a certain reward value at the beginning of the task. As a part of a task, the reward can be awarded to the agent, either for collecting the item or discarding it. There are several such items, from which we use apple, cake, water bucket and stew. The distance within which the agent can collect the item can also be configured.

Actions. Suitable to the 3D world, the agent is capable of doing actions like moving, turning and jumping. In order for the agent to reach an object or a position, it uses, from the in-built set of actions, predominantly the turn action (to orient towards a stimulus) and move (to approach a stimulus or keep exploring). Any of the actions can be stopped when required.

State. The state of the world, at any given instant, is constituted by the attributes of both the agent and the objects present in the vicinity of the agent. At any instant, the agent has information about its current levels of vital variables and how far they are from critical or fatal limits. It also has information about its own position and orientation with respect to the environment. Information about the item like its name, position and the reward it carries is also accessible the agent. As explained earlier, context also is a part of the state, describing the type of the floor for a requested subset of blocks.

EMBODIMENT OF THE AGENT

We describe here, the adaptations that we made to Malmo, in order to connect our model of cerebral loops to the world simulated in Malmo, including the characteristics of the environment and the agent. In addition to the technical considerations, these adaptations allow us to distinguish several actors in our tasks, namely the brain, the body and the environment, which have been often reported to constitute embodied cognition [START_REF] Varela | The Embodied Mind: Cognitive Science and Human Experience[END_REF]. Hence, attributing bodily features to the agent and associating them to its motivational and emotional characteristics form a key aspect of our model. These characteristics have been respectively implemented in terms of needs and preferences. Also, from a functional point of view, we adapted few aspects like visibility of the agent and the information about the positions (of items as well as the agent itself). These adaptations were important to add certain biologically plausible restrictions to the task.

Needs and Preferences. The agent has two vital variableshunger and thirst -which increase with time as well as with its efforts (meaning a move or turn action). Instead of a one dimensional reward, each item carries a value that is relevant to the hunger or the thirst level it would satisfy, and a value indicating the level of preference of the agent for this item.

Visibility. Malmo provides information about the items all around the agent's vicinity of chosen range. However, we restrict its 'Field Of Vision' to a biologically plausible value (in this case, 120 • ), which is further divided into 3 different zones viz., Appear, See and Reach, depending on the distance from the agent. When the agent is moving and some items are present in the Appear zone, the agent has no precise information about the stimuli (the items that are perceived by the agent) such as the precise location of each, or their preference appetitive values. Rather, the agent has minimal information about the presence or absence of some items in some direction. When the stimuli are within the See zone, all the information about the stimuli is provided as inputs to the model. In the Reach zone, an additional information is provided, that the stimuli are accessible for the agent to consume.

Positions In regard to the positions of the agent and the items in the environment, Malmo provides their exact coordinates, the absolute yaw details with respect to the environment. But to demonstrate a very important feature within each loop of the model, we avoid using these exact position details. Instead, we take the agent as the origin, convert the relative distance and orientations of the items into signals that regulate the activity within the loops of the model. It is usually these feedback signals relative to the desired state and the current state of the agent that sustain the execution of a selected goal.

SCENARIOS

We intentionally define a world with simple and few characteristics, in order to study precisely, not only the functioning of each loop, but also the way they interact with one another to emulate specific kinds of behavior. We present here for illustration, some behaviors that could be demonstrated with the model to emulate and others that are part of our ongoing work.

Exploration Behavior

Actions for spatial exploration can be triggered for several reasons. They can be triggered if, as in goaldirected behaviors described below, the agent must explore the environment to find desired stimuli. They can be also triggered, as in the stimulus-driven behavior described below, if the agent has no current need, to give it the opportunity to discover new options. For any kind of behavior, the agent can also apply an exploration/exploitation strategy [START_REF] Humphries | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia[END_REF] and at any moment interrupt the current behavior to explore. The exploration behavior is also particularly important at the beginning of the task, when the agent rotates until it can perceives some stimuli. If nothing is perceived, the agent selects a random direction, moves by a random distance and rotates again. When some stimuli are perceived, if they are in the Appear zone, the agent moves in that direction until they are in the See zone and can be discriminated. Then depending on the current behavior, several actions can be triggered as described below.

This basic behavior also forms an interesting basis to learn or update the contingencies in the environment or between characteristics of the environment and those of the agent. Particularly, in the limbic loops, this can contribute to set the values of the preferences and help connect some items to the needs they can satisfy. In the sensori-motor loops, this can help calibrate the movements of the agent and learn the consequences (in terms of modification in the perception) of their activation. However, since the work we report here concerns the dynamics of the loops, we provide the agent with this initial learning of fundamental contingencies as a pre-existing set of values, thereby enabling the agent to exploit them in its behavior.

Goal Directed Behavior (GD)

The system has been initially designed for a simple survival task, corresponding to activate the loops in a hierarchical way. First the Why limbic loop monitors the levels of the needs and when one of them passes above a critical threshold, satisfying this need becomes the primary goal of the agent. In the What limbic loop, the objects associated to the satisfaction of this need, are set as potential secondary goals and are activated as desired. If none of them is presently perceived, an exploration behavior is triggered in the Where sensori-motor loop, making the agent rotate until it perceives some stimuli. If they are too far (in the Appear zone), the agent approaches for them to be in the See zone and gathers their characteristics including the preferences. If several stimuli are perceived, the one with the highest preference is selected as the secondary goal of the behavior. The agent approaches the stimulus until it reaches and consumes it, thus satisfying the goals of the behavior. An implementation of this behavior is described in the section Illustrations.

Stimulus Driven Behavior (SD)

Without a specific goal or motivation, the agent can wander in the environment and discover by chance one or several items. In this case, the What limbic loop (estimating agent's preference from external information) is triggered and can select the most preferred item. Although the Why limbic loop (defin-ing the levels of need) has not triggered the decision making process beforehand, it can be activated by this preference and, depending on the corresponding level of need, it can decide to activate the sensori-motor loops to execute an action in order to reach and consume the selected item.

Opportunistic Behavior

As a part of our ongoing work, we would like to be able to interrupt a behavior in an opportunistic way. This is specifically the case when the agent is engaged in a goal-driven behavior and suddenly perceives a stimulus corresponding to the non-selected need but with a strong preference. In this case, it is conceivable to reason that, in some condition, it is preferable to choose a stimulus with a strong impact on a minor need as compared to another stimulus with a minor impact on the current need. This could be particularly the case if the stimulus with the strong preference is rare or if stimuli detected to satisfy the major need have very low levels of preference (or both). Better understanding the mutual influences between the limbic loops [START_REF] Haber | Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum[END_REF] is one of the major challenges in our ongoing work.

ILLUSTRATIONS

Figure 3 shows a sequence of snapshots from a goaldirected behavior, as implemented in Malmo. In this basic scenario, the agent has already selected its most urgent need, (hunger in figure 3(b) inset). The satisfaction of this need now becomes a desired state as a primary goal, which remains active in the Why loop until the need is satisfied. From previous experience to address the current need, the Why loop triggers the desired state of the stimuli known to satisfy the need, in the What loop. Figure 3(a) illustrates this exploration behavior, where the agent starts to move with a desired activation for some items (apple and cake in this case). If, as it is the case here, perceived stimuli are too far (in the Appear zone), the agent will have to move until they are in the See zone and can be discriminated.. When the detected stimuli are in the See zone, the simplest of the cases is when only one of the items is desired and can be directly selected. However, when encountered with multiple desired items, the agent has to decide the suitable choice among the items, suitable meaning the one with the highest preference (as illustrated in figure 3(b)).

Once the decision has been made, as a secondary goal, the selected stimulus becomes desired in the What loop and remains active until it is reached. The execution of the behavior involves two steps. (i) To evoke the necessary sequence of actions to reach the goal and (ii) to sustain the selected goal until the agent actually reaches the selected stimulus. In the case considered here, once an item is chosen, the other secondary goals are to orient towards the selected item and the reach it. The agent starts turning towards the chosen item. Here turning doesn't stop by using the target yaw provided by Malmo. Instead, we derive a feedback signal to the Where loop to sustain the act of turning until the agent is oriented towards the item, as illustrated in figure 3(c)&(d).

Then, the agent can move towards the target to reach it. With exactly the same mechanisms as described above, but here applied in the How loop, the goal of reaching is maintained until the item is reached. And once reached, this goal is considered achieved. In our current implementation, the internal action of consumption is automatically triggered when an item is reached. In this case, the goal in the What loop (sustained from the initial selection of the goal) is considered achieved. This consumption will also modify the level of need and similarly, the goal in the Why loop is also considered satisfied. This terminates the behavior as this was the primary goal of the scenario illustrated. The work that we have presented here can be considered under three points of view.

Firstly, this work is original by its deep anchoring in biological inspiration. The behavior of our autonomous agent is elaborated thanks to a model built on four loops described as playing an important role in the brain of most animals [START_REF] Alexander | Parallel organization of functionally segregated circuits linking basal ganglia and cortex[END_REF]. This inspiration is anatomical, considering the nature of information flows brought by several sensory and motor regions. This inspiration is also functional, particularly considering mechanisms to select actions and to sustain goals until they are achieved. A major characteristic of this work is to consider similar architectural and functional properties to build four loops and to build all the considered behaviors only by emergence, on the basis of the loops and their interactions. This biological inspiration is also very precious because most of the questions and orientations for future works we have evoked in the paper will be addressed by going deeper into biological details.

Secondly, we have argued that, even if these loops are mostly studied for the understanding of higher cognitive functions like reward-based decision making, considering them to implement survival scenarios is very important to design autonomous systems. Particularly, considering such basic scenarios is very convenient to study all the loops together, which is hardly addressed in the modeling literature. It is also interesting to understand how the two basic processes of signaling and regulation have evolved to allow for more abstract behaviors, which can still be described as interactions between limbic and sensorimotor loops, originally built for survival.

Thirdly, another major innovation of this paper is to propose that Malmo, a platform originally designed for experimentation in Artificial Intelligence, is a very powerful tool to build basic autonomous systems performing survival tasks. Not only it offers many interesting characteristics for the simulation and the visualization of a survival task, but it also eases the design of the most critical part, corresponding to the interface between the computational parts of the model and the internal and bodily aspects of the agent. In addition, this platform has another usefulness. To address the critical questions we have evoked in this work, we are currently seeking insights from biology, to improve and augment our model. Some of these questions are clearly unanswered by the current state of the art and must be investigated jointly with neuroscientists. In this perspective, Malmo offers a striking advantage to describe our model and its behavior to them, who are more accustomed to biological observations than algorithms and equations.

  sensori-motor Figure 1: Implementation of two classes of generic loops in the model. In both the classes, a blue component represents a region from Sensory Cortex, green represents a region from Frontal cortex and the red BG represents the corresponding sub-cortical BG nuclei involved in the loop. A black component, Information (through Interoception or Exteroception) feeds the blue component of the sensory cortex which propagates it to both the Frontal Cortical regions and the BG. (a) generic limbic loop -based on which the What? and Why? loops are implemented. (b) generic sensori-motor loop -based on which, the Where? and How? loops are implemented.

Figure 2 :

 2 Figure 2: Zones of visibility in the field of agent's vision. Zone marked 'R' is Reach, 'S' is See and 'A' is Appear.

Figure 3 :

 3 Figure 3: Snapshots at different stages in the task. The figure shows several steps involved in a goal-directed behavior of the agent. (a) exploration until the agent finds some stimuli in the Appear zone. (b) decision among the stimuli in the See zone, corresponding to the current need (inset:hunger). (c) orienting towards the selected stimulus until it is in the line of sight. (d) ready to approach towards the oriented stimulus.


A Palacios-Laloy

F Nguyen

F Mallet

P Bertet

D Vion

D Esteve

Tunable resonators for quantum circuits

Keywords: 74.78.-w, 84.40.Dc, 85.25.Am, 85.25.Dq

We have designed, fabricated and measured high-Q λ/2 coplanar waveguide microwave resonators whose resonance frequency is made tunable with magnetic field by inserting a DC-SQUID array (including 1 or 7 SQUIDs) inside. Their tunability range is 30% of the zero field frequency. Their quality factor reaches up to 3×10 4 . We present a model based on thermal fluctuations that accounts for the dependance of the quality factor with magnetic field.

Introduction

On-chip high quality factor superconducting resonators have been extensively studied in the past years due to their potential interest for ultra-high sensitivity multi-pixel detection of radiation in the X-ray, optical and infrared domains [1,2]. They consist of a stripline waveguide of well-defined length, coupled to measuring lines through input and output capacitors. The TEM modes they sustain have quality factors defined by the coupling capacitors and reaching in the best cases the 10 6 range [2].

It has also been demonstrated recently [3] that superconducting resonators provide very interesting tools for superconducting quantum bit circuits [4]. Indeed, a resonator can be used to measure the quantum state of a qubit [3,5,6,7]. Moreover, another resonator may serve as a quantum bus and mediate a coherent interaction between qubits to which it is coupled. The use of resonators might thus lead to a scalable quantum computer architecture [5]. The coupling of two qubits mediated by a coplanar waveguide (CPW) resonator has already been demonstrated [8,9]. In experiment [9], each qubit needs to be tuned in and out of resonance with the resonator for the coupling to be effective. Reference [10] proposed an alternative solution that consists in tuning the resonator in and out of resonance with each qubit. Here we report on the measurement of high quality factor resonators whose frequency can be tuned. Measurements similar to ours have been reported by other groups on lumped element [11] and distributed [12] resonators.

Tunable resonator with DC SQUID: model

Our tunable resonators consist of λ/2 coplanar waveguides with an array of N DC-SQUIDs in series inserted in the middle of the central strip (see Fig. 1a). Each DC SQUID is a superconducting loop with self-inductance L l intersected by two nominally identical Josephson junctions of critical current I c0 ; the loop is threaded by a magnetic flux Φ. The SQUID array behaves as a lumped non-linear inductance that depends on Φ, which allows to tune the resonance frequency.

A λ/2 CPW resonator without any SQUID consists of a transmission line of length l, capacitance and inductance per unit length C and L, and characteristic impedance Z 0 = L/C. We consider here only the first resonance mode that happens when Fig. 2 A DC SQUID with two junctions of critical current I c0 and loop inductance L l , biased by a magnetic flux Φ and by a current i, is equivalent to a lumped flux-dependent non-linear inductance L J (Φ, i) that can be decomposed in an inductance L J 0 (Φ) and a non-linear element SNL(Φ) in series. l = λ/2 at a frequency ωr = π/ √ LC, where L = Ll and C = Cl are the total inductance and capacitance of the resonator. The quality factor Q results from the coupling of the resonator to the R 0 = 50 Ω measurement lines through the input and output capacitors Cc leading to

Qc = π 4Z 0 R 0 C 2 c ω 2 r , (1)
from internal losses (Q int ), and from possible inhomogeneous broadening mechanisms (Q inh ). These combined mechanisms yield

Q -1 = Q -1 c + Q -1 int + Q -1 inh . (2)
As shown in Fig. 2, we model a SQUID as a non-linear inductance L J (Φ, i) that depends on Φ and on the current i passing through it, so that the voltage across the SQUID is

V = L J (Φ, i) di dt . (3)
All SQUID properties are periodic in Φ with a period Φ 0 = h/2e, the superconducting flux quantum. Introducing the reduced flux quantum ϕ 0 = Φ 0 /2π, the SQUID frustration f = πΦ/Φ 0 , the effective critical current Ic(Φ) = 2I c0 | cos f | of the SQUID at zero loop inductance, and the parameter β = L l I c0 /ϕ 0 , our calculation of L J (Φ, i) to first order in β and to second order in i/Ic(Φ)

yields for f ∈ ]-π/2, π/2[ L J (Φ, i) = L J0 (Φ) + A(Φ)i 2 , (4)
with

L J0 (Φ) = ϕ 0 Ic(Φ) 1 + β cos 2f 2 cos f , (5)
A(Φ) = ϕ 0 2I 3 c (Φ) . ( 6
)
Equation 4 shows that the SQUID can be modelled as the series combination of a lumped inductance L J0 (Φ) and of a non-linear device SNL(Φ) [13] (see Fig. 2).

In the linear regime i ≪ Ic(Φ) corresponding to low intra-cavity powers, one can neglect the non-linear term in Eq. 4. The N-SQUID array then simply behaves as a lumped inductance N L J0 (Φ). The device works in that case as a tunable harmonic oscillator. Introducing the ratio ε(Φ) = L J0 (Φ)/L between the total effective inductance of the SQUID and the resonator inductance, the frequency and quality factor are

ω 0 (Φ) = ωr 1 1 + N ε(Φ) , (7)
Q ext (Φ) = Qc [1 + 4N ε(Φ)] . (8)
At larger peak current in the resonator i Ic(Φ), the non-linear element SNL(Φ) has to be taken into account. The equation of motion of the oscillator acquires a cubic term, similar to that of a Duffing oscillator [START_REF] Landau | Mechanics[END_REF]. This leads to a small additional shift of the resonance frequency δω 0 (E) proportional to the total electromagnetic energy E stored in the resonator. Retaining first order terms in ε(Φ), we find

δω 0 (Φ, E) ω 0 (Φ) = -N 2ω 0 (Φ) πR 0 [1 + 2N ε(Φ)] 2 ϕ 0 8I 3 c (Φ) E. (9)
As shown by Eq. 7, a resonator including an array of N SQUIDs of critical current N I c0 has approximately the same resonant frequency and same tunability range as a resonator including one SQUID of critical current I c0 . However, an interesting advantage of using an array is to obtain a linear regime that extends to larger currents, allowing measurements at larger powers and therefore higher signal-to-noise ratios.

Sample fabrication

The design and fabrication of our resonators closely followed Ref. [START_REF] Frunzio | [END_REF]. The coupling capacitors were simulated using an electromagnetic solver. Test niobium resonators without any SQUIDs were first fabricated. They were patterned using optical lithography on a 200 nm thick niobium film sputtered on a high-resistivity (> 1000 Ω cm) oxidized 2-inch silicon wafer. The niobium was etched away using either dry or wet etching. Dry etching was done in a plasma of pure SF 6 at a pressure of 0.3 mbar and at a power such that the self-bias voltage was 30 V and the etching rate 1.3 nm/s. We observed that adding oxygen to the plasma gave consistently lower quality factors. Wet etching was done in a solution of HF, H 2 O, and FeCl 3 having an etching rate of approximately 1 nm/s at room-temperature. A typical resonator and its coupling capacitor are shown in panels b and c of Fig. 1. Its 3.2 cm length yields a resonance frequency around 1.8 GHz.

In addition to these test structures, some resonators had a gap in the middle (see Fig. 1d) used in a later step to fabricate a SQUID array by e-beam lithography and double-angle aluminum deposition (see panels e and f in Fig. 1). Before depositing the aluminum, the niobium surface was cleaned by argon ion-milling (dose 10 18 neutralized 500 eV ions per square centimeter). The Nb/Al contact resistance was found to be in the ohm range, yielding tunnel junctions of negligible inductance compared to that of the SQUID.

Fig. 3 Experimental setup. The sample is thermally anchored at the mixing chamber of a dilution refrigerator with temperature 40 -60 mK. It is connected to a vector network analyzer (VNA) at room-temperature that measures the amplitude and phase of the S 21 coefficient. The input line (top) is strongly attenuated (120 to 160 dB in total) with cold attenuators to protect the sample from external and thermal noise, and filtered above 2 GHz. The output line (bottom) includes a cryogenic amplifier with a 3 K noise temperature and 3 cryogenic isolators.

Design

Experimental setup

The chips were glued on a TMM10 printed-circuit board (PCB). The input and output port of the resonator were wire-bonded to coplanar waveguides on the PCB, connected to coaxial cables via mini-SMP microwave launchers. The PCB was mounted in a copper box. The S 21 coefficient (amplitude and phase) of the scattering matrix was measured as a function of frequency using a vector network analyzer. Test resonators were measured in a pumped 4 He cryostat reaching temperatures of 1.3 K, with typical input power of -50 dBm and using room-temperature amplifiers. We measured internal quality factors up to 2×10 5 with both etching methods.

The tunable resonators were measured in a dilution refrigerator operated at 40 -60 mK, using the microwave setup shown in Fig. 3. The input line includes roomtemperature and cold attenuators. The output line includes 3 cryogenic isolators, a cryogenic amplifier (from Berkshire) operated at 4 K with a noise temperature of 3 K, and additional room-temperature amplifiers. The attenuators and isolators protect the sample from external and thermal noise. This setup allowed to measure the sample with intra-cavity energies as small as a few photons in order to operate in the linear regime, corresponding to typical input powers of -140 dBm at the sample level.

Experimental results

Two tunable resonators were measured: sample A has only one SQUID (see Fig. 1e) and large coupling capacitors (27 fF) so that its total quality factor is determined by Qc = 3.4 ×10 3 . Sample B has an array of 7 SQUIDs in series (see Fig. 1f) and smaller coupling capacitors (2 fF) so that its quality factor is likely to be dominated by internal losses or inhomogeneous broadening. Relevant sample parameters are listed in table 1.

A typical resonance curve, obtained with sample A at Φ = 0 for an input power of -143 dBm corresponding to a mean photon number in the cavity n ≈ 1.2, is shown in Fig. 4. The |S 21 | curve was normalized to the maximum measured value. By fitting both the amplitude and the phase response of the resonator, we extract the resonance frequency and the quality factor Q. When the flux through the SQUID is varied, the resonance frequency shifts periodically as shown in Fig. 4b, as expected.

The resonance frequency f 0 (Φ) and quality factor Q(Φ) are shown for both samples in Fig. 5 over one flux period. The f 0 (Φ) curves in panels a and c are fitted with Eq. 7. The agreement is good over the whole frequency range, which extends from 1.3 to 1.75 GHz, yielding a tunability range of 30%. The small discrepancy observed for sample B might be due to a dispersion in the various SQUID loop areas that is not taken into account in our model. The parameters obtained by this procedure for both samples are shown in table 1; they are in good agreement with design values and test-structure measurements.

The Q(Φ) dependance for both samples is shown in panels b and d of Fig. 5. Both samples show a similar behaviour: the quality factor depends weakly on Φ when the flux is close to an integer number of flux quanta, whereas it shows a pronounced dip around Φ 0 /2.

The largest quality factors are 3.5×10 3 for sample A and 3×10 4 for sample B. This difference is due to the different coupling capacitors. For sample A, the maximum quality factor is the same as measured on test resonators with similar capacitors and corresponds to the expected Qc for Cc = 27 fF. Therefore sample A quality factor is limited by the coupling to the 50 Ω lines around integer values of Φ 0 . The situation is different for sample B: the measured value is one order of magnitude lower than both the quality factor Qc =6×10 5 expected for Cc = 2 fF and the measured Q of test resonators with the same capacitors (see table 1). This unexplained broadening of the resonance in presence of a SQUID array might be due either to the presence of low-frequency noise in the sample, or to a dissipation source specifically associated with the SQUIDs. We note that flux-noise is not plausible since our measurements show no clear correlation with the sensitivity of the resonator to flux-noise. However, critical-current noise could produce such effect. Another possibility could be dielectric losses in the tunnel barriers. We now turn to the discussion of the dip in Q(Φ) observed around Φ 0 /2. We attribute it to thermal noise. Indeed, as discussed in section 2, the resonance frequency depends on the energy stored in the resonator. At thermal equilibrium, fluctuations in the photon number translate into a fluctuation of the resonance frequency and cause an inhomogeneous broadening. At temperature T , the resonator stores an average energy given by Planck's formula E = ω 0 (Φ)n, n = 1/{exp[ ω 0 (Φ)/kT ]-1} being the average photon number. The photon number and energy fluctuations are n 2 -n 2 = n(n + 1) and

δE 2 = E 2 + ω 0 (Φ)E. (10)
The characteristic time of these energy fluctuations being given by the cavity damping time Q/ω 0 with Q ≫ 1, a quasi-static analysis is valid and leads to an inhomogeneous broadening δω inh = |dω 0 /dE| δE 2 . Using Eq. 9, we get

Q -1 inh (Φ) = δω inh (Φ) ω 0 (Φ) = N 2ω 0 (Φ) πR 0 [1 + 2N ε(Φ)] 2 ϕ 0 8I 3 c (Φ) δE 2 . ( 11
)
The resulting quality factor is Q -1 = Q -1 inh +Q -1 ext , which is plotted as full curves in panels b and d of Fig. 5, for T = 60 mK. The agreement is good, although Eq. 11 results from a first-order expansion that is no longer valid in the close vicinity of Φ 0 /2. We have also observed that Q values significantly degrade around Φ 0 /2 when the samples are heated, while remaining unchanged around integer numbers of Φ 0 . These observations suggest that thermal noise is the dominant contribution to the drop of Q. Note that our model does not take into account flux-noise, which evidently contributes to Q inh and could account for the residual discrepancy between experimental data and theoretical curves in panels b and d of Fig. 5.

Conclusion

We have designed and measured SQUID-based stripline resonators that can be tuned between 1.3 GHz and 1.75 GHz, with a maximum Q =3×10 4 limited by an unknown mechanism. The quality factor degrades due to thermal noise around Φ 0 /2. This limitation would be actually lifted with higher frequency resonators matching typical Josephson qubit frequencies. Their tunability range at high Q would then be wide enough to couple a large number of qubits.

Fig. 1 a

 1 Fig. 1 a: Tunable resonator scheme : a DC SQUID array is inserted between two λ/4 waveguides coupled to a 50 Ω measurement line through input and output capacitors Cc. b: Optical micrograph of a CPW niobium resonator. c: Typical coupling capacitor (design value : Cc = 27fF). d: Gap in the middle of the resonator, before SQUID patterning and deposition. e: Electron micrograph of an aluminum SQUID (sample A), fabricated using electron-beam lithography and double-angle evaporation. f: Electron micrograph of a 7-SQUID array (sample B).

Fig. 4 (

 4 Fig. 4 (color online) a: Measured (thin line) amplitude (top) and phase (bottom) transmission of sample A for Φ = 0 and fit (bold line) yielding a quality factor Q = 3300. b: Measured resonance frequency of sample A (squares) as a function of applied magnetic flux and corresponding fit (full line) according to Eq. 7.

Fig. 5 (

 5 Fig. 5 (color online) a and c: Measured resonance frequency f 0 as a function of Φ/Φ 0 (squares) for samples A and B, respectively, and fit according to Eq. 7 (solid line). b and d: Measured quality factor Q (disks) as a function of Φ/Φ 0 . The solid line is calculated according to the model (see text) for a temperature T = 60 mK.

Table 1

 1 Summary of sample parameters. See text for definitions.

	Measurements

Acknowledgements This work has been supported by the European project EuroSQIP. We acknowledge technical support from P. Sénat, P.F. Orfila and J.C. Tack, and fruitful discussions within the Quantronics group and with A. Lupascu, A. Wallraff, M. Devoret, and P. Delsing.