Prof Nicolas Bredeche

Dr David Ha

Dr Emmanuel Dupoux

PHD Clément Moulin-Frier
email: clement.moulin-frier@inria.fr

Peter Stone

Satinder Singh

Mark Ring

Ismael Tito Freire González

Jules Brochard

Karch, T C Colas

Emergence of communication systems as a way to maintain cooperative networks in multi-agent simulations

Keywords:

An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquiring new skills in order to solve them: what is called open-ended skill acquisition (oesa). Understanding the mechanisms underlying oesa is an important scientific challenge in both cognitive science (e.g. by studying infant cognitive development) and in artificial intelligence (aiming at computational architectures capable of open-ended learning). Both fields, however, mostly focus on cognitive and social mechanisms at the scale of an individual's life. It is rarely acknowledged that oesa, an ability that is fundamentally related to the characteristics of human intelligence, has been necessarily shaped by ecological, evolutionary and cultural mechanisms interacting at multiple spatiotemporal scales.

In this thesis, I present a research program aiming at understanding, modeling and simulating the dynamics of oesa in artificial systems, grounded in theories studying its eco-evolutionary bases in the human species. It relies on a conceptual framework expressing the complex interactions between environmental, adaptive, multi-agent and cultural dynamics. Three main research questions are developed and I present a selection of my contributions for each of them.

• What are the ecological conditions favoring the evolution of skill acquisition?

• How to bootstrap the formation of a cultural repertoire in populations of adaptive agents?

• What is the role of cultural evolution in the open-ended dynamics of human skill acquisition?

By developing these topics, we will reveal interesting relationships between theories in human evolution and recent approaches in artificial intelligence. This will lead to the proposition of a humanist perspective on AI: using it as a family of computational tools that can help us to explore and study the mechanisms driving open-ended skill acquisition in both artificial and biological systems, as a way to better understand the dynamics of our own species within its whole ecological context. This document presents an overview of my scientific trajectory since the start of my PhD thesis in 2007, the detail of my current research program, a selection of my contributions as well as perspectives for future work.

Chapter 1 Introduction

Outline

This thesis is organized into two parts. In Part I I first detail my scientific trajectory in Chapter 2, from the start of my PhD thesis in 2007 to my recruitment as a permanent researcher in the Flowers group at Inria in 2019. Then, in Chapter 3, I present the research program I have initiated since my recruitment as a permanent researcher. This research program aims at understanding, modeling and simulating the dynamics of open-ended skill acquisition in artificial systems, grounded in theories studying its eco-evolutionary bases in the human species. It relies on a conceptual framework expressing the complex interactions between environmental, adaptive, multi-agent and cultural dynamics.

Then, in Part II, I present selected contributions from my entire career -attempting to reinterpret them in light of the proposed framework -as well as perspectives for future work.

How to read this thesis

I obviously recommend reading this thesis from the start to the end. However, I provide below recommendations for readers who have limited time to read it or are looking for specific information.

The main contribution of this thesis is developed in Chapter 3, which takes the form of a new, self-contained position paper where I explain in detail my current research program. I propose a conceptual framework for grounding artificial intelligence in the origins of the human behavior, proposing to understand, model and simulate the emergence of open-ended skill acquisition in humans and machines through an artificial ecology approach. This chapter details the motivation behind this framework and its structure, as well as its position with respect to the existing literature in biological and artificial open-ended skill acquisition. The core arguments of the framework can be found in Section 3.3. It is structured around three main research questions:

• What are the ecological conditions favoring the evolution of skill acquisition? (Section 3.4)

Part I

From my scientific trajectory to my current research program

Short curriculum

My research mostly focuses on understanding and modeling the origins of complex social behavior in humans and machines. The underlying theoretical question pertains to the origins of cooperation and communication and the role of self-organization mechanisms in the co-structuration of sensorimotor skills and social interactions. My approach aims to analyze and model those mechanisms in their evolutionary, developmental and sociocultural contexts in order to extract strong computational principles. In turn, those principles are implemented and simulated to support, invalidate or reframe existing theoretical hypotheses from the literature. For this aim, I rely on theories and hypotheses from life science (language evolution, speech science, infant development and motor control, human evolution, behavioral ecology, evolutionary biology) and use methods from computer science as modeling tools (cognitive modeling, control theory, machine learning, multi-agent systems, complex systems, developmental robotics, evolutionary computation). As a result of this modeling and simulation effort, testable hypotheses on the origins of social behaviors, as well as real-world applications in artificial intelligence and robotics, can be proposed. My scientific trajectory evolved from a focus on the evolution and acquisition of speech and language, to a broader scope on the origins of open-ended skill acquisition in humans and machines. I detail below this scientific trajectory, step by step (I am co-author of all the references cited in this section).

My research career started in 2007 when I started my PhD at the Gipsa-Lab 1 in Grenoble, under the supervision of Jean-Luc Schwartz, Pierre Bessière and Julien Diard. At this point, I became interested in modeling and simulating the emergence of speech and language in populations of sensorimotor agents [START_REF] Moulin-Frier | Emergence of a language through deictic games within a society of sensory-motor agents in interaction[END_REF][START_REF] Steels | Modeling the cultural evolution of language[END_REF], 2015a;[START_REF] Schwartz | On the cognitive nature of speech sound systems[END_REF], see Section 6.3 for my main contribution on this topic). I studied the theories of the origin of language as well as the neuroanatomical circuits of speech perception and production, notably during a 6-month visit at the University of Southern California in Los Angeles with Michael A. Arbib [START_REF] Arbib | Recognizing speech in a novel accent: The motor theory of speech perception reframed[END_REF].

At the end of my thesis in September 2011, I realized a short post-doctoral project of 4 months at the Laboratoire de Physiologie de la Perception et de l'Action (LPPA) of the Collège de France2 with Jacques Droulez, where I worked on probabilistic methods of optimal control (N' Guyen et al., 2013).

I was then recruited in January 2012 for a 20-month post-doctoral position in the Flowers3 research team at Inria Bordeaux Sud-Ouest in France. My contract will be extended by 16 months (for a total of 36 months) as a research scientist. With Pierre-Yves Oudeyer, director of the team, I worked on the main topics forming the research program of developmental robotics in the context of his ERC Starting Grant EXPLORERS. Making the link with my thesis work, I applied in a novel way the algorithmic architectures for intrinsically motivated autonomous exploration (Moulin-Frier & Oudeyer, 2013a,b;Moulin-Frier et al., 2014b) to the modeling of early vocal development in humans and machines [START_REF] Moulin-Frier | Curiosity-driven phonetic learning[END_REF][START_REF] Moulin-Frier | The role of intrinsic motivations in learning sensorimotor vocal mappings: A developmental robotics study[END_REF]Moulin-Frier et al., 2014a, 2017a, see Section 6.2 for my main contribution on this topic).

As of January 2015, I was recruited at the Synthetic, Perceptive, Emotive and Cognitive Systems (SPECS)4 laboratory of the Universitat Pompeu Fabra in Barcelona as a post-doctoral researcher coordinating the scientific and technological integration of the European project WYSIWYD 5 , with Paul Verschure. This project involved six research labs on an ambitious project focusing on language acquisition in human-robot interaction [START_REF] Fischer | iCub-HRI: A Software Framework for Complex Human-Robot Interaction Scenarios on the iCub Humanoid Robot[END_REF]Moulin-Frier et al., 2016, 2018;Puigbò et al., 2015cPuigbò et al., ,b,a, 2016;;[START_REF] Sánchez-Fibla | A sensorimotor account of visual and tactile integration for depth perception: An iCub robot experiment[END_REF]. During this period, I also worked on emerging social conventions in multi-agent reinforcement learning [START_REF] Freire | Modeling the formation of social conventions from embodied real-time interactions[END_REF]Moulin-Frier et al., 2015b) and made several theoretical contributions in artificial intelligence and cognitive science [START_REF] Moulin-Frier | Embodied Artificial Intelligence through Distributed Adaptive Control: An Integrated Framework[END_REF][START_REF] Arsiwalla | The Morphospace of Consciousness[END_REF]Moulin-Frier & Verschure, 2016).

Then, in October 2017, I joined the US-based artificial intelligence research company Cogitai6 (now Sony-AI America), founded by renowned scientists in reinforcement learning (Peter Stone, Satinder Bajeva, Mark Ring) and whose goal is to build the first operational architecture for continuous learning from experience. I was trained in the methods resulting from recent advances in artificial intelligence, in particular deep reinforcement learning and its variations, including those inspired by developmental robotics that I had studied during my projects at Inria-Flowers.

Finally, in October 2019 I was recruited as a permanent researcher in the Flowers 3 research group at Inria in France. I am co-supervising 3 PhD students and 1 postdoctoral researcher on topics ranging from multi-agent reinforcement learning (Moulin-Frier & Oudeyer, 2021;Nisioti et al., 2022, see Section 7.3 for a contribution on this topic), language-based reinforcement learning [START_REF] Colas | Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration[END_REF](Colas et al., , 2022a;;[START_REF] Karch | Deep Sets for Generalization in RL. In Beyond "Tabula Rasa[END_REF], see Section 7.2 for a contribution on this topic), artificial life [START_REF] Etcheverry | Hierarchically-Organized Latent Modules for Exploratory Search in Morphogenetic Systems[END_REF]Hamon et al., 2022, see Section 5.4 for a contribution on this topic) and evolutionary computation (Nisioti & Moulin-Frier, 2022;Colas et al., 2021, see Section 5.2 for a contribution on this topic). More generally, I have initiated a long-term research program aiming at understanding and modeling how open-ended skill acquisition emerges in populations of artificial agents through the complex interaction between environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms operating at multiple spatiotemporal scales. For this aim, I have obtained funding from Inria (Exploratory Action origins and Cordi programs) as well as the French National Agency for Research (ANR, project ecocurl). Chapter 3, which can be considered the main contribution of this thesis, is explaining in detail the motivation behind this research program.

An integrative effort

During this career, a recurrent pattern has emerged: I have a particular interest in integrating different scientific trends into unified conceptual or computational frameworks.

Here are a few examples. The model I developed during my PhD thesis, originally for simulating the emergence of speech sound systems in agent populations, has actually contributed to the unification of major theories in speech production and perception processing in a unified Bayesian formulation (Moulin-Frier et al., 2012, 2010). During my post-doc position at Flowers, I have proposed a probabilistic unification of the major exploration strategies proposed in developmental robotics (Moulin-Frier & Oudeyer, 2013a). During my post-doc position at SPECS, I have proposed that major existing architectures in artificial intelligence (from top-down symbolic reasoning, to behavioral robotics, to modern deep reinforcement learning) can be expressed within the context of an integrated layered cognitive architecture [START_REF] Moulin-Frier | Embodied Artificial Intelligence through Distributed Adaptive Control: An Integrated Framework[END_REF].

The conceptual framework I will propose in Chapter 3 is the last and most developed instantiation of this integrative effort. It highlights strong relationships between recent contributions in artificial intelligence and hypotheses in human behavioral ecology, a research field studying how ecological constraints and opportunities could have shaped human behavior throughout its evolutionary history. This leads to the proposition of a conceptual framework emphasizing that open-ended skill acquisition is not only implemented as a cognitive mechanism, but is the product of complex interactions between environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms.

Curriculum Vitae (including publication list)

Chapter 3

Research program: Grounding Artificial

Intelligence in the Origins of Human Behavior

Context

This chapter has been written for the purpose of this thesis in the form of a selfcontained position paper, which I then plan to submit to a journal. It describes in detail a conceptual framework resulting from the Exploratory Action1 I am leading at Inria: the so-called origins project2 (2020-2022). This chapter further develops previous short position papers [START_REF] Nisioti | Grounding Artificial Intelligence in the Origins of Human Behavior[END_REF][START_REF] Nisioti | Grounding an Ecological Theory of Artificial Intelligence in Human Evolution[END_REF] written with my colleague Eleni Nisioti, the post-doc researcher recruited on the origins project. Eleni Nisioti has had a major role in co-elaborating this research program.

I first present different approaches to artificial intelligence (ai) and argue that a central feature of human-like intelligence is our tendency to autonomously invent new problems and to proactively learn how to solve them: what is called open-ended skill acquisition (oesa) (Section 3.1). Then, I highlight strong relationships between recent contributions in ai and hypotheses in human behavioral ecology (hbe), a research field studying how ecological constraints and opportunities could have shaped human behavior throughout its evolutionary history (Section 3.2). This leads to the propositon of a conceptual framework emphasizing that oesa is not only implemented as a cognitive mechanism but is the product of complex interactions between environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms (Section 3.3). I provide a detailed analysis of the relationships between these two fields (ai and hbe) developing the original research questions it raises (Sections 3.4 to 3.6) and proposing a roadmap to address them (Section 3.7).

Introduction

What does intelligence means in the context of artificial intelligence (ai)? There is no clear consensus about this question in the ai community, which is reflected in the diversity of terms used to characterize the end-goal of ai, e.g. artificial general intelligence (Goertzel & Pennachin, 2007), strong ai [START_REF] Searle | Minds, brains, and programs[END_REF], human-level ai [START_REF] Mccarthy | From here to human-level AI[END_REF], unified theories of cognition [START_REF] Newell | Unified Theories of Cognition[END_REF], universal intelligence (Legg & Hutter, 2007), enactive ai [START_REF] Froese | Enactive artificial intelligence: Investigating the systemic organization of life and mind[END_REF] or open-ended intelligence [START_REF] Weinbaum | Open ended intelligence: The individuation of intelligent agents[END_REF]).

An early proposition from [START_REF] Newell | Unified Theories of Cognition[END_REF] states that "a system is intelligent to the degree that it approximates a knowledge-level system", i.e. a system "embedded in an external environment, with which it interacts by a set of possible actions [. . .] to attain its goals, using all the knowledge that it has". Many other definitions have been proposed: as paraphrased by [START_REF] Chollet | On the Measure of Intelligence[END_REF], Legg et al. (2007) summarized in 2007 no fewer than 70 definitions from the literature into a single statement: "Intelligence measures an agent's ability to achieve goals in a wide range of environments". Goertzel & Pennachin (2007) define artificial general intelligence as "AI systems that possess a reasonable degree of self-understanding and autonomous self-control, and have the ability to solve a variety of complex problems in a variety of contexts, and to learn to solve new problems that they didn't know about at the time of their creation". More recently, [START_REF] Chollet | On the Measure of Intelligence[END_REF] proposes that "the intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty".

The definitions mentioned above are however purely mechanistic, in the sense that they focus on the final state of an intelligent agent and charaterize it through measurable properties. This comes in contrast with the enactive view of cognition and ai, which instead emphasizes the processes driving the emergence of autonomy and intelligence [START_REF] Varela | The Embodied Mind: Cognitive Science and Human Experience[END_REF][START_REF] Brooks | Intelligence without representation[END_REF][START_REF] Froese | Enactive artificial intelligence: Investigating the systemic organization of life and mind[END_REF][START_REF] Weinbaum | Open ended intelligence: The individuation of intelligent agents[END_REF][START_REF] De Jaegher | Participatory sense-making[END_REF]. I describe below the main hypotheses and methods driving these two approaches to artificial intelligence.

Current approaches in the quest for artificial intelligence

This quest for artificial intelligence has been revived by recent advances in machine learning (ml), leading many renowned researchers in the field to elaborate on prospective paths toward human-level ai. We can roughly categorize them into two main categories, mirroring the mechanistic vs. enactive views mentioned above.

3.1.1.1 Cognition-centric approach: Focus on architectures and optimization .

This approach considers achieving human-level ai mostly as an engineering problem. The main assumption is that intelligence must be implemented in a structured cognitive architecture (integrating e.g. control, learning and memory mechanisms) which is optimized (using ml methods) through pre-defined objective functions. The recent rise of deep neural networks as powerful function approximators has strongly revived this approach by allowing key advances in e.g. representation learning and reinforcement learning in high-dimensional spaces [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF].

Recent position papers and debates among influential researchers in the field illustrate well the principles behind this dominant approach. A very recent and prototypical example is a position paper from Yann Lecun where he sketches an integrated cognitive architecture for achieving human-level intelligence [START_REF] Lecun | A Path Towards Autonomous Machine Intelligence[END_REF]. This extract from the paper's abstract illustrates well the usual workflow of the cognition-centric approach: "This position paper proposes an architecture and training paradigms with which to construct autonomous intelligent agents. It combines concepts such as configurable predictive world model, behavior driven through intrinsic motivation, and hierarchical joint embedding architectures trained with self-supervised learning" (LeCun ( 2022)). As we can see, the approach clearly focuses on how to engineer a cognitive architecture in terms of interacting modules as well as how to optimize it using machine learning algorithms.

Within this cognition-centric approach, a debate has been initiated between proponents of a data-driven view (e.g. [START_REF] Silver | Reward is enough[END_REF], with the provocative claim that achieving human-level ai is nowadays mostly a matter of scaling up existing architectures, datasets and environments; vs. proponents of an architecture-driven view emphasizing the central role of priors and symbolic reasoning, that are central in human intelligence but poorly addressed by dnn models (e.g. [START_REF] Marcus | Rebooting AI: Building Artificial Intelligence We Can Trust[END_REF]. In both views, the proposed methods are evaluated in benchmarks designed to capture various aspects of intelligence. For example, [START_REF] Chollet | On the Measure of Intelligence[END_REF] defines intelligence as "a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty" and proposes a benchmark to evaluate it inspired by psychometric intelligence tests and called the The Abstraction and Reasoning Corpus (ARC).

In summary, the cognition-centric approach proposes to formalize a given definition of intelligence as an objective function, optimize the parameters of a cognitive architecture against this objective and evaluate progress through adequate benchmarks testing their ability to generalize on test environments. In this sense it mirrors the standard methodology adopted in cognitive psychology and neuroscience, where researchers measure the behavioral and neural response of participants attempting to achieve pre-defined tasks.

3.1.1.2 Emergentist approach: Focus on environmental and multi-agent dynamics .

In contrast to the previous approach considering intelligence as the result of a welloptimized cognitive architecture, an alternative approach is to view it as the emergent product of adaptive systems interacting with complex environmental dynamics. In other words, if the cognition-centric approach attempts to reverse-engineer the brain (or at least its main functions), the emergentist approach instead attempts to reverse-engineer the environmental conditions that lead to intelligence. There are two main propositions here. Some contributions study how competition and cooperation pressures in populations of co-adapting agents can result in a behavioral arms race where each agent has to continuously improve its skills against those of other agents, an approach called multi-agent autocurriculum [START_REF] Leibo | Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research[END_REF]. Other contributions study how learning algorithms themselves can be meta-learned for operating in a diversity of environments. [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF] calls this approach ai-Generating Algorithms (AI-GA), with three main pillars: "(1) meta-learning architectures, (2) meta-learning the learning algorithms themselves, and (3) generating effective learning environments". In both propositions (autocurriculum and AI-GA), it is the complexity of the environment (either through the presence of other co-adapting agents or through its intrinsic diversity) that drives the ability to continuously acquire new skills and generalize them in novel environments. This approach is usually less concerned with human cognition than the cognition-centric approach. Instead, it takes its inspiration from the fields of complex systems [START_REF] Ha | Collective Intelligence for Deep Learning: A Survey of Recent Developments[END_REF] and evolutionary biology. This limited grounding in human intelligence is well illustrated in this quote from [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF]:

A major open question that remains is how we can constrain the generation of environments to be those we find interesting and/or that produce intelligence that helps us solve real-world problems. [. . .] For example, one might argue that such a system could produce intelligence that is alien to us and that we cannot communicate with. However, if it is truly general intelligence, presumably through its learning efforts and our own we could learn to communicate with it. Additionally, creating alien forms of intelligence would be fascinating as it would teach us about the limits and possibilities within the space of intelligent beings.

In conclusion, the cognition-centric and the emergentist approaches are dual. On the one hand, the cognition-centric approach is focused on human intelligence as a measurable property and attempts at simulating, at different levels of abstraction, the cognitive processes underlying it. Its search space is the space of cognitive architectures and processes, while environmental properties belong to the experimental benchmarks designed for evaluating the proposed architectures. In this view, ai is strongly connected to human cognition through the modeling of mechanisms such as learning and memory for solving complex tasks. The cognition-centric approach however shows major limitations: its search space is incredibly vast (including all possible combinations of all possible implementations of diverse cognitive modules, [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF]; it is highly subject to deception (focusing on improving performance measures can prevent to explore alternative promising paths to ai, [START_REF] Stanley | Why Greatness Cannot Be Planned: The Myth of the Objective[END_REF] and it mostly ignores non-cognitive mechanisms that could have played a major role in the emergence of intelligence (e.g. environmental complexity or multi-agent dynamics). On the other hand, the emergentist approach instead considers intelligence as the emergent product of a larger dynamical system involving complex environmental and multi-agent dynamics. Its search space is the space of dynamical systems potentially driving the emergence of intelligent behavior, i.e. environmental properties are considered as experimental variables. While the emergentist approach addresses most of the limitations of the cognition-centric approach, it usually considers intelligence in all its generality and rarely cares about the specificities of human intelligence. For this reason, it has rarely resulted in real-world applications and will unlikely lead to human-like intelligence (although it could, in theory, discover other interesting types of intelligence).

The main objective of this chapter is to propose a conceptual framework for grounding the emergentist approach in the origins of human behavior.

Human-like intelligence as open-ended skill acquisition

As mentioned above, the ai community has not yet converged on a clear consensus about what is meant by intelligence, nor to a consensus on how to achieve artificial intelligence. Which phenomena is considered as the most relevant to study mostly depends on the considered approach, e.g. benchmarked learning and generalization abilities in the cognition-centric approach vs. co-adaptation patterns between adaptive agents and dynamic environments in the emergentist approach. In this thesis, I propose to focus on what I consider to be the most interesting distinctive feature of human intelligence: open-ended skill acquisition ; and I propose a path to study its emergence in artificial systems.

Open-endedness is the property of a mechanism to continuously generate increasingly diverse and complex structures [START_REF] Stanley | Why Greatness Cannot Be Planned: The Myth of the Objective[END_REF][START_REF] Banzhaf | Defining and simulating open-ended novelty: Requirements, guidelines, and challenges[END_REF]. A typical example is biological evolution, where the relatively simple mechanism of natural selection with variation has produced the open-ended diversity of biological organisms that have lived on Earth. But many other mechanisms are open-ended in the natural world. The dynamics of the environment is to some extent open-ended, generating cyclic patterns at multiple spatio-temporal scales (e.g. seasonal cycles and climate variations) as well as complex chemical reactions (some of these reactions being at the origins of life itself, [START_REF] Walker | Re-conceptualizing the origins of life[END_REF]. Developmental and cognitive mechanisms in the human species are also open-ended (e.g. skill and knowledge acquisition), as well as socio-cultural mechanisms (e.g. technological innovation, social organization, cultural evolution, [START_REF] Stanley | Why Open-Endedness Matters[END_REF][START_REF] Solé | The evolutionary ecology of technological innovations[END_REF].

In fact, a distinctive feature of the human species is our tendency to continuously invent new problems and to proactively acquire Understanding and modeling oesa has been already recognized as an important challenge in both cognitive science and in ai. In cognitive science, pioneer work from the 40s attempted to theorize the manifest drive of human individuals to perform activities not only for obtaining external rewards (e.g. related to feeding or mating) but also because those activities are intrinsically enjoyable in and of themselves. They coined the term intrinsic motivation to express this human drive to continuously invent their own challenges and try to solve them, what we call in everyday language curiosity [START_REF] Deci | Intrinsic Motivation and Self-Determination in Human Behavior[END_REF][START_REF] Berlyne | A theory of human curiosity[END_REF]Csikszentmihalyi, 1997). More recently, some research groups have been seeking the neural correlates of curiosity in primates [START_REF] Gottlieb | Information-seeking, curiosity, and attention: Computational and neural mechanisms[END_REF]. In ai, the computational modeling of oesa has first attracted interest in the 90s (Schmidhuber, 1991;[START_REF] Schmidhuber | Shifting Inductive Bias with Success-Story Algorithm, Adaptive Levin Search, and Incremental Self-Improvement[END_REF]. In the 2000s, developmental robotics emerged as a scientific field attempting to implement mechanisms inspired by infant development into algorithms that can operate on simulated or physical robots [START_REF] Barto | Intrinsically Motivated Learning of Hierarchical Collections of Skills[END_REF]Baranes & Oudeyer, 2013). More recently, principles first proposed in developmental robotics have been integrated to deep reinforcement learning algorithm [START_REF] Pathak | Curiosity-driven exploration by self-supervised prediction[END_REF][START_REF] Colas | Intrinsically motivated modular multi-goal reinforcement learning[END_REF], what is sometimes called developmental ai (Colas et al., 2022b).

As we can see, the cognition-centric approach is here again dominating the field: research in both cognitive science and ai most often considers that the mechanisms underlying oesa are mostly cognitive. It is still rarely acknowledged that open-ended skill acquisition, an ability that is fundamentally related to the characteristics of human intelligence [START_REF] Banzhaf | Defining and simulating open-ended novelty: Requirements, guidelines, and challenges[END_REF], has necessarily been shaped by our environmental, evolutionary, social and cultural history, through complex feedforward and feedback effects operating at multiple spatiotemporal scales.

What are the mechanisms promoting open-ended skill acquisition in biological and artifical agents?

In this thesis, I instead propose to zoom out from this cognition-centric view, by considering oesa within its whole substrate. This requires acknowledging that oesa is not only implemented as a cognitive mechanism but is the product of complex interactions between environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms. For this aim, we will turn our attention towards human behavioral ecology (hbe), a research field studying how ecological constraints and opportunities could have shaped human behavior throughout its evolutionary history [START_REF] Borgerhoff Mulder | Human behavioural ecology[END_REF][START_REF] Nettle | Human behavioral ecology: Current research and future prospects[END_REF]. Research in this field have studied, among others: how speciation, extinction and dispersal arose in the human history depending on climate variation [START_REF] Maslin | A synthesis of the theories and concepts of early human evolution[END_REF], the formation of cooperative groups [START_REF] Chapman | Constraints on group size in red colobus and red-tailed guenons: Examining the generality of the ecological constraints model[END_REF], the rise of resource management [START_REF] Gowlett | The discovery of fire by humans: A long and convoluted process[END_REF], of tool use [START_REF] Debeaune | The invention of technology: Prehistory and cognition[END_REF], the evolution of human language [START_REF] Freeberg | Social complexity as a proximate and ultimate factor in communicative complexity[END_REF] and the emergence of cultural norms and institutions [START_REF] Tomasello | The Cultural Origins of Human Cognition[END_REF]. What principles can we extract from this literature that could help us to achieve oesa in artificial agents?

In other words, our objective is not to directly implement agents capable of oesa -this is the role of the cognition-centric approach -but instead (1) to understand the origins of oesa in the human species in the light of the hbe literature, (2) to extract computational principles from this literature and to implement them in computer simulations, (3) to study how these principles can results in novel ai algorithms and, cherry on the cake, (4) to propose or reframe hypotheses in hbe.

Before going further we must acknowledge that, admittedly, there are many paths to the acquisition of open-ended skills in ai. Grounding our study in human behavioral ecology seems to be but one of the options, but several reasons may persuade us to explore it.

1. Examining all possible paths to oesa is infeasible considering our modern and foreseeable computational power [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF].

2. Species that are more familiar to ours make it easier to define evaluation criteria. For example, human-inspired metrics such as equality, sustainability and social welfare have been employed to evaluate agents on their ability to forage [START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF], find optimal taxation strategies [START_REF] Zheng | The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies[END_REF] and cooperate [START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF].

3. Darwinian evolution offers an existence proof for human-like oesa [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF], as well as empirical data and testable hypotheses.

4. Similar attempts at grounding ai research in a non-computational field have already proven to be a fruitful approach. For example, concepts from Development Science such as intrinsic motivation [START_REF] Oudeyer | Intrinsic Motivation Systems for Autonomous Mental Development[END_REF][START_REF] Pathak | Curiosity-driven exploration by self-supervised prediction[END_REF] and embodied language acquisition [START_REF] Cangelosi | Integration of Action and Language Knowledge: A Roadmap for Developmental Robotics[END_REF] have had a significant impact on modern ai research.

5. The potential of knowledge transfer between human behavioral ecology and artificial intelligence has already been recognized [START_REF] Frankenhuis | Enriching behavioral ecology with reinforcement learning methods[END_REF], with the transfer of ideas having the opposite direction from the one proposed here. A proposal to study major evolutionary transitions in order to understand the general laws that underlie innovation and transfer insights to artificial evolution is presented in [START_REF] Solé | The major synthetic evolutionary transitions[END_REF]. Our proposal follows a similar direction but focuses on highlighting the overlap between concepts in ai and in hbe. In addition, a number of works in ai have recently resorted to theories from ecology, psychology and economics for inspiration (Wang et al., 2019a;[START_REF] Hughes | Inequity aversion improves cooperation in intertemporal social dilemmas[END_REF][START_REF] Jaques | Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning[END_REF][START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF][START_REF] Köster | Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents[END_REF].

The rest of this chapter is organized as follows. In Section 3.2, I provide a review of existing contributions studying open-ended skill acquisition in both ai and in the human species. In Section 3.3, I present the core contribution of this thesis: a novel conceptual framework for understanding, modeling and simulating the emergence of oesa in artificial systems. Then, in Sections 3.4, 3.5 and 3.6, I detail the different parts of this framework and their underlying dynamics. A concluding discussion is provided in Section 3.7.

Existing approaches to open-ended skill acquisition in ai and the human species

In this section, I present existing approaches studying oesa in both ai and in the human species. I propose to organize these approaches into three categories according to their main focus: behavioral, environmental or social diversity.

Approaches focusing on behavioral diversity

In the human species

Open-ended skill acquisition in humans is most often considered as a product of an intrinsic motivation to acquire new knowledge and skills, or in everyday language: curiosity. The term intrinsic motivation refers to a mechanism driving individuals to select and engage in activities for their own sake because they are inherently interesting (in opposition to extrinsic motivation, which refers to doing something because it leads to a separable outcome, e.g. food) [START_REF] Deci | Intrinsic Motivation and Self-Determination in Human Behavior[END_REF].

In particular, children exploration seems to be driven by intrinsically motivated brain processes that trigger spontaneous exploration for the mere purpose of experiencing novelty, surprise or learning progress (Gopnik et al., 1999;Kaplan & Oudeyer, 2007;[START_REF] Kidd | The psychology and neuroscience of curiosity[END_REF]. During exploratory play, children can also invent and pursue their own problems [START_REF] Chu | Exploratory play, rational action, and efficient search[END_REF].

Contemporary theory in cognitive psychology maintains that curiosity is a crucial developmental element that allows humans to acquire useful knowledge [START_REF] Oudeyer | How Evolution May Work Through Curiosity-Driven Developmental Process[END_REF][START_REF] Gopnik | Childhood as a solution to explore-exploit tensions[END_REF]. Curiosity lets us seek out certain experiences for the inherent value of learning from them, not because they are rewarding in a more tangible way. The feeling of curiosity itself is borne out by a cognitive mechanism that prioritizes potential learning experiences in order to prepare the learner for future challenges that are as yet unknown [START_REF] Oudeyer | Computational theories of curiosity-driven learning[END_REF][START_REF] Gopnik | Childhood as a solution to explore-exploit tensions[END_REF].

In artificial intelligence

Intrinsically motivated learning. Precisely how the curiosity-inducing mechanism operates in biological organisms is poorly understood [START_REF] Gottlieb | Towards a neuroscience of active sampling and curiosity[END_REF]. However, the importance of curiosity-driven learning is increasingly recognized in ai, as a potential solution for building autonomous machines that can discover and learn an open-ended repertoire of skills.

Algorithmic models of intrinsic motivation were successfully used in developmental robotics [START_REF] Oudeyer | Intrinsic Motivation Systems for Autonomous Mental Development[END_REF]Baldassarre & Mirolli, 2013), in reinforcement learning [START_REF] Singh | Intrinsically motivated reinforcement learning[END_REF][START_REF] Schmidhuber | Formal Theory of Creativity, Fun, and Intrinsic Motivation[END_REF] and more recently in deep reinforcement learning (drl) [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Pathak | Curiosity-driven exploration by self-supervised prediction[END_REF]. Intrinsically motivated goal exploration processs (Imgeps), in particular, enable agents to sample and pursue their own goals without external rewards (Baranes & Oudeyer, 2013;[START_REF] Forestier | Modular active curiosity-driven discovery of tool use[END_REF][START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF] and can be formulated within the drl framework [START_REF] Held | Automatic Goal Generation for Reinforcement Learning Agents[END_REF][START_REF] Nair | Visual Reinforcement Learning with Imagined Goals[END_REF][START_REF] Colas | Intrinsically motivated modular multi-goal reinforcement learning[END_REF][START_REF] Pong | Skew-Fit: State-Covering Self-Supervised Reinforcement Learning[END_REF][START_REF] Venkattaramanujam | Self-supervised learning of distance functions for goal-conditioned reinforcement learning[END_REF][START_REF] Racaniere | Automated curricula through setter-solver interactions[END_REF]. Intrinsically motivated agents that learn new skills by generating and pursuing their own goals are referred as autotelic agents, from the Greek auto (self) and telos (goal) [START_REF] Steels | The Autotelic Principle[END_REF].

Quality-Diversity algorithms. In a parallel line of research, the recent introduction of quality-diversity algorithms [START_REF] Lehman | Evolving a diversity of virtual creatures through novelty search and local competition[END_REF][START_REF] Colas | Robots that can adapt like animals[END_REF][START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF] has signified a departure from a purely performance-based view of ai and has renewed interest in mechanisms related to the preservation of diversity arising in natural evolution. Quality-Diversity algorithms employ populations of agents that find a set of diverse and well-performing solutions, rather than a single one. This has been proven vital in real-world settings, such as robots that need to recover from damages [START_REF] Colas | Robots that can adapt like animals[END_REF].

Approaches focusing on environmental diversity

In the human species

Human behavioral ecology. Human behavioral ecology (Hbe) is a field studying how ecological constraints and opportunities could have shaped human behavior throughout its evolutionary history [START_REF] Maslin | A synthesis of the theories and concepts of early human evolution[END_REF][START_REF] Brown | Evolutionary accounts of human behavioural diversity[END_REF][START_REF] Sear | Synthesis in the human evolutionary behavioural sciences[END_REF]. The spotlight is on the Rift Valley at East Africa during a period that approximately lasted from 7 to 2 million years ago and is hypothesized to have constituted a turning point in our evolutionary trajectory. This is the time when hominin diversity reached its highest level (with the first specimens attributed to genus Homo), with evidence of episodic migration of hominins out of the Rift Valley and into Eurasia, as well as the most dramatic increases in hominin brain size [START_REF] Maslin | A synthesis of the theories and concepts of early human evolution[END_REF]. Data from paleoclimatology suggests that a main feature of that period is extreme climatic variability, characterized by the presence of lakes that could expand quickly and contract in a highly chaotic way. The interesting end result from a behavioral perspective, is that the presence of these lakes creates niches of wide diversity that vary quickly with time, presumably within a few dozen generations. This extreme climate variability could have had a strong influence on early human evolution, in particular regarding our ability to quickly generalize and extend our behavioral repertoire to changing environments. Due to partial observability of the environmental conditions and an even more partial understanding of how they affect the dynamics of populations, research in hbe have multiple hypotheses on how the environmental dynamics affected the species at that time and place. This evolutionary leap was originally studied under theories that lay emphasis on specific environmental changes. The Savannah hypothesis, for example, suggests that the change in fauna favored bipedal walking, which enabled migration and the creation of new niches for humans. Later hypotheses under the Pulsed Climate Variability (pcv) framework , however, suggest that the key change was instead the general environmental complexity characterizing that period [START_REF] Potts | Hominin evolution in settings of strong environmental variability[END_REF][START_REF] Maslin | A synthesis of the theories and concepts of early human evolution[END_REF]Collard et al., 2005). The pcv framework was recently introduced in an attempt to juxtapose different mechanisms under which environmental conditions may have affected speciation and extinction rates, which further influence the population's diversity. This framework constitutes an important step towards a unified model of the ecological and evolutionary dynamics of that period.

Niche construction theories. Niche construction, the process with which species shape their environment to increase their control over it or reduce its uncertainty [START_REF] Krakauer | Dilemmas, and Monopolies of Niche Construction[END_REF]Constant et al., 2018), has been termed "the neglected process in evolution" [START_REF] Odling-Smee | Niche Construction: The Neglected Process in Evolution[END_REF]), as natural life research previously disregarded it as a minor mechanism driving evolution, compared to natural selection. The contemporary opinion has however shifted towards viewing niche construction as an important evolutionary mechanism that accompanies natural selection. In the introduction of their book, [START_REF] Odling-Smee | Niche Construction: The Neglected Process in Evolution[END_REF] explain:

Organisms play two roles in evolution. The first consists of carrying genes; organisms survive and reproduce according to chance and natural selection pressures in their environments. This role is the basis for most evolutionary theory, it has been subject to intense qualitative and quantitative investigation, and it is reasonably well understood. However, organisms also interact with environments, take energy and resources from environments, make micro-and macrohabitat choices with respect to environments, construct artifacts, emit detritus and die in environments, and by doing all these things, modify at least some of the natural selection pressures present in their own, and in each other's, local environments. This second role for phenotypes in evolution is not well described or well understood by evolutionary biologists and has not been subject to a great deal of investigation. We call it "niche construction" (Odling-Smee 1988) and it is the subject of this book. [START_REF] Odling-Smee | Niche Construction: The Neglected Process in Evolution[END_REF] This original concept of ecological niche construction has then been extended to describe more specific mechanisms, including cognitive [START_REF] Clark | Niche and Super-Niche: How Language Makes Minds Matter More[END_REF], social [START_REF] Bergmüller | Animal personality due to social niche specialisation[END_REF][START_REF] Saltz | What, if anything, is a social niche?[END_REF] and cultural niche construction [START_REF] Laland | Cultural niche construction and human evolution[END_REF][START_REF] Laland | Cultural Niche Construction: An Introduction[END_REF].

Cognitive niche construction, which is defined "as the process by which animals build physical structures that transform problem spaces in ways that aid (or sometimes impede) thinking and reasoning about some target domain or domains" [START_REF] Clark | Niche and Super-Niche: How Language Makes Minds Matter More[END_REF]).

Social niche construction is the mechanism in which individuals, singly or collectively, influence the composition and dynamics of their social environments [START_REF] Flack | Policing stabilizes construction of social niches in primates[END_REF][START_REF] Saltz | What, if anything, is a social niche?[END_REF]. [START_REF] Saltz | What, if anything, is a social niche?[END_REF] propose that "social niche construction has occurred whenever an individual, through its behavior or other traits, modifies its social environment, and the modified social environment is within the focal individual's social niche."

Finally, another type of niche construction, which is particularly important in the human species, is cultural niche construction. According to [START_REF] Laland | Cultural niche construction and human evolution[END_REF]:

Human innovation and technology has had an enormous impact on the environment: it has made many new resources available via both agriculture and industry; it has influenced human population size and structure via hygiene, medicine and birth control; it is drastically reducing bio-diversity; and it may already have resulted in the degradation of large areas of our global environment. These are all potential sources of modified natural selection pressures. Thus cultural processes that precipitate niche construction might be expected to have played a critical role in human evolution for many thousands, perhaps millions of years. [START_REF] Laland | Cultural niche construction and human evolution[END_REF])

In artificial intelligence

Environment design. Progressing in the design of efficient learning architectures is often associated with the proposition of more complex environments to evaluate them. Reinforcement learning (Rl) environments consisting of a large collection of retro video games (Arcade Learning Environment, [START_REF] Bellemare | The Arcade Learning Environment: An Evaluation Platform for General Agents[END_REF] have been widely used since the rise of deep reinforcement learning (drl) algorithms [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. Challenging board games with very large search spaces and requiring long-horizon planning have served as a key challenge driving the integration of drl, Monte-Carlo tree search and self-play [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF][START_REF] Silver | Mastering the game of Go without human knowledge[END_REF]. Surpassing human level in multi-player strategy video games has driven the development of large scale multi-agent reinforcement learning (marl) algorithms [START_REF] Vinyals | Mastering the Real-Time Strategy Game StarCraft II[END_REF][START_REF] Openai | OpenAI Five[END_REF]. Procedurally generated environments [START_REF] Risi | Increasing Generality in Machine Learning through Procedural Content Generation[END_REF] are used to benchmark the ability of (meta-)drl algorithms to generalize their learned policy to novel tasks that were not experienced during training. Among the available simulation environments, the Minecraft video game is considered has one of the most challenging for the ai community [START_REF] Johnson | The Malmo Platform for Artificial Intelligence Experimentation[END_REF][START_REF] Fan | Building Open-Ended Embodied Agents with Internet-Scale Knowledge[END_REF]. It consists in a procedurally generated 3D world with virtually infinite terrain where the player may discover and extract raw materials, craft tools and items, and build structures, earthworks, and simple machines (Wikipedia, 2022). I will refer to the possibility to produce new elements by composing other elements as a an environment with compositional dynamics (as in Minecraft but also others, see e.g. [START_REF] Wang | Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents[END_REF][START_REF] Platanios | Jelly bean world: A testbed for never-ending learning[END_REF][START_REF] Ortiz | Gärdenfors, P. Cooperation and the Evolution of Symbolic Communication[END_REF], offering opportunities for e.g. tool crafting (Figure 5.13). Interestingly, Minecraft is relevant to evaluate both cognition-centric and emergentist approaches (Section 3.1.1). On the one hand, it allows defining a large repertoire of challenging tasks on which to benchmark learning architectures (in the spirit of the cognition-centric approach). On the other hand, it is perhaps the most open-ended environment available to the community, in the sense that the player can explore these large and complex worlds without any externally provided goals (i.e. in a fully autotelic fashion, see Section 3.2.1.2). Minecraft therefore appears as a particularly relevant environment for the emergentist approach, which considers intelligence as inextricably linked to environmental dynamics.

Meta reinforcement learning. Meta reinforcement learning (Meta-rl) aims at equipping agents with the ability to generalize to tasks or environments that have not been encountered during training. Two nested processes of adaptation are traditionally considered: the inner level is a standard drl algorithm operating on a given environment. The outer level is tuning the (hyper-)parameters of the inner loop such that it performs well on a distribution of environments. The end result of this nested training process is an algorithm that learns how to learn, i.e. learns how to adapt to unseen tasks. Works differ in the optimization technique and adaptation mechanism they consider in the two loops. The outer loop often optimizes higher-level structures such as the architecture of a neural network [START_REF] Baker | Designing neural network architectures using reinforcement learning[END_REF], the morphology [START_REF] Gupta | Embodied intelligence via learning and evolution[END_REF], a curiosity module providing intrinsic rewards [START_REF] Alet | Meta-learning curiosity algorithms[END_REF] or plasticity rules [START_REF] Najarro | Meta-Learning through Hebbian Plasticity in Random Networks[END_REF], employing either evolutionary or gradient-based optimization. Impressively, the meta-learning paradigm has managed to learn algorithms from scratch [START_REF] Oh | Discovering reinforcement learning algorithms[END_REF][START_REF] Kirsch | Meta learning backpropagation and improving it[END_REF], showcasing that human-engineered solutions often stumble upon solutions found by meta-learning.

Curriculum learning. Curriculum learning refers to a family of methods aiming at improving learning agent performance by carefully choosing the task sequence the agent is exposed to. It is inspired by curriculum learning in humans and in particular in education, where children are first exposed to easy task (e.g. learning how to count integers) then progressively to more and more difficult tasks (e.g. learning additions, then multiplication, then division). The task sequence is supposed to follow a logical order taking into account dependences across the different tasks, e.g. learning how to multiply integers first requires mastering how to perform additions. This task sequence can be either predefined or adapted dynamically according to the agent's learning dynamics (see [START_REF] Portelas | Automatic curriculum learning for deep RL: A short survey[END_REF], for a recent review focusing on drl agents). While meta-learning approaches sample environments from a distribution and therefore have no control over the order in which different environments are presented to the agent, curriculum learning instead focuses on the ordering in which tasks are presented to the agent. Note however that curriculum learning does not necessarily act on environments, they can also control goals or reward functions (as in intrinsically motivated approaches presented above) or the selection of opponents in multi-agent settings (see below). See again [START_REF] Portelas | Automatic curriculum learning for deep RL: A short survey[END_REF] for a complete taxonomy of curriculum learning approaches.

Approaches focusing on social diversity

In the human species

If cooperation requires that an agent pays a reproductive cost for someone else's benefit, how can cooperation emerge in a population of agents evolving selfishly? Under the big mistake hypothesis [START_REF] Burnham | The biological and evolutionary logic of human cooperation[END_REF], altruism emerged in small-scale groups due to kin selection or reciprocity. In contrast, the interdependence hypothesis [START_REF] Tomasello | Two Key Steps in the Evolution of Human Cooperation[END_REF] proposes a theory for the emergence of cooperation that replaces altruism with mutualistic collaboration. According to it, the need for foraging led to the selective helping of those who were needed as collaborative partners in the future. In sufficiently small groups, social selection was performed based on reputation. The size and structure of groups was dynamically shaped by their need to maintain stability and defend themselves against other groups.

Competition pressures can result in arms race dynamics where different groups or species continuously co-adapt to each other, potentially leading to increased behavioral complexity (the Red Queen hypothesis, [START_REF] Van Valen | A new evolutionary law[END_REF], see also [START_REF] Maslin | A synthesis of the theories and concepts of early human evolution[END_REF] for its interpretation in the context of hominin evolution). Competition between co-existing groups can also result in an increase in socio-cultural diversity through the mechanism of schismogenesis [START_REF] Bateson | Culture Contact and Schismogenesis[END_REF], i.e. the tendency to define oneself by opposition to one another in a self-amplifying process of divergence (see also [START_REF] Graeber | The Dawn of Everything: A New History of Humanity[END_REF], for the relevance of this concept in anthropology).

Several theories of the origins of human language postulate a joint evolution of cooperative and communicative behaviors [START_REF] Smith | Communication and collective action: Language and the evolution of human cooperation[END_REF]Gärdenfors, 2002;[START_REF] Ghazanfar | The evolution of speech: Vision, rhythm, cooperation[END_REF][START_REF] Tomasello | Two Key Steps in the Evolution of Human Cooperation[END_REF]. It is in particular the central thesis of the theory developed by Michael Tomasello, who proposes that "humans' species-unique forms of cooperation -as well as their species-unique forms of cognition, communication, and social life-all derive from mutualistic collaboration (with social selection against cheaters)" [START_REF] Tomasello | Two Key Steps in the Evolution of Human Cooperation[END_REF]). In this view, it is the constraints imposed by the social niche occupied by human beings that has driven them to jointly develop complex collaborative and communicative behaviors, in a context of interdependence requiring the sharing of intentions. We also find compatible arguments in the mirror system hypothesis developed by Michael Arbib [START_REF] Arbib | From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics[END_REF] proposing that language evolution is grounded in the sensory-motor integration required for the execution and the observation of transitive actions towards objects, enabling other's intention recognition and providing the bases of a syntactic structure [START_REF] Roy | The syntactic motor system[END_REF]) (see also [START_REF] Iriki | Triadic (ecological, neural, cognitive) niche construction: A scenario of human brain evolution extrapolating tool use and language from the control of reaching actions[END_REF], for theoretical propositions on the coevolution of tool use and language in humans). Finally, the social complexity hypothesis suggests that groups with complex social structures require more complex communication systems to regulate interactions between group members [START_REF] Freeberg | Social complexity as a proximate and ultimate factor in communicative complexity[END_REF].

Social behaviors are also shaped through the mechanism of cultural transmission, where behaviors of an individual arises through induction on the basis of observations of behavior in another individual. The compositionality of language could have emerged not only for its expressive power, but also as a way to facilitate its learning by new infants in a population: a mechanism called iterated learning [START_REF] Kirby | Iterated learning and the evolution of language[END_REF] which has been studied both in human experiments [START_REF] Tessler | Learning to solve complex tasks by growing knowledge culturally across generations[END_REF] and ai [START_REF] Ren | Compositional languages emerge in a neural iterated learning model[END_REF][START_REF] Li | Ease-of-Teaching and Language Structure from Emergent Communication[END_REF].

In his seminal work, the developmentalist Led Vygotsky promotes a social conception of intelligence. He first proposed that linguistic social interactions such as descriptions, explanations, corrections, or play start as interpersonal mechanisms before they are turned into intrapersonal cognitive mechanisms through the process of internalization [START_REF] Vygotsky | Tool and Symbol in Child Development[END_REF][START_REF] Vygotsky | Play and Its Role in the Mental Development of the Child[END_REF][START_REF] Vygotsky | Thought and Language[END_REF]. Following his vision, many psychologists [START_REF] Berk | Why Children Talk to Themselves[END_REF][START_REF] Lupyan | What Do Words Do? Toward a Theory of Language-Augmented Thought[END_REF][START_REF] Gentner | Analogy and Abstraction[END_REF], linguists [START_REF] Whorf | Thought, and Reality: Selected Writings of Benjamin Lee Whorf[END_REF][START_REF] Rumelhart | Sequential Thought Processes in Pdp Models[END_REF][START_REF] Lakoff | Metaphors We Live By[END_REF] and philosophers [START_REF] Hesse | The cognitive claims of metaphor[END_REF][START_REF] Dennett | Consciousness Explained[END_REF][START_REF] Carruthers | Modularity, Language, and the Flexibility of Thought[END_REF] argued for the importance of socio-cultural interactions in the development of human intelligence (see Colas et al., 2022a, for our recent roadmap on how to introduce Vygotskian mechanisms in ai agents).

In artificial intelligence

Works in multi-agent reinforcement learning (marl) have shown that the mere presence of multiple co-existing agents can give rise to open-ended learning dynamics [START_REF] Leibo | Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research[END_REF]. In addition to self-play originally used in two-player problem settings [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF], the presence of multiple agents can give rise to an arms race, where competing agents find solutions that act as new challenges for their opponents (a phenomenon called autocurricula in multi-agent ai, [START_REF] Leibo | Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research[END_REF][START_REF] Bansal | Emergent Complexity via Multi-Agent Competition[END_REF][START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF]. Notable examples of emergent social behavior in marl include the emergence of cooperation in social dilemmas [START_REF] Leibo | Multi-agent Reinforcement Learning in Sequential Social Dilemmas[END_REF][START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF] communication systems for solving referential games or sequential marl tasks [START_REF] Lazaridou | Emergent Multi-Agent Communication in the Deep Learning Era[END_REF][START_REF] Mordatch | Emergence of Grounded Compositional Language in Multi-Agent Populations[END_REF], tool use in competing groups of drl agents playing Hide and Seek [START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF], social norms such as road traffic rules [START_REF] Pal | Emergent road rules in multi-agent driving environments[END_REF] and silly rules [START_REF] Köster | Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents[END_REF], reputation mechanisms [START_REF] Anastassacos | Cooperation and reputation dynamics with reinforcement learning[END_REF], social influence as a form of intrinsic motivation [START_REF] Jaques | Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning[END_REF]) and cultural transmission [START_REF] Team | Learning Robust Real-Time Cultural Transmission without Human Data[END_REF].

The need for an integrative framework

This short review reveals interesting relationships between existing works studying oesa in the human species and in ai, that we will further analyze below. Conceptualizing these relationships can help to initiate an interdisciplinary dialog and enable the crossfertilization and transfer of ideas between the two fields. For this aim, I propose in the next section an integrated conceptual framework modeling important components of oesa and their interactions at multiple spatiotemporal scales. I will then explain in detail their underlying dynamics.

The origins framework

Motivation: Grounding artificial intelligence in the origins of human behavior

We can extract important relationships between open-ended skill acquisition in ai and the human species from the previous section.

There is a diversity of approaches studying oesa in both ai and in the human species. I have proposed above to organize these approaches in three categories: focusing on behavioral, environmental and socio-cultural diversity. These are however just focuses, as most of these contributions actually consider the interaction between several mechanisms. For instance, both meta-rl and hbe studies how environmental diversity can encourage behavioral diversity by improving the generalization ability of acquired behaviors to novel environments.

Intrinsic motivation is considered as a key mechanism of oesa in both ai and in the human species (Section 3.2.1). However, the evolutionary basis of intrinsic motivation is overlooked in both fields. Interestingly, most contributions studying this questions come from the ai community. [START_REF] Barto | Intrinsic Motivation and Reinforcement Learning[END_REF] proposes that the evolutionary origins of intrinsic motivation can be explained because it maximizes long-term evolutionary fitness under changing environmental conditions, "promot[ing] behavior that is ubiquitously useful across many different environments" [START_REF] Barto | Intrinsic Motivation and Reinforcement Learning[END_REF]). This hypothesis is supported by computational rl experiments in [START_REF] Singh | Intrinsically motivated reinforcement learning: An evolutionary perspective[END_REF]. With [START_REF] Ten | Curiosity-driven exploration: Diversity of mechanisms and functions[END_REF], we build upon this framework to extract key functional aspects of intrinsic motivation in both biological and artificial agents: procurement of extrinsic primary rewards, learning internal models, goal discovery and cultural innovation.

Adaptation occurs at different timescales, each one employing a mechanism of distinct nature. On the evolutionary time scale, populations of agents adapt through natural selection [START_REF] Darwin | On the Origin of Species by Means of Natural Selection or the Natural Selection of Favoured Races in the Struggle for Life[END_REF], with periods of environmental instability resulting in speciation and extinction events [START_REF] Potts | Hominin evolution in settings of strong environmental variability[END_REF]. Within their lifetime, individuals employ learning mechanisms of evolutionary origins hypothesized to be favored when the environment is unpredictable between generations but predictable enough within a lifetime for exploratory behavior to offer an evolutionary advantage [START_REF] Niv | Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors[END_REF][START_REF] Stephens | Change, regularity, and value in the evolution of animal learning[END_REF][START_REF] Johnston | Selective Costs and Benefits in the Evolution of Learning[END_REF] . In the ai community, learning initially acquired a broad, mechanistic definition that encompassed all processes with which an agent improves its performance on a given task based on its experience [START_REF] Mitchell | Machine Learning[END_REF]. However, the polyphony of mechanisms employed to achieve learning in the sub-fields of supervised, unsupervised and reinforcement learning, suggests that learning mechanisms are associated with opportunities and compromises. The recent introduction of meta-rl resonated with the needs of the community for agents with stronger generalization abilities (Vanschoren, 2018). Meta-rl is achieved through a bi-level optimization process: in the inner loop, a standard learning algorithm is optimized on a given task, while the outer tunes the algorithm in the inner loop for improving its performance on a wide distribution of tasks. This resembles the nested adaptive mechanisms of evolution (outer loop) and development (inner loop) in biological organisms. The end result of this nested training process is an algorithm that learns how to learn, i.e. learns how to solve unseen tasks faster.

It is recognized that environmental variability promotes adaptability in both ai and in the human species. Contributions in meta-rl study how training an agent on a distribution of environments can improve its generalization abilities on unseen test environments, while curriculum learning instead focus on the order in which tasks are presented to the agent (Section 3.2.2.2). Under the pcv framework discussed in Section 3.2.2.1, climate variability has served as a drive for the ability of humans to adapt to complex and rapidly changing environments, with a particular focus on the alternation between stable periods with either low or high resource availability and periods with more chaotic climate variability. Adaptation is achieved through mechanisms whose form partially depends on properties of the environment. If the environment is constant across time and space, natural selection may favor innate behaviors. By contrast, if the environment varies, natural selection might favor behavioral plasticity [START_REF] Stephens | Change, regularity, and value in the evolution of animal learning[END_REF][START_REF] Johnston | Selective Costs and Benefits in the Evolution of Learning[END_REF][START_REF] Hougen | The Evolution of Reinforcement Learning[END_REF][START_REF] Eskridge | Nurturing promotes the evolution of learning in uncertain environments[END_REF][START_REF] Frankenhuis | Enriching behavioral ecology with reinforcement learning methods[END_REF].

Socio-cultural dynamics is an important mechanism driving oesa. Many theories on the evolution and acquisition of complex social behavior in the human species -including language-emphasize a strong role of the pre-existing ecological and social conditions in the environment of our distant ancestors (Section 3.2.3.1). These conditions shape the cooperative and competitive pressures driving the acquisition of complex social behavior, which in turn modulate those pressures. These arguments resonate with results in multi-agent learning from the ai community -in particular recently in marl-where properties of the physical and social environment are the main driver for the joint learning of cooperative, communicative and normative social behavior (Section 3.2.3.2).

Niche construction is a neglected process in both fields, and especially in ai . Niche construction has been in the past termed as the neglected process in evolution [START_REF] Odling-Smee | Niche Construction: The Neglected Process in Evolution[END_REF]. It is however now admitted that agents modify their own environments and therefore their own evolutionary fitness, which are then inherited by their descendants and can have a significant impact on the course of evolution (Section 3.2.2.1). Niche construction increases the control that agents have over their environments, examples being tool use, agriculture and nest building. Whereas niche construction is recognized as a key driving force of evolution, in particular in the human species [START_REF] Kendal | Human niche construction in interdisciplinary focus[END_REF], it is still widely ignored in ai research (see however Clune, 2020, for recent propositions).

This analysis shows that open-ended skill acquisition is not a mere property of some advanced cognitive architectures. Instead, I propose in this thesis to conceive oesa as an emergent property resulting from the coupling of heterogeneous mechanisms operating at multiple spatiotemporal scales, namely environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms.

Overview of the proposed framework

Components

Informed by the analysis of the relationships between artificial and human oesa proposed above, we can extract key components driving open-ended skill acquisition in biological and artificial agents. These components are represented as boxes in Figure 3.2 and are explained below. In the text, I will refer to them in bold, e.g. environmental complexity. Each component can also be referred in abbreviated form indicated in parentheses in each paragraph below (e.g. ec for environmental complexity). Each component contains items, that I indicate in italics in the text, e.g. resource availability is an item of the environmental complexity component (see Figure 3.2).

Environmental complexity (ec). The environment exhibits multiscale and compositional dynamics (Section 3.2.2). Multiscale dynamics refers to alternating cycles at various spatiotemporal scales (day/night, seasonal cycles, climate variations). These cycles modulate constraints and opportunities, e.g. in terms of the resources available to the agents, as well as other environmental properties affecting their fitness (e.g. exposition to predator ). Compositional dynamics (Section 3.2.2.2) refers to the possibility to produce new elements in the environment, or to change the properties of existing ones, by composing other elements (offering opportunities for e.g. tool crafting, see cultural repertoire below). The environment contains other agents with possibly converging or conflicting interests (see multi-agent dynamics below).

Adaptability (a). The agents adapt to their environment on two main scales: evolutionary and developmental. At the evolutionary scale, natural selection shapes the morphological, physiological, sensorimotor and cognitive apparatuses from generations to generations according to environmental selective pressures, driving speciation and extinction events (Section 3.2.2.1). At the developmental scale, morphogenesis shapes the organism and its interaction with the environment. This results in built-in self-regulatory sensorimotor loops for anticipating needs and preparing to satisfy them before they arise (e.g. allostasis Sterling, 2012), acting as behavioral priors bootstrapping higher-level cognitive mechanisms [START_REF] Verschure | The why, what, where, when and how of goal-directed choice: Neuronal and computational principles[END_REF] such as learning and exploration (Section 3.2.1). Adaptability is thus structured in two nested mechanisms: an outer evolutionary loop optimizing fitness at the generational scale by selecting for morphological, sensorimotor and cognitive priors shaping the dynamics of an inner developmental loop at the individual scale.

Multi-agent dynamics (md). Multiple agents interact in a shared environment. Environment properties induce a game-theoretic context encouraging cooperation or competition among agent populations co-adapting at the evolutionary and developmental scales. Multi-agent interactions are shaped by this context, from simple signalling or manipulation to the self-organization of more complex social behavior (Section 3.2.3, see also cultural repertoire below). The game-theoretic context is however perturbed by environmental multiscale dynamics (see environmental complexity), driving the agents to continuously readapt to changing conditions.

Cultural repertoire (cr). Multi-agent dynamics in populations of adaptive agents can potentially lead to the formation of a cultural repertoire (Section 3.2.3) through pressures and opportunities to acquire more complex skills such as technology (e.g. tool use), communication (e.g. language) and social organization (e.g. social norms). Although there is no strong consensus on the definition of culture, a standard one refers to a behavior that is transmitted repeatedly through social or observational learning to become a populationlevel characteristic [START_REF] Gruber | The knowns and unknowns of chimpanzee culture[END_REF]. The behaviors acquired in a generation influence the experience of the next one through cultural transmission (i.e. the process by which cultural behaviors are transmitted), potentially bootstrapping cultural evolution (i.e. the cumulative change of culture over time, [START_REF] Richerson | Not by Genes Alone: How Culture Transformed Human Evolution[END_REF].

Environmental complexity (EC)

• Multiscale dynamics

• Day/night • Seasons • Climate variations

• Constraints and opportunities

• Resource availability • Exposition to predators • Compositional dynamics

Cultural repertoire (CR)

• Technology

• Tool use • Engineering

• Communication

• Language • Arts

• Social organization

• Norms and values • Institutions

Multi-agent dynamics (MD)

• Interactions

• Signaling • Manipulation • Game-theoretic context • Cooperative • Competitive • Social dilemmas Adaptability (A)
• Evolutionary scale

• Natural selection • Speciation • Extinction • Developmental scale • Self-regulatioin • Learning • Exploration

Interactions between the components

In Figure 3.2, we introduce a conceptual framework, called origins, that recognizes important ecological components as well as the feedforward and feedback links driving the whole system dynamics. I provide below a summarized overview of the main aspects of this dynamics. In the next sections, I will explain in more detail the different subparts of the diagram. As mentioned above, I indicate the framework components (i.e. the boxes in Figure 3.2) in bold, e.g. environmental complexity. Links between the framework components (i.e. arrows in the diagram of Figure 3.2) are referred as x→y, where x is the source component and y is the target component, in abbreviated form as indicated above. For example, ec→a indicates the link from environmental complexity (ec) to adaptability (a). Finally, I indicate specific items of a framework component in italics, e.g. multiscale dynamics is an item of the environmental complexity component.

Under the proposed framework, environmental complexity is characterized by multi-scale dynamics, which is driven by daily, seasonal and climate variations (Section 3.2.2.1). It modulates constraints and opportunities available to the agents, in particular through changes in resource availability, exposition to predators and the items available for compositional dynamics (Section 3.2.2). This complexity has a strong influence on two major phenomena. First, it drives adaptability (ec→a) both at the evolutionary scale, through natural selection, speciation and extinction, and at the developmental scale by shaping cognitive mechanisms for learning and exploration (Section 3.3.1). Extending existing theories in hbe and informed by recent advances in ai presented in previous sections, a key hypothesis of the origins framework is that there exist certain levels of environmental complexity able to drive the evolution of autotelic exploration (Section 3.2.1.2), i.e. of an intrinsic motivation to discover new goals and learn how to achieve them (see Section 3.2.1 above about intrinsic motivation, as well as Section 3.4 below for more detail). Second, varying the levels of resource availability and exposition to predators has a strong influence on multi-agent dynamics (ec→md) through the modulation of the game-theoretic context, i.e. cooperation and competition pressures (Section 3.2.3).

The influence of environmental complexity on adaptability and multi-agent dynamics can then have feedback and feedforward effects on the whole system dynamics. Firstly, increased morphological and cognitive complexity resulting from adaptability, as well as increased complexity in the multi-agent dynamics, feed back to environmental complexity (a→ec and md→ec) through mechanisms of niche construction modifying resource availability and predation pressure (Section 3.2.2). Competition among different groups or species is also considered as an important driver of adaptation, possibly resulting in an arms race between co-adapting species (a→md and md→a, see also Section 3.2.3). Secondly, evolved exploration strategies in adaptability and complex multi-agent dynamics can bootstrap in a feedforward manner the formation of a cultural repertoire (a→cr and md→cr), i.e. of socially transmitted skills. These skills, first of relatively low complexity (e.g. tool use), then evolve into increasingly complex technology (e.g. engineering), communication (e.g. language) and social organization (e.g. social norms and values and institutions) (Section 3.2.3). Such cultural evolution is made possible through feedback effects from the cultural repertoire to all other components of the system, e.g. technology modifying the structure of the environment (cr→ec), using language as a cognitive tool (cr→a, coresponding to the social conception of intelligence mentioned in Section 3.2.3.1), or social organization promoting cooperation or competition (cr→md). Cultural evolution thus induces a positive feedback loop driving the ever-expanding complexity of the human cultural repertoire (see Section 3.2.2 above about niche construction, as well as Section 3.6 below for more detail).

I believe this framework to be of interest for three main reasons. First, it highlights the links between two different domains that usually do not interact together: namely artificial intelligence and human behavioral ecology. Second, it can help to understand how processes operating at multiple spatiotemporal scales can support open-ended skill acquisition (both its origins expressed as feedforward links, and its open-ended aspect expressed as feedback links). Third, it provides a tentative roadmap for achieving human-like open-ended skill acquisition in artificial systems, grounded in theories and hypotheses from hbe.

As an interesting example of how the origins framework is able to express complex mechanisms driving oesa, we can analyze how it is able to represent each type of niche construction mechanism presented in Section 3.2.2.1. In the proposed framework, niche construction corresponds to feedback effects from any component of the system to environmental complexity (i.e. a→ec, md→ec and cr→ec). a→ec corresponds to the original concept of ecological niche construction as well as to cognitive niche construction where adaptive agents modify the properties of their environment, potentially modifying their own fitness landscape at the evolutionary scale or aiding their own behavior at the developmental scale. md→ec corresponds to social niche construction, where multi-agent dynamics can modulate resource availability and exposition to predators in the environment. As mentioned by [START_REF] Saltz | What, if anything, is a social niche?[END_REF]: "Simply joining a group increases its size, illustrating how individuals are closely intertwined with their social environments". In turn, increasing group size reduces both resource availability and exposition to predators [START_REF] Brown | Optimal group size in territorial animals[END_REF]. Finally, cr→ec corresponds to cultural niche construction, i.e. how technology, communication and social organization modify the structure of the environment and the fitness landscape of new generations. Cultural niche construction is considered as a central mechanism driving human-like oesa [START_REF] Laland | Cultural niche construction and human evolution[END_REF][START_REF] Laland | Cultural Niche Construction: An Introduction[END_REF][START_REF] Fogarty | The niche construction of cultural complexity: Interactions between innovations, population size and the environment[END_REF][START_REF] Iriki | Triadic (ecological, neural, cognitive) niche construction: A scenario of human brain evolution extrapolating tool use and language from the control of reaching actions[END_REF].

The proposed framework thus highlights the importance of niche construction mechanisms, as feedback effects on environmental complexity, in human-like open-ended skill acquisition (oesa). By inducing positive feedback loops that continuously increase environmental complexity, which in turn encourage the acquisition of increasingly complex skills, these different types of niche construction provide the driving force that makes human skill acquisition truly open-ended. I believe these mechanisms to be of central importance in order to achieve oesa in artificial systems.

The next three sections explain in more detail the underlying dynamics of subparts of the origins framework and how it relates to specific hypotheses in both hbe and ai. I propose to distinguish three levels of dynamics (Figure 3.3).

• Level-1 dynamics (Section 3.4) focuses on the interaction between environmental complexity, adaptability and multi-agent dynamics and is concerned with the research question:

-What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies?

• Level-2 dynamics (Section 3.5) focuses on the feedforward effects from environmental complexity, adaptability and multi-agent dynamics to the cultural repertoire and is concerned with the research question:

-What is the joint role of intrinsically motivated goal exploration and multiagent dynamics in the formation of a cultural repertoire?

• Level-3 dynamics (Section 3.6) focuses on feedback effects from the cultural repertoire to all the other components of the system and is concerned with the research question:

-What is the role of feedback effects from a cultural repertoire in the openended dynamics of human skill acquisition?

Environmental complexity (EC)

• Multiscale dynamics

• Day/night • Seasons • Climate variations

• Constraints and opportunities

• Resource availability • Exposition to predators • Compositional dynamics

Cultural repertoire (CR)

• Technology

• Tool use • Engineering

• Communication

• Language • Arts

• Social organization

• Norms and values • Institutions

Multi-agent dynamics (MD)

• Interactions

• Signaling • Manipulation • Game-theoretic context • Cooperative • Competitive • Social dilemmas

Adaptability (A)

• Evolutionary scale

• Natural selection • Speciation • Extinction • Developmental scale • Self-regulatioin • Learning • Exploration
(a) Level-1 dynamics: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies? .

Environmental complexity (EC)

• Multiscale dynamics

• Day/night • Seasons • Climate variations

• Constraints and opportunities

• Resource availability • Exposition to predators • Compositional dynamics

Cultural repertoire (CR)

• Technology

• Tool use • Engineering

• Communication

• Language • Arts

• Social organization

• Norms and values • Institutions

Multi-agent dynamics (MD)

• Interactions

• Signaling • Manipulation • Game-theoretic context • Cooperative • Competitive • Social dilemmas

Adaptability (A)

• Evolutionary scale

• Natural selection • Speciation • Extinction • Developmental scale • Self-regulatioin • Learning • Exploration (b) Level-2 dynamics:
What is the joint role of intrinsically motivated goal exploration and multiagent dynamics in the formation of a cultural repertoire? .

Environmental complexity (EC)

• Multiscale dynamics

• Day/night • Seasons • Climate variations

• Constraints and opportunities

• Resource availability • Exposition to predators • Compositional dynamics

Cultural repertoire (CR)

• Technology

• Tool use • Engineering

• Communication

• Language • Arts

• Social organization

• Norms and values • Institutions

Multi-agent dynamics (MD)

• Interactions

• Signaling • Manipulation • Game-theoretic context • Cooperative • Competitive • Social dilemmas

Adaptability (A)

• Evolutionary scale

• Natural selection • Speciation • Extinction • Developmental scale • Self-regulatioin • Learning • Exploration (c) Level-

Level-1 dynamics: the eco-evolutionary origins of autotelic agents

Level-1 dynamics concerns the interaction between environmental complexity, adaptability and multi-agent dynamics. These three components of the framework reflect the categorization I proposed in Section 3.2 for the existing approaches to openended skill acquisition (oesa), focusing on environmental, behavioral and social diversity, respectively. However, the proposed framework emphasizes the emergent effects resulting from the interaction between these three components (i.e. ec→a, ec→md, md→a, a→md, a→ec and md→ec).

Interactions between environmental complexity and adaptability (ec→a and a→ec)

Under the Pulsed Climate Variability (pcv) framework mentioned in Section 3.2.2, environmental factors such as climate variability, resource availability and exposition to predators have served as a drive for the ability of humans to adapt to complex environments:

A new consensus is emerging that suggests the unusual geology and climate of East Africa created a complex, environmentally very variable setting. This new understanding of East African climate has led to the pulsed climate variability hypothesis that suggests the long-term drying trend in East Africa was punctuated by episodes of short alternating periods of extreme humidity and aridity which may have driven hominin speciation, encephalization and dispersals out of Africa (Maslin et al. ( 2015))

The role of such complex environmental dynamics is also suggested in [START_REF] Potts | Hominin evolution in settings of strong environmental variability[END_REF], who presents a synthesis of paleoclimate and human evolution data over the past 7 million years suggesting that the adaptations that unpin much of human environmental interactions reflect past evolutionary responses to ecological instability and environmental novelty.

Adaptation to environmental complexity operates on two scales: the evolutionary one drives natural selection, speciation and extinction according to the environment conditions, shaping the developmental one which drives learning and exploration to adapt to those conditions. For example, during dry periods with strong limitations on resource availability, specialist species would struggle having lost their environmental niche and their competitive advantage [START_REF] Brown | Evolutionary accounts of human behavioural diversity[END_REF] while generalist species, such as those having evolved more advanced exploration strategies, would as a consequence increase their relative fitness.

In ai, most contributions consist in the proposition of adaptive decision-making algorithms (adaptability) allowing sensorimotor agents to learn how to optimally behave in an environment with a predefined dynamics. In hbe, however, the interaction between environmental complexity and adaptability is recognized to also operate in the reverse direction (i.e. a→ec) through the process of ecological niche construction (see Sections 3.2.2 and 3.3). Although recent theoretical propositions in ai have identified feedback effects from an agent behavior to its environment as an important mechanism [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF][START_REF] Stanley | Why Greatness Cannot Be Planned: The Myth of the Objective[END_REF], computational contributions of ecological niche construction mechanisms in ai are mostly nonexistent. Notable exceptions can be found in the study of non-episodic reinforcement learning (rl) [START_REF] Co-Reyes | Ecological Reinforcement Learning[END_REF] and in common pool resource appropriation in multi-agent reinforcement learning [START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF]) but these contributions do not interpret their results in the light of niche construction theories.

In fact, while the rl approach has recently made impressive progress, in particular due to the rise of deep reinforcement learning (drl, [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], it only covers a very limited part of level-1 dynamics in the proposed framework (i.e. mostly ec→a). There are three main limitations in the rl approach.

First, the dynamics of the environment is mostly influenced by the agent's actions. This comes in contrast to the theories in hbe described above, where environments are supposed to display rich intrinsic dynamics independently of the presence of agents (e.g. complex African paleoclimate was mostly independent of human activity). As seen above, it is this rich environmental dynamics which is supposed to have driven the emergence of complex adaptive mechanisms in humans and in particular advanced exploration strategies such as intrinsically motivated goal exploration.

Second, the standard training paradigm in rl is episodic, i.e. the environment is regularly reset to its initial conditions. While this procedure has the benefit to facilitate training from a machine learning perspective [START_REF] Pardo | Time Limits in Reinforcement Learning[END_REF], it strongly differs from natural settings where environments are persistent, i.e. where the behavior of agents affects the environment in which the next generations will further evolve and learn (see however Co-Reyes et al., 2020, for a recent attempt at studying ecologically valid non-episodic rl settings). Episodic training in rl prevents the study of both niche construction and eco-evolutionary feedback effects, which require that populations alter their environment and that those changes in the environment influence the subsequent evolution of the population [START_REF] Post | Eco-evolutionary feedbacks in community and ecosystem ecology: Interactions between the ecological theatre and the evolutionary play[END_REF]. The field of automatic curriculum learning [START_REF] Portelas | Automatic curriculum learning for deep RL: A short survey[END_REF] studies such feedback effects but focuses on how to adaptively sample novel environments of increasing complexity, using episodic training, with the explicit objective to improve an agent's learning performance.

Third, the rl paradigm traditionally assumes that rewards are provided by the environment. From a biological perspective, however, the environment does not contain any reward whatsoever. Rewards instead result from the agent's own physiology and selfregulation and have emerged from evolution as a way to guide learning and exploration (see e.g. the metaphor of evolved stick/carrot mechanisms in Sterling, 2012). Despite these major limitations of the rl paradigm, ai approaches focusing on environmental diversity described in Section 3.2.2.2 (including emergentist approaches described in Section 3.1.1) are relevant to better model the whole complexity of level-1 dynamics. Meta reinforcement learning (Meta-rl) approaches have shown how an outer optimization loop, resembling an evolutionary mechanism, can meta-learn learning algorithms able to adapt to a distribution of diverse environments. Within the meta-rl framework, few contributions have shown how intrinsic rewards can emerge as a way to quickly adapt to changing environments [START_REF] Singh | Intrinsically motivated reinforcement learning: An evolutionary perspective[END_REF][START_REF] Alet | Meta-learning curiosity algorithms[END_REF]. The evolution of autotelic exploration has however, to my knowledge, not yet been studied. In addition, curriculum learning methods have shown how the learning of complex behavior can be facilitated by exposing the agent to an adequate sequence of training environments with increasing difficulty. In both approaches, however, diverse environments are presented to the agent through an external mechanism (sampling from a distribution in meta-rl or as a sequence in curriculum learning), with weak feedback effects from adaptability to environmental complexity (see however automatic curriculum learning as in Wang et al., 2019b, studying interesting feedback effects that are nevertheless not ecologically valid). This is radically different from a natural mechanism, where species explore diverse ecological niches with complex multi-scale dynamics which is intrinsic to a persistent environment and with recognized feedback effects from adaptability to environmental complexity shaping evolutionary mechanisms.

Interactions with multi-agent dynamics (ec→md, md→a, a→md, md→ec)

Another important aspect of level-1 dynamics is the presence of multiple agents that jointly adapt in a shared environment, i.e. multi-agent dynamics. The interactions between adaptability and multi-agent dynamics (md→a and a→md) depend on the game-theoretic structure of the environment: either cooperative, competitive or mixed (e.g. social dilemma, [START_REF] Leibo | Multi-agent Reinforcement Learning in Sequential Social Dilemmas[END_REF][START_REF] Lanctot | A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning[END_REF]. For example, as seen in Section 3.2.3, cooperative environments can favor the learning of a shared communication system as a solution to solve complex cooperative tasks (as studied in emergent communication, a subfield of multi-agent ai). Mixed game-theoretic structures such as social dilemmas can lead to complex learning dynamics where agents first acquire individual skills for their own benefit, then jointly converge toward (potentially costly) cooperation strategies for their common good [START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF].

In addition, co-adaptation increases environmental complexity by influencing variations in resource availability and exposition to predators (a→ec and md→ec) that in turn modulates cooperation and competition pressures in agent populations (ec→md). In competitive environments, this feedback loop results in complex co-adaptation mechanisms potentially leading to arms race dynamics where agents continuously adapt new behaviors and morphologies in reaction to the adaptation of other agents. It has been shown that these co-adaption mechanisms can result in open-ended dynamics in both ecology (where this mechanism is called and arms race, [START_REF] Van Valen | A new evolutionary law[END_REF][START_REF] Pearson | Red queen hypothesis[END_REF] and in ai (where it is called multi-agent autocurricula, [START_REF] Leibo | Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research[END_REF][START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF]).

Another important aspect concerns the role of environmental complexity in shaping the topology of social networks (ec→md) and the role of this topology on adaptability (md→a). Complex environmental dynamics results in the appearance and disappearance of ecological barriers inducing spatial structures producing population isolation and vicariance [START_REF] Shultz | Early Human Speciation, Brain Expansion and Dispersal Influenced by African Climate Pulses[END_REF]Larrasoaña, 2012) (ec→md). Human studies in the lab have shown how such a dynamic social network topology, alternating between periods of relative isolation between populations with periods of denser connectivity, can improve collective innovation in complex crafting tasks [START_REF] Derex | Partial connectivity increases cultural accumulation within groups[END_REF] (md→a, see also Sections 6.4.2.2 and 7.3 for our propositions on this topic).

Summary of level-1 dynamics

The main research question related to level-1 dynamics is: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies? The notion of autotelic agents refers to agents that autonomously learn new skills by generating and pursuing their own goals (Section 3.2.1.2). In the proposed framework I hypothesize that such exploration strategies evolve from certain conditions on the dynamics of environmental complexity. This claim is supported by both recent theories in hbe based on paleoclimatology data as well as computational experiments in ai. In hbe, chaotic paleoclimate dynamics on the African continent is hypothesized to have driven the evolution of humans as a generalist species able to quickly adapt to abrupt environmental variations (Section 3.4.1). This claim mirrors computational experiments in ai, in particular in meta-rl, showing how intrinsic motivation mechanisms can emerge as a solution to adapt to changing environments [START_REF] Hougen | The Evolution of Reinforcement Learning[END_REF][START_REF] Singh | Intrinsically motivated reinforcement learning: An evolutionary perspective[END_REF][START_REF] Alet | Meta-learning curiosity algorithms[END_REF]. In addition, environmental complexity influences multi-agent dynamics through the modulation of cooperation and competition pressures, potentially leading to positive feedback loops such as arms race, as well as dynamic social network topologies favoring innovation (Section 3.4.2). The resulting complex multi-agent dynamics increases environmental complexity and further drives the evolution of complex exploration strategies (adaptability) as a way to explore and construct new niches.

In Chapter 5 I will present my existing contributions on this topic as well as future research directions for addressing the main research question of level-1 dynamics: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies?

Level-2 dynamics: the formation of a cultural repertoire

As seen above, level-1 dynamics results in populations of autotelic agents having evolved an intrinsic motivation to constantly explore new goals and learn how to achieve them, as a solution to quickly adapt to rapidly changing environments. Whereas they originally emerged as a way to prepare agents to abrupt environmental changes, this intrinsic motivation has the side effect of also generating skills that have no direct relationship to evolutionary fitness but could, later on, be recruited by the evolutionary process [START_REF] Oudeyer | How Evolution May Work Through Curiosity-Driven Developmental Process[END_REF]. These skills depend on the opportunities offered by the environment and are transmitted from generation to generation to form a cultural repertoire (ec→cr, a→cr and md→cr).

Studying a cultural repertoire can be divided in three subproblems [START_REF] Oudeyer | Self-Organization in the Evolution of Speech[END_REF]Moulin-Frier & Oudeyer, 2021). Firstly, the study of the forms of a cultural repertoire: in the case of language for instance, this corresponds to studying the phonemic, semantic, syntactic or pragmatic structures constituting it. Secondly, the study of the formation of a cultural repertoire, i.e. of the genesis of its forms through sensorimotor, cognitive, environmental, social, cultural or evolutionary mechanisms. Thirdly, the study of its origins, i.e. of the biological and environmental conditions that could have bootstrapped the formation process. Due to multi-agent dynamics, culturally transmitted skills are strongly shaped by social interactions (Section 3.2.3). While non-human species often exhibit impressive behavioral repertoires [START_REF] Krams | Linking social complexity and vocal complexity: A parid perspective[END_REF], human ecology is however characterized by a uniquely large behavioral repertoire involving complex technology (e.g. engineering), communication (e.g. language) and social organization (e.g. institutions). This complex cultural ecosystem has led scientists in the search for factors that differentiated us from other species [START_REF] Tomasello | Two Key Steps in the Evolution of Human Cooperation[END_REF][START_REF] Tomasello | Becoming Human: A Theory of Ontogeny[END_REF][START_REF] Botero | The ecology of religious beliefs[END_REF]. According to the inter-dependence hypothesis, social norms and institutions emerged to counteract the fact that reputation alone could no longer alleviate the problem of free riding in large groups. In addition, the social complexity hypothesis [START_REF] Krams | Linking social complexity and vocal complexity: A parid perspective[END_REF] states that language worked as a bonding mechanism that replaced grooming, practiced in small-scale societies, and thus helped with maintaining group stability in larger groups [START_REF] Dunbar | Coevolution of neocortical size, group size and language in humans[END_REF]. Tool use has also been investigated under a number of, often contesting, hypotheses. Based on the data analysis in [START_REF] Fogarty | The niche construction of cultural complexity: Interactions between innovations, population size and the environment[END_REF], environmental variability such as the risk of resource failure, mobility and climate characteristics correlate significantly with tool use in food-gathering societies. However, it is the group size and not these factors that affect tool use in food-producing societies. It is therefore conjectured that the feedback link of societies with a larger cultural repertoire has a stabilizing effect, dampening the forward impact of environmental variability (Collard et al., 2011).

Contributions in multi-agent ai, and notably in multi-agent reinforcement learning (marl), have also studied the formation of a cultural repertoire in populations of adaptive agents from a computational perspective. As mentioned in Section 3.2.3.2, this includes the emergence of cooperation in social dilemmas [START_REF] Leibo | Multi-agent Reinforcement Learning in Sequential Social Dilemmas[END_REF][START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF] communication systems for solving referential games or sequential marl tasks [START_REF] Lazaridou | Emergent Multi-Agent Communication in the Deep Learning Era[END_REF][START_REF] Mordatch | Emergence of Grounded Compositional Language in Multi-Agent Populations[END_REF], tool use in competing groups of drl agents playing Hide and Seek [START_REF] Baker | Emergent Tool Use From Multi-Agent Autocurricula[END_REF], social norms such as road traffic rules [START_REF] Pal | Emergent road rules in multi-agent driving environments[END_REF] and silly rules [START_REF] Köster | Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents[END_REF], reputation mechanisms [START_REF] Anastassacos | Cooperation and reputation dynamics with reinforcement learning[END_REF], social influence as a form of intrinsic motivation [START_REF] Jaques | Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning[END_REF]) and cultural transmission [START_REF] Team | Learning Robust Real-Time Cultural Transmission without Human Data[END_REF].

In Chapter 6 I will present my existing contributions on this topic as well as future research directions for addressing the main research question of level-2 dynamics: What is the joint role of intrinsically motivated goal exploration and multi-agent dynamics in the formation of a cultural repertoire?

Level-3 dynamics: towards human-like open-ended skill acquisition through cultural feedback effects

Level-1 and level-2 dynamics results in agent populations forming a repertoires of collective skills with no necessary direct link to evolutionary fitness, called cultural repertoire in the origins framework. Level-3 dynamics focuses on the feedback effects from the cultural repertoire to all the other components of the system (i.e. cr→ec, cr→a and cr→md), driving the open-ended dynamics of human skill acquisition.

While a large part of contributions in modern ai study how skills emerge from environmental pressures, there are surprisingly few works that study how skills affect the environment. This comes in contrast to the increasing importance of niche construction theories in ecology (Section 3.2.2 and 3.3). Adaptive agents change their environments, which are also inherited by their descendants and can have a significant impact on the course of evolution, a key mechanism which is widely ignored in ai research (see Section 3.3.2 on how the different types of niche construction theories are expressed in the origins framework, in particular cultural niche construction, i.e. cr→ec).

An example of cultural niche construction concerns the potential role of fire control in language evolution. Some contributions studying the discovery of fires by humans propose that having to maintain a fire during the night considerably increased the waking time (about 16 hours in humans, compared to 8 hours in other mammals, [START_REF] Gowlett | The discovery of fire by humans: A long and convoluted process[END_REF]. This "extended day time" could have resulted in a novel niche without strong functional pressures, where the campfire protected the group from predators and the absence of natural light prevented any hunter-gatherer activity (cr→ec). This could have played an important role in language evolution, dynamically modulating the structure of our social networks (cr→md) by providing free time at night that could be dedicated to communication [START_REF] Wiessner | Embers of society: Firelight talk among the Ju/'hoansi Bushmen[END_REF][START_REF] Dunbar | How conversations around campfires came to be[END_REF].

Another important feedback effect highlighted by the proposed framework is from the cultural repertoire to adaptability (cr→a). In a recent position paper (Colas et al., 2022a), we propose a roadmap towards Vygotskian autotelic agents able to interact with others and, more importantly, able to internalize socio-cultural interactions to transform them into cognitive tools supporting the development of new cognitive functions. This social conception of intelligence is grounded in the seminal work of the developmentalist Led Vygotsky, as described in Section 3.2.3.1. In Section 7.2, I will present a computational architecture of a Vygotskian autotelic agent showing how the compositionality of language can be leveraged to imagine creative goals and foster exploration.

In Chapter 7 I will present my existing contributions on this topic as well as future research directions for addressing the main research question of level-3 dynamics: What is the role of feedback effects from a cultural repertoire in the open-ended dynamics of human skill acquisition?

General conclusions and discussion

In this chapter I have proposed an integrated conceptual framework, called origins, highlighting the main mechanisms driving open-ended skill acquisition (oesa) in the human species and in artificial intelligence (ai). The framework is grounded in hypotheses and theories from human behavioral ecology (hbe) and motivated by the observation of strong relationships between hbe and experimental results in ai (Section 3.3.1). In this sense, the framework can be considered as an invitation to dialog between researchers in hbe (and more generally in life science, including evolutionary biology, developmental psychology and niche construction theories) and ai.

Main hypotheses. The proposed framework identifies key components driving oesa, namely environmental complexity, adaptability, multi-agent dynamics and cultural repertoire (Section 3.3.2.1). It describes the complex interactions between these components in terms of feedforward and feedback effects at different levels (Section 3.3.2.2).

Level-1 dynamics describes the interaction between environmental complexity, adaptability and multi-agent dynamics (Section 3.4). The main hypothesis at this level is that environmental complexity is the main driver of behavioral diversity, as proposed by recent theories in hbe (such as the Pulsed Climate Variability framework) as well ai experiments in meta reinforcement learning and curriculum learning. Feedback effects of co-adaptation also play an important role in behavioral diversity at this level, in line with the concepts of arms race dynamics in behavioral ecology (Red Queen hypothesis) and of multi-agent autocurriculum in ai. Environmental complexity and arms race dynamics both contribute to the evolution of autotelic agents equipped with an intrinsic motivation to discover and explore new niches and skills as a solution to adapt to abrupt environmental changes. Level-2 dynamics describes the feedforward effects from environmental complexity, adaptability and multi-agent dynamics to the formation of a cultural repertoire (Section 3.5). Intrinsically motivated goal exploration coupled with varying cooperation and competition pressures, emerged from level-1 dynamics, result in the acquisition of complex skills that are not necessarily linked to evolutionary fitness. These contribute to the formation of a cultural repertoire through level-2 dynamics, i.e. an expanding collection of behaviors that are transmitted repeatedly through social or observational learning to become a population-level characteristic. These skills include technology, communication and social organization, originally of relatively low complexity. Finally, level-3 dynamics describes how a cultural repertoire influences all the other components of the system, resulting in increased complexity of the environment, of the agents' cognitive abilities and of their multi-agent interactions. This results in positive feedback loops continuously feeding the dynamics of the three proposed levels. The main hypothesis here is that these positive feedback loops, which are especially prominent in the human species due to the complexity of our cultural repertoire, are what make human skill acquisition truly open-ended.

The origins framework as a roadmap. Thus, the framework can be considered as a roadmap for future research in ai. The underlying methodology requires ai researchers to dive into the hbe literature and to engage in an interdisciplinary dialog with this field. In ai, the objective is to extract important hypotheses and mechanisms that can potentially advance the state of the art in artificial oesa and be tested in computational experiments. This effort can in turn benefit research in hbe by proposing computational tools to test their hypotheses in simulated environments (as already proposed to some extent in [START_REF] Frankenhuis | Enriching behavioral ecology with reinforcement learning methods[END_REF]. In Part II I will present my existing contributions and will show how the origins framework allows the proposition of concrete and original future experiments, at each of the three levels.

Limitations. As with any conceptual model, the framework relies on several simplifications. In particular, each component contains different mechanisms operating at several spatiotemporal scales. Environmental complexity involves multiscale dynamics at the scale of days, seasons and climate variations. Adaptability and multi-agent dynamics operate at both the developmental at evolutionary scales. Finally, while behaviors in the cultural repertoire can be formed at the level of a generation, their effects on other components of the system are supposed to span several generations. This abstraction of spatiotemporal scales in the framework can limit the representation of important interactions, e.g. between evolutionary and developmental processes. The underlying design choice is that a component of the framework represents a certain type of functional process, e.g. adaptability represents mechanisms adapting phenotypes to environmental conditions at multiple spatiotemporal scales. Distinguishing these different time scales and their interactions can however be realized when implementing parts of the framework in computational architectures, as we will see in Part II.

Related conceptual frameworks. The origins framework displays strong relationships with other theoretical frameworks in both biology and in ai. In biology, the extended evolutionary synthesis [START_REF] Pigliucci | Evolution-the extended synthesis[END_REF][START_REF] Laland | The extended evolutionary synthesis: Its structure, assumptions and predictions[END_REF] considers that, in the standard evolution theory as well as its modern synthesis, "too much causal significance is afforded to genes and selection, and not enough to the developmental processes that create novel variants, contribute to heredity, generate adaptive fit, and thereby direct the course of evolution" [START_REF] Laland | The extended evolutionary synthesis: Its structure, assumptions and predictions[END_REF]). It recognizes important feedback effects in terms of constructive development, i.e. the ability of an organism to shape its own developmental trajectory by constantly responding to, and altering, internal and external states, as well as of reciprocal causation, i.e. that developing organisms are not solely products, but are also causes, of evolution. Such feedback effects are also central in the proposed origins framework which can be considered as a call for an extended synthesis in ai focusing on open-ended skill acquisition. On the ai side, the origins framework shares several principles with the AI-GA framework [START_REF] Clune | AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence[END_REF] mentioned in Section 3.1.1, in particular regarding the the central role of bi-level optimization such as meta-learning and feedback effects from adaptability to environmental complexity. However, the AI-GA framework is entirely focused on an ai perspective and places little emphasis on biological and ecological grounding, in particular with respect to the human species. Finally, Steels (2011) also emphasizes the role of cultural evolution within constraints provided by the biology and the ecological niches in which human populations operate, recognizing strong interactions between three evolutionary processes (biological, social and cultural). Although his analysis can to some extent map to some of the components and interactions proposed in our origins framework, it is however entirely focused on language evolution.

A humanist perspective on ai. To conclude this chapter, I would like to illustrate how the origins framework can potentially help us to think about important opportunities and challenges met by our species. As mentioned in the introduction (Section 3.1.2), the framework considers open-ended skill acquisition (oesa) as the most distinctive and interesting feature in humans, in particular because it is responsible for the most beautiful and the most harmful human activities including science, arts and technological innovation, as well as the over-exploitation of natural resources leading to a major ecological issues. Interestingly we can consider science as a result of the interaction between the three main items of the cultural repertoire component in the framework: technology, communication and social organization. Indeed, modern science is made possible through a collective agreement (communication) for allocating a part of the population resource budget (social organization) to allow scientists to invest their time to explore new opportunities (e.g. technology) [START_REF] Harari | [END_REF], see also [START_REF] Graeber | The Dawn of Everything: A New History of Humanity[END_REF] for alternative theories). In turn, scientific outcomes have a strong influence on all other components of the system by increasing the understanding of our environment (e.g. physics), of our multi-agent dynamics (e.g. social science), of our adaptability (e.g. cognitive science) and of our cultural repertoire (e.g. history). The resulting knowledge and skills also inevitably increase our control over all these components, augmenting our cultural repertoire with new technologies and cultural practices (e.g. medicine or agriculture). However, perhaps ironically, this strong feedback loop is currently resulting in an unprecedented loss of control, with the over-exploitation of resources leading to a major collapse in bio-diversity and to the global warming of the Earth climate (e.g. Cook et al., 2016). It is therefore essential for our species to better understand this complex dynamics in order to (1) identify and find agreements on which outcomes of oesa we want to encourage or to discourage, (2) characterize potentially harmful feedback loops that will lead to irreversible effects and (3) take advantage of our self-determination, as autotelic agents, to shape our own cultural repertoire in a way that self-regulate the entire system and prevent it to reach critical states that could put our own survival at risk. Better understanding the dynamics of open-ended skill acquisition appears as an important research direction in this context. In this sense, the origins framework proposes a humanist perspective on ai: using it as a family of computational tools that can help us to explore and study the mechanisms driving open-ended skill acquisition in both artificial and biological systems, as a way to better understand the dynamics of our own species within its whole ecological context.

Chapter 4

Introduction

In this Part, I present some selected contributions realized during my scientific career (see Chapter 2 for a summary of this career). Two main criteria have driven the selection of these contributions: (a) their impact and (b) their relevance to the origins framework presented in Chapter 3. I have taken the occasion to re-interpret my previous contributions in the light of this framework, even though many of them were realized prior to its proposition (i.e. prior to 2021). I found this exercise interesting, as it allows to test the expressiveness of the framework (i.e. is it able to express prior contributions which were not realized with this framework in mind?).

This part is organized in three sections mirroring the three levels proposed in Sections 3.4, 3.5 and 3.6. In consequence, it does not follow a chronogical order but instead:

• Contributions related to level-1 dynamics in Chapter 5 were all realized after having proposed the first version of the origins framework in [START_REF] Nisioti | Grounding Artificial Intelligence in the Origins of Human Behavior[END_REF].

It is at this time that I became strongly interested in the main research question of level-1 dynamics: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies?

• Contributions related to level-2 dynamics in Chapter 6 were realized during the first part of my career (PhD thesis at GIPSA-Lab and post-doc at Inria-Flowers, i.e. 2007-2015). As mentioned in the description of my scientific trajectory in Chapter 2, I was mostly interested at this time in modeling the evolution and acquisition of language. In the proposed framework, this corresponds to the main research question regarding level-2 dynamics: What is the joint role of intrinsically motivated goal exploration and multi-agent dynamics in the formation of a cultural repertoire?

• Contributions related to level-3 dynamics in Chapter 7 were realized after my recruitment as a permanent researcher at Inria-Flowers in 2019. In particular, the contribution in Section 7.2 corresponds to a research project which started before my recruitment and the proposition and the origins framework, but it addresses central aspects of the main research question of level-3 dynamics: What is the role of feedback effects from a cultural repertoire in the open-ended dynamics of human skill acquisition?

Thus, the contributions related to level-1 dynamics presented in Chapter 5 are the most recent (from 2021) and, in consequence, have yet had a limited impact in the community. I have chosen to present them in this thesis because they illustrate well my most recent research interests, as well as the methodology adopted for the implementation of concrete experiments related to the origins framework.

As we will see, the presented contributions rely on a diversity of computational methods including: abstract eco-evolutionary models (Section 5.2), simulation environments (Section 5.3), artificial life (Section 5.4), multi-agent representation learning (Section 5.5), intrinsically-motivated machine learning (Section 6.2), Bayesian cognitive modeling (Section 6.3), deep reinforcement learning (7.2) and multi-agent reinforcement learning (Section 7.3).

I conclude each chapter with concrete propositions for future work.

Chapter 5

The eco-evolutionary origins of autotelic agents (Level-1 dynamics)

Introduction

In this chapter, I present some of my contributions and propose future work related to the modeling of level-1 dynamics. As presented in Section 3.4, this part of the origins framework considers pre-existing environmental complexity operating at multiple spatiotemporal scales (e.g. seasonal cycles and climate variation), modulating constraints and opportunities (e.g. in terms of resource availability and exposition to predators) exposed to the agents. The objective is to study how this environmental complexity interacts with adaptability and multi-agent dynamics toward the evolution of learning and exploration mechanisms. The main research question at this level is: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies?

In Section 5.2, I first present a recent contribution on a minimal evo-eco-devo simulation model where a population of agents adapts to environmental changes both at the developmental scale (through plasticity) and at the evolutionary scale (through fitnessbased selection and niche-limited competition). In Section 5.3, I present our first attempt at designing an ecologically valid simulation environment for reinforcement learning which is grounded in hypotheses from human behavioral ecology. Then, in Section 5.4, we study how low-level sensorimotor control can emerge from a self-organizing system where there is initially no distinction between agents and the environment. Finally, in Section 5.5, I present another recent contribution showing how multi-agent environments, where agents do not have access to the observations of others but can communicate within a limited range, guarantees a common context that can be leveraged in individual representation learning. Finally, in Section 5.6, I mention my other contributions on this topic and propose concrete future work. Open access version of this publication is available at https://arxiv.org/abs/2202. 08834. The rest of this section is a summary of this paper. Here we study the interactions between environmental complexity, adaptability and multi-agent dynamics within a minimal evo-eco-devo simulation model. We consider a population of agents that can adapt both at the developmental scale (through plasticity) and at the evolutionary scale (through fitness-based selection and niche-limited competition). Agents evolve in environments that varies from generation to generation and that are structured in multiple ecological niches with different capacities. We study how different types of environmental variability influences plasticity and evolvability under different selection mechanisms.

Relevance to the proposed origins framework

Abstract

The diversity and quality of natural systems have been a puzzle and inspiration for communities studying artificial life. It is now widely admitted that the adaptation mechanisms enabling these properties are largely influenced by the environments they inhabit. Organisms facing environmental variability have two alternative adaptation mechanisms operating at different timescales: plasticity, the ability of a phenotype to survive in diverse environments and evolvability, the ability to adapt through mutations. Although vital under environmental variability, both mechanisms are associated with fitness costs hypothesized to render them unnecessary in stable environments. In this work, we study the interplay between environmental dynamics and adaptation in a minimal model of the evolution of plasticity and evolvability. We experiment with different types of environments characterized by the presence of niches and a climate function that determines the fitness landscape. We empirically show that environmental dynamics affect plasticity and evolvability differently and that the presence of diverse ecological niches favors adaptability even in stable environments. We perform ablation studies of the selection mechanisms to separate the role of fitness-based selection and niche-limited competition. Results obtained from our minimal model allow us to propose promising research directions in the study of open-endedness in biological and artificial systems.

Methods

We model environmental variability through a climate function L (e.g. a sinusoid) from time (in number of generations) to a 1D environmental state in R (Figure 5.2, Left). We consider N niches differing from an offset in their respective climate functions. The environmental state of a niche at a given time specifies its current capacity, i.e. the maximal number of agents that can occupy it.

Each individual agent is characterized by a tolerance curve specifying its probability of surviving depending on the current environmental state in a given niche (Figure 5.2, Right). A tolerance curve is a Gaussian pdf, with its mean corresponding to the preferred environmental state of the agent (highest fitness) and its standard deviation corresponding to the plasticity of the agent (how much the fitness is reduced when moving away from the preferred state).

The genome of an individual agent contains both the mean and standard deviation of its tolerance curve, as well as its mutation rate. Therefore, both plasticity (through the tolerance curve standard deviation) and evolvability (through the mutation rate) can evolve in this model. Our proposed selection mechanism entails two independent assumptions inspired from natural evolution:

1. niche-limited competition (N-selection): when deciding which individuals will repro- duce, we study each niche independently;

2. fitness-based selection (F-selection): within a niche, individuals produce offspring until its capacity is filled, with fitter individuals being chosen with higher probability.

Only individuals that can survive in a niche are considered for reproduction within it. We refer to this mechanism as NF-selection.

To evaluate the population we compute the following metrics at the end of each generation g:

1. μg , the value of the preferred environmental state, averaged over the population.

This metric indicates that the population is well-adapted when μg tracks the form of the climate function L.

2. σg , the value of the standard deviation σ g for preferred environmental states, averaged over the population. We refer to this metrics as the population-average plasticity.

3. rg , the mutation rate r component averaged over the population, which denotes the population-average evolvability.

4. X g = k X g k , the number of extinctions. We denote the survival of individual k in niche n at generation g as a binary variable:

s g k,n = (e n,g ∈ [µ g k -2σ g k , µ g k + 2σ g k ]) (5.1)
Thus, an individual goes extinct (X g k = 1) if n s g k,n is zero and survives (X g k = 0) if n s g k,n is positive.

5. Population survival A g , the percentage of generations that a run of our algorithm survived for. Values smaller than 1 are indicative of a mass extinction.

6. V g , the diversity of the population defined as the standard deviation of the population's genes, formally:

V = σ µ g + σ σ g + σ r g (5.2)
This metric captures the genetic diversity of the population.

7. D g , the dispersal of the population, computed as the number of niches over which at least one individual survives for a temporal window of at least w generations.

As this metric arises from the interaction of the genome and environment, we can view it as a measure of phenotypic diversity.

Results

Evolving in a stable environment

We define a stable environment as one where the climate function, and hence, the reference environmental state is constant, i.e. e g 0 = e 0 0 , ∀g ∈ G. The environmental state of the different niches is, therefore, equal to e g n = e 0 0 + n • . We experiment with high-quality environments (e 0 0 > 4) which can support a large population, low-quality environments (e 0 0 ≤ 0.5) where the capacity is low with the majority of niches being unable to support any individuals and medium-quality environments (0.5 < e 0 0 < 4) where some of the niches become uninhabitable for large enough N .

Low-quality

environments with multiple niches favor plasticity. environmental conditions and number of niches. We observe that when there is a single niche (N = 1) plasticity converges to a very low value regardless of the state. This is intuitive as the cost of plasticity captured by our genome model renders individuals with the smallest σ * k the fittest. This agrees with previous studies in constant environments Grove (2014) (note that F-selection is identical with NFselection when there is a single niche). However, the picture differs significantly when there are multiple niches and low-quality environments: as an individual can reproduce in any of the niches it can survive in, higher plasticity means higher chances of reproduction, which counteracts the cost of plasticity. As the quality of environments increases the benefit of plasticity disappears: non-plastic individuals dominate the available niches even though some individuals choose to disperse. In contrast to these interesting dynamics of plasticity, we observed that the population-average evolvability remained very low (r * < 10 -10 ) in all conditions. This observation is inline with the intuition that mutations disappear in stable environments as they incur fitness costs [START_REF] Lynch | Genetic drift, selection and the evolution of the mutation rate[END_REF][START_REF] Giraud | Costs and Benefits of High Mutation Rates: Adaptive Evolution of Bacteria in the Mouse Gut[END_REF].

In the full paper (Nisioti & Moulin-Frier, 2022), we also show that:

• niche-limited competition is necessary for plasticity to persist,

• diversity is highest under NF-selection,

• F-selection leads to more early extinction events.

Evolving under periodic variability

We model periodic variability as a sinusoid with period T e and amplitude A e that dictates the evolution of the reference environmental state e 0,g .

Plastic and evolvable individuals emerge under NF-selection only when the number of niches is sufficient. In Figure 5.4 (Left and Middle) we observe the ability of the population to survive when we vary the amplitude of oscillations (A e ) and number of niches (N ) respectively, for different values of the oscillation period T e . We can draw various conclusions from these results:

1. increasing the number of niches enables the population to survive longer in environments changing more frequently 2. survival is guaranteed for small-amplitude variations (A e = 0.2) regardless of their frequency 3. in the case of large-amplitude variations (A e = 8) high frequency does not allow the population to adapt at all 4. In the case of medium-amplitude (A e = 1) survival is possible only under lowfrequency variation

In Figure 5.4 (Right), we further analyze the ability of the population to survive by looking at how the evaluation metrics evolve with increasing generations. We observe that the population manages to track the environmental variability by keeping both plasticity and evolvability high, with oscillations in plasticity and evolvability occuring at twice the period of variability T e , as they increase at both transition points. Diversity is slightly lower during the low peaks of e 0 . We should note that we did not find high-frequency cases where the population reacted solely through plasticity or low-frequency cases where the population adapted solely through evolvability, as indicated by previous studies [START_REF] Cuypers | Evolution of evolvability and phenotypic plasticity in virtual cells[END_REF].

In the full paper (Nisioti & Moulin-Frier, 2022), we also show that F-selection and N-selection in isolation favor the evolution maladapted plastic individuals in periodic environment. In addition, we also analyze the dynamics of the system in noisy environments.

Discussion

We have designed a simple model of the evolution of plasticity and evolvability and studied the complex interactions between environmental and population dynamics. Experiments have revealed many insights into the evolution of adaptation mechanisms. Taking into account the effect of niche-limited competition gives rise to qualitatively different solutions to the plasticity-evolvability trade-off, in turn affecting population properties such as diversity and dispersal. We hope that our work sheds light into the plethora of related works and will prove useful in future studies of both artificial and natural systems.

We believe that Quality-Diversity algorithms [START_REF] Lehman | Evolving a diversity of virtual creatures through novelty search and local competition[END_REF][START_REF] Colas | Robots that can adapt like animals[END_REF][START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF] can be a particularly promising application area for such studies. Similarly to our proposal, this community lays emphasis on the benefits of combining niche-limited competition and fitness-based selection; however, as our empirical results indicate, parameters such as the number and quality of niches, as well as the form and presence of environmental variability show great qualitative impact and can potentially act as a curriculum for the emergence of adaptation. existing hypotheses:

From the perspective of human behavioral ecology our results offer insights into

1. the observation that low-quality environments favor plastic individuals, while highquality environments favor non-plastic individuals (see Section 5.2.5.1) hints to the turnover pulse hypothesis [START_REF] Vrba | Environment and evolution: Alternative causes of the temporal distribution of evolutionary events[END_REF];

2. the observation that adaptability is favored by abrupt transitions (see Section 5.2.5.2) and high variability (see published paper Nisioti & Moulin-Frier, 2022) hint to the variability selection hypothesis [START_REF] Potts | Hominin evolution in settings of strong environmental variability[END_REF] We should note that these hypotheses share a common model of environmental variability characterized by a large diversity of niches [START_REF] Maslin | A synthesis of the theories and concepts of early human evolution[END_REF].

It is important to also note the assumptions made by our study and how future work in artificial intelligence and artificial life can help overcome its limitations. First, tolerance curves assume that plasticity comes at a cost, an assumption that is often questioned in natural systems; we believe that studies with artificial systems can reveal whether such costs indeed arise. Second, our model does not capture the mechanism of species co-adaptation and can therefore not offer insights on how the dynamics of arms races are influenced by resource availability, as proposed by the Red Queen hypothesis [START_REF] Van Valen | A new evolutionary law[END_REF][START_REF] Solé | Revisiting Leigh Van Valen's "A New Evolutionary Law[END_REF][START_REF] Pearson | Red queen hypothesis[END_REF]. Then, our model assumes that there are no constraints on plasticity; we believe that studies with a more complex genotype to phenotype mapping that employ different adaptation mechanisms to ensure plasticity can reveal mechanism-specific limits that will extent the conclusions reached in this work. From an evaluation perspective, we have limited ourselves to measuring easily quantifiable properties of the genome and behavior space, such as adaptability and diversity, that have been linked to open-endedness [START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF]. As a next step we plan to investigate direct measures of open-endedness [START_REF] Dolson | The MODES toolbox: Measurements of open-ended dynamics in evolving systems[END_REF]. Finally, we believe that progress in open-endedness requires a better understanding of niching in recent simulation environments employed by the deep reinforcement learning community [START_REF] Suarez | Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents[END_REF][START_REF] Cobbe | Quantifying generalization in reinforcement learning[END_REF][START_REF] Nisioti | Grounding an Ecological Theory of Artificial Intelligence in Human Evolution[END_REF].

Finally, as proposed by the origins framework presented in Chapter 3, the effect of environmental variability on adaptation mechanisms is only part of the overall picture. To understand the complexity of the human ecological niche, we need to take into account multi-agent dynamics and culture, which modulate the processes of selection and niche construction [START_REF] Eppe | Intelligent behavior depends on the ecological niche: Scaling up AI to human-like intelligence in socio-cultural environments[END_REF][START_REF] Nisioti | Grounding an Ecological Theory of Artificial Intelligence in Human Evolution[END_REF]. We believe that exploring these links in social-centric multi-agent reinforcement learning environments [START_REF] Kovac | Benchmarking socio-cognitive abilities in deep reinforcement learning agents[END_REF] will further improve our understanding of adaptability.

The eco-evolutionary origins of autotelic agents (Level-1 dynamics)

Towards an ecologically valid simulation environment

Context

This work was realized in 2021-2022 in the Flowers team at Inria (France), in collaboration with Eleni Nisioti. It was partially funded by the Inria Exploratory action ORIGINS (https://www.inria.fr/en/origins) as well as the French National Research Agency (https://anr.fr/, project ECOCURL, Grant ANR-20-CE23-0006). This work also benefited from access to the HPC resources of IDRIS under the allocation 2020-[A0091011996] made by GENCI, using the Jean Zay supercomputer. Open access version of this publication is available at https://hal.archives-ouvertes. fr/hal-03446961/.

This paper presents both an earlier version of the framework presented in Chapter 3 as well as a novel simulation environment grounded in hypotheses from human behavioral ecology (hbe).

The rest of this section only focuses on the simulation environment. Here we entirely focus on environmental complexity, leaving the interaction with all the other components of the framework for future work. The presented environment aims at simulating some aspects of climate dynamics supposed to have played a key role in human behavioral ecology (Sections 3.2.2.1 and 3.4).

Relevance to the proposed origins framework

Proposed simulation environment

Many exciting environments have been recently proposed in the ai community [START_REF] Hafner | Benchmarking the spectrum of agent capabilities[END_REF][START_REF] Johnson | The Malmo Platform for Artificial Intelligence Experimentation[END_REF]. However, most existing environments do not display rich intrinsic dynamics independently of the agents' actions and, to the best of our knowledge, none implements climate dynamics. The Jelly Bean World (JBW) is a two-dimensional grid-world where agents navigate and collect items [START_REF] Platanios | Jelly bean world: A testbed for never-ending learning[END_REF]. Originally introduced as a benchmark for continuous learning, this environment automatically expands the world when the agent approaches its boundaries. The necessity for generating new parts of the world on demand in JBW led to the adoption of a low-complexity yet powerful mechanism for creating new items (e.g. new resources or environmental elements). Specifically, the creation and deletion of items are controlled by a probability distribution that can be configured through an intensity and interaction function, the former determining the probability of existence of an item independently of others, and the latter in relation to them. Using this mechanism, one can form a variety of item generation patterns, such as clusters and custom, spatially non-stationary distributions.

To enable the empirical experimentation of hypotheses proposed under the Pulsed Climate Variability (pcv) framework (see Section 3.2.2.1) we have equipped an existing simulation, The Jelly Bean World (JBW), with climate dynamics. Our objective is to observe the appearance of lakes with interesting dynamics, such as quick expansion and chaotic contraction [START_REF] Trauth | Human evolution in a variable environment: The amplifier lakes of Eastern Africa[END_REF], which in its turn will modulate the presence of resources available to the agents. To achieve this, we have extended the existing item generation mechanism in JBW with context-dependent resource generation. From an ecological perspective, the intensity function can be used to model an external climaterelated parameter, which in our case is the level of precipitation. Then, the interaction function can be used to model climate-related constraints, such as "resources grow only near water" and "lakes change their size based on humidity". To implement this functionality, we defined new types of items, i.e. , water cells that can be used to form lakes, resources (called "jelly-beans" and "bananas" as in the original JBW environment) that grow near lakes and trees.

A simplified model of our proposed climate dynamics is depicted in Figure 5.6(a), while Figure 5.6(b) presents the item presence patterns that arise from it. In this example, precipitation has a pulse form, which allows us to compare item patterns between periods of low (in Figure 5.6(c)) and high precipitation (5.6(d)).

Discussion

This first step towards an ecologically-valid environment has obvious limitations: it only consider a single component of the origins framework (environmental complexity) the introduction of adaptive agents has been left for future work (see Section 5.6.2). Our

The eco-evolutionary origins of autotelic agents (Level-1 dynamics) prediction is that our proposed environment will pose a challenge for standard methods in reinforcement learning that will struggle with non-stationarity and the sequential nature of the climate dynamics. We believe that bi-level optimization [START_REF] Hutchison | Indirectly Encoding Neural Plasticity as a Pattern of Local Rules[END_REF][START_REF] Najarro | Meta-Learning through Hebbian Plasticity in Random Networks[END_REF]) is an interesting direction as it can model the interaction between evolutionary and developmental processes (see Section 5.6.2 for more detail on future work).

Learning Sensorimotor Agency in Cellular Automata

Context

This work was realized in 2020-2022 as a collaboration between the Flowers group at Inria (France) and Bert Chan1 , now at Google Brain Tokyo (Japan). It builds upon recent exploration algorithms for automated discovery in complex systems developed with Mayalen Etcheverry (PhD student at Flowers) and Chris Reinke (former post-doc at Flowers) (Reinke et al., 2020(Reinke et al., -02-17, 2020-2-17;-2-17;[START_REF] Etcheverry | Hierarchically-Organized Latent Modules for Exploratory Search in Morphogenetic Systems[END_REF]. This work benefited from access to the HPC resources of IDRIS under the allocation 2020-[A0091011996] made by GENCI, using the Jean Zay supercomputer.

The main publication presenting this contribution takes the form of an extended online blog post, including videos and an online demo: Open access version of this publication is available at https://developmentalsystems. org/sensorimotor-lenia/. The rest of this section is a summary of this blog post. We are currently working on a journal article, extending it with a quantitative analysis. .7 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. This contribution adopts a radical enactive view of cognition where, contrarily to all the other contributions presented in this thesis, there is originally no distinction between agents and the environment. Instead, we use a multi-channel continuous cellular automaton where the local rules governing some channels are fixed, modeling an imposed environmental dynamics, while the rules governing other channels can be adapted according to some objective. We study how such a self-organizing system, under some conditions of environmental complexity, can evolve low-level sensorimotor skills from scratch (adaptability). Interestingly we observe that the evolved proto-agents (called creatures in the paper), while trained in isolation for spatial localization and navigation skills, show impressive generalization abilities to unseen environmental conditions. In particular, rich multi-agent dynamics emerges when multiple creatures are simultaneously placed in a shared environment at test time.

Relevance to the proposed origins framework

Abstract

Novel classes of Cellular Automata (CA) have been recently introduced in the Artificial Life (ALife) community, able to generate a high diversity of complex self-organized patterns from local update rules. These patterns can display certain properties of biological systems such as a spatially localized organization, directional or rotational movements, etc. In fact, CA have a long relationship with biology and especially the origins of life/cognition as it is a self-organizing system that can serve as a computational testbed and toy model for such theories [START_REF] Beer | Autopoiesis and Cognition in the Game of Life[END_REF] but also as a source of inspiration on what are the basic building block of "life". However, while the notions of embodiment within an environment, individuality and self-maintenance are central in theoretical biology and in particular in the definition of agency (e.g. [START_REF] Maturana | Autopoiesis and Cognition: The Realization of the Living[END_REF][START_REF] Varela | Patterns of life: Intertwining identity and cognition[END_REF], it remains unclear how such mechanisms and properties can emerge from a set of local update rules in a CA. In this contribution, we propose an approach enabling to learn self-organizing agents capable of reacting to the perturbations induced by the environment, i.e. robust agents with sensorimotor capabilities. We provide a method based on curriculum learning, on diversity search and on gradient descent over a differentiable CA able to discover the rules leading to the emergence of such creatures (Figure 5.8 A and B). The creatures obtained, using only local update rules, are able to regenerate and preserve their integrity and structure while dealing with the obstacles or other creatures in their way (Figure 5.8 C). They also show great generalization, with robustness to changes of scale, random updates or perturbations from the environment not seen during training. We believe that the field of artificial intelligence could benefit from these capabilities of self-organizing systems to make robust intelligent systems that can quickly adapt to new environments and to perturbations. Add the growth of every kernel

Update of Leariable channel

A t A t+1

Clip between 0 and 1 Learnable channel A B C Figure 5.8: A. We use a multi-channel continuous cellular automaton (Lenia). The state A t at time t is a square grid with two colored channels: an Obstacle channel in blue and a Learnable channel in yellow. Each channel is associated with a set of convolution kernel filters and growth functions, whose outputs are summed up to compute of next step A t+1 . B) Parameters of kernels and growth functions are optimized using backpropagation through time, minimizing a L2 loss between the final state of the learnable channel and a target shape at a specific location. Target locations and obstacle positions are randomized across episodes. C) Optimized creatures in the Learnable channel are able to maintain their integrity while navigating in a field of obstacles (top left). They also generalize their skills in environments not encountered during training, e.g. when placed in multiple-creature environments or when damaged (top-right and bottom). .9 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Here we study how particular multiagent dynamics, where agents do not have access to the observations of others but can communicate within a limited range, guarantees a common context that can be leveraged in individual representation learning to improve adaptability. In the presented experiments, environmental complexity is highly simplified.

Relevance to the proposed origins framework

Environmental complexity (EC)

• Multiscale dynamics

• Day/night • Seasons • Climate variations

• Constraints and opportunities

• Resource availability • Exposition to predators • Compositional dynamics

Cultural repertoire (CR)

• Technology

• Tool use • Engineering

• Communication

• Language • Arts

• Social organization

• Norms and values • Institutions

Multi-agent dynamics (MD)

• Interactions

• Signaling • Manipulation

• Game-theoretic context

• Cooperative • Competitive • Social dilemmas

Adaptability (A)

• Evolutionary scale

• Natural selection • Speciation • Extinction • Developmental scale • Self-regulatioin • Learning • Exploration Figure 5
.9: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

Introduction

In this work, we propose that aligning internal subjective representations, which naturally arise in a multi-agent setup where agents receive partial observations of the same underlying environmental state, can lead to more data-efficient representations. We propose that multi-agent environments, where agents do not have access to the observations of others but can communicate within a limited range, guarantees a common context that can be leveraged in individual representation learning. The reason is that subjective observations necessarily refer to the same subset of the underlying environmental states and that communication about these states can freely offer a supervised signal. To highlight the importance of communication, we refer to our setting as socially supervised representation learning. We present a minimal architecture comprised of a population of autoencoders, where we define loss functions, capturing different aspects of effective communication, and examine their effect on the learned representations.

Contributions.We summarize our contributions as follows:

1. We highlight an interesting link between data-augmentation traditionally used in single-agent self-supervised setting and a group of agents interacting in a shared environment (Figure 5.10). 3. We present a detailed analysis of the conditions ensuring both the learning of efficient individual representations and the alignment of those representations across the agent population (Section 5.5.4). Green boxes indicate conceptual agents, while we assume that a singular representation learning method may be interpreted as a single agent.

We introduce

The eco-evolutionary origins of autotelic agents (Level-1 dynamics)

.11: Proposed architecture. Two agents i and j (top and bottom coloured boxes) are presented with different observations (oi and oj) of the same underlying environment state (s). Each agent (say i) implements a standard autoencoder architecture with an encoder enci mapping the input observations to latent representations and a decoder deci mapping latent representations to reconstructed observations. Each agent i communicates its latent representation mi (also called a message) of its own observation input oi to the other agents. This way, each agent i is able to reconstruct the observation from its own latent representation (oii: reconstruction by agent i from its own message mi) as well as from the latent representation of the other (oij: reconstruction by agent i from the other's message mj).

The architecture can be trivially extended to a larger population, where each agent communicates their latent representations to with each other. We show that our proposed architecture allows the emergence of aligned representations. This means that different agents find similar encodings for the same sensory inputs. The subjectivity introduced by presenting agents with distinct perspectives of the environment state contributes to learning abstract representations that outperform those learned by a single autoencoder and a population of autoencoders, presented with identical perspectives of the environment state, which is shown in the left column of Fig. 5.12. Furthermore, in Fig. 5.12 (right) we show that the learned representations are dataefficient, i.e. they enjoy the most benefit when evaluated on small testing splits. This is Socially Supervised Representation Learning: the Role of Subjectivity in Learning Efficient

Results

Yes

Representations 75

important, because good representations should allow agents to adapt to downstream tasks quickly and with few samples. Altogether, our results demonstrate how communication from subjective perspectives can lead to the acquisition of more abstract representations in multi-agent systems, opening promising perspectives for future research at the intersection of representation learning and emergent communication.

The eco-evolutionary origins of autotelic agents (Level-1 dynamics)

5.6 Other contributions and future work

Other contributions

I have presented in this chapter most of my contributions related to level-1 dynamics, which is the most recent topic I have been interested in (from 2020). On the methodological side, we are currently developing software tools for automated discovery in science and art. These tools aim at providing easy-to-use interactive interfaces for assisting the exploration of complex systems. An automated discovery algorithm was published at the NeurIPS conference [START_REF] Etcheverry | Hierarchically-Organized Latent Modules for Exploratory Search in Morphogenetic Systems[END_REF], with its application in the context of the Minecraft Open-Endedness Challenge2 in a blog post [START_REF] Etcheverry | Meta-Diversity Search in Complex Systems,A Recipe for Artificial Open-Endedness ?[END_REF]. We plan to release a beta version of an open-source software for automated discovery by the end of 2022. I believe that such automated discovery algorithms can help to search for interesting levels of environmental complexity in the context of studying level-1 dynamics.

Future work

Level-1 dynamics considers pre-existing environmental complexity operating at multiple spatiotemporal scales (e.g. seasonal cycles and climate variation), modulating constraints and opportunities (e.g. in terms of resource availability and exposition to predators) exposed to the agents. The objective is to study how this environmental complexity interacts with adaptability and multi-agent dynamics toward the evolution of learning and exploration mechanisms. The main research question at this level is: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies?

Based on the theoretical analysis of level-1 dynamics in Section 3.4 as well as my existing contributions presented in the current chapter, future work will focus on the following directions.

Simulation environments grounded in human behavioral ecology

A central aspect of the origins framework is that the evolution of oesa is originally driven by certain characteristics of environmental complexity. As we have seen in Section 3.4, a variety of hypotheses in hbe highlight the importance of climate dynamics in providing a wide diversity of environmental constraints and opportunities for human evolution. We have also highlighted the importance of diverse ecological niches promoting behavioral diversity. In consequence, simulation environments for studying the evolution of oesa in ai should respect the following desiderata:

• Unbounded and realistic dynamics at multiple spatiotemporal scales.

Rather than requiring explicit design, patterns of resource availability and exposition to predators should emerge naturally and exhibit complex dynamics (e.g. through resource regrowth as in [START_REF] Pérolat | A multi-agent reinforcement learning model of common-pool resource appropriation[END_REF]. This feature will enable the study of how environmental variability at different scales influence the interaction between evolution, development and learning (see Section 5.6.2.3 below for more detail).

• Spatial open-endedness, a requirement for the appearance of diverse niches. The environment should simulate the co-existence of diverse ecological niches resulting in diverse constraints and opportunities for the agents and potentially allowing their speciation and dispersal [START_REF] Lehman | Evolvability is inevitable: Increasing evolvability without the pressure to adapt[END_REF].

• A variety of tasks relevant to human evolution, such as navigation, harvesting, hunting and crafting through tool use. In particular, the environment should display compositional dynamics, i.e. offering opportunities for agents to produce new elements (e.g. a hammer) by composing other existing elements (e.g. wood and stone), as illustrated in Figure 5.13. This is crucial for enabling behavioral diversity and innovation, an important property of oesa [START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF][START_REF] Hintze | Open-endedness for the sake of open-endedness[END_REF].

We have recently achieved preliminary steps in this direction including spatial openendedness in a grid world with complex climate dynamics (see Section 5.3), diverse niches with various types of abstract environmental dynamics (see Section 5.2), as well as compositional dynamics defined as large "recipe books" defining how elements of the environment can be combined together to create new ones (see Section 7.3 and 7.2).

Design of bi-level optimization algorithms mimicking the interplay between evolution and development

For this we want to integrate mechanisms inspired by meta reinforcement learning (meta-rl) [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF], curriculum learning [START_REF] Portelas | Automatic curriculum learning for deep RL: A short survey[END_REF] and neuroevolution [START_REF] Papavasileiou | A systematic literature review of the successors of "NeuroEvolution of augmenting topologies[END_REF] (see Section 3.2.2.2). A central conceptual limitation of the meta-rl paradigm, in particular, is the fact that environments are sampled randomly, ignoring the curriculum potential of continuous interactions with dynamic environments. In contrast, theories from hbe suggest a major role of spatiotemporal dynamics in human evolution (e.g. the Pulsed Climate Variability (pcv) framework, see Section 3.2.2.1). On the other hand, curriculum learning takes into account the order in which environments are presented to the agents, but existing contributions in this field do not integrate with bi-level optimization as in meta-rl. We instead propose to study algorithms with the following desiderata: a) the outer-loop optimization (analogous to evolution) is influenced by the order in which different tasks are experienced b) the inner-loop (analogous to development) can generate a wide range of behavioral patterns, from hard-wired behavior to adaptive learning dynamics to autotelic exploration.

Study the role of environmental variability at multiple spatiotemporal scales on the evolution of autotelic agents

As we have seen in Section 3.4, ecological [START_REF] Johnston | Selective Costs and Benefits in the Evolution of Learning[END_REF][START_REF] Stephens | Change, regularity, and value in the evolution of animal learning[END_REF] and computational [START_REF] Singh | Intrinsically motivated reinforcement learning: An evolutionary perspective[END_REF][START_REF] Niv | Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors[END_REF][START_REF] Lange | Learning not to learn: Nature versus nurture in silico[END_REF] studies point to the following hypotheses: a) low environmental variability between generations favors innate behaviors b) high environmental variability between generations favors the evolution of learning c) advanced exploration strategies, e.g. in the form of intrinsic motivation, emerge when environmental variability within a lifetime is high enough for exploration to be vital d) prohibitively high variability during a lifetime renders adaptation futile. In a recent contribution, we evaluated the evolution of both developmental plasticity and evolvability under various spatiotemporal environmental dynamics (Nisioti & Moulin-Frier, 2022, see Section 5.2). We now plan to benchmark the performance of bilevel optimization algorithms (as described in 5.6.2.2 above) to gauge their ability to appropriately respond to various types of variability in richer environments (as described in 5.6.2.1). We are particularly interested in environments containing multiple niches, each with diverse sets of available elements, and environmental variability modulating how these elements can be combined through compositional dynamics. We predict that such environments will encourage the evolution of autotelic agents continuously exploring new crafting opportunities even if they don't necessarily influence their evolutionary fitness over their own lifetime, as a solution to be better prepared for upcoming environmental changes.

Empirically evaluate hypotheses featured under the Pulsed Climate Variability (pcv) framework

To achieve this, we will model environmental dynamics following the climate curves suggested by paleoclimatology data (as described in 5.6.2.1) and evaluate populations of agents in foraging tasks where resource availability patterns are modulated by these climate curves in our proposed simulation environment [START_REF] Nisioti | Grounding an Ecological Theory of Artificial Intelligence in Human Evolution[END_REF], see Section 5.3). Our objective will measure the proposed metrics of speciation, extinction and diversity, which will enable the evaluation and comparison of the hypotheses proposed under the pcv framework. We believe that this type of interdisciplinary work will prove fruitful for both communities: hbe is in need of advanced computational methods [START_REF] Marean | A new research strategy for integrating studies of paleoclimate, paleoenvironment, and paleoanthropology[END_REF][START_REF] Frankenhuis | Enriching behavioral ecology with reinforcement learning methods[END_REF] and research in ai will go beyond studying ungrounded types of environmental variability. We have recently proposed a first step in this direction in an evo-eco-devo model (Nisioti & Moulin-Frier, 2022, see Section 5.2) and in a simulation environment [START_REF] Nisioti | Grounding an Ecological Theory of Artificial Intelligence in Human Evolution[END_REF], see Section 5.3).

Figure 5.13: Compositional dynamics as a key feature of simulation environments for oesa. An environment is said to support compositional dynamics when the agent can produce new elements (e.g. an axe) by composing other existing elements (e.g. wood and stone). On the left, Little Alchemy (https://littlealchemy.com/) is a game with compositional dynamics in a non-embodied setting. On the right, Krafter [START_REF] Hafner | Benchmarking the spectrum of agent capabilities[END_REF], a simulation environment inspired from Minecraft where agents encounter elements upon navigating. Here we consider an adaptive agent equipped with a general mechanism of intrinsically-motivated goal exploration (i.e. adaptability at the developmental scale). This agent is equipped with a model of the human vocal tract and of the auditory system. In a first experiment, there is no environmental complexity (except the agent's own morphology, i.e. its vocal tract producing sounds) and no multi-agent dynamics. We study how a general mechanism of intrinsically motivated spontaneous exploration can self-organize developmental stages during early vocal learning, without any imposed constraint to communicate. We observe that the agent spontaneously produces vocalizations of increasing complexity (from no phonation, to unarticulated sounds, to articulated sounds resembling proto-syllables in infants). Then, in a second experiment, the agent also perceives mature vocalizations from surrounding "adult" agents (hence introducing very basic multi-agent dynamics) and we study how it influences the agent's exploration toward the ambient sounds. We argue that such developmental structures, self-organizing from general mechanisms of intrinsically-motivated learning, can provide the building blocks in the emergence of a cultural repertoire (here providing a repertoire of diverse vocalizations that can potentially bootstrap language evolution and acquisition, see [START_REF] Oudeyer | How Evolution May Work Through Curiosity-Driven Developmental Process[END_REF], for a complete explanation of the theoretical argument).

Relevance to the proposed origins framework

Abstract

We propose and experimentally test the hypothesis that general mechanisms of intrinsically motivated spontaneous exploration, also called curiosity-driven learning, can self-organize developmental stages during early vocal learning. We introduce a computational model of intrinsically motivated vocal exploration, which allows the learner to autonomously structure its own vocal experiments, and thus its own learning schedule, through a drive to maximize competence progress (Figure 6.2.B). This model relies on a physical model of the vocal tract, the auditory system and the agent's motor control (Figure 6.2.A), as well as vocalizations of social peers. Computational experiments show how such a mechanism can explain the adaptive transition from vocal self-exploration with little influence from the ambient speech environment, to a later stage where vocal exploration becomes influenced by vocalizations of peers (Figure 6.2.D). Within the initial self-exploration phase, a sequence of vocal production stages self-organizes (Figure 6.2.C), and shares properties with data from infant developmental psychology (Figure 6.2.E): the vocal learner first discovers how to control phonation, then focuses on vocal variations of unarticulated sounds, and finally automatically discovers and focuses on babbling with articulated proto-syllables. As the vocal learner becomes more proficient at producing complex sounds, imitating vocalizations of peers starts to provide high learning progress explaining an automatic shift from self-exploration to vocal imitation.

Discussion

Our main contribution with respect to previous computational models of speech acquisition is that we do not presuppose the existence of successive developmental stages, but rather they can emerge from an intrinsic drive to maximize competence progress in continuous sensorimotor spaces. We showed that vocal developmental stages can self-organize autonomously, from simple sensorimotor activities to more complex ones. The agent starts producing no phonation and unarticulated vocalizations, which are easy to produce because limited in the range of their auditory effects. This can be related B) The agent learns an inverse model from a goal space T of desired auditory trajectories, to the motor trajectories in Π required to achieve those goals. For this aim it uses a generic intrinsically motivated goal exploration process (imgep, see Section 3.2.1.2) where it self-generates its own goals in T and iteratively learns a mapping T → Π. C) When exploring autonomously how articulatory commands produce auditory effects, the model first produces non-speech sounds (no phonation), then unarticulated sounds and finally articulated sounds on the form of proto-syllables. D) When the agent has the choice to either self-explore its vocal abilities or imitate sounds from an ambient language, it chooses to imitate only once it has acquired sufficient motor control over its vocal tract. This choice is made by sampling the strategy (self-exploration or imitation) where it observes maximal learning progress. E) These results are coherent with the main developmental stages observed in infant vocal development [START_REF] Kuhl | Early language acquisition: Cracking the speech code[END_REF].

to the first stage in infant vocal development (Figure 6.2.E), where the agent produces non speech-sounds (e.g., growls, squeals. . . ) before learning phonation and then produces not well-articulated quasi-vowels. Later on, once the agent does not progress much in producing unarticulated vocalizations, it focuses on more complex vocalizations of the articulated class. The reason is that, due to the properties of the sensorimotor system and internal model, the mastering of complex tasks require first the mastering of simpler tasks in order to yield significant competence progress, so that these complex tasks are selected as interesting goals.

We also showed that intrinsically motivated exploration can lead to a progressive interest toward the sounds of the ambient language. Whereas the first vocalizations are mainly the result of self-exploration, they progressively lead to mastering necessary speech principles (e.g., phonation). This progressive mastering allows in turn to make significant progress in adult-speech imitation, which explains why the vocal learner starts to choose more often as targets the sound of its environment. Competence-progress based curiosity-driven exploration could thus explain a progressive influence of the ambient language on infant vocalizations.

We therefore showed that intrinsically motivated active exploration can self-organize a coherent developmental sequence, without any external clock or preset specification of this sequence. This possible role of intrinsic motivation, providing a mechanism to discover autonomously necessary developmental stages to structure the learning process, is here validated computationally. We believe that it could be of major interest for understanding the structuration of early vocal development in infants (see [START_REF] Oller | The Emergence of the Speech Capacity[END_REF], for theoretical arguments on the role of intrinsic motivation in vocal development). Moreover, active exploration can spontaneously generate diverse behaviors from modality-independent and task-independent internal drives. Such spontaneous behavior can result in vocal activity that may have bootstrapped the evolution of an articulated vocal language in the human species [START_REF] Oudeyer | How Evolution May Work Through Curiosity-Driven Developmental Process[END_REF]. Open access version of this publication is available at https://www.sciencedirect. com/science/article/pii/S0095447015000352. The rest of this section is a summary of this paper.

Relevance to the proposed origins framework

Figure 6.3 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Here we consider a population of interacting agents, each equipped with a model of the human vocal tract and the ear. In terms of adaptability, each agent is equipped with a Bayesian cognitive architecture allowing to learn a joint probability distribution of motor commands to its vocal tract, produced vocal sounds and perceived objects in the environment. This distribution is learned through repeated interactions with other agents and objects, with minimal environmental complexity. Multi-agent dynamics is modeled as deictic games, where agents share their attention on objects of the environment and produce and perceive their vocalizations. We study how a shared sound system for communicating about the surrounding objects can self-organize out of the interactions between adaptive agents, thus forming a cultural repertoire in the form of a proto-language.

Abstract

While the origin of language remains a somewhat mysterious process, understanding how human language takes specific forms appears to be accessible by the experimental method. Languages, despite their wide variety, display obvious regularities. This contribution attempts to derive some properties of phonological systems (the sound systems for human languages) from computer simulations of interacting learning agents. It in- troduces a model of the cognitive architecture of a communicating agent, called cosmo (for "Communicating about Objects using Sensory-Motor Operations") that allows a probabilistic expression of the main theoretical trends found in the speech production and perception literature. This enables a computational comparison of these theoretical trends, which helps to identify the conditions that favor the emergence of linguistic codes. It provides realistic simulations of phonological system emergence where sensory-motor agents equipped with a computer model of the human vocal tract and the ears interact together and learn to associate speech sounds with the objects they perceive in the environment. cosmo can predict the main regularities in vowel, consonant and syllable systems in human languages.

Methods

Cognitive architecture

The scenario that we propose for the emergence of speech communication starts from the inherent structure of a communication situation. In cosmo, speech communication (C) is a success when an object O S in the speaker's mind is transferred, via sensory and motor mechanisms S and M , to the listener's mind, where it is correctly recovered as O L (Figure 6.4.A).

The central hypothesis of the cosmo model is that a communicating agent, which is both a speaker and a listener, is able to internalize fully the structure of the communication situation inside an internal model (Figure 6.4.B).

This "internalization" hypothesis therefore results in an agent cognitive architecture (cosmo agent box in Figure 6.4) combining: • an auditory system able to associate communication objects O L with auditory stimuli S,

• a sensory-motor link able to associate motor gestures M with auditory stimuli S, (providing an internal model of the articulatory-to-acoustic transformation), and • a fusion system able to associate the communication objects in both the motor (O S ) and auditory (O L ) branches through C.

This internalization hypothesis could be discussed within the framework of general cognitive theories of social communication and human evolution [START_REF] Baron-Cohen | Mindblindness: An Essay on Autism and Theory of Mind[END_REF][START_REF] Tomasello | Understanding and sharing intentions: The origins of cultural cognition[END_REF] (see also [START_REF] Moore | Spoken language processing: Piecing together the puzzle[END_REF] for similar views about internalization, expressed in a control theory framework).

The Bayesian model of cosmo is a direct translation of the conceptual cognitive architecture into the joint distribution P (C, O S , S, M, O L ) (Figure 6

Emergence of phonological systems in populations of deictic agents

Following the language-game paradigm introduced by Steels [START_REF] Steels | The synthetic modeling of language origins[END_REF] and applied to phonological systems [START_REF] Berrah | From Form to Formation of Phonetic Structures: An evolutionary computing perspective[END_REF][START_REF] Berrah | Evolution d'une Société Artificielle d'agents de Parole : Un Modéle Pour l'émergence Des Structures Phonétiques[END_REF][START_REF] De Boer | Self-organization in vowel systems[END_REF][START_REF] Oudeyer | The self-organization of speech sounds[END_REF], we simulate societies of evolving communicating agents by exploiting language games, based on deixis, that we call "deictic games" and that may be considered as a variant of Steels' "naming games" [START_REF] Steels | The Artificial Life Roots of Artificial Intelligence[END_REF][START_REF] Steels | The synthetic modeling of language origins[END_REF].

In a deictic game, two agents communicate in front of a given "object" on which their attention is jointly focused via a deictic process of some kind, such as pointing (Figure 6.5.A).

Each agent embeds it own copy of the cosmo Bayesian model, with no prior knowledge in the motor and auditory subsystems (P (M | O S ) and P (O L | S) distributions respectively). Each agent will iteratively update the distributions of its own subsystems through it interaction with other agents and various objects (see Moulin-Frier et al. (2015a) for computational detail).

Results

We ran large-scale simulations of cosmo agents interacting with objects through deitic games. We studied how their coupled learning processes can self-organize speech sound systems shared among the population to name the objects they interact with. We found that the structure of the emerged speech sound systems share the main statistical tendency the structure of phonological systems in human languages (Figure 6.5.B). In the paper (Moulin-Frier et al., 2015a), we analyze in detail the emergence of vowel, consonant and syllable systems.

Emergent language as a solution to align goals in agent populations

The imgep framework introduced by our research group (see Section 3.2.1.2) encompasses a family of learning mechanisms with which an agent autonomously defines its own goals and self-supervises their achievement. It allows to implement learning agents autonomously discovering a large diversity of goals and associated behaviors to achieve those goals (i.e. autotelic agents). Extending this framework to multi-agent settings would enable the study of emergent language as a solution to align the goals of multiple autotelic agents within a group. The marl community have recently studied multi-goal multiagent rl, but with externally provided goals that are already aligned [START_REF] Yang | CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning[END_REF][START_REF] Wang | Model-based reinforcement learning for decentralized multiagent rendezvous[END_REF]. We believe that by combining the benefits of intrinsically motivated goal exploration and multi-agent learning, we will be able to improve the generalization abilities of groups of agents in diverse game-theoretic contexts. Our proposal is grounded in theories from human studies suggesting that shared intentionality is a key element in achieving human-like cooperation [START_REF] Tomasello | Shared intentionality[END_REF].

Open-ended collective innovation

In recent work (Nisioti et al., 2022, see Section 7.3) we have proposed the concept of innovation tasks, where groups of rl agents explore how to combine existing elements to create new ones through compositional dynamics in the environment. Innovation tasks are often characterized by a deceptive optimization landscape with strong local optima and usually require collective exploration to be solved [START_REF] Derex | Partial connectivity increases cultural accumulation within groups[END_REF]. Recent human studies of the innovation abilities of groups [START_REF] Mason | Propagation of innovations in networked groups[END_REF][START_REF] Muthukrishna | Innovation in the collective brain[END_REF][START_REF] Derex | Partial connectivity increases cultural accumulation within groups[END_REF] have revealed that properties of the task, such as the existence of compositional dynamics and path-dependence, as well as the form of social connectivity, significantly influence the group's performance. An intriguing observation is that fully-connected structures can lead to suboptimal performance due to the over-exploitation of learned behaviors, while partial connectivity can help achieve the right balance between exploration and exploitation. Based on these observations, we have recently tested this hypothesis in a non-embodied simulation environment, the wordcraft playground [START_REF] Jiang | An environment for benchmarking commonsense agents[END_REF], a simplified version of the Little Alchemy game (Figure 5.13,left) where drl agents continuously crafts elements out of existing ones (Nisioti et al., 2022, see Section 7.3). We now plan to extend these experiments in an embodied environment with compositional dynamics as described in Section 5.6.2.1 and will explore the possibility to meta-learn the social connectivity structure itself using bilevel optimization algorithms as proposed in Section 5.6.2.2, potentially in the context of the multi-agent imgep framework proposed in Section 6.4.2.1.

Introduction

In this chapter, I present some of my contributions and propose future work in the modeling of level-3 dynamics. As presented in Section 3.6, this part of the proposed framework is interested in the feedback effects from a cultural repertoire to all the other components of the system. The main research question at this level is: What is the role of feedback effects from a cultural repertoire in the open-ended dynamics of human skill acquisition?

In Section 7.2, I first present a contribution showing how the compositionality of language can be used as a cognitive tool for imagining novel creative goals, thus augmenting the learning and exploration abilities of reinforcement learning agents. Then, in Section 7. Open access version of this publication is available at https://proceedings.neurips. cc/paper/2020/hash/274e6fcf4a583de4a81c6376f17673e7-Abstract.html. The rest of this section is a summary of this paper.

Demonstration videos are available at https://sites.google.com/view/imaginedrl. The source code of playground environment can be found at https://github. com/flowersteam/playground_env and the source code of the imagine architecture https://github.com/flowersteam/Imagine.

Relevance to the proposed origins framework

This contribution proposes a novel intrinsically motivated deep reinforcement learning architecture, called imagine, where an agent learn how to achieve a wide variety of goals through its interaction with a social partner (Figure 7.2). We design a novel environment, called Playground, displaying complex dynamics, where various objects can interact together in a compositional way. The learning agent can navigate in the environment and move the objects around, observing their potential interactions. For instance, animals grow when water or food is placed on them, while plants only grow with water. At the end of each episode, a simulated social partner provides linguistic descriptions of the interaction between the agent and the environment (e.g. you grow the green cat). The agent interprets these descriptions as potential future goals and learn a reward function and a goal-conditioned policy for achieving them. In a second phase where the social peer is absent, we then study how the agent can leverage the compositionality of language to invent new creative goals. We show how this mechanism of goal imagination improves generalization and exploration over agents lacking this capacity. partner. We assume a pre-existing cultural repertoire, where a simulated social partner provides descriptions of the interaction between the agent and the environment in language form. The objective of this contribution is to show how the agent can internalize this cultural knowledge, augmenting its exploration and generalization abilities (adaptability) through a compositional goal imagination mechanism. In this sense, imagine demonstrates an important influence from the cultural repertoire to the adaptability of a learning agent, mediated through environmental complexity and multi-agent dynamics.

Environmental complexity (EC)

• Multiscale dynamics • Technology

• Tool use • Engineering

• Communication

• Language • Arts

• Social organization

• Norms and values • Institutions

Multi-agent dynamics (MD)

• Interactions

• Signaling • Manipulation • Game-theoretic context • Cooperative • Competitive • Social dilemmas

Adaptability (A)

• Evolutionary scale

• Natural selection • Speciation • Extinction • Developmental scale • Self-regulatioin • Learning • Exploration

Abstract

Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states [START_REF] Held | Automatic Goal Generation for Reinforcement Learning Agents[END_REF][START_REF] Nair | Visual Reinforcement Learning with Imagined Goals[END_REF][START_REF] Colas | Intrinsically motivated modular multi-goal reinforcement learning[END_REF][START_REF] Pong | Skew-Fit: State-Covering Self-Supervised Reinforcement Learning[END_REF][START_REF] Venkattaramanujam | Self-supervised learning of distance functions for goal-conditioned reinforcement learning[END_REF][START_REF] Racaniere | Automated curricula through setter-solver interactions[END_REF]. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play [START_REF] Tomasello | The Cultural Origins of Human Cognition[END_REF][START_REF] Bruner | The narrative construction of reality[END_REF][START_REF] Piaget | The Language and Thought of the Child[END_REF][START_REF] Vygotsky | Tool and Symbol in Child Development[END_REF] . We introduce imagine, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.

Methods

General overview

This contribution proposes Intrinsic Motivations And Goal INvention for Exploration (imagine): a learning architecture which leverages natural language (nl) interactions with a descriptive social partner (sp) to explore procedurally-generated scenes and interact with objects. imagine discovers meaningful environment interactions through its own exploration (Figure 7.2a) and episode-level nl descriptions provided by sp (7.2b). These descriptions are turned into targetable goals by the agent (7.2c). The agent learns to represent goals by jointly training a language encoder mapping nl to goal embeddings and a goal-achievement reward function (7.2d). The latter evaluates whether the current scene satisfies any given goal. These signals (ticks in Figure 7.2d-e) are then used as training signals for policy learning. More importantly, imagine can invent new goals by composing known ones (7.2f). Its internal goal-achievement function allows it to train autonomously on these imagined goals.

The imagine Architecture

imagine agents build a repertoire of goals and train two internal models: 1) a goalachievement reward function R to predict whether a given description matches a behavioral trajectory; 2) a policy π to achieve behavioral trajectories matching descriptions. The architecture is presented in Figure 7.3 and follows this logic:

1. The Goal Generator samples a target goal g target from known and imagined goals (G known ∪ G im ).

2. The agent (RL Agent) interacts with the environment using its policy π conditioned on g target .

3. State-action trajectories are stored in a replay buffer mem(π).

4. sp's descriptions of the last state are considered as potential goals G sp (s T ) = D sp (s T ).

5. mem(R) stores positive pairs (s T , G sp (s T )) and infers negative pairs (s T , G known \ G sp (s T )).

6. The agent then updates:

• Goal Gen.: G known ← G known ∪ G sp (s T ) and G im ← Imagination(G known ).

• Language Encoder (L e ) and Reward Function (R) are updated using data from mem(R).

• RL agent: We sample a batch of state-action transitions (s, a, s ) from mem(π).

Then, we use Hindsight Replay and R to bias the selection of substitute goals to train on (g s ) and compute the associated rewards (s, a, s , g s , r). Substituted goals g s can be known or imagined goals. Finally, the policy and critic are trained via RL.

All algorithmic details are provided in the paper [START_REF] Colas | Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration[END_REF] and the source code is available at https://github.com/flowersteam/Imagine.

Results

We carried out experiments in order to evaluate the benefits from goal imagination in intrinsically motivated learning. Experiments are split into two phases. In the first one, the agent interacts with the social partners, collects descriptions of goals and stores them in a set of known goal descriptions. The agent uses these descriptions paired with its observations in order to learn an internal reward function that detects when the goal represented by the descriptions are achieved in a given scene. Once this internal reward function is obtained, the agent uses its output (the reward signal) in order to train a goal-conditioned policy enabling it to reach any goal. In the second phase, the social partner disappears and the agent starts imagining new goals by composing the descriptions stored in the set of known goals. The agent then targets these new goals and by doing so, discovers new interactions.

We measured the success rate of agents on a wide set of different skills and observed that agents that do not imagine goals (that stop at phase 1) master a smaller set of skills than agents that do imagine goals (Figure 7.4.A). A particular generalization: growing plants. Agents learn to grow animals from sp's descriptions, but are never told they could grow plants. When evaluated offline on the growing-plants goals before goal imagination, agents' policies perform a sensible zero-shot generalization and bring them water or food with equal probability, as they would do for animals (Figure 7.4.B,left). As they start to imagine and target these goals, their behavior adapts (Figure 7.4.B, right). If the reward function shows good zero-shot abilities, it only provides positive rewards when the agent brings water. The policy slowly adapts to this internal reward signal and pushes agents to bring more water. We call this phenomenon behavioral adaptation. The full paper [START_REF] Colas | Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration[END_REF] details the generalization abilities of imagine for 5 different types of generalizations involving predicates, attributes and categories.

A B B C

Exploration. In the full paper [START_REF] Colas | Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration[END_REF], we provide additional results answering the following questions:

Towards human-like open-ended skill acquisition through cultural feedback effects (Level-3 dynamics)

• What if we used other goal imagination mechanisms?

• How does modularity interact with goal imagination?

• Can we use more realistic feedbacks? The source code of the experiments can be found at https://github.com/eleninisioti/ SAPIENS. origins framework presented in Chapter 3. Environmental complexity is modeled as a compositional environment with various elements that can be combined together to create new elements. Regarding adaptability we consider vanilla dqn agents, each learning their own action policy by interacting with the environment in order to maximize their cumulative reward. In terms of multi-agent dynamics, agents are placed in several groups and can regularly share their experience (by sharing transitions from their replay buffer) according to different social network topologies. We study how these topologies influence the ability of the agents to solve complex hierarchical innovation tasks, forming a cultural repertoire. This cultural repertoire in turn influences both adaptability and multi-agent dynamics through the sharing of experiences and innovations to other groups.

Relevance to the proposed origins framework

Abstract

The human cultural repertoire relies on innovation: our ability to continuously and hierarchically explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective accumulation and merging of previous solutions. Machine learning approaches commonly assume that fully connected multiagent networks are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic communication topologies [START_REF] Derex | Partial connectivity increases cultural accumulation within groups[END_REF]. In dynamic topologies, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of multi-agent topology on innovation has not been systematically studied in machine learning. It remains unclear a) which communication topologies are optimal for which innovation tasks, and b) which properties of experience sharing improve multi-level innovation. Here we use a multi-level hierarchical problem setting (WordCraft), with three different innovation tasks. We systematically design networks of DQNs sharing experiences from their replay buffers in varying topologies (fully connected, small world, dynamic, ring). Comparing the level of innovation achieved by different experience-sharing topologies across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic topology achieves the highest level of innovation across tasks. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, two metrics we propose, conformity and diversity of shared experience, can explain the success of different topologies on different tasks. These contributions can advance our understanding of optimal AI-AI, human-human, and human-AI collaborative networks, inspiring future tools for fostering collective innovation in large organizations.

Methods

We used a multi-level hierarchical innovation environment (WordCraft, [START_REF] Jiang | An environment for benchmarking commonsense agents[END_REF], illustrated Figure 7.6) with 3 innovation tasks. Each task had an initial set of elements (e.g. Earth, Water). Some elements can be combined to make new elements. Creating a new element moves player one innovation level higher and provides a reward.

We defined 3 innovation tasks (Figure 7.7):

Single innovation path. The recipe book consists of an initial set of 3 base elements (X valid = {a 1 , a 2 , a 3 }) and 8 innovation levels. To create the first element an agent needs to combine two of them. To progress further the agent needs to combine the most recently created element with an appropriate one from the initial set. This optimization problem contains a single global optimum.

Merging paths. There are two independent paths, A and B, and at level 2 there is a cross-road that presents the player with three options: moving forward on path A, moving forward on path B, or combining elements from path A and B to progress on path C. The latter path is more rewarding and is, thus, the optimal choice. This innovation task is particularly challenging: to explore path C the player needs to first go to lower innovation levels instead of just progressing on a single path. This task thus exhibits two local optimum (8 elements on A, 8 elements on B) and one global optimum (2 elements on path A + 2 elements on path B + 4 elements on path C).

Best-of-ten paths. Here, there are ten paths, one of which is the most rewarding. The optimal strategy is to move only on the most rewarding path but, to do so, the player must first explore and reject the other paths. This optimization task is characterized by a single global optimum and aims at evaluating the ability of agents to explore effectively in large spaces.

We systematically designed networks of dqn agents sharing experiences from their replay buffers in varying communication topologies: fully connected, small world, ring and dynamic (Figure 7.8). We call this architecture sapiens; for Structuring multi-Agent toPology for Innovation through ExperieNce Sharing.

Results

Comparing the level of innovation achieved by different experience-sharing topologies across different tasks shows that (Figure 7.9):

= = = = a 1 A 1 A 2 B 2 C 3 C 1 C 4 A 2 A 2 A 8 A 3 A 7 A 1 ...
= = = = j 1 J 1 J 2 J 2 J 8 J 3 J 7 J 1 ... j 2 j 3 j 1 j 3 x i i th base element in path X x i i th crafted element in path X
Reuse of a crafted element in a nonoptimal trajectory

Reuse of a crafted element in an optimal trajectory .9: Overall comparison of performances for the single path task (first column), merging paths task (second column) and best-of-ten paths task (third column). We present two metrics: S G %, of group success denotes the percentage of trials in which at least one agent in the group found the optimal solution (top row) and T + , Time to first success, which is the number of training time steps required (second for group success (bottom row). We compare the performance of our proposed social network topologies and baseline distributed rl algorithms (A2C and Ape-X).

Moreover, we propose metrics of shared experience:

• Conformity C t is a group-level metric that denotes the percentage of agents in a group that followed the same trajectory in a give evaluation trial

• Average Volatility V t is an agent-level metric that denotes the cumulative number of changes in the trajectory followed by an agent On the top row, the Average volatility (Vt) is a behavioral metric indicating the cumulative number of changes in the trajectory followed by an agent, averaged by all agents. On the bottom row, Group Diversity D G t is a mnemonic metric that captures the diversity of the aggregated group buffer.

Discussion

From tool use and language to music and mathematics, human innovation is characterized by cumulative solutions. Human studies have shown that different communication networks are best suited to solve different innovation tasks. However, most decentralized RL approaches assume static, often fully-connected networks. Here we test the hypothesis that the topology of experience sharing in a group of deep RL agents can shape its performance using our proposed algorithm sapiens. In line with human experiments, sapiens experiments show that dynamic topologies of experience sharing are best suited to solve complex innovation tasks. Our work extends upon the classical decentralized RL paradigm by explicitly measuring diversity, which has been associated with improved exploration [START_REF] Horgan | Distributed Prioritized Experience Replay[END_REF]Christianos et al., 2020), and demonstrating that certain multi-agent topologies are good at maintaining it, without requiring agents with different hyper-parameters.

Our study shows that both multi-agent network topology and task structure affect the performance of our proposed sapiens architecture. Based on our experimental results, we can provide general recommendations on which topology to use for which task class. The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks as in Gupta et al. (2019)). In this case, our experimental results show no benefit of experience sharing. The merging-path task exhibits strong local optima that have to be explored up to a certain point in order to discover the global optimum (in the spirit of hard exploration tasks as in [START_REF] Baker | VPT): Learning to act by watching unlabeled online videos[END_REF]; [START_REF] Ecoffet | First return, then explore[END_REF]). Our results show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths. The dynamic topology shows the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one. Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones (in the spirit of combinatorial optimization tasks as in [START_REF] Mazyavkina | Reinforcement learning for combinatorial optimization: A survey[END_REF]). In this case, our results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.

Limitations and future works We currently limit our experiments to symbolic innovation tasks implemented in Wordcraft. We believe that scaling up our study by applying sapiens in environments commonly employed by the multi-agent reinforcement learning community to study innovation [START_REF] Leibo | Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research[END_REF] is an important next step. In the appendix of the paper (Nisioti et al., 2022), we therefore provide results on a deceptive game task implemented in a grid world. While the present implementation considers groups of DQN agents, sapiens is in theory agnostic to the choice of off-policy algorithm. In future studies we plan to investigate sapiens networks with different types of RL agents. For instance, we could identify what network characteristics (e.g. centrality, brokering, bridging, etc) is best suited for which learning agents across different tasks. We also envision a meta-learning version of sapiens where an outer-loop optimizes the social network structure. Societal implications. Such future studies can have wide societal implications, paving the way toward predicting optimal social network structures for human-AI, AI-AI human-human cooperation networks. In an era of pending climate catastrophes and global pandemics, better understanding collaboration networks appears relevant. We believe sapiens can shed light on how to solve a wide range of innovation tasks and lead to future tools for fostering human and machine innovation in large organizations. Particular attention will however be required to prevent potential negative impact, where controlling the dynamics of social network topologies could result in unequal access to information and privacy issues.

Towards human-like open-ended skill acquisition through cultural feedback effects (Level-3 dynamics) by avoiding resets and ensuring environmental persistence across generations, allowing agents' behavior to modify their own niche and fitness landscape.

Iterated learning as a driver of niche construction

Non-episodic persistent environments proposed above, combined with multiscale dynamics, spatial open-endedness and compositional dynamics as proposed in Section 5.6.2.1, will enable the study of typical niche-constructive behaviors encountered in nature. For example, agents will be able to build some sort of shelters for the winter (keeping themselves warm and protecting resources), which then become part of the environment and create new constraints and opportunities for the next generations. It will enable the empirical evaluation of the recently proposed computational hypothesis that niche construction can be viewed as an iterated meta-learning process, where previous generations increase the saliency of the environment so that future agents interacting with it will encounter a more informative niche (Constant et al., 2018;[START_REF] Kirby | Iterated learning and the evolution of language[END_REF] with potential benefits for exploration and survival. • It structures an interdisciplinary dialog among these fields as well as a roadmap for progressing towards oesa in ai (Sections 3.3, 3.4, 3.5 and 3.6),

• It considers ai methods as computational tools to better understand the dynamics of the human species at multiple spatiotemporal scales (Section 3.7).

The origins framework is structured around three levels of dynamics (Figure 3.3), each associated with a main research question.

Level-1 dynamics: What are the ecological conditions favoring the evolution of autotelic agents with intrinsically motivated goal exploration strategies? The theoretical arguments related to this question are developed in Section 3.4. My contributions and future work on the topic are presented in Chapter 5. The main hypothesis at this level is that environmental complexity is the main driver of behavioral diversity, as proposed by recent theories in human behavioral ecology as well as ai experiments in e.g. meta reinforcement learning and curriculum learning. Multi-agent dynamics through feedback effects of co-adaptation also play an important role in promoting behavioral diversity at this level. These mechanisms contribute to the evolution of autotelic agents equipped with an intrinsic motivation to discover and explore new niches and skills as a solution to adapt to abrupt environmental changes.

Level-2 dynamics: What is the joint role of intrinsically motivated goal exploration and multi-agent dynamics in the formation of a cultural repertoire? The theoretical arguments related to this question are developed in Section 3.5. My contributions and future work on the topic are presented in Chapter 6. The main hypothesis at this level is that intrinsically motivated goal exploration coupled with varying cooperation and competition pressures (emerged from level-1 dynamics above) result in the acquisition of complex skills that are not necessarily linked to evolutionary fitness. These contribute to the formation of a cultural repertoire, i.e. an expanding collection of behaviors that are transmitted repeatedly through social or observational learning to become a population-level characteristic. These skills include technology, communication and social organization, originally of relatively low complexity.

Level-3 dynamics: What is the role of feedback effects from a cultural repertoire in the open-ended dynamics of human skill acquisition? The theoretical arguments related to this question are developed in Section 3.6. My contributions and future work on the topic are presented in Chapter 7. The main hypothesis at this level is that a cultural repertoire, when reaching a certain level of complexity, can bootstrap positive feedback loops continuously increasing the complexity of the environment, of the agents' cognitive abilities and of their multi-agent dynamics. These feedback loops, which are especially prominent in the human species due to the complexity of our cultural repertoire, are what make human skill acquisition truly open-ended.

In the discussion at the end of Chapter 3 (Section 3.7) I detail the interdisciplinary aspects and the limitations of the proposed origins framework.

In the longer term, the objective is to study the interactions between these levels in an integrated simulation environment. This will require the integration of (1) complex environmental dynamics inspired by paleo-climatology and human behavioral ecology, providing diverse constraints and opportunities through compositional dynamics (Section 5.6.2) ;

(2) artificial evolution able to generate morphogenetic and neural structures with diverse functionalities, including the potential evolution of autotelic agents (Section 5.6.2) (3) reciprocal influences between agent populations modifying their environmental niches (Section 6.4.2) ; (4) cross-generational influence mediated by environmental changes and agent's interactions as a bootstrap of cultural evolution (Section 7.4.2).

I believe that simulating these mechanisms is the most promising path toward achieving human-like open-ended skill acquisition in artificial systems. Beyond this perspective for computer science, I also believe that the origins framework can help us to better understand the dynamics of our own species within its eco-evolutionary and cultural context (see a humanist perpective on ai in Section 3.7). are presented with different observations (o i and o j ) of the same underlying environment state (s). Each agent (say i) implements a standard autoencoder architecture with an encoder enc i mapping the input observations to latent representations and a decoder dec i mapping latent representations to reconstructed observations. Each agent i communicates its latent representation m i (also called a message) of its own observation input o i to the other agents. This way, each agent i is able to reconstruct the observation from its own latent representation (o ii : reconstruction by agent i from its own message m i ) as well as from the latent representation of the other (o ij : reconstruction by agent i from the other's message m j ). The architecture can be trivially extended to a larger population, where each agent communicates their latent representations to with each other. . . . . . . . . . . . . . . . . . . . . . . . . 5.12 Representation quality in terms of standard linear probing and data efficiency.

List of Figures

AE+MTM and DTI are our proposed methods whereas AE is an autoencoding baseline which does not benefit from perspectives. Left: Classification accuracy using linear probing on top of the learned representations, comparing a condition with different perspectives (Yes) and without (No). Right: Linear probing using validation datasets of varying sizes to assess the data efficiency of representations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.13

Compositional dynamics as a key feature of simulation environments for oesa. An environment is said to support compositional dynamics when the agent can produce new elements (e.g. an axe) by composing other existing elements (e.g. wood and stone). On the left, Little Alchemy (https://littlealchemy.com/) is a game with compositional dynamics in a non-embodied setting. On the right, Krafter [START_REF] Hafner | Benchmarking the spectrum of agent capabilities[END_REF] where it self-generates its own goals in T and iteratively learns a mapping T → Π. C) When exploring autonomously how articulatory commands produce auditory effects, the model first produces non-speech sounds (no phonation), then unarticulated sounds and finally articulated sounds on the form of protosyllables. D) When the agent has the choice to either self-explore its vocal abilities or imitate sounds from an ambient language, it chooses to imitate only once it has acquired sufficient motor control over its vocal tract. This choice is made by sampling the strategy (self-exploration or imitation) where it observes maximal learning progress. E) These results are coherent with the main developmental stages observed in infant vocal development [START_REF] Kuhl | Early language acquisition: Cracking the speech code[END_REF]. 83 6.3 Relevance of the contribution to the origins framework presented in Chapter 3

. Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 86 6.4 The cosmo model. A. Schema of the speech communication situation. B.

The cognitive architecture of a cosmo agent is based on the internalization hypothesis. C. Bayesian model of a cosmo agent as a joint probability distribution. D and E. Production and perception are expressed as Bayesian inferences in the joint distribution. . . . . . . . . . . . . . . . . . . . . . . . . 87 6.5 A. Illustration of a deitic game, where two agents share their attention on the same object. The speaker agent produces a motor gesture M for that object using its production model (Figure 6.4.D). This produces in a sound wave S perceived by a listener agent, which infers the corresponding object using its perception model (Figure 6.4.E). B. Examples of two speech sound systems emerging from repeating many deictic games in an agent population with several objects, where each agent alternates between the speaker and the listener role in a random fashion. These sound systems are structurally coherent with the two main vowel systems used in human world language. For example, /i,a,u/ (top) and /i,e,a,o,u/ (bottom) are very common in human languages as well as in cosmo simulations (see Moulin-Frier et al., 2015a, for the full statistics of emerged phonological systems in cosmo). . . . . . . . . .

Contents 3 . 1

 31 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Existing approaches to open-ended skill acquisition in ai and the human species . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3 The origins framework . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.4 Level-1 dynamics: the eco-evolutionary origins of autotelic agents . . 44 3.5 Level-2 dynamics: the formation of a cultural repertoire . . . . . . . . 48 3.6 Level-3 dynamics: towards human-like open-ended skill acquisition through cultural feedback effects . . . . . . . . . . . . . . . . . . . . . 49 3.7 General conclusions and discussion . . . . . . . . . . . . . . . . . . . . 50

  new skills in order to solve them, in an open-ended fashion: what is called open-ended skill acquisition (oesa) (Figure 3.1). This process is open-ended both at the individual scale (each of us is able to continuously acquire new knowledge and skills) and at the socio-cultural scale (our societies generate an open-ended repertoire of cultural innovations). Oesa is part of what makes us truly humans, in the sense that it has enabled our most impressive achievements through science, art and technological innovation. It is however also resulting in the over-exploitation of natural resources and the misuse of technology, putting at risk our own survival as a species through the collapse of biodiversity and global warming (e.g. Cook et al., 2016).

Figure 3

 3 Figure 3.1: An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquire new skills in order to solve them: what is called open-ended skill acquisition (oesa). This ability results from heterogeneous mechanisms operating at multiple spatiotemporal scales: environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms.

Figure 3 . 2 :

 32 Figure 3.2: The proposed origins framework aims at Grounding Artificial Intelligence in the Origins of Human Behavior. It identifies central components (boxes) and their interactions (arrows) driving open-ended skill acquisition, both in terms of its evolution from environmental complexity (roughly: left to right arrows) as well its open-ended aspect through feedback mechanisms (right to left arrows). The employed terminology reflects a diversity of mechanisms considered in both ai and human behavioral ecology. See text for details.

  3 dynamics: What is the role of feedback effects from a cultural repertoire in the openended dynamics of human skill acquisition? .

Figure 3 . 3 :

 33 Figure 3.3: The origins framework proposes to distinguish three levels of dynamics. Each level focuses on specific interactions between the framework components and is associated with a main research question. Same conventions as in Figure 3.2.

  Level-3 dynamics: towards human-like open-ended skill acquisition through cultural feedback effects 49

  Plasticity and Evolvability Under Environmental Variability: the Joint role of Fitness-based Selection and Niche-limited Competition 59 5.2 Plasticity and Evolvability Under Environmental Variability: the Joint role of Fitness-based Selection and Niche-limited Competition 5.2.1 Context This work was realized in 2021-2022 in the Flowers team at Inria (France), in collaboration with Eleni Nisioti. It was partially funded by the Inria Exploratory action ORIGINS (https://www.inria.fr/en/origins) as well as the French National Research Agency (https://anr.fr/, project ECOCURL, Grant ANR-20-CE23-0006). This work also benefited from access to the HPC resources of IDRIS under the allocation 2020-[A0091011996] made by GENCI, using the Jean Zay supercomputer. This contribution has been published at the GECCO 2022 conference: Nisioti, E. and Moulin-Frier, C. Plasticity and evolvability under environmental variability: The joint role of fitness-based selection and niche-limited competition. In Proceedings of the 2022 Genetic and Evolutionary Computation Conference (GECCO 2022), 2022

Figure 5 .

 5 Figure 5.1 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Here we study the interactions between environmental complexity, adaptability and multi-agent dynamics within a minimal evo-eco-devo simulation model. We consider a population of agents that can adapt both at the developmental scale (through plasticity) and at the evolutionary scale (through fitness-based selection and niche-limited competition). Agents evolve in environments that varies from generation to generation and that are structured in multiple ecological niches with different capacities. We study how different types of environmental variability influences plasticity and evolvability under different selection mechanisms.

Figure 5 . 1 :

 51 Figure 5.1: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

Figure 5 .

 5 Figure 5.2: (Left) The latitudinal model we employ to describe how the environmental state varies across niches: a single climate function L (illustrated here as a sinusoidal curve) specifies the global environmental dynamics across niches, each niche n differing from its vertical offset equal to • n. Thus niches with higher index n have higher states, and therefore, higher capacity. (Right) Modeling plasticity as a normal distribution N (µ k , σ k ). A non-plastic individual (k) has small σ k and a high peak at their preferred niche, while a plastic individual (k ) has large σ k and a lower peak at their preferred niche. Fitness in a given niche n is computed as the probability density function of the distribution at the environmental state en. This figure also illustrates the cost and benefit of plasticity, assuming that µ k = µ k . If en = µ k (the actual environmental state is identical to the preferred niche of both individuals) the plastic individual has lower fitness (cost of plasticity).If the environmental state is different from the preferred state, e.g. e n > µ k , the plastic individual has higher fitness (benefit of plasticity).

Figure 5

 5 Figure 5.3: Population-average plasticity after convergence (σ * ) in a constant environment under NF-selection.

Figure 5 .

 5 Figure 5.3 presents the population-average plasticity after convergence, σ * , under NF-selection, various environmental conditions and number of niches. We observe that when there is a single niche (N = 1) plasticity converges to a very low value regardless of the state. This is intuitive as the cost of plasticity captured by our genome model renders individuals with the smallest σ * k the fittest. This agrees with previous studies in constant environments Grove (2014) (note that F-selection is identical with NFselection when there is a single niche). However, the picture differs significantly when there are multiple

  Plasticity and Evolvability Under Environmental Variability: the Joint role of Fitness-based Selection and Niche-limited Competition 63

Figure 5 . 4 :

 54 Figure 5.4: Results in a periodic environment. Left: Survival (A) as the percentage of generations without a mass extinction under NF-selection with genome R evolve , N =100 niches and varying period Te and amplitude Ae. Middle: Survival (A) as the percentage of generations without a mass extinction under NF-selection with genome R evolve , Ae = 4 and varying period Te and Number of niches. Right: Evolution under NF-selection with genome R evolve , N =100 niches, Te = 750 and Ae=8.

  Plasticity and Evolvability Under Environmental Variability: the Joint role of Fitness-based Selection and Niche-limited Competition 65

Figure 5 .

 5 Figure 5.5 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Here we entirely focus on environmental complexity, leaving the interaction with all the other components of the framework for future work. The presented environment aims at simulating some aspects of climate dynamics supposed to have played a key role in human behavioral ecology (Sections 3.2.2.1 and 3.4).

Figure 5 . 5 :

 55 Figure 5.5: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

Figure 5

 5 Figure 5.6: Climate dynamics in our proposed environment: (a) simplified model of the climate dynamics (b) temporal patterns of lake and item presence during simulations with a precipitation function having a pulse form (c) a top-view of a gridworld where an agent navigates in a grid-world populated by lakes (green), jelly-beans (purple), bananas (yellow) and trees (green), whose presence is influenced by a user-designed precipitation function during a low-precipitation period (c) and a high-precipitation period (d)

Figure 5

 5 Figure 5.7 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. This contribution adopts a radical enactive view of cognition where, contrarily to all the other contributions presented in this thesis, there is originally no distinction between agents and the environment. Instead, we use a multi-channel continuous cellular automaton where the local rules governing some channels are fixed, modeling an imposed environmental dynamics, while the rules governing other channels can be adapted according to some objective. We study how such a self-organizing system, under some conditions of environmental complexity, can evolve low-level sensorimotor skills from scratch (adaptability). Interestingly we observe that the evolved proto-agents (called creatures in the paper), while trained in isolation for spatial localization and navigation skills, show impressive generalization abilities to unseen environmental conditions. In particular, rich multi-agent dynamics emerges when multiple creatures are simultaneously placed in a shared environment at test time.

Figure 5 . 7 :

 57 Figure 5.7: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

  steps to optimize parameters and initialization Evaluate Creature channel compared with the target shape (placed at target location). Compute the MSE and backpropagate

Figure 5

 5 Figure 5.9 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Here we study how particular multiagent dynamics, where agents do not have access to the observations of others but can communicate within a limited range, guarantees a common context that can be leveraged in individual representation learning to improve adaptability. In the presented experiments, environmental complexity is highly simplified.

  Socially Supervised Representation Learning, a new learning paradigm for unsupervised learning of efficient representations in a multi-agent setup (Figure 5.11).

Figure 5 .

 5 Figure 5.10: Data augmentation in self-supervised representation learning using stochastic image augmentations t ∼ T (top) vs. Socially Supervised Representation Learning that substitutes engineered augmentations for perspectives (oi, oj, . . . ) that arise naturally in multi-agent systems (bottom).Green boxes indicate conceptual agents, while we assume that a singular representation learning method may be interpreted as a single agent.

Figure 5 .

 5 Figure 5.12: Representation quality in terms of standard linear probing and data efficiency. AE+MTM and DTI are our proposed methods whereas AE is an autoencoding baseline which does not benefit from perspectives. Left: Classification accuracy using linear probing on top of the learned representations, comparing a condition with different perspectives (Yes) and without (No). Right: Linear probing using validation datasets of varying sizes to assess the data efficiency of representations.

Figure 6 .

 6 Figure6.1 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Here we consider an adaptive agent equipped with a general mechanism of intrinsically-motivated goal exploration (i.e. adaptability at the developmental scale). This agent is equipped with a model of the human vocal tract and of the auditory system. In a first experiment, there is no environmental complexity (except the agent's own morphology, i.e. its vocal tract producing sounds) and no multi-agent dynamics. We study how a general mechanism of intrinsically motivated spontaneous exploration can self-organize developmental stages during early vocal learning, without any imposed constraint to communicate. We observe that the agent spontaneously produces vocalizations of increasing complexity (from no phonation, to unarticulated sounds, to articulated sounds resembling proto-syllables in infants). Then, in a second experiment, the agent also perceives mature vocalizations from surrounding "adult" agents (hence introducing very basic multi-agent dynamics) and we study how it influences the agent's exploration toward the ambient sounds. We argue that such developmental structures, self-organizing from general mechanisms of intrinsically-motivated learning, can provide the building blocks in the emergence of a cultural repertoire (here providing a repertoire of diverse vocalizations that can potentially bootstrap language evolution and acquisition, see[START_REF] Oudeyer | How Evolution May Work Through Curiosity-Driven Developmental Process[END_REF], for a complete explanation of the theoretical argument).

  Figure 6.1: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

  Figure 6.2: The model is able to reproduce the main stages of the developmental sequence observed in infant vocal development. A) The agent produces vocalizations through a realistic model of the human vocal tract.B) The agent learns an inverse model from a goal space T of desired auditory trajectories, to the motor trajectories in Π required to achieve those goals. For this aim it uses a generic intrinsically motivated goal exploration process (imgep, see Section 3.2.1.2) where it self-generates its own goals in T and iteratively learns a mapping T → Π. C) When exploring autonomously how articulatory commands produce auditory effects, the model first produces non-speech sounds (no phonation), then unarticulated sounds and finally articulated sounds on the form of proto-syllables. D) When the agent has the choice to either self-explore its vocal abilities or imitate sounds from an ambient language, it chooses to imitate only once it has acquired sufficient motor control over its vocal tract. This choice is made by sampling the strategy (self-exploration or imitation) where it observes maximal learning progress. E) These results are coherent with the main developmental stages observed in infant vocal development[START_REF] Kuhl | Early language acquisition: Cracking the speech code[END_REF].

  cosmo: A Bayesian modeling framework for studying speech communication and the emergence of phonological systems 856.3 cosmo: A Bayesian modeling framework for studying speech communication and the emergence of phonological systems6.3.1 ContextThis work was realized in 2007-2012 during my PhD thesis at GIPSA-Lab at Grenoble University (France), in collaboration with Jean-Luc Schwartz, Julien Diard and Pierre Bessière. It was partially funded by the French ministry of research.The main publication presenting this contribution has been published as the target article of a Special Issue in Journal of Phonetics:Moulin-Frier, C., Diard, J., Schwartz, J.-L., and Bessière, P. COSMO ('Communicating about Objects using Sensory-Motor Operations'): A Bayesian modeling framework for studying speech communication and the emergence of phonological systems. Journal of Phonetics, 53:5-41, 2015a5-41, . ISSN 00954470. doi: 10.10165-41, /j.wocn.2015.06.001 .06.001

  Figure 6.3: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

Figure 6

 6 Figure 6.4: The cosmo model. A. Schema of the speech communication situation. B. The cognitive architecture of a cosmo agent is based on the internalization hypothesis. C. Bayesian model of a cosmo agent as a joint probability distribution. D and E. Production and perception are expressed as Bayesian inferences in the joint distribution.

  .4.C), which can be interpreted as the structure of a Bayesian Network model. (The cosmo acronym, for "Communicating about Objects using Sensory-Motor Operations", also refers the five variables of the model: C, O S , S, M, O L .) An agent with this Bayesian architecture possesses a model of the entire communication situation and is therefore able to perform both production (Figure 6.4.D) and perception (Figure 6.4.E) tasks as Bayesian inferences (P (M | O S C = 1) and (P (O L | S C = 1), respectively) on the joint distribution.

  Figure6.5: A. Illustration of a deitic game, where two agents share their attention on the same object. The speaker agent produces a motor gesture M for that object using its production model (Figure6.4.D). This produces in a sound wave S perceived by a listener agent, which infers the corresponding object using its perception model (Figure6.4.E). B. Examples of two speech sound systems emerging from repeating many deictic games in an agent population with several objects, where each agent alternates between the speaker and the listener role in a random fashion. These sound systems are structurally coherent with the two main vowel systems used in human world language. For example, /i,a,u/ (top) and /i,e,a,o,u/ (bottom) are very common in human languages as well as in cosmo simulations (seeMoulin- Frier et al., 2015a, for the full statistics of emerged phonological systems in cosmo).

  3, I present a recent contribution studying the dynamics of open-ended technological innovation in groups of adaptive agents sharing experiences with each other. Finally, in Section 7.4, I mention my other contributions on this topic and propose concrete future work.

7. 2

 2 Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration 7.2.1 Context This work was realized in 2020-2021 in the Flowers group at Inria (France), in collaboration with Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Peter Dominey, Pierre-Yves Oudeyer. Cédric Colas and Tristan Karch were partly funded by the French Ministère des Armées -Direction Générale de l'Armement. Nicolas Lair was supported by ANRT/CIFRE contract No. 151575A20 from Cloud Temple. The main publication presenting this contribution has been published at the NeurIPS 2020 conference: Colas, C., Karch, T., Lair, N., Moulin-Frier, C., Dussoux, J.-M., Dominey, P. F., and Oudeyer, P.-Y. Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration. In Advances in Neural Information Processing Systems (NeurIPS 2020), 2020

Figure 7 .

 7 Figure 7.1 provides a visual representation of the relevance of this contribution to the origins framework presented in Chapter 3. Environmental complexity corresponds here to the compositional dynamics of the Playground environment. Multi-agent dynamics is limited to the learning agent perceiving the utterances from the social

Figure 7 . 1 :

 71 Figure 7.1: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

Figure 7

 7 Figure 7.2: imagine overview. In the Playground environment, the agent (hand) can move, grasp objects and grow some of them. Scenes are generated procedurally with objects of different types, colors and sizes. A social partner provides descriptive feedback (orange), that the agent converts into targetable goals (red bubbles).

Figure 7

 7 Figure 7.3: imagine architecture. Colored boxes show the different modules of imagine. Lines represent update signals (dashed) and function outputs (plain). The language encoder Le is shared.

Figure 7 . 4 :

 74 Figure 7.4: Goal imagination drives exploration and generalization. Vertical dashed lines mark the onset of goal imagination. A) Success rate on testing set depending on when phase 2 starts (early, half-way or never. B) Behavioral adaptation, empirical probabilities that the agent brings supplies to a plant when trying to grow it. C) Evaluation metrics (ic) computed on the testing set. Stars indicate significance (A and C are tested against never ).

Figure 7 .

 7 4.C presents the ic metric (interesting interaction count, see Colas et al. (2020) for detail) computed on the set of interactions related to G test and demonstrates the exploration boost triggered by goal imagination.

Figure 7 .

 7 Figure 7.5 provides a visual representation of the relevance of this contribution to the

Figure 7 . 5 :

 75 Figure 7.5: Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution.

Figure 7

 7 Figure 7.6: (Left) Illustration of an innovation task: the task consists of an initial set of elements (Earth, Water) and a recipe book that determines which element combinations create new elements. Some elements, such as Earth + Mud, cannot be combined. Upon creating a new element the player moves one innovation level higher and receives a reward that increases monotonically with levels. (Right) Dynamic social network structures oscillate between phases of low connectivity, where experience sharing takes place within clusters, and high connectivity, where experiences spread between clusters.

  Towards human-like open-ended skill acquisition through cultural feedback effects (Level-

Figure 7 . 7 :

 77 Figure 7.7: We evaluate our algorithm on three innovation tasks called single path, merging paths and best-of-ten paths. Each task contains one or more paths, labeled by an uppercase letter (A to J). Each path X has its own initial set of three base elements {x1, x2, x3}, which are represented in dashed circles. Crafted elements in path X are represented in upper case (Xi) in solid circles. Optimal trajectories for each tasks are represented by solid red arrows, with their corresponding reward in bold red.

Figure 7 Figure 7

 77 Figure 7.8: (Left) Visualizing communication graphs (a) fully-connected (b) small-world (c) ring (d) dynamic. (Right) Schematic of two neighboring DQNs sharing experiences: agent 1 shares experiences from its own replay buffer to that of agent 2 (red arrow) and vice versa (blue arrow) while both agents are independently collecting experiences by interacting with their own copy of the environment.

Figure 7 .Figure 7 .

 77 Figure 7.10: Analyzing group behavior in the single path task:: (left) Conformity Ct is a behavioral metric that denotes the percentage of agents in a group that followed the same trajectory in a given evaluation trial (right) Average Diversity Dt is a mnemonic metric that denotes the number of unique experiences in the replay buffer of an agent, averaged over all agents

Chapter 8 Conclusion

 8 In this thesis, I have first detailed my scientific trajectory in Chapter 2, from the start of my PhD thesis in 2007 to my recruitment as a permanent researcher in the Flowers group at Inria in 2019. Then in Chapter 3, I have developed in detail the current research program I have initiated since my recruitment as a permanent researcher. It relies on the proposition of a novel conceptual framework, called origins. This framework aims at Grounding Artificial Intelligence in the Origins of Human Behavior through the modeling of complex interactions between environmental, adaptive, multi-agent and cultural dynamics. In particular:• The framework analyzes relationships between different fields studying open-ended skill acquisition (oesa) in humans (in particular human behavioral ecology) and in ai (Sections 3.2 and 3.3.1),

3. 1

 1 An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquire new skills in order to solve them: what is called open-ended skill acquisition (oesa). This ability results from heterogeneous mechanisms operating at multiple spatiotemporal scales: environmental, evolutionary, morphological, sensorimotor, developmental, cognitive, social and cultural mechanisms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 The proposed origins framework aims at Grounding Artificial Intelligence in the Origins of Human Behavior. It identifies central components (boxes) and their interactions (arrows) driving open-ended skill acquisition, both in terms of its evolution from environmental complexity (roughly: left to right arrows) as well its open-ended aspect through feedback mechanisms (right to left arrows). The employed terminology reflects a diversity of mechanisms considered in both ai and human behavioral ecology. See text for details. . . 3.3 The origins framework proposes to distinguish three levels of dynamics. Each level focuses on specific interactions between the framework components and is associated with a main research question. Same conventions as in Figure 3.2.5.1 Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . .5.2 (Left)The latitudinal model we employ to describe how the environmental state varies across niches: a single climate function L (illustrated here as a sinusoidal curve) specifies the global environmental dynamics across niches, each niche n differing from its vertical offset equal to • n. Thus niches with higher index n have higher states, and therefore, higher capacity. (Right) Modeling plasticity as a normal distribution N (µ k , σ k ). A non-plastic individual (k) has small σ k and a high peak at their preferred niche, while a plastic individual (k ) has large σ k and a lower peak at their preferred niche. Fitness in a given niche n is computed as the probability density function of the distribution at the environmental state e n . This figure also illustrates the cost and benefit of plasticity, assuming that µ k = µ k . If e n = µ k (the actual environmental state is identical to the preferred niche of both individuals) the plastic individual has lower fitness (cost of plasticity). If the environmental state is different from the preferred state, e.g. e n > µ k , the plastic individual has higher fitness (benefit of plasticity). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Population-average plasticity after convergence (σ * ) in a constant environment under NF-selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Results in a periodic environment. Left: Survival (A) as the percentage of generations without a mass extinction under NF-selection with genome R evolve , N =100 niches and varying period T e and amplitude A e . Middle: Survival (A) as the percentage of generations without a mass extinction under NF-selection with genome R evolve , A e = 4 and varying period T e and Number of niches. Right: Evolution under NF-selection with genome R evolve , N =100 niches, T e = 750 and A e =8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 5.6 Climate dynamics in our proposed environment: (a) simplified model of the climate dynamics (b) temporal patterns of lake and item presence during simulations with a precipitation function having a pulse form (c) a top-view of a gridworld where an agent navigates in a grid-world populated by lakes (green), jelly-beans (purple), bananas (yellow) and trees (green), whose presence is influenced by a user-designed precipitation function during a low-precipitation period (c) and a high-precipitation period (d) . . . . . . . . . . . . . . . . . 5.7 Relevance of the contribution to the origins framework presented in Chapter 3. Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 5.8 A. We use a multi-channel continuous cellular automaton (Lenia). The state A t at time t is a square grid with two colored channels: an Obstacle channel in blue and a Learnable channel in yellow. Each channel is associated with a set of convolution kernel filters and growth functions, whose outputs are summed up to compute of next step A t+1 . B) Parameters of kernels and growth functions are optimized using backpropagation through time, minimizing a L2 loss between the final state of the learnable channel and a target shape at a specific location. Target locations and obstacle positions are randomized across episodes. C) Optimized creatures in the Learnable channel are able to maintain their integrity while navigating in a field of obstacles (top left). They also generalize their skills in environments not encountered during training, e.g. when placed in multiple-creature environments or when damaged (top-right and bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 5.10 Data augmentation in self-supervised representation learning using stochastic image augmentations t ∼ T (top) vs. Socially Supervised Representation Learning that substitutes engineered augmentations for perspectives (o i , o j , . . . ) that arise naturally in multi-agent systems (bottom). Green boxes indicate conceptual agents, while we assume that a singular representation learning method may be interpreted as a single agent. . . . . . . . . . . . . . . . . . . 5.11 Proposed architecture. Two agents i and j (top and bottom coloured boxes)

7. 1

 1 Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 7.2 imagine overview. In the Playground environment, the agent (hand) can move, grasp objects and grow some of them. Scenes are generated procedurally with objects of different types, colors and sizes. A social partner provides descriptive feedback (orange), that the agent converts into targetable goals (red bubbles). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 imagine architecture. Colored boxes show the different modules of imagine. Lines represent update signals (dashed) and function outputs (plain). The language encoder L e is shared. . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Goal imagination drives exploration and generalization. Vertical dashed lines mark the onset of goal imagination. A) Success rate on testing set depending on when phase 2 starts (early, half-way or never. B) Behavioral adaptation, empirical probabilities that the agent brings supplies to a plant when trying to grow it. C) Evaluation metrics (ic) computed on the testing set. Stars indicate significance (A and C are tested against never ). . . . . . . 7.5 Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 7.6 (Left) Illustration of an innovation task: the task consists of an initial set of elements (Earth, Water) and a recipe book that determines which element combinations create new elements. Some elements, such as Earth + Mud, cannot be combined. Upon creating a new element the player moves one innovation level higher and receives a reward that increases monotonically with levels. (Right) Dynamic social network structures oscillate between phases of low connectivity, where experience sharing takes place within clusters, and high connectivity, where experiences spread between clusters. . . . . . . . . .

  Plasticity and Evolvability Under Environmental Variability: the Joint role of Fitness-based Selection and Niche-limited Competition 61

	...	...
	Env. state	, Generations
	Env. state	, Generations	Fitness	of Benefit Cost of plasticity
				plasticity
	Env. state	, Generations	Environmental state	Environmental state
	...	...

  This contribution has been published at the Ecological Theory of Reinforcement Learning workshop of the Conference on Neural Information Processing Systems (NeurIPS 2021): Nisioti, E., Jodogne-del Litto, K., and Moulin-Frier, C. Grounding an Ecological Theory of Artificial Intelligence in Human Evolution. In NeurIPS 2021 -Conference on Neural Information Processing Systems / Workshop: Ecological Theory of Reinforcement Learning, virtual event, France, 2021

  Hamon, G., Etcheverry, M., Chan, B. W.-C., Moulin-Frier, C., and Oudeyer, P.-Y.

	Learning	Sensorimotor	Agency	in	Cellular	Automata	(blog	post:
	https://developmentalsystems.org/sensorimotor-lenia/), 2022

  This work was realized in 2020-2022 as a collaboration between the Flowers group at Inria (France) and Microsoft Research NYC (USA), in collaboration with Eleni Nisioti, Mateo Mahaut, Pierre-Yves Oudeyer and Ida Momennejad. This research was partially funded by the Inria Exploratory action ORIGINS (https://www.inria.fr/en/origins) as well as the French National Research Agency (https://anr.fr/, project ECOCURL, Grant ANR-20-CE23-0006). This work also benefited from access to the HPC resources of IDRIS under the allocation 2020-[A0091011996] made by GENCI, using the Jean Zay supercomputer.The main publication presenting this contribution has been recently submitted to a major ai conference and is available as a preprint:Nisioti, E., Mahaut, M., Oudeyer, P.-Y., Momennejad, I., and Moulin-Frier, C. Social Network Structure Shapes Innovation: Experience-sharing in RL withSAPIENS, 2022 Open access version of this publication is available at https://arxiv.org/abs/2206. 05060. The rest of this section is a summary of this paper, part of it being adapted from a Twitter thread by Ida Momennejad: https://twitter.com/criticalneuro/status/ 1540004222858657792.

	7.3 Social Network Structure Shapes Innovation: Experience-
	sharing in RL with sapiens
	7.3.1 Context

•

  Average Diversity D t (mnemonic metric): number of unique experiences in the replay buffer of an agent, averaged over all agents.

	Conformity, C t	0.4 0.6 0.8 1.0	0	2 Training step, t 4	6	1e5	t Average Diversity, D	0 100 200 300 400	0	2 Training step, t 4 no-sharing 6 dynamic fully-connected 1e5 ring small-world
	These metrics can explain the success of different topologies on different tasks (Fig-
	ures 7.10 and 7.11).

  Towards human-like open-ended skill acquisition through cultural feedback effects (Level-3 dynamics)

  , a simulation environment inspired from Minecraft where agents encounter elements upon navigating. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.1 Relevance of the contribution to the origins framework presented in Chapter 3 . Low, medium or high opacity indicates the respective importance of the components (boxes) and their interactions (arrows) in this contribution. . . . 82 6.2 The model is able to reproduce the main stages of the developmental sequence observed in infant vocal development. A) The agent produces vocalizations through a realistic model of the human vocal tract. B) The agent learns an inverse model from a goal space T of desired auditory trajectories, to the motor trajectories in Π required to achieve those goals. For this aim it uses a generic intrinsically motivated goal exploration process (imgep, see Section 3.2.1.2)

http://www.gipsa-lab.grenoble-inp.fr/en/home.php

Scientific trajectory

https://www.college-de-france.fr/site/en-college/index.htm

https://flowers.inria.fr/

https://specs-lab.com/

http://wysiwyd.specs-lab.com/

https://cogitai.com/

https://www.inria.fr/en/inrias-exploratory-actions-taking-risks

https://www.inria.fr/en/origins-grounding-artificial-intelligence-origins-humanbehaviour

https://chakazul.github.io/

https://evocraft.life/

Towards human-like open-ended skill acquisition through cultural feedback effects (Level-3 dynamics)

Acknowledgments

To my wife, Jasmine Garside, the most fascinating human I have ever met.

§ What You Say Is What You Did, WYSIWYD project (FP7 ICT 612139) § Socialising Sensori-Motor Contingencies, socSMC project (641321-H2020FETPROACT-2014). § Role of Consciousness in Adaptive Behavior ERC's CDAC project (ERC-2013ADG 341196) § Permanent position at Inria funded by the French government § Inria Exploratory Action ORIGINS (2020-2022): Grounding AI in the Origins of Human Behavior § French National Research Agency (ANR) (2021-2025): Emergent Communication through Curiosity-driven Multi-Agent Reinforcement Learning 2020 -2021 MASTER PROJECT SUPERVISION

Context

This work was realized in 2021-2022 in the Flowers team at Inria (France), in collaboration with Julius Taylor and Eleni Nisioti. It was partially funded by the Inria Exploratory action ORIGINS (https://www.inria.fr/en/origins) as well as the French National Research Agency (https://anr.fr/, project ECOCURL, Grant ANR-20-CE23-0006). This work also benefited from access to the HPC resources of IDRIS under the allocation 2020-[A0091011996] made by GENCI, using the Jean Zay supercomputer. This contribution has been published at the AAMAS 2022 conference: Taylor, J., Nisioti, E., and Moulin-Frier, C. Socially Supervised Representation Learning: The Role of Subjectivity in Learning Efficient Representations. In International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2022), 2022

Open access version of this publication is available at https://arxiv.org/abs/2109. 09390. The rest of this section is a summary of this paper.

Chapter 6

The formation of a cultural repertoire (Level-2 dynamics)

Introduction

In this chapter, I present some of my contributions and propose future work in modeling the formation of a cultural repertoire. As presented in Section 3.5, this part of the proposed framework considers pre-existing mechanisms for adaptability (emerged from level-1 dynamics, in particular intrinsically motivated goal exploration) as well as for multi-agent dynamics (e.g. cooperation and competition pressures with other agents in a shared environment). The objective is to study how the interaction between these mechanisms can bootstrap the formation of a cultural repertoire, from its most primitive forms (e.g. structured vocalizations) to more advanced ones (e.g. phonological systems for vocal communication). The main research question at this level is: What is the joint role of intrinsically motivated goal exploration and multi-agent dynamics in the formation of a cultural repertoire?

In Section 6.2, I present a computational contribution showing how some aspects of early vocal development, required for articulated language to emerge, can self-organize out of general mechanisms of intrinsically motivated goal exploration. Then, in Section 6.3, I present a contribution showing how shared speech sound systems can emerge at the population level through the interaction of vocal agents. Finally, in Section 6.4, I mention my other contributions on this topic and propose concrete future work.

The formation of a cultural repertoire (Level-2 dynamics) 6.4 Other contributions and future work

Other contributions

Studying the formation of a cultural repertoire (i.e. level-2 dynamics) has occupied a large part of my scientific carrier, especially between 2007 and 2015. In addition to the two contributions presented in this chapter, my other contributions on the topic include:

• Emergence of shared sensory-motor graphical language from visual input [START_REF] Lemesle | Emergence of Shared Sensory-motor Graphical Language from Visual Input[END_REF],

• Emergent communication as a solution to the architect-builder problem [START_REF] Barde | Learning to Guide and to Be Guided in the Architect-Builder Problem[END_REF],

• Modeling the formation of social conventions from embodied real-time interactions [START_REF] Freire | Modeling the formation of social conventions from embodied real-time interactions[END_REF]),

• Self-organization of turn-taking behavior in rl agent populations (Moulin-Frier et al., 2015b),

• A unified probabilistic framework (Moulin-Frier & Oudeyer, 2013a) and an opensource library (Moulin-Frier et al., 2014b) for exploration strategies in developmental robotics,

• Cognitive architectures for speech production and perception (Moulin-Frier et al., 2012;[START_REF] Moulin-Frier | Recognizing speech in a novel accent: The motor theory of speech perception reframed[END_REF],

• Distinguishing self, other, and autonomy from visual feedback [START_REF] Demirel | Distinguishing Self, Other, and Autonomy From Visual Feedback: A Combined Correlation and Acceleration Transfer Analysis[END_REF],

• Theoretical propositions on the origins of communication (Moulin-Frier & Oudeyer, 2021;Moulin-Frier & Verschure, 2016) and consciousness [START_REF] Arsiwalla | The Morphospace of Consciousness[END_REF] in animals and machines.

Future work

Level-2 dynamics considers pre-existing mechanisms for adaptability (emerged from level-1 dynamics, for example learning and exploration) as well as for multi-agent dynamics (e.g. cooperation and competition pressures with other agents in a shared environment). The objective is to study how the interaction between these mechanisms can bootstrap the formation of a cultural repertoire. The main research question at this level is: What is the joint role of intrinsically motivated goal exploration and multi-agent dynamics in the formation of a cultural repertoire?

Based on the theoretical analysis of level-2 dynamics in Section 3.5 as well as my existing contributions presented in the current chapter, future work will focus on the following directions.

Other contributions and future work

Other contributions

Some of my existing contributions on level-2 dynamics presented in the last chapter (Chapter 6) also consider, to some extent, aspects of level-3 dynamics, i.e. of feedback effects from a cultural repertoire on other components of the system. It is in particular the case of most contributions studying the emergence of proto-cultural structures such as communication systems [START_REF] Lemesle | Emergence of Shared Sensory-motor Graphical Language from Visual Input[END_REF][START_REF] Barde | Learning to Guide and to Be Guided in the Architect-Builder Problem[END_REF]Moulin-Frier et al., 2015b) and social norms [START_REF] Freire | Modeling the formation of social conventions from embodied real-time interactions[END_REF] in agent populations. In these contributions, emerging proto-cultural structures necessarily influence adaptability and multi-agent dynamics. This illustrates the strong coupling between the three levels of dynamics proposed in the origins framework.

In addition to these contributions and the two main ones presented in this chapter, my other contributions on the topic also include:

• A recent position paper on Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (Colas et al., 2022a),

• Grounding spatio-temporal language with transformers (Karch et al., 2021),

• Language acquisition in human-robot interaction (Moulin-Frier et al., 2018),

• Epidemioptim: A toolbox for the optimization of control policies in epidemiological models (Colas et al., 2021).

Future work

Level-3 dynamics is interested in the feedback effects from a cultural repertoire to all the other components of the system. The main research question at this level is: What is the role of feedback effects from a cultural repertoire in the open-ended dynamics of human skill acquisition? .

Based on the theoretical analysis of level-3 dynamics in Section 3.6 as well as my existing contributions presented in the current chapter, future work will focus on the following directions.

Non-episodic persistent environments

Most existing environments used in rl relies on an episodic training paradigm, where the state of the environment is regularly reset to an initial configuration, either when the agent has solved the task or after a predefined timeout (Section 3.4.1). This prevents the study of niche construction effects as proposed in Section 3.6, where adaptability, multi-agent dynamics and the cultural repertoire all feed back to environmental complexity, bootstrapping a positive feedback loop driving the open-ended aspect of human-like skill acquisition. Studying the role of niche construction mechanisms in oesa therefore requires the design of simulation environments that enable niche construction 7.7 We evaluate our algorithm on three innovation tasks called single path, merging paths and best-of-ten paths. Each task contains one or more paths, labeled by an uppercase letter (A to J). Each path X has its own initial set of three base elements {x 1 , x behavioral metric that denotes the percentage of agents in a group that followed the same trajectory in a given evaluation trial (right) Average Diversity Dt is a mnemonic metric that denotes the number of unique experiences in the replay buffer of an agent, averaged over all agents . . . . . . . . . . . . . . . . 104 7.11 Analyzing group behavior in the single path task (first column), merging paths task (second column) and best-of-10 paths task (third column). On the top row, the Average volatility (V t ) is a behavioral metric indicating the cumulative number of changes in the trajectory followed by an agent, averaged by all agents. On the bottom row, Group Diversity D G t is a mnemonic metric that captures the diversity of the aggregated group buffer. . . . . . . . . . . . 104