Cédric Colas
email: cedric.colas@inria.fr

Tristan Karch
email: tristan.karch@inria.fr

Olivier Sigaud
email: olivier.sigaud@upmc.fr

Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey

Building autonomous machines that can explore open-ended environments, discover possible interactions and autonomously build repertoires of skills is a general objective of artificial intelligence. Developmental approaches argue that this can only be achieved by autonomous and intrinsically motivated learning agents that can generate, select and learn to solve their own problems. In recent years, we have seen a convergence of developmental approaches, and developmental robotics in particular, with deep reinforcement learning (rl) methods, forming the new domain of developmental machine learning. Within this new domain, we review here a set of methods where deep rl algorithms are trained to tackle the developmental robotics problem of the autonomous acquisition of open-ended repertoires of skills. Intrinsically motivated goal-conditioned rl algorithms train agents to learn to represent, generate and pursue their own goals. The self-generation of goals requires the learning of compact goal encodings as well as their associated goal-achievement functions, which results in new challenges compared to traditional rl algorithms designed to tackle pre-defined sets of goals using external reward signals. This paper proposes a typology of these methods at the intersection of deep rl and developmental approaches, surveys recent approaches and discusses future avenues.

Introduction

Building autonomous machines that can explore large environments, discover interesting interactions and learn open-ended repertoires of skills is a long-standing goal in Artificial Intelligence. Humans are remarkable examples of this lifelong, open-ended learning. They learn to recognize objects and crawl as infants, then learn to ask questions and interact with peers as children. Across their lives, humans build a large repertoire of diverse skills from a virtually infinite set of possibilities. What is most striking, perhaps, is their ability to invent and pursue their own problems, using internal feedback to assess completion. We would like to build artificial agents able to demonstrate equivalent lifelong learning abilities.

We can think of two approaches to this problem: developmental approaches, in particular developmental robotics, and reinforcement learning (rl). Developmental robotics takes inspirations from artificial intelligence, developmental psychology and neuroscience to model cognitive processes in natural and artificial systems [Asada et al., 2009, Cangelosi and[START_REF] Cangelosi | Developmental robotics: From babies to robots[END_REF]. Following the idea that intelligence should be embodied, robots are often used to test learning models. Reinforcement learning, on the other hand, is the field interested in problems where agents learn to behave by experiencing the consequences of their actions under the form of rewards and costs. As a result, these agents are not explicitly taught, they need to learn to maximize cumulative rewards over time by trialand-error [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. While developmental robotics is a field oriented towards answering particular questions around sensorimotor, cognitive and social development (e.g. how can we model language acquisition?), reinforcement learning is a field organized around a particular technical framework and set of methods. Now powered by deep learning optimization methods leveraging the computational efficiency of large computational clusters, rl algorithms have recently achieved remarkable results including, but not limited to, learning to solve video games at a super-human level [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], to beat chess and go world players [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF], or even to control stratospheric balloons in the real world [START_REF] Bellemare | Autonomous navigation of stratospheric balloons using reinforcement learning[END_REF]. Although standard rl problems often involve a single agent learning to solve a unique task, rl researchers recently extended rl problems to multi-goal rl problems. Instead of pursuing a single goal, agents can now be trained to pursue goal distributions [START_REF] Kaelbling | Learning to achieve goals[END_REF][START_REF] Sutton | Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction[END_REF][START_REF] Schaul | Universal value function approximators[END_REF]. As the field progresses, new goal representations emerge: from the specific goal states to the high-dimensional goal images or the abstract language-based goals [START_REF] Loynd | Working memory graphs[END_REF]. However, most approaches still fall short of modeling learning abilities of natural agents because they build agents that learn to solve predefined sets of tasks, via external and hand-defined learning signals.

Developmental robotics directly aims to model children learning and, thus, takes inspiration from the mechanisms underlying autonomous behaviors in humans. Most of the time, humans are not motivated by external rewards but spontaneously explore their environment to discover and learn about what is around them. This behavior seems to be driven by intrinsic motivations (ims) a set of brain processes that motivate humans to explore for the mere purpose of experiencing novelty, surprise or learning progress [START_REF] Berlyne | Curiosity and exploration[END_REF][START_REF] Gopnik | The scientist in the crib: Minds, brains, and how children learn[END_REF][START_REF] Kidd | The psychology and neuroscience of curiosity[END_REF][START_REF] Oudeyer | How evolution may work through curiosity-driven developmental process[END_REF][START_REF] Gottlieb | Towards a neuroscience of active sampling and curiosity[END_REF]. The integration of ims into artificial agents thus seems to be a key step towards autonomous learning agents [Schmidhuber, 1991, Kaplan andOudeyer, 2007]. In developmental robotics, this approach enabled sample efficient learning of high-dimensional motor skills in complex robotic systems [START_REF] Santucci | Intrinsically motivated openended learning in autonomous robots[END_REF], including locomotion [Baranes andOudeyer, 2013b, Martius et al., 2013], soft object manipulation [Rolf andSteil, 2013, Nguyen and[START_REF] Nguyen | Socially guided intrinsic motivation for robot learning of motor skills[END_REF], visual skills [START_REF] Lonini | Robust active binocular vision through intrinsically motivated learning[END_REF] and nested tool use in real-world robots [START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF]. Most of these approaches rely on population-based optimization algorithms, non-parametric models trained on datasets of (policy, outcome) pairs. Population-based algorithms cannot leverage automatic differentiation on large computational clusters, often demonstrate limited generalization capabilities and cannot easily handle high-dimension perceptual spaces (e.g. images) without hand-defined input pre-processing. For these reasons, developmental robotics could benefit from new advances in deep rl.

Recently, we have been observing a convergence of these two fields, forming a new domain that we propose to call developmental machine learning, or developmental artificial intelligence. Indeed, rl researchers now incorporate fundamental ideas from the development robotics literature in their own algorithms, and reversely developmental robotics learning architecture are beginning to benefit from the generalization capabilities of deep rl techniques. These convergences can mostly be categorized in two ways depending on the type of intrinsic motivation (ims) being used [START_REF] Oudeyer | What is intrinsic motivation? a typology of computational approaches[END_REF]]:

• Knowledge-based IMs compare the situations experienced by the agent to its current knowledge and expectations, and reward it for experiencing dissonance (or resonance). This family includes ims rewarding prediction errors [START_REF] Schmidhuber | A possibility for implementing curiosity and boredom in model-building neural controllers[END_REF][START_REF] Pathak | Curiosity-driven exploration by selfsupervised prediction[END_REF], novelty [START_REF] Burda | Exploration by random network distillation[END_REF][START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF], surprise [START_REF] Achiam | Surprise-based intrinsic motivation for deep reinforcement learning[END_REF], negative surprise [START_REF] Berseth | Smirl: Surprise minimizing rl in dynamic environments[END_REF], learning progress [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF][START_REF] Kim | Active world model learning with progress curiosity[END_REF] or information gains [START_REF] Houthooft | Vime: Variational information maximizing exploration[END_REF], see a review in [START_REF] Linke | Adapting behaviour via intrinsic reward: A survey and empirical study[END_REF]. This type of ims is often used as an auxiliary reward to organize the exploration of agents in environments characterized by sparse rewards. It can also be used to facilitate the construction of world models [START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF][START_REF] Kim | Active world model learning with progress curiosity[END_REF][START_REF] Sekar | Planning to explore via self-supervised world models[END_REF].

• Competence-based IMs, on the other hand, reward agents to solve self-generated problems, to achieve self-generated goals. In this category, agents need to represent, select and master self-generated goals. As a result, competence-based ims were often used to organize the acquisition of repertoires of skills in task-agnostic environments [START_REF] Baranes | Intrinsically motivated goal exploration for active motor learning in robots: A case study[END_REF][START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Santucci | Grail: a goal-discovering robotic architecture for intrinsically-motivated learning[END_REF], Forestier and Oudeyer, 2016[START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF][START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF], Colas et al., 2019, Pong et al., 2019, Colas et al., 2020c]. Just like knowledge-based ims, competence-based ims organize the exploration of the world and, thus, might be used to facilitate learning in sparse reward settings [START_REF] Colas | Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms[END_REF] or train world models [Baranes andOudeyer, 2013a, Chitnis et al., 2021].

rl algorithms using knowledge-based ims leverage ideas from developmental robotics to solve standard rl problems. On the other hand, rl algorithms using competence-based ims organize exploration around self-generated goals and can be seen as targeting a developmental robotics problem: the open-ended and self-supervised acquisition of repertoires of diverse skills. Intrinsically Motivated Goal Exploration Processes (imgep) is the family of algorithms that bake competence-based ims into learning agents [START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF]. imgep agents generate and pursue their own goals as a way to explore their environment, discover possible interactions and build repertoires of skills. This framework emerged from the field of developmental robotics [START_REF] Oudeyer | What is intrinsic motivation? a typology of computational approaches[END_REF], Baranes and Oudeyer, 2009b, 2010[START_REF] Rolf | Goal babbling permits direct learning of inverse kinematics[END_REF] and originally leveraged population-based learning algorithms (popimgep) [Baranes and Oudeyer, 2009a[START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF], Forestier and Oudeyer, 2016[START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF]. Recently, goal-conditioned rl agents were also endowed with the ability to generate and pursue their own goals and learn to achieve them via self-generated rewards. We argue that this set of methods form a sub-category of imgeps that we call goal-conditioned imgeps or gc-imgeps. In contrast, one can refer to externally-motivated goal-conditioned rl agents as gc-emgeps.

Intrinsically Motivated Processes (IMP)

Competence-based IMP aka Intrinsically Motivated Goal Exploration Processes (IMGEP)

Goals are generated internally by the agent

Goal-Conditioned RL Agents

GC-IMGEP

Knowledge-based IMP

No concept of goals

GC-EMGEP

Goals come from the env.

POP-IMGEP

UFVA [START_REF] Schaul | Universal value function approximators[END_REF] AGILE [START_REF] Bahdanau | Systematic generalization: What is required and can it be learned?[END_REF] Disagreement (Pathak et al., 2019) RIDE [START_REF] Campero | Learning with amigo: Adversarially motivated intrinsic goals[END_REF] VIME (Houthooft et al., 2017) Map-Elites [START_REF] Mouret | Illuminating search spaces by mapping elites[END_REF] RND [START_REF] Burda | Exploration by random network distillation[END_REF] Modular-IMGEP (Forestier et. al., 2016) Count-Based [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF] SAGG-RIAC [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF] HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] CURIOUS [START_REF] Colas | CURIOUS: intrinsically motivated modular multi-goal reinforcement learning[END_REF] IMAGINE (Colas et al., 2020c) RIG [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF] DISCERN [START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF] Figure 1: A typology of intrinsically-motivated and/or goal-conditioned rl approaches. pop-imgep, gc-imgep and gc-emgep refer to population-based intrinsically motivated goal exploration processes, goal-conditioned imgep and goal-conditioned externally motivated goal exploration processes respectively. pop-imgep, gcimgep and gc-imgep all represent goals, but knowledge-based ims do not. While imgeps (pop-imgep and gc-imgep) generate their own goals, gc-emgeps require externally-defined goals. This paper is interested in gc-imgeps, the intersection of goal-conditioned rl agents and intrinsically motivated processes that is, the set of methods that train learning agents to generate and pursue their own goals with goal-conditioned rl algorithms.

This paper proposes a formalization and a review of the gc-imgep algorithms at the convergence of rl methods and developmental robotics objectives. Figure 1 proposes a visual representation of intrinsic motivations approaches (knowledge-based ims vs competencebased ims or imgeps) and goal-conditioned rl (externally vs intrinsically motivated). Their intersection is the family of algorithms that train agents to generate and pursue their own goals by training goal-conditioned policies. We define goals as the combination of a compact goal representation and a goal-achievement function to measure progress. This definition highlights new challenges for autonomous learning agents. While traditional rl agents only need to learn to achieve goals, gc-imgep agents also need to learn to represent them, to generate them and to measure their own progress. After learning, the resulting goal-conditioned policy and its associated goal space form a repertoire of skills, a repertoire of behaviors that the agent can represent and control. We believe organizing past gc-rl methods at the convergence of developmental robotics and rl into a common classification and towards the resolution of a common problem will help organize future research.

Scope of the survey We are interested in algorithms from the gc-imgep family as algorithmic tools to enable agents to acquire repertoires of skills in an open-ended and self-supervised setting. Externally motivated goal-conditioned rl approaches do not enable agents to generate their own goals and thus cannot be considered imgeps. However, these approaches can often be converted into gc-imgeps by integrating the goal generation process within the agent. For this reason, we include some gc-emgeps approaches when they present interesting mechanisms that can directly be leveraged in intrinsically motivated agents.

What is not covered. This survey will not discuss some related but distinct approaches including multi-task rl [START_REF] Caruana | Multitask learning[END_REF], rl with auxiliary tasks [START_REF] Riedmiller | Learning by playing-solving sparse reward tasks from scratch[END_REF][START_REF] Jaderberg | Reinforcement learning with unsupervised auxiliary tasks[END_REF] and rl with knowledge-based ims [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Pathak | Curiosity-driven exploration by selfsupervised prediction[END_REF][START_REF] Burda | Exploration by random network distillation[END_REF]. None of these approaches do represent goals or see the agent's behavior affected by goals. The subject of intrinsically motivated goal-conditioned rl also relates to transfer learning and curriculum learning. This survey does not cover transfer learning approaches, but interested readers can refer to [START_REF] Taylor | Transfer learning for reinforcement learning domains: A survey[END_REF]. We will discuss automatic curriculum learning approaches that organize the generation of goals according to the agent's abilities in Section 6 but, for a broader picture on the topic, readers can refer to the recent review [START_REF] Portelas | Automatic curriculum learning for deep rl: A short survey[END_REF].

Survey organization

We first formalize the notion of goals and the problem of the openended and self-supervised acquisition of skill repertoires, building on the formalization of the rl and multi-goal rl problems. After presenting a definition of the gc-imgep family, we organize the surveyed literature along three axes: 1) What are the different types of goal representations? (Section 4); 2) How can we learn goal representations? (Section 5) and 3) How can we prioritize goal selection? (Section 6). From this coherent picture of the literature, we identify properties of what humans call goals that were not addressed in the surveyed approaches. This serves as the basis for a discussion of potential future avenues for the design of new gc-imgep approaches.

Self-Supervised Acquisition of Skills Repertoires with Deep RL

This section builds towards the definition of our main objective: enabling agents to acquire repertoires of skills in an open-ended and self-supervised setting. To this end, we present the traditional reinforcement learning (rl) problem, propose a formal generalized definition of goals in the context of rl, and define the multi-goal rl problem which will serve as a basis to introduce our problem.

The Reinforcement Learning Problem.

In a reinforcement learning (rl) problem, the agent learns to perform sequences of actions in an environment so as to maximize some notion of cumulative reward [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. rl problems are commonly framed as Markov Decision Processes (mdps): M = {S, A, T , ρ 0 , R} [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. The agent and its environment, as well as their interaction dynamics are defined by the first components {S, A, T , ρ 0 }, where s ∈ S describes the current state of the agent-environment interaction and ρ 0 is the distribution over initial states. The agent can interact with the environment through actions a ∈ A. Finally, the dynamics are characterized by the transition function T that dictates the distribution of the next state s from the current state and action T (s | s, a). The objective of the agent in this environment is defined by the remaining component of the mdp: R. R is the reward function, it computes a reward for any transition: R(s, a, s ). Note that, in a traditional rl problem, the agent only receives the rewards corresponding to the transitions it experiences but does not have access to the function itself. The objective of the agent is to maximize the cumulative reward computed over complete episodes. The aggregation of rewards over time is often modulated by the discounting function Γ so that the total aggregated reward from time step t, R tot t , is computed as

R tot = ∞ i=t Γ(s i , a i , i) × R(s i-1 , a i , s i ).
Γ is usually an exponentially-decreasing function of time: Γ(t) = γ t with a constant discount factor γ ∈ [0, 1[. Each instance of an mdp implements an rl problem, also called a task.

Defining Goals for Reinforcement Learning

This section takes inspiration from the notion of goal in psychological research to inform the formalization of goals for reinforcement learning.

Goals in psychological research. Working on the origin of the notion goal and its use in past psychological research, [START_REF] Elliot | The goal construct in psychology[END_REF] propose a general definition:

A goal is a cognitive representation of a future object that the organism is committed to approach or avoid [START_REF] Elliot | The goal construct in psychology[END_REF].

Because goals are cognitive representations, only animate organisms that represent goals qualify as goal-conditioned. Because this representation relates to a future object, goals are cognitive imagination of future possibilities: goal-conditioned behavior is proactive, not reactive. Finally, organisms commit to their goal, their behavior is thus influenced directly by this cognitive representation.

Generalized goals for reinforcement learning. rl algorithms seem to be a good fit to train such goal-conditioned agents. Indeed, rl algorithms train learning agents (organisms) to maximize (approach) a cumulative (future) reward (object). In rl, goals can be seen as a set of constraints on one or several consecutive states that the agent seeks to respect. These constraints can be very strict and characterize a single target point in the state space (e.g. image-based goals) or a specific sub-space of the state space (e.g. target x-y coordinate in a maze, target block positions in manipulation tasks). They can also be more general, when expressed by language for example (e.g. 'find a red object or a wooden one'). To represent these goals, rl agents must be able to 1) have a compact representation of them and 2) assess their achievement. This is why we propose the following formalization for rl goals: each goal is a g = (z g , R G ) pair where z g is a compact goal parameterization or goal embedding and R G is a goal-achievement function -also called goal-parameterized or goal-conditioned reward function -that measures progress towards goal achievement and is shared across goals. We note R g (•) = R G (• | z g ) the reward function measuring the achievement of goal g. With this definition we can express a diversity of goals, see Section 4 and Table 1.

The goal-achievement function and the goal-conditioned policy both assign meaning to a goal. The former defines what it means to achieve the goal, it describes how the world looks like when it is achieved. The latter characterizes the process by which this goal can be achieved, what the agent needs to do to achieve it. In this search for the meaning of a goal, the goal embedding can be seen as the map: the agent follows this map and via the two functions above, experiences the meaning of the goal.

The Multi-Goal Reinforcement Learning Problem.

By replacing the unique reward function R by the space of reward functions R G , rl problems can be extended to handle multiple goals: M = {S, A, T , ρ 0 , R G }. The term goal should not be mistaken for the term task, which refers to a particular mdp instance. As a result, multi-task rl refers to rl algorithms that tackle a set of mdps that can differ by any of their components (e.g. T , R, S 0 , etc.). The multi-goal rl problem can thus be seen as the particular case of the multi-task rl problem where mdps differ by their reward functions. In the standard multi-goal rl problem, the set of goals -and thus the set of reward functionsis pre-defined by engineers. The experimenter sets goals to the agent, and provides the associated reward functions. gc-rl is the set of agents targeting multi-goal problems.

The Intrinsically Motivated Skills Acquisition Problem

In the intrinsically motivated skills acquisition problem, the agent is set in an open-ended environment without any pre-defined goal and needs to acquire a repertoire of skills. Here, a skill is defined as the association of a goal embedding z g and the policy to reach it Π g . A repertoire of skills is thus defined as the association of a repertoire of goals G with a goal-conditioned policy trained to reach them Π G . The intrinsically motivated skills acquisition problem can now be modeled by a reward-free mdp M = {S, A, T , ρ 0 } that only characterizes the agent, its environment and their possible interactions. Just like children, agents need to represent and generate their own goals, to measure their progress and to learn to achieve them.

Evaluation It is often straightforward to evaluate externally motivated agents as the set of possible interactions is known in advance. One can, for instance, easily measure generalization by computing the agent's success rate on a held-out set of testing goals. Similarly, exploration can be estimated by several metrics such as the count of task-specific state visitations. In contrast, intrinsically motivated agents evolve in open-ended environments and learn to represent and form their own set of skills. In this context, the space of possible behaviors might quickly become intractable for the experimenter, which is maybe the most interesting feature of such agents. For these reasons, designing evaluation protocols is not trivial.

Interestingly, the question of how to evaluate intrinsically motivated agents is quite similar to the question of how to evaluate self-supervised learning systems such as Generative Adversarial Networks (gan) [START_REF] Goodfellow | Generative adversarial networks[END_REF] or self-supervised language models [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF]. In both cases, learning is task-agnostic and it is often hard to compare models in terms of their outputs (e.g. comparing the quality of gan output images, or comparing output repertoires of skills in intrinsically motivated agents). Let us list some approaches to evaluate such models:

• Measuring exploration: one can compute task-agnostic exploration proxies such as the entropy of the visited state distribution, or measures of state coverage (e.g. coverage of the high-level x-y state space in mazes) [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF]. Exploration can also be measured as the number of interactions from a set of interesting interactions defined subjectively by the experimenter (e.g. interactions with objects in Colas et al. [2020c].

• Measuring generalization: The experimenter can subjectively define a set of relevant target goals and prevent the agent from training on them. Evaluating agents on this held-out set at test time provides a measure of generalization [START_REF] Ruis | A benchmark for systematic generalization in grounded language understanding[END_REF], although it is biased towards what the experimenter assesses as relevant goals.

• Measuring transfer learning: Here, we view the intrinsically motivated exploration of the environment as a pre-training phase to bootstrap learning in a subsequent downstream task. In the downstream task, the agent is trained to achieve externallydefined goals. We report its performance and learning speed on these goals. This is akin to the evaluation of self-supervised language models, where the reported metrics evaluate performance in various downstream tasks (e.g. [START_REF] Brown | Language models are few-shot learners[END_REF]).

• Opening the black-box: Investigating internal representations learned during intrinsically motivated exploration is often informative. One can investigate properties of the goal generation system (e.g. does it generate out-of-distribution goals?), investigate properties of the goal embeddings (e.g. are they disentangled?). One can also look at the learning trajectories of the agents across learning, especially when they implement their own curriculum learning (e.g. [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF], Colas et al.

[2019], [START_REF] Pong | Skew-fit: State-covering self-supervised reinforcement learning[END_REF], [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF]).

• Measuring robustness: Autonomous learning agents evolving in open-ended environment should be robust to a variety of properties than can be found in the real-world. This includes very large environments, where possible interactions might vary in terms of difficulty (trivial interactions, impossible interactions, interactions whose result is stochastic thus prevent any learning progress). Environments can also include distractors (e.g. non-controllable objects) and various forms of non-stationarity. Evaluating learning algorithms in various environments presenting each of these properties allows to assess their ability to solve the corresponding challenges.

Intrinsically Motivated Goal Exploration Processes with Goal-Conditioned Policies

Until recently, the imgep family was powered by population-based algorithms (pop-imgep).

The emergence of goal-conditioned rl approaches that generate their own goals gave birth to a new type of imgeps: the goal-conditioned imgeps (gc-imgep). We build from traditional rl algorithms and goal-conditioned rl algorithms towards intrinsically motivated goalconditioned rl algorithms (gc-imgep). [2020]). This removes the need to represent a policy explicitly. In this survey, we focus on drl methods that represent a policy. Most of the goal-related mechanisms discussed in this paper can be directly applied to other optimization methods as well (e.g. il or model-based rl).

Reinforcement

Goal-Conditioned RL Algorithms

Goal-conditioned agents see their behavior affected by the goal they pursue. This is formalized via goal-conditioned policies, that is policies which produce actions based on the environment state and the agent's current goal: Π : S × Z G → A, where Z G is the space of goal embeddings corresponding to the goal space G [START_REF] Schaul | Universal value function approximators[END_REF]. Note that ensembles of policies can also be formalized this way, via a meta-policy Π that retrieves the particular policy from a one-hot goal embedding z g (e.g. [START_REF] Kaelbling | Learning to achieve goals[END_REF], [START_REF] Sutton | Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction[END_REF]). The idea of using a unique rl agent to target multiple goals dates back to [START_REF] Kaelbling | Learning to achieve goals[END_REF]. Later, the horde architecture proposed to use interaction experience to update one value function per goal, effectively transferring to all goals the knowledge acquired while aiming at a particular one [START_REF] Sutton | Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction[END_REF]. In these approaches, one policy is trained for each of the goals and the data collected by one can be used to train others. Building on these early results, [START_REF] Schaul | Universal value function approximators[END_REF] introduced Universal Value Function Approximators (uvfa). They proposed to learn a unique goal-conditioned value function and goal-conditioned policy to replace the set of value functions learned in horde.

Using neural networks as function approximators, they showed that uvfas enable transfer between goals and demonstrate strong generalization to new goals. The idea of hindsight learning further improves knowledge transfer between goals [START_REF] Kaelbling | Learning to achieve goals[END_REF], Andrychowicz et al., 2017]. Learning by hindsight, agents can reinterpret a past trajectory collected while pursuing a given goal in the light of a new goal. By asking themselves, what is the goal for which this trajectory is optimal?, they can use the originally failed trajectory as an informative trajectory to learn about another goal, thus making the most out of every trajectory [START_REF] Eysenbach | Rewriting history with inverse rl: Hindsight inference for policy improvement[END_REF]. This ability dramatically increases the sample efficiency of goal-conditioned algorithms and is arguably an important driver of the recent interest in goal-conditioned rl approaches. Sample goal embedding z g = GS(s 0 , Z G ).

GC-IMGEP Agent

Roll-out goal-conditioned policy 7:

Execute a roll-out with

Π g = Π G (• | z g ) 8:
Store collected transitions τ = (s, a, s ) in M.

Update internal models

9:

Sample a batch of B transitions: M ∼ {(s, a, s )} B .

10:

Perform Hindsight Relabelling {(s, a, s , z g )} B .

11:

Compute internal rewards r = R G (s, a, s | z g ).

12:

Update policy Π via rl on {(s, a, s , z g , r)} B .

13:

Update goal representations Z G .

14:

Update goal-conditioned reward function R G .

15:

Update goal sampling policy GS. 16: return Π } , R G , Z G

GC-IMGEP

gc-imgep are intrinsically motivated versions of goal-conditioned rl algorithms. They need to be equipped with mechanisms to represent and generate their own goals in order to solve the intrinsically motivated skills acquisition problem, see Figure 2. Concretely, this means that, in addition to the goal-conditioned policy, they need to learn: 1) to represent goals g by compact embeddings z g ; 2) to represent the support of the goal distribution, also called goal space Z } = {z g } g∈G ; 3) a goal distribution from which targeted goals are sampled D(z g ); 4) a goal-conditioned reward function R G . Algorithm 1 details the pseudo-code of gc-imgep algorithms.

A Typology of Goal Representations in the Literature

Now that we defined the problem of interest and the overall framework to tackle it, we can start reviewing relevant approaches from the literature and how they fit in this framework. This section presents a typology of the different kinds of goal representations found in the literature. Each goal is represented by a pair: 1) a goal embedding and 2) a goal-conditioned reward function.

Goals as choices between multiple objectives

Goals can be expressed as a list of different objectives the agent can choose from.

Goal embedding. In that case, goal embeddings z g are one-hot encodings of the current objective being pursued among the N objectives available. z i g is the i th one-hot vector:

z i g = (1 j=i ) j=[1..N ]
. This is the case in [START_REF] Oh | Zero-shot task generalization with multi-task deep reinforcement learning[END_REF], [START_REF] Mankowitz | Unicorn: Continual learning with a universal, offpolicy agent[END_REF], [START_REF] Codevilla | End-to-end driving via conditional imitation learning[END_REF].

Reward function. The goal-conditioned reward function is a collection of N distinct reward functions R G (•) = R i (•) if z g = z i
g . In [START_REF] Mankowitz | Unicorn: Continual learning with a universal, offpolicy agent[END_REF] and [START_REF] Chan | Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning[END_REF], each reward function gives a positive reward when the agent reaches the corresponding object: reaching guitars and keys in the first case, monsters and torches in the second.

Goals as target features of states

Goals can be expressed as target features of the state the agent desires to achieve.

Goal embedding. In this scenario, a state representation function ϕ maps the state space to an embedding space Z = ϕ(S). Goal embeddings z g are target points in Z that the agent should reach. In manipulation tasks, z g can be target block coordinates [START_REF] Andrychowicz | Hindsight experience replay[END_REF][START_REF] Nair | Overcoming exploration in reinforcement learning with demonstrations[END_REF][START_REF] Plappert | Multi-goal reinforcement learning: Challenging robotics environments and request for research[END_REF], Colas et al., 2019, Fournier et al., 2019[START_REF] Blaes | Control what you can: Intrinsically motivated task-planning agent[END_REF][START_REF] Li | Towards practical multi-object manipulation using relational reinforcement learning[END_REF][START_REF] Lanier | Curiosity-driven multi-criteria hindsight experience replay[END_REF], Ding et al., 2019]. In navigation tasks, z g can be target agent positions (e.g. in mazes [START_REF] Schaul | Universal value function approximators[END_REF], [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF]). Agent can also target image-based goals. In that case, the state representation function ϕ is usually implemented by a generative model trained on experienced image-based states and goal embeddings can be sampled from the generative model or encoded from real images [START_REF] Zhu | Targetdriven visual navigation in indoor scenes using deep reinforcement learning[END_REF], Pong et al., 2019[START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF][START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF][START_REF] Codevilla | End-to-end driving via conditional imitation learning[END_REF][START_REF] Florensa | Self-supervised learning of image embedding for continuous control[END_REF][START_REF] Venkattaramanujam | Self-supervised learning of distance functions for goal-conditioned reinforcement learning[END_REF], Lynch et al., 2020, Lynch and Sermanet, 2020[START_REF] Nair | Contextual imagined goals for self-supervised robotic learning[END_REF][START_REF] Kovač | Grimgep: Learning progress for robust goal sampling in visual deep reinforcement learning[END_REF].

Reward function. For this type of goals, the reward function R G is based on a distance metric D. One can define a dense reward as inversely proportional to the distance between features of the current state and the target goal embedding: R g = R G (s|z g ) = -α × D(ϕ(s), z g ) (e.g. [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF]). The reward can also be sparse: positive whenever that distance falls below a pre-defined threshold: R G (s|z g ) = 1 if D(ϕ(s), z g ) < , 0 otherwise.

Goals as abstract binary problems

Some goals cannot be expressed as target state features but can be represented by binary problems, where each goal expresses as set of constraint on the state (or trajectory) such that these constraints are either verified or not (binary goal achievement).

Goal embeddings. In binary problems, goal embeddings can be any expression of the set of constraints that the state should respect. [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF], [START_REF] Ecoffet | First return then explore[END_REF] both propose a pre-defined discrete state representation. These representations lie in a finite embedding space so that goal completion can be asserted when the current embedding ϕ(s) equals the goal embedding z g . Another way to express sets of constraints is via languagebased predicates. A sentence describes the constraints expressed by the goal and the state or trajectory either verifies them, or does not [START_REF] Hermann | Grounded Language Learning in a Simulated 3D World, jun[END_REF][START_REF] Chan | Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning[END_REF][START_REF] Jiang | Language as an Abstraction for Hierarchical Deep Reinforcement Learning[END_REF], Bahdanau et al., 2019, 2018, Hill et al., 2019[START_REF] Cideron | Self-educated language agent with hindsight experience replay for instruction following[END_REF], Colas et al., 2020c, Lynch and Sermanet, 2020], see [START_REF] Loynd | Working memory graphs[END_REF] for a recent review. Language can easily characterize generic goals such as "grow any blue object" [Colas et al., 2020c], relational goals like "sort objects by size" [START_REF] Jiang | Language as an Abstraction for Hierarchical Deep Reinforcement Learning[END_REF], "put the cylinder in the drawer " [Lynch and Sermanet, 2020] or even sequential goals "Open the yellow door after you open a purple door " [START_REF] Chevalier-Boisvert | Baby{AI}: First Steps Towards Grounded Language Learning With a Human In the Loop[END_REF]. When goals can be expressed by language sentences, goal embeddings z g are usually language embeddings learned jointly with either the policy or the reward function. Note that, although rl goals always express constraints on the state, we can imagine time-extended goals where constraints are expressed on the trajectory (see a discussion in Section 7.1).

Reward function.

The reward function of a binary problem can be viewed as a binary classifier that evaluates whether state s (or trajectory τ ) verifies the constraints expressed by the goal semantics (positive reward) or not (null reward). This binary classification setting has directly been implemented as a way to learn language-based goal-conditioned reward functions R g (s | z g ) in [START_REF] Bahdanau | Learning to Understand Goal Specifications by Modelling Reward[END_REF] and Colas et al. [2020c]. Alternatively, the setup described in Colas et al. [2020a] proposes to turn binary problems expressed by language-based goals into goals as specific target features. To this end, they train a language-conditioned goal generator that produces specific target features verifying constraints expressed by the binary problem. As a result, this setup can use a distance-based metric to evaluate the fulfillment of a binary goal.

Goals as a Multi-Objective Balance

Some goals can be expressed, not as desired regions of the state or trajectory space but as more general objectives that the agent should maximize. In that case, goals can parameterize a particular mixture of multiple objectives that the agent should maximize.

Goal embeddings. Here, goal embeddings are simply sets of weights balancing the different objectives z g = (β i ) i=[1..N ] where β i is the weights applied to objective i and N is the number of objectives. Note that, when β j = 1 and β i = 0, ∀i = j, the agent can decide to pursue any of the objective alone. In Never Give Up, for example, rl agents are trained to maximize a mixture of extrinsic and intrinsic rewards [Badia et al., 2020b]. The agent can select the mixing parameter β that can be viewed as a goal. Building on this approach, Agent57 adds a control of the discount factor, effectively controlling the rate at which rewards are discounted as time goes by [Badia et al., 2020a].

Reward function. When goals are represented as a balance between multiple objectives, the associated reward function cannot be represented neither as a distance metric, nor as a binary classifier. Instead, the agent needs to maximize a convex combination of the objectives: R g

(s) = N i=1 β i g R i (s) where R i is the i th of N objectives and z g = β = β g i | i∈[1.
.N ] is the set of weights.

Goal-Conditioning

Now that we described the different types of goal embeddings found in the literature, remains the question of how to condition the agent's behavior -i.e. the policy -on them. Originally, the uvfa framework proposed to concatenate the goal embedding to the state representation to form the policy input. Recently, other mechanisms have emerged. When languagebased goals were introduced, [START_REF] Chaplot | Gated-attention architectures for task-oriented language grounding[END_REF] proposed the gated-attention mechanism where the state features are linearly scaled by attention coefficients computed from the goal representation ϕ(z g ): input = s ϕ(z g ), where is the Hadamard product. Later, the Feature-wise Linear Modulation (film) approach [START_REF] Perez | Film: Visual reasoning with a general conditioning layer[END_REF] generalized this principle to affine transformations: input = s ϕ(z g ) + ψ(z g ). Alternatively, [START_REF] Andreas | Neural module networks[END_REF] came up with Neural Module Networks (nmn), a mechanism that leverages the linguistic structure of goals to derive a symbolic program that defines how states should be processed [START_REF] Bahdanau | Learning to Understand Goal Specifications by Modelling Reward[END_REF].

Conclusion

This section presented a diversity of goal representations, corresponding to a diversity of reward functions architectures. However, we believe this represents only a small fraction of the diversity of goal types that humans pursue. Section 7 discusses other goal representations that rl algorithms could target.

How to Learn Goal Representations?

The previous section discussed various types of goal representations. Intrinsically motivated agents actually need to learn these goal representations. While individual goals are represented by their embeddings and associated reward functions, representing multiple goals also requires the representation of the support of the goal space, i.e. how to represent the collection of valid goals that the agent can sample from, see Figure 2. This section reviews different approaches from the literature.

Assuming Pre-Defined Goal Representation

Most approaches tackle the multi-goal rl problem, where goal spaces and associated rewards are pre-defined by the engineer and are part of the task definition. Navigation and manipu-lation tasks, for example, pre-define goal spaces (e.g. target agent position and target block positions respectively) and use the Euclidean distance to compute rewards [START_REF] Andrychowicz | Hindsight experience replay[END_REF][START_REF] Nair | Overcoming exploration in reinforcement learning with demonstrations[END_REF][START_REF] Plappert | Multi-goal reinforcement learning: Challenging robotics environments and request for research[END_REF], Colas et al., 2019[START_REF] Li | Towards practical multi-object manipulation using relational reinforcement learning[END_REF][START_REF] Lanier | Curiosity-driven multi-criteria hindsight experience replay[END_REF], Ding et al., 2019[START_REF] Schaul | Universal value function approximators[END_REF][START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF]. [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF], [START_REF] Ecoffet | First return then explore[END_REF] hand-define abstract state representation and provide positive rewards when these match target goal representations. This falls short of solving the intrinsically motivated goal exploration problem. The next sub-section investigates how goal representations can be learned.

Learning Goal Embeddings

Some approaches assume the pre-existence of a goal-conditioned reward function, but learn to represent goals by learning goal embeddings. This is the case of language-based approaches, which receive rewards from the environment (thus are gc-emgep), but learn goal embeddings jointly with the policy during policy learning [START_REF] Hermann | Grounded Language Learning in a Simulated 3D World, jun[END_REF][START_REF] Chan | Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning[END_REF][START_REF] Jiang | Language as an Abstraction for Hierarchical Deep Reinforcement Learning[END_REF][START_REF] Bahdanau | Systematic generalization: What is required and can it be learned?[END_REF], Hill et al., 2019[START_REF] Cideron | Self-educated language agent with hindsight experience replay for instruction following[END_REF], Lynch and Sermanet, 2020]. When goals are target images, goal embeddings can be learned via generative models of states, assuming the reward to be a fixed distance metric computed in the embedding space [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF], Pong et al., 2019[START_REF] Florensa | Self-supervised learning of image embedding for continuous control[END_REF][START_REF] Nair | Contextual imagined goals for self-supervised robotic learning[END_REF].

Learning the Reward Function

A few approaches go even further and learn their own goal-conditioned reward function. [START_REF] Bahdanau | Learning to Understand Goal Specifications by Modelling Reward[END_REF], Colas et al. [2020c] learn language-conditioned reward functions from an expert dataset or from language descriptions of autonomous exploratory trajectories respectively. However, the agile approach from [START_REF] Bahdanau | Learning to Understand Goal Specifications by Modelling Reward[END_REF] does not generate its own goals. In the domain of image-based goals, [START_REF] Venkattaramanujam | Self-supervised learning of distance functions for goal-conditioned reinforcement learning[END_REF][START_REF] Goyal | Dynamical distance learning for semisupervised and unsupervised skill discovery[END_REF] learn a distance metric estimating the square root of the number of steps required to move from any state s 1 to any s 2 and generates internal signals to reward agents for getting closer to their target goals. [START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF] learn a similarity metric in the space of controllable aspects of the environment that is based on a mutual information objective between the state and the goal state s g . [START_REF] Goyal | Dynamical distance learning for semisupervised and unsupervised skill discovery[END_REF][START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF] train agents to develop a set of skills leading to maximally different areas of the state space. Agents are rewarded for experiencing states that are easy to discriminate, while a discriminator is trained to better infer the skill z g from the visited states. [START_REF] Wu | The laplacian in rl: Learning representations with efficient approximations[END_REF] compute a distance metric representing the ability of the agent to reach one state from another using the Laplacian of the transition dynamics graph, where nodes are states and edges are actions. More precisely, they use the eigenvectors of the Laplacian matrix of the graph given by the states of the environment as basis to compute the L2 distance towards a goal configuration. All these methods set their own goals and learn their own goal-conditioned reward function. For these reasons, they can be considered as complete intrinsically motivated goal-conditioned rl algorithms.

Learning the Support of the Goal Distribution

The previous sections reviewed several approaches to learn goal embeddings and reward functions. To represent collections of goals, one also needs to represent the support of the goal distribution, which embeddings correspond to valid goals and which do not. Most approaches consider a pre-defined, bounded goal space in which any point is a valid goal (e.g. target positions within the boundaries of a maze, target block positions within the gripper's reach) [START_REF] Andrychowicz | Hindsight experience replay[END_REF][START_REF] Nair | Overcoming exploration in reinforcement learning with demonstrations[END_REF][START_REF] Plappert | Multi-goal reinforcement learning: Challenging robotics environments and request for research[END_REF], Colas et al., 2019[START_REF] Li | Towards practical multi-object manipulation using relational reinforcement learning[END_REF][START_REF] Lanier | Curiosity-driven multi-criteria hindsight experience replay[END_REF], Ding et al., 2019[START_REF] Schaul | Universal value function approximators[END_REF]. However, not all approaches assume pre-defined goal spaces. Some approaches use the set of previously experienced representations to form the support of the goal distribution [START_REF] Veeriah | Many-goals reinforcement learning[END_REF], Akakzia et al., 2020[START_REF] Ecoffet | First return then explore[END_REF]. In [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF], a Generative Adversarial Network (gan) is trained on past representations of states (ϕ(s)) to model a distribution of goals and thus its support. In the same vein, approaches handling image-based goals usually train a generative model of image states based on Variational Auto-Encoders (vae) to model goal distributions and support [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF], Pong et al., 2019[START_REF] Nair | Contextual imagined goals for self-supervised robotic learning[END_REF]. In both cases, valid goals are the one generated by the generative model. We saw that the support of valid goals can be pre-defined, a simple set of past representations or approximated by a generative model trained on these. In all cases, the agent can only sample goals within the convex hull of previously encountered goals (in representation space). We say that goals are within training distribution. This drastically limits exploration and the discovery of new behaviors. Children, on the other hand, can imagine creative goals. Pursuing these goals is thought to be the main driver of exploratory play in children [START_REF] Chu | Exploratory play, rational action, and efficient search[END_REF]. This is made possible by the compositionality of language, where sentences can easily be combined to generate new ones. The imagine algorithm leverages the creative power of language to generate such out-of-distribution goals [Colas et al., 2020c]. The support of valid goals is extended to any combination of language-based goals experienced during training. They show that this mechanism augments the generalization and exploration abilities of learning agents. In Section 6, we discuss how agents can learn to adapt the goal sampling distribution to maximize the learning progress of the agent.

Conclusion

This section presented how previous approaches tackled the problem of learning goal representations. While most approaches rely on pre-defined goal embeddings and/or reward functions, some approaches proposed to learn internal reward functions and goal embeddings jointly.

How to Prioritize Goal Selection?

Intrinsically motivated goal-conditioned agents also need to select their own goals. While goals can be generated by uninformed sampling of the goal space, agents can benefit from mechanisms optimizing goal selection. In practice, this boils down to the automatic adaptation of the goal sampling distribution as a function of the agent performance.

Automatic Curriculum Learning for Goal Selection

In real-world scenarios, goal spaces can be too large for the agent to master all goals in its lifetime. Some goals might be trivial, others impossible. Some goals might be reached by chance sometimes, although the agent cannot make any progress on them. Some goals might be reachable only after the agent mastered more basic skills. For all these reasons, it is important to endow intrinsically motivated agents learning in open-ended scenarios with the ability to optimize their goal selection mechanism. This ability is a particular case of Automatic Curriculum Learning applied for goal selection: mechanisms that organize goal sampling so as to maximize the long-term performance improvement (distal objective). As this objective is usually not directly differentiable, curriculum learning techniques usually rely on a proximal objective. In this section, we look at various proximal objectives used in automatic curriculum learning (ACL) strategies to organize goal selection. Interested readers can refer to [START_REF] Portelas | Automatic curriculum learning for deep rl: A short survey[END_REF], which present a broader review of ACL methods for rl.

Intermediate difficulty. Intermediate difficulty has been used as a proxy for long-term performance improvement, following the intuition that focusing on goals of intermediate difficulty results in short-term learning progress that will eventually turn into long-term performance increase. goalgan assigns feasibility scores to goals as the proportion of time the agents successfully reaches it [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF]. Based on this data, a gan is trained to generate goals of intermediate difficulty, whose feasibility scores are contained within an intermediate range. [START_REF] Sukhbaatar | Intrinsic motivation and automatic curricula via asymmetric self-play[END_REF] and [START_REF] Campero | Learning with amigo: Adversarially motivated intrinsic goals[END_REF] train a goal policy with rl to propose challenging goals to the rl agent. The goal policy is rewarded for setting goals that are neither too easy nor impossible. [START_REF] Zhang | Automatic curriculum learning through value disagreement[END_REF] select goals that maximize the disagreement in an ensemble of value functions. Value functions agree when the goals are too easy (the agent is always successful) or too hard (the agent always fails) but disagree for goals of intermediate difficulty.

Uniform feasibility. [START_REF] Racanière | Automated curricula through setter-solver interactions[END_REF] generalize the idea of intermediate difficulty and train a goal generator to sample goals of uniform feasibility. This approach seems to lead to better stability and improved performance on more complex tasks compared to goalgan [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF].

Novelty -diversity. [START_REF] Pong | Skew-fit: State-covering self-supervised reinforcement learning[END_REF], [START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF], [START_REF] Pitis | Maximum entropy gain exploration for long horizon multi-goal reinforcement learning[END_REF] all bias the selection of goals towards sparse areas of the goal space. For this purpose, they train density models in the goal space. While [START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF], [START_REF] Pong | Skew-fit: State-covering self-supervised reinforcement learning[END_REF] aim at a uniform coverage of the goal space (diversity), [START_REF] Pitis | Maximum entropy gain exploration for long horizon multi-goal reinforcement learning[END_REF] skew the distribution of selected goals even more, effectively maximizing novelty. [START_REF] Kovač | Grimgep: Learning progress for robust goal sampling in visual deep reinforcement learning[END_REF] proposed to enhance these methods with a goal sampling prior focusing goal selection towards controllable areas of the goal space.

Short-term learning progress. Some approaches estimate the learning progress of the agent in different regions of the goal space and bias goal sampling towards areas of high absolute learning progress using bandit algorithms [START_REF] Colas | CURIOUS: intrinsically motivated modular multi-goal reinforcement learning[END_REF][START_REF] Blaes | Control what you can: Intrinsically motivated task-planning agent[END_REF][START_REF] Fournier | Accuracy-based curriculum learning in deep reinforcement learning[END_REF], 2019, Akakzia et al., 2020]. [START_REF] Colas | CURIOUS: intrinsically motivated modular multi-goal reinforcement learning[END_REF] and [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF] organize goals into modules and compute average lp measures over modules. [START_REF] Fournier | Accuracy-based curriculum learning in deep reinforcement learning[END_REF] defines goals as a discrete set of precision requirements in a reaching task and computes lp for each requirement value. The use of absolute lp enables agents to focus back on goals for which performance decreases (due to perturbations or forgetting). [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF] introduces the success rate in the value optimized by the bandit: v = (1sr) × lp, so that agents favor goals with high absolute lp and low competence.

6.2 Hierarchical Reinforcement Learning for Goal Sequencing.

Hierarchical reinforcement learning (hrl) can be used to guide the sequencing of goals [START_REF] Dayan | Feudal reinforcement learning[END_REF][START_REF] Sutton | Intra-option learning about temporally abstract actions[END_REF], 1999[START_REF] Precup | Temporal abstraction in reinforcement learning[END_REF]. In hrl, a high-level policy is trained via rl or planning to generate sequence of goals for a lower level policy so as to maximize a higher-level reward. This allows to decompose tasks with long-term dependencies into simpler sub-tasks. Low-level policies are implemented by traditional goal-conditioned rl algorithms [START_REF] Levy | Hierarchical reinforcement learning with hindsight[END_REF][START_REF] Röder | Curious hierarchical actor-critic reinforcement learning[END_REF] and can be trained independently from the high-level policy [START_REF] Kulkarni | Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation[END_REF][START_REF] Frans | Meta learning shared hierarchies[END_REF] or jointly [START_REF] Levy | Hierarchical reinforcement learning with hindsight[END_REF], Nachum et al., 2018[START_REF] Röder | Curious hierarchical actor-critic reinforcement learning[END_REF]. Most approaches consider handdefined spaces for the sub-goals (e.g. positions in a maze). Recent approaches propose to use the state space directly [START_REF] Nachum | Data-efficient hierarchical reinforcement learning[END_REF] or to learn the sub-goal space (e.g. [START_REF] Vezhnevets | Feudal networks for hierarchical reinforcement learning[END_REF], or with generative model of image states in [START_REF] Nasiriany | Planning with goal-conditioned policies[END_REF]).

Future Avenues

This section proposes several directions to improve over current gc-imgep approaches towards solving the intrinsically motivated skills acquisition problem.

Towards a Greater Diversity of Goal Representations

This section proposes new types of goal representation that rl agents could leverage to tackle the intrinsically motivated skills acquisition problem.

Time-extended goals. All rl approaches reviewed in this paper consider time-specific goals, that is, goals whose completion can be assessed from any state s. This is due to the Markov property requirement, where the next state and reward need to be a function of the previous state only. Time-extended goals -i.e. goals whose completion can be judged by observing a sequence of states (e.g. jump twice) -can however be considered by adding timeextended features to the state (e.g. the difference between the current state and the initial state Colas et al. [2020c]). To avoid such ad-hoc state representations, one could imagine using reward function architectures that incorporate forms of memory such as Recurrent Neural Network (rnn) architectures [START_REF] Elman | Learning and development in neural networks: the importance of starting small[END_REF] or Transformers [START_REF] Vaswani | Attention is all you need[END_REF]. Although recurrent policies are often used in the literature [START_REF] Chevalier-Boisvert | Baby{AI}: First Steps Towards Grounded Language Learning With a Human In the Loop[END_REF], Hill et al., 2019[START_REF] Loynd | Working memory graphs[END_REF][START_REF] Goyal | Dynamical distance learning for semisupervised and unsupervised skill discovery[END_REF], recurrent reward functions have not been investigated. Time-extended goals include goals expressed as repetitions of a given interaction (e.g. knock three times). Such goals call for novel inductive biases in reward function architectures (e.g. memory, counting ability etc.). Note that pop-imgep approaches are not limited by the Markov property and have used time-extended goals (e.g. Forestier and Oudeyer [2016]).

Learning goals. Goal-driven learning is the idea that humans use learning goals, goals about their own learning abilities as a way to simplify the realization of task goals [START_REF] Ram | Goal-driven learning[END_REF]. Here, we refer to task goals as goals that express constraints on the physical state of the agent and/or environment. On the other hand, learning goals refer to goals that express constraints on the knowledge of the agent. Although most rl approaches target task goals, one could envision the use of learning goals for rl agents. In a way, learningprogress-based learning is a form of learning goal: as the agent favors regions of the goal space to sample its task goals, it formulates the goal of learning about this specific goal region [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Fournier | Accuracy-based curriculum learning in deep reinforcement learning[END_REF], 2019, Colas et al., 2019[START_REF] Blaes | Control what you can: Intrinsically motivated task-planning agent[END_REF], Akakzia et al., 2020]. Embodied Question Answering problems can also be seen as using learning goals. The agent is asked a question (i.e. a learning goal) and needs to explore the environment to answer it (acquire new knowledge) [START_REF] Das | Embodied question answering[END_REF][START_REF] Yuan | Interactive language learning by question answering[END_REF].

Goals as optimization under selected constraints. We discussed the representations of goals as a balance between multiple objectives. An extension of this idea is to integrate the selection of constraints on states or trajectories. One might want to maximize a given metric (e.g. walking speed), while setting various constraints (e.g. maintaining the power consumption below a given threshold or controlling only half of the motors). The agent could explore in the space of constraints, setting constraints to itself, building a curriculum on these, etc. This is partially investigated in Colas et al. [2020b], where the agent samples constraint-based goals in the optimization of control strategies to mitigate the economic and health costs in simulated epidemics. This approach however, only considers constraints on minimal values for the objectives and requires the training of an additional Q-function per constraint.

Towards Goal Composition and Imagination

It is commonly admitted that children play supports learning [START_REF] Piaget | The construction of reality in the child[END_REF][START_REF] Montessori | The Montessori Method[END_REF][START_REF] Sutton-Smith | The ambiguity of play[END_REF]. Recently, [START_REF] Chu | Exploratory play, rational action, and efficient search[END_REF] proposed to see play as an efficient behavior towards imagined goals. In a similar way, imagining creative out-ofdistribution goals might help agents to explore more efficiently their environment. Humans are indeed very good at imagining and understanding out-of-distribution goals using compositional generalization (e.g. put a hat on a goat). If an agent knows how to perform atomic goals (e.g. build a tower with the blue blocks and build a pyramid with the red blocks), we would like to automatically generalize to composition of these goals, as well as their logical combinations (e.g. build a tower of red blocks, but do not build a pyramid of blue blocks). The logical combinations of atomic goals was investigated in [START_REF] Tasse | A boolean task algebra for reinforcement learning[END_REF], [START_REF] Chitnis | Glib: Efficient exploration for relational model-based reinforcement learning via goal-literal babbling[END_REF], and Colas et al. [2020a], [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF]. The first approach represents the space of goals as a Boolean algebra, which allows immediate generalization to compositions of goals (and, or, not). The second approach considers using general symbolic and logic languages to express goals, but uses symbolic planning techniques that are not yet fully integrated in the goal-conditioned deep rl framework. The third and fourth train a generative model of goals conditioned on language inputs. Because it generates discrete goals, it can compose language instructions by composing the finite sets of discrete goals associated to each instruction (and is the intersection, or the union etc). However, these works fall short of exploring the richness of goal compositionality and its various potential forms. [START_REF] Tasse | A boolean task algebra for reinforcement learning[END_REF] seem to be limited to specific goals as target features, while [START_REF] Akakzia | DECSTR: Learning goal-directed abstract behaviors using pre-verbal spatial predicates in intrinsically motivated agents[END_REF] requires discrete goals.

Language as a Tool for Creative Goal Generation

Whether they are specific or abstract, time-specific or time-extended, whether they represent mixture of objectives, constraints, or logical combinations, all goals can be expressed easily by humans through language. Language, thus, seems like the ideal candidate to express goals in rl agents. So far, language was only used to formulate a few forms of goals (see Section 4). In the future, it might be used to express any type of goals. Recurrent Neural Networks (rnn) [START_REF] Elman | Learning and development in neural networks: the importance of starting small[END_REF], Deep Sets [START_REF] Zaheer | Deep sets[END_REF], Graph Neural Networks (gnn) or Transformers are all architectures that benefits from inductive biases and could be leveraged to facilitate new forms of goal representations (time-extended, set-based, relational etc.). As a learning signal, Colas et al. [2020c] propose to leverage descriptive sentences from social partners as a way to train goal representations. Mixing self-supervised learning and language inputs from humans as in Lynch and Sermanet [2020] might be the way forward.

Discussion & Conclusion

This paper defined the intrinsically motivated skills acquisition problem and proposed to view intrinsically motivated goal-conditioned rl algorithms or gc-imgep as computational tools to tackle it. These methods belong to the new field of developmental machine learning, the intersection of the developmental robotics and rl fields. We reviewed current goalconditioned rl approaches under the lens of intrinsically motivated agents that learn to represent and generate their own goals in addition of learning to achieve them.

We propose a new general definition of the goal construct: a pair of compact goal representation and an associated goal-achievement function. Interestingly, this viewpoint allowed us to categorize some rl approaches as goal-conditioned, even though the original papers did not explicitly acknowledge it. For instance, we view the Never Give Up [Badia et al., 2020b] and Agent 57 [Badia et al., 2020a] architectures as goal-conditioned, because agents actively select parameters affecting the task at hand (parameter mixing extrinsic and intrinsic objectives, discount factor) and see their behavior affected by this choice (goal-conditioned policies). This point of view also offers a direction for future research. Intrinsically motivated agents need to learn to represent goals and to measure goal achievement. Future research could extend the diversity of considered goal representations, investigate novel reward function architectures and inductive biases to allow time-extended goals, goal composition and to improve generalization.

The general vision we convey in this paper is the vision of the learning agent as a curious scientist. A scientist that would formulate hypotheses about the world and explore it to find out whether they are true. A scientist that would ask questions, and setup intermediate goals to explore the world and find answers. A scientist that would set challenges to itself to learn about the world, to discover new ways to interact with it and to grow its collection of skills and knowledge. Such a scientist could decide of its own agenda. It would not need to be instructed and could be guided only by its curiosity, by its desire to discover new information and to master new skills.

Approach Goal Type

Goal

Rep.

Reward Function

Goal sampling strategy GC-IMGEPs that assume goal embeddings and reward functions Table 1: A classification of GC-IMGEP approaches. The classification groups algorithms depending on their degree of autonomy: 1) gc-imgeps that rely on pre-defined goal representations (embeddings and reward functions); 2) gcimgeps that rely on pre-defined reward functions but learn goal embeddings and 3) gc-imgeps that learn complete goal representations (embeddings and reward functions). For each algorithm, we report the type of goals being pursued (see Section 4), whether goal embeddings are learned (Section 5), whether reward functions are learned (Section 5.3) and how goals are sampled (Section 6). We mark in bold algorithms that use a developmental approaches and explicitly pursue the intrinsically motivated skills acquisition problem.

Figure 2 :

 2 Figure 2: Representation of the different learning modules in a gc-imgep algorithm. In contrast, externally motivated goal exploration processes (gc-emgeps) only train the goal-conditioned policy and assume external goal generator and goal-conditioned reward function.

Fournier

  Learning Algorithms for the RL Problem rl methods use transitions collected via interactions between the agent and its environment (s, a, s , R(s, a, s )) to train a policy π: a function generating the next action a based on the current state s so as to maximize a cumulative function of rewards. The deep rl family (drl) leverages deep neural networks as function approximators to represent policies, reward and value functions. Other set of methods can also be used to train policies. Imitation Learning (il) leverages demonstrations, i.e. transitions collected by another entity (e.g.[START_REF] Ho | Generative adversarial imitation learning[END_REF],[START_REF] Hester | Deep q-learning from demonstrations[END_REF]). Evolutionary Computing (ec) is a group of population-based approaches where populations of policies are trained to maximize cumulative rewards using episodic samples (e.g.[START_REF] Lehman | Evolving a diversity of virtual creatures through novelty search and local competition[END_REF],[START_REF] Mouret | Illuminating search spaces by mapping elites[END_REF],[START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF],[START_REF] Colas | Scaling map-elites to deep neuroevolution[END_REF]). Finally, model-based rl approaches can be used to 1) learn a model of the transition function T and 2) perform planning towards reward maximization in that model (e.g.[START_REF] Chua | Deep reinforcement learning in a handful of trials using probabilistic dynamics models[END_REF],Charlesworth and Montana