Keywords: Apprentissage par renforcement, Motivation intrinsèque, Apprentissage développemental, Apprentissage tout au long d Reinforcement learning, Intrinsic motivation, Developmental learning, Lifelong learning, Repre-

En apprentissage par renforcement (RL), un agent apprend à résoudre une tâche en interagissant avec son environnement. Afin de faire passer à l'échelle ces agents sur des tâches complexes, les méthodes récentes ont proposé avec succès d'intégrer les méthodes d'apprentissage profond au RL, créant le domaine d'apprentissage profond par renforcement (DRL). Cependant, la signification sémantique d'une tâche est toujours pourvue par une fonction de récompense experte qui guide l'agent dans son processus d'apprentissage. Ce paradigme contraste avec la manière dont les animaux et humains apprennent: les travaux de psychologie suggèrent que les humains sont intrinsèquement motivés à acquérir de nouvelles connaissances à propos de leur environnement.

Dans cette thèse, notre objectif est d'étudier comment la motivation intrinsèque permet de résoudre les problèmes expérimentés par le DRL. Tout d'abord, nous mettons en évidence comment les motivations intrinsèques actuelles attaquent certains problèmes du DRL. Nous classifions et formalisons les méthodes, puis analysons leurs limites. Afin d'exhiber leur importance, nous mettons en avant que ces verrous peuvent empêcher un agent d'apprendre des compétences et représentations de l'environnement de plus en plus complexes. Ce sont des éléments-clés pour faire apprendre des agents autonomes comme des humains. Á partir de cette analyse, nous introduisons deux nouveaux modèles qui peuvent apprendre des compétences diverses et spécifiques à une tâche de bout en bout. Le premier, ELSIM, construit un arbre discret de compétences dans la direction des récompenses de l'environnement. Nos résultats montrent que ce paradigme d'apprentissage améliore l'exploration dans des environnements avec des récompenses éparses et permet d'utiliser des compétences sur différentes tâches corrélées. Nous mettons en avant les inconvénients d'ELSIM et proposons un autre modèle, DisTop, pour les corriger. DisTop construit progressivement une topologie de l'environnement en utilisant une fonction de coût contrastive, un réseau auto-organisé et une politique dépendante d'objectifs. L'agent peut alors intelligemment contrôler quelles compétences apprendre ou oublier.

De cette manière, DisTop est compétitif avec des algorithmes de l'état de l'art sur trois types de tâches différentes, incluant une tâche hiérarchique avec des récompenses éparses.

Pour conclure la thèse, nous discutons des perspectives du domaines et des directions futures de notre recherche.

above all, you let me the chance to autonomously experiment my own research while supporting me with strategic advices and critical methodologies. Salima, now that you made the famous home barbecue, I can peacefully end my thesis and I will not be there to harass you anymore ! I thank all the members of the SyCoSMA team for being involved in our regular meetings. Finally, almost everyone participated ! Especially Mathieu, for your dynamic implication and for always being open to scientific discussions. Overall, I hope we will have the opportunity to regularly meet again ! I would like to mention all the persons that made this experience socially pleasant and interesting. This particularly includes Theo, Rémy, Yacine, Mehdi, Simon, Julien, Huan, Antoine, but also Bastien, Mouna, Pierre and other temporary interns. Simon, I'm sure we only got a glimpse of all your games ! Mehdi, I hope you will manage to improve your football skills and one day you may make a true nutmeg.

Finally, I would like to particularly thank my brother Antoine for his wise advices and comforting speeches; I hope we will have the opportunity to make a paper together ! I am grateful to my partner, Clarisse, whose supported me all along the scientific journey despite some hard times. That is only the beginning and like in artificial intelligence, the path will be long and tough; however, nowadays, that is a three-way adventure that begins.

Introduction . Context

In this dissertation, we consider the setting where an agent interacts with its environment through reinforcement learning (RL)[Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF]. In RL, an agent learns to solve a task by maximizing the cumulative reward it gathers throught its interactions with its environment. An example of such agent could be a wheeled robot that perceives its environment with a camera (state) and moves (action) to post letters (task). While these works have been limited to very simple tasks for a long time, recent works proposed to merge the generalization power of deep neural networks with standard approaches of RL, creating the field of deep reinforcement learning (DRL) [Mnih et al., ]. These approaches have demonstrated their ability to learn specific tasks using only ground high-dimensional inputs [Mnih et al., ] and ground actions [Haarnoja et al., erarchically organized skills with intrinsic motivation ? This is the motivation of our next contributions.

. . End-to-end learning of reusable skills

Scientific issues. In our analysis, we exhibit that previous approaches mostly learn the skills during an unsupervised pre-training phase (or developmental period [Metzen and Kirchner, ]), this prevents end-to-end exploration and the specialization of skills to a given task or goal. If it is not possible to specialize skills according to a goal, it may also prevent an integration in a hierarchically organized structure. Based on this simple observation, we hypothetize that an agent has to be able to simultenaously learn to solve a task or a goal and learn its bottom-up skills; it results that its learning process has to be end-to-end, i.e without pre-training phase. To validate the hypothesis, we temporally puts aside the hierarchical organization of skills and focus on how to discover skills with intrinsic motivation in an end-to-end way.

ELSIM: end-to-end learning of reusable skills through intrinsic motivation. The second contribution is a model, ELSIM, that improves a previous approach in order to learn a discrete set of skills in an end-to-end way. In particular, it progressively builds a tree of skills in the direction of the feedbacks of the environment. This improvement opens new perspectives: -the agent can now directly solves a task or a goal, making it suitable to be incorporated in a hierarchical setting; -it can explore its environment even though it does not get extrinsic rewards, thus improving exploration. While being end-to-end, skills can still be used across several tasks. Our proof-of-concept experiments validate our hypothesis. However, our thorough analysis highlight that: firstly, exploration is sub-optimal; secondly, learning a discrete set of skills makes it difficult to integrate it in a continually growing hierarchical architecture. These issues motivate our third contribution.

. . Discovering a topological representation while learning diverse skills

The third contribution is a model, DisTop, that learns a topological state-goal representation of its environment while learning the skills to navigate inside it. It overcomes the limitations of ELSIM by learning a continuous goal representation and its discretized version, it selects the skills to improve according to their density (for exploration) or their interest relatively to a task. The agent still learns in an end-to-end way, keeping the best properties of ELSIM.

In our experiments, we show that: -while DisTop enjoys the properties of ELSIM, it clearly outperforms it on several benchmarks; -by forgetting skills and smartly targetting skills to improve, it gets competitive results with state-of-the-art methods on dense rewards single-task, multi-skill discovery benchmarks without rewards and hierarchical tasks with sparse rewards.

With our model, we expect further works to exhibit the possibility and interest of using DisTop while using a hierarchical policy.

. .6 Discussion

In the last chapter, we will summarize our contributions and highlight their perspectives and limitations. In particular we will discuss the questions raised by our works: how to learn a good representation of the environment ? Can we use a unified concept of intrinsic motivation to guide the model proposals ? These questions may be critical for future research in developmental learning, both in the short and long term.

Chapter 2

Background of intrinsic motivation and deep RL

The motivation of an autonomous agent can be two-folds: either it tries to solve a task given by an external entity, in which case it can learn through RL, or it acts according to its own will, in which case it is intrinsically motivated. Both methods have different properties and origins. Intrinsic motivation is a concept derived from the psychological literature [Deci and Ryan, ] to describe the spontaneous inclination of humans to learn and explore the environment [Ryan and Deci, ].

This definition is intuitive but it is unclear how it could be integrated in a computational framework in order for an autonomous agent to enjoy the potential benefits of an intrinsic motivation. In contrast, DRL agents evolve in a well-defined computational framework where an agent perceives the ground state of the environment and executes ground low-level actions to solve one singletask. The main algorithms of RL and DRL are built inside this framework. It results that integrating the intrinsic motivation in the computational framework of RL requires to revisit the standard vision of RL.

In this chapter, we review the basics of RL, DRL, intrinsic motivation and their association. In Section . , we define the framework of RL; this includes the modelisation of the problem as a Markov Decision Process, the core algorithms that drive RL and their recent extensions to DRL algorithms. In particular we highlight a recent state-of-the-art method [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] we will use throughout the rest of the dissertation. Then, in Section . , we focus on developmental learning, in particular lifelong learning and intrinsic motivation: we define them and identify the key properties that characterize an intrinsic motivation. With all these elements in hand, in Section . , we describe the computational framework that merges intrinsic motivation and DRL. Importantly, within this model, we give general key elements that guarantee an efficient learning process to an intrinsically motivated agent.

Throughout the dissertion, we will assume the reader has a basic understanding of deep learning methods, in particular: Convolutional neural networks (CNN), Multi-layer perceptrons (MLP), stochastic gradient descent (SGD), optimizers . . . . For further informations about these aspects, we refer to Goodfellow et al. [ 6].

. . REINFORCEMENT LEARNING Figure . : Illustration of the interaction loop in RL.

. Reinforcement learning

This section is dedicated to the description of the reinforcement learning framework, including the modeling of the problem and the core algorithms that adress it.

. . Markov Decision Process

Considering an autonomous agent that interacts in its environment, we can model this as a Markov Decision Process (MDP) [Puterman, ]. An MDP is defined by:

• S the set of possible states;

• A the set of possible actions;

• T the transition function T : S × A × S → R. T (s ′ , a, s) = p(s ′ |s, a) defines the probability to go in state s ′ ∈ S after making an action a ∈ A in state s ∈ S;

• R the reward function R : S × A × S → R;

• γ ∈ [0, 1] the discount factor;

• d 0 : S → R the initial probability distribution of states.

In this dissertation, T and d 0 define probabilities (discrete argument) or probability density functions (continuous arguments); these functions are strictly positive and their integral equals one.

The interaction loop of an RL agent is illustrated by Figure . . An agent starts in a state s 0

given by d 0 . At each time step t, the agent perceives a state s t and performs an action a t . Then, it waits for the feedback from the environment composed of a state s t+1 sampled from the transition function s t+1 ∼ p(•|s t , a t ), and a reward given by the reward function r t = R(s t , a t , s t+1 ). The agent repeats this interaction loop until the end of an episode. The episode ends when the agent reaches particular states s end ∈ S or after a fixed number of timesteps T episode . At the end of an episode, the agent resets in a starting state s 0 ∼ d 0 (•) and can repeat the loop. This approach is very generic and allows to model a large number of problems that can be described with an agent interacting with its environment.

CHAPTER . BACKGROUND OF INTRINSIC MOTIVATION AND DEEP RL

It may happen that the agent does not access its full state. For instance, as humans, we do not visually perceive what is behind us and we must infer it from our previous observations. In this case, the MDP can be extended to a Partially Observable Markov Decision Process (POMDP) [Kaelbling et al.,[START_REF] Bengio | Curriculum learning[END_REF]. In comparison with a MDP, it adds a set of possible observations O which defines what the agent can perceive and an observation function Ω : S × O → R that defines the probability of observing o ∈ O when the agent is in the state s, i.e Ω(s, o) = p(o|s).

To solve a MDP or POMDP, we need to assess the performance of the algorithm according to a reward-based criteria. For example, a commonly used criteria can be to maximize the discounted cumulated reward gathered throughout an infinite or finite horizon defined by Equation ., where T episode ∈ [1; ∞[. The discount factor γ ∈ [0; 1] defines how much the long-term reward is important in comparison with the short-term reward.

J =   T episode t=0 γ t r t   .
( . )

Similarly, we can also define the finite-horizon undiscounted objective with T episode = ∞:

J =   T episode t=0 r t   . ( . )
Here, the reward is considered as extrinsic (or as a feedback) because the reward function is provided expertly and specifically for the task. We will now study how standard RL agents manage to solve an MDP.

. . Reinforcement learning

A reinforcement learning algorithm aims to associate states s ∈ S to actions a ∈ A through a stochastic policy π : S × A → R. In the specific case of π(a|s) = {0, 1}, the policy becomes deterministic. This policy induces a t-steps state distribution that can be recursively defined as:

d π t (s) = S d π t-1 (s t-1 )
A p(s t |s t-1 , a)π(a|s t-1 )da ds t-1 ( . )

with d π 0 (s) = d 0 . The goal of the agent is then to find the optimal policy π * maximizing the cumulative reward defined in Equation . :

π * = arg max π E at∼π(•|st) ∞ t=0 γ t R(s t , a t , s t+1 )| s0∼d0(•) .
( . )

The simplest strategy to optimize this objective is to -randomly generate policies; -execute them a large number of times to estimate the expectation of cumulated reward. This estimation is called a Monte-Carlo approximation; -Keep the best policy. But this is strongly inefficient since the number of possible policies scales exponentially with the number of states and actions.

Figure . : Illustration of the difficulty to differentiate optimal actions from sub-optimal ones. The black line represents the trajectory of an orange agent that looks for the star.

To make easier the problem, one can search for the optimal actions in each state of the MDP, dividing the issue into more straightforwards issues. This approach amounts to find the action that maximizes the long-term reward in a state s. Following our criteria, we can maximize the expected discounted gain following a policy π from a state s, noted V π (s) (cf. Equation .), or from a state-action tuple, noted Q π (s, a) (cf. Equation .6). It allows to measure the impact of the state-action tuple in obtaining the cumulative reward [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF].

V π (s) = E at∼π(•|st) st+1∼p(•|st,at) ∞ t=0
γ t R(s t , a t , s t+1 )| s0=s .

( . ) 6) This computation also deals with what is called the credit assignment problem, i.e the difficulty to identify the value of an action while marginalizing the effect of other actions from the same trajectory. Figure . illustrates the issue. Before passing the bottleneck, the orange agent made a sub-optimal action; even though it managed to achieve the task, which was reaching the star, it could perform better by fixing this sub-optimal action. Q-learning algorithm tackles this by computing the value of each pair (s, a). When the policy selects the actions, it efficiently identifies the contribution of each action with respect to the final performance of the policy.

Q π (s, a) = E at∼π(•|st) st+1∼p(•|st,at) ∞ t=0 γ t R(s t , a t , s t+1 )| s0=s,a0=a . ( .
There is an optimal Q-function Q * that has the highest value in all states. This is defined by Equation . .

Q * (s, a) = max π Q π (s, a) ∀s ∈ S, ∀a ∈ A.
( . )

CHAPTER . BACKGROUND OF INTRINSIC MOTIVATION AND DEEP RL

If an agent finds out the optimal Q-function, it will be able to act optimally in the environment according to its given task. More formally, the optimal policy is defined by: π * (a t |s t ) = ✶[a t = arg max a Q * (s t , a)] ( .8)

where ✶[cond] is the indicator function which equals if cond is verified, elsewhere. Since the optimal policy derives from the optimal Q-values, the problem amounts to find the optimal Q-function.

One way to approximate the optimal Q-function is to take advantage of the Bellman equation verified by the optimal Q-function [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF]:

Q * (s t , a t ) = E st+1∼p(•|st,at)
R(s t , a t , s t+1 ) + γ max a Q * (s t+1 , a) .

( . )

It tells us that the optimal Q-value of a state-action tuple (s t , a t ) is the reward that results from the interaction plus the maximum future cumulative reward starting from the next state. One can notice that this approximation of Q is off-policy, i.e the right-hand part does not depend on the policy .

We can now apply the Q-learning algorithm: it uses Equation . to define a recursive operator that approximates Q(s t , a t ) using true interactions from the environment. The application of this recursive operator is sometimes called bootstrapping. Equation . describes this recursive approximation where α ∈ [0; 1] is the learning rate and Q prev denotes the Q-value at the previous learning step [Watkins,[START_REF] Bengio | Curriculum learning[END_REF].

Q(s t , a t ) = Q prev (s t , a t ) + α -Q prev (s t , a t ) + R(s t , a t , s t+1 ) + γ max a Q prev (s t+1 , a) Target . ( . )
We recall that an agent does not access the transition function s t+1 ∼ p(•|s t , a t ). So it still has to gather true interesting interactions to apply this operator and correct its Q-values. A naive approach would be to apply its optimal policy according to its current estimation of the optimal Q-function, π(a t |s t ) = ✶[a t = arg max a Q(s t , a)]. But under-approximated Q(s, a) would never be corrected, since the agent would never execute sub-optimal actions in s. In contrast, it has to keep exploring using currently sub-optimal actions to fix badly approximated Q-values. Of course, at the opposite, if it only executes random actions, it may never discover interesting parts of its state space. This is called the exploitation-exploration trade-off: how much should the agent follow its actual optimal policy and how much it should explore ? We briefly introduce two standard exploration strategies:

• The simplest one is the method ǫ-greedy. It allows to select with probability ǫ a random action, and with probability (1ǫ) the current optimal action.

• The Boltzmann exploration selects an action a k according to the probability defined by Equa-

In this dissertation, so as to be concise, we only detail the off-policy setting since this is the one we used in the work, but a large different set of algorithms derive from an on-policy setting. Essentially, the difference comes from whether it takes into consideration the current exploration policy of an agent when it approximates its Q-value [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF]. tion . where τ ∈ R is an hyper-parameter that rules how much the agent explores. The greater it is, the more random is the policy. Intuitively, Boltzmann exploration ponderates the probability to select an action according to its current Q-value.

π(a k |s) = e Q(s,a k ) τ a∈A e

Q(s,a) τ .

( . )

Assuming we access a large table that describes the Q-values of all state-action tuples, by iteratively applying the Bellman operator and stochastically acting in the environment, Q-values progressively converge towards the optimal Q-values [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF].

However, when the states are high-dimensionnal inputs, e.g images filled with 84 × 84 pixels (84×84×3 values), the number of states is huge and directly approximating Q-values using a table turns out to be impossible. In this case, a learning agent has to generalize over its state space to efficiently compute Q-values. For similar reasons, it is not clear how one can solve a MDP with a continuous action or state space. The two next sections will be dedicated to summarize the recent advances in the domain based on deep neural networks.

. . Deep reinforcement learning

To scale on high-dimensional and continuous state and action spaces, recent approaches successfully proposed to approximate Q and π with neural networks, which leaded to the emergence of the field of deep reinforcement learning.

Generalizing over the state space

To generalize over the state space, Mnih et al. [ ] proposed to approximate Q(s, a) with a neural network parameterized by θ: Q θ (s, a) ≈ Q(s, a). This is illustrated in Figure . ; the neural network takes in a state (image with 210 × 160 × 3 dimensions) and outputs the Q-values of each of the two possible actions. However it is not possible to directly apply Equation . due to what has been called the deadly triad [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF]. When an agent uses complex function approximators along with off-policy learning and bootstrapping (like in the Q-learning), the approximation may diverge. Intuitively, this is because modifying a Q-value with Equation .

also modifies its target Q-value (right part of Equation .). It follows that, if some Q-values are more often updated than others, the Q-values can diverge [van Hasselt et al.,[START_REF] Bengio | Curriculum learning[END_REF]. To bypass this issue, the Deep Q-Network (DQN) [Mnih et al., ] proposes two mechanisms:

. To learn on uncorrelated different states, they use a very large buffer B of interactions and learn from interactions sampled from it.

. Rather than using the true Q-value as a target, they use a slowly moving approximation Qθ ′ . Parameters θ ′ are either updated towards θ at a predefined frequency [Mnih et al., ] or they slowly move at each learning step using an exponential moving average [Lillicrap et al., ].

Equation . sums up the resulting objective function, which can be maximized using stochastic gradient descent. Using buffers, we have r t ∼ R(s t , a t , s t+1 ).

L dqn = E

(st,at,rt,st+1)∼B Q θ (s t , a t ) -(r t + γ max a Qθ ′ (s t+1 , a)) 2 ( . )

RL with continuous action spaces

To keep generalizing over the state space while generating continuous actions, SVG [Heess et al., ] also approximates π with a neural network π φ . Figure . illustrates a typical architecture. An Actor neural network takes in a state s and computes the mean µ φ (s) and the co-variance matrix σ φ (s) of a diagonal Gaussian. Actions are sampled from this parameterized Gaussian distribution. Then, actions are bounded by the action space of the MDP using a tanh function (actions generally range in [-1; 1]). After that, a Critic neural network computes the Q-value of an incoming (state,action) tuple. The Critic objective function is essentially the same as in the previous subsection, but takes in actions deterministically output by the Actor to compute the one-dimensional

Image taken from https://montreal.ubisoft.com

. . REINFORCEMENT LEARNING target objective. Thus, in the following, we focus on how to learn the continuous policy. With this architecture, Equation . highlights that the cumulative reward, approximated by Q θ , can be maximized by updating the Actor using SGD. ∇ at Q θ (s t , tanh(a t )) computes the gradient of action with respect to the Q-value approximation, i.e how to change the action to increase the Q-value;

∇ θ π φ (a t |s t ) translates the required changes of action into an update of the parameters of the policy.

∇ θ π J ≈ E st∼B ∇ at Q θ (s t , tanh(a t )) ∇ θ π φ (a t |s t ) with a t ∼ N (µ θ (s t ), σ θ (s t )) ( . )

To allow back-propagation of the gradient through action sampling a t ∼ N (µ θ (s t ), σ θ (s t )), we can apply the reparameterization trick [Kingma and Welling, ]:

. sample ǫ noise ∼ N (0, 1);

. compute a t = µ θ (s t ) + ǫ noise σ θ (s t ).

From now, in continuous cases, we will refer to π as being the parameterized distribution induced by ǫ noise with bounded actions. A variant of this architecture, DDPG [Lillicrap et al., ],

can also learn a deterministic actor. With these methods, the agent typically explores by manually adding either a spherical Gaussian Noise to the actions output by the policy, or a Ornstein-Uhlenbeck process to provide temporally coherent behaviors. In the next subsection, we study a state-of-the-art DRL algorithm based on these improvements.

Soft Actor Critic

Soft Actor-Critic (SAC) [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] proposes to maximize the entropy augmented cumulative reward defined by Equation .. By adding an entropy term H(A) = a∈A p(a) log p(a)

weighted by the hyper-parameter α, SAC learns a stochastic actor that directly learns to explore. α directly controls the degree of exploration, the more α is important, the more the agent explores independently from the rewards. They show that it considerably improves and stabilizes the learning process on several environments.

π * = arg max π E at∼π(•|st) ∞ t=0 γ t (R(s t , a t , s t+1 )| s0∼d0 + αH(π(•|s t )) . ( . )
It follows two new objectives to learn the Q-values (Equation .) and the policy (Equation .6). For the Q-value approximation, the gradient associated to the entropy term is approximated through Monte-Carlo sampling of log-probabilities. Intuitively, the policy outputs a distribution of actions that matches the α-weighted distribution of exponential Q-values over actions. Of course, the repartition of the Q-values over the state space changes considerably in comparison with the usual objective (Equation .), the entropy term strongly favors states where the agent can act stochastically.

CHAPTER . BACKGROUND OF INTRINSIC MOTIVATION AND DEEP RL

J Q (θ) = E (st,at,rt,st+1)∼B at+1∼π φ (•|st+1) (Q θ (s t , a t ) -(r t + γ ( Qθ ′ (a t+1 , s t+1 ) -α log π φ (a t+1 |s t+1 ))) 2 . ( . ) J π (φ) = E st∼B E at∼π φ (•|st) α log π φ (a t |s t ) -Q θ (s t , a t ) . ( . 6)
Additionally to these objective functions, it is possible to fix the over-estimation bias of Qlearning based models [Fujimoto et al.,[START_REF] Bengio | Curriculum learning[END_REF]Van Hasselt et al.,6,Hasselt,]. This bias occurs when an agent bootstraps its Q-values using the best possible action in the next state, making Q-values over-estimated in comparison with the true Q-value. Over-estimated Q-values tend to be more often bootstraped making other Q-values also over-estimated. Thus, it can prevent an agent to efficiently converge, being stuck in some over-estimated local optimums. To overcome this issue, TD [Fujimoto et al.,[START_REF] Bengio | Curriculum learning[END_REF] proposes to under-estimate Q-values using a double-critic architecture; in fact, an under-estimation is better than an over-approximation since the error may not be recursively propagated. It is possible to incorporate this improvement in the SAC objective; there is now two approximations of the Q-function, Q 1 θ 1 and Q 2 θ 2 with their own slowly moving approximation Q1

θ ′ 1 and Q2

θ ′ 2 . Using Q min θ ′ = min( Q1 θ ′1 (a t+1 , s t+1 ), Q2 θ ′2 (a t+1 , s t+1
)), Equation . sums up the update when the minimum over two Q-function acts as the target Q-value. The same equation applies to update Q 2 θ2 .

J Q 1 (θ 1 ) = E (st,at,rt,st+1)∼B at+1∼π φ (•|st+1) Q θ (s t , a t ) -(r t + γ (Q min θ ′ -α log π φ (a t+1 |s t+1 ))) 2 . ( . )
Because of its efficiency, most of the methods we will study in the rest of the dissertation are based on SAC. Now that we reviewed some core components of DRL, we will study how we can take advantage of the generalization ability and long-term maximization of DRL agents to build intrinsically motivated agents. We first define two components of develomental learning: lifelong learning and intrinsic motivation, then we study the typical properties of an intrinsic motivation.

.

Developmental learning

Developmental learning refers to the design of models that directly draws inspiration from human cognitive development; such model of agent should autonomously build its knowledge by interacting with its world [Guerin, , Nguyen et al., ]. In this section, we describe two aspects of developmental learning: a developmental agent ( ) continually acts and perceives its environment even though its objective may change and ( ) autonomously and actively acquires knowledge through an intrinsic motivation.

. . DEVELOPMENTAL LEARNING

. . Definition of lifelong learning

Biological organisms face an uninterrompted stream of data all along their lifetime. During this lifetime, they keep acting according to different objectives that are assigned to them, whether they are self-assigned or attributed by an external entity. Typically, a child may play with its cubes and change its activity to play with, for instance, a puzzle: This ability to switch between activities is called lifelong learning or continual learning [Thrun and Mitchell, , Parisi et al., , Lesort et al., ].

In the machine learning domain, this generally results in an agent that learns to solve several different tasks sequentially. However, such setting introduces several new considerations. Firstly, the agent should be able to transfer and reuse the knowledge acquired while carrying out a task to another task [Parisi et al., ]. For instance, if a baby knows how to grasp a ball, it should provide him/her information about how to grasp a cube. This is refered as transfer learning. Secondly, the agent must avoid forgetting how to solve previously learnt tasks, which is not trivial when an agent uses neural networks [Parisi et al., , Lesort et al., ] since a newly learnt task may overwrite a previous learnt task. For instance, if a child learns to shoot a ball after learning to grasp it, it must not forget how to grasp it. This forgetting issue is refered as catastrophic forgetting.

. . Definition of intrinsic motivation

According to Singh et al. [ ], evolution provides a general intrinsic motivation (IM) function that maximizes a fitness function based on the survival of an individual. Curiosity, for instance, does not immediately produce selective advantages but enables the acquisition of skills providing by themselves some selective advantages. More widely, the use of intrinsic motivation allows to obtain intelligent behaviors which may later serve goals more efficiently than with only a standard reinforcement [Baldassarre and Mirolli,a,Baldassarre,,Lehman and Stanley,[START_REF] Bengio | Curriculum learning[END_REF].

Typically, a student doing his mathematical homework because he/she thinks it is interesting is intrinsically motivated whereas his/her classmate doing it to get a good grade is extrinsically motivated [Ryan and Deci, ]. In this future, the intrinsically motivated student may be more successful in math than the other one. It questions the relevance of using only standard reinforcement methods.

Simply stated, intrinsic motivation is about doing something for its inherent satisfaction rather than to get a positive feedback from the environment [Ryan and Deci, ]. Looking at this definition, one can notice that intrinsic motivation is defined by contrast with extrinsic motivation;

it highlights the difference between the two paradigms. Intrinsic motivation assumes the agent learns on its own while extrinsic motivation assumes there exits an expert that supervises the learning process.

More rigorously, Oudeyer and Kaplan [START_REF] Bengio | Curriculum learning[END_REF] explain that an activity is intrinsically motivating for an autonomous entity if its interest depends primarily on the collation or comparison of information from different stimuli and independently of their semantics. At the opposite, an extrinsic reward results of an unknown environment static function which does not depend on previous experience of the agent on the considered environment. The main point is that the agent must not have any

CHAPTER . BACKGROUND OF INTRINSIC MOTIVATION AND DEEP RL

a priori on the semantic of the observations it receives. Here the term stimuli does not refer to sensory inputs, but more generally to the output of a system which may be internal or external to the independent entity, thereby including homeostatic body variables (temperature, hunger, thirst, attraction to sexual activities . . . ) [Baldassarre,,Berlyne,6 ]. Broadly speaking, the motivation of an agent can be internal (source of motivation) while still being extrinsic (why of the actions). For instance, when an agent is looking for food because of the hunger, hunger is a stimuli coming to the cognitive system of the agent such that it is an internal but extrinsic motivation.

As an other example, a child may do his/her homeworks because he/she thinks it will be crucial to latter get a job. While the source of the motivation is internal, the true outcome comes from the environment.

Now that the we clarified the notion of intrinsic motivation, the next section provides a computational perspective on intrinsic motivation and study its properties. An extensive overview of IM can be found in Barto [ ].

. .

Properties of intrinsic motivation

While the previously mentioned definition encompasses the search for desirable stimulus properties such as surprisingness [Berlyne,6 ], an intrinsically motivated entity could also search their exact opposite, such as boredom. It highlights that it is not enough to say that an agent is intrinsically motivated; different kind of intrinsic motivations can co-exist and each one may have different properties. Examples include "novelty", "surprisingness", "complexity", "ambiguity" . . . [Oudeyer and Kaplan,[START_REF] Bengio | Curriculum learning[END_REF]Berlyne,6 ]. However it is possible to exhibit typical properties.

The notion of intrinsic motivation is strongly related to its homeostatic property. The agent, while acting to fulfil its intrinsic motivation is progressively satisfied and the interest of the agent moves towards other things. As an example, this is especially visible when one consider exploratory motivations. Let us consider an agent that moves in a maze. When an agent discovers a new part of its environment, at first it is satisfied and keeps exploring this area; then, one expect it will progressively get bored and may move away, this is in fact the core interest of exploratory behaviors. It highlights that intrinsic motivations do not bring static behaviors but temporary behaviors that rule the search for information.

We can sum up and translate the definition of intrinsic motivation into several key properties that characterize how intrinsic is a motivation for an agent.

• The motivation has to be task-agnostic. By task, we refer to a goal attributed by an external entity (like in the RL framework described in Section . . ).

• The agent must be guided by the information contained in its previous interactions. This directly follows the definition of Oudeyer and Kaplan [START_REF] Bengio | Curriculum learning[END_REF].

• The motivation has to be dynamic, or in other words, non-stationary.

• The agent does not access a priori knowledge on its environment.

Looking at these properties, we can see how intrinsic motivation relates to standard learning schemes in machine learning. In the next section, we will see how to formally incorporate these properties to make DRL agents intrinsically motivated.

. Intrinsic motivation in DRL . . A model of RL with intrinsic rewards

Reinforcement learning is derived from behaviorism [Skinner,[START_REF] Bengio | Curriculum learning[END_REF] and usually uses extrinsic rewards [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF]. However Singh et al. [ ] and Barto et al. [ ] reformulated the RL framework to incorporate IM. We can differentiate rewards, which are events in the environment, and reward signals which are internal stimulis to the agent. Thus, what is named reward in the RL community is in fact a reward signal. Inside the reward signal category, there is a distinction between primary reward signals and secondary reward signals. The secondary reward signal is a local reward signal computed through expected future rewards and is related to the value function (cf. Equation .6) whereas the primary reward signal is the standard reward signal received from the MDP.

In addition, rather than considering the MDP environment as the environment in which the agent achieves its task, it suggests that the MDP environment can be formed of two parts: the external part which corresponds to the potentiel task and the environment of the agent; the internal part which computes the MDP states and the secondary reward signal using potentially previous interactions. Consequently, we can consider an intrinsic reward as a reward signal received from the MDP environment. The MDP state is no more the external state but an internal state of the agent. However, from now, we will follow the terminology of RL and the term reward will refer to the primary reward signal.

Figure . summarizes the new framework: the critic is in the internal part of the agent, it computes the intrinsic reward and deals with the credit assignment. The agent can merge intrinsic rewards and extrinsic rewards in its internal part. The state includes sensations and any form of internal context; in this section we refer to this state as a contextual state. The decision can be a This conceptual model incorporates intrinsic motivations into the formalism of MDP. Now, we will review how this model is instantiated in practice. Indeed it is possible to extend RL to incorporate the three new components that are intrinsic rewards, high-level decisions and contextual states. We separately study them in the following sections.

. . Intrinsic rewards and information theory

Throughout our definition of intrinsic motivation, one can notice that the notion of information comes up a lot. As we will see in Chapter , this is not hazardous and quantifying information proves useful to generate intrinsic rewards. In this section, we provide the basics about information theory and explain how to use it to combine intrinsic and extrinsic rewards. However, we emphasize that intrinsic rewards are not restricted to information measures and their characterization mostly depends one whether the reward function fits the properties of an intrinsic motivation.

Information theory. The Shannon entropy quantifies the mean necessary information to determine the value of a random variable. Let X be a random variable with a law of density p(X) satisfying the normalization and positivity requirements, we define its entropy by:

H(X) = - X p(x) log p(x). ( . 8)
In other words, it allows to quantify the disorder of a random variable. The entropy is maximal when X follows an uniform distribution, and minimal when p(X) is equal to zero everywhere except in one value, which is a Dirac distribution. From this, we can also define the entropy conditioned on a random variable S. It is similar to the classical entropy and quantifies the mean necessary information to find X knowing the value of an other random variable S: ) .

H(X|S) = - S p(s) X p(x|s) log p(x|s). ( .

. INTRINSIC MOTIVATION IN DRL

The mutual information allows to quantify the information contained in a random variable X about an other random variable Y . It can also be viewed as the decrease of disorder brought by a random variable Y on a random variable X. The mutual information is defined by:

I(X; Y ) = H(X) -H(X|Y ) ( . )
We can notice that the mutual information between two independent variables is zero (since

H(X|Y ) = H(X)).
Similarly to the conditional entropy, the conditional mutual information allows to quantify the information contained in a random variable about an other random variable, knowing the value of a third one. It can be written in various ways:

I(X; Y |S) = H(X|S) -H(X|Y, S) ( . ) = H(Y |S) -H(Y |X, S) ( . ) = H(X|S) + H(Y |S) -H(X, Y |S) = D KL p(X, Y |S)||p(X|S)p(Y |S) ( . )
We can see with Equation . and Equation . that the mutual information is symmetric and that it characterizes the decrease in entropy on X brought by Y (or inversely). Equation . defines the conditional mutual information as the Kullback-Leibler divergence [Cover and Thomas, ] between distribution P (Y, X|S) and the same distribution if Y and X were independent variables (the case where

H(Y |X, S) = H(Y |S)).
It is straightforward to generalize mutual information to several random variables, leading to the multi-information [Slonim et al., ]:

I(X 1 , . . . , X N ) = D KL p(X 1 , . . . , X N )||p(X1) . . . p(X N ) ( . )
It quantifies the information that variables X 1 , . . . , X N contain about each other. As above, it is described as the discrepancy between the joint distribution p(X 1 , . . . , X N ) and the same distribution if variables were independent.

For further information on these notions, the interested reader can refer to Cover and Thomas

[

]. Let us focus again on intrinsic rewards to give an example. By computing the mutual information between actions and the resulting states, an agent can provide intrinsic rewards that measure how much it controls its environment using one action. Other methods compute reward so that the agent learns a policy that maximizes the entropy of visited states. We give more details about this aspect in Chapter .

Combining intrinsic rewards and extrinsic reward

In practice, there are multiple ways to integrate an intrinsic reward into a RL framework. The main approach is to compute the agent's reward r as a weighted sum of an intrinsic reward r int and an extrinsic reward r ext : r = αr int + βr ext [Kakade and Dayan,,Burda et al.,[START_REF] Bengio | Curriculum learning[END_REF]Gregor et al.,6]. Of course, one of the weighting coefficient α and β can be set to . In this version, one can think of the intrinsic reward as an intrinsic bonus. When the extrinsic value function is important to compute the intrinsic reward or CHAPTER . BACKGROUND OF INTRINSIC MOTIVATION AND DEEP RL when the hyper-parameters α and β may change, the sum can be made at the Q-value level, i.e.

Q(s, a) = αQ int (s, a) + βQ ext (s, a) [Kim et al., b] where α and β ponderate each Q-value. As noticed in Section . . , intrinsic rewards generally evolve over time; while it may often decrease to zero [Baldassarre, ], this is not a general rule and it may happen that the agent cannot find an optimal stationary policy. ]. This problem refers to the fact that rewards can occur with a temporal delay and will only very weakly affect all temporally distant states that have preceded it, although these states may be important to obtain that reward. Indeed, the agent must propagate the reward along the entire sequence of actions (through Equation .) to reinforce the first involved state-action tuple. This process can be very slow when the action sequence is large. This problem also concerns determining which action is decisive for getting the reward, among all actions of the sequence (see also Section . . . In contrast, if an agent can take advantage of temporally-extended actions, a large sequence of low-level actions become a short sequence of time-extended decisions that make easier the propagation of rewards. This goal setting mechanism can be extended to create managers of managers so that an agent can recursively define increasingly abstract decisions as the hierarchy of RL algorithms increases. Relatively to Figure . , the internal environment of a RL module becomes the lower level module.

. . Decisions and hierarchical RL

We can model these decisions as options. An option op ∈ O is defined through components:

• A set of starting states I ⊂ S from which an option can be applied.

. . INTRINSIC MOTIVATION IN DRL

• A policy (or worker) that is responsible of achieving the options with lower-level actions. This is studied in the next section.

• A completion function F that specifies the probability of completing the option in each state.

Typically, the starting state can derive from d 0 (all options start at the beginning of an episode)

or the full set of states S (options can start everywhere). The completion function can also set a probability 0 everywhere [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF], in this case, it ends at the same time as an episode. Such specific cases often occur [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]. Options where originally learnt during a pre-training phase with exclusively extrinsic rewards [Sutton et al., ], it was meant to take advantage of expert knowledge on the task. However, in our framework, we are interested in intrinsically motivated agent, so, in the next section, we take a closer look on how to learn the policies that learn to achieve goals using intrinsic motivation. In particular, we will define goals, skills and explain how to build a contextual state.

. . Goal-parameterized RL

Usually, RL agents solve only one task and are not suited to learn multiple tasks. Thus, an agent is unable to generalize across different variants of a task. For instance, if an agent learns to grasp a circular object, it will not be able to grasp a square object. In the developmental model described in Section . . , the decisions can be hierarchically organized into several levels where an upperlevel takes decision (or sets goals) that a lower-level has to satisfy. This questions: -how a DRL algorithm can make its policy dependent on the goal set by its upper-level decision module ? -How to compute the intrinsic reward using the goal ? These issues rise up a new formalism based on developmental machine learning [Colas et al., b].

In this formalism, a goal is defined by the pair (g, R G ) where G ⊂ R d , R G is a goal-conditioned reward function and g ∈ G is the d-dimensional goal embedding. This contrasts with the notion of task which is proper to an extrinsic reward function assigned by an expert to the agent.

With such embedding, one can generalize DRL to multi-goal learning, or even to every available goal in the state space, with the Universal Value Function Approximator (UVFA) [Schaul et al., ].

UVFA integrates, by concatenating, the state goal embedding g with the state of the agent to create a contextual state c = (g, s). Depending on the semantic meaning of a skill, we can further enhance the contextual states with other actions or states executed after starting executing the skill (cf. Section . ).

We can now define the skill associated to each goal as the goal-conditioned policy π g (a|s) = π(a|g, s); in other words, a skill refers to the sensorimotor mapping that achieve a goal [Thill et al., ]. This skill may by learnt or unlearnt according to the expected intrinsic rewards it gathers. It implies that, if the goal space is well-constructed (as often a ground state space for example, R G = S), the agent can generalize its policy across the goal space, i.e the corresponding skills of two close goals are similar.

For example, let us consider an agent moving in a closed maze where every position in the maze can be a goal. We can set G = S and set the intrinsic reward function to be the euclidian distance between the goal and the current state of the agent R G : S × G → R, (s, g) → ||s -g|| 2 .

CHAPTER . BACKGROUND OF INTRINSIC MOTIVATION AND DEEP RL

This formalism completes the instantiation of the architectures described in Section . . . Now we will explain how, in practice, one can efficiently learn the goal-conditioned policy.

. . Efficient learning with goal relabelling

When the goal space is a continuous state space, it is difficult to determine whether a goal is reached or not, since two continuous values are never exactly equal. Hindsight experience replay (HER) [Andrychowicz et al., ] tackles this issue by providing a way to learn on multiple objectives with only one interaction. With author's method, the agent can use an interaction done to accomplish one goal to learn on an other goal, by modifying the associated intrinsic reward. This mechanism greatly improves the sample efficiency since it avoids to try all interactions for every goals.

Let us roll out an example. An agent acts in the environment to gather a tuple (s, s ′ , r g , a, g) where r g is the reward associated to the goal g. The agent can learn on this interaction, but can also use this interaction to learn other goals; to do so, it can change the goal into a new goal and recompute the reward, resulting in a new interaction (s, s ′ , r g ′ , a, g ′ ). The only constraint for doing this is that the reward function R(s, a, s ′ , g ′ ) has to be known, which is the case with an intrinsic reward function. Typically, an agent can have a goal state and a reward function which is 1 if it is into that state and 0 otherwise. At every interaction, it can change its true goal state for its current state and learn with a positive reward.

. Conclusion

We considered the setting where an agent interacts with its environment. We have highlighted several recent advances allowing to scale RL methods to high-dimensional state space and continuous action space (Section . ). But one needs other mechanism to make a learning agent truly independent from external interventions. Intrinsic motivation may play this role; while it comes from psychological literature, we emphasized its key properties so that we can use it in computational frameworks in Section . . By theoretically reconsidering the RL framework and taking advantage of how DRL methods learn, we illustrated how intrinsic motivation can be efficiently translated into intrinsic rewards, HRL and goal-conditioned RL (Section . ).

However, we notice there are several ways to instantiate and apply an intrinsic motivation; the effects are different for each of them. This is reflected in practice: one may use goal-conditioned RL at a single decision-level or may not use any notions of goals, using only flat intrinsic rewards. Even across similar architectures, intrinsic rewards may be computed using very different heuristics; in addition, it may happen that R G depends on the structure of G so that computing G may also influence the intrinsic reward. In each method, it may differently overcome a limitation of DRL. This observation motivates our work and we analyze the current literature in the next chapter.

. . CONCLUSION Chapter 3

Survey on intrinsic motivation in DRL

In the previous chapter, we studied the evolution of DRL and gave a general overview on how to formally integrate intrinsic motivations with DRL. In this chapter, we study the role of intrinsic motivation in the framework of DRL.

Indeed, despite the recent improvements of DRL approaches (several are detailed in Section . ), they turn out to be most of the time unsuccessful when the rewards are scattered in the environment, as the agent is then unable to learn the desired behavior for the targeted task [START_REF] Bengio | Curriculum learning[END_REF]. Moreover, the behaviors learned by the agent are hardly reusable, both within the same task and across many different tasks [START_REF] Bengio | Curriculum learning[END_REF]. It is difficult for an agent to generalize its skills so as to learn to make high-level decisions in the environment. For example, such skill could be go to the door using primitive actions consisting in moving in the four cardinal directions; or even to move forward controlling different joints of a humanoid robot like in the robotic simulator MuJoCo [Todorov et al., ].

For several years now, IM is increasingly used in RL, fostered by important results and the emergence of deep learning. This paradigm offers a greater learning flexibility, through the use of a more general reward function, allowing to tackle the issues raised above when only an extrinsic reward is used. Typically, IM improves the agent ability to explore its environment, to incrementally learn skills independently of its main task, to choose an adequate skill to be improved and even to create a representation of its state with meaningful properties. In addition, as a consequence of its definition, IM does not require additional expert supervision, making it easily generalizable across environments.

A lot of very different works pretend to fit in the framework of IMs by proposing task-agnostic losses [Kulkarni et al.,6,Zhang et Scope and motivation of our review. In this chapter, we make the choice to study and group together methods that maximize information theoretic objectives. We closely match the definition of IM provided in Section . . and we characterize it as the search for correlations among a set of internal variables. This way, we revisit the notions of surprise, novelty and skill learning and show that they can encompass numerous works. This allows us to situate a large body of works, to highlight important directions of research and to unify these intrinsic motivations using information theory.

Related works.

The overall literature on IM is huge [Barto, ] and we only consider its application to DRL. Therefore, our study of IMs is not meant to be exhaustive. on intrinsic motivation through the lens of information theory. We assume that our work is in line with the work of Schmidhuber [START_REF] Bengio | Curriculum learning[END_REF], which postulates that organisms are guided by the desire to compress the information they receive. However, by reviewing the more recent advances in the domain, we formalize the idea of compression with the tools from information theory.

To sum up, in this chapter, we investigate the use of IM in the framework of DRL and consider the following aspects:

• The role of IM in addressing the challenges of DRL.

• Classifying current works through information theoretic principles.

• Important outlooks of IM in RL.

• Proposal of an unified view of IMs.

This chapter is organized as follows. As a first step, we highlight the main current challenges of RL and identify the need for an additional outcome (Section . ). Then, we briefly explain our classification (Section . ), namely surprise, novelty and skill learning and we detail how current works fit it (respectively Section . , Section . and Section . ). Thereafter, we highlight some important outlooks of the domain (Section .6). Finally, in Section . , we unify intrinsic motivations under one information theoretic objective function and discuss its potential role as a universal guiding principle. as a result, it stays in its surrounded area because of the temporal inconsistency of its behaviour.

On the right side, we can imagine an ideal exploration strategie where the agent covers the whole state space to discover where rewards are located.

. Challenges . . Sparse rewards

Classic RL algorithms operate in environments where the rewards are dense, i.e. the agent receives a reward after almost every completed action. In this kind of environment, naive exploration policies such as ǫ-greedy [Sutton and Barto,[START_REF] Bengio | Curriculum learning[END_REF] Rather than working on an exploration policy, it is common to shape an intermediary dense reward function that adds to the reward associated to the task in order to make the learning process easier for the agent [Su et al., ]. However, the building of a reward function often reveals several unexpected errors [Ng et al.,,Amodei et al.,6] and most of the time requires expert knowledge. For example, it may be difficult to shape a local reward for navigation tasks. Indeed, one has to be able to compute the shortest path between the agent and its goal, which is the same as solving the navigation problem. On the other side, the automation of the shaping of the local reward (without calling on an expert) requires too high computational resources [Chiang et al., ].

We will see in Section . , Section . how IM is a valuable method to encourage exploration in a sparse rewards setting.

. . Temporal abstraction of actions

As argued in Section . . , skills, through hierarchical RL, are a key element to speed up the learning process since the number of decisions to take is significantly reduced when skills are used. In particular, they make easier the credit assignment. Skills can be manually defined, but it requires some extra expert knowledge [Sutton et al., ]. To avoid providing hand-made skills, several works proposed to learn them with extrinsic rewards [Bacon et al., , Li et al., b]. However, if an agent rather learns skills in a bottom-up way, i.e with intrinsic rewards rather than extrinsic rewards, learnt skills become independent from possible tasks. This way, skills can be reused across several tasks to improve transfer learning [Aubret et al.,,Heess et al.,6] and an agent can learn skills even though it does not access rewards, improving exploration when rewards are sparse [Machado et al., ]. Let us illustrate both advantages.

Exploration when rewards are sparse. Figure . illustrates the benefit in terms of exploration when an agent hierarchically uses skills. The green circle can use a skill Go to the far right, to reach the rewarding star while the orange agent can only use low-level cardinal movements. The problem of exploration becomes trivial for the agent using skills, since one exploratory action can lead to the reward. In contrast, it requires an entire sequence of specific low-level actions for the other agent to find the reward. This problem arises from the minimal number of specific actions needed to get a reward (see also Section . . ). A thorough analysis of this aspect can be found in [Nachum et al., b].

Reusing skills across several tasks. Skills learnt with intrinsic rewards are not specific to a task. Assuming an agent is required to solve several tasks in a similar environment, i.e a single MDP with a changing extrinsic reward function, an agent can execute its discovered skills to solve all tasks. Typically, in Figure . , if both agents learnt to reach the star and we move the star somewhere else in the environment, the green agent would still be able to execute Go to the far right and executing this skill may make the agent closer to the new star. In contrast, the orange agent would have to learn a whole new policy.

In Section . , we provide insights on how an agent can discover skills in a bottom-up way.

.

Classification of methods

In order to tackle the problem of exploration, an agent may want to identify and return in rarely visited states or unexpected states, which can be quantified with current intrinsic motivations. We will particularly focus on two objectives that address the challenge of exploring with sparse rewards, each with different properties: maximizing novelty and surprise. We formalize novelty and surprise through the lens of information theory (in respectively Section . and Section . ) and the works that instantiate it. Surprise and novelty are specific notions that have often been used in an interchanged way and we are not aware of a currently unanimous definition of novelty [Barto et al., ]. The third notion we study, skill learning, focuses on the issue of skill abstraction.

Table . sums up our taxonomy. We classify intrinsic motivations in three categories of objectives based on information theory that reflects the high-level studied concepts of novelty, surprise and skill learning. In practice, we mostly take advantage of the mutual information to provide a quantity for our conceptual objectives. These objectives are compatible with each other and may be used simultaneously, as argued Section .6. . Within each category of objectives, we additionnally highlight several ways to maximize each objective and provide details about the underlying methods of the literature. (1 + α skew ) log p(s g )

Surprise

α skew < 0 (Goal selection policy)
Table . : Summary of our taxonomy of intrinsic motivations in DRL. The function f outputs a part of the trajectories T , Z and G are internal random variables respectively denoting state representations and self-assigned goals. Please, refer to the corresponding sections for more details about methods and notations. I(S; Z). We divide this objective maximization into two kinds of methods: a direct maximization of a parametric entropy of states, and an entropy maximization based on a k-nearest neighbors approximation.

Skill learning:

We formalize skill learning as maximizing the mutual information between a goal representation G and a part of a time-extended trajectory f (T ), I(G; f (T )) while following G. We will consider two ways to achieve this: -fixing the goal distribution; -deriving the goal representation from the state space. We will see that the second point also needs to maximize the entropy of goals-states.

We justify our objective within each category and study the different ways to maximize this objective and the advantages/disadvantages. In practice, surprise and novelty are currently maximized as a flat intrinsic motivation, i.e without using hierarchical decisions. This mostly helps to improve exploration when rewards are sparse. In contrast, skill learning allows to define timeextended hierarchical skills which enjoy all the benefits argued in Section . . .

. Surprise

In this section, we study methods that maximize the surprise. Firstly, we formalize the notions of surprise, then we will study three approaches for computing intrinsic rewards based of these notions.

. . Definition of surprise

In this section, we assume the agent learns either a density model (Section . . ) or a forward model of the environment (Sections . . and . . ) parameterized by φ ∈ Φ. The density model induces a marginal distribution of state p(S|φ) and a forward model computes the nextstate distribution conditioned on a tuple state-action p(S ′ |S, A, φ). Typically, this can be the parameters of a neural network. Trying to approximate the true model, the agent maintains an approximate distribution p(Φ|h) of models, where h t = h refers to the ordered history of interactions ((s 0 , a 0 , s 1 ), (s 1 , a 1 , s 2 ), . . . , (s t-1 , a t-1 , s t )). In this section, h simulates a dataset of interactions, we use it to clarify the role of the dataset. It is important to notice that the policy feeds this h.

In this case, surprise quantifies the mismatch between an expectation and the true experience of an agent [Barto et al., , Ekman and Davidson, ]. In this paper, we refer to the definition of Itti and Baldi [ ], which define it as the discrepancy between a prior distribution of beliefs and the posterior probability distribution following an observation [Itti and Baldi, , Storck et al., ]. If an agent maximizes the surprise over a model through interactions with the environment, which is often the case [Barto et al., ], it leads to the expected information gain objective [Sun et al., ]. Intuitively, the agent returns in states where it experienced an unexpected transition. Using the KL-divergence to assess the discrepancy, surprise can be computed as D KL (p(Φ|h t+1 )||p(Φ|h t )) where φ ∈ Φ are parameters of a model and t denotes the timestep.

. . SURPRISE

In this case, the agent has a prior distribution about model parameters p(Φ) and this model can be updated using the Bayes rule:

p(φ|h, s, a, s ′ ) = p(φ|h) p(s ′ |h, s, a, φ) p(s ′ |h, s, a) . ( . )
The expected information gain [Sun et al., , Little and Sommer, ] over a forward or density model parameterized by φ can be formulated as:

IG(h, A, S ′ , S, Φ) = E (s,a)∼π s ′ ∼p T (•|s,a,h) D KL (p(Φ|h, s, a, s ′ )||p(Φ|h)).
( . )

≈ I(S ′ ; Φ|h, A, S)
where p T refers, in this section, to the true probability induced by the transition fonction of the environment. Actively maximizing the expected information gain amounts to reduce the uncertainty of the model. We emphasize that p(φ|h) = p(φ|h, a, s) since only full transitions provide informations about the true dynamics of the environment. In this case, p(s ′ |s, a, h) does not refer to the probability induced by the environment, but rather to the probability induced by the current history of transitions. This is stressed out by writing:

p(s ′ |s, a, h) = φ∈Φ p(s ′ |s, a, h, φ)p(φ|s, a, h).
( . )

Let us assume φ are parameters of a forward model. When the history gets large enough, we hypothetize that the distribution p(φ|s, a, h) becomes not null only for φ being equal to the probability induced by the true environment transition function φ T . In this case, p(s ′ |s, a, h) indeed refers to the probability of a transition, i.e p(s ′ |s, a, h) = p(s ′ |s, a, h, φ T ) . Similar reasoning can be obtained if φ encodes a density model and if the environment's marginal distribution of states is the uniform one.

In the following, we will study three objectives: the expected information gain over forward models, an approximation of surprise over forward models leading to a minimax game and the expected information gain over density models.

. . Information gain over forward model

We first study the works that maximize the expected information gain over forward models.

Here, φ are parameters of a forward model. Using Equation . , we can extract an intrinsic reward:

R(s, a, s ′ ) = D KL (p(Φ|h, s, a, s ′ )||p(Φ|h)). ( . )
This way, an agent executes actions that provide information about the dynamics of the environment. This allows, on one side, to push the agent towards areas it does not know, and on the ]. The interest of Bayesian approaches is to be able to measure the uncertainty of the learned model [Blundell et al., ]. This way, assuming a fully factorized Gaussian distribution over model parameters, the KLdivergence has a simple analytic form [Houthooft et al.,6,Linke et al.,], making it easy to compute. However, the interest of the proposed algorithm is shown only on simple environments and the reward can be computationally expensive to compute. Achiam and Sastry [ ] propose a similar method (AKL), with comparable results, using deterministic neural networks, which are simpler and quicker to apply. The weak performance of both models is probably due to the difficulty to retrieve the uncertainty reduction by rigorously following the mathematical formalism of information gain.

In JDRX [Shyam et al., ], authors show that one can maximize the information gain by computing the Jensen-Shannon or Jensen-Rényi divergence between distributions of states induced by several forward models. The more the models are trained on a state-action tuple, the more they will converge to the expectated distribution of next states. Intuitively, the reward represents how much the different transition models disagree on the next-state distribution. Other works also maximize a similar form of disagreement ]. However the main intrinsic issue is computational since it requires multiple forward models to train.

To conclude, despite the theorical power of the information gain for improving exploration, it remains hard to efficiently estimate it and use it in difficult tasks. In the next paragraph, we review how we can simplify the expected information gain.

. . Prediction error of forward model

Using a deterministic forward model φ, one can also maximize the surprise using several approximations.

I(S

′ ; Φ|h, A, S) ≈ H T (S ′ |h, A, S) -H T (S ′ |A, Φ, S, h) = - E (s ′ ,s,a)∼π log p(s ′ |h, s, a) + E φ∼p(•|h,s,a,s ′ ) (s ′ ,s,a)∼π log p(s ′ |s, a, φ, h) ( . )
where H T refers to the entropy with true transitions in its expected part. Equation . means that, in order to maximize the expected information gain, the agent should go in areas where it reduces the entropy of its approximation; indeed, the right-hand term predicts with one more transition than the left-hand term. Now, we will make the strong assumption that the second term of Equation . can be optimally maximized through the loss of the predictive model if

H T (S ′ |h, A, S)
is maximized using actions. A fully deterministic environment is a particular case of such setting . . SURPRISE since the second term would converge to 0 for all transitions. In this case, the agent just has to go in stochastic areas. One can lower-bound the first term using the stochastic forward model:

- E (s ′ ,s,a)∼π log p(s ′ |h, a, s) = - E (s ′ ,s,a)∼π log E φ∼p(•|h) p(s ′ |h, a, s, φ) ( .6) ≥ - E φ∼p(•|h) (s ′ ,s,a)∼π log p(s ′ |φ, h, a, s) ( . )
where we applied the Jensen inequality. The two final terms of Equation . are essentially similar while being optimized by different parts of the model (actions and forward model), it results a minimax objective. The predictive model strives to reduce the mismatch between true nextstates and predictions while the agent acts to maximize it.

One can model p(s ′ |h, s, a, φ) with a unit-variance Gaussian distribution in order to obtain a tractable loss. This way, we have:

E φ∼p(•|h) (s ′ ,s,a)∼π -log p(s ′ |φ, h, a, s) ≈ E φ∼p(•|h) (s ′ ,s,a)∼π ŝ′ ∼p(•|s,a,φ) -log 1 (2π) d/2 e -0.5(s ′ -ŝ ′ ) T (s ′ -ŝ ′ )
( . 8)

∝ E φ∼p(•|h) (s ′ ,s,a)∼π ŝ′ ∼p(•|s,a,φ) ||s ′ -ŝ′ || 2 2 + Const ( . )
As explained in Section . . , we assume p(s ′ |h, a, s) represents the the probability induced by the transition function of the environment and ŝ′ represents the mean prediction. It follows a line of works that generates intrinsic rewards equal to ||s ′ -ŝ′ || 2 2 where ŝ′ = arg max

s ′′ ∈S p(s ′′ |h, a, s, φ) ( . )
assuming φ paramaterizes a deterministic forward model. Therefore, the agent should go towards areas where the prediction of the state following a state-action tuple is difficult. Following the objective, we can compute an intrinsic reward as:

R(s, a, s ′ ) = ||g(s ′ ) -g(ŝ ′ )|| 2 2 ( . )
where g is a generic function (e.g. identity or a learnt one) encoding the state space into a feature space. Equation . amounts to reward the predictor error of φ in the representation g. In the following, we will see that learning a relevant function g is the main challenge.

The first natural idea to test is whether a function g is required. Burda et al. [ ] learn the forward model from the ground state space and observe it is inefficient when the state space is large. In fact, the L distance is meaningless in such high-dimensional state space. In contrast, they raise up that random features extracted from a random neural network can be very competitive with other state-of-art methods. However they poorly generalize to environment changes. An Euclidian distance.

CHAPTER . SURVEY ON INTRINSIC MOTIVATION IN DRL

other model, Dynamic Auto-Encoder (Dynamic-AE) [Stadie et al., ], computes the distance between the predicted and the real state in a state space compressed with an auto-encoder [Hinton and Salakhutdinov,6]. g is then the encoding part of the auto-encoder. However this approach only slightly improves the results over Boltzmann exploration on some standard Atari games. Other works also consider a dynamic-aware representation [Ermolov and Sebe, ].

These methods are unable to handle the local stochasticity of the environment [Burda et al., ]. For example, it turns out that adding random noise in a D environment attracts the agent; it passively watches the noise since it is unable to predict the next observation. This problem is also called the white-noise problem [Pathak et al., , Schmidhuber, ]. This problem emerges by considering only the maximization of Equation . , making us assume environments are deterministic. Therefore, exploration with prediction error breaks down when this assumption is no longer true.

To tackle exploration with local stochasticity, the intrinsic curiosity module (ICM) [Pathak et al., ] learns a state representation function g end-to-end with an inverse model (i.e. a model which predicts the action done between two states). Thus, the function g is constrained to represent things that can be controlled by the agent during next transitions. Secondly, the forward model used in ICM predicts, in the feature space computed by g, the next state given the action and the current state. The prediction error does not incorporate the white-noise that does not depend on actions, so it will not be represented in the feature state space. ICM notably allows the agent to explore its environment in the games VizDoom et Super Mario Bros. Building a similar action space, Exploration with Mutual Information (EMI) [Kim et al., a] significantly outperforms previous works on Atari but at the cost of several complex layers. EMI transfers the complexity of learning a forward model into the learning of a space and action representation through the maximization of I([s, a]; s ′ ) and I([s, s ′ ]; a). Then, the forward model φ is constrained to be a simple linear model in the representation space. Furthermore, EMI introduces a model error which offloads the linear model when a transition remains strongly non-linear (such as a screen change). However one major drawback of ICM and EMI is the incapacity of their agent to keep in their representation what depends on their long-term control. For instance, in a POMDP, an agent may perceive the consequences of its actions several steps later.

An other way to tackle local stochasticity can be to maximize the improvement of prediction error, or learning progress, of a transition model [Schmidhuber, , Azar et al., , Lopes et al., , Oudeyer et al., , Kim et al., ]. However, it turns out to be hard to estimate since the reward depends on the efficiency of the gradient update of the forward model. In its stochastic variant, rewarding the learning progress correlates with the reduction of entropy, thereby merging the information gain formalism of Section . . .

Conclusion.

While these methods perform well in deterministic environments, they struggle to offset the determinism assumption that underpines the focus on Equation .; it results that standard methods focus on the more stochastic areas. Methods that tackle stochasticity may not predict important long-term information about the environment or they need to compute a learning progress measure, which is non-trivial. In the next paragraph, we explore the information gain related to an other kind of models: a density model.

. . SURPRISE

. . Information gain over density model

Surprisal can also arise by quantifying the discrepancy between its probability of occurring and the fact that it actually occurred [Barto et al., ]. To quantify this probability of occuring, in this paragraph, we assume the agent tries to learn a density model φ ∈ Φ that approximates the true current marginal density distribution of states p(s ′ ). In this setting, we can define the expected information gain over a density model φ [Bellemare et al.,6]:

IG(h, S, A, S ′ ) ≈ E s ′ ∼π D KL (p(φ|h, s ′ )||p(φ|h)). ( . )
We hypothetize that the adversarial training that results from the objective (active maximization of the KL-divergence and density fitting) results in an approximately uniform distribution of states (and auniform density estimation). This may be due to the convexity of the KL-divergence in p(φ|h, a, s, s ′ ) and p(φ|h) but we leave the proof to future work. To our knowledge, no works directly optimize this objective, but it has been shown that this objective lower-bounds the squared inverse pseudo-count objective [Bellemare et al.,6], which derives from count-based objectives;

in the following, we will review count and pseudo-count objectives.

To efficiently explore its environment, an agent can count the number of times it visits a state and returns in rarely visited states. Such methods are said to be count-based [Strehl and Littman,[START_REF] Bengio | Curriculum learning[END_REF]. As the agent visits a state, the intrinsic reward associated with this state decreases. It can be formalized with:

R(s, a, s ′ ) = 1 N (s ′ ) ( . )
where N (s) is the number of times that the state s has been visited.

Although this method is efficient and tractable in a tabular environment (with a discrete state space), it hardly scales when states are numerous or continuous since an agent never really returns in the same state. A first solution proposed by Tang et al. [ ], called TRPO-AE-hash, is to hash the latent space of an auto-encoder fed with states. However, these results are only slightly better than those obtained with a classic exploration policy. An other line of works propose to adapt counting to high-dimensional state spaces via pseudo-counts [Bellemare et al.,6]. Essentially, pseudo-counts allow the generalization of the count from a state towards neighbourhood states using a learnt density model φ. This is defined as:

N (s ′ ) = p(s ′ |φ)(1 -p(s ′ |φ ′ ) p(s ′ |φ ′ ) -p(s ′ |φ) ( . )
where φ ′ (s) computes the density of s after having learnt on s. In fact, Bellemare et al. [ 6] show that, under some assumptions, pseudo-counts increase linearly with the true counts. In this category, DDQN-PC [Bellemare et al.,6] ].

There exists several other well-performing tractable exploration methods like RND [Burda et al.,[START_REF] Bengio | Curriculum learning[END_REF], DQN+SR [Machado et al.,[START_REF] Bengio | Curriculum learning[END_REF], RIDE [Raileanu and Rocktaschel, ] or BeBold [Zhang et al., b]. These papers argue the reward they propose more or less relate to a count estimation.

Conclusion.

Maximizing the information gain over a density model may maximize the pseudocount, which relates to count-based objectives. They provide interesting feedbacks for exploration, but in practice, pseudo-counts are hard to approximate since they rely on a powerfull density model, a strict online estimation of density and they assume p(s|φ) strictly increases ∀s ∈ S [Ostrovski et al., ]. In addition, they also struggle with the problem of randomness. For instance, let us assume that one (state, action) tuple can lead to two very different states with % chance each. The algorithm will manage to count for both states the number of visits, although it would take twice as long to avoid to be too much attracted. However, these methods do not address the white-noise problem since next states may be randomly generated at every steps. In this case, it is unclear how these methods could resist the temptation of going into this area since the counting associated to this state will never increase.

. Novelty maximization

Novelty quantifies how much a stimuli contrasts with a previous set of experiences [Barto et al.,,Berlyne,66]. More formally, Barto et al. [ ] defend that an observation is novel when a representation of it is not found in memory, or, more realistically, when it is not "close enough" to any representation found in memory. Previous experiences may be collected in a bounded memory or distilled in a learnt representation.

Several works propose to formalize novelty seeking as looking for low-density states [Becker-Ehmck et al., ], or similarly (cf. Section . . ), states that are different from others [Lehman and Stanley,,Conti et al.,[START_REF] Bengio | Curriculum learning[END_REF]. In our case, this would result in maximizing the entropy of a state distribution. This distribution can be the t-steps state distribution (cf. Equation .) H(d π t (S)) or the entropy of the stationary state-visitation distribution over a finite horizon T :

H(d π (S)) = H( 1 T T t=1 d π t (S)). ( .
)
This formalization is not perfect and does not fit several intuitions about novelty [Barto et al., ]. Barto et al. [ ] criticize such definition by stressing out that very distinct and memorable events may have low probabilites of occuring while not being novel (e.g a wedding). They suggest that novelty may rather relates to the acquisition of a representation of the incoming sensory data. Following this definition, we propose to formalize novelty seeking behaviors as those that actively maximize the mutual information between states and their representation I(S; Z) = H(S) -H(S|Z) where Z is a low-dimensional space (|Z| ≤ |S|). This objective is commonly . . NOVELTY MAXIMIZATION known as the infomax principle. [Linsker,88,Almeida,,Bell and Sejnowski,,Hjelm et al.,]; in our case, it amounts to actively learning a representation of the environment. Most of works focus on actively maximizing the entropy of state distribution while a representation learning function minimizes H(S|Z).

There are several ways to maximize the state-entropy, we separate them based on how they maximize the entropy. We found two kind of methods: low-density search and k-nearest neighbors methods.

. . Direct entropy maximization

The most evident way to maximize the entropy of states consists in maximizing H(ρ(s)) where ρ(s) approximates a density model p(s). If we access this density model, it becomes straightforward to discover a policy that maximizes the entropy of a stationary state distribution ] and q decoder a neural network that ends with a diagonal Gaussian. ρ(s) ≈ q decoder (s|z)q encoder (z|s) ( . 6)

≈ 1 N N i=1 p(z) q encoder (z|s) q decoder (s|z) ( . )
Equation . is unbiased but more expensive to compute than Equation . 6 since it requires decoding several samples. Basically, this estimation allows to reward an agent [Berseth et al., , Lee et al., , Zhang et al., ] according to: ] propose to condition the state density estimation with policy parameters in order to directly back-propagate the gradient of state-entropy into policy parameters. Although MaxRenyi achieves good scores on Montezuma's revenge with pure exploration, maximizing the ground state entropy may not be adequate since two closed ground states are not necessarily neighbors in the true environment [Aubret et al., ]. Following this observation, GEM [Guo et al., ] rather maximizes the entropy of the estimated density of states considering the dynamic-aware proximity of states, H(R). However they do not actively consider H(R|S).

R(s, a, s ′ ) = -log ρ(s ′ ).

Conclusion.

Generally speaking, these methods need an accurate density model to provide Circles represent states and red dotted lines show the distance between a state and its fourth nearest neighbor.

rewards. In the next paragraph, we study methods that avoid learning a density model.

. . K-nearest neighbors approximation of entropy

Several works propose to approximate the entropy of a distribution using samples and their k-nearest neighbors [Singh et al., , Kraskov et al., ]. In fact such objective has already been refered to as novelty [Conti et al.,[START_REF] Bengio | Curriculum learning[END_REF]. Assuming nn k (S, s i ) is a function that outputs the k-th closest state to s i in S, this approximation can be written as:

H(S) ∝ 1 |S| si∈S log ||s i -nn k (S, s i )|| 2 + χ(|S|) + Const ( .
R(s, a t , s ′ ) = log(1 + 1 K K 0 ||g(s ′ ) -nn k (g(S), g(s ′ ))|| 2 ) ( . )
where g is a representation function learnt with a contrastive loss based on data augmentation ]. The intrinsic reward for a state is then the KL-divergence between a fixed diagonal Gaussian prior and the posterior of the distribution of latent variables. In this case, common latent states fit the prior while novel latents diverge from the prior.

Intra-episode novelty. K-nearest neighbors intrinsic rewards have also been employed to improve intra-episode novelty [Stanton and Clune,[START_REF] Bengio | Curriculum learning[END_REF]. It contrasts with standard exploration since the agent looks for novel states in the current episode: typically it can try to reach all states after every resets. This setting is possible when the policy depends on all its previous interactions, which is often the case when an agent evolves in a POMDP, since the agent has to be able to predict its value function even though varies widely during episodes. This way, ECO [Savinov et al.,8b] and Never give up [Badia et al., ] uses an episodic memory and learn to reach states that have not been visited during the current episode.

Conclusion K-nn methods turn out to be simple to experiment, but they strongly rely on learnt dynamic-aware representations, their theoretical connection to the rigorous approximation of entropy remains most of the time unclear and the approach badly scales with an increase of the memory size. We note that simple methods can tackle the issue of finding the neighbors by partionning together close states [Yarats et al., ]. We observe efficient exploration and the methods easily translate to intra-episode exploration.

. . Conclusion

In this section, we reviewed works that maximize novelty to improve exploration with flat policies. We formalized novelty as actively discovering a representation according to the infomax principle despite that most of works maximize the entropy of states. But works manages to learn .

Skill learning

In our everyday life, nobody has to think about having to move his arms' muscles to grasp an object. A command to take the object is just issued. This can be done because an acquired skill can be effortlessly reused.

Skill abstraction denotes the ability of an agent to learn a representation of diverse skills. We formalize skill abstraction as maximizing the mutual information between the goal g ∈ G and some of the rest of the contextual states f (τ ) ∈ f (T ), denoted as I(G; f (T )) where τ ∈ T is a trajectory and f a function that extracts a subpart of the trajectory (last state for example). The definition of f depends on the wanted semantic meaning of a skill. Let s 0 refers to the state at which the skill started and s a random state from the trajectory, we highlight two settings based on the literature:

• f (T ) = S, the agent learns skills that target a particular state of the environment [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF].

• f (T ) = T , the agent learns skills that follow a particular trajectory. This way, two different skills can end in the same state if they cross different areas [START_REF] Bengio | Curriculum learning[END_REF].

Most of works maximize I(G; S) so that, unless stated otherwise, we refer to this objective.

In the following, we will study the different ways to maximize I(G; S) which can be written under its reversed form

I(S; G) = H(G) -H(G|S) or forward form I(G; S) = H(S) -H(S|G) [Campos et al., ].
In particular, we emphasize that:

-H(G|S) = g∈G,s∈S p(g, s) log p(g|s) ( . ) = E g∼p(g) s∼π g log p(g|s) ( . )
where, to simplify, p(g) is the current distribution of goals (approximated with a buffer) and s ∼ π g denotes the distribution of states that results from the policy that achieves g. Note that p(g, s) = p(s|g)p(g).

In this section, we first focus on methods that assume they can learn all skills induced by a given goal space/goal distribution and they assign parts of trajectories to every goal. The second set of methods directly derives the goal space from visited states, so that there are two different challenges that we treat separately: the agent has to learn to reach a selected goal and it must maximize the diversity of goals it learns to reach. We make this choice of decomposition because some contributions focus on only one part of the objective function.

. . Fixing the goal distribution

The first approach assumes the goal space is arbitrarily provided except for the semantic meaning of a goal. In this setting, the agent samples goals uniformly from G, ensuring that H(G) is maximal, and it progressively assigns all possible goals to a part of the state space. To do this assignment, the agent maximizes the reward provided by Equation .:

R(g, s, a, s ′ ) = -log q ω (g|s ′ ) ( . )
where q ω (g|s ′ ) represents a learnt discriminator (often a neural network) that approximates p(g|s ′ ).

At first, we focus on discrete number of skills, where p(g) represents a uniform categorical distribution. Figure . sums up the learning process with two discrete skills: -skills and discriminator q are randomly initialized; -the discriminator tries to differentiate the skills with states s from its trajectories, in order to approximate p(g|s); -skills are rewarded with Equation . in order to make them go in the area assigned to it by the discriminator; - ] make it hard to use a uniform (for instance) H(G|S 0 ), because every skill may not be executable everywhere in the state space. Therefore, they also maximize the entropy term with another reward bonus similar to log p(g|s 0 ). They learn discriminable skills, but still struggle to combine them on complex benchmarks [Baumli et al., ]. Keeping p(g) uniform, DADS [Sharma et al., ] maximizes the forward form of mutual information I(S; G|S 0 ) = H(S|S 0 ) -H(S|G, S 0 ) by approximating p(s|s 0 ) and p(s|s 0 , g). This method makes possible to plan over skills and can combine several locomotion skills. However this requires several conditional probability density estimation on the ground state space, which may badly scale on higher-dimensional environments.

These methods tend to stay close from their starting point [Campos et al., ] and do not learn skills that cover the whole state space. In fact, it is easier for the discriminator to overfit over a small area than to make a policy go in a novel area, this results with a lot of policies that target [START_REF] Bengio | Curriculum learning[END_REF]. More consistent dynamic-aware methods may further improve regularization; however it remains hard to scale the methods to a large number of skills which are necessary to scale to a large environment. In above-mentioned methods, the number of skills greatly increases [Achiam et al.,[START_REF] Bengio | Curriculum learning[END_REF]Aubret et al.,] and the discrete skill embedding does not provide information about proximity of skills. Therefore learning a continuous embedding may be more efficient.

a

Continuous embedding.

The prior uniform distribution p(g) is far more difficult to set in a continuous embedding. One can introduce the continuous DIAYN with a prior p(G) = N (0 d , I) where d is the number of dimensions, or the continuous DADS with a uniform distribution over

[-1; 1] [Sharma et al.,
], yet it remains unclear how the skills could adapt to complex environments, where the prior does not globally fit the inherent structure of the environment. VISR [Hansen et al., ] seems to, at least partially, overcome this issue with a long unsupervised training phase and successor features. They uniformly sample goals on the unit-sphere and computes the reward as a dot product between unit-normed goal vectors and successor features log q ω (g|s) = φ successor (s) T g.

Conclusion.

This set of methods manages to learn discrete skills that can be combined, yet, despite regularization, discrete skills struggle to cover a very large state space [Aubret et al., ] (cf. Chapter ). Successfull adaptations to scale it up to large states spaces currently rely on the relevance of successor features. In the next two sections, we study how to maximize the mutual information by assuming the goal space derives from the state space.

. . Achieving a state-goal

In this section, we review how current methods maximize the goal achievement part of the objective of the agent, -H(S g |S) where S g refers to the goal-relative embedding of states. We temporally set aside H(S g ) and we will come back to this in the next subsection, Section . . , mainly because the two issues are tackled separately in the literature.

Obviously, maximizing -H(S g |S) can be written:

-H(S g |S) = Sg p(s g , s) log p(s g |s) ( . ) = E sg∼p(s) s∼π g log p(s g |s) ( . )
where, to simplify, p(s) is the current distribution of states (approximated with a buffer) and s ∼ π g denotes the distribution of states that results from the policy that achieves g. If log p(s g |s ′ ) is modelled as an unparameterized Gaussian with a unit-diagonal co-variance matrix, we have

log p(s g |s ′ ) ∝ -||s g -s ′ || 2 2 +
Const so that we can reward an agent according to:

R(s g , s, a, s ′ ) = -||s g -s ′ || 2 2 . ( . )
Trivially, it means that if the goal is a state, the agent must minimize the distance between its state and the goal state. To achieve this, it can take advantage of a goal-conditioned policy π sg (s).

Ground state space. This way, Hierarchical Actor-Critic (HAC) [Levy et al., ] directly uses the state space as a goal space to learn three levels of option (the options from the second level are selected to fulfill the chosen option from the third level). A reward is given when the distance between states and goals (the same distance as in Equation .) is below a threshold and they take advantage of HER to avoid to directly use the threshold.

f (T ) = S f -S 0 .
This approach is relatively simple and does not require extra neural networks. However, there are two problems in using the state space in the reward function. Firstly, a distance (like L ) makes little sense in a very large space like images composed of pixels. Secondly, it is difficult to make a manager policy learn on a too large action space. Typically, an algorithm having as goals images can imply an action space of 84 × 84 × 3 dimensions for a goal-selection policy (in the case of an image with standard shape). Such a wide space is currently intractable, so these algorithms can only work on low-dimensional state spaces.

Learning a representation of goals. To tackle this issue, an agent can learn low-dimensional embedding of space φ e and maximize the reward of Equation . 6 using a goal-conditioned policy π φe(sg) (s): ] and growing when required [Marsland et al., b] to build a topological map. A hierarchical agent then uses nodes of the map, representing positions in the world, as a goal space. However the authors do not compare their contribution to previous approaches.

R(s g , s, a, s ′ ) = -||φ e (s g ) -φ e (s ′ )|| 2 2 . ( .
Other approaches learn a state embedding that captures the proximity of states with contrastives losses. For instance, DISCERN learns the representation function by maximizing the mutual information between the last state representation and the state-goal representation. Similarly to works in Section . . , the fluctuations around the objective allow to bring states around s g closer to it in the representation. More explicitly, the representation of NOR [Nachum et al., a] maximizes I(φ e (S t+k ); φ e (S t ), A t:t+k ) and the one of LESSON [Li et al., b] and DisTop (cf. Chapter ) approximately maximizes I(φ e (S t+1 ); φ e (S t )); LESSON and NOR target a change in the representation and manage to navigate in a high-dimensional maze while learning the intrinsic euclidian structure of the mazes. Their skills can be reused on several environments. However, experiments are made in -dimensional embedding spaces and it remains unclear how relevant may be goals as state changes in an embedding space with higher dimensions. The more the number of dimensions increase, the more difficult it will be to distinguish possible skills from impossible skills in a state. DisTop targets goal-state, it has more difficulties to navigate in large environments, but also work in non-maze environments. We discuss again this issue in the next section.

Conclusion.

To sum up, representation learning methods allows to learn state-based skills over complex state spaces. Learning this representation function combined with the use of the euclidian distance as reward function amounts to learn a particular form of reward function in addition for providing pre-computed features to the goal-conditioned policy. In the next paragraph, we study how to maximize H(S) so that to make sure learnt skills are diverse.

. . Proposing diverse state-goals

To make sure the agent maximizes the mutual information between its goals and all visited states, it must sample a diverse set of goal-states. In other words, it has to maximize H(S g ) but through goal selection rather than with an intrinsic bonus as in Section . . Similarly to works on novelty (cf. Section . ), such entropy maximization along with skill acquisition (cf. Section . . ) tackles the exploration challenge, but without facing catastrophic forgetting (cf. Section .6. ) since the agent does not forget its skills.

A naive approach would be to generate random values in the goal space, but this faces a considerable problem: the set of achievable goals is often a very small subset of the entire goal space. To tackle this, a first approach can be to explicitly learn to differentiate these two sets of goals [Florensa et al.,[START_REF] Bengio | Curriculum learning[END_REF]Racaniere et ].

[ Pong et al., ]. Other works obtain good results on imagining new goals, but using either a particularly structured goal space [Colas et al., b] or dataset [Khazatsky et al., ]. In contrast, an agent can simply set a previously met state as a goal, this way, it ensures that goals are reachable, since they have already been achieved. In the rest of this section, we focus on this set of methods.

Figure . illustrates different strategies for sampling goals using previously met states. In RIG [Nair et al.,[START_REF] Bengio | Curriculum learning[END_REF], the agent randomly samples states as goals from its buffer, but it does not increase the diversity of states, and thus, the diversity of learnt skills. Pong et al. [ ] showed theoretically and empirically that, by sampling goals following a α-more uniform distribution over the support of visited states than the "achieved" distribution, the distribution of states of the agent can converge to the uniform distribution. Intuitively, the agent just samples more often low-density goals: this setting typically applies in the two right-most distributions of Figure . and we illustrate it in Figure .6. There are several ways to increase the importance of low-density goal-states that we introduce in the following.

Density estimation in the ground state space. DISCERN [Warde-Farley et al., ] proposes to sample uniformly over the support of visited stated with a simple procedure. Every time the agent wants to add an observation to its buffer, it randomly samples an other observation from its buffer and only keeps the one that is the farthest to all other states of the buffer. This way, it progressively builds an uniform distribution of states inside its buffer. However, it uses the euclidian distance to compare images, which may not be relevant. Other approaches select the state that has the lower density (OMEGA) [Pitis et al., ] ] to estimate the learning progress of each partition and avoid distractors.

The density weighting can also operate in a learnt embedding. HESS [Li et al., a] partitions the embedding space of LESSON and rewards with a variant of a count-based bonus (see Section . ). It improves exploration in a two-dimensional latent embedding but the size of partitions may not scale well if the agent considers more latent dimensions. In contrast, DisTop [Aubret et al., ] (cf. Chapter ) dynamically clusters a dynamic-aware embedding space using a variant of a Growing When Required [Marsland et al., b]; they estimate the density of state according to how much its partition contains states and skew the distribution of sampled similarly to Skewfit. HESS and DisTop demonstrate their ability to explore and navigate with an ant inside complex mazes without extrinsic rewards.

Conclusion.

Entropy maximization methods improves over standard skill learning methods by learning to reach as many states as possible. We expect further works to show the ability to scale to even more complex environments, with higher-dimensional latent structure [Li et al., a].

.

Outlooks of the domain

In this section, we take a step back and thoroughly analyze the results of our overall review. We first study the exploration process of flat intrinsic motivation in comparison with hierarchical intrinsic motivations in Section .6. ; then, this will motivate our focus on the challenges induced by learning a deep hierarchy of skills in Section .6. . Finally, in Section .6. , we discuss how flat and hierarchical intrinsic motivation can and should cohabit in such hierarchy. .

Long-term exploration, detachment and derailment

The most challenging used benchmarks in flat intrinsic motivations (surprise and novelty) are DMLab and Montezuma's revenge, yet very sparse reward games such as Pitfall! are not currently addressed and should be investigated. In Pitfall!, the first reward is reached only after multiple rooms where it requires specific action sequences to go through each room. State of the art on IM methods [Ostrovski et al., ] achieve mean reward in this game. At the opposite, imitation RL methods [Aytar et al.,[START_REF] Bengio | Curriculum learning[END_REF]Hester et al.,[START_REF] Bengio | Curriculum learning[END_REF] are insensitive to such a specific reward, and thus, exceed IM methods with a mean reward of on Montezuma's revenge and on Pitfall!. Even though these methods use expert knowledge, this performance gap exhibits their resilience to long-term rewards. Compared with flat intrinsic reward methods, which do not exceed a score on Montezuma's revenge [Burda et al.,[START_REF] Bengio | Curriculum learning[END_REF] and hardly achieve a score on Pitfall! [Ostrovski et al., ], it shows that flat IMs is still far from solving the overall problem of exploration.

Furthermore, we want to emphasize that the challenge is harder when the intrinsic reward itself is sparse [Burda et al.,[START_REF] Bengio | Curriculum learning[END_REF]. In Montezuma's revenge, it is about avoiding to use a key too quickly in order to be able to use it later. In every day life, it can be about avoiding to spend money too quickly. In fact, it looks like there is an exploration issue in the intrinsic reward function. Intrinsic reward can guide the exploration at the condition that the agent finds this intrinsic reward. There may be two reasons causing the intrinsic reward to be sparse:

. The first comes from partial observability, with which most models are incompatible. Typically, if an agent has to push a button and can only see the effect of this pushing after a long sequence of actions, density models and predictive models may not provide meaningfull intrinsic rewards. There would be a too large distance between the event "push a button" and the intrinsic reward.

. Most works presented in Section . abstract actions on a restricted number of hierarchies (generally one hierarchy). This is necessary to well-understand the mechanism of abstraction, but we want to argue that imposing deeper hierarchies could considerably enhance the semantic comprehension of the environment of an agent. Organisms are often assumed to deal with composition of behaviors, which in turn serve as building block for more complex behaviors [Flash and

Hochner, ]. Using a limited vocabulary of skills makes easier avoiding the curse of dimensionality associated to the redundancy of a whole set of ground behaviors. Skill focus. In a developmental process, multi-level hierarchical RL questions the ability of the agent to learn all policies of the hierarchy simultenaously. This obviously relates to the ability of organisms to continually learn throughout their lifetime; but in more practical way, it may allow to focus the learning process of skills that are interesting for higher-level skills. This focus avoids

CHAPTER . SURVEY ON INTRINSIC MOTIVATION IN DRL

learning everything in the environment [Aubret et al., ], which is hard and obviously not done by biological organisms.

Critical periods and lifelong learning. Considering a goal representation that changes over time introduces new issues for the agent. In this case, the goal-conditioned policy may be perturbed by the changes of inputs and may no longer be able to reach the goal [Li et al., a]. Current methods consider -developmental periods (unsupervised pre-training [Metzen and Kirchner, ]); -to modify the representation every k-steps epochs [Pong et al., ]; -to impose slowly changes of the representation [Aubret et al., , Li et al., a]. Further works may thoroughly investigate the relation and transitions between these methods since they can relate to the concept of critical periods [Hensch, , Konczak, ]. Critical periods assume that the brain is more plastic at some periods of development in order to acquire specific knowledge. Despite this mechanism, the brain slowly keeps learning throughout the lifetime. In the hierarchy of skills, the introduction of a new level may first result in a quick/plastic learning process, followed by slower changes.

.6.

The role of flat intrinsic motivations

In Section .6. , we essentially criticized the limited role that flat intrinsic motivation like surprise or novelty can play in favor of exploration and we hypothetized in Section .6. that deeper hierarchies could make emerge an understanding of more complex affordances. Then, what could be the roles of surprise and novelty ?

Novelty. We saw in Section . that novelty seeking behaviors allow to learn a correct representation of the whole environment; this can be a basis for learning diverse skills. While some methods consider a goal as a state and manage to avoid using novelty bonuses [Pong et al., ],

this is harder to do when skills have a different semantic (like a change in the state space). Nachum et al.

[ a] provide a meaningful example of this: the agent acts to simultaneously discover a representation of the environment and achieve upper-level goals.

Surprise. We leave aside the interest of surprise for learning a forward model that could be used for planning [Hafner et al., ] and rather focus on the learning process. Surprise amounts to look for the learning progress of forward models so that, in a hierarchy of skills, it quantifies whether skills can currently be better learnt or not. This links surprise to curriculum learning [Bengio et al., ], i.e can we find a natural order to efficiently learn skills ? For example, assuming an agent want to learn to reach state-goal in a maze, it would be smarter to learn to start learning skills that target goals close to its starting position and to progressively extend its goal selection while learning other skills. To sum up, we propose that the role of surprise and novelty may rather be to support the learning of skills. Novelty seeking helps to learn the representation required by the skill learning module and surprise speeds up the maximization of the skill learning objective. Considering this, it would result several surprises and novelties: an agent can experiment a novel or surprise interaction for a level of decision (injure the toy while walking), yet it does not mean other levels would be surprised . . UNIFYING INTRINSIC MOTIVATIONS (it is still on the same road). This emphasizes the multi-dimensionality and relativity of the notion of surprisingness ou novelty [Berlyne,6 ], only a part of the incoming stimuli may arouse the agent.

.

Unifying intrinsic motivations

In this section, we more thoroughly investigate the links between information theory and intrinsic motivation. We saw in Section . . that an intrinsic motivation is defined by how it is computed, i.e the comparison on previous data. While this definition encompasses the search for desirable stimulus properties such as surprisingness, novelty or a large number of distinctly described motivations [Berlyne,6 ], an intrinsically motivated entity could also search their exact opposite, such as boredom. Therefore, the comparison of data appears more as a property of an IM rather than as its definition. It follows that IM appears to the DRL community like a fuzzy concept that admits a lot of different approaches. Despite this, intrinsic motivation could be a core component in the development of organisms [Guerin, ]. As such, a precise formalism may converge to a computational principle guiding the development of agents.

In order to quantify the information processed by an agent, previous works propose to model the sensori-motor loop of an agent interacting with its environment using a Bayesian network [Pearl, ]; This modeling allows to quantitatively analyze the perception-action loop [Touchette and Lloyd, , Klyubin et al., a]. This framework opens the path for a quantitatization of intrinsic motivations.

We noticed in Section .6 that maximizing the correlations of some variables can/could lead to complex behaviors that reflect the emergence of increasingly more complex cognitive structures as a result of interactions with the environment [Guerin, , Zlatev and Balkenius, ]. Having argues in Section .6. for the compatibility of intrinsic motivations, we now propose to go considerably further and to join current forms of intrinsic motivations with the information-theoretic framework of the perception-action loop in order to exhibit the information-theoretic general principle that underpins current IMs. Precisely, we formally show that maximizing the multi-information of a simple hierarchical cognitive model amounts, to some extent, to simultaneously maximizing novelty, surprise and skill learning objectives Our methodology is two-folds:

• Proposing a plausible cognitive architecture, modelled as a Bayesian Network. It essentially requires variables with causal dependencies.

• Deriving local maximization terms from the multi-information of the model.

Proposing a complete cognitive architecture explaining the full complexity of human behaviors is out of the scope of our study. This rather aims to demonstrate the validity of our objective recipe.

In the following, we first introduce Bayesian networks in Section . . , then we explictly give the assumptions (Section . . ) that we use in Section . . to derive the objective that unifies skill learning, novelty and surprise. We end the section with a discussion about this objective.

CHAPTER . SURVEY ON INTRINSIC MOTIVATION IN DRL

. . Bayesian networks

Bayesian networks (or graphical models) are directed acyclic graphs for which -vertices correspond to random variables which represent part of the state of the system; -edges reflect statistical dependencies between these variables, i.e a causal relationship. (V 0 , . . . , V N ) are the N + 1 vertices of a graph, and P a(V ) sums up the parents of V with respect to the edges, we can write [Pearl, ]: ], we assume that the standard graphical model is simpler and makes our results more understandable.

P (V 0 , . . . , V N ) = N i=0 P (V i |P a(V i )). (
When variables can be described with a graphical model, the multi-information of the whole set of variables can be rewritten as the sum of mutual information between a node and its parents [Friedman et al., , Slonim et al., ]:

. In practice, all variables are dependent on the content of the memory and the decisions/actions depend on DRL parameters but we omit these here for clarity since it does not bring out more informations for our analysis.

. UNIFYING INTRINSIC MOTIVATIONS

M I(V 0 , . . . , V N ) = N i=0 I(V i ; P a(V i )). ( . )
. .

Assumptions about agent's cognitive model

A plausible candidate for a simplified cognitive architecture can be inspired from hierarchical reinforcement learning for two reasons. First there are bio-inspired correlations between hierarchical behaviors in humans [Botvinick, 8, Mussa-Ivaldi and Bizzi, ] and the option framework [Botvinick et al., ]. Indeed, human behaviors are building blocks that can be hierarchically or serially combined to perform more complex building blocks, which can be combined again [Mussa-Ivaldi and Solla, , Mussa-Ivaldi and Bizzi, ]. Secondly it is widely used in computational models since it can bring out complex behaviors, as argued in Section .6. . We also introduce the concept of representations that have been discovered in the brain [Quiroga et al., ], e.g some neurons respond to particular objects.

We In our case, the probability distributions are defined over a memory of the agent, which is filled with new interactions. We made some common-sense assumptions: first, we assume the agent learns a forward model φ that integrates the perception-action loop. It does not access the true state and the agent is agnostic to it but rather learns a representation of the environment that allows to produce high-level decisions; the information contained in the representation grows the higher up the hierarchy it is. The decisions impact the time-extended representation, as well as the ground actions of the agent.

One can notice that we can easily add more hierarchies to the model; we can recursively reproduce the interactions between low-level and high-level decisions so that high-level modules become low-level modules for the next level of the hierarchy.

In practice, the true causal model that usually results from the interactions of the agent may be essentially similar, except for the links between z t and z t+3 and the absence of the hidden causal variables of the environment.

Maximizing the multi-information of such architecture is particularly hard since each combination of time-observation is a random variable. For instance, the probability p(O 0 = o) may be different from p(O 25 = o). In addition, in a lifelong learning scenario without episodes, the timesteps t may infinitely increase, leaving one value for each variable at each timestep. To overcome this issue, we take inspiration from off-policy RL [Mnih et al., ] and the brain [Wilson and McNaughton, ] and assume the agent considers a restricted set of time-independent variables while replaying its interactions. This facilitates the memorization of interactions so that the agent only needs to keep local ordering (a variable and its direct parents) rather than the whole time-index. Thus, we write p(

V t ) = p(V |t) = P (V ) and p(V t |P a(V t )) = p(V |P a(V ))
where V can represent any random variables and P a represent the parents. In other words, we assume that the joint distribution of tuples of locally dependent variables is time-independent. In this case we have, with t = 0, . . . , N :

t I(X t ; X t-1 , Y t ) = t I(X ′ ; X, Y |t) = t H(X ′ |t) -H(X ′ |X, Y, t) = t - X ′ p(x ′ , t) log p(x ′ |t) + X ′ ,X,Y p(x ′ , x, y, t) log p(x ′ |x, y, t) = - X ′ t p(x ′ |t)p(t) log p(x ′ |t) + X ′ X,Y t p(x ′ , x, y|t)p(t) log p(x ′ |x, y, t) (1)
= -

X ′ t p(x ′ )p(t) log p(x ′ ) + X ′ X,Y t p(x ′ , x, y)p(t) log p(x ′ |x, y) (2)
= -

X ′ t p(x ′ ) 1 N log p(x ′ ) + X ′ X,Y t p(x ′ , x, y) 1 N log p(x ′ |x, y) = t 1 N - X ′ p(x ′ ) log p(x ′ ) + X ′ X,Y p(x ′ , x, y) log p(x ′ |x, y) = I(X ′ ; X, Y ) ( . )
where we applied p(V |t) = p(V ) in ( ) and noticed that p(t) = 1 N in ( ). This essentially means that, by storing interactions in memory, the agent considerably simplifies the objective to timeindependent variables. In the next section, we use this results to derive the multi-information of the model.

. . Derivation of core objectives

Now we will use Equation . to derive the locally structured multi-information of the model

B: M I B (1) = t I(O t+1 ; O t , A t , φ) + I(A t ; G 1 t , O t ) + I(Z t ; φ 1 , G 1 t-3 , Z t-3 , O t ) + I(G 1 t ; G 2 t , Z t ) (2) = I(O ′ ; O, A, φ) + I(A; G 1 , O) + I(Z ′ ; φ 1 , G 1 , Z, O ′ ) + I(G 1 ; G 2 , Z) ( . ) (3) = I(O ′ ; φ|A, O) Low-level surprise + I(O ′ ; A, O) Controllability + I(A; G 1 , O) Low-level policy + I(Z ′ ; Z, O ′ ) Novelty + I(Z ′ ; φ 1 |G 1 , Z, O ′ )
High-level surprise

+ I(Z ′ ; G 1 |Z, O ′ ) Skill learning + I(G 1 ; G 2 , Z)
High-level policy ( . )

Since our model is a directed acyclic graph, in ( ) we can rewrite the multi-information as the sum of mutual informations between the nodes and their parents using Equation .. In ( ), we simply apply Equation . to each term. Finally, in ( ), we unroll the first and third terms with the chain rule of mutual information. Let us now analyze each term of Equation . :

Surprise: two terms relate to surprise, one per level of the hierarchy. It directly refers to our formalism of surprise in Section . where the agent acts to reduce its uncertainty over its forward or density model. This allows to explore the environment through different levels of decisions.

Controllability: controllability has been discussed in Section . and relates to the ability of the agent to control through actions the observations it gathers [Touchette and Lloyd, ]. This way, the agent can act to avoid stochastic areas. This may also be maximized through evolution of sensors and actuators [Klyubin et al., ]. In upper-levels of the hierarchy, this term can be retrieve through the skill learning term, however, in this setting, the policy underpinning a skill defines the semantic meaning of a skill, so that no skills lead to intrinsically stochastic areas [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF].

Policies: these terms essentially means the agent must learn policies dependent on the current representations and high-level decisions while each action should have low marginal probabilities. This is most of the time respected since current exploration policies often progressively converge to an almost deterministic policy when they become able to solve a task. This is the case, for instance of decaying ǫ-greedy, where ǫ progressively converges to almost 0, or Boltzmann exploration. Along with exploration through surprise and novelty, it allows for directed exploration.

Novelty: we discussed about novelty seeking behaviors in Section . , it essentially allows to learn a representation of the environment. This time, we propose to maximize I(Z ′ ; Z, O ′ ) rather than I(Z ′ ; O ′ ). We argued in Section . that novelty aims at dynamically discovering a representation of the environment, but what kind of representations ? Equation . suggests that this representation should capture both low-level features and temporally-extended features. This is in line with our analysis in Section .6. which emphasizes the role of the slow moving principle for learning representations. One can see this as transforming the temporal information into abstract representations, without the need for explicit memory-based networks [Greff et al.,6]. Conceptually, this shares ideas with clockwork recurrent neural networks [Koutnik et al., ] which has been applied on classification of sequences of data. Since Z ′ must keep informations about all O ′ and add information relatively to previous Z, the representation size should grow along with the hierarchy in order to capture more information about observations. If the agent transmits ground representations towards high levels of the hierarchy (maximizing the objective), this can be deducted with Z ′ = (Z ′′ , O ′ ) and:

I(Z ′ ; Z, O ′ ) = I(Z ′′ , O ′ ; Z, O ′ ) = I(Z ′′ ; Z, O ′ |O ′ ) + I(O ′ ; Z, O ′ ) = I(Z ′′ ; Z|O ′ ) + H(O ′ ). ( . )
Considering a large hierarchy of decisions and representations, we generalize this to an arbitrary level:

I(Z ′k ; Z k , Z ′k-1 ) = I(Z ′′k ; Z k |Z ′k-1 ) + H(Z ′k-1 ).
( . )

In practice, we just need to make possible H(O ′k-1 ) < H(Z ′k ) which may be a consequence of |Z ′k | > |O ′k-1 |. Now let us further investigate why different temporal resolution should encode different representations. We can write:

I(Z ′k ; Z k , O ′k-1 ) = H(Z k ) + H(O ′k-1 ) -I(O ′k-1 ; Z k ) -H(Z k , O ′k-1 |Z ′k ). ( . 6)
Equation . 6 tells us that, under size constraints, the agent has to minimize I(O ′k-1 ; Z k ), thereby encoding different informations about the observations with Z k and O ′k-1 . The causal model directly implies that O ′k-1 should focus over short-term information and Z k over longerterm information since Z k-1 does not access local information.

Skill learning: Finally, we observe the last component we are looking for when maximized through the low-level policy: a skill learning objective that we described in Section . . This suggests an agent should not look for ground states or trajectories [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF], but rather a change in the input space [Gregor et al.,6]. In our case, this appears that the skill learning should focus on changes in the temporally-extended part of the representation.

As a result, the multi-information of a hierarchical model can indeed explain surprise, novelty and skill learning. As emphasized by Equation . , the roles of I(Z ′ ; φ 1 , G 1 , Z, O ′ ) and I(O ′ ; O, A, φ) are essential for such derivation; the two others only incite the policies to act according to their .8. CONCLUSION goal and actual observations/representations.

. . Discussion

We hope future works will investigate whether multi-information would provide explanations for other phenomenas, like visual tracking [Eckmann et al., ], vergence control [Zhang et al., ],

disentanglement [Bengio et al., ], language [Vygotsky,[START_REF] Bengio | Curriculum learning[END_REF] . . . . Multi-information principle arises in a bottom-up way by studying works on intrinsic motivation but the building of a Bayesian cognitive model should be intertwinned with biological evidences about the structure of the cortex, in particular the interactions between cortical areas [Mumford, , Felleman and Van Essen, ]. In fact, we expect that an investigation about correlated neurobiology aspects to provide more evidences of the principle.

Widening the explanation strength of multi-information could make it a plausible candidate to be a guiding principle for action and perception, thereby competing with the free-energy principle [Friston, ]. Yet, we highlight that the multi-information shares some properties with the freeenergy principle which itself has been justified with neurobiological studies [Friston, ]. This includes minimizing the conditional entropy of representations, or errors of prediction, and the building of a causal model. The most proeminent difference lies in the fact that multi-information maximizes the total amount of information, or marginal entropy of states, representations and decisions. This is a critical add-on since it allows to tackle the dark room issue of the free-energy principle: in their case, an agent staying motionless in the darkness minimizes its prediction error and thus, acts optimally. In the case of multi-information, the dark-room scenario does not maximize the marginal entropy of observations/representations, and thus, does not maximize the objective.

However, we think that the most important advantage of multi-information in comparison with the free-energy comes from our methodology: by extrapolating multi-information principle from machine learning objectives, it deeply connects with the truth on the ground, making it useful for deriving new objectives. In contrast, to the best of our knowledge, the free-energy principle composes with only few applications in complex environments [Berseth et al., ].

.

Conclusion

In this survey, we have presented the current challenges faced by DRL: namely -learning with sparse rewards through exploration;building a hierarchy of skills in order to make easier credit assignment, exploration with sparse rewards and transfer learning.

We identified several types of IM to tackle these issues, that we classified into three categories based on the maximized information theoretic objective, which are surprise, novelty and skill learning. Surprise and novelty based intrinsic motivations implicitly improve exploration while skill learning allows to create a hierarchy of skills.

Looking for surprise maximizes the mutual information between a model parameters and the next state, knowing the previous state, the action and the history of interactions. We have shown that it can be maximized through three set of works: information gain over predictive models, density models or prediction errors. Novelty seeking can be assimilated to learning a representation of the environment, through the maximization of mutual information between states and their representation. The most important term is the state-entropy maximization. Finally, using skill learning objective that amount to maximize the mutual information between a goal and a part of trajectories of the corresponding skill, an agent can learn hierarchies of temporally-extended skills.

The three objectives are compatible and we have discussed how they could interact to provide a more robust exploration process along with reusable and hierarchical skills, a quick and focused skill acquisition and multi-semantic representations. In addition, we showed that these objectives can by summarized by considering the multi-information of a simple hierarchical cognitive model. We expect that further works will show the value of multi-information as a way to induce a large set of complex behaviors and cognitive abilities.

The core part of the developmental architecture presented in Section .6. is mostly based on the integration of bottom-up hierarchical skill discovery in HRL. We recall that we identified three advantages of discovering skills in a bottom-up way:

. The time extended commitment resulting from a hierarchical random walk avoids the usual wanderlust due to the sparsity of rewards.

. Objective functions for skill abstraction encourages skill diversity, which favors exploration of the state space.

. Skills can be hierarchically executed and transfered across several different tasks.

In particular, such exploration pathways may be robust against detachment, which is a hard issue for flat exploration methods. All together, these elements highlight the potential role of bottom-up skill discovery to tackle RL deadlocks. In the rest of the dissertation, we propose to investigate how bottom-up hierarchical skill discovery could concretely be at the origin of this developmental architecture. Thus we reformulate and refine our problem statement: Can an agent continually learn increasingly complex hierarchical skills using DRL and intrinsic motivation ?

As a first step, we put aside the hierarchical combination of skills and focus on their discovery. In fact, we hypothetize that, even though an agent does not take advantage of a straightforward credit assignment or hierarchical random walk, exploration and skills transfer can still benefit from bottom-up skill discovery. In the next chapter, we introduce a novel model that validate this hypothesis.

.8. CONCLUSION

Chapter 4

End-to-end learning of reusable skills through intrinsic motivation .

Introduction

In the previous chapter, we thoroughly studied what is missing to DRL agents to tackle some of their more important issues (exploration, transfer learning and credit assignment problem). We proposed a developmental architecture and noticed that it could be based on a bottom-up skill discovery. In this chapter, we thoroughly investigate how to discover such skills, we identify the shortcomings of the related approaches and propose a novel model that tackle them.

Several works [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]Achiam et al.,[START_REF] Bengio | Curriculum learning[END_REF]Pong et al.,] recently proposed to intrinsically learn skills (see definition in Section . . ) by maximizing the mutual information between states s ∈ S and goals g ∈ G, I(G, S), such that different states are covered by the learned skills. While they discover a great diversity of skills, these works neither learn skills sequentially, nor execute skills sequentially [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF], but rather always sample goals uniformly.

Learning skills uniformly brings up several issues:

. Most skills target uninteresting parts of the environment relatively to some tasks; thereby it requires prior knowledge about which features to diversify [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]Sharma et al., ]. This is illustrated in the first step of Figure . , the agent learns a skill that goes to the top part of the environment, even though there are no extrinsic rewards.

. Time-extended skills used in a hierarchical setting are often sub-optimal for a task. With diversity heuristic, skills are indeed not expressive enough to efficiently target a goal [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]Achiam et al.,[START_REF] Bengio | Curriculum learning[END_REF]. This is showed in the second step of Figure . , even though the agent knows which of its skill performs the best, it may often reach sub-optimal areas.

. The agent suffers from catastrophic forgetting when it tries to learn a task while learning skills [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]. This issue appears in steps two and three of Let us suppose there is a hierarchical module that provides goals to the skill learning algorithm.

Again, the algorithm will not focus its learning process on upper-levels assignments and will discover skills in uninterested areas of the state space. Thus, the issue is two-folds: -it prevents an agent to learn a task; -it prevents skill discovery methods to be efficiently integrated in a hierarchical way. In fact, uniform skill discovery occurs during a developmental period [Metzen and Kirchner, ] which is just an unsupervised pretraining. However, a truly open-ended learning agent does not consider pre-training phases and always expands its repertoire of skill, whether high-level goals or tasks are provided to it.

To implement such open-ended learning agent, hierarchical skills should be discovered in a continual learning framework (cf. Section . . ).

In the rest of the chapter, we propose to improve the approaches for continually learning increasingly difficult skills with diversity heuristics. We introduce ELSIM (End-to-ended Learning of reusable Skills through Intrinsic Motivation), a method for learning representations of skills in a bottom-up way. The agent autonomously builds a tree of abstract skills where each skill is a refinement of its parent. First of all, skills are learned independently from the tasks but along with tasks; it guarantees they can be easily transferred to other tasks and may help the agent to explore its environment. Secondly, the agent selects a skill to refine with extrinsic or intrinsic rewards, and learns new sub-skills; it ensures that the agent learns specific skills useful for tasks through an intelligent curriculum, among millions of possible skills.

We believe our paradigm, by removing the requirement of an unsupervised pretraining, makes compatible skill discovery methods with an open-ended architecture. Therefore, we emphasize three properties of our ELSIM method :

Learning is bottom-up : the agent does not require an expert supervision to expand the set of skills. It can use its skills to solve different sequentially presented tasks or to explore its environment.

Learning is end-to-end : the agent never stops training and keeps expanding its tree of skills. It gradually self-improves and avoids catastrophic forgetting.

Learning is focused : the agent only learns skills useful for its high-level extrinsic/intrinsic objectives when provided.

Our contributions are the following: we introduce a new curriculum algorithm based on an adaptation of diversity-based skill learning methods. Our objective is not to be competitive when the agent learns one specific goal, but to learn useful and reusable skills along with sequentially presented goals in an end-to-end fashion. We show experimentally that ELSIM achieves good asymptotic performance on several single-task benchmarks, improves exploration over standard DRL algorithms and manages to easily reuse its skills. Thus, this is a step towards lifelong learning agents.

This chapter is organized as follows. First, we provide more details on diversity-based intrinsic motivation (Section . . ) and exhibit how ELSIM relates to existing works that learn skills in a lifelong learning scenario. In Section . , the core of our method is presented. Then, we explain and visualize how ELSIM works on simple gridworlds and compare its performances with stateof-the-art DRL algorithms on single and sequentially presented tasks learning (Section . ). We sum up the method in Section . just before we take a step back, will evaluate how ELSIM could be integrated in a hierarchical setting and discuss the limitations of ELSIM in Section .6. Based on this discussion, we investigate a potential solution in Section . .

. Background . . Obtaining diverse skills through mutual information objective

One way to learn, without extrinsic rewards, a set of different skills is to use the objective discussed in Section . . . We already discussed the intuition of the method in Section . . , here we provide more technical details.

In DIAYN [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF], learned skills should be as diverse as possible (different skills should visit different states) and distinguishable (it should be possible to infer the goal from the states visited by the skill). It follows that the learning process of DIAYN is -step with two learning parts: -the agent samples one skill from an uniform distribution; -the agent executes the skill (randomly initialized); -a discriminator q ω learns to categorize the resulting states to the assigned skill; -at the same time, these approximations reward (cf. Equation .).

The global objective can be formalized as maximizing the MI between the set of skills G and states S ′ visited by skills, defined by [Gregor et al.,6]:

I(G; S ′ ) = H(G) -H(G|S ′ ) ( . ) = E g∼p(g) s ′ ∼p(s ′ |π g θ ,s) [log p(g|s ′ ) -log p(g)] ( . )
where π g θ is the skill associated to the goal g ∈ G and is parameterized by θ; p(g) is the uniform distribution of skills the agent samples on; and p(g|s ′ ) is the probability to infer g knowing the next state s ′ and skills. This MI quantifies the reduction in the uncertainty of G due to the knowledge of S ′ . By maximizing it, states visited by a skill π g have to be informative about the given goal g.

A bound on the MI can be used as an approximation to avoid the difficulty to compute p(g|s ′ ) [Barber and Agakov,,Gregor et al.,6] :

I(G, S ′ ) ≥ E g∼p(g) s ′ ∼p(s ′ |π g θ ,s)
[log q ω (g|s ′ )log p(g)] ( . )

where q ω (g|s ′ ) is the discriminator approximating p(g|s ′ ). In our case, the discriminator is a neural network parameterized by ω. q ω minimizes the standard cross-entropy -E g∼p(g|s ′ ) log q ω (g|s ′ )

where s ′ ∼ π g θ .

To discover skills, it is more efficient to set p(g) to be uniform as it maximizes the entropy of G

[ Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]. Using the uniform distribution, log p(g) is constant and can be removed from Equation .. It follows that one can maximize Equation . using an intrinsic reward to learn the skill g ∈ G [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]:

r g (s ′ ) = log q ω (g|s ′ ). ( . )
Similarly to [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF], we use an additional entropy term to encourage the diversity of covered states. In practice, this bonus is maximized through the use of DRL algorithms: Soft Actor Critic (SAC) [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] for continuous action space and Deep Q network . .

Related works

Since we already studied works related to skill learning in Section . , we focus on sequential learning methods and other lifelong learning aspects tackled in the literature. We also compare our technical contribution to previous approaches.

Sequential learning.

Contrary to methods that learn skills uniformly (Section . . ), some methods manage to execute skills sequentially, considering skills as options and hierarchically running several of them inside one episode. However, either they do not explore efficiently [Nachum et al., a, Levy et al., ], or explore with a hierarchical random walk [Nachum et al.,[START_REF] Bengio | Curriculum learning[END_REF]Li et al.,b]. While this is more efficient than a low-level random walk [Li et al., b], it is not as efficient as discovering novel states through novel skill discovery. Deep Covering Options (DCO) explores efficiently [Jinnai et al., ] but it is unclear whether it can learn a large set of options. Approaches also learn skills directly with the tasks [Bacon et al., , Li et al., a], but skills are biased towards a task and this prevents exploration when rewards are sparse.

Continual learning.

Other works proposed a lifelong learning architecture. Some assume that skills are already learned and learn to reuse them; for example, Hierarchical Deep Reinforcement Learning Network (H-DRLN) [Tessler et al., ] uses a hierarchical policy to choose between ground actions and skills. They also propose to distill previously learned skills into a larger architecture, making their approach scalable. In contrast, we tackle the problem of learning skills in an end-to-end fashion, thereby our approach may be compatible. Similarly to us, Continual Curiosity driven Skill Acquisition (CCSA) [Kompella et al., ] addresses the catastrophic forgetting problem by freezing the learning of some experts. They mix two unsupervised learning methods to find and represent goal states, and then learn to reach them. However, their unsupervised algorithm only extracts linear features and they manually define a first set of skills. One particular aspect of continual learning is Meta-RL: how can an agent learn how to learn ? Traditional methods assume there exists a task distributions and try to generalize over it [Finn et al.,,Duan et al.,6]; this task distribution serves as prior knowledge. In [Gupta et al.,[START_REF] Bengio | Curriculum learning[END_REF], the authors address this issue and apply Model-Agnostic Meta-Learning (MAML) [Finn et al., ] on an uniform distribution of tasks learned by DIAYN [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]. However, learning is neither focused, nor end-to-end. In the continuity of this work, Curricula for Unsupervised Meta-Reinforcement Learning (CARML) [Jabri et al., ] mixes the objective of DADS [Sharma et al., ] and Meta-RL; it alternates between generating trajectories of the distribution of tasks and fitting the task distribution to new trajectories. While CARML discovers diverse behaviors with pixel-level state space, it cannot learn a global objective end-to-end like ELSIM.

State abstraction.

Our method can be viewed as a way to perform state abstraction [Li et al.,6]. Rather than using this abstraction as inputs to make learning easier, we use it to target specific states. The application of our refinement method bounds the suboptimality of the representation, while the task-independent clustering ensures that skills are transferable. In contrast to our objective, existing methods usually tackle suboptimality for a task without addressing transfer learning or exploration [Akrour et al.,[START_REF] Bengio | Curriculum learning[END_REF]Abel et al.,6]. The k-d tree algorithm [Friedman et al., ] has been used to perform state abstraction over a continuous state space [Uther and Veloso, 8], but as above, the splitting process takes advantage of extrinsic reward and previously defined partitions are not adapted throughout the learning process. In the domain of developmental robotics, Robust Intelligent Adaptive Curiosity (RIAC) and SAGG-RIAC [Baranes and Oudeyer, , ] already implement a splitting algorithm building a tree of subregions in order to efficiently explore the environment and learn a forward model. More precisely, they split the state space to maximize either the sum of variance of interactions already collected or the difference of learning progress between subregions. However, these heuristics do not scale to larger continuous environments. In contrast, we assign states to subregions according to the proximity of states and use these subregions as reusable skills to solve several tasks. Adaptive skills adaptive partitions (ASAP) [Mankowitz et al.,6] partitions the goal space, but does not use intrinsic motivation and the partitions are limited to hyper-plans.

. Method

In this section, we first give an overview of our method and then detail the building of the tree of skills, the learning of the skill policy, the selection of the skill to refine and how ELSIM integrates this in an end-to-end framework.

. . METHOD .

. Overview: building a tree of skills

To get both bottom-up skills and interesting skills relatively to some tasks, our agent has to choose the skills to improve thanks to the extrinsic rewards, but we want that our agent improves its skills without extrinsic rewards. The agent starts by learning a discrete set of diverse and distinguishable skills using the method presented in Section . . . Once the agent clearly distinguishes these skills using the covered skill-conditioned states with its discriminator, it splits them into new sub-skills. For instance, for a creature provided with proprioceptive data, a moving forward skill could be separated into running and walking. The agent only trains on sub-skills for which the parent skill is useful for the global task. Thus it incrementally refines the skills it needs to accomplish its current task. If the agent strives to sprint, it will select the skill that provides the greater speed.

The agent repeats the splitting procedure until its skill either reach the maximum number of splits or become too deterministic to be refined.

The first splitting is illustrated in Figure . a and Figure . b, where the agent is a circle that moves in the squared enviromnent. The hierarchy of skills is maintained using a tree where each node refers to an abstract skill that has been split and each leaf is a skill being learned.

We formalize the hierarchy using sequence of letters where a letter's value is assigned to each node:

• The set of skills G is the set of leaf nodes. A skill g ∈ G is represented by a sequence of k + 1 letters : g = (l 0 , l 1 , ..., l k ). When g is split, a letter is added to the sequence of its new sub-skills. For instance, the skill g = (l 0 = 0, l 1 = 1) can be split into two sub-skills (l 0 = 0, l 1 = 1, l 2 = 0) and (l 0 = 0, l 1 = 1, l 2 = 1).

• The vocabulary V refers to the values which can be assigned to a letter. For example, to refine a skill into sub-skills, we should define V = {0, 1, 2, 3}.

• The length L(g) of a skill is the number of letters it contains. Note that the length of a skill is always larger than its parent's.

• l :k is the sequence of letters preceding l k (excluded). We use two kind of policies: the first are the goal-conditioned policies defining a skill. The learning of these skills is described in Section . . . The second type of policy is task-dependent and responsible to choose which skill to execute; we call it the tree-policy (see Section . . ).

. . Learning skills

In this section, we detail how skills are learned. We adapt the method presented in Section . . to our hierarchical skills context. Two processes are simultaneously trained to obtain diverse skills: the skills learn to maximize the intrinsic reward (cf. Equation .), which requires to learn a discriminator q ω (g|s ′ ). This is difficult because it requires to provides a lot of trajectories from each π g .

Given our hierarchic skills, we can formulate the probability inferred by the discriminator as a product of the probabilities of achieving each letter of g knowing the sequence of preceding letters, by applying the chain rule:

r g (s ′ ) = log q ω (g|s ′ ) = log q ω (l 0 , l 1 , . . . , l k |s ′ ) = log k i=0 q ω (l i |s ′ , l :i ) = k i=0 log q ω (l i |s ′ , l :i ) = k-1 i=0 log q ω (l i |s ′ , l :i ) + log q ω (l k |s ′ , l :k ).
( . )

Gathering this value is difficult and requires an efficient discriminator q ω . As it will be explained in Section . . , in practice, we use one different discriminator for each node of our tree:

∀i, q ω (l i |s ′ , l :i ) ≡ q :i ω (l i |s ′ ) where i indicates the level in the tree. For instance, if |V | = 2, one discriminator q ∅ ω will be used to discriminate (l 0 = 0) and (l 0 = 1) but an other one, q l 0 =0 ω will discriminate (l 0 = 0, l 1 = 0) and (l 0 = 0, l 1 = 1).

Training several discriminators at the same time induce several issues, -we can not simulteneaously learn all of them; -we must avoid to learn correlated discriminators that similarly partition the state space.

Primary learning for the leaves discriminators. It would be difficult for the discriminators to learn over all letters at once; the agent would gather states for several inter-level discriminators at the same time and a discriminator would not know which part of the gathered states it should focus on. This is due to the fact that discriminators and skills simultaneously train. Furthermore, there are millions of possible combinations of letters when the maximum size of sequence is large. We do not want to learn them all.

To address these issues, we introduce a new curriculum learning algorithm that refines a skill only when it is distinguishable. When discriminators successfully learn, they progressively extends the sequence of letters; in fact, we split a skill (add a letter) only when its discriminator has managed to discriminate the values of its letter. Let's define the following probability: 6) where s f inal is the state reached by the skill at the last timestep. We assume the discriminator q :k ω has finished its primary learning phase when: ∀v ∈ V, p :k f inish (l k = v) ≥ δ where δ ∈ [0, 1] is an hyperparameter. Choosing a δ close to 1 ensures that the skill is learned, but a skill always explores, thereby it may never reach an average probability of exactly ; we found empirically that . works well.

p :k f inish (l k ) = E s f inal ∼π :k+1 q :k ω (l k |s f inal ) . ( .
To approximate Equation .6 for each letters' value v, we use an exponential moving average

p :k f inish (l k = v) = (1 -β)p :k f inish (l k = v) + βq :k ω (l k = v|s f inal )
where s f inal ∼ π :k+1 and β ∈ [0; 1]. Since we use buffers of interactions (see Section . . ), we entirely refill the buffer before the split.

Learning uncorrelated discriminated areas. Let us reconsider Equation . . The left-hand part represents the reward assigned by the previously learned discriminators. It forces the skill to stay close to the states of its parent skills since this part of the reward is common to all the rewards of its parent skills. In contrast, the right-hand part represents the reward assigned by the discriminator that actively learns a new discrimination of the state space. Since the agent is constrained to stay inside the area of previous discriminators, the new discrimination is uncorrelated from previous parent discriminations. In practice, we increase the importance of previous discriminations with a hyper-parameter α ∈ R:

r g (s ′ ) = log q ω (l k |s ′ , l :k ) + α k-1 i=0 log q ω (l i |s ′ , l :i ).
( . )

This hyper-parameter is important to prevent the agent to deviate from previously discriminated areas to learn more easily the new discrimination.

. . Learning which skill to execute and train

For each task, a stochastic policy, called tree-policy and noted π T (with T the tree of skills), is responsible to choose the skill to train by navigating inside the tree at the beginning of a taskepisode. This choice is critical in our setting: while expanding its tree of skills, the agent cannot learn to discriminate every leaf skill at the same time since discriminators need states resulting from the skills. We propose to choose the skill to refine according to its benefit in getting an other reward (extrinsic or intrinsic), thereby ELSIM executes and learns only interesting skills (relatively to an additional reward).

To learn the tree-policy, we propose to model the tree of skills as an MDP solved with a Qlearning and Boltzmann exploration. The action space is the vocabulary V ; the state space is the set of nodes, which include abstract and actual skills; the deterministic transition function Figure . : Representation of a part of the tree of skills with |V | = 2 and the value of tree-policy in each node. White nodes are actual leaves of the tree (there are no discriminators since there are no children to discriminate). Yellow nodes represent nodes for which the discriminator can not differentiate its sub-skills; the tree-policy samples uniformly. Nodes are blue when the discriminator can distinguish its sub-skills; the tree-policy samples using Q-values.

is the next node selection; if the node is not a leaf, the reward function R T is 0, else this is the discounted reward of the skill executed in the environment divided by the maximal episode length.

Each episode starts with the initial state as the root of the tree, the tree-policy selects the next nodes using Q-values. Each episode ends when a leaf node has been chosen, i.e. a skill for which all its letters has been selected; the last node is always chosen uniformly (see Section . . ).

Let us roll out an example using the tree-policy displayed in Figure . . The episode starts at the root of the tree; the tree-policy samples the first letter, for example it selects l 0 = 0. Until it reaches a leaf-node, it samples new letters, e.g. l 1 = 1 and l 2 = 0. The tree-policy has reached a leaf, thereby it will execute and learn the skill (0, 1, 0). Then, the state-action tuple ((0, 1), ( 0)) is rewarded with the scaled discounted reward of the task. This reward is propagated via the Qlearning update to previous state-action tuples ((∅), (0)) and ((0), ( 1)) to orientate the tree-policy to (0, 1, 0).

The MDP evolves during the learning process since new letters are progressively added. The Q-values of new skills are initialized with their parent Q-values. However, Equation . ensures that adding letters at the leaf of the tree monotonically increases Q-values of their parent nodes. The intuition is that, when splitting a skill, at least one of the child is equal or better than the skill of its parent relatively to the task. We experimentally show this in Section . . . The resulting curriculum can be summarized as follows: the tree will be small at the beginning, and will grow larger in the direction of feedbacks of the environment.

We now sum up the process of the tree-policy: -an agent runs an episode inside the MDP of skills; the sequence of actions represents a skill; -the agent executes the skill; -the tree-policy is rewarded according to how well the skill fits the task and the Q-learning applies. When the tree policy gathers a constant (possibly null) reward, it becomes uniform. The full algorithm of the tree-policy is given in Appendix A. .

CHAPTER . END-TO-END LEARNING OF REUSABLE SKILLS THROUGH INTRINSIC MOTIVATION

. .

Simultaneous training of the tree-policy and skills

The MI objective requires the skill distribution to remain uniform (cf. Equation .), however that is not our case: the agent strives to avoid some useless skills while focusing on others. In our preliminary experiments, ignoring this leads us to catastrophic forgetting of the learned skills since discriminators forget how to categorize states of the skills they never learn on. To bypass this issue and sample uniformly, we assign to each node i of our tree a replay buffer containing interactions of the skill with the environment, a RL algorithm and a discriminator (q :i ω ). At each split, skills and buffers of a node are copied to its children; for the first node, its skills are randomly initialized and its buffer is empty. This way, the entire training is off-policy: the skill fills the replay buffer while the discriminator and skills learn from the interactions that are uniformly extracted from their buffers. We divide the lifetime of a node into two phases, before and after the split: -the learning phase during which next letter's values are sampled uniformly; the tree-policy is uniform at this node; -the exploitation phase during which the tree-policy chooses letters with its Boltzmann policy (Section . . ).

Then, at each step, the agent runs the tree-policy to select the discriminator in the learning phase that will learn. The discriminator samples a mini-batch of data from its children's (all leaves) buffers and learns on it. Then, all children skills learn from the intrinsic feedback of the same interactions, output by the selected discriminator and all its parents according to Equation .. Once a node enters the exploitation phase, an hyper-parameter η regulates the probability that each parent's discriminator learn on its children data. Their learning interactions are recursively sampled uniformly on their children. This post-exploration learning allows a node to expand its high-reward area. Without this mechanism, different uncovered states of the desired behaviour may be definitively attributed to different fuzzy skills, as shown in Section . . . The full learning algorithm is given in Appendix A. . Figure . gives an example of a potential tree and how different phases coexist; the skills starting by (0, 1) seem to be the most interesting for the task since each letter sampling probability is high. Skills (0, 1, 0), (0, 1, 1), (1, 0) and (1, 1) are being learned, therefore the sampling probability of their last values is uniform.

. Experiments

The first objective of this section is to study the behavior of our ELSIM algorithm on basic gridworlds to make the visualization easier. The second purpose is to show that ELSIM can scale with high-dimensional environments. We also compare its performance with a non-hierarchical algorithm SAC [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] in a single task setting. Finally we show the potential of ELSIM for transfer learning.

. . Study of ELSIM in gridworlds

In this section, we analyze how skills are refined on simple gridworlds adapted from gymminigrid [START_REF] Bengio | Curriculum learning[END_REF]. Unless otherwise stated, there is no particular task (or extrinsic reward), thereby the tree-policy is uniform.

Experimental setup. The observations of the agent are its coordinates; its actions are the movements into the four cardinal directions. Our hyperparameters can be found in Appendix A.

To maximize the entropy of the skill with a discrete action space, we use the DQN algorithm [Mnih et al., ]. The agent starts an episode in the middle of the grid (see figures) and an episode resets every steps, thus the skill lasts steps. At the end of the training phase, the skills of all the nodes are evaluated through an evaluation phase lasting steps for each skill. In all figures, each tile corresponds to a skill that is displayed at the top-left of the tile. We set |V | = 4, so that there are four skills at the beginning of the tree and each skill splits into four new subskills. Figure . 6,Figure . and .8 display the density of the states visited by skills during the evaluation phase: the more red the state, the more the agent goes over it.

Can ELSIM refine a high-stochastic skills into low-stochastic skills ?

The first and sixth column of Figure . shows, for each possible skill g when L(g) = 1, the states covered by the agent during the evaluation phase of the skills. We can see that the agent clearly separates its skills since the states covered by one skill are distinct from the states of the other skill. In our example, the first skill (0) makes the agent go at the right of the grid, the second one (1) at the top, the third one (2) at the left and the fourth one at the bottom. In contrast, columns two to five and seven to eleven of Figure . show the skills learned with L(g) = 2. As evidenced by goals' numbers, the four rightmost skills are the refinement of the leftmost fuzzy policy on the same row. We see that the refinement allows to get lower-stochastic policies. For example, skill (1) is very fuzzy while its children target very specific areas of the world. This emphasizes the benefits of using more latent variables to control the environment. Do the split of skills improve the exploration of an agent ? Figure .6 shows some skills learned in an environment of rooms separated by a bottleneck. The full set of skills is displayed in Appendix A. . We first notice that the agent clearly separates its first skills (0), ( 1), ( 2 3) and their sub-skills in every state (i.e. q(g|s)). The more red the state, the more rewarding it is for the skill. The left side corresponds to the preliminary stage of the learning process (timestep 128.10 4 ); the right side corresponds to the end of the learning process (timestep 640.10 4 ).

(2, 0), (2, 2), (2, 3)) rooms. With a second refinement, (2, 3, 0) even manages to reach the farthest room (bottom-right). This stresses out that the refinement of a skill also allows to expand the states covered by the skill, and thus can improve the exploration of an agent when the rewards are sparse in an environment.

Does the split of skills correct a wrong over-generalization of a parent skill ? Figure . shows the evolution of the intrinsic reward function for some skills (see Appendix A.6 for the full set of skills). The environment contains a vertical wall and settings are the same as before, except the Boltzmann parameter set to . . At the beginning, skill (1) is rewarding identically left and right sides of the wall. This is due to the generalization over coordinates and to the fact that the agent has not yet visited the right side of the wall. However it is a wrong generalization because left and right sides are not close to each other (considering actions). After training, when the agent begins to reach the right side through the skill (3, 2), it corrects this wrong generalization. The reward functions better capture the distance (in actions) between two states: states on the right side of the wall are attributed to skill (3) rather than (1). We can note that other parts of the reward function remain identical.

Can the agent choose which skill to develop as a priority ?

In this part, we use the same environment as previously, but states on the right side of the wall give an extrinsic reward of 1. Thus the agent follows the tree-policy to maximize its rewards, using Boltzmann exploration, and focus its refinement on rewarding skills. Figure .8 shows all the parent skills of the most refined skill which reaches L(g) = 6. The agent learns more specialized skills in the rewarding area than when no reward is provided (cf. Appendix A. for the full set of skills learned).

Summary. We illustrated the following properties of ELSIM: -it expands a previously learned rewarding area when it discovers new states; -we show in Section . . that it improves exploration when the rewards are sparse; -adding letters corrects over-generalization of their parent MountainCarContinuous-v [Moore, ] and Pendulum-v ) for SAC and ELSIM (averaged over seeds). We use our own implementation of SAC except for HalfCheetah for which the blue curve is the average reward of SAC on seeds, taken from [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF]. We stopped the simulation after convergence of SAC.

discriminator; -it can focus the skill expansion towards task-interesting areas.

. . Performance on a single task

In this part, we study the ability of ELSIM to be competitive on continuous benchmarks without any prior knowledge. Action and state spaces of used environments are summarized in Table . . Unless stated otherwise, used hyper-parameters can be found in Appendix A. . For ELSIM, we set the maximum skill length to , which is reached in HalfCheetah. Shaded areas color are upper-bounded (resp. lower-bounded) by the maximal (resp. minimal) average reward.

First, the MountainCarContinuous environment represents a challenge for the exploration as it is a sparse reward environment: the agent receives the reward only when it reaches the goal. In this environment, ELSIM outperforms SAC by getting a higher average reward. It confirms our results

(cf. Section . . ) on the positive impact of ELSIM on the exploration. There is a slight decrease after reaching an optima, in fact, ELSIM keeps discovering skills after finding its optimal skill. On Pendulum and LunarLander, ELSIM achieves the same asymptotic average reward than SAC, even though ELSIM may require more timesteps. On HalfCheetah, SAC is on average better than ELSIM.

Environment

However we emphasize that ELSIM also learns other skills. For example in HalfCheetah, ELSIM learns to walk and flip while SAC, which is a non-hierarchical algorithm, only learns to sprint.

. . Transfer learning

In this section, we evaluate the interest of ELSIM for transfer learning. We take skills learned by intra-policies in Section . . , reset the tree-policy and restart the learning process on HalfCheetah and HalfCheetah-Walk. HalfCheetah-Walk is a slight modification of HalfCheetah which makes the agent target a speed of (cf. Appendix A.8 for more details on the new reward function). skills learning was stopped in HalfCheetah.

In contrast with our previous experiment, the performance exclusively depends on the exploration of the tree. Therefore we use [START_REF] Bengio | Curriculum learning[END_REF] to more efficiently explore the tree. This method adds a count-based bonus to the Q-value:

Q(s, a) = Q(s, a) + β n(s, a) ( .8)
where n(s, a) is the number of time we chose a in s and β is an hyper-parameter. We set it to for classic HalfCheetah and for walking HalfCheetah. The agent selects the action that has the larger Q.

Figure . shows that the tree-policy learns to reuse its previously learned skills on HalfCheetah since it almost achieves the same average reward as in Figure . . On HalfCheetah-Walk, we clearly see that the agent has already learned skills to walk and that it easily retrieves them. In both environments, ELSIM learns faster than SAC, which learns from scratch. It demonstrates that skills learned by ELSIM can be used for other tasks than the one it has originally been trained on.

. . CONCLUSION

. Conclusion

We proposed ELSIM, a novel algorithm that continually refines discrete skills using a recently defined diversity heuristic [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF]. To do so, the agent progressively builds a tree of different skills in the direction of a high-level objective. As shown in Section . , ELSIM expands the area associated to a skill thanks to its exploratory behavior which comes from adding latent variables to the overall policy. ELSIM also focuses its training on interesting skills relatively to some tasks. Even though the agent is often learning a task, the skills can be defined independently from a specific task and we showed that ELSIM possibly makes them transferable across different tasks of a similar environment. Since the agent does not need extrinsic reward to learn, we show that it can improve exploration on sparse rewards environments. We trained ELSIM on relatively simple benchmarks, however, it is a strong proof-of-concept that emphasizes the potential ability of bottom-up skill discovery to tackle exploration and transfer learning.

Let us recall that our final objective is to integrate this paradigm of skill discovery in a hierarchical scenario where several upper-level modules set different higher-level goals inside one episode. Therefore, in the next section, we assess the ability of ELSIM to integrate a hierarchical setting. We also discuss its exploration and transfer performance.

.6 Limitations of ELSIM

ELSIM learns a discrete set of skills and, when it does not access rewards, it explores uniformly over its set of skills while building the tree of skills. In this section, we will explain why these properties hurt the integration of ELSIM in a hierarchical framework and why it sub-optimally bounds both its exploration process and its ability to reuse skills. We will sketch some outlooks, however these observations will incite us to tackle the problem of continual bottom-up skill discovery in a different way in the next chapter.

.6. Sub-optimal exploration

ELSIM is able to keep improving its skills in an end-to-end way, even though it does not access rewards. However it uniformly samples skills to execute, and thus uniformly tries to improve them. Such uniform discovery is, in fact, strongly sub-optimal. To illustrate this, let us reconsider our previous examples. In Figure . , the agent starts in the middle of an empty environment, it can explore all around its starting position and each skill can be improved; in this situation, the exploration process is going well: its uniform sampling matches an uniform distribution of the state space. sampling skills at each level of its tree, it more thoroughly focuses on the top-left room. Indeed, of its first-level skills ( and ) can not leave the room. Uniform sampling over each level of the tree hierarchy results in at least 50% of skills sampled in the top-left room, despite they can not be improved. There is no mechanism for inciting the agent to focus on skills that can be improved or novel areas. This issue is highlighted on much more complicated benchmarks in the next chapter. This issue could be addressed by taking advantage of learning progress measures [Oudeyer and Kaplan, ] or count-based entropy maximization [Watters et al., , Pong et al., ]. In the next section, we will focus on the limitations proper to the discretization of the goal space. .6.

Discrete set of skills

Currently, our method allows to avoid the problem of catastrophic forgetting, but the counterpart is an increase of the memory footprint, which is a recurrent issue in methods based on trees. ELSIM creates novel neural networks at each level of the hierarchy, it prevents the transfer of knowledge between the different neural networks. But even though one neural network may approximate several same-level policies, the input goal space is strongly unstructured (one hot encoding vector) so that the goal-conditioned policy of an agent can not generalize on it: the distance between two goal representations does not give information about the proximity of the resulting skills.

This generalization issue also occurs on the top-down vertical representation of the tree. Knowing that a skill performs badly does not give sufficient information about the performance of skills lower in the tree, making samples unefficient. This is stressed out by our transfer learning expriment in Figure . : while ELSIM indeed finds the correct already learnt skill, it took almost steps to find out which one is the best, even though it takes advantage of the tree. This problem is even more prominent if one wants to integrate ELSIM in a multi-level hierarchical architecture. Assuming the goal space is the action space of an upper-level agent, its upper-level action space would be a very large discrete action space, which is intractable for a DRL agent: typical approaches that scale to large action spaces take advantage of a continuous embedding of actions [Dulac-Arnold et al., ].

To sum up, the discrete set of goals is not adequate, neither as an input for a low-level goalconditioned policies, nor as an output for a high-level policy. These observations suggest that a continuous embedding would more easily integrate a hierarchical organization and make easier transfer learning. There are two alternatives to build a continuous embedding:

. We can map the discrete tree representation to a continuous space.

. The agent should directly learn a continuous embedding.

In the last section of the chapter, we briefly study the relevance of the first option: can we map a discrete tree to a continuous embedding ? By highlighting our preliminary investigation, we mean to provide explanation and a basic overview of this research outlook. We more thoroughly study the second option in the Chapter . .

Learning a continuous hyperbolic embedding of skills

This section provides basic materials about how to learn a continuous embedding of trees so that we can discuss if ELSIM could integrate a hierarchical setting. It ends up with a discussion on the perspectives of this approach.

. . A short introduction to hyperbolic spaces

Creating a continuous embedding of the tree of skills is not trivial; this embedding should be induced with a metric that captures how much skills are similar. Similarity between skills here refers to the length of the path in the tree that connects skills, this comes from how we build our tree of skills (Section . ). For instance, in the right part of Figure . , z is far from z ′ since three nodes separate them, but close to z ′′ since one node separate them. While the induced metric usually follows an euclidian distance, the required dimensionality grows too fast with the number of levels when the metric is intrinsically hierarchical like in a tree [Linial et al.,,Sala et al.,[START_REF] Bengio | Curriculum learning[END_REF],

the required dimension of the embeddings may be unbounded. This is especially due to the fact that the number of nodes of a tree grows exponentially with the depth of a tree. More specifically, assuming each node of a tree has b children, with maximum depth l, we can compute the number of nodes as n = 1 + b l-1 . The right part of Figure . illustrates this issue: the euclidian space quickly saturates and, although nodes z and z ′ should be very different, their euclidian distance is low.

Recent studies rather suggest that hyperbolic embeddings may more appropriately fit the hierarchical structure of a tree than embeddings provided with an euclidian metric [Nickel and Kiela, , Sonthalia and Gilbert, , Law et al., , Chamberlain et al., ]. Essentially, hyperbolic geometry is a non-Euclidian geometry that studies spaces provided with a constant negative sectional curvature. This curvature modifies the notion of distance between points so that the area of a disc grows exponentially with its radius (instead of quadratically for an uncurved euclidian geometry). It follows that hyperbolic geometry can be thought as a continuous version of a tree [Krioukov et al., ], making intuitive its relevance for embedding a tree.

CHAPTER . END-TO-END LEARNING OF REUSABLE SKILLS THROUGH INTRINSIC MOTIVATION

Several models exist to study an hyperbolic geometry, but we focus on the Poincare Ball, since this is largely used in machine learning [Nickel and Kiela, , Khrulkov et al., ], even though other models may be used [Nickel and Kiela,[START_REF] Bengio | Curriculum learning[END_REF]. The Poincare Ball model corresponds to the Riemannian manifold with (B d c , g c p ) where c < 0 is an arbitrary negative curvature coefficient,

B d c is the d-dimensional open-ball of radius 1 √ |c| and g c p = 2 1+c||z|| 2 2
g e is its metric tensor. g e is the euclidian metric tensor, i.e it defines a dot-product. Using this model, all points belong to B d c and we can analytically compute the distance between two points using Equation . .

d c p (z, z ′ ) = 1 |c| arcosh 1 + 2c ||z -z ′ || 2 2 (1 + c||z|| 2 2 )(1 + c||z ′ || 2 2 . ( . )
The Poincare disc model B 2 1 illustrated in the left part of Figure . shows how distances are altered when points get closer to the boundary of the ball: they exponentially increase with small variations. This is reflected in the distance estimation of Equation . , the denominator exhibits that, when vectors are closed to the boundary, distances tend to be larger than those of vectors close to the origin of the ball. Consequently, the center of the ball is close to all points relatively to the boundary points; this gives an intuitive understanding of the induced hierarchy: the root of the tree should be close to the origin and leaves should be close to the boundary of the ball.

. . Hyperbolic skill embedding

In practice, there are two ways to compute an hyperbolic embedding in a Poincare Ball. We can use optimization methods based on SGD, contrastive learning and hyperbolic distance [Nickel and Kiela, ]. Equation . provides an example of loss function which simultaneously finds out embeddings of all nodes, using a dataset D of connected nodes. The numerator brings together connected nodes while the denominator moves away unconnected nodes.

L Z = (z,z ′ )∈D log e -d c p (z,z ′ ) zn|(z,zn) / ∈D e -d c p (z,zn) + 1 ( . )
Alternatively, one can analytically embeds a provided tree [Sala et al., 8, Sarkar, ] using geometric operators [Brannan et al., ]. This method, called Sarkar's construction, progressively embeds nodes by starting from the root and adding nodes with a depth-first strategy. In both cases, our preliminary experiments highlight several shortcomings. We briefly detail them below.

Scaling factor.

An example of already learnt skill embedding resulting from Sarkar's construction applied on skills previously stored is displayed in Figure . . We see that skill (0, 0, 0) is close to skill (2, 2), even though we consider an hyperbolic distance. In fact Sarkar's construction trades off how quickly close-to-the-root nodes gets closer to the boundary with a scale factor τ . A high scale factor will set a lot of nodes very close to the boundary. This keeps the distances of the original tree but makes embeddings sensitive to the precision of float numbers [Sala et al.,[START_REF] Bengio | Curriculum learning[END_REF]: for an embedding e, 1 √

e should be different from 0, but as it gets closer to (e.g 10 -15 ), it may be confounded with . A low scale factor reverses the trade-off. This issue is shared by the . . LEARNING A CONTINUOUS HYPERBOLIC EMBEDDING OF SKILLS Graph versus tree. Figure . highlighted that ELSIM can refine a high-stochastic skill into a lowstochastic one. This is interesting because it highlights that the tree of skills captures a hierarchical structure of the state space built by the discriminators; close-to-the-origin skills target larger part of the state space whereas leaf skills target smaller parts. However, in practice, the tree and skills most of time do not follow this semantic. This is emphasized by Figure .

. . Conclusion

These observations suggest that our hierarchical representation is not adequate to explore and match the structure of the environment, a graph may be more appropriate by being able to handle such setting. However, further works on hyperbolic embeddings may open new research perspectives by considering a continuous hierarchical abstraction of the state space. We expect further improvements in hyperbolic DRL algorithms to open this path. In the next chapter, we thoroughly investigate our second option which was about directly learning a continuous representation of the state space. In particular, we will introduce a new model that aims to tackle the shortcomings of ELSIM.

Chapter 5

Discovering a topological representation to learn diverse and rewarding skills

In the last chapter, we stressed out several mandatory properties to learn skills that can be simultaneously learnt and organized in a hierarchical way. We sum up these desideratas:

. An agent should stop trying to improve a skill if it can not extend it.

. Learning a continuous skill representation may make easier both a hierarchical integration of skills and the generalization of a goal-conditioned policy.

. A graph-based representation of the environment may better fit the structure of the environment.

In particular, an efficient way to explore an environment can be to learn diverse skills that maximize the state entropy [Hazan et al., , Pong et al., ]. To follow such principle, we propose that the agent should not focus on what to learn, as commonly done in RL, but rather on where to learn in its environment. Thus the agent should learn a representation of states that keeps the structure of the environment to be able to select where it is worth learning, instead of learning a representation that fits a wanted behavior which is most of the time unknown. Indeed, a standard DRL agent tackles the issue of learning a representation of its states from scratch using only extrinsic rewards. In a sparse-reward environment, the agent turns out to be unable to locate the reward and learn a representation of its surrounding.

In this chapter, we introduce a new way to learn a continuous goal space and select the goals, keeping the learning process end-to-end. We propose a new model that progressively Discovers a Topological representation (DisTop) of the environment. DisTop bridges the gap between acquiring skills that reach a uniform distribution of terminal embedded states and solving a task.

It makes diversity-based skill learning suitable for end-to-end exploration in a single-task setting irrespective of the ground state space. DisTop simultaneously optimizes three components:

. it learns a continuous representation of its states using a contrastive loss that brings together consecutive states;

. this representation allows the building of a discrete topology of the environment with a new variation of a Growing When Required network [Marsland et al., a]. Using the clusters of the topology, DisTop can sample from an almost-arbitrary distribution of visited embedded states using a very small set of parameters;

. it trains a goal-conditioned deep RL policy to reach embedded states.

Upon these components, a hierarchical state-independent Boltzmann policy selects the cluster of skills to improve according to an extrinsic reward and a diversity-based intrinsic reward.

We show, through DisTop, that the paradigm of choosing where to learn and what to forget using a learnt discrete topology is more generic than previous approaches. Our contribution is -folds: -we visualize the representation learnt by DisTop and exhibit that, unlike previous approaches [Pong et al., ], DisTop is agnostic to the shape of the ground state space and works with states being images, high-dimensional proprioceptive data or high-dimensional binary data;

-we demonstrate that DisTop clearly outperforms our previous method ELSIM; -we show that DisTop achieves similar performance with state-of-the-art (SOTA) algorithms on both single-task dense rewards settings [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] and multi-skills learning benchmarks [Pong et al., ]; -we show that it improves exploration over a state-of-the-art hierarchical method on hard hierarchical benchmarks.

In Section . , we first describe the basic elements required to understand DisTop . Then, in Section Section . we describe the motivation of DisTop and its core components. After that, we describe the experiments that support the effectiveness of DisTop (Section . ). Finally, we explain how our work relates with previous approaches (Section . ) and we discuss the perspectives and limitations of DisTop (Section . ).

. Background

In this section, we introduce the key tools required to understand how DisTop works. This includes the framework of entropy maximization of Skew-fit, basic elements on contrastive learning and a well-known topology-mapping growing network.

. . Goal-conditioned RL for skill discovery Skew-Fit [Pong et al., ] strives to learn goal-conditioned policies that visit a maximal entropy of states. We already discussed Skew-fit in Section . . but we deepen some aspects in this section.

To generate goal-states with high entropy, they learn a generative model, i.e a model that generates states, that incrementally increases the entropy of generated state-goals and makes visited states progressively tend towards a uniform distribution. Assuming a parameterized generative model of states q ψ (s) is available, an agent would like to learn ψ to maximize the log-likelihood of SKILLS the state distribution according to a uniform distribution: E s∼U (S) log q ψ (s). But sampling from the uniform distribution U (S) is hard since the set of reachable states is unknown.

Skew-Fit uses importance sampling and approximates the true ratio with a skewed distribution of the generative model q ψ (s) α skew where α skew < 0. This way, it maximizes E s∼B q ψ (s) α skew log q ψ (s)

where s are uniformly sampled from the buffer B of the agent. α skew defines the importance given to low-density states; if α skew = 0, the distribution will stay similar, if α skew = -1, it strongly overweights low-density states to approximate U (S). In practice, they show that directly sampling states s ∼ 1 Z q ψ (s) α skew , with Z being the normalization constant reduces the variance.

. . Contrastive learning and InfoNCE

Contrastive learning aims at discovering a similarity metric between data points [Chopra et al., ]. First, positive pairs that represent similar data and fake negative pairs are built. Second they are given to an algorithm that computes a data representation able to discriminate the positive from the negative ones. Figure . illustrates the learning process when no labels are provided. The agent samples an anchor and embeds in a low-dimensional state represented by a circle. Then a process transforms the image in positive examples and the agent other data points denoted labeled as negatives, here images of elephants and cats. The agent brings together the anchor and its transformation in the embedding space and separates the anchor from negative samples. In unsupervised learning, the positive sample building process is a critical step since the whole embedding builds over this. Several methods [Laskin et al., , Schwarzer et al., ] propose to build them using data augmentation methods; this includes random shifts, crops or color jitter.

InfoNCE [van den Oord et al.,[START_REF] Bengio | Curriculum learning[END_REF] proposed to build positive pairs from states that belong to the same trajectory, making the process scale to all input representations. The negative pairs are built with states randomly sampled. For a sampled trajectory, they build a set B that concatenates N -1 negative states and a positive one s t+p . Then, they maximize L N Inf oN CE = -E B log fp(st+p,vt) s j ∈B fp (sj ,vt) where v t represents an aggregation of previous states learnt with an autoregressive model, p indicates the additional number of steps needed to select the positive sample and f p are positive similarity functions. The numerator brings together states that are part of the same trajectory and the denominator pushes away negative pairs. InfoNCE maximizes a lower bound of the mutual information I(v t , s t+p ).

One can also see such optimization function as metric learning [Tschannen et al., ]. The objective of metric learning is to learn a function g : R d → R p that maps semantically similar inputs x ∈ R d onto metrically close points in R p . In the case of InfoNCE, we can set f p (x, y) = sim(g(x), g(y)) where sim is a similarity function that defines the wanted metric (euclidian distance for example).

. . Growing when required

Several methods introduced unsupervised neural networks to learn a discrete representation of a fixed input distribution [Kohonen, , Fritzke et al., , Marsland et al., a]. The objective is to have clusters (or nodes) that cover the space defined by the input points . Each node c ∈ C has a reference vector w c which represents its position in the input space. We are interested in the Growing when required algorithm (GWQ) [Marsland et al., a] since it updates the structure of the network (creation and deletion of edges and nodes) and does not impose a frequency on node addition. Figure . shows the network learnt while facing points sampled inside the two squares. The network globally matches the input distribution and quickly changes its topology along with the input distribution (between the second and third squares).

In the following, we will summarize the algorithm.It starts with nodes and an edge that connects them. A d dimensional weight vector w i is associated to each node i, denoting its position in the input space. Input points x t are sequentially given to the network and each one go through several steps:

. get the closest node c 1 t to x t (firing one) and its second closest node c 2 t ;

. add (or update) a winning edge with age = 0 between the two closest nodes;

. if the distance between c 1 t and x t exceeds a threshold and the c 1 t firing count is greater than SKILLS . if no node is added, move c 1 t and its neighbors towards x t according to the respective learning rates α and α neighbors ;

. increment the age of the edges of c 1 t and the firing count of c 1 t ;

6. the age of an edge is deleted when it exceeds a threshold and a node is deleted when it does not have neighbors. The procedure repeats until a stopping criterion is met.

We provide additional details about the algorithm in Section . . .

. Method

In this section, we introduce our model. Firstly, we give an overview of the method, then we will successively present the core components of the model, namely a new state-representation learning method (Section . . ), a way to discretize the state-representation in order to bias state sampling (Section . . ), how we mix the previous sampling with extrinsic rewards (Section . . ) and technical details about the reward computation and our goal-relabeling strategies (Section . . ).

. . Overview

As in the previous chapter, we define the interest of an agent so that its default behavior is to learn skills that achieve a uniform distribution of terminal states while focusing its skill discovery in extrinsically rewarding areas when a task is provided. If the agent understands the true intrinsic environment structure and can execute the skills that navigate in the environment, a hierarchical policy would just have to select the state to target and to call for the corresponding skill. However, this is difficult to learn both skills and the structure of the environment. First the ground representation of states may give no clue of this structure and may be high-dimensional. Second, the agent has to autonomously discover new reachable states. Third, the agent must be able to easily sample goals where it wants to learn. . It samples an interaction (s prev , a, s, s g ) from different buffers (see below), where s g is the original goal-state and embeds the associated states with a neural network φ ω (cf. Section . . ). Here, it selects the second buffer.

. Then, φ ω is trained with a new contrastive loss that brings together consecutive states in the embedded space (cf. Section . . ). This makes the embeddings of states reflect the topology of the environment, which is important to make the next step meaningful.

. After that, our growing network dynamically clusters the embedded states in order to uniformly cover the embedded state distribution. In Figure . , the two nearests clusters, three and four gets closer to the input embedding. DisTop approximates the probability density of visiting a state with the probability of visiting its cluster (cf. Section . . ).

Finally, it assigns buffers to clusters, and can sample goal-states or states with almost-arbitrary density over the support of visited states, e.g the uniform or reward-weighted ones (cf. Section . . ).

Figure . describes how DisTop gathers states:

. At the beginning of each episode, a state-independent Boltzmann policy π high selects a cluster.

. Then a goal-state s is selected that belongs to the buffer of the selected cluster and the agent finally computes its representation g = φ ω ′ (s) where weights ω ′ are a slow exponential moving average of ω.

. A goal-conditioned policy π g θ , trained to reach g, acts in the environment and discovers new reachable states close to its goal.

. The interactions made by the policy are stored in two buffers: according to their initial objective and according to the embedding of the reached state.

We will now detail each step of the DisTop process. . .

Learning the topology of the states

In order to learn the discrete topology of the known states, the agent passes through two steps: -it learns a continuous representation of states undistorted with respect to the environment dynamics; -it discretizes this representation and tracks the changes.

Learning an undistorted continuous representation of states. DisTop strives to learn a state representation function φ ω that maps a given state to an embedded space. To learn this function, we propose to maximize the constrained mutual information between the consecutive states resulting from the interactions of an agent. We call consecutive states, pair of states that are separated by one action, i.e pairs for which there exists a transition. We define the binary operator ✶(s, s ′ ) to be equal to if s and s ′ follow each other, otherwise.

To learn φ ω , DisTop takes advantage of the InfoNCE loss (cf. Section . . ). In contrast with previous approaches [Lu et al.,,van den Oord et al.,[START_REF] Bengio | Curriculum learning[END_REF], we want to keep our representation locally undistorted [Lesort et al.,[START_REF] Bengio | Curriculum learning[END_REF]Indyk,], so that it reflects the true topology of the environment and we can apply a L -based clustering algorithm on this representation. We call undistorted a representation that respects:

||φ ω (s) -φ ω (s ′ )|| 2 ≤ k c , if ✶(s, s ′ ). ( . ) ||φ ω (s) -φ ω (s ′ )|| 2 > k c , otherwise. ( . )
To compute such representation, we do not consider the whole sequence of interactions since this makes it more dependent on the policy. Typically, a constant and deterministic policy would lose the structure of the environment and could emerge once the agent has converged to an optimal deterministic policy. DisTop considers its local variant and builds positive pairs with only consecutive states. This way, it keeps distinct the states that cannot be consecutive. We propose to select our unique similarity function as f ω (s t , s t+1 ) = e -k||φω(st)-φω(st+1)||2 where φ ω is a neural network parameterized by ω, || 2 is the L norm and k is a temperature hyper-parameter [Chen et al., ]. If B is a batch of N -1 negative states and 1 consecutive state, the local InfoNCE objective, LInfoNCE, is described by Equation ..

L Inf oN CE = E (st,st+1)∈B log f ω (s t , s t+1 ) s∈B f ω (s, s t+1 ) = E (st,st+1)∈B -k||φ ω (s t ) -φ ω (s t+1 )|| 2 -log( s∈B f ω (s, s t+1 )) ( . ) = E (st,st+1)∈B -k||φ ω (s t ) -φ ω (s t+1 )|| 2 -log(f ω (s t , s t+1 ) + s∈B s =s t f (s, s t+1 )) ( . ) ≥ E (st,st+1)∈B -k||φ ω (s t ) -φ ω (s t+1 )|| 2 -log(1 + s∈B s =s t f (s, s t+1 )) ( . )
In the last line of Equation . , we upper-bound f ω (s t , s t+1 ) with since e -v < 1 when v is . . METHOD positive. The logarithmic function is monotonic, so the negative logarithm inverses the bound. We introduce this bound since we found it to stabilize the learning process.

Intuitively, Equation . brings together consecutive states, and pushes away states that are separated by a large number of actions. This is illustrated in the second step of Figure . . There are several drawbacks with this objective function. Firstly, the representation may still be strongly distorted, making a clustering algorithm inefficient since the pushing away term (righthand term) can override the other one. Secondly, the DRL algorithm requires semantically stable inputs to compute Q-values: if the representation of its goals quickly changes, it can not take into account the changes and may output bad approximations of Q-values. To tackle this, Equation . 6reformulates the objective as a constrained maximization. Firstly, DisTop forces the distance between consecutive states to be below a threshold δ (first constraint of Equation . 6). Secondly, it enforces our representation to stay consistent over time, i.e lower than a close to constant ǫ (second constraint of Equation . 6). In consequence, we avoid using a hand-engineered environmentspecific scheduling [Pong et al., ] and update representations in an online fashion.

max ω E (st+1)∈B -log(1 + s∈B f ω (s, s t+1 )) s.t. E (st,st+1)∈B ||φ ω (s t ) -φ ω (s t+1 )|| 2 ≤ δ E st+1∈B ||φ ω (s t+1 ) -φ ω ′ (s t+1 )|| 2 2 ≤ ǫ ( .6)
Transforming the constraints into penalties, the agent maximizes Equation . . k c is the temperature hyper-parameter that brings closer consecutive states, β is the coefficient that slows down the speed of change of the representation. In practice we set k c > 1 to avoid distortions (cf.

Section . ).

L DisT op = E (st,st+1)∈B -k c (ReLU (||φ ω (s t ) -φ ω (s t+1 )|| 2 -δ)) -log(1 + s∈B f ω (s, s t+1 )) -β||φ ω (s t+1 ) -φ ω ′ (s t+1 )|| 2 2 ( . )
By applying this objective function, DisTop learns a consistent, not distorted representation that keeps close consecutive states while avoiding collapse (see Figure . for an example). In fact, by increasing k c and/or k, one can select the level of distortion of the representation (cf. Section . for an analysis). One can notice that the function depends on the distribution of consecutive states in B; we experimentally found that using tuples (s t , s t+1 ) from sufficiently stochastic policy is enough to keep the representation stable. We discuss the distribution of states that feed B in Section . . . In practice, to learn φ ω we do not consider the whole batch of states as negative samples. For each positive pair, we randomly sample only states within the batch of states. In the following, we will study how DisTop takes advantage of this specific representation to sample diverse or rewarding state-goals.

CHAPTER . DISCOVERING A TOPOLOGICAL REPRESENTATION TO LEARN DIVERSE AND REWARDING SKILLS

Mapping the continuous representation to a discrete topology Since our representation strives to avoid distortion by keeping close consecutive states, DisTop can match the topology of the embedded environment by clustering the embedded states. Using this topology, the next Section will detail how to bias the goal and state sampling procedure to favor exploration and the maximization of an extrinsic reward.

In order to adapt the building process to temporally-related and goal-related interactions, we adapt the GWQ and propose the Off-policy Embedded Growing Network (OEGN). OEGN dynamically creates, moves and deletes clusters so that clusters generates a network that uniformly cover the whole set of embedded states, independently of their density; this is illustrated in the two last steps of Figure . . Each node (or cluster) c ∈ C has a reference vector w c which represents its position in the input space. The update operators make sure that all embedded states φ ω (s) are within a ball centered on the center of its cluster, i.e min c (φ ω (s)-w c ) ⊤ (φ ω (s)-w c ) ≤ δ new where δ new is the radius of the ball and the threshold for creating clusters. δ new is particularly important since it is responsible of the granularity of the clustering: if it is low, we will have a lot of small clusters, else we will obtain few large clusters that badly approximate the topology.

end for

Algorithm describes the major steps of OEGN and GWQ. Operators and relative changes are described below. Specific operations of OEGN are colored in red, the ones of GWQ in blue and the common parts are black. Following Algorithm , we define e = φ ω ′ (s) and closest(e) = min c∈C (||c -e|| 2 ) for all s ∈ B. Delete operator: Delete c i if c i .error is above a threshold δ error to verify that the node is still active; delete the less filled neighbors of two neighbors if their distance is below δ prox to avoid too much overlapping; check it has been selected n del times before deleting it. Delete if a node does not have edges anymore.

Both creation and moving operators: Check that the distance between the original goal g i and . . METHOD the resulting embedding ||g ie i || 2 is below a threshold δ success .

Creation operator: Check if ||closeste i || 2 > δ new . Verify closest has already been selected δ count times by the goal-selection policy. If the conditions are filled, a new node is created at the position of the incoming input e i and is connected to closest. Update a firing count of the winning node and its neighbors and check it is below a threshold. A node is created halfway between the winning node and the second closest node. The two winning nodes are connected to the new node.

Moving operator: If no node is created or deleted, which happens most of the time, we apply the moving operator described by Equation .8. In our case, we use a very low α neighbors to avoid the nodes to concentrate close to high-density areas.

w j =          w j + α(φ ω ′ (s) -w j ), if j = closest w j + α neighbors (φ ω ′ (s) -w j ), if j ∈ neighbors(closest)
w j , otherwise.

(

Edge operator: edges are added or updated with attribute age = 0 between the winning node of e i and the one of e prev i . Edges with age = 0 are added or updated between the two closest nodes of e i . When an edge is added or updated, increment the age of the other neighbors of closest and delete edges that are above a threshold δ age .

In the next section, we detail how we take advantage of the topology to bias the state and goal sampling process.

. . Selecting novel or rewarding skills

While sampling low-density or rewarding states is attractive to solve a task, it is not easy to sample new reachable goals. For instance, using an embedding space R 10 , the topology of the environment may only exploit a small part of it, making most of the goals pointless. Similarly to previous works [Warde-Farley et al., ], DisTop generates goals by sampling previously visited states. To sample the states, DisTop first samples a cluster, and then samples a state that belongs to the cluster.

Building a skewed distribution

To sample more often low-density embedded states, we assume that the density of a visited state is reflected by the marginal probability of its cluster. So we approximate the density of a state with the density parameterized by w, reference vector of e = φ ω ′ (s):

q w (φ ω ′ (e)) ≈ count(c s ) c ′ ∈C count(c ′ ) ( . )
where count(c s ) denotes the number of interactions that belong to the cluster that contains e.

Using this approximation, we skew the distribution very efficiently by first sampling a cluster with CHAPTER . DISCOVERING A TOPOLOGICAL REPRESENTATION TO LEARN DIVERSE AND REWARDING SKILLS the probability given by

p α skew (c) = count(c) 1+α skew c ′ ∈C count(c ′ ) 1+α skew ( . )
where α skew is the skewed parameter (cf. Section . . ).

While our approximation q w (s) decreases the quality of our skewed objective, it makes our algorithm very efficient: we associate a buffer to each cluster and only weight the distribution of clusters; unlike Pong et al. [ ], DisTop does not weight each visited state , but only a limited set of clusters. In practice we can trade-off the bias and the instability of OEGN by decreasing the radius of the balls δ new : the smaller the clusters are, the smaller the bias is, but the more unstable are OEGN and the sampling distribution (see Section . ). So, in contrast with Skew-Fit, we do not need to compute the approximated density of each state, we just need to keep states in the buffer of their closest node. In practice, we associate to each node a second buffer that takes in interactions according to the proximity of the original goal with respect to the node. This results in two sets of buffers: B G and B S to respectively sample goal-states and states with the skewed distribution. Formally, given an interactions (s g , s, s ′ , a) :

• in B S , DisTop selects the cluster arg min c∈C ||φ(s) -c|| 2 ;

• in B G , DisTop selects the cluster arg min c∈C ||φ(s g ) -c|| 2 .

In the next sections, we will detail how we use this skewed distribution over B G and B S for sampling low-density goal-states (Section . . ) and sampling learning interactions (Section . . ).

Sampling from the skewed distribution It is not easy to sample new reachable goals. For instance, using an embedding space R 10 , the topology of the environment may only exploit a small part of it, making most of the goals pointless. Similarly to previous works [Warde-Farley et al., ], DisTop generates goals by sampling previously visited states. To sample the states, DisTop first samples a cluster, and then samples a state that belongs to the cluster.

Sampling a cluster:

To increase the entropy of states, DisTop samples goal-states with the skewed distribution defined in Section . . , which can be reformulated as:

p α skew (c) = e (1+α skew ) log count(c) c ′ ∈C e (1+α skew ) log count(c ′ ) .
( . )

It is equivalent to sampling with a simple Boltzmann policy π high , using a novelty bonus reward log count(c) and a fixed temperature parameter 1 + α skew . In practice, we can use a different We can add a second reward to modify our skewed policy π high so as to take into consideration extrinsic rewards. We associate to a cluster c ∈ C a value r c that represents the mean average extrinsic rewards received by the skills associated to its goals :

r c = E s∈c,g=φω(s) 1 T T t=0 E at∼π g θ (•|st),st+1∼p(•|st,at) R(s t , a t , s t+1 ). ( . )
The extrinsic value of a cluster R ext c is updated with an exponential moving average

R ext c = (1 - α c ) * R ext c + α c * r c
where α c is the learning rate. To favor exploration, we can also update the extrinsic value of the cluster's neighbors with a slower learning rate. Our sampling rule can then be :

π high (c) = softmax C (t ext R ext c + (1 + α ′ skew ) log count(c)) ( . )
Sampling a state: Once the cluster is selected, we keep exploring independently of the presence of extrinsic rewards. To approximate a uniform distribution over visited states that belongs to the cluster, DisTop samples a vector in the ball of radius δ new that surrounds the center of its cluster (Figure . b). It rejects the vector and samples another one if it does not belong to the cluster.

Finally, it selects the closest embedded state to the random vector and extracts the corresponding interaction (Figure . c).

For example, one can imagine a ball with states close to the center of the ball and two states on its surface; with a random sampling system, the agent would give priority to states close to the center. In contrast, we want to simulate uniform sampling in the ball that corresponds to the selected cluster. By computing the state that is the closest to a uniformly sampled point in the ball, our preliminary experiments suggested it increases the uniformity of selection inside the ball and favors exploration.

. . Learning goal-conditioned policies

Training We now briefly introduce the few mechanisms used to efficiently learn the skills π g θ that achieve sampled goals. Our implementation of the goal-conditioned policy is trained with Soft CHAPTER . DISCOVERING A TOPOLOGICAL REPRESENTATION TO LEARN DIVERSE AND REWARDING SKILLS Actor-Critic (SAC) [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] and the reward is the L distance between the state and the goal in the learnt representation:

r g t (s t-1 , a t-1 , s t , s g t ) = ||φ ω ′ (s t ) -φ ω ′ (s g t )|| 2 like Pong et al. [ ].
Our goal-conditioned policy π g θ needs a uniform distribution of goals to avoid forgetting previously learnt skills. Our representation function φ ω requires a uniform distribution over visited states to quickly learn the representation of novel areas. In consequence, DisTop samples a cluster c ∼ p α skew and randomly takes half of the interactions from B G and the rest from B S . We can also sample a ratio of clusters with π high if we do not care about forgetting skills (cf. Section . ). θ, ω and w are learnt over the same mini-batch and we relabel the goals extracted from B S as detailed below.

Relabeling strategies. To increase the learning efficiency, we proposed three relabeling strategies:

. π high relabelling: we take samples from B S and relabel the goal using clusters sampled with π high and randomly sampled states. This is interesting when the agent focuses on a task; this gives more importance to rewarding clusters and allows to forget uninteresting skills. Indeed, since it does not learn of these skills, it forgets how to execute them and the corresponding nodes of the graph gets deleted.

. Uniform relabelling: we take samples from B S and relabel the goal using states sampled from from B S . When α skew ≈ 0, this is equivalent to relabeling uniformly over the embedded . Topological relabelling: we take samples from both B S and B G and relabel each goal with a state that belongs to a neighboring cluster. This is interesting when the topology is very large, making an uniform relabelling inefficient.

. Experiments

Our experiments aim at studying whether the ability of DisTopto select skills to learn and forget makes the approach generic to different task settings. We compare DisTop to three SOTA algorithms on three kinds of tasks with very different ground state spaces, thereby we compare DisTop to: the SOTA algorithm SAC [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF] on dense rewards task; the SOTA algorithm Skew-Fit [Pong et al., ] on no-rewards task; SAC and the SOTA algorithm LESSON [Li et al., b] on sparse rewards task. We also compare DisTop to ELSIM, which follows the same paradigm than ours, on the ability to explore the environment and solve a dense-reward task. That is unclear to us how to fairly adapt Skew-fit to solve a particular task and we can not assess the skills diversity of SAC since it does not learn skills. Similarly, LESSON requires a task to learn skills and it explores with hierarchical random walk, making a fair comparison unrelevant in no/denserewards task. We also compare the learnt representation with the representation learnt by a VAE to exhibit its properties. Welling, ] and periodically updates its representation. We use the same evaluation protocol than Skew-Fit: a set of images are sampled, given to the representation function and the goalconditioned policy executes the skill. In Figure .8, we observe that skills of DisTop are learnt quicker on Visual Pusher but are slightly worst on Visual Door. Since both Skew-Fit and DisTop generates rewards with the L distance, we hypothesize that this is due to the structure of the learnt goal space. In practice we observed that DisTop is more stochastic than Skew-Fit, probably because the intrinsic reward function is required to be smooth over consecutive states. It results that a one-step complete change of the door angle creates a small change in the representation, thereby a small negative intrinsic reward. Despite the stochasticity, it is able to reach the goal as evidenced by the minimal distance reached through the episode by Distop(min). Stochasticity does not bother the evaluation on Visual Pusher since the arm moves elsewhere after pushing the puck in the right position. Therefore, DisTop manages to learn diverse skills, but may more or less fluctuate according to the overall proximity of states. Does DisTop discover the environment topology even though the ground state space is unstructured ? In this experiment, we analyze the representation learnt by a VAE [Kingma and Welling, ] and DisTop. To make sure that the best representation is visualizable, we adapted the gymminigrid environment [Chevalier-Boisvert and Willems, 8] (cf. Figure . ) where a randomly initialized agent moves for fifty timesteps in the four cardinal directions. We use the ground state space, either an unstructured one with -dimensional one-hot vector with at the position of SKILLS the agent (binary) or a structured one composed of (x,y) coordinates of the agent. Interactions are given to the representation learning algorithm. Except for OEGN, we display as a node the learnt representation of each possible state and connect states that are reachable in one step. For OEGN, we simply displays the learnt network. In Figure . , we clearly see that DisTop discovers the topology of its environment since each connected points are distinct but close with each other. Similary, OEGN network closely resembles the environment topology. In contrast, the VAE representation collapses since it cannot take advantage of the proximity of states in the ground state space; it only learns a correct representation when it gets (x,y) positions, which are well-structured. This toy visualization highlights that, unlike VAE-based models, DisTop does not depend on well-structured ground state spaces to learn a suitable environment representation for learning skills.

Can DisTop solve non-hierarchical dense rewards tasks? We test DisTop on MuJoCo environments Halfcheetah-v and Ant-v [Todorov et al., ], where the agent gets rewards to move forward as fast as possible. We fix the maximal number of timesteps to . We compare DisTop to our implementation of SAC [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF], a SOTA algorithm, and to ELSIM, the method introduced in the previous chapter which follows the same paradigm than DisTop. We obtain better results than our previous experiments with ELSIM using similar hyper-parameters to DisTop. In curriculum that changes the extrinsic goal across episodes; here we train only on the farthest goal and use a sparse reward function. Thus, the agent gets a non-negative reward only when it gets close to the top-right goal. We compare our method with a SOTA hierarchical method, LESSON [Li et al., b] since it performs well on hierarchical environments with extrinsic rewards like mazes, ELSIM and our implementation of SAC [Haarnoja et al.,[START_REF] Bengio | Curriculum learning[END_REF]. For ELSIM, LESSON and DisTop, we only pass the "image" to the representation learning part of the algorithms, assuming that an agent can separate its proprioceptive state from the sensorial state. In Figure . , we can see that DisTop is the only method that manages to regularly reach the goal; in fact, looking at a learnt D OEGN network in Figure . , we can see that it successfully represents the U-shape form of the maze and sets goals close to the extrinsic goal. LESSON discovers the goal but does not learn SKILLS .

Ablation study

In this section, we study the impact of the different key hyper-parameters of DisTop. Except for previous results (entitled "paper"), we average the results over random seeds. For visualizations based on the topology, the protocol is the same as the one described in Section . . In addition, we select the most viewable one among learnt topologies; while we did not observe variance in our analysis through the seeds, the D angle of rotation can bother our perception of the topology. In Figure . , we see that the distortion threshold δ, which prevents consecutive embeddings to be equal in Equation . , also impacts the distortion of the representation, however it mostly limits the compression of the representation in the areas the agent often interacts in. In the borders, the agent often hurts the wall and stays in the same position. So in comparison with large rooms, the bring together is less important than the move away part. δ limits such asymmetries and keeps large rooms dilated. Overall, this asymmetry also occurs when the number of states increases due to the exploration of the agent: the agent progressively compresses its representation since interesting negative samples are less frequent.

Controlling

The size of the clusters of OEGN has to match the distortion of the representation. Figure .   emphasizes the importance of the creation threshold parameter δ new that rules the radius of cluster in Section . . . With a low δ new = 0.2, clusters do not move and a lot of clusters are created.

OEGN waits a long time to delete them. with a high δ new = 1, the approximation of the topology becomes very rough, and states that belong to very different parts of the topology are classified in the same cluster; this hurts our density approximation.

Selection of interactions:

Figure . shows the importance of the different hyper-parameters that rule the sampling of goals and states in Equation . and Equation . . In Ant environment, we see that the agent has to sample a small ratio of learning interactions from π high rather than p α skew ; it speeds up its learning process by making it focus on important interactions relatively to extrinsic rewards. Otherwise, it remains hard to learn all skills at the same time. However, the learning process becomes unstable if it deterministically samples from a very small set of clusters (ratios 0.9 and 0.7).

Then we evaluate the importance of 1 + α ′ skew on Visual Pusher. We see that the agent learns quicker with a low 1 + α ′ skew . It hardly learns when the high-level policy almost does not over- sample low-density clusters (-0.1). This makes our results consistent with the analysis provided in the paper of Skew-Fit [Pong et al., ].

In the last two graphics of Figure . , we show the impact of 1 + α skew ; we observe that the agent learns quicker when 1 + α skew is close to 0. It highlights that the agent quicker learns both a good representation and novel skills by sampling uniformly over the clusters.

Overall, we also observe that some seeds become unstable since they coincide with large deletions of clusters. We expect that an hyper-parameter search on the delete operators of OEGN may solve this issue in these specific cases.

.

Related works

In this section, we explain how our work relates to previous approaches.

Intrinsic skills in hierarchical RL Some works propose to learn hierarchical skills, but do not introduce a default behavior that maximizes the visited state entropy [Hazan et al., ], limiting the ability of an agent to explore. For example, it is possible to learn skills that target a ground state or a change in the ground state space [Nachum et al.,[START_REF] Bengio | Curriculum learning[END_REF]Levy et al.,]. These approaches do not generalize well with high-dimensional states. To address this, one may want to generate rewards with a learnt representation of goals. In addition, DisTop strives to learn either skills that are diverse or extrinsically rewarding. It differs from a set of prior methods that learn only diverse skills during a pre-training phase, pre-. . RELATED WORKS venting exploration for end-to-end learning. Some of them maximize the mutual information (MI) between a set of states and skills. Typically, DIAYN [Eysenbach et al.,[START_REF] Bengio | Curriculum learning[END_REF], VALOR [Achiam et al.,[START_REF] Bengio | Curriculum learning[END_REF] and SNN [Florensa et al., ] learn a discrete set of skills, but hardly generalize over skills. It has been further extended to continuous set of skills, using a generative model [Sharma et al., ] or successor features [Hansen et al., , Borsa et al., ]. In both case, directly maximizing this MI may incite the agent to focus only on simple skills [Campos et al., ]. DISCERN [Warde-Farley et al., ] maximizes the MI between a skill and the last state of an episode using a contrastive loss. Unlike us, they use the true goal to generate positive pairs and a L distance over pixels to define a strategy that improves the diversity of skills. In addition, unlike VAE-based models, our method better scales to any ground state space (see Section . ). Typically, RIG [Nair et al.,[START_REF] Bengio | Curriculum learning[END_REF] uses a VAE [Kingma and Welling, ] to compute a goal representation before training the goal-conditioned policies. Using a VAE, it is possible to define a frontier with a reachability network, from which the agent should start stochastic exploration [Bharadhwaj et al., ]; but the gradual extension of the frontier is not automatically discovered, unlike approaches that maximize the entropy of states (including DisTop Skill discovery for end-to-end exploration. Like DisTop, ELSIM (cf. Chapter ) discovers diverse and rewarding skills in an end-to-end way. It builds a tree of skills and selects the branch to improve with extrinsic rewards. DisTop outperforms ELSIM for both dense and sparse-rewards settings (cf.

Section . ). This end-to-end setting has also been experimented through multi-goal distribution matching [Pitis et al., , Lee et al., ] where the agent tries to reduce the difference between the density of visited states and a given distribution (with high-density in rewarding areas). Yet, either they approximate a distribution over the ground state space [Lee et al., ] or assume a well-structured state representation [Pitis et al., ]. Similar well-structured goal space is assumed when an agent maximizes the reward-weighted entropy of goals [Zhao et al., ].

Dynamic-aware representations. A set of RL methods try to learn a topological map without addressing the problem of discovering new and rewarding skills. Some methods [Savinov et al.,8b,a,Eysenbach et al.,] consider a topological map over direct observations, but to give flat intrinsic rewards or make planning possible. We emphasize that SFA-GWR-HRL [Zhou et al., ] hierarchically takes advantage of a topological map built with two GWQ placed over two Slow Feature Analysis algorithms [Wiskott and Sejnowski, ]; it is unclear whether it can be applied to other environments than their robotic setting. Functional dynamic-aware representations can be discovered by making the distance between two states match the expected difference of trajectories to go to the two states [Ghosh et al., ]; interestingly, they exhibit the interest of topological representations for HRL and propose to use a fix number of clusters to create goals.

Previous work also showed that an active dynamic-aware search of independent factors can disentangle the controllable aspects of an environment [Bengio et al., ]. Other methods take advantage of temporal contrastive losses for other functional uses; therefore, unlike DisTop, they CHAPTER . DISCOVERING A TOPOLOGICAL REPRESENTATION TO LEARN DIVERSE AND REWARDING SKILLS do not try to learn a topology of the environment by preventing the distortion of the representation.

For example, successor representations [Jinnai et al.,,Wu et al.,[START_REF] Bengio | Curriculum learning[END_REF] orthogonalize the features of the representation and standard temporal contrastive losses use the representation for imitation [Sermanet et al.,[START_REF] Bengio | Curriculum learning[END_REF], end-to-end task solving [Anand et al., , Stooke et al., ] or flat exploration [Yarats et al., , Guo et al., ].

.

Conclusion

We introduced a new model, DisTop, that simultaneously learns a discrete topology of its environment and the skills that navigate into it. In contrast with previous approaches ] could both take advantage of the topology and make easier states discovery; Frontier-based exploration [Bharadhwaj et al., ] could also be explored to reduce skill stochasticity. Disentangling the topology [Bengio et al., ] could improve the scalability of the approach: currently, the number of created cluster may exponentially grow with respect to the number of independent factors. .6. CONCLUSION Chapter 6

Conclusion

The concept of intrinsic motivation has been introduced in the psychological literature to describe the tendency of organisms to actively understand the structure of the world. Intrinsic motivation is able to describe parts of complex organism behaviors, highlighting its potential interest to develop agents that learn and grow like humans. On an other side, the computational framework of DRL has recently shown substantial improvements in being able to provide learning agents that can achieve complex tasks in difficult environments. Yet there are critical limitations in standard DRL: an agent requires handmade feedback to guide its learning process. This dissertation aimed to study how intrinsic motivations can complement DRL.

Synthesis of contributions

The first part of our work (Chapter ) consisted in describing the framework of DRL and defining intrinsic motivation from a computational perspective. This way, we then managed to highlight that DRL and intrinsic motivation can be theoretically used together through goal-conditioned policy, hierarchical RL and information theory. This provided the basis of a computational framework for developmental DRL. Having these key elements in hand, Chapter studied on-going research in this domain under the perspective of information theory. We proposed/reconsidered mathematical information theoretic definitions of the notions of surprise seeking, novelty seeking, and skill learning. This way, this can serve as a categorization to encompass a large body of works that tackle DRL issues like exploration in sparse rewards or skill abstraction.

Our analysis stressed out the interest of learning abstract skills: it allows surprise and novelty seeking behaviors to avoid catastrophic forgetting, and thus, avoid the collapse of the exploration process. Learning more hierarchies of abstract skills could also be crucial to imitate the hierarchy of skills learnt by biological organisms. However, we noticed that, to learn deep hierarchies of skill, an agent must be able to focus its skill discovery on what interests its upper-level skills. The analogy with humans clearly testifies this: there is no need for all humans to learn all gymnastic skills, most humans only learn the main locomotion skills like walking, turning or running. To be able to select what to improve, the agent must simultaneously select the skills it wants to improve 6. . OUTLOOKS OF RESEARCH and to learn them. This motivated the rest of the dissertation. In Chapter , we introduced ELSIM, which is a method that learns skills in a bottom-up way while being able to select which skill to improve according to extrinsic rewards. In particular, ELSIM creates a tree of discrete skills that either expands widthwise when the agent wants to explore or expands depthwise if the agent knows which skills it wants to improve according to extrinsic feedbacks. We showed that ELSIM improves transfer learning and exploration when rewards are sparse in comparison with a SOTA DRL algorithm. Despite the properties of ELSIM, we also evidended that ELSIM is unable to efficiently explore and can not easily integrate a deep hierarchy of DRL agents. In addition, the tree of skills can not capture the graph-based structure of an environment.

To tackle the challenges of ELSIM, we proposed another model, DisTop (Chapter ). DisTop learns a continuous representation of goal-states that matches the topological structure of the environment. This representation is then discretized into a graph. Thanks to this structured representation, DisTop selects which node is worth to be improved according to both exploration and extrinsic rewards. If it needs to focus on particular skills, it also forgets uninteresting skills.

Therefore, DisTop enjoys the same properties than ELSIM, but using a continuous representation of goals-states. This makes DisTop eligible for an integration in a deep hierarchy of DRL agents.

In the following section, we try to take a step back to project ourselves on what remains to be done in the domain of intrinsic motivation.

6.

Outlooks of research

Our works focused on analyzing the potential of intrinsic motivation to build open-ended learning agents. We proposed two models that can learn to achieve intrinsically generated goals while learning to achieve a task in an end-to-end way. In this section, we discuss the lessons from our works on two time-scales: -the short-term perspective and -the long-term perspective.

. On learning a representation of the environment

Here, we focus on the short-term perspectives based on lessons we learnt about our survey analysis and our models DisTop and ELSIM.

ELSIM and DisTop only consider one hierarchy of skills. Typically, the agent selects a goal (in the graph or tree) and it executes the corresponding skill in the environment. We argued in Section .6. for the need of deep composite hierarchies of skills and we hope that further works will quantitatively exhibit the interest of using several hierarchies of skills that correspond to different temporal scales. But, as highlighted in Section . , the nature of skills essentially depends on the learnt representation of states that underpins the reward function. a] and our preliminary analysis highlights that some losses do not dynamically adapt [Choi et al., ]. Studying the properties of these representations under one common framework could enlighten the path of representation learning.

Disentangling the representation. The biggest challenge for contrastive losses may be to be able to disentangle this representation to better capture the causal model of the world. This is consistent with our analysis of the difficulty of DisTop to tackle environments with several factors of variation. In the case of DisTop, such disentangled continuous representation could open the path for a disentangled discrete representation, i.e a disentangled topology of the environment. In this case, contrastive loss may go beyond the VAE approach.

. Multi-information as a universal guiding princple

We discussed in Section . how we can unify intrinsic motivations with multi-information over a hierarchical agent's model. Furthermore, we shortly argued that this single objective could be a plausible candidate for being a universal guiding principle that rules action selection and perception in organisms. In addition, the multi-information may clarify the role of ontogenetic learning and phylogenetic learning. Ontogenetic learning may refer to the multi-information maximization while phylogenetic learning could refer to learning the overall structure of the brain, considering the trade-off between processing information and minimizing the required energy, i.e following the parsimony principle [Polani, , Polani et al., ].

That is a very strong hypothesis that deserves to be verified since it could open the path to the derivation of precise objective functions. In fact, this principle transforms the problem of proposing a learning objective into the problem of proposing a cognitive architecture composed of elementary components and inductive biases. Fortunately, such proposals can be enlightened with the help of neurobiology, in contrast to the problem of proposing objectives, which hardly takes inspiration from neuroscience. Briefly, this may strengthen the relation between the domain of neurobiology and machine learning. In results, our recipe may directly tackle the actual problem of erratically proposing a plethora of objectives for an agent. Surprise may allow DisTop to target improvable skills, this includes skills to improve in sparse reward areas, but most importantly, it would make DisTop avoid purely stochastic areas where the agent could not learn skills. We currently avoid such setting in our experiments.

Skill learning: the skill learning term of the multi-information suggests to maximize I(R ′ ; G 1 |R, O ′ ) rather than I(R ′ ; G 1 ) for the goal-conditioned policy. Thus, DisTop agent should learn skills that impose a change in the representation rather than skills that target a particular area of the environment. Along with surprise, the agent may also avoid making skills that can not be achieved in particular states.

Overall, even though the discrete topological representation has several advantages presented in Chapter , this suggests that DisTop could also benefit from using objective functions as the one of HESS [Li et al., a] and NOR [Nachum et al., a], which integrate skills as changes in the representation space, incoroporate novelty-seeking behaviors and autonomously compute temporally coherent features. This example ilustrates how instrinsic motivation as a guiding principle could help us to define future objectives.

Conclusion

Developmental approaches attract a lot of attention, because of the ability of the theory to explain the development process of humans [Nguyen et al., ]. Our pathway is composed of two parts: firstly, we argued that information-theoretic intrinsic motivations in DRL and lifelong learning are plausible candidates for a computational theory of developmental learning, despite the fact that they were currently mostly addressed separately. Secondly, we introduced two models that allow to integrate the lifelong learning paradigm in works on intrinsic motivation, it follows that choosing which skills to keep and deepen makes the learning models generic to several types of tasks. This may be a key aspect to integrate skill discovery in a multi-level hierarchy of skills. To go further in this direction, we argued for the need of further evidences showing that our unification of intrinsic motivations based on the multi-information can be a universal guiding principle of autonomous development. This may provide a methological way to keep exploring the path towards human-like learning agents.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 1

Figure . :

 . Figure . : Illustration of a Deep Q-network on Atari Breakout. The state space is composed of images with dimensions 210 × 160 × 3 and the actions consist in moving the horizontal bar to the left or the right (|A| = 2).

Figure . :

 . Figure . : Illustration of a typical off-policy variant of an actor-critic architecture (value gradient architecture).

Figure . :

 . Figure . : Model of RL integrating IM, taken in Singh et al. []. The environment is factored into an internal and external environment, with all reward coming from the former.

Figure . 6 :

 6 Figure .6: Visualization of trajectories of agents that start at the white circle and try to reach the star.The blue agent uses only low-level actions while the orange can set goals.

Figure . :

 . Figure . : Illustration of the sparse reward issue in a very simple setting. The agent, represented by a circle, strives to reach the star.The reward function is one when the agent reaches the star and zero otherwise. On the left side, the agent explores with standard methods such as ǫ-greedy; as a result, it stays in its surrounded area because of the temporal inconsistency of its behaviour. On the right side, we can imagine an ideal exploration strategie where the agent covers the whole state space to discover where rewards are located.

  Figure . : Illustration of the benefits of using options. Agents, represented by circles, have to reach the star. The green agent can use a skill Go to the far right; the orange agent can only use primitive actions to reach the star.

Figure . :

 . Figure . : Illustration of the correlation between density and the fourth-nearest neighbor distance.Circles represent states and red dotted lines show the distance between a state and its fourth nearest neighbor.

  a representation that match the inherent structure of the environment [Tao et al., ]. It suggests that it is most of the time enough to learn a good representation. For instance, Guo et al. [ ] and Tao et al. [ ] compute a reward based on a learnt representation, but perhaps a bad representation tends to be located in low-density areas. It would result that active representation entropy maximization correlates with state-conditional entropy minimization.We are not aware of a lot of methods that actively learn a representation maximizing I(R; S). Yet, we stress out two methods that strive to actively learn a representation of states. In CRL[Du et al., ] and NOR[Nachum et al., a], the agent plays a minimax game. A module learns a representation function with a constrastive loss and the agent actively challenges the representation by looking for states with a large loss.

  . . SKILL LEARNING (a) Skills are not learnt yet. (b) The discriminator tries unsuccessfully to distinguish the skills. (c) Each skill learns to go in the area assigned to it by the discriminator. (d) Skills locally spread out by maximizing action entropy, as standardly done in DRL [Haarnoja et al., 8].

Figure . :

 . Figure . : The agent (circle) starts an episode in the center of the environment, colors denote the trajectories of their corresponding skills.

Figure . :

 . Figure . : Examples of state-goals selection strategies, extracted and adapted from Pitis et al. []. In the RIG strategy, the agent samples goals according to its current distribution of states; the DISCERN strategy tries to samples uniformly and the MEGA strategy prioritizes very low density states.

Figure . 6 :

 6 Figure .6: Illustration of the reweighting process. Left: probability of visited states to be selected as goals before density-reweighting. Right: probability of visited states to be selected as goals after density-reweighting. This figure simplifies the figure of Pong et al. [ ].

. 6 .

 6 Figure . : Illustration of the detachment issue. Image extracted from Ecoffet et al. [ ]. Green color represents intrinsically rewarding areas, white color represents no-rewards areas and purples areas are currently being explored. ( ) The agent starts to learn and has not explored the environment yet. ( ) It discovers the rewarding area at the left of its starting position and explores it. ( ) It consumed close intrinsic rewards on the left part, thus it prefers gathering the right-part intrinsic rewards. ( ) Due to catastrophic forgetting, it forgot how to reach the intrinsically rewarding area on the left.

Figure . 8 :

 8 Figure .8: Illustration of a three-levels HRL architecture where each DRL algorithm learns with intrinsic motivation. Each incoming state is built based on the original state and the dynamics of the lower-level policies.

Figure . 8

 8 Figure .8 displays a simplified example of such architecture. We saw in Section . that obtaining good skills essentially relies on the quality of the learnt representation which can depend on the potentially temporally-extended dynamics [Nachum et al., a, Aubret et al., , Li et al., b]. The representation can be learnt using the slowness principle [Wiskott and Sejnowski, ] which assumes temporally close states are similar. By configuring the time-extension of the representation, one may focus on different semantic parts of the state space. This can be illustrated in Aubret et al. [ ] (cf. Chapter ): -the agent can learn a very low level representation that provides skills that can manipulate torques of a creature; -skills can also orientate an agent in a maze assuming it accesses the maze representation. While they do not try to combine and learn several representations at the same time, further works could consider separate different parts of states (e.g. agent positions and object positions [Zhao et al., ]) or learning these representations at different time scales.

Figure . :

 . Figure . : Simple example of Bayesian network.

Figure . :

 . Figure . : Bayesian network which sums up the perception-action loop, extracted from Klyubin et al. [ a]. S are sensory inputs, R the true state of the environment, A ground actions on the agent, M the memory of the agent. We use a different notation in comparison with the rest of the dissertation to remain faithful to the original figure.

  Figure . illustrates a simple example of graphical model. Given the dependencies, a joint probability p(X, Y, C, Z) is conform to the graphical model if p(X, Y, C, Z) = p(X)p(Y |X)p(Z)p(C|Z, Y ). More generally, if

  . ) These models are convenient to model action-perception loops [Touchette and Lloyd, , Klyubin et al., a, Levine, 8] and allow to compute information theoretic measures. In this setting, parameters, actions, states, decisions etc. . . are all random variables. Figure . shows the graphical model induced by a typical perception-action loop which is unrolled through time. This kind of unrollment is typical of Dynamical Bayesian Network [Dagum et al., ]. While we could also use a Structured Graphical Model [Pearl,

Figure . :

 . Figure . : Bayesian network which sums up a simplified HRL-based cognitive model of an agent.o are observations, φ a distribution of parameters encoding a forward model, a ground actions of the agent, z representations of the state of the agent, g k the k-level decisions and φ k the k-level forward model. Black arrows represent inter-hierarchies interactions, yellow and blue components represent respectively the first decision level and the second decision level. We assume the model is consistent through time. In practice, all variables are dependent on the content of the memory and the decisions/actions depend on DRL parameters but we omit these here for clarity since it does not bring out more informations for our analysis.

  model the cognitive model of an agent with a Bayesian network [Pearl, ], as in Friston et al. [ ]. However, in contrast with them, we do not assume an agent is a generative model and only follows the internal causality of its processing. Thereby we do not explicitly model the hidden variables of the environment and rather consider an implicit and abstract understanding of the dynamics at several levels of representations. This directly follows the Bayesian brain hypothesis [Knill and Pouget, ] which assumes that the brain models uncertainty and performs Bayesian inference over variables. However, Figure . shows the Bayesian Network B of a decision-making step at the second level of a HRL framework, assuming the high-level decisions last for three timesteps. Of course, the framework can be made generic over a decision's level-dependent du-CHAPTER . SURVEY ON INTRINSIC MOTIVATION IN DRL ration l k .

Figure

Figure . :

 . Figure . : Illustration of the problem inherent to previous skill learning methods. The blue circle represents the starting position of the agent and it executes one skill par episode. ) the agent learns diverse skills, skill goes up and skill goes down; ) it executes its rewards skill, i.e skill ;) It forgets the difference between skills and .

(

  DQN) with Boltzmann exploration [Mnih et al., ] for discrete one.

  Figure . : The agent (circle) starts an episode in the center of the environment, colors denote the trajectories of their corresponding skills.

  (b) The agent separates the best skill, skill , into two new subskills ( , ) and ( , ) (c) The agent separates its new best skills into two new subskills ( , , ) and ( , , ) (d) The agent keeps expanding its tree of skill in the direction of the feedback from the environment.

Figure . :

 . Figure . : The yellow path represents the goal selected to be executed and splitted and r denotes the average extrinsic rewards gathered by a skill and . Here, |V | = 2.

  Figure . : Different states covered by the agent while doing a skill. The first and sixth columns display the skills learned with a message length equal to ; once the learning has been completed, the agent refines each skill into four new sub-skills, displayed on each row.

  Figure .6: Some skills learned by the agent in an environment composed of four rooms.

Figure . :

 . Figure . : Discriminator's probability of achieving skills (1), (3) and their sub-skills in every state (i.e. q(g|s)). The more red the state, the more rewarding it is for the skill. The left side corresponds to the preliminary stage of the learning process (timestep 128.10 4 ); the right side corresponds to the end of the learning process (timestep 640.10 4 ).

Figure . 8 :Figure . :

 8. Figure .8: One path in the tree of skills learned by the agent with an extrinsic reward of 1 on the upper right side of the wall.

Figure . respectively shows

  Figure . respectively shows the average reward per episode for different environments.

Figure . :

 . Figure . : Average reward per episode in HalfCheetah and HalfCheetah-Walk. We use our own implementation of SAC for HalfCheetah-Walk. The black curve is the average reward of a random tree-policy that uses the transfered skills.

Figure

  Figure . : Skill of Figure .6.

Figure . :

 . Figure . : Left: Illustration extracted from. "Circle Limit I" by M.C. Escher? It illustrates how the Poincare disc models the hyperbolic space. The area of each tile is constant in the hyperbolic space while the area gradually decreases as tiles get closer to the boundary in an euclidian space. Right: A tree output in an euclidian space modified. Source: https://bjlkeng.github.io/posts/

Figure . :

 . Figure . : Embedding of a set of skills in the Poincare disc.

  Figure . : Skills ( , , ) and ( , , ) and their parents. Extracted from Figure .6.

Figure . :

 . Figure . : Illustration of an embedding of a unsupervised contrastive loss. Extracted from https: //ai.googleblog.com/2021/06/extending-contrastive-learning-to.html.

Figure . :

 . Figure . : Illustration of the networks learnt while varying the input distribution. Extracted from Marsland et al. [ a].

Figure . :

 . Figure . : Illustration of a learning step of our growing network and contrastive loss (cf. text). Cylinders are buffers associated to each cluster, blue circles are states embedded with φ and pink circles represent clusters. The image is an example of state in the Visual Door [Nair et al., 8] environment.

  Figure . : Illustration of the selection of a skill and an interaction of the skill. See text for explanations.

  88CHAPTER . DISCOVERING A TOPOLOGICAL REPRESENTATION TO LEARN DIVERSE AND REWARDING SKILLS

  α ′ skew than in Section . . to trade-off the importance of the novelty reward in comparison with an extrinsic reward (see below) or decrease the speed at which we gather low-density states[Pong et al., The agent randomly samples a vector in the ball associated to the cluster.Uniform sampling in the ballState sampled (c) The agent selects the closest embedded state to the vector.

Figure . :

 . Figure . : Illustration of how an agent samples interactions from a buffer.

  state space. It is close to the stategy used by Pong et al. [ ].

Figure . 6 :

 6 Figure .6: Examples of 8 terminal states after the execution of 8 different skills learnt in Visual Pusher.

Figure . :Figure . 8 :

 .8 Figure . : Visualization of the representations learnt by a VAE and DisTopon the gridworld displayed at the far left. From left to right, we respectively see a-the rendering of the maze; b-the continuous representation learnt by DisTop with -dimensional binary inputs; c-a VAE representation with true (x,y) coordinates; d-a VAE representation with -dimensional binary inputs; e-OEGN network learnt from binary inputs.

Figure . 8 (

 8 Figure .8 (Right), we observe that DisTop obtains high extrinsic rewards and clearly outperforms ELSIM. It also outperforms SAC on Halfcheetah-v and is close to SAC on Ant-v . In contrast, we highlight that SAC overfits to dense rewards settings and cannot learn in sparse or no-reward settings (see below). Despite the genericity of DisTop and the narrowness of SAC, DisTop competes with SAC on two of SAC's favourite environments.

Figure . :

 . Figure . : Average rewards throughout training episodes. We use a plot for each comparison to avoid curve overlaps.

Figure . :

 . Figure . : Top-down view of Ant Maze environment and visualization of the topology that is progressively learnt by DisTop. The blue circle represents the extrinsically rewarding area. Small green points represent selected goal-states and the red circle highlights the area under exploitation.

Figure . :

 . Figure . : Different Topologies learnt on the gridworld displayed in Figure . . From left to right, we show the learnt topology with k c = 2, k c = 10, k c = 50.

Figure . :

 . Figure . : Different Topologies learnt on the gridworld displayed in Figure . . From left to right, we show the learnt topology with δ = 0.01, δ = 0.05, δ = 0.1, δ = 0.2.

  Figure . : Different Topologies learnt on the gridworld displayed in Figure . . From left to right, we show the learnt topology with k = 0.5, k = 1, k = 3, k = 10.

Figure . :

 . Figure . : Different OEGN networks learnt according to δ new . From left to right, we show the OEGN network with δ new = 0.2,δ new = 0.4,δ new = 0.6,δ new = 0.8,δ new = 1

Figure . :

 . Figure . : Different learning curves showing the impact of the choice of interactions. -We study the impact choosing learning interactions with π high rather than p α skew . -we study the importance of 1 + α ′ skew in Visual Pusher. and -we assess the importance of 1 + α skew in Visual Pusher and Visual Door.

  NOR [Nachum et al., a] bounds the sub-optimality of such representation to solve a task and LESSON[Li et al., b] uses a slow dynamic heuristic to learn the representation. In fact, it uses an InfoNCE-like objective function; this is similar to [Lu et al., ] which learns the representation during pre-training with random walks. Above-mentioned methods use a hierarchical random walk to explore the environment, we have shown in Section . that DisTop explores quicker by maximizing the entropy of states in its topological representation. DCO [Jinnai et al., ] generates options by approximating the second eigenfunction of the combinatorial graph Laplacian of the MDP. It extends previous works [Machado et al., , Bar et al., ] to continuous state spaces.

  [Pitis et al., , Pong et al., ], there is no pre-training, particular scheduling [Pong et al., ] or random walks [Lu et al., ]. It manages to select whether it wants to forget a skill, does not need a well-structured goal space like Pitis et al. [ ] or dense rewards as required by Li et al. [ b]. Our main take-away message is as follow: computing a discrete topology of the environment allows to control which skills to forget, improve or explore. With this control capacity, DisTop is generic enough to compete with SOTA algorithms on three very different reward settings and state spaces. Yet, there are limitations and exciting perspectives: HRL and planning based approaches [Nasiriany et al.,

  aspect, let us naturally ask the following question: what does multi-information teach us about the objective function of DisTop ? Let us discuss the three objectives derived from multi-information: Novelty could improve the exploration process by inciting to go into badly represented areas of the state space.

Table . :

 . Type of learning. feedback here refers to an expert supervision.

			With feedback	Without feedback
		Active	Reinforcement	Intrinsic motivation
		Passive	Supervised	Unsupervised
	an active process since the agent learns from its interactions with the environment, unlike classi-
	fication or regression which are supervised methods. Unsupervised learning is a passive learning
	process, i.e. it does not use predefined labels, or in other words, learns without a feedback. Finally,
	the substitution of the feedback by an intrinsic motivation allows to break free from an expert
	supervision; however, the difference remains between IM and unsupervised learning in the sense
	that IM is an active process which implies interactions. In the literature of machine learning, intrin-
	sic motivation-based learning may also be called self-supervised learning [Liu et al.,	, Pathak
	et al.,	].

Table . shows the difference between RL and the use of IM. RL is . . INTRINSIC MOTIVATION IN DRL

  CHAPTER . SURVEY ON INTRINSIC MOTIVATION IN DRLother side to prevent attraction towards stochastic areas. Indeed, if the area is deterministic, environment transitions are predictable and the uncertainty about its dynamics can decrease. At the opposite, if transitions are stochastic, the agent turns out to be unable to predict transitions and does not reduce uncertainty. The exploration strategy VIME[Houthooft et al., 6] computes this intrinsic reward by modelling p(φ|h) with Bayesian neural networks [Graves,

  [Hazan et al., ]. But computing ρ(s) is challenging in high-dimensional state spaces. Several methods propose to estimate ρ(s) using variational inference[Zhang et al.,

, Islam et al., , Lee et al., , Pong et al., ] based on autoencoder architectures. In this setting, we can use either Equation . 6 [Lee et al., ] or Equation . [Pong et al., ], assuming z is a compressed latent variable, p(z) a prior distribution [Kingma and Welling,

  Instead of relying on euclidian distance, one can try to learn a similarity function. EX 2[Fu et al., ] learns a discriminator to differentiate states from each other: when the discriminator does not manage to differentiate the current state from those in the buffer, it means that the agent has not visited this state enough and it will be rewarded. States are sampled from a buffer, implying the necessity to have a large buffer. To avoid this, some methods distill recent states in a prior distribution of latent variables[Kim et al., b, Klissarov et al.,

	. . NOVELTY MAXIMIZATION
	consecutive states [Tao et al.,	, Yarats et al.,	].
	Identifying different states.
	[Srinivas et al.,	] and K denotes the number of k-nn estimations. By looking for distant state
	embeddings during an unsupervised pre-training phase, they manage to considerably speed up

task-learning in the DeepMind Control Suite. The representation g can also derive from a random encoder

[Liu and Abbeel, ]

or a constrastive loss that ensures the euclidian proximity between

  Hetereogenous methods address the problem of overfitting of the discriminator. The naive way can be to regularize the learning process of the discriminator. ELSIM (Chapter ) takes advantages of L regularization and progressively expand the goal space G to cover larger areas of the state space and Choi et al.

restricted part of the state space [Choi et al., ]. Accessing the whole set of true possible states and deriving the set of goals by encoding states can considerably improve the coverage of skills [Campos et al., ]. Approaches for a better coverage of states. [ ] propose to use spectral normalization [Miyato et al.,

  6)CHAPTER . SURVEY ON INTRINSIC MOTIVATION IN DRLSimilarly to Equation . , this amounts to maximize -H(S g |S). RIG[Nair et al.,[START_REF] Bengio | Curriculum learning[END_REF] proposes to build the feature space independently with a variational auto-encoder (VAE); but this approach can be very sensitive to distractors (i.e. useless features for the task or goal, inside states) and does not allow to correctly weight features. Similar approaches also encode part of trajecto-

	ries [Kim et al.,	, Co-Reyes et al.,	8] for similar mutual information objectives. SFA-GWR-
	HRL [Zhou et al.,	] uses unsupervised methods like the algorithms of slow features analysis
	[Wiskott and Sejnowski,

  Skew-fit[Pong et al., ] provides more flexibility on how uniform one want its distribution of states. Skew-fit extends RIG and learns a parameterized generative model q φ (S) ≈ p(S) and skews the generative model (VAE) with the ratio:q φ (s) α skew . ( . )where α skew < 0 determines the speed of uniformisation. This way it gives more importance to low-density states. Then it weights all visited states according to the density approximated by the generative model at the beginning of each epoch, which is made of a predefined number of timesteps. Skew-fit manages to explore image-based environments very efficiently. As shown in[Aubret et al., ] (cf. Chapter ), this ratio applied on a discrete number of skills, amount to rewards a Boltzmann goal-selection policy with: R(s g ) = (1 + α skew ) log p(s g ).

		( . 8)
	Density reweighting by partitioning the embedding space. With a different objective, GRIM-
	GREP [Kovač et al.,	] partitions the VAE embedding of Skew-fit with a Gaussian Mixture Model
	[Rasmussen et al.,

according to a kernel density estimation or use the rank of state-densities [Zhao and Tresp, ] estimated with a Variational Gaussian Mixture Model [Blei

and Jordan, 6]

. In contrast with them,

Table . :

 . Summary of environments and the size of action and state spaces.CHAPTER . END-TO-END LEARNING OF REUSABLE SKILLS THROUGH INTRINSIC MOTIVATION

		HalfCheetah				HalfCheetah-Walk
		ELSIM with transfer
		SAC
		Random with transfer
	average reward	4 8
		0
		0.0	0.2	0.4	0.6	0.8	0.0	0.2	0.4	0.6	0.8
				million steps				million steps
								State space	Action space
			HalfCheetah						6
		MountainCarContinous
				Pendulum
			LunarLander				8

  The algorithm works as follows: assuming a new low-density embedded state is discovered (rare state located at the border of a ball), there are two possibilities: -the balls currently overlap: OEGN progressively moves the clusters closer to the new state and reduces overlapping; -the clusters almost do not overlap and OEGN creates a new cluster next to the embedded state. In the case of a high-density embedded state, it does not much modify the spread of balls. A learnt discrete topology can be visualized at the far right of Figure.. Sample an input e i ← sample(B).closest ← min c∈C (||ce i || 2 ).

	Algorithm Algorithm of OEGN (red) and GWQ (blue)
	Initialize network with two random nodes, set theirs attributes to and connect them.
	for each learning iteration do Sample a tuple s i , c i , s prev i Embed states: e i ← φ ω ′ (s i ); e prev ← sample(B). i ← φ ω ′ (s prev i	)
	Increase error count of c i by . Reset error count of closest to
	Apply DeleteOperator().
	If a node is deleted, stop the learning iteration
	Apply CreationOperator().
	Apply M ovingOperator().
	Apply EdgeOperator().

  ).Skew-Fit [Pong et al., ] further extended RIG to improve the diversity of learnt skills by making the VAE over-weight low-density states. Unlike DisTop, it is unclear how Skew-Fit could target another distribution over states than a uniform one. Approaches based on learning progress (LP) have already been built over VAEs[Kovač et al., ,[START_REF] Bengio | Curriculum learning[END_REF]; we believe that DisTop could make use of LP to avoid distractors or further improve skill selection.

  Currently, there are clearly two distinct approaches, one based on VAE [Pong et al., ] and the other based on temporal (and marginally data-augmentation based for images) contrastive losses (cf. Chapter ). In the first set of methods, the disentangled representation opens a large set of possibilities for affordance CHAPTER 6. CONCLUSION learning [Khazatsky et al., ], while contrastive losses capture the temporal proximity whatever the structure of the ground state space is. We perceive two challenges for active contrastive representation learning: Clarifying actual losses. Contrastive losses are the subject of intensive research [Chen et al., , Srinivas et al., , Schwarzer et al., ]. In particular, several works (including DisTop) Their specific properties are currently unclear since each representation is sometimes learnt in a different context for a different purpose. The same mutual information objective can lead to a spherically distributed representation [Ermolov and Sebe, ], a distorted one Li et al. [ b], an undistorted one (like DisTop, cf. Chapter ), action-dependent ones [Nachum et al.,

	introduce temporally contrastive losses with different structures [Li et al.,	b, Oord et al.,	8,
	Warde-Farley et al.,	, Nachum et al.,	a], [Shu et al.,	, Ermolov and Sebe,	, Tao
	et al.,	, Yarats et al.,	] and non-contrastive losses also show similar properties [Choi et al.,
	, Zhang et al.,	a].

, .

Acknowledgements

I would first like to thank my supervisors, Laëtitia and Salima. Your empathy and kindness made these years enjoyable to me and I appreciated your caring and constructive remarks. But

Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, .

Paul Dagum, Adam Galper, and Eric Horvitz. Dynamic network models for forecasting. In Uncertainty in artificial intelligence, pages -8. Elsevier, .

P Dayan and GE Hinton. Feudal reinforcement learning. nips' (pp.

-8), .

Peter Dayan and Geoffrey E Hinton. Varieties of helmholtz machine. Neural Networks, (8): 8 -,

Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine.

Neural computation, ( ):88 -, .

Edward Yilun Du, Chuang Gan, and Phillip Isola. Curious representation learning for embodied intelligence.

arXiv preprint arXiv: . 6 , .

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl$ˆ $:

Fast reinforcement learning via slow reinforcement learning. CoRR, abs/ 6 . , 6.

Gabriel

-, .

Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Adaptive skills adaptive partitions (asap).

In Advances in Neural Information Processing Systems, pages 88-6, 6.

Stephen Marsland, Jonathan Shapiro, and Ulrich Nehmzow. A self-organising network that grows when required. Neural Networks, (8-): -8, a.

Stephen Marsland, Jonathan Shapiro, and Ulrich Nehmzow. A self-organising network that grows when required. . Illustration of the difficulty to differentiate optimal actions from sub-optimal ones. The black line represents the trajectory of an orange agent that looks for the star. . .6 Visualization of trajectories of agents that start at the white circle and try to reach the star.The blue agent uses only low-level actions while the orange can set goals.

List of abbreviations

. Illustration of the sparse reward issue in a very simple setting. The agent, represented by a circle, strives to reach the star. The reward function is one when the agent reaches the star and zero otherwise. On the left side, the agent explores with standard methods such as ǫ-greedy; as a result, it stays in its surrounded area because of the temporal inconsistency of its behaviour. On the right side, we can imagine an ideal exploration strategie where the agent covers the whole state space to discover where rewards are located. .

LIST OF FIGURES

Appendix A

Complements on ELSIM experiments A. Tree-policy algorithm

Algorithm shows how ELSIM runs an episode in the environment without the learning part of the intra-skill policy and the discriminator. There are steps: -an agent runs an episode inside the MDP of skills; the sequence of actions represents a skill; -the agent executes the intra-skill policy of the skill; -the tree-policy is rewarded according to how well the intra-skill policy fits the task and the Q-learning applies.

Learning algorithm

Algorithm shows one learning step in ELSIM for discriminators and intra-skill policies. The agent executes its tree-policy on its tree; the discriminators of the intermediary encountered nodes learn with probability η on a batch of interactions of their recursively and uniformly chosen children leaves. The last discriminator learns with probability 1 over a batch of its leaves' interactions. This batch is labeled with the intrinsic reward computed through Equation . and leaves' intra-skill policies learn with this batch.

Algorithm

A.

Hyper-parameters on gridworlds

A.

Hyper-parameters on continuous environments.

Since the goal of ELSIM is to learn in a continual learning setting, we record the performances of ELSIM in training mode, i.e. with its stochastic policies. Transfer learning . In experiments on transfer learning, we fix the number of parallel environments to and reduce the length of an episode to . For the first updates of a node, the tree-policy remains uniform.

A. Skills learned in four rooms environment

A.6 Skills learned with a vertical wall

A. Skill expansion

A.8 HalfCheetah-Walk

We introduce a slight modification of HalfCheetah-v to make the creature walk. The reward function used in HalfCheetah-v :

where C(a) is the cost for making moves and Sf (s, s ′ ) is the speed of the agent. The reward function used in HalfCheetah-Walk is:

Complements on DisTop experiments B. Ablation study

Except for δ new and δ success , we emphasize that thresholds parameters have not been fine-tuned and are common to all experiments; we give them in Table B. .

B. Comparison methods

LESSON: we used the code provided by the authors and reproduced some of their results in dense rewards settings. Since the environments are similar, we used the same hyper-parameter as in the original paper Li et al. [ b].

Skew-Fit: since we use the same evaluation protocol, we directly used the results of the paper.

In order to fairly compare DisTop to Skew-Fit, the state given to the DRL policy of DisTop is also the embedding of the true image. We do not do this in other environments. We also use the exact same convolutional neural network (CNN) architecture for weights ω as in the original paper of Skew-Fit. x ; number of channels: 6, , and 6 ; strides: , and . Finally, there is a last linear layer with neurons that corresponds to the topology dimensions d. This latent dimension is different from the ones of Skew-Fit, but this is not tuned and set to .

ELSIM:

In contrast with our first experiments, we set the batch size to 256 and use neural networks with 2×256 hidden layers. The weight decay of the discriminator is set to 1•10 -4 in the maze environment and 1 • 10 -3 in Ant and Half-Cheetah. In Ant and Half-Cheetah, the learning process was too slow since the agent sequentially runs up to neural networks to compute the intrinsic reward; so we divided the number of updates by two. In our results, it did not bring significant changes.

SAC: we made our own implementation of SAC. We made a hyper-parameter search on entropy scale, batch size and neural networks structure. Our results are consistent with the results from the original paper Haarnoja et al. [START_REF] Bengio | Curriculum learning[END_REF].

B. Hyper-parameters

Table B. shows the hyper-parameters used in our main experiments. We emphasize that tasks are very heterogeneous and we did not try to homogenize hyper-parameters across environments.

B. Environment details B. . Robotic environments

B. . Maze environments

These environments are described in Nachum et al.

[ a] and we used the code modified by Li et al. [ b]. For convenience, we provide again some details and explain our sparse version. The environment is composed of 8x8x8 fixed blocks that confine the agent in a U-shaped corridor displayed in Figure . . Similarly to Li et al. [ b], we zero-out the (x,y) coordinates and append a low-resolution top view of the maze to the proprioceptive state. This "image" is a -dimensional vector. In our sparse version, the agent gets reward when its distance to the target position is below . and getsreward otherwise. The fixed goal is set at the top-left part of the maze.

Sparse Point Maze: the proprioceptive state is composed of dimensions and its -dimensional action space ranges in the intervals [-, ] for forward/backward movements and [-. , . ] for rotation movements.

Sparse Ant Maze: the proprioceptive state is composed of dimensions and its 8-dimension action space ranges in the intervals [-6, 6].

B. Computational resources

Each simulation runs on one GPU during to hours according to the environment. Here are the settings we used:

• Nvidia Tesla K8 , CPU cores from of a Xeon E -6 v , G of RAM.

• Nvidia Tesla V , CPU cores from a Xeon Silver , G of RAM.

• Nvidia Tesla V SXM , CPU cores from a Intel Cascade Lake 6 6 processors, 8G of RAM. (Least used).

B.6 Examples of skills