Léonard Hussenot

Robert Dadashi

Matthieu Geist

Olivier Pietquin

Show me the Way: Intrinsic Motivation from Demonstrations

The study of exploration in the domain of decision making has a long history but remains actively debated. From the vast literature that addressed this topic for decades under various points of view (e.g., developmental psychology, experimental design, artificial intelligence), intrinsic motivation emerged as a concept that can practically be transferred to artificial agents. Especially, in the recent field of Deep Reinforcement Learning (RL), agents implement such a concept (mainly using a novelty argument) in the shape of an exploration bonus, added to the task reward, that encourages visiting the whole environment. This approach is supported by the large amount of theory on RL for which convergence to optimality assumes exhaustive exploration. Yet, Human Beings and mammals do not exhaustively explore the world and their motivation is not only based on novelty but also on various other factors (e.g., curiosity, fun, style, pleasure, safety, competition, etc.). They optimize for life-long learning and train to learn transferable skills in playgrounds without obvious goals. They also apply innate or learned priors to save time and stay safe. For these reasons, we propose to learn an exploration bonus from demonstrations that could transfer these motivations to an artificial agent with little assumptions about their rationale. Using an inverse RL approach, we show that complex exploration behaviors, reflecting different motivations, can be learnt and efficiently used by RL agents to solve tasks for which exhaustive exploration is prohibitive.

INTRODUCTION

Intrinsic motivation [START_REF] Edward | Intrinsic motivation. The corsini encyclopedia of psychology[END_REF] has emerged as one explanation for humans' and animals' impressive learning capabilities. Steered by the need to explore their environment (whether this need is satiable or not has been a fierce debate in the behavioral psychological community [START_REF] Hunt | Intrinsic motivation and its role in psychological development[END_REF]), they are able to discover near-optimal strategies in very efficient ways. Designing artificial agents presenting such capabilities is a central goal of modern artificial intelligence and Reinforcement Learning (RL) is a popular candidate to do so. RL has addressed a variety of sequential-decision-making problems whether in games [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF][START_REF] Tesauro | Temporal difference learning and TD-Gammon[END_REF] or robotics [START_REF] Abbeel | Apprenticeship learning via inverse reinforcement learning[END_REF][START_REF] Openai ; Marcin Andrychowicz | Learning dexterous in-hand manipulation[END_REF]. Nevertheless, some simple problems remain unsolved. Current state-of-the-art methods struggle to find good policies in environments [START_REF] Abbeel | Apprenticeship learning via inverse reinforcement learning[END_REF] where constant negative rewards may discourage the agent to explore (e.g., the Pitfall! game from Atari), [START_REF] Openai ; Marcin Andrychowicz | Learning dexterous in-hand manipulation[END_REF] where the reward is so sparse that an agent does not find any (e.g., the Montezuma's Revenge Atari game), [START_REF] Bagnell | Boosting structured prediction for imitation learning[END_REF] where state and action space are big (e.g., text worlds). These tasks remain fairly easy for humans, though. In order to tackle these specific problems, the use of reward bonuses, inspired by animal curiosity, was proposed to steer the agent's exploration [START_REF] Şimşek | An intrinsic reward mechanism for efficient exploration[END_REF][START_REF] Alexander | An analysis of model-based interval estimation for Markov decision processes[END_REF]. Even though different intrinsic bonuses have been proposed, a large majority rely on the same principle: reward for novelty. These methods mostly differ in how they compute this notion of newness. Count-based methods do it by counting how often the agent has encountered a given state [START_REF] Alexander | An analysis of model-based interval estimation for Markov decision processes[END_REF]. Pseudo-counts methods [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Ostrovski | Count-based exploration with neural density models[END_REF] allow to approximate counts in large state spaces. Prediction error is also used to measure novelty, either by computing the agent's ability to predict the future [START_REF] Pathak | Curiosity-driven exploration by self-supervised prediction[END_REF] or random statistics about the current state [START_REF] Burda | Exploration by random network distillation[END_REF]. Some restrict novelty to state-action pairs that have an impact on the agent [START_REF] Raileanu | RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments[END_REF] or derive empowerment metrics [START_REF] Mohamed | Variational information maximisation for intrinsically motivated reinforcement learning[END_REF] using mutual information. All these methods naturally encourage the discovery of new states through exhaustive exploration. Yet, in most realistic environments, exhaustive exploration is (1) not feasible due to the size of the state-action space, (2) not desirable as most behaviors are unlikely to be relevant for the task at hand.

Nonetheless, human and more generally mammals exploration behaviors are governed by various motivations and constraints. Intelligent Beings do not have unlimited resources of time and energy. They optimize these resources to survive and reproduce but also to have fun [START_REF] Sarah | Children's geographies: Playing, living, learning[END_REF], to help others [START_REF] Richard | Machiavellian intelligence: social expertise and the evolution of intellect in monkeys, apes, and humans[END_REF] or to satisfy their curiosity. Oudeyer and Kaplan [START_REF] Oudeyer | What is intrinsic motivation? A typology of computational approaches[END_REF] make the difference between homeostatic motivations (that encourage to stay in the "comfort zone" and generally correspond to desires that can be satiated) and heterostatic motivations (that push organisms out of equilibrium but cannot be satiated). These many desires shape the way organisms interact with their environment, encouraging them to discover new things but also to protect themselves, avoiding over-surprising events with mechanisms like fear [START_REF] Peter | Fear and anxiety: animal models and human cognitive psychophysiology[END_REF]. Berseth et al. [START_REF] Berseth | SMiRL: Surprise Minimizing RL in Dynamic Environments[END_REF] exemplified how to exploit such priors by implementing a "homeostasis" objective for RL, thereby showing how different from "novelty seeking" these priors can be. Eventually, the resource constraints stop organisms from exploring exhaustively their environment and push them to transfer knowledge from past experience. Hunt [START_REF] Hunt | Intrinsic motivation and its role in psychological development[END_REF] developed the idea of optimal incongruity: high-novelty is not rewarded as much as intermediate-level novelty, suggesting how curiosity is tightly connected to fear. More recently, Kidd et al. [START_REF] Kidd | The Goldilocks effect: Human infants allocate attention to visual sequences that are neither too simple nor too complex[END_REF] supported this hypothesis with experiments on children curiosity. Overall, novelty methods fail to model correctly human curiosity as they consider that "the newer, the better". This failure calls for a new way of definingour agent's intrinsic motivations.

In an arbitrary environment, exhaustive exploration is desirable and leads to convergence with theoretical guarantees [START_REF] Alexander L Strehl | Reinforcement learning in finite MDPs: PAC analysis[END_REF]. But when the exploration presents some structure, one can transfer skills and priors from similar environments. Dubey et al. [START_REF] Dubey | Investigating human priors for playing video games[END_REF] exemplified, in the case of simple video games, how humans priors help us to solve new problems. The authors enlighten how humans struggle to play the same underlying video game with change of the object semantics, physics modifications (e.g., the gravity is rotated) or with visual similarities transformations. Overall, they show how much of the human's ability to solve a new game in a zero-shot manner is due to their prior on the environment.

In this paper, instead of hard coding what we think an agent's motivations should be (e.g. novelty), we propose to learn a bonus that captures these sources of motivation from demonstrations. By adopting this approach, we expect to learn a bonus that implicitly helps reproducing a structured exploration behavior (i.e. using priors from the demonstrations to reduce the search space), in lieu of an exhaustive one. We also argue that, to a certain extent at least, this can happen without the need of extra modelling inspired by cognitive or behavioral research. To do so, we cast this problem as an inverse RL problem with the difference that only some fraction of the reward optimized by an observed agent is hidden: the intrinsic motivation bonus. The task-related reward remains provided by the environment. We then build upon Klein et al. [START_REF] Klein | A cascaded supervised learning approach to inverse reinforcement learning[END_REF] to propose a method that allows us to recover the intrinsic motivation from demonstrations.

Therefore, our contributions are the following:

(1) a modelling that allows for disentangling the reward optimized by a demonstrator from its intrinsic motivation bonus; (2) an architecture, that we call "Show me the Way" (SmtW), based on a cascade of supervised learning methods that extracts that exploration bonus from demonstrations; (3) an empirical evaluation showing that SmtW is able to capture different exploration priors explained by various types of motivations.

To evaluate SmtW, we validate a set of hypotheses on a controlled environment. We notably find that our method can learn structures and styles, transfer useful priors and encourages long-term planning.

BACKGROUND

Markov Decision Processes. In Reinforcement Learning (RL), an agent learns to behave optimally through interactions with an environment. This is usually formalized as a Markov Decision Process (MDP) [START_REF] Martin L Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF][START_REF] Richard | Reinforcement learning: An introduction[END_REF], a tuple (S, A, P, R, 𝛾) with S the set of states, A the set of actions (assumed discrete here), P : S × A → P(S) the Markovian transition kernel defining the dynamic of the environment, R : S × A → R a bounded reward function and 𝛾 ∈ [0, An optimal policy 𝜋 * satisfies component-wise 𝑄 𝜋 * ≥ 𝑄 𝜋 , for any policy 𝜋. Let 𝑄 * = 𝑄 𝜋 * be the associated (unique) optimal 𝑄-function, any deterministic optimal policy is greedy with respect to it: 𝜋 * (𝑠) ∈ argmax 𝑎 𝑄 * (𝑠, 𝑎). Exploration Bonus. A common strategy to encourage exploration is to augment the reward function with a bonus. This bonus generally depends on past history. For example, a bonus rewarding novelty requires remembering what has been experienced so far. Write ℎ 𝑡 = (𝑠 0 , 𝑎 0 , . . . , 𝑠 𝑡 -1 , 𝑎 𝑡 -1 , 𝑠 𝑡 ) the history up to time 𝑡, and H the set of all histories. Generally speaking, we abstract a bonus as B : H × A → R, and use it for addressing the dilemma between exploration and exploitation, which thus amounts for the agent to optimize for 𝑅(𝑠 𝑡 , 𝑎 𝑡 ) + 𝐵(ℎ 𝑡 , 𝑎 𝑡 ) instead of simply 𝑅(𝑠 𝑡 , 𝑎 𝑡 ). This state-of-the-art exploration bonuses all rely on memory, e.g. by counting the state visitation [START_REF] Alexander | An analysis of model-based interval estimation for Markov decision processes[END_REF] or through updates of a neural networks [START_REF] Burda | Exploration by random network distillation[END_REF][START_REF] Ostrovski | Count-based exploration with neural density models[END_REF]. These exploration bonuses are designed to express the prior that any source of novelty is good for exploration.

Such a prior on what is good for exploration is task-specific. Taıga et al. [START_REF] Ali Taıga | ON BONUS-BASED EXPLORATION METHODS IN THE AR-CADE LEARNING ENVIRONMENT[END_REF] showed that state-of-the-art bonuses were degrading performances in most Atari games.

SHOW ME THE WAY

Rather than handcrafting a bonus that encodes what we think intrinsic motivation should be (e.g. using novelty), we propose to learn it from demonstrations of exploratory behaviours.

We thus assume that the demonstrator learns to solve a task by exploring its environment and a simple solution would be to perform behavior cloning. Because the demonstrator is likely to use past interactions to make decisions (remembering what has been already tried so far), we could frame our problem as learning, in a supervised manner, a mapping from histories to actions. Yet, behavioral cloning suffers the behavioral drift [START_REF] Ross | Efficient reductions for imitation learning[END_REF], which would be exacerbated in the case of history dependent policies . Moreover, we would like to transfer this behavior to new environments and possibly to new tasks.

Imitation learning classically assumes that experts are optimizing an MDP with an unknown reward function R 𝐸 (𝑠, 𝑎). Note that this introduces a modelling bias, i.e. a human performing a task is not necessarily explicitly solving an MDP. In this study, we do not tackle the standard problem of inferring an optimal behavior from demonstrations, but of estimating an exploratory behavior. Yet, the latter can be reduced to the former. Taking inspiration from RL, and especially from the body of work about the exploration-exploitation dilemma, we assume that our demonstrator is optimizing for an unknown reward function R 𝐸 (𝑠, 𝑎), standing for the task objective, augmented with a trajectory-dependent intrinsic bonus B 𝐸 (ℎ, 𝑎), standing for how the environment is explored. Making this bonus depend on past interactions is important as we can reasonably assume that exploration is based on memory (one would not try to always reproduce situations that were already seen). Then, we assume that the expert is optimal for the bonus-augmented reward R 𝐸 (𝑠, 𝑎) + B 𝐸 (ℎ, 𝑎), in the augmented MDP {H, 𝐴, 𝑃, R 𝐸 + B 𝐸 , 𝛾 }. That is the original MDP, with the state space being replaced by the set of all trajectories and the reward function being augmented with a bonus function. So, we reduced our original problem to Inverse Reinforcement Learning (IRL): learn a function 𝑅𝐵 : H × A → R such that the demonstrator is the (unique) optimal policy. By design, 𝑅𝐵 = R 𝐸 + B 𝐸 is a solution to this problem (even if it is not learnable exactly, as the optimal policy is invariant to many reward transformations [START_REF] Andrew Y Ng | Policy invariance under reward transformations: Theory and application to reward shaping[END_REF]). Yet, we also assume that we know the task's reward 𝑅, or at least that we observe it in the demonstrations. Formally, it may be different from the reward 𝑅 𝐸 (even if we assume that it leads to a similar optimal behavior), but we can leverage it to disentangle the task contribution and the exploration one. For doing so, we propose to learn a bonus function B : H × 𝐴 → R such that the demonstrator is optimal for 𝑅 + B. Notice that it does not change the problem, as it is just a reparameterization of the previous one (by setting B = 𝑅𝐵 -R).

Thus, our IRL problem has additional constraints. First, we want to recover the bonus, for transferring to new environments or new tasks, this preclude using imitation learning methods that do not explicitly recover rewards (IRL is mandatory). Second, our specific parameterization (R + B) precludes using IRL methods that would not allow using the observation of the task reward R along the expert trajectories. Third, the function we want to estimate is history-dependent which requires an IRL method able to use sequences as inputs.

Formalism. We assume to have access to demonstrations that are optimal according to the (known) reward of the environment plus an (unknown) intrinsic bonus. The environment being assumed Markovian, knowing the current state is enough to act optimally according to the task (optimizing for the environment's reward). Yet, the demonstrator also optimizes its exploration bonus, that depends on the past. To formalize things, we consider that the demonstrations are provided by a policy 𝜋 𝐸 : H → A, and that the policy is optimal for the augmented MDP (H, A, P, R 𝐸 + B 𝐸 ), where H replaces S and R 𝐸 + B 𝐸 replaces R.

We frame our problem as learning the bonus B 𝐸 from trajectories sampled from 𝜋 𝐸 .

Our approach. If we cannot naively apply any existing IRL algorithm to our problem, it can be a source of inspiration. Especially, one suits well our problem: the set-policy framework [START_REF] Piot | Bridging the gap between imitation learning and inverse reinforcement learning[END_REF]. It shows that a formal bijection between supervised learning and IRL exists. Among the covered algorithms, the Cascaded Supervised approach to IRL (CSI) [START_REF] Klein | A cascaded supervised learning approach to inverse reinforcement learning[END_REF] is of particular interest to us. We refer the reader to these papers for more details and we explain in details here how the CSI paradigm can be readily applied to our setting. The demonstrator's policy, 𝜋 𝐸 , is assumed optimal for R 𝐸 + B 𝐸 (which is unknown), so

𝜋 𝐸 = 𝜋 * R 𝐸 + B 𝐸 . Write 𝑄 𝐸 (ℎ, 𝑎) = 𝑄 * R 𝐸 + B 𝐸
𝑄 𝐸 (ℎ, 𝑎) = R 𝐸 (𝑠, 𝑎) + B 𝐸 (ℎ, 𝑎) + 𝛾E 𝑠 ′ |𝑠,𝑎 [max 𝑎 ′ 𝑄 𝐸 (ℎ ′ , 𝑎 ′ )] = R 𝐸 (𝑠, 𝑎) + B 𝐸 (ℎ, 𝑎) + 𝛾E 𝑠 ′ |𝑠,𝑎 [𝑄 𝐸 (ℎ ′ , 𝜋 𝐸 (ℎ ′ ))].
Would the optimal policy and 𝑄-function be known, we could use them to recover the optimized bonus-augmented reward using this Bellman equation:

R 𝐸 (𝑠, 𝑎) + B 𝐸 (ℎ, 𝑎) = 𝑄 𝐸 (ℎ, 𝑎) -𝛾E 𝑠 ′ |𝑠,𝑎 [𝑄 𝐸 (ℎ ′ , 𝜋 𝐸 (ℎ ′ ))].
Now, the quantities in the right hand side are unknown, but they can be estimated in an indirect way.

Assuming that the actions are discrete, we can learn the policy 𝜋 𝐸 by mapping histories to actions (using, for example, an LSTM network). Write π : H → S the resulting policy, or classifier. If we train it by minimizing a cross-entropy loss, what we learn indeed are logits Q (ℎ, 𝑎), and the classifier is π (ℎ) = argmax 𝑎 Q (ℎ, 𝑎). Said otherwise, π is greedy with respect to Q, that can thus be interpreted as an optimal Q-function for an unknown reward. Using the Bellman equation, we can recover this reward: R(𝑠, 𝑎)

+ B(ℎ, 𝑎) = Q (ℎ, 𝑎) -𝛾E 𝑠 ′ |𝑠,𝑎 [ Q (ℎ ′ , π (ℎ ′ ))]. (1)
By Bellman, as π is greedy w.r.t. Q, we have that π and Q are respectively the optimal policy and 𝑄-function for the reward R + B. We cannot use directly Eq. ( 1), the model being unknown, but we can sample the right hand side and estimate B by solving a regression problem. Therefore, we have reduced our initial problem to a sequence of supervised learning problem. Our algorithm is indeed CSI, up to the fact that we consider trajectories instead of states, and parameterize the bonus with the reward task. As such, the theoretical results of Klein et al. [START_REF] Klein | A cascaded supervised learning approach to inverse reinforcement learning[END_REF] applies to our setting. Notably, we would have that

0 ≤ E ℎ∼𝜋 𝐸 [max 𝑎 𝑄 * 𝑅+ B (ℎ, 𝑎) -𝑄 𝜋 𝐸 𝑅+ B (ℎ, 𝜋 𝐸 (ℎ)] ≤ O 𝜖 1 + 𝜖 2 1 -𝛾 ,
with 𝜖 1 the classification error and 𝜖 2 the regression error. This means that the demonstrator policy is close to optimal if these errors are small, for the learnt bonus function. One could argue that this bound trivially holds for 𝑅 + B = 0, when all behaviors are optimal. Yet, this is unlikely, as for having the classification error 𝜖 1 small, we must have Q (ℎ, 𝜋 𝐸 (ℎ)) > Q (ℎ, 𝑎 ≠ 𝜋 𝐸 (ℎ)) w.h.p., and thus learn an informative bonus.

Figure 1: Trajectories (𝑠, 𝑎, . . . ) are generated by a demonstrator exploring its environment. In order to recover a bonus that can explain its behavior, a BC policy parameterized with an LSTM is trained to predict the actions of the demonstrator from its trajectories of states, by minimizing L BC . The policy's logits 𝑄 𝜙 are interpreted as optimal 𝑄-values and used to compute a regression target. A bonus function B 𝜃 , parameterized with an LSTM, is then trained to predict it, by minimizing L reg .

Implementation. More concretely, we consider softmax(𝑄 𝜙 ) to be a neural network classifier with LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] units, 𝜙 being the set of parameters and 𝑄 𝜙 being the logits. We train 𝜋 𝜙 to do behavioral cloning, that is to predict the demonstrator actions 𝑎 𝐸 based on its past interactions ℎ 𝐸 , by minimizing a cross-entropy loss:

L BC = -ln(softmax(𝑄 𝜙 (ℎ 𝐸 , 𝑎 𝐸 )),
with 𝑄 𝜙 (ℎ, 𝑎) the 𝑎 th logit for input ℎ. If the classifier learns correctly, the logits of the resulting network should satisfy 𝑄 𝜙 (ℎ 𝐸 , 𝑎 𝐸 ) > 𝑄 𝜙 (ℎ 𝐸 , 𝑎) for 𝑎 ≠ 𝑎 𝐸 , and the class predicted by the classifier will be 𝜋 𝜙 (ℎ) = argmax 𝑎 𝑄 𝜙 (ℎ, 𝑎). Hence, as explained above, one can interpret 𝑄 𝜙 as an optimal 𝑄-function (hence the notation), and 𝜋 𝜙 as the associated optimal policy. Recall that these quantities can be related to the bonus-augmented reward through Bellman:

𝑄 𝜙 (ℎ, 𝑎) = R(𝑠, 𝑎) + B(ℎ, 𝑎) + E 𝑠 ′ |𝑠,𝑎 [𝑄 𝜙 (ℎ ′ , 𝜋 𝜙 (ℎ ′ ))].
L reg = 𝑄 𝜙 (ℎ 𝐸 , 𝑎 𝐸 ) -𝛾𝑄 𝜙 (ℎ 𝐸 ′ , 𝜋 𝜙 (ℎ 𝐸 ′ )) -𝑅(𝑠 𝐸 , 𝑎 𝐸 ) -B 𝜃 (ℎ 𝐸 , 𝑎 𝐸 ) 2 + B min -B 𝜃 (ℎ 𝐸 , 𝑎 𝐸 ) 2 .
To sum up, we train a BC policy by minimizing L BC . The implicit resulting logits are considered optimal 𝑄-values, that are in turn used to learn the bonus 𝐵 𝜃 by minimizing the loss L reg (Figure 1).

EXPERIMENTS

We aim at providing insights on what priors SmtW is able to extract from the demonstrations and specifically, we wish to verify that SmtW is able to encourage a structured exploration of the environment. In order to thoroughly study the method, we test it on a grid-world where we are able to design controllers with specific behaviors. As in IRL, studying the return of an agent trained with our bonus is only a proxy to evaluate SmtW's quality and is not informative on the priors the bonus conveys. We thus focus our experiments on analyzing the priors that were extracted from the demonstrations by the method. More specifically we wish to answer the following questions: (1) Is SmtW encouraging the demonstrator's behavior more than a random one? (2) Is SmtW capturing the demonstrator's style, its way of exploring the environment? (3) Does SmtW capture the skills required to solve the task? (4) Does SmtW encourage novelty seeking? (5) Does SmtW capture the constraints the demonstrator may be submitted to? To do so, we design controlled behaviors and study the bonus returned along these specific behaviors by SmtW, as described in Fig. 2. Given a behavior 𝐴 and a behavior 𝐵, this allows to check if a given bonus encourages behavior 𝐴 over 𝐵 or vice versa or rewards them equivalently.

After addressing these questions, we also verify that a simple agent can benefit from SmtW to actually solve efficiently a task.

The environment. We introduce a specific environment to answer these. We require this environment to be procedurallygenerated in order to test SmtW's ability to generalize to unseen SmtW encourages behaviour B over behaviour A while the other intrinsic reward does not.

Test environment

SmtW Other Intrinsic Reward

Behavior A Behavior B environments. We require the environment to be complex enough so that exhaustive exploration is prohibitively expensive. To achieve this, we introduce the KeysDoors grid-world of size NxN. When an agent makes the action "take" on a key, it is then able to move with it. Actions "open" or "take" make the agent lose the key it was previously holding. To solve the task, the agent has to go to the correct key, take it, go to the correct door without doing action "take" or "open" on the way (so as not to lose the key), and then "open" the door. We need the environment to require perseverance so we made the reward function -1 for any actions but the wait action, that is rewarded 0. Opening the correct door with the correct key gives a reward of 100 and terminates the episode. It requires perseverance as a "lazy" policy would get a return of 0 whereas trying to find the 100 reward gives -1 at each step. This is a well known issue in RL that simple exploration leads to such lazy solutions. The KeysDoors environment is generated procedurally. For each column, locations for a door and a key are sampled uniformly without replacement. Thus, there is exactly one key and one door on each column and these cannot be at the same location. The "correct" key is then uniformly sampled among the keys and the "correct" door is sampled uniformly among the doors. The initial position of the agent is sample uniformly on the grid. The environment gives both a ground-truth-state (an integer representing the current state), unused by SmtW as well as an RGB observation (as shown in Fig. 3), used by SmtW. Figure 4 shows a trajectory in one possible instance of the KeysDoors environment with 𝑁 = 5. Every observation 𝑥 (an 𝑁 × 𝑁 × 3 tensor) is normalized between 0 and 1 by dividing by 255.

The demonstrations. For a given instance of the environment, the demonstrator navigates between keys and doors and tries key/door pairs in a precise order. It takes the first key on the left and tries it on the first door on the left, then it tries the same key on the second door etc. Once it has tried the first key on every door, it repeats the operation with the second key and proceeds further this way. The episode ends when the demonstrator finds the right key/door pair and obtains the reward. Then it "exploits", taking the correct key and opening directly the correct door five consecutive times. Note that this also simulates the non-stationnarity happening in most goal-directed task solving processes. One first mainly explores and then exploits more and more.

Train vs. Test. The bonus is always used in new test environments, unseen in the demonstrations. SmtW's ability to generalize to new environments is thus tested in all the following experiments. Given the possible positions of the keys, of the doors and then of the correct key and the correct door, there are (𝑁 -1)𝑁 3 possible instances of the environment. The behaviors that are designed to study what is actually encouraged or discouraged by SmtW are the following. Their associated bonus is always studied in test instances of the environment.

• The demonstrator behavior acts as described previously, sequentially trying key/door pairs. • The random behavior takes random actions. Trajectories are limited to 1000 steps. • The demonstrator inverse behavior is similar to the demonstrator as it navigates to a key, takes it, navigates to a door and opens it. However, the key/door pairs are tried in the reverse order to the demonstrations. • The demonstrator random behavior is also similar but tries the key/door pairs in a random order. • The dummy demonstrator behavior navigates exactly like the demonstrator but drops the key at a random time on the way to the door (uniformly sampled on the path to the door) by taking action open. The trajectories are limited to 1000 steps. • The standing still behavior remains in its original position by only taking the wait action. • The waiting demonstrator behavior acts like the demonstrator but has a probability 0.1 of waiting at each step. • The unsafe demonstrator acts like the demonstrator but takes this action take each time it moves until it has a key. Taking action take somewhere else than on a key can be considered as breaking a safety constraint that the demonstrator respects strictly.

A trajectory of an agent moving to a key, taking it, moving to a door and trying to open it with the key is shown in Fig. 4

Bonus analysis

We train the SmtW bonus on 200 KeysDoors(N=5) training environments with 10 demonstrations for each of them. The implementation choices are detailed in Sec. 4.3. In order to study the priors extracted from the demonstrations, we study the bonus given by SmtW along various trajectories following a given behavior. We thereafter plot the distribution of received bonus along the various controlled behaviors on 20 test environments, unseen during the training of SmtW. We compare the bonus given by SmtW along these trajectories to the one that would be given by a count-based [START_REF] Alexander | An analysis of model-based interval estimation for Markov decision processes[END_REF] and a random network distillation bonus [START_REF] Burda | Exploration by random network distillation[END_REF]. The very same trajectories are presented to each bonus.

Does SmtW encourage a structured exploration more than a random one? We compare in Figure 5 the distribution of bonus received along random trajectories to the ones obtained by the demonstrator's behavior. Recall that SmtW has been trained on similar environments but is here tested on different ones. It is thus provided with trajectories unseen during training.

As shown on Figure 5, the demonstrator's behavior (top) is more rewarded by SmtW than the random behavior (bottom). We can conclude that SmtW encourage the agent to follow a demonstratorlike behavior on new unseen environments more than a random behavior. The count-based bonus also rewards the demonstrator's behavior more than a random one as a random behavior explores the environment very locally. Surprisingly, RND rewards the random behavior more than the demonstrator's one. This might be explained by the fact that the demonstrator visits several times the same state in order to explore correctly. Indeed the demonstrator has to go several times to the same key to take it and try it on the several doors.

Does SmtW capture the demonstrator's style, his way of exploring the environment? We show in Figure 6 the distribution of bonus received along different behaviors: the demonstrator one, the demonstrator inverse one as well as the demonstrator random one. These three behaviors lead to the same outcome but we hope to capture the demonstrator's exploration bias and see if it encourages the behaviors that tries the key/door pair in the same order as in the demonstrations. As shown on Figure 6, the count-based bonus and RND reward similarly the three behaviors, as they lead to the same amount of novelty. Only the order in which the key/door pairs are tried is change. SmtW, on the contrary, encourages to reproduce the demonstrator bias. It rewards more the behavior trying the key/door pairs in the same order as in the demonstrations. Does SmtW capture the priors useful to solve the task? Figure 7 shows the distribution of bonus received by the demonstrator behavior and compares the bonus received to the one received when following the dummy demonstrator one. As shown on Figure 7, the count-based bonus and RND reward equivalently these two behaviors as they bring the same amount of novelty (both in term of ground-truth-state and observations). SmtW does not reward the dummy demonstrator behavior as much as the expert one and we can interpret the lower distribution mode (SmtW-bottom) as the bonus obtained after loosing the key. We can argue that SmtW has somehow captured the prior that it is useful to navigate from the key to the door without loosing the key, as it rewards more the demonstrator behavior than the dummy demonstrator one. Does SmtW encourage long-term exploration? As the environment gives a reward of -1 for taking any action but the wait action, an agent not exploring sufficiently would quickly converge to the policy only taking action wait to avoid negative rewards (verified in Figure 11). This same problem is visible in the Pitfall! game, where the best agents learn a policy obtaining 0 reward, while persevering humans get much higher scores. We show in Figure 8 the distribution of bonus obtained by the standing still behavior. As shown on Figure 8, SmtW rewards much less a behavior not seeking novelty. As expected the count based gives a bonus very close to 0 for such a behavior. Perhaps surprisingly, RND rewards negatively this behavior but not with an average bonus lower than the demonstrator's behavior. This might be also due to the designed bonus normalization that RND uses (zero-mean unit-variance).

Does SmtW capture the constraints the demonstrator may be submitted to? A demonstrator can be subject to time or energy constraints. In the demonstrations, the demonstrator tries to explore the environment as fast as possible and does not take action wait on his way to keys and doors. We compare the bonus distribution obtained by the waiting demonstrator behavior to the one obtained by the demonstrator one. As shown on Figure 9, RND and the count-based bonus reward equivalently these two behaviors. On the other hand, SmtW rewards less the waiting demonstrator behavior. We argue it has somehow captured the prior resulting from the resource constraint that leads the demonstrator to try the key/door pairs as fast as possible. In other words, it favors behaviors that, as shown in the demonstrations, discard the wait action to simplify exploration of the MDP. What is more, a demonstrator might be subject to safety constraints. As example, it might be dangerous for a robot to try an action in an inappropriate place. The demonstrations minimize the number of time they use the action "take" and only do it when on keys. We can consider that the demonstrator's behavior complied with safety constraints. We show in Figure 10 the bonus distribution obtained by the demonstrator's behavior and compare it with the one obtained by the unsafe demonstrator. As shown on Figure 10, the RND and the count-based bonuses reward equivalently these two behaviors. This is expected as they bring the same amount of novelty. In contrast, SmtW rewards less the unsafe demonstrator behavior, capturing the safety prior the demonstrator have been subject to.

Overall, we argue that SmtW is able to recover some important bias and constraints inherent to the demonstrations. Hand-crafting a bonus expressing these motivations could be extremely complicated and we demonstrated that SmtW is able to generalize these motivations to unseen environments.

Training an agent on the bonus

We now wish to check that an agent can benefit from SmtW. We thus train a 𝑄-learning agent with SmtW and compare the results with that of a simple 𝜖-greedy (𝜖=0.1) exploration strategy and a count-based bonus with 𝐵(𝑠, 𝑎) = 𝑁 (𝑠, 𝑎) -1/2 . The results are averaged over 10 newly generated environments, unseen during SmtW training. For each of these environments, the experiment is repeated twice. We present, for each algorithm, the best result after a hyper-parameter search. The bonus given by our method is computed to capture the exploratory behavior of the demonstrator. In order for the agent not to keep exploring forever, our bonus is here divided by

√

𝑘 with k the number of step of training.

As Figure 11 shows, the Q-learning with an 𝜖-greedy exploration strategy quickly gets stuck in "waiting" at each timestep. SmtW encourages the agent to visit its environment and solves the 10 new environments much faster than the count-based method that push for exhaustive exploration.

Implementation Details

Our method works directly with visual inputs, as shown in Fig. 3. The network used for the behavioral cloning policy 𝜋 𝜙 has the following architecture: an LSTM with 64 units, a fully-connected layer with 512 units and relu activation and an output layer with as many units as there are actions available in the environment (7 for KeysDoors). It is trained with the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with a learning rate of 10 -3 and a batch size of 1. It uses the visual input from the environment and not the ground-truth state.

The network used for the regression of the bonus B 𝜃 has the same architecture but an output layer with a single unit. It is trained with the Adam optimizer, a learning rate of 10 -4 and a batch size of 1. The discount factor used in SmtW is set to 𝛾 = 0.99.

For experiment shown in Figure 11, the tabular 𝑄-learning is trained on the 10 test environments twice and the figure shows the median and the min/max values. For each of the compared algorithms, we sweep over the agent learning rate over the following values: [0.01, 0.1, 0.5, 0.7]. Only the result of the learning rate with the highest median return over the 10 × 2 runs is shown for each algorithm. The 𝜀-greedy strategy is used for all methods with 𝜀 = 0.1. Even though the agent is tabular, we recall that SmtW itself does not access the ground-truth state of the environment. It works from observations. The count-based bonus, on the contrary, counts ground-truth states.

RELATED WORK

Intrinsic Motivation. Intrinsic motivation is essential to mental development [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] and we can argue that this may, in consequence, be an essential component for computational learning. Oudeyer and Kaplan [START_REF] Oudeyer | What is intrinsic motivation? A typology of computational approaches[END_REF] argue that all humans respond to intrinsic motivations. Young infants motivations can be qualified as more chaotic as they push children to bite, throw, grasp or shout in order to learn. Adults, in contrast, have more structured intrinsic motivations, activated, for instance, when they play games, read novels or watch movies. Correctly using these numerous intrinsic motivations can be key to train agents that solve more and more difficult tasks. Instead of modeling such intrinsic motivations to mimic cognitive processes, we learn them from demonstrations.

Exploration. In order to provide an exploration signal to the agent, [START_REF] Alexander | An analysis of model-based interval estimation for Markov decision processes[END_REF] proposed the very intuitive count-based method in order to measure novelty. Counting how many times the agent has been in a given state, it rewards less visited states. Several methods extended this idea to large state-space problems [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Marlos | Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents[END_REF][START_REF] Ostrovski | Count-based exploration with neural density models[END_REF][START_REF] Tang | # exploration: A study of count-based exploration for deep reinforcement learning[END_REF], where it is not possible to count state occupancy. Intrinsic curiosity is also commonly computed as a prediction error, either trying to predict the environment's dynamics [START_REF] Pathak | Curiosity-driven exploration by self-supervised prediction[END_REF][START_REF] Raileanu | RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments[END_REF] or random statistics about the current state [START_REF] Burda | Exploration by random network distillation[END_REF]. Different methods try also to measure surprise as a prediction gain [START_REF] Houthooft | Variational information maximizing exploration[END_REF][START_REF] Schmidhuber | Curious model-building control systems[END_REF]. Instead of designing such a bonus, we aim at learning one from demonstrations.

Learning from demonstrations. Imitation learning, the problem of learning from demonstrations, is typically folded into two different paradigms. (1) Behavioral cloning [START_REF] Bagnell | Boosting structured prediction for imitation learning[END_REF][START_REF] Dean | Efficient training of artificial neural networks for autonomous navigation[END_REF][START_REF] Ross | Efficient reductions for imitation learning[END_REF] tries to directly match the demonstrator's behavior, generally using supervised learning techniques. (2) Inverse Reinforcement Learning [START_REF] Andrew | Algorithms for inverse reinforcement learning[END_REF][START_REF] Russell | Learning agents for uncertain environments[END_REF] first tries to recover a reward explaining the demonstrator's behavior, before optimizing the reward for imitating the demonstrator. Some methods output an explicit reward [START_REF] Abbeel | Apprenticeship learning via inverse reinforcement learning[END_REF][START_REF] Klein | A cascaded supervised learning approach to inverse reinforcement learning[END_REF][START_REF] Andrew | Algorithms for inverse reinforcement learning[END_REF][START_REF] Ziebart | Maximum entropy inverse reinforcement learning[END_REF] while adversarial imitation learning can be seen as IRL with implicit reward recovery [START_REF] Finn | Guided cost learning: Deep inverse optimal control via policy optimization[END_REF][START_REF] Fu | Learning robust rewards with adversarial inverse reinforcement learning[END_REF][START_REF] Ho | Generative adversarial imitation learning[END_REF]. Overall these methods all assume that the nearoptimality of the demonstrations. Some works try to relax this assumption and to learn from sub-optimal demonstrations [START_REF] Brown | Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations[END_REF][START_REF] Jacq | Learning from a Learner[END_REF]. IRL methods typically control the quality of their algorithm through the proxy of the return obtained by an agent trained on the inferred reward.

Our methods differs from these methods it does not assume that demonstrations are optimal but rather try to answer the question: "In what way is the demonstrator's behavior deviating from an optimal policy?". Moreover, we do not seek to recover a reward as in IRL but rather to recover a bonus explaining which, added to the environment reward, explains the demonstrator's behavior. Facing the same problem that the usual proxy to control the algorithm quality (training an agent on the inferred bonus) is not informative, we decided to study our method through its response to various behaviors.

CONCLUSION

In this work, we present a novel method for extracting an intrinsic bonus from the demonstrations. The method we introduce is offline and does not require environment interactions to recover the bonus, unlike recent adversarial imitation methods who need numerous interaction in order to recover a reward function. Anyway, those methods could not be readily applied to our problem, as they do not explicitly compute a reward function. Moreover, to the best of our knowledge, this is one of the very first method to recover some kind of reward that is history-dependent. We show how this bonus generalizes to unseen environments and is able to convey long-term priors. We exemplified the approach on a simple yet didactic and challenging example. Yet, testing the method on a larger-scale environment would require human exploratory demonstrations. Gathering such a dataset is costly and very few are already available, none of them really covering our setting. Even though the given example is simple, this novel approach of capturing the demonstrator's bias could potentially lead to new lines of work in RL. For instance, one could use our method to implement behavioral style-transfer in RL and show to an agent a specific way to solve the task thanks to demonstrations. Combining a reward and biases extracted from demonstrations may also help for robotic tasks, where some aspects of the task are easily programmable with a reward but some expectations on how to solve the task may be easier to transmit thanks to demonstrations. This could also lead to some advances in tackling mispecified rewards. Using both a reward, that would contain information on the task to solve but not fully describe the constraints of the problem and demonstrations to correct the reward can be key to train sequential controllers in complex dynamics.

  (ℎ, 𝑎) the associated optimal 𝑄-function. It satisfies the Bellman optimality equation (writing ℎ = (. . . , 𝑠), that is 𝑠 the last state of the trajectory ℎ and ℎ ′ = (ℎ, 𝑎, 𝑠 ′ )):

  Then, we can learn a network B 𝜃 (parameterized by 𝜃 , with LSTM units) by minimizing a square-loss, the regression target being𝑄 𝜙 (ℎ 𝐸 , 𝑎 𝐸 ) -𝛾𝑄 𝜙 (ℎ 𝐸 ′ , 𝜋 𝜙 (ℎ 𝐸 ′ )) -𝑅(𝑠 𝐸 , 𝑎 𝐸 ), an unbiased sampleof what would give the true Bellman equation. However, we only observe optimal actions (according to R + B), so this alone would hardly generalize to suboptimal ones. Therefore, we propose a heuristic, that consists in regressing for suboptimal actions towards B min , a hyperparameter of the algorithm. For example, it could be set to min(𝑄 𝜙 (ℎ 𝐸 , 𝑎 𝐸 ) -𝛾𝑄 𝜙 (ℎ 𝐸 ′ , 𝜋 𝜙 (ℎ 𝐸 ′ )) -𝑅(𝑠 𝐸 , 𝑎 𝐸 )), the minimum being over transitions in the dataset. This gives the following loss, for a transition (ℎ 𝐸 , 𝑎 𝐸 , ℎ 𝐸 ′ ), and for 𝑎 𝐸 being sampled randomly in A \{𝑎 𝐸 }:

Figure 2 :

 2 Figure 2: Comparing intrinsic rewards on what behavior they encourage or discourage on new unseen environments.

Figure 3 :

 3 Figure 3: KeysDoors(N=5). It contains 𝑁 keys and 𝑁 doors, modeled by two different colors. The agent has a third color. The goal is to find the correct key and to open the correct door with it. As doors (resp. keys) are indistinguishable (except by their locations), an explorer has to try the different keys on the different doors. Actions available are {go left, go right, go up, go down, take, open, wait}.When an agent makes the action "take" on a key, it is then able to move with it. Actions "open" or "take" make the agent lose the key it was previously holding. To solve the task, the agent has to go to the correct key, take it, go to the correct door without doing action "take" or "open" on the way (so as not to lose the key), and then "open" the door. We need the environment to require perseverance so we made the reward function -1 for any actions but the wait action, that is rewarded 0. Opening the correct door with the correct key gives a reward of 100 and terminates the episode. It requires perseverance as a "lazy" policy would get a return of 0 whereas trying to find the 100 reward gives -1 at each step. This is a well known issue in RL that simple exploration leads to such lazy solutions.The KeysDoors environment is generated procedurally. For each column, locations for a door and a key are sampled uniformly without replacement. Thus, there is exactly one key and one door on each column and these cannot be at the same location. The "correct" key is then uniformly sampled among the keys and the "correct" door is sampled uniformly among the doors. The initial position of

Figure 4 :

 4 Figure 4: A trajectory of length 9 in an instance the Keys-Doors(N=5) environment.

Figure 5 :Figure 6 :Figure 7 :

 567 Figure5: Bonus distribution received by an demonstrator's behavior (top) and a random behavior (bottom). We can say that a bonus encourages more a behavior A than a behavior B if the distribution of bonus along trajectories following A are globally higher than the one along trajectories following B. SmtW encourages the demonstrator behavior over the random behavior.SmtW

Figure 8 :Figure 9 :Figure 10 :

 8910 Figure 8: Bonus distribution received by the demonstrator behavior (top) and by the standing still behavior (bottom).SmtW

Figure 11 :

 11 Figure 11: Median and min/max values of the return per episode (left) and of the total bonus per episode(right).

  1[ a discount factor. The agent interacts with the environment through a (here deterministic) policy 𝜋 : S → A. The quality of a given policy is quantified by the associated state-action value function, or 𝑄function. It is the expected discounted cumulative reward for starting from 𝑠, taking action 𝑎, and following 𝜋 afterward: 𝑄 𝜋 (𝑠, 𝑎) = E 𝜋 [ 𝑡 ≥0 𝛾 𝑡 𝑟 𝑡 |𝑠 0 = 𝑠, 𝑎 0 = 𝑎], with 𝑎 𝑡 = 𝜋 (𝑠 𝑡 ), 𝑟 𝑡 = R(𝑠 𝑡 , 𝑎 𝑡 ) and 𝑠 𝑡 +1 ∼ P(.|𝑠 𝑡 , 𝑎 𝑡 ). By construction, it satisfies the Bellman equation: for any 𝑠, 𝑎, 𝑄 𝜋 (𝑠, 𝑎) = R(𝑠, 𝑎) + 𝛾 𝑠 ′ P(𝑠 ′ |𝑠, 𝑎)𝑄 𝜋 (𝑠 ′ , 𝜋 (𝑠 ′ )).