Lina Mezghani
email: linamezghani@fb.com

Meta Ai

Sainbayar Sukhbaatar

Piotr Bojanowski

Alessandro Lazaric

Meta Ai Karteek

Alahari Inria

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping

Keywords: Offline RL, Self-Supervised Learning, Goal-Conditioned RL

Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works [1, 2] targeted these challenges by learning goal-conditioned policies from offline datasets without manually specified rewards, through hindsight relabeling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that our model significantly outperforms existing approaches [1, 2], especially on tasks that involve long-term planning.

Introduction

While the goal of realizing general autonomous agents requires mastery of a large and diverse set of skills, achieving this by focusing on each skill individually with standard reinforcement learning (RL) frameworks is prohibitive. This is primarily due to the need for manually designed reward functions and environment interactions for each skill. Unsupervised RL has opened a way for learning agents that can execute diverse abilities without supervision (i.e., hand-crafted rewards), and then be further adapted to downstream tasks through few-shot or zero-shot generalization [START_REF] Pathak | Curiosity-driven exploration by selfsupervised prediction[END_REF][START_REF] Burda | Exploration by random network distillation[END_REF][START_REF] Yarats | Reinforcement learning with prototypical representations[END_REF][START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF]. However, learning policies with such methods is impractical with real robots as they require millions of interactions when trained online.

Recently, a line of study has emerged that uses pre-collected datasets of trajectories and trains policies offline (i.e., without additional interactions with the environment) [START_REF] Yarats | Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[END_REF][START_REF] Lambert | The challenges of exploration for offline reinforcement learning[END_REF]. More precisely, given a dataset of reward-free trajectories and a reward function designed to solve a specific task, the agent learns offline by relabeling the transitions in the dataset with the reward function. This setting is particularly relevant in robotics, where data collection is extremely time-consuming: disentangling data collection and policy learning in this context allows for faster policy iteration. However, it would require designing one specific reward function and learning one policy for each individual task.

An important question to scale offline robot learning is therefore to find ways of learning multi-task policies from already collected datasets. Recent works [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF][START_REF] Yang | Rethinking goal-conditioned supervised learning and its connection to offline rl[END_REF][START_REF] Li | Hierarchical planning through goal-conditioned offline reinforcement learning[END_REF], have targeted this problem from a goal-conditioned perspective: given a dataset of previously collected trajectories, the objective is to learn a goal-oriented agent that can reach any state in the dataset. The advantages of this formulation are two-fold: first, it makes it easy to interpret skills, and second it does not require any adaptation at test time. Making this framework unsupervised requires to break free from hand-crafted rewards, as proposed by Chebotar et al. [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF], where they learn goal-conditioned policies offline through hindsight relabeling [START_REF] Andrychowicz | Hindsight experience replay[END_REF]. However, their approach is subject to the pitfall of learning from sparse rewards, and can be inefficient in long-horizon tasks.

In this work, we present a self-supervised reward shaping method that enables building an offline dataset with dense rewards. To this end, we develop a self-supervised learning phase that aims at learning the structure and dynamics of the environment before training the policy. During this phase, we: (i) train a reachability network [START_REF] Savinov | Episodic curiosity through reachability[END_REF] to estimate the local distance in the state space S, then (ii) extract a set of representative states that covers S, and finally (iii) build a graph on this set to approximate the global distance in S. When training the goal-conditioned policy, we use the graph in two ways: to compute rewards through shortest path distance, and to create transitions of intermediate difficulty on the path to the goal. We evaluate our method on complex continuous control tasks, and compare it to previous stateof-the-art offline [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF][START_REF] Andrychowicz | Hindsight experience replay[END_REF] approaches. We show that our graph-based reward method learns good goal-conditioned policies by leveraging transitions from a dataset of past experience with neither any additional interactions with the environment nor manually-designed rewards. Moreover, we show that, contrary to prior work that uses datasets collected with a policy trained with supervised rewards [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF], our method allows for learning goal-conditioned policies even from datasets of poor quality, e.g. containing trajectories sampled with a random policy. Our work is thus the first to learn goal-conditioned policies from offline datasets without any supervision, as it does not require any hand-crafted reward function at any stage: data collection, policy training and evaluation.

Related Work

Goal-conditioned RL. In its original formulation, goal-conditioned reinforcement learning was tackled by several methods [START_REF] Kaelbling | Learning to achieve goals[END_REF][START_REF] Schaul | Universal value function approximators[END_REF][START_REF] Andrychowicz | Hindsight experience replay[END_REF][START_REF] Nasiriany | Planning with goal-conditioned policies[END_REF]. The policy learning process is supervised in these works: the set of evaluation goals is available at train time as well as a reward function that guides the agent to the goal. Several works propose solutions for generating goals automatically when training goalconditioned policies, including self-play [START_REF] Sukhbaatar | Intrinsic motivation and automatic curricula via asymmetric self-play[END_REF][START_REF] Sukhbaatar | Learning goal embeddings via self-play for hierarchical reinforcement learning[END_REF][START_REF] Openai | Asymmetric self-play for automatic goal discovery in robotic manipulation[END_REF], and adversarial student-teacher policies [START_REF] Campero | Learning with amigo: Adversarially motivated intrinsic goals[END_REF]. A recent line of research [START_REF] Warde-Farley | Unsupervised control through non-parametric discriminative rewards[END_REF][START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF][START_REF] Ecoffet | Go-explore: a new approach for hard-exploration problems[END_REF][START_REF] Pong | Skew-fit: State-covering selfsupervised reinforcement learning[END_REF][START_REF] Venkattaramanujam | Self-supervised learning of distance functions for goal-conditioned reinforcement learning[END_REF][START_REF] Hartikainen | Dynamical distance learning for semisupervised and unsupervised skill discovery[END_REF][START_REF] Mendonca | Discovering and achieving goals via world models[END_REF][START_REF] Mezghani | Walk the random walk: Learning to discover and reach goals without supervision[END_REF] focuses on learning goal-conditioned policies in an unsupervised fashion. The objective is to train general agents that can reach any goal state in the environment without any supervision (reward, goal-reaching function) at train time. In particular, Mendonca et al. [START_REF] Mendonca | Discovering and achieving goals via world models[END_REF] trains a model-based agent that learns to discover novel goals with an explorer model, and reach them with an achiever policy via imagined rollouts.

Offline RL. The data collection technique is an important aspect when studying the training of policies from pre-collected datasets. In this context, the first works assumed access to policies trained with task-specific rewards [START_REF] Fu | D4rl: Datasets for deep data-driven reinforcement learning[END_REF][START_REF] Gulcehre | Rl unplugged: A suite of benchmarks for offline reinforcement learning[END_REF]. More recently, methods proposed to leverage unsupervised exploration to collect datasets for offline RL [START_REF] Yarats | Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[END_REF][START_REF] Lambert | The challenges of exploration for offline reinforcement learning[END_REF]. In particular, Yarats et al. [START_REF] Yarats | Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[END_REF] creates a dataset of pre-collected trajectories, ExoRL, on the DeepMind control suite [START_REF] Tassa | Deepmind control suite[END_REF] generated without any hand-crafted rewards. Similar to URLB [START_REF] Laskin | Urlb: Unsupervised reinforcement learning benchmark[END_REF], ExoRL benchmarks a number of exploration algorithms [START_REF] Pathak | Curiosity-driven exploration by selfsupervised prediction[END_REF][START_REF] Eysenbach | Diversity is all you need: Learning skills without a reward function[END_REF][START_REF] Pathak | Self-supervised exploration via disagreement[END_REF][START_REF] Yarats | Reinforcement learning with prototypical representations[END_REF], and evaluates the performance of a policy trained on the corresponding offline datasets relabeled with task-specific rewards.

Multi-task Offline RL. Recent works proposed to learn multiple tasks from pre-collected datasets, starting with methods [START_REF] Endrawis | Efficient self-supervised data collection for offline robot learning[END_REF] that generate goals to improve the offline data collection process in a selfsupervised way. This connection has also been studied in the supervised setting [START_REF] Yang | Rethinking goal-conditioned supervised learning and its connection to offline rl[END_REF][START_REF] Ma | How far i'll go: Offline goal-conditioned reinforcement learning via f -advantage regression[END_REF] and when learning hierarchical policies [START_REF] Li | Hierarchical planning through goal-conditioned offline reinforcement learning[END_REF]. In a setting closely related to our work, Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] considers the problem of learning goal-conditioned policies from offline datasets without interacting with the environment, and with no task-specific rewards. They employ goal-conditioned Q-learning with hindsight relabeling [START_REF] Andrychowicz | Hindsight experience replay[END_REF]. As opposed to their work that relies on learning from sparse rewards, we propose to leverage a self-supervised training stage to densely shape rewards.

Preliminaries

Let E = (S, A, P, p 0 , γ, T ) define a reward-free Markov decision process (MDP), where S and A are state and action spaces respectively, P : S × A × S → IR + is a state-transition probability add edge add node Figure 1: Overview of the graph building algorithm. Given a transition (s i , s i+1 ) ∈ D, we add s i as node if it is distant enough from existing nodes in the graph. Moreover, we add an edge in the graph between the incoming nearest neighbor of s i and the outgoing nearest neighbor of s i+1 . function, p 0 : S → IR + is an initial state distribution, γ is the discount factor, and T is the task horizon. In the goal-conditioned setting, the objective is to learn a policy π : S × G → A that maximizes the expectation of the cumulative return over the goal distribution, where G denotes the goal space. Here, we make the common assumption that states and goals are defined in the same form, i.e., G ⊂ S.

We assume that we have access to a dataset D of pre-collected episodes generated by using any data collection algorithm in E. Each episode is stored in D as a series of (s, a, s ) tuples, where s, s ∈ S and a ∈ A. In the general offline formulation introduced by Yarats et al. [START_REF] Yarats | Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[END_REF], the dataset D can be relabeled by evaluating any reward function r : S × A → IR at each tuple in D, and adding the resulting tuple (s, a, r(s, a), s ) in the relabeled dataset D r . We can extend this protocol to the goaloriented setting by considering a goal distribution p G in the goal space, and any goal-conditioned reward function r : S × A × G → IR. Given a tuple (s, a, s ) in D, we relabel it by sampling a goal g ∼ p G , computing r(s, a, g) and adding the resulting tuple (s, a, g, r(s, a, g), s ) in the relabeled dataset D r,p G .

Once the relabeled dataset D r,p G is generated, we can learn a goal-conditioned policy by executing any offline RL algorithm. The algorithm runs completely offline, by sampling tuples from D r,p G and without any interaction with the environment. The goal-conditioned policy is then evaluated online in E on a set of fixed evaluation goals that is not known during training.

Self-supervised Reward Shaping

We now describe our self-supervised reward shaping method. It comprises three stages that we will detail below. In the first stage, we train a Reachability Network (RNet) [START_REF] Savinov | Episodic curiosity through reachability[END_REF] on the trajectories in D to predict whether two states are reachable from one another. The second stage consists in building a directed graph M whose nodes are a subset of states in D, and edges connect reachable states. We employ the RNet as a criterion to avoid adding similar states to M so that its nodes cover the states in D uniformly. The final stage consists in training the goal-conditioned policy on transitions and goals sampled from D. It is trained with dense rewards computed as the sum of a global (based on the graph distance in M) and local (based on the RNet) distance terms. The important aspect of our method is that the whole training only uses trajectories from the pre-collected dataset D without running a single action in the environment. We now describe each component in more detail.

Reachability network

In order to learn a good local distance between states in D, we adopt an asymmetric version of the Reachability Network (RNet) [START_REF] Savinov | Episodic curiosity through reachability[END_REF]. The general idea of RNet is to approximate the distance between states in the environment by the average number of steps it takes for a random policy to go from one state to another. We adapted the original formulation with two modifications: first, we use exploration trajectories from D instead of random trajectories and second, we leverage the temporal direction because a state can be reachable from another without the converse being true. Let (s a 1 , ..., s a T ) denote a trajectory in D, where a is a trajectory index. We define a reachability label y ab ij for each pair of observations (s a i , s b j ) by

y ab ij = 1 if a = b and 0 ≤ j -i ≤ τ reach , 0 otherwise, for 1 ≤ i, j ≤ T, (1)
where the reachability threshold τ reach is a hyperparameter. The reachability label is equal to 1 iff the states are in the same trajectory and the number of steps from s a i to s b j is below τ reach , as shown 3 in Figure 2a. Note that y ab ij = y ab ji . We train a siamese neural network R, the RNet, to predict the reachability label y ab ij from a pair of observations (s a i , s b j ) in D. The RNet consists of an embedding network g, and a fully-connected network f to compare the embeddings, i.e.,

R(s

a i , s b j ) = σ f (g(s a i ), g(s b j )) , (2)
where σ is a sigmoid function. A higher R value indicates two states reachable easily with random walk, so they can be considered close in the environment. More precisely, R takes values in (0, 1) and s is reachable from s if R(s, s ) ≥ 0.5. RNet is learned in a self-supervised fashion, as the ground-truth labels needed to train the network are generated automatically.

Directed graph

In the next phase, we use trajectories in D to build a directed graph M that captures high-level dynamics of the environment, as illustrated in Figure 1. We want the nodes of M to evenly represent the states in D. This is achieved by filtering the states in D: a state is added to M only if it is distant enough from all the other nodes in M. More precisely, a state s ∈ D is added to M if and only if R(s, n) < 0.5 and R(n, s) < 0.5, for all n ∈ M.

Note that we require both the directions to be novel. This filtering avoids redundancy by preventing similar states to be added to the memory. It also has a balancing effect because it limits the number of states that can be added from a certain area even if it is visited by the agent many times in D.

Once the nodes are selected, we connect pairs that are reachable from one to another. To this end, we employ trajectories in D because they contain actual feasible transitions. Given a transition s i → s j in D, we add edge n i → n j if s i can be reached from node n i and node n j can be reached from s j . This way, we have a chain n i → s i → s j → n j and can assume n j is reachable from n i . Concretely, we select node n i to be the incoming nearest neighbor (NN in ) to s i , and n j to be the outgoing nearest neighbor (NN out ) from s j , i.e.,

n i = NN in (s i ) = argmax n∈M R(n, s i ), n j = NN out (s j ) = argmax n∈M R(s j , n). (4)
By performing this action over all the transitions in D, we turn M into a directed graph where edges represent reachability from one node to another.

Distance function for policy training

We then use the obtained directed graph to compute a global distance in the state space. Indeed, RNet predicts reachability between s i and s j so we can directly use it as a distance metric

d l (s i , s j ) = 1 -R(s i , s j ), ∀s i , s j ∈ S. (5)
However, this reachability metric is confined to a certain threshold, so there is no guarantee that the RNet predictions will have good global properties.

In contrast, the directed graph M captures high-level global dynamics of the environment. We can easily derive a distance function d M (n i , n j ) between any pair of nodes in M by computing the length of the shortest path in this graph, provided the graph is connected. In practice, we can use a trick to connect the graph if necessary, by adding an edge between the pair of nodes from different connected components with the maximum RNet value. Moreover, we can extend this distance d M to a global distance function d g in the state space S by finding, for any pair s i and s j in S their nearest neighbors in the corresponding direction. More precisely,

d g (s i , s j ) = d M (NN out (s i ), NN in (s j )), ∀s i , s j ∈ S. (6)
The distance d g between two states in the state space becomes the length of the shortest path between their respective closest nodes in the graph. This process, summarized in Figure 2b, propagates the good local properties of RNet to get a well-shaped distance function for states that are further away. Since d g captures global distances while d l captures local fine-grained distance, we use their combination as a final distance function: ∀s i , s j ∈ S, d(s i , s j ) = d g (s i , s j ) + d l (s i , s j ).

Policy training

The last phase of our method is training the goal-conditioned policy offline. Here, we create an offline replay buffer B that is filled with relabeled data. We randomly sample a transition (s t , a t , s t+1 ) from D as well as a goal g and relabel the transition with reward r t = -d(s t+1 , g). We then push the relabeled transition (s t , a t , g, r t , s t+1 ) to B. In order to create a curriculum that artificially guides the agent towards the goal, we experimented with two different transition augmentation techniques:

Sub-goal augmentation. Let (s t , a t , g, r t , s t+1 ) denote a relabeled transition and (n 0 , ..., n P -1 ) the shortest path in the graph M between n 0 = NN out (s t ) and n P -1 = NN in (g). The augmentation technique consists in adding to the replay buffer every transition (s t , a t , n i , r i t , s t+1 ) for all i ∈ {0, P -1}, where r i t = -d(s t+1 , n i ). In other words, given a transition (s t , a t , s t+1 ) and a goal g from D, we push to the replay buffer a set of relabeled transitions with all goals on the shortest path from s t to g (and their corresponding rewards).

Edge augmentation. Similar to the subgoal augmentation technique, we consider a relabeled transition (s t , a t , g, r t , s t+1 ) and the associated shortest path (n 0 , ..., n P -1 ). This time, we keep the same goal g for every augmented transition, but for every edge (n i-1 , n i ), i ∈ {1, P -1}, we add the relabeled transition (s i t , a i t , g, r i t , s i t+1 ) to B where (s i t , a i t , s i t+1 ) ∈ D, NN out (s i t ) = n i-1 , NN in (s i t+1 ) = n i and r i t = -d(s i t , g). Note that the existence of such a transition in D is guaranteed by construction: an edge is added to the graph from one node to another iff there exist a transition in D whose corresponding nearest neighbors are these two nodes (in the same order).

Once the replay buffer B is filled, the goal-conditioned policy can be trained using any off-policy algorithm. In our implementation, we chose Soft Actor-Critic [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF], as it is known to require few hyper-parameter tuning, and is widely used in the literature.

Experiments

Environments & data collection

We perform experiments on three continuous control tasks with state-based inputs.

UMaze [START_REF] Kanagawa | mujoco-maze[END_REF]. The first environment, shown in Figure 3a, is a two-dimensional U-shaped maze with continuous action space and a fixed initial position. We generate the training data for this environment by deploying a random policy with randomized start position in the maze. We collect 10k trajectories of length 1k. We evaluate the goal-conditioned agent by giving the agent a goal sampled at random in the environment and computing the final euclidean distance to the goal.

RoboYoga Walker [START_REF] Mendonca | Discovering and achieving goals via world models[END_REF]. Introduced by Mendonca et al. [START_REF] Mendonca | Discovering and achieving goals via world models[END_REF], the challenging RoboYoga benchmark is based on the Walker domain of the DeepMind control suite [START_REF] Tassa | Deepmind control suite[END_REF], and consists of 12 goals that correspond to body poses inspired from yoga (e.g. lying down, raising one leg or balancing). We consider the state-based version of the task, and use the task-agnostic dataset from Yarats et al. [START_REF] Yarats | Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[END_REF] generated with an unsupervised exploration policy. It contains 10k trajectories of length 1k obtained by deploying the "proto" [START_REF] Yarats | Reinforcement learning with prototypical representations[END_REF] algorithm in the Walker domain. The success metric of the evaluation policy is assessed by the pose of the humanoid at the end of the episode.

Pusher [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF]. We also apply our method on Pusher, a realistic robotic environment shown in Figure 7 (left), where a robot arm (red) needs to push a puck (blue) to a specified location on a table.

To build the offline dataset, we generated 10k random trajectories of length 200. Similar to prior works [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF][START_REF] Pong | Skew-fit: State-covering selfsupervised reinforcement learning[END_REF][START_REF] Mezghani | Walk the random walk: Learning to discover and reach goals without supervision[END_REF], we generated 500 goals at random in the state space, and we measured the performance as the final Euclidean distance between the puck and its target location.

Ablation & design choices

We first show that the graph structure is necessary for long-term planning. Then, we explain the importance of the directness of the graph on tasks with asymmetric behaviours. Finally, we show the impact of transition augmentation techniques when labeling data for the goal-conditioned policy.

Necessity of graph-based rewards. An important component of our method is the construction of the graph M that enables computing a distance with good global properties. To empirically validate this hypothesis, we performed a comparison between the goal-conditioned policy trained with RNet rewards (i.e., by using the distance d l from equation ( 5)) and the one trained with both distance terms as reward. We run this experiment on the UMaze environment, and show results in Figure 3d.

We note that the model trained with graph rewards outperforms the one trained with RNet rewards overall, particularly for distant goals (ie. rooms 3 and 4). We also notice that the model trained with RNet rewards is slightly better for goals that are close to the initial position. This highlights the fact that RNet is good at estimating local distances. The qualitative visualization in Figure 3b & 3c confirms this observation, as it shows low values between states in the first and fourth rooms.

Importance of graph directness. We then investigate the importance of the asymmetry of the RNet and the directness of the graph. To this end, we implement an undirected version of our method where the RNet is symmetric and the graph is undirected. All other components of our method are unchanged. First, we compare the performance of both variants in the UMaze task in Figure 4a, and note that asymmetric RNet and directed graph in our approach significantly improve the goal-conditioned policy performance (+11% on success rate), especially on goals close to the initial location, i.e., goals in rooms 1 and 2. We then analyze qualitative visualizations of the shortest path in the undirected and directed graphs in the RoboYoga task, as shown in Figure 4b. In the undirected case, the humanoid defies the laws of gravity and is encouraged to stand its head by flipping backwards, which might be extremely difficult, or even infeasible. In the directed case, the shortest path fosters the agent to first get back on its legs, and then lean forward. In this exemple, the gravity makes the dynamics of the environment non-symmetric and non-fully reversible, which justifies the directed formulation described in our method.

Transition sampling strategy. As a final ablation study, we study the utility of the transition augmentation techniques described in subsection 4.4. We evaluate four possible variants of our method: Comparison on Pusher. As a final experiment, we compared our method to prior work on a realistic robotic environment, as shown in Figure 7. Our policy trained offline is evaluated by sampling a goal at random in the state space, and measuring three different metrics: (i) the hand distance, which corresponds to the final distance between the end of the robot arm and the target, (ii) the puck distance, which measures the distance between the final puck location and the target, and (iii) the average distance, the average of the first two metrics. Our method outperforms the baselines on this task, and our goal-conditioned agent is able to sequentially place the puck at the goal location, and then place the hand at its target location. On the contrary, HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] places the puck at the target location with a performance similar to our method, but lacks precision on the hand location.

Conclusion: Summary and Limitations

We proposed a method for learning multi-task policies from pre-generated datasets in an offline and unsupervised fashion, i.e., without requiring any additional interaction with the environment, nor manually designed rewards. Our method leverages a self-supervised stage that aims at learning the dynamics of the environment from the offline dataset, and that allows for shaping a dense reward function. It shows significant improvement over prior works based on hindsight relabeling, especially on long-horizon tasks, where dense rewards are crucial for learning a good policy.

The main limitation of our method is that it relies on the availability of a pre-collected dataset of trajectories, with a sufficiently large coverage of the state space for proper policy learning. Although such data can be already available, as for the RoboYoga Walker task, or that offline dataset collection could be done with random policies, as we did on the UMaze and Pusher tasks, this step can be challenging for other environments. Another limitation is that we evaluated our method exclusively on simulated environments, and we did not perform any experiments on real robots, for which precollected dataset with expert demonstrations can be available [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF]. 0 for all transitions except the final one that gets a reward of 1. The pseudo-code for this method is shown in Figure 8.

Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] relies on a similar idea as in HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF], but contains two additional steps to improve the method. The first step is a form of goal chaining and it consists in using the Q-value at the final state of the trajectory to compute the reward for the final transition. The second step aims at balancing the unseen action effect, in order to regularize the action space. In practice, it consists in sampling negative actions from the policy and label zero-reward transitions with these actions. The implementation of both tricks is shown in Figure 8.

For the implementation of the third baseline, HER + random negative action, the overall algorithm is the same as HER, except that we also generate transitions with negative actions, similar to Actionable Models. This time, the negative actions are not sample from the policy, but are generated uniformly at random from the action space. Randomly cut τ at step i for j ∈ {0, ..., i -2} do (s j , a j , g, 0, s j+1 ) → B a 1 ∼ π(s j , g) (s j , a 1 , g, 0, s j+1 ) → B (s j , a j , s i , 0, s j+1 ) → B a 2 ∼ π(s j , s i ) (s j , a 2 , s i , 0, s j+1 ) → B end for (s i-1 , a i-1 , g, Q(s i , a i-1 , g), s i ) → B a 3 ∼ π(s i-1 , g) (s i-1 , a 3 , g, 0, s i ) → B (s i-1 , a i-1 , s i , 1, s i ) → B a 4 ∼ π(s i-1 , s i ) (s i-1 , a 4 , s i , 0, s i ) → B end while Figure 8: Pseudo-code for replay buffer filling with HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] and Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] methods. We compare both implementations by showing in red modifications related to goal chaining, and in blue edits related to unseen action regularization.

A.3 Hyper-parameters

We first list the hyper-parameters for the self-supervised reward shaping phase in Table 1. Table 2 details the hyper-parameters for the offline policy training stage with SAC [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. For the Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] and HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] baselines, we used the same parameters as in our approach, with the exception of some parameters specific to these methods, shown in Table 3.

These hyper-parameters were obtained by performing a random search for all the methods over several parameter values. All the experiments in this work were performed on 3 random seeds. Table 3: Hyper-parameters for offline policy learning with SAC [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] specific to Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] and HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] baselines.

  Visualisation of the reward computation

Figure 2 :

 2 Figure 2: Visualization of our dense reward shaping method. (a) shows how training labels are generated for training the RNet: given a state s i , positive pairs are sampled in the same trajectory within a threshold τ reach , and the rest of the trajectory forms negative pairs. (b) presents how rewards are implemented as a combination of a global distance term (green), computed with the shortest path in the graph between the outgoing nearest neighbor (NN out ) of the state s t+1 and the incoming nearest neighbor of the goal (NN in ), and a local distance term (red) computed using the RNet value between NN in and g.

Figure 3 :

 3 Figure 3: (a) UMaze environment, Heatmap of rewards computed with RNet (b) and graph (c) distances, and (d) Performance of the goal-conditioned policy trained with RNet and graph-based rewards on UMaze. In (b) and (c), high rewards are shown in yellow, and low rewards in black.

Figure 6 :Figure 7 :

 67 Figure 6: Performance on the RoboYoga Walker taskComparison on RoboYoga Walker. In a second experiment, we compare our method to baselines on the RoboYoga task, as shown in Figure6a. Here again, our method outperforms prior work, and Actionable Models does not make any significant improvement over HER. The results broken down by goal are shown in the supplementary material. Overall these results suggest that our dense reward shaping method allows for faster and more robust offline goal-conditioned policy training.

Table 1 :

 1 Hyper-parameters for reachability network training and directed graph construction.

	Common hyper-parameter			Value
	Task	UMaze RoboYoga
	Number of training pairs	5 × 10 5	5 × 10 5
	Ratio of negatives		0.5	0.5
	Ratio of negatives from same trajectory	0.5	0.5
	Reachability threshold (τ reach )		5	2
	Batch size		512	512
	Learning rate	0.001	0.0003
	Weight decay	0.00001	0.00001
	Total number of training epochs		20	100
	Capacity of the directed graph	1000	10000
	Hyper-parameter		Value
	Task	UMaze RoboYoga
	Replay buffer capacity	10 6		10 6
	Batch size	2048		2048
	Discount (γ)	0.90		0.95
	Number of updates per epoch	1000		1000
	Total number of epochs	1000		1000
	Target update interval	1		1
	Soft update coefficient (τ )	0.005		0.005
	SAC entropy parameter (α)	0.05		0.01
	Optimizer	Adam		Adam
	Learning rate	0.0003	0.0003
	Action repeat	1		2
	Reward scaling factor	0.1		0.5

Table 2 :

 2 Hyper-parameters for offline policy learning with SAC[START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] with our method.

	Hyper-parameter	Value
	Task	UMaze RoboYoga
	Discount (γ)	0.99	0.99
	SAC entropy parameter (α)	0.01	0.001
	Learning rate	0.0001	0.0001
	Reward scaling factor	1	10

Acknowledgments

Karteek Alahari is supported in part by the ANR grant AVENUE (ANR-18-CE23-0011).

(i) without any augmentation, (ii) with edge augmentation only, (iii) with subgoal augmentation only, and (iv) with both augmentations. We execute this experiment on the RoboYoga task, and show results in Figure 6b. We observe that both of the augmentation techniques improve the performance of the goal-conditioned agent, with subgoal augmentation showing greater improvement. Moreover, we note that combining both augmentations improves the performance further. For the reminder of the experiments, we use both these augmentation techniques. We show the success rate for goals sampled at random in each of the four rooms, as well as the average over all rooms.

Comparison to prior work

Baselines. We compare our method to prior work on unsupervised goal-conditioned policy learning.

We perform an apples-to-apples comparison by implementing the baselines using the same learning framework as our method, and changing the reward relabeling process. We compare with the following baselines:

• Hindsight Experience Replay [HER] [START_REF] Andrychowicz | Hindsight experience replay[END_REF] This is a re-implementation of the standard unsupervised RL technique, adapted to the offline setting. More precisely, we relabel subtrajectories from D with a sparse reward, which is equal to 1 only for the final transition of the sub-trajectory, and 0 everywhere else. Following Chebotar et al. [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF], we also label sub-trajectories with goals sampled at random in D and zero reward. • HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF] with random negative action is a variant of HER where, for a transition in D we sample an action uniformly at random in the action space and label it with zero reward. This helps overcoming the problem of over-estimation of the Q-values for unseen actions mentioned in Chebotar et al. [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF]. • Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] This approach is based on goal-conditioned Q-learning with hindsight relabeling. We re-implemented the goal relabeling procedure that uses the Q-value at the final state of sub-trajectories in D to enable goal chaining, as well as the negative action sampling trick.

Comparison on UMaze. We compare our method to the baselines on the UMaze task, and show results in Figure 5. We observe that our model outperforms all baselines overall, and shows greater improvements on challenging goals that are far from the initial position. Interestingly, we note that Actionable Models reaches goals in the first room only. This confirms the intuition that sparse rewards make it difficult for the policy to learn long-horizon tasks.

A Implementation details

A.1 Self-supervised reward shaping

We provide pseudo-code for the two stages of our approach: the graph building process is shown in Algorithm

Relabel transition with goal g and reward r t , and Push (s t , a t , g, r t , s t+1 ) to B end while

A.2 Actionable Models baselines re-implementation

In this section, we provide details for our re-implementation of Actionable Models [START_REF] Chebotar | Actionable models: Unsupervised offline reinforcement learning of robotic skills[END_REF] and HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF]. Since we are using the same optimization algorithm for offline policy training for these baselines and our method, the only difference lies in how the transitions in the pre-collected dataset D are relabeled to build the replay buffer B.

In HER [START_REF] Andrychowicz | Hindsight experience replay[END_REF], the idea is to sample a trajectory and a goal g at random in D, and to cut the trajectory at a step i. Each transition in the trajectory (until step i) is then relabelled twice: once with the goal g and reward 0 for all transitions, and once with goal s i (the final state of the trajectory) and reward

A.4 Architecture details

Reachability Network [START_REF] Savinov | Episodic curiosity through reachability[END_REF] The RNet has a siamese architecture with two embedding heads (one for each observation of the pair) with tied weights, and a comparator network that compares both embeddings and returns a reachability score. For the UMaze task, we used an embedding head with 3 fully-connected layers with batch normalization and Tanh activations, with a hidden size of 64 and an embedding size of 16. For the Roboyoga Walker task, the embedding network has the same architecture, but we increased both the hidden and embedding sizes to 128. The comparator network is also a fully-connected network. It contains batch normalization and ReLU activations. The hidden size for the UMaze (respectively the RoboYoga Walker) task is set to 16 (resp. 128) and the number of layers is 2 (resp. 4).

Policy Network

The goal-conditioned policy network takes as input both the observation and the goal, in separate heads with the same architecture but independent weights. These heads are implemented as 3-layer fully-connected networks with Tanh activations, hidden size of dimension 64, and 16 dimensions for the feature size. The output from both the heads is then concatenated and fed into a 2-layer fully-connected network of width 256. The critic network has the same architecture for both observation and goal heads, and is followed by 3 fully connected-layers of width 256.

B Full results on RoboYoga Walker task

We show the comparison of our method against the aforementioned baselines on each of the 12 goals of the RoboYoga Walker task in Figure 9. These goals are illustrated in Figure 10. We see that our method masters most of the goals that do not require balancing (Lie Back & Front, Legs Up, Lunge), and succeeds quite well at more complicated goals like Side Angle, Lean Back and Bridge, but is unable to progress in complex goals like Head Stand or Arabesque.