Cédric Colas

Ahmed Akakzia

Pierre-Yves Oudeyer

Mohamed Chetouani

Olivier Sigaud

Language-Conditioned Goal Generation: a New Approach to Language Grounding in RL

In the real world, linguistic agents are also embodied agents: they perceive and act in the physical world. The notion of Language Grounding questions the interactions between language and embodiment: how do learning agents connect or ground linguistic representations to the physical world ? This question has recently been approached by the Reinforcement Learning community under the framework of instruction-following agents. In these agents, behavioral policies or reward functions are conditioned on the embedding of an instruction expressed in natural language. This paper proposes another approach: using language to condition goal generators. Given any goal-conditioned policy, one could train a language-conditioned goal generator to generate language-agnostic goals for the agent. This method allows to decouple sensorimotor learning from language acquisition and enable agents to demonstrate a diversity of behaviors for any given instruction. We propose a particular instantiation of this approach and demonstrate its benefits.

Introduction

Language Grounding describes the idea that language acquisition is strongly shaped by one's experience of the physical world. This idea emerged in Cognitive Science [START_REF] Harnad | The symbol grounding problem[END_REF][START_REF] Glenberg | Grounding language in action[END_REF][START_REF] Zwaan | Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking[END_REF] and quickly inspired Artificial Intelligence approaches in natural language processing [START_REF] Roy | Language-Conditioned Goal Generation: a New Approach to Language Grounding in RL Schaul[END_REF], humanmachine interactions [START_REF] Dominey | Emergence of grammatical constructions: evidence from simulation and grounded agent experiments[END_REF][START_REF] Madden | A cognitive neuroscience perspective on embodied language for human-robot cooperation[END_REF] and more recently deep Reinforcement Learning (deep RL) [START_REF] Luketina | A survey of reinforcement learning informed by natural language[END_REF].

In the RL community, this has taken the form of languageconditioned agents [START_REF] Hermann | Grounded language learning in a simulated 3D world[END_REF][START_REF] Chan | Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning[END_REF] * Equal contribution 1 Flowers Team, Inria Bordeaux, FR. 2 Université Paris-Sorbonne, Paris, FR.. Correspondence to: Cédric Colas <cedric.colas@inria.fr>. [START_REF] Bahdanau | Learning to understand goal specifications by modelling reward[END_REF][START_REF] Cideron | Self-educated language agent with hindsight experience replay for instruction following[END_REF][START_REF] Jiang | Language as an abstraction for hierarchical deep reinforcement learning[END_REF][START_REF] Zhong | Generalising to novel environment dynamics via reading[END_REF][START_REF] Waytowich | Grounding natural language commands to starcraft ii game states for narration-guided reinforcement learning[END_REF]Colas et al., 2020b). Language can be used to build representations [START_REF] Waytowich | Grounding natural language commands to starcraft ii game states for narration-guided reinforcement learning[END_REF] or to characterize the dynamics of the environment [START_REF] Zhong | Generalising to novel environment dynamics via reading[END_REF]. However, it is mostly used to represent instructions or goals [START_REF] Hermann | Grounded language learning in a simulated 3D world[END_REF][START_REF] Chan | Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning[END_REF][START_REF] Bahdanau | Learning to understand goal specifications by modelling reward[END_REF][START_REF] Cideron | Self-educated language agent with hindsight experience replay for instruction following[END_REF][START_REF] Jiang | Language as an abstraction for hierarchical deep reinforcement learning[END_REF]Colas et al., 2020b). In these approaches, natural language (NL) sentences are embedded through recurrent networks and merged with the agent's state to form the input of the policy or reward function. The language encoder is then trained jointly with either the former [START_REF] Hermann | Grounded language learning in a simulated 3D world[END_REF][START_REF] Chan | Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning[END_REF][START_REF] Jiang | Language as an abstraction for hierarchical deep reinforcement learning[END_REF][START_REF] Cideron | Self-educated language agent with hindsight experience replay for instruction following[END_REF][START_REF] Hill | Emergent systematic generalization in a situated agent[END_REF] or the latter [START_REF] Bahdanau | Learning to understand goal specifications by modelling reward[END_REF]Colas et al., 2020b). These approach demonstrate benefits over traditional goalconditioned methods:

1. Targeting abstract goals. Language can express abstract goals characterized by sets of properties the scene should verify (e.g. block A above block B). This contrasts with previous goal-as-state approaches, where goals are specific states of the agent (e.g. target pixel image, target block positions). 2. Systematic generalization.

Previous languageconditioned approaches demonstrate strong generalization capabilities including generalizations to new combinations of action verbs and object attributes (colors, shapes, categories etc.) [START_REF] Hermann | Grounded language learning in a simulated 3D world[END_REF][START_REF] Bahdanau | Learning to understand goal specifications by modelling reward[END_REF][START_REF] Hill | Emergent systematic generalization in a situated agent[END_REF]Colas et al., 2020b).

However, this comes with drawbacks:

1. Language becomes a pre-requisite for sensorimotor learning. Pre-verbal infants are known to demonstrate goal-oriented behavior [START_REF] Wood | The role of tutoring in problem solving[END_REF]. Languageconditioned agents require language inputs to act in the world and, thus, cannot account for this decoupling. 2. Lack of behavioral diversity. Because policies are directly conditioned on language embeddings, an instruction in a given context only generates a low diversity of behaviors: a unique behavior for a deterministic policy and minor noise-induced behavioral variations for stochastic policies.

Goal-Conditioned Policy

Language-Conditioned Goal Generator goal (language-agnostic)

Figure 1. Language-conditioned policy, goal-conditioned policy and language-conditioned goal generator.

This paper proposes to leverage the abstraction and generalization capacities of language while 1) decoupling language acquisition from sensorimotor learning; 2) allowing behavioral diversity. This is achieved via language-conditioned goal generation (Figure 1). Instead of using language to condition policies directly, we use it to condition a generator of language-agnostic goals.

Language-agnostic goals can be represented by specific states (e.g. pixels inputs) [START_REF] Péré | Unsupervised learning of goal spaces for intrinsically motivated goal exploration[END_REF][START_REF] Laversanne-Finot | Curiosity driven exploration of learned disentangled goal spaces[END_REF][START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF][START_REF] Van De Wiele | Unsupervised control through non-parametric discriminative rewards[END_REF][START_REF] Nair | Contextual imagined goals for self-supervised robotic learning[END_REF][START_REF] Pong | Skew-fit: State-covering self-supervised reinforcement learning[END_REF] or handcrafted representations (e.g. target block coordinates) (Schaul et al., 2015;[START_REF] Andrychowicz | Hindsight Experience Replay[END_REF][START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF][START_REF] Racaniere | Automated curricula through setter-solver interactions[END_REF][START_REF] Fournier | Clic: Curriculum learning and imitation for object control in nonrewarding environments[END_REF][START_REF] Colas | Intrinsically motivated multi-task, multi-goal reinforcement learning[END_REF]. These can come from any source: uniform sampling of the goal space (Schaul et al., 2015), sampling from low-density areas [START_REF] Pong | Skew-fit: State-covering self-supervised reinforcement learning[END_REF], from high learning progress areas [START_REF] Fournier | Clic: Curriculum learning and imitation for object control in nonrewarding environments[END_REF][START_REF] Colas | Intrinsically motivated multi-task, multi-goal reinforcement learning[END_REF], using generative models of states [START_REF] Nair | Visual reinforcement learning with imagined goals[END_REF]2019) or generative models targeting intermediate difficulty [START_REF] Florensa | Automatic goal generation for reinforcement learning agents[END_REF][START_REF] Racaniere | Automated curricula through setter-solver interactions[END_REF]. In our approach, language-conditioned goal generators are a source of language-agnostic goals among others. As a result, pure sensorimotor training does not require any language input: agents can simply use any other source of goals.

Language-conditioned goal generators are generative models of language-agnostic goals conditioned on instructions (e.g. Variational Auto-Encoders, Generative Adversarial Networks). For any given instruction, agent can thus sample a set of matching goals. This results in behavioral diversity, a set of diverse behaviors matching any given instruction.

Contributions. This paper introduces a novel approach to the problem of language grounding in RL agents: languageconditioned goal generation. This leverages the capability of language to represent abstract goals and to gener-alize, while avoiding the lack of behavioral diversity and the dependence to language inputs usually associated with language-conditioned agents. This paper presents a particular implementation of this approach and disentangles language acquisition from policy learning by assuming access to pre-trained goal-conditioned policies. Our policies are pre-trained to reach high-level configurations characterizing spatial relations between objects in the scene.

Methods

This paper presents a particular implementation of languageconditioned goal generation. Section 2.1 describes the learning environment and the pre-trained goal-conditioned policies. Assuming these, Section 2.2 describes an implementation of a language-conditioned goal generator and how it is used for language grounding.

Behavioral Policies Conditioned on Abstract Semantic Representations

The agents considered in this paper were pre-trained without language input by an algorithm presented in a companion paper.1

The Fetch Manipulate environment Agents evolve in the Fetch Manipulate environment: a robotic manipulation domain based on MUJOCO [START_REF] Todorov | A physics engine for model-based control[END_REF] and adapted from the Fetch tasks [START_REF] Plappert | Multi-goal reinforcement learning: Challenging robotics environments and request for research[END_REF]. Agents are 4 DoF robotic arms that face 3 colored blocks on a table (see Figure 2). They are given innate representations called semantic configurations that characterize spatial relations between blocks. During sensorimotor training, agents discovered all reachable configurations in that space and learned to master them.

Semantic configurations. In contrast to traditional approaches, goals are not defined as particular targets for each block but as high-level semantic configurations. These configurations are based on two spatial predicates infants demonstrate early in their development [START_REF] Mandler | On the spatial foundations of the conceptual system and its enrichment[END_REF]: the close and the above binary predicates. These two predicates are applied to all permutations of object pairs, i.e. 6 permutations for the 3 objects we consider. Because the close predicate is order invariant, we only need to evaluate it on 3 object combinations. The above predicate being order dependent, we need all 6 permutations. The resulting binary vector of size 9 forms the semantic configuration. It represents the spatial relations between objects in the scene.

In the resulting semantic configuration space {0, 1} 9 , the agent can reach 35 physically valid configurations, including stacks of 2 or 3 blocks and pyramids. Supplementary

Section 5 provides formal definitions and properties of predicates and semantic configurations. Figure 2 displays visual representations of some example configurations.

Figure 2. Some semantic configurations in Fetch Manipulate.

Language-Conditioned Goal Generation

The language-conditioned goal generator, or language module, generates semantic configurations matching the agent's initial configuration and a sentence describing an expected transformation of a relation between a pair of objects. This section explains its training and use for language grounding.

Training the language module. The languageconditioned goal generator is implemented by a conditional Variational Auto-Encoder (C-VAE) [START_REF] Sohn | Learning structured output representation using deep conditional generative models[END_REF] trained in a supervised setting. The training data is collected via interactions between a trained agent and a social partner.

For each goal-directed trajectory the agent performs, the social partner provides the description of one of the resulting transformations in the object relations. The set of possible descriptions contains 102 sentences, each describing in a simplified language a positive or negative shift for each of the 9 predicates (e.g. get red above green).

NL descriptions are encoded via a recurrent network that is jointly trained with the C-VAE. Supplementary Section 6 provides the list of sentences and implementation details.

Language grounding. At test time, agents are instructed by one of the 102 sentences to transform a relation between objects. The trained language module acts as a translator: agent can sample a goal configuration matching their current state and instruction. This module effectively enables agents to ground NL in their internal semantic representations and set of sensorimotor skills. We consider three evaluation settings: 1) performing a single instruction; 2) performing a sequence of instructions; 3) performing a logical combination of instructions. As the agent can generate a set of goals matching any instruction, it can easily combine these sets to perform logical functions of instructions: and is an intersection, or is an union and not is the complement within the set of goals the agent discovered during sensorimotor training. Given sets of compatible goal configurations, agents can also try again: find other goal configurations that match the required instruction when previous attempts failed.

Experiments

We first train a language-conditioned goal generator from a training dataset D collected via interactions between the agent and a social partner (Section 2.2). For a given initial configuration and a given sentence, we want the language module to generate all compatible final configurations, and just these.

Language-conditioned goal generation performance.

To evaluate the language module, we construct a synthetic, oracle dataset O of triplets (c i , s, C f (c i , s)), where c i is the initial configuration, s is the sentence describing the expected transformation and C f (c i , s) is the set of all final configurations compatible with (c i , s). Note that, on average, C f in O contains 16.7 configurations, while the training dataset D only contains 3.4 (20%). We are interested in two metrics: 1) The Compatibility Probability (CP) is the probability that a goal sampled from the generator belongs to C f ; 2) The Coverage (Cov) is the size of the intersection between C f and the set resulting from sampling the language-conditioned generator 100 times. We compute these metrics on 5 different sets of input pairs (c i , s), each calling for a different type of generalization:

1. Pairs found in D, except pairs removed to form the following test sets. This calls for the extrapolation of known initialization-effect pairs (c i , s) to new final configurations c f (D contains only 20% of C f on average). 2. Pairs that were removed from D, calling for a recombination of known effects s on known c i . 3. Pairs for which the c i was entirely removed from D.

This calls for the transfer of known effects s on unknown c i . 4. Pairs for which the s was entirely removed from D.

This calls for generalization in the language space, to generalize unknown effects s from related sentences and transpose this to known c i . 5. Pairs for which both the c i and the s were entirely removed from D. This calls for the generalizations 3 and 4 combined.

Our language module demonstrates these 5 types of generalization (see Table 1). Agents can generate goals from situations they never encountered (Test 3). They can generalize the meaning of sentences they never heard (Test 4) and even apply the latter to unknown situations (Test 5). We detail the testing sets in Supplementary Section 6. Grounding language in sensorimotor behavior. We investigate how the language module interacts with the sensorimotor skills of the agent. We consider three evaluation settings. In the transition setup, we look at the average success rate of the agent when asked to perform the 102 instructions 5 times each, resetting the environment each time. In the expression setup, we evaluate the agent on 500 randomly generated logical functions of sentences. In both setups, we give the agent 5 attempts, enabling it to resample new compatible goals when the previous failed (without reset). Success rates after 1 (SR1) and 5 (SR5) attempts are reported in Table 2. In the sequence setup, we ask the agent to execute 20 random sequences of instructions without reset and report the average number of successes before the agent fails: N s = 14.9 ± 5.7 (std). The 10 RL agents are evaluated with the 10 C-VAE models evaluated above. These results show that the language module efficiently implements language grounding. Agents achieve instructed transitions almost all the time, resampling alternative goals when previous ones failed. They only fail when a previous trajectory kicked the blocks out of reach.

Discussion

This paper introduces language-conditioned goal generation: a new approach to the problem of language grounding in RL agents. It shows the following advantages over traditional language-conditioned RL agents:

• Decoupled language grounding. Our approach enables agents to decouple language acquisition from sensorimotor learning, as it is observed in infants [START_REF] Wood | The role of tutoring in problem solving[END_REF]. However, nothing in the architecture prevents language from being grounded during sensorimotor learning. This would result in "overlapping waves" of sensorimotor and linguistic development [START_REF] Siegler | Emerging minds: The process of change in children's thinking[END_REF]. • Behavioral diversity. The language module generates a diversity of goals matching any instruction (see Coverage metrics in Table 1). This results in a behav-ioral diversity that language-conditioned agents cannot demonstrate. Indeed, while a language-conditioned agent trained on put red close_to green would only push the red block towards the green one, our agent can generate many matching goal configurations. It could build a pyramid, make a blue-green-red pile or target a dozen other compatible configurations. • Trying again. Generating a diversity of goals matching any instruction enables agents to try again: to find alternative approaches to satisfy a same instruction when previous attempts failed. Table 2 shows that benefiting from several attempts significantly improves the chances of success. This could also improve robustness to perturbed environments where former successful behaviors fail. • Logical expressions. Generating sets of compatible goals makes it easy to scale language understanding to any logical combination of instructions. This cannot be achieved with language-conditioned policies.

One strength of the language-conditioned approach is their capability of systematic generalization [START_REF] Hermann | Grounded language learning in a simulated 3D world[END_REF][START_REF] Bahdanau | Learning to understand goal specifications by modelling reward[END_REF][START_REF] Hill | Emergent systematic generalization in a situated agent[END_REF]Colas et al., 2020b). Our language module also seems to demonstrate generalization abilities: agents can generate goals from situations they never encountered (Type 3). They can generalize the meaning of sentences they never heard (Type 4) and even apply the latter to unknown situations (Type 5). Type 4, especially, involves recombinations of action verbs and attributes similar to [START_REF] Hill | Emergent systematic generalization in a situated agent[END_REF]; Colas et al. (2020b).

C-VAE were already used to generate goals matching initial states in [START_REF] Nair | Contextual imagined goals for self-supervised robotic learning[END_REF]. However, the additional condition on instructions brings the benefits of language use: 1) the representation of abstract goals; 2) the systematic generalization capacities. In addition instructions enable agents to control their goal generation. 2019), we believe our languageconditioned goal generator could be trained to generate imaged-based goals.

Further work could investigate the extension of languageconditioned goal generator to diverse goal-conditioned settings such as, for example, Quality-Diversity algorithms. These algorithms train populations of diverse and highperforming solutions to a problem [START_REF] Cully | Quality and diversity optimization: A unifying modular framework[END_REF]. Each solution is associated with its behavioral characterization: a low-dimensional description of its behavior. Our language-conditioned goal generator could be used to map language instructions to that behavioral space, enabling a language-based control of these diversity-seeking evolutionary algorithms [START_REF] Mouret | Illuminating search spaces by mapping elites[END_REF][START_REF] Lehman | Evolving a diversity of virtual creatures through novelty search and local competition[END_REF][START_REF] Conti | Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents[END_REF]Colas et al., 2020a).

Supplementary Material

This supplementary material includes:

• Section 5: a formal definition of semantic configurations and the definiton of the semantic configurations used in the Fetch Manipulate environment. • Section 6: further details about the training of our languageconditioned goal generation module.

Formal Definition of Semantic Configurations

Semantic configurations are based on a collection of formal systems known as predicate logic or first-order logic. Examples of binary predicates. We consider the objects o1 and o2.

• The expression "o1 is close to o2" describes the predicate close evaluated on (o1, o2).

• The expression "o2 is above o1" describes the predicate above evaluated on (o2, o1).

Semantic mapping functions. To achieve symbol grounding into non-symbolic sensorimotor interactions using predicates, we define a semantic mapping function f associated with the binary predicate p as the probability that p is true given the states of the considered objects. Formally, if we consider the objects oi, oj and their respective states si, sj, then: f (si, sj) = P (p(oi, oj) | si, sj). This paper assumes oracle deterministic semantic mapping functions, i.e. f is a Boolean function in {0, 1}. Practically, we hardcode a function, assumed internal to the agent, that uses predefined fixed thresholds to determine whether a predicate is true or false given the states of the considered objects. For example, for the close predicate, it outputs 1 if and only if the Euclidean distance between the two considered objects is below a defined threshold. For the sake of simplicity, we omit the word deterministic.

Symmetry and asymmetry. Consider a finite set of objects O = {o1, o2, ..., oM } and a binary predicate p. The predicate p is said to be symmetric if and only if, for any ordered pair of objects (oi, oj) ∈ O 2 , "oi r oj" and "oj r oi" are equivalent. As a result, the corresponding semantic mapping function f needs to be symmetric, i.e. f (oi, oj) = f (oj, oi). The predicate p is said to be asymmetric iff, for any ordered pair (oi, oj) ∈ O 2 , "oi r oj" implies not "oj r oi".

Examples. We consider the objects o1 and o2.

• close is symmetric: "o1 is close to o2" ⇔ "o2 is close to o1".

The corresponding semantic mapping function is based on the Euclidean distance, which is symmetric.

• above is asymmetric: "o1 is above o2" ⇒ not "o2 is above o1". The corresponding semantic mapping function evaluates the sign of the difference of the object Z-axis coordinates.

Effective number of predicate relations. Consider a finite set of M objects O = {o1, o2, ..., oM } and a binary predicate p.

• If p is not symmetric, then the effective number of relations Kp that can be described without redundancy is equal to the number of permutations of 2 objects among M , i.e. Kp = AM,2 = M (M -1).

• If p is symmetric, then the effective number of relations Kp is equal to the number of combinations of 2 objects among

M , i.e. Kp = M 2 = M (M -1) 2 .
Semantic configurations based on spatial relations. Let (pi) i∈[1..P ] be a list of P binary predicates. The concatenation of the evaluations of the semantic mapping functions fi on the Kpi pairs of objects forms a semantic configuration. It is an abstract representation of a scene which characterizes all relations defined by the (pi) predicates among the M objects. This defines a binary semantic configuration space Cp = {0, 1} Kc , where Kc = P i=1 Kp i . If any world configuration can be mapped to Cp, not all configurations are reachable (e.g. o1 cannot be above and below o2 at the same time).

Semantic representation space in Fetch Manipulate. In the Fetch Manipulate environment, we restrict semantic representations to the use of the close and above binary predicates applied on M = 3 objects. The resulting semantic configurations are formed by: cp = [c(o1, o2), c(o1, o3), c(o2, o3), a(o1, o2), a(o2, o1), a(o1, o3), a(o3, o1), a(o2, o3), a(o3, o2)],

where c() and a() refer to the close and above predicates respectively and (o1, o2, o3) are the red, green and blue blocks respectively.

Language-Conditioned Goal Generator Training

We use a conditional Variational Auto-Encoder (C-VAE) [START_REF] Sohn | Learning structured output representation using deep conditional generative models[END_REF]. Conditioned on the initial configuration and a sentence describing the expected transformation of one object relation, it generates compatible goal configurations. After the first phase of goal-directed sensorimotor training, the agent interacts with a hard-coded social partner as described in Main Section 2.2. From these interactions, we obtain a dataset of 5000 triplets: initial configuration, final configuration and sentence describing one change of predicate from the initial to the final configuration. The list of sentences used by the synthetic social partner are provided in Table 3. Note that red, green and blue refer to objects o1, o2, o3 respectively.

Content of test sets. We describe the 5 test sets:

1. Test set 1 is made of input pairs (ci, s) from the training set, but tests the coverage of all compatible final configurations

  Because image-based goal generation was shown to work in Nair et al. (2018); Pong et al. (2019); Nair et al. (

Policy Langage Encoder instruction state action goal (language embedding) language embedding Policy Langage Encoder Any source of goals Any source of goals instruction state initial state noise action Goal Generator language-agnostic goal goal OR Language-Conditioned Policy Policy state action

Table 1 .

 1 Language module average metrics over 10 seeds. Std is below 0.07 for Cov and 0.06 for CP.

	Metr. Test 1 Test 2 Test 3 Test 4 Test 5
	CP	0.93	0.94	0.95	0.90	0.92
	Cov	0.97	0.93	0.98	0.99	0.98

Table 2 .

 2 Language grounding performance metrics over 10 seeds (av±std).

	Metr. Transition Expression
	SR1	0.89 ± 0.05 0.74 ± 0.08
	SR5	0.99 ± 0.01 0.94 ± 0.06

  They use k-ary relations to describe possible connections between k quantified variables. This paper focuses on spatial binary predicates characterizing spatial relations between pairs of physical objects. We provide formal definitions, properties and examples below.Binary predicates Consider a finite set of objects O = {o1, o2, ..., oM }. A binary predicate p associated with a semantic relation r is an expression that takes as input any ordered pair of objects (oi, oj) ∈ O 2 . p(oi, oj) is said true if and only if "oi r oj" is verified. For simplicity, we refer to p and r interchangeably.

This paper is not cited here to respect anonymity. It will be after reviews.

Acknowledgments

Cédric Colas is partly funded by the French Ministère des Armées -Direction Générale de l'Armement.

C f , 80% of which are not found in the training set. In that sense, it is partly a test set.

2. Test set 2 contains two input pairs: {[0 1 0 0 0 0 0 0 0], put blue close_to green} and {[0 0 1 0 0 0 0 0 0], put green below red} corresponding to 7 and 24 compatible final configurations respectively.

3. Test set 3 corresponds to all pairs including the initial configuration ci = [1 1 0 0 0 0 0 0 0] (29 pairs), with an average of 13 compatible final configurations.

4. Test set 4 corresponds to all pairs including one of the sentences put green on_top_of red and put blue far_from red, i.e. 20 pairs with an average of 9.5 compatible final configurations.

5. Test set 5 is all pairs that include both the initial configuration of test set 3 and one of the sentences of test set 4, i.e. 2 pairs with 6 and 13 compatible goals respectively. Note that pairs of set 5 are removed from sets 3 and 4. Testing on logical expressions of instructions. To evaluate our agents on logical functions of instructions, we generate three types of expressions:

1. 100 instructions of the form "A and B" where A and B are basic instructions corresponding to shifts of the form above 0 → 1 (see Table 3). These intersections correspond to stacks of 3 or pyramids.

2. 200 instructions of the form "A and B" where A and B are above and close instructions respectively. B can be replaced by "not B" with probability 0.5. Here also, any instruction can be replaced by its negation with probability 0.5.

Hyperparameters. The encoder is a fully-connected neural network with two layers of size 128 and ReLU activations. It takes as input the concatenation of the final binary configuration and its two conditions: the initial binary configuration and an embedding of the NL sentence. The NL sentence is embedded with an recurrent network with embedding size 100, tanh non-linearities and biases. The encoder outputs the mean and log-variance of the latent distribution of size 27. The decoder is also a fully-connected network with two hidden layers of size 128 and ReLU activations. It takes as input the latent code z and the same conditions as the encoder. As it generates binary vectors, the last layer uses sigmoid activations. We train the architecture with a mixture of Kullback-Leibler divergence loss (KDloss) w.r.t a standard Gaussian prior and a binary Cross-Entropy loss (BCEloss). The combined loss is loss = BCEloss + β × KDloss with β = 0.6. We use an Adam optimizer with a learning rate of 5 × 10 -4 , a batch size of 128 and optimize for 150 epochs. As training is fast (≈ 2 min on a single cpu), we conducted a quick hyperparameter search over β, layer sizes, learning rates and latent sizes (see Table 4). We found robust results for various layer sizes, various β below 1 and latent sizes above 9. [9,18,27]