Geoffrey Cideron

Mathieu Seurin

Florian Strub

Olivier Pietquin

HIGhER: Improving instruction following with Hindsight Generation for Experience Replay

Keywords: Reinforcement Learning, Representation Learning, Natural Language Processing

Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay approach to the language-conditioned policy setting. Whenever the agent does not fulfill its instruction, HIGhER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, HIGhER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We show the efficiency of our approach in the BabyAI environment, and demonstrate how it complements other instruction following methods.

I. INTRODUCTION

Language has slowly evolved to communicate intents, to state objectives, or to describe complex situations [START_REF] Kirby | Compression and communication in the cultural evolution of linguistic structure[END_REF]. It conveys information compactly by relying on composition and highlighting salient facts. As language can express a vast diversity of goals and situations, it may help conditioning the training of interactive agents over heterogeneous and composite tasks [START_REF] Luketina | A Survey of Reinforcement Learning Informed by Natural Language[END_REF] and help transfer [START_REF] Narasimhan | Grounding language for transfer in deep reinforcement learning[END_REF]. Unfortunately, conditioning a policy on language also entails a supplementary difficulty as the agent needs to understand linguistic cues to alter its behavior. The agent thus needs to ground its language understanding by relating the words to its observations, actions, and rewards before being able to leverage the language structure [START_REF] Kiela | Virtual embodiment: A scalable long-term strategy for artificial intelligence research[END_REF][START_REF] Moritz | Grounded language learning in a simulated 3d world[END_REF]. Once the linguistic symbols are grounded, the agent may then take advantage of language compositionality to condition its policy on new goals.

In this work, we use instruction following as a natural testbed to examine this question [START_REF] Tellex | Understanding natural language commands for robotic navigation and mobile manipulation[END_REF][START_REF] David | Learning to interpret natural language navigation instructions from observations[END_REF][START_REF] Artzi | Weakly supervised learning of semantic parsers for mapping instructions to actions[END_REF][START_REF] Luketina | A Survey of Reinforcement Learning Informed by Natural Language[END_REF][START_REF] Moritz | Learning to follow directions in street view[END_REF]. In this * Those authors contributed equally setting, the agent is given a text description of its goal (e.g. "pick the red ball") and is rewarded when achieving it. The agent has thus to visually ground the language, i.e., linking and disentangling visual attributes (shape, color) from language description ("ball", "red") by using rewards to condition its policy toward task completion. On one side, the language compositionality allows for a high number of goals, and offers generalization opportunities; but on the other side, it dramatically complexifies the policy search space. Besides, instruction following is a notoriously hard RL problem as the training signal is very sparse since the agent is only rewarded over task completion. In practice, the navigation and language grounding problems are often circumvented by warm-starting the policy with labeled trajectories [START_REF] Zang | Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation[END_REF][START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF]. Although scalable, these approaches require numerous human demonstrations, whereas we here want to jointly learn the navigation policy and language understanding from scratch. In a seminal work, [START_REF] Moritz | Grounded language learning in a simulated 3d world[END_REF] successfully ground language instructions, but the authors used unsupervised losses and heavy curriculum to handle the sparse reward challenge.

In this paper, we take advantage of language compositionality to tackle the lack of reward signals. To do so, we extend Hindsight Experience Replay (HER) to language goals [START_REF] Andrychowicz | Hindsight experience replay[END_REF]. HER originally deals with the sparse reward problems in spatial scenario; it relabels unsuccessful trajectories into successful ones by redefining the policy goal a posteriori. As a result, HER creates additional episodes with positive rewards and a more diverse set of goals. Unfortunately, this approach cannot be directly applied when dealing with linguistic goals. As HER requires a mapping between the agent trajectory and the goal to substitute, it requires expert supervision to describe failed episodes with words. Hence, this mapping should either be handcrafted with synthetic bots [START_REF] Chan | ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning[END_REF], or be learned from human demonstrations, which would both limit HER generality. More generally, language adds a level of semantics, which allows generating textual objective that could not be encoded by simple spatial observations as in regular HER, e.g., "fetch a ball that is not blue" or "pick any red object".

In this work, we introduce Hindsight Generation for Experience Replay (HIGhER), a training procedure where the agent jointly learns the language-goal mapping and the navigation policy by solely interacting with the environment illustrated Fig. 1. Upon positive trajectory, the agent trajectory is added to the RL replay buffer and the goal mapper dataset. Upon failed trajectory, the goal mapper is used to relabel the episode, and both trajectories are appended to the replay buffer. In the original HER paper, the mapping function is bypassed since they are dealing with spatial goals, and therefore, vanilla HER cannot be applied without external expert.

in Figure 1. HIGhER leverages positive trajectories to learn a mapping function, and HIGhER then tackles the sparse reward problem by relabeling language goals upon negative trajectories in a HER fashion. We evaluate our method on the BabyAI world [START_REF] Chevalier-Boisvert | BabyAI: First Steps Towards Grounded Language Learning With a Human In the Loop[END_REF], showing a clear improvement over RL baselines while highlighting the robustness of HIGhER to noise.

II. BACKGROUND AND NOTATION

In reinforcement learning, an agent interacts with the environment to maximize its cumulative reward [START_REF] Richard | Reinforcement learning: An introduction[END_REF]. At each time step t, the agent is in a state s t ∈ S, where it selects an action a t ∈ A according its policy π : S → A. It then receives a reward r t from the environment's reward function r : S × A → R and moves to the next state s t+1 with probability p(s t+1 |s t , a t ). The quality of the policy is assessed by the Q-function defined by Q π (s, a) = E π [ t γ t r(s t , a t )|s 0 = s, a 0 = a] for all (s, a) where γ ∈ [0, 1] is the discount factor. We define the optimal Q-value as Q * (s, a) = max π Q π (s, a), from which the optimal policy π * is derived. We here use Deep Q-learning (DQN) to evaluate the optimal Q-function with neural networks and perform offpolicy updates by sampling transitions (s t , a t , r t , s t+1 ) from a replay buffer [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF].

In this article, we augment our environment with a goal space G which defines a new reward function r : S ×A×G → R and policy π : S × G → A by conditioning them on a goal descriptor g ∈ G. Similarly, the Q-function is also conditioned on the goal, and it is referred to as Universal Value Function Approximator (UVFA) [START_REF] Schaul | Universal value function approximators[END_REF]. This approach allows learning holistic policies that generalize over goals in addition to states at the expense of complexifying the training process. In this paper, we explore how language can be used for structuring the goal space, and how language composition eases generalization over unseen scenarios in a UVFA setting.

a) Hindsight Experience Replay (HER): [START_REF] Andrychowicz | Hindsight experience replay[END_REF] is designed to increase the sample efficiency of off-policy RL algorithms such as DQN in the goal-conditioning setting. It reduces the sparse reward problem by taking advantage of failed trajectories, relabelling them with new goals. An expert then assigns the goal that was achieved by the agent when performing its trajectory, before updating the agent memory replay buffer with an additional positive trajectory.

Formally, HER assumes the existence of a predicate f : S × G → {0, 1} which encodes whether the agent in a state s satisfies the goal f (s, g) = 1, and defines the reward function r(s t , a, g) = f (s t+1 , g). At the beginning of an episode, a goal g is drawn from the space G of goals. At each time step t, the transition (s t , a t , r t , s t+1 , g) is stored in the DQN replay buffer, and at the end of an unsuccessful episode, an expert provides an additional goal g that matches the trajectory. New transitions (s t , a t , r t , s t+1 , g ) are thus added to the replay buffer for each time step t, where r = r(s t , a t , s t+1 , g ). DQN update rule remains identical to [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], transitions are sampled from the replay buffer, and the network is updated using one step td-error minimization.

HER assumes that a mapping m between states s and goals g is given. In the original paper, this requirement is not restrictive as the goal space is a subset of the state space. Thus, the mapping m is straightforward since any state along the trajectory can be used as a substitution goal. In the general case, the goal space differs from the state space, and the mapping function is generally unknown. In the instruction following setting, there is no obvious mapping from visual states to linguistic instructions. It thus requires expert intervention to provide a new language goal given the trajectory, which drastically reduces the interest of HER. Therefore, we here explore how to learn this mapping without any form of expert knowledge nor supervision.

III. HINDSIGHT GENERATION FOR EXPERIENCE REPLAY

Hindsight Generation for Experience Replay (HIGhER) aims to learn a mapping from past experiences that relates a trajectory to a goal in order to apply HER, even when no expert are available. The mapping function relabels unsuccessful trajectories by predicting a substitute goal ĝ as an expert would do. The transitions are then appended to the replay buffer. This mapping learning is performed alongside agent policy training.

Besides, we wish to discard any form of expert supervision to learn this mapping as it would reduce the practicability of the approach. Therefore, the core idea is to use environment signals to retrieve training mapping pairs. Instinctively, in the sparse reward setting, trajectories with positive rewards encode ground-truth mapping pairs, while trajectories with negative rewards are mismatched pairs. These cues are thus collected to train the mapping function for HIGhER in a supervised fashion. We emphasize that such signals are inherent to the environment, and an external expert does not provide them. In the following, we only keep positive pairs in order to train a discriminative mapping model.

Formally, HIGhER is composed of a dataset D of s, g pairs, a replay buffer R and a parametrized mapping model m w . For each episode, a goal g is picked, and the agent generates transitions (s t , a t , r t , s t+1 , g) that are appended to the replay buffer R. The Q-function parameters are updated with an off-policy algorithm by sampling minibatches from D. Upon episode termination, if the goal is achieved, i.e. f (s T , g) = 1, the s T , g pair is appended to the dataset D. If the goal is not achieved, a substitute goal is sampled from the mapping model1 m w (s T ) = ĝ and the additional transitions {(s t , a t , r t , s t+1 , ĝ )} T t=0 are added to the replay buffer. At regular intervals, the mapping model m w is optimized to predict the goal g given the trajectory τ by sampling mini-batches from D. Noticeably, HIGhER can be extended to partially observable environments by replacing the predicate function f (s, g) by f (τ, g), i.e., the completion of a goal depends on the full trajectory rather than one state. Although we assess HIGhER in the instruction following setting, the proposed procedure can be extended to any other goal modalities.

IV. EXPERIMENTS A. Experimental Setting a) Environment: We experiment our approach on a visual domain called Minigrid [START_REF] Chevalier-Boisvert | BabyAI: First Steps Towards Grounded Language Learning With a Human In the Loop[END_REF]. This environment offers a variety of instruction-following tasks using a synthetic language for grounded language learning. We use a 10x10 grid with 10 objects randomly located in the room. Each object has 4 attributes (shade, size, color, and type) inducing a total of 300 different objects (240 objects are used for training, 60 for testing). To the best of our knowledge, the number of different objects and its diversity is greater than concurrent works ( [START_REF] Singh | Gated-attention architectures for task-oriented language grounding[END_REF] used 55 train instructions and 15 test instructions and [START_REF] Hill | Understanding grounded language learning agents[END_REF] has a total of 40 different objects). The agent has four actions {forward, left, right, pick}, and it can only see the 7x7 grid in front of it. For each episode, one object's attribute is randomly picked as a goal, and the text generator translates it in synthetic language. , e.g., "Fetch a tiny light blue ball." The agent is rewarded when picking one object matching the goal description, which ends the episode; otherwise, the episode stops after 40 steps or after taking an incorrect object. b) Task Complexity: It is important to underline the complexity of this task. To get rewards over multiple episodes, the agent must learn to navigate and inspect objects in the room while simultaneously learning the meaning of each word and how they relate to visual characteristics. The burden comes from reward sparsity as the replay buffer is filled with unsuccessful trajectories RL fails to learn. Alleviating this problem is essential and minigrid is an excellent testbed to assess algorithmic performances as an agent deals with partial observability, visual representation learning, and language grounding only from sparse rewards signal. c) Models: In this experiment, HIGhER is composed of two separate models. The instruction generator is a neural network outputting a sequence of words given the final state of a trajectory. It is trained by gradient descent using a cross-entropy loss on the dataset D collected as described in section III. We train a DQN network following [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] with a dueling head [START_REF] Wang | Dueling Network Architectures for Deep Reinforcement Learning[END_REF], double Q-learning [START_REF] Hado Van Hasselt | Deep reinforcement learning with double Q-Learning[END_REF], and a prioritized replay buffer [START_REF] Schaul | Prioritized experience replay[END_REF] over trajectories. The network receives a tuple < s, g > as input and output an action corresponding to the argmax over states-actions values Q(s, a, g). We use -greedy exploration with decaying .

Algorithm 1: Hindsight Generation for Experience Replay (HIGhER)

Given:

• an off-policy RL algorithm (e.g. DQN) A

• a reward function r : S × A × G → R.

• a language score (e.g. parser accuracy, BLEU etc.) Replace g by ĝ in the transitions of the last episode and set r = r(st, at, ĝ ).

B. Building Intuition

This section examines the feasibility of HIGhER by analysing two potential issues. We first show that HER is robust to a noisy mapping function (or partially incorrect goals), we then estimate the accuracy and generalisation performance of the instruction generator.

1) Noisy instruction generator and HER: We investigate how a noisy mapping m affects performance compared to a perfect mapping. As the learned instruction generator is likely to be imperfect, it is crucial to assess how a noisy mapping may alter the training of the agent. To do so, we train an agent with HER and a synthetic bot to relabel unsuccessful trajectories. We then inject noise in our mapping where each attribute has a fixed probability p to be swapped, e.g. color blue may be changed to green. For example, when p = 0.2, the probability of having the whole instruction correct is 0.8 4 ≈ 0.4. The resulting agent performance is depicted in Figure 2 (left).

The agent performs 80% as well as an agent with perfect expert feedback even when the mapping function has a 50% noise-ratio per attribute. Surprisingly, even highly noisy mappers, with a 80% noise-ratio, still provides an improvement over vanilla DQN-agents. Hence, HER can be applied even when relabelling trajectories with partially correct goals.

We also examine whether this robustness may be induced by the environment properties (e.g. attribute redundancy) rather than HER. We thus compute the number of discriminative features required to pick the correct object. On average, an object can be discriminated with 1.7 features in our setting -which eases the training, but any object shares at least one property with any other object 70% of the time -which tangles the training. Besides, the agent does not know which features are noisy or important. Thus, the agent still has to disentangle the instructions across trajectories in the replay buffer, and this process is still relatively robust to noise.

2) Learning an instruction generator: We briefly analyze the sample complexity and generalization properties of the instruction generator. If training the mapping function is more straightforward than learning the agent policy, then we can thus use it to speed up the navigation training.

We first split the set of missions G into two disjoint sets G train and G test . Although all object features are present in both sets, they contain dissimilar combinations of target objects. For instance, blue, dark, key, and large are individually present in instructions of G train and G test but the instruction to get a large dark blue key is only in G test . We therefore assess whether a basic compositionality is learned. In the following, we use train/split ratio of 80/20, i.e., 240 vs 60 goals.

We here observe than 1000 positive episodes are necessary to reach around 20% accuracy with our model, and 5000 pairs are enough to reach 70% accuracy. The instruction generator also correctly predicts unseen instructions even with fewer than 1000 samples and the accuracy gap between seen and unseen instructions slowly decrease during training, showing basic compositionality acquisition. As further discussed in section V, we here use a vanilla mapping architecture to assess the generality of our HIGhER, and more advanced architectures may drastically improve sample complexity [START_REF] Bahdanau | Systematic Generalization: What Is Required and Can It Be Learned?[END_REF].

C. HIGhER for instruction following

In the previous section, we observe that: (1) HER is robust to noisy relabeled goals, (2) an instructor generator requires few positive samples to learn basic language compositionality. We thus here combine those two properties to execute HIGhER, i.e. jointly learning the agent policy and language prediction in a online fashion for instruction following.

a) Baselines: We want to assess if the agent benefits from learning an instruction generator and using it to substitute goals as done in HER. We denote this approach DQN+HIGhER. We compare our approach to DQN without goal substitution (called DQN) and DQN with goal substitution from a perfect mapping provided by an external expert (called DQN+HER) available in the BabyAI environment. We emphasize again that it is impossible to have an external expert to apply HER in the general case. Therefore, DQN is a lower bound that we expect to outperform, whereas DQN+HER is the upper bound as the learned mapping can not outperform the expert. Note that we only start using the parametrized mapping function after collecting 1000 positive trajectories, which is around 18% validation accuracy. Finally, we compute an additional DQN baseline denoted DQN+reward: we reward the agent with 0.25 for each matching properties when picking a object given an instruction. It enforces a hand-crafted curriculum and dramatically reduces the reward sparsity, which gives a different perspective on the current task difficulty.

b) Results: In Figure 3 (left), we show the success rate of the benchmarked algorithms per environment steps. We first observe that DQN does not manage to learn a good policy, and its performance remains close to that of a random policy. On the other side, DQN+HER and DQN+reward quickly manage to pick the correct object 40% of the time. Finally, DQN+HIGhER sees its success rates increasing as soon as we use the mapping function, to rapidly perform nearly as well as DQN+HER. Figure 3 (right) shows the performance accuracy of the mapping generator by environment steps. We observe a steady improvement of the accuracy during training before reaching 78% accuracy after 5M steps. In the end, DQN+HIGhER outperforms DQN by using the exact same amount of information, and even matches the conceptual upper bond computed by DQN+HER. Besides, HIGhER does not alter the optimal policy which can occur when reshaping the reward [START_REF] Andrew Y Ng | Policy invariance under reward transformations: Theory and application to reward shaping[END_REF]. As stated in paragraph IV-C0a, a gap remains between DQN+HER and HIGhER as the latter is building an approximate model of the instruction, thus sometimes failing to relabel correctly as pointed out in Figure 2 D. Discussion a) Improvements over DQN: As observed in the previous noisy-HER experiment, the policy success rate starts increasing even when the mapping accuracy is 20%, and DQN+HIGhER becomes nearly as good as DQN+HER despite having a maximum mapping accuracy of 78%. It demonstrates that DQN+HIGhER manages to trigger the policy learning by better leveraging environment signals compared to DQN. As the instruction generator focuses solely on grounding language, it quickly provides additional training signal to the agent, initiating the navigation learning process.

b) Generator Analysis: We observe that the number of positive trajectories needed to learn a non-random mapping m w is lower than the number of positive trajectories needed to obtain a valid policy with DQN (even after 5M environment steps the policy has 10% success rate). Noticeably, we artificially generate a dataset in subsubsection IV-B2 to train the instruction generator, whereas we follow the agent policy to collect the dataset, which is a more realistic setting. For instance, as the instructor generator is trained on a moving dataset, it could overfit to the first positive samples, but in practice it escapes from local minima and obtains a final high accuracy.

Different factors may also explain the learning speed discrepancy: supervised learning has less variance than reinforcement learning as it has no long-term dependency. The agent instructor generator can also rely on simpler neural architectures than the agent. Although HIGhER thus takes advantage of those training facilities to reward the agent ultimately.

c) Robustness: Finally, we observe a virtuous circle that arises. As soon as the mapping is correct, the agent success rate increases, initiating the synergy. The agent then provides additional ground-truth mapping pairs, which increases the mapping accuracy, which improves the quality of substitute goals, which increases the agent success rate further more. As a result, there is a natural synergy that occurs between language grounding and navigation policy as each module iteratively provides better training samples to the other model. If we ignore time-out trajectories, around 90% of the trajectories are negative at the beginning of the training. As soon as we start using the instruction generator, 40% the transitions are relabelled by the instructor generator, and 10% of the transitions belong to positive trajectories. As training goes, this ratio is slowly inverted, and after 5M steps, there is only 15% relabelled trajectories left while 60% are actual positive trajectories.

E. Limitations

Albeit generic, HIGhER also faces some inherent limitations. From a linguistic perspective, HIGhER cannot transcribe negative instructions (Do not pick the red ball), or alternatives (Pick the red ball or the blue key) in its current form. However, this problem could be alleviated by batching several Fig. 4. The instruction generator is triggered after collecting 0, 1000 and 2000 positive trajectories (i.e, approximately 0%, 20%, 50% accuracy). Even when the instruction generator is not accurate, the policy still makes steady progress and the final success rate is not impacted. Delaying the generator instructor does not provide additional benefit Fig. 5. Transition distributions in the replay buffer between successful, unsuccessful and relabeled trajectories. We remove time-out trajectories for clarity, which accounts for 54% of the transition in average (±3% over training).

trajectories with the same goal. Therefore, the model would potentially learn to factorize trajectories into a single language objective. On the policy side, HIGhER still requires a few trajectories to work, and it thus relies on the navigation policy. In other word, historical HER could be applied in the absence of reward signals, while HIGhER only alleviate the sparse reward problem by better leveraging successful trajectories. A natural improvement would be to couple HIGhER with other exploration methods, e.g, intrinsic motivation [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF] or DQN with human demonstration [START_REF] Hester | Deep q-learning from demonstrations[END_REF]. Finally, under-trained goal generators might hurt the training in some environments although we did not observe it in our setting as shown in Figure 4. However, a simple validation accuracy allows to circumvent this risk while activating the goal mapper (More details in algorithm 1). We emphasize again that the instruction generator can be triggered anytime to kick-start the learning as the it is independent of the agent.

V. RELATED WORK

Instruction following have recently drawn a lot of attention following the emergence of several 2D and 3D environments [START_REF] Chevalier-Boisvert | BabyAI: First Steps Towards Grounded Language Learning With a Human In the Loop[END_REF][START_REF] Brodeur | HoME: a Household Multimodal Environment[END_REF][START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF]. This section first provides an overview of the different approaches, i.e, fully-supervised agent, reward shaping, auxiliary losses, before exploring approaches related to HIGhER.

A. Vision and Language Navigation

Instruction following is sometimes coined as Vision and Language Navigation tasks in computer vision [START_REF] Anderson | Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments[END_REF][START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF]. Most strategies are based on imitation learning, relying on expert demonstrations and knowledge from the environment. For example, [START_REF] Zang | Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation[END_REF] relate instructions to an environment graph, requiring both demonstrations and high-level navigation information. Closer to our work, [START_REF] Fried | Speaker-follower models for visionand-language navigation[END_REF] also learns a navigation model and an instruction generator, but the latter is used to generate additional training data for the agent. The setup is hence fully supervised, and requires human demonstrations. These policies are sometimes finetuned to improve navigation abilities in unknown environments. Noticeably, [START_REF] Wang | Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation[END_REF] optimizes their agent to find the shortest path by leveraging language information. The agent learns an instruction generator, and they derive an intrinsic reward by aligning the generator predictions over the ground truth instructions. Those approaches complete long sequences of instructions in visually rich environments but they require a substantial amount of annotated data. In this paper, we intend to discard human supervision to explore learning synergies. Besides, we needed a synthetic environments with experts to evaluate HIGhER. Yet, HIGhER could be studied on natural and visually rich settings by warmstarting the instruction generator, and those two papers give a hint that HIGhER could scale up to larger environment. Recent works using pretrained language model [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF][START_REF] Hill | Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text[END_REF] could also complement HIGhER both in the generator and in the instruction understanding.

B. IRL for instruction following

[5] learn a mapping from <instruction, state> to a reward function. The method's aim is to substitute the environment's reward function when instructions can be satisfied by a great diversity of states, making hand-designing reward function tedious. Similarly, [START_REF] Fu | From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following[END_REF] directly learn a reward function and assess its transferability to new environments. Those methods are complementary to ours as they seek to transfer reward function to new environment and we are interested in reducing sample complexity.

C. Improving language compositionality

HIGhER heavily relies on leveraging the language structure in the instruction mapper toward initiating the learning synergy. For instance, [START_REF] Bahdanau | Systematic Generalization: What Is Required and Can It Be Learned?[END_REF] explore the generalization abilities of various neural architectures. They show that the sample efficiency of feature concatenation can be considerably improved by using feature-wise modulation [START_REF] Perez | Film: Visual reasoning with a general conditioning layer[END_REF], neural module networks [START_REF] Andreas | Neural module networks[END_REF] or compositional attention networks [START_REF] Drew | Compositional Attention Networks for Machine Reasoning[END_REF]. In this spirit, [START_REF] Bahdanau | Learning to Understand Goal Specifications by Modelling Reward[END_REF] take advantage of these architectures to quickly learn a dense reward model from a few human demonstrations in the instruction following setup. Differently, the instructor generator can also be fused with the agent model to act as an auxiliary loss, reducing further the sparse reward issue.

D. HER variants

HER has been extended to multiple settings since the original paper. These extensions deal with automatic curriculum learning [START_REF] Liu | Competitive experience replay[END_REF], dynamic goals [START_REF] Fang | DHER: Hindsight Experience Replay for Dynamic Goals[END_REF], or they adapt goal relabelling to policy gradient methods [START_REF] Rauber | Hindsight policy gradients[END_REF]. Closer to our work, [START_REF] Sahni | Visual Hindsight Experience Replay[END_REF] train a generative adversarial network to hallucinate visual near-goals state over failed trajectories. However, their method requires heavy engineering as visual goals are extremely complex to generate, and they lack the compact generalization opportunities inherent to language. [START_REF] Chan | ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning[END_REF] also studies HER in the language setting, but the authors only consider the context where a language expert is available.

E. Conditioned Language Policy

There have been other attempts to leverage language instruction to improve the agent policy. For instance, [START_REF] Jiang | Language as an abstraction for hierarchical deep reinforcement learning[END_REF] computes a high-level language policy to give textual instruction to a lowlevel policy, enforcing a hierarchical learning training. The authors manage to resolve complicated manipulating task by decomposing the action with language operation. The language mapper performs instruction retrieval into a predefined set of textual goals and yet, the low-level policy benefits from language compositionality and is able to generalize to unseen instructions, as mentioned by the authors. [START_REF]Guiding policies with language via meta-learning[END_REF] train an agent to refine its policy by collecting language corrections over multiple trajectories on the same task. While the authors focus their effort on integrating language cues, it could be promising to learn the correction function in a HIGhER fashion.

VI. CONCLUSION

We introduce Hindsight Generation for Experience Replay (HIGhER) as an extension to HER for language. We define a protocol to learn a mapping function to relabel unsuccessful trajectories with predicted consistent language instructions. We show that HIGhER nearly matches HER performances despite only relying on signals from the environment. We provide empirical evidence that HIGhER manages to alleviate the instruction following task by jointly learning language grounding and navigation policy with training synergies. HIGhER has mild underlying assumptions, and it does not require human data, making it valuable to complement to other instruction following methods. More generally, HIGhER can be extended to any goal modalities, and we expect similar procedures to emerge in other setting.

978- 1 -

 1 7281-2547-3/20/$31.00 ©2020 IEEE Text Goal: Pick the red ball

Fig. 2 .

 2 Fig. 2. Top: Agent performance with noisy mapping function. Bottom: Instruction generator accuracy over 5k pairs. Figures are averaged over 5 seeds and error bars shows one standard deviation.

Fig. 3 .

 3 Fig. 3. Left: learning curves for DQN, DQN+HER, DQN+HIGhER in a 10x10 gridworld with 10 objects with 4 attributes. The instruction generator is used after the vertical bar. Right: the mapping accuracy for the prediction of instructions. mw starts being trained after collecting 1000 positive trajectories. Results are averaged over 5 seeds with one standard deviation.

The mapping model can be utilized with an accuracy criterion over a validation set to avoid random goal sampling.

end

end

ACKNOWLEDGEMENTS

The authors would like to acknowledge the stimulating research environment of the SequeL INRIA Project-Team. Special thanks to Edouard Leurent, Piotr Mirowski and Florent Altché for fruitful discussions and reviewing the final manuscript.

We acknowledge the following agencies for research funding and computing support: Project BabyRobot (H2020-ICT-24-2015, grant agreement no.687831), and CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020.

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).


RECHERCHE EISAI

Nature de la demande : Projet de recherche ACRONYME et titre du projet : EISAI Ethical Information System based on Artificial Intelligence Auteurs : Saïd Assar, Professeur en Systèmes d'Information Christine Balagué, Professeur en Sciences de Gestion Loréa Baiada-Hirèche, Maître de Conférence en Sciences de Gestion Institut Mines-Télécom Business School, Laboratoire LITEM, Université Paris Saclay said.assar@imt-bs.eu christine.balague@imt-bs.eu lorea.baiada-hireche@imt-bs.eu

Contexte du projet de recherche :

Durant ces dernières années, l'explosion des données massives issues de technologies numériques et les capacités accrues de l'intelligence artificielle ont généré de nombreuses applications et services au sein des entreprises et plus largement des organisations (systèmes de recommandations, traitement automatique d'images, analyse de réseaux sociaux, modèles prédictifs, détection de fraude, traitement automatique de données vidéos pour le recrutement, prix dynamique, etc..). Demain, le développement de la 5G dans de nombreux pays va accroître la collecte des données et leur stockage dans le cloud, connecter des millions d'objets entre eux quasiment en temps réel, et créer de nouveaux usages. Cependant, ces évolutions souvent présentées comme innovations positives ont des impacts négatifs à la fois sur les individus et la société, montrés par de nombreux rapports et publications scientifiques. Dès 2016, le rapport « Big Data : a report on algorithmic systems, opportunity and civil rights » du bureau exécutif du président Obama montrait les effets de discriminations des systèmes algorithmiques sur les minorités et de l'impact des technologies sur les libertés individuelles. La même année, la FTC publiait le rapport « Big data : a tool for inclusion or exclusion ? » montrant que les technologies utilisant des données massives entraînaient une exclusion d'une partie de la population. La sociologue Virginia Eubanks en 2017 dans son ouvrage « Automating inequalities » décrit une automatisation des inégalités entre les individus par la très large diffusion des services numériques et des systèmes informatiques aux Etats-Unis. Plusieurs articles de recherche ont montré les impacts sociétaux de l'intelligence artificielle et des algorithmes : Eipstein (2015) a montré les effets de manipulation des moteurs de recherche sur les intentions de vote ; Basky et al. (2015) ont révélé dans la revue Science le phénomène des bulles filtrantes sur Facebook et la limite d'accès à une diversité d'opinion ; [START_REF] Lambrecht | Algorithmic Bias? An Empirical Study of Apparent Gender-Based Discrimination in the Display of STEM Career Ads[END_REF] ont mis en avant des effets de discrimination selon le genre dans les algorithmes de recrutement ; Obermayer et al. ( 2019) dans Science montrent la discrimination des populations noires dans les algorithmes de santé largement utilisés aux Etats-Unis pour l'accès aux soins. Cette prise en compte des impacts sociétaux négatifs des algorithmes de l'intelligence artificielle et du big data a entraîné ces dernières années plusieurs mobilisations et travaux internationaux sur la nécessité de développer des technologies responsables et éthiques. La déclaration de Montréal en décembre 2018, portée par l'Université de Montréal, porte dix principes pour une intelligence artificielle responsable et éthique. Le HLEG (High Level Expert Group) sur l'intelligence artificielle, mis en place par la commission européenne en 2018, a publié un rapport en 2019 sur les sept exigences pour une intelligence artificielle de confiance, et a mis en place des groupes pilotes dans les différents pays européens pour avancer sur ces sujets. Plus récemment, le 19 février, la commission européenne a publié le livre blanc « Intelligence artificielle : une approche européenne axée sur l'excellence et la confiance », prônant un cadre de régulation européen, une plus grande transparence des systèmes algorithmiques, et leur certification pour des usages dits sensibles (santé, recrutement, etc…). Selon Virginia Dignum (2019), « it is not an option to ignore our responsibility. AI systems are artefacts decided upon, designed, implemented and used by people. We, people, are responsible ». Notre projet de recherche se positionne dans ce courant international de nécessité de développer des technologies plus responsables et éthiques, qui est aussi l'objectif de la chaire Good in Tech.

Etat de l'art et question de recherche :

Les publications sur l'éthique de l'intelligence artificielle sont majoritairement dans les champs disciplinaires de l'informatique, la philosophie, le droit et la sociologie. Paradoxalement, les recherches en systèmes d'information, qui pourtant implémentent et souvent pilotent ces technologies au sein des organisations, restent peu nombreuses sur le sujet. Nous proposons avec ce projet de recherche de combler cet écart et d'étudier comment un projet en systèmes d'informations, utilisant l'intelligence artificielle, peut être mené de manière éthique. Les enjeux éthiques dans les systèmes d'information plus généralement ont fait l'objet de plusieurs publications : Cordoba (2006) mobilise les théories de Foucault, [START_REF] Leclercq-Vandelannoitte | From sovereign IT governance to liberal IT governmentality? A Foucauldian analogy[END_REF] reprennent également le concept de gouvernementalité de Foucault pour l'appliquer à la gouvernance des systèmes d'information. [START_REF] Leclercq-Vandelannoitte | An Ethical Perspective on Emerging Forms of Ubiquitous IT-Based Control[END_REF] présente une perspective éthique pour contrôler les systèmes d'information. [START_REF] Paraschakis | Towards an ethical recommendation framework[END_REF] propose un cadre éthique pour les systèmes de recommandations, mais plus globalement, peu de recherches portent sur l'éthique de systèmes d'information utilisant de l'intelligence artificielle. Or, l'un des challenges des systèmes d'information combinant big data, analytics et intelligence artificielle est de transformer ces données en connaissances et systèmes d'aide à la décision [START_REF] Kersting | From Big Data to Big Artificial Intelligence? Algorithmic Challenges and Opportunities of Big Data[END_REF]. Selon Zuboof (2015), nous vivons dans une civilisation de l'information, à la source d'un capitalisme de la surveillance fondé sur une logique d'accumulation de données visant à prédire et modifier le comportement humain comme un moyen de produire du revenu et de contrôler les marchés. Les enjeux éthiques des systèmes d'information utilisant le big data et l'intelligence artificielle sont donc majeurs.

Notre recherche vise d'abord à mieux comprendre les pratiques actuelles dans la prise en compte des questions éthiques dans les projets en systèmes d'information, et ensuite, à proposer un cadre conceptuel de ces pratiques qui servira à élaborer des recommandations sur les différentes dimensions dans un projet.


Mathieu Seurin
email: mathieu.seurin@inria.fr

Florian Strub

Philippe Preux

Olivier Pietquin

A Machine of Few Words Interactive Speaker Recognition with Reinforcement Learning

Keywords: active speaker recognition, reinforcement learning, deep learning, iterative representation learning

Speaker recognition is a well known and studied task in the speech processing domain. It has many applications, either for security or speaker adaptation of personal devices. In this paper, we present a new paradigm for automatic speaker recognition that we call Interactive Speaker Recognition (ISR). In this paradigm, the recognition system aims to incrementally build a representation of the speakers by requesting personalized utterances to be spoken in contrast to the standard text-dependent or text-independent schemes. To do so, we cast the speaker recognition task into a sequential decision-making problem that we solve with Reinforcement Learning. Using a standard dataset, we show that our method achieves excellent performance while using little speech signal amounts. This method could also be applied as an utterance selection mechanism for building speech synthesis systems.

Introduction

"Good words are worth much and cost little." -George Herbert In many speech-based applications, the construction of a speaker representation is required [START_REF] Irum | Speaker verification using deep neural networks: A[END_REF]. Automatic Speaker Recognition (ASR) and Speech Synthesis are some examples that have recently made steady progress by leveraging large amounts of data and neural networks. Text-To-Speech (TTS) convincingly encompasses someone's voice [START_REF] Shen | Natural tts synthesis by conditioning wavenet on mel spectrogram predictions[END_REF], and modern Speaker Recognition systems identify a speaker [START_REF] Irum | Speaker verification using deep neural networks: A[END_REF] among thousands of possible candidates with high accuracy. Speaker recognition systems are trained to extract speaker-specific features from speech signals, and during evaluation, test speaker utterances are compared with the already existing utterances. However, dozen of test recordings are necessary, limiting usage when interacting with humans. When identifying a speaker or trying to create a convincing TTS system, only some key features might be necessary, such as certain inflexions or speech mannerisms. In this paper, we build a speaker recognition system that can identify a speaker by using a limited and personalized number of words. Instead of relying on full test utterance across all individuals, we interact with the speakers to iteratively select the most discriminative words. Some pronunciation might be typical of certain speakers. For example, the phoneme 'r' might be pronounced differently depending on your accent. Thus starting with general phoneme and refining based on the utterances received could result in better recognition systems. More generally, a desirable feature of speaker recognition is to adapt its strategy to the current speaker as important features vary from person to person.

Here we propose to envision the problem of building a representation of the speaker as a sequential decision-making prob-lem. The system we want to develop will select words that a speaker must utter so that it can be recognized as fast as possible. Reinforcement learning (RL) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] is a framework to solve such sequential decision-making problems. It has been used in speech-based applications such as dialog [START_REF] Chandramohan | Optimizing spoken dialogue management with fitted value iteration[END_REF][START_REF] Strub | End-to-end optimization of goal-driven and visually grounded dialogue systems[END_REF] but not to the problem of speaker identification (note that [START_REF] Pietquin | Comparing asr modeling methods for spoken dialogue simulation and optimal strategy learning[END_REF] combines RL and phones similarity). We adapt a standard RL algorithm to interact with a speaker to maximize the identification accuracy given as little data as possible. After introducing an Interactive Speaker Recognition (ISR) game based on the TIMIT dataset to simulate the speaker ASR interaction, we show that the RL agent builds an iterative strategy that achieves better recognition performance while querying only a few words.

Our contributions are thus:

1. to introduce the Interactive Speaker Recognition as an interactive game between the SR module and a human (Sec. 2);

2. to formalize ISR as a Markov Decision Process [START_REF] Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF] so as to solve the problem with RL (Sec. 3);

3. to introduce a practical Deep RL ISR model, and train it on actual data (Sec. 4).

Finally, we test our method on the TIMIT dataset and show that ISR model successfully personalized the words it requests toward improving speaker identification, outperforming two noninteractive baselines (Sec. 5).

Interactive Speaker Recognition Game

In this paper, we aim to design an Interactive Speaker Recognition (ISR) module that identifies a speaker from a list of speakers only by requesting to utter a few user-specific words. To do so, we first formalize the ISR task as an interactive game involving the speaker and the ISR module. We then define the notation used to formally describe the game before detailing how we designed the ISR module.

Game Rules

To instantiate the ISR game, we first build a list of random individuals, or guests. Each guest is characterized by a few spoken sentences (enrolment phase), which act as their signature that we call voice print. In a second step, we label one of the guests as the target speaker that we aim to identify. Hence a game is defined by K guests characterized with K voice prints, and one of these guests is labeled as the speaker.

As the game starts, the K voice prints are provided to the ISR module, and it needs to identify the speaker among the guests. To do so, the ISR engine may interact with the speaker, but it can only request the speaker to utter T words within a predefined vocabulary list. At each turn of the game, the ISR module asks the speaker to say a word, the speaker pronounces Again, the ISR module may only request T words. Thus, it needs to carefully choose them to correctly identify the speaker.

Game notation

A game is composed of a list of K guests characterized by their voice print g = [g k ] K k=1 where g is a subset from a larger group of registered guests G of size G, and a predefined vocabulary V of size V . The ISR module aims at building a list of words w = [wt] T t=1 ∈ V to be uttered by the speaker. The uttered version of w is x = {xt} T t=1 , where xt is the representation of word wt pronounced by the speaker. Note that, for a given w, x differs from one speaker to another.

Modelling the Speaker Recognition Module

From a machine learning perspective, we aim to design an ISR module that actively builds an internal speaker representation to perform voice print classification. As further discussed in subsection 4.2, this setting differs from standard SR methods that rely on generic but often long utterances [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF]. In practice, we can split this task into two sub-modules: 1) an interactive module that queries the speaker to build the representation, and 2) a module that performs the voice print classification. In the following, we refer to these modules as enquirer and guesser.

Formally, the guesser must retrieve the speaker in a list of K guests characterized by their voice print g k ∈ g and a sequence of words x uttered by the speaker g * ∈ g. Thus, the guesser has to link the speaker's uttered words to the speaker's voice print. The enquirer must select the next word wt+1 ∈ V that should be pronounced by the speaker given a list of K guests and the sequence of t previously spoken words [x t ] t t =1 . Thus, the enquirer's goal is to pick the word that maximizes the guesser's success rate. Therefore, the ISR module first queries the speaker with the enquirer. Once the T words are collected, they are forwarded to the guesser to perform the speaker retrieval. In practice, this artificial split allows training the guesser with vanilla supervised learning, i.e., by randomly sampling words to retrieve speakers. The enquirer can hence be trained through reinforcement learning, as explained in the next section.

Speaker Recognition as a RL Problem

Reinforcement Learning addresses the problem of sequential decision making under uncertainty, where an agent interacts with the environment to maximize its cumulative reward [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. In this paper, we aim at maximizing the guesser success ratio by allowing the enquirer to interact with the speaker, which makes RL a natural fit to solve the task. In this section, we thus provide the necessary RL terminology before relating the enquirer to the RL setting and defining the optimization protocol.

Markov Decision Process

In RL, the environment is modeled as a Markov Decision Process (MDP), where the MDP is formally defined as a tuple {S, A, P, R, γ} [START_REF] Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF][START_REF] Howard | Dynamic Programming and Markov Processes[END_REF]. At each time step t, the agent is in a state st ∈ S, where it selects an action at ∈ A according to its policy π : S → A. The agent then moves to the state st+1 according to a transition kernel P and receives a reward rt = r(st, at) drawn from the environment's reward function R : S × A. In this paper, we define the enquirer as a parametric policy π θ where θ is a vector of neural network weights that will be learnt with RL. At the beginning of an episode, the initial state corresponds to the list of guests: s0 = {G}. At each time step t, the enquirer picks the action at by selecting the next word to utter wt, where wt ∼ π θ (st). The speaker then pronounces the word wt, which is processed to obtain xt before being appended to the state st+1 = st ∪ {xt}. After T words, the state sT = {g, x} is provided to the guesser. The enquirer is rewarded whenever the guesser identifies the speaker, i.e. r(st, at) = 0 if t < T and r(sT , aT

) = 1 [argmax k p(g k |s T )=g * ]
where 1 is the indicator function.

Enquirer optimization Process

In RL, policy search aims at learning the policy π θ * that maximizes the expected return by directly optimizing the policy parameters θ. More precisely, we search to maximize the mean value defined as J(θ) = Eπ θ T t=1 γ t-1 r(st, at) . To do so, the policy parameters are updated in the direction of the gradient of J(θ). In practice, direct approximation of ∇J(θ) may lead to destructively large policy updates, may converge to a poor deterministic policy at early training and it has a high variance. In this paper, we thus use the recent Proximal Policy Optimization approach (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. PPO clips the gradient estimate to have smooth policy updates, adds an entropy term to soften the policy distribution [START_REF] Geist | A theory of regularized markov decision processes[END_REF], and introduce a parametric baseline to reduce the gradient variance [START_REF] Schulman | Proximal policy optimization algorithms[END_REF][START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF].

Experimental Protocol

We first detail the data we used to create the ISR game before describing the speech processing phase. Finally, we present the neural training procedure.

Dataset

We build the ISR game using the TIMIT corpus [START_REF] Garofolo | Timit acoustic-phonetic continuous speech corpus[END_REF]. This dataset contains the voice recordings of 630 speakers with eight different accents. Each speaker uttered ten sentences, where two sentences are shared among all the speakers, and the eight others differ. Sentences are encoded as 16-bit, 16kHz waveforms. First, we define the ISR vocabulary by extracting the words of the two shared sentences, so the enquirer module may always request these words whatever the target speaker. In total, we obtained twenty distinct words such as dark, year, carry while dropping the uninformative specifier a. Second, we use the eight remaining sentences to build the speakers' voice print.

Audio Processing

Following [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF][START_REF] Snyder | Deep neural network embeddings for text-independent speaker verification[END_REF][START_REF] Snyder | Deep neural network-based speaker embeddings for end-to-end speaker verification[END_REF], we first down-sample the waveform to 8kHz before extracting the Mel Frequency Cepstral Coefficient (MFCC). We use MFCCs of dimension 20 with a frame-length of 25ms, mean-normalized over a sliding window of three seconds. We then process the MFCCs features through a pretrained X-Vector network to obtain a high quality voice embedding of fixed dimension 128, where the X-Vector network is trained on augmented Switchboard [START_REF] Godfrey | Switchboard: Telephone speech corpus for research and development[END_REF], Mixer 6 [START_REF] Chodroff | New release of mixer-6: Improved validity for phonetic study of speaker variation and identification[END_REF], and NIST SREs [START_REF] Doddington | The nist speaker recognition evaluation-overview, methodology, systems, results, perspective[END_REF] 1 . To get the spoken word representation (word that the enquirer will query), we split the two shared sentences into individual words by following the TIMIT word timestamps. We then extract the X-Vector of each word wt of every speaker k to obtain x k t . We compute the voice print by extracting the X-Vector of the eight remaining sentences before averaging them into a final vector of size 128 for each guest g k .

Speaker Recognition Neural Modules

We here describe the ISR training details and illustrate the neural architectures in Figure 2. Guesser. To model the guesser, we first model the guest by averaging the voice print into a single vector ĝ = 1 K g k . We then pool the X-Vectors with an attention layer conditioned on ĝ to get the guesser embedding x [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]:

et = M LP ([xt, ĝ]) ; α = sof tmax(e) ; x = t αtxt,
where [.,.] is the concatenation operator and MLP is a multilayer perceptron with one hidden layer of size 256. We concatenate the guesser embedding with the guest voice print before projecting them through a MLP of size 512. Finally, we use a softmax to estimate the probability p k of each guest to be the speaker, i.e. p(g k = g * |x, g) = sof tmax M LP ([g k , x]) . Both MLP have ReLU activations [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF] with a dropout ratio of 0.5% [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]. The guesser is trained by minimizing the crossentropy with ADAM [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], a batch size of 1024 and an initial learning rate of 3.10 -4 over 45k games with five random guests.

Enquirer. To model the enquirer, we first represent the pseudosequence of words by feeding the X-Vectors into a bidirectional LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] to get the word hidden state xt of dimension 2*128. Note that we use a start token for the first iteration. In parallel, we average the voice print into a single vector ḡ = 1 K g k to get the guest context. We then concatenate the word hidden state and the guest context before 1 available in kaldi library [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] at http://www.kaldi-asr. org/models/m3) processing them through a one-hidden-layer MLP of size 256 with ReLU. Finally, a softmax activation estimates the probability of requesting the speaker to utter wt+1 as the next word:

p(wt+1|xt, • • • , x1, g) = sof tmax M LP ([xt, ḡ]
) . The enquirer is trained by maximizing the reward encoded as the the guesser success ratio with PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. We use the ADAM optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a learning rate of 5e-3 and gradient clipping of 1 [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF]. We performed 80k episodes of length T = 3 steps and K = 5 random guests. When applying PPO, we use an entropy coefficient of 0.01, a PPO clipping of 0.2, a discount factor of 0.9, an advantage coefficient of 0.95, and we apply four training batches of size 512 every 1024 transitions.

Experiments

We run all experiments over five seeds, and report the mean and one-standard deviation when not specified otherwise.

Guesser Evaluation

In this section, we evaluate the guesser accuracy in different settings. As mentioned, we opt to request T = 3 words to identify the speaker among K = 5 guests. In this default setting, a random policy has a success ratio of 20%, whereas the neural model reaches 74.1% ± 0.2 on the test set. As the guesser is trained on random words, these scores may be seen as an ISR lower-bound for the enquirer, which would later refine the word selection toward improving the guesser success ratio. Thus, this setting shows an excellent ratio between task difficulty and guesser initial success, allowing to train the enquirer with a relatively dense reward signal.

Word Sweep. We assess the guesser quality to successfully perform speaker recognition when increasing the number of words T in Figure 3a. We observe that a single word only gives 50% speaker retrieval, but the accuracy keeps improving when requesting more words. Noticeably, collecting the full vocabulary only scores up to 97% accuracy.

Guest Sweep. We report the impact of the number of guests K in Figure 3b. The guesser accuracy quickly collapses when increasing the number of guests with K = 50 having a 46% success ratio. As the number of words remains small, the guesser experiences increasing difficulty in discriminating the guests. One way to address this problem would be to use a Probabilistic Linear Discriminant Analysis (PLDA) [START_REF] Ioffe | Probabilistic linear discriminant analysis[END_REF] to enforce a discriminative space and explicitly separate the guests based on their class. Baseline. We compare our approach to two baselines: a random policy, and a heuristic policy. As the name suggests, the random baseline picks T random words without replacement. To obtain a strong baseline, we pre-select words by taking advantage of the guesser model, where we value a sequence of words by computing the guesser accuracy over η = 20000 games. Optimally, we want to iterate over every tuple of words to retrieve the optimal set; yet, it is computationally intractable as it requires η * V T estimations. Therefore, we opt for a heuristic sampling mechanisms. We curated a list of the most discriminant words (words that increase globally the recognition scores) and sample among those instead of the whole list.

Results. In our default setting, the random baseline reaches 74.1% ± 0.2 speaker identification, and the heuristic baseline scores up to 85.1%. The RL enquirer obtains up to 88.6%±0.5, showing that it successfully leverages the guests' voice prints to refine its policy. We show the RL training in Figure 4. At early training, we observe that the ISR module still has high variance, and may behave randomly. However, RL enquirer steadily improves upon training, and it consistently outperforms the heuristic baseline.

Word Diversity. To verify whether the enquirer adapts its policy to the guests, we generate a game for every speaker in the test set, and collect the requested words. We then compute the overlap Ω between the tuple of words by estimating the aver-aged Jaccard-index [START_REF] Jaccard | Distribution de la flore alpine dans le bassin des dranses et dans quelques regions voisines[END_REF] of every pair of speakers as follow:

Ω = 1 N -1 n n N -1 i=1 N j=i J(w i , w j ) ; where J(A, B) = A ∩ B A ∪ B
where N is the number of speakers in the test set and w i is the word tuple of game i. Intuitively, the lower this number, the more diverse the policy, e.g, the deterministic policy have a Jaccard-index of 1. In the default setting, the random policy has an index of 0.14 while the RL agent has an index of 0.65. Thus, the requested words are indeed diverse.

Requesting Additional Words We here study the impact of increasing the number of words T requested by the enquirer (see Figure 3c for results). First, we observe that the ISR module manages to outperform the heuristic policy when requesting two to four words, showing that the interaction with the speaker is beneficial in the low data regime. This effect unsurprisingly diminishes when increasing the number of words. However, we noticed that the enquirer always outputs the same words when t = 1. It suggests that the model faces some difficulties contextualizing the guests' voice print before listening to the first speaker utterance. We assume that more advanced multimodal architecture, e.g., multimodal transformers [START_REF] Lu | Vilbert: Pretraining taskagnostic visiolinguistic representations for vision-and-language tasks[END_REF][START_REF] Tan | LXMERT: Learning cross-modality encoder representations from transformers[END_REF], may ease representation learning, further improving the ISR agent.

Conclusions and Future Directions

In this paper, we introduced the Interactive Speaker Recognition paradigm as an interactive game to improve speaker recognition accuracy while querying only a few words. We formalize it as a Markov Decision Process and train a neural model using Reinforcement Learning. We showed empirically that the ISR model successfully personalizes the words it requests to improve speaker identification, outperforming two non-interactive baselines. Future directions can include : scaling to bigger datasets [START_REF] Nagrani | Voxceleb: a largescale speaker identification dataset[END_REF][START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF], scaling up vocabulary size [START_REF] Dulac-Arnold | Deep reinforcement learning in large discrete action spaces[END_REF][START_REF] He | Deep reinforcement learning with a combinatorial action space for predicting popular reddit threads[END_REF][START_REF] Seurin | I'm sorry dave, i'm afraid i can't do that -deep q-learning from forbidden action[END_REF] Our protocol may go beyond speaker recognition. The model can be adapted to select speech segments in the context of Text-To-Speech training. Interactive querying may also prevent malicious voice generator usage by asking complex words to the generator in a speaker verification setting.

Figure 1 :

 1 Figure 1: Interactive Speaker Recognition game overview it, and the ISR engine updates its internal speaker representation, as detailed in subsection 4.3, before asking the next word.Again, the ISR module may only request T words. Thus, it needs to carefully choose them to correctly identify the speaker.

Figure 2 :

 2 Figure 2: (Left) The guesser must retrieve the speaker in the list of guests. (Right) The enquirer must select the next word to utter.

Figure 3 :

 3 Figure 3: (a-b) Guesser test accuracy, respectively varying the number of words (resp. guests) being used (c) Enquirer test accuracy varying the number of queried words. The RL enquirer outperforms the heuristic baseline when selecting a low number of words.

Figure 4 :

 4 Figure 4: Enquirer test accuracy averaged over 3 random seeds5.2. Enquirer EvaluationModel. As previously mentioned, the enquirer aims to find the best sequence of words w that maximizes the guesser accuracy by interacting with the speaker. At each time step, we thus select the word with the highest probability p(wt+1|xt, • • • , x1, g) according to the policy without replacement, i.e., the model never requests the same word twice.

Acknowledgements

We would like to thank Corentin Tallec for his helpful recommendations and the SequeL-Team for being a stimulating environment. We acknowledge the following agencies for research funding and computing support: Project BabyRobot (H2020-ICT-24-2015, grant agreement no.687831), and CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020


Sur le volet socio économique

Les plus : Création de richesse, reconnaissance de savoir faire, valorisation produits de terroir, entretien de paysages.

Les moins :

Conditions de travail difficiles, perte de revenu, mise en cause de l'élevage, souffrance animale

Sur la santé humaine et animale

Les plus : Production de protéines animales de qualité, forte teneur en oméga 3 dans production à l'herbe. L'élevage extensif favorise la robustesse des animaux et leur bien-être.

Les moins :

Consommation excessive de viande, acide gras saturés. Antibiorésistance ; zoonoses, non prise en compte du bien être animal en élevage intensif

Outre leur fonction de production pour nourrir les hommes, les secteurs agricole et agro-alimentaire sont au coeur de l'aménagement durable des territoires.

Ils contribuent notamment au développement économique, à la protection de l'environnement et à la vitalité du tissu social, en particulier dans les territoires ruraux les plus vulnérables.

Mais quelles sont les finalités de l'agriculture en France ?

https://agriculture.gouv.fr/agriculture-et-environnement-0

Dans la politique agricole, l'objectif de production de l'agriculture française est réaffirmé tout en intégrant les grands enjeux environnementaux ramenés à la question de la gestion durable :

Quels sont les enjeux environnementaux ? Vers une transition agro écologique

Des politiques transversales intégrant les enjeux environnementaux visent à accompagner les exploitations et les entreprises :

• favoriser les pratiques innovantes (Produisons autrement),

• réduire les impacts (Ecophyto)

• développer des stratégies (projet agro-écologique pour la France, plan régional de l'agriculture durable,...),

• des outils de valorisation (Agriculture Biologique et certification environnementale des exploitations agricoles, trophées de l'agriculture durable, accords collectifs, GIEE,...),

• des leviers financiers (PAC, MAEC).

Une vision plus multifonctionnelle de l'agriculture

Le cas des systèmes d'élevages herbagers

Le projet Semences locales

Le projet ELEVE -Eleveur, vétérinaire et environnement.

Développer des solutions de traitement antiparasitaires adaptées à chaque exploitation pour préserver le réseau trophique des prairies.

Gérer durablement les Tourbières

Développer des itinéraires de gestion adaptés pour transformer une contrainte en élément de fierté et préserver les milieux humides.

Bon Films et on se retrouve pour le débat

  Agriculture et environnement : de quoi parle t-on ? Un objet de débat sociétal Les conditions actuelles de la production alimentaire et son empreinte environnementale sont questionnées. Les choses ne sont ni toutes noires, ni toutes blanches lorsque l'on examine les impacts Sur l'environnement physique Les plus : Régulation des débits d'eau, filtration des eaux par les couverts végétaux, activité biologique du sol, stockage de carbone Services écosystémiques. Les moins : Pollution du milieu liée à forte utilisation d'intrants, phytosanitaires et antibiotiques ; émission de gaz à effet de serres ; déforestation ; Banalisation de la flore, perte de biodiversité, destruction d'habitats, conflit avec la faune sauvage, destruction des réseaux trophiques

➢

  des ressources naturelles (eau, sol, air, biodiversité) ➢ des espaces agricoles et des paysages, ➢ la lutte contre le changement climatique et l'adaptation à ces effets, ➢ la production d'énergie renouvelable et l'efficacité énergétique, la valorisation de la biomasse (méthanisation). ➢ la réduction des risques sanitaires Et à la prise en compte de nouveaux enjeux :

-

  Mondialisation du commerce (intrants et productions agricoles)-Évolution des attentes sociales : des produits sûrs et sains ; bien être animal ; limitation de l'impact environnemental (GES, nitrates, pesticides,…); etc.-Demande sociétale concernée par la production de biens immatériels : paysages, fonctionnalités des écosystèmes (fertilité des sols, biodiversité ) …qui poussent à reconsidérer les schémas classiques de production -Limitation des resources (énergie, phosphate, eau) De nouveaux défis et enjeux …  Vouées à la production fourragère, les prairies contribuent à l'identité territoriale et accueillent une importante diversité d'espèces végétales, animales et microorganismes En Europe, la majorité des surfaces en herbe sont intimement liées aux activités d'élevage, qui en exploitant régulièrement la biomasse produite bloquent la dynamique végétale vers la forêt. Ces surfaces représentent 30% de la surface du territoire national soit près de 10 millions d'ha en France métropolitaine. La prairie : un écosystème géré, élément d'un système d'élevage La majorité des surfaces toujours en herbe du Massif central sont des PRAIRIES NATURELLES composées d'une forte diversité d'adaptation des systèmes à des perturbations ou aléas (hypothèse assurance écologique) * protection des espèces ou des milieux rares (conservation biologique ou des habitats) Un élevage bien géré a des impacts positifs : la ressource en herbe en maintenant les milieux ouverts * ressources hydriques (filtration prairies), sol (fertilité), atmosphère (bilan GES) * pour le bien être des résidents et les usages récréatifs * pour limiter les risques (incendies, avalanches) => éviter l'enfrichement généralisé et reconquérir des espaces enfrichés. Quelques illustrations et témoignages Concevoir et mettre en oeuvre des actions concertées pour concilier activité agricole et préservation de la biodiversité et de l'environnement Reconstruire une cohérence et une identité qui concilie rentabilité économique des exploitations économiques (élevage), préservation des ressources naturelles et qualité des productions animales et du patrimoine culturel Prairies fleuries -concilier les usages Reconnaitre la multiplicité des services rendus par les prairies et concilier les usages dans les élevages herbagers. Maintenir et restaurer une flore prairiale native typique des terroirs par des pratiques d'auto régénération des prairies Expérimentation de récolte et de semis de semences de prairies naturelles locales.